On the recipe detail page we provide a link to the actual recipe file
for reference purposes. However, for recipes that include a common .inc
file, many of the definitions of variable values will be inside the
.inc, therefore if you just look at the recipe you won't see the full
picture. Of course you can just go up to the parent directory in the
repository web interface, but for convenience's sake add links to any
files that are included/required by the recipe that are adjacent to
the recipe itself. (We already have the data in the form of the
RecipeFileDependency records that are intended to ensure we know when
the recipe needs to be updated if one of the files it includes changes).
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Specifying the covering recipe in the comparison recipe detail page was
always a bit awkward - you could only type the name, if you wanted to
actually find a recipe or look up the currently selected one's details
then you had to open another browser tab/window. To fix this, replace
the form on the comparison recipe detail page with a side-by-side
display of the covering recipe's information, along with a button that
lets you search and then select the covering recipe and at the same time
enter comments or set any of the other cover fields.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Comparison updates might involve some custom fetch process, so provide a
mechanism to register these via settings.py on a per-branch basis. If
an update command is defined for a branch and the logged-in user has the
new "update_comparison_branch" permission, an "Update" button will show
up on the recipes page for the comparison branch for authenticated
users that will trigger the command in the background (as a celery job)
and then show a page that displays the status. The status isn't shown in
real-time since that requires quite a lot of plumbing, but the page at
least auto-refreshes.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Make it clearer what this field is for (it should be set for every
comparison layer, so that they don't show up in places they shouldn't).
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
When doing the reversed query it's often desirable to exclude recipes
by inherited class, for example those that inherit the packagegroup,
image and meta classes as they don't actually build anything and thus
aren't going to match up with anything in the other distribution.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Provide a lazy-loaded popup for selecting layers to include in the query
instead of having it as a simple drop-down, so you can select more than
one layer.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Enable "reversing" the comparison, so you can see which recipes on the
OE side match up (or don't) with the other distro. The filtering for
this is a bit awkward, since we don't have an actual foreign key for the
link, hence the hairiness of the code.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Add a flag that can be set and searched for to indicate that we need to
take care of importing a package or a patch applied by a package.
Ideally the comments would elaborate on what's needed (if it's not
obvious from the cover status).
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Packages with the "Not available" status were inexplicably excluded from
the graph, include them.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Allow searching on:
* Any "available" status (i.e. other than "Unknown" or "Not available")
* Whether the package has patches or not
* What the covering layer is (assuming there is one).
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
It's a bit crude, but accept '' or "" as meaning search for entries
with an empty category.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Add a script to import "recipe" information from other distro metadata,
based on the import_classic.py script. At the moment, this assumes a
directory where each subdirectory is a package directory containing a
spec file; this would be suitable for distributions such as Fedora
assuming you have all of the package repos checked out locally. Since
you can add additional information to these records (the cover fields
pointing to matching recipes), existing records are updated rather than
deleting everything and re-importing, and we only mark records as
deleted rather than actually deleting them (in case you accidentally
point the script at an empty directory or similar).
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Turn the existing OE-Classic support into something a bit more
generic so we can import data from other distributions and compare it to
what we have in layers.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
If you're running a testing / internal instance then you really don't
want to be emailing maintainers on publish, so provide a setting you can
use to disable that.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Add the ability to export the recipe listing for a layer to a CSV file
for importing into external tools. At the moment we include name,
version and license, but there is a parameter that lets you specify the
fields to include in the URL if desired.
Implements [YOCTO #12722].
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
We're about to replace this with a proper CSV export function, so we
don't need this dead code hanging around anymore.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
If the RRS is enabled, then add a way to get from the layer detail page
to any maintenance plans in which the layer is included.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Sometimes we get back UTF-8 characters (particularly when picking up
names from git commits), and the ascii codec will error out if that
happens, so switch over to utf-8.
We can as a result remove the decode() calls from the bulk change view.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
In e902b67bcc I missed a couple of Context
usages in the layer publish view and the result was that it broke
publishing a layer (and apparently I didn't run a final test on that,
shame on me).
Thanks to Yi Zhao <yi.zhao@windriver.com> for pointing this out.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
With Django 1.10+, if you use get_template() to retrieve a template,
then you can't pass a context when calling .render() on it, you need to
pass a dict instead.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
We use django-reversion to track history of admin/maintainer changes to
the site, and part of our extension on top of django-reversion involves
annotating each "revision" with a description of the changes that were
made. In django-reversion 2.0.0+ the pre_revision_commit signal that we
were using to do this annotation is gone in favour of just using
Django's standard pre-save signal. This was a little challenging to
adapt to for our purposes but not impossible.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Rename publish() to publish_view() and call it directly from the view
rather than using a string name (which is not possible in Django 1.10+).
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Add a page with basic statistics for the index - number of layers,
recipes, classes, machines and distros on an overall basis (distinct
names) and per branch, since I've been asked a few times for this kind
of information. It's currently only linked from the Tools menu for
logged-in users, but the URL will work for anyone.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
If your classic recipe search returned a single result, then since
commit c8c25fb641 you got a standard
recipe page instead of the proper page with fields you can edit. To fix
it, just drop the redirection functionality from the classic recipe
search, since we don't really want it here.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
We were using the layerbranch id to search for the specified layer,
which is most likely to return either no results or results for the
wrong layer. We can also avoid specifying the id field at all here as
the filter() function can handle real objects.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
We need to set a default for this variable or you get an
UnboundLocalError if no non-root superuser/staff accounts are set up in
the database.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Added a model for the PACKAGECONFIG variable, which has a one to
many relationship with the Recipe model.
Added models for static build dependencies and dynamic build
dependencies, both of which have a many to many relationship with
the Recipe model.
These objects are created in update_layer.py and are displayed on the
Recipe detail page.
Added a depends search option for recipes, allowing users to search for
recipes that depend on any particular recipe. Use "depends:recipename"
in the recipe search to activate this.
Fixes [YOCTO #12129]
Fixes [YOCTO #11415]
Signed-off-by: Amanda Brindle <amanda.r.brindle@intel.com>
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Use JavaScript to check if the search box for recipe search is
empty before querying the database. This will prevent the "502
Bad Gateway" error that occurs when the query takes too long due
to the large list of recipes. Since there are so many recipes
spread across the layers in the OE index, there's no point in
allowing a user to search without a keyword in order to browse
the list; it simply isn't digestible as a whole.
For the Machines, Classes, and Distros pages, the search behaviour is
unaffected, however to make it more obvious that you can browse the list
add an explicit "browse" button.
Fixes [YOCTO #11930]
Signed-off-by: Amanda Brindle <amanda.r.brindle@intel.com>
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Add another tab to search for classes.
Fixes [YOCTO #11207]
Signed-off by: Amanda Brindle <amanda.r.brindle@intel.com>
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
When publishing a layer, send an email notification to all of that
layer's maintainers. Include information on how to edit the layer, plus
contact details for the first active staff user if there are any
problems (we could make this configurable in future, but for now this is
sufficient).
Fixes [YOCTO #11208]
Signed-off-by: Amanda Brindle <amanda.r.brindle@intel.com>
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
This patch adds asynchronous task execution using a Celery backend
and RabbitMQ task queue, so that the layer submission process to
proceed even in the event that sending the notification email fails,
and establishing an asynchronous execution mechanism that we can use
in the future e.g. for triggering parse operations from the web UI.
This pertains to bug 11197:
https://bugzilla.yoctoproject.org/show_bug.cgi?id=11197
It updates the README to reflect the installation and configuration
of a basic RabbitMQ setup, adds a 'tasks.py' file to contain task
definitions, updates the 'edit_layer_view' function to send
emails to administrators about new and updated layers asynchronously,
modifies the 'settings.py' to include a default configuration
for a RabbitMQ connection, and updates the Dockerfile to start a Celery
worker alongside the Gunicorn daemon.
Fixes [YOCTO #11197].
Signed-off-by: Diana Thayer <garbados@gmail.com>
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Before, if a user edited a layer's name, they would be redirected
to a url utilizing the old name and then receive a 404 Page not
found error. Now, the url utilizes the new name.
Fixes [YOCTO #11932]
Signed-off-by: Amanda Brindle <amanda.r.brindle@intel.com>
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
At the moment the URL that is presented in the review email will have
http:// or https:// prefix depending on what the user who submitted the
layer was using, but that's irrelevant - we actually want https:// if
the server is capable of it since the reviewer may be redirected to log
in (and Django's login_required decorator will always redirect to the
login page if http is being used as far as I observed, which is a bit
annoying if you are already logged in.) Add a setting which if enabled
will substitute https:// as the prefix.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
This change supports querying recipes that belong to a specific layer
by using the prefix "layer:" + the desired layer name, for example:
"layer: openembedded-core" and this string can be used by itself or
combined with other supported options.
A descriptive error message is displayed when the query string has an
unexpected formatting or a non-valid layer name is searched.
[YOCTO #6618]
Signed-off-by: Jose Lamego <jose.a.lamego@linux.intel.com>
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Add a status for a layer indicating it should not be updated. I don't
expect this to be widely used (and is only settable from the admin
interface) but would be useful if you have a legacy sub-layer that you
want to prevent from being visible on certain branches - it will prevent
the update script from doing anything with the layer and thus avoid
branch records from being auto-created on branches where you've deleted
it.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
When the recipe search returns a single result, redirect to it
instead of showing the full result list view.
Part of the fix for [YOCTO #6618].
Signed-off-by: Alex Franco <alejandro.franco@linux.intel.com>
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
On the main layer index page we want openembedded-core to appear before
meta-oe - naturally openembedded-core is the layer everyone starts from.
We already have an "index_preference" column that we use for search
results, so just sort by that here before sorting by name.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
It's a little crude and certainly not optimal performance-wise, but we
can support querying for recipes that inherit a particular class without
too much trouble. This allows you to add "inherits:cmake" to the query
and have it return only recipes that inherit the cmake class. You can
use more than one inherits: item to filter down to recipes that inherit
all of the specified classes.
Note: this does not otherwise change the behaviour of specifying
multiple words - all of the words other than those that start with
"inherits:" are treated as part of a single phrase that will be searched
for - not separate keywords.
Fixes [YOCTO #9879].
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
At the moment it's a bit difficult to get update logs out of the
environment in which the update script is being run. In order to make
the logs more accessible, create a LayerUpdate model to record the
output of update_layer.py separately for each layerbranch and tie the
created LayerUpdates together with a single Update model per session.
We provide two ways to look at this - a Tools->Updates page for
logged-in users, and there's also an "Updates" tab on each layer that is
accessible to anyone; which one is useful depends on whether you are
looking at the index as a whole or an individual layer.
Update records older than 30 days are deleted automatically by default.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Improves slightly on 3155206e54 by doing
an exact match on name and showing that first - now when you search for
"git" you really do get the git recipe first in the list.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Usage of itertools.chain() that was introduced in
3155206e54 in order to prioritise matches
in the recipe name resulted in recipes showing up twice in the results
if they matched in both the name and the recipe name. Use a custom
chaining function that skips duplicate results in order to fix this.
Fixes [YOCTO #10177].
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Add the distros to the index. This looks a lot like the machines
and allows users to search for a particular distro.
Signed-off-by: Liam R. Howlett <Liam.Howlett@WindRiver.com>
Added associated migration.
Signed-off-by: Mark Hatle <mark.hatle@windriver.com>
With Python 3 we need to take care of decoding the output so we can
treat it as a string. At the same time, it's more useful to see the
output string rather than the exception if there is some output.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
We need to be able to support Python 3 so that we can parse master of
OE-Core with bitbake (which now requires it). This now means the
interface itself and the update script require Python 3.4+.
Part of the implementation for [YOCTO #9704].
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
We have to upgrade django-reversion to 1.8 due to upgrading Django, but
unfortunately in that same version the author has removed the type field
on Version model, without a particularly good explanation as to why. This
is really annoying as we were using it to provide a reasonable audit
including for deletes. I suspect we'll need to move away from
django-reversion and do our own thing in future, but for now at least
allow the layer index to keep working.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Fix a greedy regex in the recipe detail view resulting in the git recipe
listing all bbappends named *_git.bbappend as its bbappends (quite a few
in the public instance).
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
