With Django 1.10+, if you use get_template() to retrieve a template,
then you can't pass a context when calling .render() on it, you need to
pass a dict instead.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
We use django-reversion to track history of admin/maintainer changes to
the site, and part of our extension on top of django-reversion involves
annotating each "revision" with a description of the changes that were
made. In django-reversion 2.0.0+ the pre_revision_commit signal that we
were using to do this annotation is gone in favour of just using
Django's standard pre-save signal. This was a little challenging to
adapt to for our purposes but not impossible.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
I can't quite tell which Django version broke this, but in any case
based on what's in pagination.html it seems we ought to have been
using == already.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
resolve_variable isn't available with Django 1.10+, and it's not even
used here, so just drop it from the import line.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
The patterns() function is deprecated in Django 1.8 and gone in 1.10, so
we should switch over to the new list format.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Rename publish() to publish_view() and call it directly from the view
rather than using a string name (which is not possible in Django 1.10+).
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Django 1.8 introduced this as a soft requirement, it is a hard
requirement for later Django releases.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
It's useful if the script is executable so we don't have to specify
python on the command line. We also only need one shebang line (not sure
how we ended up with two) and it should point to python3.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
The utils.setup_django() costs a lot of time, but both update.py and
update_layer.py call it, so move layer validation from update_layer.py
to update.py to avoid calling update_layer.py when possible can save a
lot of time.
Now we don't have to call update_layer.py in the following cases:
* The branch doesn't exist
* The layer is already update to date on specified branch (when no
reload)
* The layer dir or conf/layer.layer doesn't exist
We can save up to 98% time in my testing:
$ update.py -b master --nofetch [--fullreload]
Before Now Reduced
No update: 276s 3.6s 98%
Partial update: 312s 87s 72%
Full reload: 1016s 980s 3%
Note:
* All of the testing are based on --nofetch
* "No update" means all layers on the branch is up-to-date, for
example, when we run it twice, there is no update in the second run,
so we only need about 3s now, which is the most common case when we
use cron to run it per half an hour.
* "Partial update" means part of the layers have been updated.
* "Full reload" means all of the layers have been updated.
Signed-off-by: Robert Yang <liezhi.yang@windriver.com>
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
This makes it easy to see which layers failed. For example:
ERROR: Failed layers on branch master: openembedded-core meta-python
Signed-off-by: Robert Yang <liezhi.yang@windriver.com>
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
We have an update.py running periodically in background, but we also
need to run it manually, for example, run it to update actual_branch,
the manually run usually failed because can't get lockfile, we have to
run it again and again. A timeout option helps a lot in such a case. Now
the following command can make sure we can run the command successfully:
$ update.py -b master -a actual_branch -t 2000
Signed-off-by: Robert Yang <liezhi.yang@windriver.com>
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
The previous commit broke the layer order, e.g.:
A -> B -> C -> D
The algorithm is checking the dependencies one by one, and until we find D, add
D to layerquery_sorted, and add it "collections", the one in "collections"
means it's dependencies are OK, then C, B and A will check against collections,
so that update_layer.py will update them one by one. The previous commit added
A/B/C/D to collections directly, so that when check against it, all the
dependencies are met, thus broke the layer sorting, and then there would be
failures if we pass layer A to update_layer.py before B, C and D (suppose they
are newly to database). This commit fix the problem.
BTW., why I use collections to record the one whose dependencies are matched,
but not directly use layerquery_sorted, it is because collections contains both
the ones to be added/updated and the ones in database, but layerquery_sorted
only contains the ones to be updated/added.
Signed-off-by: Robert Yang <liezhi.yang@windriver.com>
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Update the minor version of Django, and replace the list of dependencies
with a pointer to requirements.txt since there's not much point
maintaining that in two places (and the README wasn't complete anyway).
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Use latest tested versions (though we pin djangorestframework at 3.6.4
since that is the last version that supports Django 1.8), and add new
resulting dependencies.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
A lot of people enter line breaks in their layer descriptions hoping
that these will show up in the layer description, but of course since
they are being displayed as part of an HTML document, they don't by
default. Use a style in the description paragraph to ensure that they
do.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Add a page with basic statistics for the index - number of layers,
recipes, classes, machines and distros on an overall basis (distinct
names) and per branch, since I've been asked a few times for this kind
of information. It's currently only linked from the Tools menu for
logged-in users, but the URL will work for anyone.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
If the user hit Ctrl+C during the initial info gathering then it didn't
break out of the loop in update.py, so you had to hit Ctrl+C for as many
layers as were involved in the update. Look for exit code 254 from
update_layer.py and stop if it is returned since that indicates Ctrl+C
has been used.
Additionally, ensure we return exit code 254 and print a message from
the main update script when it is interrupted in this way.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
For reasons that are not immediately clear to me, if you don't specify a
name when adding the ViewSet it takes one from the model used in the
queryset in the ViewSet. The new layers view uses LayerBranch and so it
silently replaced the layerBranches URL in the router, even though the
URL itself was still present. Unfortunately that's broken Toaster.
Specify a name to restore the old URL.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
It was a bit awkward to query layers externally - you had to do multiple
queries and you couldn't get the YP Compatible version info at all. Add
an additional LayerBranch-based view that exposes the branch name,
layer fields, YP Compatible Version and active maintainer information
with just one call.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Django REST Framework now requires a field specification for every
ModelSerializer, so specify '__all__' to retain the current behaviour.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Layer maintainer and layer note information wasn't available through the
REST API since it wasn't needed for Toaster, but for other uses it is
useful.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
If you query on a boolean field you can use the string "False" to match
False in the database; however if you try the same with __isnull then
the query will match every record which is obviously undesirable. If
__isnull is being used, then convert the value to a boolean so that the
query works properly.
An example of this type of query:
http://127.0.0.1:8000/layerindex/api/layerBranches/?filter=yp_compatible_version__isnull:false
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
tinfoil is not needed in cases like the layer is already up-to-date or the
layer is invalid, so only init it when needed.
This can save about 1min when run "update.py -b <branch>" (124 layers).
Signed-off-by: Robert Yang <liezhi.yang@windriver.com>
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
This can save a lot of time, here is my testing data when PARALLEL_JOBS is 10,
this is the fetch time only, I hacked it to stop when the fetch is done to get
the data (124 layers):
$ update.py -b <branch>
Before: 2m30
Now: 16s
Signed-off-by: Robert Yang <liezhi.yang@windriver.com>
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Fixed:
$ ./update.py -l foo -b new_branch
INFO: Fetching remote repository foo
DEBUG: run cmd 'git fetch -p' in 'foo'
[snip]
DEBUG: run cmd 'git checkout origin/new_branch' in oe-core
ERROR: error: pathspec 'origin/new_branch' did not match any file(s) known to git.
The "new_branch" is newly created, it doesn't exist in local repo since it
isn't fetched, it only fetches the layer specified by -l, so only the foo layer
is fetched. This patch fixes problem.
Signed-off-by: Robert Yang <liezhi.yang@windriver.com>
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
The obsolete dependency is the one which is in database but not in
conf/layer.conf anymore. The old code had a problem for newly created
layerbranch, the new layerbranch has no dependencies, so no need remove. And it
had a side effect was that when need_remove was cleaned up, it would be set
again in the next for loop, thus might wrongly remove dependencies. This patch
can fix the problem.
Signed-off-by: Robert Yang <liezhi.yang@windriver.com>
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
If your classic recipe search returned a single result, then since
commit c8c25fb641 you got a standard
recipe page instead of the proper page with fields you can edit. To fix
it, just drop the redirection functionality from the classic recipe
search, since we don't really want it here.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
We were using the layerbranch id to search for the specified layer,
which is most likely to return either no results or results for the
wrong layer. We can also avoid specifying the id field at all here as
the filter() function can handle real objects.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Unsatisfied layer dependencies shouldn't error out of the script -
broken metadata isn't supposed to terminate the index update process.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
We need to set a default for this variable or you get an
UnboundLocalError if no non-root superuser/staff accounts are set up in
the database.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
If you use a traditional database engine (such as MySQL/MariaDB) then
maximum character field lengths are enforced, however depending on the
configuration this may result in an exception rather than a warning and
truncation and will also break the per-layer transaction. To avoid that
ugliness, add a signal handler to do it internally, which as a bonus
lets us know if field lenghts are too short for data when using database
engines that don't enforce lengths (e.g. SQLite).
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
If a database error occurs when we save a recipe (e.g. because a
database-level constraint is voilated) it will mess up the transaction.
Unfortunately that means we need to break out of updating the entire
layer rather than catching the error, because if we do catch it we just
get errors on every update after the initial error; failing early and
giving up on the transaction is a little better in terms of not filling
up the update logs with further useless errors.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
* Allow the model's __str__() function to control what is shown for each
dependency item as per other model admins
* Enable searching for PACKAGECONFIGs by recipe name
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Use the more typical handling of the return of get_or_create() and check
if the item was created before saving.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Fix duplicate PackageConfig records being created each time a recipe
is being updated - we need to delete the old ones first.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Since we added the "packageconfig" to Recipe we need to exclude it from
the admin or we see an error when we try to open ClassicRecipe in the
admin site.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Added a model for the PACKAGECONFIG variable, which has a one to
many relationship with the Recipe model.
Added models for static build dependencies and dynamic build
dependencies, both of which have a many to many relationship with
the Recipe model.
These objects are created in update_layer.py and are displayed on the
Recipe detail page.
Added a depends search option for recipes, allowing users to search for
recipes that depend on any particular recipe. Use "depends:recipename"
in the recipe search to activate this.
Fixes [YOCTO #12129]
Fixes [YOCTO #11415]
Signed-off-by: Amanda Brindle <amanda.r.brindle@intel.com>
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Use JavaScript to check if the search box for recipe search is
empty before querying the database. This will prevent the "502
Bad Gateway" error that occurs when the query takes too long due
to the large list of recipes. Since there are so many recipes
spread across the layers in the OE index, there's no point in
allowing a user to search without a keyword in order to browse
the list; it simply isn't digestible as a whole.
For the Machines, Classes, and Distros pages, the search behaviour is
unaffected, however to make it more obvious that you can browse the list
add an explicit "browse" button.
Fixes [YOCTO #11930]
Signed-off-by: Amanda Brindle <amanda.r.brindle@intel.com>
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Add another tab to search for classes.
Fixes [YOCTO #11207]
Signed-off by: Amanda Brindle <amanda.r.brindle@intel.com>
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Don't assume that every model will have a "search_allowed_fields"
attribute - if it doesn't, default to all CharFields in the model. This
fixes searching via the REST API.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
When publishing a layer, send an email notification to all of that
layer's maintainers. Include information on how to edit the layer, plus
contact details for the first active staff user if there are any
problems (we could make this configurable in future, but for now this is
sufficient).
Fixes [YOCTO #11208]
Signed-off-by: Amanda Brindle <amanda.r.brindle@intel.com>
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
For newly added layers it may not be immediately obvious that you have
to wait for it to be indexed (since the update script is usually run on
a scheduled basis). If the layer branch hasn't been indexed, add a note
mentioning this.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
