To provide statistics about RecipeUpstream information based on
Milestones we need to store RecipeUpstreamHistory that contains when the
update script was executed.
rrs/admin.py: Add admin page for RecipeUpstreamHistory.
rrs/models.py: Add model for RecipeUpstreamHistory with helper functions
for get last and last by date range also add migration.
Signed-off-by: Aníbal Limón <anibal.limon@linux.intel.com>
Add Marius, Saul, and Anibal since they contributed to the RRS code that
is being integrated.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Add RRS-specific settings and bring in RRS. Based on work by
Aníbal Limón <anibal.limon@linux.intel.com>.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Add models for store Milestone, Maintainer and Recipe{Maintainer,
Upgrade, Upstream, Distro}, add initial data for Milestone and
Maintainer, initial migration and admin sites.
Add rrs/context_processors.py for return the site_name and application
for use in templates.
Signed-off-by: Aníbal Limón <anibal.limon@linux.intel.com>
In 3adddf6a25 I changed the style to allow
pre-formatted text to work in the layer description, but this broke
wrapping so that text could go behind the "Dependencies" box. Use the
"pre-wrap" style instead so that both pre-formatting and wrapping work
here.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
pip (strangely only in the python 2 version when I test it here) reports
that some of the versions in requirements.txt were incompatible:
django-nvd3 0.9.7 has requirement python-nvd3==0.14.2, but you'll have
python-nvd3 0.15.0 which is incompatible.
django-registration 2.4.1 has requirement confusable-homoglyphs~=3.0,
but you'll have confusable-homoglyphs 2.0.2 which is incompatible.
python-nvd3 0.14.2 has requirement python-slugify==1.1.4, but you'll
have python-slugify 1.2.5 which is incompatible.
I'm not particularly keen on downgrading these but it seems like we
don't have much choice. Luckily looking over the changelogs it doesn't
seem like that will cause us any problems though.
Thanks to Yi Zhao <yi.zhao@windriver.com> for pointing this out.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
In e902b67bcc I missed a couple of Context
usages in the layer publish view and the result was that it broke
publishing a layer (and apparently I didn't run a final test on that,
shame on me).
Thanks to Yi Zhao <yi.zhao@windriver.com> for pointing this out.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Since Django 1.8 is now out of support, and we've cleaned up the issues,
bump the Django requirement to 1.11 and update other dependencies at the
same time.
Fixes [YOCTO #12696] (requires preceding commits)
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
There were several issues with this code, including that it used
SortedDict which was removed in Django 1.9 and that it seemed not to
have been fully updated to accommodate changes in bitbake's recipe
parsing API. In the end I decided the simplest thing would be to move it
over to using oe.recipeutils.patch_recipe() which is actually a now much
more mature version of the code that originally started life here. With
that we can get the bulk change functionality working again and gain
some of the improvements in behaviour that we've developed in
oe.recipeutils.patch_recipe(), as well as avoiding effectively
duplicated code.
Implements [YOCTO #9730].
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
It seems that in df492b1277 the original
intention was to move setup_layer to the utils module, but that didn't
actually get done in the final patch - however the change was made here
to accommodate the move, meaning it's been broken since then. Revert
that so we're actually calling the function in the place it exists.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
With Django 1.10+, if you use get_template() to retrieve a template,
then you can't pass a context when calling .render() on it, you need to
pass a dict instead.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
We use django-reversion to track history of admin/maintainer changes to
the site, and part of our extension on top of django-reversion involves
annotating each "revision" with a description of the changes that were
made. In django-reversion 2.0.0+ the pre_revision_commit signal that we
were using to do this annotation is gone in favour of just using
Django's standard pre-save signal. This was a little challenging to
adapt to for our purposes but not impossible.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
I can't quite tell which Django version broke this, but in any case
based on what's in pagination.html it seems we ought to have been
using == already.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
resolve_variable isn't available with Django 1.10+, and it's not even
used here, so just drop it from the import line.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
The patterns() function is deprecated in Django 1.8 and gone in 1.10, so
we should switch over to the new list format.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Rename publish() to publish_view() and call it directly from the view
rather than using a string name (which is not possible in Django 1.10+).
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Django 1.8 introduced this as a soft requirement, it is a hard
requirement for later Django releases.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
It's useful if the script is executable so we don't have to specify
python on the command line. We also only need one shebang line (not sure
how we ended up with two) and it should point to python3.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
The utils.setup_django() costs a lot of time, but both update.py and
update_layer.py call it, so move layer validation from update_layer.py
to update.py to avoid calling update_layer.py when possible can save a
lot of time.
Now we don't have to call update_layer.py in the following cases:
* The branch doesn't exist
* The layer is already update to date on specified branch (when no
reload)
* The layer dir or conf/layer.layer doesn't exist
We can save up to 98% time in my testing:
$ update.py -b master --nofetch [--fullreload]
Before Now Reduced
No update: 276s 3.6s 98%
Partial update: 312s 87s 72%
Full reload: 1016s 980s 3%
Note:
* All of the testing are based on --nofetch
* "No update" means all layers on the branch is up-to-date, for
example, when we run it twice, there is no update in the second run,
so we only need about 3s now, which is the most common case when we
use cron to run it per half an hour.
* "Partial update" means part of the layers have been updated.
* "Full reload" means all of the layers have been updated.
Signed-off-by: Robert Yang <liezhi.yang@windriver.com>
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
This makes it easy to see which layers failed. For example:
ERROR: Failed layers on branch master: openembedded-core meta-python
Signed-off-by: Robert Yang <liezhi.yang@windriver.com>
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
We have an update.py running periodically in background, but we also
need to run it manually, for example, run it to update actual_branch,
the manually run usually failed because can't get lockfile, we have to
run it again and again. A timeout option helps a lot in such a case. Now
the following command can make sure we can run the command successfully:
$ update.py -b master -a actual_branch -t 2000
Signed-off-by: Robert Yang <liezhi.yang@windriver.com>
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
The previous commit broke the layer order, e.g.:
A -> B -> C -> D
The algorithm is checking the dependencies one by one, and until we find D, add
D to layerquery_sorted, and add it "collections", the one in "collections"
means it's dependencies are OK, then C, B and A will check against collections,
so that update_layer.py will update them one by one. The previous commit added
A/B/C/D to collections directly, so that when check against it, all the
dependencies are met, thus broke the layer sorting, and then there would be
failures if we pass layer A to update_layer.py before B, C and D (suppose they
are newly to database). This commit fix the problem.
BTW., why I use collections to record the one whose dependencies are matched,
but not directly use layerquery_sorted, it is because collections contains both
the ones to be added/updated and the ones in database, but layerquery_sorted
only contains the ones to be updated/added.
Signed-off-by: Robert Yang <liezhi.yang@windriver.com>
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Update the minor version of Django, and replace the list of dependencies
with a pointer to requirements.txt since there's not much point
maintaining that in two places (and the README wasn't complete anyway).
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Use latest tested versions (though we pin djangorestframework at 3.6.4
since that is the last version that supports Django 1.8), and add new
resulting dependencies.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
A lot of people enter line breaks in their layer descriptions hoping
that these will show up in the layer description, but of course since
they are being displayed as part of an HTML document, they don't by
default. Use a style in the description paragraph to ensure that they
do.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Add a page with basic statistics for the index - number of layers,
recipes, classes, machines and distros on an overall basis (distinct
names) and per branch, since I've been asked a few times for this kind
of information. It's currently only linked from the Tools menu for
logged-in users, but the URL will work for anyone.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
If the user hit Ctrl+C during the initial info gathering then it didn't
break out of the loop in update.py, so you had to hit Ctrl+C for as many
layers as were involved in the update. Look for exit code 254 from
update_layer.py and stop if it is returned since that indicates Ctrl+C
has been used.
Additionally, ensure we return exit code 254 and print a message from
the main update script when it is interrupted in this way.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
For reasons that are not immediately clear to me, if you don't specify a
name when adding the ViewSet it takes one from the model used in the
queryset in the ViewSet. The new layers view uses LayerBranch and so it
silently replaced the layerBranches URL in the router, even though the
URL itself was still present. Unfortunately that's broken Toaster.
Specify a name to restore the old URL.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
It was a bit awkward to query layers externally - you had to do multiple
queries and you couldn't get the YP Compatible version info at all. Add
an additional LayerBranch-based view that exposes the branch name,
layer fields, YP Compatible Version and active maintainer information
with just one call.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Django REST Framework now requires a field specification for every
ModelSerializer, so specify '__all__' to retain the current behaviour.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Layer maintainer and layer note information wasn't available through the
REST API since it wasn't needed for Toaster, but for other uses it is
useful.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
If you query on a boolean field you can use the string "False" to match
False in the database; however if you try the same with __isnull then
the query will match every record which is obviously undesirable. If
__isnull is being used, then convert the value to a boolean so that the
query works properly.
An example of this type of query:
http://127.0.0.1:8000/layerindex/api/layerBranches/?filter=yp_compatible_version__isnull:false
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
tinfoil is not needed in cases like the layer is already up-to-date or the
layer is invalid, so only init it when needed.
This can save about 1min when run "update.py -b <branch>" (124 layers).
Signed-off-by: Robert Yang <liezhi.yang@windriver.com>
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
This can save a lot of time, here is my testing data when PARALLEL_JOBS is 10,
this is the fetch time only, I hacked it to stop when the fetch is done to get
the data (124 layers):
$ update.py -b <branch>
Before: 2m30
Now: 16s
Signed-off-by: Robert Yang <liezhi.yang@windriver.com>
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Fixed:
$ ./update.py -l foo -b new_branch
INFO: Fetching remote repository foo
DEBUG: run cmd 'git fetch -p' in 'foo'
[snip]
DEBUG: run cmd 'git checkout origin/new_branch' in oe-core
ERROR: error: pathspec 'origin/new_branch' did not match any file(s) known to git.
The "new_branch" is newly created, it doesn't exist in local repo since it
isn't fetched, it only fetches the layer specified by -l, so only the foo layer
is fetched. This patch fixes problem.
Signed-off-by: Robert Yang <liezhi.yang@windriver.com>
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
The obsolete dependency is the one which is in database but not in
conf/layer.conf anymore. The old code had a problem for newly created
layerbranch, the new layerbranch has no dependencies, so no need remove. And it
had a side effect was that when need_remove was cleaned up, it would be set
again in the next for loop, thus might wrongly remove dependencies. This patch
can fix the problem.
Signed-off-by: Robert Yang <liezhi.yang@windriver.com>
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
If your classic recipe search returned a single result, then since
commit c8c25fb641 you got a standard
recipe page instead of the proper page with fields you can edit. To fix
it, just drop the redirection functionality from the classic recipe
search, since we don't really want it here.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
We were using the layerbranch id to search for the specified layer,
which is most likely to return either no results or results for the
wrong layer. We can also avoid specifying the id field at all here as
the filter() function can handle real objects.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Unsatisfied layer dependencies shouldn't error out of the script -
broken metadata isn't supposed to terminate the index update process.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
