Improve CSS in order to handle column width better also
use styles to display upstream status column.
Signed-off-by: Aníbal Limón <anibal.limon@linux.intel.com>
Recipe detail page gives information about Recipe like summary,
section, license, file, etc. also display's upgrade history.
rrs/models.py: Milestone add get_by_date and rewrite get_current
for use get_by_date and RecipeDistro add get_distros_by_recipe.
rrs/urls.py: Add url for recipe_detail with pk.
rrs/views.py: Add RecipeUpgradeDetail view.
templates/rrs/recipedetail.html: Add recipedetail template.
templates/rrs/recipes.html: Add link to Recipe detail by row.
Signed-off-by: Aníbal Limón <anibal.limon@linux.intel.com>
Add url validation for Milestone and Upstream status and maintainer name
params if isn't valid raise HTTP 404.
Signed-off-by: Aníbal Limón <anibal.limon@linux.intel.com>
templates/rrs/base_toplevel.html: Add support for display statistics by
Milestone.
templates/rrs/recipes.html: Add initial page that display Recipe
status by Milestone also details of every recipe.
rrs/views.py: Add RecipeLitView for support recipes page.
rrs/models.py: Add helper functions.
rrs/static/*: Add css and js resources.
Signed-off-by: Aníbal Limón <anibal.limon@linux.intel.com>
To provide statistics about RecipeUpstream information based on
Milestones we need to store RecipeUpstreamHistory that contains when the
update script was executed.
rrs/admin.py: Add admin page for RecipeUpstreamHistory.
rrs/models.py: Add model for RecipeUpstreamHistory with helper functions
for get last and last by date range also add migration.
Signed-off-by: Aníbal Limón <anibal.limon@linux.intel.com>
Add Marius, Saul, and Anibal since they contributed to the RRS code that
is being integrated.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Add RRS-specific settings and bring in RRS. Based on work by
Aníbal Limón <anibal.limon@linux.intel.com>.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Add models for store Milestone, Maintainer and Recipe{Maintainer,
Upgrade, Upstream, Distro}, add initial data for Milestone and
Maintainer, initial migration and admin sites.
Add rrs/context_processors.py for return the site_name and application
for use in templates.
Signed-off-by: Aníbal Limón <anibal.limon@linux.intel.com>
In 3adddf6a25 I changed the style to allow
pre-formatted text to work in the layer description, but this broke
wrapping so that text could go behind the "Dependencies" box. Use the
"pre-wrap" style instead so that both pre-formatting and wrapping work
here.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
pip (strangely only in the python 2 version when I test it here) reports
that some of the versions in requirements.txt were incompatible:
django-nvd3 0.9.7 has requirement python-nvd3==0.14.2, but you'll have
python-nvd3 0.15.0 which is incompatible.
django-registration 2.4.1 has requirement confusable-homoglyphs~=3.0,
but you'll have confusable-homoglyphs 2.0.2 which is incompatible.
python-nvd3 0.14.2 has requirement python-slugify==1.1.4, but you'll
have python-slugify 1.2.5 which is incompatible.
I'm not particularly keen on downgrading these but it seems like we
don't have much choice. Luckily looking over the changelogs it doesn't
seem like that will cause us any problems though.
Thanks to Yi Zhao <yi.zhao@windriver.com> for pointing this out.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
In e902b67bcc I missed a couple of Context
usages in the layer publish view and the result was that it broke
publishing a layer (and apparently I didn't run a final test on that,
shame on me).
Thanks to Yi Zhao <yi.zhao@windriver.com> for pointing this out.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Since Django 1.8 is now out of support, and we've cleaned up the issues,
bump the Django requirement to 1.11 and update other dependencies at the
same time.
Fixes [YOCTO #12696] (requires preceding commits)
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
There were several issues with this code, including that it used
SortedDict which was removed in Django 1.9 and that it seemed not to
have been fully updated to accommodate changes in bitbake's recipe
parsing API. In the end I decided the simplest thing would be to move it
over to using oe.recipeutils.patch_recipe() which is actually a now much
more mature version of the code that originally started life here. With
that we can get the bulk change functionality working again and gain
some of the improvements in behaviour that we've developed in
oe.recipeutils.patch_recipe(), as well as avoiding effectively
duplicated code.
Implements [YOCTO #9730].
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
It seems that in df492b1277 the original
intention was to move setup_layer to the utils module, but that didn't
actually get done in the final patch - however the change was made here
to accommodate the move, meaning it's been broken since then. Revert
that so we're actually calling the function in the place it exists.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
With Django 1.10+, if you use get_template() to retrieve a template,
then you can't pass a context when calling .render() on it, you need to
pass a dict instead.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
We use django-reversion to track history of admin/maintainer changes to
the site, and part of our extension on top of django-reversion involves
annotating each "revision" with a description of the changes that were
made. In django-reversion 2.0.0+ the pre_revision_commit signal that we
were using to do this annotation is gone in favour of just using
Django's standard pre-save signal. This was a little challenging to
adapt to for our purposes but not impossible.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
I can't quite tell which Django version broke this, but in any case
based on what's in pagination.html it seems we ought to have been
using == already.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
resolve_variable isn't available with Django 1.10+, and it's not even
used here, so just drop it from the import line.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
The patterns() function is deprecated in Django 1.8 and gone in 1.10, so
we should switch over to the new list format.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Rename publish() to publish_view() and call it directly from the view
rather than using a string name (which is not possible in Django 1.10+).
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Django 1.8 introduced this as a soft requirement, it is a hard
requirement for later Django releases.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
It's useful if the script is executable so we don't have to specify
python on the command line. We also only need one shebang line (not sure
how we ended up with two) and it should point to python3.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
The utils.setup_django() costs a lot of time, but both update.py and
update_layer.py call it, so move layer validation from update_layer.py
to update.py to avoid calling update_layer.py when possible can save a
lot of time.
Now we don't have to call update_layer.py in the following cases:
* The branch doesn't exist
* The layer is already update to date on specified branch (when no
reload)
* The layer dir or conf/layer.layer doesn't exist
We can save up to 98% time in my testing:
$ update.py -b master --nofetch [--fullreload]
Before Now Reduced
No update: 276s 3.6s 98%
Partial update: 312s 87s 72%
Full reload: 1016s 980s 3%
Note:
* All of the testing are based on --nofetch
* "No update" means all layers on the branch is up-to-date, for
example, when we run it twice, there is no update in the second run,
so we only need about 3s now, which is the most common case when we
use cron to run it per half an hour.
* "Partial update" means part of the layers have been updated.
* "Full reload" means all of the layers have been updated.
Signed-off-by: Robert Yang <liezhi.yang@windriver.com>
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
This makes it easy to see which layers failed. For example:
ERROR: Failed layers on branch master: openembedded-core meta-python
Signed-off-by: Robert Yang <liezhi.yang@windriver.com>
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
We have an update.py running periodically in background, but we also
need to run it manually, for example, run it to update actual_branch,
the manually run usually failed because can't get lockfile, we have to
run it again and again. A timeout option helps a lot in such a case. Now
the following command can make sure we can run the command successfully:
$ update.py -b master -a actual_branch -t 2000
Signed-off-by: Robert Yang <liezhi.yang@windriver.com>
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
The previous commit broke the layer order, e.g.:
A -> B -> C -> D
The algorithm is checking the dependencies one by one, and until we find D, add
D to layerquery_sorted, and add it "collections", the one in "collections"
means it's dependencies are OK, then C, B and A will check against collections,
so that update_layer.py will update them one by one. The previous commit added
A/B/C/D to collections directly, so that when check against it, all the
dependencies are met, thus broke the layer sorting, and then there would be
failures if we pass layer A to update_layer.py before B, C and D (suppose they
are newly to database). This commit fix the problem.
BTW., why I use collections to record the one whose dependencies are matched,
but not directly use layerquery_sorted, it is because collections contains both
the ones to be added/updated and the ones in database, but layerquery_sorted
only contains the ones to be updated/added.
Signed-off-by: Robert Yang <liezhi.yang@windriver.com>
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Update the minor version of Django, and replace the list of dependencies
with a pointer to requirements.txt since there's not much point
maintaining that in two places (and the README wasn't complete anyway).
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Use latest tested versions (though we pin djangorestframework at 3.6.4
since that is the last version that supports Django 1.8), and add new
resulting dependencies.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
A lot of people enter line breaks in their layer descriptions hoping
that these will show up in the layer description, but of course since
they are being displayed as part of an HTML document, they don't by
default. Use a style in the description paragraph to ensure that they
do.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Add a page with basic statistics for the index - number of layers,
recipes, classes, machines and distros on an overall basis (distinct
names) and per branch, since I've been asked a few times for this kind
of information. It's currently only linked from the Tools menu for
logged-in users, but the URL will work for anyone.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
If the user hit Ctrl+C during the initial info gathering then it didn't
break out of the loop in update.py, so you had to hit Ctrl+C for as many
layers as were involved in the update. Look for exit code 254 from
update_layer.py and stop if it is returned since that indicates Ctrl+C
has been used.
Additionally, ensure we return exit code 254 and print a message from
the main update script when it is interrupted in this way.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
For reasons that are not immediately clear to me, if you don't specify a
name when adding the ViewSet it takes one from the model used in the
queryset in the ViewSet. The new layers view uses LayerBranch and so it
silently replaced the layerBranches URL in the router, even though the
URL itself was still present. Unfortunately that's broken Toaster.
Specify a name to restore the old URL.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
It was a bit awkward to query layers externally - you had to do multiple
queries and you couldn't get the YP Compatible version info at all. Add
an additional LayerBranch-based view that exposes the branch name,
layer fields, YP Compatible Version and active maintainer information
with just one call.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Django REST Framework now requires a field specification for every
ModelSerializer, so specify '__all__' to retain the current behaviour.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Layer maintainer and layer note information wasn't available through the
REST API since it wasn't needed for Toaster, but for other uses it is
useful.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
If you query on a boolean field you can use the string "False" to match
False in the database; however if you try the same with __isnull then
the query will match every record which is obviously undesirable. If
__isnull is being used, then convert the value to a boolean so that the
query works properly.
An example of this type of query:
http://127.0.0.1:8000/layerindex/api/layerBranches/?filter=yp_compatible_version__isnull:false
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
tinfoil is not needed in cases like the layer is already up-to-date or the
layer is invalid, so only init it when needed.
This can save about 1min when run "update.py -b <branch>" (124 layers).
Signed-off-by: Robert Yang <liezhi.yang@windriver.com>
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
This can save a lot of time, here is my testing data when PARALLEL_JOBS is 10,
this is the fetch time only, I hacked it to stop when the fetch is done to get
the data (124 layers):
$ update.py -b <branch>
Before: 2m30
Now: 16s
Signed-off-by: Robert Yang <liezhi.yang@windriver.com>
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
