Commit Graph

547 Commits

Author SHA1 Message Date
Paul Eggleton
8370c10306 rrs/tools: avoid unnecessary checkouts
We don't need to create branches here, and we don't need to actually
check anything out unless we're going to parse, so we can save a bit of
time by not doing so.

Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
2018-05-04 23:57:52 +12:00
Paul Eggleton
6f34b7fce7 rrs: show full name in admin
It's neater to show "Recipe Reporting System" instead of "Rrs".

Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
2018-05-04 23:57:52 +12:00
Paul Eggleton
24960bb7be rrs: add Release link to MaintenancePlan
If any Releases exist then we create a default MaintenancePlan and then
link all Releases to it - this needs to be done in multiple upgrade
steps since the field needs to be non-null.

Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
2018-05-04 23:57:52 +12:00
Paul Eggleton
1f49943aee rrs: add maintenance plans
The MaintenancePlan will provide some context for the RRS records so we
can effectively enable RRS functionality only on certain layers where we
want it. You can also consider the maintenance of multiple layers
together under a single plan if desired.

Here we add just the MaintenancePlan and the corresponding migration;
a non-null link from the Release requires a separate migration and thus
that will be done in a separate commit.

Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
2018-05-04 23:57:52 +12:00
Paul Eggleton
8a9a846d1e rrs: add initial migration
This is of course a new Django migration rather than the previous South
one. This will have to be applied using --fake-initial on a database
that already has the RRS tables.

Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
2018-05-04 23:57:52 +12:00
Paul Eggleton
cecb5924a8 rrs: drop old migrations
These south migrations aren't useful.

Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
2018-05-04 23:57:52 +12:00
Paul Eggleton
fd786875c3 rrs/tools: add dry-run option to each script
Add the ability to run the scripts without writing changes back to the
database, for debugging purposes.

Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
2018-05-04 23:57:52 +12:00
Paul Eggleton
6ca3d6649e Use this layer index's URL for layer branch
If we're integrating the layer index and the RRS, no need to jump off to
the OE index if for example this is an internal index.

Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
2018-05-04 23:57:52 +12:00
Paul Eggleton
e50c2e6841 Rename RRS URLs to have rrs_ prefix
Make these distinct from the layerindex ones.

Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
2018-05-04 23:57:52 +12:00
Paul Eggleton
bb20ca4fb0 Drop "load url from future"
This is no longer needed with current Django versions.

Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
2018-05-04 23:57:52 +12:00
Paul Eggleton
10a639a6b7 rrs/tools/rrs_unique_recipes: drop
We can't just delete arbitrary recipes that do exist in the layer if
we're in a general layer index database. We'll need to handle this in a
different way, or just live with the fact that there will be duplicate
entries.

Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
2018-05-04 23:57:52 +12:00
Paul Eggleton
f58754c998 rrs/models: move Raw class to views.py
This is something that really belongs in with the views, so move it
there.

Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
2018-05-04 23:57:52 +12:00
Aníbal Limón
510dec2ee8 rrs/tools/rrs_{distros,upstream_history}: Add path for poky meta lib
Due to bitbake client/server changes now the meta path isn't included
to sys.path when load tinfoil.

Signed-off-by: Aníbal Limón <anibal.limon@linux.intel.com>
2018-05-04 23:57:52 +12:00
Aníbal Limón
4c2e9ec1f9 rrs/tools/rrs_upstream_history.py: Change regex variables to tuple
This are causing a failure when access the remote datastore.

File "rrs/rrs/tools/rrs_upstream_history.py", line 169, in <module>
    set_regexes(recipe_data)
  File "rrs/rrs/tools/rrs_upstream_history.py", line 53, in set_regexes
    if any(d.getVar(var, True) for var in variables):
  File "rrs/rrs/tools/rrs_upstream_history.py", line 53, in <genexpr>
    if any(d.getVar(var, True) for var in variables):
  File "/srv/rrs/layers/bitbake/lib/bb/data_smart.py", line 608, in
getVar
    return self.getVarFlag(var, "_content", expand, noweakdefault,
parsing)
  File "/srv/rrs/layers/bitbake/lib/bb/data_smart.py", line 723, in
getVarFlag
    local_var, overridedata = self._findVar(var)
  File "/srv/rrs/layers/bitbake/lib/bb/data_smart.py", line 480, in
_findVar
    return connector.getVar(var)
  File "/srv/rrs/layers/bitbake/lib/bb/tinfoil.py", line 62, in getVar
    value = self.tinfoil.run_command('dataStoreConnectorFindVar',
self.dsindex, name)
  File "/srv/rrs/layers/bitbake/lib/bb/tinfoil.py", line 316, in
run_command
    raise TinfoilCommandFailed(result[1])
bb.tinfoil.TinfoilCommandFailed: Traceback (most recent call last):
  File "/srv/rrs/layers/bitbake/lib/bb/command.py", line 81, in
runCommand
    result = command_method(self, commandline)
  File "/srv/rrs/layers/bitbake/lib/bb/command.py", line 465, in
dataStoreConnectorFindVar
    datastore = command.remotedatastores[dsindex]
  File "/srv/rrs/layers/bitbake/lib/bb/remotedata.py", line 39, in
__getitem__
    return self.datastores[key]
TypeError: unhashable type: 'list'

Signed-off-by: Aníbal Limón <anibal.limon@linux.intel.com>
2018-05-04 23:57:52 +12:00
Aníbal Limón
029796b103 templates/rrs/recipedetail.html: Change Maintainer to Committer
To avoid confusion about who is the current maintainer.

Signed-off-by: Aníbal Limón <anibal.limon@linux.intel.com>
2018-05-04 23:57:52 +12:00
Aníbal Limón
66193c153b rrs/views.py: When export to csv use release_milestone.csv
To be more descriptive about what report is.

Signed-off-by: Aníbal Limón <anibal.limon@linux.intel.com>
2018-05-04 23:57:52 +12:00
Aníbal Limón
f8753345c0 rrs/tools/rrs_upstream_history.py: Disable threads due to new tinfoil API
The new tinfoil API don't support pythreads so set to 1 while
tinfoil have support.

Signed-off-by: Aníbal Limón <anibal.limon@linux.intel.com>
2018-05-04 23:57:52 +12:00
Aníbal Limón
8fc103d3d6 rrs/tools/rrs_distros.py: Don't open files in binary mode
It ends on failures with Python 3.x when do a split method
over binary data.

Signed-off-by: Aníbal Limón <anibal.limon@linux.intel.com>
2018-05-04 23:57:52 +12:00
Aníbal Limón
d94a3524b2 rrs/tools/rrs_upgrade_history.py: Remove mp process related code
The new tinfoil API acts as a client/server so isn't need to work
around creating process to avoid errors in cooker with different
data revisions.

Also add a try/except when create an upgrade display the error.

Signed-off-by: Aníbal Limón <anibal.limon@linux.intel.com>
2018-05-04 23:57:52 +12:00
Aníbal Limón
fbe5857877 rrs/tools/common.py: Fix typo when load recipes
[YOCTO #10863]

Signed-off-by: Aníbal Limón <anibal.limon@linux.intel.com>
2018-05-04 23:57:52 +12:00
Aníbal Limón
d13329be1d rrs/tools: Upgrade to use new API of tinfoil
The new client/server API of tinfoil requires explicit call of
shutdown method to send the event for finalize cooker process.

[YOCTO #10863]

Signed-off-by: Aníbal Limón <anibal.limon@linux.intel.com>
2018-05-04 23:57:52 +12:00
Aníbal Limón
8ed223e13c rrs/tools: Upgrade to use transaction.atomic() in Django 1.6
Django 1.6 provides a context manager for atomic transactions so
update the way for make db transactions.

Signed-off-by: Aníbal Limón <anibal.limon@linux.intel.com>
2018-05-04 23:57:52 +12:00
Aníbal Limón
1c5b79a312 templates/rrs/recipedetail.html: Fix unneeded quotes in url call
The url template function resolve an URL associated with a model, so
in Django 1.6 quotes around variables don't get the variable value.

Signed-off-by: Aníbal Limón <anibal.limon@linux.intel.com>
2018-05-04 23:57:52 +12:00
Aníbal Limón
818288f17c rrs/tools/common.py: When load a recipe use tinfoil cache
Due to multi-configuration builds in bitbake now is needed
to use a tinfoil cache instance instead of call directly in
bb.cache.

Signed-off-by: Aníbal Limón <anibal.limon@linux.intel.com>
2018-05-04 23:57:52 +12:00
Aníbal Limón
2c6610ce5d rrs/tools/rrs_upgrade_history.py: Fix import of split_recipe_fn
split_recipe_fn function was moved to update_layer during python3
upgrade in layerindex.

Signed-off-by: Aníbal Limón <anibal.limon@linux.intel.com>
2018-05-04 23:57:52 +12:00
Aníbal Limón
34be7b80c2 rrs/tools/common.py: common_setup() Add layerindex to the path.
Add explicity layerindex folder to the path for avoid errors
in imports.

Signed-off-by: Aníbal Limón <anibal.limon@linux.intel.com>
2018-05-04 23:57:52 +12:00
Aníbal Limón
6b8cfecf48 layerindex,rrs: Minor changes to work with python3.
[YOCTO #9746]

Signed-off-by: Aníbal Limón <anibal.limon@linux.intel.com>
2018-05-04 23:57:52 +12:00
Aníbal Limón
48375d481c rrs: Frontend update to work with Python3 and django
[YOCTO #9746]

Signed-off-by: Aníbal Limón <anibal.limon@linux.intel.com>
2018-05-04 23:57:52 +12:00
Aníbal Limón
af8275db0f rrs/tools/rrs_maintainer_history.py: Update to point new meta-poky repository.
Signed-off-by: Aníbal Limón <anibal.limon@linux.intel.com>
2018-05-04 23:57:52 +12:00
Aníbal Limón
0a5781ae07 tools/rrs_distros.py: Fix distro_check.create_distro_packages_list call
Now create_distro_packages_list requires d to be passed, this change was
done to enable proxy support into create_distro_packages_list function.

Signed-off-by: Aníbal Limón <anibal.limon@linux.intel.com>
2018-05-04 23:57:52 +12:00
Aníbal Limón
86d77504f2 rrs: Fix frontpage redirect
We was experimenting fixed redirects due to urls are
cached at init of rrs.

Move the redirect Release, Milestone selection to a view
called FrontPageRedirect.

Signed-off-by: Aníbal Limón <anibal.limon@linux.intel.com>
2018-05-04 23:57:52 +12:00
Aníbal Limón
2943d182e5 rrs/tools/rrs_upstream_email.py: Update message to point recipe file
instead of deprecated upstream tracking file.

Since upstream_tracking.inc file is deprecated because now every recipe
has it's own upstream information then update email message according it.

Signed-off-by: Aníbal Limón <anibal.limon@linux.intel.com>
2018-05-04 23:57:52 +12:00
Belen Barros Pena
cd5c4ff7d1 Show the 'Upstream version' column only when relevant
Currently the recipes table always shows the 'Upstream version' column.
However, for recipes up-to-date, this column is irrelevant, since for such
recipes 'version' always equals 'upstream version'.

Since the vast majority of the recipes are up-to-date, when you are
showing all the recipes the 'upstream version' column is also pretty
useless.

The column is only really relevant when you are looking at recipes
with status 'not updated' or 'can't be updated'. I would have thought
there would be nothing to show in the 'Upstream version' column for
recipes with status 'Unknown', but it turns out some of those do have
an upstream version value, that I thought might be useful to the
maintainer somehow.

So, this patch hides the 'Upstream version' column whenever you are
looking at all the recipes or up-to-date recipes; and shows it when
you select any of the other recipe statuses

Signed-off-by: Belen Barros Pena <belen.barros.pena@intel.com>
2018-05-04 23:57:52 +12:00
Aníbal Limón
3ccb422aeb rrs/recipes: Add support for persist search and filters.
Now when search something the URL is modified and you can
share the URL for access to the data.

[YOCTO #7809]

Signed-off-by: Aníbal Limón <anibal.limon@linux.intel.com>
2018-05-04 23:57:52 +12:00
Aníbal Limón
aab0a9ad3f rrs/urls.py: Disable permanent (301) redirection on main view.
Don't redirect permanently in main view because the first URL needs
to point to the current release, so use permanent = False to return
a 302..

Signed-off-by: Aníbal Limón <anibal.limon@linux.intel.com>
2018-05-04 23:57:52 +12:00
Belen Barros Pena
1bd89388e1 Update tool URL in status email
The status email sent to the Yocto Project mailing
list by the Recipe Reporting Tool still shows the
old URL (http://packages.yoctoproject.org).

This patch changes it to the new URL:

http://recipes.yoctoproject.org

Signed-off-by: Belen Barros Pena <belen.barros.pena@linux.intel.com>
Signed-off-by: Aníbal Limón <anibal.limon@linux.intel.com>
2018-05-04 23:57:52 +12:00
Aníbal Limón
32391e968a rrs_upstream_history.py: Fix use regexes in packages
Make difference when handling suffixes or preffixes like 'nativesdk-'
when try to use regexes in packages that have suffixes like '-crosssdk'
it can contain the arch into it like binutils-crosssdk-x86_64.

Use split method in suffixes and replace method into preffixes, this
fixes issues with suffixes containing archs at end.

[YOCTO #8102]

Signed-off-by: Aníbal Limón <anibal.limon@linux.intel.com>
2018-05-04 23:57:52 +12:00
Mariano Lopez
ef32d52b29 rrs/views: Fix mismatch in summary and recipe list
This fixes the issue with the mismatch in the
statistics bar and the recipe list. The mismatch
is caused by the gcc-source recipe, because every
version of the recipe is counted in Upstream History
but not in Recipe Upgrade.

Also added the TODO.rrs to track bugs and issues with
rrs.

Signed-off-by: Mariano Lopez <mariano.lopez@linux.intel.com>
Signed-off-by: Aníbal Limón <anibal.limon@linux.intel.com>
2018-05-04 23:57:52 +12:00
Aníbal Limón
5e8eeaa8ac templates/rrs/recipes.html: Re-add the sort property into Maintainer.
When last updated colum comes we lost the sort property.

Signed-off-by: Aníbal Limón <anibal.limon@linux.intel.com>
2018-05-04 23:57:52 +12:00
Mariano Lopez
742128c517 rrs/base_toplevel.html: Navbar redesign
This provides changes in the front end for the
navbar, now it shows the percentage of recipes
up-to-date, not update, unknown and can't be
updated along with the number of recipes.

This also moves the update percentage to the
end and adds clarity to what it means.

This also moves the export list button the the
top bar.

[YOCTO #8020]

Signed-off-by: Mariano Lopez <mariano.lopez@linux.intel.com>
Signed-off-by: Aníbal Limón <anibal.limon@linux.intel.com>
2018-05-04 23:57:52 +12:00
Mariano Lopez
6aa369757c rrs/views.py: Added percentages for navbar
This adds the percentage for all the recipes types
(up-to-date, not updated, unknown, can't be updated)
so it can be displayed in the navbar.

This also adds the number of the recipes not updated
at the begining of the period and number of recipes
updated in the period.

The changes in the frontend are still missing, this
just adds the functionality.

[YOCTO #8020]

Signed-off-by: Mariano Lopez <mariano.lopez@linux.intel.com>
Signed-off-by: Aníbal Limón <anibal.limon@linux.intel.com>
2018-05-04 23:57:52 +12:00
Belen Barros Pena
66c0b28982 rrs: avoid too many columns in the recipes table
Currently, when you select 'Can't be updated' in the upstream
status filter, the resulting table will add the 'No update reason'
column to the default column set.

Although this is probably useful information to see in the
table itself, it results in too many columns, and a rather
unpleasant layout change.

This patch hides the 'Summary' column whenever you select
'Can't be updated' in the upstream status filter, effectively
replacing the 'Summary' with the 'No update reason' column,
which is probably more relevant in this context. Now you
have less columns distracting you, and a slightly less
jumpy layout change.

A designer would have come up with this solution in the
first place. Sadly, she was never asked.

Signed-off-by: Belen Barros Pena <belen.barros.pena@linux.intel.com>
Signed-off-by: Aníbal Limón <anibal.limon@linux.intel.com>
2018-05-04 23:57:52 +12:00
Belen Barros Pena
671d06aaa7 rrs: remove sorting from the 'No update reason' column
Nothing useful can come from sorting by this column.

Signed-off-by: Belen Barros Pena <belen.barros.pena@linux.intel.com>
Signed-off-by: Aníbal Limón <anibal.limon@linux.intel.com>
2018-05-04 23:57:52 +12:00
Aníbal Limón
d1d108507c rrs_upstream_history.py: Use regexes in SPECIAL_PKGSUFFIX packages
When SPECIAL_PKGSUFFIX packages don't have regexes use it from package
without SPECIAL_PKGSUFFIX.

[YOCTO #8102]

For example: python-native use regex from python if don't have one.

Signed-off-by: Aníbal Limón <anibal.limon@linux.intel.com>
2018-05-04 23:57:52 +12:00
Mariano Lopez
e325f317aa recipes.html: Format change for last updated column
This changes the format of the last updated column.
Now the column text is set to the default text color.

[YOCTO #8018]

Signed-off-by: Mariano Lopez <mariano.lopez@linux.intel.com>
2018-05-04 23:57:52 +12:00
Mariano Lopez
bffe94dd66 rrs-additional.css: Added min-width to last update column
This adds the min-width for the last updated column
to the css.

[YOCTO #8018]

Signed-off-by: Mariano Lopez <mariano.lopez@linux.intel.com>
2018-05-04 23:57:52 +12:00
Mariano Lopez
b08004f778 views.py: Show last updated date for all recipes
This changes the behavior of the last update column
and show the date for all the recipes, including the
up to date recipes.

[YOCTO #8018]

Signed-off-by: Mariano Lopez <mariano.lopez@linux.intel.com>
2018-05-04 23:57:52 +12:00
Aníbal Limón
fac5b245d4 rrs/tools: UpstreamHistory only load current recipe files.
Don't load all recipes avaiable in layer because was previous
filtered to have the last version.

Signed-off-by: Aníbal Limón <anibal.limon@linux.intel.com>
2018-05-04 23:57:52 +12:00
Aníbal Limón
e81daf2e0f rrs/tools: get_recipe_pv_without_srcrev add support for discard prefixes
Avoid version prefixes like v|r into all versions.

Signed-off-by: Aníbal Limón <anibal.limon@linux.intel.com>
2018-05-04 23:57:52 +12:00
Aníbal Limón
b2e04c0a9e rrs: Fix last updated.
Don't conditional last updated to upstream status.

Signed-off-by: Aníbal Limón <anibal.limon@linux.intel.com>
2018-05-04 23:57:52 +12:00