Commit Graph

121 Commits

Author SHA1 Message Date
Paul Eggleton
e50c2e6841 Rename RRS URLs to have rrs_ prefix
Make these distinct from the layerindex ones.

Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
2018-05-04 23:57:52 +12:00
Paul Eggleton
10a639a6b7 rrs/tools/rrs_unique_recipes: drop
We can't just delete arbitrary recipes that do exist in the layer if
we're in a general layer index database. We'll need to handle this in a
different way, or just live with the fact that there will be duplicate
entries.

Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
2018-05-04 23:57:52 +12:00
Paul Eggleton
f58754c998 rrs/models: move Raw class to views.py
This is something that really belongs in with the views, so move it
there.

Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
2018-05-04 23:57:52 +12:00
Aníbal Limón
510dec2ee8 rrs/tools/rrs_{distros,upstream_history}: Add path for poky meta lib
Due to bitbake client/server changes now the meta path isn't included
to sys.path when load tinfoil.

Signed-off-by: Aníbal Limón <anibal.limon@linux.intel.com>
2018-05-04 23:57:52 +12:00
Aníbal Limón
4c2e9ec1f9 rrs/tools/rrs_upstream_history.py: Change regex variables to tuple
This are causing a failure when access the remote datastore.

File "rrs/rrs/tools/rrs_upstream_history.py", line 169, in <module>
    set_regexes(recipe_data)
  File "rrs/rrs/tools/rrs_upstream_history.py", line 53, in set_regexes
    if any(d.getVar(var, True) for var in variables):
  File "rrs/rrs/tools/rrs_upstream_history.py", line 53, in <genexpr>
    if any(d.getVar(var, True) for var in variables):
  File "/srv/rrs/layers/bitbake/lib/bb/data_smart.py", line 608, in
getVar
    return self.getVarFlag(var, "_content", expand, noweakdefault,
parsing)
  File "/srv/rrs/layers/bitbake/lib/bb/data_smart.py", line 723, in
getVarFlag
    local_var, overridedata = self._findVar(var)
  File "/srv/rrs/layers/bitbake/lib/bb/data_smart.py", line 480, in
_findVar
    return connector.getVar(var)
  File "/srv/rrs/layers/bitbake/lib/bb/tinfoil.py", line 62, in getVar
    value = self.tinfoil.run_command('dataStoreConnectorFindVar',
self.dsindex, name)
  File "/srv/rrs/layers/bitbake/lib/bb/tinfoil.py", line 316, in
run_command
    raise TinfoilCommandFailed(result[1])
bb.tinfoil.TinfoilCommandFailed: Traceback (most recent call last):
  File "/srv/rrs/layers/bitbake/lib/bb/command.py", line 81, in
runCommand
    result = command_method(self, commandline)
  File "/srv/rrs/layers/bitbake/lib/bb/command.py", line 465, in
dataStoreConnectorFindVar
    datastore = command.remotedatastores[dsindex]
  File "/srv/rrs/layers/bitbake/lib/bb/remotedata.py", line 39, in
__getitem__
    return self.datastores[key]
TypeError: unhashable type: 'list'

Signed-off-by: Aníbal Limón <anibal.limon@linux.intel.com>
2018-05-04 23:57:52 +12:00
Aníbal Limón
66193c153b rrs/views.py: When export to csv use release_milestone.csv
To be more descriptive about what report is.

Signed-off-by: Aníbal Limón <anibal.limon@linux.intel.com>
2018-05-04 23:57:52 +12:00
Aníbal Limón
f8753345c0 rrs/tools/rrs_upstream_history.py: Disable threads due to new tinfoil API
The new tinfoil API don't support pythreads so set to 1 while
tinfoil have support.

Signed-off-by: Aníbal Limón <anibal.limon@linux.intel.com>
2018-05-04 23:57:52 +12:00
Aníbal Limón
8fc103d3d6 rrs/tools/rrs_distros.py: Don't open files in binary mode
It ends on failures with Python 3.x when do a split method
over binary data.

Signed-off-by: Aníbal Limón <anibal.limon@linux.intel.com>
2018-05-04 23:57:52 +12:00
Aníbal Limón
d94a3524b2 rrs/tools/rrs_upgrade_history.py: Remove mp process related code
The new tinfoil API acts as a client/server so isn't need to work
around creating process to avoid errors in cooker with different
data revisions.

Also add a try/except when create an upgrade display the error.

Signed-off-by: Aníbal Limón <anibal.limon@linux.intel.com>
2018-05-04 23:57:52 +12:00
Aníbal Limón
fbe5857877 rrs/tools/common.py: Fix typo when load recipes
[YOCTO #10863]

Signed-off-by: Aníbal Limón <anibal.limon@linux.intel.com>
2018-05-04 23:57:52 +12:00
Aníbal Limón
d13329be1d rrs/tools: Upgrade to use new API of tinfoil
The new client/server API of tinfoil requires explicit call of
shutdown method to send the event for finalize cooker process.

[YOCTO #10863]

Signed-off-by: Aníbal Limón <anibal.limon@linux.intel.com>
2018-05-04 23:57:52 +12:00
Aníbal Limón
8ed223e13c rrs/tools: Upgrade to use transaction.atomic() in Django 1.6
Django 1.6 provides a context manager for atomic transactions so
update the way for make db transactions.

Signed-off-by: Aníbal Limón <anibal.limon@linux.intel.com>
2018-05-04 23:57:52 +12:00
Aníbal Limón
818288f17c rrs/tools/common.py: When load a recipe use tinfoil cache
Due to multi-configuration builds in bitbake now is needed
to use a tinfoil cache instance instead of call directly in
bb.cache.

Signed-off-by: Aníbal Limón <anibal.limon@linux.intel.com>
2018-05-04 23:57:52 +12:00
Aníbal Limón
2c6610ce5d rrs/tools/rrs_upgrade_history.py: Fix import of split_recipe_fn
split_recipe_fn function was moved to update_layer during python3
upgrade in layerindex.

Signed-off-by: Aníbal Limón <anibal.limon@linux.intel.com>
2018-05-04 23:57:52 +12:00
Aníbal Limón
34be7b80c2 rrs/tools/common.py: common_setup() Add layerindex to the path.
Add explicity layerindex folder to the path for avoid errors
in imports.

Signed-off-by: Aníbal Limón <anibal.limon@linux.intel.com>
2018-05-04 23:57:52 +12:00
Aníbal Limón
6b8cfecf48 layerindex,rrs: Minor changes to work with python3.
[YOCTO #9746]

Signed-off-by: Aníbal Limón <anibal.limon@linux.intel.com>
2018-05-04 23:57:52 +12:00
Aníbal Limón
48375d481c rrs: Frontend update to work with Python3 and django
[YOCTO #9746]

Signed-off-by: Aníbal Limón <anibal.limon@linux.intel.com>
2018-05-04 23:57:52 +12:00
Aníbal Limón
af8275db0f rrs/tools/rrs_maintainer_history.py: Update to point new meta-poky repository.
Signed-off-by: Aníbal Limón <anibal.limon@linux.intel.com>
2018-05-04 23:57:52 +12:00
Aníbal Limón
0a5781ae07 tools/rrs_distros.py: Fix distro_check.create_distro_packages_list call
Now create_distro_packages_list requires d to be passed, this change was
done to enable proxy support into create_distro_packages_list function.

Signed-off-by: Aníbal Limón <anibal.limon@linux.intel.com>
2018-05-04 23:57:52 +12:00
Aníbal Limón
86d77504f2 rrs: Fix frontpage redirect
We was experimenting fixed redirects due to urls are
cached at init of rrs.

Move the redirect Release, Milestone selection to a view
called FrontPageRedirect.

Signed-off-by: Aníbal Limón <anibal.limon@linux.intel.com>
2018-05-04 23:57:52 +12:00
Aníbal Limón
2943d182e5 rrs/tools/rrs_upstream_email.py: Update message to point recipe file
instead of deprecated upstream tracking file.

Since upstream_tracking.inc file is deprecated because now every recipe
has it's own upstream information then update email message according it.

Signed-off-by: Aníbal Limón <anibal.limon@linux.intel.com>
2018-05-04 23:57:52 +12:00
Aníbal Limón
3ccb422aeb rrs/recipes: Add support for persist search and filters.
Now when search something the URL is modified and you can
share the URL for access to the data.

[YOCTO #7809]

Signed-off-by: Aníbal Limón <anibal.limon@linux.intel.com>
2018-05-04 23:57:52 +12:00
Aníbal Limón
aab0a9ad3f rrs/urls.py: Disable permanent (301) redirection on main view.
Don't redirect permanently in main view because the first URL needs
to point to the current release, so use permanent = False to return
a 302..

Signed-off-by: Aníbal Limón <anibal.limon@linux.intel.com>
2018-05-04 23:57:52 +12:00
Belen Barros Pena
1bd89388e1 Update tool URL in status email
The status email sent to the Yocto Project mailing
list by the Recipe Reporting Tool still shows the
old URL (http://packages.yoctoproject.org).

This patch changes it to the new URL:

http://recipes.yoctoproject.org

Signed-off-by: Belen Barros Pena <belen.barros.pena@linux.intel.com>
Signed-off-by: Aníbal Limón <anibal.limon@linux.intel.com>
2018-05-04 23:57:52 +12:00
Aníbal Limón
32391e968a rrs_upstream_history.py: Fix use regexes in packages
Make difference when handling suffixes or preffixes like 'nativesdk-'
when try to use regexes in packages that have suffixes like '-crosssdk'
it can contain the arch into it like binutils-crosssdk-x86_64.

Use split method in suffixes and replace method into preffixes, this
fixes issues with suffixes containing archs at end.

[YOCTO #8102]

Signed-off-by: Aníbal Limón <anibal.limon@linux.intel.com>
2018-05-04 23:57:52 +12:00
Mariano Lopez
ef32d52b29 rrs/views: Fix mismatch in summary and recipe list
This fixes the issue with the mismatch in the
statistics bar and the recipe list. The mismatch
is caused by the gcc-source recipe, because every
version of the recipe is counted in Upstream History
but not in Recipe Upgrade.

Also added the TODO.rrs to track bugs and issues with
rrs.

Signed-off-by: Mariano Lopez <mariano.lopez@linux.intel.com>
Signed-off-by: Aníbal Limón <anibal.limon@linux.intel.com>
2018-05-04 23:57:52 +12:00
Mariano Lopez
742128c517 rrs/base_toplevel.html: Navbar redesign
This provides changes in the front end for the
navbar, now it shows the percentage of recipes
up-to-date, not update, unknown and can't be
updated along with the number of recipes.

This also moves the update percentage to the
end and adds clarity to what it means.

This also moves the export list button the the
top bar.

[YOCTO #8020]

Signed-off-by: Mariano Lopez <mariano.lopez@linux.intel.com>
Signed-off-by: Aníbal Limón <anibal.limon@linux.intel.com>
2018-05-04 23:57:52 +12:00
Mariano Lopez
6aa369757c rrs/views.py: Added percentages for navbar
This adds the percentage for all the recipes types
(up-to-date, not updated, unknown, can't be updated)
so it can be displayed in the navbar.

This also adds the number of the recipes not updated
at the begining of the period and number of recipes
updated in the period.

The changes in the frontend are still missing, this
just adds the functionality.

[YOCTO #8020]

Signed-off-by: Mariano Lopez <mariano.lopez@linux.intel.com>
Signed-off-by: Aníbal Limón <anibal.limon@linux.intel.com>
2018-05-04 23:57:52 +12:00
Aníbal Limón
d1d108507c rrs_upstream_history.py: Use regexes in SPECIAL_PKGSUFFIX packages
When SPECIAL_PKGSUFFIX packages don't have regexes use it from package
without SPECIAL_PKGSUFFIX.

[YOCTO #8102]

For example: python-native use regex from python if don't have one.

Signed-off-by: Aníbal Limón <anibal.limon@linux.intel.com>
2018-05-04 23:57:52 +12:00
Mariano Lopez
bffe94dd66 rrs-additional.css: Added min-width to last update column
This adds the min-width for the last updated column
to the css.

[YOCTO #8018]

Signed-off-by: Mariano Lopez <mariano.lopez@linux.intel.com>
2018-05-04 23:57:52 +12:00
Mariano Lopez
b08004f778 views.py: Show last updated date for all recipes
This changes the behavior of the last update column
and show the date for all the recipes, including the
up to date recipes.

[YOCTO #8018]

Signed-off-by: Mariano Lopez <mariano.lopez@linux.intel.com>
2018-05-04 23:57:52 +12:00
Aníbal Limón
fac5b245d4 rrs/tools: UpstreamHistory only load current recipe files.
Don't load all recipes avaiable in layer because was previous
filtered to have the last version.

Signed-off-by: Aníbal Limón <anibal.limon@linux.intel.com>
2018-05-04 23:57:52 +12:00
Aníbal Limón
e81daf2e0f rrs/tools: get_recipe_pv_without_srcrev add support for discard prefixes
Avoid version prefixes like v|r into all versions.

Signed-off-by: Aníbal Limón <anibal.limon@linux.intel.com>
2018-05-04 23:57:52 +12:00
Aníbal Limón
b2e04c0a9e rrs: Fix last updated.
Don't conditional last updated to upstream status.

Signed-off-by: Aníbal Limón <anibal.limon@linux.intel.com>
2018-05-04 23:57:52 +12:00
Aníbal Limón
4936eaa65a rrs/tools: Add daily_run.sh script for provide updates.
Signed-off-by: Aníbal Limón <anibal.limon@linux.intel.com>
2018-05-04 23:57:52 +12:00
Aníbal Limón
5410eb9ccc rrs/tools/rrs_upgrade_history.py: Don't take into account git recipes
without versioning.

Don't detect upgrade in git recipes that don't have versioning only
PV=git.

Signed-off-by: Aníbal Limón <anibal.limon@linux.intel.com>
2018-05-04 23:57:52 +12:00
Aníbal Limón
6added950f rrs/views.py: RecipeListView always use recipes from history.
If recipe don't exist in history means that don't need to be displayed.

Signed-off-by: Aníbal Limón <anibal.limon@linux.intel.com>
2018-05-04 23:57:52 +12:00
Mariano Lopez
ac50ffd82e models.py: Code cleanup
Cleaned up the class Raw to have better readability.
Also added docstring to methods in this class.

Signed-off-by: Mariano Lopez <mariano.lopez@linux.intel.com>
Signed-off-by: Aníbal Limón <anibal.limon@linux.intel.com>
2018-05-04 23:57:52 +12:00
Mariano Lopez
45ba3a280d views.py: Changed date format to iso
Changed the day format to iso. Also, the outdated
field in the recipes views was changed from days
to the date the recipe was last updated.

Signed-off-by: Mariano Lopez <mariano.lopez@linux.intel.com>
Signed-off-by: Aníbal Limón <anibal.limon@linux.intel.com>
2018-05-04 23:57:52 +12:00
Mariano Lopez
7f24895f6e views.py: Changed the behavior of percentage done
This changes the percetage done from recipes up to date
in the period to percentage of not updated recipes that
were updated in the period.

Signed-off-by: Mariano Lopez <mariano.lopez@linux.intel.com>
Signed-off-by: Aníbal Limón <anibal.limon@linux.intel.com>
2018-05-04 23:57:52 +12:00
Mariano Lopez
aab05507d5 models.py: Added Raw SQL for percentage updated
This adds SQL queries that will be used for the
percentage of recipes updated in a period.

Signed-off-by: Mariano Lopez <mariano.lopez@linux.intel.com>
Signed-off-by: Aníbal Limón <anibal.limon@linux.intel.com>
2018-05-04 23:57:52 +12:00
Mariano Lopez
58ea26418e views.py: Fix current up to date percentage
There was an issue where the current up to date was always
using the latest upstream version to get the percentage.
This uses the upstream version in the period displayed.

Signed-off-by: Mariano Lopez <mariano.lopez@linux.intel.com>
2018-05-04 23:57:52 +12:00
Mariano Lopez
034bf57f71 views.py: Fix version in current milestone
Fixed the version displayed in current milestone.
Before the data was obtained from recipe upgrad table
and this allows to fetch the data from the recipe table.

Signed-off-by: Mariano Lopez <mariano.lopez@linux.intel.com>
2018-05-04 23:57:52 +12:00
Mariano Lopez
faafae13fa views.py: Add how long a recipe hasn't been updated
This add a tooltip in the upstream status field that
show how long the recipe hasn't been updated.

Signed-off-by: Mariano Lopez <mariano.lopez@linux.intel.com>
Signed-off-by: Aníbal Limón <anibal.limon@linux.intel.com>
2018-05-04 23:57:52 +12:00
Mariano Lopez
a9aff3cae6 rrs/views.py: Changed ORM to raw SQL calls
Allow to use the new raw SQL call to get better performance in the
web page. Also removed SQL queries inside loops.
This is only for two views: recipes and maintainers

Signed-off-by: Mariano Lopez <mariano.lopez@linux.intel.com>
Signed-off-by: Aníbal Limón <anibal.limon@linux.intel.com>
2018-05-04 23:57:52 +12:00
Mariano Lopez
35b4d0fb34 rrs/models.py: Added raw SQL calls
The raw calls are going to be used instead of Django's ORM.
This improves the performance of the webpage.

Signed-off-by: Mariano Lopez <mariano.lopez@linux.intel.com>
Signed-off-by: Aníbal Limón <anibal.limon@linux.intel.com>
2018-05-04 23:57:52 +12:00
Aníbal Limón
22eb60487f rrs/tools/rrs_maintainer_history.py: Set new recipes to No maintainer if don't have
When add new recipes sometimes don't have maintainer defined causing
errors on frontend when search for one.

Signed-off-by: Aníbal Limón <anibal.limon@linux.intel.com>
2018-05-04 23:57:52 +12:00
Aníbal Limón
1c2f2b8edb rrs/tools: Add rrs_distros script.
This script updates the information about recipes in other distros.

Signed-off-by: Aníbal Limón <anibal.limon@linux.intel.com>
2018-05-04 23:57:52 +12:00
Aníbal Limón
ffee2377d1 rrs/tools/rrs_upstream_history.py: Remove unneeded histroy save at end.
Signed-off-by: Aníbal Limón <anibal.limon@linux.intel.com>
2018-05-04 23:57:52 +12:00
Aníbal Limón
d745458018 rrs/tools: Convert unicode strings from Django models to strings.
bb.cache.Cache.loadDatafull() expects an filepath in asciii, if not
is provided it fails with character mapping error.

Signed-off-by: Aníbal Limón <anibal.limon@linux.intel.com>
2018-05-04 23:57:52 +12:00