Commit Graph

99 Commits

Author SHA1 Message Date
Paul Eggleton
10a639a6b7 rrs/tools/rrs_unique_recipes: drop
We can't just delete arbitrary recipes that do exist in the layer if
we're in a general layer index database. We'll need to handle this in a
different way, or just live with the fact that there will be duplicate
entries.

Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
2018-05-04 23:57:52 +12:00
Aníbal Limón
510dec2ee8 rrs/tools/rrs_{distros,upstream_history}: Add path for poky meta lib
Due to bitbake client/server changes now the meta path isn't included
to sys.path when load tinfoil.

Signed-off-by: Aníbal Limón <anibal.limon@linux.intel.com>
2018-05-04 23:57:52 +12:00
Aníbal Limón
4c2e9ec1f9 rrs/tools/rrs_upstream_history.py: Change regex variables to tuple
This are causing a failure when access the remote datastore.

File "rrs/rrs/tools/rrs_upstream_history.py", line 169, in <module>
    set_regexes(recipe_data)
  File "rrs/rrs/tools/rrs_upstream_history.py", line 53, in set_regexes
    if any(d.getVar(var, True) for var in variables):
  File "rrs/rrs/tools/rrs_upstream_history.py", line 53, in <genexpr>
    if any(d.getVar(var, True) for var in variables):
  File "/srv/rrs/layers/bitbake/lib/bb/data_smart.py", line 608, in
getVar
    return self.getVarFlag(var, "_content", expand, noweakdefault,
parsing)
  File "/srv/rrs/layers/bitbake/lib/bb/data_smart.py", line 723, in
getVarFlag
    local_var, overridedata = self._findVar(var)
  File "/srv/rrs/layers/bitbake/lib/bb/data_smart.py", line 480, in
_findVar
    return connector.getVar(var)
  File "/srv/rrs/layers/bitbake/lib/bb/tinfoil.py", line 62, in getVar
    value = self.tinfoil.run_command('dataStoreConnectorFindVar',
self.dsindex, name)
  File "/srv/rrs/layers/bitbake/lib/bb/tinfoil.py", line 316, in
run_command
    raise TinfoilCommandFailed(result[1])
bb.tinfoil.TinfoilCommandFailed: Traceback (most recent call last):
  File "/srv/rrs/layers/bitbake/lib/bb/command.py", line 81, in
runCommand
    result = command_method(self, commandline)
  File "/srv/rrs/layers/bitbake/lib/bb/command.py", line 465, in
dataStoreConnectorFindVar
    datastore = command.remotedatastores[dsindex]
  File "/srv/rrs/layers/bitbake/lib/bb/remotedata.py", line 39, in
__getitem__
    return self.datastores[key]
TypeError: unhashable type: 'list'

Signed-off-by: Aníbal Limón <anibal.limon@linux.intel.com>
2018-05-04 23:57:52 +12:00
Aníbal Limón
f8753345c0 rrs/tools/rrs_upstream_history.py: Disable threads due to new tinfoil API
The new tinfoil API don't support pythreads so set to 1 while
tinfoil have support.

Signed-off-by: Aníbal Limón <anibal.limon@linux.intel.com>
2018-05-04 23:57:52 +12:00
Aníbal Limón
8fc103d3d6 rrs/tools/rrs_distros.py: Don't open files in binary mode
It ends on failures with Python 3.x when do a split method
over binary data.

Signed-off-by: Aníbal Limón <anibal.limon@linux.intel.com>
2018-05-04 23:57:52 +12:00
Aníbal Limón
d94a3524b2 rrs/tools/rrs_upgrade_history.py: Remove mp process related code
The new tinfoil API acts as a client/server so isn't need to work
around creating process to avoid errors in cooker with different
data revisions.

Also add a try/except when create an upgrade display the error.

Signed-off-by: Aníbal Limón <anibal.limon@linux.intel.com>
2018-05-04 23:57:52 +12:00
Aníbal Limón
fbe5857877 rrs/tools/common.py: Fix typo when load recipes
[YOCTO #10863]

Signed-off-by: Aníbal Limón <anibal.limon@linux.intel.com>
2018-05-04 23:57:52 +12:00
Aníbal Limón
d13329be1d rrs/tools: Upgrade to use new API of tinfoil
The new client/server API of tinfoil requires explicit call of
shutdown method to send the event for finalize cooker process.

[YOCTO #10863]

Signed-off-by: Aníbal Limón <anibal.limon@linux.intel.com>
2018-05-04 23:57:52 +12:00
Aníbal Limón
8ed223e13c rrs/tools: Upgrade to use transaction.atomic() in Django 1.6
Django 1.6 provides a context manager for atomic transactions so
update the way for make db transactions.

Signed-off-by: Aníbal Limón <anibal.limon@linux.intel.com>
2018-05-04 23:57:52 +12:00
Aníbal Limón
818288f17c rrs/tools/common.py: When load a recipe use tinfoil cache
Due to multi-configuration builds in bitbake now is needed
to use a tinfoil cache instance instead of call directly in
bb.cache.

Signed-off-by: Aníbal Limón <anibal.limon@linux.intel.com>
2018-05-04 23:57:52 +12:00
Aníbal Limón
2c6610ce5d rrs/tools/rrs_upgrade_history.py: Fix import of split_recipe_fn
split_recipe_fn function was moved to update_layer during python3
upgrade in layerindex.

Signed-off-by: Aníbal Limón <anibal.limon@linux.intel.com>
2018-05-04 23:57:52 +12:00
Aníbal Limón
34be7b80c2 rrs/tools/common.py: common_setup() Add layerindex to the path.
Add explicity layerindex folder to the path for avoid errors
in imports.

Signed-off-by: Aníbal Limón <anibal.limon@linux.intel.com>
2018-05-04 23:57:52 +12:00
Aníbal Limón
6b8cfecf48 layerindex,rrs: Minor changes to work with python3.
[YOCTO #9746]

Signed-off-by: Aníbal Limón <anibal.limon@linux.intel.com>
2018-05-04 23:57:52 +12:00
Aníbal Limón
af8275db0f rrs/tools/rrs_maintainer_history.py: Update to point new meta-poky repository.
Signed-off-by: Aníbal Limón <anibal.limon@linux.intel.com>
2018-05-04 23:57:52 +12:00
Aníbal Limón
0a5781ae07 tools/rrs_distros.py: Fix distro_check.create_distro_packages_list call
Now create_distro_packages_list requires d to be passed, this change was
done to enable proxy support into create_distro_packages_list function.

Signed-off-by: Aníbal Limón <anibal.limon@linux.intel.com>
2018-05-04 23:57:52 +12:00
Aníbal Limón
2943d182e5 rrs/tools/rrs_upstream_email.py: Update message to point recipe file
instead of deprecated upstream tracking file.

Since upstream_tracking.inc file is deprecated because now every recipe
has it's own upstream information then update email message according it.

Signed-off-by: Aníbal Limón <anibal.limon@linux.intel.com>
2018-05-04 23:57:52 +12:00
Belen Barros Pena
1bd89388e1 Update tool URL in status email
The status email sent to the Yocto Project mailing
list by the Recipe Reporting Tool still shows the
old URL (http://packages.yoctoproject.org).

This patch changes it to the new URL:

http://recipes.yoctoproject.org

Signed-off-by: Belen Barros Pena <belen.barros.pena@linux.intel.com>
Signed-off-by: Aníbal Limón <anibal.limon@linux.intel.com>
2018-05-04 23:57:52 +12:00
Aníbal Limón
32391e968a rrs_upstream_history.py: Fix use regexes in packages
Make difference when handling suffixes or preffixes like 'nativesdk-'
when try to use regexes in packages that have suffixes like '-crosssdk'
it can contain the arch into it like binutils-crosssdk-x86_64.

Use split method in suffixes and replace method into preffixes, this
fixes issues with suffixes containing archs at end.

[YOCTO #8102]

Signed-off-by: Aníbal Limón <anibal.limon@linux.intel.com>
2018-05-04 23:57:52 +12:00
Aníbal Limón
d1d108507c rrs_upstream_history.py: Use regexes in SPECIAL_PKGSUFFIX packages
When SPECIAL_PKGSUFFIX packages don't have regexes use it from package
without SPECIAL_PKGSUFFIX.

[YOCTO #8102]

For example: python-native use regex from python if don't have one.

Signed-off-by: Aníbal Limón <anibal.limon@linux.intel.com>
2018-05-04 23:57:52 +12:00
Aníbal Limón
fac5b245d4 rrs/tools: UpstreamHistory only load current recipe files.
Don't load all recipes avaiable in layer because was previous
filtered to have the last version.

Signed-off-by: Aníbal Limón <anibal.limon@linux.intel.com>
2018-05-04 23:57:52 +12:00
Aníbal Limón
e81daf2e0f rrs/tools: get_recipe_pv_without_srcrev add support for discard prefixes
Avoid version prefixes like v|r into all versions.

Signed-off-by: Aníbal Limón <anibal.limon@linux.intel.com>
2018-05-04 23:57:52 +12:00
Aníbal Limón
4936eaa65a rrs/tools: Add daily_run.sh script for provide updates.
Signed-off-by: Aníbal Limón <anibal.limon@linux.intel.com>
2018-05-04 23:57:52 +12:00
Aníbal Limón
5410eb9ccc rrs/tools/rrs_upgrade_history.py: Don't take into account git recipes
without versioning.

Don't detect upgrade in git recipes that don't have versioning only
PV=git.

Signed-off-by: Aníbal Limón <anibal.limon@linux.intel.com>
2018-05-04 23:57:52 +12:00
Aníbal Limón
22eb60487f rrs/tools/rrs_maintainer_history.py: Set new recipes to No maintainer if don't have
When add new recipes sometimes don't have maintainer defined causing
errors on frontend when search for one.

Signed-off-by: Aníbal Limón <anibal.limon@linux.intel.com>
2018-05-04 23:57:52 +12:00
Aníbal Limón
1c2f2b8edb rrs/tools: Add rrs_distros script.
This script updates the information about recipes in other distros.

Signed-off-by: Aníbal Limón <anibal.limon@linux.intel.com>
2018-05-04 23:57:52 +12:00
Aníbal Limón
ffee2377d1 rrs/tools/rrs_upstream_history.py: Remove unneeded histroy save at end.
Signed-off-by: Aníbal Limón <anibal.limon@linux.intel.com>
2018-05-04 23:57:52 +12:00
Aníbal Limón
d745458018 rrs/tools: Convert unicode strings from Django models to strings.
bb.cache.Cache.loadDatafull() expects an filepath in asciii, if not
is provided it fails with character mapping error.

Signed-off-by: Aníbal Limón <anibal.limon@linux.intel.com>
2018-05-04 23:57:52 +12:00
Aníbal Limón
1f0935240d rrs/tools: Remove call to tinfoil shutdown method.
Tinfoil shutdown was not accepted as patch into bitbake because
this is a simple solution to a big problem.

Signed-off-by: Aníbal Limón <anibal.limon@linux.intel.com>
2018-05-04 23:57:52 +12:00
Aníbal Limón
e8b258e717 rrs/tools/rrs_upstream_history.py: Add recipe upstream script.
This script generates upstream version information and recipe status.

Signed-off-by: Aníbal Limón <anibal.limon@linux.intel.com>
2018-05-04 23:57:52 +12:00
Aníbal Limón
ad5cfbed01 rrs/tools/rrs_unique_recipes.py: Now only filter same pn recipes.
Don't remove native, cross, crossinitial and sdk recipes now only
keep major version of recipe based by pn.

Signed-off-by: Aníbal Limón <anibal.limon@linux.intel.com>
2018-05-04 23:57:52 +12:00
Aníbal Limón
1b6ba9fb0f rrs/tools/rrs_upgrade_history.py: WORKAROUND Run recipe parse as different proccess.
When runs recipe parse with different revisions in same process it
generates errors due to cooker parser uses global data cache's.

Signed-off-by: Aníbal Limón <anibal.limon@linux.intel.com>
2018-05-04 23:57:52 +12:00
Aníbal Limón
72cc4f72ba rrs/tools/common.py: Add import of FetchError and display parse errors
Import of FetchError when try to expand PV somethings requires to have
SRCREV/SRCPV causing an FetcherError.

Signed-off-by: Aníbal Limón <anibal.limon@linux.intel.com>
2018-05-04 23:57:52 +12:00
Aníbal Limón
ec0ae0dc29 rrs/tools.py: Create standalone get_recipe_pv_without_srcpv function.
Copied get_recipe_pv_without_srcpv from oe-core/recipeutils library
to avoid import errors caused by have imported twice oe-core from poky
and from tinfoil.

rrs/tools/rrs_upgrade_history.py: Update to use get_recipe_pv_without_srcpv
from common.

Signed-off-by: Aníbal Limón <anibal.limon@linux.intel.com>
2018-05-04 23:57:52 +12:00
Aníbal Limón
d08787e05e rrs/tools/rrs_unique_recipes.py: Also filter -source recipes.
Signed-off-by: Aníbal Limón <anibal.limon@linux.intel.com>
2018-05-04 23:57:52 +12:00
Aníbal Limón
45bd164233 rrs/tools: Change to use get_logger from common.
Signed-off-by: Aníbal Limón <anibal.limon@linux.intel.com>
2018-05-04 23:57:52 +12:00
Aníbal Limón
00045cb6dc rrs/tools/rrs_upgrade_history.py: Only run upgrade detection when detect recipe files in commit.
Signed-off-by: Aníbal Limón <anibal.limon@linux.intel.com>
2018-05-04 23:57:52 +12:00
Aníbal Limón
115bd943d9 rrs/tools/common.py: Handle exception when load_recipes caused by expand PV variable.
Signed-off-by: Aníbal Limón <anibal.limon@linux.intel.com>
2018-05-04 23:57:52 +12:00
Aníbal Limón
12e6d9a9e0 rrs/tools/rrs_unique_recipes.py: Change to use get_logger from common.
Signed-off-by: Aníbal Limón <anibal.limon@linux.intel.com>
2018-05-04 23:57:52 +12:00
Aníbal Limón
669d58d191 rrs/tools/common.py: Add get_logger function.
get_logger function creates a stream logger and a rotate
file logger to store logs.

Signed-off-by: Aníbal Limón <anibal.limon@linux.intel.com>
2018-05-04 23:57:52 +12:00
Aníbal Limón
5a2ea7713b rrs/tools/rrs_upgrade_history.py: Improve version detection.
Use load_recipes from common instead of parse PV by hand.

Signed-off-by: Aníbal Limón <anibal.limon@linux.intel.com>
2018-05-04 23:57:52 +12:00
Aníbal Limón
a04ce69942 rrs/tools/common.py: Add load_recipes function.
load_recipes function setup tinfoil and parses recipe for given
directory.

Signed-off-by: Aníbal Limón <anibal.limon@linux.intel.com>
2018-05-04 23:57:52 +12:00
Aníbal Limón
e10543ce9f rrs/tools/rrs_unique_recipes.py: Remove wrong info at top comment.
Signed-off-by: Aníbal Limón <anibal.limon@linux.intel.com>
2018-05-04 23:57:52 +12:00
Aníbal Limón
fd01d9c35f rrs/tools: Add script for send email to maintainers.
Signed-off-by: Aníbal Limón <anibal.limon@linux.intel.com>
2018-05-04 23:57:52 +12:00
Aníbal Limón
bdde6f5c1b rrs/tools/rrs_: Reorder imports.
Signed-off-by: Aníbal Limón <anibal.limon@linux.intel.com>
2018-05-04 23:57:52 +12:00
Aníbal Limón
dc0fd18ca1 rrs/tools/rrs_upgrade_history.py: Add script for update the recipe upgrades history.
Signed-off-by: Aníbal Limón <anibal.limon@linux.intel.com>
2018-05-04 23:57:52 +12:00
Aníbal Limón
69d71f80b0 rrs/tools/rrs_unique_recipes.py: Add script for filter recipes.
In the RRS only one recipe (PN) per layerbranch makes sense because
the RRS reports the current status of the recipe.

Signed-off-by: Aníbal Limón <anibal.limon@linux.intel.com>
2018-05-04 23:57:52 +12:00
Aníbal Limón
1208f6bc67 rrs/tools/common.py: Add get_pv_type function.
get_pv_type returns the type of pv can be git, svn, hg,
etc.

Signed-off-by: Aníbal Limón <anibal.limon@linux.intel.com>
2018-05-04 23:57:52 +12:00
Aníbal Limón
607e5f141b rrs/tools: Add recipe maintainer history script.
This script creates the manianter history of recipes based on
maintainer.inc file.

Signed-off-by: Aníbal Limón <anibal.limon@linux.intel.com>
2018-05-04 23:57:52 +12:00
Aníbal Limón
31c2120edd rrs/tools/common.py: Add library with common functionality.
Signed-off-by: Aníbal Limón <anibal.limon@linux.intel.com>
2018-05-04 23:57:52 +12:00