Commit Graph

22 Commits

Author SHA1 Message Date
Paul Eggleton
369b03c6db rrs_upstream_history: make more robust
Avoid exceptions / blank versions during recipe upstream crashing the
entire script.

Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
2018-05-04 23:57:53 +12:00
Paul Eggleton
7606664eeb rrs: link maintenance/upstream history to layerbranch
RecipeUpstreamHistory was not linked to the layer it was produced from,
which meant that it wasn't easy to query for a different maintenance
plan (i.e. a different layer) and thus the maintenance plan selection
on the recipe list didn't really work. Add a link field, populate it in
a migration and then make it required.

We had added a link earlier from RecipeMaintainerHistory to LayerBranch
but it was optional; for the same reasons we now populate it and make it
required.

Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
2018-05-04 23:57:53 +12:00
Paul Eggleton
201c74ca0f rrs/tools: add -p/--plan option
Add an option to specify which maintenance plan to operate on (largely
for debugging purposes).

Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
2018-05-04 23:57:53 +12:00
Paul Eggleton
70084bd48d rrs/tools: ensure recipe parsing code deletes temporary dir
load_recipes() was leaving files around in /tmp; on my Fedora system
this eventually resulted in /tmp running out of space which we do not
want.

Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
2018-05-04 23:57:53 +12:00
Paul Eggleton
291f6bfde7 rrs/tools: use layer index lock
We check out different revisions while we do this processing, and so
does the layer index update script, so we shouldn't be allowing both to
run at once or nasty stuff will happen.

Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
2018-05-04 23:57:53 +12:00
Paul Eggleton
000ebf4d59 rrs_upstream_history: use try...finally to ensure tinfoil is shut down
We want tinfoil.shutdown() to be called even if an exception occurs.

Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
2018-05-04 23:57:53 +12:00
Paul Eggleton
50aab7c03a rrs/tools: run all tools scripts with python3
These scripts should all now be run under python 3, so update the
shebangs accordingly.

Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
2018-05-04 23:57:53 +12:00
Paul Eggleton
4ea3834033 rrs_upstream_history: properly handle missing Recipe
If the Recipe object doesn't exist here then an exception will be raised
rather than None being returned, and this will also trigger if multiple
recipes match. This may have never triggered in the past because this
would have been run right after updating all the recipes in the layer and
clearing out duplicates (which we were doing earlier), and thus what is
in the database would match the recipe files in the repository, assuming
no errors occurred during parsing). We can't remove duplicates though so
we need to switch over to using filter() and taking the first recipe.

Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
2018-05-04 23:57:53 +12:00
Paul Eggleton
0c1763bf56 rrs_upstream_history.py: support maintenance plans and remove poky hardcoding
Remove hardcoded references to the poky repository, and process
layerbranches for all enabled maintenance plans.

Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
2018-05-04 23:57:52 +12:00
Paul Eggleton
0bab50462c rrs_upstream_history.py: drop threading
This is never going to work, the only way we can practically do parallel
execution is to run these things in a task which would basically amount
to distrodata's do_checkpkg; we may move to that in future but for now
just drop the threading code.

Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
2018-05-04 23:57:52 +12:00
Paul Eggleton
fd786875c3 rrs/tools: add dry-run option to each script
Add the ability to run the scripts without writing changes back to the
database, for debugging purposes.

Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
2018-05-04 23:57:52 +12:00
Aníbal Limón
510dec2ee8 rrs/tools/rrs_{distros,upstream_history}: Add path for poky meta lib
Due to bitbake client/server changes now the meta path isn't included
to sys.path when load tinfoil.

Signed-off-by: Aníbal Limón <anibal.limon@linux.intel.com>
2018-05-04 23:57:52 +12:00
Aníbal Limón
4c2e9ec1f9 rrs/tools/rrs_upstream_history.py: Change regex variables to tuple
This are causing a failure when access the remote datastore.

File "rrs/rrs/tools/rrs_upstream_history.py", line 169, in <module>
    set_regexes(recipe_data)
  File "rrs/rrs/tools/rrs_upstream_history.py", line 53, in set_regexes
    if any(d.getVar(var, True) for var in variables):
  File "rrs/rrs/tools/rrs_upstream_history.py", line 53, in <genexpr>
    if any(d.getVar(var, True) for var in variables):
  File "/srv/rrs/layers/bitbake/lib/bb/data_smart.py", line 608, in
getVar
    return self.getVarFlag(var, "_content", expand, noweakdefault,
parsing)
  File "/srv/rrs/layers/bitbake/lib/bb/data_smart.py", line 723, in
getVarFlag
    local_var, overridedata = self._findVar(var)
  File "/srv/rrs/layers/bitbake/lib/bb/data_smart.py", line 480, in
_findVar
    return connector.getVar(var)
  File "/srv/rrs/layers/bitbake/lib/bb/tinfoil.py", line 62, in getVar
    value = self.tinfoil.run_command('dataStoreConnectorFindVar',
self.dsindex, name)
  File "/srv/rrs/layers/bitbake/lib/bb/tinfoil.py", line 316, in
run_command
    raise TinfoilCommandFailed(result[1])
bb.tinfoil.TinfoilCommandFailed: Traceback (most recent call last):
  File "/srv/rrs/layers/bitbake/lib/bb/command.py", line 81, in
runCommand
    result = command_method(self, commandline)
  File "/srv/rrs/layers/bitbake/lib/bb/command.py", line 465, in
dataStoreConnectorFindVar
    datastore = command.remotedatastores[dsindex]
  File "/srv/rrs/layers/bitbake/lib/bb/remotedata.py", line 39, in
__getitem__
    return self.datastores[key]
TypeError: unhashable type: 'list'

Signed-off-by: Aníbal Limón <anibal.limon@linux.intel.com>
2018-05-04 23:57:52 +12:00
Aníbal Limón
f8753345c0 rrs/tools/rrs_upstream_history.py: Disable threads due to new tinfoil API
The new tinfoil API don't support pythreads so set to 1 while
tinfoil have support.

Signed-off-by: Aníbal Limón <anibal.limon@linux.intel.com>
2018-05-04 23:57:52 +12:00
Aníbal Limón
d13329be1d rrs/tools: Upgrade to use new API of tinfoil
The new client/server API of tinfoil requires explicit call of
shutdown method to send the event for finalize cooker process.

[YOCTO #10863]

Signed-off-by: Aníbal Limón <anibal.limon@linux.intel.com>
2018-05-04 23:57:52 +12:00
Aníbal Limón
8ed223e13c rrs/tools: Upgrade to use transaction.atomic() in Django 1.6
Django 1.6 provides a context manager for atomic transactions so
update the way for make db transactions.

Signed-off-by: Aníbal Limón <anibal.limon@linux.intel.com>
2018-05-04 23:57:52 +12:00
Aníbal Limón
32391e968a rrs_upstream_history.py: Fix use regexes in packages
Make difference when handling suffixes or preffixes like 'nativesdk-'
when try to use regexes in packages that have suffixes like '-crosssdk'
it can contain the arch into it like binutils-crosssdk-x86_64.

Use split method in suffixes and replace method into preffixes, this
fixes issues with suffixes containing archs at end.

[YOCTO #8102]

Signed-off-by: Aníbal Limón <anibal.limon@linux.intel.com>
2018-05-04 23:57:52 +12:00
Aníbal Limón
d1d108507c rrs_upstream_history.py: Use regexes in SPECIAL_PKGSUFFIX packages
When SPECIAL_PKGSUFFIX packages don't have regexes use it from package
without SPECIAL_PKGSUFFIX.

[YOCTO #8102]

For example: python-native use regex from python if don't have one.

Signed-off-by: Aníbal Limón <anibal.limon@linux.intel.com>
2018-05-04 23:57:52 +12:00
Aníbal Limón
fac5b245d4 rrs/tools: UpstreamHistory only load current recipe files.
Don't load all recipes avaiable in layer because was previous
filtered to have the last version.

Signed-off-by: Aníbal Limón <anibal.limon@linux.intel.com>
2018-05-04 23:57:52 +12:00
Aníbal Limón
ffee2377d1 rrs/tools/rrs_upstream_history.py: Remove unneeded histroy save at end.
Signed-off-by: Aníbal Limón <anibal.limon@linux.intel.com>
2018-05-04 23:57:52 +12:00
Aníbal Limón
1f0935240d rrs/tools: Remove call to tinfoil shutdown method.
Tinfoil shutdown was not accepted as patch into bitbake because
this is a simple solution to a big problem.

Signed-off-by: Aníbal Limón <anibal.limon@linux.intel.com>
2018-05-04 23:57:52 +12:00
Aníbal Limón
e8b258e717 rrs/tools/rrs_upstream_history.py: Add recipe upstream script.
This script generates upstream version information and recipe status.

Signed-off-by: Aníbal Limón <anibal.limon@linux.intel.com>
2018-05-04 23:57:52 +12:00