Added SPDX identifiers to all .py files except those in migrations directory.
Fixes: [YOCTO #13527]
Signed-off-by: Meh Mbeh Ida Delphine <idadelm@gmail.com>
Signed-off-by: Paul Eggleton <bluelightning@bluelightning.org>
We have at least one instance where two versions of a recipe were added
at the same time and then later one was deleted - sed. We didn't detect
more than one recipe being added and thus the delete was seen as
removing the recipe entirely, causing the recipe to vanish. Fix the
filter so that we see the other addition and adjust the debug printing
so that we can see what type of deletions are occurring.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Going back in OE-Core recipe upgrade history, we kept GPLv2 and GPLv3
versions of a number of recipes around, so this is the source of quite a
few situations where we had multiple versions of recipes with the same
recipe name around. Add means of grouping upgrades by license so that we
can keep these versions separate in the upgrade history instead of
detecting lots of apparent upgrades and downgrades if they are
intermingled.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Aligning with recent changes in the layer index proper, handle where PV
is not changing but SRCREV is - typically this happens when PV does not
contain ${SRCPV} - ncurses in OE-Core is one example.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
We were picking up lib/bblayers/example.bb in OE-Core, and it's possible
we might add similar templates in future. There shouldn't ever be files
we're interested in under lib/, and in the absence of the ability to
follow BBFILES, just exclude the directory explicitly.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
We need to capture the re-addition properly or the recipe simply won't
show up in the recipe list. Examples from meta-oe:
* psqlodbc removed in ec9e5ed06256ad92c818474cdb490dc0d3a0d0a3 and
added back in 16a6fee6c0455863ed5df15afc49efe8cc617d9c
* libgxim removed in 5dd01c5175f518658d8ee5627ede4f593111b872 and
added back in af602920594a9cc2e9b397fe311fda7f531be7f3
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
In OE-Core commit 309a02931779f32d1139cc1169a039cbe4638706, a reference
to BBINCLUDED was added to HOSTTOOLS in conf/bitbake.conf, however when
we use tinfoil to parse this BBINCLUDED is not set (probably too early)
and the result is an immediate parsing failure. The issue was eventually
fixed in 40a904bf8bc1279c3da0893c003f740f1d2066c2 however there are some
commits in this range that we care about, so within this range we hack
bitbake.conf to have a default for BBINCLUDED since it's an easy
workaround.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Sometimes in the past it has been desirable to create a shared .inc
file from a recipe, e.g. d5a95dc8985a42bb7e50bc4e7dc6b012d711ff08 in
OE-Core for tzdata. Git detects this type of change as a rename of the
.bb to a .inc with some changes, and an addition of a new .bb with new
content; however we want to treat it as a change to the .bb file and
ignore the .inc, otherwise it can look like the recipe was renamed and
the history becomes broken (it wasn't, the recipe name stayed the same).
Detect this situation and handle it properly.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Record the previous version in RecipeUpgrades, and use it to more
accurately record upgrades where there are multiple versions present at
a given time (common with e.g. kernel recipes).
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Make it possible to re-collect all the history for a given path.
(Typically this would only be used for debugging, as it saves time if
you are trying to correct an issue with upgrade data collection for a
single recipe.)
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
If we ever want to analyse the upgrade chain later on then we need to
avoid overwriting the paths when we identify a moved recipe - instead,
store a "move" upgrade record (not shown in the UI) that we can later
pick up when we are going through and deleting.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Silently catching all exceptions like this is evil. Use .first() and
check the return instead.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Version downgrades (or what appear to be downgrades) do occasionally
happen, and if they did then the RRS was previously simply ignoring
them, resulting in the latest version being reported incorrectly.
Allow downgrades to be recorded as an upgrade with a new 'Downgrade'
type option set, and display a label on such records in the UI.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
There are a range of commits in OE-Core which cause parsing problems;
map them to the one that fixes it in order to avoid the problem. (This
will only be done if we're dealing with OE-Core as a dependency, rather
than the actual layer we're parsing).
(The second set are some commits during the python 3 conversion time.)
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Now that we're using RecipeSymbols we have the complete list of recipes
that ever existed in a layer. We only want to see the ones that are
valid for the selected milestone, so when a recipe gets deleted (or
renamed or moved outside of the layer subdirectory, if any) we need to
record that - do so using a RecipeUpgrade record with a new field
upgrade_type set to 'R'. Additionally we need to store the file path so
that deletion events (where we don't parse the contents of the recipe,
thus we don't have PN) are easy to match up with RecipeUpgrade records;
naturally we need to keep the paths "up-to-date" when we notice recipe
files being moved around.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Recipes come and go over time, so when a recipe gets deleted the history
for it goes away, which means that if you look back in time you do not
see an accurate picture - you only see the subset of recipes that are
currently present. Introduce an indirection between recipes and history
that allows for old recipes to persist (mostly in name only).
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
It's best practice for security reasons to use shell=False and pass
command line arguments as a list; it also avoids some pain with
escaping, so let's use it everywhere we can (in fact we're only left
with one place in layerindex/tasks.py where we now pass shell=True).
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Errors deleting bitbake.sock and bitbake.lock have been observed when
shutting down tinfoil at the end of some of these scripts. Move the code
used in the main layer index update script to a function in utils.py and
use it everywhere in order to avoid the issue.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
* Consolidate the code for checking out a repository, using the newly
added utils.checkout_repo() function
* Check out a layer's dependencies, not just the layer itself
* Only check out if the desired revision isn't already checked out
(mostly useful for bitbake which we would otherwise be checking out
much more frequently than necessary since it may not have changed
even if we've moved to a new commit in the layer).
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
If a recipe upgrade only consists of a .inc file changing, we were not
picking it up, since we were only looking specifically for recipes
(.bb). If a .inc file changes, assume all .bb files in the same
directory (if any) should be parsed to look for an upgrade.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
load_recipes() was leaving files around in /tmp; on my Fedora system
this eventually resulted in /tmp running out of space which we do not
want.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
We were parsing recipes that were in the repository but not inside the
actual layer we're dealing with (e.g. we have meta-selftest within the
OE-Core repository, containing a number of recipes that are only
intended for testing purposes and should not be looked at by this
script).
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
If an exception occurs during parsing, let's actually see what kind of
exception it was in the output.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
In OE-Core revision e0531174119bff21e9014b95ed1bbd0e1c01af26 we
accidentally committed a new e2fsprogs recipe with ..bb at the end of
its name instead of .bb. This was fixed immediately afterwards, but when
the RRS hits this commit, it doesn't fail immediately, but the bogus
version "1.43." gets into the database and all subsequent commits
touching the e2fsprogs recipe cause bb.utils.vercmp_part() to blow up
because one of the version parts in the "previous" version in the database
is apparently empty. To work around this and any similar issues, just
reject any change that results in such a broken version string (on the
assumption that it'll be corrected in a subsequent commit and thus we
will get to re-parse the recipe then and therefore not miss the
upgrade.)
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
We should expect multiple matches for layerbranch + pn, so use filter()
instead of get() and take the first id that matches.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Use try...finally to ensure we shut down tinfoil, but since we are
potentially dealing with older bitbake releases when importing older
upgrades, only call shutdown() if it's actually there (and although it's
unlikely, guard against the broken shutdown() in fido as we do in the
main layer index update script).
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
If you want to go back and get history for the earlier releases (krogoth
and previous) then we need to be able to support both python 2 and 3,
which practically means we need the same split for this script as we
have for the main layer index update script.
The catch here is that since we are going back and following the history
of changes forward, we basically need to use the same version of bitbake
that was current at that time. This works except for around the
transition between python 2 to 3 where the metadata lagged behind a bit,
so we need to take that into account. In order to keep things generic we
have a date field on the maintenance plan layer branch that specifies
the date in the metadata where we should switch over to python 3, and
then link to PythonEnvironment records that should be used for python 2
and 3 respectively.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
