Since we are only using parse_version for comparison (typically checking
that we are greater than some minimum version for tool or package), one
would think we can use packaging.version.parse as if it was parse_version
Unfortunately, this requires conforming to PEP-440 version definitions,
which does not work for e.g. autotools (2.72d) nor older openssl (1.1.1p).
We rely in these (and to be sure other) cases on the LegacyVersion behavior.
https://packaging.python.org/en/latest/specifications/version-specifiers/#summary-of-differences-from-pkg-resources-parse-version
"This specification purposely restricts the syntax which constitutes a
valid version while pkg_resources.parse_version attempts to provide some
meaning from any arbitrary string."
In order to have the least impact to the overall code, we instead add
packaging_legacy to requirements.txt and use packaging_legacy.version.parse
as if it was parse_version.
https://pypi.org/project/packaging-legacy/https://github.com/pypa/packaging/pull/407
Since pypi.org itself is depending on packaging_legacy (in fact, a pypi dev
developed the package), we can expect it to be supported for quite some time.
https://github.com/pypi/warehouse/pull/13500
[YOCTO #15348]
Signed-off-by: Tim Orling <tim.orling@konsulko.com>
'result' is a list of RecipeUpstream objects... drop legacy 'recipe, ru'
syntax. In the current code base, result.append((recipe, ru)) was
appending a tuple of the last value of 'recipe' (in 'for recipe_qry:' under
'for maintplan in maintplans:') and the RecipeUpstream object.
Fixes:
AttributeError: 'tuple' object has no attribute 'history'
Signed-off-by: Tim Orling <tim.orling@konsulko.com>
The get_recipe_pv_without_srcpv function was renamed to
get_recipe_pv_with_pfx_sfx in:
84794b59 lib/oe/recipeutils.py: accommodate SRCPV being optional and deprecated in version check regex
Try to import/call the old method and fail over to the new method.
Signed-off-by: Tim Orling <tim.orling@konsulko.com>
Fixes:
Traceback (most recent call last):
File "/opt/layerindex/./rrs/tools/rrs_upstream_history.py", line 210, in <module>
set_regexes(recipe_data)
File "/opt/layerindex/./rrs/tools/rrs_upstream_history.py", line 73, in set_regexes
bb.data.update_data(localdata)
AttributeError: module 'bb.data' has no attribute 'update_data'
bb.data.update_data was dropped in:
584989ed 'data/data_smart/build: Clean up datastore finalize/update_data references'
While we are here, drop some trailing whitespace.
Signed-off-by: Tim Orling <tim.orling@konsulko.com>
When performing rrs_upgrade_history for commit date 8/24/2023,
the 'fetch2: Add new srcrev fetcher API' bitbake commit is not
automatically added.
Fix this by remapping 'cooker: Fix error message' (8/22/2023) to
the tag on 8/24/2023 'fetch2: git: Check if clone directory is
a git repo'.
This allows upgrade history for oe-core to continue across the
8/24/2023 date.
Fixes:
2024-01-08 23:59:07,114: DEBUG: run cmd '['git', 'show', '-s', '--format=%ci']' in /opt/workdir/git___git_openembedded_org_openembedded-core
2024-01-08 23:59:07,215: DEBUG: output: 2023-08-24 13:34:35 +0100
2024-01-08 23:59:07,227: DEBUG: run cmd '['git', 'rev-parse', 'HEAD']' in /opt/workdir/bitbake
2024-01-08 23:59:07,329: DEBUG: output: a06619951a43acb80b80d92e0caac560657ca249
2024-01-08 23:59:07,448: DEBUG: Running "GIT_DIR=/opt/workdir/bitbake/.git git merge-base --is-ancestor 87104b6a167188921da157c7dba45938849fb22a HEAD"
NOTE: Starting bitbake server...
ERROR: Failure expanding variable fetcher_hashes_dummyfunc[vardepvalue], expression was ${@bb.fetch.get_hashvalue(d)} which triggered exception AttributeError: module 'bb.fetch2' has no attribute 'get_hashvalue'
The variable dependency chain for the failure is: fetcher_hashes_dummyfunc[vardepvalue]
Signed-off-by: Tim Orling <tim.orling@konsulko.com>
* Allow the regex for RECIPE_MAINTAINER to use either _ or : override
syntax to allow processing maintainer changes across the change
boundary.
Signed-off-by: Tim Orling <tim.orling@konsulko.com>
I noticed in the branch comparison plain text view ("Plain text" button
in the Tools -> Branch Comparison page) that in current versions of
Firefox the ellipses were coming through corrupted, though they looked
fine in the HTML version, and it turns out this is because I wasn't
specifying a character set encoding. It should be UTF-8, so add a
charset to the content type stating as such for this and other similar
views.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.microsoft.com>
Removed obsolete references to django.core.urlresolvers. Added the
newly required on_delete parameters to foreign key relationships in
models.py and in all migrations.
(Amber Elliot's 2d526f9b0d served as a
reference for these changes).
Signed-off-by: Paul Eggleton <paul.eggleton@linux.microsoft.com>
Added SPDX identifiers to all .py files except those in migrations directory.
Fixes: [YOCTO #13527]
Signed-off-by: Meh Mbeh Ida Delphine <idadelm@gmail.com>
Signed-off-by: Paul Eggleton <bluelightning@bluelightning.org>
The regex we were using here explicitly only matched numeric characters
in version numbers - presumably the assumption was that any non-numeric
characters were not significant. However, for upstream projects such as
OpenSSL and BIND for example, alphabetic characters are an explicit part
of the version number, so if we ignore them then we miss detecting most
of the upgrades. Fix the regex so that that doesn't happen.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
We have at least one instance where two versions of a recipe were added
at the same time and then later one was deleted - sed. We didn't detect
more than one recipe being added and thus the delete was seen as
removing the recipe entirely, causing the recipe to vanish. Fix the
filter so that we see the other addition and adjust the debug printing
so that we can see what type of deletions are occurring.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Going back in OE-Core recipe upgrade history, we kept GPLv2 and GPLv3
versions of a number of recipes around, so this is the source of quite a
few situations where we had multiple versions of recipes with the same
recipe name around. Add means of grouping upgrades by license so that we
can keep these versions separate in the upgrade history instead of
detecting lots of apparent upgrades and downgrades if they are
intermingled.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
By dumping the recipe upgrade data using the mostly same code as the web
application but in plain text format, this tool gives us an easy way to
compare recipe upgrade history data from multiple runs.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Aligning with recent changes in the layer index proper, handle where PV
is not changing but SRCREV is - typically this happens when PV does not
contain ${SRCPV} - ncurses in OE-Core is one example.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
In the "Maintenance Plan" drop-down the maintenance plans point to the
"default" release and milestone, but it was picking the most recently
added record in the database rather than the latest one by date. Use an
order_by() to ensure we get the most recent release/milestone by date
rather than just the most recently added in case they have been added
out-of-order.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
We were picking up lib/bblayers/example.bb in OE-Core, and it's possible
we might add similar templates in future. There shouldn't ever be files
we're interested in under lib/, and in the absence of the ability to
follow BBFILES, just exclude the directory explicitly.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
We need to capture the re-addition properly or the recipe simply won't
show up in the recipe list. Examples from meta-oe:
* psqlodbc removed in ec9e5ed06256ad92c818474cdb490dc0d3a0d0a3 and
added back in 16a6fee6c0455863ed5df15afc49efe8cc617d9c
* libgxim removed in 5dd01c5175f518658d8ee5627ede4f593111b872 and
added back in af602920594a9cc2e9b397fe311fda7f531be7f3
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
In bitbake commit 5796ed550d127853808f38257f8dcc8c1cf59342, line
numbering functionality was improved with the starting line number for
python functions being stored in a "lineno" varflag; however, mapped
functions (using EXPORT_FUNCTIONS) did not have a line number set, which
caused parse failures. This bug was not fixed until
547128731e62b36d2271c4390b3fee2b16c535dc so we should be avoiding any
bitbake commit inside that range.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
In meta-oe there are two commits (d91f92cf04 and 57492d40b5) which have
the same commit date and thus don't deterministically order; the result
was that the mercurial-native recipe might or might not show up. Add id
to the order_by to make it deterministic.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
In OE-Core commit 309a02931779f32d1139cc1169a039cbe4638706, a reference
to BBINCLUDED was added to HOSTTOOLS in conf/bitbake.conf, however when
we use tinfoil to parse this BBINCLUDED is not set (probably too early)
and the result is an immediate parsing failure. The issue was eventually
fixed in 40a904bf8bc1279c3da0893c003f740f1d2066c2 however there are some
commits in this range that we care about, so within this range we hack
bitbake.conf to have a default for BBINCLUDED since it's an easy
workaround.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Sometimes in the past it has been desirable to create a shared .inc
file from a recipe, e.g. d5a95dc8985a42bb7e50bc4e7dc6b012d711ff08 in
OE-Core for tzdata. Git detects this type of change as a rename of the
.bb to a .inc with some changes, and an addition of a new .bb with new
content; however we want to treat it as a change to the .bb file and
ignore the .inc, otherwise it can look like the recipe was renamed and
the history becomes broken (it wasn't, the recipe name stayed the same).
Detect this situation and handle it properly.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Add an option to stop at a particular commit (so we can then repeat a
specific commit afterwards easily for debugging purposes).
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Record the previous version in RecipeUpgrades, and use it to more
accurately record upgrades where there are multiple versions present at
a given time (common with e.g. kernel recipes).
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Make it possible to re-collect all the history for a given path.
(Typically this would only be used for debugging, as it saves time if
you are trying to correct an issue with upgrade data collection for a
single recipe.)
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
If we ever want to analyse the upgrade chain later on then we need to
avoid overwriting the paths when we identify a moved recipe - instead,
store a "move" upgrade record (not shown in the UI) that we can later
pick up when we are going through and deleting.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Silently catching all exceptions like this is evil. Use .first() and
check the return instead.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Version downgrades (or what appear to be downgrades) do occasionally
happen, and if they did then the RRS was previously simply ignoring
them, resulting in the latest version being reported incorrectly.
Allow downgrades to be recorded as an upgrade with a new 'Downgrade'
type option set, and display a label on such records in the UI.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
There are a range of commits in OE-Core which cause parsing problems;
map them to the one that fixes it in order to avoid the problem. (This
will only be done if we're dealing with OE-Core as a dependency, rather
than the actual layer we're parsing).
(The second set are some commits during the python 3 conversion time.)
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Now that we're using RecipeSymbols we have the complete list of recipes
that ever existed in a layer. We only want to see the ones that are
valid for the selected milestone, so when a recipe gets deleted (or
renamed or moved outside of the layer subdirectory, if any) we need to
record that - do so using a RecipeUpgrade record with a new field
upgrade_type set to 'R'. Additionally we need to store the file path so
that deletion events (where we don't parse the contents of the recipe,
thus we don't have PN) are easy to match up with RecipeUpgrade records;
naturally we need to keep the paths "up-to-date" when we notice recipe
files being moved around.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Recipes come and go over time, so when a recipe gets deleted the history
for it goes away, which means that if you look back in time you do not
see an accurate picture - you only see the subset of recipes that are
currently present. Introduce an indirection between recipes and history
that allows for old recipes to persist (mostly in name only).
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
It's best practice for security reasons to use shell=False and pass
command line arguments as a list; it also avoids some pain with
escaping, so let's use it everywhere we can (in fact we're only left
with one place in layerindex/tasks.py where we now pass shell=True).
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
The original TableSorter is unmaintained, move to the latest version of
the active fork:
https://mottie.github.io/tablesorter/docs/
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
This should have been added in 2c8f979f9c
to make it possible to delete an account that is the administrator for a
maintenance plan without removing the maintenance plan.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
If you had more than one layer on a maintenance plan, only the recipes
from the last layer were shown because we were using a recipe list
variable from the last iteration. The fix just turned out to be to
indent the code that read from that list so it runs every iteration
instead of at the end.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
For security reasons it's best practice to use parameters to pass values
into SQL statements and not substitute them in as strings (with Django's
database API the distinction is subtle, but we pass in the parameters in
a second list parameter instead of using % to substitute them before
passing the query in).
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
The recipe upstream history can't be reconstructed, so we need to be
able to export it out of an old database an import it into the new one.
There's also a "remove-duplicates" function; whilst not strictly
necessary it did help us to remove duplicate records resulting from
running the import twice (due to a bug) and may be useful in future
under similar circumstances.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
The revision that happens to be checked out at the moment could be
anything and might not actually have a maintainers.inc, so check out the
proper branch before looking for it.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Best practices state that you should use a mechanism that ensures files
get closed in case of any error, so let's do that.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Errors deleting bitbake.sock and bitbake.lock have been observed when
shutting down tinfoil at the end of some of these scripts. Move the code
used in the main layer index update script to a function in utils.py and
use it everywhere in order to avoid the issue.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
