Since we are only using parse_version for comparison (typically checking
that we are greater than some minimum version for tool or package), one
would think we can use packaging.version.parse as if it was parse_version
Unfortunately, this requires conforming to PEP-440 version definitions,
which does not work for e.g. autotools (2.72d) nor older openssl (1.1.1p).
We rely in these (and to be sure other) cases on the LegacyVersion behavior.
https://packaging.python.org/en/latest/specifications/version-specifiers/#summary-of-differences-from-pkg-resources-parse-version
"This specification purposely restricts the syntax which constitutes a
valid version while pkg_resources.parse_version attempts to provide some
meaning from any arbitrary string."
In order to have the least impact to the overall code, we instead add
packaging_legacy to requirements.txt and use packaging_legacy.version.parse
as if it was parse_version.
https://pypi.org/project/packaging-legacy/https://github.com/pypa/packaging/pull/407
Since pypi.org itself is depending on packaging_legacy (in fact, a pypi dev
developed the package), we can expect it to be supported for quite some time.
https://github.com/pypi/warehouse/pull/13500
[YOCTO #15348]
Signed-off-by: Tim Orling <tim.orling@konsulko.com>
Some layers now have one branch with many supported
LAYERSERIES_COMPAT. If this branch name does not match
one of the stable releases, LayerBranches might not
have been created. When actual_branch is set, it is
only set in a LayerBranch object. We previously
could not update (create) a stable branch with actual_branch
except manually in the admin interface.
Add --force-create option to be used in conjunction with
--actual-branch (which already requires --branch) in the
update.py script. This tells the script to ignore the
fact that no layerbranch exists already.
Add --actual-branch to update_layer.py so that we can create
(and more importantly checkout) an actual_branch for the
given stable --branch.
Update utils.py to allow checking out of actual_branch when
a LayerBranch does not yet exist.
While we are at it, ensure that any Branch that is marked
as no update will be skipped even with --force-create. The
main reason that a Branch has updates disabled is because the
bitbake or python syntax has changed enough to cause exceptions.
This script can now be run with:
./layerindex/update.py \
--layer meta-weird-one \
--branch kirkstone \
--actual-branch=nonstandard \
--force-create
Which will attempt to create a meta-weird-one:kirkstone layerbranch
checked out at the 'nonstandard' branch from that layer's git repo.
This allows layerindex admins to at least populate the database
without tedious creation of layerbranches in the admin interface.
Helps make the "branch mapping" actually work and be useful:
[YOCTO #8008]
Signed-off-by: Tim Orling <tim.orling@konsulko.com>
We have never been checking out dependent layers at the same
release/branch. With the introduction of 'addpylib', this
became obvious due to parsing errors.
Ensure that known LayerDependency objects are checked out at
the expected branch/release. Since openembedded-core has already
been handled elsewhere, we skip it.
[YOCTO #15236]
Signed-off-by: Tim Orling <tim.orling@konsulko.com>
So with honister / current master we can no longer get away with
bypassing BBLAYERS - it now needs to point to the core layer at
minimum. This is fine, we just need to skip parsing layer.conf if we're
parsing the core layer or we get some extra warnings we don't need.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.microsoft.com>
Added SPDX identifiers to all .py files except those in migrations directory.
Fixes: [YOCTO #13527]
Signed-off-by: Meh Mbeh Ida Delphine <idadelm@gmail.com>
Signed-off-by: Paul Eggleton <bluelightning@bluelightning.org>
If a recipe dependency (either static or dynamic) is removed from the
recipe when it is parsed, then we should ensure it gets removed from the
database as well.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Add an extra tool that lets you view all of the recipe dependencies in
a layer. There is also a mode that shows only cross-layer dependencies,
which can be useful to find dependencies on recipes in other layers
that aren't declared in the layer's dependencies (or conversely where a
layer dependency is no longer necessary).
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
For the purposes of the branch comparison function I'm about to add it
would be useful to track the value of SRCREV, so we can see if it has
changed even if PV hasn't.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Add a new BITBAKE_PATH to the settings file to specify the path within the
BITBAKE_REPO_URL where bitbake lives. This is useful when using a combined
repository, such as poky, that contains bitbake, openembedded-core and other
layers.
This change also changes the default path, in the fetch directory, for the
bitbake checkout. It no longer uses the path 'bitbake', but instead uses the
same URL processing as the layer fetching.
There is a side effect that, when using a shared fetch, the branch of the
layer will be used instead of the specified bitbake branch. Generally this
is a reasonable compromise, since in a combined repository bitbake and
openembedded-core component should already match.
Signed-off-by: Mark Hatle <mark.hatle@kernel.crashing.org>
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
This change is to record PE (epoch) and PR (release) values for recipes.
These values will be reflected in the REST API but will not be exposed
in the UI. The aim of this change is to enrich metadata availability to
consumers of the feed.
Signed-off-by: Pranay Mankad <pranaymankad@gmail.com>
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Record the configure script options when importing recipe / package
information so we can display them.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Patches often need to be applied in a specific order. For OE recipes we
were always storing the Patch objects correct order as they are
refreshed every time the recipe itself is refreshed, however for other
distro comparisons, import_otherdistro.py attempts to preserve existing
records, adds new ones and then deletes whatever is left over, which may
result in the order getting messed up over time. To avoid this issue,
record the order next to the patch and set the model meta-info to use
this to sort Patch queries by default.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Errors deleting bitbake.sock and bitbake.lock have been observed when
shutting down tinfoil at the end of some of these scripts. Move the code
used in the main layer index update script to a function in utils.py and
use it everywhere in order to avoid the issue.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
It's not too common but there are instances where people have copied
.inc files into their own layer and modified them, and if you are using
such a layer that could result in unexpected behaviour. In order to get
a handle on when this is being done, collect data about all .inc files
and show duplicates in the Duplicates screen.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
If we want to be able to read in patch information on python2-based
branches (e.g. fido) then we need to use codecs.open() instead of open()
here since python2's open() did not support the encoding parameter.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
The code in recipeparse.setup_layer() was trying to log a warning in the
case where LAYERRECOMMENDS not being satisfied, however there is no
actual logger object in this context. Pass it in via a parameter and
update all callers to pass it.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
In c26604146a I made a fix to change where
the bitbake code writes out bitbake.lock and other files it creates
during parsing, but didn't adequately test it and it turns out our
call to delete the temp directory races against bitbake deleting
bitbake.lock and bitbake.sock. For now the simplest way to deal with
this is to ignore the errors since we don't care about these files,
we just want the temp dir gone.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
During debugging a parsing issue, we don't really want to continue if a
parsing error occurs, and in that situation I usually end up using
Ctrl+C to exit early. Add an option to exit immediately upon error to
avoid having to do that.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
When an exception occurs during the main part of update_layer, we were
catching and printing it but that's not enough - we need to do the
following as well:
* Use logger.error() to print the exception information, so that it gets
logged and highlighted as an error in the layer update
* Exit with a non-zero return code so that update.py knows it has failed
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
If a file is moved (renamed) to a path outside of the layer, e.g.
another layer within a multi-layer repository, then we need to treat it
as a delete. Up until now we were updating the path and continuing, and
then the recipe was also picked up as an add in the other layer, leading
to duplicate recipe entries. I'd noticed these duplicates before but up
until now I'd thought that they were due to another bug we already
fixed, apparently not.
In order to remove these erroneous duplicate entries in existing
databases I have also added a layerindex/tools/fixup_duplicates.py
script. I've also made the -r/--reload option delete them as well.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
If a file is modified and renamed it will show up in both
iter_change_type('M') and iter_change_type('R'), however naturally the
file that will exist will be the b path and not the a one, so we should
be looking at the b path or we will get errors.
FYI you can reproduce this with OE-Core (in a scratch database) using
the following procedure:
1) (in the OE-Core layer directory):
git checkout 59285b324f6d9ed270b0bef209ef5da22a620a83
2) update.py -l openembedded-core -b master -x --nofetch -r --fullreload
3) (in the OE-Core layer directory):
git checkout 086308aa2a5e332de6f00ed397c4a55d132f158f
4) update.py -l openembedded-core -b master -x --nofetch
Without this change you'll see the following error:
ERROR: Unable to read /opt/layerindex/layers/git___git_openembedded_org_openembedded-core/meta/recipes-devtools/python-numpy/python-numpy_1.13.1.bb: Traceback (most recent call last):
File "/opt/layerindex/layers/bitbake/lib/bb/command.py", line 84, in runCommand
result = command_method(self, commandline)
File "/opt/layerindex/layers/bitbake/lib/bb/command.py", line 568, in parseRecipeFile
envdata = bb.cache.parse_recipe(config_data, fn, appendfiles)['']
File "/opt/layerindex/layers/bitbake/lib/bb/cache.py", line 315, in parse_recipe
bb_data = bb.parse.handle(bbfile, bb_data)
File "/opt/layerindex/layers/bitbake/lib/bb/parse/__init__.py", line 117, in handle
return h['handle'](fn, data, include)
File "/opt/layerindex/layers/bitbake/lib/bb/parse/parse_py/BBHandler.py", line 132, in handle
abs_fn = resolve_file(fn, d)
File "/opt/layerindex/layers/bitbake/lib/bb/parse/__init__.py", line 141, in resolve_file
raise IOError(errno.ENOENT, "file %s not found" % fn)
FileNotFoundError: [Errno 2] file /opt/layerindex/layers/git___git_openembedded_org_openembedded-core/meta/recipes-devtools/python-numpy/python-numpy_1.13.1.bb not found
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Fixed:
Assume there is no master branch in hello layer:
$ update.py -l hello -b master
INFO: Skipping update of layer hello - branch master doesn't exist
This is correct since hello layer doesn't have master branch, but when --nocheckout:
$ update.py -l hello -b master --nocheckout
[snip]
INFO: Sorting layers for branch mater:
WARNING: Cannot find required collections on branch master:
WARNING: hello: LAYERDEPENDS: <snip>
This is incorrect, this patch fixed the problem, now it skips it since the
branch doesn't exists when --nocheckout.
Signed-off-by: Robert Yang <liezhi.yang@windriver.com>
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Fixed:
$ update.py -b <new_branch>
[snip]
NOTE: Starting bitbake server...
Traceback (most recent call last):
File "update_layer.py", line 471, in main
utils.setup_core_layer_sys_path(settings, branch.name)
File "/buildarea1/lyang1/layerindex-web/layerindex/utils.py", line 376, in setup_core_layer_sys_path
core_layerdir = os.path.join(core_repodir, core_layerbranch.vcs_subdir)
AttributeError: 'NoneType' object has no attribute 'vcs_subdir'
[snip]
This is because core_layerbranch is not in database yet for completely new
branch, so it is None and we will get the error. Avoid calling
setup_core_layer_sys_path() when "update_layer.py --initial" will fix the
problem.
And also only add core layer's sys path when it is present, since core layer
may not be added yet for completely new branch.
Signed-off-by: Robert Yang <liezhi.yang@windriver.com>
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Collect information about patches applied by a recipe, and record each
patch along with the upstream status, presenting them in the recipe
detail.
Implements [YOCTO #7909].
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Save each remote SRC_URI so we can use these for the recipe reporting
system. This replaces an earlier implementation in the rrs branch where
we simply stored SRC_URI verbatim.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
The utils.setup_django() costs a lot of time, but both update.py and
update_layer.py call it, so move layer validation from update_layer.py
to update.py to avoid calling update_layer.py when possible can save a
lot of time.
Now we don't have to call update_layer.py in the following cases:
* The branch doesn't exist
* The layer is already update to date on specified branch (when no
reload)
* The layer dir or conf/layer.layer doesn't exist
We can save up to 98% time in my testing:
$ update.py -b master --nofetch [--fullreload]
Before Now Reduced
No update: 276s 3.6s 98%
Partial update: 312s 87s 72%
Full reload: 1016s 980s 3%
Note:
* All of the testing are based on --nofetch
* "No update" means all layers on the branch is up-to-date, for
example, when we run it twice, there is no update in the second run,
so we only need about 3s now, which is the most common case when we
use cron to run it per half an hour.
* "Partial update" means part of the layers have been updated.
* "Full reload" means all of the layers have been updated.
Signed-off-by: Robert Yang <liezhi.yang@windriver.com>
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
tinfoil is not needed in cases like the layer is already up-to-date or the
layer is invalid, so only init it when needed.
This can save about 1min when run "update.py -b <branch>" (124 layers).
Signed-off-by: Robert Yang <liezhi.yang@windriver.com>
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
If a database error occurs when we save a recipe (e.g. because a
database-level constraint is voilated) it will mess up the transaction.
Unfortunately that means we need to break out of updating the entire
layer rather than catching the error, because if we do catch it we just
get errors on every update after the initial error; failing early and
giving up on the transaction is a little better in terms of not filling
up the update logs with further useless errors.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Use the more typical handling of the return of get_or_create() and check
if the item was created before saving.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Fix duplicate PackageConfig records being created each time a recipe
is being updated - we need to delete the old ones first.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Added a model for the PACKAGECONFIG variable, which has a one to
many relationship with the Recipe model.
Added models for static build dependencies and dynamic build
dependencies, both of which have a many to many relationship with
the Recipe model.
These objects are created in update_layer.py and are displayed on the
Recipe detail page.
Added a depends search option for recipes, allowing users to search for
recipes that depend on any particular recipe. Use "depends:recipename"
in the recipe search to activate this.
Fixes [YOCTO #12129]
Fixes [YOCTO #11415]
Signed-off-by: Amanda Brindle <amanda.r.brindle@intel.com>
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
If you're diagnosing problems with the bitbake server when running the
update script, then you need to be able to look at
bitbake-cookerdaemon.log, but you couldn't do that after the fact
because the temporary directory it gets written out to was being
unconditionally deleted. Add a --keep-temp option which preserves it and
some debug messages to tell you where it is.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
The branch is not needed any more when it has been removed from the repo, so we
also need remove its layerbranch, otherwise it still can be got from the web,
which causes confusions.
Note, we have to move the location of tinfoil's code to avoid creating tinfoil
when not needed, otherewise, if tinfoil is created (but not needed), and the
program exists earlier before "try ... finally" block, then tinfoil.shutdown()
doesn't run so that it is not shutdown. Move the code back, right before where
it is needed can fix the problem.
Signed-off-by: Robert Yang <liezhi.yang@windriver.com>
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
It doesn't need to be so ahead since we only need it when writing database, and
a following patch will remove layerbranch from database when the branch had
been removed from the repo, it's not easy to do the work in
transaction.atomic() block.
Signed-off-by: Robert Yang <liezhi.yang@windriver.com>
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
It never works since it is in the middle of transaction.atomic() block, and
update.py doesn't need it any more, so remove it.
Signed-off-by: Robert Yang <liezhi.yang@windriver.com>
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
* Problems
The update.py couldn't handle new (not only new branch, in fact, but also
existing branches, see below for more info) branch well, for example, there are
3 layers: layer_A, layer_B and layer_C, and create new branch "branch_1" for
them, and they have depends:
layer_A -> layer_B -> layer_C
The "->" means depends on.
Then run "update.py -b branch_1", there would be errors like:
ERROR: Dependency layer_B of layer_A does not have branch record for branch branch_1
Though update.py runs "update_layer.py" twice, but it didn't help since
layerbranch was None when it was failed to create in the first run.
The reason is if update.py updates layer_A firstly, it would fail since it
can't find layer_B:branch_1 in database (not added to database yet), similarly,
if add layer_B before layer_C, it would also fail. Only layer_C can be added
(assume it has no dependencies). So we have to re-run update.py again and again
to make it work, here we may have to run update.py 3 times, and more runs are
needed if the dependency chain is longer.
* Solutions:
Make update.py pass layers orderly to update_layer.py according to dependencies
can fix the problem, we can get intial dependencies from tinfoil, add an option
"-i, --initial" to update_layer.py to get them.
Not only new branch, but also existing branches may have the problem, because
collections and dependencies maybe changed in the coming update, so we can't
trust database when the layer is going to be updated, for example, if there are
10 layers in database, and 3 of them will be updated (-l layer1,layer2,layer3),
then we can not use the 3 layers' collections data from database, we need get
them from tinfoil.
* Performance improvement:
It should be the same as before in theory, I have tested it with 97 layers:
- Before: 11m6.472s, but only 75 layers were added, 22 ones were failed, I
have to re-run update.py again and again (maybe 4 times to make all of
them added). So:
(11 * 60 + 6)/75*97/60 = 14m35s
- Now 12m10.350s, all the layers are added in the first run.
So about 2 minutes are saved.
Signed-off-by: Robert Yang <liezhi.yang@windriver.com>
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
The _add_dependency() uses:
if layerbranch.collection:
var_name = layerbranch.collection
The layerbranch.collection is none if it is newly created, thus it can't get
LAYERDEPENDS, because what defined in layer.conf is LAYERDEPENDS_<collection>,
but what it would get is LAYERDEPENDS_<layer_name>, this patch can fix the
problem.
Reproducer:
$ python3 update_layer.py -l mete-xfce -b <newbranch> --fullreload -d
It would get None LAYERDEPENDS.
Signed-off-by: Robert Yang <liezhi.yang@windriver.com>
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
The layerbranch is not used in parse_layer(), so remove it.
Signed-off-by: Robert Yang <liezhi.yang@windriver.com>
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
When reading conf/distro/*.conf to create distro records, attempt to
parse the config file and use DISTRO_NAME (if set) to populate the
description field for the record. If that's not set then fall back to
the less commonly used meta-comment that we supported previously.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
The fido branch of OE-Core expects BitBake 1.26.x. Unfortunately that
version had a commit backported which introduced a non-working
tinfoil.shutdown() method, so we can't simply rely on its presence to
determine whether or not it should be called. Use a check on the BitBake
version number instead.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
* We were passing the incorrect path (to the top of the layer repo) if
the layer had a subdirectory, so this doesn't seem to have been able
to work for such layers previously.
* Doing this update in the main update.py script meant that this could
never work across branches requiring a python version change (using
PythonEnvironment records) since the code was running within the same
environment in which update.py was launched - the entire point of the
separation of the two scripts. Move the checking to update_layer.py
and call it separately to perform these updates, splitting out some
common code in order to do so.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Fix a couple of problems with shutting down tinfoil:
1) tinfoil.shutdown() wasn't being called on error because it wasn't
inside a finally: section, thus on error the script hung with bitbake
master (since the tinfoil2 changes) - this is almost certainly a bug
in bitbake but let's handle it here anyway.
2) We need to check whether the tinfoil object actually has a shutdown
attribute since we still want to support updating for older branches
where tinfoil didn't have a shutdown() method.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
