Added SPDX identifiers to all .py files except those in migrations directory.
Fixes: [YOCTO #13527]
Signed-off-by: Meh Mbeh Ida Delphine <idadelm@gmail.com>
Signed-off-by: Paul Eggleton <bluelightning@bluelightning.org>
Fixes the bitbakepath variable not being defined with -x/--nofetch
specified.
(Regression introduced in c91372587bbddd4c595d7202e51a8740b787a06e.)
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Add a new BITBAKE_PATH to the settings file to specify the path within the
BITBAKE_REPO_URL where bitbake lives. This is useful when using a combined
repository, such as poky, that contains bitbake, openembedded-core and other
layers.
This change also changes the default path, in the fetch directory, for the
bitbake checkout. It no longer uses the path 'bitbake', but instead uses the
same URL processing as the layer fetching.
There is a side effect that, when using a shared fetch, the branch of the
layer will be used instead of the specified bitbake branch. Generally this
is a reasonable compromise, since in a combined repository bitbake and
openembedded-core component should already match.
Signed-off-by: Mark Hatle <mark.hatle@kernel.crashing.org>
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
It's best practice for security reasons to use shell=False and pass
command line arguments as a list; it also avoids some pain with
escaping, so let's use it everywhere we can (in fact we're only left
with one place in layerindex/tasks.py where we now pass shell=True).
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
It's been reported to me that it's possible in a custom setup to have a
bitbake repo that intentionally doesn't have a master branch, so pick
up the bitbake branch from the master Branch record if it exists and the
bitbake_branch field is set.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
If you update a python2-based branch then the python2-compatible version
of bitbake will be checked out, but we are calling into bitbake's
bb.utils directly here from python 3 and thus you get an error about
commands.getstatusoutput being missing (since that is not available in
python 3 and the old version of bitbake refers to it). To fix this,
check out origin/master in the bitbake repo right before we call the
code in question.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
We don't want to allow any other arguments to be injected into these
commands, so disable the shell and pass the parameters in the form of a
list to prevent that.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
We do not want to be prompting the user for a password during layer
updates or upstream checks, e.g. in the case where a repo requires
authentication, or on github where any fetch of a nonexistent repo
apparently triggers authentication.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
During debugging a parsing issue, we don't really want to continue if a
parsing error occurs, and in that situation I usually end up using
Ctrl+C to exit early. Add an option to exit immediately upon error to
avoid having to do that.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
If an exception occurred during the update then we were managing to save
the update record, but we did not include the exception traceback in the
log for the update. Catch the exception and log it which ensures it gets
captured in the update record and still gets printed as well.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Make layerupdate collection slightly more reliable and make it easier
to see when updates have actually been captured:
* Split layerbranch into separate layer and branch fields, since there
may not be a layerbranch in existence but we might want to log an
error relating to the branch and layer.
* Show all layerupdates on the update detail page, not just those with
log messages
* Record before and after revisions and show these in the update detail
and layerupdate detail (with links)
* Record return code of update_layer process
* Highlight layer updates with a non-zero return code, errors or
warnings in the output on the update detail page
* Show duration on the layerupdate detail page
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
If for some reason update_layer.py does not print out the values we
expect then we shouldn't be throwing a traceback, we should be handling
it gracefully - i.e., print an error and then move on to the next layer.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Make it clearer what this field is for (it should be set for every
comparison layer, so that they don't show up in places they shouldn't).
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Fixed:
Assume there is no master branch in hello layer:
$ update.py -l hello -b master
INFO: Skipping update of layer hello - branch master doesn't exist
This is correct since hello layer doesn't have master branch, but when --nocheckout:
$ update.py -l hello -b master --nocheckout
[snip]
INFO: Sorting layers for branch mater:
WARNING: Cannot find required collections on branch master:
WARNING: hello: LAYERDEPENDS: <snip>
This is incorrect, this patch fixed the problem, now it skips it since the
branch doesn't exists when --nocheckout.
Signed-off-by: Robert Yang <liezhi.yang@windriver.com>
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
When layer_a RECOMMENDS layer_b, try to add layer_b before layer_a, but if
layer_b is not found, still add layer_a.
And print summary error mesage:
$ update.py -b master
ERROR: Issues found on branch master:
openembedded-core: Added without LAYERRECOMMENDS
meta-secure-env: Failed to add since LAYERDEPENDS is not satisfied
Signed-off-by: Robert Yang <liezhi.yang@windriver.com>
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
The utils.setup_django() costs a lot of time, but both update.py and
update_layer.py call it, so move layer validation from update_layer.py
to update.py to avoid calling update_layer.py when possible can save a
lot of time.
Now we don't have to call update_layer.py in the following cases:
* The branch doesn't exist
* The layer is already update to date on specified branch (when no
reload)
* The layer dir or conf/layer.layer doesn't exist
We can save up to 98% time in my testing:
$ update.py -b master --nofetch [--fullreload]
Before Now Reduced
No update: 276s 3.6s 98%
Partial update: 312s 87s 72%
Full reload: 1016s 980s 3%
Note:
* All of the testing are based on --nofetch
* "No update" means all layers on the branch is up-to-date, for
example, when we run it twice, there is no update in the second run,
so we only need about 3s now, which is the most common case when we
use cron to run it per half an hour.
* "Partial update" means part of the layers have been updated.
* "Full reload" means all of the layers have been updated.
Signed-off-by: Robert Yang <liezhi.yang@windriver.com>
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
This makes it easy to see which layers failed. For example:
ERROR: Failed layers on branch master: openembedded-core meta-python
Signed-off-by: Robert Yang <liezhi.yang@windriver.com>
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
We have an update.py running periodically in background, but we also
need to run it manually, for example, run it to update actual_branch,
the manually run usually failed because can't get lockfile, we have to
run it again and again. A timeout option helps a lot in such a case. Now
the following command can make sure we can run the command successfully:
$ update.py -b master -a actual_branch -t 2000
Signed-off-by: Robert Yang <liezhi.yang@windriver.com>
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
The previous commit broke the layer order, e.g.:
A -> B -> C -> D
The algorithm is checking the dependencies one by one, and until we find D, add
D to layerquery_sorted, and add it "collections", the one in "collections"
means it's dependencies are OK, then C, B and A will check against collections,
so that update_layer.py will update them one by one. The previous commit added
A/B/C/D to collections directly, so that when check against it, all the
dependencies are met, thus broke the layer sorting, and then there would be
failures if we pass layer A to update_layer.py before B, C and D (suppose they
are newly to database). This commit fix the problem.
BTW., why I use collections to record the one whose dependencies are matched,
but not directly use layerquery_sorted, it is because collections contains both
the ones to be added/updated and the ones in database, but layerquery_sorted
only contains the ones to be updated/added.
Signed-off-by: Robert Yang <liezhi.yang@windriver.com>
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
If the user hit Ctrl+C during the initial info gathering then it didn't
break out of the loop in update.py, so you had to hit Ctrl+C for as many
layers as were involved in the update. Look for exit code 254 from
update_layer.py and stop if it is returned since that indicates Ctrl+C
has been used.
Additionally, ensure we return exit code 254 and print a message from
the main update script when it is interrupted in this way.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
This can save a lot of time, here is my testing data when PARALLEL_JOBS is 10,
this is the fetch time only, I hacked it to stop when the fetch is done to get
the data (124 layers):
$ update.py -b <branch>
Before: 2m30
Now: 16s
Signed-off-by: Robert Yang <liezhi.yang@windriver.com>
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Fixed:
$ ./update.py -l foo -b new_branch
INFO: Fetching remote repository foo
DEBUG: run cmd 'git fetch -p' in 'foo'
[snip]
DEBUG: run cmd 'git checkout origin/new_branch' in oe-core
ERROR: error: pathspec 'origin/new_branch' did not match any file(s) known to git.
The "new_branch" is newly created, it doesn't exist in local repo since it
isn't fetched, it only fetches the layer specified by -l, so only the foo layer
is fetched. This patch fixes problem.
Signed-off-by: Robert Yang <liezhi.yang@windriver.com>
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Unsatisfied layer dependencies shouldn't error out of the script -
broken metadata isn't supposed to terminate the index update process.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
If you're diagnosing problems with the bitbake server when running the
update script, then you need to be able to look at
bitbake-cookerdaemon.log, but you couldn't do that after the fact
because the temporary directory it gets written out to was being
unconditionally deleted. Add a --keep-temp option which preserves it and
some debug messages to tell you where it is.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
-p, --prune
Before fetching, remove any remote-tracking references that no longer exist on the remote.
Fixed:
$ git push origin :test_branch
$ ./update.py
The test_branch was still in fetched local repo which was incorrect, it should
be gone since it has been removed by upstream.
Signed-off-by: Robert Yang <liezhi.yang@windriver.com>
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
* Problems
The update.py couldn't handle new (not only new branch, in fact, but also
existing branches, see below for more info) branch well, for example, there are
3 layers: layer_A, layer_B and layer_C, and create new branch "branch_1" for
them, and they have depends:
layer_A -> layer_B -> layer_C
The "->" means depends on.
Then run "update.py -b branch_1", there would be errors like:
ERROR: Dependency layer_B of layer_A does not have branch record for branch branch_1
Though update.py runs "update_layer.py" twice, but it didn't help since
layerbranch was None when it was failed to create in the first run.
The reason is if update.py updates layer_A firstly, it would fail since it
can't find layer_B:branch_1 in database (not added to database yet), similarly,
if add layer_B before layer_C, it would also fail. Only layer_C can be added
(assume it has no dependencies). So we have to re-run update.py again and again
to make it work, here we may have to run update.py 3 times, and more runs are
needed if the dependency chain is longer.
* Solutions:
Make update.py pass layers orderly to update_layer.py according to dependencies
can fix the problem, we can get intial dependencies from tinfoil, add an option
"-i, --initial" to update_layer.py to get them.
Not only new branch, but also existing branches may have the problem, because
collections and dependencies maybe changed in the coming update, so we can't
trust database when the layer is going to be updated, for example, if there are
10 layers in database, and 3 of them will be updated (-l layer1,layer2,layer3),
then we can not use the 3 layers' collections data from database, we need get
them from tinfoil.
* Performance improvement:
It should be the same as before in theory, I have tested it with 97 layers:
- Before: 11m6.472s, but only 75 layers were added, 22 ones were failed, I
have to re-run update.py again and again (maybe 4 times to make all of
them added). So:
(11 * 60 + 6)/75*97/60 = 14m35s
- Now 12m10.350s, all the layers are added in the first run.
So about 2 minutes are saved.
Signed-off-by: Robert Yang <liezhi.yang@windriver.com>
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Add an option "-a" to update actual branch for layer and bitbake, it is
useful when there are many layers and need update actual branches
frequently. We only can update them via website without this patch,
which is not fun and easy to make mistakes.
* It works with "-l", and "-l bitbake" means update bitbake branch.
* It requires "-b" to work, and only one branch is supported in a run.
For example:
$ update.py -b master -a branch_20170526
All the layers which have branch master and actual_branch branch_20170526
will be updated to branch_20170526.
$ update.py -b master -l meta-oe -a branch_20170526
Only meta-oe layer will be updated.
$ update.py -b master -l bitbake -a branch_20170526
The bitbake's bitbake_branch will be updated.
Signed-off-by: Robert Yang <liezhi.yang@windriver.com>
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Add a status for a layer indicating it should not be updated. I don't
expect this to be widely used (and is only settable from the admin
interface) but would be useful if you have a legacy sub-layer that you
want to prevent from being visible on certain branches - it will prevent
the update script from doing anything with the layer and thus avoid
branch records from being auto-created on branches where you've deleted
it.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
We can't decode UTF-8 characters byte-by-byte, as soon as we hit a
character that's more than one byte long then we'll fail. Use a reader
object to do it properly. This fixes parsing current meta-angstrom on
master. At the same time, specify errors="surrogateescape" to avoid the
update process dying at this point in case of characters that aren't
valid UTF-8.
Thanks to Jiajie Hu <jiajie.hu@intel.com> who fixed this in devtool's
very similar code.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
* We were passing the incorrect path (to the top of the layer repo) if
the layer had a subdirectory, so this doesn't seem to have been able
to work for such layers previously.
* Doing this update in the main update.py script meant that this could
never work across branches requiring a python version change (using
PythonEnvironment records) since the code was running within the same
environment in which update.py was launched - the entire point of the
separation of the two scripts. Move the checking to update_layer.py
and call it separately to perform these updates, splitting out some
common code in order to do so.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
A minor optimisation - it's not going to change between layers so get it
in the branch loop.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
* Setting of the object has to be before the try: or otherwise the
finally block can get called if that doesn't succeed with the result
that the layerconfparser object won't have a value, which will
trigger an exception
* We shouldn't be using the config_data object after shutting down
tinfoil, so avoid doing that
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
At the moment it's a bit difficult to get update logs out of the
environment in which the update script is being run. In order to make
the logs more accessible, create a LayerUpdate model to record the
output of update_layer.py separately for each layerbranch and tie the
created LayerUpdates together with a single Update model per session.
We provide two ways to look at this - a Tools->Updates page for
logged-in users, and there's also an "Updates" tab on each layer that is
accessible to anyone; which one is useful depends on whether you are
looking at the index as a whole or an individual layer.
Update records older than 30 days are deleted automatically by default.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
update_layer may create the layerbranch record (e.g. for a new branch)
so we should be looking for it after that has been run. Also, we cannot
assume that last_rev will get set because a layer might fail to fetch,
so take that into account as well.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
If a layer is removed by its subdirectory being deleted then we want to
pick up on that and produce an appropriate error message - so let the
layer_update code do the checking out and verifying things are correct
before trying to parse layer.conf.
This also fixes --nocheckout still checking out a branch.
Additionally, drop some code that gets the layerbranch which we just
retrieved a few lines above.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
If the layer failed to fetch, we shouldn't be attempting to update its
contents. (This got broken when we split the update script into two in
c64e4c57a9).
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Parse layer.conf and add dependencies that are not required from
LAYERRECOMMENDS_<name>. Update the layerindex/template to support
recommends. Uses bitbake parsing code & checks versions.
Signed-off-by: Liam R. Howlett <Liam.Howlett@WindRiver.com>
Added associated migration.
Signed-off-by: Mark Hatle <mark.hatle@windriver.com>
Read dependencies from layer.conf and try to create the LayerDependency
entry by looking up the correct database object. Dependencies are found
by layer name only - no collection support. layer.conf parsing is
handled by the bitbake code.
Once all layers are added, the dependencies have to be rechecked in case
the layers are not added in order.
Signed-off-by: Liam R. Howlett <Liam.Howlett@WindRiver.com>
Add a model to support setting a python command and virtualenv per
branch, which allows you to parse master with python3 and krogoth with
python2 for example.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
We need to be able to support Python 3 so that we can parse master of
OE-Core with bitbake (which now requires it). This now means the
interface itself and the update script require Python 3.4+.
Part of the implementation for [YOCTO #9704].
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Allow updating multiple branches, and if no branches are specified,
update all branches that have a new "updates_enabled" flag field set to
True. This avoids the need to have a separate shell script which runs
update.py for each branch (and thus has hardcoded knowledge of each
active branch in the index, i.e. it needs to be kept up-to-date in
addition to the database.)
The migration will default updates_enabled to True for all branches so
if you wish to take advantage of this functionality, the flag will need
to be set to False for any branches that shouldn't be updated.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
In order to try to avoid problems with leaking memory, context bleeding
from one layer to another, and lay the groundwork for supporting
updating all branches in one operation, split the updating of a single
layer out to its own internal script. This means that the tinfoil
instantiation and metadata parsing is in a completely separate process
per layer.
Implements [YOCTO #9647].
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Display blacklist information for recipes in the recipe details,
as well as the recipe search page, and layer page tables. This
information is pulled from the PNBLACKLIST variable. Includes a
hover text containing the reason for blacklist labels.
Changes to Django and Layerindex files:
- models.py, update.py
- migration file
- templates and static content (stylesheet)
[YOCTO #7855]
Signed-off-by: Alex Franco <alejandro.franco@linux.intel.com>
Signed-off-by: Elliot Smith <elliot.smith@intel.com>
The code to populate the inherits information was using a function
designed to get the recipe name from a file path, but unlike recipe
filenames, the underscore isn't treated as special in class filenames
and in fact it's quite common to use underscores there; we were ending
up with items in the inherit list such as "populate" and "rootfs". Use
the standard python os.path functions to do the splitting instead and
avoid this.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
To identify image recipes and provide inheritance data for recipes, an
inherits field was added to the recipe model, and then populated using
refactored data from __inherit_cache. Finally the field was also added
to page templates, along with style changes proposed in attachment, and
an additional style change to display the inherits field as a list in
detailed view. The field skips globally inherited data as proposed.
[YOCTO #7575]
Signed-off-by: Alex Franco <alejandro.franco@linux.intel.com>
Signed-off-by: Aníbal Limón <anibal.limon@linux.intel.com>
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Fix up memory leak fix for bitbake in daisy and earlier which did not
have bb.codeparser.codecache.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
