It never works since it is in the middle of transaction.atomic() block, and
update.py doesn't need it any more, so remove it.
Signed-off-by: Robert Yang <liezhi.yang@windriver.com>
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
* Problems
The update.py couldn't handle new (not only new branch, in fact, but also
existing branches, see below for more info) branch well, for example, there are
3 layers: layer_A, layer_B and layer_C, and create new branch "branch_1" for
them, and they have depends:
layer_A -> layer_B -> layer_C
The "->" means depends on.
Then run "update.py -b branch_1", there would be errors like:
ERROR: Dependency layer_B of layer_A does not have branch record for branch branch_1
Though update.py runs "update_layer.py" twice, but it didn't help since
layerbranch was None when it was failed to create in the first run.
The reason is if update.py updates layer_A firstly, it would fail since it
can't find layer_B:branch_1 in database (not added to database yet), similarly,
if add layer_B before layer_C, it would also fail. Only layer_C can be added
(assume it has no dependencies). So we have to re-run update.py again and again
to make it work, here we may have to run update.py 3 times, and more runs are
needed if the dependency chain is longer.
* Solutions:
Make update.py pass layers orderly to update_layer.py according to dependencies
can fix the problem, we can get intial dependencies from tinfoil, add an option
"-i, --initial" to update_layer.py to get them.
Not only new branch, but also existing branches may have the problem, because
collections and dependencies maybe changed in the coming update, so we can't
trust database when the layer is going to be updated, for example, if there are
10 layers in database, and 3 of them will be updated (-l layer1,layer2,layer3),
then we can not use the 3 layers' collections data from database, we need get
them from tinfoil.
* Performance improvement:
It should be the same as before in theory, I have tested it with 97 layers:
- Before: 11m6.472s, but only 75 layers were added, 22 ones were failed, I
have to re-run update.py again and again (maybe 4 times to make all of
them added). So:
(11 * 60 + 6)/75*97/60 = 14m35s
- Now 12m10.350s, all the layers are added in the first run.
So about 2 minutes are saved.
Signed-off-by: Robert Yang <liezhi.yang@windriver.com>
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
The _add_dependency() uses:
if layerbranch.collection:
var_name = layerbranch.collection
The layerbranch.collection is none if it is newly created, thus it can't get
LAYERDEPENDS, because what defined in layer.conf is LAYERDEPENDS_<collection>,
but what it would get is LAYERDEPENDS_<layer_name>, this patch can fix the
problem.
Reproducer:
$ python3 update_layer.py -l mete-xfce -b <newbranch> --fullreload -d
It would get None LAYERDEPENDS.
Signed-off-by: Robert Yang <liezhi.yang@windriver.com>
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
The layerbranch is not used in parse_layer(), so remove it.
Signed-off-by: Robert Yang <liezhi.yang@windriver.com>
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
When reading conf/distro/*.conf to create distro records, attempt to
parse the config file and use DISTRO_NAME (if set) to populate the
description field for the record. If that's not set then fall back to
the less commonly used meta-comment that we supported previously.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
The fido branch of OE-Core expects BitBake 1.26.x. Unfortunately that
version had a commit backported which introduced a non-working
tinfoil.shutdown() method, so we can't simply rely on its presence to
determine whether or not it should be called. Use a check on the BitBake
version number instead.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
* We were passing the incorrect path (to the top of the layer repo) if
the layer had a subdirectory, so this doesn't seem to have been able
to work for such layers previously.
* Doing this update in the main update.py script meant that this could
never work across branches requiring a python version change (using
PythonEnvironment records) since the code was running within the same
environment in which update.py was launched - the entire point of the
separation of the two scripts. Move the checking to update_layer.py
and call it separately to perform these updates, splitting out some
common code in order to do so.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Fix a couple of problems with shutting down tinfoil:
1) tinfoil.shutdown() wasn't being called on error because it wasn't
inside a finally: section, thus on error the script hung with bitbake
master (since the tinfoil2 changes) - this is almost certainly a bug
in bitbake but let's handle it here anyway.
2) We need to check whether the tinfoil object actually has a shutdown
attribute since we still want to support updating for older branches
where tinfoil didn't have a shutdown() method.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
If a layer is removed by its subdirectory being deleted then we want to
pick up on that and produce an appropriate error message - so let the
layer_update code do the checking out and verifying things are correct
before trying to parse layer.conf.
This also fixes --nocheckout still checking out a branch.
Additionally, drop some code that gets the layerbranch which we just
retrieved a few lines above.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
In order to keep primary keys from constantly changing, preserve the
existing keys as much as possible.
Signed-off-by: Mark Hatle <mark.hatle@windriver.com>
Signed-off-by: Liam R. Howlett <Liam.Howlett@WindRiver.com>
Parse layer.conf and add dependencies that are not required from
LAYERRECOMMENDS_<name>. Update the layerindex/template to support
recommends. Uses bitbake parsing code & checks versions.
Signed-off-by: Liam R. Howlett <Liam.Howlett@WindRiver.com>
Added associated migration.
Signed-off-by: Mark Hatle <mark.hatle@windriver.com>
Collection and version will be pulled from the layer.conf if it exists
and dependencies will be resolved by first checking for layers with the
dependency name and then checking for collections.
Signed-off-by: Liam R. Howlett <Liam.Howlett@WindRiver.com>
Added associated migration.
Signed-off-by: Mark Hatle <mark.hatle@windriver.com>
Read dependencies from layer.conf and try to create the LayerDependency
entry by looking up the correct database object. Dependencies are found
by layer name only - no collection support. layer.conf parsing is
handled by the bitbake code.
Once all layers are added, the dependencies have to be rechecked in case
the layers are not added in order.
Signed-off-by: Liam R. Howlett <Liam.Howlett@WindRiver.com>
Add the distros to the index. This looks a lot like the machines
and allows users to search for a particular distro.
Signed-off-by: Liam R. Howlett <Liam.Howlett@WindRiver.com>
Added associated migration.
Signed-off-by: Mark Hatle <mark.hatle@windriver.com>
The multiconfig changes broke the calls here to loadDataFull(). To avoid
this being an issue in future, make use of tinfoil's new parse_recipe_file()
function (if available) to isolate the code here from any future changes to
bitbake's internal code.
Part of the fix for [YOCTO #10192].
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
The old transaction API has been removed in Django 1.8 and was
deprecated at 1.6. There's no explicit open transaction, commit or
rollback now - we just wrap the layer operations in
"with transaction.atomic()"; if we need to roll back we just raise a
"dummy" exception.
Part of the implementation for [YOCTO #9620].
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Newer versions of GitPython implement rename detection, which means that
such changes don't show up as adds and deletes anymore and you get a
bunch of ENOENT errors for renamed files. Add some code to explicitly
look for renames and process them appropriately.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Stop using "d" (sometimes multiple levels!) and use proper descriptive
variable names instead.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
In order to try to avoid problems with leaking memory, context bleeding
from one layer to another, and lay the groundwork for supporting
updating all branches in one operation, split the updating of a single
layer out to its own internal script. This means that the tinfoil
instantiation and metadata parsing is in a completely separate process
per layer.
Implements [YOCTO #9647].
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
