Checking out a revision in the bitbake/layer repositories is something
we are doing in a few places, so add a checkout_repo() function that
does this, ensuring that we don't get tripped up by any junk files,
and avoids checking out if the repository is already at the desired
revision (thus avoiding the clean operation and e.g. preserving any
.pyc files that aren't stale and would speed things up a little). Note
that we do the clean before checking out in case there are untracked
files that are tracked in the commit we are checking out.
In addition to adding this function, change the existing code that we
use in the update script to check out a layer use the new function.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Sometimes we get back UTF-8 characters (particularly when picking up
names from git commits), and the ascii codec will error out if that
happens, so switch over to utf-8.
We can as a result remove the decode() calls from the bulk change view.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
We have an update.py running periodically in background, but we also
need to run it manually, for example, run it to update actual_branch,
the manually run usually failed because can't get lockfile, we have to
run it again and again. A timeout option helps a lot in such a case. Now
the following command can make sure we can run the command successfully:
$ update.py -b master -a actual_branch -t 2000
Signed-off-by: Robert Yang <liezhi.yang@windriver.com>
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
The obsolete dependency is the one which is in database but not in
conf/layer.conf anymore. The old code had a problem for newly created
layerbranch, the new layerbranch has no dependencies, so no need remove. And it
had a side effect was that when need_remove was cleaned up, it would be set
again in the next for loop, thus might wrongly remove dependencies. This patch
can fix the problem.
Signed-off-by: Robert Yang <liezhi.yang@windriver.com>
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
If a log level is set on the command line with -q/-d,
set tinfoil's log level to the appropriate log level.
Fixes [YOCTO #11931]
Signed-off-by: Amanda Brindle <amanda.r.brindle@intel.com>
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
* Problems
The update.py couldn't handle new (not only new branch, in fact, but also
existing branches, see below for more info) branch well, for example, there are
3 layers: layer_A, layer_B and layer_C, and create new branch "branch_1" for
them, and they have depends:
layer_A -> layer_B -> layer_C
The "->" means depends on.
Then run "update.py -b branch_1", there would be errors like:
ERROR: Dependency layer_B of layer_A does not have branch record for branch branch_1
Though update.py runs "update_layer.py" twice, but it didn't help since
layerbranch was None when it was failed to create in the first run.
The reason is if update.py updates layer_A firstly, it would fail since it
can't find layer_B:branch_1 in database (not added to database yet), similarly,
if add layer_B before layer_C, it would also fail. Only layer_C can be added
(assume it has no dependencies). So we have to re-run update.py again and again
to make it work, here we may have to run update.py 3 times, and more runs are
needed if the dependency chain is longer.
* Solutions:
Make update.py pass layers orderly to update_layer.py according to dependencies
can fix the problem, we can get intial dependencies from tinfoil, add an option
"-i, --initial" to update_layer.py to get them.
Not only new branch, but also existing branches may have the problem, because
collections and dependencies maybe changed in the coming update, so we can't
trust database when the layer is going to be updated, for example, if there are
10 layers in database, and 3 of them will be updated (-l layer1,layer2,layer3),
then we can not use the 3 layers' collections data from database, we need get
them from tinfoil.
* Performance improvement:
It should be the same as before in theory, I have tested it with 97 layers:
- Before: 11m6.472s, but only 75 layers were added, 22 ones were failed, I
have to re-run update.py again and again (maybe 4 times to make all of
them added). So:
(11 * 60 + 6)/75*97/60 = 14m35s
- Now 12m10.350s, all the layers are added in the first run.
So about 2 minutes are saved.
Signed-off-by: Robert Yang <liezhi.yang@windriver.com>
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Fixed:
1) set LAYERDEPENDS_openembedded-layer = "core"
2) $ "update.py -l meta-oe -b master"
Check from web, its dependency is "openembedded-core"
3) Change LAYERDEPENDS_openembedded-layer = "foo"
4) Run "update.py -l meta-oe -b master"
5) Check from web, its dependency is "openembedded-core and foo", this might
be incorrect, now if set REMOVE_LAYER_DEPENDENCIES to true, the old
dependency openembedded-core will be removed, the default is False which
prints warnings to notify users.
And also the existing checking should filter(required=required), otherwise it
can't work well when a layer is in both depends and recommends, this can't
happen in a normal case, but it would surprise the user when this happens.
Signed-off-by: Robert Yang <liezhi.yang@windriver.com>
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Add an option "-a" to update actual branch for layer and bitbake, it is
useful when there are many layers and need update actual branches
frequently. We only can update them via website without this patch,
which is not fun and easy to make mistakes.
* It works with "-l", and "-l bitbake" means update bitbake branch.
* It requires "-b" to work, and only one branch is supported in a run.
For example:
$ update.py -b master -a branch_20170526
All the layers which have branch master and actual_branch branch_20170526
will be updated to branch_20170526.
$ update.py -b master -l meta-oe -a branch_20170526
Only meta-oe layer will be updated.
$ update.py -b master -l bitbake -a branch_20170526
The bitbake's bitbake_branch will be updated.
Signed-off-by: Robert Yang <liezhi.yang@windriver.com>
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
If we want to parse a configuration file (e.g. a distro conf file) then
we need convenient access to bitbake's conf parsing code, so create a
parse_conf() function to provide that.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
At the moment it's a bit difficult to get update logs out of the
environment in which the update script is being run. In order to make
the logs more accessible, create a LayerUpdate model to record the
output of update_layer.py separately for each layerbranch and tie the
created LayerUpdates together with a single Update model per session.
We provide two ways to look at this - a Tools->Updates page for
logged-in users, and there's also an "Updates" tab on each layer that is
accessible to anyone; which one is useful depends on whether you are
looking at the index as a whole or an individual layer.
Update records older than 30 days are deleted automatically by default.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
If you specified printerr=False we were referring to the output variable
that hadn't been set. Looks like I broke this back in 2013 in
93ce26f21c.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Usage of itertools.chain() that was introduced in
3155206e54 in order to prioritise matches
in the recipe name resulted in recipes showing up twice in the results
if they matched in both the name and the recipe name. Use a custom
chaining function that skips duplicate results in order to fix this.
Fixes [YOCTO #10177].
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Parse layer.conf and add dependencies that are not required from
LAYERRECOMMENDS_<name>. Update the layerindex/template to support
recommends. Uses bitbake parsing code & checks versions.
Signed-off-by: Liam R. Howlett <Liam.Howlett@WindRiver.com>
Added associated migration.
Signed-off-by: Mark Hatle <mark.hatle@windriver.com>
Collection and version will be pulled from the layer.conf if it exists
and dependencies will be resolved by first checking for layers with the
dependency name and then checking for collections.
Signed-off-by: Liam R. Howlett <Liam.Howlett@WindRiver.com>
Added associated migration.
Signed-off-by: Mark Hatle <mark.hatle@windriver.com>
Read dependencies from layer.conf and try to create the LayerDependency
entry by looking up the correct database object. Dependencies are found
by layer name only - no collection support. layer.conf parsing is
handled by the bitbake code.
Once all layers are added, the dependencies have to be rechecked in case
the layers are not added in order.
Signed-off-by: Liam R. Howlett <Liam.Howlett@WindRiver.com>
Convert binary strings to strings and strip leading/trailing whitespace
prior to returning errors and output.
Signed-off-by: Liam R. Howlett <Liam.Howlett@WindRiver.com>
We need to be able to support Python 3 so that we can parse master of
OE-Core with bitbake (which now requires it). This now means the
interface itself and the update script require Python 3.4+.
Part of the implementation for [YOCTO #9704].
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
I'd like to be upgrading to 1.8 but that causes problems with South, and
we're not quite ready to dispense with our existing migrations yet.
Part of the implementation for [YOCTO #9620].
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Avoid the possibility of these two clashing especially when multiple
branches are being used.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
To allow re-use outside of the update script, split out parsing setup
code to a new recipeparse module. Also split out runcmd, get_layer,
get_branch and logger_create functions to a separate utils module.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
