The base configuration needs key expansion and anon python execution,
the parsed configurations do not. Fix this consistently, its been
broken and causing double key expansion for a while, only relised
when we started double anonymous python exeution too.
(Bitbake rev: 6138897de5ac6becf3bff56ce7a78f3ec208fcdf)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The output of bitbake -e can differ from what actually is used due
to anonymous python making changes to the data store. Execute any
anonymous python added in the base configuration to make things
more consistent.
(Bitbake rev: bcdc2f73e3b4a10b1e479c2891f251d9507a9e30)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
This allows us to call this code from other contexts without
duplicating it.
(Bitbake rev: c6be487f9bd5d95915f2495d555b9f539adb1d44)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
If a standalone tasks adds a dependency on X:do_build, the code in runqueue would
currently remove it if that do_build was part of an image recipe which uses
recrdeptask on do_build.
Such individual tasks shouldn't do this, therefore tweak the recursive reference code
to only process recurseive tasks, not all tasks.
(Bitbake rev: 4cfca360891e1ed876a9c19487b4f6210686af26)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
In order to allow users to manually populate the download directory with
valid content change the assumption that missing the donestamp file
means unfetched content.
This allows users to populate the download dir without needing to create
dummy .done files such that a user does not need a PREMIRROR when using
BB_NO_NETWORK to provide valid content files in the download directory.
To ensure the correct result this change also fails first if the
localpath does not exist. This prevents further parts of the function
attempting to calculating the checksum on non-existent files. This also
fixes some edge conditions around where if the donestamp exists but the
localpath does not it returns, and did not remove the donestamp.
Also added test cases to cover this use case and additional use cases
where for example the fetcher does not support checksums.
(Bitbake rev: a335dbbb65d5b56e71d98cf3e4fa9bfbec1dcde6)
Signed-off-by: Nathan Rossi <nathan@nathanrossi.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
At least the cli-color node module has dependencies that have
cyclic dependency among themselves. npm.py is prepared to deal
with such a case but the condition is handled only for downloading
or not a dependency again, but then it goes checking the its
dependency which causes an infinite loop in _getdependencies().
Make this function simply return when a dependency is already
downloaded and only download and check its dependencies when not.
(Bitbake rev: 545540420112992e53f4a83104af10452df168d0)
Signed-off-by: Zoltán Böszörményi <zboszor@pr.hu>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The resolve_file() calls mark_dependency(), so the one which calls
resolve_file() doesn't need call mark_dependency() again.
(Bitbake rev: 4682571107323a39b42cd9ec8ee67419e7f15acc)
Signed-off-by: Robert Yang <liezhi.yang@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* Print message when cachefile is found/not can help debug.
* Update "Using cache in" to "Cache dir:" since it was the same as the debug
message of "codeparser & file checksum caches", which caused confusion. And
whether the cache file will be used or not is still unknown at that time, so
just print the cache dir.
(Bitbake rev: c8d3a2016f432e8ed9e99d9c28850149ab6fd6d8)
Signed-off-by: Robert Yang <liezhi.yang@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
It should be live/exited with server rather than cooker, fixed:
$ bitbake --server-only -T -1
Set MACHINE = "qemux86" in conf/local.conf
$ bitbake quilt
Set MACHINE = "qemuppc" in conf/local.conf
$ bitbake quilt
[snip]
ERROR: When reparsing /workspace1/lyang1/poky/meta/recipes-connectivity/openssl/openssl_1.0.2m.bb.do_package, the basehash value changed from c216f7f4fdd3cf4a0b10b975a636426c to d5a8e9431ab261381752d7a64c7b2fa9. The metadata is not deterministic and this needs to be fixed.
[snip]
This is because the server doesn't know local.conf is changed since the
notifiers are stopped, so it doesn't reparse, and then we would get the errors,
let the notifiers live/exited with server can fix the problem.
(Bitbake rev: a6a641cb9c5f3abe901b150da915372e295383d7)
Signed-off-by: Robert Yang <liezhi.yang@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The old code couldn't handle nestled layers correctly, e.g.:
parent_layer/sub_layer/foo.bb
Note there are two layers, parent_layer and sub_layer.
And in parent_layer/conf/layer.conf:
BBFILE_PATTERN_parent_layer = ""^${LAYERDIR}/"
This setting is incorrect since it also matches parent_layer/sub_layer/foo.bb,
so it warns that no files matched sub_layer, this is the expected behavior, but
it doesn't warn when there is a parent_layer/sub_layer/bar.bbappend, this was
incorrect since the bbappend is also matched by BBFILE_PATTERN_parent_layer, it
should warn and let the user fix the problem. Check the bbappend in already
"matched set" before return it as matched by "unmatched set" can fix the problem.
(Bitbake rev: ec90245d28e52ea718d2ce084eb304cdc4355c9c)
Signed-off-by: Robert Yang <liezhi.yang@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
There is nothing to do when BBFILE_PATTERN is empty.
(Bitbake rev: b7da31641c24c53d47ea45a7119d1bd353011b39)
Signed-off-by: Robert Yang <liezhi.yang@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Move the code that existed in tests/fetch.py for determining the path to
'git-make-shallow' into the git module and reference it.
This ensures that 'git-make-shallow' is always available and the desired
version regardless of the path variable or whether git exposes the
command.
(Bitbake rev: 6b508ab8fd5aa796c1c00c970e81e5e93f84d35d)
Signed-off-by: Nathan Rossi <nathan@nathanrossi.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The runall commandline option was confusing people. There are in fact two
different behaviours people may want.
a) For a given target (or set of targets) look through the task graph and
run task X only if its present and would have been built.
b) For a given target (or set of targets) look through the task graph and
run task X if any recipe in the taskgraph has such a target even if it wasn't
in the original task graph.
I've decided to interpret the existing "runall" option as b), even if right
now if behaves like a). For a), which is a valid use case, this patch adds
a "runonly" option.
With both behaviours present, I'm hoping we can then kill off the "fetchall",
"checkuriall" and other tasks from OE metadata and replace them with this
option. This would significantly speed up task graph processing.
(Deleting the checkuriall and fetchall tasks takes "bitbake core-image-sato -g"
from 22s to 8s).
(Bitbake rev: 546a662c877b2d3af35e3996950582ed2df41fe4)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
This is a performance sensitive piece of code and the shear number
of recursive loops is causing a significant and unscalable performance
pain point.
This change moves to a two step approach, firstly generating a list of recursive
dependencies for any task, then applying this to the recursive tasks, iterating
over things until no further dependencies are added.
It was noticed an optimisation is possible and the list of recursive tasks need not
contain the taskname, only the base task id. This allows a significant performance
improvement and limits the size of the resursive task lists, improving speed.
(Bitbake rev: eba738ac5672556eaab4f3374c8025c322761c4a)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
We can optimise the loops slightly so we only process given substrings
once rather than many times. This means expanding out add_resolved_dependencies.
Also add a function which allows replacement of the task element of a
task id, reducing the amount of string handling we're doing in a performance
critical loop.
Its also clear that later code adds to the tasks depends so we don't need
to add .depends() to extradeps at the start.
(Bitbake rev: 4ad281224e92b5f94e3a9c17e8898ec8f1086cdc)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Currently we only run through the recidepends/recrdepends code once. This
means that we can miss some expansions of dependency trees where one
rec{r,i}depends tasks depends on another rec{r,i}depends task.
In reality we need to iterate over the data until we stop adding
dependencies.
In doing this we can't show quite so granular progress information since
we don't know how many times we'll need to do this.
This does slow down the runqueue prepare phase however some optimisations
are possible and can be handled in subsequent patches.
This fix means some missing dependencies, such as:
<image>:do_fetchall -> <image>:do_rootfs -> <pkgs>:do_package_write_X
-> <ca-certs>:do_package_write_X -> debianutils-native
(via PAKAGE_WRITE_DEPS)
are now found/added.
[YOCTO #12510]
(Bitbake rev: aec2f07d56a19b97b6515897532b113cdead8338)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
We've noticed issues on our infrastucture iterating over the many
tag/branch/head reference files that some git repositories may contain.
By issuing the pack-refs command, we move these all to a single file
which speeds up operations with the mirror repos in the downloads
directory in general.
(Bitbake rev: f8126aaf774186a6eaf0bd4067b89c074594886c)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
This means two different graph files can easily be compared, currently
you'd have to sort them as the output is randomized.
(Bitbake rev: 5f7c6ec785f70beb1a4a1bbc0eb83cfa6cd7740d)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
If a fetcher, e.g., git, is run when pseudo is active it will think it
is running as root. If it in turn uses ssh (as git does), ssh too will
think it is running as root. This will cause it to try to read root's
ssh configuration from /root/.ssh which will fail. If ssh then needs to
ask for credentials it will hang indefinitely as there is nowhere for it
to ask the user for them (and even if there was it would not access the
correct private keys).
The solution to the above is to temporarily disable pseudo while
executing any fetcher commands. There should be no reason for them to be
executed under pseudo anyway so this should not be a problem.
RP Ammendum:
We finally did get more information about how to reproduce this problem,
something needs to trigger bb.fetch2.get_srcrev() in a pseudo context,
for example when AUTOREV is in use or the recipe doesn't have a defined
SRCREV. That SRC_URI needs to be using protocol=ssh. This would trigger
an ls-remote of the remote repo and if that happens under pseudo, the
wrong ssh credentials may be attempted which can hang.
[YOCTO #12464]
(Bitbake rev: ceaca281cafa662aa2385b95641bce309dce843d)
Signed-off-by: Peter Kjellerstedt <peter.kjellerstedt@axis.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The cfg property on events doesn't add much, all code appears to access
"data" at this point. Remove it to clean up the interface.
(Bitbake rev: bd4b9e4460b60f142c3bf346fb04e360e512eaee)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Without this the paths to git-make-shallow are incorrect and cause test
failures if bitbake isn't executed from cwd or PATH.
(Bitbake rev: 643eacb162b8710330ef292bfda21cfeab97f95c)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
This change adds EventClassesTest class to bb/tests/event.py,
including 47 new test cases for the public interfaces of the
bitbake event and related helper classes.
[YOCTO #10773]
(Bitbake rev: ee5fe4431713b8a29bdb424a29460965374b3234)
Signed-off-by: Jair Gonzalez <jair.de.jesus.gonzalez.plascencia@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Test the triggering of bb.event.worker_fire callback.
(Bitbake rev: daa59a2057c811b20d75235526ac6c2079ac6e10)
Signed-off-by: Jair Gonzalez <jair.de.jesus.gonzalez.plascencia@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
This change includes unit tests for the following functions,
helper class and methods in bitbake.event:
- set_eventfilter(func)
- set_UIHmask(handlerNum, level, debug_domains, mask)
- getName(e)
- class UIEventFilter(object)
def __init__(self, level, debug_domains)
def update(self, eventmask, level, debug_domains)
def filter(self, event)
[YOCTO #10773]
(Bitbake rev: 4a19dde704fd0bf262ea991ef530f991a4897d31)
Signed-off-by: Jair Gonzalez <jair.de.jesus.gonzalez.plascencia@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
If for example you deltask do_build but it has recrdeps set, its confusing
to have that list returned when the task no longer exists (same would apply
to deps too if it was set after the deltask).
(Bitbake rev: b7a81cb91e82ba64b63c9153dc161a2ab4696715)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Allow for "SinglePackageInfo" events that do not include package data,
for example OPKGN equal 'lib32-*' or 'lib64-*'.
[YOCTO #12204]
(Bitbake rev: 567f072ff260614cde3da220a40a95d5a8b9ab92)
Signed-off-by: David Reyna <David.Reyna@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
We saw builds where runtime providers were sometimes changing order and the
build result was therefore non-deterministic. For example it could show:
DEBUG: providers for lib32-initd-functions are: ['lib32-lsbinitscripts', 'lib32-initscripts']
or
DEBUG: providers for lib32-initd-functions are: ['lib32-initscripts', 'lib32-lsbinitscripts']
which could cause a test to pass or fail.
This change ensures we don't rely on the random order of dictonaries in
memory and act deterministically.
(Bitbake rev: ebce92bf8d71f8a6e8af1c6cf6ba335faf9d67c8)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
.txz is the same as .tar.xz, and can be found in the wild.
(Bitbake rev: 2ba8a6b25ccc12e7b543e8450121e5311c7a701d)
Signed-off-by: André Draszik <git@andred.net>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The connection cache class uses a dummy file object but it doesn't have a closed
attribute, so we can't use it in a context manager.
(Bitbake rev: 7b072ef91d16331eae11bd60f229ce1f0c175995)
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
If a matching recipe is not found then return Null instead of raising
KeyError because we were blindly using None as a key for pkg_fn.
(Bitbake rev: 431e89e322850a2497157c3c0843da9df6bc9a3e)
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
This is primarily paranoid but ensure we remove any loggers we setup
either directly or indirectly so the initial state is restored after
we exit.
(Bitbake rev: af7d63b1f76fd3f7fa92ed15ae61ca47d9e13472)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Whilst we're likely exiting in this case, clean up the loggers we add
so that in the case of certain server retries there is no possibility
multiple loggers stack up.
(Bitbake rev: 25b7bf6672be66bcbfe5760610dce7d3e866cdcc)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Add in a tets ftp url so we ensure ftp urls contnue to work after the loss
of the ftp.gnu.org ones.
(Bitbake rev: e1e8565b5e19dd3f7ef6e7e41932456adaa3df81)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
As we test multiple URLs in this these tests and one failing abandons the test,
use subtests so all URLs are tested. This should help us identify patterns in
the failing URLs.
(Bitbake rev: c4c4465b32e82d4b6e46a44e776be5039aef6b18)
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The ftp server at ftp.gnu.org is likely to be retired at some point soon
so siwtch over to the http/https services.
This means bitbake-selftest doesn't have ftp test urls, however finding stable
ftp test servers is proving increasingly hard.
(Bitbake rev: 892a08245ddb21a464aeb37d3e32377e99dd7e2b)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
After cleaning deprecated API usage repo fetcher is missing
logger as it was indirectly imported via deprecated bb.data.
Fix this by importing logger directly.
Fixes: 9752fd1c10b8 ("fetch2: don't use deprecated bb.data APIs")
(Bitbake rev: f8e027d26603db2f1fe757dca767ea35d95174c7)
Signed-off-by: Oleksandr Andrushchenko <oleksandr_andrushchenko@epam.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Instead of not even having the test functions if network tests are disabled, use
a custom decorator to mark the network tests and skip them.
(Bitbake rev: cc420f430b1dafd9ca944bea259a564aaab34595)
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Currently if you run bitbake in an invalid directory, the user experience
is poor:
birbake/lib/bb/main.py", line 427, in setup_bitbake
topdir, lock = lockBitbake()
File "./bitbake/lib/bb/main.py", line 494, in lockBitbake
lockfile = topdir + "/bitbake.lock"
TypeError: unsupported operand type(s) for +: 'NoneType' and 'str'
This ensures we exit straight away with a better error message.
[YOCTO #12163]
(Bitbake rev: 562f9ee674a8b392437096422b9cceab9c3cba2e)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
In Python 3.6, glob.glob() was reimplemented to use os.scandir() (which
itself appeared in Python 3.5), thus our monkey patching of os.listdir()
here was no longer effective. The end result was not only that bitbake
wouldn't notice added recipes or bbappends with BB_SERVER_TIMEOUT set
when being run with Python 3.6 (the shipped Python version on Fedora 26
and some other distribution versions), it also broke devtool modify,
devtool upgrade and devtool extract since they rely on the ability to
create a bbappend on the fly and have bitbake pick it up.
To fix it, do the same monkey patching for os.scandir(), which needs to
be conditional upon that actually existing since we have to support
Python 3.4 that doesn't have it. Long term we should probably look for a
better way to handle this that doesn't involve monkey patching Python
library code.
Fixes [YOCTO #12185].
(Bitbake rev: d02e90db32e7ee341c2ba3be79b0627d8796bdd6)
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
When the destination is a directory, building the the destination file
path is always needed. That's because even if the copy fallback is
taken, it's always followed by a rename.
(Bitbake rev: 14c17480827ced2e03c1b62dc839696421fc4de8)
Signed-off-by: Alberto Mardegan <amardegan@luxoft.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
We were seeing cases where we could hit the 5s timeout on large/fast machines
running many different tasks at once. Increase this to 30s since the main
connection timeout path should no longer hit this slow path.
[YOCTO #12116]
(Bitbake rev: 131d4b8a5834781a93ed41e2967d8dcd4d80f29a)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
In collect_bbfiles() we're monkey-patching os.listdir in order to find
which directories to watch, and then undoing that when we're finished -
however if an exception occurred for any reason there was nothing to
ensure the latter occurred. This may not have caused any issues, but as
this kind of thing really ought to be secured using try...finally just
in case, so do that.
(Bitbake rev: 013047484a03185c0ce281c53c1db4949cdc4e69)
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The code that was supposed to watch directories along BBFILES for
creation of new files wasn't working in the case where the directory did
not initially contain any matching files - since in updateCache() we are
passing the directory path to add_filewatch() and the latter function
calls os.path.dirname() on the path on the assumption that it is a file
path, and thus the parent of the directory got watched but not the
directory itself. (If the directory wasn't empty everything worked fine
since add_filewatch() was called elsewhere with the path to one of the
files in that directory, and thus the directory got watched). Add a
parameter to add_filewatch() to tell it we are passing it directory
path(s) rather than file path(s).
(Bitbake rev: 47a34dee08fcc25d896a1bdf16fa86267f0b898f)
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
