This adds some basic tests for task checksums to ensure that the
checksums:
* change when variables change
* change when active _remove operators are present
* don't change when the _remove operators are not active
* change when an active contains() expression is present
* dont' change a contains() expression isn't active
There is a lot of other functionality which should be added to this
test but its a start.
(Bitbake rev: 5463c16e3619d324aed137f47f93f0997a227d29)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
By creating a standalone function, we can add better functional testing
of this code.
(Bitbake rev: 796a20d24dc18479de1975a37b9e52a5ac75c73f)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Currently remove operations are not being accounted for in the task
checksums. This is a fairly serious oversight and needs to be fixed.
To do so, we need internal data from getVarFlag combined with the
expanded variable data so that only "active" remove operators are
accounted for in the task checksum. We can get this from the new
optional removes attribute in the returned parser object.
The code can then use the data on active remove operators to account
for the removals in task checksum but only when the removal is active.
We have to be careful here not to reference any expanded data since this
may for example contain build paths. This means we can only map back
and reference the unsplit (and hence unexpanded) remove string which may
expand to multiple removal values.
[YOCTO #12913]
(Bitbake rev: 57d2ee17ae83a139a37081eb082e6184fa883581)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The contents of the expand_cache is meant to match the return value of
getVarFlag() but the implementation was mostly in expandWithRefs(). If
an incorrect key was passed to expandWithRefs(), or a variable was only
partially expanded with no remove processing, the cache could become
corrupted.
Move the code to getVarFlag making the data lifecycle very clear, meaning
other calls to expandWithRefs() cannot corrupt the cache.
The expand_cache reset code needs to be moved ahead of any remote data
connectors too, since the expand_cache is now on the local side of the
connection.
(Bitbake rev: a039052f9b680eae53f3f12b7381b945f1d69253)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
This brings _remove handling into line with _append/_prepend with regard
to the parsing flag to getVarFlag.
This is an internal flag and the only times this is used is through getVar
during renameVar operations and when processing ?= operations to see if
a variable is set. In either case we don't need to process remove operations.
Therefore take the minor speedup and skip processing for parsing=True.
[YOCTO #10945]
(Bitbake rev: 6d19eb3226b59922c0f888e33b28443635151501)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Calling getVarFlag with flag=None makes no sense, don't do it. Bitbake
used to silently ignore this, it now warns so avoid the warning.
(Bitbake rev: a68de8ace62eaba23856bfb301efbbe1824322aa)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Provide more information in the case the sources are not found in the
unpack step.
(Bitbake rev: 27a2214bf6f2e7c61bfc422a20959a55f7e0d25d)
Signed-off-by: Urs Fässler <urs.fassler@bbv.ch>
Signed-off-by: Pascal Bach <pascal.bach@siemens.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
A user friendly error is throw when neither the clonedir nor
fullshallow exist. Without the check, a difficult to interpret error
is throw from within the fetch command.
(Bitbake rev: 30cf2506007d25162f0805051212f54c39034ff3)
Signed-off-by: Urs Fässler <urs.fassler@bbv.ch>
Signed-off-by: Pascal Bach <pascal.bach@siemens.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
For better readability and future use, we extract the function which
ensures that a given symlink exists.
(Bitbake rev: 5e69ca56533666a097bb23d09ab673e5c862051c)
Signed-off-by: Urs Fässler <urs.fassler@bbv.ch>
Signed-off-by: Pascal Bach <pascal.bach@siemens.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The mapping of the URLs to the local shallow tarballs is not obvious. For
easier understanding, we add this tests to explicitly showing the mapping.
(Bitbake rev: 05f2ac8e19a027d103921b5ae0070db609a83042)
Signed-off-by: Urs Fässler <urs.fassler@bbv.ch>
Signed-off-by: Pascal Bach <pascal.bach@siemens.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The mapping of the URLs to the local tarballs is not obvious. For easier
understanding, we add this tests to explicitly showing the mapping.
(Bitbake rev: c604d9402b4c9ad7eb5c794ab24f2f348d9719a9)
Signed-off-by: Urs Fässler <urs.fassler@bbv.ch>
Signed-off-by: Pascal Bach <pascal.bach@siemens.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The mapping of the URLs to the local directory is not obvious. For easier
understanding, we add this tests to explicitly showing the mapping.
(Bitbake rev: 5f92682389fee437d6df2ff7718c571b7444e179)
Signed-off-by: Urs Fässler <urs.fassler@bbv.ch>
Signed-off-by: Pascal Bach <pascal.bach@siemens.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Some recent changes broke SRCPV and workaround was introduced
to avoid circular dependency if SRCPV is in PV.
However there is still the same error if SRCPV is in PR.
(Bitbake rev: 05ee4845f925b8528a7ce9cffb4bae425b8fa1e9)
Signed-off-by: Peter Marko <peter.marko@siemens.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Searching in the task explorer requires one to focus the task list. A
readily visible and focused search bar makes searching intuitive.
(Bitbake rev: 43f8a23d56995f552f98a666e86b6cc124e235a4)
Signed-off-by: Tobias Olausson <tol@hms.se>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
We have some slightly odd behaviours with the current implementation of
_remove operations. For example:
TEST = " A B"
TEST_remove = "C"
would trigger TEST to become "A B" even thought it doesn't contain "C".
In particular, this means that an inactive remove operator added in a
bbappend could change the task checksum which is not desireable.
Fix the operation to preserve whitespace, adding new tests to make this
explict and test further corner cases. Also update the manual to match.
(Bitbake rev: c0a23dd9155c50a6b7df796980bc7b612cac7994)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Currently when a variable is renamed, all the variable history is lost.
This makes it hard for users to understand/debug where values came from.
Improve the code so that history is preserved across renamed variables.
(Expanded variable key names are a special case of variable renaming)
(Bitbake rev: 11722c989077a8751a5d0653c523760bf91e6efa)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The environment was being cleared before the UI imports occurred which
caused problems for graphical UIs like taskexp. The full environment was
intended to be available to UI clients and it was only meant to be cleared
for the server/cooker, so tweak the code order so this is the case.
This fixes problems reported for taskexp.
[YOCTO #12670]
(Bitbake rev: ac15028391fdcc3fec2dd0e64a4f763ce63e5449)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Some repositories may specify a relative submodule path. If this happens,
it is our responsibility to use the parents URL (ud) and handle any relative
processing ourselves.
(Bitbake rev: fd9893c338df7828b2c01f1d548aa86dfcf7c50a)
Signed-off-by: Mark Hatle <mark.hatle@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The new fetcher did not run 'git submodule update' recursively.
(Bitbake rev: 377ed943357b6d47d41d84edbf971741f44affa9)
Signed-off-by: Laurent Bonnans <laurent.bonnans@here.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The new gitsm fetcher assumed that submodules were living directly in
the 'modules' directory, whereas they can be arbitrarily nested inside
subdirectories.
Solve it by first creating the parent of the destination directory for
the symlink and copy steps.
(Bitbake rev: 3dbc84e9c572f43209b79f3656d7dc35a6d982ba)
Signed-off-by: Laurent Bonnans <laurent.bonnans@here.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The method 'store_log_event' in 'buildinfohelper.py' always puts log
messages from CLI builds into the backlog but never takes them out.
The "close" method now forces all backlogged CLI events to be registered.
[YOCTO #12813]
(Bitbake rev: 6458cc4234337f551dfe189b6f8800d8da886c24)
Signed-off-by: David Reyna <David.Reyna@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
'qbareclone' in place of 'bareclone'
(Bitbake rev: 90a3181f1397ae05862f4e89a9bbac606e74504e)
Signed-off-by: Laurent Bonnans <laurent.bonnans@here.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The shallow_tarball check is never true due a check on the caller side.
The tarball check is not related to the code on the caller side.
(Bitbake rev: 086eddcf8c7520ff5c52ce2a11ca9bf5b5fe5d7e)
Signed-off-by: Urs Fässler <urs.fassler@bbv.ch>
Signed-off-by: Pascal Bach <pascal.bach@siemens.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
To improve the readability we extract the different scenarios of why
the clonedir needs an update.
(Bitbake rev: 9038e029f4f0ab413727de76c74248cbb3cdc9ea)
Signed-off-by: Urs Fässler <urs.fassler@bbv.ch>
Signed-off-by: Pascal Bach <pascal.bach@siemens.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
GitLab returns a redirect from /foo to /foo.git which older releases of Git
don't handle when cloning. These tests don't clone so the redirect works, but
let's be consistant with the structure of these URLs and add the .git suffix.
(Bitbake rev: d47febbae4ff24be259037f12bafbc14b9e2d6a8)
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The prior fetcher did not know how to work with MIRRORS, and did not
honor BB_NO_NETWORK and similar.
The new fetcher approach recursively calls 'gitsm' download on each
submodule detected. This ensures that it will go throug the
standard download process.
Each downloaded submodule is then 'attached' to the original download in
the 'modules' directory. This mimics the behavior of:
git submodule init
but there is no chance it will contact the network without permission.
It then corrects upstream reference URIs.
The unpack steps simply copies the items from the downloads to the destdir.
Once copied the submodules are connected and we then run:
git submodule update
According to the git documentation, git submodule init can and will modify
the project configuration and may connect to the network. Doing the
work manually prevents this. (This manual process is allowed based
on my reading of the documentation.)
See: https://git-scm.com/book/en/v2/Git-Tools-Submodules
The small change to the existing test is due to this new code always assuming
the code is from a remote system, and not a 'local' repository. If this
assumption proves to be incorrect -- code will need to be added to deal
with local repositories without an upstream URI.
(Bitbake rev: 9c6b39adf9781fa6745f48913a97c859fa37eb5b)
Signed-off-by: Mark Hatle <mark.hatle@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The test case was changing the current directory, but was never restoring it
to the original location. This causes occasional failures in later test cases.
(Bitbake rev: 8c222c45148d1f21c2390d66ddd9d3e33b397f05)
Signed-off-by: Mark Hatle <mark.hatle@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
A nasty corner case leads to a hang when utils.lockfile is called from
oe-core's package-manager:deploy_dir_lock (in turn called from
rootfs:_create further up the call stack) with "name" owned by root
and the user running bitbake has no write access.
Because this code runs under pseudo, the UID and EUID of the bitbake
worker process are 0, so the os.access(dirname, os.W_OK) returns True
i.e. it thinks the path is writable when in fact it's not writable.
Only later when trying to open the file an Exception it thrown because
the OS prohibits writing, but the Exception is ignored and the open is
retried leading to an infinite loop.
So this fix is to not ignore the "Permission Denied" exception.
An alternative fix would be to replace the os.access() call with an
try: open() except() at the beginning of the function.
(Bitbake rev: 0cb64d0f85b41b2fa764baf6ff7ea1b13f95004e)
Signed-off-by: Ioan-Adrian Ratiu <adrian.ratiu@ni.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
At the end of bitbake selftest we see:
sys:1: ResourceWarning: unclosed file <_io.TextIOWrapper name='/dev/null' mode='r+' encoding='UTF-8'>
Clean up the /dev/null handling to drop the unused entry in build.by and
ensure the other open() calls are cleaned up.
NULL was unused since http://git.yoctoproject.org/cgit.cgi/poky/commit/bitbake/lib/bb/build.py?id=4a081b5a52e3d27da8d4b062f3fda292e8d8fb0a
back in 2012.
(Bitbake rev: e72be96cfa9f05fda5f420c7cfa8bcfa9304b884)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Fix the warning:
WARNING: lib/bb/data_smart.py:235: DeprecationWarning: generator 'COWDictMeta.iter' raised StopIteration
for k, v in self.variables.iteritems():
by using return from the generator, not raising StopIteration.
(Bitbake rev: 407d6e07b09123c12c382b4a92107f002c314b05)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Running "bitbake gconf-native -c cleansstate; bitbake core-image-sato:do_populate_sdk"
results in a build where it fails to find gconf-native and fails to build it,
merrily trying to build the SDK without gconf being present.
The issue is the missing setscene tasks are effectively ignored as the later
code in runqueue thinks that since other sstate tasks are present, these
'cover' the missing one. In reality we need to call BB_SETSCENE_DEPVALID
to make that decision. To do that we need a "reduced" setscene dependency
graph which we don't have in main task graph context.
Since that was already done in setscene, we should just assume anything
in the non-covered list needs to be built.
(Bitbake rev: 464d0339add15bc8b4344ddd1e4c49706e3c0a02)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
If the user puts universe on the commandline, they don't really want warnings
so use the new verbnote level instead.
(Bitbake rev: 0c87ade5678e503899e3a6cdda5329f6fc212b63)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
It has become apparant we need a log level which reaches the console but
isn't a warning/error. Add "verbnote" as a way of doing this, behaves as
a note but with a higher priority.
(Bitbake rev: 2076f12cc2f809345108b1606bd6201f41287505)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
There were various problems in the server startup loggin:
a) stdout/stderr were not being flushed before forking which could potentially
duplicate output
b) there were separate buffers for stdout/stderr leading to confusing logs
where the entries could be reordered. This was particularly confusing
due to the separator the logs use to idendify new messages
c) an fd wasn't being closed during server startup meaning if the
server failed to start, the closed fd wasn't detected as it was held
open by the other reference
d) If the pipe was detected as being closed, the code incorrectly retried
server startup
e) The event code would remap stdout/stderr without flushing them, leading
to lose log messages
(Bitbake rev: 0594faa0b52ce5dbd948d836d88617d38d9862d1)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
In the case of a sublayer of an existing layer, where the sublayer and
main layer share a path, the system may not match the paths properly resulting
in:
No bb files matched BBFILE_PATTERN_sublayer '^/path/main/sublayer'
because it has already matched the main layer.
Fix this issue by sorting the collection items based on the pattern, using
longest to shortest. Obviously regex wildcards could still be an issue
but these are typically not used, so this simply fix should work in the
existing cases.
(Bitbake rev: 1787cef7221b88f6920ea70fadaffc117d84c7aa)
Signed-off-by: Mark Hatle <mark.hatle@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Before this fix it is assumed that the removal of the
remote can only fail because there is not remote to remove. This
is a false assumption. Example error which would be ignored:
git -c core.fsyncobjectfiles=0 remote rm origin failed with exit code 1, output:
Note: A branch outside the refs/remotes/ hierarchy was not removed;
to delete it, use:
git branch -d master
error: could not lock config file config
error: Could not remove config section 'remote.origin'
Due to the masking of this error a stranger error will be
presented to the user, because this time we do not mask the
exception:
git -c core.fsyncobjectfiles=0 remote add --mirror=fetch origin https://github.com/ptsneves/tl-wn722.git failed with exit code 128, output:
fatal: remote origin already exists.
The most likely reason that the remote cannot be removed nor
modified is that the DL_DIR/git2 does not have permissions
compatible with the user running bitbake.
This commit fixes:
https://bugzilla.yoctoproject.org/show_bug.cgi?id=12728
(Bitbake rev: 9c86c582a10c9b23abad7d34b6cbf12f7086294d)
Signed-off-by: Paulo Neves <ptsneves@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
In finalize() we save event handlers, register the ones relevant to the
recipe being finalised, trigger events, and then restore the handlers so
that one recipe's custom handlers (actually implemented within a class
inherited by the recipe) do not affect other recipes. However, if an
exception occurs during parsing, the saved handlers were not being
restored. Use a try...finally block to ensure that the handlers are
always restored.
This issue became apparent since in OpenEmbedded-Core we have recently
introduced a find_intercepts() handler for the
bb.event.RecipePreFinalise event in image-postinst-intercepts.bbclass
that images and old-style SDK recipes will end up inheriting. So far it
doesn't seem that the the error has manifested itself in normal builds,
but when parsing OE-Core recipes in the OE layer index it has:
core-image-rt-* image recipes were parsed which in the default
configuration raise SkipRecipe. The next non-image recipe that is parsed
will trigger a real exception, because the find_intercepts() handler is
still registered and gets fired, but in the context of the new recipe
the POSTINST_INTERCEPTS_PATHS variable is not set, and the code in
find_intercepts() is written with the reasonable assumption that that
isn't possible given that the class itself sets a default, and thus it
fails.
(Bitbake rev: e5f1f8fa201774e0c3c554d59b277baa2128708f)
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
In spite of a comment suggesting otherwise, os._exit() does not flush
buffered output from file descriptors before exiting the process like
os.exit() does. This means that any un-flushed output is lost in the
daemon process, in particular the traceback from any thrown exceptions,
making debugging exceptions in the daemon quite difficult.
The solution is to flush stdout and stderr before exiting.
(Bitbake rev: fcc8e1ff53696f78dd64b4ee32f3c433b7a47df0)
Signed-off-by: Joshua Watt <JPEWhacker@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
bb.utils.copyfile is called in a few places with identical src and dst
in order to create an st_nlinks==1 version of the file. That that even
works relies on an implementation detail of copyfile (namely, that it
creates a temporary file and then does a rename). Moreover, it's a waste
of time if the file already has st_nlinks==1.
So create a helper that optimizes away the copy in the st_nlinks==1
case. Of course, this helper relies on the same implementation detail,
but that's now contained within bb.utils itself.
To test that we do at least sometimes hit the no-copy path, I tested
locally with
if sstat[stat.ST_NLINK] == 1:
+ bb.note("Woohoo, 2*%d bytes I/O avoided" % sstat[stat.ST_SIZE])
return True
(and the obvious places in oe-core patched), and the do_package log files
are indeed filled with woohoo notes.
(Bitbake rev: 7ae93cf40ab91965147055100432961436bce46c)
Signed-off-by: Rasmus Villemoes <rasmus.villemoes@prevas.dk>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Opening a file in binary mode and iterating it seems like the simple solution
but will still break on newlines, which for binary files isn't really useful as
the size of the chunks could be huge or tiny.
Instead, let's be a bit more clever: we'll be MD5ing lots of files, but we don't
want to fill up memory: use mmap() to open the file and read the file in 8k
blocks.
(Bitbake rev: a0ac8d67f1471a0c611d691b856fede67efb53f6)
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
In case something goes tragically wrong, catch a request to checksum / and
refuse.
(Bitbake rev: e7cd4c86ef8a2c2bbf068e84c83fdc9e052b6e3d)
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
(Bitbake rev: d49483eb733ee27325349246fa37a84140c0941d)
Signed-off-by: Kai Kang <kai.kang@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
There is an oversight in the current hash validation API in that the
function can't know how many setscene tasks already completed. Rather
than trying to add additional parameters to the function, causing
incompatibilities, store the value in the datastore.
This is useful to allow build status reporting to the user for
figures on sstate reusage and build completion.
(Bitbake rev: ec037d3e49264037b81212f498d98e292ae7c334)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
This patch adds the capability for tasks from different
multiconfigs to depend on one another.
These dependencies can be enabled using the following format:
task[mcdepends] = "multiconfig:FROM-MC:TO-MC:PN:task-to-depend-on"
For the sake of simplicity consider the following example:
Assuming we have set up multiconfig builds, one for qemux86 and one for
qemuarm, named x86 and arm respectively.
Adding the following line to an image recipe (core-image-sato):
do_image[mcdepends] = "multiconfig:x86:arm:core-image-minimal:do_rootfs"
Would state that core-image-sato:do_image from x86 will depend on
core-image-minimal:do_rootfs from arm so it can be executed.
This patch makes modifications to:
- cooker: To glue both multiconfigs in one place and make sure
the dependencies can be provided.
- taskdata: To parse and add a new kind of dependency (mcdepends) to
the taskdata object.
- runqueue: To differentiate tasks from different multiconfigs,
add the specified dependencies to the corresponding tasks, and
create a working runqueue that contains tasks from both multiconfigs.
- siggen: To avoid looking for tasks from different multiconfigs on
objects where they dont belong.
The taskdata objects are still not aware of the concept of multiconfig,
so each object doesnt know which multiconfig its building, hence why
the mcdepends are added to all taskdata objects equally (we really
dont expect many of these), but the actual dependencies are added only
to the required tasks by the runqueue.
(Bitbake rev: da8cb8633504bdc815bdcefc538340b9bce5065d)
Signed-off-by: Alejandro Enedino Hernandez Samaniego <alejandr@xilinx.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Add a callback that lets you modify or remove items in addition to the
current scheme where you can only add or remove. This enables you to for
example replace a layer with a temporary copy (which is what we will use
this for first in OE's oe-selftest).
(Bitbake rev: bfedb4e85a84e817dbe5d8694b8f8fcdd6f2f22a)
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Currently the bitbake-cookerdaemon.log is placed into cwd. This seems like a
bad idea, we can place it in the build directory alongside the lockfile that
represents the server instead.
(Bitbake rev: 1620dbc48ffb2a882371cf9174a7b12648befc8a)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The file_name parameter passed to bb.parse.siggen.invalidate_task
should be a virtual file name instead of a real file name, or else you
will encounter a following error, for instance, when you run:
$ bitbake nativesdk-lzip -c unpack -f
the error arise:
| ERROR: An uncaught exception occurred in runqueue
| if file_name:
| > taintfn = d.stamp[file_name] + '.' + task + '.taint'
| else:
| KeyError: 'virtual:nativesdk:/opt/poky/meta/recipes-extended/lzip/lzip_1.19.bb'
when multilib builds are used on OE.
(Bitbake rev: da37bdad46e11e7ce93ba7a59d58757b769dc16b)
Signed-off-by: Ming Liu <liu.ming50@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
If uri_find contain parameters then original URI parameters should
be checked against parameters from uri_find instead of parameters
from uri_replace.
(Bitbake rev: 8efa7826a61501589afa33eb698c0ab3a622bf2e)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
On high core machines, in do_fetch, it is possible to DDoS your own machine.
A method to limit any arbitrary task type to a certain number of simultaneous
threads is needed. (Similar to how BB_NUMBER_THREADS works in the general
case.) The format of this new limitation is:
do_fetch[number_threads] = "2"
This should be set globally. If it is set in individual recipes it could
result in unpredictable behavior.
Note: a value for number_threads > BB_NUMBER_THREADS will have no effect.
(Bitbake rev: 055865047c63b9c3b213b47a1884924ce0adeda0)
Signed-off-by: Mark Hatle <mark.hatle@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The RunQueueStats:taskCompleted and RunQueueStats:taskSkipped can take
multiple arguments. However, nowehere in bitbake are multiple arguments used.
Change this to match the behavior of the other APIs where it needs to be
called once for each task.
Additionally, these two functions were usually called in tandem, however in
the wrong order. It really doesn't matter as there is no specific preemption
point between the calls. But the taskSkipped should be called first to
increment the 'active' count, and then taskCompleted called to decrement it.
(Bitbake rev: 26d5ea9bb892bd6a2e1fd29a9023e0b0644edc16)
Signed-off-by: Mark Hatle <mark.hatle@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Currently there is quite some variation between the fetchers in terms
of how they determine the subdirectory within DL_DIR and the base
fetch command to run. Some rely on variables being set externally
(e.g. from bitbake.conf in oe-core), some respect these external
variables but provide fallback defaults and some use only hardcoded
internal values. Try to unify the approach used across the various
fetchers.
(Bitbake rev: efd5e35af4b08501c67e8b30f30d9457f6fdf610)
Signed-off-by: Andre McCurdy <armccurdy@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Although the submodules' histories have been fetched during the
do_fetch() phase, the mechanics used to clone the workdir copy
of the repo haven't been transferring the actual .git/modules
directory from the repo fetched into downloads/ during the
fetch task.
Fix that, and for good measure also explicitly tell Git to avoid
hitting the network during do_unpack() of the submodules.
[YOCTO #12739]
(Bitbake rev: 11b6a5d5c1b1bb0ce0c5bb3983610d13a3e8f84a)
Signed-off-by: Matt Hoosier <matt.hoosier@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
When running bitbake command with Python 3.6.5 always result in
import error causing by the change of distutils module.
This patch replaces the method to search executable in PATH by
"/usr/bin/env <command>".
(Bitbake rev: bd9a1b063633af2936ba1dd87b19202424900151)
Signed-off-by: Tzu Hsiang Lin <t9360341@ntut.org.tw>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
For mirrors or premirrors defined like: "http://.*/.* http://somewhere.org"
fetching ends with errors because function fetch2/__init__.py:encodeurl()
creates url like "http://somewhere.orgsomefile.tar.gz".
It happens because function fetch2/__init__.py:decodeurl()
for url "http://somewhere.org" returns
['http', 'somewhere.org', '', '', '', {}]
and then in function fetch2/__init__.py:uri_replace()
variable result_decode will be
['http', 'somewhere.org', 'somefile.tar.gz', '', '', {}]
(because of line: result_decoded[loc] = os.path.join(result_decoded[loc], basename))
for which encodeurl returns "http://somewhere.orgsomefile.tar.gz".
In addition for mirror "http://.*/.* http://somewhere.org/"
everything works fine.
(Bitbake rev: d822ae24ef5485e550804cbd9130ebd73b2aa48e)
Signed-off-by: Jakub Dębski <jdebski@enigma.com.pl>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Updating a local git repo clone currently results in multiple calls
to self._contains_ref(), some of which appear to be redundant and can
be eliminated by minor tweaks to the logic in download().
Also drop redundant calls to os.path.exists(ud.clonedir) before
self.need_update(), since need_update() includes its own built-in
check for the existance of ud.clonedir.
(Bitbake rev: 61b0df5523afc8f805043f3adc9c106690e6f133)
Signed-off-by: Andre McCurdy <armccurdy@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Prevent movefile from falsely setting the source file's owner and
permissions on the destination directory instead of the destination
file when moving between devices.
This bug caused the last file moved into a directory to dictate the
directory's owner and permissions.
(Bitbake rev: 82ea737a0b42a8b53e11c9cde141e9e9c0bd8c40)
Signed-off-by: Mattias Hansson <mattias.hansson@axis.com>
Signed-off-by: Peter Kjellerstedt <peter.kjellerstedt@axis.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
We'd like layers to set this variable so that we know which layers are compatible
with which others, even if the branch is a generic un-updated "master" branch.
Start printing a warning to highlight this issue.
(Bitbake rev: cca81e33b58c390dcf5cc3a31555a43b79177166)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
This was removed in the bitbake repository back in 2010 (bitbake
revision 24857e2ceb405916b0b0b3e75c6c2375a909b9ba) - it's not clear
to me what happened, but for some reason the file remained in the poky
repository. However it does not appear that it has been used since then
and the reasons for removing even more valid now than they were then -
the code has moved on even further, we have memory resident bitbake
server, and if we were to re-implement this we would do it in a
different way. Drop the file and bring us back in sync with the bitbake
repo.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Before, explode_dep_versions2 would sort the OrderedDict before
returning. This function will still sort the OrderedDict by default, but
will now have the option to return the OrderedDict unsorted. This option will
allow us to check if the order of the package list has changed.
(Bitbake rev: 39d6a30a28f66c599e18beddbd847f40dcff623c)
Signed-off-by: Amanda Brindle <amanda.r.brindle@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The base configuration needs key expansion and anon python execution,
the parsed configurations do not. Fix this consistently, its been
broken and causing double key expansion for a while, only relised
when we started double anonymous python exeution too.
(Bitbake rev: 6138897de5ac6becf3bff56ce7a78f3ec208fcdf)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The output of bitbake -e can differ from what actually is used due
to anonymous python making changes to the data store. Execute any
anonymous python added in the base configuration to make things
more consistent.
(Bitbake rev: bcdc2f73e3b4a10b1e479c2891f251d9507a9e30)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
This allows us to call this code from other contexts without
duplicating it.
(Bitbake rev: c6be487f9bd5d95915f2495d555b9f539adb1d44)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
If a standalone tasks adds a dependency on X:do_build, the code in runqueue would
currently remove it if that do_build was part of an image recipe which uses
recrdeptask on do_build.
Such individual tasks shouldn't do this, therefore tweak the recursive reference code
to only process recurseive tasks, not all tasks.
(Bitbake rev: 4cfca360891e1ed876a9c19487b4f6210686af26)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
In order to allow users to manually populate the download directory with
valid content change the assumption that missing the donestamp file
means unfetched content.
This allows users to populate the download dir without needing to create
dummy .done files such that a user does not need a PREMIRROR when using
BB_NO_NETWORK to provide valid content files in the download directory.
To ensure the correct result this change also fails first if the
localpath does not exist. This prevents further parts of the function
attempting to calculating the checksum on non-existent files. This also
fixes some edge conditions around where if the donestamp exists but the
localpath does not it returns, and did not remove the donestamp.
Also added test cases to cover this use case and additional use cases
where for example the fetcher does not support checksums.
(Bitbake rev: a335dbbb65d5b56e71d98cf3e4fa9bfbec1dcde6)
Signed-off-by: Nathan Rossi <nathan@nathanrossi.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
At least the cli-color node module has dependencies that have
cyclic dependency among themselves. npm.py is prepared to deal
with such a case but the condition is handled only for downloading
or not a dependency again, but then it goes checking the its
dependency which causes an infinite loop in _getdependencies().
Make this function simply return when a dependency is already
downloaded and only download and check its dependencies when not.
(Bitbake rev: 545540420112992e53f4a83104af10452df168d0)
Signed-off-by: Zoltán Böszörményi <zboszor@pr.hu>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The resolve_file() calls mark_dependency(), so the one which calls
resolve_file() doesn't need call mark_dependency() again.
(Bitbake rev: 4682571107323a39b42cd9ec8ee67419e7f15acc)
Signed-off-by: Robert Yang <liezhi.yang@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* Print message when cachefile is found/not can help debug.
* Update "Using cache in" to "Cache dir:" since it was the same as the debug
message of "codeparser & file checksum caches", which caused confusion. And
whether the cache file will be used or not is still unknown at that time, so
just print the cache dir.
(Bitbake rev: c8d3a2016f432e8ed9e99d9c28850149ab6fd6d8)
Signed-off-by: Robert Yang <liezhi.yang@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
It should be live/exited with server rather than cooker, fixed:
$ bitbake --server-only -T -1
Set MACHINE = "qemux86" in conf/local.conf
$ bitbake quilt
Set MACHINE = "qemuppc" in conf/local.conf
$ bitbake quilt
[snip]
ERROR: When reparsing /workspace1/lyang1/poky/meta/recipes-connectivity/openssl/openssl_1.0.2m.bb.do_package, the basehash value changed from c216f7f4fdd3cf4a0b10b975a636426c to d5a8e9431ab261381752d7a64c7b2fa9. The metadata is not deterministic and this needs to be fixed.
[snip]
This is because the server doesn't know local.conf is changed since the
notifiers are stopped, so it doesn't reparse, and then we would get the errors,
let the notifiers live/exited with server can fix the problem.
(Bitbake rev: a6a641cb9c5f3abe901b150da915372e295383d7)
Signed-off-by: Robert Yang <liezhi.yang@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The old code couldn't handle nestled layers correctly, e.g.:
parent_layer/sub_layer/foo.bb
Note there are two layers, parent_layer and sub_layer.
And in parent_layer/conf/layer.conf:
BBFILE_PATTERN_parent_layer = ""^${LAYERDIR}/"
This setting is incorrect since it also matches parent_layer/sub_layer/foo.bb,
so it warns that no files matched sub_layer, this is the expected behavior, but
it doesn't warn when there is a parent_layer/sub_layer/bar.bbappend, this was
incorrect since the bbappend is also matched by BBFILE_PATTERN_parent_layer, it
should warn and let the user fix the problem. Check the bbappend in already
"matched set" before return it as matched by "unmatched set" can fix the problem.
(Bitbake rev: ec90245d28e52ea718d2ce084eb304cdc4355c9c)
Signed-off-by: Robert Yang <liezhi.yang@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
There is nothing to do when BBFILE_PATTERN is empty.
(Bitbake rev: b7da31641c24c53d47ea45a7119d1bd353011b39)
Signed-off-by: Robert Yang <liezhi.yang@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Move the code that existed in tests/fetch.py for determining the path to
'git-make-shallow' into the git module and reference it.
This ensures that 'git-make-shallow' is always available and the desired
version regardless of the path variable or whether git exposes the
command.
(Bitbake rev: 6b508ab8fd5aa796c1c00c970e81e5e93f84d35d)
Signed-off-by: Nathan Rossi <nathan@nathanrossi.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The runall commandline option was confusing people. There are in fact two
different behaviours people may want.
a) For a given target (or set of targets) look through the task graph and
run task X only if its present and would have been built.
b) For a given target (or set of targets) look through the task graph and
run task X if any recipe in the taskgraph has such a target even if it wasn't
in the original task graph.
I've decided to interpret the existing "runall" option as b), even if right
now if behaves like a). For a), which is a valid use case, this patch adds
a "runonly" option.
With both behaviours present, I'm hoping we can then kill off the "fetchall",
"checkuriall" and other tasks from OE metadata and replace them with this
option. This would significantly speed up task graph processing.
(Deleting the checkuriall and fetchall tasks takes "bitbake core-image-sato -g"
from 22s to 8s).
(Bitbake rev: 546a662c877b2d3af35e3996950582ed2df41fe4)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
This is a performance sensitive piece of code and the shear number
of recursive loops is causing a significant and unscalable performance
pain point.
This change moves to a two step approach, firstly generating a list of recursive
dependencies for any task, then applying this to the recursive tasks, iterating
over things until no further dependencies are added.
It was noticed an optimisation is possible and the list of recursive tasks need not
contain the taskname, only the base task id. This allows a significant performance
improvement and limits the size of the resursive task lists, improving speed.
(Bitbake rev: eba738ac5672556eaab4f3374c8025c322761c4a)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
We can optimise the loops slightly so we only process given substrings
once rather than many times. This means expanding out add_resolved_dependencies.
Also add a function which allows replacement of the task element of a
task id, reducing the amount of string handling we're doing in a performance
critical loop.
Its also clear that later code adds to the tasks depends so we don't need
to add .depends() to extradeps at the start.
(Bitbake rev: 4ad281224e92b5f94e3a9c17e8898ec8f1086cdc)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Currently we only run through the recidepends/recrdepends code once. This
means that we can miss some expansions of dependency trees where one
rec{r,i}depends tasks depends on another rec{r,i}depends task.
In reality we need to iterate over the data until we stop adding
dependencies.
In doing this we can't show quite so granular progress information since
we don't know how many times we'll need to do this.
This does slow down the runqueue prepare phase however some optimisations
are possible and can be handled in subsequent patches.
This fix means some missing dependencies, such as:
<image>:do_fetchall -> <image>:do_rootfs -> <pkgs>:do_package_write_X
-> <ca-certs>:do_package_write_X -> debianutils-native
(via PAKAGE_WRITE_DEPS)
are now found/added.
[YOCTO #12510]
(Bitbake rev: aec2f07d56a19b97b6515897532b113cdead8338)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
We've noticed issues on our infrastucture iterating over the many
tag/branch/head reference files that some git repositories may contain.
By issuing the pack-refs command, we move these all to a single file
which speeds up operations with the mirror repos in the downloads
directory in general.
(Bitbake rev: f8126aaf774186a6eaf0bd4067b89c074594886c)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
This means two different graph files can easily be compared, currently
you'd have to sort them as the output is randomized.
(Bitbake rev: 5f7c6ec785f70beb1a4a1bbc0eb83cfa6cd7740d)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
If a fetcher, e.g., git, is run when pseudo is active it will think it
is running as root. If it in turn uses ssh (as git does), ssh too will
think it is running as root. This will cause it to try to read root's
ssh configuration from /root/.ssh which will fail. If ssh then needs to
ask for credentials it will hang indefinitely as there is nowhere for it
to ask the user for them (and even if there was it would not access the
correct private keys).
The solution to the above is to temporarily disable pseudo while
executing any fetcher commands. There should be no reason for them to be
executed under pseudo anyway so this should not be a problem.
RP Ammendum:
We finally did get more information about how to reproduce this problem,
something needs to trigger bb.fetch2.get_srcrev() in a pseudo context,
for example when AUTOREV is in use or the recipe doesn't have a defined
SRCREV. That SRC_URI needs to be using protocol=ssh. This would trigger
an ls-remote of the remote repo and if that happens under pseudo, the
wrong ssh credentials may be attempted which can hang.
[YOCTO #12464]
(Bitbake rev: ceaca281cafa662aa2385b95641bce309dce843d)
Signed-off-by: Peter Kjellerstedt <peter.kjellerstedt@axis.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The cfg property on events doesn't add much, all code appears to access
"data" at this point. Remove it to clean up the interface.
(Bitbake rev: bd4b9e4460b60f142c3bf346fb04e360e512eaee)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Without this the paths to git-make-shallow are incorrect and cause test
failures if bitbake isn't executed from cwd or PATH.
(Bitbake rev: 643eacb162b8710330ef292bfda21cfeab97f95c)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
This change adds EventClassesTest class to bb/tests/event.py,
including 47 new test cases for the public interfaces of the
bitbake event and related helper classes.
[YOCTO #10773]
(Bitbake rev: ee5fe4431713b8a29bdb424a29460965374b3234)
Signed-off-by: Jair Gonzalez <jair.de.jesus.gonzalez.plascencia@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Test the triggering of bb.event.worker_fire callback.
(Bitbake rev: daa59a2057c811b20d75235526ac6c2079ac6e10)
Signed-off-by: Jair Gonzalez <jair.de.jesus.gonzalez.plascencia@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
This change includes unit tests for the following functions,
helper class and methods in bitbake.event:
- set_eventfilter(func)
- set_UIHmask(handlerNum, level, debug_domains, mask)
- getName(e)
- class UIEventFilter(object)
def __init__(self, level, debug_domains)
def update(self, eventmask, level, debug_domains)
def filter(self, event)
[YOCTO #10773]
(Bitbake rev: 4a19dde704fd0bf262ea991ef530f991a4897d31)
Signed-off-by: Jair Gonzalez <jair.de.jesus.gonzalez.plascencia@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
If for example you deltask do_build but it has recrdeps set, its confusing
to have that list returned when the task no longer exists (same would apply
to deps too if it was set after the deltask).
(Bitbake rev: b7a81cb91e82ba64b63c9153dc161a2ab4696715)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Allow for "SinglePackageInfo" events that do not include package data,
for example OPKGN equal 'lib32-*' or 'lib64-*'.
[YOCTO #12204]
(Bitbake rev: 567f072ff260614cde3da220a40a95d5a8b9ab92)
Signed-off-by: David Reyna <David.Reyna@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
We saw builds where runtime providers were sometimes changing order and the
build result was therefore non-deterministic. For example it could show:
DEBUG: providers for lib32-initd-functions are: ['lib32-lsbinitscripts', 'lib32-initscripts']
or
DEBUG: providers for lib32-initd-functions are: ['lib32-initscripts', 'lib32-lsbinitscripts']
which could cause a test to pass or fail.
This change ensures we don't rely on the random order of dictonaries in
memory and act deterministically.
(Bitbake rev: ebce92bf8d71f8a6e8af1c6cf6ba335faf9d67c8)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
.txz is the same as .tar.xz, and can be found in the wild.
(Bitbake rev: 2ba8a6b25ccc12e7b543e8450121e5311c7a701d)
Signed-off-by: André Draszik <git@andred.net>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The connection cache class uses a dummy file object but it doesn't have a closed
attribute, so we can't use it in a context manager.
(Bitbake rev: 7b072ef91d16331eae11bd60f229ce1f0c175995)
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
If a matching recipe is not found then return Null instead of raising
KeyError because we were blindly using None as a key for pkg_fn.
(Bitbake rev: 431e89e322850a2497157c3c0843da9df6bc9a3e)
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
This is primarily paranoid but ensure we remove any loggers we setup
either directly or indirectly so the initial state is restored after
we exit.
(Bitbake rev: af7d63b1f76fd3f7fa92ed15ae61ca47d9e13472)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Whilst we're likely exiting in this case, clean up the loggers we add
so that in the case of certain server retries there is no possibility
multiple loggers stack up.
(Bitbake rev: 25b7bf6672be66bcbfe5760610dce7d3e866cdcc)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Add in a tets ftp url so we ensure ftp urls contnue to work after the loss
of the ftp.gnu.org ones.
(Bitbake rev: e1e8565b5e19dd3f7ef6e7e41932456adaa3df81)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
As we test multiple URLs in this these tests and one failing abandons the test,
use subtests so all URLs are tested. This should help us identify patterns in
the failing URLs.
(Bitbake rev: c4c4465b32e82d4b6e46a44e776be5039aef6b18)
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The ftp server at ftp.gnu.org is likely to be retired at some point soon
so siwtch over to the http/https services.
This means bitbake-selftest doesn't have ftp test urls, however finding stable
ftp test servers is proving increasingly hard.
(Bitbake rev: 892a08245ddb21a464aeb37d3e32377e99dd7e2b)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
After cleaning deprecated API usage repo fetcher is missing
logger as it was indirectly imported via deprecated bb.data.
Fix this by importing logger directly.
Fixes: 9752fd1c10b8 ("fetch2: don't use deprecated bb.data APIs")
(Bitbake rev: f8e027d26603db2f1fe757dca767ea35d95174c7)
Signed-off-by: Oleksandr Andrushchenko <oleksandr_andrushchenko@epam.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Instead of not even having the test functions if network tests are disabled, use
a custom decorator to mark the network tests and skip them.
(Bitbake rev: cc420f430b1dafd9ca944bea259a564aaab34595)
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Currently if you run bitbake in an invalid directory, the user experience
is poor:
birbake/lib/bb/main.py", line 427, in setup_bitbake
topdir, lock = lockBitbake()
File "./bitbake/lib/bb/main.py", line 494, in lockBitbake
lockfile = topdir + "/bitbake.lock"
TypeError: unsupported operand type(s) for +: 'NoneType' and 'str'
This ensures we exit straight away with a better error message.
[YOCTO #12163]
(Bitbake rev: 562f9ee674a8b392437096422b9cceab9c3cba2e)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
In Python 3.6, glob.glob() was reimplemented to use os.scandir() (which
itself appeared in Python 3.5), thus our monkey patching of os.listdir()
here was no longer effective. The end result was not only that bitbake
wouldn't notice added recipes or bbappends with BB_SERVER_TIMEOUT set
when being run with Python 3.6 (the shipped Python version on Fedora 26
and some other distribution versions), it also broke devtool modify,
devtool upgrade and devtool extract since they rely on the ability to
create a bbappend on the fly and have bitbake pick it up.
To fix it, do the same monkey patching for os.scandir(), which needs to
be conditional upon that actually existing since we have to support
Python 3.4 that doesn't have it. Long term we should probably look for a
better way to handle this that doesn't involve monkey patching Python
library code.
Fixes [YOCTO #12185].
(Bitbake rev: d02e90db32e7ee341c2ba3be79b0627d8796bdd6)
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
When the destination is a directory, building the the destination file
path is always needed. That's because even if the copy fallback is
taken, it's always followed by a rename.
(Bitbake rev: 14c17480827ced2e03c1b62dc839696421fc4de8)
Signed-off-by: Alberto Mardegan <amardegan@luxoft.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
We were seeing cases where we could hit the 5s timeout on large/fast machines
running many different tasks at once. Increase this to 30s since the main
connection timeout path should no longer hit this slow path.
[YOCTO #12116]
(Bitbake rev: 131d4b8a5834781a93ed41e2967d8dcd4d80f29a)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
In collect_bbfiles() we're monkey-patching os.listdir in order to find
which directories to watch, and then undoing that when we're finished -
however if an exception occurred for any reason there was nothing to
ensure the latter occurred. This may not have caused any issues, but as
this kind of thing really ought to be secured using try...finally just
in case, so do that.
(Bitbake rev: 013047484a03185c0ce281c53c1db4949cdc4e69)
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The code that was supposed to watch directories along BBFILES for
creation of new files wasn't working in the case where the directory did
not initially contain any matching files - since in updateCache() we are
passing the directory path to add_filewatch() and the latter function
calls os.path.dirname() on the path on the assumption that it is a file
path, and thus the parent of the directory got watched but not the
directory itself. (If the directory wasn't empty everything worked fine
since add_filewatch() was called elsewhere with the path to one of the
files in that directory, and thus the directory got watched). Add a
parameter to add_filewatch() to tell it we are passing it directory
path(s) rather than file path(s).
(Bitbake rev: 47a34dee08fcc25d896a1bdf16fa86267f0b898f)
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Printing the last 10 lines of bitbake-cookerdaemon.log when the server
fails to start can sometimes result in printing the output from a
previous run, which could lead the user completely down the wrong path
in terms of the cause of the failure. Use a known start text containing
the time which we can then look for when scanning through the log, and
then grab the last 10 lines of that part instead.
Fixes [YOCTO #11903].
(Bitbake rev: 567f2cf1bc455b4f3cfb1cbd7f25145360b05a62)
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The whitelist shouldn't have to be populated in order for the
enforcement to work properly - check if the list is not None in order to
determine whether the functionality is enabled or not since that is how
the function that sets up the list behaves.
(Bitbake rev: 7b1e79c352ca6eef1693d8abfacf7505544f1caa)
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Issues in start are not being correctly detected by the current readypipe
code. Change it to use specific "ready" or "fail" messages to correctly
determine the correct failure mode and avoid bitbake seeming to hang
(it does currently timeout eventually).
[YOCTO #12062]
(Bitbake rev: 60d4791e3dd05729d2a2adf6f3b203c80d466a73)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
If there is a network failure the return value from latest_versionstring() is
('','') which later causes an exception when comparing versions.
Improve this by checking the return value and failing the test early.
[ YOCTO #12053 ]
(Bitbake rev: 3f034d2172bf64ecc43577b43e0cf032a54b1358)
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Since 2017-08-17 (git version 2.14.1.473.g3ec7d702a) using deprecated
git branch parameter "--set-upstream" causes a fetcher error. Replace
it by "--set-upstream-to".
https://git.kernel.org/pub/scm/git/git.git/commit/?id=52668846ea2d41ffbd87cda7cb8e492dea9f2c4d
says, it's deprecated since 2012-08-30 so hopefully all still supported
host distributions have new enough git to support "--set-upstream-to".
ERROR: PACKAGE do_unpack: Fetcher failure: ...;
git -c core.fsyncobjectfiles=0 branch --set-upstream master origin/master failed with exit code 128, output:
fatal: the '--set-upstream' option is no longer supported. Please use '--track' or '--set-upstream-to' instead.
ERROR: PACKAGE do_unpack: Function failed: base_do_unpack
(Bitbake rev: 2ab50074c1a6c56a8a178755de108447d7b7acaf)
Signed-off-by: Andre Rosa <andre.rosa@lge.com>
Signed-off-by: Martin Jansa <Martin.Jansa@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
When the UI disconnects, we can throw away any server-side remote
datastores we created in response to calls from the UI, and we *must*
drop everything in extraconfigdata or it will taint any future
operations.
Dropping extraconfigdata upon disconnect fixes taskhash mismatch errors
when running devtool.DevtoolTests.test_devtool_update_recipe_local_files
within oe-selftest with BB_SERVER_TIMEOUT=100 in OpenEmbedded.
(Bitbake rev: 1ca2eec459424892391f060442ef38cf28d6a54a)
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
If you have a layer with a blank BBFILE_PATTERN the layer was ignored
when processing the list of layers with priorities. This list is not
only used for processing recipes, but also by additional programs such
as bitbake-layers show-layers.
Without this change, a layer that provides configuration or classes
only does now show up in show-layers, which is used by the
yocto-compat-layer.py script. This causes a failures in the compatibility
check.
(Bitbake rev: a0eaf8c0f228f984bafff09e4e9739f758dc1a9b)
Signed-off-by: Mark Hatle <mark.hatle@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
If we're implementing reset() in SignatureGenerator at all (and we need
to for a basic non-OE BitBake setup where that is the default signature
generator), then we need it to be clearing out the internal values
properly.
(Bitbake rev: 13f52d38fdbcb84c2a0c46f85baa44b22d53fdc1)
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
BB_SERVER_TIMEOUT=100 oe-selftest -r bblayers
was failing and highlighted that since parseConfiguation clears data
structures, it needs to also clear parsecache_valid as it no longer
contains correct data.
(Bitbake rev: 7234f33a7eb38ad51a8345f6689bc26e29f29f92)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
I can't actually see how this was working, nothing connected the commandline option
to the data in TaskData(). Drop the remaining pieces of this option, it was a relic
from a decade ago and we want deterministic builds, not random tries until something
might work.
(Bitbake rev: 767c7ba8fc76ec667ac1567de1c971c3575f2ecd)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Fix failure after commit "cooker/siggen: Reset siggen when reparsing"
(e4c6ca9440f63761560b49bbe12654441f54687e) when executing without
specifying a BB_SIGNATURE_HANDLER.
(Bitbake rev: 2a78c2d09aea0323632bbc927f370f1d3c9c249e)
Signed-off-by: Caio Marcelo de Oliveira Filho <caio.oliveira@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
BB_SERVER_TIMEOUT=100 oe-selftest -r devtool.DevtoolTests.test_devtool_build_image
fails, the reason is that internally, the limited_deps flag is set in
the bitbake server and then never cleared. This causes the sysroots to
be setup incorrectly (as per the limited dependency case) and builds
break. There is also potential for corruption of recipecaches.
Add shutdown/cleanup code to ensure these effects don't 'stick'.
This bug is particularly nasty as you can destroy TMPDIR with large
sysroots in build work directories which are prone to break.
Also ensure mtime cache is cleared (to match buildTargets) and
that no lasting changes are made to siggen either which ensures:
BB_SERVER_TIMEOUT=100 oe-selftest -r devtool.DevtoolTests.test_devtool_upgrade_git devtool.DevtoolTests.test_devtool_virtual_kernel_modify
works.
(Bitbake rev: 0a7ee8c8378bba9877c260b1aee782878f1935b4)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
When fetching source for the first time within scripts such as
OpenEmbedded's recipetool, we don't want to be showing warnings about
NPM_SHRINKWRAP or NPM_LOCKDOWN not being set since there's no way we
could have set them in advance. Previously we were using
ud.ignore_checksums to suppress these but since we are now using a more
standard task-based path to fetch the source, we need to disable these
through the metadata. Look for a "noverify" parameter set on the npm URL
and skip the checks if it is set to "1".
(Bitbake rev: 8c4b35d1e4d31bae9fddd129d5ba230acb72c3bb)
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
If you run some other operations that result in the cache being
populated, and then call buildFileInternal(), then you can end up in a
situation where the cache already contains information about the recipe.
For example in OE this can now happen when you use devtool upgrade.
Normally this doesn't cause any problems, unless you have a non-absolute
path in BBLAYERS - in buildFileInternal() we are calling matchfile() which
will convert the filename to absolute, but later when taskdata goes to find
the providers of the recipe it finds the non-absolute path, sets up the
task information using this and then the runqueue can't find any tasks
matching the absolute path. To fix this, back out the optimisation I did
earlier in bitbake rev ba53e067a2d448dd63b4ca252557ce98aa8e6321 to avoid
calling parseConfiguration() again, which is unfortunate but does result
in the cached information being that causes the problem being cleared
out.
This fixes "Task do_unpack does not exist for target ..." running
devtool upgrade within intel-iot-refkit.
(Bitbake rev: f120355eaec4571ba6d60fc5f7ae9e1f31d846d1)
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
If you set tracking=True when creating the tinfoil object, that ensures
history is collected for the main datastore, but at the end of parsing
the configuration, history tracking gets turned off to save time with
the result that we don't collect history for any recipes we parse.
Enable tracking when we parse a recipe (and disable it afterwards if we
enabled it) in order to fix this.
This fixes functionality in OE's devtool that relies upon variable
history (such as devtool upgrade updating PV when it's set within a
recipe).
(Bitbake rev: cc8b4c81bb589fb70774a0151f87a8d277f40f06)
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
If a task fails during build_targets(), we need to print out the log
lines as knotty does or the user will be missing information about the
failure.
(This should get some deeper refactoring, but now isn't the time for
that.)
(Bitbake rev: 24879df071d4803db3d39ae1d5cad852daa92f28)
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
With config_only=False we launch the UI and it sets up a logger, whereas
when config_only=True we don't, with the result that with True we are
seeing log messages from both our logger and the one set up by the UI.
Suppress our loggers with config_only=True to avoid this.
Fixes [YOCTO #11275] (again).
(Bitbake rev: b5e3b28b7c982dd8a3991d727f25710dbf58bb80)
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
If for example you run:
bitbake -r somefile.inc
rm somefile.inc
bitbake -e
bitbake will crash with an error about not being able to find somefile.inc. This
is because it tries to reparse the base config for the early getVariable requests
before it sees the updated missing -r option.
Send the updateConfig command earlier to avoid this.
(Bitbake rev: a38164620ebdc770690c5f39ff9ed69d3f82719e)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
cooker.reset() can be called before we've actually setup the datastore. Gracefully
handle this case instead of the current traceback+exit.
(Bitbake rev: 8fd30ca6d271c125a8ea03ef0c5d7ab176900701)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
When we reload the configuration, PRSERV_HOST can change. Therefore
restart the PR Server depending on the new configuration at reparse.
Note that the server has to be started from the right process, it
can't be in the UI which shuts down as that shutdown triggers a shutdown
of its children and the PR Server shuts down too. This is why we need
pre_serve() which ensures its executed in the right context.
(Bitbake rev: 971272e84f4efe7ebd0037e164ba54f013a2a34e)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Currently the original prefile and postfile passed when starting bitbake
server are 'sticky'. With the new memory resident model this doesn't make
sense as the server the system is started with isn't special.
This patch changes the code so the prefile/postfile are used if specified
on the commandline and not used otherwise. This makes the behaviour much
more predictable and expected and as an added bonus simplifies the code.
(Bitbake rev: 638d366234fad78f283d3a13a12b07cb0ccbe914)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The current passing of "tracking" backwards and forwards, client to server
is ugly and complex and error prone. Instead, set this during showEnvironment
commands triggering a reset there if/as required.
(Bitbake rev: 9dc7f384db0479569ff93a76a623d5395fecaf47)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Drop pointless unused function parameter.
(Bitbake rev: 8104b33656de0b619943bd7a9884eb650ccafbf4)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The 'needconfig' flag was meant to be the default as most commands do need
it and the "False" cases were the exception. The code was written backwards
with a default False value. Invert this to match the intent, resulting in
the config being reparsed if metadata has changed.
Also ensure the second level configuration is parsed for the getLayerPriorities
command as otherwise it can return stale info.
With these changes:
BB_SERVER_TIMEOUT=100 oe-selftest -r bblayers.BitbakeLayers.test_bitbakelayers_add_remove
passes instead of fails.
(Bitbake rev: af3c8928a69f204d5ced02c947485990ac04a776)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Rather than just waiting for 5s, try and get the lockfile. If we gain
the lock, we know we're ready to retry and can skip any remaining timeout.
(Bitbake rev: 8a60106c6f7d586c793b965c5e9460b6016fab15)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Its possible for a connection to connect to the server as its shutting down
but before its removed the socket file. This patch:
a) Removes the socket file earlier to avoid connections.
b) Handles EOFError in initial connections gracefully. These occur if the
socket is closed during the server shutdown.
c) Ensure duplicate events aren't shown on the console. This makes debugging
these issues very very confusing.
With these changes the backtrace that was concerning users is hidden and the
server works as expected with a reconnect when it catches it in a bad state.
(Bitbake rev: f45196cf84669723382730944dddc7eaf50826f2)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
URLs do not have to have a path; currently our npm URLs don't, so
encodeurl() needs to handle if the path element isn't specified. This
fixes errors using OpenEmbedded's devtool add / recipetool create on an
npm URL after OE-Core revision ecca596b75cfda2f798a0bdde75f4f774e23a95b
that uses decodeurl() and encodeurl() to change URL parameter values.
(Bitbake rev: d5cab2dbf5682d2fd08e58316a3bf39a10f63df2)
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The underlying model is already sorted for use in the other view,
add a sorting model for the 'Dependent Tasks' view.
(Bitbake rev: 27ca94c33234f0ef9753f8285213dde2871a3fcf)
Signed-off-by: Jussi Kukkonen <jussi.kukkonen@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
There is an interesting bug in the current code where a sync command
is not seen until the current async command completes, by which time
the UI may have shut down.
The reason is that if there are idle commands, we may not end up sleeping
in the select call at all, partiularly under heavy load like parsing.
Fix this by calling select with a zero timeout so that we see active
fds and know to read from them. This fixes various problems toaster was
having with the recent server changes.
[YOCTO #11898]
(Bitbake rev: bbcce58e824b2793abf50efa52db158ae16e23e4)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
We're seeing the server fail to start within 8s on heavily loaded
autobuilders so increase this timeout to 30s which should be more
than enough time.
(Bitbake rev: 8d4c120ec46d6d7a54947c64d33e18cb60b60505)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
There have been cases where the server could loop indefinitely and incorrectly
handle client disconnects. In the EOFError case, ensure a full disconnect
happens in the alternative disconnect path to avoid this.
(Bitbake rev: 5e267f14bb0155889615f567a920af4a37eb3c6b)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Writes to the cookerdaemon log and/or the lockfile were meaning the parser
cache was always being invalidated and reparsed. This is unnecessary so
spot accesses to these two common cases and ignore the files from a reparse
perspective.
This doesn't remove many sources of reparse but does improve several
common cases.
(Bitbake rev: 218e4b6418992588312b8ef5949b84ef43263d1a)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Add a proper function for clearing the mtime cache. Clean up the inotify
event overflow case to err on the side of caution and clear any potentially
now out of sync caches.
(Bitbake rev: ec60459fe2ba16966544eebff43b061abb7ed3ba)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The inotify watch events are handled "at idle" which could in theory mean
a command could run before any preceeding inotify events have been processed.
This leads to a theoretical race window where those events may have a
signficicant effect on the command.
Add a mechanism to allow us to ensure all pending events are processed before
running commands.
(Bitbake rev: bf76cd7e5881adf264b8ba64e27a5b6ca9df4fde)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Some of the directories searched by BBFILES are not currently being added
to the inotify watch list. This can mean that added append files are not
noticed leading to misleading metadata results when using
BB_SERVER_TIMEOUT != 0.
We use glob to expand the BBFILES references and without writing our own
implentation, figuring out the directories it searches is hard. We use
some horrible hacks here to intecept the listdir calls, I'm open
to better ways to handle this but this does ensure we have the right
watches set.
(Bitbake rev: 4d508d35a224e3a25d2d59c8415ab7985964b14f)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The server currently crashes if we hit an EOFError due to controllersock
still being in ready and the continue meaning ready isn't re-evaluated.
Setting the value to False can mean the shutdown code doesn't handle the
situation cleanly.
Clear ready to avoid the crash/loop instead and handle any OSError whilst
we're in here.
(Bitbake rev: 2bc47e887c3b41417edaa89a8708c223fd2085de)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
If we don't do this, we get basehash mismatch errors occurring from the reparse
which would then set bitbake's error exit code.
This for example would cause oe-selftest -r bbtests.BitbakeTests.test_bbappend_order
to fail with a non-zero BB_SERVER_TIMEOUT.
(Bitbake rev: e4c6ca9440f63761560b49bbe12654441f54687e)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Make BB_SERVER_TIMEOUT = -1 mean no unload forever.
(Bitbake rev: 923d5567be6fd9969ca74c166f36817ec09305e3)
Signed-off-by: Robert Yang <liezhi.yang@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Fixed:
$ export BB_SERVER_TIMEOUT=10000
$ bitbake --server-only
$ bitbake --status-only
[snip]
File "/buildarea/lyang1/poky/bitbake/lib/bb/server/process.py", line 472, in recvfds
msg, ancdata, flags, addr = sock.recvmsg(1, socket.CMSG_LEN(bytes_size))
OSError: [Errno 9] Bad file descriptor
And:
$ export BB_SERVER_TIMEOUT=10000
$ bitbake --server-only -B localhost:-1
$ bitbake --status-only # Everything is fine in first run
$ bitbake --status-only
[snip]
File "/buildarea/lyang1/poky/bitbake/lib/bb/server/process.py", line 472, in recvfds
msg, ancdata, flags, addr = sock.recvmsg(1, socket.CMSG_LEN(bytes_size))
OSError: [Errno 9] Bad file descriptor
This was because self.controllersock was not set to False, so it still ran
sock.recvmsg() when sock was closed.
And also need set command_channel to Flase, otherwise the
self.command_channel.get() will always run when EOF, and cause infinite loop.
(Bitbake rev: 7b739a38601b053d9bea4df2c0b44a952ab670c4)
Signed-off-by: Robert Yang <liezhi.yang@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
There is already a type=float, so the float() is not needed, which also makes
the error clearer:
$ export BB_SERVER_TIMEOUT=10000AA
With float():
$ bitbake quilt-native
[snip]
ValueError: could not convert string to float: '10000AA'
Without float():
$ bitbake quilt-native
[snip]
optparse.OptionValueError: option --idle-timeout: invalid floating-point value: '10000AA'
The second one tells clearly where is wrong.
(Bitbake rev: 7a4ea1e6a10f20d80009a78f4f0aebf8f90095fd)
Signed-off-by: Robert Yang <liezhi.yang@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
In combination with the recent server reworking, this change actually
prevents messages sent from tasks from being logged properly. This will
of course give us the duplicated messages back, and I really hate to do
that effectively a second time, but that's better than seeing no error
at all in the case of a failure - we'll have to find the proper way of
avoiding the duplication that doesn't result in some messages going
missing.
This reverts commit 8a5bae76f91f2411187c638a42fa3c762052cf11.
(Bitbake rev: 645c8dd15762516ae5ab64a1df47fadb95d072d1)
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
If there are events in the UI queue we want to print them regardless of
whether we're handling an exception or something like SystemExit.
This improves error messages for some other failure cases where currently
no logging would get printed and leave the user confused as to what went
wrong.
(Bitbake rev: 76c4f2c20216719736766e8ae7d089ccd061b71b)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
"bitbake -g m4-native -c do_unpack" doesn't list any dependencies in
task-depends.dot. This is because no header describing the task was
added unless a task has dependencies. Tweak the code to fix this.
[YOCTO #10893]
(Bitbake rev: e2fbe728769f09ef4178e2ef4d69a2f60ffe3ebb)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
This was originally based on some other code but its mostly misleading
comments now. Massively clean this up and accept its now a total fork.
There are no funciton changes here, just cleanup.
(Bitbake rev: 3daa9001269e5b9156d0eb3299d2628d2d219fff)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
When we create the child, if an exception occurred it was transfering
back into the parent context. We don't want to do that us use a try/finally
to ensure we exit.
We need to ensure a traceback is printed and any queued UI messages which
may not have made it to the client UI at this point.
(Bitbake rev: dec1d2c26f6cb3ffeb44beaab0129cd531a6d08b)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
We have cases where the server is being started but we're not seeing any messages
from it. Add some earlier logging so we can try and better understand where
issues may be occurring.
(Bitbake rev: 10a6a48c0c5a2fe2051e90143e66075356853971)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
If the connection control socket and the command channel close together,
we can race and hit EOFError exceptions before we close the channel.
Reorder the code to handle this in the correct order and ignore the
EOFError exceptions as they mean the client is disconnecting and shouldn't
terminate the server.
(Bitbake rev: 974281ed72d8366baa16ee85f7e93970689b5956)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
My previous assertion about FusionForge appears to have been wrong, or
FusionForge has changed behaviour, or both.
FusionForge now mandates that downloads have the Accept header set, despite that
header being optional, and returns a 406 Not Acceptable error if it isn't set.
As we were pretending that 406 was actually 405 (Moved) and tried to handle it as a
redirect this results in an infinite loop until Python kills the recursion.
Delete the handling of 406 as 405, and pass Accept: */* in the headers.
(Bitbake rev: bb70ae0c9aac5ec688026d23a64ac0cac1947187)
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Having this as one huge long line isn't easy to manipulate, split it into
multiple lines for ease of debugging issues.
(Bitbake rev: 5753fe81194f75fbcf4ccdc733cc585d02794cb1)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The checkstatus() code was expecting checkstatus to throw exceptions if it
failed, but in general it should return False.
(Bitbake rev: 57be5cc6228518e60f564570a39cebbeb6cf564e)
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
`git fetch --tags` seems to interact badly with `mirror=fetch`, resulting in
the regular branches not being fetched, so drop the unnecessary `--tags`. This
fixes this unit test failure: `bb.fetch2.FetchError: Fetcher failure: Unable
to resolve 'master' in upstream git repository in git ls-remote output for
/tmp/tmp4ag_mgmn/gitsource`
[YOCTO #11698]
(Bitbake rev: 2d0203fae08c5ff8dc3e9afaa9a819abc4a1af6f)
Signed-off-by: Christopher Larson <chris_larson@mentor.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
If the server isn't running return 0 as to do otherwise complicates
scripts which do cleanup of bitbake servers which would potentially
be memory resident.
(Bitbake rev: 0f85c54385f325f3ebbf911b35bbac11b8863b6b)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
This timeout path was commonly hit due to errors starting the server. Now we
have a better way to handle that, the retry logic can be improved and cleaned
up. This patch:
* Makes the timeout 5s rather than intervals of 1s with a message. Paul
noted some commands can take around 1s to run on a server which has just
been started on a loaded system.
* Allows a broke connection to exit immediately rather than retrying something
which will never work.
* Drops the Ctrl+C masking, we shouldn't need that anymore and any issues
would be better handled in other ways.
This should make things clearer and less confusing for users and is much cleaner
code too.
(Bitbake rev: 8633b7cd03cfaba3e0359aa5da22fc76b66768c7)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
If we reconnect to the server, stop leaking pipes and clean up
after ourselves.
(Bitbake rev: f41e4e971e807157be68cf4496580494b8b60643)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Currently if the server fails to start, the user sees no error message and
the server will be repeatedly attempted to be started until some longer
timeouts expire. There are error messages in the cookerdeamon log but
nobody thinks to look there.
Add in a pipe which can be used to tell the starting process whether the cooker
did actually start or not. If it fails to start, no further attempts can be
made and if present, the log file can be shown to the user.
[YOCTO #11834]
(Bitbake rev: 57000d44beb1aeba37dfc70782b0d6418943acc5)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
This cleans up the socket keep alive into better class structured code
and adds cleanup of the open file descriptors upon shutdown.
(Bitbake rev: 77fd3a3a29a569e212374b27aea742ddbaafcdd5)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Currently the monitoring thread exits with some error code or runs indefinitely. Allow
closure of the pipe its monitoring to have the thread exit cleanly/silently.
(Bitbake rev: 930d077637928213e13a07c78fee3bf7a8c37ebf)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Expose the underlying close() and fileno() methods which allow connection
monitoring and cleanup.
(Bitbake rev: f79187f4ebfad7969be47b429995e7f7a3e33c1e)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
This code would duplicate messages to stdout and stderr when no UI connected
and there were error level messages.
Rework the code so it either uses stderr (for errors and above) or
stdout for warnings/debug but not both for the same messages.
(Bitbake rev: 45cff5734ba2ba8c8d36d17d722a5804d39b258b)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
We were bridging the gap between the server and UI here by calling a
bb.siggen.find_siginfo, a function defined and set on that module from
the metadata. This worked from the UI side before but since the recent
server changes is no longer accessible. Create a new command so this can
execute on the server side and return the result by way of a new event.
(We're still running compare_sigfiles() on the signature generator but
that isn't quite the same thing and does still work.)
Fixes [YOCTO #11844].
(Bitbake rev: fdcea991baa4f83d9c98d468d7b49c8c388a4a15)
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
When wget is fetching a listing for a directory over FTP it writes to a
temporary file called .listing in the current directory. If there are many such
operations happening in parallel - for example during 'bitbake world -c
checkpkg' - then up to BB_NUMBER_THREADS instances of wget will be racing to
write to, read, and delete the same file.
This results in various failures such as the file disappearing before wget has
processed it or the file changing contents, which causes checkpkg to randomly
fail.
Mitigate the race condition by creating a temporary directory to run wget in
when doing directory listings.
[ YOCTO #11828 ]
(Bitbake rev: 91d4ca93df092cf86ab84faaa94cc66ff9f43057)
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The warning message currently shown can occur more frequently than previously
if a previous bitbake server is shutting down and we're reconnecting to a new
server. Change it to a note message to match the higher level connection
logging retry messages and so as not to interfer with selftests.
(Bitbake rev: b7514340cd6a2753eb217b059229bb279c3849ec)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Now that there is only one server, this abstraction is no longer needed
and causes indrection/confusion. The server shutdown is also broken with
the cooker post_server calls happening too late, leading to "lock held"
warnings in the logs if PRServ is enabled.
Remove the abstraction and put the shutdown calls in the right order
with respect to the locking.
(Bitbake rev: c0ddde7cf680225127d6285685652b905ed176c3)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
When the sstate is accessed via HTTP, the existence check can fail due
to network issues, in which case bitbake silently continues without
sstate.
One such network issue is an HTTP server like Python's own SimpleHTTP
which closes the TCP connection despite an explicit "Keep-Alive" in
the HTTP request header. The server does that without a "close" in the
HTTP response header, so the socket remains in the connection cache,
leading to "urlopen failed: <urlopen error [Errno 9] Bad file
descriptor>" (only visible in "bitbake -D -D" output) when trying to
use the cached connection again.
The connection might also get closed for other reasons (proxy,
timeouts, etc.), so this is something that the client should be able
to handle.
This is achieved by checking for the error, removing the bad
connection, and letting the check_status() method try again with a new
connection. It is necessary to let the second attempt fail
permanently, because bad proxy setups have been observed to also lead
to such broken connections. In that case, we need to abort for real
after trying twice, otherwise a build would just hang forever.
[YOCTO #11782]
(Bitbake rev: 6fa07752bbd3ac345cd8617da49a70e0b2dd565f)
Signed-off-by: Patrick Ohly <patrick.ohly@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
If BB_STRICT_CHECKSUMS is set to anything other than "1" i.e. we're not
going to raise an error, then fire an event so that scripts can listen
for it and get the checksums.
(Bitbake rev: 8b2ccb4b865f2df118ef668847df682a83f9c500)
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
If BB_STRICT_CHECKSUMS is set to "ignore" then don't display a warning
if no checksums are specified in the recipe. This is not intended to be
used from recipes - it is needed when we move to using more standard
code paths to fetch new files from scripts i.e. where we don't know what
the checksums are in advance.
(Bitbake rev: f15ca7339de8a448a93a14cf6130b3925178a920)
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
We want this API to be easier to use, so add missing function
documentation to help with that.
(Bitbake rev: 3e0e002d6497caa987f327cd83ad4db82cca6810)
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
A common task for tinfoil-using scripts is to iterate over all recipes.
This isn't too difficult with the current API, but the pkg_* variables
are a little awkward and are really designed for bitbake's internal
usage - and it gets a bit more difficult when you want to access some of
the other information such as packages and rprovides. To resolve this,
create a new recipe info class and add an all_recipes() function to
generate this for all recipes. Also add a get_recipe_info() function to
get the information for a specific recipe (by PN).
(It might perhaps be suggested that we already have a structure similar
to this in the cache, however the one we add here is designed for
external use and allows the internal structures to change if needed
without affecting the API).
(Bitbake rev: 308994028e59735ca726c5d2c1f0f85baccfe89d)
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Add access to fn_provides, packages, packages_dynamic and rproviders on
the recipecache object. This requires an additional corresponding
command plumbing to be added.
(Bitbake rev: 3df9b7c615174a6557581f3cd157842a28f6bb26)
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Up to this point, if you wanted to run build tasks in the normal way
they get run from a python script, there was no other way than to shell
out to bitbake. Worse than that, you couldn't have tinfoil active during
that because only one bitbake instance could be running at once. As long
as we're prepared to handle the events produced, we can create a wrapper
around calling the buildTargets command. Borrow code from knotty to do
this in such a way that we get the expected running task display
(courtesy of TermFilter) and Ctrl+C handling.
(Bitbake rev: 43761eee756be52a1021be53a40dc591a6c35fa7)
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
This isn't useful for knotty itself, but for use from tinfoil
in case we can't get access to either the console or errconsole, allow
either to be unspecified (None).
(Bitbake rev: 7544de437fc66b81502ecdb5db859182c45827cb)
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
