For example:
$ bitbake core-image-minimal --runall unpack
...
NOTE: Tasks Summary: Attempted 326 tasks of which 88 didn't need to be rerun and all succeeded.
$ bitbake core-image-minimal --runall patch
NOTE: Tasks Summary: Attempted 453 tasks of which 332 didn't need to be rerun and all succeeded.
This can replace fetchall as well:
$ bitbake core-image-minimal --runall fetch
NOTE: Tasks Summary: Attempted 135 tasks of which 119 didn't need to be rerun and all succeeded.
(Bitbake rev: 7c0fa6ba66cdb956b37d94055307cde857633df9)
Signed-off-by: Matthew McClintock <msm-oss@mcclintock.net>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
When we removed the postinst sstate dependency handling code from
setscene_depvalid, we noticed things being installed into the old
style sysroot for rootfs tasks which should not have been there,
causing a performance regression.
Analysis revealed that setscene dependencies were "bubbling" over
sstate tasks when they should have been stopping there. The
'continue' added by this patch avoids this issue and eusures
sstate tasks remain contained to their specific chains.
There was another bug in the code this exposed where the acconting
for tasks as they were removed from sq_revdeps was not correct. In
fixing this, what looks like a workaround in another test can then
be simplified.
After this change, populate_sysroot tasks are no longer depending
on package_write_rpm tasks for example, which would make no sense.
A before/after analysis of image dependencies only revealed improved
dependencies after this change.
Recipe specific sysroots did highlight the issue here since the
behaviour of the sysroot dependencies (and processing with depvalid)
was not matching what bitbake itself was doing, with bitbake being
incorrect. Failures were 'safe' in that too many dependencies would
get installed.
(Bitbake rev: 5ef2cb50041fa7106c8de170af73d2a54cb0b1f0)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The idea is that tasks which complete building a recipe (like
do_package_qa) are more important than tasks which start building new
recipes (do_fetch) or those which increase disk usage
(do_compile). Therefore tasks get ordered like this (most important
first, do_rm_work before do_build because the enhanced rm_work.bbclass
was used):
1. ID /work/poky/meta/recipes-support/popt/popt_1.16.bb:do_build
2. ID /work/poky/meta/recipes-core/readline/readline_6.3.bb:do_build
3. ID /work/poky/meta/recipes-connectivity/libnss-mdns/libnss-mdns_0.10.bb:do_build
...
464. ID /work/poky/meta/recipes-sato/images/core-image-sato.bb:do_build
465. ID /work/poky/meta/recipes-graphics/xorg-proto/inputproto_2.3.2.bb:do_rm_work
466. ID /work/poky/meta/recipes-devtools/python/python3_3.5.2.bb:do_rm_work
467. ID /work/poky/meta/recipes-core/packagegroups/packagegroup-base.bb:do_rm_work
...
3620. ID virtual:native:/work/poky/meta/recipes-extended/pbzip2/pbzip2_1.1.13.bb:do_install
3621. ID /work/poky/meta/recipes-devtools/qemu/qemu-helper-native_1.0.bb:do_install
3622. ID /work/poky/meta/recipes-core/zlib/zlib_1.2.8.bb:do_compile_ptest_base
3623. ID /work/poky/meta/recipes-extended/bzip2/bzip2_1.0.6.bb:do_compile_ptest_base
...
3645. ID /work/poky/meta/recipes-support/libevent/libevent_2.0.22.bb:do_compile_ptest_base
3646. ID /work/poky/meta/recipes-core/busybox/busybox_1.24.1.bb:do_compile_ptest_base
3647. ID /work/poky/meta/recipes-kernel/linux/linux-yocto_4.8.bb:do_uboot_mkimage
3648. ID /work/poky/meta/recipes-kernel/linux/linux-yocto_4.8.bb:do_sizecheck
3649. ID /work/poky/meta/recipes-kernel/linux/linux-yocto_4.8.bb:do_strip
3650. ID /work/poky/meta/recipes-kernel/linux/linux-yocto_4.8.bb:do_compile_kernelmodules
3651. ID /work/poky/meta/recipes-kernel/linux/linux-yocto_4.8.bb:do_shared_workdir
3652. ID /work/poky/meta/recipes-kernel/linux/linux-yocto_4.8.bb:do_kernel_link_images
3653. ID /work/poky/meta/recipes-devtools/quilt/quilt-native_0.64.bb:do_compile
3654. ID /work/poky/meta/recipes-extended/texinfo-dummy-native/texinfo-dummy-native.bb:do_compile
...
The order of the same task between different recipes is the same as
with the speed scheduler, i.e. more important recipes come first.
(Bitbake rev: 70e297e5c285ce0a02e9efd3117ff62cdc77ec12)
Signed-off-by: Patrick Ohly <patrick.ohly@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Without this, bitbake -b of image recipes cause tracebacks since
the list of providers is empty.
(Bitbake rev: c53e43f3c6a675c0934a7a4e358fd66b049ffca3)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
This change significantly shortens the time on reparsing stage
of '-S' option.
Each file is reparsed and then dumped within a dedicated
process. The maximum number of the running processes is not
greater than the value of BB_NUMBER_PARSE_THREADS if it is set.
The dump_sigs() in class SignatureGeneratorBasic is _replaced_
by a new dump_sigfn() interface, so calls from the outside and
subclasses are dispatched to the implementation in the base
class of SignatureGeneratorBasic.
Fixes [YOCTO #10352]
(Bitbake rev: 99d3703edd77a21770b366c6ad65a3c0f5183493)
Signed-off-by: Jianxun Zhang <jianxun.zhang@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Fakeroot workers usually have dependencies that need to be ready before they
can be started. Starting them as a block therefore doesn't work as the
dependencies may or may not have been built. Therefore start the multiconfig
fakeworkers individually upon demand.
[YOCTO #10344]
(Bitbake rev: ab15486c636e451ed5a62e561980946887901c3b)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
In multiconfig, mutliple tasks can execute which share the same stamp file. These
must not execute in parallel, the idea is the first should execute, the subsequent
ones should see a valid stamp and get skipped.
The normal task execution code has stamps code to handle this, this adds similar
code to the setscene execute() function to handle the issue there too.
(Bitbake rev: 937acf267fa9e45f538695b2cf8aa83232a96240)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
setscene was being entirelu skipped for multiconfig variants as the tasks
were simply not being spotted. If the default config was also being built
it masked the problem. When this was fixed by using taskfn instead of fn
in lookups against dataCache, several other instances of this problem were
highlighted.
This goes through and corrects the setscene code to correclty use taskfn
instead of fn in the appropriate places meaning setscene tasks for multiconfig
now work correctly.
(Bitbake rev: cead85cf219c07b40aeb7c0d948c25f0429136db)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The location of the fakeroot command and the various environmental values need
to be taken from the right multiconfig datastore, not the shared one. This
patch ensures the right one is used for cases like a split TMPDIR.
[YOCTO #10344]
(Bitbake rev: 813a7600bc26b7132ac36d1515571d321f566afb)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
For the purposes BB_SETSCENE_ENFORCE is designed for (in OE, it is used
by the installation process for the extensible SDK), we don't actually
need the whitelisted real tasks to execute - we just need to have them
in the dependency tree so that we get all of the setscene tasks they
depend on to run. Therefore we can actually dry-run those real tasks
i.e. they won't be run (and thus we won't waste a significant amount of
time doing so) and won't be stamped as having run either. We do already
have a dry-run mode in BitBake (activated by the -n or --dry-run command
line option), but it dry-runs the setscene tasks as well which we don't
want here.
Note that this has no effect on the checking we are doing with
BB_SETSCENE_ENFORCE to ensure that only whitelisted real tasks are
scheduled to run - that's handled separately.
Fixes [YOCTO #10369].
(Bitbake rev: 58f084291beb3a87d8d9fdb36dfe7eff911fa36b)
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Some of our metadata assumes that BuildStarted and BuildCompleted events
see the same data store. This is the case for buildTarget but not for
buildFile and recent changes mean this is now a problem.
The update_data() call is now an empty operation and there is no difference
between the expanded_data and data so we can simply remove the expanded_data
and its references and use data everywhere. This has been inteded for a while
but the above issue makes this more pressing to finally clean up.
(Bitbake rev: e3694e738e98f26f413ada6860ca7d829d3662f0)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Allow the client to set variables with the setVariable command and have
those changes take effect when running tasks. This is accomplished by
collecting changes made by setVariable separately and pass these to the
worker so it can be applied on top of the datastore it creates.
(Bitbake rev: 69a3cd790da35c3898a8f50c284ad1a4677682a4)
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
We now have code in OE that needs BB_TASKDEPDATA for setscene tasks. Therefore
generate and send this data. In this case its a "pre collapsed" tree
but that is fine for the use cases in question.
(Bitbake rev: 38b857d086af43af6ea3aa60d3876a2c9b225401)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Its useful to know the task hash in code using TASKDEPDATA so add this
data to the data structure. The recipe specific sysroots in OE
need this data.
(Bitbake rev: 758867e8dc74283bb1f031e158ec54cefdd5c2a6)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Hooking the disk monitor into the regular heatbeat event instead
of the runqueue solves two problems:
- When there is just one long running task which fills up the disk,
the previous approach did not notice that until after the completion
of the task because _execute_runqueue() only gets called on task
state changes. As a result, aborting a build did not work in this
case.
- When there are many short-lived tasks, disk space was getting
checked very frequently. When the storage that is getting checked
is on an NFS server, that can lead to noticable traffic to the
server.
(Bitbake rev: 4547eea26803a9cd355d8b045197bcbdbb36a9ad)
Signed-off-by: Patrick Ohly <patrick.ohly@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
getVar() now defaults to expanding by default, thus remove the True
option from getVar() calls with a regex search and replace.
Search made with the following regex: getVar ?\(( ?[^,()]*), True\)
(Bitbake rev: 3b45c479de8640f92dd1d9f147b02e1eecfaadc8)
Signed-off-by: Joshua Lock <joshua.g.lock@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The whitelist checks for BB_SETSCENE_ENFORCE were running for every call
to execute(). Since the task list doesn't change for each call into
execute, the checks only need to be ran once.
[YOCTO #10369]
(Bitbake rev: f65e631ab6705dfd9188f19ee423eca33bca7d7d)
Signed-off-by: Randy Witt <randy.e.witt@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Some task id manipulations were suboptimal:
* taskfn_fromtid and fn_from_tid were effectively the same function
* many calls to split_tid(), then taskfn_fromtid()
* taskfn_fromtid() called split_tid() internally
This patch adds split_tid_mcfn() to replace split_tid() and returns the
"taskfn" variant being used in many places. We update all core calls
to the new function and ignore the return values we don't need since the
function call overhead of the split_tid wrapper is higher than ignoring
a return value.
The one remaining standalone use of taskfn_fromtid is replaced with
fn_from_tid. I couldn't see any external usage so it was dropped.
There is external usage of split_tid so a wrapper remains for it.
Combined together these changes should improve some of the runqueue task
manipulation performance.
(Bitbake rev: 1bf2ef874fbe47f1320007efa0bdeef8d630b8a1)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
`if w in self.rq.worker` when w *is* self.rq.worker doesn't make a great deal
of sense, and results in this error:
File ".../poky/bitbake/lib/bb/runqueue.py", line 2372, in runQueuePipe.read():
name = None
> if w in self.rq.worker:
name = "Worker"
TypeError: unhashable type: 'dict'
Most likely this was meant to be 'is' rather than 'in', but rather than
checking after the fact, just include the name in the iteration, instead.
While we're here, also clean up and fix the broken error message.
(Bitbake rev: 267e025cad44c8bd0fb157f1f7a2e08df117ba84)
Signed-off-by: Christopher Larson <chris_larson@mentor.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
If the worker fails to launch, ensure the task is shown as failed rather
than a confusing "all succeeded" message.
Patch from Juro Bystricky
[YOCTO #10335]
(Bitbake rev: 0e9a2ff96d138641501874a1cd7aa6cc7e94d727)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
If you "bitbake glibc-locale" then delete the libpcre-native sstate
and "bitbake glibc-locale -C package_write_rpm", it will fail with
rpmbuild missing the libprce library.
The reason is that libpcre-native fails to install from sstate (since
it isn't present) but doesn't get built and hence rpm-native tries to
run without its dependencies.
The simplest fix is not to add "covered" tasks which have failed to
install sstate. I can't help feeling there is more to this issue but
this does fix the current problem and shouldn't have adverse affects.
It is an unusual situation to have missing dependencies in sstate since
they're usually all present or not at all.
I've taken the opportunity to remove some old cruft from when we had
numeric task ids, the code can be simpler now.
(Bitbake rev: ba566b46d530b495f12f3a74f76434717b22a020)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
We can't execute the same task for the same package_arch multiple
times as the current setup has conflicting directories. Since
these would usually have the same stamp/hash, we want to execute in
sequence rather than in parallel, so for the purposes of task execution,
don't consider the "extra-info" on the stamp files. We need to add
a parameter to the stamp function to achieve this.
This avoids multiple update-rc.d populate_sysroot tasks executing in
parallel and breaking multiconfig builds.
(Bitbake rev: a9041fc96a14e718c0c1d1676e705343b9e872d3)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
This patch adds the notion of supporting multiple configurations within
a single build. To enable it, set a line in local.conf like:
BBMULTICONFIG = "configA configB configC"
This would tell bitbake that before it parses the base configuration,
it should load conf/configA.conf and so on for each different
configuration. These would contain lines like:
MACHINE = "A"
or other variables which can be set which can be built in the same
build directory (or change TMPDIR not to conflict).
One downside I've already discovered is that if we want to inherit this
file right at the start of parsing, the only place you can put the
configurations is in "cwd", since BBPATH isn't constructed until the
layers are parsed and therefore using it as a preconf file isn't
possible unless its located there.
Execution of these targets takes the form "bitbake
multiconfig:configA:core-image-minimal core-image-sato" so similar to
our virtclass approach for native/nativesdk/multilib using BBCLASSEXTEND.
Implementation wise, the implication is that instead of tasks being
uniquely referenced with "recipename/fn:task" it now needs to be
"configuration:recipename:task".
We already started using "virtual" filenames for recipes when we
implemented BBCLASSEXTEND and this patch adds a new prefix to
these, "multiconfig:<configname>:" and hence avoid changes to a large
part of the codebase thanks to this. databuilder has an internal array
of data stores and uses the right one depending on the supplied virtual
filename.
That trick allows us to use the existing parsing code including the
multithreading mostly unchanged as well as most of the cache code.
For recipecache, we end up with a dict of these accessed by
multiconfig (mc). taskdata and runqueue can only cope with one recipecache
so for taskdata, we pass in each recipecache and have it compute the result
and end up with an array of taskdatas. We can only have one runqueue so there
extensive changes there.
This initial implementation has some drawbacks:
a) There are no inter-multi-configuration dependencies as yet
b) There are no sstate optimisations. This means if the build uses the
same object twice in say two different TMPDIRs, it will either load from
an existing sstate cache at the start or build it twice. We can then in
due course look at ways in which it would only build it once and then
reuse it. This will likely need significant changes to the way sstate
currently works to make that possible.
(Bitbake rev: 5287991691578825c847bac2368e9b51c0ede3f0)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
With the introduction of multi-config and the possibility of distributed
builds we need arrays of workers rather than the existing two.
This refactors the code to have a dict() of workers and a dict of
fakeworkers, represented by objects. The code can iterate over these.
This is separated out from the multi-config changes since its separable
and clearer this way.
(Bitbake rev: 8181d96e0a4df0aa47287669681116fa65bcae16)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
A couple of fixes for the "Initialising tasks" progress bar behaviour:
* Properly finish the progress bar when using bitbake -S
* Finish the progress bar before calling BB_HASHCHECK_FUNCTION (so that
in OE when that shows its own "Checking sstate mirror object
availability" progress bar it gets shown on the next line as it
should).
(Bitbake rev: de6759d8e9990e426e6d6464a2e05381cd4c12d6)
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Runqueue errors direct the user to view the "failure below",
but no additional error message is available.
Log the stacktrace so that the user can see what went wrong.
Also fix a typo in the log message.
(Bitbake rev: e191f401e372ee181bc02250232ad9cb9a0e9477)
Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com>
Signed-off-by: bavery <brian.avery@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
When "Preparing RunQueue" shows up you can expect to wait up to 30
seconds while it works - which is a bit long to leave the user waiting
without any kind of output. Since the work being carried out during this
time is divided into stages such that it's practical to determine
internally how it's progressing, replace the message with a progress
bar.
Actually what happens during this time is two major steps rather than
just one - the runqueue preparation itself, followed by the
initialisation prior to running setscene tasks. I elected to have the
progress bar cover both as one (there doesn't appear to be much point in
doing otherwise from a user perspective). I did however describe it as
"initialising tasks".
(Bitbake rev: 591e9741e108487ff437e77cb439ef2dbca42e03)
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Add the ability to enter a mode where only a specified whitelist of
tasks can be executed outright; everything else must be successfully
provided in the form of a setscene task (or covered by a setscene task).
Any setscene failure outside of the whitelist will cause the build to
fail immediately instead of running the real task, and any real tasks
that would execute outside of the whitelist cause an immediate build
failure when it comes to executing the runqueue as well.
The mode is enabled by setting BB_SETSCENE_ENFORCE="1", and the
whitelist is specified through BB_SETSCENE_ENFORCE_WHITELIST, consisting
of pn:taskname pairs. A single % character can be substituted for the pn
value to match any target explicitly specified on the bitbake command
line. Wildcards * and ? can also be used as per standard unix file name
matching for both pn and taskname.
Part of the implementation of [YOCTO #9367].
(Bitbake rev: 624722c067a7fdd0c0f5d8be611e1f6666ecc4a2)
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
In 2c88afb6 find_chains()'s taskid argument was renamed to tid but
taskid is still used as key to explored_deps dictionary. Use tid instead
of taskid.
(Bitbake rev: 29a34ae8f5306d2779bcc761c52f1f9d13a0c0c5)
Signed-off-by: George McCollister <george.mccollister@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
I'm not sure what possesed me when I wrote this code originally but its
indirection of everyting to use numeric IDs and position dependent lists
is horrific. Given the way python internals work, its completely and
utterly pointless from performance perspective. It also makes the code
hard to understand and debug since any numeric ID has to be translated
into something human readable.
The hard part is that the IDs are infectous and spread from taskdata
into runqueue and even partly into cooker for the dependency graph
processing. The only real way to deal with this is to convert everything
to use a more sane data structure.
This patch:
* Uses "<fn>:<taskname>" as the ID for tasks rather than a number
* Changes to dict() based structures rather than position dependent lists
* Drops the build name, runtime name and filename ID indexes
On the most part there shouldn't be user visible changes. Sadly we did
leak datastructures to the setscene verify function which has to be
rewritten. To handle this, the variable name used to specifiy the version
changes from BB_SETSCENE_VERIFY_FUNCTION to BB_SETSCENE_VERIFY_FUNCTION2
allowing multiple versions of bitbake to work with suitably written
metadata. Anyone with custom schedulers may also need to change them.
I believe the benefits in code readability and easier debugging far
outweigh those issues though. It also means we have a saner codebase
to add multiconfig support on top of.
During development, I did have some of the original code coexisting with
the new data stores to allow comparision of the data and check it was
working correcty, particuarly for taskdata. I have also compared
task-depends.dot files before and after the change. There should be no
functionality changes in this patch, its purely a data structure change
and that is visible in the patch.
(Bitbake rev: 2c88afb60da54e58f555411a7bd7b006b0c29306)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Using positions in lists for flags is an odd choice and makes the code
hard to maintain. Maintaining a list is slow since list searches are
slow (watch bitbake -n slow massively with it) but we can use a set()
instead.
This patch uses python sets to maintain the lists of tasks in each state
and this prepares for changing the task IDs from being integers.
(Bitbake rev: 8c1ed57f6ea475b714eca6673b48e8e5f5f0f9c3)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Various misc changes to convert bitbake to python3 which don't warrant
separation into separate commits.
(Bitbake rev: d0f904d407f57998419bd9c305ce53e5eaa36b24)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The data included in the event is useful for implementing a pre-build
check that warns about unexpected components, for example because of
an incorrect configuration or changed dependencies.
Such a check can be done in a .bbclass that gets inherited
globally. But in contrast to a UI, such a class cannot request that
the event shall be emitted, and thus the event has to be emitted
whether there is a consumer or not.
This was done conditionally earlier out of concerns about the
performance impact. But now events are handled more efficiently, so
that concern no longer seems valid: in some simple testing (admittedly
on a fast build workstation), the two lines (generating the data and
emitting the event with it) only took about 0.05 seconds (measured
with timeit). That was for a build with roughly 500 recipes (from
pn-buildlist aka depgraph['pn']), triggered via the command line. That
was even with a consumer of the data active and doing some work, so it
should be even faster when there is no consumer.
(Bitbake rev: 5ddaf5b7ed1001d2dd3f67e7a6d704afa85479d2)
Signed-off-by: Patrick Ohly <patrick.ohly@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
python3 cares more about invalid type comparisons. Add break statements
and better tests to make the code paths clearer and avoid type issues
in python3. No code functionality change.
(Bitbake rev: 2c39ebdd2762d027f007a6a769fdf023cdf3da2b)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
We shouldn't try and use fakeworker when performing a dry_run. This
makes the core match the other fakeworker execution points.
(Bitbake rev: 49bea821a2edad5e19c3a566d1a80c23718dede9)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
python deprecated logger.warn() in favour of logger.warning(). This is only
used in bitbake code so we may as well just translate everything to avoid
warnings under python 3. Its safe for python 2.7.
(Bitbake rev: 676a5f592e8507e81b8f748d58acfea7572f8796)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
When multiple recipes which both provide something are being built, bitbake
informs us that most likely one of them provides something the other doesn't,
which is usually correct, but unfortunately it's rather painful to figure out
exactly what that is.
This patch dumps two sets of information, one is the provides information for
each recipe, filtered so only common components are removed. The other is a list
of dependees on the recipe, since sometimes this can easily identify why something
is being built.
Its not straightforward for bitbake to obtain the information but since the
warning/error code path isn't the normal one, we can afford to go through some
less than optimal processing to aid debugging.
Also provide the same information even if we're showing a warning since its still
useful.
[YOCTO #8032]
(Bitbake rev: 96fc889b8e62ba4463c71158c4b7286c48d68cd8)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Before this patch the usage of cache was quite useless as the file
checksums were not actually cached on disk but re-calculated every time.
This patch utilises the new writeout_file_checksum_cache() method of the
SignatureGenerator class to do the job.
(Bitbake rev: 5ac9cbf405841ed3f65e6f99a3cee032567fb182)
Signed-off-by: Markus Lehtonen <markus.lehtonen@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
This means that when you view the process tree, the processes
have meaningful names, aiding debugging:
$ pstree -p 30021
bash(30021)───KnottyUI(115579)───Cooker(115590)─┬─PRServ(115592)───{PRServ Handler}(115593)
├─Worker(115630)───bash:sleep(115631)───run.do_sleep.11(115633)───sleep(115634)
└─{ProcessEQueue}(115591)
$ pstree -p 30021
bash(30021)───KnottyUI(117319)───Cooker(117330)─┬─Cooker(117335)
├─PRServ(117332)───{PRServ Handler}(117333)
├─Parser-1:2(117336)
└─{ProcessEQueue}(117331)
Applies to parse threads, PR Server, cooker, the workers and execution
threads, working within the 16 character limit as best we can.
Needed to tweak the bitbake-worker magic values to tell the
workers apart.
(Bitbake rev: 539726a3b2202249a3f148d99e08909cb61902a5)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
This reverts commit b22592af81.
That commit isn't entirely clear about why this change is needed but
I do have a usecase where this breaks things. If for example you run
"bitbake X -c packagedata" and that packagedata is in sstate, you'd
expect this to work.
If sstate doesn't contain a do_populate_sysroot for a dependency, you
would still expect the command above to succeed and you would not
expect
it to rebuild that dependency. With the current code, this isn't what
happens. The code finds the sstate for do_populate_sysroot missing,
this makes the task "uncovered" and this in turn makes it unskippable.
The example I found with this was avahi-ui, where it would trigger
a build of libdaemon to obtain its populate_sysroot.
Since this behaviour seems completely incorrect, revert the older patch
and we'll address any issues that crop up as a result.
(Bitbake rev: 36a9840a5da17cc14561881fdd6a4f2cb0a75e49)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Debugging suggests that setscene tasks are being a little greedy about their
dependencies, for example, lsof is insisting that gcc-runtime's do_package
is installed. If it isn't, its requiring gcc to rebuild.
If gcc-runtime do_package_write_xxx and do_packagedata is available, there
is no reason do_package should be needed.
The reason this is happening appears to be from the batching up of task
dependencies code, rather than setscene tasks stopping when passing over
a setscene task, they were being carried forward. This patch fixes it
so the data is 'zeroed' when passing over a setscene task boundary,
which gives the dependency graph that is expected.
After this patch, lsof will rebuild quite happily without
gcc-runtime:do_package being present, as expected. This should lead to
less dependencies being installed for builds from sstate and generally
better performance in general.
(Bitbake rev: f8bcb0a1e3b008b71c9a7cd21f76d0906f2d8068)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Added optional parameter 'prefix' to filter out names that
don't start with specified prefix. Changed existing call
of get_providermap according to changed API.
Optimized the code: got rid of extra loop and temporary
list variable virts.
(Bitbake rev: df5a1392d6f91ccb44a99721c7d847da242121bb)
Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com>
Signed-off-by: Elliot Smith <elliot.smith@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Its turning out that we really need a way to have bitbake just run
the setscene tasks but not any real tasks, particularly for SDK
operations.
Add an option for this since its pretty straight forward. This allows
various nasty workarounds in OE-Core to be removed.
(Bitbake rev: e4a2aafa1650a227a04d92a8a0b31efaed2c310e)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
It can be useful to run all tasks up to but not including a specific task. The
main reason this was never added was the lack of a good syntax. This patch
uses the syntax <taskname>- to denote this behaviour which is simple, not
invasive and fits what we need from good syntax IMO, hence we can add this.
(Bitbake rev: 99ccfd411ab3f7baa111f9f3d50fae68816a9a83)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
This firstly prints debug messages which show how bitbake decided to resolve
the virtual/xxx providers which is useful for debugging.
If the siggen has a tasks_resolved() method, it calls this, passing in
the mappings, allowing that to do things with the resolved names.
(Bitbake rev: d473fc84acddfd69a7207affcd89f65ea2ecf730)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The output of "bitbake, -c listtasks pkg" lists tasks with their real names
(starting with "do_"), but then "bitbake -c do_task" fails, as "do_" always
gets unconditionally prepended to task names. This patch handles this error
by checking whether a task starts with "do_" prior to prepending it with it
when the task runlist is being constructed (and a few other corner cases).
[YOCTO #7818]
(Bitbake rev: dd3050ceef37ac556546e940aa596ce96ef6c8df)
Signed-off-by: Alex Franco <alejandro.franco@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Currently bitbake is the only one that knows the relation
between PN and PROVIDES. In some cases it is needed to have
this relation in the data store (the bootloader it's a good
case).
This adds the PROVIDES to the taskdata, so it would be in
the data store as one field of BB_TASKDEPDATA.
(Bitbake rev: a660787311d2961c66c0443bf0e2e094c9baef1b)
Signed-off-by: Mariano Lopez <mariano.lopez@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Bitbake does really weird things with negative or zero numbers of threads
which is confusing to the user. Add a sanity check for this.
When you have code doing arithmetic on the values and a VM reconfigures
to only a single thread, negative numbers are easier than you'd think.
(Bitbake rev: 32166ac3c85ff3c04081580ae76bd63590d6ff3e)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
If we see a BBHandledException in runqueue, the understanding is the system
handled it, printing a log and traceback is just confusing.
Therefore only print these in the cases where its an unknown/unhandled
exception.
(Bitbake rev: 29d28e22ce431c3d3aabdb88ff4d8cca67a1cfad)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Currently if the fakeworker failes to start the output from bitbake is confusing.
Improve the error handling to give a clear indication of what failed.
Patch from Chris Larson.
(Bitbake rev: ad286d6fed7a580bec36a92c7b7e205322ac407b)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
In some cases we need to check specifically for siginfo files, in
some cases we need to check for the actual sstate objects themselves.
Therefore make this a parameter to the function. A fallback to the
previous function style is maintained for now.
(Bitbake rev: 18d3a03e1b07c98b2dce46eb94f30de1a2b4320b)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
This ensures that _append, _prepend, overrides, etc are functional when used
on sstate variables (e.g. SSTATE_DIR).
[YOCTO #7564]
(Bitbake rev: 2e683c25b856b431198573f7f352d841587275e6)
Signed-off-by: Christopher Larson <kergoth@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
After Ctrl+C is pressed to interrupt bitbake, it loops continually, running
at 100% cpu. This patch selects on the correct file descriptors resolving
the excess cpu usage.
(Bitbake rev: 497404e8484b7ca7c11e459bf0845642156eb677)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
This has always bothered me:
NOTE: Preparing runqueue
NOTE: Executing RunQueue Tasks
This patch changes the messages to be consistent.
(Bitbake rev: 72ac9f9227fbfb4dc8b933b357d21aa0e4060959)
Signed-off-by: Gary Thomas <gary@mlbassoc.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
We need to transfer some of the siggen data from the core/cooker into
the worker instances. There was a partial API created for this but
its ugly and its not possible to extend it from the siggen class.
This patch completes the interface/abstraction for the data and
means the class can extend/customise it in any siggen class.
(Bitbake rev: cf2d642052979d236185c5b8ca2c5478c06e62ae)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Currently we get no profiling oversight into either the main bitbake worker
process, or the overall parsing before task execution. This adds in extra
profiling hooks so we can truly capture all parts of bitbake's execution
into the profile data.
To do this we modify the 'magic' value passed to bitbake-worker to trigger
the profiling, before the configuration data is sent over to the worker.
(Bitbake rev: 446e490bf485b712e5cee733dab5805254cdcad0)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Currently, if you have hard dependencies between setscene tasks (like avahi on
base-passwd through useradd.bbclass), other dependencies may not be installed
even if these exist in sstate. For example, avahi -> expat -> pigz-native
(and avahi -> base-passwd) yet if you cleansstate base-passwd:
bitbake gzip-native:do_clean avahi:do_clean expat:do_clean pigz-native:do_clean base-passwd:do_cleansstate
bitbake avahi | tee
you will currently see pigz-native being rebuilt even though it was in
sstate. The fix for this is to continue to iterate dependency chains
around hard blocked dependencies as per this patch.
After this patch is applied, you will see pigz-native installed from sstate.
(Bitbake rev: f787957a224e8c2682a19e5c4a4d9c86bdce52ba)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
(Bitbake rev: 000fa81013205dd9bc907ff7a61f06f57637212d)
Signed-off-by: Robert P. J. Day <rpjday@crashcourse.ca>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
(Bitbake rev: 4e9aef14d747c37444a4fc683f9641906906afe9)
Signed-off-by: Robert P. J. Day <rpjday@crashcourse.ca>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Its useful to have an event emitted when all of the sceneQueue tasks
have completed since the metadata can hook this for processing.
Therefore add such an event.
(Bitbake rev: 38d4f65bf1cbcdd5a2d60dff0e1d2859c34ed62e)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
If you hit Ctrl+C at the right point, the system processes the request
but merrily continues building. It turns out finish_runqueue() is called
but this doesn't stop the later generation and execution of the
runqueue.
This patch adjusts some of the conditionals to ensure the build really
does stop.
(Bitbake rev: 39b08c604ba713100e174c136f81f18eca6ef33d)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
In dry run mode, stamps for noexec tasks are being written out which
is incorrect. Avoid this.
(Bitbake rev: aa6448a0552ba2947ac262b8b5314a593d1058d3)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
When looking at a list of tasks, do_patch and do_unpack were being
given equal priority when one clearly depends on another. The
reason for this was the default task weights of 0 being to tasks.
This is therefore changed to 1 to allow correct weighting of dependencies
which means the scheduler has better information available to it about
tasks.
Weight endpoints differently (10) for clearer debugging of priorities.
(Bitbake rev: 12dc1d17fac3e8ec420fcafb06186d32fd847d89)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The zero priority task should be run first but was being confused with
the None value the priority field defaulted to. Check for None
explicitly to avoid this error.
In the real world this doesn't change much but it confused the debug
output from the schedulers.
(Bitbake rev: 49c9d8c9400f74c804c2f36462639236e0841ff0)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The first part of the sstate code checks en-mass whether given checksums
are available. The next part of the code then either triggers those
setscene tasks either running them or skipping them if they've been
covered by others.
The problems was that this second part would always skip a task if it
was unavailable in the first part, even if it would have otherwise been
covered by other tasks.
This mean the mere presence of an artefact (or lack of presence) could
cause a different build failure.
The issue reproduces if you run a build and populate an sstate feed, then
run a second build off that feed, then run a third build off the sstate
feed of the second build (which is reduced in size).
The fix is rather than immediately skipping tasks if the checksum is
unavailable, create a list of missing tasks, then, if that task cannot
be covered by others we can skip it later. The deferral makes the
behaviour the same even when the cache is "incomplete".
[YOCTO #6081]
(Bitbake rev: 5edb1a3e3f454ba6e65551174d86229db2f99636)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
If a setscene task has [depends], its possible they may still get executed out
of order. The issue is that the dependencies are set to set() for all tasks
involved. This patch adds back in explict dependencies within these chains
to avoid the setscene task failures.
[YOCTO #6069]
(Bitbake rev: 724c889eed3b03d3199810c185086d3973af826c)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
This allows the commandline options to be processed in the dump signature
code.
(Bitbake rev: ef8537a2e9b48f4fe065a165c102935aee2c9029)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
There is no easy way to make this change. We really need parameters for the -S
(dump signatures) handling code. Such a parameter can then be used within the
codebase to handle the signatures in different ways.
For now, "none" is the recommended default and "printdiff" will execute the
new (and more expensive) comparison algorithms.
(Bitbake rev: b9873588696507dfb6aade6821f6f75cb9a19e0a)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The runqueue should be using the "realtask" ID to lookup the task
hash, not the "task" ID. This patch resolves corruption issues where
incorrect task hashes were displayed within toaster.
(Bitbake rev: 84be1a27f89d1bf63c21f06d831df0a66a5db860)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Use of waitpid on the worker processes is a bad idea since it conflicts
directly with subprocess internals. Instead use the poll() method
and returncode to determine if the process has exitted, if it has,
we can shut down the system.
This should resolve the hangs once and for all, famous last words.
(Bitbake rev: 60969cd62e21e7d4af161bf8504b7643a879c73f)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
We're running into processes using 100% cpu. It appears theses are locked in
a subprocess.poll() type loop where the process has exited but the code is
looping as its not handling the ECHILD error.
http://bugs.python.org/issue14396http://bugs.python.org/issue15756
This is likely due to one or both of the above bugs. The question is what actually
grabbed the child exit code as it wasn't this code. Its likely there is therefore
some other code racing and taking that code, it may be some kind of race like:
http://hg.python.org/cpython/rev/767420808a62/
where the fix effectively catches the childs codes in a different part of the system.
We could try and get everyone onto python 2.7.4 where the above bugs are fixed however
for now its safer to admit defeat and go back to polling explictly for our worker exit
codes.
(Bitbake rev: 5b9a099ec2a1dc954b614e12a306595f55b6a99e)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Catching all child exit status values is a bad idea. Setting an http sstate mirror
is a great way to view that spectacularly break things. The previous change did
have good code changes so don't revert those parts.
(Bitbake rev: fa7ffb62d510ac1124ae7e08fa4d190a710f5b54)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
There are several problems. Firstly, a return value of "None" can mean
there is a C signal handler installed so we need to better handle that
case. signal.SIG_DFL is 0 which equates to false so we also need to
handle that by testing explicitly for None.
Finally, the signal handler *must* call waitpid on all child processes
else it will just get called repeatedly, leading to the hanging behaviour
we've been seeing. The solution is to only error for the worker children,
we warn about any other stray children which we'll have to figure out the
sources of in due course.
Hopefully this patch gets things working again properly though.
(Bitbake rev: 973876c706f08735c1b68c791a5a137e5f083dd2)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Failures on the autobuilder look like this handler is recursing. That
shouldn't be possible but it doesn't hurt to code as such.
(Bitbake rev: e39e85803cbe1ef9413a118868c19087c0546d01)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
We've noticed hanging processes which appear to be looping around
waitpid. Its possible multiple calls to teardown are causing problem
or in theory multiple registrations (although the code should not
allow that). Regardless, put better guards around signal handler
registration.
(Bitbake rev: 79acfb0853aa3215215cee89a945f8e97b0a8fae)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
If we didn't setup any workers (such as bitbake -S), this would error
since we're trying to set a signal handler to None. This patch
avoids that problem.
(Bitbake rev: ce17478c8197abf178c00774f5bbe23fd4375ee2)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The sigchld handler was reaping any processes and this was leading to
confusion with any other process handling code that could be active.
This patch:
a) Ensures we only read any process results for the worker processes
we want to monitor
b) Ensures we pass the event to any other sigchld handler if
it isn't an event we're interested in so the functions are properly
chained.
Together this should resolve some of the reports of unknown processes
people have been reporting.
(Bitbake rev: fe8baaa2f533db7a1b7203476c675588923d8d45)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Instead of a significant number of calls to waitpid, register a SIGCHLD
handler instead.
(Bitbake rev: 76029d08ad56a0a264ff9738a0336971a455b7f5)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
A run of "bitbake bash -c unpack" when the task has already been
completed resulted in about 9000 calls to logger.debug(). With this
patch which comments out some noisy/less usefull logging and moves
other logging calls outside loops, this number is reduced to 1000
calls. This results in cleaner logs and gives a small but
measurable 0.15s speedup. The log size dropped from 900kb to 160kb.
(Bitbake rev: d2677f084fe1d8846db77d89ef5e6ffb18dc171a)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
If the worker has already gone missing (e.g. SIGTERM), we should
gracefully handle the write failures at exit time rather than throwing
ugly tracebacks.
(Bitbake rev: 1b1672e1ceff17fc4ff8eb7aa46f59fce3593873)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
If the worker (or fakeworker) process disappears for some reason, the
system doesn't currently even notice. To fix this, we call waitpid
periodically, looking for exit events of our children. If these
occur, we can gracefully shutdown the server.
(Bitbake rev: ee28ddadaa7ef91e4d4b7d22fc267382aaad6d01)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
slef.self is clearly meant to be self, fix typo.
Otavio spotted and reported, thanks.
(Bitbake rev: 316daad7928a58cdfc42e27b20e739f4dd74a02a)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* exception like this keeps spinning quite quickly generating GBs of logs
better to kill it asap and show invalid pickle
(Bitbake rev: a69eb4c12c71bba9d742c4e5578f25c388d9f825)
Signed-off-by: Martin Jansa <Martin.Jansa@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
A previous commit of mine used the target variable for two different uses
resulting in a lot more sstate being installed than is needed.
Fix the variable to use two different names and unbreak the setscene
behaviour.
(Bitbake rev: f975ca2cf728561bd6317ed8f76303598546113a)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
If you specify multiple targets on bitbake's commandline and some of them are
setscene tasks which are "masked" by other tasks they may not get run.
For example <image>:do_rootfs <kernel>:do_populate_sysroot
the rootfs tasks "masks" the populate_sysroot task so bitbake would currently
decide not to run it. In this case, we do really want it to be run.
The fix is not to skip anything which has been given as an explict target.
(Bitbake rev: 0753899d1e855795cc18671357609a86f169b379)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Commit c54e738e2b5dc0d8e6fd8e93b284ed96e7a83051 added in the idea of hard dependencies
such as the case a setscene has a hard dependency on pseudo-native and that
dependency wasn't available from sstate for some reason.
Unfortunately the implementation was a bit too enthusiastic, causing rebuilds
of things when it wasn't necessary. A test case was:
bitbake quilt-native
bitbake quilt-native -c clean
bitbake <some-image>
and then you'd watch quilt-native get rebuilt for no good reason.
The clue to the problem is in the for loop where it never depends on
the item being iterated over.
The fix is to include the exact list of hard dependencies rather than
guessing. With these changes, the use case above works, the one in
the original commit also works.
This patch also adds in or cleans up various pieces of logging to
allow issues like this to be more easily debugged in future.
(Bitbake rev: 81bd475585ff1b44b390036b1eca0feae7c149eb)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Shared work directories work by assuming bitbake will not run
more than one task with a specific stamp name. Recent runqueue optimisations
accidentally broke this meaning there could be races. This fixes the code.
(Bitbake rev: b1628b1a260ddf43fc9985535b1ddcfcebbb1e5b)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Based upon the list of difference starting points, we can use the siggen.find_siginfo()
function call and the difference printing code to provide a list of differences
between the current build target and whatever can be obtained from the sstate cache.
(Bitbake rev: 7a77861feb62750ef166d2d1e89ed1f444ca8dc7)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The way hash_deps was being generated was different to the way siggen generated
the data internally which lead to seemingly different sigdata/siginfo files
for the same checksum. The -S output correct but the files written during
builds contained superflous data which would look like a difference.
This patch removes the badly duplicated data and uses it from the source
which ensures its consistent.
(Bitbake rev: e6d5e925c402cd2cc7ee034e9de4cc6df8944a34)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Its useful to understand where the delta starts against an existing sstate cache
for a given target. Adding this to the output of the -S option seems like a
natural fit.
We use the hashvalidate function to figure this out and assume it can find siginfo
files for more than just the setscene tasks.
(Bitbake rev: c18b8450640ebfd55a2b35b112959f9ea3e0a700)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Currently tasks have no knowledge of which other tasks they depend
upon. This makes it impossible to do at least two things which would be
desirable/interesting:
a) Have the ability to create per recipe sysroots
b) Allow the aclocal files to be present only for the entries in
DEPENDS (directly and indirectly)
By exporting task data through this new variable, tasks can inspect
their dependencies and then take actions based upon this.
(Bitbake rev: 84f1dde717dac22435005b79d03ee0b80a3e8e62)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
This unlikely looking function was found to be eating a lot of CPU time
since it gets called once per trip through the idle loop if we're not
running a maximum number of processes. This was particularly true in
world builds of 13,000 tasks.
Calling the computation code is pretty pointless because until some
other task finishes nothing is going to become available to build.
We can know when things become available so this patch teaches the
scheduler this knowledge.
It also:
* skips any coputation when nothing can be built
* if there is only one available item to build, ignore the priority map
* precomputes the stamp filenames, rather than doing it every time
* saves the length of the array rather than calculating it each time
(the extra function overhead is significant)
Timing wise, initially, 5000 iterations through here was 20s, with
the patch 200000 calls takes the same time. The end result is that
builds get up and running faster.
(Bitbake rev: 4841c1d37c503a366f99e3a134dca7440e3a08ea)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
When using the dry run option (-n), bitbake would still try and fire
a specific fakeroot worker. This is doomed to failure since it might
well not have been built.
Add in some checks to prevent the failures.
[YOCTO #5367]
(Bitbake rev: f34d0606f87ce9dacadeb78bac35879b74f10559)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
We have do_bundle_initramfs which is a task inserted after compile and
before build. It is not covered by sstate.
If we run a build with a valid sstate cache present, the setsceneverify
function realises it will rerun the do_compile step (due to the
bundle_initramfs task) and hence marks do_populate_sysroot to rerun.
do_install, a dependency of do_populate_sysroot is left as marked as
covered by sstate.
What we need to do is traverse the dependency tree for any setsceneverify
invalided task and ensure any dependencies are also invalidated. We can
stop at any point we reach another setscene task though.
This means the do_populate_sysroot task has the data from do_install
available and doesn't crash.
(Bitbake rev: f21910157d873c030b149c4cdc5b57c5062ab5a6)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Adding the sstate-related hash for all runqueue and
scenequeue tasks, as it's needed in the WebHob data.
(Bitbake rev: b6e2ce1cf7a0ede890f08fabf536a556dc4263c5)
Signed-off-by: Alexandru DAMIAN <alexandru.damian@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
This patch adds task identifying information for all
runQueue and sceneQueue events, and for bb.build.Task* events.
This will allow matching event to specific tasks in the UI
handlers processing these events.
Adds RunQueueData functions to get the task name and task
file for usage with the runQueue* events.
Adds taskfile and taskname properties to bb.build.TaskBase.
Adds taskfile and taskname properties to the *runQueue* events
(Bitbake rev: b4a5e4be50d871a80dbe0993117d73f5ad82e38f)
Signed-off-by: Alexandru DAMIAN <alexandru.damian@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Adding a CookerFeature that allows UIs to enable
receving a dependency tree once the task data has been
computed and the runQueue is ready to start.
This will allow the clients to display dependency
data in an efficient manner, and not recompute the runqueue
specifically to get the dependency data.
(Bitbake rev: 75466a53b6eece5173a9bfe483414148e4c06517)
Signed-off-by: Alexandru DAMIAN <alexandru.damian@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Adding a runQueueTaskSkipped to notify that the tasks that are not
run either because they are set-scened or they don't need an update
(timestamp was ok).
(Bitbake rev: cf4a0c7aa82090876ae652b611acfab3ce2688f7)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Adding an event to be fired when a scene task is completed.
It is analogous to the run task completed event, and has
been missing for some reason.
(Bitbake rev: 73b8f4d3fbeaf1b330a66d76012d0a5cef8dbe2d)
Signed-off-by: Alexandru DAMIAN <alexandru.damian@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
There would be an race issue if we:
$ bitbake make-3.81 make-3.82
This because they are being built at the same time which would cause
unexpected problems, for example:
[snip]
ERROR: Package already staged (/path/to/tmp/sstate-control/manifest-qemux86-make.populate-sysroot)?!
ERROR: Function failed: sstate_task_postfunc
[snip]
Or there would be python's strack trace such as:
[snip]
*** 0004: mfile = open(manifest)
0005: entries = mfile.readlines()
0006: mfile.close()
0007:
0008: for entry in entries:
Exception: IOError: [Errno 2] No such file or directory: xxx
[snip]
[YOCTO #5094]
We can quit earlier to avoid this kind of issue when two versions of the same PN
are going to be built since this isn't supported.
(Bitbake rev: ab377c00c33a2d296bfda1b0b6c2a62b29d1004f)
Signed-off-by: Robert Yang <liezhi.yang@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Due to the worker split the ${DATE} and ${TIME} variables could end up
with different values for different workers.
E.g., a task like do_rootfs that is run within a fakeroot environment
had a slightly different view of the time than another task that was not
fakerooted which made it impossible to correctly refer to the image
generated by do_rootfs from the other task.
(Bitbake rev: 756cc69ebf8bfe8455d0c90f288dd51be2499773)
Signed-off-by: Peter Kjellerstedt <peter.kjellerstedt@axis.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
BUILDNAME is set from cooker by default, so since the worker split it
will not be set when executing functions. In OpenEmbedded this results
in /etc/version (which is populated from BUILDNAME) not having any
content. Pass this variable value through to the worker explicitly to
fix the issue.
Fixes [YOCTO #4818].
(Bitbake rev: 92940b0427d9b2b3f95e27c230ec1e36638a34bc)
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
This was missed off in a previous patch.
(Bitbake rev: ad7664edd40fa46e6f6fec2144403e3b6fc3a639)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The existing backend server implementations were inefficient since they
were sleeping for the full length of the timeouts rather than being woken when
there was data ready for them. It was assumed they would wake and perhaps did
when we forked processes directory but that is no longer the case.
This updates both the process and xmlrpc backends to wait using select(). This
does mean we need to pass the file descriptors to wait on from the internals
who know which these file descriptors are but this is a logical improvement.
Tests of a pathaolgical load on the process server of ~420 rapid tasks
executed on a server with BB_NUMBER_THREAD=48 went from a wall clock
measurement of the overall command execution time of 75s to a much more
reasonable 24s.
(Bitbake rev: 9bee497960889d9baa0a4284d79a384b18a8e826)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Help to pick up mistakes such as "bitbake -c cleanstate xyz" (instead
of "bitbake -c cleansstate xyz".)
(Bitbake rev: 15c3db1cffdffd85641c6b12e77f19ce7a553472)
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Add a warning if 'bitbake -C' is executed with a task that does not
exist.
Fixes [YOCTO #4877]
(Bitbake rev: 6459c1d0eb8f1007246df36149e2496ee531e25f)
Signed-off-by: Petter Mabäcker <petter@technux.se>
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Currently, tasks like fetchall are slightly broken since if a recipe
has specific [depends] which occur after do_fetch and add items not listed
in DEPENDS and RDEPENDS, they are not caught by recrdeptask. We've gone
around in circles on this issue (e.g
http://git.yoctoproject.org/cgit.cgi/poky/commit/bitbake/lib/bb/runqueue.py?id=5fa6036d49ed7befe6ad50ec95c61a50aec48195
) and in many cases the behaviour of recrdepends is correct but tasks like
fetchall need the other behaviour.
To address this we add a recideptask flag which can be used in conjuction
with the recrdeptask flag to specify which task to to the inspection upon.
This means entries like do_rootfs[depends] which have do_fetch tasks are
caught and run.
I'm not 100% happy with needing another flag but I don't see any rational
way to get the correct behaviour in all cases without it.
[YOCTO #4597]
(Bitbake rev: f8c9b292b02ce2c28741b74901205f5e5807ca87)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
If a setscene task has a hard dependency on a task like pseudo-native, its
expected that the setscene task will not run unless the dependency is met.
This adds code to ensure that is the case, otherwise a bug would show up
with a usecase like:
bitbake gnome-common
bitbake pseudo-native -c cleansstate
bitbake gnome-common -c clean
bitbake gnome-common
With the double wrapper script environment, we'd not see issues like
this as it would be masked. The problem theoretically affects code like
useradd too as well as anything using a sstate postinstall.
(Bitbake rev: c54e738e2b5dc0d8e6fd8e93b284ed96e7a83051)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
With the change to bitbake-worker we need to ensure the workers know
how to contact the PR service, the magic 0 port and singleton is
no longer enough.
(Bitbake rev: c761751e259bb8e940552a28794b45887b5a72d9)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
We're going to need a fakeroot/pseudo version of the worker so
abstract the code to start the worker process.
(Bitbake rev: b5d0f12f9df3ab211700473ed145ee6fbd9ca8e9)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The worker was being executed by each execution queue so would get
constructed twice for each build. This is wasteful so move execution
to the main runqueue so we only have to start the worker once.
(Bitbake rev: 8117f8480125b121b2b5ac0afc31b108d9e670ae)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
This is a pretty fundamental change to the way bitbake operates. It
splits out the task execution part of runqueue into a completely
separately exec'd process called bitbake-worker.
This means that the separate process has to build its own datastore and
that configuration needs to be passed from the cooker over to the
bitbake worker process.
Known issues:
* Hob is broken with this patch since it writes to the configuration
and that configuration isn't preserved in bitbake-worker.
* We create a worker for setscene, then a new worker for the main task
execution. This is wasteful but shouldn't be hard to fix.
* We probably send too much data over to bitbake-worker, need to
see if we can streamline it.
These are issues which will be followed up in subsequent patches.
This patch sets the groundwork for the removal of the double bitbake
execution for psuedo which will be in a follow on patch.
(Bitbake rev: b2e26f1db28d74f2dd9df8ab4ed3b472503b9a5c)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The reasons for combining these objects is ancient history, it makes
sense to clean things up and separate them out now. This follows on
logically from the configuration cleansups and leads well into the
bitbake-worker changes.
(Bitbake rev: 89ffd62661ebcf2a97ce0c8dfd5e4d5bfbe27de7)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
This means the variables show up in the shell execution "run" files since
its useful to know what the fakeroot environment is and how to set it up
manually.
(Bitbake rev: bdf437747b664479acde6deaa9096e2a6bcdf483)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The Cooker class is too large and needs to be split up into different
functional units. Splitting out the collections code into its own class
seems like a good place to start to try and disentangle things.
(Bitbake rev: ca1fcbb6e214c155a05328779d3d326e10c5eac0)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* dep variable was removed in
commit 3190cb83e2af195a464f669c5aa8aedbf795160e
Author: Richard Purdie <richard.purdie@linuxfoundation.org>
Date: Wed Jun 27 11:04:06 2012 +0100
taskdata: Add gettask_id_fromfnid helper function
This is like gettask_id but doesn't require translation of fnid -> fn
first which the function then translates back. This gives a sizeable
performance improvement since a significant number of lookups are avoided.
* now it fails completely instead of showing which task is missing
(Bitbake rev: 58847fabd389e5b8d02d5a9c6827aabedb30312f)
Signed-off-by: Martin Jansa <Martin.Jansa@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
When the setscene code was originally written it was thought that we'd
allow "partial" coverage. For example, if we just want to build the target
"bash:do_populate_sysroot" and its available from sstate, it makes no sense
to install gcc-cross's sstate package as its simply not needed.
Due to various other issues in the codebase, this functionality was
disabled/removed to allow the setscene code and sstate to stabilise and allow
us to concentrate on other problems.
The time has now come to enable "partial" coverage. There are two major changes
in this patch:
a) Creation of an unskippable list. This lists direct dependencies of
build targets and hence things that cannot be skipped.
b) Addition of a handler which looks at a given setscene target and what depends
on it and then decides whether its necessary.
(Bitbake rev: 2a937cd6a6c3110030b40bc4d85e349b85cb4db7)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Mainly intended for the purpose of debugging or forcing builds
from source, the --no-setscene will prevent any setscene
tasks from running.
(Bitbake rev: 440e479f3e248482c38c149643403c6907ac7034)
Signed-off-by: Jason Wessel <jason.wessel@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
If a target dependency is marked as failed and yet we are continuing on
because -k has been specified, don't try to access the dependency's data
in taskData.build_targets since it will have been removed. This fixes
"IndexError: list index out of range" errors in this situation.
Also, do not print the "unhandled exception" message when SystemExit is
raised since we will have reported the actual error already in this
case (e.g. when -k has been specified and some targets failed).
Fixes [YOCTO #3133].
(Bitbake rev: 70eebc184eb1ab3678be87bed019b5beadecdc89)
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The stamp files can change during setscene and the cache should be cleared to
account for this.
(Bitbake rev: 5ec12f586a50fce675b268965b3dc487aaa96c43)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
There was some odd behaviour if some task was run from setcene whilst there were
existing valid stamps for a depepdency. For example, do_populate_sysroot might
be installed at setscene time but if there were other tasks not installed from
setscene such as do_populate_lic which depend on do_configure, the setsceneverify
function would think that do_configure needed to be rerun and would hence void the
do_populate_sysroot and force that to rerun too.
The setsceneverify function needs to know which tasks are going to be rerun, not just
what the overall task list is and what setscene functions have run. This patch adds
that information and maintains backwards compatibility in a slightly ugly but effective
way. The metadata needs updating to take advantage of this change.
(Bitbake rev: 1423aafff97f17169e95ec3ba973eb002ff98c1c)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The scenequeue code could result in one setscene function masking out another
which had been marked as notcovered. This change ensures the notcovered list
is taken into account when deciding which tasks need to be run, ensuring
that tasks that should be run do get run.
(Bitbake rev: 95b31127d8ba12ccb061d67481ece134c18c39f2)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Currently we call the hashvalidate for anything without a valid setscene stamp.
This improves the code to account for existing stamp files so that we only process
setscene hashes for things we might actually end up using. This avoids hash processing
and makes things slightly more efficient and the logs less confusing.
(Bitbake rev: 2e71a65bd910e8be894b8284eb0a50de7270c081)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
There was a bug where an exception in the runqueue code would cause an infinite loop
of debug messages. The exception would get reported but would loop since runqueue was
still registered as an idle handler.
This patch adds an exception handler to ensure in the case of errors, the system
more gracefully shuts down and doesn't loop.
(Bitbake rev: 99467c7e387f6e71358b40b8afae450d72cd68e0)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
We check the stamp cache before comparing t2 and t3 which means that we can miss
a level in the stamp file chains. The result of this is that a stamp can be accepted
as valid when in fact it isn't. Some weird behaviour alerted me to this in a local
build.
This patch also fixes to only uses the cache in recurse mode, there was a corner
case where stamps not in recurse mode could get added to the cache which could cause
an issue potentially.
(Bitbake rev: 7a5f776d325c0e37f58003424beda7cae2695325)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
In some cases we want to pull in DEPENDS and RDEPENDS of recrdeptask
dependencies but we need a way to trigger or avoid this behaviour
depending on context. The logical syntax to trigger such behaviour
would be a self referencing recrdeptask:
do_a[recrdeptask] = "do_a do_b"
The dependency chains already recurse this kind of expression correctly, the
missing piece is to avoid any circular reference errors. Since the dependencies
have already been recursively resolved, simply removing any recrdeptask
references is enough to break the circular references.
This patch therefore removes any circular references using the set
difference_update() operator. There will be metadata tweaks required to
add any references needed to the extra taskname.
(Bitbake rev: a5324da9b8a0c9307a6c511ea9009f34be70c92b)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
I'm not sure why we don't currently allow multiple entries in rdeptask when
we do in deptask. This makes the handling match between the two since
its trivial to fix.
(Bitbake rev: 19c84fe8854639768c874cc1449963a9867ad397)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Currently, recrdepends is extremely greedy. For example:
do_foo[rdepends] = "somedep:sometask"
addtask foo
which adds foo with *no* dependencies, will suddenly start appearing
as a dependency in every task which uses recrdepends. So far this has
been mildy annoying but we now have use cases where this makes no sense
at all.
This reworks the recrdepends code to avoid this problem. To do this we
can no longer collapse things into lists just based on file ID. The problem
is this code is extremely performance sensitive. The "preparing runqueue"
phase spends a lot of time in these recursive dependency calculations so any
change here could negatively impact the user experience.
As such, this code has been carefully tested on convoluted dependency trees
with operations like "time bitbake world -g". The net result of this change
and the preceeding changes combined is a net speed up of these operations in
all cases measured.
Tests were made comparing "bitbake world -g" task-depends.dot before and after
this patch. There *are* differences for example -nativesdk do_build dependencies
on -native recipes are no longer present. All removed dependencies appear to
be sensible improvements to the system. The "rdepends" cross contamination
issue above is also fixed.
(Bitbake rev: 82d73423c57569b984ee0ae3d93e3c3bd5dc5216)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
This gives some small performance gains and sets the scene for other
improvements by removing the need for duplicate detection code.
(Bitbake rev: 6fd723479e8d49227fd58040b3485c1d5afc4bc5)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
This is like gettask_id but doesn't require translation of fnid -> fn
first which the function then translates back. This gives a sizeable
performance improvement since a significant number of lookups are avoided.
(Bitbake rev: 3190cb83e2af195a464f669c5aa8aedbf795160e)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Currently its not possible to add arbitrary RDEPENDS to a specific task.
This can be useful and this patch adds functionality equivalent to the
'depends' task flag.
(Bitbake rev: db65080a6199baecc5c422294a4c4a9fd12dc29e)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
This new command line option forces the specified task and all dependent
tasks up to the default task to re-run. This means that the following
single step:
bitbake -C compile somerecipe
is equivalent to the following two steps (with the recent change to -f):
bitbake -c compile -f somerecipe
bitbake somerecipe
Note that to work this option needs full hashing enabled (i.e.
BB_SIGNATURE_HANDLER must be set to a signature handler that inherits
from BasicHash). If this is not the case, -C effectively does nothing.
Based on a previous implementation of this option by Jason Wessel
<jason.wessel@windriver.com>.
Implements [YOCTO #2615].
(Bitbake rev: 2530e0faada5775897cfd1b93aba6925826dca73)
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
If -f is specified, force dependent tasks to be re-run next time. This
works by changing the force behaviour so that instead of deleting the
task's stamp, we write a "taint" file into the stamps directory, which
will alter the taskhash randomly and thus trigger the task to re-run
next time we evaluate whether or not that should be done as well as
influencing the taskhashes of any dependent tasks so that they are
similarly re-triggered. As a bonus because we write this file as
<stamp file name>.taskname.taint, the existing code which deletes the
stamp files in OE's do_clean will already handle removing it.
This means you can now do the following:
bitbake somepackage
[ change the source code in the package's WORKDIR ]
bitbake -c compile -f somepackage
bitbake somepackage
and the result will be that all of the tasks that depend on do_compile
(do_install, do_package, etc.) will be re-run in the last step.
Note that to operate in the manner described above you need full hashing
enabled (i.e. BB_SIGNATURE_HANDLER must be set to a signature handler
that inherits from BasicHash). If this is not the case, -f will just
delete the stamp for the specified task as it did before.
This fix is required for [YOCTO #2615] and [YOCTO #2256].
(Bitbake rev: f7b55a94226f9acd985f87946e26d01bd86a35bb)
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
This function was used by old code such as packaged staging but is thankfully
obsolete now and replaced with better mechanisms. Its time to remove it and
the horrible internal only variables associated with it.
(Bitbake rev: 2995b8d551e0532eca20f8862730acd062c608ad)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
This was originally used to check the consistency of the stamps in one function
call. This turns out to be inefficient, unnecessary and if it were necessary,
check_stamp_task() could be called in a loop. The function has been unmodified
for a while and likely contains bugs. Its best simply removed.
(Bitbake rev: 728ffde1bd69b880d48fe8523b1616956d13616e)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
This should fix issues where bitbake would seemingly lock up when checking
certain configurations of stampfiles.
The cache is kept within the runqueue since that feels like the right
place to associate this cache data.
(Bitbake rev: e95755c4931b26d9f8102ed3652dff969145cfc9)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
- for a normal exit, use WEXITSTATUS, rather than manually shifting
- for exit via signal, set the exit code to 128+N, per shell convention
- if a process was stopped, return and don't handle it, as the process can yet
be continued
This should fix the case where bitbake says a task failed with an exit code of
0 (we assumed failure based on the overall status, but didn't pass all the
information along to task_fail).
(Bitbake rev: 84ea614bc56d35a414eb5bf5658891b340bfc569)
Signed-off-by: Christopher Larson <chris_larson@mentor.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
When "Force stop" is performed during the build stage, after os.kill() kills the build sub-processes, there are many <defunct> python processes in the system. In Hob, when the user initiates a new build, os.waitpid() in runqueue_process_waitpid() will be called, and the pids of those <defunct> processes will be returned as result[0], then self.build_pids[result[0]] will throw KeyError exception because now for the new build self.build_pids is empty.
This patch is to address the above issue to collect the results and handle the sub-processes as soon as they are killed.
[Yocto #2186]
(Bitbake rev: e9f4ca467e795bbc520d12b0e7a5985b6ff0a20e)
Signed-off-by: Shane Wang <shane.wang@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
(Bitbake rev: c4c8df0e1e77f99f3ee0b07e1748116fc362819f)
Signed-off-by: Robert P. J. Day <rpjday@crashcourse.ca>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Currently the code checking whether stamps are valid only traverses one step
of the dependency graph. This works fine in the normal cases where we've already
validated dependencies but for the setscene code, it doesn't work well. A typical
problem usecase is something like:
bitbake gcc-cross -c unpack -f
bitbake gcc-cross
which will ignore any sstate files already cached which could be used to speed
up the gcc-cross build. This becomes particularly problematic with multiple gcc
builds where only one should rebuild yet they all do.
This patch teaches the stamp code to be able to recurse within a given fn
which gives the behaviour people would expect from the code and allows
bitbake to make better use of prebuild sstate objects.
(Bitbake rev: e68814cb2e8da523d4ddf05e8ceddfaa19244851)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Fixes a regression introduced in commit
e8a3499c95a6d4f2b8fed002fb9504733c5be3c6 which resulted in a
backtrace on setscene task failure due to trying to dereference
the setscene task ID twice.
(Bitbake rev: 8b846a92a58b5c20d7cfd2efd32b763e95c3c2fd)
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Sometimes user stops the build before runqueue is established,
for example, at the stage of running add_unresolved() function. This
will cause RunQueue to use rqexe field in finish_runqeue() before
initialized. This will cause endless print of "Running idle function"
if use process server.
This commit initialize rqexe variable in RunQueue's init function,
and add a judgement in finish_runqueue().
(Bitbake rev: 59f817723172092a87738c79f555e605f55ea375)
Signed-off-by: Dongxiao Xu <dongxiao.xu@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* When a setscene task starts, print out that it's starting in the UI
(ensuring we get the correct task name)
* When a setscene task fails, ensure we remove it from the list of
running tasks so that if you break out any time afterwards it
is not still listed.
(Bitbake rev: e8a3499c95a6d4f2b8fed002fb9504733c5be3c6)
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Make dry-run do everything except executing the task, instead of
cutting it off earlier. This fully tests the code path for running the
child task (parsing and fakeroot), as well as enabling future
functionality such as using dry-run to produce signature files.
(Bitbake rev: bf1d7739618dabf3872a868230c0112b9ad2a2c0)
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Monitor disk availability and take action when the free disk space or
amount of free inode is running low, it is enabled when BB_DISKMON_DIRS
is set.
* Variable meanings(from meta-yocto/conf/local.conf.sample):
# Set the directories to monitor for disk usage, if more than one
# directories are mounted in the same device, then only one directory
# would be monitored since the monitor is based on the device.
# The format is:
# "action,directory,minimum_space,minimum_free_inode"
#
# The "action" must be set and should be one of:
# ABORT: Immediately abort
# STOPTASKS: The new tasks can't be executed any more, will stop the build
# when the running tasks have been done.
# WARN: show warnings (see BB_DISKMON_WARNINTERVAL for more information)
#
# The "directory" must be set, any directory is OK.
#
# Either "minimum_space" or "minimum_free_inode" (or both of them)
# should be set, otherwise the monitor would not be enabled,
# the unit can be G, M, K or none, but do NOT use GB, MB or KB
# (B is not needed).
#BB_DISKMON_DIRS = "STOPTASKS,${TMPDIR},1G,100K WARN,${SSTATE_DIR},1G,100K"
#
# Set disk space and inode interval (only works when the action is "WARN",
# the unit can be G, M, or K, but do NOT use the GB, MB or KB
# (B is not needed), the format is:
# "disk_space_interval, disk_inode_interval", the default value is
# "50M,5K" which means that it would warn when the free space is
# lower than the minimum space(or inode), and would repeat the action
# when the disk space reduces 50M (or the amount of inode reduces 5k)
# again.
#BB_DISKMON_WARNINTERVAL = "50M,5K"
[YOCTO #1589]
(Bitbake rev: 4d173d441d2beb8e6492b6b1842682f8cf32e6cc)
Signed-off-by: Robert Yang <liezhi.yang@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
If we call finish_now(True), rq.state is not updated to match. This
makes the behaviour of finish_now(False) and finish_now(True) consistent
so both leave rq.state consistently.
(Bitbake rev: 9079ae0ab74f9232b7e9853b2013b051d4fcf623)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
This commit implements a new design for hob
Some of the new features:
- Friendly new designed GUI. Quick response to user actions.
- Two step builds support package generation and image generation.
- Support running GUI seprarately from bitbake server.
- Recipe/package selection and deselection.
- Accurate customization for image contents and size.
- Progress bars showing the parsing and build status.
- Load/save user configurations from/into templates.
(Bitbake rev: 4dacd29f9c957d20f4583330b51e5420f9c3338d)
Signed-off-by: Dongxiao Xu <dongxiao.xu@intel.com>
Signed-off-by: Shane Wang <shane.wang@intel.com>
Signed-off-by: Liming An <limingx.l.an@intel.com>
Signed-off-by: Fengxia Hua <fengxia.hua@intel.com>
Designed-by: Belen Barros Pena <belen.barros.pena@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The current code prints a log when a setscene task starts, therefore
the progressbar in hob will not receive it. Use a sceneQueueTaskStarted
event instead.
Besides, change the sceneQueueTaskFailed event to inherit runQueueEvent
directly to avoid confusion to event receiver.
(Bitbake rev: 7c07cc93d6558d7d9c3144b13493901b7ebae050)
Signed-off-by: Shane Wang <shane.wang@intel.com>
Signed-off-by: Dongxiao Xu <dongxiao.xu@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
When building the universe target:
* Show a warning about universe likely producing errors
* Multiprovider errors are now shown as warnings
Also remove an unused "error" variable in runqueue.py.
Fixes [YOCTO #1936]
(Bitbake rev: 293c796e8a3d1f74ca1e51017b6dede261612281)
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
This is being added to resolve setscene race issues where we do have
particular dependencies required between setscene tasks. This allows
specific dependencies to be specified. This allows us to fix the races
in sstate with the useradd class in OE-Core.
Any tasks being depended upon have their reverse dependencies cleared to
ensure we don't have circular references.
(Bitbake rev: e1b157d26374a70e6274edcb4c0b9f3bc48f765c)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
When constructing the setscene inter-dependencies, we need to account for all task,
not just the last one found. This patch corrects this oversight and ensures all
dependencies are added, not just the first one found.
(Bitbake rev: b9b5b5129d066e1ff7d3effda116afc3c6657beb)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
It's unclear from the history but at some point we stopped logging the
"Tasks Summary" NOTE when tasks failed. Reinstate this for failure, and
also make the count of attempted tasks include the failed task.
(Bitbake rev: e44d5be98fc5d2589cd929ce143638395936b936)
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Seeing the word "failed" alone without reading the whole context has
occasionally triggered an automatic assumption on the part of some users
(myself included) that something has gone wrong, even when this message
is telling you that "0 [tasks] failed". To avoid this let's just say
"all succeeded" in this case instead.
As a bonus this means you can now search the output for "fail" and not
find anything if all went well.
(Bitbake rev: b6f067af12d4661758a78788f1db472684b9aba8)
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
When debugging setscene problems it was found that some extra debug messages were useful.
This patch adds them.
(Bitbake rev: cff2c258b77fde01d530a5923e553e6111b15eb5)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The existing looping code can mask an existing "found = True"
by forcing it to False each time. This can lead to dependency
lists not being fully searched and results in dependency errors.
An exmaple of this was the autobuilder building linux-yocto from
sstate but then rebuilding some of the recipe's tasks for no
apparent reason. Separating the logic into two variables solves this
problem since any "found = True" value is now always preserved.
(Bitbake rev: 61017fc5d30b7a13308d038872ec92efc1a84cef)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
If A depends upon B which depends upon C and the setscene for B
succeeds but C is going to get rebuilt, we should wait for C to
try and build A but currently we don't.
This is due to the timing of when we run the task_skip() as this
triggers other tasks to become buildable. This patch moves the timing
of that call to a more appropriate place allowing dependencies to
behave as expected.
(Bitbake rev: b7114d8e5d9b0720339bd5d24d243c0f2a7c1f3b)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The OE metadata has cases where it needs to prevent some setscene
functions from running. An example of this is where we know a task
is going to run do_configure (which would clean out do_populate_sysroot)
and hence we don't want do_populate_sysroot_setscene to run.
This change adds in a hook so that the metadata can allow any
such policy decision to filter back up to bitbake.
It removes the existing code which attempted to do this in a generic
way but failed.
(Bitbake rev: 2f6d987d9957a5d713ce119c24c2e87540611f53)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The code which populates setscene_covered list was adding a task to the
covered list if all of the tasks that depend upon it were also covered;
however, this means that tasks that would have installed "runtime"
dependencies were being marked as covered also, e.g. gmp-native and
mpfr-native are needed by gcc-cross at runtime since they are shared
libraries that gcc links to, but their do_populate_sysroot tasks were
being marked as covered, resulting in failures later on if gcc-cross was
available from sstate but mpfr-native and gmp-native weren't.
Since we currently have no real way to handle runtime dependencies for
native packages, add a workaround which avoids marking tasks as covered
if one or more of their revdeps are from a different recipe.
Fixes [YOCTO #1536].
(Bitbake rev: e492eb4dc9016cd0bed194377c6f2b85cf0ad113)
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Add a FAKEROOTNOENV which does the opposite of the FAKEROOTENV variable
and is data loaded into the environment for tasks without the fakeroot
flag.
The intent here is to provide a way to control the environment when we
aren't needing fakeroot context which allows us to unload the preload
from memory entirely and gain a valuable speedup.
I'm not 100% happy with needing this at the bitbake level, particularly
with the cache hit but it does give a valuable speedup.
(Bitbake rev: 05c29ab0b2ad3c521414cabb6a92bca15c6e919c)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Without this, variables can be set to one thing in one part of the environment and something
different in another part. This change ensures the datastore and the environment
are consistent.
(Bitbake rev: 459addf13721a6847406f215650fa1882fb83ea9)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
This fixes two problems:
a) Variables which were in the parent environment but not set as "export"
variables in the datastore could end up in the task environment
b) oe.environ.update() can't cope with the generator returned by
bb.data.exported_vars()
Whilst the updated code isn't as neat, it does do the expected thing,
sets the environment correctly and stops unwanted values leaking into
the task environment.
(Bitbake rev: 67e5e23034c5ec2b9efcca935242830306c0048d)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
This patch, based on proof of concept code from Richard adds code to
bitbake to allow individual tasks to optionally specify their stamp
file using the stamp-base flag. This takes the same form as the STAMP
variable but can be specified on a per task basis.
Code is also added to runqueue to ensure that if two tasks share the
same stamp file, only one will be executed at once.
A significant usecase for this code is to share source code (${S})
between recipes where separate build directories (${B}) are used.
(Bitbake rev: 41bef02bef8379590ba012319aebe05068a8081e)
Signed-off-by: Robert Yang <liezhi.yang@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The umask for a task can now be set as:
task[umask] = 022
task[umask] = '022'
If specified as a text string, it must be octal. (This is due to
recipe parsing where it's always set to a string.)
[RP tweaked to use None instead of -1]
(Bitbake rev: a5caaaaba8f0db1af5d8f2e610021d6d4b56894e)
Signed-off-by: Mark Hatle <mark.hatle@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Update exception handling syntax to use the modern style:
except ExcType as localvar
(Bitbake rev: dbf5f42b06bef81749b13aa99945cc1292a6676d)
Signed-off-by: Scott Garman <scott.a.garman@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
JaMa reported issues where bitbake would rebuild things instead of using the
existing built tasks. This was tracked to a case where:
a) rm_work is uses
b) A depends on B
c) B has a version change (e.g. PR bump)
and A *and* B would then rebuild.
It turns out that rm_work was correctly turning stamp files into the correct
_setscene varients but bitbake was then ignoring them during setscene processing.
If the correct sstate checksumed files didn't exist, everything would seemingly
rebuild.
The fix is to check for existing *_setscene stamps and if present, honour them.
If "basichash" is enabled, the hash is included with the stamps so everything
should then function as intended.
(Bitbake rev: 0a4d857aabc86b973170ba9ce32b6b449a4e2165)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
If a build causes a real task to be run when the setscene task has already
run then it was possible for dependent packages to be rebuilding at the same
time as a rebuild of the packages they depended on, resulting in failures
when files were missing. This change looks in the setscene covered list and
removes anything where a dependency of the real task is going to be run (e.g.
do_install is going to be run even though the setscene equivalent of
do_populate_sysroot has already been run).
As an additional safeguard we also delete the stamp file for the setscene
task under these circumstances.
Fixes [YOCTO #792]
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
This patch is a quick proof of concept to show how source code could
be shared between recipes which use ${B} to have a separate build
directory compared to source directory ${S}.
Issues:
a) gcc uses sed and creates config files against ${S} which means
the directory should not be shared. Need to change the way that works.
b) Could be extended to cover eglibc except there is a patch applied
against nativesdk versions which again makes the source incompatible.
c) Need to clean up the layout in work-shared and make a directory level deeper
to ensure patch separation.
d) clean task does not remove stamps
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
If a setscene task failed previously it was showing an incorrect task
name in the error line. This patch ensures we show the correct name, also
including the "_setscene" suffix.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Parsing the recipe in the parent before forking off the child worker
can mean the parent doesn't hit the idle loop and becomes a bottleneck
when lauching many short lived processes.
The reason we need this in the parent is to figure out the fakeroot
environmental options. To address this, add the fakeroot variables
to the cache and move recipe loadData into the child task.
For a poky-image-sato build this results in about a 2 minute speedup
(1.8%).
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
This avoids cases where the stats are modified after the event is fired but
before it's dispatched to the UI.
(Bitbake rev: 1954f182687a0bd429175dda87f05d8a94bb403a)
Signed-off-by: Chris Larson <chris_larson@mentor.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The server UI was reading 1024 bytes, then sleeping for 0.25 seconds. Since
most new LogRecord events are larger than this it leads to a build up of data
which is only processed slowly, leading to a bottleneck and a slow down of
all bitbake processes.
Thanks to Dongxiao Xu <dongxiao.xu@intel.com> for the great work in debugging
this. A large value has been left in for the read() command just to ensure some
fairness amongst process handling if a task tries to log truly huge amounts of
data to the server, or goes crazy and ensures the main loop doesn't stall.
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
For certain tasks, we need additional information in build stamp file
other than the task name and file name. stamp-extra-info is introduced as
a task flag which is appended to the stamp file name.
[Code simplifcations/tweaks from Richard]
Signed-off-by: Dongxiao Xu <dongxiao.xu@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The current parameters are not useful to the stampfile generator function
as they can't uniquely define a task. This updated things so the
parameters can identify unique tasks.
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Previously, the cache was actually being loaded from disk twice whenever using
-b or -e -b. This also moves the bb_cache instance into the CookerParser, as
it's not needed by the cooker itself at all.
(Bitbake rev: dd0ec2f7b18e2a9ab06c499b775670516bd06ac8)
Signed-off-by: Chris Larson <chris_larson@mentor.com>
Signed-off-by: Richard Purdie <rpurdie@linux.intel.com>
range() allocates an actual list when called. xrange() is just an iterator
and creates the next range item on demand. This provides a slight
performance increase.
In python 3, range will do what xrange does currently, but the upgrade will
be handled by the 2to3 tool.
(Bitbake rev: 73b40f06444cb877a5960b2aa66abf7dacbd88f0)
Signed-off-by: Bob Foerster <robert@erafx.com>
Signed-off-by: Richard Purdie <rpurdie@linux.intel.com>
Per the python documentation, os.waitpid returns the exitcode shifted up by 8
bits, and we weren't compensating, resulting in a display of 'failed with 256'
when a worker process exits with a code of 1.
(Bitbake rev: 90c2b6cb24dc9c82f0a9aa9d23f2d1ed2e6ff301)
Signed-off-by: Chris Larson <chris_larson@mentor.com>
Signed-off-by: Richard Purdie <rpurdie@linux.intel.com>
We use a custom Logger subclass for our loggers
This logger provides:
- 'debug' method which accepts a debug level
- 'plain' method which bypasses log formatting
- 'verbose' method which is more detail than info, but less than debug
(Bitbake rev: 3b2c1fe5ca56daebb24073a9dd45723d3efd2a8d)
Signed-off-by: Chris Larson <chris_larson@mentor.com>
Signed-off-by: Richard Purdie <rpurdie@linux.intel.com>
This kills firing of Msg* events in favor of just passing along LogRecord
objects. These objects hold more than just level and message, but can also
have exception information, so the UI can decide what to do with that.
As an aside, when using the 'none' server, this results in the log messages in
the server being displayed directly via the logging module and the UI's
handler, rather than going through the server's event queue. As a result of
doing it this way, we have to override the event handlers of the base logger
when spawning a worker process, to ensure they log via events rather than
directly.
(Bitbake rev: c23c015cf8af1868faf293b19b80a5faf7e736a5)
Signed-off-by: Chris Larson <chris_larson@mentor.com>
Signed-off-by: Richard Purdie <rpurdie@linux.intel.com>
It needs to be a generator, so scheduler subclasses have the option to skip
buildable tasks and return a later one.
(Bitbake rev: a8c61e41bc6277222e4cde667ad0b24bd1597aa0)
Signed-off-by: Chris Larson <chris_larson@mentor.com>
Signed-off-by: Richard Purdie <rpurdie@linux.intel.com>
If you create a runqueue scheduler class in a python module, available in the
usual python search path, you can now make it available to bitbake via the
BB_SCHEDULERS variable, and the user can then select it as they select any
other scheduler.
Example usage:
In a test.py I placed appropriately:
import bb.runqueue
class TestScheduler(bb.runqueue.RunQueueScheduler):
name = "myscheduler"
In local.conf, to make it available and select it:
BB_SCHEDULERS = "test.TestScheduler"
BB_SCHEDULER = "myscheduler"
(Bitbake rev: 4dd38d5cfb80f9bb72bc41a629c3320b38f7314d)
Signed-off-by: Chris Larson <chris_larson@mentor.com>
Signed-off-by: Richard Purdie <rpurdie@linux.intel.com>
SIGINT should be from the user, not a script. It also doesn't work as
reliably to shut down processes, as it's not always interpreted as a
termination request. In addition, it causes KeyboardInterrupt exceptions in
the worker processes, which can interfere with our exception handling.
(Bitbake rev: e5f6e0e9de4c6d1dfdd269d2bf7f83c00c415a27)
Signed-off-by: Chris Larson <clarson@kergoth.com>
Signed-off-by: Richard Purdie <rpurdie@linux.intel.com>
In previous exec() model, cooker is re-initialized from scratch with environmental
variable exported accordingly. Now in fork() model, environmental variables are
not exported again, and thus original method to export BB_TASKHASH doesn't apply
now which breaks all sstate packages. Now we can set data variable directly instead.
Signed-off-by: Kevin Tian <kevin.tian@intel.com>
sstate hash validation is done at initialization of RunQueueExecuteScenequeue.
However the index of 'valid' list returned from the validation doesn't
correspond to setscene task ID. It's just an intermediate namespace between
runqueue and sstate hash func. Use it as setscene task ID fully mess the flow.
Previously this doesn't cause trouble because all setscene tasks are passed. Commit
58396a5d24 add 'noexec' concept to setscene
tasks which grabs some tasks out of the list and thus trigger this problem
Without this fix there're ~50 recipes (gzip-native, glib, ...) rebuilt weirdly
with a minimal build, even though existing sstate packages could accelerate them.
there's another typo using wrong task ID in a debug message which further hide
this issue
Signed-off-by: Kevin Tian <kevin.tian@intel.com>
Currently, anything whitelisted in the environment makes it into the worker
processes. This is undesireable and the worker environment should be as
clean as possible. This patch adapts bitbake sosme variables are loaded into
bitbake's datastore but not exported by default. Any variable can be exported
by setting its export flag.
Currently, this code only finalises the environment in he worker as doing so
in the server means variables are unavailable in the worker. If we switch
back to fork() calls instead of exec() this code will need revisting.
Signed-off-by: Richard Purdie <rpurdie@linux.intel.com>
This means the noexec messages are only shown once as the stamp files are now
correctly created.
Signed-off-by: Richard Purdie <rpurdie@linux.intel.com>
If sstate was used to accelerate a build, the pseudo directory might not have
been created leading to subsequent task failures.
Also, sstate packages were not being installed under pseudo context meaning
file permissions could have been lost.
Fix these problems by creating a FAKEROOTDIRS variable which bitbake ensures
exists before running tasks and running the appropriate setscene tasks under
fakeroot context.
Signed-off-by: Richard Purdie <rpurdie@linux.intel.com>
