According to sstate_checkhashes which is defined in sstate.bbclass, the
currentcoun should be a number (0, not None).
Fixed:
$ bitbake base-files -Sprintdiff
> bb.plain("Sstate summary: Wanted %d Found %d Missed %d Current %d (%d%% match, %d%% complete)" % (total, len(found), len(missed), currentcount, match, complete))
TypeError: %d format: a number is required, not NoneType
(Bitbake rev: 45cb73e2846eaffe8964a573875f54808e8f3633)
Signed-off-by: Robert Yang <liezhi.yang@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Reworks the hash equivalence server to address performance issues that
were encountered with the REST mechanism used previously, particularly
during the heavy request load encountered during signature generation.
Notable changes are:
1) The server protocol is no longer HTTP based. Instead, it uses a
simpler JSON over a streaming protocol link. This protocol has much
lower overhead than HTTP since it eliminates the HTTP headers.
2) The hash equivalence server can either bind to a TCP port, or a Unix
domain socket. Unix domain sockets are more efficient for local
communication, and so are preferred if the user enables hash
equivalence only for the local build. The arguments to the
'bitbake-hashserve' command have been updated accordingly.
3) The value to which BB_HASHSERVE should be set to enable a local hash
equivalence server is changed to "auto" instead of "localhost:0". The
latter didn't make sense when the local server was using a Unix
domain socket.
4) Clients are expected to keep a persistent connection to the server
instead of creating a new connection each time a request is made for
optimal performance.
5) Most of the client logic has been moved to the hashserve module in
bitbake. This makes it easier to share the client code.
6) A new bitbake command has been added called 'bitbake-hashclient'.
This command can be used to query a hash equivalence server, including
fetching the statistics and running a performance stress test.
7) The table indexes in the SQLite database have been updated to
optimize hash lookups. This change is backward compatible, as the
database will delete the old indexes first if they exist.
8) The server has been reworked to use python async to maximize
performance with persistently connected clients. This requires Python
3.5 or later.
(Bitbake rev: 2124eec3a5830afe8e07ffb6f2a0df6a417ac973)
Signed-off-by: Joshua Watt <JPEWhacker@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
We only have hash equivalence for setscene tasks so only query the server
for those, reducing the number of connections needed.
(Bitbake rev: 22082c7b3ca0cffcedb7d1d8c6681d35286376db)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Specifying the force flag will now cause runall and runonly to
invalidate the tasks before running them. This allows a --runall or
--runonly to force the tasks to run, even if they would have otherwise
been skipped, e.g.:
bitbake -f --runall fetch
Will run all do_fetch tasks even if they wouldn't be necessary (for
example, skipped by setscene)
(Bitbake rev: 71e52d3822016027106f2a2e74b8dfdf20f5dc1e)
Signed-off-by: Joshua Watt <JPEWhacker@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Rather than repeatedly calling mc_from_tid() do this in the parent,
removing around a million function calls. Takes time spent in this
function from 40s to 36s.
(Bitbake rev: 28b3f0d8867804799420689c314ac4a8f01efb8c)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
There are other data structures which can be reprocessed at the same
time as holdoff_tasks, further improving build efficiency in various
places.
(Bitbake rev: 02090b3456b7a2de12e72dfeaabfd3b631609924)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
We don't need to process the holdoff task list until we're executing tasks
which saves some data manipulation, at the cost of some data structures
not being correct at all times. This saves significant amounts of time
in various profile charts of larger builds.
(Bitbake rev: 270f076111b12eab358417b0c4cf9c70d7cc787a)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
We've observed do_package and do_package_setscene running in parallel. The
reason is that holdoff_tasks wasn't getting updated. Looking at the code, it
would seem the reason is that the task was in pending_migrations and hence
changed wasn't set and holdoff_tasks wasn't updated.
Fix this. It only affects builds with rehashing enabled.
(Bitbake rev: e26e61e84575669bd223f6ab316798097ed95ec8)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
bitbake <target> --runonly=fetch
failed as the target_tids list included entries which were no longer targeted
task ids. Fix this.
(Bitbake rev: 94e848ae6544e628a19cb97115279b0b1678967c)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
This doesn't appear to have ill effects right now but there is a correctness
issue which this so fix it.
(Bitbake rev: a5e084a266f63c2fd370122327615e49beaeb94e)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
This was overwriting data in the parent which caused all kinds of
odd/weird failures.
(Bitbake rev: 4c5aeb424247a9d0c907524ffacd9c61fcdc0852)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
When the task hashes change we need to ensure the stampcache is cleared out
else tasks don't rerun when they should as we're basing decisions on stale
cache data.
(Bitbake rev: 08962092d3bb7887d82f97d442a6103c0677eae7)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
We weren't marking this special case of setscene task as buildable leading to
runqueue task failures.
(Bitbake rev: 930efbc563443d82df8d692bb8ff172ca2bae192)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Further testing shows we should test some extra datastructures to help pinpoint logic
errors more precisely. This adds some further data structure sanity checks.
(Bitbake rev: 83c4370b25c3a14cc946965c5c5f83ea28f488a1)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
This debug statement could result in a long list of tasks which when
repeatedly sent over our IPC, slowed down the builds immensely. Remove
it in favour of other more targeted debugging added recently, bringing
back some lost performance, particularly on builds with large numbers
of tasks.
(Bitbake rev: 85fe627fdb6510f0942917964386fad9d8c479c8)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The event from the task notifiing of hash equivalency should only be processed
when the task completes. This can otherwise result in a race where a dependent
task may run before the original task completes causing various failures.
To make this work reliably, the code had to be restructured quite a bit.
(Bitbake rev: 1bf5be46f92f125193638cf41ff207d68f592259)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The changed_setscene variable here is just odd and not needed. Worse,
it could prevent some tasks from being removed from the holdoff tasks
list. The list is being rebuilt and should work as intended just from
the other data, this is a leftover from previous versions of the code
as far as I can tell.
(Bitbake rev: 030b9f2b3ce6ed40e79304eb0ffee6c6613f43be)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Whilst we had good runqueue failure mode debug, it hadn't adapted to the
scenequeue changes. Run the scenequeue sanity tests at the end of
a build and output the results regardless of whether all setscene tasks
completed or not. This *massively* improves the ability to debug runqueue
problems.
(Bitbake rev: b9b2177473c0b95a23bd519a201e1d2ba101c6c1)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Looking at the profile information, a lot of time is being spent in
next_buildable_task. This is probably due to the generator expressions
not working well with the empty test.
The easiest way to improve things is to switch to using set manipulations.
We also don't need to update self.buildable the way the original code did
as we don't rely on that anywhere.
(Bitbake rev: 3bcf9ad4964b7e42d1a02ce231e9db42a81ead2a)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
If tasks are in the covered list of tasks for a given setscene task,
it needs to wait for those to complete before we can start.
(Bitbake rev: fdee640c26750b852eb68f5c80437377aa300ed8)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The previous tasks_covered and tasks_notcovered were basically unstable
data structures. We couldn't always tell whether tasks should be covered
or not when trying to repair the sturcture if sstate tasks reran.
In the end its simpler to throw the lists away and rebuild them based upon
current data rather than trying to patch it adhoc. This turns out to be
simpler and much more reliable and I've much more confidence in this code.
(Bitbake rev: 52ee2ba2c617d928569f5afa404925c8b6f317bc)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
We need to copy this set, not modify the original else all kinds
of weird and bad things break, mostly from circular references.
We'll not go into how much sleep I lost tracking down the fallout
from this.
(Bitbake rev: 49927546d2b306830c98f6f9da4a6ad828f6a3a6)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Currently siggen uses the format "<filename>.<taskname>" for referencing tasks
whilst runqueue uses "<filename>:<taskname>". This converts to use ":" as the
separator everywhere.
This is an API breaking change since the cache is affected, as are siginfo files
and any custom signature handlers such as those in OE-Core.
Ultimately this will let us clean up and the accessor functions from runqueue,
removing all the ".rsplit(".", 1)[0]" type code currently all over the place.
Once a standard is used everwhere we can update the code over time to be more
optimal.
(Bitbake rev: 07e539e1c566ca3434901e1a00335cb76c69d496)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
This function uses an old API which uses offsets into lists as a communication
mechanism. Update the API to use "tid" which is used universally in runqueue now.
We can also add kwargs support to the funciton definition to drop some of the
backwards compaiblility hoops we had to jump though with different function
argument combinations.
(Bitbake rev: dc23550047e5078da491ce9a6f30989cb5260df6)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Its useful, particularly in the local developer model of usage, for
bitbake to start and stop a hash equivalence server on local port,
rather than relying on one being started by the user before the build.
The new BB_HASHSERVE variable supports this.
The database handling is moved internally into the hashserv code so that
different threads/processes can be used for the server without errors.
(Bitbake rev: a4fa8f1bd88995ae60e10430316fbed63d478587)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Whilst this isn't strictly necessary, its helpful if the log output is
consistent and its also helpful if bugs either appear or don't appear
for a specific configuration. Ensuring the various iterations we make
are deterministic (sorted) helps with this.
(Bitbake rev: 6a901bb904a97ca90d88be2c6901d3d32346282f)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
There is a compelling usecase for tasks being able to notify runqueue
that their "unihash" has changed. When this is recieved, the hashes of
all subsequent tasks should be recomputed and their new hashes checked
against existing setscene validity. Any newly available setscene tasks
should then be executed.
Making this work effectively needs several pieces. An event is added
which the cooker listen for. If a new hash becomes available it can
send an event to notify of this.
When such an event is seen, hash recomputations are made. A setscene
task can't be run until all the tasks it "covers" are stopped. The
notion of "holdoff" tasks is therefore added, these are removed from
the buildable list with the assumption that some setscene task will
run and cover them.
The workers need to be notified when taskhashes change to update their
own internal siggen data stores. A new worker command is added to do this
which will affect all newly spawned worker processes from that worker.
An example workflow which tests this code is:
Configuration:
BB_SIGNATURE_HANDLER = "OEEquivHash"
SSTATE_HASHEQUIV_SERVER = "http://localhost:8686"
$ bitbake-hashserv &
$ bitbake automake-native
$ bitbake autoconf-native automake-native -c clean
$ bitbake m4-native -c install -f
$ bitbake automake-native
with the test being whether automake-native is installed from sstate.
(Bitbake rev: 1f630fdf0260db08541d3ca9f25f852931c19905)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
We need to preserve unihash task hashes between runs. Use the new SimpleCache
class to create such a class within the signature generator which is loaded
at init time and saved when builds complete. The default is unpopulated but
metadata sig handlers can populate this cache.
(Bitbake rev: 1f326f2c29c2664a5daaeeb0c1fd332630efbdba)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Rather than a special copy of the data structure which we change, compute
the logic using set operations from other data we have. This means
we can add tasks back into the scenequeue without having to worry about
reversing operations on this variable with all the potential bugs that
might involve.
(Bitbake rev: b707d0cbc25fa336a1e95ff588f1ea37eee063eb)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
If you specify both setscene and non-setscene tasks on the commandline, the
non-setscene tasks could be missed, e.g. "bitbake X:do_patch X:do_populate_sysroot"
and do_patch would fail to run.
Fix the problem in runqueue and add a testcase.
(Bitbake rev: 75292fdec5d9c0b5b3c554c4b7474a63656f7e12)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Currently if a multiconfig build contains different configurations which
have overlapping sstate artefacts, it will build them multiple times.
This is clearly suboptimal and not what users want/expect.
This adds code to detect this and stall all but one of the setscne tasks
so that once its built, it can be found by the other tasks.
We take care to iterate the multiconfigs in order so try and avoid
dependency loops. We also match on PN+taskname+taskhash since this is
what we know sstate in OE-Core would use. There are some tasks even within
a multiconfig which match hashes (mostly do_populate_lic tasks) but those
have a much higher chance of circular dependency so aren't work attempting
to optimise.
If a deadlock does occur the build will be slower but there is code to
unbreak such a deadlock so it hopefully doens't break anything.
Comments are injected into the test tasks so they have different task
hashes and a new test for this optimisation is added.
(Bitbake rev: a75c5fd6d4ec56836de0be2fe679c81297a080ad)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Pull the common pieces of the hash verification code into a single function
and reduce code duplication.
(Bitbake rev: d0c39e05cef841c6f29cc6c919df6cbf271a9bda)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Fix some unwanted extra indentation.
(Bitbake rev: 460a5c2e3e1d72f2da16fbc96832fadc82e72c52)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
This was left from when task IDs complicated the code, simplify.
(Bitbake rev: ae36b5c693bb9f13c88199e78e3c31616852eafb)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
This combines the scqenequeue and normal task execution into one function
and simplifies the state engine accordingly.
This is the final set of cleanup to fully merge things without adding the
extra noise to the previous commits.
(Bitbake rev: 56f3396d8c7cfbebd175877c9d773e4e35f8dea1)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Since there is now parallel execution of setscene and normal tasks, the way
setscenewhitelist handling worked can't function the way it did. Paul and I
never liked its error output anyway.
This code tries a different approach, checking the task at execution time
but printing the uncovered task list.
This code may need improvement after real world usage but can
work with the new task flows.
(Bitbake rev: a08d8ba5f5194a09391b1904ee31c04c5f0b1e28)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
This is the serious functionality change in this runqueue patch series of
changes.
Rather than two phases of execution, the scenequeue setscene phase, followed
by normal task exeuction, this change allows them to execute in parallel
together.
To do this we need to handle marking of tasks as covered/uncovered in a piecemeal
fashion on a task by task basis rather than in a single function.
The code will block normal task exeuction until any setcene task which could
cover that task is executed and its status is known. There is a slight
optimisation which could be possible here at the risk of races but that
doesn't seem worthwhile.
The state engine isn't entirely cleaned up in this commit (see FIXME) and
the setscenewhitelist functionality is broken by it (see following patches)
however its good enough to test with normal workflows.
(Bitbake rev: 58b3f0847cc2d47e76f74d59dcbbf78fe41b118b)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
It wasn't clear whether the variable contained just setscene covered
tasks or all covered tasks. We need both sets of data so lets just have
two clearly named variables.
(Bitbake rev: a9fb55627762e7c8b3df30b335ad0b2f1adc080e)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The code for setting up buildable tasks can be simplified.
(Bitbake rev: ce3cd2df5b034f8dbdcf9834e8b9a393b6b01aad)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Its now clear a variable is pointless, remove it and tweak the logic
so the data structure of the existing variable matches what we need.
(Bitbake rev: c257c7b93b86dd794d31307e820215301c7ccf3b)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Simplfy some looping code which no longer has any purpose.
(Bitbake rev: 01dfc37095e5c661f275917d22aa1c1ad7f24d8d)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Add some extra comments to build_scenequeue_data() and fix the debug code
so it actually works.
(Bitbake rev: 8ea6d8193fc89b4596da69e400fbc50e5a443f9f)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The existing code to compute the 'unskippable' setscene task list is overcomlicated,
so replace it with something functionally equivalent but simpler and more efficient.
We don't need to process all chains, just the 'top' ones to the first setscene tasks.
This also makes the code more readable.
(Bitbake rev: 06982c82f10cbdbea0b601e5cf0450a2a99c14c2)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Its useful to have a list of all the tasks a given setscene task covers
and we can easily generate this data whilst doing other data processing.
This is used in later changes to runqueue rather than trying to compute it
on the fly which is difficult.
(Bitbake rev: 63ddc2fec40bd1b456702b97091f9dc5ef70a941)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Whilst this class has existed for years, it doesn't have any
users and has a questionable interface. Drop it to allow for further
simplification and changes.
(Bitbake rev: 3ab51764f7965d696bb2c5a872bf161473df4289)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Also move the scheduler init over, apart for the builtable tasks part which need
to remain called later.
(Bitbake rev: ad30a16cd30f9eab0224eb271f98f9a24516b621)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Cleanup to the _execute_runqueue logic to reduce indentation, drop the
dummy executor class concept and prepare for further changes.
(Bitbake rev: 726e3c61a69fef16e605ba9b911a17cd99f1a2c3)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Replace the remains of the Tasks and Scenequeue Tasks classes with simple
function calls. Also drop the dummy version of the execution class to
simplify further changes as its not needed.
(Bitbake rev: 33805394310046cd58c2194f6d063b3946811014)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Some tasks were not being marked as covered/notcovered since internal
calls were being made without using the external call points.
Fix the accounting issues by using the correct external call points.
(Bitbake rev: fe0a7be03e8baed22f6b0915cd5f7956ba3fbf83)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Merge the unique functions from the Tasks and Scenequeue Tasks classes
into the common base class.
(Bitbake rev: 7539fe22bc831bb835901e3aca77985ab4ebc4c7)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Use a seperate stats class for scenequeue tasks and move the setup
into the base class. Update references accordingly.
(Bitbake rev: 32f39bbd5d3b7394689da9ba05be2c15b4523b27)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
In preparation for merging the setscene and normal task execution,
uniquely namespace the scenequeue specific functions.
For the one shared function, add the "sq_live" variable so we know
which functions to send the results to.
(Bitbake rev: 2cbe9399902ba67dca566c7344b2247412cf4d5c)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
For ease of refactoring, move this code to its own separate function
until it becomes clear what we should do with it.
(Bitbake rev: 4b96b204f986dd62fba485876b7208665c14268d)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The function is only used by setscene code so the parameter is pointless,
remove it.
(Bitbake rev: b52dbf5e9cb327f8434213d286ad333f5dbad1d3)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Move the bulk of the scenequeue data generation to its own function
allowing for refactoring of the code.
Create the start of an object to represent this data.
(Bitbake rev: 68326e0426f25a1bbfd5ae3aa278656a3744053e)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
With the removal of the setcene verify code, this additional code block
is also now unneeded since tasks can't be forced at this point in the code
any move. This effectively reverts f21910157d873c030b149c4cdc5b57c5062ab5a6.
(Bitbake rev: 4514fe4f045d595cc9b938f9326f66f2b3e99f71)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Nothing in OE-Core uses this and hasn't since 2017. It wasn't needed by core
metadata since the switch to recipe specific sysroots.
Since this function would be hard to implement with the planned changes to
runqueue, drop it which allows simplification and further code cleanup.
(Bitbake rev: 5deaa5df730a8a846f3192b4a639b7a2a72c1b71)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Work off a copy of the 'buildable' class variable, allowing easier
future code changes.
(Bitbake rev: e851169acfebba404514135bf512e6f045739a13)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Adds an option to skip _setscene only if they would normally be
executed, without ignoring sstate completely.
Previously, '--no-setscene' would allow a build that completely ignored
sstate and _setscene tasks, and '--setscene-only' would allow a build
that only ran _setscene tasks, but there was no option do a build that
would respect tasks previously restored from sstate and build everything
else. Now one can run:
bitbake --setscene-only IMAGE; bitbake --skip-setscene IMAGE
which is functionally equivalent to:
bitbake IMAGE
The indented use is to allow a build to complete successfully in the
presence of _setscene task failures by splitting apart the two phases
e.g.:
(bitbake -k --setscene-only IMAGE || true) && bitbake --skip-setscene IMAGE
(Bitbake rev: 813ba5b7c13b573a0b813b628a819bdbf0627540)
Signed-off-by: Joshua Watt <JPEWhacker@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
(Bitbake rev: cc712f3257904960247a7532cfc4611f3dccd36c)
Signed-off-by: Robert Yang <liezhi.yang@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
After real world use its clear the "multiconfig:" prefix to multiconfig tasks,
whilst clear, is also clumbersome. Switch to use the short version instead.
mcdepends will continue to work with "multiconfig:" for now as well. The commandline
will only accept mc: going forward.
[YOCTO #11168]
(Bitbake rev: 821daf093b76504067a8b77dfa4b181af6ec92b4)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
There are much better ways to handle this and most editors shouldn't need this
in modern times, drop the noise from the files. Its not consitently applied
anyway.
(Bitbake rev: 5e43070e3087d09aea2f459b033d035c5ef747d0)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
With the introduction of SPDX-License-Identifier headers, we don't need a ton
of header boilerplate in every file. Simplify the files and rely on the top
level for the full licence text.
(Bitbake rev: 695d84397b68cc003186e22f395caa378b06bc75)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
This adds the SPDX-License-Identifier license headers to the majority of
our source files to make it clearer exactly which license files are under.
The bulk of the files are under GPL v2.0 with one found to be under V2.0
or later, some under MIT and some have dual license. There are some files
which are potentially harder to classify where we've imported upstream code
and those can be handled specifically in later commits.
The COPYING file is replaced with LICENSE.X files which contain the full
license texts.
(Bitbake rev: ff237c33337f4da2ca06c3a2c49699bc26608a6b)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
It used 2 spaces as indent which wasn't clear enough, and might cause
confusions, people might think it was in wrong format.
Fixed:
$ bitbake bc-native -ccleansstate -Snone
$ bitbake bc-native -ccleansstate -Snone
$ bitbake-diffsigs tmp/stamps/x86_64-linux/bc-native/1.07.1-r0.do_cleansstate.sigdata.*
* Before:
Hash for dependent task bc/bc_1.07.1.bb.do_clean:virtual:native changed from [foo]
Taint (by forced/invalidated task) changed from [foo]
Taint (by forced/invalidated task) changed from [foo]
* Now
Hash for dependent task bc/bc_1.07.1.bb.do_clean:virtual:native changed from [foo]
Taint (by forced/invalidated task) changed from [foo]
Taint (by forced/invalidated task) changed from [foo]
(Bitbake rev: 5127a8d8e6d53f5f43a6ada7fd09b6b0c24ae989)
Signed-off-by: Robert Yang <liezhi.yang@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The consumers of BB_TASKDEPDATA in OE metadata can't cope with multiconfig
dependencies. The choice is either to start adding code to each of them to
filter out multiconfig dependencies, or do this at source.
After consideration we've decided to do this at source as doing otherwise
is code duplication and error prone and in any case we've looked at, they
don't make sense.
[YOCTO #13090]
[YOCTO #13130]
(Bitbake rev: 531dcd221a10853f45cc057b52bb2d5083e0ee42)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Currently the mechanism for breaking out of the dependnecy loop analysis
code is broken and doesn't work leading to bitbake appearing to hang.
Add in a custom exception for this purpose and fix the code to exit
as intended, fixing the hang and making the dependency loop code
usable again.
(Bitbake rev: 8756e4ade67c16e35269ea0659e10b9ebaa6117f)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Changes FAKEROOTCMD so that it can accept additional arguments to pass
to the fakeroot implementation instead of being treated as a simple
command
(Bitbake rev: 4fa51afb56b090cf1f746842acd602c9536715d5)
Signed-off-by: Joshua Watt <JPEWhacker@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The unihash should be fetched using the task filename that includes the
multiconfig prefixes.
[YOCTO #13124]
(Bitbake rev: 5e7f4e77e27bceaf6c68137cacb4f8d7d7de49dd)
Signed-off-by: Joshua Watt <JPEWhacker@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
If the unique hash is being used to track task dependencies, the hash
validation function needs to know about it in order to properly validate
the hash.
[YOCTO #13030]
(Bitbake rev: 9a529bb2658a4046dafbf32e1eb503d84e64e947)
Signed-off-by: Joshua Watt <JPEWhacker@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The unique hash is now passed to the task in the BB_UNIHASH variable
[YOCTO #13030]
(Bitbake rev: aab80b099f6f259e4b57cba2c26dd385d07c5947)
Signed-off-by: Joshua Watt <JPEWhacker@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Requests the task unique hash from siggen and tracks it
[YOCTO #13030]
(Bitbake rev: 1ecc47f0831b35c8c92b37a81cef4e43ff9f67b2)
Signed-off-by: Joshua Watt <JPEWhacker@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Pass the task hash as a parameter to the 'runtask' message instead of
passing the entire dictionary of hashes when the worker is setup. This
is possible less efficient, but prevents the worker taskhashes from
being out of sync with the runqueue in the event that the taskhashes in
the runqueue change.
[YOCTO #13030]
(Bitbake rev: 1e86d8c1bec7ea5d016a5ad2097f999362e29033)
Signed-off-by: Joshua Watt <JPEWhacker@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Currently if there are no setscene tasks, the disk monitor isn't started.
Move the startup code to somewhere to ensure it always is started. This
issue would partially explain occasional selftest failures.
(Bitbake rev: 5ba83ee25c1c9cba349edb68a22476b1d5fca6ce)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Running "bitbake gconf-native -c cleansstate; bitbake core-image-sato:do_populate_sdk"
results in a build where it fails to find gconf-native and fails to build it,
merrily trying to build the SDK without gconf being present.
The issue is the missing setscene tasks are effectively ignored as the later
code in runqueue thinks that since other sstate tasks are present, these
'cover' the missing one. In reality we need to call BB_SETSCENE_DEPVALID
to make that decision. To do that we need a "reduced" setscene dependency
graph which we don't have in main task graph context.
Since that was already done in setscene, we should just assume anything
in the non-covered list needs to be built.
(Bitbake rev: 464d0339add15bc8b4344ddd1e4c49706e3c0a02)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
If the user puts universe on the commandline, they don't really want warnings
so use the new verbnote level instead.
(Bitbake rev: 0c87ade5678e503899e3a6cdda5329f6fc212b63)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
There is an oversight in the current hash validation API in that the
function can't know how many setscene tasks already completed. Rather
than trying to add additional parameters to the function, causing
incompatibilities, store the value in the datastore.
This is useful to allow build status reporting to the user for
figures on sstate reusage and build completion.
(Bitbake rev: ec037d3e49264037b81212f498d98e292ae7c334)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
This patch adds the capability for tasks from different
multiconfigs to depend on one another.
These dependencies can be enabled using the following format:
task[mcdepends] = "multiconfig:FROM-MC:TO-MC:PN:task-to-depend-on"
For the sake of simplicity consider the following example:
Assuming we have set up multiconfig builds, one for qemux86 and one for
qemuarm, named x86 and arm respectively.
Adding the following line to an image recipe (core-image-sato):
do_image[mcdepends] = "multiconfig:x86:arm:core-image-minimal:do_rootfs"
Would state that core-image-sato:do_image from x86 will depend on
core-image-minimal:do_rootfs from arm so it can be executed.
This patch makes modifications to:
- cooker: To glue both multiconfigs in one place and make sure
the dependencies can be provided.
- taskdata: To parse and add a new kind of dependency (mcdepends) to
the taskdata object.
- runqueue: To differentiate tasks from different multiconfigs,
add the specified dependencies to the corresponding tasks, and
create a working runqueue that contains tasks from both multiconfigs.
- siggen: To avoid looking for tasks from different multiconfigs on
objects where they dont belong.
The taskdata objects are still not aware of the concept of multiconfig,
so each object doesnt know which multiconfig its building, hence why
the mcdepends are added to all taskdata objects equally (we really
dont expect many of these), but the actual dependencies are added only
to the required tasks by the runqueue.
(Bitbake rev: da8cb8633504bdc815bdcefc538340b9bce5065d)
Signed-off-by: Alejandro Enedino Hernandez Samaniego <alejandr@xilinx.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The file_name parameter passed to bb.parse.siggen.invalidate_task
should be a virtual file name instead of a real file name, or else you
will encounter a following error, for instance, when you run:
$ bitbake nativesdk-lzip -c unpack -f
the error arise:
| ERROR: An uncaught exception occurred in runqueue
| if file_name:
| > taintfn = d.stamp[file_name] + '.' + task + '.taint'
| else:
| KeyError: 'virtual:nativesdk:/opt/poky/meta/recipes-extended/lzip/lzip_1.19.bb'
when multilib builds are used on OE.
(Bitbake rev: da37bdad46e11e7ce93ba7a59d58757b769dc16b)
Signed-off-by: Ming Liu <liu.ming50@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
On high core machines, in do_fetch, it is possible to DDoS your own machine.
A method to limit any arbitrary task type to a certain number of simultaneous
threads is needed. (Similar to how BB_NUMBER_THREADS works in the general
case.) The format of this new limitation is:
do_fetch[number_threads] = "2"
This should be set globally. If it is set in individual recipes it could
result in unpredictable behavior.
Note: a value for number_threads > BB_NUMBER_THREADS will have no effect.
(Bitbake rev: 055865047c63b9c3b213b47a1884924ce0adeda0)
Signed-off-by: Mark Hatle <mark.hatle@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The RunQueueStats:taskCompleted and RunQueueStats:taskSkipped can take
multiple arguments. However, nowehere in bitbake are multiple arguments used.
Change this to match the behavior of the other APIs where it needs to be
called once for each task.
Additionally, these two functions were usually called in tandem, however in
the wrong order. It really doesn't matter as there is no specific preemption
point between the calls. But the taskSkipped should be called first to
increment the 'active' count, and then taskCompleted called to decrement it.
(Bitbake rev: 26d5ea9bb892bd6a2e1fd29a9023e0b0644edc16)
Signed-off-by: Mark Hatle <mark.hatle@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
If a standalone tasks adds a dependency on X:do_build, the code in runqueue would
currently remove it if that do_build was part of an image recipe which uses
recrdeptask on do_build.
Such individual tasks shouldn't do this, therefore tweak the recursive reference code
to only process recurseive tasks, not all tasks.
(Bitbake rev: 4cfca360891e1ed876a9c19487b4f6210686af26)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The runall commandline option was confusing people. There are in fact two
different behaviours people may want.
a) For a given target (or set of targets) look through the task graph and
run task X only if its present and would have been built.
b) For a given target (or set of targets) look through the task graph and
run task X if any recipe in the taskgraph has such a target even if it wasn't
in the original task graph.
I've decided to interpret the existing "runall" option as b), even if right
now if behaves like a). For a), which is a valid use case, this patch adds
a "runonly" option.
With both behaviours present, I'm hoping we can then kill off the "fetchall",
"checkuriall" and other tasks from OE metadata and replace them with this
option. This would significantly speed up task graph processing.
(Deleting the checkuriall and fetchall tasks takes "bitbake core-image-sato -g"
from 22s to 8s).
(Bitbake rev: 546a662c877b2d3af35e3996950582ed2df41fe4)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
This is a performance sensitive piece of code and the shear number
of recursive loops is causing a significant and unscalable performance
pain point.
This change moves to a two step approach, firstly generating a list of recursive
dependencies for any task, then applying this to the recursive tasks, iterating
over things until no further dependencies are added.
It was noticed an optimisation is possible and the list of recursive tasks need not
contain the taskname, only the base task id. This allows a significant performance
improvement and limits the size of the resursive task lists, improving speed.
(Bitbake rev: eba738ac5672556eaab4f3374c8025c322761c4a)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
We can optimise the loops slightly so we only process given substrings
once rather than many times. This means expanding out add_resolved_dependencies.
Also add a function which allows replacement of the task element of a
task id, reducing the amount of string handling we're doing in a performance
critical loop.
Its also clear that later code adds to the tasks depends so we don't need
to add .depends() to extradeps at the start.
(Bitbake rev: 4ad281224e92b5f94e3a9c17e8898ec8f1086cdc)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Currently we only run through the recidepends/recrdepends code once. This
means that we can miss some expansions of dependency trees where one
rec{r,i}depends tasks depends on another rec{r,i}depends task.
In reality we need to iterate over the data until we stop adding
dependencies.
In doing this we can't show quite so granular progress information since
we don't know how many times we'll need to do this.
This does slow down the runqueue prepare phase however some optimisations
are possible and can be handled in subsequent patches.
This fix means some missing dependencies, such as:
<image>:do_fetchall -> <image>:do_rootfs -> <pkgs>:do_package_write_X
-> <ca-certs>:do_package_write_X -> debianutils-native
(via PAKAGE_WRITE_DEPS)
are now found/added.
[YOCTO #12510]
(Bitbake rev: aec2f07d56a19b97b6515897532b113cdead8338)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The whitelist shouldn't have to be populated in order for the
enforcement to work properly - check if the list is not None in order to
determine whether the functionality is enabled or not since that is how
the function that sets up the list behaves.
(Bitbake rev: 7b1e79c352ca6eef1693d8abfacf7505544f1caa)
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
I can't actually see how this was working, nothing connected the commandline option
to the data in TaskData(). Drop the remaining pieces of this option, it was a relic
from a decade ago and we want deterministic builds, not random tries until something
might work.
(Bitbake rev: 767c7ba8fc76ec667ac1567de1c971c3575f2ecd)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Having this as one huge long line isn't easy to manipulate, split it into
multiple lines for ease of debugging issues.
(Bitbake rev: 5753fe81194f75fbcf4ccdc733cc585d02794cb1)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
We really ought to have just one place where the string representation
of these events is produced. This doesn't take any real control away
from the UI - if an alternative representation is desired, that can
still be made.
(Bitbake rev: cb15db2a799be6d8eab9a2a43a9a573f89229cff)
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Differentiate between fn and taskfn in "execute".
This was somehow missed in "fakeroot" handling.
(Bitbake rev: d848bff7cf78f63986467b51f701a998a480eb25)
Signed-off-by: Juro Bystricky <juro.bystricky@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Currently if you specify a recrdeptask, it must exist for all recipes or
you get a python traceback. This is a bug and it should be possible to have
recipes which don't have the specified task.
As well as preventing such a traceback (which shouldn't happen, it should be
a user readable error), this allows us to fix issues in OE-Core which would
otherwise trigger the traceback.
(Bitbake rev: f93a77f2f188e18de9e3d812e86d77c2f3c71889)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
For example:
$ bitbake core-image-minimal --runall unpack
...
NOTE: Tasks Summary: Attempted 326 tasks of which 88 didn't need to be rerun and all succeeded.
$ bitbake core-image-minimal --runall patch
NOTE: Tasks Summary: Attempted 453 tasks of which 332 didn't need to be rerun and all succeeded.
This can replace fetchall as well:
$ bitbake core-image-minimal --runall fetch
NOTE: Tasks Summary: Attempted 135 tasks of which 119 didn't need to be rerun and all succeeded.
(Bitbake rev: 7c0fa6ba66cdb956b37d94055307cde857633df9)
Signed-off-by: Matthew McClintock <msm-oss@mcclintock.net>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
When we removed the postinst sstate dependency handling code from
setscene_depvalid, we noticed things being installed into the old
style sysroot for rootfs tasks which should not have been there,
causing a performance regression.
Analysis revealed that setscene dependencies were "bubbling" over
sstate tasks when they should have been stopping there. The
'continue' added by this patch avoids this issue and eusures
sstate tasks remain contained to their specific chains.
There was another bug in the code this exposed where the acconting
for tasks as they were removed from sq_revdeps was not correct. In
fixing this, what looks like a workaround in another test can then
be simplified.
After this change, populate_sysroot tasks are no longer depending
on package_write_rpm tasks for example, which would make no sense.
A before/after analysis of image dependencies only revealed improved
dependencies after this change.
Recipe specific sysroots did highlight the issue here since the
behaviour of the sysroot dependencies (and processing with depvalid)
was not matching what bitbake itself was doing, with bitbake being
incorrect. Failures were 'safe' in that too many dependencies would
get installed.
(Bitbake rev: 5ef2cb50041fa7106c8de170af73d2a54cb0b1f0)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The idea is that tasks which complete building a recipe (like
do_package_qa) are more important than tasks which start building new
recipes (do_fetch) or those which increase disk usage
(do_compile). Therefore tasks get ordered like this (most important
first, do_rm_work before do_build because the enhanced rm_work.bbclass
was used):
1. ID /work/poky/meta/recipes-support/popt/popt_1.16.bb:do_build
2. ID /work/poky/meta/recipes-core/readline/readline_6.3.bb:do_build
3. ID /work/poky/meta/recipes-connectivity/libnss-mdns/libnss-mdns_0.10.bb:do_build
...
464. ID /work/poky/meta/recipes-sato/images/core-image-sato.bb:do_build
465. ID /work/poky/meta/recipes-graphics/xorg-proto/inputproto_2.3.2.bb:do_rm_work
466. ID /work/poky/meta/recipes-devtools/python/python3_3.5.2.bb:do_rm_work
467. ID /work/poky/meta/recipes-core/packagegroups/packagegroup-base.bb:do_rm_work
...
3620. ID virtual:native:/work/poky/meta/recipes-extended/pbzip2/pbzip2_1.1.13.bb:do_install
3621. ID /work/poky/meta/recipes-devtools/qemu/qemu-helper-native_1.0.bb:do_install
3622. ID /work/poky/meta/recipes-core/zlib/zlib_1.2.8.bb:do_compile_ptest_base
3623. ID /work/poky/meta/recipes-extended/bzip2/bzip2_1.0.6.bb:do_compile_ptest_base
...
3645. ID /work/poky/meta/recipes-support/libevent/libevent_2.0.22.bb:do_compile_ptest_base
3646. ID /work/poky/meta/recipes-core/busybox/busybox_1.24.1.bb:do_compile_ptest_base
3647. ID /work/poky/meta/recipes-kernel/linux/linux-yocto_4.8.bb:do_uboot_mkimage
3648. ID /work/poky/meta/recipes-kernel/linux/linux-yocto_4.8.bb:do_sizecheck
3649. ID /work/poky/meta/recipes-kernel/linux/linux-yocto_4.8.bb:do_strip
3650. ID /work/poky/meta/recipes-kernel/linux/linux-yocto_4.8.bb:do_compile_kernelmodules
3651. ID /work/poky/meta/recipes-kernel/linux/linux-yocto_4.8.bb:do_shared_workdir
3652. ID /work/poky/meta/recipes-kernel/linux/linux-yocto_4.8.bb:do_kernel_link_images
3653. ID /work/poky/meta/recipes-devtools/quilt/quilt-native_0.64.bb:do_compile
3654. ID /work/poky/meta/recipes-extended/texinfo-dummy-native/texinfo-dummy-native.bb:do_compile
...
The order of the same task between different recipes is the same as
with the speed scheduler, i.e. more important recipes come first.
(Bitbake rev: 70e297e5c285ce0a02e9efd3117ff62cdc77ec12)
Signed-off-by: Patrick Ohly <patrick.ohly@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Without this, bitbake -b of image recipes cause tracebacks since
the list of providers is empty.
(Bitbake rev: c53e43f3c6a675c0934a7a4e358fd66b049ffca3)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
This change significantly shortens the time on reparsing stage
of '-S' option.
Each file is reparsed and then dumped within a dedicated
process. The maximum number of the running processes is not
greater than the value of BB_NUMBER_PARSE_THREADS if it is set.
The dump_sigs() in class SignatureGeneratorBasic is _replaced_
by a new dump_sigfn() interface, so calls from the outside and
subclasses are dispatched to the implementation in the base
class of SignatureGeneratorBasic.
Fixes [YOCTO #10352]
(Bitbake rev: 99d3703edd77a21770b366c6ad65a3c0f5183493)
Signed-off-by: Jianxun Zhang <jianxun.zhang@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Fakeroot workers usually have dependencies that need to be ready before they
can be started. Starting them as a block therefore doesn't work as the
dependencies may or may not have been built. Therefore start the multiconfig
fakeworkers individually upon demand.
[YOCTO #10344]
(Bitbake rev: ab15486c636e451ed5a62e561980946887901c3b)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
In multiconfig, mutliple tasks can execute which share the same stamp file. These
must not execute in parallel, the idea is the first should execute, the subsequent
ones should see a valid stamp and get skipped.
The normal task execution code has stamps code to handle this, this adds similar
code to the setscene execute() function to handle the issue there too.
(Bitbake rev: 937acf267fa9e45f538695b2cf8aa83232a96240)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
setscene was being entirelu skipped for multiconfig variants as the tasks
were simply not being spotted. If the default config was also being built
it masked the problem. When this was fixed by using taskfn instead of fn
in lookups against dataCache, several other instances of this problem were
highlighted.
This goes through and corrects the setscene code to correclty use taskfn
instead of fn in the appropriate places meaning setscene tasks for multiconfig
now work correctly.
(Bitbake rev: cead85cf219c07b40aeb7c0d948c25f0429136db)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The location of the fakeroot command and the various environmental values need
to be taken from the right multiconfig datastore, not the shared one. This
patch ensures the right one is used for cases like a split TMPDIR.
[YOCTO #10344]
(Bitbake rev: 813a7600bc26b7132ac36d1515571d321f566afb)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
For the purposes BB_SETSCENE_ENFORCE is designed for (in OE, it is used
by the installation process for the extensible SDK), we don't actually
need the whitelisted real tasks to execute - we just need to have them
in the dependency tree so that we get all of the setscene tasks they
depend on to run. Therefore we can actually dry-run those real tasks
i.e. they won't be run (and thus we won't waste a significant amount of
time doing so) and won't be stamped as having run either. We do already
have a dry-run mode in BitBake (activated by the -n or --dry-run command
line option), but it dry-runs the setscene tasks as well which we don't
want here.
Note that this has no effect on the checking we are doing with
BB_SETSCENE_ENFORCE to ensure that only whitelisted real tasks are
scheduled to run - that's handled separately.
Fixes [YOCTO #10369].
(Bitbake rev: 58f084291beb3a87d8d9fdb36dfe7eff911fa36b)
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Some of our metadata assumes that BuildStarted and BuildCompleted events
see the same data store. This is the case for buildTarget but not for
buildFile and recent changes mean this is now a problem.
The update_data() call is now an empty operation and there is no difference
between the expanded_data and data so we can simply remove the expanded_data
and its references and use data everywhere. This has been inteded for a while
but the above issue makes this more pressing to finally clean up.
(Bitbake rev: e3694e738e98f26f413ada6860ca7d829d3662f0)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Allow the client to set variables with the setVariable command and have
those changes take effect when running tasks. This is accomplished by
collecting changes made by setVariable separately and pass these to the
worker so it can be applied on top of the datastore it creates.
(Bitbake rev: 69a3cd790da35c3898a8f50c284ad1a4677682a4)
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
We now have code in OE that needs BB_TASKDEPDATA for setscene tasks. Therefore
generate and send this data. In this case its a "pre collapsed" tree
but that is fine for the use cases in question.
(Bitbake rev: 38b857d086af43af6ea3aa60d3876a2c9b225401)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Its useful to know the task hash in code using TASKDEPDATA so add this
data to the data structure. The recipe specific sysroots in OE
need this data.
(Bitbake rev: 758867e8dc74283bb1f031e158ec54cefdd5c2a6)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Hooking the disk monitor into the regular heatbeat event instead
of the runqueue solves two problems:
- When there is just one long running task which fills up the disk,
the previous approach did not notice that until after the completion
of the task because _execute_runqueue() only gets called on task
state changes. As a result, aborting a build did not work in this
case.
- When there are many short-lived tasks, disk space was getting
checked very frequently. When the storage that is getting checked
is on an NFS server, that can lead to noticable traffic to the
server.
(Bitbake rev: 4547eea26803a9cd355d8b045197bcbdbb36a9ad)
Signed-off-by: Patrick Ohly <patrick.ohly@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
getVar() now defaults to expanding by default, thus remove the True
option from getVar() calls with a regex search and replace.
Search made with the following regex: getVar ?\(( ?[^,()]*), True\)
(Bitbake rev: 3b45c479de8640f92dd1d9f147b02e1eecfaadc8)
Signed-off-by: Joshua Lock <joshua.g.lock@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The whitelist checks for BB_SETSCENE_ENFORCE were running for every call
to execute(). Since the task list doesn't change for each call into
execute, the checks only need to be ran once.
[YOCTO #10369]
(Bitbake rev: f65e631ab6705dfd9188f19ee423eca33bca7d7d)
Signed-off-by: Randy Witt <randy.e.witt@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Some task id manipulations were suboptimal:
* taskfn_fromtid and fn_from_tid were effectively the same function
* many calls to split_tid(), then taskfn_fromtid()
* taskfn_fromtid() called split_tid() internally
This patch adds split_tid_mcfn() to replace split_tid() and returns the
"taskfn" variant being used in many places. We update all core calls
to the new function and ignore the return values we don't need since the
function call overhead of the split_tid wrapper is higher than ignoring
a return value.
The one remaining standalone use of taskfn_fromtid is replaced with
fn_from_tid. I couldn't see any external usage so it was dropped.
There is external usage of split_tid so a wrapper remains for it.
Combined together these changes should improve some of the runqueue task
manipulation performance.
(Bitbake rev: 1bf2ef874fbe47f1320007efa0bdeef8d630b8a1)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
`if w in self.rq.worker` when w *is* self.rq.worker doesn't make a great deal
of sense, and results in this error:
File ".../poky/bitbake/lib/bb/runqueue.py", line 2372, in runQueuePipe.read():
name = None
> if w in self.rq.worker:
name = "Worker"
TypeError: unhashable type: 'dict'
Most likely this was meant to be 'is' rather than 'in', but rather than
checking after the fact, just include the name in the iteration, instead.
While we're here, also clean up and fix the broken error message.
(Bitbake rev: 267e025cad44c8bd0fb157f1f7a2e08df117ba84)
Signed-off-by: Christopher Larson <chris_larson@mentor.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
If the worker fails to launch, ensure the task is shown as failed rather
than a confusing "all succeeded" message.
Patch from Juro Bystricky
[YOCTO #10335]
(Bitbake rev: 0e9a2ff96d138641501874a1cd7aa6cc7e94d727)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
If you "bitbake glibc-locale" then delete the libpcre-native sstate
and "bitbake glibc-locale -C package_write_rpm", it will fail with
rpmbuild missing the libprce library.
The reason is that libpcre-native fails to install from sstate (since
it isn't present) but doesn't get built and hence rpm-native tries to
run without its dependencies.
The simplest fix is not to add "covered" tasks which have failed to
install sstate. I can't help feeling there is more to this issue but
this does fix the current problem and shouldn't have adverse affects.
It is an unusual situation to have missing dependencies in sstate since
they're usually all present or not at all.
I've taken the opportunity to remove some old cruft from when we had
numeric task ids, the code can be simpler now.
(Bitbake rev: ba566b46d530b495f12f3a74f76434717b22a020)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
We can't execute the same task for the same package_arch multiple
times as the current setup has conflicting directories. Since
these would usually have the same stamp/hash, we want to execute in
sequence rather than in parallel, so for the purposes of task execution,
don't consider the "extra-info" on the stamp files. We need to add
a parameter to the stamp function to achieve this.
This avoids multiple update-rc.d populate_sysroot tasks executing in
parallel and breaking multiconfig builds.
(Bitbake rev: a9041fc96a14e718c0c1d1676e705343b9e872d3)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
This patch adds the notion of supporting multiple configurations within
a single build. To enable it, set a line in local.conf like:
BBMULTICONFIG = "configA configB configC"
This would tell bitbake that before it parses the base configuration,
it should load conf/configA.conf and so on for each different
configuration. These would contain lines like:
MACHINE = "A"
or other variables which can be set which can be built in the same
build directory (or change TMPDIR not to conflict).
One downside I've already discovered is that if we want to inherit this
file right at the start of parsing, the only place you can put the
configurations is in "cwd", since BBPATH isn't constructed until the
layers are parsed and therefore using it as a preconf file isn't
possible unless its located there.
Execution of these targets takes the form "bitbake
multiconfig:configA:core-image-minimal core-image-sato" so similar to
our virtclass approach for native/nativesdk/multilib using BBCLASSEXTEND.
Implementation wise, the implication is that instead of tasks being
uniquely referenced with "recipename/fn:task" it now needs to be
"configuration:recipename:task".
We already started using "virtual" filenames for recipes when we
implemented BBCLASSEXTEND and this patch adds a new prefix to
these, "multiconfig:<configname>:" and hence avoid changes to a large
part of the codebase thanks to this. databuilder has an internal array
of data stores and uses the right one depending on the supplied virtual
filename.
That trick allows us to use the existing parsing code including the
multithreading mostly unchanged as well as most of the cache code.
For recipecache, we end up with a dict of these accessed by
multiconfig (mc). taskdata and runqueue can only cope with one recipecache
so for taskdata, we pass in each recipecache and have it compute the result
and end up with an array of taskdatas. We can only have one runqueue so there
extensive changes there.
This initial implementation has some drawbacks:
a) There are no inter-multi-configuration dependencies as yet
b) There are no sstate optimisations. This means if the build uses the
same object twice in say two different TMPDIRs, it will either load from
an existing sstate cache at the start or build it twice. We can then in
due course look at ways in which it would only build it once and then
reuse it. This will likely need significant changes to the way sstate
currently works to make that possible.
(Bitbake rev: 5287991691578825c847bac2368e9b51c0ede3f0)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
With the introduction of multi-config and the possibility of distributed
builds we need arrays of workers rather than the existing two.
This refactors the code to have a dict() of workers and a dict of
fakeworkers, represented by objects. The code can iterate over these.
This is separated out from the multi-config changes since its separable
and clearer this way.
(Bitbake rev: 8181d96e0a4df0aa47287669681116fa65bcae16)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
A couple of fixes for the "Initialising tasks" progress bar behaviour:
* Properly finish the progress bar when using bitbake -S
* Finish the progress bar before calling BB_HASHCHECK_FUNCTION (so that
in OE when that shows its own "Checking sstate mirror object
availability" progress bar it gets shown on the next line as it
should).
(Bitbake rev: de6759d8e9990e426e6d6464a2e05381cd4c12d6)
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Runqueue errors direct the user to view the "failure below",
but no additional error message is available.
Log the stacktrace so that the user can see what went wrong.
Also fix a typo in the log message.
(Bitbake rev: e191f401e372ee181bc02250232ad9cb9a0e9477)
Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com>
Signed-off-by: bavery <brian.avery@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
When "Preparing RunQueue" shows up you can expect to wait up to 30
seconds while it works - which is a bit long to leave the user waiting
without any kind of output. Since the work being carried out during this
time is divided into stages such that it's practical to determine
internally how it's progressing, replace the message with a progress
bar.
Actually what happens during this time is two major steps rather than
just one - the runqueue preparation itself, followed by the
initialisation prior to running setscene tasks. I elected to have the
progress bar cover both as one (there doesn't appear to be much point in
doing otherwise from a user perspective). I did however describe it as
"initialising tasks".
(Bitbake rev: 591e9741e108487ff437e77cb439ef2dbca42e03)
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Add the ability to enter a mode where only a specified whitelist of
tasks can be executed outright; everything else must be successfully
provided in the form of a setscene task (or covered by a setscene task).
Any setscene failure outside of the whitelist will cause the build to
fail immediately instead of running the real task, and any real tasks
that would execute outside of the whitelist cause an immediate build
failure when it comes to executing the runqueue as well.
The mode is enabled by setting BB_SETSCENE_ENFORCE="1", and the
whitelist is specified through BB_SETSCENE_ENFORCE_WHITELIST, consisting
of pn:taskname pairs. A single % character can be substituted for the pn
value to match any target explicitly specified on the bitbake command
line. Wildcards * and ? can also be used as per standard unix file name
matching for both pn and taskname.
Part of the implementation of [YOCTO #9367].
(Bitbake rev: 624722c067a7fdd0c0f5d8be611e1f6666ecc4a2)
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
In 2c88afb6 find_chains()'s taskid argument was renamed to tid but
taskid is still used as key to explored_deps dictionary. Use tid instead
of taskid.
(Bitbake rev: 29a34ae8f5306d2779bcc761c52f1f9d13a0c0c5)
Signed-off-by: George McCollister <george.mccollister@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
I'm not sure what possesed me when I wrote this code originally but its
indirection of everyting to use numeric IDs and position dependent lists
is horrific. Given the way python internals work, its completely and
utterly pointless from performance perspective. It also makes the code
hard to understand and debug since any numeric ID has to be translated
into something human readable.
The hard part is that the IDs are infectous and spread from taskdata
into runqueue and even partly into cooker for the dependency graph
processing. The only real way to deal with this is to convert everything
to use a more sane data structure.
This patch:
* Uses "<fn>:<taskname>" as the ID for tasks rather than a number
* Changes to dict() based structures rather than position dependent lists
* Drops the build name, runtime name and filename ID indexes
On the most part there shouldn't be user visible changes. Sadly we did
leak datastructures to the setscene verify function which has to be
rewritten. To handle this, the variable name used to specifiy the version
changes from BB_SETSCENE_VERIFY_FUNCTION to BB_SETSCENE_VERIFY_FUNCTION2
allowing multiple versions of bitbake to work with suitably written
metadata. Anyone with custom schedulers may also need to change them.
I believe the benefits in code readability and easier debugging far
outweigh those issues though. It also means we have a saner codebase
to add multiconfig support on top of.
During development, I did have some of the original code coexisting with
the new data stores to allow comparision of the data and check it was
working correcty, particuarly for taskdata. I have also compared
task-depends.dot files before and after the change. There should be no
functionality changes in this patch, its purely a data structure change
and that is visible in the patch.
(Bitbake rev: 2c88afb60da54e58f555411a7bd7b006b0c29306)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Using positions in lists for flags is an odd choice and makes the code
hard to maintain. Maintaining a list is slow since list searches are
slow (watch bitbake -n slow massively with it) but we can use a set()
instead.
This patch uses python sets to maintain the lists of tasks in each state
and this prepares for changing the task IDs from being integers.
(Bitbake rev: 8c1ed57f6ea475b714eca6673b48e8e5f5f0f9c3)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Various misc changes to convert bitbake to python3 which don't warrant
separation into separate commits.
(Bitbake rev: d0f904d407f57998419bd9c305ce53e5eaa36b24)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The data included in the event is useful for implementing a pre-build
check that warns about unexpected components, for example because of
an incorrect configuration or changed dependencies.
Such a check can be done in a .bbclass that gets inherited
globally. But in contrast to a UI, such a class cannot request that
the event shall be emitted, and thus the event has to be emitted
whether there is a consumer or not.
This was done conditionally earlier out of concerns about the
performance impact. But now events are handled more efficiently, so
that concern no longer seems valid: in some simple testing (admittedly
on a fast build workstation), the two lines (generating the data and
emitting the event with it) only took about 0.05 seconds (measured
with timeit). That was for a build with roughly 500 recipes (from
pn-buildlist aka depgraph['pn']), triggered via the command line. That
was even with a consumer of the data active and doing some work, so it
should be even faster when there is no consumer.
(Bitbake rev: 5ddaf5b7ed1001d2dd3f67e7a6d704afa85479d2)
Signed-off-by: Patrick Ohly <patrick.ohly@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
python3 cares more about invalid type comparisons. Add break statements
and better tests to make the code paths clearer and avoid type issues
in python3. No code functionality change.
(Bitbake rev: 2c39ebdd2762d027f007a6a769fdf023cdf3da2b)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
We shouldn't try and use fakeworker when performing a dry_run. This
makes the core match the other fakeworker execution points.
(Bitbake rev: 49bea821a2edad5e19c3a566d1a80c23718dede9)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
python deprecated logger.warn() in favour of logger.warning(). This is only
used in bitbake code so we may as well just translate everything to avoid
warnings under python 3. Its safe for python 2.7.
(Bitbake rev: 676a5f592e8507e81b8f748d58acfea7572f8796)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
When multiple recipes which both provide something are being built, bitbake
informs us that most likely one of them provides something the other doesn't,
which is usually correct, but unfortunately it's rather painful to figure out
exactly what that is.
This patch dumps two sets of information, one is the provides information for
each recipe, filtered so only common components are removed. The other is a list
of dependees on the recipe, since sometimes this can easily identify why something
is being built.
Its not straightforward for bitbake to obtain the information but since the
warning/error code path isn't the normal one, we can afford to go through some
less than optimal processing to aid debugging.
Also provide the same information even if we're showing a warning since its still
useful.
[YOCTO #8032]
(Bitbake rev: 96fc889b8e62ba4463c71158c4b7286c48d68cd8)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Before this patch the usage of cache was quite useless as the file
checksums were not actually cached on disk but re-calculated every time.
This patch utilises the new writeout_file_checksum_cache() method of the
SignatureGenerator class to do the job.
(Bitbake rev: 5ac9cbf405841ed3f65e6f99a3cee032567fb182)
Signed-off-by: Markus Lehtonen <markus.lehtonen@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
This means that when you view the process tree, the processes
have meaningful names, aiding debugging:
$ pstree -p 30021
bash(30021)───KnottyUI(115579)───Cooker(115590)─┬─PRServ(115592)───{PRServ Handler}(115593)
├─Worker(115630)───bash:sleep(115631)───run.do_sleep.11(115633)───sleep(115634)
└─{ProcessEQueue}(115591)
$ pstree -p 30021
bash(30021)───KnottyUI(117319)───Cooker(117330)─┬─Cooker(117335)
├─PRServ(117332)───{PRServ Handler}(117333)
├─Parser-1:2(117336)
└─{ProcessEQueue}(117331)
Applies to parse threads, PR Server, cooker, the workers and execution
threads, working within the 16 character limit as best we can.
Needed to tweak the bitbake-worker magic values to tell the
workers apart.
(Bitbake rev: 539726a3b2202249a3f148d99e08909cb61902a5)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
This reverts commit b22592af81.
That commit isn't entirely clear about why this change is needed but
I do have a usecase where this breaks things. If for example you run
"bitbake X -c packagedata" and that packagedata is in sstate, you'd
expect this to work.
If sstate doesn't contain a do_populate_sysroot for a dependency, you
would still expect the command above to succeed and you would not
expect
it to rebuild that dependency. With the current code, this isn't what
happens. The code finds the sstate for do_populate_sysroot missing,
this makes the task "uncovered" and this in turn makes it unskippable.
The example I found with this was avahi-ui, where it would trigger
a build of libdaemon to obtain its populate_sysroot.
Since this behaviour seems completely incorrect, revert the older patch
and we'll address any issues that crop up as a result.
(Bitbake rev: 36a9840a5da17cc14561881fdd6a4f2cb0a75e49)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Debugging suggests that setscene tasks are being a little greedy about their
dependencies, for example, lsof is insisting that gcc-runtime's do_package
is installed. If it isn't, its requiring gcc to rebuild.
If gcc-runtime do_package_write_xxx and do_packagedata is available, there
is no reason do_package should be needed.
The reason this is happening appears to be from the batching up of task
dependencies code, rather than setscene tasks stopping when passing over
a setscene task, they were being carried forward. This patch fixes it
so the data is 'zeroed' when passing over a setscene task boundary,
which gives the dependency graph that is expected.
After this patch, lsof will rebuild quite happily without
gcc-runtime:do_package being present, as expected. This should lead to
less dependencies being installed for builds from sstate and generally
better performance in general.
(Bitbake rev: f8bcb0a1e3b008b71c9a7cd21f76d0906f2d8068)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Added optional parameter 'prefix' to filter out names that
don't start with specified prefix. Changed existing call
of get_providermap according to changed API.
Optimized the code: got rid of extra loop and temporary
list variable virts.
(Bitbake rev: df5a1392d6f91ccb44a99721c7d847da242121bb)
Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com>
Signed-off-by: Elliot Smith <elliot.smith@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Its turning out that we really need a way to have bitbake just run
the setscene tasks but not any real tasks, particularly for SDK
operations.
Add an option for this since its pretty straight forward. This allows
various nasty workarounds in OE-Core to be removed.
(Bitbake rev: e4a2aafa1650a227a04d92a8a0b31efaed2c310e)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
It can be useful to run all tasks up to but not including a specific task. The
main reason this was never added was the lack of a good syntax. This patch
uses the syntax <taskname>- to denote this behaviour which is simple, not
invasive and fits what we need from good syntax IMO, hence we can add this.
(Bitbake rev: 99ccfd411ab3f7baa111f9f3d50fae68816a9a83)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
This firstly prints debug messages which show how bitbake decided to resolve
the virtual/xxx providers which is useful for debugging.
If the siggen has a tasks_resolved() method, it calls this, passing in
the mappings, allowing that to do things with the resolved names.
(Bitbake rev: d473fc84acddfd69a7207affcd89f65ea2ecf730)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The output of "bitbake, -c listtasks pkg" lists tasks with their real names
(starting with "do_"), but then "bitbake -c do_task" fails, as "do_" always
gets unconditionally prepended to task names. This patch handles this error
by checking whether a task starts with "do_" prior to prepending it with it
when the task runlist is being constructed (and a few other corner cases).
[YOCTO #7818]
(Bitbake rev: dd3050ceef37ac556546e940aa596ce96ef6c8df)
Signed-off-by: Alex Franco <alejandro.franco@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Currently bitbake is the only one that knows the relation
between PN and PROVIDES. In some cases it is needed to have
this relation in the data store (the bootloader it's a good
case).
This adds the PROVIDES to the taskdata, so it would be in
the data store as one field of BB_TASKDEPDATA.
(Bitbake rev: a660787311d2961c66c0443bf0e2e094c9baef1b)
Signed-off-by: Mariano Lopez <mariano.lopez@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Bitbake does really weird things with negative or zero numbers of threads
which is confusing to the user. Add a sanity check for this.
When you have code doing arithmetic on the values and a VM reconfigures
to only a single thread, negative numbers are easier than you'd think.
(Bitbake rev: 32166ac3c85ff3c04081580ae76bd63590d6ff3e)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
If we see a BBHandledException in runqueue, the understanding is the system
handled it, printing a log and traceback is just confusing.
Therefore only print these in the cases where its an unknown/unhandled
exception.
(Bitbake rev: 29d28e22ce431c3d3aabdb88ff4d8cca67a1cfad)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Currently if the fakeworker failes to start the output from bitbake is confusing.
Improve the error handling to give a clear indication of what failed.
Patch from Chris Larson.
(Bitbake rev: ad286d6fed7a580bec36a92c7b7e205322ac407b)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
