When setting up the worker we were transfering large amounts of data
which aren't needed until task execution time.
Defer the fakeroot and taskdeps data until they're needed for a specific
task. This will duplicate some information when executing different tasks
for a given recipe but as is is spread over the build run, it shouldn't
be an issue overall.
Also take the opportunity to clean up the silly length argument lists
that were being passed around at the expense of extra dictionary keys.
(Bitbake rev: 3a82acdcf40bdccd933c4dcef3d7e480f0d7ad3a)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
When 'NoCache' was written, databuilder/cookerdata didn't exist. It does
now and the recipe parsing functionality contained in NoCache clearly
belongs there, it isn't a cache function. Move those functions, renaming
to match the style in databuilder but otherwise not changing functionality
for now. Fix up the callers to match (which make it clear this is the right
move).
(Bitbake rev: 783879319c6a4cf3639fcbf763b964e42f602eca)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
(Bitbake rev: 82f40261a06d39f0e7748942f480da5b44282fa3)
Signed-off-by: Oliver Lang <quantenkeks@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The user does need to be told about this but it isn't really a warning,
just something they may need to be aware of. Drop the level accordingly.
(Bitbake rev: 9bdedc8074990e613c9567e2cd8072f8d885f07f)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Prevent new tasks from being scheduled if the memory pressure is above
a certain threshold, specified through the "BB_MAX_PRESSURE_MEMORY"
variable in the conf/local.conf file. This is an extension to the
following commit and hence regulates pressure in the same way:
48a6d84de1 bitbake: runqueue: add cpu/io pressure regulation
Memory pressure is experienced when time is spent swapping, refaulting
pages from the page cache or performing direct reclaim. This is why
memory pressure is rarely seen but might be useful as a last resort to
prevent OOM errors.
(Bitbake rev: 44c395434c7be8dab968630a610c8807f512920c)
Signed-off-by: Aryaman Gupta <aryaman.gupta@windriver.com>
Signed-off-by: Randy Macleod <Randy.Macleod@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
I'm 99% certain this failing of a scenequeue task corrupts runqueue and
causes all kinds of breakage. I'd rather runqueue deadlocked than corrupted
and did weird things so drop this code.
We've seen builds where the deadlock triggers and it then tries to run tasks
where the SQ task already ran with very confusing failures. It is likely it
is this code causing it.
(Bitbake rev: 8efced47fcb47851a370fd6786df6fb377f99963)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Tweak the deadlock breaking messages to be explict about which task is
blocked on which other task. The messages currently imply it is "freeing"
the blocking task which is confusing.
(Bitbake rev: cf7f60b83adaded180f6717cb4681edc1d65b66d)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
We have to prefer one multiconfig over another when deferring tasks, else
we'll have cross-linked build trees and nothing will be able to build.
In the original population code, we sort like this but we don't after
rehashing. Ensure we have the same sorting after rehashing toa void
deadlocks.
(Bitbake rev: 27228c7f026acb8ae9e1211d0486ffb7338123a2)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Prevent the scheduler from starting new tasks if the current cpu or io
pressure is above a certain threshold and there is at least one active
task. This threshold can be specified through the
"BB_PRESSURE_MAX_{CPU|IO}" variables in conf/local.conf.
The threshold represents the difference in "total" pressure from the
previous second. The pressure data is discussed in this oe-core commit:
061931520b buildstats.py: enable collection of /proc/pressure data
where one can see that the average and "total" values are available.
From tests, it was seen that while using the averaged data was somewhat
useful, the latency in regulating builds was too high. By taking the
difference between the current pressure and the pressure seen in the
previous second, better regulation occurs. Using a shorter time period
is appealing but due to fluctations in pressure, comparing the current
pressure to 1 second ago achieves a reasonable compromise. One can look
at the buildstats logs, that usually sample once per second, to decide a
sensible threshold.
If the thresholds aren't specified, pressure is not monitored and hence
there is no impact on build times. Arbitary lower limit of 1.0 results
in a fatal error to avoid extremely long builds. If the limits are higher
than 1,000,000, then warnings are issued to inform users that the specified
limit is very high and unlikely to result in any regulation.
The current bitbake scheduling algorithm requires that at least one
task be active. This means that if high pressure is seen, then new tasks
will not be started and pressure will be checked only for as long as at
least one task is active. When there are no active tasks, an additional task
will be started and pressure checking resumed. This behaviour means that
if an external source is causing the pressure to exceed the threshold,
bitbake will continue to make some progress towards the requested target.
This violates the intent of limiting pressure but, given the current
scheduling algorithm as described above, there seems to be no other option.
In the case where only one bitbake build is running, the implications of
the scheduler requirement will likely result in pressure being higher
than the threshold. More work would be required to ensure that
the pressure threshold is never exceeded, for example by adding pressure
monitoring to make and ninja.
(Bitbake rev: 502e05cbe67fb7a0e804dcc2cc0764a2e05c014f)
Signed-off-by: Aryaman Gupta <aryaman.gupta@windriver.com>
Signed-off-by: Randy Macleod <randy.macleod@windriver.com>
Signed-off-by: Alexandre Belloni <alexandre.belloni@bootlin.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Very occasionally we see errors in eSDK testing on the autobuilder where the task
hashes in the eSDK don't match what was just built. I was able to inspect one of
these build directories and noticed that the bb_unihashes.dat file in the eSDK
was zero sized. Whilst inspecting the code to understand the cause, I noticed that
updated hashes are not saved out in subsequent updates of the values in the rehash
process.
Add a missing sync call to ensure this happens.
(Bitbake rev: 7912dabbcf444a3c3d971cca4a944a8b931e301b)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
We're using the wrong data store when trying to locate siginfo files,
fix this. Thanks to Gregory Lumen <gregorylumen@microsoft.com> for
spotting.
[YOCTO #14774]
(Bitbake rev: 0ed800e19a3197f8e622c8d3b630aae384e60aba)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
This is set at the start of the loop anyway so it does nothing. Drop
the pointless code.
(Bitbake rev: e6a3173c9cdf349ccbd4cf612868f92cce8717c8)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
In line with the inclusive language migration defined at:
https://wiki.yoctoproject.org/wiki/Inclusive_language
replace the use of "abort" with "halt" in code related to handling
task failure.
(Bitbake rev: 831fb7f2329a3cd95b71e9c85d7d7f0d717f947f)
Signed-off-by: Scott Murray <scott.murray@konsulko.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
In line with the inclusive language migration defined at:
https://wiki.yoctoproject.org/wiki/Inclusive_language
rename:
MULTI_PROVIDER_WHITELIST -> BB_MULTI_PROVIDER_ALLOWED
(Bitbake rev: a09546b725fda13c0279638c7c904110da7bf6cd)
(Bitbake rev: d035435c1a4951a45481867cf932faa4a6f8f936)
Signed-off-by: Scott Murray <scott.murray@konsulko.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
In line with the inclusive language migration defined at:
https://wiki.yoctoproject.org/wiki/Inclusive_language
rename:
BB_SETSCENE_ENFORCE_WHITELIST -> BB_SETSCENE_ENFORCE_IGNORE_TASKS
(Bitbake rev: 2e243ac06581c4de8c6e697dfba460ca017d067c)
(Bitbake rev: f8f7b80a0df4646247e58238a52a7d85a37116d4)
Signed-off-by: Scott Murray <scott.murray@konsulko.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The different stamp policies were poor versions of the siggen code and task
hashes, predating it and being used by packaged staging. They had many
limitations, hence their replacement. I'm not aware of any users of that
code any more so I believe it and the assoicated stamp whitelist variable
can simply be removed.
(Bitbake rev: 98407efc8c670abd71d3fa88ec3776ee9b5c38f3)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The previous fix for runall option handling had a small bug in it, it
didn't clear the originally processed task list which meant it was running
too many tasks. Fix this so the list is reset and rebuild correctly.
(Bitbake rev: 87c9e120897ed04dfc64d4752fc602f9bfcb8645)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The runbuild option handling in runqueue was flawed as items deleted from the
main task list may be dependencies and hence cause index errors.
Rather than modify runtaskentries straight away, compute a new shorted list
and use that as an input to the second phase. This avoids the need to add tasks
back to the list meaning delcount can be simplifed to a simple counter.
The second use case in runonly doen't re-add items so doesn't have this
issue.
(Bitbake rev: 3428e3c54eb5cc03ff96f9cee6dc839afee7a419)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Python scales badly when concatinating strings in loops. Most of these
references aren't problematic but at least one (in data.py) is probably
a performance issue as the issue is compounded as strings become large.
The way to handle this in python is to create lists which don't reconstruct
all the objects when appending to them. We may as well fix all the references
since it stops them being copy/pasted into something problematic in the future.
This patch was based on issues highligthted by a report from AWS Codeguru.
(Bitbake rev: d654139a833127b16274dca0ccbbab7e3bb33ed0)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The recent fixes to merge setscene and normal task accounting in runqueue
fixed some display issues but broke the task numbering of setscene tasks.
Add new accounting methods to the stats structure specifically designed
for setscene. This accounts for the fact that setscene tasks can rerun
multiple times in the build.
Then use the new data in the UI to correctly display the numbers the
user wants to see to understand progress.
(Bitbake rev: ed7e2da88bf4b7bfc7ebfc12b9bd6c0fb7d8c1aa)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
When we parallelised normal and setscene tasks, the task stats
handling was left separate pending further thought. We had to remove
handling of the setscene tasks from the UI in order to maintain
consistent task numbering.
Currently, "0 of 0" tasks can be shown as setscene tasks execute
until the first normal task runs.
The only use left for sq_stats is in the active task numbers which
we can use the length of sq_ive for instead. We can therefore
drop it and return stats in all cases. This removes the "0 of 0" task
problem since the stats in all normal and setscene tasks matches.
[YOCTO #14479]
(Bitbake rev: eae6e947e37e18cded053814bd2a268b44fb25cd)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
In multiconfig builds with large numbers of identical tasks, builds were
deadlocking after recent runqueue changes upon rebuilds where there was
heavy sstate usage (i.e. on second builds after a first completed).
The issue was that deferred tasks were being left indefinitely on
the deferred list. The deadlock handler was then "breaking" things
by failing tasks that had already succeeded, leading to the task
being on both covered and not covered lists, giving a further error.
The fix is to clean up the deferred task list when each setscene task
completes. I'd previously been hoping to avoid iterating that list
but it appears unavoidable.
[YOCTO #14342]
(Bitbake rev: ae24a0f2d2d8b4b5ec10efabd0e9362e560832ea)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
If the deferred task deadlock avoidance code triggers, it could mark an executed
task as failed which leads to "covered and not covered" error messages. Improve
the logic so if the deadlock code is triggered, it doesn't cause the errors.
(Bitbake rev: 51bdd6cb3bd9e2c02e261fb578bb945b86b82c75)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The previous patches have exposed new issues with this code path,
the issues being around what should happen when the hash of a task
changes and the task is or is not on the deferred task list.
Rather than rebuilding the deferred task list during each rehash
event, build it once at the start of a build. This avoids the problem
of tasks being added back after they have run and also avoids problems
of always ensuring the same task is deferred. It also allows the
'outrightfail' codepath to be handled separately as the conditions
are subtly differnt.
One significant win for the new approch is the build is not continually
printing out lists of deferred tasks, that list remains fairly static
from the start of the build. Logic is added in to ensure a rehashed
task with a hash matching other deferred tasks is deferred along with
them as a small optimization.
An interesting test case for this code was reported by Mark Hatle
with four multiconfigs, each the same apart from TMPDIR and running a
build of:
bitbake buildtools-tarball mc:{one,two,three,four}:core-image-minimal
which is interesting in that the build of buildtools partially overlaps
core-image-minimal and the build has a rehash event for qemuwrapper-cross
even without any external hash equivalence server or preexisting data.
(Bitbake rev: bb424e0a6d274d398f434f7df63951da9ce305b3)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
If the hash of a task changes and that hash is a deferred task (e.g. a multiconfig
build), we need to ensure that the hash change propagates through to all the tasks
else the build will run multiple copies of the task, sometimes with oddly differing
results as the outhashes of native tasks built in differing locations can confuse
things.
(Bitbake rev: 2db571324f755edc4981deecbcfdf0aaa5a97627)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
We were testing the validity of deferred tasks setscene status "up front" which
is very unlikely to succeed and leads to cache invalidation issues. With the
change to rebuild the deferred task list, this status becomes out of sync. The
result was tasks being executed when they should not have been leading to extra
work for the build unnecessarily.
Instead, don't process validity status for deferred tasks and assume their
data will become available. If it doesn't, this will now result in a build
error as the setscene task will fail and the main task will run instead.
In theory we could try and track the state changes in the deferred list and
re-test validity then but I'm not sure it is worth the effort when the other
code path and errors in setscene tasks will give a pretty good idea of what
is happening anyway.
(Bitbake rev: edcafac13b3b241b6687419e59018d21811507a1)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
In a multiconfig situation there are circumstances where firstly, tasks
are deferred when they shouldn't be, then later, tasks can end up as
both covered and not covered.
This patch fixes two related issues. Firstly, the stamp validity checking
is done up front in the build and not reevaulated. When rebuilding the
deferred task list after scenequeue hash change updates, we need therefore
need to check if a task was in notcovered *or* covered when deciding to
defer it. This avoids strange logs like:
NOTE: Running setscene task X of Y (mc:initrfs_guest:/A/alsa-state.bb:do_deploy_source_date_epoch_setscene)
NOTE: Deferring mc:initrfs_guest:/A/alsa-state.bb:do_deploy_source_date_epoch after mc:host:/A/alsa-state.bb:do_deploy_source_date_epoch
where tasks have run but are then deferred.
Since we're recalculating the whole list, we also need to clear it before
iterating to rebuild it. By ensuring covered tasks aren't added to the
deferred queue, the covered + notcovered issue should also be avoided.
in the task deadlock forcing code.
[YOCTO #14342]
(Bitbake rev: 3c8717fb9ee1114dd80fc1ad22ee6c9e312bdac7)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
There is further evidence of tasks ending up being "covered" and "notcovered"
which shouldn't happen and is bad. The code that caused this problem last
time appears to have issues where stamps for tasks already exist.
Split out the setscene stamp checking code to a separate function and
use this when checking "hard dependencies" (like pseudo-native) so
that if the stamps exist and it will be "covered", it is not put on
the notcovered list.
(Bitbake rev: a1848a481e36b729c8e4130c394b1d462d4b488a)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Use the new functionality in build.py to identify stale setscene tasks
and send an event to the metadata listing them. The metadata then
has the option of performing cleanup operations if it thinks that
appropriate.
(Bitbake rev: ef8c980a3ae92c168b7ca16a4d19cd38a9574761)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Currently if pseudo fails we can only see the path to pseudo.log. If we
have no access to server and can only rely on bitbake log then debugging
becomes impossible. This printing needs to be added in runqueue level,
not inside task execution, because in some cases task fails with pseudo
abort really early and we don't even see any log.
In this change I'm adding pseudo log printing in every fakeroot task
failure that logged `mismatch`, `error` or `fatal` to logfile, because
we have no other way to communicate with pseudo if it failed or not.
Only lines from last pseudo server execution will be printed.
(Bitbake rev: e7c664a947903ed7b868abef62af2ff5f8ef0dc6)
Signed-off-by: Tomasz Dziendzielski <tomasz.dziendzielski@gmail.com>
Signed-off-by: Jan Brzezanski <jan.brzezanski@gmail.com>
Signed-off-by: Adrian Walag
Signed-off-by: Paulo Neves <ptsneves@gmail.com>
Signed-off-by: Mikolaj Lasota <mikolaj.lasota@protonmail.com>
Signed-off-by: Wiktor Baura <wbaura@gmail.com>
Signed-off-by: Kamil Kwiek <kamil.kwiek@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
We've seen hard to debug issues where a task ends up in both the
covered and notcovered list. Add a sanity check to ensure if this
happens in future, we see it in the logs.
(Bitbake rev: 6e001410854792f9bb66a0409a2ac176171b0507)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
We've seen occasional issues where linux-yocto:do_compile_kernelmodules would
run without do_shared_workdir running before it. do_shared_workdir is an
setscene task but never has an sstate object generated so it will always
rerun. This should not happen since compile_kernemodules should only
execute if a setscene that depends on it didn't run and that should trigger
do_shared_workdir not to be marked as covered.
The issue is that build-appliance-image:do_package is one of the tasks which
covers linux-yocto:do_compile_kernelmodules but it is also a noexec task
and has a dependecy on pseudo-native:do_populate_sysroot.
In the problem case, pseudo-native:do_populate_sysroot is unavailable but
marked as covered since it is noexec. The "harddeps" code then also marks it
as notcovered. No task should ever be both covered and notcovered and this
is where the problems come from.
The solution is for the harddeps code only to to fail tasks if they've not
already been handled in some way. The code is assuming code couldn't have
handled revdeps at this point but we now have clear evidence they can.
(Bitbake rev: f66556bbb38449789ceea2fd105e9f68df7fb660)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Currently each task has to have a umask specified individually. This
is leading to determinism issues since it is easy to miss specifying
this for an extra task.
Add support for specifing the default task umask globally which
simplifies the problem.
(Bitbake rev: 3e664599fd54a8a37ce587022fcbce5ca26f2ed3)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
When multiconfig is used bitbake might try to run events that don't
exist for specific mc target. In cooker.py we pass
`self.databuilder.mcdata[mc]` data that contains names of events'
handlers per mc target, but fire_class_handlers uses global _handlers
variable that is created during parsing of all the targets.
This leads to a problem where bitbake runs event handler that don't
exist for a target or even overrides them - if multiple targets use
event handler with the same name but different code then only one
version will be executed for all targets.
See [YOCTO #13071] for detailed bug information.
Add mc target name as a prefix to event handler name so there won't be
two different handlers with the same name. Add internal __BBHANDLERS_MC
variable to have the handlers lists per machine.
(Bitbake rev: 5f7fdf7b2d8c59805c8ef4dae84f536baa5e172b)
Signed-off-by: Tomasz Dziendzielski <tomasz.dziendzielski@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The bitbake logger overrode the definition of the debug() logging call
to include a debug level, but this causes problems with code that may
be using standard python logging, since the extra argument is
interpreted differently.
Instead, change the bitbake loggers debug() call to match the python
logger call and add a debug2() and debug3() API to replace calls that
were logging to a different debug level.
[RP: Small fix to ensure bb.debug calls bbdebug()]
(Bitbake rev: f68682a79d83e6399eb403f30a1f113516575f51)
Signed-off-by: Joshua Watt <JPEWhacker@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
When we run `devtool build mc` recipe's task dependencies are expanded
to "mc:do_populate_sysroot" where "mc" name is treated as multiconfig
and "do_package_sysroot" as multiconfigname.
| ERROR: Multiconfig dependency mc:do_populate_sysroot depends on
| nonexistent multiconfig configuration named do_populate_sysroot
(Bitbake rev: 3ce4b2caccfe608a54dff159459f3687ea610597)
Signed-off-by: Tomasz Dziendzielski <tomasz.dziendzielski@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
We should not be using sys.argv inside the server to decide which targets
the user added on the commandline. There might not even be a commandline.
Thankfully the targets variable is easily accessible in this context
and contains this exact data we want.
(Bitbake rev: 5b12bf30bccdd00262e74964223220c649040be4)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The levels of indirection to set these verbose logging options is rather
crazy. This attempts to turn things into two specific options with
much more specific meanings. For now its all still controlled by the
commandline verbose option and should funciton as previously, with
the addition that the BB_VERBOSE_LOGS option can now be task specific.
(Bitbake rev: 423c046f2173aaff3072dc3d0882d01b8a0b0212)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
In rare cases the pickled data from a task contains "</event>" which
causes backtrace. This can be reproduced with something like:
do_unpack_prepend () {
bb.warn("</event>")
}
There are several solutions but the easiest is to catch this exception
and look for the next marker instead as this should be the only way such
an unpickle error could occur.
This fixes rare exceptions seen on the autobuilder.
Also add in other potential exceptions listed in the pickle manual
page so that better debug is obtained should there be an error in
this code path in future. exitcode doesn't need the same handling
since we control what is in that data field and it could never contain
</exitcode>
(Bitbake rev: 5ada512d6f9cbbdf1172ff7818117c38b12225ca)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Passing all the data caches to the task hashing functions allows them to
correctly account for mcdepends in task signatures. This allows tasks to
be correctly re-run when a mcdepends changes.
By default, the legacy behavior is maintained for derived signature
generators by passing a special proxy object that can either be used to
access all multiconfigs or the legacy behavior. If a derived signature
generator is updated, it can set the supports_multiconfig_datacaces
property to instruct bitbake it deals with multiconfigs properly.
[YOCTO #13724]
(Bitbake rev: 8ff9203de4fce9c104c2987d86980c9f34036b97)
Signed-off-by: Joshua Watt <JPEWhacker@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
All of the other multiconfig splitting functions are located in runqueue
so move the function to split a pn/fn there also so that its easier to
see them all together. Fixes a case where the findBestProvider() command
wasn't working for multiconfig because it was looking for a prefix of
"multiconfig:" instead of the newer "mc:"
(Bitbake rev: 325827af66434affc2da460cc8b9a5c460e38056)
Signed-off-by: Joshua Watt <JPEWhacker@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Splits the cooker to track a collection per multiconfig instead of a
single collection for all multiconfigs. Practically speaking, this
allows each multiconfigs to each have different BBMASKs that apply to it
instead of each one using the mask specified in the base configuration.
(Bitbake rev: dd6d8eca2027f8d9be8a734a493227b440075e49)
Signed-off-by: Joshua Watt <JPEWhacker@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Lowers the level of the log message when setscene tasks have completed.
This message can occur multiple times when hash equivalence is enabled,
since the runqueue switches between executing setscene tasks and normal
tasks. Since this is primarily of use when debugging hash equivalence,
use the hash equivalence logger at VERBOSE level.
[YOCTO #13813]
(Bitbake rev: 7dd5b3900622008ff34ec70d71c6e994f460a46f)
Signed-off-by: Joshua Watt <JPEWhacker@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Lowers the level at which hash equivalence messages are logged so as to
not annoy the majority of users. The autobuilder can use a custom
logging configuration to log these to a file for debugging (see
contrib/autobuilderlog.json)
[YOCTO #13813]
(Bitbake rev: 2ddb649ea31afe052f08e3969e36abf6fb515bc2)
Signed-off-by: Joshua Watt <JPEWhacker@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Switches the hash equivalence logging to use a different logger so that
it can be easily filtered out with python's structured logging.
(Bitbake rev: 20bb29ef973e9c5483eb50a74550ea207637367b)
Signed-off-by: Joshua Watt <JPEWhacker@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Passes around the actual logging level as the default log level variable
instead of the debug count. This makes it easier to deal with logging
levels since the conversion from debug count and verbose flag only has
to occur once when logging is initialized and after that actual log
levels can be used
(Bitbake rev: 41bd155faf7f65cb0727fcce972715769b26ca89)
Signed-off-by: Joshua Watt <JPEWhacker@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Previously --runonly=do_task would give a misleading error like:
ERROR: Could not find any tasks with the tasknames ['do_task'] to run
within the recipes of the taskgraphs of the targets...
The problem is that BitBake tried to find "do_do_task". So teach it to
only add the do_ prefix if it's not already there.
(Bitbake rev: 694904bde980606dc67c201da61f4fb685679b17)
Signed-off-by: Chris Laplante <chris.laplante@agilent.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
removed unused imports which made the code harder to read, and slightly
but less efficient
(Bitbake rev: 4367692a932ac135c5aa4f9f2a4e4f0150f76697)
Signed-off-by: Frazer Clews <frazer.clews@codethink.co.uk>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
This should give performance improvements to functions using this list of
tasks (sets are used for most of the other code for this reason, not sure
why this wasn't a set in the first place).
(Bitbake rev: f5daef68703481a3c243dfecc7de404e6ebfdbb6)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
A corner case was identified where tasks with valid stamps from previous
builds need to be accounted for in the new sanity test in the migration
code. Add a variable to track such completed tasks to ensure the sanity
test works correctly.
(Bitbake rev: d517b1ef13ca7ab2fb4d761d3bd3b9fb7c591514)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
We've seen a number of mystery failures where task B would run despite
task A, its dependency not having run. An example would be do_compile
when do_unpack didn't run.
This has been tracked down to this code block. In theory it shouldn't
trigger however it can and has due to bugs elsewhere. When it does, it
causes significant weird failures and possible build corruption.
Change the code to abort the build. This avoids any chance of corruption
and should ensure the issues get reported, putting an end to the weird
build failures.
There may be some cases where this triggers and it shouldn't, we'll work
through those as they arise and are identified.
(Bitbake rev: 7a92b7f58ab187eddfe550bd6fb687240c7b11bb)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Regardless of whether we remapped the hash on the server or not, we need
to have bitbake work as if we did as we need to match how the stamp files
look.
This change resolves build failures where tasks were rerunning when they
shouldn't.
(Bitbake rev: 40928f6991436cf687821015324483b205abfcb1)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
This doesn't save much time but does make the profile counts for the
function more accurate which is in itself useful.
(Bitbake rev: d446fa89d206fbc6d098215163c968ea5a8cf4a9)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
We were seeing this running thousands of times with hashequiv, do
the filtering where it makes more sense and make it persist.
(Bitbake rev: 2cfeb9998a8ad5b1dcda0bb4e192c5e4306dab17)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Move the calls to difference_update out a code level which improves efficiency
significantly.
Also further combine the outer loop for efficiency too.
These two changes remove a bottleneck from the performance charts.
(Bitbake rev: e28ec69356f1797de3e4e3fca0fef710bc4564de)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Creating a new sorted list of sstate tasks each iteration through runqueue is
extremely ineffecient and was compounded by the recent change from a list to set.
Create one sorted list instead of recreating it each time.
(Bitbake rev: de18824996841c3f35f54ff5ad12f94f6dc20d88)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
There are two operations happening in get_taskhash, the building of the
underlying data and the calculation of the hash.
Split these into two funtions since the preparation part doesn't need
to rerun when unihash changes, only the calculation does.
This split allows sigificant performance improvements for hashequiv
in builds where many hashes are equivalent and many hashes are changing.
(Bitbake rev: 6a32af2808d748819f4af55c443578c8a63062b3)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The "no dependencies" task case was not being correctly considered in this
code and seemed to be the cause of occasionaly task hash mismatch errors
that were being seen as the dependencies were never accounted for properly.
(Bitbake rev: 608b9f821539de813bfbd9e65950dbc56a274bc2)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The looping over multiple changed hashes causes many calls to get_taskhash
and get_unihash which are potentially slow and then overwritten.
Instead, batch up all the tasks which have changed unihashes and then
do one big loop over the changed tasks rather than each in turn.
This makes worlds of difference to the performance graphs and should speed
up build where many tasks are being rehashed.
(Bitbake rev: c9c68d898985cf0bec6fc95f54c151cc50255cac)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Batch all updates to scenequeue data together in a single invocation
instead of checking each task serially. This allows the checks for
sstate object to happen in parallel, and also makes sure the log
statement only happens once (per set of rehashes).
(Bitbake rev: db033a8f8a276d864bdb2e1eef159ab5794a0658)
Signed-off-by: Joshua Watt <JPEWhacker@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Based on the hashserv's new ability to accept hash mappings, update runqueue
to use this through a helper function in siggen.
This addresses problems with meta-extsdk-toolchain and its dependency on
gdb-cross which caused errors when building eSDK. See the previous commit
for more details.
(Bitbake rev: 39098b4ba2133f4d9229a0aa4fcf4c3e1291286a)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
When follow the instructions of multiconfig from Yocto dev manual that
set in core-image-sato recipe:
do_image[mcdepends] = "multiconfig:x86:arm:core-image-minimal:do_rootfs"
it show too many annoying warnings look like:
| WARNING: Deferring mc:x86:virtual:native:/buildarea6/kkang/poky/meta/recipes-support/libxslt/libxslt_1.1.33.bb:do_populate_sysroot
| after mc:arm: virtual:native:/buildarea6/kkang/poky/meta/recipes-support/libxslt/libxslt_1.1.33.bb:do_populate_sysroot
Treat them as infomations rather than warnings.
(Bitbake rev: 0fb4b5153237af4a13b2c65711ab798b0de06c2f)
Signed-off-by: Kai Kang <kai.kang@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Setscene hard dependencies were not being correctly handled during task migration.
For example, do_package of recipe X might become valid due to hashequiv yet we're
still rebuilding pseudo-native, a harddep of do_package. This would mean
it would try to execute that setscene task despite pseudo not being present.
Fix this by ignoring tasks with failed harddependencies. This does mean
stlightly more rebuilds than is optimal but it avoids build crashes. Ultimately
the new runqueue model can likely better handle these cases than the older codebase
could but that is for another more invasive patch.
(Bitbake rev: 0dc61e8b9964f7fe41d51b6a481ca7aeaacd8bce)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The key problem is that currently running setscene tasks are not
accounted for when processing task migrations. This means can allow
two of the same task to execute at the same time with unpredictable
effects.
This change allows us to stop doing that and refactor the code slightly
to make it clearer that these conditions don't arrive even with
deferred tasks.
(Bitbake rev: ac1922d348613ca46a1047ad5ddf755eac16d568)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Bibake is currently too 'chatty' when hash equivalence is enabled. Fix
this by only printing the log output if a rehash happens and it matches
an sstate object.
Also, pass a summary option to the hash checking function. This was
already changed to a mechanism which allows addition of new parameters
so this should be backwards and forwards compatible.
(Bitbake rev: 0c4515603ad08775e3b0404cba5374367e49f236)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
A minor performance optmisation to keep lists smaller when running large
builds. We can do this since once a task has been built, we don't need
to worry about it. This improves a major bottleneck that shows up on
performance profile charts in dryruns.
(Bitbake rev: cd6b89230823707c3c9bb9e6883bf5a971916581)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
There are some runqueue code paths where the unihash cache would not be
saved where for example only parsing or an occurred. Save the cache at the
end of runqueue generation to ensure entries are cached.
(Bitbake rev: 9eee0d36870c11dd303894a6151c33a83bd3a1bc)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
We need to set the setscene tasklist before we call into the
taskhash/unihash code else the behaviour is inconsistent.
Avoid reporting hashes for non setscene tasks since we'd never
query that.
(Bitbake rev: 419a7840b8627278db694029c25df00214d01d96)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Currently runqueue will rerun setscene tasks multiple times as hashes
change. This has caused numerous problems since a setscene task may
become "unavailable" for some future signature combination and the code
then can't easily "unskip" tasks its already passed into the execution
queue.
At least for now, only run setscene once and assume they're equivalent
at that point. In practise that has been much more stable in testing.
Tweak the test to match the change in behaviour.
(Bitbake rev: 4205a3ef23834f317642bba155d67cd772176fb6)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Tasks were not migrating consistently, particularly:
* if a task was rehashed which had already run
* if a task which was valid became invalid due to a rehash
We need to always run the migration code for rehashed tasks and then
reprocess them for hash validity. This means rearranging the code.
It also means several tests are no longer correct and can't be written
correctly to work on all possible workflows so those are removed.
(Bitbake rev: 8443989ee41e9b162972935513e437b5c66ea74d)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
According to sstate_checkhashes which is defined in sstate.bbclass, the
currentcoun should be a number (0, not None).
Fixed:
$ bitbake base-files -Sprintdiff
> bb.plain("Sstate summary: Wanted %d Found %d Missed %d Current %d (%d%% match, %d%% complete)" % (total, len(found), len(missed), currentcount, match, complete))
TypeError: %d format: a number is required, not NoneType
(Bitbake rev: 45cb73e2846eaffe8964a573875f54808e8f3633)
Signed-off-by: Robert Yang <liezhi.yang@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Reworks the hash equivalence server to address performance issues that
were encountered with the REST mechanism used previously, particularly
during the heavy request load encountered during signature generation.
Notable changes are:
1) The server protocol is no longer HTTP based. Instead, it uses a
simpler JSON over a streaming protocol link. This protocol has much
lower overhead than HTTP since it eliminates the HTTP headers.
2) The hash equivalence server can either bind to a TCP port, or a Unix
domain socket. Unix domain sockets are more efficient for local
communication, and so are preferred if the user enables hash
equivalence only for the local build. The arguments to the
'bitbake-hashserve' command have been updated accordingly.
3) The value to which BB_HASHSERVE should be set to enable a local hash
equivalence server is changed to "auto" instead of "localhost:0". The
latter didn't make sense when the local server was using a Unix
domain socket.
4) Clients are expected to keep a persistent connection to the server
instead of creating a new connection each time a request is made for
optimal performance.
5) Most of the client logic has been moved to the hashserve module in
bitbake. This makes it easier to share the client code.
6) A new bitbake command has been added called 'bitbake-hashclient'.
This command can be used to query a hash equivalence server, including
fetching the statistics and running a performance stress test.
7) The table indexes in the SQLite database have been updated to
optimize hash lookups. This change is backward compatible, as the
database will delete the old indexes first if they exist.
8) The server has been reworked to use python async to maximize
performance with persistently connected clients. This requires Python
3.5 or later.
(Bitbake rev: 2124eec3a5830afe8e07ffb6f2a0df6a417ac973)
Signed-off-by: Joshua Watt <JPEWhacker@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
We only have hash equivalence for setscene tasks so only query the server
for those, reducing the number of connections needed.
(Bitbake rev: 22082c7b3ca0cffcedb7d1d8c6681d35286376db)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Specifying the force flag will now cause runall and runonly to
invalidate the tasks before running them. This allows a --runall or
--runonly to force the tasks to run, even if they would have otherwise
been skipped, e.g.:
bitbake -f --runall fetch
Will run all do_fetch tasks even if they wouldn't be necessary (for
example, skipped by setscene)
(Bitbake rev: 71e52d3822016027106f2a2e74b8dfdf20f5dc1e)
Signed-off-by: Joshua Watt <JPEWhacker@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Rather than repeatedly calling mc_from_tid() do this in the parent,
removing around a million function calls. Takes time spent in this
function from 40s to 36s.
(Bitbake rev: 28b3f0d8867804799420689c314ac4a8f01efb8c)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
There are other data structures which can be reprocessed at the same
time as holdoff_tasks, further improving build efficiency in various
places.
(Bitbake rev: 02090b3456b7a2de12e72dfeaabfd3b631609924)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
We don't need to process the holdoff task list until we're executing tasks
which saves some data manipulation, at the cost of some data structures
not being correct at all times. This saves significant amounts of time
in various profile charts of larger builds.
(Bitbake rev: 270f076111b12eab358417b0c4cf9c70d7cc787a)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
We've observed do_package and do_package_setscene running in parallel. The
reason is that holdoff_tasks wasn't getting updated. Looking at the code, it
would seem the reason is that the task was in pending_migrations and hence
changed wasn't set and holdoff_tasks wasn't updated.
Fix this. It only affects builds with rehashing enabled.
(Bitbake rev: e26e61e84575669bd223f6ab316798097ed95ec8)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
bitbake <target> --runonly=fetch
failed as the target_tids list included entries which were no longer targeted
task ids. Fix this.
(Bitbake rev: 94e848ae6544e628a19cb97115279b0b1678967c)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
This doesn't appear to have ill effects right now but there is a correctness
issue which this so fix it.
(Bitbake rev: a5e084a266f63c2fd370122327615e49beaeb94e)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
This was overwriting data in the parent which caused all kinds of
odd/weird failures.
(Bitbake rev: 4c5aeb424247a9d0c907524ffacd9c61fcdc0852)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
When the task hashes change we need to ensure the stampcache is cleared out
else tasks don't rerun when they should as we're basing decisions on stale
cache data.
(Bitbake rev: 08962092d3bb7887d82f97d442a6103c0677eae7)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
We weren't marking this special case of setscene task as buildable leading to
runqueue task failures.
(Bitbake rev: 930efbc563443d82df8d692bb8ff172ca2bae192)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Further testing shows we should test some extra datastructures to help pinpoint logic
errors more precisely. This adds some further data structure sanity checks.
(Bitbake rev: 83c4370b25c3a14cc946965c5c5f83ea28f488a1)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
This debug statement could result in a long list of tasks which when
repeatedly sent over our IPC, slowed down the builds immensely. Remove
it in favour of other more targeted debugging added recently, bringing
back some lost performance, particularly on builds with large numbers
of tasks.
(Bitbake rev: 85fe627fdb6510f0942917964386fad9d8c479c8)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The event from the task notifiing of hash equivalency should only be processed
when the task completes. This can otherwise result in a race where a dependent
task may run before the original task completes causing various failures.
To make this work reliably, the code had to be restructured quite a bit.
(Bitbake rev: 1bf5be46f92f125193638cf41ff207d68f592259)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The changed_setscene variable here is just odd and not needed. Worse,
it could prevent some tasks from being removed from the holdoff tasks
list. The list is being rebuilt and should work as intended just from
the other data, this is a leftover from previous versions of the code
as far as I can tell.
(Bitbake rev: 030b9f2b3ce6ed40e79304eb0ffee6c6613f43be)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Whilst we had good runqueue failure mode debug, it hadn't adapted to the
scenequeue changes. Run the scenequeue sanity tests at the end of
a build and output the results regardless of whether all setscene tasks
completed or not. This *massively* improves the ability to debug runqueue
problems.
(Bitbake rev: b9b2177473c0b95a23bd519a201e1d2ba101c6c1)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Looking at the profile information, a lot of time is being spent in
next_buildable_task. This is probably due to the generator expressions
not working well with the empty test.
The easiest way to improve things is to switch to using set manipulations.
We also don't need to update self.buildable the way the original code did
as we don't rely on that anywhere.
(Bitbake rev: 3bcf9ad4964b7e42d1a02ce231e9db42a81ead2a)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
If tasks are in the covered list of tasks for a given setscene task,
it needs to wait for those to complete before we can start.
(Bitbake rev: fdee640c26750b852eb68f5c80437377aa300ed8)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The previous tasks_covered and tasks_notcovered were basically unstable
data structures. We couldn't always tell whether tasks should be covered
or not when trying to repair the sturcture if sstate tasks reran.
In the end its simpler to throw the lists away and rebuild them based upon
current data rather than trying to patch it adhoc. This turns out to be
simpler and much more reliable and I've much more confidence in this code.
(Bitbake rev: 52ee2ba2c617d928569f5afa404925c8b6f317bc)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
We need to copy this set, not modify the original else all kinds
of weird and bad things break, mostly from circular references.
We'll not go into how much sleep I lost tracking down the fallout
from this.
(Bitbake rev: 49927546d2b306830c98f6f9da4a6ad828f6a3a6)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Currently siggen uses the format "<filename>.<taskname>" for referencing tasks
whilst runqueue uses "<filename>:<taskname>". This converts to use ":" as the
separator everywhere.
This is an API breaking change since the cache is affected, as are siginfo files
and any custom signature handlers such as those in OE-Core.
Ultimately this will let us clean up and the accessor functions from runqueue,
removing all the ".rsplit(".", 1)[0]" type code currently all over the place.
Once a standard is used everwhere we can update the code over time to be more
optimal.
(Bitbake rev: 07e539e1c566ca3434901e1a00335cb76c69d496)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
This function uses an old API which uses offsets into lists as a communication
mechanism. Update the API to use "tid" which is used universally in runqueue now.
We can also add kwargs support to the funciton definition to drop some of the
backwards compaiblility hoops we had to jump though with different function
argument combinations.
(Bitbake rev: dc23550047e5078da491ce9a6f30989cb5260df6)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Its useful, particularly in the local developer model of usage, for
bitbake to start and stop a hash equivalence server on local port,
rather than relying on one being started by the user before the build.
The new BB_HASHSERVE variable supports this.
The database handling is moved internally into the hashserv code so that
different threads/processes can be used for the server without errors.
(Bitbake rev: a4fa8f1bd88995ae60e10430316fbed63d478587)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Whilst this isn't strictly necessary, its helpful if the log output is
consistent and its also helpful if bugs either appear or don't appear
for a specific configuration. Ensuring the various iterations we make
are deterministic (sorted) helps with this.
(Bitbake rev: 6a901bb904a97ca90d88be2c6901d3d32346282f)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
There is a compelling usecase for tasks being able to notify runqueue
that their "unihash" has changed. When this is recieved, the hashes of
all subsequent tasks should be recomputed and their new hashes checked
against existing setscene validity. Any newly available setscene tasks
should then be executed.
Making this work effectively needs several pieces. An event is added
which the cooker listen for. If a new hash becomes available it can
send an event to notify of this.
When such an event is seen, hash recomputations are made. A setscene
task can't be run until all the tasks it "covers" are stopped. The
notion of "holdoff" tasks is therefore added, these are removed from
the buildable list with the assumption that some setscene task will
run and cover them.
The workers need to be notified when taskhashes change to update their
own internal siggen data stores. A new worker command is added to do this
which will affect all newly spawned worker processes from that worker.
An example workflow which tests this code is:
Configuration:
BB_SIGNATURE_HANDLER = "OEEquivHash"
SSTATE_HASHEQUIV_SERVER = "http://localhost:8686"
$ bitbake-hashserv &
$ bitbake automake-native
$ bitbake autoconf-native automake-native -c clean
$ bitbake m4-native -c install -f
$ bitbake automake-native
with the test being whether automake-native is installed from sstate.
(Bitbake rev: 1f630fdf0260db08541d3ca9f25f852931c19905)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
