Currently if the fakeworker failes to start the output from bitbake is confusing.
Improve the error handling to give a clear indication of what failed.
Patch from Chris Larson.
(Bitbake rev: ad286d6fed7a580bec36a92c7b7e205322ac407b)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
In some cases we need to check specifically for siginfo files, in
some cases we need to check for the actual sstate objects themselves.
Therefore make this a parameter to the function. A fallback to the
previous function style is maintained for now.
(Bitbake rev: 18d3a03e1b07c98b2dce46eb94f30de1a2b4320b)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
This ensures that _append, _prepend, overrides, etc are functional when used
on sstate variables (e.g. SSTATE_DIR).
[YOCTO #7564]
(Bitbake rev: 2e683c25b856b431198573f7f352d841587275e6)
Signed-off-by: Christopher Larson <kergoth@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
After Ctrl+C is pressed to interrupt bitbake, it loops continually, running
at 100% cpu. This patch selects on the correct file descriptors resolving
the excess cpu usage.
(Bitbake rev: 497404e8484b7ca7c11e459bf0845642156eb677)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
This has always bothered me:
NOTE: Preparing runqueue
NOTE: Executing RunQueue Tasks
This patch changes the messages to be consistent.
(Bitbake rev: 72ac9f9227fbfb4dc8b933b357d21aa0e4060959)
Signed-off-by: Gary Thomas <gary@mlbassoc.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
We need to transfer some of the siggen data from the core/cooker into
the worker instances. There was a partial API created for this but
its ugly and its not possible to extend it from the siggen class.
This patch completes the interface/abstraction for the data and
means the class can extend/customise it in any siggen class.
(Bitbake rev: cf2d642052979d236185c5b8ca2c5478c06e62ae)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Currently we get no profiling oversight into either the main bitbake worker
process, or the overall parsing before task execution. This adds in extra
profiling hooks so we can truly capture all parts of bitbake's execution
into the profile data.
To do this we modify the 'magic' value passed to bitbake-worker to trigger
the profiling, before the configuration data is sent over to the worker.
(Bitbake rev: 446e490bf485b712e5cee733dab5805254cdcad0)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Currently, if you have hard dependencies between setscene tasks (like avahi on
base-passwd through useradd.bbclass), other dependencies may not be installed
even if these exist in sstate. For example, avahi -> expat -> pigz-native
(and avahi -> base-passwd) yet if you cleansstate base-passwd:
bitbake gzip-native:do_clean avahi:do_clean expat:do_clean pigz-native:do_clean base-passwd:do_cleansstate
bitbake avahi | tee
you will currently see pigz-native being rebuilt even though it was in
sstate. The fix for this is to continue to iterate dependency chains
around hard blocked dependencies as per this patch.
After this patch is applied, you will see pigz-native installed from sstate.
(Bitbake rev: f787957a224e8c2682a19e5c4a4d9c86bdce52ba)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
(Bitbake rev: 000fa81013205dd9bc907ff7a61f06f57637212d)
Signed-off-by: Robert P. J. Day <rpjday@crashcourse.ca>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
(Bitbake rev: 4e9aef14d747c37444a4fc683f9641906906afe9)
Signed-off-by: Robert P. J. Day <rpjday@crashcourse.ca>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Its useful to have an event emitted when all of the sceneQueue tasks
have completed since the metadata can hook this for processing.
Therefore add such an event.
(Bitbake rev: 38d4f65bf1cbcdd5a2d60dff0e1d2859c34ed62e)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
If you hit Ctrl+C at the right point, the system processes the request
but merrily continues building. It turns out finish_runqueue() is called
but this doesn't stop the later generation and execution of the
runqueue.
This patch adjusts some of the conditionals to ensure the build really
does stop.
(Bitbake rev: 39b08c604ba713100e174c136f81f18eca6ef33d)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
In dry run mode, stamps for noexec tasks are being written out which
is incorrect. Avoid this.
(Bitbake rev: aa6448a0552ba2947ac262b8b5314a593d1058d3)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
When looking at a list of tasks, do_patch and do_unpack were being
given equal priority when one clearly depends on another. The
reason for this was the default task weights of 0 being to tasks.
This is therefore changed to 1 to allow correct weighting of dependencies
which means the scheduler has better information available to it about
tasks.
Weight endpoints differently (10) for clearer debugging of priorities.
(Bitbake rev: 12dc1d17fac3e8ec420fcafb06186d32fd847d89)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The zero priority task should be run first but was being confused with
the None value the priority field defaulted to. Check for None
explicitly to avoid this error.
In the real world this doesn't change much but it confused the debug
output from the schedulers.
(Bitbake rev: 49c9d8c9400f74c804c2f36462639236e0841ff0)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The first part of the sstate code checks en-mass whether given checksums
are available. The next part of the code then either triggers those
setscene tasks either running them or skipping them if they've been
covered by others.
The problems was that this second part would always skip a task if it
was unavailable in the first part, even if it would have otherwise been
covered by other tasks.
This mean the mere presence of an artefact (or lack of presence) could
cause a different build failure.
The issue reproduces if you run a build and populate an sstate feed, then
run a second build off that feed, then run a third build off the sstate
feed of the second build (which is reduced in size).
The fix is rather than immediately skipping tasks if the checksum is
unavailable, create a list of missing tasks, then, if that task cannot
be covered by others we can skip it later. The deferral makes the
behaviour the same even when the cache is "incomplete".
[YOCTO #6081]
(Bitbake rev: 5edb1a3e3f454ba6e65551174d86229db2f99636)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
If a setscene task has [depends], its possible they may still get executed out
of order. The issue is that the dependencies are set to set() for all tasks
involved. This patch adds back in explict dependencies within these chains
to avoid the setscene task failures.
[YOCTO #6069]
(Bitbake rev: 724c889eed3b03d3199810c185086d3973af826c)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
This allows the commandline options to be processed in the dump signature
code.
(Bitbake rev: ef8537a2e9b48f4fe065a165c102935aee2c9029)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
There is no easy way to make this change. We really need parameters for the -S
(dump signatures) handling code. Such a parameter can then be used within the
codebase to handle the signatures in different ways.
For now, "none" is the recommended default and "printdiff" will execute the
new (and more expensive) comparison algorithms.
(Bitbake rev: b9873588696507dfb6aade6821f6f75cb9a19e0a)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The runqueue should be using the "realtask" ID to lookup the task
hash, not the "task" ID. This patch resolves corruption issues where
incorrect task hashes were displayed within toaster.
(Bitbake rev: 84be1a27f89d1bf63c21f06d831df0a66a5db860)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Use of waitpid on the worker processes is a bad idea since it conflicts
directly with subprocess internals. Instead use the poll() method
and returncode to determine if the process has exitted, if it has,
we can shut down the system.
This should resolve the hangs once and for all, famous last words.
(Bitbake rev: 60969cd62e21e7d4af161bf8504b7643a879c73f)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
We're running into processes using 100% cpu. It appears theses are locked in
a subprocess.poll() type loop where the process has exited but the code is
looping as its not handling the ECHILD error.
http://bugs.python.org/issue14396http://bugs.python.org/issue15756
This is likely due to one or both of the above bugs. The question is what actually
grabbed the child exit code as it wasn't this code. Its likely there is therefore
some other code racing and taking that code, it may be some kind of race like:
http://hg.python.org/cpython/rev/767420808a62/
where the fix effectively catches the childs codes in a different part of the system.
We could try and get everyone onto python 2.7.4 where the above bugs are fixed however
for now its safer to admit defeat and go back to polling explictly for our worker exit
codes.
(Bitbake rev: 5b9a099ec2a1dc954b614e12a306595f55b6a99e)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Catching all child exit status values is a bad idea. Setting an http sstate mirror
is a great way to view that spectacularly break things. The previous change did
have good code changes so don't revert those parts.
(Bitbake rev: fa7ffb62d510ac1124ae7e08fa4d190a710f5b54)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
There are several problems. Firstly, a return value of "None" can mean
there is a C signal handler installed so we need to better handle that
case. signal.SIG_DFL is 0 which equates to false so we also need to
handle that by testing explicitly for None.
Finally, the signal handler *must* call waitpid on all child processes
else it will just get called repeatedly, leading to the hanging behaviour
we've been seeing. The solution is to only error for the worker children,
we warn about any other stray children which we'll have to figure out the
sources of in due course.
Hopefully this patch gets things working again properly though.
(Bitbake rev: 973876c706f08735c1b68c791a5a137e5f083dd2)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Failures on the autobuilder look like this handler is recursing. That
shouldn't be possible but it doesn't hurt to code as such.
(Bitbake rev: e39e85803cbe1ef9413a118868c19087c0546d01)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
We've noticed hanging processes which appear to be looping around
waitpid. Its possible multiple calls to teardown are causing problem
or in theory multiple registrations (although the code should not
allow that). Regardless, put better guards around signal handler
registration.
(Bitbake rev: 79acfb0853aa3215215cee89a945f8e97b0a8fae)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
If we didn't setup any workers (such as bitbake -S), this would error
since we're trying to set a signal handler to None. This patch
avoids that problem.
(Bitbake rev: ce17478c8197abf178c00774f5bbe23fd4375ee2)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The sigchld handler was reaping any processes and this was leading to
confusion with any other process handling code that could be active.
This patch:
a) Ensures we only read any process results for the worker processes
we want to monitor
b) Ensures we pass the event to any other sigchld handler if
it isn't an event we're interested in so the functions are properly
chained.
Together this should resolve some of the reports of unknown processes
people have been reporting.
(Bitbake rev: fe8baaa2f533db7a1b7203476c675588923d8d45)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Instead of a significant number of calls to waitpid, register a SIGCHLD
handler instead.
(Bitbake rev: 76029d08ad56a0a264ff9738a0336971a455b7f5)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
A run of "bitbake bash -c unpack" when the task has already been
completed resulted in about 9000 calls to logger.debug(). With this
patch which comments out some noisy/less usefull logging and moves
other logging calls outside loops, this number is reduced to 1000
calls. This results in cleaner logs and gives a small but
measurable 0.15s speedup. The log size dropped from 900kb to 160kb.
(Bitbake rev: d2677f084fe1d8846db77d89ef5e6ffb18dc171a)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
If the worker has already gone missing (e.g. SIGTERM), we should
gracefully handle the write failures at exit time rather than throwing
ugly tracebacks.
(Bitbake rev: 1b1672e1ceff17fc4ff8eb7aa46f59fce3593873)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
If the worker (or fakeworker) process disappears for some reason, the
system doesn't currently even notice. To fix this, we call waitpid
periodically, looking for exit events of our children. If these
occur, we can gracefully shutdown the server.
(Bitbake rev: ee28ddadaa7ef91e4d4b7d22fc267382aaad6d01)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
slef.self is clearly meant to be self, fix typo.
Otavio spotted and reported, thanks.
(Bitbake rev: 316daad7928a58cdfc42e27b20e739f4dd74a02a)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* exception like this keeps spinning quite quickly generating GBs of logs
better to kill it asap and show invalid pickle
(Bitbake rev: a69eb4c12c71bba9d742c4e5578f25c388d9f825)
Signed-off-by: Martin Jansa <Martin.Jansa@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
A previous commit of mine used the target variable for two different uses
resulting in a lot more sstate being installed than is needed.
Fix the variable to use two different names and unbreak the setscene
behaviour.
(Bitbake rev: f975ca2cf728561bd6317ed8f76303598546113a)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
If you specify multiple targets on bitbake's commandline and some of them are
setscene tasks which are "masked" by other tasks they may not get run.
For example <image>:do_rootfs <kernel>:do_populate_sysroot
the rootfs tasks "masks" the populate_sysroot task so bitbake would currently
decide not to run it. In this case, we do really want it to be run.
The fix is not to skip anything which has been given as an explict target.
(Bitbake rev: 0753899d1e855795cc18671357609a86f169b379)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Commit c54e738e2b5dc0d8e6fd8e93b284ed96e7a83051 added in the idea of hard dependencies
such as the case a setscene has a hard dependency on pseudo-native and that
dependency wasn't available from sstate for some reason.
Unfortunately the implementation was a bit too enthusiastic, causing rebuilds
of things when it wasn't necessary. A test case was:
bitbake quilt-native
bitbake quilt-native -c clean
bitbake <some-image>
and then you'd watch quilt-native get rebuilt for no good reason.
The clue to the problem is in the for loop where it never depends on
the item being iterated over.
The fix is to include the exact list of hard dependencies rather than
guessing. With these changes, the use case above works, the one in
the original commit also works.
This patch also adds in or cleans up various pieces of logging to
allow issues like this to be more easily debugged in future.
(Bitbake rev: 81bd475585ff1b44b390036b1eca0feae7c149eb)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Shared work directories work by assuming bitbake will not run
more than one task with a specific stamp name. Recent runqueue optimisations
accidentally broke this meaning there could be races. This fixes the code.
(Bitbake rev: b1628b1a260ddf43fc9985535b1ddcfcebbb1e5b)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Based upon the list of difference starting points, we can use the siggen.find_siginfo()
function call and the difference printing code to provide a list of differences
between the current build target and whatever can be obtained from the sstate cache.
(Bitbake rev: 7a77861feb62750ef166d2d1e89ed1f444ca8dc7)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The way hash_deps was being generated was different to the way siggen generated
the data internally which lead to seemingly different sigdata/siginfo files
for the same checksum. The -S output correct but the files written during
builds contained superflous data which would look like a difference.
This patch removes the badly duplicated data and uses it from the source
which ensures its consistent.
(Bitbake rev: e6d5e925c402cd2cc7ee034e9de4cc6df8944a34)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Its useful to understand where the delta starts against an existing sstate cache
for a given target. Adding this to the output of the -S option seems like a
natural fit.
We use the hashvalidate function to figure this out and assume it can find siginfo
files for more than just the setscene tasks.
(Bitbake rev: c18b8450640ebfd55a2b35b112959f9ea3e0a700)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Currently tasks have no knowledge of which other tasks they depend
upon. This makes it impossible to do at least two things which would be
desirable/interesting:
a) Have the ability to create per recipe sysroots
b) Allow the aclocal files to be present only for the entries in
DEPENDS (directly and indirectly)
By exporting task data through this new variable, tasks can inspect
their dependencies and then take actions based upon this.
(Bitbake rev: 84f1dde717dac22435005b79d03ee0b80a3e8e62)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
This unlikely looking function was found to be eating a lot of CPU time
since it gets called once per trip through the idle loop if we're not
running a maximum number of processes. This was particularly true in
world builds of 13,000 tasks.
Calling the computation code is pretty pointless because until some
other task finishes nothing is going to become available to build.
We can know when things become available so this patch teaches the
scheduler this knowledge.
It also:
* skips any coputation when nothing can be built
* if there is only one available item to build, ignore the priority map
* precomputes the stamp filenames, rather than doing it every time
* saves the length of the array rather than calculating it each time
(the extra function overhead is significant)
Timing wise, initially, 5000 iterations through here was 20s, with
the patch 200000 calls takes the same time. The end result is that
builds get up and running faster.
(Bitbake rev: 4841c1d37c503a366f99e3a134dca7440e3a08ea)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
When using the dry run option (-n), bitbake would still try and fire
a specific fakeroot worker. This is doomed to failure since it might
well not have been built.
Add in some checks to prevent the failures.
[YOCTO #5367]
(Bitbake rev: f34d0606f87ce9dacadeb78bac35879b74f10559)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
We have do_bundle_initramfs which is a task inserted after compile and
before build. It is not covered by sstate.
If we run a build with a valid sstate cache present, the setsceneverify
function realises it will rerun the do_compile step (due to the
bundle_initramfs task) and hence marks do_populate_sysroot to rerun.
do_install, a dependency of do_populate_sysroot is left as marked as
covered by sstate.
What we need to do is traverse the dependency tree for any setsceneverify
invalided task and ensure any dependencies are also invalidated. We can
stop at any point we reach another setscene task though.
This means the do_populate_sysroot task has the data from do_install
available and doesn't crash.
(Bitbake rev: f21910157d873c030b149c4cdc5b57c5062ab5a6)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Adding the sstate-related hash for all runqueue and
scenequeue tasks, as it's needed in the WebHob data.
(Bitbake rev: b6e2ce1cf7a0ede890f08fabf536a556dc4263c5)
Signed-off-by: Alexandru DAMIAN <alexandru.damian@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
This patch adds task identifying information for all
runQueue and sceneQueue events, and for bb.build.Task* events.
This will allow matching event to specific tasks in the UI
handlers processing these events.
Adds RunQueueData functions to get the task name and task
file for usage with the runQueue* events.
Adds taskfile and taskname properties to bb.build.TaskBase.
Adds taskfile and taskname properties to the *runQueue* events
(Bitbake rev: b4a5e4be50d871a80dbe0993117d73f5ad82e38f)
Signed-off-by: Alexandru DAMIAN <alexandru.damian@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Adding a CookerFeature that allows UIs to enable
receving a dependency tree once the task data has been
computed and the runQueue is ready to start.
This will allow the clients to display dependency
data in an efficient manner, and not recompute the runqueue
specifically to get the dependency data.
(Bitbake rev: 75466a53b6eece5173a9bfe483414148e4c06517)
Signed-off-by: Alexandru DAMIAN <alexandru.damian@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Adding a runQueueTaskSkipped to notify that the tasks that are not
run either because they are set-scened or they don't need an update
(timestamp was ok).
(Bitbake rev: cf4a0c7aa82090876ae652b611acfab3ce2688f7)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Adding an event to be fired when a scene task is completed.
It is analogous to the run task completed event, and has
been missing for some reason.
(Bitbake rev: 73b8f4d3fbeaf1b330a66d76012d0a5cef8dbe2d)
Signed-off-by: Alexandru DAMIAN <alexandru.damian@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
There would be an race issue if we:
$ bitbake make-3.81 make-3.82
This because they are being built at the same time which would cause
unexpected problems, for example:
[snip]
ERROR: Package already staged (/path/to/tmp/sstate-control/manifest-qemux86-make.populate-sysroot)?!
ERROR: Function failed: sstate_task_postfunc
[snip]
Or there would be python's strack trace such as:
[snip]
*** 0004: mfile = open(manifest)
0005: entries = mfile.readlines()
0006: mfile.close()
0007:
0008: for entry in entries:
Exception: IOError: [Errno 2] No such file or directory: xxx
[snip]
[YOCTO #5094]
We can quit earlier to avoid this kind of issue when two versions of the same PN
are going to be built since this isn't supported.
(Bitbake rev: ab377c00c33a2d296bfda1b0b6c2a62b29d1004f)
Signed-off-by: Robert Yang <liezhi.yang@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Due to the worker split the ${DATE} and ${TIME} variables could end up
with different values for different workers.
E.g., a task like do_rootfs that is run within a fakeroot environment
had a slightly different view of the time than another task that was not
fakerooted which made it impossible to correctly refer to the image
generated by do_rootfs from the other task.
(Bitbake rev: 756cc69ebf8bfe8455d0c90f288dd51be2499773)
Signed-off-by: Peter Kjellerstedt <peter.kjellerstedt@axis.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
BUILDNAME is set from cooker by default, so since the worker split it
will not be set when executing functions. In OpenEmbedded this results
in /etc/version (which is populated from BUILDNAME) not having any
content. Pass this variable value through to the worker explicitly to
fix the issue.
Fixes [YOCTO #4818].
(Bitbake rev: 92940b0427d9b2b3f95e27c230ec1e36638a34bc)
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
This was missed off in a previous patch.
(Bitbake rev: ad7664edd40fa46e6f6fec2144403e3b6fc3a639)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The existing backend server implementations were inefficient since they
were sleeping for the full length of the timeouts rather than being woken when
there was data ready for them. It was assumed they would wake and perhaps did
when we forked processes directory but that is no longer the case.
This updates both the process and xmlrpc backends to wait using select(). This
does mean we need to pass the file descriptors to wait on from the internals
who know which these file descriptors are but this is a logical improvement.
Tests of a pathaolgical load on the process server of ~420 rapid tasks
executed on a server with BB_NUMBER_THREAD=48 went from a wall clock
measurement of the overall command execution time of 75s to a much more
reasonable 24s.
(Bitbake rev: 9bee497960889d9baa0a4284d79a384b18a8e826)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Help to pick up mistakes such as "bitbake -c cleanstate xyz" (instead
of "bitbake -c cleansstate xyz".)
(Bitbake rev: 15c3db1cffdffd85641c6b12e77f19ce7a553472)
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Add a warning if 'bitbake -C' is executed with a task that does not
exist.
Fixes [YOCTO #4877]
(Bitbake rev: 6459c1d0eb8f1007246df36149e2496ee531e25f)
Signed-off-by: Petter Mabäcker <petter@technux.se>
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Currently, tasks like fetchall are slightly broken since if a recipe
has specific [depends] which occur after do_fetch and add items not listed
in DEPENDS and RDEPENDS, they are not caught by recrdeptask. We've gone
around in circles on this issue (e.g
http://git.yoctoproject.org/cgit.cgi/poky/commit/bitbake/lib/bb/runqueue.py?id=5fa6036d49ed7befe6ad50ec95c61a50aec48195
) and in many cases the behaviour of recrdepends is correct but tasks like
fetchall need the other behaviour.
To address this we add a recideptask flag which can be used in conjuction
with the recrdeptask flag to specify which task to to the inspection upon.
This means entries like do_rootfs[depends] which have do_fetch tasks are
caught and run.
I'm not 100% happy with needing another flag but I don't see any rational
way to get the correct behaviour in all cases without it.
[YOCTO #4597]
(Bitbake rev: f8c9b292b02ce2c28741b74901205f5e5807ca87)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
If a setscene task has a hard dependency on a task like pseudo-native, its
expected that the setscene task will not run unless the dependency is met.
This adds code to ensure that is the case, otherwise a bug would show up
with a usecase like:
bitbake gnome-common
bitbake pseudo-native -c cleansstate
bitbake gnome-common -c clean
bitbake gnome-common
With the double wrapper script environment, we'd not see issues like
this as it would be masked. The problem theoretically affects code like
useradd too as well as anything using a sstate postinstall.
(Bitbake rev: c54e738e2b5dc0d8e6fd8e93b284ed96e7a83051)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
With the change to bitbake-worker we need to ensure the workers know
how to contact the PR service, the magic 0 port and singleton is
no longer enough.
(Bitbake rev: c761751e259bb8e940552a28794b45887b5a72d9)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
We're going to need a fakeroot/pseudo version of the worker so
abstract the code to start the worker process.
(Bitbake rev: b5d0f12f9df3ab211700473ed145ee6fbd9ca8e9)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The worker was being executed by each execution queue so would get
constructed twice for each build. This is wasteful so move execution
to the main runqueue so we only have to start the worker once.
(Bitbake rev: 8117f8480125b121b2b5ac0afc31b108d9e670ae)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
This is a pretty fundamental change to the way bitbake operates. It
splits out the task execution part of runqueue into a completely
separately exec'd process called bitbake-worker.
This means that the separate process has to build its own datastore and
that configuration needs to be passed from the cooker over to the
bitbake worker process.
Known issues:
* Hob is broken with this patch since it writes to the configuration
and that configuration isn't preserved in bitbake-worker.
* We create a worker for setscene, then a new worker for the main task
execution. This is wasteful but shouldn't be hard to fix.
* We probably send too much data over to bitbake-worker, need to
see if we can streamline it.
These are issues which will be followed up in subsequent patches.
This patch sets the groundwork for the removal of the double bitbake
execution for psuedo which will be in a follow on patch.
(Bitbake rev: b2e26f1db28d74f2dd9df8ab4ed3b472503b9a5c)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The reasons for combining these objects is ancient history, it makes
sense to clean things up and separate them out now. This follows on
logically from the configuration cleansups and leads well into the
bitbake-worker changes.
(Bitbake rev: 89ffd62661ebcf2a97ce0c8dfd5e4d5bfbe27de7)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
This means the variables show up in the shell execution "run" files since
its useful to know what the fakeroot environment is and how to set it up
manually.
(Bitbake rev: bdf437747b664479acde6deaa9096e2a6bcdf483)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The Cooker class is too large and needs to be split up into different
functional units. Splitting out the collections code into its own class
seems like a good place to start to try and disentangle things.
(Bitbake rev: ca1fcbb6e214c155a05328779d3d326e10c5eac0)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* dep variable was removed in
commit 3190cb83e2af195a464f669c5aa8aedbf795160e
Author: Richard Purdie <richard.purdie@linuxfoundation.org>
Date: Wed Jun 27 11:04:06 2012 +0100
taskdata: Add gettask_id_fromfnid helper function
This is like gettask_id but doesn't require translation of fnid -> fn
first which the function then translates back. This gives a sizeable
performance improvement since a significant number of lookups are avoided.
* now it fails completely instead of showing which task is missing
(Bitbake rev: 58847fabd389e5b8d02d5a9c6827aabedb30312f)
Signed-off-by: Martin Jansa <Martin.Jansa@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
When the setscene code was originally written it was thought that we'd
allow "partial" coverage. For example, if we just want to build the target
"bash:do_populate_sysroot" and its available from sstate, it makes no sense
to install gcc-cross's sstate package as its simply not needed.
Due to various other issues in the codebase, this functionality was
disabled/removed to allow the setscene code and sstate to stabilise and allow
us to concentrate on other problems.
The time has now come to enable "partial" coverage. There are two major changes
in this patch:
a) Creation of an unskippable list. This lists direct dependencies of
build targets and hence things that cannot be skipped.
b) Addition of a handler which looks at a given setscene target and what depends
on it and then decides whether its necessary.
(Bitbake rev: 2a937cd6a6c3110030b40bc4d85e349b85cb4db7)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Mainly intended for the purpose of debugging or forcing builds
from source, the --no-setscene will prevent any setscene
tasks from running.
(Bitbake rev: 440e479f3e248482c38c149643403c6907ac7034)
Signed-off-by: Jason Wessel <jason.wessel@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
If a target dependency is marked as failed and yet we are continuing on
because -k has been specified, don't try to access the dependency's data
in taskData.build_targets since it will have been removed. This fixes
"IndexError: list index out of range" errors in this situation.
Also, do not print the "unhandled exception" message when SystemExit is
raised since we will have reported the actual error already in this
case (e.g. when -k has been specified and some targets failed).
Fixes [YOCTO #3133].
(Bitbake rev: 70eebc184eb1ab3678be87bed019b5beadecdc89)
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The stamp files can change during setscene and the cache should be cleared to
account for this.
(Bitbake rev: 5ec12f586a50fce675b268965b3dc487aaa96c43)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
There was some odd behaviour if some task was run from setcene whilst there were
existing valid stamps for a depepdency. For example, do_populate_sysroot might
be installed at setscene time but if there were other tasks not installed from
setscene such as do_populate_lic which depend on do_configure, the setsceneverify
function would think that do_configure needed to be rerun and would hence void the
do_populate_sysroot and force that to rerun too.
The setsceneverify function needs to know which tasks are going to be rerun, not just
what the overall task list is and what setscene functions have run. This patch adds
that information and maintains backwards compatibility in a slightly ugly but effective
way. The metadata needs updating to take advantage of this change.
(Bitbake rev: 1423aafff97f17169e95ec3ba973eb002ff98c1c)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The scenequeue code could result in one setscene function masking out another
which had been marked as notcovered. This change ensures the notcovered list
is taken into account when deciding which tasks need to be run, ensuring
that tasks that should be run do get run.
(Bitbake rev: 95b31127d8ba12ccb061d67481ece134c18c39f2)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Currently we call the hashvalidate for anything without a valid setscene stamp.
This improves the code to account for existing stamp files so that we only process
setscene hashes for things we might actually end up using. This avoids hash processing
and makes things slightly more efficient and the logs less confusing.
(Bitbake rev: 2e71a65bd910e8be894b8284eb0a50de7270c081)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
There was a bug where an exception in the runqueue code would cause an infinite loop
of debug messages. The exception would get reported but would loop since runqueue was
still registered as an idle handler.
This patch adds an exception handler to ensure in the case of errors, the system
more gracefully shuts down and doesn't loop.
(Bitbake rev: 99467c7e387f6e71358b40b8afae450d72cd68e0)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
We check the stamp cache before comparing t2 and t3 which means that we can miss
a level in the stamp file chains. The result of this is that a stamp can be accepted
as valid when in fact it isn't. Some weird behaviour alerted me to this in a local
build.
This patch also fixes to only uses the cache in recurse mode, there was a corner
case where stamps not in recurse mode could get added to the cache which could cause
an issue potentially.
(Bitbake rev: 7a5f776d325c0e37f58003424beda7cae2695325)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
In some cases we want to pull in DEPENDS and RDEPENDS of recrdeptask
dependencies but we need a way to trigger or avoid this behaviour
depending on context. The logical syntax to trigger such behaviour
would be a self referencing recrdeptask:
do_a[recrdeptask] = "do_a do_b"
The dependency chains already recurse this kind of expression correctly, the
missing piece is to avoid any circular reference errors. Since the dependencies
have already been recursively resolved, simply removing any recrdeptask
references is enough to break the circular references.
This patch therefore removes any circular references using the set
difference_update() operator. There will be metadata tweaks required to
add any references needed to the extra taskname.
(Bitbake rev: a5324da9b8a0c9307a6c511ea9009f34be70c92b)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
I'm not sure why we don't currently allow multiple entries in rdeptask when
we do in deptask. This makes the handling match between the two since
its trivial to fix.
(Bitbake rev: 19c84fe8854639768c874cc1449963a9867ad397)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Currently, recrdepends is extremely greedy. For example:
do_foo[rdepends] = "somedep:sometask"
addtask foo
which adds foo with *no* dependencies, will suddenly start appearing
as a dependency in every task which uses recrdepends. So far this has
been mildy annoying but we now have use cases where this makes no sense
at all.
This reworks the recrdepends code to avoid this problem. To do this we
can no longer collapse things into lists just based on file ID. The problem
is this code is extremely performance sensitive. The "preparing runqueue"
phase spends a lot of time in these recursive dependency calculations so any
change here could negatively impact the user experience.
As such, this code has been carefully tested on convoluted dependency trees
with operations like "time bitbake world -g". The net result of this change
and the preceeding changes combined is a net speed up of these operations in
all cases measured.
Tests were made comparing "bitbake world -g" task-depends.dot before and after
this patch. There *are* differences for example -nativesdk do_build dependencies
on -native recipes are no longer present. All removed dependencies appear to
be sensible improvements to the system. The "rdepends" cross contamination
issue above is also fixed.
(Bitbake rev: 82d73423c57569b984ee0ae3d93e3c3bd5dc5216)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
This gives some small performance gains and sets the scene for other
improvements by removing the need for duplicate detection code.
(Bitbake rev: 6fd723479e8d49227fd58040b3485c1d5afc4bc5)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
This is like gettask_id but doesn't require translation of fnid -> fn
first which the function then translates back. This gives a sizeable
performance improvement since a significant number of lookups are avoided.
(Bitbake rev: 3190cb83e2af195a464f669c5aa8aedbf795160e)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Currently its not possible to add arbitrary RDEPENDS to a specific task.
This can be useful and this patch adds functionality equivalent to the
'depends' task flag.
(Bitbake rev: db65080a6199baecc5c422294a4c4a9fd12dc29e)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
This new command line option forces the specified task and all dependent
tasks up to the default task to re-run. This means that the following
single step:
bitbake -C compile somerecipe
is equivalent to the following two steps (with the recent change to -f):
bitbake -c compile -f somerecipe
bitbake somerecipe
Note that to work this option needs full hashing enabled (i.e.
BB_SIGNATURE_HANDLER must be set to a signature handler that inherits
from BasicHash). If this is not the case, -C effectively does nothing.
Based on a previous implementation of this option by Jason Wessel
<jason.wessel@windriver.com>.
Implements [YOCTO #2615].
(Bitbake rev: 2530e0faada5775897cfd1b93aba6925826dca73)
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
If -f is specified, force dependent tasks to be re-run next time. This
works by changing the force behaviour so that instead of deleting the
task's stamp, we write a "taint" file into the stamps directory, which
will alter the taskhash randomly and thus trigger the task to re-run
next time we evaluate whether or not that should be done as well as
influencing the taskhashes of any dependent tasks so that they are
similarly re-triggered. As a bonus because we write this file as
<stamp file name>.taskname.taint, the existing code which deletes the
stamp files in OE's do_clean will already handle removing it.
This means you can now do the following:
bitbake somepackage
[ change the source code in the package's WORKDIR ]
bitbake -c compile -f somepackage
bitbake somepackage
and the result will be that all of the tasks that depend on do_compile
(do_install, do_package, etc.) will be re-run in the last step.
Note that to operate in the manner described above you need full hashing
enabled (i.e. BB_SIGNATURE_HANDLER must be set to a signature handler
that inherits from BasicHash). If this is not the case, -f will just
delete the stamp for the specified task as it did before.
This fix is required for [YOCTO #2615] and [YOCTO #2256].
(Bitbake rev: f7b55a94226f9acd985f87946e26d01bd86a35bb)
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
This function was used by old code such as packaged staging but is thankfully
obsolete now and replaced with better mechanisms. Its time to remove it and
the horrible internal only variables associated with it.
(Bitbake rev: 2995b8d551e0532eca20f8862730acd062c608ad)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
This was originally used to check the consistency of the stamps in one function
call. This turns out to be inefficient, unnecessary and if it were necessary,
check_stamp_task() could be called in a loop. The function has been unmodified
for a while and likely contains bugs. Its best simply removed.
(Bitbake rev: 728ffde1bd69b880d48fe8523b1616956d13616e)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
This should fix issues where bitbake would seemingly lock up when checking
certain configurations of stampfiles.
The cache is kept within the runqueue since that feels like the right
place to associate this cache data.
(Bitbake rev: e95755c4931b26d9f8102ed3652dff969145cfc9)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
- for a normal exit, use WEXITSTATUS, rather than manually shifting
- for exit via signal, set the exit code to 128+N, per shell convention
- if a process was stopped, return and don't handle it, as the process can yet
be continued
This should fix the case where bitbake says a task failed with an exit code of
0 (we assumed failure based on the overall status, but didn't pass all the
information along to task_fail).
(Bitbake rev: 84ea614bc56d35a414eb5bf5658891b340bfc569)
Signed-off-by: Christopher Larson <chris_larson@mentor.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
When "Force stop" is performed during the build stage, after os.kill() kills the build sub-processes, there are many <defunct> python processes in the system. In Hob, when the user initiates a new build, os.waitpid() in runqueue_process_waitpid() will be called, and the pids of those <defunct> processes will be returned as result[0], then self.build_pids[result[0]] will throw KeyError exception because now for the new build self.build_pids is empty.
This patch is to address the above issue to collect the results and handle the sub-processes as soon as they are killed.
[Yocto #2186]
(Bitbake rev: e9f4ca467e795bbc520d12b0e7a5985b6ff0a20e)
Signed-off-by: Shane Wang <shane.wang@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
(Bitbake rev: c4c8df0e1e77f99f3ee0b07e1748116fc362819f)
Signed-off-by: Robert P. J. Day <rpjday@crashcourse.ca>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Currently the code checking whether stamps are valid only traverses one step
of the dependency graph. This works fine in the normal cases where we've already
validated dependencies but for the setscene code, it doesn't work well. A typical
problem usecase is something like:
bitbake gcc-cross -c unpack -f
bitbake gcc-cross
which will ignore any sstate files already cached which could be used to speed
up the gcc-cross build. This becomes particularly problematic with multiple gcc
builds where only one should rebuild yet they all do.
This patch teaches the stamp code to be able to recurse within a given fn
which gives the behaviour people would expect from the code and allows
bitbake to make better use of prebuild sstate objects.
(Bitbake rev: e68814cb2e8da523d4ddf05e8ceddfaa19244851)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
