Commit Graph

366 Commits

Author SHA1 Message Date
Joshua Watt
4fd9ee4670 bitbake: bitbake: Add --skip-setscene option
Adds an option to skip _setscene only if they would normally be
executed, without ignoring sstate completely.

Previously, '--no-setscene' would allow a build that completely ignored
sstate and _setscene tasks, and '--setscene-only' would allow a build
that only ran _setscene tasks, but there was no option do a build that
would respect tasks previously restored from sstate and build everything
else. Now one can run:

 bitbake --setscene-only IMAGE; bitbake --skip-setscene IMAGE

which is functionally equivalent to:

 bitbake IMAGE

The indented use is to allow a build to complete successfully in the
presence of _setscene task failures by splitting apart the two phases
e.g.:

 (bitbake -k --setscene-only IMAGE || true) && bitbake --skip-setscene IMAGE

(Bitbake rev: 813ba5b7c13b573a0b813b628a819bdbf0627540)

Signed-off-by: Joshua Watt <JPEWhacker@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-07-03 17:00:57 +01:00
Robert Yang
5161799993 bitbake: bitbake: lib: Cleanup /usr/bin/env python
(Bitbake rev: cc712f3257904960247a7532cfc4611f3dccd36c)

Signed-off-by: Robert Yang <liezhi.yang@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-06-28 13:29:04 +01:00
Richard Purdie
1f68b0bd98 bitbake: multiconfig: Switch from 'multiconfig' -> 'mc'
After real world use its clear the "multiconfig:" prefix to multiconfig tasks,
whilst clear, is also clumbersome. Switch to use the short version instead.

mcdepends will continue to work with "multiconfig:" for now as well. The commandline
will only accept mc: going forward.

[YOCTO #11168]

(Bitbake rev: 821daf093b76504067a8b77dfa4b181af6ec92b4)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-06-10 14:46:38 +01:00
Richard Purdie
9501864db8 bitbake: bitbake: Strip old editor directives from file headers
There are much better ways to handle this and most editors shouldn't need this
in modern times, drop the noise from the files. Its not consitently applied
anyway.

(Bitbake rev: 5e43070e3087d09aea2f459b033d035c5ef747d0)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-05-04 10:44:10 +01:00
Richard Purdie
cf9c0be3f6 bitbake: bitbake: Drop duplicate license boilerplace text
With the introduction of SPDX-License-Identifier headers, we don't need a ton
of header boilerplate in every file. Simplify the files and rely on the top
level for the full licence text.

(Bitbake rev: 695d84397b68cc003186e22f395caa378b06bc75)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-05-04 10:44:10 +01:00
Richard Purdie
79834a7144 bitbake: bitbake: Add initial pass of SPDX license headers to source code
This adds the SPDX-License-Identifier license headers to the majority of
our source files to make it clearer exactly which license files are under.

The bulk of the files are under GPL v2.0 with one found to be under V2.0
or later, some under MIT and some have dual license. There are some files
which are potentially harder to classify where we've imported upstream code
and those can be handled specifically in later commits.

The COPYING file is replaced with LICENSE.X files which contain the full
license texts.

(Bitbake rev: ff237c33337f4da2ca06c3a2c49699bc26608a6b)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-05-04 10:44:04 +01:00
Robert Yang
5c5a0938de bitbake: bitbake-diffsigs: Use 4 spaces as indent for recursecb
It used 2 spaces as indent which wasn't clear enough, and might cause
confusions, people might think it was in wrong format.

Fixed:
$ bitbake bc-native -ccleansstate -Snone
$ bitbake bc-native -ccleansstate -Snone
$ bitbake-diffsigs tmp/stamps/x86_64-linux/bc-native/1.07.1-r0.do_cleansstate.sigdata.*

* Before:
Hash for dependent task bc/bc_1.07.1.bb.do_clean:virtual:native changed from [foo]
  Taint (by forced/invalidated task) changed from [foo]
Taint (by forced/invalidated task) changed from [foo]

* Now
Hash for dependent task bc/bc_1.07.1.bb.do_clean:virtual:native changed from [foo]
    Taint (by forced/invalidated task) changed from [foo]
Taint (by forced/invalidated task) changed from [foo]

(Bitbake rev: 5127a8d8e6d53f5f43a6ada7fd09b6b0c24ae989)

Signed-off-by: Robert Yang <liezhi.yang@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-04-23 23:30:45 +01:00
Richard Purdie
0b19e52b85 bitbake: runqueue: Filter out multiconfig dependencies from BB_TASKDEPDATA
The consumers of BB_TASKDEPDATA in OE metadata can't cope with multiconfig
dependencies. The choice is either to start adding code to each of them to
filter out multiconfig dependencies, or do this at source.

After consideration we've decided to do this at source as doing otherwise
is code duplication and error prone and in any case we've looked at, they
don't make sense.

[YOCTO #13090]
[YOCTO #13130]

(Bitbake rev: 531dcd221a10853f45cc057b52bb2d5083e0ee42)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-02-25 10:43:31 +00:00
Richard Purdie
8dac3ce424 bitbake: runqueue: Fix dependency loop analysis 'hangs'
Currently the mechanism for breaking out of the dependnecy loop analysis
code is broken and doesn't work leading to bitbake appearing to hang.

Add in a custom exception for this purpose and fix the code to exit
as intended, fixing the hang and making the dependency loop code
usable again.

(Bitbake rev: 8756e4ade67c16e35269ea0659e10b9ebaa6117f)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-02-17 10:20:21 +00:00
Joshua Watt
9025bd11ec bitbake: bitbake: Allow arguments in FAKEROOTCMD
Changes FAKEROOTCMD so that it can accept additional arguments to pass
to the fakeroot implementation instead of being treated as a simple
command

(Bitbake rev: 4fa51afb56b090cf1f746842acd602c9536715d5)

Signed-off-by: Joshua Watt <JPEWhacker@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-02-15 08:18:19 +00:00
Robert Yang
34ce32a75b bitbake: bitbake: runqueue: __find_md5__ -> __find_sha256
Fixed:
Create a new build
$ bitbake quilt-native -ccleansstate -Snone
$ bitbake quilt-native -ccleansstate -Sprintdiff

[snip]
     latestmatch = sorted(matches.keys(), key=lambda f: matches[f])[-1]
>    prevh = __find_md5__.search(latestmatch).group(0)
     output = bb.siggen.compare_sigfiles(latestmatch, match, recursecb)
AttributeError: 'NoneType' object has no attribute 'group'

(Bitbake rev: 15d20d948359fa1d7a7a754b2a1d8ed9f4ca0480)

Signed-off-by: Robert Yang <liezhi.yang@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-01-14 11:36:49 +00:00
Joshua Watt
46f6f6a5f9 bitbake: bitbake: runqueue: Use multiconfig name to fetch unihash
The unihash should be fetched using the task filename that includes the
multiconfig prefixes.

[YOCTO #13124]

(Bitbake rev: 5e7f4e77e27bceaf6c68137cacb4f8d7d7de49dd)

Signed-off-by: Joshua Watt <JPEWhacker@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-01-11 10:39:30 +00:00
Joshua Watt
445acdffe0 bitbake: runqueue: Pass unique hash to hash validate
If the unique hash is being used to track task dependencies, the hash
validation function needs to know about it in order to properly validate
the hash.

[YOCTO #13030]

(Bitbake rev: 9a529bb2658a4046dafbf32e1eb503d84e64e947)

Signed-off-by: Joshua Watt <JPEWhacker@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-01-03 22:47:11 +00:00
Joshua Watt
3b5bdb227b bitbake: runqueue: Pass unique hash to task
The unique hash is now passed to the task in the BB_UNIHASH variable

[YOCTO #13030]

(Bitbake rev: aab80b099f6f259e4b57cba2c26dd385d07c5947)

Signed-off-by: Joshua Watt <JPEWhacker@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-01-03 22:47:11 +00:00
Joshua Watt
57e3c55f6d bitbake: runqueue: Track task unique hash
Requests the task unique hash from siggen and tracks it

[YOCTO #13030]

(Bitbake rev: 1ecc47f0831b35c8c92b37a81cef4e43ff9f67b2)

Signed-off-by: Joshua Watt <JPEWhacker@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-01-03 22:47:11 +00:00
Joshua Watt
86cc1a4104 bitbake: bitbake-worker: Pass taskhash as runtask parameter
Pass the task hash as a parameter to the 'runtask' message instead of
passing the entire dictionary of hashes when the worker is setup. This
is possible less efficient, but prevents the worker taskhashes from
being out of sync with the runqueue in the event that the taskhashes in
the runqueue change.

[YOCTO #13030]

(Bitbake rev: 1e86d8c1bec7ea5d016a5ad2097f999362e29033)

Signed-off-by: Joshua Watt <JPEWhacker@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2018-12-07 12:38:58 +00:00
Richard Purdie
c5ca140d38 bitbake: runqueue: Ensure disk monitor is started when no setscene tasks are run
Currently if there are no setscene tasks, the disk monitor isn't started.

Move the startup code to somewhere to ensure it always is started. This
issue would partially explain occasional selftest failures.

(Bitbake rev: 5ba83ee25c1c9cba349edb68a22476b1d5fca6ce)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2018-12-01 11:32:08 +00:00
Richard Purdie
530aceebc1 bitbake: runqueue: Ensure setscene tasks that aren't covered get built
Running "bitbake gconf-native -c cleansstate; bitbake core-image-sato:do_populate_sdk"

results in a build where it fails to find gconf-native and fails to build it,
merrily trying to build the SDK without gconf being present.

The issue is the missing setscene tasks are effectively ignored as the later
code in runqueue thinks that since other sstate tasks are present, these
'cover' the missing one. In reality we need to call BB_SETSCENE_DEPVALID
to make that decision. To do that we need a "reduced" setscene dependency
graph which we don't have in main task graph context.

Since that was already done in setscene, we should just assume anything
in the non-covered list needs to be built.

(Bitbake rev: 464d0339add15bc8b4344ddd1e4c49706e3c0a02)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2018-09-10 12:13:38 +01:00
Richard Purdie
479a059c96 bitbake: cooker/runqueue: Turn universe warnings into verbnotes
If the user puts universe on the commandline, they don't really want warnings
so use the new verbnote level instead.

(Bitbake rev: 0c87ade5678e503899e3a6cdda5329f6fc212b63)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2018-09-10 12:13:38 +01:00
Andreas Müller
331463011f bitbake: runqueue: Move decision if a task can be started to one common place
(Bitbake rev: 99d9e4389e1f1d78d17a23ee078fe3f4a12cb31d)

Signed-off-by: Andreas Müller <schnitzeltony@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2018-08-15 21:45:58 +01:00
Richard Purdie
dbda297fd9 bitbake: runqueue: Allow the hash validate function to have an idea of build completion
There is an oversight in the current hash validation API in that the
function can't know how many setscene tasks already completed. Rather
than trying to add additional parameters to the function, causing
incompatibilities, store the value in the datastore.

This is useful to allow build status reporting to the user for
figures on sstate reusage and build completion.

(Bitbake rev: ec037d3e49264037b81212f498d98e292ae7c334)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2018-08-08 10:52:00 +01:00
Alejandro Enedino Hernandez Samaniego
b8730f68d2 bitbake: bitbake: Add support for multiconfig dependencies
This patch adds the capability for tasks from different
multiconfigs to depend on one another.

These dependencies can be enabled using the following format:

task[mcdepends] = "multiconfig:FROM-MC:TO-MC:PN:task-to-depend-on"

For the sake of simplicity consider the following example:

Assuming we have set up multiconfig builds, one for qemux86 and one for
qemuarm, named x86 and arm respectively.

Adding the following line to an image recipe (core-image-sato):
do_image[mcdepends] = "multiconfig:x86:arm:core-image-minimal:do_rootfs"

Would state that core-image-sato:do_image from x86 will depend on
core-image-minimal:do_rootfs from arm so it can be executed.

This patch makes modifications to:
- cooker: To glue both multiconfigs in one place and make sure
  the dependencies can be provided.
- taskdata: To parse and add a new kind of dependency (mcdepends) to
  the taskdata object.
- runqueue: To differentiate tasks from different multiconfigs,
  add the specified dependencies to the corresponding tasks, and
  create a working runqueue that contains tasks from both multiconfigs.
- siggen: To avoid looking for tasks from different multiconfigs on
  objects where they dont belong.

The taskdata objects are still not aware of the concept of multiconfig,
so each object doesnt know which multiconfig its building, hence why
the mcdepends are added to all taskdata objects equally (we really
dont expect many of these), but the actual dependencies are added only
to the required tasks by the runqueue.

(Bitbake rev: da8cb8633504bdc815bdcefc538340b9bce5065d)

Signed-off-by: Alejandro Enedino Hernandez Samaniego <alejandr@xilinx.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2018-08-01 10:07:22 +01:00
Joshua Watt
9b034e9a83 bitbake: runqueue: Stop on first unsatisfied dependency
(Bitbake rev: 9876b5da1c65bf09a790542cb4057f2d93868cf7)

Signed-off-by: Joshua Watt <JPEWhacker@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2018-07-09 11:26:56 +01:00
Joshua Watt
53d1027625 bitbake: runqueue: Remove unused variable
(Bitbake rev: 91e5540f53aca93e3489255cfd95feaa0afd0498)

Signed-off-by: Joshua Watt <JPEWhacker@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2018-07-09 11:26:56 +01:00
Joshua Watt
714f2f5bae bitbake: runqueue: Remove trailing whitespace
(Bitbake rev: 9b9aecbbb1a3fa67f7d3b1669186c9f4ced3a590)

Signed-off-by: Joshua Watt <JPEWhacker@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2018-07-09 11:26:56 +01:00
Ming Liu
7cb125d7ee bitbake: runqueue.py: Fix a virtual class extension stamps issue
The file_name parameter passed to bb.parse.siggen.invalidate_task
should be a virtual file name instead of a real file name, or else you
will encounter a following error, for instance, when you run:

$ bitbake nativesdk-lzip -c unpack -f

the error arise:
| ERROR: An uncaught exception occurred in runqueue
| if file_name:
|    > taintfn = d.stamp[file_name] + '.' + task + '.taint'
| else:
| KeyError: 'virtual:nativesdk:/opt/poky/meta/recipes-extended/lzip/lzip_1.19.bb'

when multilib builds are used on OE.

(Bitbake rev: da37bdad46e11e7ce93ba7a59d58757b769dc16b)

Signed-off-by: Ming Liu <liu.ming50@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2018-07-02 21:38:06 +01:00
Mark Hatle
a0cb1021c0 bitbake: runqueue.py: Initial implementation of per task process limits
On high core machines, in do_fetch, it is possible to DDoS your own machine.
A method to limit any arbitrary task type to a certain number of simultaneous
threads is needed.  (Similar to how BB_NUMBER_THREADS works in the general
case.)  The format of this new limitation is:

    do_fetch[number_threads] = "2"

This should be set globally.  If it is set in individual recipes it could
result in unpredictable behavior.

Note: a value for number_threads > BB_NUMBER_THREADS will have no effect.

(Bitbake rev: 055865047c63b9c3b213b47a1884924ce0adeda0)

Signed-off-by: Mark Hatle <mark.hatle@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2018-06-15 17:57:30 +01:00
Mark Hatle
27101e647f bitbake: runqueue.py: Minor cleanup for RunQueueStats and users
The RunQueueStats:taskCompleted and RunQueueStats:taskSkipped can take
multiple arguments.  However, nowehere in bitbake are multiple arguments used.
Change this to match the behavior of the other APIs where it needs to be
called once for each task.

Additionally, these two functions were usually called in tandem, however in
the wrong order.  It really doesn't matter as there is no specific preemption
point between the calls.  But the taskSkipped should be called first to
increment the 'active' count, and then taskCompleted called to decrement it.

(Bitbake rev: 26d5ea9bb892bd6a2e1fd29a9023e0b0644edc16)

Signed-off-by: Mark Hatle <mark.hatle@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2018-06-15 17:57:30 +01:00
Richard Purdie
4bc71faf76 bitbake: runqueue: Ensure only recursive task dependencies are pruned
If a standalone tasks adds a dependency on X:do_build, the code in runqueue would
currently remove it if that do_build was part of an image recipe which uses
recrdeptask on do_build.

Such individual tasks shouldn't do this, therefore tweak the recursive reference code
to only process recurseive tasks, not all tasks.

(Bitbake rev: 4cfca360891e1ed876a9c19487b4f6210686af26)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2018-03-01 22:14:12 +00:00
Richard Purdie
c75985f40a bitbake: main/runqueue: Rework runall task and add runonly option
The runall commandline option was confusing people. There are in fact two
different behaviours people may want.

a) For a given target (or set of targets) look through the task graph and
run task X only if its present and would have been built.

b) For a given target (or set of targets) look through the task graph and
run task X if any recipe in the taskgraph has such a target even if it wasn't
in the original task graph.

I've decided to interpret the existing "runall" option as b), even if right
now if behaves like a). For a), which is a valid use case, this patch adds
a "runonly" option.

With both behaviours present, I'm hoping we can then kill off the "fetchall",
"checkuriall" and other tasks from OE metadata and replace them with this
option. This would significantly speed up task graph processing.

(Deleting the checkuriall and fetchall tasks takes "bitbake core-image-sato -g"
from 22s to 8s).

(Bitbake rev: 546a662c877b2d3af35e3996950582ed2df41fe4)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2018-02-14 15:26:03 +00:00
Richard Purdie
dafa1ac864 bitbake: runqueue: Rewrite and optimize recrdepends handling
This is a performance sensitive piece of code and the shear number
of recursive loops is causing a significant and unscalable performance
pain point.

This change moves to a two step approach, firstly generating a list of recursive
dependencies for any task, then applying this to the recursive tasks, iterating
over things until no further dependencies are added.

It was noticed an optimisation is possible and the list of recursive tasks need not
contain the taskname, only the base task id. This allows a significant performance
improvement and limits the size of the resursive task lists, improving speed.

(Bitbake rev: eba738ac5672556eaab4f3374c8025c322761c4a)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2018-02-06 11:06:30 +00:00
Richard Purdie
2ae62f0d2b bitbake: runqueue: Optimize recrdepends handling
We can optimise the loops slightly so we only process given substrings
once rather than many times. This means expanding out add_resolved_dependencies.

Also add a function which allows replacement of the task element of a
task id, reducing the amount of string handling we're doing in a performance
critical loop.

Its also clear that later code adds to the tasks depends so we don't need
to add .depends() to extradeps at the start.

(Bitbake rev: 4ad281224e92b5f94e3a9c17e8898ec8f1086cdc)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2018-02-06 11:06:30 +00:00
Richard Purdie
38fd8bbb7f bitbake: runqueue: Remove unused variables
(Bitbake rev: 7e56c285f5ebae9b4b367514e60e3a6ba8cd5693)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2018-02-06 11:06:30 +00:00
Richard Purdie
0899c332f7 bitbake: runqueue: Fix recidepends handling
Currently we only run through the recidepends/recrdepends code once. This
means that we can miss some expansions of dependency trees where one
rec{r,i}depends tasks depends on another rec{r,i}depends task.

In reality we need to iterate over the data until we stop adding
dependencies.

In doing this we can't show quite so granular progress information since
we don't know how many times we'll need to do this.

This does slow down the runqueue prepare phase however some optimisations
are possible and can be handled in subsequent patches.

This fix means some missing dependencies, such as:

<image>:do_fetchall -> <image>:do_rootfs -> <pkgs>:do_package_write_X
  -> <ca-certs>:do_package_write_X -> debianutils-native
(via PAKAGE_WRITE_DEPS)

are now found/added.

[YOCTO #12510]

(Bitbake rev: aec2f07d56a19b97b6515897532b113cdead8338)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2018-02-06 11:06:30 +00:00
Richard Purdie
b77321ab24 bitbake: runqueue: Fix typo builable -> buildable
(Bitbake rev: 1e59ae8729513e19a801c723b67911491c2a66fe)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-11-21 13:06:45 +00:00
Paul Eggleton
1bb49ce322 bitbake: runqueue: fix BB_SETSCENE_ENFORCE with empty BB_SETSCENE_ENFORCE_WHITELIST
The whitelist shouldn't have to be populated in order for the
enforcement to work properly - check if the list is not None in order to
determine whether the functionality is enabled or not since that is how
the function that sets up the list behaves.

(Bitbake rev: 7b1e79c352ca6eef1693d8abfacf7505544f1caa)

Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-09-12 23:55:29 +01:00
Richard Purdie
2355a9b6f0 bitbake: cookerdata/taskdata/runqueue: Drop remaining tryaltconfigs code and commandline option
I can't actually see how this was working, nothing connected the commandline option
to the data in TaskData(). Drop the remaining pieces of this option, it was a relic
from a decade ago and we want deterministic builds, not random tries until something
might work.

(Bitbake rev: 767c7ba8fc76ec667ac1567de1c971c3575f2ecd)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-09-02 00:52:10 +01:00
Richard Purdie
3f71378fde bitbake: runqueue: Tweak debug message to make it more readable/diffable
Having this as one huge long line isn't easy to manipulate, split it into
multiple lines for ease of debugging issues.

(Bitbake rev: 5753fe81194f75fbcf4ccdc733cc585d02794cb1)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-07-31 15:13:53 +01:00
Paul Eggleton
21bb330f46 bitbake: lib/bb/event: refactor printing events
We really ought to have just one place where the string representation
of these events is produced. This doesn't take any real control away
from the UI - if an alternative representation is desired, that can
still be made.

(Bitbake rev: cb15db2a799be6d8eab9a2a43a9a573f89229cff)

Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-07-21 08:41:12 +01:00
Juro Bystricky
68847d9ab5 bitbake: bitbake: runqueue: multiconfig fix
Differentiate between fn and taskfn in "execute".
This was somehow missed in "fakeroot" handling.

(Bitbake rev: d848bff7cf78f63986467b51f701a998a480eb25)

Signed-off-by: Juro Bystricky <juro.bystricky@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-06-14 14:54:24 +01:00
Richard Purdie
ee0ef7d13c bitbake: runqueue: Allow recrdeptask not to exist for all recipes
Currently if you specify a recrdeptask, it must exist for all recipes or
you get a python traceback. This is a bug and it should be possible to have
recipes which don't have the specified task.

As well as preventing such a traceback (which shouldn't happen, it should be
a user readable error), this allows us to fix issues in OE-Core which would
otherwise trigger the traceback.

(Bitbake rev: f93a77f2f188e18de9e3d812e86d77c2f3c71889)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-04-10 23:00:32 +01:00
Andre McCurdy
812fabbd30 bitbake: runqueue: minor typo fix
(Bitbake rev: f2c151cd345788a135452f63f622a2bc0b10c2d4)

Signed-off-by: Andre McCurdy <armccurdy@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-03-15 17:38:33 +00:00
Matthew McClintock
404f2aef94 bitbake: runqueue: add option to run all tasks in specific build target
For example:

$ bitbake core-image-minimal --runall unpack
...
NOTE: Tasks Summary: Attempted 326 tasks of which 88 didn't need to be rerun and all succeeded.

$ bitbake core-image-minimal --runall patch
NOTE: Tasks Summary: Attempted 453 tasks of which 332 didn't need to be rerun and all succeeded.

This can replace fetchall as well:
$ bitbake core-image-minimal --runall fetch
NOTE: Tasks Summary: Attempted 135 tasks of which 119 didn't need to be rerun and all succeeded.

(Bitbake rev: 7c0fa6ba66cdb956b37d94055307cde857633df9)

Signed-off-by: Matthew McClintock <msm-oss@mcclintock.net>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-03-13 09:43:40 +00:00
Richard Purdie
ff734f0560 bitbake: runqueue: Fix collapsed setscene dependency tree
When we removed the postinst sstate dependency handling code from
setscene_depvalid, we noticed things being installed into the old
style sysroot for rootfs tasks which should not have been there,
causing a performance regression.

Analysis revealed that setscene dependencies were "bubbling" over
sstate tasks when they should have been stopping there. The
'continue' added by this patch avoids this issue and eusures
sstate tasks remain contained to their specific chains.

There was another bug in the code this exposed where the acconting
for tasks as they were removed from sq_revdeps was not correct. In
fixing this, what looks like a workaround in another test can then
be simplified.

After this change, populate_sysroot tasks are no longer depending
on package_write_rpm tasks for example, which would make no sense.

A before/after analysis of image dependencies only revealed improved
dependencies after this change.

Recipe specific sysroots did highlight the issue here since the
behaviour of the sysroot dependencies (and processing with depvalid)
was not matching what bitbake itself was doing, with bitbake being
incorrect. Failures were 'safe' in that too many dependencies would
get installed.

(Bitbake rev: 5ef2cb50041fa7106c8de170af73d2a54cb0b1f0)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-02-19 06:28:47 -08:00
Patrick Ohly
bf0ea57e4a bitbake: runqueue.py: revised completion scheduler
The idea is that tasks which complete building a recipe (like
do_package_qa) are more important than tasks which start building new
recipes (do_fetch) or those which increase disk usage
(do_compile). Therefore tasks get ordered like this (most important
first, do_rm_work before do_build because the enhanced rm_work.bbclass
was used):

1. ID /work/poky/meta/recipes-support/popt/popt_1.16.bb:do_build
2. ID /work/poky/meta/recipes-core/readline/readline_6.3.bb:do_build
3. ID /work/poky/meta/recipes-connectivity/libnss-mdns/libnss-mdns_0.10.bb:do_build
...
464. ID /work/poky/meta/recipes-sato/images/core-image-sato.bb:do_build
465. ID /work/poky/meta/recipes-graphics/xorg-proto/inputproto_2.3.2.bb:do_rm_work
466. ID /work/poky/meta/recipes-devtools/python/python3_3.5.2.bb:do_rm_work
467. ID /work/poky/meta/recipes-core/packagegroups/packagegroup-base.bb:do_rm_work
...
3620. ID virtual:native:/work/poky/meta/recipes-extended/pbzip2/pbzip2_1.1.13.bb:do_install
3621. ID /work/poky/meta/recipes-devtools/qemu/qemu-helper-native_1.0.bb:do_install
3622. ID /work/poky/meta/recipes-core/zlib/zlib_1.2.8.bb:do_compile_ptest_base
3623. ID /work/poky/meta/recipes-extended/bzip2/bzip2_1.0.6.bb:do_compile_ptest_base
...
3645. ID /work/poky/meta/recipes-support/libevent/libevent_2.0.22.bb:do_compile_ptest_base
3646. ID /work/poky/meta/recipes-core/busybox/busybox_1.24.1.bb:do_compile_ptest_base
3647. ID /work/poky/meta/recipes-kernel/linux/linux-yocto_4.8.bb:do_uboot_mkimage
3648. ID /work/poky/meta/recipes-kernel/linux/linux-yocto_4.8.bb:do_sizecheck
3649. ID /work/poky/meta/recipes-kernel/linux/linux-yocto_4.8.bb:do_strip
3650. ID /work/poky/meta/recipes-kernel/linux/linux-yocto_4.8.bb:do_compile_kernelmodules
3651. ID /work/poky/meta/recipes-kernel/linux/linux-yocto_4.8.bb:do_shared_workdir
3652. ID /work/poky/meta/recipes-kernel/linux/linux-yocto_4.8.bb:do_kernel_link_images
3653. ID /work/poky/meta/recipes-devtools/quilt/quilt-native_0.64.bb:do_compile
3654. ID /work/poky/meta/recipes-extended/texinfo-dummy-native/texinfo-dummy-native.bb:do_compile
...

The order of the same task between different recipes is the same as
with the speed scheduler, i.e. more important recipes come first.

(Bitbake rev: 70e297e5c285ce0a02e9efd3117ff62cdc77ec12)

Signed-off-by: Patrick Ohly <patrick.ohly@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-01-20 11:52:52 +00:00
Richard Purdie
d8b2257f57 bitbake: runqueue: Fix traceback when using -b
Without this, bitbake -b of image recipes cause tracebacks since
the list of providers is empty.

(Bitbake rev: c53e43f3c6a675c0934a7a4e358fd66b049ffca3)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-01-20 11:52:52 +00:00
Jianxun Zhang
7eb02e837e bitbake: use multiple processes to dump signatures.
This change significantly shortens the time on reparsing stage
of '-S' option.

Each file is reparsed and then dumped within a dedicated
process. The maximum number of the running processes is not
greater than the value of BB_NUMBER_PARSE_THREADS if it is set.

The dump_sigs() in class SignatureGeneratorBasic is _replaced_
by a new dump_sigfn() interface, so calls from the outside and
subclasses are dispatched to the implementation in the base
class of SignatureGeneratorBasic.

Fixes [YOCTO #10352]

(Bitbake rev: 99d3703edd77a21770b366c6ad65a3c0f5183493)

Signed-off-by: Jianxun Zhang <jianxun.zhang@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-01-17 11:29:28 +00:00
Richard Purdie
4fc02c5eaa bitbake: runqueue: Only start fakeroot workers when needed
Fakeroot workers usually have dependencies that need to be ready before they
can be started. Starting them as a block therefore doesn't work as the
dependencies may or may not have been built. Therefore start the multiconfig
fakeworkers individually upon demand.

[YOCTO #10344]

(Bitbake rev: ab15486c636e451ed5a62e561980946887901c3b)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-12-22 12:36:40 +00:00
Richard Purdie
fe11d18ffd bitbake: runqueue: Ensure setscene tasks with overlapping stamps don't parallel execute
In multiconfig, mutliple tasks can execute which share the same stamp file. These
must not execute in parallel, the idea is the first should execute, the subsequent
ones should see a valid stamp and get skipped.

The normal task execution code has stamps code to handle this, this adds similar
code to the setscene execute() function to handle the issue there too.

(Bitbake rev: 937acf267fa9e45f538695b2cf8aa83232a96240)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-12-22 12:36:40 +00:00
Richard Purdie
f048db1c64 bitbake: runqueue: Fix setscene issues with multiconfig
setscene was being entirelu skipped for multiconfig variants as the tasks
were simply not being spotted. If the default config was also being built
it masked the problem. When this was fixed by using taskfn instead of fn
in lookups against dataCache, several other instances of this problem were
highlighted.

This goes through and corrects the setscene code to correclty use taskfn
instead of fn in the appropriate places meaning setscene tasks for multiconfig
now work correctly.

(Bitbake rev: cead85cf219c07b40aeb7c0d948c25f0429136db)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-12-22 12:36:40 +00:00