When multiconfig is used bitbake might try to run events that don't
exist for specific mc target. In cooker.py we pass
`self.databuilder.mcdata[mc]` data that contains names of events'
handlers per mc target, but fire_class_handlers uses global _handlers
variable that is created during parsing of all the targets.
This leads to a problem where bitbake runs event handler that don't
exist for a target or even overrides them - if multiple targets use
event handler with the same name but different code then only one
version will be executed for all targets.
See [YOCTO #13071] for detailed bug information.
Add mc target name as a prefix to event handler name so there won't be
two different handlers with the same name. Add internal __BBHANDLERS_MC
variable to have the handlers lists per machine.
(Bitbake rev: 5f7fdf7b2d8c59805c8ef4dae84f536baa5e172b)
Signed-off-by: Tomasz Dziendzielski <tomasz.dziendzielski@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Lets use valid python even if it the original happens to work.
(Bitbake rev: 343187c57e1459b0e57f90463843782f3a3a8443)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Calling sys.exit() in the middle of the code is rather antisocial. We catch
this in various places but we shouldn't have to. In all these cases we have
already sent events explaining to the user what happened. This means the
correct exception is BBHandledException.
The recent startup changes have moved the point a lot of this code gets
called to inside the UI, with memres it would have always been possible
from there anyway. This change makes things much more consistent.
(Bitbake rev: 91699f366d24480ff3b19faec78fb9f3181b3e14)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The first thing the UIs do is update the server config from the UI. We
can just rely upon that and start the server with a standard config,
removing the need to pass the confusing configuration object around
as well as configParams, which contains a similar copy of some of the
data.
This makes memory resident bitbake work the same way as the normal
mode, removing the opportunity for some class of bugs.
The xmlrpcinterface and server_timeout values are passed in at server
startup time now and there no longer a second option in the
configuration which is effective ignored once the server starts.
(Bitbake rev: 783a03330802e83c525c55522e3ee2a933bded3a)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Currently the eventlog is not handled correctly for memory resident
bitbake. Fix this by allowing adpations to configuration changes.
(Bitbake rev: f7d2c9116116659ea42260a3bb96dca100aadae7)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
There were some options which were not being passed to the server. This
was breaking, particularly in memory resident bitbake mode.
Add the missing options to the correct place to ensure the server is
updated correctly.
(Bitbake rev: 5dc178e7fae3ca8558146e385a05f5d96a021777)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Rather than passing debug/verbose/debug_domains around, pass the
computed output of these. Ensure that the cooker sets the levels
to match the levels currently set in the UI and generally try and
make it easier to understand what the code is doing.
(Bitbake rev: f0535beecc692a6213be2a22f9eef5956450ecf8)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The levels of indirection to set these verbose logging options is rather
crazy. This attempts to turn things into two specific options with
much more specific meanings. For now its all still controlled by the
commandline verbose option and should funciton as previously, with
the addition that the BB_VERBOSE_LOGS option can now be task specific.
(Bitbake rev: 423c046f2173aaff3072dc3d0882d01b8a0b0212)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Having the idle callbacks abstracted via the configuration object
makes no sense. Its like this for historical reasons from the
multiple server backends but we don't need this now so simplfy.
(Bitbake rev: e56c49717355c9493b07d5fc80981a95ad8a0ec8)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Add an inverse mode for BBFILES_DYNAMIC.
Entries where the layername is prefix with '!' will act in inverted mode
including given glob mask in absence of the referenced layer.
This is useful to backfill recipes from potentially missing layer.
(Bitbake rev: bfd729cfe06351ccd0071004e23af1d04546c723)
Signed-off-by: Konrad Weihmann <kweihmann@outlook.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The variable values that result from parsing multiconfig should be
included in the cooker data hash, otherwise changes to these files won't
be detected, which will allow the parsing cache to be loaded with the
old values for the multiconfigs. This can either manifest as the
variable values simply not updating, or getting basehash changed errors
when building.
This bug was previously undetected because all of the multiconfig base
files were a direct file dependency in all parsed recipes. This was
fixed in 34137a00f60 ("bitbake: bitbake: cooker: Rename __depends in all
multiconfigs"), exposing this bug.
[YOCTO #13541]
(Bitbake rev: c74481aa15226e1bff9d53e4ee4b702ebfa1ad32)
Signed-off-by: Joshua Watt <JPEWhacker@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
If we define a metadata siggen it can fail due to the early init here. Move
slightly later to avoid those failures which allows fixes in OE to the
check-layer script related to the hash equiv siggen.
(Bitbake rev: fdf5c341f3393173876a753c46c9bd067eb2b353)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Adds an option to skip _setscene only if they would normally be
executed, without ignoring sstate completely.
Previously, '--no-setscene' would allow a build that completely ignored
sstate and _setscene tasks, and '--setscene-only' would allow a build
that only ran _setscene tasks, but there was no option do a build that
would respect tasks previously restored from sstate and build everything
else. Now one can run:
bitbake --setscene-only IMAGE; bitbake --skip-setscene IMAGE
which is functionally equivalent to:
bitbake IMAGE
The indented use is to allow a build to complete successfully in the
presence of _setscene task failures by splitting apart the two phases
e.g.:
(bitbake -k --setscene-only IMAGE || true) && bitbake --skip-setscene IMAGE
(Bitbake rev: 813ba5b7c13b573a0b813b628a819bdbf0627540)
Signed-off-by: Joshua Watt <JPEWhacker@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
(Bitbake rev: cc712f3257904960247a7532cfc4611f3dccd36c)
Signed-off-by: Robert Yang <liezhi.yang@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Check existence of all the bblayer direcotories at once and print them
all, so if there are multiple nonexistent directories, user does not
have to correct bblayers.conf and restart bitbake multiple times.
[YOCTO #11647]
(Bitbake rev: 19291f7c4d17086ebb6a7b80c3cb06333d7fd55b)
Signed-off-by: Oleksandr Kravchuk <open.source@oleksandr-kravchuk.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
There are much better ways to handle this and most editors shouldn't need this
in modern times, drop the noise from the files. Its not consitently applied
anyway.
(Bitbake rev: 5e43070e3087d09aea2f459b033d035c5ef747d0)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
With the introduction of SPDX-License-Identifier headers, we don't need a ton
of header boilerplate in every file. Simplify the files and rely on the top
level for the full licence text.
(Bitbake rev: 695d84397b68cc003186e22f395caa378b06bc75)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
This adds the SPDX-License-Identifier license headers to the majority of
our source files to make it clearer exactly which license files are under.
The bulk of the files are under GPL v2.0 with one found to be under V2.0
or later, some under MIT and some have dual license. There are some files
which are potentially harder to classify where we've imported upstream code
and those can be handled specifically in later commits.
The COPYING file is replaced with LICENSE.X files which contain the full
license texts.
(Bitbake rev: ff237c33337f4da2ca06c3a2c49699bc26608a6b)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
It shouldn't work when there are duplicated BBFILE_COLLECTIONS.
(Bitbake rev: a1f251e5ab859d6d4a2cb908408d4ddcab5a5de1)
Signed-off-by: Robert Yang <liezhi.yang@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
We'd like layers to set this variable so that we know which layers are compatible
with which others, even if the branch is a generic un-updated "master" branch.
Start printing a warning to highlight this issue.
(Bitbake rev: cca81e33b58c390dcf5cc3a31555a43b79177166)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The runall commandline option was confusing people. There are in fact two
different behaviours people may want.
a) For a given target (or set of targets) look through the task graph and
run task X only if its present and would have been built.
b) For a given target (or set of targets) look through the task graph and
run task X if any recipe in the taskgraph has such a target even if it wasn't
in the original task graph.
I've decided to interpret the existing "runall" option as b), even if right
now if behaves like a). For a), which is a valid use case, this patch adds
a "runonly" option.
With both behaviours present, I'm hoping we can then kill off the "fetchall",
"checkuriall" and other tasks from OE metadata and replace them with this
option. This would significantly speed up task graph processing.
(Deleting the checkuriall and fetchall tasks takes "bitbake core-image-sato -g"
from 22s to 8s).
(Bitbake rev: 546a662c877b2d3af35e3996950582ed2df41fe4)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
I can't actually see how this was working, nothing connected the commandline option
to the data in TaskData(). Drop the remaining pieces of this option, it was a relic
from a decade ago and we want deterministic builds, not random tries until something
might work.
(Bitbake rev: 767c7ba8fc76ec667ac1567de1c971c3575f2ecd)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
cooker.reset() can be called before we've actually setup the datastore. Gracefully
handle this case instead of the current traceback+exit.
(Bitbake rev: 8fd30ca6d271c125a8ea03ef0c5d7ab176900701)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Currently the original prefile and postfile passed when starting bitbake
server are 'sticky'. With the new memory resident model this doesn't make
sense as the server the system is started with isn't special.
This patch changes the code so the prefile/postfile are used if specified
on the commandline and not used otherwise. This makes the behaviour much
more predictable and expected and as an added bonus simplifies the code.
(Bitbake rev: 638d366234fad78f283d3a13a12b07cb0ccbe914)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The current passing of "tracking" backwards and forwards, client to server
is ugly and complex and error prone. Instead, set this during showEnvironment
commands triggering a reset there if/as required.
(Bitbake rev: 9dc7f384db0479569ff93a76a623d5395fecaf47)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
This changes the way bitbake server works quite radically. Now, the
server is always a process based server with the option of starting
an XMLRPC listener on a specific inferface/port.
Behind the scenes this is done with a "bitbake.sock" file alongside
the bitbake.lock file. If we can obtain the lock, we know we need
to start a server. The server always listens on the socket and UIs
can then connect to this. UIs connect by sending a set of three file
descriptors over the domain socket, one for sending commands, one for
receiving command results and the other for receiving events.
These changes meant we can throw away all the horrid server abstraction
code, the plugable transport option to bitbake and the code becomes
much more readable and debuggable. It also likely removes a ton of
ways you could hang the UI/cooker in weird ways due to all the race
conditions that existed with previous processes.
Changes:
* The foreground option for bitbake-server was dropped. Just tail
the log if you really want this, the codepaths were complicated enough
without adding one for this.
* BBSERVER="autodetect" was dropped. The server will autostart and
autoconnect in process mode. You have to specify an xmlrpc server
address since that can't be autodetected. I can't see a use case
for autodetect now.
* The transport/servetype option to bitbake was dropped.
* A BB_SERVER_TIMEOUT variable is added which allows the server
to stay resident for a period of time after the last client
disconnects before unloading. This is used if the -T/--idle-timeout
option is not passed to bitbake.
This change is invasive and may well introduce new issues however I
believe the codebase is in a much better position for further
development and debugging.
(Bitbake rev: 72a3dbe13a23588e24c0baca6d58c35cdeba3f63)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Finding the top level build directory is currently hard and relies on
having a complete cooker being setup. Add a helper function which
does the same thing without all the extra overhead. This is needed
to be able to locate the bitbake lockfile and hence the socket
for connecting clients in the new server model.
(Bitbake rev: d196afe68032898c31a8599ca7d3ceba58d96b0a)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Fix one incorrect indentation and several lines with trailing spaces.
(Bitbake rev: 4dbf4f1907a32f303c2e9acf2f27817cf3c9ec38)
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
There are some cases where the metadata needs to be aware a multiconfig build
is happening and have access to the multiconfig data stores to merge data into
the common build. This adds such an event allowing access to these datastores.
(Bitbake rev: 160e47f5df90850e64dcb857c81a5039abc9235f)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Currently, someone trying a master branch of any given layer may
or may not find it works with the version of OE-Core that they
have picked.
This patchset introduces LAYERSERIES_CORENAMES and
LAYERSERIES_COMPAT_<layername> with the intention of allowing layers
to indicate which versions of the core project they're compatible with.
In general the master branch will use the next release codename as
the value of LAYERSERIES_CORENAMES, meaning that the waterfall model
of layer releases is supported with layers updating their
LAYERSERIES_COMPAT_ values as they branch for release.
Both variables support multiple strings and a given layer is accepted
if there is overlap of any one value. This means a layer can be compatible
with multiple core versions.
Setting LAYERSERIES_COMPAT_ will be required by the Yocto Project
Compatible v2 standard but the system will not error if its not set at this
point.
This should make it clear when a given layer is unmaintained and untested
with new releaes of OE-Core, a common user pain point.
(Bitbake rev: 92c49b9b1a16dfd35444db8143bd4cae4cda70cc)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
In OpenEmbedded's buildhistory class we want access to the bitbake
command line that launched the build, and up to now we were simply using
sys.argv from within the event handler to get that. Unfortunately that
doesn't work in memory resident mode, since the event handler is
naturally executing within the server and thus will give you the command
that launched the bitbake server which is much less interesting. Add a
dynamic variable BB_CMDLINE to provide access to this, set from sys.argv
within the UI process in updateToServer().
(Note that BB_CMDLINE isn't currently passed through to the worker, so
this is only really readable from event handlers plus any explicit
getVariable calls - in theory an observe-only UI could read it for
example.)
Part of the fix for [YOCTO #11634].
(Bitbake rev: 8ad31a3eff5cfcb7b1c462578a582dafcbc426d4)
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
CookerConfiguration sets the "tracking" option to True when -e is
specified in order to have history tracking enabled in the datastore
so that we can show variable history (which isn't enabled by default for
performance reasons), however in memory resident mode this wasn't doing
anything because it was happening on the UI side only. We do have a
mechanism for updating the cooker configuration in the server, but the
tracking option wasn't being included in the list of options we updated,
so we just need to add this option to fix the issue.
Fixes [YOCTO #10730].
(Bitbake rev: 57d4977555cf892b15dd0302dfe261fe37d49327)
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
BBFILES_DYNAMIC can be used to activate content only when some other
layers are present. The other layers are identified by the collections
that they define.
The main use case is to avoid .bbappends without the corresponding .bb
file in layers that want to modify other layers via .bbappends without
introducing a hard dependency on those other layers. .bb files could
also be handled via BBFILES_DYNAMIC.
Entries in BBFILES_DYNAMIC must have the form <collection
name>:<filename pattern>. Example usage:
BBFILES_DYNAMIC += " \
clang-layer:${LAYERDIR}/bbappends/meta-clang/*/*/*.bbappend \
core:${LAYERDIR}/bbappends/openembedded-core/meta/*/*/*.bbappend \
"
Parsing is aborted when invalid entries are found with an error
message like this:
ERROR: BBFILES_DYNAMIC entries must be of the form <collection name>:<filename pattern>, not:
/work/my-layer/bbappends/meta-security-isafw/*/*/*.bbappend
/work/my-layer/bbappends/openembedded-core/meta/*/*/*.bbappend
Based on a patch by Richard Purdie.
(Bitbake rev: 04f8bd50aa04b12cf91dd6a3154527ad2c24695c)
Signed-off-by: Patrick Ohly <patrick.ohly@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
For example:
$ bitbake core-image-minimal --runall unpack
...
NOTE: Tasks Summary: Attempted 326 tasks of which 88 didn't need to be rerun and all succeeded.
$ bitbake core-image-minimal --runall patch
NOTE: Tasks Summary: Attempted 453 tasks of which 332 didn't need to be rerun and all succeeded.
This can replace fetchall as well:
$ bitbake core-image-minimal --runall fetch
NOTE: Tasks Summary: Attempted 135 tasks of which 119 didn't need to be rerun and all succeeded.
(Bitbake rev: 7c0fa6ba66cdb956b37d94055307cde857633df9)
Signed-off-by: Matthew McClintock <msm-oss@mcclintock.net>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
When we're running with bitbake -b, BB_TASKDEPDATA is incorrect and limited.
We really need a way to know this from the metadata and this new variable
provides this in worker context. This means existing code can stop having
to guess.
(Bitbake rev: 05763bc886024dcce2ce6b3060fb00abf79a9402)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
People are struggling with multiconfig as the point the conf file
is injected into the data store is not what people expect. We can't
really use a post config since that is too late and we can't really
use a pre config file since that is too early. In OE terms, we need
something right around the local.conf point so it behaves in a similar
way.
A way to handle this is to set the new variable BB_CURRENT_MC to be the
currently selected multiconfig, then the metadata itself can choose
when to inject the approriate configuration.
(Bitbake rev: 518b9015c2be8d3894277a8e54890d6f04d656c0)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
In memory resident mode we don't really want to actually shut down since
it's only the client going away.
(Bitbake rev: 74db369c46043116359101cab70486afd82372c0)
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
getVarFlag() now defaults to expanding by default, thus remove the
True option from getVarFlag() calls with a regex search and
replace.
Search made with the following regex:
getVarFlag ?\(( ?[^,()]*, ?[^,()]*), True\)
(Bitbake rev: c19baa8c19ea8ab9b9b64fd30298d8764c6fd2cd)
Signed-off-by: Joshua Lock <joshua.g.lock@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
getVar() now defaults to expanding by default, thus remove the True
option from getVar() calls with a regex search and replace.
Search made with the following regex: getVar ?\(( ?[^,()]*), True\)
(Bitbake rev: 3b45c479de8640f92dd1d9f147b02e1eecfaadc8)
Signed-off-by: Joshua Lock <joshua.g.lock@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The event handling 'Exception' was catching and triggering a backtrace. This
trace was obscuring any errors from an event handler that had raised the
BBHandledException, which should indicate do not print additional information.
(Bitbake rev: 51ca5193a5674b27d816140b0254f485912177a2)
Signed-off-by: Mark Hatle <mark.hatle@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
This patch adds the notion of supporting multiple configurations within
a single build. To enable it, set a line in local.conf like:
BBMULTICONFIG = "configA configB configC"
This would tell bitbake that before it parses the base configuration,
it should load conf/configA.conf and so on for each different
configuration. These would contain lines like:
MACHINE = "A"
or other variables which can be set which can be built in the same
build directory (or change TMPDIR not to conflict).
One downside I've already discovered is that if we want to inherit this
file right at the start of parsing, the only place you can put the
configurations is in "cwd", since BBPATH isn't constructed until the
layers are parsed and therefore using it as a preconf file isn't
possible unless its located there.
Execution of these targets takes the form "bitbake
multiconfig:configA:core-image-minimal core-image-sato" so similar to
our virtclass approach for native/nativesdk/multilib using BBCLASSEXTEND.
Implementation wise, the implication is that instead of tasks being
uniquely referenced with "recipename/fn:task" it now needs to be
"configuration:recipename:task".
We already started using "virtual" filenames for recipes when we
implemented BBCLASSEXTEND and this patch adds a new prefix to
these, "multiconfig:<configname>:" and hence avoid changes to a large
part of the codebase thanks to this. databuilder has an internal array
of data stores and uses the right one depending on the supplied virtual
filename.
That trick allows us to use the existing parsing code including the
multithreading mostly unchanged as well as most of the cache code.
For recipecache, we end up with a dict of these accessed by
multiconfig (mc). taskdata and runqueue can only cope with one recipecache
so for taskdata, we pass in each recipecache and have it compute the result
and end up with an array of taskdatas. We can only have one runqueue so there
extensive changes there.
This initial implementation has some drawbacks:
a) There are no inter-multi-configuration dependencies as yet
b) There are no sstate optimisations. This means if the build uses the
same object twice in say two different TMPDIRs, it will either load from
an existing sstate cache at the start or build it twice. We can then in
due course look at ways in which it would only build it once and then
reuse it. This will likely need significant changes to the way sstate
currently works to make that possible.
(Bitbake rev: 5287991691578825c847bac2368e9b51c0ede3f0)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The current codepaths are rather confusing. Stop passing these
as parameters and use the ones from when the object is created.
(Bitbake rev: 8c992c148d9619b10eeae8bbd9376ecf408037a5)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Makes it easier for user to identify problems, e.g. typos, in BBLAYERS.
[YOCTO #9507]
(Bitbake rev: 32c9689e4b492dc5821749e284e397d717af2a6c)
Signed-off-by: Markus Lehtonen <markus.lehtonen@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
This variable is a regex-escaped version of LAYERDIR, for safer use in
BBFILE_PATTERN, so as to avoid issues with regex special characters in the
layer path.
[YOCTO #8402]
(Bitbake rev: 72900522778b6ff08b135bf8bb97dff3f1a20bd9)
Signed-off-by: Christopher Larson <chris_larson@mentor.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The string formatting wasn't correct and we should exit if we hit
errors here similar to the other exception handlers.
(Bitbake rev: b90a16408a5c45ce5312384f278e19d09f8dda4d)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
