The dependency graph in the bb.event.DepTreeGenerated and the
corresponding pn-buildlist and task-depends.dot from "bitbake -g"
contained entries like multiconfig:qemuarm.gcc (dot as last separator)
instead of the correct multiconfig:qemuarm:gcc (colon as separator).
(Bitbake rev: cccd1578d84f041cd0c2dcddb91f317c69af70de)
Signed-off-by: Patrick Ohly <patrick.ohly@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Displaying the environment data in a sorted manner makes it easier
to compare data between different setups and means you can know
where to find specific entries.
(Bitbake rev: baa417e44b009149eb1dfb07d5a488740b6e68ef)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
When constructing an older style single regex, it's possible for BBMASK
to end up beginning with '|', which matches and masks _everything_.
(Bitbake rev: 56ad67017e601c7e0f6085ca84e29c28d8d4519f)
Signed-off-by: Andre McCurdy <armccurdy@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The package-depends.dot and pn-depends.dot files are inaccurate, missing
out key dependencies such those made via the [depends] flags. As such
they can be misleading to the user.
They mainly exist for historical reasons, coming from a time before
we had task based execution.
This commit removes the two dated file formats and replaces them with
a recipe-depends.dot which is a flattened version of task-depends.dot.
The old format files are removed if present so that the user can't get
confused about why data might not match between files.
The code is also rewritten to use 'with f: f.write()' syntax as is more
commonly used now.
Also update the docs to match the change.
(Bitbake rev: d3e182bc18ff2894f1efc8aad3d508dd432c996e)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Also run sanity check for all additional multiconfig configurations
as listed in BBMULTICONFIG, not just the "default" configuration.
[YOCTO#10810]
(Bitbake rev: b9777374cbe63cdcf80767033f628c26d23e90f9)
Signed-off-by: Juro Bystricky <juro.bystricky@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Now that the datastore works dynamically we don't need the update_data calls
so we can just remove them. They're not actually done anything at all for
a while.
(Bitbake rev: 2300beb50333bb620013b058a7309e7f2042101d)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
When we're running with bitbake -b, BB_TASKDEPDATA is incorrect and limited.
We really need a way to know this from the metadata and this new variable
provides this in worker context. This means existing code can stop having
to guess.
(Bitbake rev: 05763bc886024dcce2ce6b3060fb00abf79a9402)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Currently "bitbake universe -c fetch" will error if every task in the system
doesn't have a fetch task. This seems unreasonable so filter the list based
upon whether the task exists. This implementation does assume if a task exits
for a given provider name, it can run, even if the preferred provider that
ends up getting built might be the namespace without the task. This is unlikely
to be a real world issue at this point.
(Bitbake rev: 438986f4e39974930ee3205ca4c4d620090e57bb)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Currently you cannot build a target for all the configured multiconfigs without
specifying a list. The list can be quite long, requiring to type several lines
of text.
This enhancement is to support globbing so that you can do this,
e.g. instead of:
$ bitbake multiconfig:A:bash multiconfig:B:bash bash
you can do:
$ bitbake multiconfig:*:bash
There are real world use cases where it is desirable to use multiconfig with
two different tasks. For example: SDKs with multiple toolchains but also
containing set of additional host tools, or multiconfig builds requiring one image for
the main CPU(s) and a different co-image for a companion CPU.
For this reason, two variations of the new syntax are supported.
For example, the following:
$ bitbake multiconfig:*:meta-toolhchain
would expand to:
$ bitbake multiconfig:A:meta-toolchain multiconfig:B:meta-toolchain meta-toolchain
However the following:
$ bitbake multiconfig:*:meta-toolhchain hosttools
would expand to:
$ bitbake multiconfig:A:meta-toolchain multiconfig:B:meta-toolchain hosttools
In other words, if the user specified the "default" task explicitly, it replaces the implicit
"default" task.
[YOCTO#10680]
(Bitbake rev: 1559ad1fc3ae8b1e4e75b210f598b90be192571b)
Signed-off-by: Juro Bystricky <juro.bystricky@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
At end of BBCooker needs to release fd's associated with pyinotify
watchers to avoid: Too many open files (EMFILE) error in different
scenarios like several instances of tinfoil.
[YOCTO #10873]
(Bitbake rev: ae6045b84978940c365c95c33d6996359c3e299d)
Signed-off-by: Aníbal Limón <anibal.limon@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Some of our metadata assumes that BuildStarted and BuildCompleted events
see the same data store. This is the case for buildTarget but not for
buildFile and recent changes mean this is now a problem.
The update_data() call is now an empty operation and there is no difference
between the expanded_data and data so we can simply remove the expanded_data
and its references and use data everywhere. This has been inteded for a while
but the above issue makes this more pressing to finally clean up.
(Bitbake rev: e3694e738e98f26f413ada6860ca7d829d3662f0)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Allow the client to set variables with the setVariable command and have
those changes take effect when running tasks. This is accomplished by
collecting changes made by setVariable separately and pass these to the
worker so it can be applied on top of the datastore it creates.
(Bitbake rev: 69a3cd790da35c3898a8f50c284ad1a4677682a4)
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
If we want to use this function/command internally, we don't want this
warning shown.
(Bitbake rev: 5cfbb60833e7b12d698c1c2970c17ccf2a4971bf)
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Rewrite tinfoil as a wrapper around the UI, instead of the earlier
approach of starting up just enough of cooker to do what we want. This
has several advantages:
* It now works when bitbake is memory-resident instead of failing with
"ERROR: Only one copy of bitbake should be run against a build
directory".
* We can now connect an actual UI, thus you get things like the recipe
parsing / cache loading progress bar and parse error handling for free
* We can now handle events generated by the server if we wish to do so
* We can potentially extend this to do more stuff, e.g. actually running
build operations - this needs to be made more practical before we can
use it though (since you effectively have to become the UI yourself
for this at the moment.)
The downside is that tinfoil no longer has direct access to cooker, the
global datastore, or the cache. To mitigate this I have extended
data_smart to provide remote access capability for the datastore, and
created "fake" cooker and cooker.recipecache / cooker.collection adapter
objects in order to avoid breaking too many tinfoil-using scripts that
might be out there (we've never officially documented tinfoil or
BitBake's internal code, but we can still make accommodations where
practical). I've at least gone far enough to support all of the
utilities that use tinfoil in OE-Core with some changes, but I know
there are scripts such as Chris Larson's "bb" out there that do make
other calls into BitBake code that I'm not currently providing access to
through the adapters.
Part of the fix for [YOCTO #5470].
(Bitbake rev: 3bbf8d611c859f74d563778115677a04f5c4ab43)
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
In memory resident mode we don't really want to actually shut down since
it's only the client going away.
(Bitbake rev: 74db369c46043116359101cab70486afd82372c0)
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
If many files change and the inotify queue overflows, rather than print
a traceback, invalidate the caches and warn the user.
[YOCTO #10676]
(Bitbake rev: 058f8517c041b80e8b591ad7d34a68281b2d03fc)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The processing of the "do_" prefix to tasks is currently inconsistent
and has resulted in "bitbake world -g" being broken as task prefixes
don't get handled correctly.
Make the "do_" task prefix handling consistent through various codepaths.
[YOCTO #10651]
(Bitbake rev: 3d7186353e804c9410096c408bc337a98c8b33fe)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
We assume that the value taken by variable v can be string,
integer or any type which can be marshalled by xmlrpc. This
change would help us to convert the non marshallable types
to string. So that we don't get exception from xmlrpc.
[YOCTO #10740]
(Bitbake rev: efb0e47479b3526bc047112f7200087c5844bba4)
Signed-off-by: Sujith H <sujith.h@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
getVar() now defaults to expanding by default, thus remove the True
option from getVar() calls with a regex search and replace.
Search made with the following regex: getVar ?\(( ?[^,()]*), True\)
(Bitbake rev: 3b45c479de8640f92dd1d9f147b02e1eecfaadc8)
Signed-off-by: Joshua Lock <joshua.g.lock@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
There would be error when BBFILE_PATTERN = None:
BBFILE_PATTERN_foo not defined
This is the correct behaviour, but when the layer sets BBFILE_PATTERN = "",
it would match all the remaining recipes, and cause "No bb files matched BBFILE_PATTERN"
warnings for all the layers which behind it.
When a layer sets BBFILE_PATTERN = "" (for example, a layer only
provides git repos and source tarballs), now it means has no recipes.
This is different from BBFILE_PATTERN_IGNORE_EMPTY, the later one means
that it *may* not have any recipes.
(Bitbake rev: 91c3b34625fac2a0f093a4b46a46e89f813e7972)
Signed-off-by: Robert Yang <liezhi.yang@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Some task id manipulations were suboptimal:
* taskfn_fromtid and fn_from_tid were effectively the same function
* many calls to split_tid(), then taskfn_fromtid()
* taskfn_fromtid() called split_tid() internally
This patch adds split_tid_mcfn() to replace split_tid() and returns the
"taskfn" variant being used in many places. We update all core calls
to the new function and ignore the return values we don't need since the
function call overhead of the split_tid wrapper is higher than ignoring
a return value.
The one remaining standalone use of taskfn_fromtid is replaced with
fn_from_tid. I couldn't see any external usage so it was dropped.
There is external usage of split_tid so a wrapper remains for it.
Combined together these changes should improve some of the runqueue task
manipulation performance.
(Bitbake rev: 1bf2ef874fbe47f1320007efa0bdeef8d630b8a1)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
A "bitbake world -c unpack" currently breaks as not all tasks have an
unpack task. This change allows addition of world targets only if the
specified task exists which makes certain commands possible when otherwise
you just get errors which can't easily be avoided.
(Bitbake rev: ca4f5e6d01b5c8cf315f59bc86194d63c0d3d042)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Add recommended layers to collection_depends[] so that dynamic
priority assignment will work for both depends and recommends.
Recommended layers do not cause an error or warning
if they are not in the collection list, but debug messages
are output for level 3 and above.
explode_dep_versions2 returns a dictionary, so we
change the variable deplist to depDict. The dictionary
values are lists which are either empty or contain only one
version specification.
(Bitbake rev: 20cdc3d609f8aea992f97c3db336574d3a549973)
Signed-off-by: Joe Slater <jslater@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Task name was incorrectly added to the targets that already
contained :task suffix and fired with BuildInit event. This
caused Toaster to create incorrect Target objects and show
them in UI.
[YOCTO #10221]
(Bitbake rev: b7faf1af3bd3110fba347fbe6e432fc4ee66590a)
Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Bitbake collects all events in special event queue when called with
-w option. However, it starts to write events to the eventlog only
after BuildStarted event is received. In some cases this event is
not received at all, e.g. when bitbake is run with --parse-only
command line option.
It makes sense to write all collected events when CookerExit event
received to make sure all events are written into the eventlog even
if BuildStarted event is not fired.
[YOCTO #10145]
(Bitbake rev: 57912de63fa83550c0ae658eb99b76e9cc91a8d1)
Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Currently, there is a check to remove the TOSTOP attribute from
a tty to avoid hangs. It assumes that sys.stdout will have a
file descriptor and this is not always true, some IO classes
will throw exceptions when trying to get its file descriptor.
This will add a check for such cases and avoid throwing an
exception.
[YOCTO #10162]
(Bitbake rev: cb4f8f6efa28ef2b13bc738a0118b876baa15b3e)
Signed-off-by: Mariano Lopez <mariano.lopez@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
This patch adds the notion of supporting multiple configurations within
a single build. To enable it, set a line in local.conf like:
BBMULTICONFIG = "configA configB configC"
This would tell bitbake that before it parses the base configuration,
it should load conf/configA.conf and so on for each different
configuration. These would contain lines like:
MACHINE = "A"
or other variables which can be set which can be built in the same
build directory (or change TMPDIR not to conflict).
One downside I've already discovered is that if we want to inherit this
file right at the start of parsing, the only place you can put the
configurations is in "cwd", since BBPATH isn't constructed until the
layers are parsed and therefore using it as a preconf file isn't
possible unless its located there.
Execution of these targets takes the form "bitbake
multiconfig:configA:core-image-minimal core-image-sato" so similar to
our virtclass approach for native/nativesdk/multilib using BBCLASSEXTEND.
Implementation wise, the implication is that instead of tasks being
uniquely referenced with "recipename/fn:task" it now needs to be
"configuration:recipename:task".
We already started using "virtual" filenames for recipes when we
implemented BBCLASSEXTEND and this patch adds a new prefix to
these, "multiconfig:<configname>:" and hence avoid changes to a large
part of the codebase thanks to this. databuilder has an internal array
of data stores and uses the right one depending on the supplied virtual
filename.
That trick allows us to use the existing parsing code including the
multithreading mostly unchanged as well as most of the cache code.
For recipecache, we end up with a dict of these accessed by
multiconfig (mc). taskdata and runqueue can only cope with one recipecache
so for taskdata, we pass in each recipecache and have it compute the result
and end up with an array of taskdatas. We can only have one runqueue so there
extensive changes there.
This initial implementation has some drawbacks:
a) There are no inter-multi-configuration dependencies as yet
b) There are no sstate optimisations. This means if the build uses the
same object twice in say two different TMPDIRs, it will either load from
an existing sstate cache at the start or build it twice. We can then in
due course look at ways in which it would only build it once and then
reuse it. This will likely need significant changes to the way sstate
currently works to make that possible.
(Bitbake rev: 5287991691578825c847bac2368e9b51c0ede3f0)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Rather than passing in a datastore to build on top of, use the data builder
object in the cache and base the parsed recipe from this. This turns
things into proper objects building from one another rather than messy
mixes of static and class functions.
This sets things up so we can support parsing and building multiple
configurations.
(Bitbake rev: fef18b445c0cb6b266cd939b9c78d7cbce38663f)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Rather that the current mix of static and class methods, refactor
so that the cache has the databuilder object internally. This becomes
useful for the following patches for multi config support.
It effectively completes some of the object oriented work we've been
working towards in the bitbake core for a while.
(Bitbake rev: 7da062956bf40c1b9ac1aaee222a13f40bba9b19)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Needing to access these static methods through a class doesn't
make sense. Move these to become module level standalone functions.
(Bitbake rev: 6d06e93c6a2204af6d2cf747a4610bd0eeb9f202)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
In situations where a bitbake run fails before the build
properly starts and BuildStarted is fired, a UI has no way
to get at the targets passed to the build. This makes it
difficult for the UI to report on the targets which failed.
Fire a BuildInit event before running buildTargets() or
buildFile(). This enables a UI to capture targets passed to
buildTargets(), even if the build fails (e.g. the targets
themselves are invalid).
[YOCTO #8440]
(Bitbake rev: ac02fda870965bf7d44ff5688eda54d2d11ab9c7)
Signed-off-by: Elliot Smith <elliot.smith@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
In the runqueue cleanup/conversion, "dep" was mistakenly used where "tid" should
be leading to incorrect task-depends.dot files and causing general confusion.
Fix this, its clearly incorrect looking at the code.
(Bitbake rev: 689730dbb068c5ea3593e7b92fe5d5e5c0c3760a)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Restructured EventWriter code to make it more readable:
- got rid of init_file method as it's called only once
- renamed exception variable e -> err
- renamed event variable e -> evt
- simplified main 'if' structure of send method
(Bitbake rev: 31977e7bb98f676197c6cee66f6ab4c12d4dcbde)
Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
class EventLogWriteHandler is a simple wrapper class with only one
class member. Replacing it with namedtuple makes code less nested and more
readable.
(Bitbake rev: 7c5b6812d32d173df36e7f9fc1d877329e79f994)
Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
There is no need to remove output file as it gets rewritten by
open(self.eventfile, 'w') anyway.
(Bitbake rev: 1fc9957837b7038dfb983217a3fcd880f143e3a4)
Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
pickle converts python objects into the binary form that can't be
decoded to text and therefore can't be converted to JSON format.
Attempt to convert event objects raises this error:
TypeError:
b'\x80\x03cbb.runqueue\nrunQueueTaskSkipped\nq\x00)...
is not JSON serializable
Encoded pickled event objects to base64 to be able to convert data
structure to JSON.
[YOCTO #9803]
(Bitbake rev: f18055237e6084f90f6221442e3ba021dcc59c50)
Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
EventLogWriteHandler object was created and used in
BBCooker.initConfigurationData.
This causes creation of multiple EventLogWriteHandler objects
and results in duplicated entries in the output event file
as BBCooker.initConfigurationData is called multiple times.
Added eventlogfile parameter to EventLogWriteHandler to avoid using
global variable DEFAULT_EVENTFILE.
Moved EventLogWriteHandler to the module level.
Created EventLogWriteHandler object in BBCooker.__init__ to ensure that only
one handler object is created.
(Bitbake rev: d3ad8eee850ec2df54aa09fae44cc7e69c12f32a)
Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
First parameter of traceback.format_exc is a 'limit' - a number
of stracktraces to format.
Passing exception object to format_exc is incorrect, but it works in
Python 2 as this code from traceback module works:
while tb is not None and (limit is None or n < limit):
Comparing integer counter n with the exception object in Python 2
always results in True. However, in Python 3 it throws exception:
TypeError: unorderable types: int() < <Exception type>()
As format_exc is used in except block of handling another
exception this can cause hard to find and debug bugs.
(Bitbake rev: a9509949d7e2adba6e3cd89f97daa19a955855b5)
Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
I'm not sure what possesed me when I wrote this code originally but its
indirection of everyting to use numeric IDs and position dependent lists
is horrific. Given the way python internals work, its completely and
utterly pointless from performance perspective. It also makes the code
hard to understand and debug since any numeric ID has to be translated
into something human readable.
The hard part is that the IDs are infectous and spread from taskdata
into runqueue and even partly into cooker for the dependency graph
processing. The only real way to deal with this is to convert everything
to use a more sane data structure.
This patch:
* Uses "<fn>:<taskname>" as the ID for tasks rather than a number
* Changes to dict() based structures rather than position dependent lists
* Drops the build name, runtime name and filename ID indexes
On the most part there shouldn't be user visible changes. Sadly we did
leak datastructures to the setscene verify function which has to be
rewritten. To handle this, the variable name used to specifiy the version
changes from BB_SETSCENE_VERIFY_FUNCTION to BB_SETSCENE_VERIFY_FUNCTION2
allowing multiple versions of bitbake to work with suitably written
metadata. Anyone with custom schedulers may also need to change them.
I believe the benefits in code readability and easier debugging far
outweigh those issues though. It also means we have a saner codebase
to add multiconfig support on top of.
During development, I did have some of the original code coexisting with
the new data stores to allow comparision of the data and check it was
working correcty, particuarly for taskdata. I have also compared
task-depends.dot files before and after the change. There should be no
functionality changes in this patch, its purely a data structure change
and that is visible in the patch.
(Bitbake rev: 2c88afb60da54e58f555411a7bd7b006b0c29306)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Fixed:
DeprecationWarning: The 'warn' method is deprecated, use 'warning' instead
(Bitbake rev: a3f464d202dafef4538e66c008cdecb7b8709ed1)
Signed-off-by: Robert Yang <liezhi.yang@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Various misc changes to convert bitbake to python3 which don't warrant
separation into separate commits.
(Bitbake rev: d0f904d407f57998419bd9c305ce53e5eaa36b24)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Under python the type conversions can mean there are float values
used for triggering the parse progress events which then fails.
Add an explict int() conversion to ensure the parse events are
generated under python3.
(Bitbake rev: 138329c58e92744c56aae3ab70ceeef09613250c)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Use open() instead of file() and close files when finished with them.
(Bitbake rev: 033c5a16ff19781ed793c2d97d285884017a2a4e)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
It's useful to see tracebacks for ExpansionErrors, but only if we skip the
leading bitbake-internal elements, otherwise we see elements of the expansion
process.
As one example:
Before:
ERROR: ExpansionError during parsing /scratch/yocto-new/external-as-needed/poky/meta/recipes-core/glibc/glibc-locale_2.23.bb: Failure expanding variable PV[:=], expression was ${@get_external_libc_version(d)} which triggered exception AttributeError: 'module' object has no attribute 'external'
After:
ERROR: ExpansionError during parsing /scratch/yocto-new/external-as-needed/poky/meta/recipes-core/glibc/glibc-locale_2.23.bb
Traceback (most recent call last):
File "PV[:=]", line 1, in <module>
File "/scratch/yocto-new/external-as-needed/meta-sourcery/recipes-external/glibc/glibc-external-version.inc", line 3, in get_external_libc_version(d=<bb.data_smart.DataSmart
object at 0x7f05d2566950>):
sopattern = os.path.join(d.getVar('base_libdir', True), 'libc-*.so')
> found_paths = oe.external.find_sysroot_files([sopattern], d)
if found_paths:
ExpansionError: Failure expanding variable PV[:=], expression was ${@get_external_libc_version(d)} which triggered exception AttributeError: 'module' object has no attribute 'external'
(Bitbake rev: 7ff5b9eed82b7f4fd138fc6d746a0b79efbea98a)
Signed-off-by: Christopher Larson <chris_larson@mentor.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Now the event is sent unconditionally we can drop this feature
as its no longer needed.
(Bitbake rev: 473deeb0fc6065693e1fcfcbb8b79753103db537)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Ensure we pass the string parameter correctly.
(Bitbake rev: 7ed82bd1fe7bdd93b0614119c42eb218dc5d83e6)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
This avoids a common issue where PACKAGECONFIG is emitted as a function in
bitbake -e when the 'python' flag exists. It isn't a python function unless
both 'func' and 'python' are set. This aligns with the behavior of
emit_func_python.
(Bitbake rev: c5e45063cb3ae17bbe3304ea5e712bd76e686c4a)
Signed-off-by: Christopher Larson <chris_larson@mentor.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
We now prefix log messages coming from worker task context with the
PF and task info, however parsing messages all have to be manually
prefixed which is ugly and error prone. This change modifies the log
handler filter so this happens automatically, meaning we don't have
to change every message to include that information. This makes error
messages longer but more usable.
(Bitbake rev: 1af0ccaac81e182c4ca520037dda362d180e5605)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
An invalid task causes bitbake to exit incorrectly, firing a
CommandCompleted event rather than a CommandFailed one. This
means that clients listening for CommandFailed events are
unable to detect the build failure even though one occurred.
Passing an exception string to finishAsyncCommand when a task
fails causes the CommandFailed event to be fired correctly.
[YOCTO #9087]
(Bitbake rev: 98a2c37e077b16e3bc8bb102bd18b293130d15a4)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Prevent a hang when shutdown() is called during parsing (e.g. after
SIGINT). We must not append 'None' to the jobs queue. Otherwise the
worker loop inside Parser.realrun() may break out at the wrong point,
causing the results queue thread blocking bitbake indefinitely.
[YOCTO #9319]
(Bitbake rev: 7ebea3e9a60232222efa8a546a0ff28a53029949)
Signed-off-by: Markus Lehtonen <markus.lehtonen@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Currently bbappend files in a layer are applied in the order they're
found on disk (as reported by glob) which means things are not
deterministic.
By sorting the glob results, the order becomes deterministic, the parsing
order for .bb files also should be deterministic as a result of this change.
[YOCTO #9138]
(Bitbake rev: 3f8febc4212fbd3485ac9bdd4ac71b8fb0a05693)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Drop unused 'd' argument from the cache save methods, simplifying the
API.
(Bitbake rev: 81bc1f20662c39ee8db1da45b1e8c7eb64abacf3)
Signed-off-by: Markus Lehtonen <markus.lehtonen@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
We're seeing issues where the self test, which uses tinfoil doesn't
notice the changed contents of include files. The issue is
cached_statements in the parser being reused when the files have changed.
Whilst looking at this, I realised there were some other issues:
* We need to also invalidate the mtime cache when cooker restarts
* We should pass full filenames to the file invalidation code
* We should process cached_statements as part of inotify invalidation
With these fixes, the caching is more reliable for memory resident
bitbake too. It does raise some questions about cache validation and
lifecycles and indicates bitbake does need more work in the area,
preferably with the removal of the globals. This at least highlights
and works around some of the current issues.
(Bitbake rev: 3f507ff8bc467fba936cf3f31bb8ea8e02f168e8)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
CookerParser.shutdown code doesn't do all required work to shutdown
parser processes. As a result bitbake hangs if interrupted during
parsing. Putting None into the parser_quit queue should fix this issue
as it makes parsers to quit main loop.
(Bitbake rev: f67307977e8f089ce6d208d3e9de2a6a1768757e)
Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Right now, if you have some python code like:
X = "a"
def somefunction(d):
d.setVar("X", "b")
d.setVar("Y", "${X}")
then any sane person would expect that Y = "b" at the end of the
function. This is not the case, Y = "a".
This is due to the python function being expanded before execution, the
executed code would read d.setVar("Y", "a"). This understandably
confuses people, it also makes it near impossible to write ${} in a
python function without unintended things happening.
I think there is general agreement we should fix this and standardise
on non-expansion of python functions. We already don't expand anonymous
python (mostly).
I've checked OE-Core with buildhistory before and after this change and
there were a small number of issues this exposed which I've sent
patches for.
I propose we default to not expanding python code and then deal with
any consequences from that if/as/where identified. This will improve
new user understanding and usability of the system, it also allows
several long standing weird expansion issues to be fixed.
(Bitbake rev: 8bf33a8e92c0e188fa392030025756196c96fcbb)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
We don't want to expand python functions since they aren't expanded
at execution time (e.g. anonymous python). They can also have side
effects.
This function is primarily used by toaster for variable dumps for later
display. The lack of expansion of python functions won't matter in this case
and actively helps some variable handling (e.g. SRCPV).
(Bitbake rev: 3f5520b4844a4bdd615046479ba08ed192bdc8cd)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Extended the dot styling of dependencies created by bitbake -g in dot syntax to differentiate between the various kinds.
depends: solid
rdepends: dashed
rrecommends: dotted
The change observed is that depends get an explicit style which is the same as dot default behavior and the runtime recommends get
dotted while before they were dashed. This helps to distinguish them graphically as well as eases post processing by script.
(Bitbake rev: 86e78e0ca7aa5452411f35239942ecee3d8824ec)
Signed-off-by: Henning Schroeder <henning.schroeder@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Currently any not catched exception in cooker causes bitbake
to hang because of not terminated children of CookerParser.
Long term solution would be to reimplement Cooker as a context
manager and terminate parser children in its __exit__ method.
Partial fix is to call CookerParser.shutdown in Cooker.shutdown in
hope that all Cooker exceptions are caught and shutdown method is
called.
[YOCTO #8900]
(Bitbake rev: 3f67600dc3292bc8208644ce89e8bf7ab95cf2e7)
Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
I before E, except after C...
(Bitbake rev: 14c9593265f7469cb8a205a46f845ac7491246df)
Signed-off-by: Phil Blundell <pb@pbcl.net>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
If we're only going to parse one recipe, no point in starting
a large number of threads.
(Bitbake rev: b977faf59dc08050a44a16032fe52d1bbb80f2a1)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
This means that when you view the process tree, the processes
have meaningful names, aiding debugging:
$ pstree -p 30021
bash(30021)───KnottyUI(115579)───Cooker(115590)─┬─PRServ(115592)───{PRServ Handler}(115593)
├─Worker(115630)───bash:sleep(115631)───run.do_sleep.11(115633)───sleep(115634)
└─{ProcessEQueue}(115591)
$ pstree -p 30021
bash(30021)───KnottyUI(117319)───Cooker(117330)─┬─Cooker(117335)
├─PRServ(117332)───{PRServ Handler}(117333)
├─Parser-1:2(117336)
└─{ProcessEQueue}(117331)
Applies to parse threads, PR Server, cooker, the workers and execution
threads, working within the 16 character limit as best we can.
Needed to tweak the bitbake-worker magic values to tell the
workers apart.
(Bitbake rev: 539726a3b2202249a3f148d99e08909cb61902a5)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Before, BBMASK was only permitted to contain one regular expression.
This made it hard to add to the BBMASK in multiple places as one was
supposed to separate the different regular expressions with a "|"
rather than with whitespace as is customary in BitBake variables.
Now one can specify any number of regular expressions in BBMASK. This
makes it possible to, e.g., mask out recipes in another layer from the
layer.conf file.
This also properly ignores any regular expressions that do not compile
(before an invalid regular expression would cause a ParseError in the
first bbappend file found stating that it was not a BitBake file...)
(Bitbake rev: 2c778ad50aceaffb855baf5f4aa0fed98c880870)
Signed-off-by: Peter Kjellerstedt <peter.kjellerstedt@axis.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
In certain circumstances it can be useful to get access to the world
targets list from a recipe in order to add dependencies on some or all
of the items in it. If a special function, 'calculate_extra_depends' is
defined in the recipe, and the recipe is to be built, then call it at
the right point before we calculate which tasks should be run. The
function can append items to the "deps" list in order to add
dependencies. This is not as tidy a solution as I would have liked, but
it does at least do the job.
As part of this change, the buildWorldTargets function was moved to
bb.providers to make it possible to call from taskdata.
Part of the implementation of [YOCTO #8600].
(Bitbake rev: aba0dce57c889495ec5c13919991a060aeff65d2)
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Added providermap information to the result of buildDependTree API.
This will be used by Toaster to map virtual dependencies to recipes.
(Bitbake rev: d3e07368549f30265f59846a260efa8230a225ca)
Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com>
Signed-off-by: Elliot Smith <elliot.smith@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The documentation for findFilesMatchingInDir() was inconsistant with the
implementation: the regex was escaped before searching so effectively it's a
pure textual substring, and the machine example was broken.
(Bitbake rev: 6bef981488ec94b46dbe3797acfecf9c4b6ecbbc)
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
No point counting all instances when we just want to know if there's any or not.
(Bitbake rev: 3369072efb653339da8dbd1ca864ff8e1ff899ca)
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Added method to convert state code into the human readable name.
It will be used in logging and error reporting.
(Bitbake rev: 9ec6379b27d210214d0b3f2e55962f721b7f5f51)
Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The SyntaxError exception simply shows the recipe that failed to parse
which is pretty useless without the actual exception. We could make it
print more info, however we can just use one of the more generic handlers
instead and remove this one.
For a python indentation error, this leads to a much more readable error
message.
(Bitbake rev: 9241eb10847634e34c5ff8767ed8c114f66ff6cf)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The console log data is written to is created at console initialisation
time and does not change over reset events. This ensures the
BB_CONSOLELOG value is correct over such resets by preserving it.
(Bitbake rev: 335eb2db228f7543a49de71f063ac72b865c947a)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Additional config files passed to bitbake server with --read and
--postread options are rewritten by client bitbake even if
it doesn't use those options.
This is a show stopper for toaster as toaster command line
builds are based on the assumption that server is aware of
toster configs, provided by --postread option.
This behaviour is fixed by preserving values of --read and
--postread options when bitbake server starts and restoring
them if client bitbake doesn't explicitly specify them.
(Bitbake rev: 02c64f7487ca8ec5d32c440f5002c4b8f64b76a6)
Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
BuildStarted event not fully represents build tasks for
the targets. If -c option is used to specify default task
it's not included into the event.
Made build targets to always look as <target>:do_<task>.
Consider default task (do_build or specified by -c command
line option) when normalizing.
(Bitbake rev: 0b0e214e6f53c97ad3d48f622c7fc0ca149956f6)
Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The output of "bitbake, -c listtasks pkg" lists tasks with their real names
(starting with "do_"), but then "bitbake -c do_task" fails, as "do_" always
gets unconditionally prepended to task names. This patch handles this error
by checking whether a task starts with "do_" prior to prepending it with it
when the task runlist is being constructed (and a few other corner cases).
[YOCTO #7818]
(Bitbake rev: dd3050ceef37ac556546e940aa596ce96ef6c8df)
Signed-off-by: Alex Franco <alejandro.franco@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Allow any listeners for this event (such as buildhistory.bbclass in
OpenEmbedded) to find out if the build was interrupted rather than
completing normally. The value will be 0 if not interrupted, 1 if
interrupted waiting for remaining tasks to complete, or 2 if force
interrupted (stopping any running tasks immediately).
(Bitbake rev: df2b778efd2ecc48f6c5a3ed446f6459f2250035)
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Now we have the bbappends list and all users have been converted over to
use it, we don't need this anymore.
(Bitbake rev: 279770c42d4c63aa2cebce331b55a92a564b50ac)
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Whilst collecting this list on the fly may be quicker, doing so within a
function that's meant to *query* the list of bbappends is poor practice.
(Bitbake rev: 5c12aa3b0010d7d1733e54a0ca7d0af465454210)
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The prefile/postfile options weren't working in memory resident mode
because they weren't being passed through to the server, so ensure that
they do get passed through and that the server is reset when the values
come through.
(Bitbake rev: a3f7dc042fc7b1c308bfd248431930eb8ba50326)
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Add a means of ensuring that synchronous commands that read the results
of the configuration trigger a reparse of the configuration if any
underlying files have changed.
(Bitbake rev: aaf3cc024315450c1674819edf2a4e5cdf293f35)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Unfortunately we were acting on inotify notifications about any files
changing within the watched directories, not just the ones we actually
care about. In OE the build directory is in BBPATH and hence it gets
watched, and we write things like bitbake.lock and
bitbake-cookerdaemon.log to that directory, hence effectively
notifications were being tripped on every bitbake invocation. To avoid
this, record which file/subdirectory we're interested in against each
watched directory so we can ignore any events for files/subdirectories
we don't care about.
Additionally, if we move up to the parent dir, ensure we haven't already
seen it before adding a watch on it (we were previously calling
watcher.add_watch() on the same directory multiple times before in a
typical OE configuration).
(Bitbake rev: bc39b8da34c046b629c43fd0a8cac2efbf1c060f)
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
With python you should not assign a list as the default value of a
function parameter - because a list is mutable, the result will be that
the first time a value is passed it will actually modify the default.
Reference:
http://docs.python-guide.org/en/latest/writing/gotchas/#mutable-default-arguments
(Bitbake rev: 7859f7388f2e3f675d0e1527cfde18625f36f637)
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
If you load memory resident bitbake, then change bblayers.conf or other
configuration files, then execute your first command, bitbake will not
notice the changes. This is because it adds the file watches during
inital parsing, which happens at the time the first command is run.
To fix this, we move the addition of the file watches to earlier in
the process, so bitbake then does track them correctly. This also
avoids some issues Paul was seeing with tinfoil2.
(Bitbake rev: b9375c73e736003e66575969c9ea244403e47aeb)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
If the PR server or indeed any other child process takes some time to
exit (which it sometimes does when saving its database), it can end up
holding bitbake.lock after the UI exits, which led to errors if you ran
bitbake commands successively - we saw this when running the PR server
oe-selftest tests in OE-Core. The recent attempt to fix this wasn't
quite right and ended up breaking memory resident bitbake. This time we
close the lock file when cooker shuts down (inside the UI process)
instead of unlocking it, and this is done in the cooker code rather than
the actual UI code so it doesn't matter which UI is in use. Additionally
we report that we're waiting for the lock to be released, using lsof or
fuser if available to list the processes with the lock open.
The 'magic' in the locking is due to all spawned subprocesses of bitbake
holding an open file descriptor to the bitbake.lock. It is automatically
unlocked when all those fds close the file (as all the processes terminate).
We close the UI copy of the lock explicitly, then close the server process
copy, any remaining open copy is therefore some proess exiting.
(The reproducer for the problem is to set PRSERV_HOST = "localhost:0"
and add a call to time.sleep(20) after self.server_close() in
lib/prserv/serv.py, then run "bitbake -p; bitbake -p" ).
Cleanup work done by Paul Eggleton <paul.eggleton@linux.intel.com>.
This reverts bitbake commit 69ecd15aece54753154950c55d7af42f85ad8606 and
e97a9f1528d77503b5c93e48e3de9933fbb9f3cd.
(Bitbake rev: a29780bd43f74b7326fe788dbd65177b86806fcf)
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
If you have a situation where you are getting a "Nothing PROVIDES" error
(for example when something you request to build DEPENDS on something
that has been skipped or doesn't exist) it would be useful to be able to
use bitbake -g or bitbake -e to debug it, but currently both of those
are blocked by the error.
This patch adds an "allowincomplete" option to taskdata and uses this
for the -e/-g bitbake options. The NoProvider errors are still
printed and bitbake does return an error exist code but the environment
and task graph files are generated.
[YOCTO #7623]
(Bitbake rev: 2ca36a9f088438a8d1db44119c704f9480b04298)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Currently, once set, DATE and TIME in the metadata remain unchanged.
This is suboptimal for cases where cooker is reused. This ties setting
the variables into the BuildStarted event which seems like a more
appropriate time to do so.
It also changes BUILDNAME to be based off DATE/TIME by default if not
already set so that the data is more consistent. We therefore need to
expand the value rather than the previous default of not doing so.
This change does mean the date/time values are in sync across all
variables too.
It does mean bitbake now has special knowledge of DATE/TIME but that
would seen unavoidable (other than doing this in event handlers which
has its own set of downsides).
[YOCTO #5187]
(Bitbake rev: f883cf240266ee7be2cbd8971a8164cf4df9e372)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Some handlers hook on BuildComplete so it avoids certain event races
to finish the command after the BuildComplete event is sent out.
This means the UI is available to handle events until the command
completes.
What appears to be a race on one of the sanity tests for event handlers
triggered this change although the failure is hard to reproduce.
[YOCTO #7921]
(Bitbake rev: e42d7c47a06fbb5981e0313478c8e3656b99f4e7)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Rather than just use d.getVar(X), use the more explict d.getVar(X, False)
since at some point in the future, having the default of expansion would
be nice. This is the first step towards that.
This patch was mostly made using the command:
sed -e 's:\(getVar([^,()]*\)\s*):\1, False):g' -i `grep -ril getVar *`
(Bitbake rev: 659ef95c9b8aced3c4ded81c48bcc0fbde4d429f)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The metadata can potentially use such an event to clean up any
"unreachable" data, solving several problems we currently have
where obsolete data may continue to exist in the shared areas.
(Bitbake rev: c5e6f929f3d5eeb7954660dea62611c58b795ff8)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Currently the cooker parsing processes each dump an individual profile
which is ok, but means absolute numbers of function calls for a given load
can be tricky to determine as parsing of recipes may go to different pool
threads on different runs.
This change collects up the individual thread parsing results and processes
them into one profile output. The profile processing function in utils
needed tweaks to allow this to work.
(Bitbake rev: d3d2541aacd1ea560da0d8b25a3ea3f0563dee70)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Introduced build mtime cache structure. Reset it before the build
to prevent bitbake from crashing when build/tmp/stamps hierarchy
is removed.
[YOCTO: #7562]
(Bitbake rev: f8590547a198a78334debdf14bf40acb50c22ecc)
Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
self.appendlist is a dict and as such unordered. This can lead to cases
where appends with different names (e.g. x_%.bbappend vs. x_123.bbappend)
can be reordered in application which in turn reorders the variables
that those bbappend files might touch. Reorderd variables changes the sstate
cache signatures causing real world issues.
To avoid this, use a list for the append files instead.
This patch is conservative and just adds a new data structure alongside
the existing one and uses it to resolve the core issue. Later patches
(post release) can handle some of the wider but less problematic ones
(e.g. issues in bitbake-layers flatten).
[YOCTO #7511]
(Bitbake rev: f980f060cd0d1e7fe5011f3c325c1b254f05eccf)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The recent inotify changes are causing a 100% cpu usage issue in the
idle handlers. To avoid this, we update the idle functions to optionally
report a float value which is the delay before the function needs to be
called again. 1 second is fine for the inotify handler, in reality its
more like 0.1s due to the default idle function sleep.
This reverts performance regressions of 1.5 minutes on a kernel build
and ~5-6 minutes on a image from scratch.
(Bitbake rev: 0e0ba408c2dce14a0fabd3fdf61d8465a031495b)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
This patch fixes a bug where if the build is force stopped,
subsequent clients cannot connect to the server due to
unnecessary limits on setFeature.
Additionally, we make sure that the state is properly reset
even if the BuildCompleted event firing excepts for some reason.
(Bitbake rev: 0b66b05169688aa4ddc4c54d175bb961b2f27fec)
Signed-off-by: Alexandru DAMIAN <alexandru.damian@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The inotify facility monitoring changes to the config files
could be overwhelmed by massive changes to the watched files
while server is running.
This patch adds verification the notification watches to the
server idle functions, in addition to the cooker updateCache
command which executes only infrequently, thus preventing
overflowing the notification buffer.
[YOCTO #7316]
(Bitbake rev: 996e663fd5c254292f44eca46f5fdc95af897f98)
Signed-off-by: Alexandru DAMIAN <alexandru.damian@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
We've had versioned dependency support in LAYERDEPENDS for quite a long
time, but I can say with pretty good certainty that almost nobody has
used it up to now because it was too strict - the specified version had
to exactly match the version in your configuration or you would get an
error; there was no "greater than or equal" option, which is usually
what you will want given that LAYERVERSION does get bumped from time to
time.
However, users mismatching layer branches and then having their builds
fail later on with some incomprehensible error is still a pretty common
problem. We can't simply use the git branch because not everyone is
always on a branch and the branch names don't always match up (and
that's not an issue). To provide a practical means to address branch
mismatching, I have reworked LAYERDEPENDS version specifications to use
the more familiar "dependency (>= version)" syntax as used with package
dependencies, support non-integer versions, and clarified the error
message a little. If we then take care to bump the version on every
breaking change, it is at least possible to have layers depend on these
changes when they update to match; we can now even support a major.minor
scheme to allow retrospectively adding a version limiter to old branches
when a new branch is created and yet still allow the old branch minor
version to be bumped if needed.
Fixes [YOCTO #5991].
(Bitbake rev: 408be9cdf2b1e32e64ea488d8051a546fb54c144)
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Users expect operators like _remove to work on this variable. We need
to use expanded_data to ensure this happens correctly.
[YOCTO #7135]
(Bitbake rev: cc4c8478fc547ea0ebf827a8d319496b39f25684)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
event_data would be better named expanded_data, then we can widen its scope
to other places in cooker where we need to access an expanded data store.
We certainly don't want multiple expanded data stores.
(Bitbake rev: 1a3c1c9203e1a1452314954f1cfd771e5c1ce89b)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Trap the ENOSPC error and translate it into a human readable error
message, which is good for debugging.
(Bitbake rev: 2b084dff6ff0d274fbbf7ab07022507f7249e427)
Signed-off-by: Robert Yang <liezhi.yang@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
We currently add crazy numbers of watches on files. The per user limit is 8192
by default and on a system handling multiple builds, this can be an issue.
We don't need to watch all files individually, we can watch the directory containing
the file instead. This gives better resource utilisation and better performance
further reverting some of the performance regression seen with the introduction
of pyinotify.
(Bitbake rev: a2d441237916a99405b800c1a3dc39f860100a8c)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
If the environment changes, we need memory resident bitbake to adapt to those
changes. This adds in functionality to handle this alongside the configuration
option handling code. This means that the common usage:
MACHINE=X bitbake Y
now works with the memory resident server.
(Bitbake rev: 4d1343010da757a0c126bc22475354da44aaf8e3)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
