Commit Graph

129 Commits

Author SHA1 Message Date
Richard Purdie
ad09c355d1 bitbake: command: Fix command complete event ordering
The command complete event triggers the UI to start new commands.
We should send this once we've cleaned up existing commands since
on very slow systems, this could otherwise cause failures. There
is a 30s timeout on incomming commands but slow systems have been
known to hit this on poorly configured I setups.

(Bitbake rev: d202298b3d989c39bc532b69c410e2e78b082a11)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2025-03-11 10:53:13 +00:00
Chris Laplante
9d1b31d254 bitbake: cooker: Make cooker 'skiplist' per-multiconfig/mc
Previously, the cooker skiplist was shared across multiconfigs
(including default ''). If you had a recipe that was incompatible with
several multiconfigs for different reasons, then the displayed reason
(i.e. the "ERROR: Nothing PROVIDES" and "* was skipped" messages) might
vary across invocations of bitbake. This was caused by the random order
in which recipes are parsed under different multiconfig contexts, with
each skip reason overwriting the previously assigned reason.

I hit this specificially when using COMPATIBLE_MACHINE, but
COMPATIBLE_HOST (or anything using bb.parse.SkipRecipe) would have done it too.

(Bitbake rev: c51f01a35ed9a928402eab0899598b5c59602eef)

Signed-off-by: Chris Laplante <chris.laplante@agilent.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2024-12-17 11:48:25 +00:00
Chris Laplante
9cabda06ab bitbake: cooker: use enum for cooker state to improve readability
enum was introduced in Python 3.4

(Bitbake rev: 35b71a94f8757fcca830f972a42edab1dd000c16)

Signed-off-by: Chris Laplante <chris.laplante@agilent.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2024-12-06 23:33:54 +00:00
Yoann Congal
30a3ac4a2e bitbake: tinfoil: add new "finalizeData" API
Create a new tinfoil API "finalizeData" which does key expantion and
runs the anonymous functions to allow bitbake-getvar to have the same
output as "bitbake -e".

(Bitbake rev: 63db64ca98564420a378f14c337074ae06da8c63)

Signed-off-by: Yoann Congal <yoann.congal@smile.fr>
Signed-off-by: Mathieu Dubois-Briand <mathieu.dubois-briand@bootlin.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2024-11-18 13:41:38 +00:00
Yoann Congal
0317315a97 bitbake: command: fix needconfig for revalidateCaches
Fixes what looked like a type since parseConfiguration.needconfig is
already set just after parseConfiguration() definition.

(Bitbake rev: d371df029ece3b9e600a89d08337c437a8ddbf63)

Signed-off-by: Yoann Congal <yoann.congal@smile.fr>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2024-11-07 22:17:43 +00:00
Richard Purdie
859786a83f bitbake: siggen: Ensure version of siggen is verified
Since we need to change the form of the siggen function, we need to add versioning
and some verison checks. This means if a newer bitbake is used with older metadata
we can detect it.

(Bitbake rev: 721556568413508213d22c29985e305a45a8d68a)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2024-01-05 11:59:08 +00:00
Peter Kjellerstedt
370de0fa91 bitbake: command: Make parseRecipeFile() handle virtual recipes correctly
Running `devtool search gcc` would result in errors like this for
virtual recipes:

  ERROR: When reparsing
  .../meta/recipes-devtools/gcc/libgcc-initial_13.2.bb:do_populate_sysroot,
  the basehash value changed from
  b1cd809ed98cef9db0fb1b17d34c4083e739c336f9d5619b89715b0294d81af5 to
  44c2f92781dc4a20e98b7bb4724e204e64b101905fa75e71241a574b725997dc. The
  metadata is not deterministic and this needs to be fixed.
  ERROR: The following commands may help:
  ERROR: $ bitbake libgcc-initial -cdo_populate_sysroot -Snone
  ERROR: Then:
  ERROR: $ bitbake libgcc-initial -cdo_populate_sysroot -Sprintdiff

The reason was the newly introduced :layer-<layername> override, which
is used, e.g., in meta/classes-global/insane.bbclass to add the
patch-status QA test only for the meta layer:

  ERROR_QA:append:layer-core = " patch-status"

When tinfoil parsed the recipes using the parseRecipeFile() function, it
failed to properly identify the layername for virtual recipes, which
resulted in the error above.

The correct thing to do is to make parseRecipeFile() call
bb.cache.virtualfn2realfn() to convert the virtual filename into a real
filename and virtual class.

(Bitbake rev: da2aed134412f5de04d7b540f92d735983ad0108)

Signed-off-by: Peter Kjellerstedt <peter.kjellerstedt@axis.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2023-11-23 12:06:06 +00:00
Julien Stephan
8eeb58cf4a bitbake: bitbake: cooker: add a new function to retrieve task signatures
adding a new command in cooker to compute and get task signatures

this commit also add the associated command and event needed to get the
signatures using tinfoil

(Bitbake rev: 05c15162de90c41dad67e37a95ec9fdb440a7864)

Signed-off-by: Julien Stephan <jstephan@baylibre.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2023-09-28 12:38:26 +01:00
Richard Purdie
37c31a5adc bitbake: lib: Drop inotify support and replace with mtime checks
With the flush in serverlog() removed and a memory resident bitbake with a
60s timeout, the following could fail in strange ways:

rm bitbake-cookerdaemon.log
bitbake-layers add-layer ../meta-virtualization/
bitbake-layers add-layer ../meta-openembedded/meta-oe/
bitbake -m

specifically that it might error adding meta-oe with an error related to meta-virt.

This clearly shows that whilst bblayers.conf was modified, bitbake was not
recognising that. This would fit with the random autobuilder issues seen when
the serverlog flush() call was removed.

The issue appears to be that you have no way to "sync()" the inotify events with
the command stream coming over the socket. There is no way to know if there are
changes in the IO queue which bitbake needs to wait for before proceeding with
the next command.

I did experiment with os.sync() and fsync on the inotify fd, however nothing
addressed the issue. Since it is extremely important we have accurate cache data,
the only realistic thing to do is to switch to stat() calls and check mtime.

For bitbake commands, this is straightforward since we can revalidate the cache
upon new connections/commands. For tinfoil this is problematic and we need to
introduce and explict command "revalidateCaches" that the code can use to force
bitbake to re-check it's cache validity. I've exposed this through tinfoil with
a new "modified_files" function.

So, this patch:

a) drops inotify support within bitbake's cooker/server and switch to using mtime
b) requires a new function call in tinfoil when metadata has been modified

(Bitbake rev: da3ec3801bdb80180b3f1ac24edb27a698415ff7)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2023-09-18 11:35:05 +01:00
Richard Purdie
3bdabde2df bitbake: command: Avoid time intensive distractions for ping
We noticed some users were seeing very slow ping response times which
caused 'server timeout' issues. There were some function calls in runCommand
which could be slow such as the inotify callback processing.

Mark up the ping command such that it doesn't need configuration information,
it is allowed on a readonly server and specifically skip the inofity processing
too since ping would never need that.

This will hopefully resolve various ping timeout issues that were being reported.

(Bitbake rev: 0fc821a22f2b49cbd336d9658d92942c0d733be1)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2023-09-02 07:46:40 +01:00
Richard Purdie
695998f921 bitbake: cooker: Add FILE_LAYERNAME variable containing the layername for a recipe
There are times when it would be useful for code to know which layer
(or collection in old bitbake terms) it is contained within.

Add support for FILE_LAYERNAME to be set by bitbake when parsing a recipe
so that it is possible to determine this. To do it, we need to pass data
from the cooker into the recipe endpoints, since only the top level cooker
information knows about the layer structure which makes the patch a bit
painful.

The idea is that this would make layer overrides possible:

OVERRIDES .= ":layer-${FILE_LAYERNAME}"

which then opens possibilities like:

WARN_QA:append:layer-core = " patch-fuzz"

as an example where OE-Core could enable specific QA tests only for that
specific layer.

(Bitbake rev: 7090a14b0035842112d073acf7f2ed1a01fdeccf)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2023-05-25 13:16:24 +01:00
Richard Purdie
165e8b563d bitbake: process/cooker/command: Fix currentAsyncCommand locking/races
currentAsyncCommand currently doesn't have any locking and we have
a conflict in "idle" conditions since the idle functions count needs
to be zero *and* there needs to be no active command.

Move the changes/checks of currentAsyncCommand to within the lock
and then we can add it to the condition for idle, simplifying some
of the code.

(Bitbake rev: b5215887d2f8ea3f28f1ebda721bd5b8f93ec7f3)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2023-01-11 10:58:36 +00:00
Richard Purdie
4e4f040a73 bitbake: cooker/command: Drop async command handler indirection via cooker
Indirecting the async command handler via cooker is confusing and no
longer needed. Drop it to make things slightly clearer.

(Bitbake rev: 4a41a7d0594e6a84a67b9de70a505858aebcd84a)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2023-01-11 10:58:36 +00:00
Richard Purdie
82e77db0ce bitbake: command: Ensure that failure cases call finishAsyncComand
There are a couple of failure cases that runCommands needs to better
handle, ensuring finishAsyncCommand is called. This doesn't matter if the
server is about to shut down but for memory resident bitbake and with
threading enabled, correctness is more important this could could in
theory lead to problems with the recent code changes.

By using the idleFinish class, it ensures the current async command
is terminated correctly and the various state pieces around commands
finishing are called. This also makes the code more uniform matching
the other exception handling code.

(Bitbake rev: 367a83ed46c7fbcdd06579b6cc3a6e48a89ca7fb)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2023-01-02 17:32:57 +00:00
Richard Purdie
4c57c6eeec bitbake: server/process: Run idle commands in a separate idle thread
When bitbake is off running heavier "idle" commands, it doesn't service it's
command socket which means stopping/interrupting it is hard. It also means we
can't "ping" from the UI to know if it is still alive.

For those reasons, split idle command execution into it's own thread.

The commands are generally already self containted so this is easier than
expected. We do have to be careful to only handle inotify poll() from a single
thread at a time. It also means we always have to use a thread lock when sending
events since both the idle thread and the command thread may generate log messages
(and hence events). The patch depends on  previous fixes to the builtins locking
in event.py and the heartbeat enable/disable changes as well as other locking
additions.

We use a condition to signal from the idle thread when other sections of code
can continue, thanks to Joshua Watt for the review and tweaks squashed into this
patch. We do have some sync points where we need to ensure any currently executing
commands have finished before we can start a new async command for example.

(Bitbake rev: 67dd9a5e84811df8869a82da6a37a41ee8fe94e2)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2022-12-31 17:05:17 +00:00
Richard Purdie
c4ecfc4dc5 bitbake: server/process: Improve idle loop exit code
When idle handlers want to exit, returning "False" isn't very clear
and also causes challenges with the ordering of the removing the idle
handler and marking that no async command is running.

Use a specific class to signal the exit condition allowing clearer code
and allowing the async command to be cleared after the handler has been
removed, reducing any opportunity for races.

(Bitbake rev: 102e8d0d4c5c0dd8c7ba09ad26589deec77e4308)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2022-12-31 17:05:17 +00:00
Richard Purdie
4895e78234 bitbake: command: Tweak finishAsyncCommand ordering to avoid races
(Bitbake rev: 89435442946767cfe58eedde363802add8f1ab29)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2022-12-31 17:05:17 +00:00
Richard Purdie
26f255da09 bitbake: command: Add ping command
Add a simple ping command so the UI can check the server is still there.

(Bitbake rev: fd3359de0b9f18fac187a629df203be0b2c87545)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2022-12-21 14:15:26 +00:00
Chen Qi
a5f1b44310 bitbake: command.py: cleanup bb.cache.parse_recipe
The parse_recipe has been removed from bb.cache, replace it with
the databuilder._parse_recipe.

(Bitbake rev: d386fa81848247a3d407debf889db8cbcce03359)

Signed-off-by: Chen Qi <Qi.Chen@windriver.com>
Signed-off-by: Alexandre Belloni <alexandre.belloni@bootlin.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2022-11-27 23:48:15 +00:00
Richard Purdie
3a4aeda0fa bitbake: cache/cookerdata: Move recipe parsing functions from cache to databuilder
When 'NoCache' was written, databuilder/cookerdata didn't exist. It does
now and the recipe parsing functionality contained in NoCache clearly
belongs there, it isn't a cache function. Move those functions, renaming
to match the style in databuilder but otherwise not changing functionality
for now. Fix up the callers to match (which make it clear this is the right
move).

(Bitbake rev: 783879319c6a4cf3639fcbf763b964e42f602eca)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2022-11-20 08:31:28 +00:00
Richard Purdie
2446bdf59a bitbake: cooker/command: Add a dummy event for tinfoil testing
We need a command genetating an event to test through the tinfoil API. The
current test has IO load issues so add a dummy version which won't have
the IO constraints.

(Bitbake rev: a144178584394ea0700ffc2f2bfac94e8690effc)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2021-09-26 16:35:00 +01:00
Richard Purdie
5a4501abeb bitbake: command: Ensure we catch/handle exceptions
If an exception occurs in early setup, bitbake could just hang. Return
the exception rather than doing that.

[YOCTO #14408]

(Bitbake rev: c8a4107132ce51f84ae84bf1ceb1c3fd90f156d3)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2021-08-06 06:34:58 +01:00
Richard Purdie
d08d362cf9 bitbake: tinfoil/data_smart: Allow variable history emit() to function remotely
We can't access the emit() function of varhistory currently as the datastore parameter
isn't handled correctly, nor is the output stream. Add a custom wrapper for this
function which handles the two details correctly.

(Bitbake rev: ba0fa084ccd2b1ade96425d158fd31e49e42f286)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2021-04-18 11:38:22 +01:00
Richard Purdie
20397f8d4b bitbake: command: Ensure exceptions inheriting from BBHandledException are visible
Previous changes allowed BBHandledException to be detected but not exceptions
which inherit from it. Fix this. The code really needs totally reworking
to preserve the exceptions.

[YOCTO #14054]

(Bitbake rev: ef762d92df6c2554c6248e80212f984d9ec4c651)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2020-10-10 13:37:56 +01:00
Richard Purdie
79adf16931 bitbake: cooker/command: Fix disconnection handling
After the recent init changes, if a client disconnects before issuing a
command, the cooker can break in the reset handlers. Add some guards
in the code to prevent this.

(Bitbake rev: 12605e30e4c4e1ae6a67c97363b892ebf0b9566c)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2020-09-12 15:29:56 +01:00
Richard Purdie
0bcc00ac51 bitbake: cooker: Defer configuration init to after UI connection
Currently we end up parsing the base configuration multiple times as
initially, the right settings haven't come from the UI. We can defer
this until later in startup using runCommand as a trigger.

The advantage to doing this is improved startup times and ultimately
we should be able to avoid the double parse of the base configuration.

(Bitbake rev: 3caa43b665604475d2c87ba505efb0b9fca9c2e9)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2020-08-25 18:14:53 +01:00
Richard Purdie
a3448ad15e bitbake: server/process: Simplfy idle callback handler function
Having the idle callbacks abstracted via the configuration object
makes no sense. Its like this for historical reasons from the
multiple server backends but we don't need this now so simplfy.

(Bitbake rev: e56c49717355c9493b07d5fc80981a95ad8a0ec8)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2020-08-13 08:07:00 +01:00
Joshua Watt
6c2bb4a4ad bitbake: bitbake: command: Handle multiconfig in findSigInfo
Changes the findSigInfo command to accept a recipe specified with the
multiconfig prefix

(Bitbake rev: 379951b6417eacbafc92ac1413ae1358bafdddfb)

Signed-off-by: Joshua Watt <JPEWhacker@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2020-07-25 15:11:48 +01:00
Richard Purdie
0aff94023f bitbake: cooker: Fix unmatched files handling leading to misleading warnings
Currently if all recipes in a layer are skipped, there are warnings that the
BBFILE_PATTERN_ entry didn't match anything. We probably shouldn't do this
for skipped recipes.

The current code is hard to understand, not least as it passes variables
which functions modify by reference rather than giving a return value.

Update calc_bbfile_priority() to return values rather than modifying them.
Refactor the code to try and make it clearer what its doing and fix the
skipped recipe issue by passing in the list of parsed files.

The function is complicated by the need to not rerun regex matching more
than we ever have to which complicates the flow, it would be easier if we
just reran operations multiple times.

(Bitbake rev: 969cb27b4d978551817612ff4558bec81cfb655c)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2020-07-21 16:57:43 +01:00
Joshua Watt
0ecab7a461 bitbake: bitbake: command: Move split_mc_pn to runqueue
All of the other multiconfig splitting functions are located in runqueue
so move the function to split a pn/fn there also so that its easier to
see them all together. Fixes a case where the findBestProvider() command
wasn't working for multiconfig because it was looking for a prefix of
"multiconfig:" instead of the newer "mc:"

(Bitbake rev: 325827af66434affc2da460cc8b9a5c460e38056)

Signed-off-by: Joshua Watt <JPEWhacker@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2020-06-10 12:30:01 +01:00
Joshua Watt
b9fdb6a426 bitbake: bitbake: cooker: Split file collections per multiconfig
Splits the cooker to track a collection per multiconfig instead of a
single collection for all multiconfigs. Practically speaking, this
allows each multiconfigs to each have different BBMASKs that apply to it
instead of each one using the mask specified in the base configuration.

(Bitbake rev: dd6d8eca2027f8d9be8a734a493227b440075e49)

Signed-off-by: Joshua Watt <JPEWhacker@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2020-06-10 12:30:01 +01:00
Richard Purdie
70ee928f8f bitbake: tinfoil: Add back ability to parse on top of a datastore
This option was removed recently as we didn't realise the layerindex
relies upon it. Add back the API which it turns out can be supported
as long as we assume the datastore passed in is a remote datastore
which it usually would be unless created locally.

(Bitbake rev: 4618da2094189e4d814b7d65672cb65c86c0626a)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2020-04-06 16:45:34 +01:00
Richard Purdie
d53958766e bitbake: tinfoil: Simplify remote datastore connections
The current approach to remote datastores used in tinfoil is breaking. For
example, adding a devupstream extension to a recipe with a git upstream,
making it the preferred version and then running "devtool modify" on it
causes get_srcrev() circular dependency issues. The problem is the override
handling in the datastore is broken.

This gets broken since remotedata:recieve_datastore() sets d.dict but doesn't
update d.overridedata (or d.inchistory or d.varhistory). We could play
whack-a-mole but the current implementation seems to be flawed to me. It
also doesn't cover, or only partially covers some datastore operations and
each needs new dedicated command API.

Instead, step back and reimplement the way the datastore connector works.

With this change, the datastore is either remote or local but the data is not
spread on two sides of the connection. All the API is proxied over the connection
by a single function for the datastore (and two to support variable history
and include history).

This code does not support using the datastore as a parameter to any data store
functions. We did have one case of that but its just bad code and can be
replaced.

The result is something which is much simpler and less invasive to the datastore
code itself, meaning its behaviour should be much more consistent. The existing
tests for the remote data no longer make any sense and are removed.

The one bug this code would have is if key/value pairs are returned over the IPC
and those values contained a DataSmart object since we don't recurse into return
values to find such things. Nothing appears to do that currently so lets worry
about it if its ever an issue. This change should simplfy a ton of other issues
and avoid a ton of other bugs so is a huge net gain.

Tested with bitbake's and OE's selftests.

(Bitbake rev: 85e03a64dd0a4ebe71009ec4bdf4192c04a9786e)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2020-03-24 22:01:03 +00:00
Joshua Watt
86aa524667 bitbake: command: Add mc parameter to findProviders command
Adds a multiconfig selection parameter to the findProviders command.
This allows a client to find the providers for a specific multiconfig
instead of the base configuration.

(Bitbake rev: 95bb446137734865f3fe04fe97681d90250deb11)

Signed-off-by: Joshua Watt <JPEWhacker@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2020-03-07 16:07:00 +00:00
Frazer Clews
fa5524890e bitbake: lib: amend code to use proper singleton comparisons where possible
amend the code to handle singleton comparisons properly so it only checks
if they only refer to the same object or not, and not bother
comparing the values.

(Bitbake rev: b809a6812aa15a8a9af97bc382cc4b19571e6bfc)

Signed-off-by: Frazer Clews <frazer.clews@codethink.co.uk>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2020-01-19 13:31:05 +00:00
Richard Purdie
cf9c0be3f6 bitbake: bitbake: Drop duplicate license boilerplace text
With the introduction of SPDX-License-Identifier headers, we don't need a ton
of header boilerplate in every file. Simplify the files and rely on the top
level for the full licence text.

(Bitbake rev: 695d84397b68cc003186e22f395caa378b06bc75)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-05-04 10:44:10 +01:00
Richard Purdie
79834a7144 bitbake: bitbake: Add initial pass of SPDX license headers to source code
This adds the SPDX-License-Identifier license headers to the majority of
our source files to make it clearer exactly which license files are under.

The bulk of the files are under GPL v2.0 with one found to be under V2.0
or later, some under MIT and some have dual license. There are some files
which are potentially harder to classify where we've imported upstream code
and those can be handled specifically in later commits.

The COPYING file is replaced with LICENSE.X files which contain the full
license texts.

(Bitbake rev: ff237c33337f4da2ca06c3a2c49699bc26608a6b)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-05-04 10:44:04 +01:00
Paul Eggleton
c697226168 bitbake: cooker: clear extra config settings and remote datastores on client disconnect
When the UI disconnects, we can throw away any server-side remote
datastores we created in response to calls from the UI, and we *must*
drop everything in extraconfigdata or it will taint any future
operations.

Dropping extraconfigdata upon disconnect fixes taskhash mismatch errors
when running devtool.DevtoolTests.test_devtool_update_recipe_local_files
within oe-selftest with BB_SERVER_TIMEOUT=100 in OpenEmbedded.

(Bitbake rev: 1ca2eec459424892391f060442ef38cf28d6a54a)

Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-09-05 15:00:49 +01:00
Richard Purdie
92903ced3a bitbake: command: Fix some bugs identified by BB_SERVER_TIMEOUT
The 'needconfig' flag was meant to be the default as most commands do need
it and the "False" cases were the exception. The code was written backwards
with a default False value. Invert this to match the intent, resulting in
the config being reparsed if metadata has changed.

Also ensure the second level configuration is parsed for the getLayerPriorities
command as otherwise it can return stale info.

With these changes:

BB_SERVER_TIMEOUT=100 oe-selftest -r bblayers.BitbakeLayers.test_bitbakelayers_add_remove

passes instead of fails.

(Bitbake rev: af3c8928a69f204d5ced02c947485990ac04a776)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-08-31 23:30:03 +01:00
Richard Purdie
019f518287 bitbake: cooker: Ensure we handle inotify before running the next command
The inotify watch events are handled "at idle" which could in theory mean
a command could run before any preceeding inotify events have been processed.
This leads to a theoretical race window where those events may have a
signficicant effect on the command.

Add a mechanism to allow us to ensure all pending events are processed before
running commands.

(Bitbake rev: bf76cd7e5881adf264b8ba64e27a5b6ca9df4fde)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-08-13 09:28:14 +01:00
Paul Eggleton
d4c3ace097 bitbake: bitbake-diffsigs: fix regression after recent server changes
We were bridging the gap between the server and UI here by calling a
bb.siggen.find_siginfo, a function defined and set on that module from
the metadata. This worked from the UI side before but since the recent
server changes is no longer accessible. Create a new command so this can
execute on the server side and return the result by way of a new event.

(We're still running compare_sigfiles() on the signature generator but
that isn't quite the same thing and does still work.)

Fixes [YOCTO #11844].

(Bitbake rev: fdcea991baa4f83d9c98d468d7b49c8c388a4a15)

Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-07-27 15:14:20 +01:00
Paul Eggleton
9e8bbe7c6f bitbake: tinfoil: enable access to additional cached items
Add access to fn_provides, packages, packages_dynamic and rproviders on
the recipecache object. This requires an additional corresponding
command plumbing to be added.

(Bitbake rev: 3df9b7c615174a6557581f3cd157842a28f6bb26)

Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-07-21 08:41:12 +01:00
Paul Eggleton
21bb330f46 bitbake: lib/bb/event: refactor printing events
We really ought to have just one place where the string representation
of these events is produced. This doesn't take any real control away
from the UI - if an alternative representation is desired, that can
still be made.

(Bitbake rev: cb15db2a799be6d8eab9a2a43a9a573f89229cff)

Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-07-21 08:41:12 +01:00
Paul Eggleton
192b2e1536 bitbake: tinfoil: add internal mode to build_file() function
In OE's devtool we want to repeatedly run build_file() without showing
unnecessary messages and triggering buildhistory for each call.
build_file() is just a wrapper around the buildFile command. Change
the final "hidewarning" parameter of the buildFile command to "internal"
and have this call a new buildFileInternal() function without triggering
any of the normal build events, silencing the normal info messages from
the runqueue ("Executing RunQueue Tasks", "Tasks Summary" etc.) and
avoiding calling parseConfiguration() which we've already done at this
point.

(Bitbake rev: ba53e067a2d448dd63b4ca252557ce98aa8e6321)

Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-07-21 08:41:11 +01:00
Richard Purdie
dd71707d5a bitbake: event/command: Allow UI to request the UI eventhander ID
The UI may want to change its event mask however to do this, it needs the
event handler's ID. Tweak the code to allow this to be stored and add
a command to query it.

Use the new command in the process server backend.

(Bitbake rev: f8cf2cb58b80ce74f756a11a9773b6b0e78d51ee)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-07-21 08:41:11 +01:00
Paul Eggleton
833520d2fb bitbake: command / cooker: drop Hob support commands and functions
Drop a number of the commands and support functions that were created
solely to support functionality in the now-removed Hob UI. In most cases
we now have this functionality elsewhere e.g. to modify config files and
recipes we have bb.utils.edit_metadata_file() and friends, and in OE we
have oe.recipeutils.patch_recipe_file() and friends which build on top of
the former. Additionally, some of it represented pretty egregious
incursion of OE metadata-specific references into BitBake code.

For now I have left in the find*File functions and commands as they are
relatively generic and possibly still useful, but they might be removed
in future.

(Bitbake rev: a322f13183c66a28d93cc4bc3d839d95fc1d90f4)

Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-06-12 15:07:39 +01:00
Paul Eggleton
3d4c64fb9f bitbake: cooker: add BB_CMDLINE to enable access to UI command line with memres
In OpenEmbedded's buildhistory class we want access to the bitbake
command line that launched the build, and up to now we were simply using
sys.argv from within the event handler to get that. Unfortunately that
doesn't work in memory resident mode, since the event handler is
naturally executing within the server and thus will give you the command
that launched the bitbake server which is much less interesting. Add a
dynamic variable BB_CMDLINE to provide access to this, set from sys.argv
within the UI process in updateToServer().

(Note that BB_CMDLINE isn't currently passed through to the worker, so
this is only really readable from event handlers plus any explicit
getVariable calls - in theory an observe-only UI could read it for
example.)

Part of the fix for [YOCTO #11634].

(Bitbake rev: 8ad31a3eff5cfcb7b1c462578a582dafcbc426d4)

Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-06-12 15:07:39 +01:00
Paul Eggleton
d45f5be977 bitbake: lib/bb/command: fix overrides in remote datastores where there is no default
If you have, for example, no value set for a variable VARIABLE and a
then VARIABLE_pn-something = "value" and then you parse something.bb,
you expect getVar('VARIABLE') on the resulting datastore to return
"value", but the code here assumed that if the variable wasn't set
without overrides then we didn't need to return the overridedata and
thus we didn't get the overridden value.

In OE this affected the ability to get RECIPE_MAINTAINER for a recipe
in a script using tinfoil (since this is only set from an inc file with
_pn- overrides for each recipe, and no default is set).

(Bitbake rev: b3d2c9917c5fd8278878328794daa107ddf79b64)

Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-04-20 07:59:40 +01:00
Juro Bystricky
00d49a05a6 bitbake: command.py: multiconfig support for findBestProvider
In a multiconfig environment, a tinfoil call such as

    tinfoil.parse_recipe("multiconfig:arduino-101-sss:gcc")

can fail with an error such as:

  File "/data/master/poky/bitbake/lib/bb/tinfoil.py", line 373, in get_recipe_file
    raise bb.providers.NoProvider('Unable to find any recipe file matching "%s"' % pn)
bb.providers.NoProvider: Unable to find any recipe file matching "multiconfig:arduino-101-sss:gcc"

The culprit is findBestProvider, which does not handle multiconfig.
This patch fixes the error and in the case mentioned above the tinfoil call returns:

  "multiconfig:arduino-101-sss:/data/master/poky/meta/recipes-devtools/gcc/gcc_6.3.bb"

[YOCTO#11210]

(Bitbake rev: e9c03fbfd7b057b28645affa263cb4aebfa24b04)

Signed-off-by: Juro Bystricky <juro.bystricky@intel.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-04-19 10:20:24 +01:00
Paul Eggleton
5b1807021e bitbake: data_smart: implement missing remote datastore operations
Enable the following operations from a remote datastore to affect the
other end:

* setVarFlag()
* delVar()
* delVarFlag()
* renameVar()

In practice I don't expect these to be used much, but they should be
present so that the implementation is at least reasonably filled out
and that the tests pass.

Also add tests for the interface, mostly by subclassing the existing
local test classes so that they are using a remote datastore. (These
don't actually test remote usage via tinfoil, just that the
datastore's interface can be used.)

(Bitbake rev: 282dc0719d22a39df746eea762ebe05c66aa8f8a)

Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-03-22 11:35:22 +00:00