In interactive bitbake sessions it is obvious what tasks are running
when one of them hangs or otherwise takes a long time. However, in
non-interactive sessions (such as automated builds) bitbake just prints
a message saying that it is "still alive" with no clues as to what tasks
are active still.
By simply listing the active tasks when printing the keep alive message,
we don't need to parse the bitbake log to identify which of the tasks
is still active and has presumably hung.
(Bitbake rev: f9f57fb7d2c8a13df1eb9d5b9766f15e229dcf97)
(Bitbake rev: 30f6c3f175617beea8e8bb75dcf255611e3fc2fd)
Signed-off-by: Ross Burton <ross.burton@arm.com>
Signed-off-by: Alexandre Belloni <alexandre.belloni@bootlin.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Adds a Dockerfile for building the PR service in a container.
Based on the hash equivalence server container [hashserv].
Use the following environment variables to configure the PR service
container:
- DBFILE: database filename. Default:
/var/lib/bbprserv/prserv.sqlite3.
- LOGFILE: log filename. Default: /var/lib/bbprserv/prserv.log.
- LOGLEVEL: logging level. Default: DEBUG.
- HOST: ip address to bind. Default: 0.0.0.0.
- PORT: port number. Default: 8585.
- DBMODE: database mode. Default: Empty (RW).
Note: DBMODE in RO: "--read-only".
[hashserv]: contrib/hashserv/Dockerfile
(Bitbake rev: a5497428b539e8598263924f63a9df8fe1ea70f9)
(Bitbake rev: 1d05abd92da56e284fcd904cf32bd12485903f10)
Signed-off-by: Daniel Gomez <daniel@qtec.com>
Signed-off-by: Alexandre Belloni <alexandre.belloni@bootlin.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Force a sync point for end-of build event handler force
the build's outcome status commit, to resolve a race
condition with the build completion takedown.
[YOCTO #14765]
(Bitbake rev: f76c13536e19d2401365c0fc240ef8dbcb3ee7e8)
Signed-off-by: David Reyna <David.Reyna@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Add a polling check on tracebacks in a build's log. This
can for example indicate that bitbake crashed, which would
stop the event stream that Toaster normally uses to detect
build errors.
[YOCTO #14085]
(Bitbake rev: 32b1c0b3477e359d2e2a61a23a294e317e417f95)
Signed-off-by: David Reyna <David.Reyna@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Creating the new log instances triggers a lot of python logging overhead
in a commonly called function (about 600,000 for parsing OE-Core).
We only need the log functionality if we're parsing, not if we just hit
from the cache. Therefore defer the log setup overhead until we know it
is a cache miss.
Whilst this complicates the code slightly, the performance gain is worth
it as for parsing OE-Core we drop 60 million funciton calls (from 225
overall).
(Bitbake rev: ac868167ad854f9bb32dcb2e63528870547805a7)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
We can call getVarFlags() instead of the multiple getVarFlag calls
which is a little more efficient. This reduces the number of overall
function calls so is good but probably isn't much faster (or slower).
(Bitbake rev: 505a4dd34e822bdf902d9b348dbcdf5b2c94e784)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The code tries to expand missing entities when they're encountered in
python expressions. Looking at traces, these are often things which
would not be in the data store such as "bb".
Optimise to skip the data store queries for things we know will never
be there. The expansion cache usually covers these but skipping
entirely cuts a few million function calls parsing OE-Core.
(Bitbake rev: 1ae712635a2eef3ecbf541e1f7cc2eeb2f3799c9)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
With the upcoming LTS, it is time we changed the bitbake version so
move to 2.0.
(Bitbake rev: 9a13bf8e20b1841ec99281d45be4c4fc1118438c)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
By alphabetical order, to get the same order as in the HTML
output, sorted thanks to the ":sorted:" directive.
(Bitbake rev: 8b97af89316a4407dd07649e8aadd495edd86ee4)
Signed-off-by: Michael Opdenacker <michael.opdenacker@bootlin.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
This template is not referenced anywhere and therefore not used.
It also refers to the obsolete "build mode" in the old docs.
(Bitbake rev: f89a35155e3b8d12ac609a165c63ed206751b8da)
Signed-off-by: Tim Orling <tim.orling@konsulko.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
connection.features.autocommits_when_autocommit_is_off
was deprecated in 3.0 and is no longer present in 3.2
(Bitbake rev: cc0f526fb4298349d0eea44c8f35d3dd226cc8d6)
Signed-off-by: Tim Orling <tim.orling@konsulko.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Follow the default guidelines from Django 3.2 setting DEFAULT_AUTO_FIELD
to django.db.models.BigAutoField which leads to these migrations.
(Bitbake rev: ccfdcf5ed6b5d2024c268bace30e53753d1f4da4)
Signed-off-by: Tim Orling <tim.orling@konsulko.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
New in Django 3.2.
Silence warnings by adding:
DEFAULT_AUTO_FIELD = 'django.db.models.BigAutoField'
in toastermain/settings.py
(Bitbake rev: cbc5a8e339a8b5eb4cdd89ad898af432f05cadfb)
Signed-off-by: Tim Orling <tim.orling@konsulko.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Django 3.2 no longer has django.utils.six module.
(Bitbake rev: ae03c1f05e37f70e72a2ac08f7c718b8fc4fc36e)
Signed-off-by: Tim Orling <tim.orling@konsulko.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Update docs links from www.yoctoproject.org to the docs.yoctoproject.org
equivalents.
(Bitbake rev: 74194962eb294c9bd1192ddd7935d57258712ee2)
Signed-off-by: Tim Orling <tim.orling@konsulko.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Django 2.2.x goes EOL April 2022.
Switch the to current LTS Django 3.2.x supported until April 2024.
(Bitbake rev: 23a15c5703d5ef4190921f9bb0273e43b1de489c)
Signed-off-by: Tim Orling <tim.orling@konsulko.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Fix a bad patch merge and move the file to the correct path.
(Bitbake rev: d173f45a88cc9460734e7cc0b87952c0ea0ff270)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Add script to safely generate the fixture files from a table
[YOCTO #14759]
(Bitbake rev: ec5e9fff8f71ba81bf5e9207abcd3011d63d86d9)
Signed-off-by: David Reyna <David.Reyna@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
[YOCTO #14757]
(Bitbake rev: 49308d1db4f7af81e31db21dc8954947de5976c7)
Signed-off-by: David Reyna <David.Reyna@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
After the override syntax change, toaster isn't working correctly. This
is because it uses IMAGE_INSTALL_append instead of IMAGE_INSTALL:append.
This tweaks the code accordingly to fix this.
I have a suspicion that exiting toaster databases may struggle with
the change and there are some migration steps missing for the whole
overrides syntax change step.
(Bitbake rev: befc93c3e03a9b1a2b630cd5badcda756fe6cd28)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
bitbake -e is not showing variable history correctly for all variables.
The issue was triggered by the change to knotty to default to enabling
variable tracking in the base datastore but that exposed another issue.
The real problem is that calling reset() inside cooker reverts to the
data tracking enabled by the UI for the base datastore, then turns
off tracking. In the case of the environment printing code, it needs
it to be left on.
Tweak the code to ensure data store tracking really is enabled.
The code here is clearly a bit of a mess but this at least fixes a
clear regression until more invasive improvements can be made.
(Bitbake rev: 9596bffc9903f023d1cc97ce531ddb1cf010c4d7)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* Use max() to clamp progress to >= 0.
* Be consistent when evaluating self.quiet (treat it as a boolean).
(Bitbake rev: 160f71372ff93894d9314619e9d3b547c1f3cda3)
Signed-off-by: Peter Kjellerstedt <peter.kjellerstedt@axis.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
In commit 8055ec36 (knotty: Improve setscene task display) the setscene
tasks got their own line in the task output. However, the progress bar
code does not handle newlines in its widgets, so the length of the
setscene line was included when calculating how much space is available
for the progress bar of the running tasks, making it much too short.
Instead of trying to teach the progress bar code to handle newlines,
separate the output of the setscene tasks from the progress bar for the
real tasks.
(Bitbake rev: a41f7792f17acdba8c7ea83b79e413ae6a49da68)
Signed-off-by: Peter Kjellerstedt <peter.kjellerstedt@axis.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Use pluralise() to correct the grammar, and drop the colon at the end if
runnning in quiet mode.
(Bitbake rev: 57396289f935892390c11afe95f3eede28fd80e7)
Signed-off-by: Peter Kjellerstedt <peter.kjellerstedt@axis.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The "Parsing halted due to errors" is shown when user has old-style
variables requiring renaming. Make it clear that it isn't an additional
error, but a result of earlier errors.
(Bitbake rev: e81b6ae61dd8b5d0f77146263d79dfe66c66b060)
Signed-off-by: Marta Rybczynska <marta.rybczynska@huawei.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
This avoids backtraces when starting toaster or using bitbake in
remote mode.
(Bitbake rev: bf723f2cb5d288ca730e4f029110b36380420a01)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Add links corresponding to recent releases
Align the title of sections with yocto-docs' releases.rst
(Bitbake rev: 5700317975e115aa31186c8cd3eddca16ec387fb)
Signed-off-by: Michael Opdenacker <michael.opdenacker@bootlin.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
(Bitbake rev: ad2307823a96e91873d804aa9d8f11c9ff39bec0)
Signed-off-by: Michael Opdenacker <michael.opdenacker@bootlin.com>
Reported-by: Peter Kjellerstedt <peter.kjellerstedt@axis.com>
Reported-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Now defunct.
(Bitbake rev: 2c7bff686c1f5eb80c89cb7f9f12287e8bb5770f)
Signed-off-by: Michael Opdenacker <michael.opdenacker@bootlin.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
There is general agreement the current display of the total setscene and
executed full tasks is confusing. Fixing it is tricky to be clear to new
users whilst not wasting screen real estate for experienced users and not
compromisng features like the progress bar.
As a compromise, move the setscene total to the previous line making
the status display a two line summary for interactive terminals. This
makes it clear what two of the numbers represent (setscene task) whilst
separating this out from the other items.
Also tweak the non-interactive output to use a comma as a separator,
the use of the forward slash was a poor choice.
(Bitbake rev: 8055ec360507e6a678ee5c4018ec1ab7f5a9cce5)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Currently the footer code doesn't quite handle multiline items correct.
Fix this to do so.
(Bitbake rev: 8eeccf73185d986c6abd6426b1d1e72da3a982df)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
This implements support for sstate mirrors using ssh as transport
protocol.
(Bitbake rev: 0a3b5b3de7bcb1c5c3748cba42d394cc484e966b)
Signed-off-by: Daniel Wagenknecht <dwagenknecht@emlix.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Fix absolute paths and paths containing the ":" character. Both is
necessary for supporting sstate mirrors via ssh (not implemented yet).
(Bitbake rev: df5505a1ba15524c3a185360d687854300aef342)
Signed-off-by: Daniel Wagenknecht <dwagenknecht@emlix.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Support URLs like ssh://HOST/PATH. They were previously not recognized
due to a missing @ sign.
(Bitbake rev: a2aa18bd27dac8902e52b466cb7118f71367d3dc)
Signed-off-by: Daniel Wagenknecht <dwagenknecht@emlix.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
hashlib.md5() is not permitted on a FIPS enabled host system. This is due
to md5 not being an approved hash algorithm.
Instead use:
hashlib.new('MD5', usedforsecurity=False)
This is allowed, as it's clear the hash is used for a non-security purpose.
Note: utils.py version should never be used to verify file integrity, but
instead be used to identify if the file may have changed. sha256 should be
used for integrity purposes.
(Bitbake rev: af866dd077867cba0129757bfcc689551445e9d7)
Signed-off-by: Mark Hatle <mark.hatle@xilinx.com>
Signed-off-by: Mark Hatle <mark.hatle@kernel.crashing.org>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
File "scripts/lib/checklayer/__init__.py", line 49, in _get_layer_collections
ldata.expandVarref('LAYERDIR')
File "build/bitbake/lib/bb/data_smart.py", line 1007, in expandVarref
if referrervalue and ref in referrervalue:
TypeError: argument of type 'bool' is not iterable
We inject True values as an internal sentinel to the datastore, fix
this codepath to handle it.
(Bitbake rev: 3b88562d87ac94725c1a683c859c2a6a3287d173)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Until now, if a renamed variable was found, bitbake exited
immediately if it was in a class, but continued after an error
message if the variable was in a recipe. This was caused by
cookerdata.py CookerDataBuilder::parseBaseConfiguration checking
a different DataSmart instance than the variable was set in.
To solve the issue, add a special variable and set it when we
find a renamed variable. Check for it in ast.py and bail out
if needed.
(Bitbake rev: d12400600e30549c88dc9e7883dc3d63b1dc1117)
Signed-off-by: Marta Rybczynska <marta.rybczynska@huawei.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
In line with the inclusive language migration defined at:
https://wiki.yoctoproject.org/wiki/Inclusive_language
replace the remaining usage of "abort" in documentation, error
messages, and comments with halt/fail/exit as appropriate.
A couple of external Javascript API calls in Toaster remain, as
they cannot currently be changed.
(Bitbake rev: bc27762bf3ffb4a20b58eace5302438c4a526626)
Signed-off-by: Scott Murray <scott.murray@konsulko.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
In line with the inclusive language migration defined at:
https://wiki.yoctoproject.org/wiki/Inclusive_language
replace the "ABORT" action in BB_DISKMON_DIRS entries with "HALT".
In order to ease migration, code has been added to warn users to
update their configurations if the old name is used, as opposed to
to throwing an error.
(Bitbake rev: 11dc65dc077398ff9818060769c99c0090291186)
Signed-off-by: Scott Murray <scott.murray@konsulko.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
In line with the inclusive language migration defined at:
https://wiki.yoctoproject.org/wiki/Inclusive_language
replace the use of "abort" with "halt" in code related to handling
task failure.
(Bitbake rev: 831fb7f2329a3cd95b71e9c85d7d7f0d717f947f)
Signed-off-by: Scott Murray <scott.murray@konsulko.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
In line with the inclusive language migration defined at:
https://wiki.yoctoproject.org/wiki/Inclusive_language
replace the remaining non-backwards-compatibility related usage
of "blacklist"/"whitelist" with "allowed"/"disallowed" in a few
places as appropriate.
(Bitbake rev: f579fb8c23d2919d25641fa4234f8a1e9c06a922)
Signed-off-by: Scott Murray <scott.murray@konsulko.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Add support for custom strings in BB_RENAMED_VARIABLES and use this to show
that BB_STAMP_WHITELIST and BB_STAMP_POLICY are no longer supported.
(Bitbake rev: 0914011f7647571ab125bbddcd7d68e3da47226a)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
In line with the inclusive language migration defined at:
https://wiki.yoctoproject.org/wiki/Inclusive_language
rename:
MULTI_PROVIDER_WHITELIST -> BB_MULTI_PROVIDER_ALLOWED
(Bitbake rev: a09546b725fda13c0279638c7c904110da7bf6cd)
(Bitbake rev: d035435c1a4951a45481867cf932faa4a6f8f936)
Signed-off-by: Scott Murray <scott.murray@konsulko.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
In line with the inclusive language migration defined at:
https://wiki.yoctoproject.org/wiki/Inclusive_language
rename:
BB_SETSCENE_ENFORCE_WHITELIST -> BB_SETSCENE_ENFORCE_IGNORE_TASKS
(Bitbake rev: 2e243ac06581c4de8c6e697dfba460ca017d067c)
(Bitbake rev: f8f7b80a0df4646247e58238a52a7d85a37116d4)
Signed-off-by: Scott Murray <scott.murray@konsulko.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
In line with the inclusive language migration defined at:
https://wiki.yoctoproject.org/wiki/Inclusive_language
rename:
BB_HASHCONFIG_WHITELIST -> BB_HASHCONFIG_IGNORE_VARS
(Bitbake rev: f344246be73d626c215f867718e45fd6cddc2aaf)
(Bitbake rev: 371deb3fe8510aadf4455810d7c5243d374e6532)
Signed-off-by: Scott Murray <scott.murray@konsulko.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
In line with the inclusive language migration defined at:
https://wiki.yoctoproject.org/wiki/Inclusive_language
rename:
BB_ENV_WHITELIST -> BB_ENV_PASSTHROUGH
BB_ENV_EXTRAWHITE -> BB_ENV_PASSTHROUGH_ADDITIONS
(Bitbake rev: fe60627839d4280cf0117ed1afbfccdff1181b6a)
(Bitbake rev: 87104b6a167188921da157c7dba45938849fb22a)
Signed-off-by: Scott Murray <scott.murray@konsulko.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
In line with the inclusive language migration defined at:
https://wiki.yoctoproject.org/wiki/Inclusive_language
rename:
BB_HASHBASE_WHITELIST -> BB_BASEHASH_IGNORE_VARS
BB_HASHTASK_WHITELIST -> BB_TASKHASH_IGNORE_TASKS
the derived code variables basewhitelist and taskwhitelist have been
renamed to basehash_ignore_vars and taskhash_ignore_tasks,
respectively.
[RP: Added compatibility mapping code and tweaked naming]
(Bitbake rev: efaafc9ec2e8c0475e3fb27e877a1c0a5532a0e5)
Signed-off-by: Scott Murray <scott.murray@konsulko.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
This change adds support for improving the user experience when variables
are renamed. This isn't as simple as it might first appear since some
bitbake variables are used through the environment before the datastore
exists, some are bitbake variables which we know about straight away
and some are metadata defined which we don't know about until later.
This patch adds support for handling these different cases, allowing a list
of bitbake renamed variables to be defined in bitbake itself and allows this
to be extended through the metadata using BB_RENAMED_VARIABLES.
In order to give the best feedback to the user, we default to turning on
variable history tracking in the base data store from knotty, which allows
filename and line number information to be shown.
(Bitbake rev: bd50a5d5e4b4fa90844464396887ebdff0d4e5f7)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
If a python function exec_func() calls a shell task, the logging wasn't working
correctly in all cases since the exception was turned into a BBHandledException()
and the logfile piece was lost which is handled at the top task level.
The easiest way to avoid this is to allow the ExecutionError exceptions to
be raised to a higher level, we don't need the traceback for them.
(Bitbake rev: 7cae11f558f9ff5fd05ef23b789aaef92fb5a327)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The changing of the environment inside the wget fetcher can race if
threading is used, such as with sstate in OE-Core. Abstract the function
so the environment can be correct before the function is called, removing
the race since the enviroment is then no longer changed.
(Bitbake rev: c73bb6023c73f003a160bb02aa4da1b580b86c23)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
BB_ORIGENV is used as a fallback environment block. It is repeatedly
accessed inside the loop. Since it is a loop invariant, move it out of
the loop.
(Bitbake rev: 346a1a6c76d40458d7b4c116147ec4d371bee74a)
Signed-off-by: Zygmunt Krynicki <zygmunt.krynicki@huawei.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
File "/home/pokybuild/yocto-worker/qemuarm/build/build/tmp/work/qemuarm-poky-linux-gnueabi/core-image-sato/1.0-r0/testsdkext/layers/build/bitbake/lib/bb/fetch2/wget.py", line 328, in checkstatus
with bb.utils.environment(**newenv):
File "/home/pokybuild/yocto-worker/qemuarm/build/build/tmp/work/qemuarm-poky-linux-gnueabi/core-image-sato/1.0-r0/testsdkext/buildtools/sysroots/x86_64-pokysdk-linux/usr/lib/python3.10/contextlib.py", line 142, in __exit__
next(self.gen)
File "/home/pokybuild/yocto-worker/qemuarm/build/build/tmp/work/qemuarm-poky-linux-gnueabi/core-image-sato/1.0-r0/testsdkext/layers/build/bitbake/lib/bb/utils.py", line 1737, in environment
del os.environ[var]
File "/home/pokybuild/yocto-worker/qemuarm/build/build/tmp/work/qemuarm-poky-linux-gnueabi/core-image-sato/1.0-r0/testsdkext/buildtools/sysroots/x86_64-pokysdk-linux/usr/lib/python3.10/os.py", line 695, in __delitem__
raise KeyError(key) from None
It is possible the wrapped function might change the environment which can lead
to KeyError issues like this. The key may no longer be in the original environment
so handle that case correctly and avoid the tracebacks.
(Bitbake rev: ae4ce4c21998bb2a12a1e1f0b8e0af8d218b87e2)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
There are some cases where the metadata needs to check if a variable
has any overrides set, even if they are currently inactive. That code
currently pokes into datastore internals. Add API instead to replace
and avoid that.
(Bitbake rev: c885e5542dcf760b8fc5881e385abb4a10020874)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Loggers created with a logging configuration file automatically get the
once filter applied to ensure that warnonce()/erroronce() only appear a
single time. The same filter needs to also be added to
bb.msg.logger_create() to ensure that manually created loggers have the
same behavior
(Bitbake rev: bd93f7baff9d56de37a1af474189e0888b82d475)
Signed-off-by: Joshua Watt <JPEWhacker@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
This adds a log level and logging function call to use it where the
warning or error will only be displayed once, regardless of how many
times the message is logged.
This has to be done either in the cooker or on the UI side. I've opted
for the UI side since display control is really a UI issue but it uses
a common library filter function to enable it which can be reused
elsewhere.
The knotty message displayed as the build summary is tweaked to
make sense when the numbers won't match since it will still count
the number of times it was logged and this is probably helpful
for debugging in some cases so I've deliberately left it that way.
(Bitbake rev: 7bd40e3003a043e3cb7efc276681054b563b5e7b)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Signed-off-by: Joshua Watt <JPEWhacker@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
When a recipe raises a BBHandledException, it means the error was already
shown to the user. Adding an additional one here isn't helpful. What is
helpful is to mention that parsing was halted.
Tweak the code to do this with improves the messages the user sees
and helps understand what happened.
(Bitbake rev: fdf6ebc8b603fcfd3ed7c64baf486a4adabd25be)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The overrides warning message is meant to show filename and line
numbers but the variable names are incorrect and this wasn't working.
Fix it.
(Bitbake rev: 551c1cb20fc9b9d0dab5d830182c2bf626e72845)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Export ClientError and ServerError classes from bb.asyncrpc.serv as
required by lib/hashserv/server.py module.
Error:
Client ('192.168.2.184', 51142) connected
Error from client: module 'bb.asyncrpc' has no attribute 'ClientError'
Traceback (most recent call last):
File "/opt/bbhashserv/lib/bb/asyncrpc/serv.py", line 192, in handle_client
await client.process_requests()
File "/opt/bbhashserv/lib/hashserv/server.py", line 200, in process_requests
await super().process_requests()
File "/opt/bbhashserv/lib/bb/asyncrpc/serv.py", line 72, in process_requests
await self.dispatch_message(d)
File "/opt/bbhashserv/lib/hashserv/server.py", line 217, in dispatch_message
raise bb.asyncrpc.ClientError("Unrecognized command %r" % msg)
AttributeError: module 'bb.asyncrpc' has no attribute 'ClientError'
(Bitbake rev: 8aca1d341500af8bbe788801c2df3da9bbfacaed)
Signed-off-by: Daniel Gomez <daniel@qtec.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
When building the Hash Equivalence server Dockerfile, some
dependencies are missing in order to run the hash server properly:
Traceback errors:
Traceback (most recent call last):
File "/opt/bbhashserv/bin/bitbake-hashserv", line 61, in <module>
ret = main()
File "/opt/bbhashserv/bin/bitbake-hashserv", line 54, in main
server = hashserv.create_server(args.bind, args.database, upstream=args.upstream, read_only=args.read_only)
File "/opt/bbhashserv/lib/hashserv/__init__.py", line 120, in create_server
from . import server
File "/opt/bbhashserv/lib/hashserv/server.py", line 14, in <module>
import bb.asyncrpc
ModuleNotFoundError: No module named 'bb'
Traceback (most recent call last):
File "/opt/bbhashserv/bin/bitbake-hashserv", line 61, in <module>
ret = main()
File "/opt/bbhashserv/bin/bitbake-hashserv", line 54, in main
server = hashserv.create_server(args.bind, args.database, upstream=args.upstream, read_only=args.read_only)
File "/opt/bbhashserv/lib/hashserv/__init__.py", line 120, in create_server
from . import server
File "/opt/bbhashserv/lib/hashserv/server.py", line 14, in <module>
import bb.asyncrpc
File "/opt/bbhashserv/lib/bb/__init__.py", line 128, in <module>
from bb import fetch2 as fetch
File "/opt/bbhashserv/lib/bb/fetch2/__init__.py", line 1930, in <module>
from . import git
File "/opt/bbhashserv/lib/bb/fetch2/git.py", line 70, in <module>
import bb.progress
File "/opt/bbhashserv/lib/bb/progress.py", line 14, in <module>
import bb.build
File "/opt/bbhashserv/lib/bb/build.py", line 27, in <module>
from bb import data, event, utils
File "/opt/bbhashserv/lib/bb/data.py", line 36, in <module>
from bb import data_smart
File "/opt/bbhashserv/lib/bb/data_smart.py", line 23, in <module>
import bb, bb.codeparser
File "/opt/bbhashserv/lib/bb/codeparser.py", line 26, in <module>
import codegen
ModuleNotFoundError: No module named 'codegen'
Traceback (most recent call last):
File "/opt/bbhashserv/bin/bitbake-hashserv", line 61, in <module>
ret = main()
File "/opt/bbhashserv/bin/bitbake-hashserv", line 54, in main
server = hashserv.create_server(args.bind, args.database, upstream=args.upstream, read_only=args.read_only)
File "/opt/bbhashserv/lib/hashserv/__init__.py", line 120, in create_server
from . import server
File "/opt/bbhashserv/lib/hashserv/server.py", line 14, in <module>
import bb.asyncrpc
File "/opt/bbhashserv/lib/bb/__init__.py", line 128, in <module>
from bb import fetch2 as fetch
File "/opt/bbhashserv/lib/bb/fetch2/__init__.py", line 1930, in <module>
from . import git
File "/opt/bbhashserv/lib/bb/fetch2/git.py", line 70, in <module>
import bb.progress
File "/opt/bbhashserv/lib/bb/progress.py", line 14, in <module>
import bb.build
File "/opt/bbhashserv/lib/bb/build.py", line 27, in <module>
from bb import data, event, utils
File "/opt/bbhashserv/lib/bb/data.py", line 36, in <module>
from bb import data_smart
File "/opt/bbhashserv/lib/bb/data_smart.py", line 23, in <module>
import bb, bb.codeparser
File "/opt/bbhashserv/lib/bb/codeparser.py", line 32, in <module>
from bb.pysh import pyshyacc, pyshlex
File "/opt/bbhashserv/lib/bb/pysh/pyshyacc.py", line 13, in <module>
import bb.pysh.pyshlex as pyshlex
File "/opt/bbhashserv/lib/bb/pysh/pyshlex.py", line 17, in <module>
from ply import lex
ModuleNotFoundError: No module named 'ply'
Traceback (most recent call last):
File "/opt/bbhashserv/bin/bitbake-hashserv", line 61, in <module>
ret = main()
File "/opt/bbhashserv/bin/bitbake-hashserv", line 54, in main
server = hashserv.create_server(args.bind, args.database, upstream=args.upstream, read_only=args.read_only)
File "/opt/bbhashserv/lib/hashserv/__init__.py", line 120, in create_server
from . import server
File "/opt/bbhashserv/lib/hashserv/server.py", line 14, in <module>
import bb.asyncrpc
File "/opt/bbhashserv/lib/bb/__init__.py", line 128, in <module>
from bb import fetch2 as fetch
File "/opt/bbhashserv/lib/bb/fetch2/__init__.py", line 1935, in <module>
from . import wget
File "/opt/bbhashserv/lib/bb/fetch2/wget.py", line 30, in <module>
from bs4 import BeautifulSoup
ModuleNotFoundError: No module named 'bs4'
(Bitbake rev: 0bd637acfcba5a44230c291889d2a5ff571cb8c6)
Signed-off-by: Daniel Gomez <daniel@qtec.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Where a git url uses a tag instead of a full source revision, breakage
can currently occur in builds. Issues include:
* the revision being looked up in multiple tasks (fetch and unpack)
* the risk a different revision may be obtained in those tasks
* that some tasks may not be allowed to access the network
* that a revision may not be consistent throughout a given build
* rerunning a specific task may given inconsistent results
To fix this, stop the workers from cleaning out the source revision store. This
should only be done in the cooker itself (based on current policy).
Also, where the code "sees" an upstream access, mark the recipe as not to be
cached. The reparse re-triggers the upstream lookup by the server.
Add a test to ensure that if get_srcrev isn't called, the user is told they're
using a configuration that is known to break.
(Bitbake rev: 4b5eed1626709ef3dc06b32fd55d40a2a6edd179)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
We started to require the branch parameter for git urls. This updates
the tests to match and quieten some noise.
(Bitbake rev: e5144887f2b6ae266f7b52151005619480e80ec6)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Previously npm files that specify downloadfilename= in the SRC_URI
would be downloaded to the root of ${DL_DIR} rather than in the
${DL_DIR}/npm2 directory where all other npm files are downloaded.
This should make it simpler when setting up and configuring a
premirror with the downloaded npm packages.
(Bitbake rev: 73fa376d6502ab6f2cccfb25a1193d9b1c3c3bc8)
Signed-off-by: Peter Kjellerstedt <peter.kjellerstedt@axis.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The solution implementated in commit 96c30007 (fetch2: fix
downloadfilename issue with premirror) missed two corner cases. The
first is if the basename of the original URI also appears somewhere
else in the replacement URI, in which case it would also be replaced.
The second is if the basename of the original URI partially matches
the basename of the replacement URI, in which case the inital part of
the basename from the replacement URI would be left behind.
The second case caused test_npm_premirrors_with_specified_filename to
fail.
The solution is to prefix the basename with a slash when matching to
avoid partial matches, and only replace the basename at the end of the
URI.
This also adds two test cases that test for these problems. Before
this they would give the following errors:
- ['file:///mirror/example/1.0.0/some-example-1.0.0.tgz;downloadfilename=some-example-1.0.0.tgz']
+ ['file:///mirror/some-example-1.0.0.tgz/1.0.0/some-example-1.0.0.tgz;downloadfilename=some-example-1.0.0.tgz']
? +++++ ++++++++++
- ['file:///mirror/some-example-1.0.0.tgz;downloadfilename=some-example-1.0.0.tgz']
+ ['file:///mirror/some-some-example-1.0.0.tgz;downloadfilename=some-example-1.0.0.tgz']
? +++++
(Bitbake rev: 5924c6f007519cd8ea6cc8b316814d17b43048ca)
Signed-off-by: Peter Kjellerstedt <peter.kjellerstedt@axis.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
There are two totally opposite use cases for how a premirror is
expected to behave in combination with specifying a downloadfilename=
parameter in the SRC_URI. On the one hand there is the expectation
that it works like any other mirror, which means the premirror is
expected to contain a file with the original name specified in the
SRC_URI. On the other hand there is the expectation that one can use
the artefacts downloaded by bitbake in ${DL_DIR} as a premirror, in
which case it is expected to contain a file with the name from the
downloadfilename= parameter.
The latter case has been how downloaded files have been handled until
commit 8a3ff9f3 (fetch2: fix premirror URI when downloadfilename
defined), where the fetcher was changed to store files as per the
first case. This is also when the test_npm_premirrors test case was
modified in commit 5ba191a0 (tests/fetch: add and fix npm tests) to
expect the first case.
The above change was later reverted in commit 96c30007 (fetch2: fix
downloadfilename issue with premirror). However the
test_npm_premirrors test case was not updated to match, and has been
failing ever since. This has probably gone unnoticed because the npm
related test cases require that npm is installed on the host.
This commit updates test_npm_premirrors to expect that premirrors use
the filenames specified by downloadfilename= as this matches the
current fetcher implementation and also is the most likely use case
for premirrors. It also tries to mimic how one typically might setup
the premirror directory by simply copying the download directory.
(Bitbake rev: 9e913ade70474aaeb928814d4763e7105569d63a)
Signed-off-by: Peter Kjellerstedt <peter.kjellerstedt@axis.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
If they are set, assume they are correct.
(Bitbake rev: a3a3a40b96c29051d97d0c96eb01784cbd1f5420)
Signed-off-by: Peter Kjellerstedt <peter.kjellerstedt@axis.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
This unifies the three separate implementations of git() from
GitMakeShallowTest, GitShallowTest and GitLfsTest into one
implementation in FetcherTest. It also makes use of this in
FetcherLocalTest and FetcherNetworkTest.
(Bitbake rev: 8c8b418c3e6f8ab2535ad405e8006ee1012f3484)
Signed-off-by: Peter Kjellerstedt <peter.kjellerstedt@axis.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
This changes:
test_foo .. skipped 'Network tests being skipped'
test_foo .. skipped 'npm not installed, tests being skipped'
to:
test_foo .. skipped 'network test'
test_foo .. skipped 'npm not installed'
to avoid double "skipped" in the output.
(Bitbake rev: e551328a4595c220b94ab5002d978ddb4deeebcf)
Signed-off-by: Peter Kjellerstedt <peter.kjellerstedt@axis.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Since commit 044fb04d (fetch2: Allow whitespace only mirror entries)
there is no need to separate the entries in mirror variables with
"\n".
(Bitbake rev: ed22b078aae25e804ca5edf062e2c905d3e49426)
Signed-off-by: Peter Kjellerstedt <peter.kjellerstedt@axis.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Both test_crate_url and test_crate_url_multi require Internet access.
(Bitbake rev: 34e9daed6a7f766bb6e27c7e02c94b545f4791d8)
Signed-off-by: Peter Kjellerstedt <peter.kjellerstedt@axis.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Since commit 044fb04d (fetch2: Allow whitespace only mirror entries)
there is no need to separate the entries in PREMIRRORS with "\n".
(Bitbake rev: 341746c7f7b8e04d3fad51ec1b9e462ced355bfc)
Signed-off-by: Peter Kjellerstedt <peter.kjellerstedt@axis.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The NIS can't work when network is dissable, so preserve network for it, the
error is like:
do_ypcall: clnt_call: RPC: Unable to send; errno = Network is unreachable
Note, enable nscd on the build machine might be a solution, but that isn't
reliable since it depends on whether the network function has been cached or
not.
(Bitbake rev: 4eafae7904bae6e5c6bc50356e8a9077f2e207fa)
Signed-off-by: Robert Yang <liezhi.yang@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
This imports the crate fetcher from OE-Core to resolve various module issues
and adds some very very basic tests of that new fetcher.
(Bitbake rev: 1f06f326fa8b47e2a4dce756d57a9369a2225201)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
If an unexpected exception occurs in a setscene task, it is currently hidden
from the user and not recorded in any logs. This isn't helpful to debug
such failures.
Change the code so that even in the "silent" or "quiet" task case (setscene
tasks), a warning is shown with the traceback unless it was an "handled"
exception. This means the failing function can show it's own warning/error
instead if it wants to and then raise a handled event.
(Bitbake rev: 41dcdc61eb40def8c14a42e8d7bb9ce5a34afa57)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
These variables were removed from the codebase, update the docs to match.
(Bitbake rev: 32180d5057c818a69987aada482e82acf3c72ef2)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The URL expected by git fetcher when the SSH protocol is used differs
from the one given by Git servers to be used with the `git clone`
command.
Add a note making this specificity known to users.
Cc: Quentin Schulz <foss+yocto@0leil.net>
(Bitbake rev: 81c09d434aed13fcfc4ba426ae6a0b849447eb5f)
Signed-off-by: Quentin Schulz <quentin.schulz@theobroma-systems.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The different stamp policies were poor versions of the siggen code and task
hashes, predating it and being used by packaged staging. They had many
limitations, hence their replacement. I'm not aware of any users of that
code any more so I believe it and the assoicated stamp whitelist variable
can simply be removed.
(Bitbake rev: 98407efc8c670abd71d3fa88ec3776ee9b5c38f3)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
This patch changes behaviour. After this change any task which does not
have the network flag set will have networking disabled on systems that
support that (kernel version dependent).
Add a "network" task specific flag which then triggers networking to
be enabled for this task, it is otherwise disabled.
This needs to happen before we enter the fakeroot environment of the task
due to the need for the real uid/gid which we save in the parent process.
(Bitbake rev: 0746b6a2a32fec4c18bf1a52b1454ca4c04bf543)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Add a function which uses the unshare glibc call to disable networking
in the current process. This doesn't work on older distros/kernels
but will on more recent ones so for now we simply ignore the cases we
can't execute on. uid/gid can be passed in externally so this can
work with pseudo/fakeroot contexts.
(Bitbake rev: 9d6341df611a1725090444f6f8eb0244aed08213)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
This is deprecated in python 3.12 and Fedora 35 is throwing warnings so
move to the new functions.
(Bitbake rev: 68a18fbcb5959e334cf307d7fa8dc63832edb942)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
These urls are no longer adding much to the test coverage but the intermittent
network issues connecting to them are painful. Drop the urls.
(Bitbake rev: bdf5739c5d831dc97a7d81568f94a0953c71017f)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Always use a temporary file for the user config 'NPM_CONFIG_USERCONFIG'
because npm otherwise failed if configs and npmrc aren't set:
double-loading config "/dev/null" as "global", previously loaded as "user"
(Bitbake rev: 9f272ad7f76c1559e745e9af686d0a529f917659)
Signed-off-by: Stefan Herbrechtsmeier <stefan.herbrechtsmeier@weidmueller.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Quote destdir in run chmod command to support special characters in
package name and to avoid syntax error for packages like
'@(._.)/execute'.
(Bitbake rev: a701dfce3f0e74b4d7c687eeda83fe9c8e7240b1)
Signed-off-by: Stefan Herbrechtsmeier <stefan.herbrechtsmeier@weidmueller.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Internal bitbake shutdown timings must have changed recently as we're
seeing race issues on the autobuilder around the removal of the bitbake.lock
file. Improve the lockfile race code to cover bitbake's lockfile too
and use it in all the tests.
[YOCTO #14658]
[YOCTO #14652]
(Bitbake rev: bd1912bed64424f9fb28396b71bb49b6090ed087)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
We should not redirect stderr to stdout if we need to get separated
stdout and stderr contents from Popen.communicate() later.
(Bitbake rev: 1ecc1d9424877df89fcda2f23c306998998a65ff)
Signed-off-by: Anton Mikanovich <amikan@ilbers.de>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Currently the console output for signature dumps e.g. with bitbake-dumpsig
isn't deterministic. Add some sorting to improve that.
(Bitbake rev: a663440b1623f97c9c169df5566e429fbc932a53)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Add a parameter that limits sending Basic authentication in the
Authorization header to only the first host and not any that we're
redirected to. Ignoring potential security concerns, temporary AWS URLs
will reject any request that includes authentication details in both the
query parameters (from the redirect) and in the Authorization header.
Temporary AWS URLs are now being used for release assets from private
Github repositories. According to the previous discussion linked below,
they're also in use by bitbucket.
See also:
https://lore.kernel.org/bitbake-devel/CAC9ffDEuZL-k8199bUyN+8frjw6bg-g=vrumxxtvt+RVParQ8Q@mail.gmail.com/
(Bitbake rev: a6ab32013a4381a1b694ed46caf2c9da932644d0)
Signed-off-by: Justin Bronder <jsbronder@cold-front.org>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Add a parameter `striplevel` to the SRC_URI to strip NUMBER leading
components (levels) from file names on extraction.
For example, if the archive `archive.tar.gz` contains `some/file`,
the SRC_URI `https://.../archive.tar.gz;subdir=other;striplevel=1`
will extract `some/file` to `other/file`.
This is useful to extract archives to a specified directory instead of
the original root component of the archive. The feature is required for
the npm support. The npm package contents should reside in a subfolder
inside a npm archive (usually it is called package/). npm strips one
directory layer when installing the package.
(Bitbake rev: aa4926e5d9c92f33b4434e2da709ff0bf3049f5b)
Signed-off-by: Stefan Herbrechtsmeier <stefan.herbrechtsmeier@weidmueller.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The tar command and its arguments are repeated for many archive types in
the unpack function. Unify the common parts in a variable to prepare
further extension.
(Bitbake rev: a08e57c9eaec1d9740a96149bf4843e576da4e5c)
Signed-off-by: Stefan Herbrechtsmeier <stefan.herbrechtsmeier@weidmueller.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Fixed when oe-core's path contians append/prepend/remove, e.g.:
/path/to/append_test/oe-core/
Initial a build in any build dirs:
$ bitbake -p
ERROR: Variable __anon_32__buildarea2_xhou_builds_append_test_layers_oe_core_meta_classes_patch_bbclass
contains an operation using the old override syntax. Please convert this
layer/metadata before attempting to use with a newer bitbake.
The anonymous fuctions has no names, so skip checking for it to fix the issue.
(Bitbake rev: ebd00330c41c75797529ff38d6a0955b93f05d1b)
Signed-off-by: Robert Yang <liezhi.yang@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
This syntax is obsolete, update to the correct modern version.
(Bitbake rev: 744e9a4c2b6f44116435feb62ac64ff256c752e9)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Using "" as a target for .replace() is a really bad idea as it duplicates the replacement
for every character in the string. Add a testcase which triggered this and correct the
code to return the correct result.
(Bitbake rev: 3af1ecf049d2eed56f6d319dc7df6eb4a3d4eebc)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The fetcher mirror code can go crazy creating lock filenames which exceed the
filesystem limits. When this happens, the code will loop/hang.
Handle the filename too long exception correctly but also truncate lockfile
lengths to under 256 since the worst case situation is lockfile overlap
and lack of parallelism.
(Bitbake rev: 63baf3440b16e41ac6601de21ced94a94bdf1509)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
This code has been unchanged since 2006 apart from attempts to optimise
performance by minimising chdir() calls.
There is no reason the modern bitbake parser should be changing directory
all the time. We did have some path assumptions in the mists of time but
those were resovled and the code is deterministic and doesn't depend on
cwd now for parsing. We can therefore drop the changes in directory.
Also, TOPDIR is now being set by cookerdata in all cases so we don't
need the fallbacks in this code (which was used to effectively initialise
a value). We don't need to change TOPDIR when parsing a recipe, that makes
no sense. If we stop all the other messing around, we don't need to expand
TMPDIR either.
These changes have the potential to break some obscure use cases such
as an anonymous function assuming the current working directory, or some
case which depends on TOPDIR changing but I believe any such uses should
be fixed at this point.
(Bitbake rev: add5d488e1d6607a98441836075d01cb1dc9c0fa)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
TOPDIR is set internally deep within the parser to os.getcwd(). Rather
than do that, set it explicitly if not set. Note that modern code will
almost always have a bblayers.conf file which would have already set
TOPDIR before this new code.
Also fix findTopdir since the conf/bitbake.conf codepath is just
plain incorrect, it would find build metadata, not the current
build directory that bitbake would use. Again, the use of bblayers.conf
means hitting the fallback code was unlikely.
This change makes everything clear and explicit.
(Bitbake rev: c03df5283408dfd089b6317677d2b7af6fa73936)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
It was clear with testing that we're asuming bitbake sets TOPDIR correctly
when running these tests. Remove that implict assumption and make it
explicit.
(Bitbake rev: c8f1eb377ceb1fc78cbfaed976107720ad78c075)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
>From contents from the Yocto Project manual
Took the opportunity to reorder SRC_URI fetchers and options
alphabetically.
(Bitbake rev: ee6a951de31471c610030d0cf745039a71706b50)
Signed-off-by: Michael Opdenacker <michael.opdenacker@bootlin.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The right hand side of dependencies in the task dependency file generated
by bitbake -g was missing multiconfig prefixes, corrupting the data. Fix
this.
[YOCTO #14621]
(Bitbake rev: 1d5ca721040c5e39aefa11219f62710de6587701)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
There have been requests to better document the contraints of fetcher
design and operation. This README attempts to start that.
(Bitbake rev: d9cda7835816ecd5a60f0575f6ce832ec9c6aced)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
When build with nativesdk-python3 (3.10) from buildtools:
/path/to/bitbake/lib/pyinotify.py:55: DeprecationWarning: The asyncore module is deprecated. The recommended replacement is asyncio
The pyinotify.py's upstream didn't have any update in recent 7 years:
https://github.com/seb-m/pyinotify
And bitbake doesn't use the asyncore module, so remove the related code.
(Bitbake rev: 58fbb01c3e2111bef4f79f88e1aac1827350c82a)
Signed-off-by: Robert Yang <liezhi.yang@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The previous fix introduced a race where the queue might not be empty
but all the parser processes have exited. Handle this correctly to avoid
occasional errors.
(Bitbake rev: 8e7f2b6500e26610f52d128b48ca0a09bf6fb2cb)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
We had an issue where a webserver serving sstate had filesystem issues so
would accept connections but effectively not do anything with them. This
causes bitbake to hang whilst processing things like sstate objects inside
the checkstatus() calls. It can be replicated by setting up a server like:
socat -u TCP4-LISTEN:NNN,fork OPEN:/dev/null
and pointing SSTATE_MIRRORS in OE at that address.
Adding a timeout to the checkstatus calls of 15s means that whilst the
system will pause, it will then continue and not hang entirely. Since there
isn't a large transfer here, 30s should be a reasonable response time after
which we should fall back to building things ourselves.
[YOCTO #13716]
(Bitbake rev: edc3b0c3953cab675e29fe295b58cfa84ba811c3)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Operations like XXX:append += "YYY" are almost always wrong and this
is a common mistake made in the metadata. Show warnings for these usages
with a view to making it a fatal error eventually.
(Bitbake rev: 8c31e75557dc6a8d8f407b5d24d6327889a3e3b1)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The previous fix for runall option handling had a small bug in it, it
didn't clear the originally processed task list which meant it was running
too many tasks. Fix this so the list is reset and rebuild correctly.
(Bitbake rev: 87c9e120897ed04dfc64d4752fc602f9bfcb8645)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Currently if you have something like:
SRC_URI = "file://foobar;subdir=${S}"
and a file like:
foobar/1/somefile
and then move it to:
foobar/2/somefile
the task checksums don't reflect/notice this. The file-checksum fields
encode two pieces of data, the file path and whether or not the file
exists. Changing the code which uses these fields is problematic.
We can however add a "/./" path element which means "include the bit
after the marker in the checksum" which the path walking code can use
to mark which bits of the path are visible to the fetcher.
I'm not convinced this is great design but it does appear to work.
(Bitbake rev: b4975d2ecf615ac4c240808fbc5a3f879a93846b)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
If one of the parse threads disappears during parsing for some reason, bitbake
currently hangs. Avoid this (and zombie threads hanging around) by joining()
threads which have exited.
(Bitbake rev: dc86a533d951d13643ce446533370da804782afc)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
vcs.pcre.org was a redirect to github which we use for subversion testing.
With the protocol changes at github and the removal of the redirect, use a
direct address for github.
(Bitbake rev: 6230ca71eb7eb2a6db162e28a01727d00af5299b)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
When the user specify an invalid upstream hash equivalence server in
BB_HASHSERVE_UPSTREAM notify the user that we can't connect the server.
(Bitbake rev: be45aeb9a84f30c28711e87e2d2a4a86320a8d94)
Signed-off-by: Jose Quaresma <quaresma.jose@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The runbuild option handling in runqueue was flawed as items deleted from the
main task list may be dependencies and hence cause index errors.
Rather than modify runtaskentries straight away, compute a new shorted list
and use that as an input to the second phase. This avoids the need to add tasks
back to the list meaning delcount can be simplifed to a simple counter.
The second use case in runonly doen't re-add items so doesn't have this
issue.
(Bitbake rev: 3428e3c54eb5cc03ff96f9cee6dc839afee7a419)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Add a test to ensure that a parameter like protocol=git can be rewritten
to a different url and protocol.
(Bitbake rev: 69b4f9a09ff74378788cc2ec1ad58cd66b27ca59)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Username or password replacements in URIs were being appended rather than
replaced in mirror url remapping. Fix this and add a test case.
[YOCTO #13823]
(Bitbake rev: 66ad58bb87e5158aced572be4f1d5726bc97fcce)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Python scales badly when concatinating strings in loops. Most of these
references aren't problematic but at least one (in data.py) is probably
a performance issue as the issue is compounded as strings become large.
The way to handle this in python is to create lists which don't reconstruct
all the objects when appending to them. We may as well fix all the references
since it stops them being copy/pasted into something problematic in the future.
This patch was based on issues highligthted by a report from AWS Codeguru.
(Bitbake rev: d654139a833127b16274dca0ccbbab7e3bb33ed0)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
In theory we can leak the so file descriptor so refactor the code
to avoid that.
(Bitbake rev: dfad69d4d8c894a5e1e2686023e41552de09bf3b)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
On master, tell the users they need to update their urls for github.
(Bitbake rev: 42526a402357e04794f4cb6f21ac18f562220a9b)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
github is dropping support for git protocol in Git urls. Add code to remap
this to https in a way that could be used in older bitbake versions.
(Bitbake rev: f05e19135b3ddee509c0cb427b3b9376bb4738d1)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
There is much uncertainty around what tools and hosting providers will
do about default git branch naming in the future. To help ensure we
can handle the various scenarios, we will make branch names required in
SRC_URI. To start that process, show users a warning if it isn't set.
This may also allow us to change the default at some point in the future.
(Bitbake rev: 86a9c26828479be55865bcce72bcc7e12b93caa7)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The git fetcher is odd in that it supports destsuffix as a parameter but not
the default documented subdir parameter. destsuffix is more limited as it can't
take absolute paths. Rework the code to correctly support subdir.
Also cleanup to use the None default .get() values and be a bit more pythonic
and use subpath as the variable name for subpath for code clarity.
We could consider dropping destsuffix as a parameter as some future point.
Also fix the tests not to pass in a subdir parameter which was never used
but now causes errors.
(Bitbake rev: 66953f06fe822e4001efabd9fc1c985ea2b03f96)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The following commit to fix [Yocto #13039] causes regression of
the behavior of PREMIRRORS.
"bitbake: fetch2: fix premirror URI when downloadfilename defined"
Take meta-openembedded/meta-networking/recipes-protocols/freediameter/freediameter_1.4.0.bb
as an example.
SRC_URI = "\
http://www.freediameter.net/hg/${fd_pkgname}/archive/${PV}.tar.gz;downloadfilename=${fd_pkgname}-${PV}.tar.gz \
...
"
With the above commit, it now tries to fetch 1.4.0.tar.gz instead of
freeDiameter-1.4.0.tar.gz. This makes https://downloads.yoctoproject.org/mirror/sources
not work for freediameter, as it holds freeDiameter-1.4.0.tar.gz.
The commit above tries to avoid fetching from invalid url such as:
https://<some_mirror>/1.4.0.tar.gz/freeDiameter-1.4.0.tar.gz.
And its solution is to make basename to be 1.4.0.tar.gz, thus causing the
regression.
This patch fixes the above regression. For Yocto #13039, it now tries
to fetch from url: https://<some_mirror>/freeDiameter-1.4.0.tar.gz.
(Bitbake rev: 96c30007dc0b32eee2b15771daec7948bc9bfd97)
Signed-off-by: Chen Qi <Qi.Chen@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Add a test case test_fetch_premirror_use_downloadfilename_to_fetch to ensure
that 'downloadfilename' is used when fetching from premirror.
Although the other two previous test cases, test_fetch_premirror_specify_downloadfilename_regex_uri
and test_fetch_premirror_specify_downloadfilename_specific_uri already
implicitly contain such verification, we still need to add a very clear
case to ensure no regression.
(Bitbake rev: 20aabc3d53f69949810ecf02295725db947ffef8)
Signed-off-by: Chen Qi <Qi.Chen@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
When downloadfilename is specified, it is used to fetch from premirror.
So fix the test cases accordingly.
(Bitbake rev: 3b4d2e3b5024324058360a2a28f33c34114218d0)
Signed-off-by: Chen Qi <Qi.Chen@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Documentation is based on the commit message of bitbake rev 5ed7d85fda
and mailing list discussion.
[YOCTO #14493]
(Bitbake rev: a183322ef56c64c97aa2b72fedcdf58e29dbb3cf)
Signed-off-by: Paul Barker <paul@pbarker.dev>
Signed-off-by: Michael Opdenacker <michael.opdenacker@bootlin.com>
Reviewed-by: Quentin Schulz <foss@0leil.net>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Our minimum python version is 3.6 so these are now unneeded.
(Bitbake rev: 15f16bf973666807e94f3b356095f8265af0282a)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The switch to using json has messed up the type handling as the code
does assume that set()s are present. Add a decoder to reconstruct
the set() objects. Also fix the change of tuples to lists for the
file checksums and fix an existing type bug where dicts insteads of
lists was used.
Drop some old siginfo format handling code which is now long since
obsolete.
(Bitbake rev: 2d704842c0928f8dbe78fd081042aa7280af96be)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
We occasionally see bitbake-worker failing and from the logs, an unpickle error
occurs. Add more debug so we can further debug this next time it fails.
[YOCTO #14595]
(Bitbake rev: fe8105cc06beca8240b76ea366a1eff5aa9c5412)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
We've seen races where the socket may be gone but the server is still writing
out it's database. Handle that case too to avoid cleanup tracebacks.
[YOCTO #14440]
(Bitbake rev: b9e4fb843cb9d3a4d4404af093a781fab5520465)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The URL for the sources mirror no longer works. Update to the new
location.
(Bitbake rev: 160feefd4af07c0b057495f19c8dd94fe243853b)
(Bitbake rev: 45ea7a937d81d68e3c1203a6e04a3f2f3cc607da)
Signed-off-by: Jon Mason <jdmason@kudzu.us>
Signed-off-by: Alexandre Belloni <alexandre.belloni@bootlin.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Since Python 3.8 visit_Num(), visit_Str() and so on are all deprecated
and replaced with visit_Constant. We can't yet remove the deprecated
functions until we require 3.8, but we can implement visit_Constant to
silence the deprecation warnings.
(Bitbake rev: 4edd5767fc6d699f5262862b763b6a99ad1f1bbf)
Signed-off-by: Ross Burton <ross.burton@arm.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
npmsw(and maybe other fetchers) may try to download same files simulatenously.
if a premirror is set, download method tries premirror location first. If file
is not found, done is set to false. In the meantime, other fetchers may download
the files required and verify_donestamp might return true.
In this scenerio, fetcher doesn't try to download again and fails. Do not
attempt to verify_donestamp if done is not set. By this way, we ensure that
fetcher attempts to do another download after premirror failure.
(Bitbake rev: 4fa25714916e84f99ecd22cb76cb5adada01e5e8)
Signed-off-by: Caner Altinbasak <cal@brightsign.biz>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
If multiple npmsw fetchers are trying to download the same npm file, one of them
can try to download the file while other is calling verify. npmsw methods gets
called without holding the lock, which causes race conditions in fetching and
verification etc. Lock the lockfile before calling proxy fetcher methods.
(Bitbake rev: fa39e6689d0f0fff772e1c81682698f4b1587b8a)
Signed-off-by: Caner Altinbasak <cal@brightsign.biz>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Since OE is about to change to zstd compression of sstate, it would make it
timely to convert the siginfo files from pickle which isn't reproducible
to json which is both reproducible and also human readable. At the same time
add zstd compression. This makes the siginfo files smaller, reprodubicle
and easier to debug.
Backwards compatibility mixing the two formats hasn't been supported since
in reality if sstate changes at the same time, files will be in one format
or the new one but comparing mixed formats won't make much sense.
Since json doesn't support sets, we translate them into lists in the files
themselves. We only use sets in bitbake since it makes things easier in
the internal code, sorted lists are fine for the file format.
[YOCTO #13973]
(Bitbake rev: 22c18494c9072788e6e26eb73de70378ae5c5bf5)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The code works fine if we pass a file descriptor in and we need to
do this from the siggen code so add that as a valid input.
(Bitbake rev: fc8d74b7de576005ecf67920501dc7e694880fda)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
For some debugging, BB_CURRENTTASK is set too late to be useful as
it isn't present in some event handlers for example. There is no
other way to know which task is actually running so set the value
earlier.
(Bitbake rev: 7daaaaa27f55b5a458656857c6d61a51b34a62fe)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The checkstatus semantics are unconventional, so document them.
(Bitbake rev: efb949020215b580e7dc3694c377b18df2fd7e9c)
Signed-off-by: Ross Burton <ross.burton@arm.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
We have a choice of policy with hashequivalence - whether to reduce
sstate duplication in the sstate feed to a minimum or have maximal
sstate reuse from the user's perspective.
The challenge is that non-matching outhashes are generated due to
determinism issues, or due to differences in host gcc version,
architecture and so on and the question is how to reconcile then.
The approach before this patch is that any new match is added and
matches can update. This has the side effect that a queried value
from the server can change due to the replacement and you may not
always get the same value from the server. With the client side
caching bitbake has, this can be suboptimal and when using the
autobuilder sstate feed, it results in poor artefact reuse.
This patch switches to the other possible behaviour, once a hash is
assigned, it doesn't change. This means some sstate artefacts may be
duplicated but dependency chains aren't invalidated which I suspect
may give better overall performance.
Update the tests to match the new behaviour.
(Bitbake rev: 20d6ac753efa364349100cdc863e5eabec8e5b78)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Fixes the hashequivalence server to resolve the diverging report race
error. This error occurs when the same task(hash) is run simultaneous on
two different builders, and then the results are reported back but the
hashes diverge (e.g. have different outhashes), and one outhash is
equivalent to a hash and another is not. If taskhash was not originally
in the database, the client will fallback to using the taskhash as the
suggested unihash and the server will see reports come in like:
taskhash: A
unihash: A
outhash: B
taskhash: C
unihash: C
outhash: B
taskhash: C
unihash: C
outhash: D
Note that the second and third reports are the same taskhash, with
diverging outhashes.
Taskhash C should be equivalent to taskhash (and unihash) A because they
share an outhash B, but the server would not do this when tasks were
reported in the order shown.
It became clear while trying to fix this that single large table to
store all reported hashes was going to make these updates difficult
since updating the unihash of all entries would be complex and time
consuming. Instead, it makes more sense to split apart the database into
two tables: One that maps taskhashes to unihashes and one that maps
outhashes to taskhashes. This should hopefully improve the parsing query
times as well since they only care about the taskhashes to unihashes
table, at the cost of more complex INNER JOIN queries on the lesser used
API.
Note this change does delete existing hash equivlance data and starts a
new database table rather than converting existing data.
(Bitbake rev: dff5a17558e2476064e85f35bad1fd65fec23600)
Signed-off-by: Joshua Watt <JPEWhacker@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Prevents `ResourceWarning: unclosed event loop` warnings when using the
synchronous client and python exits
(Bitbake rev: 8b95972bc04ce52a98c7780184af15a5e95f987b)
Signed-off-by: Joshua Watt <JPEWhacker@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
https has been the preferred way to access websites for many years now.
Change all of the URLs with a _working_ https server/certificate to use
that URL.
(Bitbake rev: da543cdaf88a387675e25d3555765f1146e4105e)
Signed-off-by: Jon Mason <jdmason@kudzu.us>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
If the build is interrupted, handle the shutdown of pseudo even in this
case to avoid data corruption inside docker containers.
[YOCTO #14555]
(Bitbake rev: a2a04c6fe94bc56efcff299c669a151746e35916)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
With the changes to task accounting, bitbake doesn't show progress
when executing setscene tasks on the summary console output.
Change to show a progress within the setscene tasks and a progress
within the main tasks. I can't see any way to display this more easily
without confusing users.
[YOCTO #14586]
(Bitbake rev: 0244acb968eb593d2ad7bc6e52f222c2b1d39aa9)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Create a configuration npmrc per npm environment to avoid repeated
creation of the same configuration file. Create the file via python to
avoid multiple npm config calls and add the ability to pass a file
path instead of a temporary file.
Deprecate the npm configs argument of the run function. The configs
should be passed to npm environment or as command specific arguments.
(Bitbake rev: 2c2df49b06a2bad7a5b8872a9998338a4660498f)
Signed-off-by: Stefan Herbrechtsmeier <stefan.herbrechtsmeier@weidmueller.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Delay directory restore and set execute/search directory mode bits in
unpack to support npm archives with a missing search directory mode.
(Bitbake rev: 60cbd34d3da8f0f523281aad7eec93eec9cd4db8)
Signed-off-by: Stefan Herbrechtsmeier <stefan.herbrechtsmeier@weidmueller.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The cached revisions which are used to decide if a repository doesn't
need to be updated are misleading when used in conjunction with mirror
tarballs and can cause partial fetches to happen, resulting in unpack
errors as repositories were not fetched.
A concrete example: edk2-firmware in meta-arm is at version 202102
(ef91b0). This is built on the autobuilder so the source mirror contains
the repository as a mirror tarball. If I build edk2-firmware 202102 the
gitsm fetcher will initially download the top-level repository and then
iterate into the submodules to also fetch those repositories, including
cmocka from cryptomilk.org. edk2-firmware will then unpack and build
successfully.
I then update edk2-firmware to 202105 (e1999b) and build it.
Gitsm.needs_update() starts by calling Git.needs_update() which returns
False, as the mirror tarball contains this revision. It then looks at
the "nuggets" which are SRCREVs it has fetched before. The mirror
tarball itself contains the nugget for e1999b as this has been built on
the autobuilder, so needs_update return False, no more fetching is done,
and the build proceeds to unpack.
However, as part of the 202105 upgrade the URL of the cmocka submodule
changed, and this new repository was never fetched. This means that
unpack fails as one of the required git repositories isn't available.
The nugget codepaths appear to be an attempt at optimising the fetch
process, but have demonstratable failure cases. Just removing them
entirely solves the edk2-firmware example, and all of the fetcher test
cases still pass.
(Bitbake rev: 51212507ce3f670ace9efb691c92887d66f7aaf8)
Signed-off-by: Ross Burton <ross.burton@arm.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
If runfetchcmd() fails with bb.process.NotFoundError, the message output
is simply "Fetch command" which doesn't really explain what the problem
is.
Add "not found" to clarify what happened.
(Bitbake rev: 8de9dc02ed6a73b47f2ab10be30d1aed7954bc72)
Signed-off-by: Ross Burton <ross.burton@arm.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Set a prefix so the temporary directories are identifable.
(Bitbake rev: c3440b82cbe9c317f9961d61e12ea37fc9541ce0)
Signed-off-by: Ross Burton <ross.burton@arm.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Avoids a warning from Python as \s isn't a valid escape.
(Bitbake rev: be39c2e926beebe78030fbe26c6737f08f960fcb)
Signed-off-by: Ross Burton <ross.burton@arm.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
When writing functions into shell scripts, write then in a deterministic
order. This is unlikely to affect anything at runtime but it does change
the signatures of the generated useradd postinst scripts in OE-Core and is
a good thing to be consistent about in general.
(Bitbake rev: 8a181dc8f3c8c9c9885ea3011cb234321a296d92)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
We need a command genetating an event to test through the tinfoil API. The
current test has IO load issues so add a dummy version which won't have
the IO constraints.
(Bitbake rev: a144178584394ea0700ffc2f2bfac94e8690effc)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The diff output isn't deterministic at the moment as the sets can have differing
ordering. Sort the output so it is consistent.
(Bitbake rev: 117830c1d7ef3e53052fa326e1ca62c5c3946c45)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
We have a problem where pseudo server processes exist after bitbake exits
and hold the pseudo database in memory. In a docker container, the processes
will be killed as the container is destroyed with no warning and no opportunity
to write the data to disk. This leads to permissions/inode corruptions and
data loss.
Send a shutdown message to pseudo which in new versions of pseudo will flush
the database, thereby fixing some of the issues people using docker containers
see.
(Bitbake rev: a07a971b40acd3eee12e203d2cfa3e49f56109f6)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
There is a race between sending the TaskFailed event which could trigger
the UI to look at the logs and flushing and closing the log files.
Reverse the order of the finally clause and the exception handling
to ensure we've handled the logfiles before sending the task events.
This should fix a race seen in bblogging.BitBakeLogging.test_python_exit_logging
(Bitbake rev: 032190aac31604d37740d8aecf6e74a5448de358)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Currently we see things like:
Log data follows:
| DEBUG: Executing python function do_pythontest_exit
| DEBUG: Python function do_pythontest_exit finished
| ERROR: 1
| This is python stdout
Whilst after the change we see things like:
Log data follows:
| DEBUG: Executing python function do_pythontest_exit
| This is python stdout
| DEBUG: Python function do_pythontest_exit finished
| ERROR: 1
since the output is now correctly mixed with the log messages. In some cases the logging
tests indicate the output is being lost entirely which is bad for debugging and makes
things rather confusing.
(Bitbake rev: 8966b43761500e0505333d8c9a3f0f2c3dbe7559)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Sometimes the peg revision functionality we use in the svn fetcher is
not the correct option. Add a parameter nopegrevision which can be added
to disable this behaviour.
[YOCTO #6258]
(Bitbake rev: ea26682b706f655a8e418f56bebe742e2d130a1d)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
If a "None" value gets into the varlist, it doesn't display properly.
Ensure we don't add one to have the exception display properly.
(Bitbake rev: ee26e258888114143e66330c256b5bfe7d071c53)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Fix:
lib/bb/tests/fetch.py:1288: DeprecationWarning: invalid escape sequence
for several lines of the fetch tests.
(Bitbake rev: 9d84fd557a3fcbae2cdd70b24e69325ad737a01e)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
os.rename will overwrite the destination file if present so we can use this
instead of the process call overhead.
(Bitbake rev: b3cccaa6a896c41d8c9be5eebc327f726542d16b)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
