* Remove double imports
* Re-roder the imports as mentioned by pylint
(Bitbake rev: 6de536bbaee8cf7664c5702a96a5ca18ad09fb9d)
Signed-off-by: Michael Estner <michaelestner@web.de>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Add support for the Go mod h1 hash. The hash is based on the Go dirhash
package. The package defines hashes over directory trees and is uses for
Go mod files and zip archives.
(Bitbake rev: deefb01592f717efba68e3997fefd04dc7611d88)
Signed-off-by: Stefan Herbrechtsmeier <stefan.herbrechtsmeier@weidmueller.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
We should release locks in the opposite order to the order we locked
them in to avoid potentially problematic behaviour. This hasn't been
observed with our current usage and is just a preventative improvement.
(Bitbake rev: 4fbef87adaca2df8c1e877aec5588c2e5466f252)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
When running in a cgroup which is limited to a subset of cpus (via
cpuset.cpus), cpu_count() should return the number of cpus that can be
used instead of the number of cpus the system has.
This also aligns the semantics with nproc.
(Bitbake rev: a029bfe96c6542f178720c72a772b7ede9898118)
Signed-off-by: Felix Moessbauer <felix.moessbauer@siemens.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
preserved_envvars() is used when the BB_ENV_PASSTHROUGH
environment variable is not set. Therefore, its code shouldn't
return this variable.
(Bitbake rev: 0a33b560233b983456178541603ab96fea22238b)
Signed-off-by: Michael Opdenacker <michael.opdenacker@bootlin.com>
Reported-by: Robert P. J. Day <rpjday@crashcourse.ca>
Tested-by: Michael Opdenacker <michael.opdenacker@bootlin.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
In some cases ${ may not be expanded in the WORKDIR (one of the cases is
undefined variable) and causes cryptic failures [1]. Guard this by
erroring out if directory name contains ${
Fixes: [Yocto #15255]
[1]
ERROR: x-native-1.0+${SRCPV}-r0 do_deploy_source_date_epoch: Error executing a python function in exec_func_python() autogenerated:
The stack trace of python calls that resulted in this exception/failure was:
File: 'exec_func_python() autogenerated', lineno: 2, function: <module>
0001:
*** 0002:sstate_hardcode_path(d)
0003:
File: '/home/mischief/src/poky/meta/classes-global/sstate.bbclass', lineno: 654, function: sstate_hardcode_path
0650: bb.note("Removing hardcoded paths from sstate package: '%s'" % (sstate_hardcode_cmd))
0651: subprocess.check_output(sstate_hardcode_cmd, shell=True, cwd=sstate_builddir)
0652:
0653: # If the fixmefn is empty, remove it..
*** 0654: if os.stat(fixmefn).st_size == 0:
0655: os.remove(fixmefn)
0656: else:
0657: bb.note("Replacing absolute paths in fixmepath file: '%s'" % (sstate_filelist_relative_cmd))
0658: subprocess.check_output(sstate_filelist_relative_cmd, shell=True)
Exception: FileNotFoundError: [Errno 2] No such file or directory: '/home/mischief/src/poky/build/tmp/work/core2-64-poky-linux/x-native/1.0+${SRCPV}-r0/sstate-build-deploy_source_date_epoch/fixmepath'
(Bitbake rev: e91c256ec076e70cf8a18e369fe7862e50618c48)
Signed-off-by: Pavel Zhukov <pavel@zhukoff.net>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Adds a utility that checks if one path is an descendant of another. This
check uses os.path.samestat() to make it immune to symlinks and bind
mounts.
(Bitbake rev: c3ae45946886ee2049939dd5a205790657a7de32)
Signed-off-by: Joshua Watt <JPEWhacker@gmail.com>
Signed-off-by: Alexandre Belloni <alexandre.belloni@bootlin.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
We currently have two lists of "proxy" or "fetcher" environment exports.
Make the one in utils match the one on the fetcher which has a more complete
list of variables now.
(Bitbake rev: f9165a798a307a6f0fee120d5c3de660d3a44ae8)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
bb.utils.export_proxies() is a poor-man's alternative for the
environment setup code in bb/fetch2, but it's used in several places
where recipes want to download manually (such as cve-update-db-native).
Notably, export_proxies() doesn't pass on the SSL certificate paths from
the original environment, so if SSL_CERT_FILE needs to be set (for
example, in a buildtools environment) then proxies work but SSL doesn't.
In an ideal world export_proxies and the same logic in fetch2 would
merge, but until then we can add the SSL_CERT_ variables and duplicate
the basic logic: check the datastore first and then the original
environment for variables.
Also remove the return value as nothing ever checked it.
[ YOCTO #15000 ]
(Bitbake rev: c19035e8e71c419c5688a86bfc9c946c96f638e8)
Signed-off-by: Ross Burton <ross.burton@arm.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The thread/process locks we use translate to futexes in Linux. If a
process dies holding the lock, anything else trying to take the lock
will hang indefinitely. An example would be the OOM killer taking out
a parser process.
To avoid bitbake processes just hanging indefinitely, add a timeout to
our lock calls using a context manager. If we can't obtain the lock
after waiting 5 minutes, hard exit out using os._exit(1). Use _exit()
to avoid locking in any other places trying to write error messages to
event handler queues (which also need locks).
Whilst a bit harsh, this should mean we stop having lots of long running
processes in cases where things are never going to work out and also
avoids hanging builds on the autobuilder.
(Bitbake rev: d2a3f662b0eed900fc012a392bfa0a365df0df9b)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Get rid of the duplicate code and add extra check that the
locale en_US.UTF-8 is available on the system. This new helper
method is now located right above the method filter_environment()
which sets LC_ALL environment variable to 'en_US.UTF-8'.
[YOCTO #10165]
(Bitbake rev: a4ce040a6fd540a1cac52f808f909f9fcf8c961c)
Signed-off-by: Frank de Brabander <debrabander@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Some variables may be set as:
X = 1
as well the more usual
X = "1"
so add support to to_boolean to handle this case.
(Bitbake rev: ef9c033b011e68bbfedf7ddf118633c14388aaaf)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
In the case where hashlib is not available, the try would fail and fall
through resulting in a backtrace on the usage of the 'sig'. The backtrace
itself was confusing and made it difficult to determine what went wrong.
Update the import to be in it's own try block with an appropriate
message to indicate what went wrong.
Note, the current version of ply all of this code has been restructured
so this is not applicable upstream.
Additionally, some versions of hashlib don't appear to implement the
second FIPS related argument. Detect this and support both versions.
(Bitbake rev: 484ab42f440070c0369b81f5c69da860fa47a798)
Signed-off-by: Mark Hatle <mark.hatle@amd.com>
Signed-off-by: Mark Hatle <mark.hatle@kernel.crashing.org>
Signed-off-by: Luca Ceresoli <luca.ceresoli@bootlin.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
It can be used to enable the loopback interface, typically after calling
disable_network().
Also correct a typo in a debug message.
(Bitbake rev: 0d317209d4234c5f05a9fcdc13c52f502f104018)
Signed-off-by: Mattias Jernberg <mattias.jernberg@axis.com>
Signed-off-by: Peter Kjellerstedt <peter.kjellerstedt@axis.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Pass additional arguments in the fileslocked() context manager to the
underlying lockfile() function. This allows the context manager to be
used for any types of locks (non-blocking, shared, etc.) that the
lockfile() function supports.
(Bitbake rev: 7a8eb8da8e8495051e174721062da08e06168024)
Signed-off-by: Joshua Watt <JPEWhacker@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Signature generation uses mkstemp() to get a file descriptor to a unique
file and then write the signature into it. However, the unique file name
generation in glibc is based on the system timestamp, which means that
with highly parallel builds it is more likely than one might expect
expected that a conflict will occur between two different builder nodes.
When operating over NFS (such as a shared sstate cache), this can cause
race conditions and rare failures (particularly with NFS servers that
may not correctly implement O_EXCL).
The signature generation code is particularly susceptible to races since
a single "sigtask." prefix used for all signatures from all tasks, which
makes collision even more likely.
To work around this, add an internal implementation of mkstemp() that
adds additional truly random entropy to the file name to eliminate
conflicts.
(Bitbake rev: 97955f3c1c738aa4b4478a6ec10a08094ffc689d)
Signed-off-by: Joshua Watt <JPEWhacker@gmail.com>
Signed-off-by: Luca Ceresoli <luca.ceresoli@bootlin.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
When we call the remove with recurse=True we first check if the
remove operation is safe in _check_unsafe_delete_path.
But the check is been done on the path instaed of the expanded
python glog.
(Bitbake rev: 7236488b898309ec5f1880936ddae22a28ccf5d3)
Signed-off-by: Jose Quaresma <jose.quaresma@foundries.io>
Signed-off-by: Alexandre Belloni <alexandre.belloni@bootlin.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
If the path to bitbake.lock is in a deep directory, bitbake will hang. The
reason was that the max file length limiting code (to 255 chars) was including
the directory name and it should only act on the filename within the directory.
Fix it to just use the base filename.
[YOCTO #14766]
(Bitbake rev: 89d70e7b71eecfe06592202f326e566c579ba01d)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
hashlib.md5() is not permitted on a FIPS enabled host system. This is due
to md5 not being an approved hash algorithm.
Instead use:
hashlib.new('MD5', usedforsecurity=False)
This is allowed, as it's clear the hash is used for a non-security purpose.
Note: utils.py version should never be used to verify file integrity, but
instead be used to identify if the file may have changed. sha256 should be
used for integrity purposes.
(Bitbake rev: af866dd077867cba0129757bfcc689551445e9d7)
Signed-off-by: Mark Hatle <mark.hatle@xilinx.com>
Signed-off-by: Mark Hatle <mark.hatle@kernel.crashing.org>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
In line with the inclusive language migration defined at:
https://wiki.yoctoproject.org/wiki/Inclusive_language
rename:
BB_ENV_WHITELIST -> BB_ENV_PASSTHROUGH
BB_ENV_EXTRAWHITE -> BB_ENV_PASSTHROUGH_ADDITIONS
(Bitbake rev: fe60627839d4280cf0117ed1afbfccdff1181b6a)
(Bitbake rev: 87104b6a167188921da157c7dba45938849fb22a)
Signed-off-by: Scott Murray <scott.murray@konsulko.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
If a python function exec_func() calls a shell task, the logging wasn't working
correctly in all cases since the exception was turned into a BBHandledException()
and the logfile piece was lost which is handled at the top task level.
The easiest way to avoid this is to allow the ExecutionError exceptions to
be raised to a higher level, we don't need the traceback for them.
(Bitbake rev: 7cae11f558f9ff5fd05ef23b789aaef92fb5a327)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
File "/home/pokybuild/yocto-worker/qemuarm/build/build/tmp/work/qemuarm-poky-linux-gnueabi/core-image-sato/1.0-r0/testsdkext/layers/build/bitbake/lib/bb/fetch2/wget.py", line 328, in checkstatus
with bb.utils.environment(**newenv):
File "/home/pokybuild/yocto-worker/qemuarm/build/build/tmp/work/qemuarm-poky-linux-gnueabi/core-image-sato/1.0-r0/testsdkext/buildtools/sysroots/x86_64-pokysdk-linux/usr/lib/python3.10/contextlib.py", line 142, in __exit__
next(self.gen)
File "/home/pokybuild/yocto-worker/qemuarm/build/build/tmp/work/qemuarm-poky-linux-gnueabi/core-image-sato/1.0-r0/testsdkext/layers/build/bitbake/lib/bb/utils.py", line 1737, in environment
del os.environ[var]
File "/home/pokybuild/yocto-worker/qemuarm/build/build/tmp/work/qemuarm-poky-linux-gnueabi/core-image-sato/1.0-r0/testsdkext/buildtools/sysroots/x86_64-pokysdk-linux/usr/lib/python3.10/os.py", line 695, in __delitem__
raise KeyError(key) from None
It is possible the wrapped function might change the environment which can lead
to KeyError issues like this. The key may no longer be in the original environment
so handle that case correctly and avoid the tracebacks.
(Bitbake rev: ae4ce4c21998bb2a12a1e1f0b8e0af8d218b87e2)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The NIS can't work when network is dissable, so preserve network for it, the
error is like:
do_ypcall: clnt_call: RPC: Unable to send; errno = Network is unreachable
Note, enable nscd on the build machine might be a solution, but that isn't
reliable since it depends on whether the network function has been cached or
not.
(Bitbake rev: 4eafae7904bae6e5c6bc50356e8a9077f2e207fa)
Signed-off-by: Robert Yang <liezhi.yang@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Add a function which uses the unshare glibc call to disable networking
in the current process. This doesn't work on older distros/kernels
but will on more recent ones so for now we simply ignore the cases we
can't execute on. uid/gid can be passed in externally so this can
work with pseudo/fakeroot contexts.
(Bitbake rev: 9d6341df611a1725090444f6f8eb0244aed08213)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
This is deprecated in python 3.12 and Fedora 35 is throwing warnings so
move to the new functions.
(Bitbake rev: 68a18fbcb5959e334cf307d7fa8dc63832edb942)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The fetcher mirror code can go crazy creating lock filenames which exceed the
filesystem limits. When this happens, the code will loop/hang.
Handle the filename too long exception correctly but also truncate lockfile
lengths to under 256 since the worst case situation is lockfile overlap
and lack of parallelism.
(Bitbake rev: 63baf3440b16e41ac6601de21ced94a94bdf1509)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
bb.utils.environment() is a context manager to alter os.environ inside
a specific block, restoring it after the block is closed.
(Bitbake rev: 9974848f67581ff7d76cef52a94f505af99b4932)
Signed-off-by: Ross Burton <ross.burton@arm.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Fix some references that missed during the overrides syntax migration or
were incorrect. Thanks to Quentin Schulz <foss@0leil.net> for the patch.
(Bitbake rev: 6184cb07dfa44f5f76f1c423533b4547d80b20ab)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
os.rename can fail for example an incremental build in Docker fails with:
OSError: [Errno 18] Invalid cross-device link
when source and destination are on different overlay filesystems.
Rather than trying to fix every call site, add a wrapper in bb.utils
for renames. We can then handle cross device failures and
fall back to shutil.move. The reason os.rename is still used is
because shutil.move is too slow for speed sensitive sections of code.
[YOCTO #14301]
(Bitbake rev: c5c4e49574ab2a65e06298a0a77bb98b041cf56b)
Signed-off-by: Devendra Tewari <devendra.tewari@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The bitbake logger overrode the definition of the debug() logging call
to include a debug level, but this causes problems with code that may
be using standard python logging, since the extra argument is
interpreted differently.
Instead, change the bitbake loggers debug() call to match the python
logger call and add a debug2() and debug3() API to replace calls that
were logging to a different debug level.
[RP: Small fix to ensure bb.debug calls bbdebug()]
(Bitbake rev: f68682a79d83e6399eb403f30a1f113516575f51)
Signed-off-by: Joshua Watt <JPEWhacker@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
A list of functions that now has a docstring.
* vercmp_string
* explode_dep_versions
* prunedir
* prune_suffix
* to_boolean
* contains_any
* export_proxies
See [YOCTO #9725] for details.
(Bitbake rev: b61ba4a18693a9e553d2a93161feb0bcc1c82384)
Signed-off-by: Milan Shah <mshah@mvista.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Add a umask context manager which can be used to temporarily change the
umask in a 'with' block.
(Bitbake rev: 6c601e68a27e1c60b04c2a61830d1812cc883e09)
Signed-off-by: Ross Burton <ross.burton@arm.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
PEP 3110 changed how exceptions work. 'e' is unbound
after the 'except' clause. See: https://www.python.org/dev/peps/pep-3110/#semantic-changes
(Bitbake rev: b69e97de53eb172ed730993e3b755debaa26f30d)
Signed-off-by: Chris Laplante <chris.laplante@agilent.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The following code costs a lot of time when there are lot of layers and recipes:
for collection in collections:
collection_res[collection] = d.getVar('BBFILE_PATTERN_%s' % collection) or ''
My build has more than 100 layers and 3000 recipes, which calls d.getVar() 300K
(3000 * 100) times and makes 'bitbake-layers show-recipes' very slow, add a
keyword argument to get_file_layer() can fix the problem, it can save about 90%
time in my build (6min -> 40s).
(Bitbake rev: f08a6601c9bb09622855d62e1cedb92fafd2f71d)
Signed-off-by: Robert Yang <liezhi.yang@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
This can make "$ bitbake-layers show-recipes" save about 60% time (14min ->
6min) in my build (more than 3000 recipes)
The command "bitbake-layers show-recipes" calls bb.utils.get_file_layer() with
each recipe, and get_file_layer() compare the file with each item in BBFILES
which makes it very time consuming when there are a lot of recipes and items in
BBFILES. So use BBFILES_PRIORITIZED and exit when file is matched, it doesn't
make sense to go on the loop when file is matched.
And use fnmatchcase to replace of fnmatch since the comparison should be
case-sensitive.
(Bitbake rev: 8d64181d29dc262e066a6114dd51e5f2d04f47de)
Signed-off-by: Robert Yang <liezhi.yang@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
I strongly suspect this function doesn't work with modern python so
and its unused now, drop it.
(Bitbake rev: a3033cea089c66c8b4614e7ee57c166f4262c590)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The value of TERM is leaking into OE-Core postinst-useradd-${PN} scripts,
which in turn can optionally be monitored by buildhistory. Prune the value in
order to make the OE-Core buildhistory output more deterministic.
(Bitbake rev: 0d5cdd0c0d65f2f81c3af0f3767fee86c4142c3a)
Signed-off-by: Jacob Kroon <jacob.kroon@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
This function checks if a string is a semantic version:
https://semver.org/spec/v2.0.0.html
The npm fetcher needs this function to validate its version parameter.
(Bitbake rev: 61ac4e825fa7afbb76282030586abc9ee4ac215c)
Signed-off-by: Jean-Marie LEMETAYER <jean-marie.lemetayer@savoirfairelinux.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The npm fetcher needs these functions to support the subresource
integrity: https://www.w3.org/TR/SRI/
(Bitbake rev: 80e2216e2b41cb6170292009064864449bc48bbe)
Signed-off-by: Jean-Marie LEMETAYER <jean-marie.lemetayer@savoirfairelinux.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Given a start expression, bb.utils.get_referenced_vars returns the
referenced variable names in a quasi-BFS order (variables within the
same level are ordered aribitrarily).
For example, given an empty data store:
bb.utils.get_referenced_vars("${A} ${B} ${d.getVar('C')}", d)
returns either ["A", "B", "C"], ["A", "C", "B"], or another
permutation.
If we then set A = "${F} ${G}", then the same call will return a
permutation of [A, B, C] concatenated with a permutation of [F, G].
This method is like a version of d.expandWithRefs().references that
gives some insight into the depth of variable references.
(Bitbake rev: 076eb5453ca35b8b75b8270efb989d5208095b27)
Signed-off-by: Chris Laplante <chris.laplante@agilent.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
removed unused imports which made the code harder to read, and slightly
but less efficient
(Bitbake rev: 4367692a932ac135c5aa4f9f2a4e4f0150f76697)
Signed-off-by: Frazer Clews <frazer.clews@codethink.co.uk>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Discovered with a recipe under devtool. The ${S}/singletask.lock file (added by
externalsrc.bbclass) was leaked, giving a warning like:
WARNING: <PN>+git999-r0 do_populate_lic: /home/laplante/yocto/sources/poky/bitbake/lib/bb/build.py:582: ResourceWarning: unclosed file <_io.TextIOWrapper name='/home/laplante/yocto/build/workspace/sources/<PN>/singletask.lock' mode='a+' encoding='UTF-8'>
exec_func(task, localdata)
(Bitbake rev: 6beddf6214e22b4002626761031a9e9d34fb04db)
Signed-off-by: Chris Laplante <chris.laplante@agilent.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
md5_file() uses a mmap() window to improve performance when hashing files, so
refactor the code and do the same for SHA1 and SHA256.
(Bitbake rev: ecf87437ff796e17c3e4f210b5803b0136a9e8a4)
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Autobuilder type infrastructure can benefit from deletion of certain files as
background IO due to the way Linux filesystem priority works.
We have problems where build directories as part of oe-selftest being
delete starves the running tasks of IO to the point builds take much
longer to compelte.
Having this option of running the deletion at "idle" helps a lot with
that.
(Bitbake rev: 797354d285f6d624d9adb52bab65823572da0e39)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
