Version bump is needed after "siggen.py: Improve taskhash reproducibility"
otherwise there will be taskhash mismatch errors
(Bitbake rev: 046871d9fd76efdca7b72718b328d8f545523f7e)
Signed-off-by: Steve Sakoman <steve@sakoman.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Signed-off-by: Steve Sakoman <steve@sakoman.com>
file checksums are part of the data checksummed
to generate the task hash. The list of file checksums
was not ordered.
In this commit we make sure the task hash checksum takes
a list of checksum data that is ordered by unique file name
thus guaranteeing reproducibility.
(Bitbake rev: da5f41996687e18b78d9c9845e621d832115aa1e)
Signed-off-by: Paulo Neves <paulo@myneves.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Signed-off-by: Martin Jansa <martin.jansa@gmail.com>
Signed-off-by: Steve Sakoman <steve@sakoman.com>
>From git 2.48 release notes:
"""
When "git fetch $remote" notices that refs/remotes/$remote/HEAD is
missing and discovers what branch the other side points with its
HEAD, refs/remotes/$remote/HEAD is updated to point to it.
"""
This means with git 2.48 onwards, there is a mystery "HEAD" revision
appearing in some of our shallow clone tests. We can avoid this by
using the same canonicalization as used for the reference revisions.
This resolves autobuilder failures on the Fedora 40 workers.
(Bitbake rev: e71f1ce53cf3b8320caa481ae62d1ce2900c4670)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
(cherry picked from commit c83444d1210740e27b1744d3aa7c5cad4e28db2f)
Signed-off-by: Steve Sakoman <steve@sakoman.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Signed-off-by: Steve Sakoman <steve@sakoman.com>
Testing shows the worst case CDN response time can be up to 100s. The wget fetcher
is used for accessing sstate from the CDN so increase our timeouts there to match
our worst case repsonse times.
(Bitbake rev: 3f88b005244a0afb5d5c7260e54a94a453ec9b3e)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Signed-off-by: Steve Sakoman <steve@sakoman.com>
Python 3.13 emits a ResourceWarning for unclosed sqlite3 `Connection`s.
See https://docs.python.org/3/whatsnew/3.13.html#sqlite3
The previous commit fixed persist_data's context manager to close the
connection, but we were never actually using `with` in the first place.
This change is not necessary on 'master' because persist_data was
removed.
(Bitbake rev: 9789c55ecc90ba074596061fa16e90d3e8accb02)
Signed-off-by: Chris Laplante <chris.laplante@agilent.com>
(cherry picked from commit 61f803c7d92a012b62837b0cdae4789a394b260e)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Signed-off-by: Steve Sakoman <steve@sakoman.com>
A Connection's __exit__ does not close the connection, but rather deals
with transactions. See https://docs.python.org/3/library/sqlite3.html#how-to-use-the-connection-context-manager
This change is not necessary on 'master' because persist_data was
removed.
(Bitbake rev: 97856a56cbf09401b34d6c9bb5444c07a88772ce)
Signed-off-by: Chris Laplante <chris.laplante@agilent.com>
(cherry picked from commit b0fba0f5af555ff970406b5cf3bf8fb113d953d1)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Signed-off-by: Steve Sakoman <steve@sakoman.com>
Commits 4591011449212c8e494ea42348acb2d27a82a51b and
6c19b6cf105ac321ec89da1a876a317020c45ab7 unconditionally changed
codeparser to rely on CPython 3.8 semantics. However, kirkstone
continues to support CPython versions >= 3.6.0 and as such string AST
nodes were no longer correctly identified.
Fix this by continuing to use `ast.Str` for Python versions < 3.8.0 and
only using the new code path for more recent versions. Detecting which
version of the AST API to use seems to be non-trivial so the Python
feature version is used instead.
Instances of this issue can be identified when executing bitbake with
debug logging:
while parsing MACHINE_ARCH, in call of d.getVar, argument
''TUNE_PKGARCH'' is not a string literal
As a consequence of these parsing issues, bitbake may assume that task
inputs haven't changed and as such erroneously reuse sstate objects when
it shouldn't.
(Bitbake rev: fb73c495c45d1d4107cfd60b67a5b4f11a99647b)
Signed-off-by: Philip Lorenz <philip.lorenz@bmw.de>
Signed-off-by: Steve Sakoman <steve@sakoman.com>
GNOME gitlab has occasional downtime which impacts bitbake-selftest
and causes autobuilder failures. Switch to our own mirror for test
purposes to avoid those issues.
(Bitbake rev: 0c30e9aadd30fc6f0dcf811eb8340687b52eb00b)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
(cherry picked from commit 91e268b11ed683bd197026f9b36001f6d54ee05c)
Signed-off-by: Steve Sakoman <steve@sakoman.com>
GNOME gitlab has occasional downtime which impacts bitbake-selftest
and causes autobuilder failures. Switch to our own mirror for test
purposes to avoid those issues.
(Bitbake rev: e4ec4267e4c0818a1682f8a1a4bf3d1419e509a1)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
(cherry picked from commit 008808755ed6cfeb6c41273e69ce718f0833c26c)
Signed-off-by: Steve Sakoman <steve@sakoman.com>
The tmpdir such as downloads/tmplp3cnemv won't be removed without this fix.
(Bitbake rev: 15582daed9a18330bcf1ad316a57d46571bbf7c6)
Signed-off-by: Robert Yang <liezhi.yang@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
(cherry picked from commit 2ba8d3214759142afc11f0a88d80eb30a8bcde3a)
Signed-off-by: Steve Sakoman <steve@sakoman.com>
There are 14 lines can be removed, and can make it easy to maintain.
(Bitbake rev: ff2dfda55258d8034ea748d87222e51124a03f02)
Signed-off-by: Robert Yang <liezhi.yang@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
(cherry picked from commit 0ea2c1ac079d63349407a69172ff80cd9acc7252)
Signed-off-by: Steve Sakoman <steve@sakoman.com>
(Bitbake rev: c092f7e6c5e07a829173b25e591ab0326e9dcb67)
Signed-off-by: Jose Quaresma <jose.quaresma@foundries.io>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Signed-off-by: Steve Sakoman <steve@sakoman.com>
The pipes module is removed in python 3.13. It was already using the
quote function from shlex so use that directly instead.
The module already imports shlex too so it is an easy substitution.
(Bitbake rev: 53264dc14554890b3a2afc83cb1749cf10d86854)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Signed-off-by: Steve Sakoman <steve@sakoman.com>
On Fedora 39 and Fedora 40 hosts, this version of the hash server
exhibits different behavior on exit when using Unix Domain sockets.
Instead of closing the client connections and exiting immediately, the
server will wait until all clients have disconnected before exiting. It
is unknown why this changed, or why it only affects Unix Domain sockets
and not TCP sockets.
Because of this behavior change, the Slow Server Start test is failing
on these hosts. This test is primarily concerned with ensuring that the
server will actually exit, even if it gets a termination signal before
it enters its main loop, and doesn't really care about clients. As such,
modify the test so that a client is not pre-connected to the server.
This allows the server to actually exit so that the test can verify the
signal behavior.
The latest version of the hash equivalence server (on master) does not
exhibit this behavior. Speculation is that the more standardized usage
of asyncio allows the server to properly exit, even if clients are still
connected. Regardless, this patch is only intended for the older
versions, and should not be applied to master
(Bitbake rev: eb5c1ce6b1b8f33535ff7b9263ec7648044163ea)
Signed-off-by: Joshua Watt <JPEWhacker@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Signed-off-by: Steve Sakoman <steve@sakoman.com>
Some distributions (namely Fedora Core 40) have started replacing
wget with wget2. There are some changes to wget2 that make it
incompatible with wget:
1. ftp/ftps is not supported anymore
2. progress 'dot' is not yet supported
3. Relative paths in -P and -O are not correctly dealt with
Item 1: Is already dealt with since Scarthgap by only adding the
option --passive-ftp when the URL specifies ftp/sftp. While that
won't help if ftp/sftp is actually required it at least does
not break http/https downloads.
Item 2: While not supported it at least does not break the operation.
Item 3: If there are relative path components in -P or -O then wget2
only deals with them correctly if there is one, and only one, relative
path component at the beginning of the path:
-P ./downloads works
-P ../downloads works
-P ../../downloads does not work
-P ./../downloads does not work
-P /home/user/downloads/../downloads does not work
In cases where there are more than one relative path component at
the beginning of the path and/or one or more reltaive path
component somewhere in the middle or end of the path, wget2 aborts
with the message Internal error: Unexpected relative path: '<path>')
Such can happen if DL_DIR includes relative path components e.g.
DL_DIR = "${TOPDIR}/../../downloads".
This patch canonicalizes DL_DIR before it is passed to wget.
(Bitbake rev: 07081a94997142746f7d345c27bc6805231d025d)
Signed-off-by: Rudolf J Streif <rudolf.streif@ibeeto.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
(cherry picked from commit 3e4208952b086adc510e78c1c5f9cf4550d79dc9)
Signed-off-by: Steve Sakoman <steve@sakoman.com>
(cherry picked from commit 47678142e26bb76d1351886060deff5e75039bc9)
Signed-off-by: Steve Sakoman <steve@sakoman.com>
On Fedora40, "localhost" sometimes resolves to ::1 and sometimes to 127.0.0.1
and python only binds to one of the addresses, leading to test failures.
Use 127.0.0.1 explicitly to avoid problems of the name resolution, we're trying
to test things other than the host networking.
(Bitbake rev: 3147ebb0edcca8f7f850aabd86ef010ab593496f)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
(cherry picked from commit 9adc6da42618f41bf0d6b558d62b2f3c13bedd61)
Signed-off-by: Steve Sakoman <steve@sakoman.com>
Fedora 40 introduces wget2 as a drop-in replacement for wget. This
rewrite does not currently have support for FTP. This causes
the wget fetcher to fail complaining about an unrecognized option.
Making --passive-ftp conditional based on the protocol used in
the SRC_URI limits the scope of the problem. It also gives us
an opportunity to build the older wget as a host tool.
(Bitbake rev: 477f90f9515cb7cd093920cee5518f662cef2e05)
Signed-off-by: Rob Woolley <rob.woolley@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
(cherry picked from commit f10e630fd7561746d835a4378e8777e78f56e44a)
Signed-off-by: Steve Sakoman <steve@sakoman.com>
Fixed:
- BBMULTICONFIG = "qemux86-64 qemuarm64" and more than 70 layers in BBLAYERS
$ bitbake -p -P
Check profile.log.processed, the record() cost more than 20 seconds, it is less
than 1 second when multiconfig is not enabled, and there would be the following
error when more muticonfigs are enabled:
Timeout while waiting for a reply from the bitbake server
Don't change the type of loginfo['detail'] or re-assign it can make record()
back to less than 1 second, this won't affect COW since loginfo is a mutable
type.
The time mainly affected by two factors:
1) The number of enabled layers, nearly 1 second added per layer when the
number is larger than 50.
2) The global var such as USER_CLASSES, about 1 ~ 2 seconds added per layer
when the layers number is larger than 50.
(Bitbake rev: ec2a99a077da9aa0e99e8b05e0c65dcbd45864b1)
Signed-off-by: Robert Yang <liezhi.yang@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
(cherry picked from commit 0596aa0d5b0e4ed3db11b5bd560f1d3439963a41)
Signed-off-by: Steve Sakoman <steve@sakoman.com>
We have been seeing obscure failures in devtool, particularly on newer
autobuilder workers where it appears the cache is assumed to be valid
when it shouldn't be.
We're using the 'seconds' granulation mtime field which is not really
a good way of telling if a file has changed. We can switch to the "ns"
version which is better however also add in inode number and size as
precautions. We already have all this data and tuples are fast so there
isn't really any cost to do so.
This hopefully fixes [YOCTO #15318].
(Bitbake rev: 5a90927f31c4f9fccbe5d9d07d08e6e69485baa8)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
(cherry picked from commit d9e5d313c79500e3c70ab9c3239b6b2180194f67)
Signed-off-by: Steve Sakoman <steve@sakoman.com>
The current implementation only performs a git lfs fetch alongside of a
regular git fetch. This causes issues when the downloaded revision is
already part of the fetched repository (e.g. because of moving back in
history or the updated revision already being part of the repository at
the time of the initial clone).
Fix this by explicitly checking whether the required LFS objects are
available in the downloade directory before confirming that a downloaded
repository is up-to-date.
This issue previously went unnoticed as git lfs would silently fetch the
missing objects during the `unpack` task. With network isolation turned
on, this no longer works, and unpacking fails.
(cherry picked from commit cfae1556bf671acec119a6c8bbc4b667a856b9ae)
(Bitbake rev: 40fd5f4eef7460ca67f32cfce8e229e67e1ff607)
Signed-off-by: Philip Lorenz <philip.lorenz@bmw.de>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Signed-off-by: Philip Lorenz <philip.lorenz@bmw.de>
Signed-off-by: Steve Sakoman <steve@sakoman.com>
Added tests that verify that git-lfs works with an actual
real git-lfs server. This was not previously the case because
the repo in the test was a simulation of git-lfs but not
a real git lfs repo.
The 2 added tests are almost the same but test that the
git lfs file checkout is successfult with or without the
lfs=1 flag. The lfs=1 URI parameter is a quirk that triggers
2 different code paths for git lfs.
lfs=1, when used on git lfs repositories triggers the git lfs
downloading at the fetch bare stage.
lfs query parameter unset triggers the git lfs downloading only
on checkout as an implicit behavior of git. This leads to possible
network access on the unpack stage and outside the DL_DIR.
lfs=0 actually disables git-lfs functionality even if supported.
(cherry picked from commit d2be7f7f652360f13cd66d0850f3e19ffe2afb0a)
(Bitbake rev: 2a6fd774405a58f3ef2953a0dc3eca063e0cf890)
Signed-off-by: Paulo Neves <paulo@myneves.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Signed-off-by: Philip Lorenz <philip.lorenz@bmw.de>
Signed-off-by: Steve Sakoman <steve@sakoman.com>
Not restoring the mocked _find_git_lfs leads to other tests
failing.
(cherry picked from commit 70f848631450bd723c223227c21c60e815ee033d)
(Bitbake rev: 1d5d4a46c772f44ab652f95b37f508b890828e67)
Signed-off-by: Paulo Neves <paulo@myneves.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Signed-off-by: Philip Lorenz <philip.lorenz@bmw.de>
Signed-off-by: Steve Sakoman <steve@sakoman.com>
Python 3 interprets string literals as Unicode strings, and therefore
\s is treated as an escaped Unicode character which is not correct.
Declaring the RegEx pattern as a raw string instead of unicode is
required for Python 3.
(Bitbake rev: 7efed7bc09c56e41d3074a26388581a62f145625)
Signed-off-by: Adrian Freihofer <adrian.freihofer@siemens.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Signed-off-by: Steve Sakoman <steve@sakoman.com>
Specifically:
/srv/work/alex/poky/bitbake/lib/bb/codeparser.py:279: DeprecationWarning: ast.Str is deprecated and will be removed in Python 3.14; use ast.Constant instead
if isinstance(node.args[0], ast.Str):
/srv/work/alex/poky/bitbake/lib/bb/codeparser.py:280: DeprecationWarning: Attribute s is deprecated and will be removed in Python 3.14; use value instead
value = node.args[0].s
(Bitbake rev: 6c19b6cf105ac321ec89da1a876a317020c45ab7)
Signed-off-by: Alexander Kanavin <alex@linutronix.de>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Signed-off-by: Steve Sakoman <steve@sakoman.com>
These have been deprecated since 3.8
(Bitbake rev: 4591011449212c8e494ea42348acb2d27a82a51b)
Signed-off-by: Chris Laplante <chris.laplante@agilent.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Signed-off-by: Steve Sakoman <steve@sakoman.com>
Adds context manager API for the asyncrcp client class which allow
writing code that will automatically close the connection like so:
with hashserv.create_client(address) as client:
...
Rework the bitbake-hashclient tool and PR server to use this new API to
fix warnings about unclosed event loops when exiting
(Bitbake rev: ee090484cc25d760b8c20f18add17b5eff485b40)
Signed-off-by: Joshua Watt <JPEWhacker@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
(cherry picked from commit d01d684a0f6398270fe35ed59b7d28f3fd9b7e41)
Signed-off-by: Steve Sakoman <steve@sakoman.com>
With python 3.8 and 3.9, we see intermittent errors of:
libgcc_s.so.1 must be installed for pthread_cancel to work
Aborted (core dumped)
which seem related to:
https://stackoverflow.com/questions/64797838/libgcc-s-so-1-must-be-installed-for-pthread-cancel-to-workhttps://bugs.ams1.psf.io/issue42888
These tend to occur on debian 11 and ubuntu 20.04.
Workaround this by ensuring libgcc is preloaded in all cases.
(Bitbake rev: 2c6183594279e2e9d03f11155ad969448869c863)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Signed-off-by: Steve Sakoman <steve@sakoman.com>
A recipe variable handles its dependencies even on the "contains"
variables within the "inline Python expressions" like bb.utils.filter().
And it also handles those in the append operator correctly, but the
problem is that it does not so in the remove operator.
Fix it by adding the missing dependencies every time the remove
operator has been handled.
Also add a test case to check if the override operators handle
dependencies correctly.
(Bitbake rev: 23639edfbbb3fced7606dce211db8a31c5766585)
Signed-off-by: Insu Park <insu0.park@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Cherry-picked from master: b90520eedb1dbc7f6a3928d089fe74fafb864eb5
- Conflicts in data.py are resolved as the master branch moved
handle_contains() and handle_remove() out of the try block.
- The test code in codeparser.py are modified as the master branch
added three more arguments to the build_dependencies().
Signed-off-by: Insu Park <insu0.park@gmail.com>
Signed-off-by: Steve Sakoman <steve@sakoman.com>
Verify that an existing layer path was given when adding a new
layer.
Manually using the shell for globbing is unnecessary, use the glob
function instead for cleaner code.
(Bitbake rev: de1cf0e31f3836a449cbd17490a2f6b6a86b17f9)
Signed-off-by: Marta Rybczynska <marta.rybczynska@syslinbit.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Signed-off-by: Steve Sakoman <steve@sakoman.com>
* it might be a bit confusing as it shows number of threads before making
the decision to start more tasks and also it can show only a few tasks
running, but not because of pressure when there just aren't many tasks
left or wait for their dependencies to be finished first
* example output:
NOTE: Pressure status changed to CPU: True, IO: None, Mem: None (CPU: 297589.5/200000.0, IO: 5522.2/None, Mem: 779.2/None) - using 7/8 bitbake threads
NOTE: Pressure status changed to CPU: False, IO: None, Mem: None (CPU: 196381.2/200000.0, IO: 2667.9/None, Mem: 556.2/None) - using 2/8 bitbake threads
(Bitbake rev: 42a1c9fe698a03feb34c5bba223c6e6e0350925b)
Signed-off-by: Martin Jansa <Martin.Jansa@gmail.com>
Signed-off-by: Steve Sakoman <steve@sakoman.com>
* with latest bitbake I'm seeing very low number of bitbake tasks
executed in parallel, probably due to pressure regulation
show the values this is based on in the note
* also simplify a bit by counting the pressure and exceeds signs
only once
(Bitbake rev: 1050145c3004861ebede4777fd4fbd89d0470716)
Signed-off-by: Martin Jansa <Martin.Jansa@gmail.com>
Signed-off-by: Steve Sakoman <steve@sakoman.com>
The current calculation is not correct because if tdiff is less than
1.0, it's not taken into consideration when calculating the current
pressure.
Also, make it clear that the 1.0s is the psi accumulation cycle,
which might be changed in the future. We have this cycle because it
could largely avoid the 0 result issue, that is, if the interval
between checks are too small, the result might be 0. With this
accumulation logic, which has been there but let's make it clear,
this 0 result problem could be mitigated.
(Bitbake rev: 76889ff0a8938a3d77603d2af176aa9e264df839)
Signed-off-by: Chen Qi <Qi.Chen@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Signed-off-by: Steve Sakoman <steve@sakoman.com>
It is currently hard to tell when bitbake is throttling task execution
due to system pressure changes. Add notes to the console output to make
this clearer, only generating output when the values change.
(Bitbake rev: b5f77e8159ad321f31999af8304f082a2c56b537)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Signed-off-by: Steve Sakoman <steve@sakoman.com>
The current PSI check calculation does not take into consideration
the possibility of the time interval between last check and current
check being much larger than 1s. In fact, the current behavior does
not match what the manual says about BB_PRESSURE_MAX_XXX, even if
the value is set to upper limit, 1000000, we still get many blocks
on new task launch. The difference between 'total' should be divided
by the time interval if it's larger than 1s.
(Bitbake rev: 2b2997ab6bdda730e4b638f416311a73e0c42156)
Signed-off-by: Chen Qi <Qi.Chen@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Signed-off-by: Steve Sakoman <steve@sakoman.com>
Error occured while running bitbake on cephfs:
WARNING: The free inode of path is running low (-0.001K left)
ERROR: Immediately halt since the disk space monitor action is "HALT"!
(Bitbake rev: a7f6c3e67bd9170e93b2b94676e84018faf0df91)
Signed-off-by: Samantha Jalabert <samantha.jalabert@syslinbit.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Signed-off-by: Steve Sakoman <steve@sakoman.com>
Using multiconfig to target baremetal pieces of the system and building
corresponding toolchains for them results in hundreds and hundreds of
"Deferring %s after %s" and "Deferred task %s now buildable".
To clean up the output and to reduce risk of missing important warnings,
convert these notice messages to debug messages.
(Bitbake rev: 3505d8d8c02b041946670ab6bc5751e54fe292ff)
Signed-off-by: Denys Dmytriyenko <denis@denix.org>
Signed-off-by: Denys Dmytriyenko <denys@konsulko.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
(cherry picked from commit 64bc00a46d1aacc23fe7e8d9a46a126f3a4bc318)
Signed-off-by: Steve Sakoman <steve@sakoman.com>
If Tinfoil is initialized with setup_logging = False and
Tinfoil.prepare() is called with config_only = False, then it fails
because self.localhandlers is only initialized when
setup_logging = True.
This is seen with, e.g., `bitbake-getvar -q -r busybox MACHINE`:
Traceback (most recent call last):
File ".../bitbake/bin/bitbake-getvar", line 41, in <module>
tinfoil.prepare(quiet=2)
File ".../bitbake/lib/bb/tinfoil.py", line 390, in prepare
for handler in self.localhandlers:
AttributeError: 'Tinfoil' object has no attribute 'localhandlers'.
Did you mean: 'oldhandlers'?
(Bitbake rev: e452c6d7ba5bb4f78a1d2bfb742794efdf171dbc)
Signed-off-by: Peter Kjellerstedt <peter.kjellerstedt@axis.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
(cherry picked from commit 616101ddb630e2c9975022068b52a87c4cf647f6)
Signed-off-by: Steve Sakoman <steve@sakoman.com>
Initializing Tinfoil with setup_logging = False only has an effect when
recipe parsing is not needed. To make it work regardless of if --recipe
is used, manipulate the quiet parameter to Tinfoil.prepare() instead.
(Bitbake rev: 161ab0d5bab74732e12d490cee50e14295be0a9f)
Signed-off-by: Peter Kjellerstedt <peter.kjellerstedt@axis.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
(cherry picked from commit 71ee69a20f21f3d37f4f060a7d8e87d9f1dc6aa1)
Signed-off-by: Steve Sakoman <steve@sakoman.com>
Add a SECURITY.md file with hints for security researchers and other
parties who might report potential security vulnerabilities.
(Bitbake rev: 6c1ffa9091d0c53a100e8c8c15122d28642034bd)
Signed-off-by: Marta Rybczynska <marta.rybczynska@syslinbit.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Signed-off-by: Steve Sakoman <steve@sakoman.com>
declaring queue=b"" creates an object of types bytes().
bytes() is an immutable object, and therefore doing "self.queue = self.queue + r"
creates a new object containing "self.queue" concatenated with "r".
On my test setup, we are passing 180MB of data of "workerdata" to the bitbake-worker,
so those copies significantly slow down the initialization of the bitbake-worker.
Rather use bytearray() which a mutable type, and use extend() to avoid copies.
In my test setup, byterray.extend() is 10.000 times faster than copying the queue,
for a queue size of 180MB.
(Bitbake rev: 9993a89e5b97dda5f3657e5a7cc3a4fa94ff7111)
Signed-off-by: Etienne Cordonnier <ecordonnier@snap.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Signed-off-by: Steve Sakoman <steve@sakoman.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Signed-off-by: Steve Sakoman <steve@sakoman.com>
bitbake-getvar does not have a way to silence bitbake
server's logger and that makes the tool hard to use for
text processing. This is especially true when one wants to
get a bitbake value to be piped to some other utility and
instead we get uncontrolled logging messages or warnings
together with bitbake's variable value.
Example without quiet:
bitbake-getvar --value MACHINE
NOTE: Starting bitbake server...
qemux86-64
With quiet:
bitbake-getvar --value MACHINE --quiet
qemux86-64
(Bitbake rev: d677daa0770c66010fc4ed0af5bbbb94752ba590)
Signed-off-by: Paulo Neves <ptsneves@gmail.com>
Signed-off-by: Alexandre Belloni <alexandre.belloni@bootlin.com>
(cherry picked from commit af354e975d0b4c26d0e91e3c82946b093bc11b45)
Signed-off-by: Markus Niebel <Markus.Niebel@ew.tq-group.com>
Signed-off-by: Steve Sakoman <steve@sakoman.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Signed-off-by: Steve Sakoman <steve@sakoman.com>
Plugins may want to use it (e.g. the layers-setup plugin that would
want to discover writer sub-plugins with it), and so it makes sense
to make tinfoil available a bit eariler.
(Bitbake rev: 41b6684489d0261753344956042be2cc4adb0159)
Signed-off-by: Alexander Kanavin <alex@linutronix.de>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
(cherry picked from commit 2f6c7523a622f59ddf84a1a196927492bc5fa7a2)
Signed-off-by: Jermain Horsman <jermain.horsman@nedap.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Signed-off-by: Steve Sakoman <steve@sakoman.com>
If there are several multiconfigs in play for example a non-multiconfig with
a task with one hash and then three multiconfigs for the same task, different
architectures but the same hash (different to the non-mc), the three mcs
will be deferred until after the non-mc task but then will all run together
and race against each other.
Change the code to re-enable deferred tasks one at a time. This way, if they do
race, they won't run in parallel against each other.
(Bitbake rev: 907416ee1062f87f5844ab0638b54616abfc1a22)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
(cherry picked from commit 9523e28658ad7fb446645b590608dfac2812afd3)
Signed-off-by: Steve Sakoman <steve@sakoman.com>
Get rid of the duplicate code and add extra check that the
locale en_US.UTF-8 is available on the system. This new helper
method is now located right above the method filter_environment()
which sets LC_ALL environment variable to 'en_US.UTF-8'.
[YOCTO #10165]
(Bitbake rev: 0c6f86b60cfba67c20733516957c0a654eb2b44c)
Signed-off-by: Frank de Brabander <debrabander@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
(cherry picked from commit a4ce040a6fd540a1cac52f808f909f9fcf8c961c)
Signed-off-by: Steve Sakoman <steve@sakoman.com>
The "git clone /path/to/git/objects_symlink" couldn't work after the following
change:
6f054f9fb3
But repo command manages the git repo as symlinks, so check whether the objects
is an symlink to fix the problem:
* Nothing is changed if git/objects is not a symlink
* Use "git clone file://" when git/objects is a symlink
(Bitbake rev: 2802adb572eb73a3eb2725a74a9bbdaafc543fa7)
Signed-off-by: Robert Yang <liezhi.yang@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
(cherry picked from commit a0d8108eba8d542707740d00c66c1c5f5b963f18)
Signed-off-by: Steve Sakoman <steve@sakoman.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
This code appears to be dangerous, it swallows exceptions, turning them into
"handled" versions which then show no errors to the user. This is a pretty
poor user experience and I can't see why this code should be swallowing
such things. Drop the worst bits of code.
(Bitbake rev: a29ffcc12df24737e702e198aeb8f6892884932b)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Signed-off-by: Steve Sakoman <steve@sakoman.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Martin Jansa reported that if you put a syntax error into an imported
module such as qa.py in OE, no error is shown.
Part of the issue appears to be that the catch_parse_error() decorator only
catches certain exceptions and SyntaxError isn't one of them. As far as I can
tell we should remove all the special cases and use the more advanced code
in all cases, not just expansion errors.
I confirmed this now prints a proper error message for a qa.py syntax error.
(Bitbake rev: e66012bfda60ffe1658473e25879aa67909ae65f)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Signed-off-by: Steve Sakoman <steve@sakoman.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
