Calling SystemExit doesn't work well with server/client usage since the string
isn't printed to the right place. Use bb.fatal() instead which prints the right
log output and raises and handled exception which then shows correctly on the
UI.
(Bitbake rev: b9ae7164d9e744e8eb9aaff79218f57233a449b7)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Signed-off-by: Steve Sakoman <steve@sakoman.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Some variables may be set as:
X = 1
as well the more usual
X = "1"
so add support to to_boolean to handle this case.
(Bitbake rev: e7df13a61911b7431802af2b4d7472b2aaf346fa)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Signed-off-by: Steve Sakoman <steve@sakoman.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* in recipe with 17 git repos in SRC_URI I've accidentally pasted one SRCREV to
be one character shorter and because fetcher uses:
if not ud.revisions[name] or len(ud.revisions[name]) != 40 or (False in [c in "abcdef0123456789" for c in ud.revisions[name]]):
to decide which SRCREV values are fixed SRCREVs this one was
considered as tag or branch name, because it was only 39 chars long
The original error message wasn't very helpful as it doesn't show
which repo or which SRCREV was considered missing:
do_fetch: Bitbake Fetcher Error: FetchError("Recipe uses a floating tag/branch without a fixed SRCREV yet doesn't call bb.fetch2.get_srcrev() (use SRCPV in PV for OE).", None)
with SRCPV included in PV as error recomments it's a bit better:
bb.data_smart.ExpansionError: Failure expanding variable SRCPV, expression was ${@bb.fetch2.get_srcrev(d)} which triggered exception FetchError: Fetcher failure: Unable to resolve '0a92994d729ff76a58f692d3028ca1b64b145d9' in upstream git repository in git ls-remote output for github.com/Maratyszcza/FP16
The variable dependency chain for the failure is: SRCPV -> PV -> WORKDIR -> T
with this change the first error will read:
do_fetch: Bitbake Fetcher Error: FetchError("Recipe uses a floating tag/branch '0a92994d729ff76a58f692d3028ca1b64b145d9' for repo 'github.com/Maratyszcza/FP16' without a fixed SRCREV yet doesn't call bb.fetch2.get_srcrev() (use SRCPV in PV for OE).", None)
(Bitbake rev: 9bbdedc0ba7ca819b898e2a29a151d6a2014ca11)
Signed-off-by: Martin Jansa <Martin.Jansa@gmail.com>
Signed-off-by: Steve Sakoman <steve@sakoman.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
compare_sigfiles() recursively calculates differences on all dependent
tasks with changed hashes. This is done in arbitrary/alphabetical order, and
only the last of those results is returned, while everything else is discarded.
This changes the behavior to instead return the first difference and not calculate
any more, which significantly speeds up diffs of tasks with many dependencies.
(Bitbake rev: 89f13cd4a927a73de98998c27082c63b07671525)
Signed-off-by: Adriaan Schmidt <adriaan.schmidt@siemens.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
(cherry picked from commit ea6a676c9aa2864c2eff40eea41ba09ce903a651)
Signed-off-by: Steve Sakoman <steve@sakoman.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
This ignores flake8 rules:
* E402 module level import not at top of file
* E501 line too long
(Bitbake rev: 60e05043f83c73a34cd154193e5c40d18a3ed3db)
Signed-off-by: Marius Kriegerowski <marius.kriegerowski@gmail.com>
Signed-off-by: Alexandre Belloni <alexandre.belloni@bootlin.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
(cherry picked from commit e8b176de448dc387c7a578c92b52aef28591038f)
Signed-off-by: Steve Sakoman <steve@sakoman.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
In the case where hashlib is not available, the try would fail and fall
through resulting in a backtrace on the usage of the 'sig'. The backtrace
itself was confusing and made it difficult to determine what went wrong.
Update the import to be in it's own try block with an appropriate
message to indicate what went wrong.
Note, the current version of ply all of this code has been restructured
so this is not applicable upstream.
Additionally, some versions of hashlib don't appear to implement the
second FIPS related argument. Detect this and support both versions.
(Bitbake rev: d26ed38c233583b35ad1451e521f07d9c2329f4e)
Signed-off-by: Mark Hatle <mark.hatle@amd.com>
Signed-off-by: Mark Hatle <mark.hatle@kernel.crashing.org>
Signed-off-by: Luca Ceresoli <luca.ceresoli@bootlin.com>
(cherry picked from commit 484ab42f440070c0369b81f5c69da860fa47a798)
Signed-off-by: Steve Sakoman <steve@sakoman.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
As discussed in https://stackoverflow.com/a/4435752/1710392 , CPython
has an optimization for statements in the form "a = a + b" or "a += b".
It seems that this line does not get optimized, because it has a form a = a + b + c:
data = data + "./" + f.split("/./")[1]
For that reason, it does a copy of data for each iteration, potentially copying megabytes
of data for each iteration.
Changing this line causes SignatureGeneratorBasic::get_taskhash to take 0.06 seconds
instead of 45 seconds on my test setup where SRC_URI points to a big directory.
Note that PEP8 recommends explicitely not to use this optimization which is specific to CPython:
"do not rely on CPython’s efficient implementation of in-place string concatenation for statements in the form a += b or a = a + b"
However, the PEP8 recommended form using "join()" also does not avoid the copy and takes 45 seconds in my test setup:
data = ''.join((data, "./", f.split("/./")[1]))
I have changed the other lines to also use += for consistency only, however those were in the form a = a + b
and were optimized already.
Co-authored-by: JJ Robertson <jrobertson@snap.com>
(Bitbake rev: 590ae6fde9da75db3a368e5c0d47920696c33ebf)
Signed-off-by: Etienne Cordonnier <ecordonnier@snap.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
(cherry picked from commit 195750f2ca355e29d51219c58ecb2c1d83692717)
Signed-off-by: Steve Sakoman <steve@sakoman.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Namespace in this context means a branch, a tag, etc., clarify
it in the description. Also, fix a typo "a any", replace with
plain "any".
This patch is based of feedback on new applied patch
d32e5b0e ("fetch2/git: Prevent git fetcher from fetching gitlab repository metadata")
(Bitbake rev: 86f2fa5261da959cda706c794a0047e5e89d4d6b)
Signed-off-by: Marek Vasut <marex@denx.de>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
(cherry picked from commit b4999425c812b25cb359d5163d11e3c1b030dc28)
Signed-off-by: Steve Sakoman <steve@sakoman.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The bitbake git fetcher currently fetches 'refs/*:refs/*', i.e. every
single object in the remote repository. This works poorly with gitlab
and github, which use the remote git repository to track its metadata
like merge requests, CI pipelines and such.
Specifically, gitlab generates refs/merge-requests/*, refs/pipelines/*
and refs/keep-around/* and they all contain massive amount of data that
are useless for the bitbake build purposes. The amount of useless data
can in fact be so massive (e.g. with FDO mesa.git repository) that some
proxies may outright terminate the 'git fetch' connection, and make it
appear as if bitbake got stuck on 'git fetch' with no output.
To avoid fetching all these useless metadata, tweak the git fetcher such
that it only fetches refs/heads/* and refs/tags/* . Avoid using negative
refspecs as those are only available in new git versions.
Per feedback on the ML, Gerrit may push commits outsides of branches or
tags during CI runs, which currently works with the 'nobranch=1' fetcher
parameter. To retain this functionality, keep fetching everything in case
the 'nobranch=1' is present. This still avoids fetching massive amount of
data in the common case, since 'nobranch=1' is rare. Update 'nobranch'
documentation.
Reviewed-by: Peter Kjellerstedt <peter.kjellerstedt@axis.com>
(Bitbake rev: c17fc1468ab84663b919e2809606b1b8ea2bebd9)
Signed-off-by: Marek Vasut <marex@denx.de>
Signed-off-by: Alexandre Belloni <alexandre.belloni@bootlin.com>
(cherry picked from commit d32e5b0ec2ab85ffad7e56ac5b3160860b732556)
Signed-off-by: Steve Sakoman <steve@sakoman.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
bb.utils.export_proxies() is a poor-man's alternative for the
environment setup code in bb/fetch2, but it's used in several places
where recipes want to download manually (such as cve-update-db-native).
Notably, export_proxies() doesn't pass on the SSL certificate paths from
the original environment, so if SSL_CERT_FILE needs to be set (for
example, in a buildtools environment) then proxies work but SSL doesn't.
In an ideal world export_proxies and the same logic in fetch2 would
merge, but until then we can add the SSL_CERT_ variables and duplicate
the basic logic: check the datastore first and then the original
environment for variables.
Also remove the return value as nothing ever checked it.
[ YOCTO #15000 ]
(Bitbake rev: ed0dcc40f80c48839bac20298013d70043858a4e)
Signed-off-by: Ross Burton <ross.burton@arm.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Commit 0361ecf7eb82c386a9842cf1f3cb706c0a112e77 introduced regression
in submodules path parsing. As the result gitsm fetcher fails on each
submodule which name begins from the name of the parent repo which is
totally valid usecase [Yocto #14045] [1]
Fix the code to error out only if submodule's name is equal to parent
name but not if it's part of it.
[1] https://bugzilla.yoctoproject.org/show_bug.cgi?id=14045#c4
(Bitbake rev: 7e268c107bb0240d583d2c34e24a71e373382509)
Signed-off-by: Pavel Zhukov <pavel@zhukoff.net>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
(cherry picked from commit 3ad27272c18f2bb9edd441f840167a3dabd5407b)
Signed-off-by: Steve Sakoman <steve@sakoman.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
We identified a use case where a native recipe (autoconf-native) was
rebuilt with no change in output yet the sstate for do_package tasks
wasn't being used.
The issue is that do_package tasks have a hard dependency on
pseudo-native:do_populate_sysroot. That task was one of the many
tasks being rehashed when autoconf-native's hash was changed.
If update_tasks processed a recipe before it had processed pseudo-native,
that recipe would be marked as not possible from sstate and would
run the full tasks.
The fix is to split the processing into two passes, first to handle
the existing covered/notcovered updates, then in the second pass,
check whether there are "harddep" issues.
This defers the do_package tasks until after pseudo-native is installed
from sstate as expected and everything works well again.
(Bitbake rev: 72a3afd99e8b785cb2a2f687e71a58e08cdd9c74)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
(cherry picked from commit e479d1e418a7d34f0a4663b4a0e22bb11503c8ab)
Signed-off-by: Steve Sakoman <steve@sakoman.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
CVE-2022-39253 in git meant file:// urls within submodules were disabled. Add
a parameter to the commands in the tests to allow this to continue to work.
(Bitbake rev: c90d57497b9bcd237c3ae810ee8edb5b0d2d575a)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The connect_unix() call had a bug where if a relative path to a socket
was passed (which the non-async client always does), and the current
working directory was changed after the initial call, it would fail to
reconnect if it became disconnected, since the socket couldn't be found
relative to the new current working directory.
To work around this, change the socket connection for UNIX domain
sockets to be synchronous and change current working before connecting.
This isn't ideal since the connection could block the entire event loop,
but in practice this shouldn't happen since the socket are local files
anyway.
Help debugging and resolving from Joshua Watt.
(Bitbake rev: 6603c3e39f1cf746669ec6c9f0be8c6e6ece426e)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
(cherry picked from commit 5964bb67bb20df7f411ee0650cf189504a05cf25)
Signed-off-by: Steve Sakoman <steve@sakoman.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
If submodule refers to specific revision of the parent repository it
causes deadlock in bitbake locking mechanism (lock is acquired to fetch
the parent and cannot be released before all submodules are fetched).
raise FetchError in such situation to prevent deadlocking.
[Yocto 14045]
(Bitbake rev: 987712c4c8fefd86a1f5116c11ee86e296e852ee)
Signed-off-by: Pavel Zhukov <pavel@zhukoff.net>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
(cherry picked from commit 0361ecf7eb82c386a9842cf1f3cb706c0a112e77)
Signed-off-by: Steve Sakoman <steve@sakoman.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Hit this error while building nlf-native recently:
{
"error": {
"summary": "URI malformed",
"detail": ""
}
}
Some poking about led me to discover that:
1) The npm.py tool replaces npm:// with http://, not https://
2) Some versions of the npm tool don't handle 301 redirects properly,
choosing to display the above error instead when using the default
nodejs registry
It would be good to go fix npm to handle the redirect properly, but it
seems like it would also be good to assume secure http when contacting a
registry, hence, this patch
(Bitbake rev: bb5c43220f5f1c3d82334c65aff1ce13008db8d9)
Signed-off-by: Neil Horman <nhorman@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
(cherry picked from commit 2cd76e8aabe4e803c760e60f06cfe1f470714ec7)
Signed-off-by: Steve Sakoman <steve@sakoman.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Where copyright headers were not present, add them to make things
clear.
(Bitbake rev: e591325b2bd901c381003deb96a7b32a7148e93e)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
(cherry picked from commit 1aa338a216350a2751fff52f866039343e9ac013)
Signed-off-by: Steve Sakoman <steve@sakoman.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Signature generation uses mkstemp() to get a file descriptor to a unique
file and then write the signature into it. However, the unique file name
generation in glibc is based on the system timestamp, which means that
with highly parallel builds it is more likely than one might expect
expected that a conflict will occur between two different builder nodes.
When operating over NFS (such as a shared sstate cache), this can cause
race conditions and rare failures (particularly with NFS servers that
may not correctly implement O_EXCL).
The signature generation code is particularly susceptible to races since
a single "sigtask." prefix used for all signatures from all tasks, which
makes collision even more likely.
To work around this, add an internal implementation of mkstemp() that
adds additional truly random entropy to the file name to eliminate
conflicts.
(Bitbake rev: 63bb5591e833de0e7b552963ad9bc4b39e56fda9)
Signed-off-by: Joshua Watt <JPEWhacker@gmail.com>
Signed-off-by: Luca Ceresoli <luca.ceresoli@bootlin.com>
(cherry picked from commit 97955f3c1c738aa4b4478a6ec10a08094ffc689d)
Signed-off-by: Steve Sakoman <steve@sakoman.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
I'm 99% certain this failing of a scenequeue task corrupts runqueue and
causes all kinds of breakage. I'd rather runqueue deadlocked than corrupted
and did weird things so drop this code.
We've seen builds where the deadlock triggers and it then tries to run tasks
where the SQ task already ran with very confusing failures. It is likely it
is this code causing it.
(Bitbake rev: f386298fc056ef130c2eb6dabf25eafbd55f55ca)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
(cherry picked from commit 8efced47fcb47851a370fd6786df6fb377f99963)
Signed-off-by: Steve Sakoman <steve@sakoman.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Tweak the deadlock breaking messages to be explict about which task is
blocked on which other task. The messages currently imply it is "freeing"
the blocking task which is confusing.
(Bitbake rev: d1b84e3cfe9fb8d282d4b700a9fe31891e00d837)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
(cherry picked from commit cf7f60b83adaded180f6717cb4681edc1d65b66d)
Signed-off-by: Steve Sakoman <steve@sakoman.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
We have to prefer one multiconfig over another when deferring tasks, else
we'll have cross-linked build trees and nothing will be able to build.
In the original population code, we sort like this but we don't after
rehashing. Ensure we have the same sorting after rehashing toa void
deadlocks.
(Bitbake rev: 513bfd771d9095fcb6a8bf93806673dbf988a4de)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
(cherry picked from commit 27228c7f026acb8ae9e1211d0486ffb7338123a2)
Signed-off-by: Steve Sakoman <steve@sakoman.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Using the term "Parameter" which is consistent with the
description of SRC_URI parameters in the following text.
(Bitbake rev: ac576d6fad6bba0cfea931883f25264ea83747ca)
Signed-off-by: Michael Opdenacker <michael.opdenacker@bootlin.com>
Reported-by: Quentin Schulz <foss@0leil.net>
Reviewed-by: Quentin Schulz <foss+yocto@0leil.net>
Signed-off-by: Luca Ceresoli <luca.ceresoli@bootlin.com>
(cherry picked from commit 87e42f1202162152c779ccc8bbd06f88f0bdab96)
Signed-off-by: Steve Sakoman <steve@sakoman.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
When we call the remove with recurse=True we first check if the
remove operation is safe in _check_unsafe_delete_path.
But the check is been done on the path instaed of the expanded
python glog.
(Bitbake rev: 280ea5a776436eab7e664fccea2df2e7ce47e586)
Signed-off-by: Jose Quaresma <jose.quaresma@foundries.io>
Signed-off-by: Alexandre Belloni <alexandre.belloni@bootlin.com>
(cherry picked from commit 7236488b898309ec5f1880936ddae22a28ccf5d3)
Signed-off-by: Steve Sakoman <steve@sakoman.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
If the mirrors code is trying to create a symlink and the
parent directory doesn't exist, as might be the case for sstate
mirrors where the fetch is into a subdir, it can silently fail.
Ensure the directory exists in this case to avoid issues.
(Bitbake rev: ff3afb1c1bb236c4a52c62a74f2917071e0af55b)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
(cherry picked from commit eff16e474ee7dc49ae433420a4c8d15d3314a618)
Signed-off-by: Steve Sakoman <steve@sakoman.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Currently if you trigger one of the comment errors, the newline characters
are stripped and the line numbers are incorrect. In one case it prints
the empty line which is also unhelpful.
Rework the code around these errors so the line numbers are correct
and the lines in question are more clearly displayed complete with newlines
so the user can more clearly see the error.
I also added a couple of simplistic test cases to ensure that errors
are raised by the two known comment format errors.
[YOCTO #11904]
(Bitbake rev: 01d27562c11d4b05eb30c7f9fefd58b6599fdd15)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
(cherry picked from commit 712da71b24445c814d79a206ce26188def8fce0a)
Signed-off-by: Steve Sakoman <steve@sakoman.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
As specified by git submodule manual relative urls can start either
with '..' or './', second case was incorrectly managed leading to an
interpretation of urls starting with './' as absoulte urls.
(Bitbake rev: d828cd2a16ddf4f084e61ffe44471483e132653a)
Signed-off-by: Gennaro Iorio <gennaro.iorio@schindler.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
(cherry picked from commit 4a0bd3bcd1f7fc25364df8bbf185ff64881c015b)
Signed-off-by: Steve Sakoman <steve@sakoman.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Like in other sections describing fetchers
(Bitbake rev: bcbe78bbaea0312d61f31f4a51b2bc9e672f1cb7)
Signed-off-by: Michael Opdenacker <michael.opdenacker@bootlin.com>
Signed-off-by: Luca Ceresoli <luca.ceresoli@bootlin.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
(cherry picked from commit c9bab35f6aecbf85ee1a19a7b70e15a80b42471f)
Signed-off-by: Steve Sakoman <steve@sakoman.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Stating that the assignment is done at the end of parsing is misleading.
The weak default value is the value which a variable will expand to if no value
has been assigned to it using any of the assignment operators.
(Bitbake rev: f28dbdf80a7fc2febca227f8cb2b474f5058281e)
Signed-off-by: Jacob Kroon <jacob.kroon@gmail.com>
Signed-off-by: Luca Ceresoli <luca.ceresoli@bootlin.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
(cherry picked from commit 8189f58d0449d16f162b6e8d98c4e5edc6bff875)
Signed-off-by: Steve Sakoman <steve@sakoman.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The f.close() statement should have been removed in
459ad524756a3f9b50feeedf31e33502dceae8d5.
(Bitbake rev: 23221378ff0d8c6908d75d1be3219aae0beee406)
Signed-off-by: Ola x Nilsson <ola.x.nilsson@axis.com>
Signed-off-by: Alexandre Belloni <alexandre.belloni@bootlin.com>
Signed-off-by: Luca Ceresoli <luca.ceresoli@bootlin.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
(cherry picked from commit 9fc1bab6b7e3c0fca3ddec4bc8c7763d2aff8bab)
Signed-off-by: Steve Sakoman <steve@sakoman.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
When atexit functions run, stdout and stderr operations may fail, e.g.
when output is piped to less but has been exited by the user.
This removes error print from output of "bitbake -e sqlite3 | less"
if user presses "q" before bitbake has finished processing:
[Errno 32] Broken pipeError in atexit._run_exitfuncs:
Traceback (most recent call last):
File "/home/builder/src/poky/bitbake/lib/bb/event.py", line 135, in print_ui_queue
sys.stdout.flush()
(Bitbake rev: 65cee11967f60c74fa89bb6d72f32135968a6b87)
Signed-off-by: Mikko Rapeli <mikko.rapeli@bmw.de>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
(cherry picked from commit 35167536c163eb6b7653cbcaad9f65b834d3e2f8)
Signed-off-by: Steve Sakoman <steve@sakoman.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
As reported by Martin Jansa <Martin.Jansa@gmail.com>:
bitbake/lib/bb/cooker.py:16: DeprecationWarning: module 'sre_constants' is deprecated
import sre_constants
it's deprecated since 3.11 with:
https://github.com/python/cpython/issues/91308
The correct replacement for our usage is re.error so use that instead.
(Bitbake rev: c98007217b8e40f1abfdcba709185dc5ddbcd0c2)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
(cherry picked from commit 3c0cd401472ffee06d5a93bdba566cb033851fcf)
Signed-off-by: Steve Sakoman <steve@sakoman.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Very occasionally we see errors in eSDK testing on the autobuilder where the task
hashes in the eSDK don't match what was just built. I was able to inspect one of
these build directories and noticed that the bb_unihashes.dat file in the eSDK
was zero sized. Whilst inspecting the code to understand the cause, I noticed that
updated hashes are not saved out in subsequent updates of the values in the rehash
process.
Add a missing sync call to ensure this happens.
(Bitbake rev: 81a6f490dd1f5f669c75cd2ceb1105ce7a09c6e4)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
(cherry picked from commit 7912dabbcf444a3c3d971cca4a944a8b931e301b)
Signed-off-by: Steve Sakoman <steve@sakoman.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Pass additional arguments in the fileslocked() context manager to the
underlying lockfile() function. This allows the context manager to be
used for any types of locks (non-blocking, shared, etc.) that the
lockfile() function supports.
(Bitbake rev: ce9fe70156e8f909a3a81da017b89ea61bc6fe38)
Signed-off-by: Joshua Watt <JPEWhacker@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The user does need to be told about this but it isn't really a warning,
just something they may need to be aware of. Drop the level accordingly.
(Bitbake rev: bc85c044ec250001855f2f9f0717ac031feab7c2)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Prevent new tasks from being scheduled if the memory pressure is above
a certain threshold, specified through the "BB_MAX_PRESSURE_MEMORY"
variable in the conf/local.conf file. This is an extension to the
following commit and hence regulates pressure in the same way:
48a6d84de1 bitbake: runqueue: add cpu/io pressure regulation
Memory pressure is experienced when time is spent swapping, refaulting
pages from the page cache or performing direct reclaim. This is why
memory pressure is rarely seen but might be useful as a last resort to
prevent OOM errors.
(Bitbake rev: 44c395434c7be8dab968630a610c8807f512920c)
(Bitbake rev: 4ada86cb6b05e6e3aabc8015a6e73aacb14a3388)
Signed-off-by: Aryaman Gupta <aryaman.gupta@windriver.com>
Signed-off-by: Randy Macleod <Randy.Macleod@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Signed-off-by: Steve Sakoman <steve@sakoman.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Prevent the scheduler from starting new tasks if the current cpu or io
pressure is above a certain threshold and there is at least one active
task. This threshold can be specified through the
"BB_PRESSURE_MAX_{CPU|IO}" variables in conf/local.conf.
The threshold represents the difference in "total" pressure from the
previous second. The pressure data is discussed in this oe-core commit:
061931520b buildstats.py: enable collection of /proc/pressure data
where one can see that the average and "total" values are available.
From tests, it was seen that while using the averaged data was somewhat
useful, the latency in regulating builds was too high. By taking the
difference between the current pressure and the pressure seen in the
previous second, better regulation occurs. Using a shorter time period
is appealing but due to fluctations in pressure, comparing the current
pressure to 1 second ago achieves a reasonable compromise. One can look
at the buildstats logs, that usually sample once per second, to decide a
sensible threshold.
If the thresholds aren't specified, pressure is not monitored and hence
there is no impact on build times. Arbitary lower limit of 1.0 results
in a fatal error to avoid extremely long builds. If the limits are higher
than 1,000,000, then warnings are issued to inform users that the specified
limit is very high and unlikely to result in any regulation.
The current bitbake scheduling algorithm requires that at least one
task be active. This means that if high pressure is seen, then new tasks
will not be started and pressure will be checked only for as long as at
least one task is active. When there are no active tasks, an additional task
will be started and pressure checking resumed. This behaviour means that
if an external source is causing the pressure to exceed the threshold,
bitbake will continue to make some progress towards the requested target.
This violates the intent of limiting pressure but, given the current
scheduling algorithm as described above, there seems to be no other option.
In the case where only one bitbake build is running, the implications of
the scheduler requirement will likely result in pressure being higher
than the threshold. More work would be required to ensure that
the pressure threshold is never exceeded, for example by adding pressure
monitoring to make and ninja.
(Bitbake rev: 502e05cbe67fb7a0e804dcc2cc0764a2e05c014f)
(Bitbake rev: f4954b878d404a72156fe98609387aa9c5747af0)
Signed-off-by: Aryaman Gupta <aryaman.gupta@windriver.com>
Signed-off-by: Randy Macleod <randy.macleod@windriver.com>
Signed-off-by: Alexandre Belloni <alexandre.belloni@bootlin.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Signed-off-by: Steve Sakoman <steve@sakoman.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
I realised only the first logging message was being displayed in a given
parsing process. The reason turned out to be the UI handler failing
with a "pop from empty list". The default handler was then lost and
no further messages were processed.
Fix this by catching the exception correctly in the connection writer code.
(Bitbake rev: b8fd6f5d9959d27176ea016c249cf6d35ac8ba03)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
(cherry picked from commit d3e64f64525187f1409531a0bd99df576e627f7f)
Signed-off-by: Steve Sakoman <steve@sakoman.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
We currently have no API to be able to remove all the potential stamps of a
task. It is unusual to need to do this, particularly as you could race against
other things happening in the system but we do have a use case for this in
cleaning up sysroots in OE-Core. The alternative is to mess with CLEANMASK in
OE-Core but that is just going to add potential for errors.
We need the first part of the make_stamp() function so separate that out so
it can be called seperately.
(Bitbake rev: 494fcfcb7c4469915c67f544997104d81c417266)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
(cherry picked from commit 4d671504a25863018ac51c21c005cef0a4d8f05c)
Signed-off-by: Steve Sakoman <steve@sakoman.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
If SRC_URI contains python function that extends vardepvalueexclude its
value is being tracked by sstate-cache, which can lead to rebuilds if
value is set dynamically (for example gerrit replicas).
Return empty string if vardepvalueexclude is checked to fix this
behaviour.
(Bitbake rev: 943701ee8cb55307996545f0237721413edfb168)
Signed-off-by: Tomasz Dziendzielski <tomasz.dziendzielski@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
(cherry picked from commit f5f9a7b89a7d8321f03184e61ad6d5ed8d0f840e)
Signed-off-by: Steve Sakoman <steve@sakoman.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The current fetcher seemed to have some issues that made it difficult when
trying to use the same. This patch fixes the following
* Make consistent use of the path that needs to be used as oscdir
* The path mentioned in os.access in download function was not same as
ud.moddir which would result into invoking of fetch command instead of
update command even if directory already existed
* Before creating oscrc, make sure oscdir exists and create it if it does
not exist
* Updated the configuration to use apiurl and added a new parameter to
control whether http or https needs to be used to connect to apiurl
(Bitbake rev: 8ac6e09447d884e658c556388d6014279c50f202)
Signed-off-by: Gunjan Gupta <viraniac@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
(cherry picked from commit 3ec78686f3c0ea2304097b86a965f9be4b0cb879)
Signed-off-by: Steve Sakoman <steve@sakoman.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
We don't have less in HOSTTOOLS in OE and this can confuse git. Force the
pager to cat to be consistent and minimal everywhere.
(Bitbake rev: 59c16ae6c55c607c56efd2287537a1b97ba2bf52)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
(cherry picked from commit d3d406e8552fdd865dc58b419a84411736475ad2)
Signed-off-by: Steve Sakoman <steve@sakoman.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
File names containing colons cause split() to return a list with more
than two elements, which will lead to a stack trace ending in:
ValueError: too many values to unpack (expected 2)
Split only once at the last colon, thereby making sure that only two
elements are returned.
(Bitbake rev: e14ed7dd866334ab40cd335d9c006e5d7b447abd)
Signed-off-by: Roland Hieber <rhi@pengutronix.de>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
(cherry picked from commit a70a7376a8708bde07959deb5d5842d7f84ee5f8)
Signed-off-by: Steve Sakoman <steve@sakoman.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
We're using the wrong data store when trying to locate siginfo files,
fix this. Thanks to Gregory Lumen <gregorylumen@microsoft.com> for
spotting.
[YOCTO #14774]
(Bitbake rev: 7eb0ef75fd08b6e4ca1e9dca9c96a7b590e5147b)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
(cherry picked from commit 0ed800e19a3197f8e622c8d3b630aae384e60aba)
Signed-off-by: Steve Sakoman <steve@sakoman.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The path has been encoded by urllib.parse.quote(), so decode it back for ssh.
Fixed when fetch from PREMIRRORS via ssh:
$ bitbake bonnie++ libsigc++-2.0 -cfetch
scp: /path/to/downloads/libsigc%2B%2B-2.10.7.tar.xz: No such file or directory
(Bitbake rev: 3969786a787eea34e096b932d52cd02978aacb8e)
Signed-off-by: Robert Yang <liezhi.yang@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
(cherry picked from commit c1c8fc678eb4783cea3974328a5fa8d1b79f1266)
Signed-off-by: Steve Sakoman <steve@sakoman.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
This probably means the osc fetcher isn't being used but fix the missing
parameter.
(Bitbake rev: 73fbb743a2def2037d4053605e77e09d2d8a9fd0)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
(cherry picked from commit a23c201cb6efc5c0abf763c26f905442f0eebb68)
Signed-off-by: Steve Sakoman <steve@sakoman.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
(Bitbake rev: 97e150550a3807c60cfa685abda8dccafc0a1268)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
(cherry picked from commit d720dfa40620e64a557edef527148d58fcb1d858)
Signed-off-by: Steve Sakoman <steve@sakoman.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
This is set at the start of the loop anyway so it does nothing. Drop
the pointless code.
(Bitbake rev: dcf78788daa177bf5c438f33b3c9f7ced7aea8ab)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
(cherry picked from commit e6a3173c9cdf349ccbd4cf612868f92cce8717c8)
Signed-off-by: Steve Sakoman <steve@sakoman.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
(Bitbake rev: 1e5c5efa6fe9175b27b21cd7a5b833bd3af55238)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
(cherry picked from commit 7254eb6b3e8ef504ef2274541dcc55f1d42238c6)
Signed-off-by: Steve Sakoman <steve@sakoman.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
(Bitbake rev: 00f10fe95393728e94339f3512171ebab96f1900)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
(cherry picked from commit 625565087d8c9e7a6a79b0b4f3e5be2d77d5f100)
Signed-off-by: Steve Sakoman <steve@sakoman.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
(Bitbake rev: f9b563dc02535b594975ac0f5dab8e12d11a6ce0)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
(cherry picked from commit 42809f6acb79e39042e81d54c28efb92b7481e44)
Signed-off-by: Steve Sakoman <steve@sakoman.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
(Bitbake rev: 43bb92a2728832eeb1a207809e21404c86ff1710)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
(cherry picked from commit 879f17ecd5ba09e217cef74f6a51339b145e8ef5)
Signed-off-by: Steve Sakoman <steve@sakoman.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The function has a loop where the variable is never used which I was going
to fix but the entire function never seems to be called so remove it entirely.
(Bitbake rev: d739799a1e68dc2ad0414d4ae7d9e079cedcee3c)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
(cherry picked from commit 3bcb20f025907f4e88bbe3d14f5638d5f01010cb)
Signed-off-by: Steve Sakoman <steve@sakoman.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
(Bitbake rev: 609d50a6ccb6f794a3ba6d73a820927eba68891c)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
(cherry picked from commit 140929b404ee1e2f5e0e1a3a1d3aa49fb3759ade)
Signed-off-by: Steve Sakoman <steve@sakoman.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
(Bitbake rev: c6df0e68ef52c10be9e8773e660c3bb5c9d1e915)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
(cherry picked from commit 1c811ad6f10560e7a7fb6830cf83707551ba04bd)
Signed-off-by: Steve Sakoman <steve@sakoman.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
(Bitbake rev: bf221631e4b591c80aadd26c2328586b6a4c9eac)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
(cherry picked from commit aca0ff85109f4b0f3c201c02c3f59cad7ee2e787)
Signed-off-by: Steve Sakoman <steve@sakoman.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
(Bitbake rev: 65413c1cecc7e667b04d73b95de9159c688f7ce3)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
(cherry picked from commit 543315e6463f15ca7ab2b4ef3e8ed41bb4207ccf)
Signed-off-by: Steve Sakoman <steve@sakoman.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Fix an issue where two tests have the same name with one overwriting the
other.
(Bitbake rev: 38842a8150f44f5d31d9bd2b0b6ec0502acc971d)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
(cherry picked from commit da812d938fd79e2cc7bdf355ccf5b0f9ead684c4)
Signed-off-by: Steve Sakoman <steve@sakoman.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
During parsing, Python raises
RuntimeError: dictionary changed size during iteration
in getRuntimeProviders, if you happen to have a recipe
with an explicit RDEPENDS on a dynamic package containing a '+'
character, such as 'gtk+3-locale-en'.
This is because we're using the modified pattern as the
key into the packages_dynamic dict to append to rproviders,
and since that key doesn't exist, the dict is getting modified
to add a new, empty, entry for it. So even without the runtime
error, we'd be generating an incorrect result.
Fix this by using a local variable for modifying the pattern
and using the original key to retrieve the value on a match.
(Bitbake rev: 69d3b86449be23b07f794e302f6e18f3a2c46424)
Signed-off-by: Matt Madison <matt@madison.systems>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
(cherry picked from commit 07de375c3e57f17ab7b47569186f24ecd9896825)
Signed-off-by: Steve Sakoman <steve@sakoman.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
logger.debug was giving an integer value (2) as event message, causing
knotty to crash when running with debug enabled.
bitbake/lib/bb/ui/knotty.py", line 685, in main
event.msg = taskinfo['title'] + ': ' + event.msg
TypeError: can only concatenate str (not "int") to str
Same issue also happens in the original code that was taken from
oe-core (openembedded-core/meta/lib/crate.py honister) / meta-rust.
(Bitbake rev: c212b0f3b542efa19f15782421196b7f4b64b0b9)
Signed-off-by: Ricardo Salveti <ricardo@foundries.io>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Use bitwise operators to manipulate the received event mask in
_ProcessEvent.
Also minor clarification & clean up of the related comments.
(Bitbake rev: 2ab60c7be124d928d304ab1fb73f0dbff29964ae)
Signed-off-by: Peter Kjellerstedt <peter.kjellerstedt@axis.com>
Signed-off-by: Alexandre Belloni <alexandre.belloni@bootlin.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
In `runtaskhashes`, the keys contain the absolute paths to the recipe. When
working with shared sstate caches (where these absolute paths can be different)
we see that compare_sigfiles does not identifiy a changed hash of a dependent
task as "changed", but instead as "removed"&"added", preventing the function
from recursing and continuing the comparison.
By calling `clean_basepaths` before comparing the `runtaskhashes` dicts, we
avoid this.
(Bitbake rev: 7358378b90b68111779e6ae72948e5e7a3de00a9)
Signed-off-by: Adriaan Schmidt <adriaan.schmidt@siemens.com>
Signed-off-by: Alexandre Belloni <alexandre.belloni@bootlin.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Currently, if you use one of the functions from EXPORT_FUNCTIONS,
the meaning of cleandirs and fakeroot are lost. This leads to the function
changing in behaviour depending upon it's caller context. This isn't intended
so add mapping for the cleandirs and fakeroot flags too.
This does break devtool in OE-Core and there is a separate fix for that.
[YOCTO #8621]
(Bitbake rev: b074f4aff00923acc5bf6649d204d541a79fd2b6)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
GIT_SSH_COMMAND is more convinient to use if arguments have to be passed
and the user doesn't want to create a wrapper script around ssh.
(Bitbake rev: 5e746cb9d26ce87d6c9d52d9022122081a9811c5)
Signed-off-by: Pavel Zhukov <pavel.zhukov@huawei.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Change the owner information in the mirror tarballs generated using
BB_GENERATE_MIRROR_TARBALLS="1". This is an extension of commit
0178ab83, which used the original pokybuild:user information, but failed
to clean up the numerical user and group ids. Now set the more canonical
values of oe:oe and 0:0.
(Bitbake rev: 37437115d3fb1a9f5d8ed7356a0fc01a408e4f8c)
Signed-off-by: Olaf Mandel <o.mandel@menlosystems.com>
CC: Marek Vasut <marex@denx.de>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
If there are spaces in the URI filenames it can break the code.
We already solved this issue once somewhere else in the code so
use the same regex trick here as well.
We should ultimately refactor this code but at least fix the issue
for now.
(Bitbake rev: 57e2fc4d7f60afea4d4b2c84761324dd99e74a87)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
While the progress bar is good at conveying information about how much
of a task has executed, the elapsed time of the task is still very
much relevant to show.
(Bitbake rev: 41eeb4f34fb2306303f7688ec5e0ae965a573aa4)
Signed-off-by: Peter Kjellerstedt <peter.kjellerstedt@axis.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
This was only meant to be added for the handle_contains function in
a previous commit, fix it.
(Bitbake rev: 7399be398df39bc29e1b5eaac23b29cfae017abd)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The kernel inotify code can set more than one of the bits in the mask,
fsnotify_change() in linux/fsnotify.h is quite clear that IN_ATTRIB,
IN_MODIFY and IN_ACCESS can arrive together. We don't care about two
of these from a bitbake perspective but it probably explains why in
real world builds, we've seen:
pyinotify.ProcessEventError: Unknown mask 0x00000006
This module code assumes only one mask bit can be present. Since we
don't care about two of these events, just mask them out for now.
The "upstream" code is unmainained since 2015.
(Bitbake rev: 7fb93c2ce6dacd9b53fc3a227133a3493e6a6a1d)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The recent inotify changes can cause entire build trees to be monitored
which is suboptimal for performance.
Rather than trying increasingly convoluted tricks to try and handle add/removed
directories, rebuild the inotify watch when we reparse the configuration or
metadata.
(Bitbake rev: 3df322a200c28b45af1f2c92478c85eb7d20c38b)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The python gc can trigger whilst we're holding the event stream lock
and when cleaning up objects, they can trigger warnings. This translates
into a new event which would then need the lock and we can deadlock.
Disable gc whilst we hold that lock to avoid this unfortunate and
problematic situation.
(Bitbake rev: 96a6303949cefd469bcf5ed250ff512271354357)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Adding bb.utils.filter('WARN_QA', 'patch-fuzz', d) when WARN_QA is in
BB_BASEHASH_IGNORE_VARS or in vardepsexclude should not add a dependency
on WARN_QA.
Fix it and add some tests.
(Bitbake rev: 6aecc2fe51a52020f6f13be08449e18d42e7a6b5)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
We're still seeing issues with unclosed asyncio event loops. At the
init site, make sure any existing one is closed first to try and avoid
this.
(Bitbake rev: 78dee3c03c75a27531fcff26f9298fce2519bdde)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
In some cases we'd never have setup a siggen so don't error in that case.
(Bitbake rev: bbaaf2cf7b5a9339d3790610e622020c19d52f5a)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Ensure any exiting hash server connection is terminated before we start
a new bitbake session. This avoids errors seen with memory resident bitbake
when the asyncio event loop isn't closed correctly.
(Bitbake rev: 42ff9de77f24e2a0bec48a14b64c4b538e00b4af)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
When memory resident bitbake is active and we re-parse, the old module
configuration is present which can lead to strange errors. Reset this
when reparsing so the state is consistent. This fixes memory resident
bitbake errors.
(Bitbake rev: 951942c3c284ec2c62e730e145688033190af9b2)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The previous fix for inotify wasn't quite correct as we need to modify
bbseen before calling add_filewatch(). We also need to ensure the parse
mtime cache is cleared when directories are added/removed. There was also
a typo in the original fix and the wrong watcher was being changed. Fix
the various issues which improves memory resident bitbake testing results.
(Bitbake rev: 66cadd6be58bce5f7a56556cf92efd8159fb0b0e)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
- Replace hyphens by em dashes when necessary
See https://www.grammarly.com/blog/hyphens-and-dashes/
- No uppercase after em dashes
- No uppercase after colons if what follows is not a
complete sentence.
- Fix spacing before colons ":"
- Replace em-dashes with colons for consistency in a section
(Bitbake rev: 72230d6a9976b3bfca1f1e6fb09736fec195e2fe)
(Bitbake rev: f1c4ac816e927f490fb9852c12aa408e8c9403b1)
Signed-off-by: Michael Opdenacker <michael.opdenacker@bootlin.com>
Signed-off-by: Alexandre Belloni <alexandre.belloni@bootlin.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The "force" option to parser shutdown was often the cause of lockups and
there is no good reason we should have two different behaviours.
Change and unify the codepaths to always:
* Wait for longer for a controlled shutdown of a process (0.5s). Usually
it will be much faster if it has finished so the delay doesn't really matter.
* Send processes a SIGINT
* Failing that, send a SIGTERM
* Call .close() if available to release zombies
This means we no longer need the "force" parameter to the function so it is removed.
(Bitbake rev: de88c62ef873e9fce78ba162f5311d846de96f2b)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
If a parser process is terminated while holding a write lock, then it
will lead to a deadlock (see
https://docs.python.org/3/library/multiprocessing.html#multiprocessing.Process.terminate).
With SIGTERM, we don't want to terminate holding the lock. We also don't
want a SIGINT to cause a partial write to the event stream.
I tried using signal masks to avoid this but it doesn't work, see
https://bugs.python.org/issue47139
Instead, add a signal handler and catch the calls around the critical section.
We also need a thread lock to ensure other threads in the same process don't
handle the signal until all the threads are not in the lock.
(Bitbake rev: a40efaa5556a188dfe46c8d060adde37dc400dcd)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Not sure why this is so convoluted but we should simplify it!
(Bitbake rev: 6195343c46ba9d2685fc2d42366922f88ff3f369)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
If an exception occurs in the parsing process, ensure the cleanup
is called via all codepaths using a try/finally.
(Bitbake rev: e1ba2a69f1fc02f01a851bce20b1badf0b991f03)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
When parsing, the parser isn't servicing the main loop so a Ctrl+C in the
UI won't be seen on the cooker/server side. Fix this by returning when queue
timeouts occur. This helps where there is a hung or slow parsing thread.
(Bitbake rev: a2cde38311a51112dca5e7bb4e7feaf4e6a281b3)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
We shouldn't be generating exception inside a generator. Rearrange the
code to improve the handling of this. Also fix the misconverted code
from when multiconfig was added and pass the exception as "result".
(Bitbake rev: ae89e23696de2f27c00ae00922933395171de5d5)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
This parameter is now required by the git fetcher module
(Bitbake rev: d61b349581c219e7f9d50f683177184fa473cb83)
Signed-off-by: Michael Opdenacker <michael.opdenacker@bootlin.com>
Signed-off-by: Alexandre Belloni <alexandre.belloni@bootlin.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
When using BB_GENERATE_MIRROR_TARBALLS="1" to generate mirror tarballs
of git repositories, they leaked local information: username, group and
time of the last fetch. Remove all these by setting fixed information:
* uname = pokybuild
* gname = users
* mtime = committer time of newest commit in repo
The username and group value were taken from the archives available on
the downloads.yoctoproject.org mirror. The modification time is chosen
so it still retains some relationship to the contents of the archive.
(Bitbake rev: 0178ab83e6312e97e528aa8c5e12105f5165d896)
Signed-off-by: Olaf Mandel <o.mandel@menlosystems.com>
Signed-off-by: Alexandre Belloni <alexandre.belloni@bootlin.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Thanks to great debugging from pavel@zhukoff.net we had a simpler reproducer
for the corruption see in oe-selftest when using BB_SERVER_TIMEOUT=60, i.e.
with bitbake in memory resident mode. This was effectively:
oe-selftest -r devtool.DevtoolUpgradeTests.test_devtool_upgrade devtool.DevtoolUpgradeTests.test_devtool_upgrade_git -j 1 -K
The issue is that if directories are removed (such as workspace), if
they are added again, we don't have the watches in place any more. This
patch adds some slightly paranoid checks to ensure we do the correct things
for directory additions and removals (we track directories, not files
specifically to avoid running out of watches).
[YOCTO #14023]
(Bitbake rev: 2c414f659d793d732041614caedd773959eb4f27)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
This debug is useful but the cooker shutdown or post_serve() may have cleanup
left so run after those.
(Bitbake rev: 1463fc0448d1a6a7265806a4a8b165b610dfb43f)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
When we have a client connection, we should close that connection when reset()
is called on the siggen. Add the missing function.
(Bitbake rev: 770b4ea81b6126b0830e51649c40f7a46c64132a)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
We have shutdown functions within the async client code but the siggen
doesn't currently call them. We notice on python 3.10 (such as on fedora35)
that at exit, there is a stray asyncio process left behind. Usually this
doesn't cause problems but it could potentially be a cause of a hang.
For general cleanliness and completness, add in hooks to call the exit handler.
(Bitbake rev: 9ee3fb95330003878fbd64b3ce8897aad96fcd0f)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
There are some commands where we want to see the events returned so allow
the caller to request this. This also allows us to fix an infamous bug in
the tinfoil testsuite in OE-Core.
(Bitbake rev: 0e8421c41d97d5d50a553d70c8f775d521f1a199)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
If the path to bitbake.lock is in a deep directory, bitbake will hang. The
reason was that the max file length limiting code (to 255 chars) was including
the directory name and it should only act on the filename within the directory.
Fix it to just use the base filename.
[YOCTO #14766]
(Bitbake rev: 89d70e7b71eecfe06592202f326e566c579ba01d)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
I'd never spotted this until it was pointed out but the task isn't dependent,
it is a dependency. Fix this confusing reference.
(Bitbake rev: 93395559c9dda734a3dec9ffd9bb2156a71a2c17)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The keep alive timeout is excessively long at 83 minutes (5000 seconds),
reduce this to 10 minutes: this should be long enough that it rarely
triggers in normal builds, but when it does it has useful information.
(Bitbake rev: 2e47346b95b09d7ab8f0603e2d62cfb549dc1f5c)
(Bitbake rev: dcf52157d3635925491783be656c6b76d1efe1a4)
Signed-off-by: Ross Burton <ross.burton@arm.com>
Signed-off-by: Alexandre Belloni <alexandre.belloni@bootlin.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
