Commit Graph

6867 Commits

Author SHA1 Message Date
Tim Orling
375055b2d7 bitbake: toaster: Add refreshed oe-core and poky fixtures
After updating gen_fixtures.py, run ./gen_fixtures.py --all

This includes the latest stable/supported releases.

(Bitbake rev: 88a9cc0318c6ef01976d8b09dfc1a92cf0a1e498)

Signed-off-by: Tim Orling <tim.orling@konsulko.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2023-03-15 23:15:31 +00:00
Tim Orling
67dcd704b1 bitbake: toaster: fixtures/gen_fixtures.py: update branches
Re-introduce Dunfell since it is supported until April 2024
Drop Honister and Hardknott as they are both EOL
Add Langdale as it is the latest stable release

(Bitbake rev: 2dda597de3fcf6911e4022a8c279ae00413ec747)

Signed-off-by: Tim Orling <tim.orling@konsulko.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2023-03-15 23:15:31 +00:00
Tim Orling
624cd9b348 bitbake: toaster: fixtures/README: django 1.8 -> 3.2
We should reference the docs for the current LTS version we are using.

(Bitbake rev: add4569195f8f64000555679e282b0b12d1ca3f8)

Signed-off-by: Tim Orling <tim.orling@konsulko.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2023-03-15 23:15:31 +00:00
Richard Purdie
9e4cc20109 bitbake: cookerdata: Drop dubious exception handling code
This code appears to be dangerous, it swallows exceptions, turning them into
"handled" versions which then show no errors to the user. This is a pretty
poor user experience and I can't see why this code should be swallowing
such things. Drop the worst bits of code.

(Bitbake rev: 2b239555f76e4e98ca704e7ef60e796d0f19463c)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Signed-off-by: Steve Sakoman <steve@sakoman.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2023-03-14 15:25:03 +00:00
Richard Purdie
191d4f2577 bitbake: cookerdata: Improve early exception handling
Martin Jansa reported that if you put a syntax error into an imported
module such as qa.py in OE, no error is shown.

Part of the issue appears to be that the catch_parse_error() decorator only
catches certain exceptions and SyntaxError isn't one of them. As far as I can
tell we should remove all the special cases and use the more advanced code
in all cases, not just expansion errors.

I confirmed this now prints a proper error message for a qa.py syntax error.

(Bitbake rev: 4b922345a40f7cc803eb46c4906269691d408940)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Signed-off-by: Steve Sakoman <steve@sakoman.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2023-03-14 15:25:03 +00:00
Richard Purdie
0ade79619e bitbake: cookerdata: Remove incorrect SystemExit usage
Calling SystemExit doesn't work well with server/client usage since the string
isn't printed to the right place. Use bb.fatal() instead which prints the right
log output and raises and handled exception which then shows correctly on the
UI.

(Bitbake rev: 8001c9b5d3a3111f1134557f221325fe2593c2d9)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Signed-off-by: Steve Sakoman <steve@sakoman.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2023-03-14 15:25:03 +00:00
Richard Purdie
dde7a392c5 bitbake: utils: Allow to_boolean to support int values
Some variables may be set as:

X = 1

as well the more usual

X = "1"

so add support to to_boolean to handle this case.

(Bitbake rev: 3cc9fe911f764e4553078dbeed9497f6f08336ce)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Signed-off-by: Steve Sakoman <steve@sakoman.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2023-03-14 15:25:03 +00:00
Frank de Brabander
69553e4365 bitbake: bin/utils: Ensure locale en_US.UTF-8 is available on the system
Get rid of the duplicate code and add extra check that the
locale en_US.UTF-8 is available on the system. This new helper
method is now located right above the method filter_environment()
which sets LC_ALL environment variable to 'en_US.UTF-8'.

[YOCTO #10165]

(Bitbake rev: 2fe0c90da89b0a4e2e133f8ffa7a93d71097bb32)

Signed-off-by: Frank de Brabander <debrabander@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
(cherry picked from commit a4ce040a6fd540a1cac52f808f909f9fcf8c961c)
Signed-off-by: Steve Sakoman <steve@sakoman.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2023-03-14 15:25:03 +00:00
Etienne Cordonnier
3a60944caf bitbake: siggen: Fix inefficient string concatenation
As discussed in https://stackoverflow.com/a/4435752/1710392 , CPython
has an optimization for statements in the form "a = a + b" or "a += b".
It seems that this line does not get optimized, because it has a form a = a + b + c:
data = data + "./" + f.split("/./")[1]

For that reason, it does a copy of data for each iteration, potentially copying megabytes
of data for each iteration.

Changing this line causes SignatureGeneratorBasic::get_taskhash to take 0.06 seconds
instead of 45 seconds on my test setup where SRC_URI points to a big directory.

Note that PEP8 recommends explicitely not to use this optimization which is specific to CPython:
"do not rely on CPython’s efficient implementation of in-place string concatenation for statements in the form a += b or a = a + b"

However, the PEP8 recommended form using "join()" also does not avoid the copy and takes 45 seconds in my test setup:
data = ''.join((data, "./", f.split("/./")[1]))

I have changed the other lines to also use += for consistency only, however those were in the form a = a + b
and were optimized already.

Co-authored-by: JJ Robertson <jrobertson@snap.com>
(Bitbake rev: 592ee222a1c6da42925fb56801f226884b6724ec)

Signed-off-by: Etienne Cordonnier <ecordonnier@snap.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
(cherry picked from commit 195750f2ca355e29d51219c58ecb2c1d83692717)
Signed-off-by: Steve Sakoman <steve@sakoman.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2023-02-17 15:05:12 +00:00
Marek Vasut
4e69335efc bitbake: fetch2/git: Clarify the meaning of namespace
Namespace in this context means a branch, a tag, etc., clarify
it in the description. Also, fix a typo "a any", replace with
plain "any".

This patch is based of feedback on new applied patch
d32e5b0e ("fetch2/git: Prevent git fetcher from fetching gitlab repository metadata")

(Bitbake rev: a40fc6a3f774bcb28cf72701ac146ceb7ae8061a)

Signed-off-by: Marek Vasut <marex@denx.de>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
(cherry picked from commit b4999425c812b25cb359d5163d11e3c1b030dc28)
Signed-off-by: Steve Sakoman <steve@sakoman.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2023-02-04 10:34:08 +00:00
Marek Vasut
581b574ebb bitbake: fetch2/git: Prevent git fetcher from fetching gitlab repository metadata
The bitbake git fetcher currently fetches 'refs/*:refs/*', i.e. every
single object in the remote repository. This works poorly with gitlab
and github, which use the remote git repository to track its metadata
like merge requests, CI pipelines and such.

Specifically, gitlab generates refs/merge-requests/*, refs/pipelines/*
and refs/keep-around/* and they all contain massive amount of data that
are useless for the bitbake build purposes. The amount of useless data
can in fact be so massive (e.g. with FDO mesa.git repository) that some
proxies may outright terminate the 'git fetch' connection, and make it
appear as if bitbake got stuck on 'git fetch' with no output.

To avoid fetching all these useless metadata, tweak the git fetcher such
that it only fetches refs/heads/* and refs/tags/* . Avoid using negative
refspecs as those are only available in new git versions.

Per feedback on the ML, Gerrit may push commits outsides of branches or
tags during CI runs, which currently works with the 'nobranch=1' fetcher
parameter. To retain this functionality, keep fetching everything in case
the 'nobranch=1' is present. This still avoids fetching massive amount of
data in the common case, since 'nobranch=1' is rare. Update 'nobranch'
documentation.

Reviewed-by: Peter Kjellerstedt <peter.kjellerstedt@axis.com>
(Bitbake rev: 2b4583922de184a2d96324cd4e3f85993341cd14)

Signed-off-by: Marek Vasut <marex@denx.de>
Signed-off-by: Alexandre Belloni <alexandre.belloni@bootlin.com>
(cherry picked from commit d32e5b0ec2ab85ffad7e56ac5b3160860b732556)
Signed-off-by: Steve Sakoman <steve@sakoman.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2023-02-04 10:34:08 +00:00
Ross Burton
f59aa3752d bitbake: bb/utils: include SSL certificate paths in export_proxies
bb.utils.export_proxies() is a poor-man's alternative for the
environment setup code in bb/fetch2, but it's used in several places
where recipes want to download manually (such as cve-update-db-native).

Notably, export_proxies() doesn't pass on the SSL certificate paths from
the original environment, so if SSL_CERT_FILE needs to be set (for
example, in a buildtools environment) then proxies work but SSL doesn't.

In an ideal world export_proxies and the same logic in fetch2 would
merge, but until then we can add the SSL_CERT_ variables and duplicate
the basic logic: check the datastore first and then the original
environment for variables.

Also remove the return value as nothing ever checked it.

[ YOCTO #15000 ]

(Bitbake rev: 28ba566c25af72afe62ea3cb62b0bafeffc47de8)

Signed-off-by: Ross Burton <ross.burton@arm.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2023-01-30 08:40:33 +00:00
Richard Purdie
da318dd088 bitbake: server/process: Add bitbake.sock race handling
We've seen cases where the bitbake.sock file appears to disappear but the
server continues to hold bitbake.lock. The most likely explaination is
that some previous build directory was moved out the way, a server there
kept running, eventually exited and removed the sock file from the wrong
directory.

To guard against this, save the inode information for the sock file and check
it before deleting the file. The new code isn't entirely race free but should
guard against what is a rare but annoying potential issue.

(Bitbake rev: 52b6b099a47555811b8d0b311f62af712dd6eb8e)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
(cherry picked from commit b02ebbffdae27e564450446bf84c4e98d094ee4a)
Signed-off-by: Steve Sakoman <steve@sakoman.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2023-01-16 11:08:39 +00:00
Frank de Brabander
bfa114bfa8 bitbake: process: log odd unlink events with bitbake.sock
Log when the socket file already exists and is removed before
recreating a new socket.

Log when unlinking the socket file failed.

(Bitbake rev: 9779fad4d9e2540b24bb91cfa38fc1984402bef2)

Signed-off-by: Frank de Brabander <debrabander@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
(cherry picked from commit cfd7c9899f988bab6d9fe7bbfbdb60603fb5ed34)
Signed-off-by: Steve Sakoman <steve@sakoman.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2023-01-16 11:08:39 +00:00
Pavel Zhukov
9591c2f298 bitbake: gitsm: Fix regression in gitsm submodule path parsing
Commit 0361ecf7eb82c386a9842cf1f3cb706c0a112e77 introduced regression
in submodules path parsing. As the result gitsm fetcher fails on each
submodule which name begins from the name of the parent repo which is
totally valid usecase [Yocto #14045] [1]
Fix the code to error out only if submodule's name is equal to parent
name but not if it's part of it.

[1] https://bugzilla.yoctoproject.org/show_bug.cgi?id=14045#c4

(Bitbake rev: f0f166aee766b4bb1f8cf8b35dfc7d406c75e6a4)

Signed-off-by: Pavel Zhukov <pavel@zhukoff.net>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
(cherry picked from commit 3ad27272c18f2bb9edd441f840167a3dabd5407b)
Signed-off-by: Steve Sakoman <steve@sakoman.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2023-01-06 17:33:18 +00:00
Richard Purdie
6361c77aca bitbake: runqueue: Fix race issues around hash equivalence and sstate reuse
We identified a use case where a native recipe (autoconf-native) was
rebuilt with no change in output yet the sstate for do_package tasks
wasn't being used.

The issue is that do_package tasks have a hard dependency on
pseudo-native:do_populate_sysroot. That task was one of the many
tasks being rehashed when autoconf-native's hash was changed.

If update_tasks processed a recipe before it had processed pseudo-native,
that recipe would be marked as not possible from sstate and would
run the full tasks.

The fix is to split the processing into two passes, first to handle
the existing covered/notcovered updates, then in the second pass,
check whether there are "harddep" issues.

This defers the do_package tasks until after pseudo-native is installed
from sstate as expected and everything works well again.

(Bitbake rev: 3b2d1331487a74863e74cc0c5564004cbb1b5a4a)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
(cherry picked from commit e479d1e418a7d34f0a4663b4a0e22bb11503c8ab)
Signed-off-by: Steve Sakoman <steve@sakoman.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2022-11-24 15:25:15 +00:00
Mark Asselstine
79434a17eb bitbake: bitbake: bitbake-layers: checkout layer(s) branch when clone exists
[YOCTO #7852]

Fixes 'bitbake-layers layerindex-fetch --branch kirkstone meta-arm'
not checking out the branch if the repo is already cloned and on a
different branch.

If a clone of a layer being added already exists check what branch it
is on and if necessary attempt to switch to the given branch. If the
switch fails to happen the git error will be reported. We also warn if
there are uncommitted changes as the changes might go unnoticed and
result in unexpected behaviors.

(Bitbake rev: 138dd7883ee2c521900b29985b6d24a23d96563c)

Signed-off-by: Mark Asselstine <mark.asselstine@windriver.com>
Signed-off-by: Alexandre Belloni <alexandre.belloni@bootlin.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
(cherry picked from commit d2cb388f58a37db2149fad34e4572d954e6e5441)
Signed-off-by: Steve Sakoman <steve@sakoman.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2022-11-10 14:43:30 +00:00
Justin Bronder
25f355e0ef bitbake: asyncrpc: serv: correct closed client socket detection
If the client socket is closed, asyncio.StreamReader.readline() will
return an empty bytes object, not None.

This prevents multiple tracebacks being logged by bitbake-hashserv each
time bitbake is started and performs a connection check.

(Bitbake rev: 4bdd9ba43f34a1473db31a6a3b10bd33e358fe3a)

Signed-off-by: Justin Bronder <jsbronder@cold-front.org>
Signed-off-by: Alexandre Belloni <alexandre.belloni@bootlin.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
(cherry picked from commit 2d07f252704dff7747fa1f9adf223a452806717f)
Signed-off-by: Steve Sakoman <steve@sakoman.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2022-11-10 14:43:30 +00:00
Ross Burton
186d179614 bitbake: fetch2/git: don't set core.fsyncobjectfiles=0
This git configuration variable is deprecated in 2.36.0 onwards, so git
warns in the logs for every git call.

Luckily the default value has always been false[1], so we can just remove
this.

[ YOCTO #14939 ]

[1] aafe9fbaf4

(Bitbake rev: 13f86aeb53cd73c03bfb2f00fe923b51ec8d1c73)

Signed-off-by: Ross Burton <ross.burton@arm.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
(cherry picked from commit 8ad310633e0c5d5593631c1196cbdde30147efce)
Signed-off-by: Steve Sakoman <steve@sakoman.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2022-11-10 14:43:30 +00:00
Michael Opdenacker
975e3fb53c bitbake: bitbake-user-manual: details about variable flags starting with underscore
Fixes [YOCTO #14140]

(Bitbake rev: 8a08e207854810b40b53946ec94065a6a560a7a5)

Signed-off-by: Michael Opdenacker <michael.opdenacker@bootlin.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
(cherry picked from commit 0f3e9d87168813ce49995ff04bccdce11c5f7b47)
Signed-off-by: Steve Sakoman <steve@sakoman.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2022-11-10 14:43:30 +00:00
Richard Purdie
6b9db5a99b bitbake: tests/fetch: Allow handling of a file:// url within a submodule
CVE-2022-39253 in git meant file:// urls within submodules were disabled. Add
a parameter to the commands in the tests to allow this to continue to work.

(Bitbake rev: 209f7ba352b60722830157054e3fc56cb9c693eb)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2022-10-26 23:02:11 +01:00
Mark Asselstine
6672cbe670 bitbake: tests: bb.tests.fetch.URLHandle: add 2 new tests
Add a test for special characters in user and password to qualify
decodeurl() inspired by a bug report describing that '=' signs in a
password was problematic.

Add a second test to qualify decodeurl() as related to the change in
commit 628c4bf6c89b [fetch2/__init__: handle @ in package names].

Relates to [YOCTO #14476]

(Bitbake rev: ee04cf09c7022168c035affa654773652a49793e)

Signed-off-by: Mark Asselstine <mark.asselstine@windriver.com>
Signed-off-by: Alexandre Belloni <alexandre.belloni@bootlin.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2022-10-26 23:02:10 +01:00
Michael Opdenacker
c58059d282 bitbake: doc: bitbake-user-manual: expand description of BB_PRESSURE_MAX variables
(Bitbake rev: 72e9847dd578c3cbed52a9c16fea23ebbeef5046)

Signed-off-by: Paul Eggleton <paul.eggleton@microsoft.com>
Signed-off-by: Michael Opdenacker <michael.opdenacker@bootlin.com>
Signed-off-by: Alexandre Belloni <alexandre.belloni@bootlin.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2022-10-26 23:02:10 +01:00
Mark Hatle
ec08faf2e4 bitbake: utils/ply: Update md5 to better report errors with hashlib
In the case where hashlib is not available, the try would fail and fall
through resulting in a backtrace on the usage of the 'sig'.  The backtrace
itself was confusing and made it difficult to determine what went wrong.

Update the import to be in it's own try block with an appropriate
message to indicate what went wrong.

Note, the current version of ply all of this code has been restructured
so this is not applicable upstream.

Additionally, some versions of hashlib don't appear to implement the
second FIPS related argument.  Detect this and support both versions.

(Bitbake rev: 484ab42f440070c0369b81f5c69da860fa47a798)

Signed-off-by: Mark Hatle <mark.hatle@amd.com>
Signed-off-by: Mark Hatle <mark.hatle@kernel.crashing.org>
Signed-off-by: Luca Ceresoli <luca.ceresoli@bootlin.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2022-10-26 23:02:10 +01:00
Johan Korsnes
7aa3ed5c37 bitbake: bitbake: user-manual: inform about spaces in :remove
Inform the reader that there should be no need for spaces in the value
when using removal override `:remove`.

Considering why spaces are used in the other override operators, it
might seem obvious that they aren't needed for the removal operator.
But, it seems like I'm not the first to be confused about this.

Cc: Richard Purdie <richard.purdie@linuxfoundation.org>
Cc: Quentin Schulz <quentin.schulz@theobroma-systems.com>
Cc: Ross Burton <ross.burton@arm.com>
Cc: Nicolas Dechesne <nicolas.dechesne@linaro.org>
(Bitbake rev: 0a493a772f83436cbe909de93c157f4ab2d2d136)

Signed-off-by: Johan Korsnes <johan.korsnes@remarkable.no>
Signed-off-by: Alexandre Belloni <alexandre.belloni@bootlin.com>
Reviewed-by: Michael Opdenacker <michael.opdenacker@bootlin.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2022-10-26 23:02:10 +01:00
Richard Purdie
1b014677b7 bitbake: bitbake: Bump to version 2.2.0
(Bitbake rev: 074da4c469d1f4177a1c5be72b9f3ccdfd379d67)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2022-09-29 21:56:55 +01:00
Mattias Jernberg
74877d2dc2 bitbake: utils: Add enable_loopback_networking()
It can be used to enable the loopback interface, typically after calling
disable_network().

Also correct a typo in a debug message.

(Bitbake rev: 0d317209d4234c5f05a9fcdc13c52f502f104018)

Signed-off-by: Mattias Jernberg <mattias.jernberg@axis.com>
Signed-off-by: Peter Kjellerstedt <peter.kjellerstedt@axis.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2022-09-29 21:26:09 +01:00
Aryaman Gupta
764bbf2a9b bitbake: doc: bitbake-user-manual: Add definition for BB_PRESSURE_MAX variables
Add the definitions for the BB_PRESSURE_MAX{CPU|IO|MEMORY} variables in the
bitbake varibales glossary. Further information on how to determine a good
threshold will be added to the Yocto reference manual in a later commit.

(Bitbake rev: c014281f72f4f54ec8e681ef2b8e1080de9ab5cf)

Signed-off-by: Aryaman Gupta <aryaman.gupta@windriver.com>
Signed-off-by: Randy Macleod <Randy.Macleod@windriver.com>
Signed-off-by: Michael Opdenacker <michael.opdenacker@bootlin.com>
Reviewed-by: Michael Opdenacker <michael.opdenacker@bootlin.com>
Reviewed-by: Quentin Schulz <foss+yocto@0leil.net>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2022-09-29 21:24:29 +01:00
Pascal Bach
f761409551 bitbake: fetch2/ssh.py: fix checkstatus
The output of runfetchcmd is always empty in this case, as
the test doesn't produce any output.
SSH either returns 0 or 1, which is handled via exceptions.

This means the current check is not only unnecessary but prevents the
function from working.

We can just assume that if we reach the end of the function that the
file exists and return True.

(Bitbake rev: d599af48635fab587e5b913591b95daf87b40080)

Signed-off-by: Pascal Bach <pascal.bach@siemens.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2022-09-29 21:24:29 +01:00
Michael Opdenacker
342d18ee78 bitbake: bitbake-user-manual: mention pydevshell in OE
(Bitbake rev: 3f23140f3b26d81452e345f56ed67d2928ae3a12)

Signed-off-by: Michael Opdenacker <michael.opdenacker@bootlin.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2022-09-22 21:40:23 +01:00
Jose Quaresma
d39a1c49b3 bitbake: fetch2/crate fixup c212b0f3 loglevel
c212b0f3 change the debug log level unintentional when tryng to fix a knotty issue.
This will maintain the same debug log level 2 as before.

(Bitbake rev: 19f8265023281f3b1d5d0a02e47f8d7d08cfcc16)

Signed-off-by: Jose Quaresma <jose.quaresma@foundries.io>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2022-09-21 20:20:22 +01:00
Jose Quaresma
e2c3bd1638 bitbake: lib/bb: warning when the debug message is invalid
There are many messed up calls for the debug log,
so is better to warm about this as they will not work
as expected.

The level need to be an integer and the msg a string.

(Bitbake rev: c1d9c1d25ce36848040dc0ce182835e497ccbb82)

Signed-off-by: Jose Quaresma <jose.quaresma@foundries.io>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2022-09-21 20:20:22 +01:00
Joshua Watt
3e0e5347e6 bitbake: bitbake: Fix a few more logger debug() calls
f68682a7 ("logging: Make bitbake logger compatible with python logger")
replaced several .debug() calls to make them comply with the standard
python logging API, but a few were missed.

(Bitbake rev: eb25cd4d64b9a4e8e2b2ce7fbccc123d00b2fc2b)

Signed-off-by: Joshua Watt <JPEWhacker@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2022-09-21 20:20:22 +01:00
Paulo Neves
a6b647af79 bitbake: fetch2: Remove unneeded conditional
The condition will always evaluate to true and
thus is redundant.

(Bitbake rev: be1ee681e8a566564549068dcf90c95c36544815)

Signed-off-by: Paulo Neves <ptsneves@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2022-09-21 20:20:22 +01:00
Richard Purdie
bdd91aa388 bitbake: fetch2: Ensure mirror tarballs don't enforce checksum
local file fetches now validate checksums. The checksums for mirror
tarballs of repositories will not match so ignore these checksums.

(Bitbake rev: 6424f4b7e9c1ba8db81346e8b3a806dd035d4551)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2022-09-07 21:28:59 +01:00
Otavio Salvador
1e2b9bafd4 bitbake: toaster: fix kirkstone version
(Bitbake rev: 04014b8b2c3d7bb80d7d8dca97b7472f0e6b4ebb)

Signed-off-by: Otavio Salvador <otavio@ossystems.com.br>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2022-09-07 08:33:53 +01:00
Richard Purdie
e490bc8262 bitbake: asyncrpc/client: Fix unix domain socket chdir race issues
The connect_unix() call had a bug where if a relative path to a socket
was passed (which the non-async client always does), and the current
working directory was changed after the initial call, it would fail to
reconnect if it became disconnected, since the socket couldn't be found
relative to the new current working directory.

To work around this, change the socket connection for UNIX domain
sockets to be synchronous and change current working before connecting.
This isn't ideal since the connection could block the entire event loop,
but in practice this shouldn't happen since the socket are local files
anyway.

Help debugging and resolving from Joshua Watt.

(Bitbake rev: 5964bb67bb20df7f411ee0650cf189504a05cf25)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2022-09-05 12:54:25 +01:00
Richard Purdie
58c42911d4 bitbake: wget: Avoid bad checksum race issues
If two recipes have conflicting checksums for a file, the code will currently
remove the existing file when a mismatch is downloaded, even if another task
successfully fetched it.

This changes the code to verify the checksum (if possible) before replacing
the file. This removes a potential race window and stops builds failing
everywhere from one incorrect checksum.

To make this work, we need to be able to override localpath and avoid
NoChecksum errors being logged.

(Bitbake rev: 4b8de2e7d12667d69d86ffe6e9f85a7932c4c9a5)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2022-09-05 12:54:25 +01:00
Richard Purdie
c9342278d7 bitbake: Revert "fetch: use BPN instead"
PN is correct here, bitbake has no knowledge of BPN.

This reverts commit d613e48c07d4b12219270c1359cbf2f390b848dd.

(Bitbake rev: cffcfacb747d41304c857b17bfea646e220b2389)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2022-09-01 12:18:36 +01:00
Mingli Yu
4ab7125180 bitbake: fetch: use BPN instead
When checking for the non-existing file, BPN is actually the acutal recipe
name. And we should use BPN for the error message and it also fix the below
test when multilib is enabled.
 $ oe-selftest -r bbtests.BitbakeTests.test_invalid_recipe_src_uri

(Bitbake rev: d613e48c07d4b12219270c1359cbf2f390b848dd)

Signed-off-by: Mingli Yu <mingli.yu@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2022-09-01 10:07:00 +01:00
Pavel Zhukov
af64b9b31b bitbake: tests: Add test for possible gitsm deadlock
(Bitbake rev: b0506480baa9bcf3ef645b0aed5a07ad9950245c)

Signed-off-by: Pavel Zhukov <pavel@zhukoff.net>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2022-09-01 10:07:00 +01:00
Pavel Zhukov
492ec14a37 bitbake: tests: Add Timeout class
The class and exception aim to test rare cases there deadlocks are
possible.
Can be used in context managers:
with Timeout(<value>):
        do_deadlock()

(Bitbake rev: c5fcdd804d422f959a189b270d72123a50e74da6)

Signed-off-by: Pavel Zhukov <pavel@zhukoff.net>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2022-09-01 10:07:00 +01:00
Pavel Zhukov
0abd64b3af bitbake: gitsm: Error out if submodule refers to parent repo
If submodule refers to specific revision of the parent repository it
causes deadlock in bitbake locking mechanism (lock is acquired to fetch
the parent and cannot be released before all submodules are fetched).
raise FetchError in such situation to prevent deadlocking.

[Yocto 14045]

(Bitbake rev: 0361ecf7eb82c386a9842cf1f3cb706c0a112e77)

Signed-off-by: Pavel Zhukov <pavel@zhukoff.net>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2022-09-01 10:07:00 +01:00
Neil Horman
9e22a3cccf bitbake: Fix npm to use https rather than http
Hit this error while building nlf-native recently:
{
  "error": {
    "summary": "URI malformed",
    "detail": ""
  }
}

Some poking about led me to discover that:
1) The npm.py tool replaces npm:// with http://, not https://
2) Some versions of the npm tool don't handle 301 redirects properly,
   choosing to display the above error instead when using the default
   nodejs registry

It would be good to go fix npm to handle the redirect properly, but it
seems like it would also be good to assume secure http when contacting a
registry, hence, this patch

(Bitbake rev: 2cd76e8aabe4e803c760e60f06cfe1f470714ec7)

Signed-off-by: Neil Horman <nhorman@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2022-09-01 10:07:00 +01:00
Alexander Kanavin
b5597638c7 bitbake: bitbake-layers: initialize tinfoil before registering command line arguments
Plugins may want to use it (e.g. the layers-setup plugin that would
want to discover writer sub-plugins with it), and so it makes sense
to make tinfoil available a bit eariler.

(Bitbake rev: 2f6c7523a622f59ddf84a1a196927492bc5fa7a2)

Signed-off-by: Alexander Kanavin <alex@linutronix.de>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2022-09-01 10:07:00 +01:00
Joshua Watt
79cc8584e6 bitbake: utils: Pass lock argument in fileslocked
Pass additional arguments in the fileslocked() context manager to the
underlying lockfile() function. This allows the context manager to be
used for any types of locks (non-blocking, shared, etc.) that the
lockfile() function supports.

(Bitbake rev: 7a8eb8da8e8495051e174721062da08e06168024)

Signed-off-by: Joshua Watt <JPEWhacker@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2022-08-31 10:40:07 +01:00
Richard Purdie
1a5c4140b0 bitbake: runqueue: Change pressure file warning to a note
The user does need to be told about this but it isn't really a warning,
just something they may need to be aware of. Drop the level accordingly.

(Bitbake rev: 9bdedc8074990e613c9567e2cd8072f8d885f07f)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2022-08-24 15:43:36 +01:00
Richard Purdie
7bd328f9d2 bitbake: BBHandler/cooker: Implement recipe and global classes
We have some confusion for users since some classes are meant to work
in the configuration space (or "globally") and some are meant to be
selected by recipes individually.

The cleanest way I could find to clarify this is to create "classes-global"
and "classes-recipe" directories which contain the approproate classes and
have bitbake switch scope between them at the appropriate point during
parsing. The existing "classes" directory is always searched as a fallback.

Once a class is moved to a specific directory, it will no longer be found
in the incorrect context. A good example from OE is that

INHERIT += "testimage"

will no longer work but

IMAGE_CLASSES += "testimage"

will, which makes the global scope cleaner by only including it where it
is useful and intended to be used (images).

(Bitbake rev: f33ce7e742f46635658c400b82558cf822690b5e)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2022-08-12 11:49:29 +01:00
Richard Purdie
e19ce4191d bitbake: bitbake: Add copyright headers where missing
Where copyright headers were not present, add them to make things
clear.

(Bitbake rev: 1aa338a216350a2751fff52f866039343e9ac013)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2022-08-12 11:49:29 +01:00
Richard Purdie
12d7157a41 bitbake: BBHandler: Make inherit calls more directly
Rather than recursing into the conf handler code, simply call into
the parse code directly when inheriting files as we've already resolved
the paths and don't need anything the other codepath brings. This
makes the codepath clearer at the expense of some slight duplication.

(Bitbake rev: 0f4f3af6d93a0018df58b8a1d8d423c78ba6526d)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2022-08-12 11:49:29 +01:00