Commit Graph

651 Commits

Author SHA1 Message Date
Richard Purdie
f209f127e6 bitbake: main: Add an option to specify what to profile
Starting with python 3.12, profiling now stays enabled over threads yet
you can't extract the profile data in the threads themselves, which makes it
difficult to use for our use case.

Our main loop starts the idle loop which starts the parsing threads and this
means we can't profile in the main loop and the parsing threads or the idle
loop at the same time due to this.

Add options to the commandline so you can specify which piece of bitbake
you want to enable profiling for. This allows some profiling with python 3.12
onwards rather than crashing.

(Bitbake rev: 09f29a4968841ee5070f70277ba8c253bb14f017)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2025-07-17 10:45:57 +01:00
Richard Purdie
5336309d6f bitbake: bitbake: Bump version to 2.15.1
(Bitbake rev: f68b513c38fa33c89236efbaab2674a25983d5e1)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2025-07-01 08:50:51 +01:00
Richard Purdie
58b0a65ada bitbake: utils: Refactor filemode variable conversion to a function
We have other places in the code where we need to take filemode/mask
information from a bitbake variable and turn it into a real python
number. Turn this internal code into public API in bb.utils and
add some tests for it.

(Bitbake rev: d89e30fb2fb15b09f2cb95c4e5aa9f749ca257ea)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2025-07-01 08:49:37 +01:00
Richard Purdie
30fe072f6a bitbake: bitbake: Bump to version 2.15.0
Update version to 2.15.0 for the development series and features needed for
toolchain selection in OE.

(Bitbake rev: c2f29c9475c4b9cdd12af1f8610f2675f8fdd964)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2025-06-16 22:23:18 +01:00
hongxu
8bd8461212 bitbake: bitbake-getvar: skip info output of bitbake for quiet
Calling oe-debuginfod in a build failed:
...
$ oe-debuginfod
|Getting sysroot...
|Error: NOTE: Reconnecting to bitbake server...
|NOTE: Retrying server connection (#1)... (18:55:53.009687)
|path-to-build/tmp/work/x86_64-linux/elfutils-native/0.192/recipe-sysroot-native doesn't exist.
|Have you run 'bitbake elfutils-native -caddto_recipe_sysroot'?
...

The script oe-debuginfod calls bitbake-getvar to get sysroot, the
output of bitbake-getvar was mixed with info output of bitbake
...
NOTE: Reconnecting to bitbake server...
NOTE: Retrying server connection (#1)... (18:55:53.009687)
...

Set logger level to logging.WARNING to skip info output
for quiet

(Bitbake rev: 873c524e1a33846df8f34b7c87b298349277b3d5)

Signed-off-by: Hongxu Jia <hongxu.jia@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2025-06-16 22:23:18 +01:00
Richard Purdie
2d7f5904a8 bitbake: bitbake: Update version to 2.12.0 for release
(Bitbake rev: 5b4e20377eea8d428edf1aeb2187c18f82ca6757)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2025-03-29 15:28:51 +00:00
Richard Purdie
2d66669b8c bitbake: bitbake: Bump version to 2.9.2
After the fetcher revisions changes, we need a new version marker to
match this with in OE-Core.

(Bitbake rev: 8cc976e2792fdde3900729f3b09dd18ab640b5e8)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2025-03-25 14:29:18 +00:00
Alexandre Marques
a4f8336982 bitbake: hashserv: Add gc-mark-stream command for batch hash marking
Implements the `gc-mark-stream` command to allow for marking equivalence entries
in batch, by  making use of stream mode communication to the server.

The aim of this is to improve efficiency by reducing the impact of latency when
marking a high volume of hash entries.

Example usage of the new `gc-mark-stream` command:

```
$ cat << HASHES | \
./bin/bitbake-hashclient --address "ws://localhost:8688/ws" gc-mark-stream "alive"
unihash f37918cc02eb5a520b1aff86faacbc0a38124646
unihash af36b199320e611fbb16f1f277d3ee1d619ca58b
taskhash a1117c1f5a7c9ab2f5a39cc6fe5e6152169d09c0 method oe.sstatesig.OEOuthashBasic
HASHES
```

(Bitbake rev: c84715f28cd36666ea07a179d91b8c32ea0df8e7)

Signed-off-by: Alexander Marques <c137.marques@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2025-03-13 16:52:44 +00:00
Richard Purdie
e1e6066934 bitbake: bitbake-server/worker: Hide os.fork() deprecation warning
We're fairly careful in bitbake about how we handle fork() calls and believe our code
to be safe. The upstream deprecation warning is problematic as it can appear in log
output as a WARNING, breaking tests. It also tends to alarm users.

Hide the warning for now to avoids the test failures.

(Bitbake rev: c636bd629896f56e5f3d4030da3d1f130590afc6)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2025-03-11 14:17:57 +00:00
Richard Purdie
6ffd1a58f9 bitbake: bin/git-make-shallow: Fix syntax to work with older git versions
The transaction model was only introduced in git 2.27 whereas Ubuntu focal
(20.04) has 2.25. This causes failures. We don't need the transations here
so simply drop the commit piece, fixing on older git versions.

Credit to Nick Owens <nick.owens@eero.com> for working out how to fix it.

(Bitbake rev: 0723ec9d4cd7c9b2d46904c3a038be123feea374)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2025-01-10 09:10:06 +00:00
Nick Owens
ef7755749b bitbake: git-make-shallow: use stdin mode
when there are many refs to delete, using xargs to exec git can take a very long
time. make this faster by only running git update-ref with stdin mode.

for a repo with over 34000 git tags this makes git-make-shallow finish
in 2 seconds instead of 3 minutes for me.

(Bitbake rev: 2b815e42ec074a7f8667bbfaccaa69fc4a0ba788)

Signed-off-by: Nick Owens <nick.owens@eero.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2025-01-08 11:34:30 +00:00
Alexander Kanavin
2a5623dddc bitbake: bitbake-config-build: add an alias to bitbake-layers
This alias is intended for managing specific local configs and it
is prompted by adding support for config fragments (in a separate
commit to oe-core); after some deliberation I concluded there should be
a separate tool, as bitbake-layers is already somewhat over-stuffed,
and this will give space for more build/conf/* operations in the future
that anyone can come up with (such as tweaking site-specific items
in site.conf etc.)

The alias completely reuses existing code via symlink and
the difference is in where it looks for plugins.

(Bitbake rev: ba90fe673aa87cb0cda9b2e465ebe2063551f527)

Signed-off-by: Alexander Kanavin <alex@linutronix.de>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2024-12-13 11:11:18 +00:00
Chris Laplante
a8a11cbc79 bitbake: bitbake-layers: use 'with' to manage tinfoil context
(Bitbake rev: bd468a5b9210043d0121a322360d976fd830f736)

Signed-off-by: Chris Laplante <chris.laplante@agilent.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2024-11-28 00:06:24 +00:00
BELHADJ SALEM Talel
f0824f9270 bitbake: bitbake-getvar: Catch NoProvider exception
When the recipe provided by (-r, --recipe) is not found
tinfoil raises an exception that is not catched for
readability, example:

Traceback (most recent call last):
  File "/.../poky/bitbake/bin/bitbake-getvar", line 45, in <module>
    d = tinfoil.parse_recipe(args.recipe)
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/.../poky/bitbake/lib/bb/tinfoil.py", line 633, in parse_recipe
    fn = self.get_recipe_file(pn)
         ^^^^^^^^^^^^^^^^^^^^^^^^
  File "/.../poky/bitbake/lib/bb/tinfoil.py", line 550, in get_recipe_file
    raise bb.providers.NoProvider('Unable to find any recipe file matching "%s"' % pn)
bb.providers.NoProvider: Unable to find any recipe file matching "aaa"

(Bitbake rev: 06aa6c292813a28c84736193b550fb2d18884d43)

Signed-off-by: Talel BELHAJSALEM <bhstalel@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2024-11-21 12:16:28 +00:00
Richard Purdie
f95a645dd4 bitbake: bitbake-worker/cooker: Increase default pipe size
The default pipe size is 64kb on builds, which can be inefficient
for larger log files from workers. Increase the pipe size to 512kb
since build systems have decent amounts of memory and this is a more
efficient way of batching the data.

Tweak the default read sizes to match the pipe size for efficiency.

Since the contstant is only present in python 3.10 onwards, add
some compatibility code.

(Bitbake rev: 69c14e46600ba5ae9703f67704ab2548875ae6d7)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2024-11-21 12:16:28 +00:00
Richard Purdie
b97de63d2f bitbake: bitbake-worker: Improve bytearray truncation performance
If there are large amounts of data being transferred to the cooker
from the worker, recreating the bytearray becomes inefficient as it
happens for every pipesize block of data, defaulting to 64kb.

Instead we can use the deletion API for bytearrays to make this more
efficient and avoid the object recreation.

We noticed this with a strace ptest image taking days to complete the
build after having 6GB of data in the testimage log. Whilst there are
other issues there, making this code more efficient doesn't hurt.

(Bitbake rev: a4a72b7edb368f352784c856a647236a887010dd)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2024-11-21 12:16:28 +00:00
Alexander Kanavin
17f1b80c06 bitbake: bitbake-layers: ensure tinfoil.shutdown() gets executed when tinfoil.prepare() fails
https://git.yoctoproject.org/poky/commit/bitbake/bin/bitbake-layers?id=f6de2b033d32c0f92f19f5a4a8c4c8874a00a8f7
erroneously moved tinfoil.prepare() out of try..finally block, where
'finally' contains a tinfoil.shutdown() call.

Without the shutdown, if there is an error in tinfoil.prepare() (such as parsing errors),
the tool locks up, as seen here:
https://valkyrie.yoctoproject.org/#/builders/71/builds/431

(Bitbake rev: 06b8a18339434be8f754e534dacb790a2c9cb91d)

Signed-off-by: Alexander Kanavin <alex@linutronix.de>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2024-11-21 12:16:28 +00:00
Yoann Congal
4be9d61225 bitbake: bitbake-getvar: use finalizeData tinfoil API to get identical result to "bitbake -e"
Fixes [YOCTO #15638]

(Bitbake rev: 68ae81dc93f86eab378fec2276561c5062263d7e)

Signed-off-by: Yoann Congal <yoann.congal@smile.fr>
Signed-off-by: Mathieu Dubois-Briand <mathieu.dubois-briand@bootlin.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2024-11-18 13:41:38 +00:00
Joshua Watt
b5f17d0334 bitbake: bitbake-hashclient: Add help for address
Adds an epilog to the help text that indicates the possible options for
the server address

(Bitbake rev: b6b703fce02057212ad11b1d1286c6178c533bad)

Signed-off-by: Joshua Watt <JPEWhacker@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2024-10-25 15:40:03 +01:00
Michael Opdenacker
39400c4a2b bitbake: bitbake-prserver: use PRSERV_UPSTREAM as default setting
Instead of PRSERVER_UPSTREAM.
The intended variable name is PRSERV_UPSTREAM, as
already used in lib/prserv/serv.py, an consistently
with the PRSERV_HOST variable name.

(Bitbake rev: b0c277f16f9fae51914024c1daecd5d3e4fac5c2)

Signed-off-by: Michael Opdenacker <michael.opdenacker@rootcommit.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2024-10-11 12:17:36 +01:00
Chris Laplante
0f83df527d bitbake: bitbake-server: use with to avoid ResourceWarning
Prevents the following warning in bitbake-cookerdaemon.log:

2386729 10:54:16.195427 Exiting (socket: True)
2386729 10:54:16.201065 Exiting as we could obtain the lock
sys:1: ResourceWarning: unclosed file <_io.TextIOWrapper name='/home/laplante/main_yocto/build/bitbake-cookerdaemon.log' mode='a+' encoding='UTF-8'>

(Bitbake rev: 8dbf1ec8139d9dd7f52c1773cccbe7696b3ec1b4)

Signed-off-by: Chris Laplante <chris.laplante@agilent.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2024-10-11 12:17:36 +01:00
Richard Purdie
2aad7b988e bitbake: persist_data: Remove it
It was never a great solution to persisting data and there are much better
ones now. The last user has been replaced so drop the code and tests.

(Bitbake rev: 681a7516e9f7027e0be6f489c54a7a5e19fa9f06)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2024-10-09 13:04:30 +01:00
Enrico Jörns
287e2ede38 bitbake: bitbake-diffsigs: fix handling when finding only a single sigfile
This fixes the following error when calling 'bitbake-dumpsig' or
'bitbake-diffsigs' when having only a single sigfile available:

| Traceback (most recent call last):
|   File "[..]/poky/bitbake/bin/bitbake-dumpsig", line 171, in <module>
|     files = find_siginfo_task(tinfoil, options.taskargs[0], options.taskargs[1])
|             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
|   File "[..]/poky/bitbake/bin/bitbake-dumpsig", line 83, in find_siginfo_task
|     sig2 = latestsigs[1]
|            ~~~~~~~~~~^^^
| IndexError: list index out of range

Handle this by adding (and returning) the path for the second sigfile
only if one is found. This way it will work for both diffsigs and
dumpsig use case.

The calling argparse code already deals with find_siginfo_task()
returning only a single file.
For 'bitbake-dumpsig' it will just dump the single sigfile, for
'bitbake-diffsigs' it will emit a proper error message again:

| ERROR: Only one matching sigdata file found for the specified task (systemd configure)

(Bitbake rev: 25057d33e9131f3214a06bbb316c916c744f8f03)

Signed-off-by: Enrico Jörns <ejo@pengutronix.de>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2024-07-02 22:29:27 +01:00
Joshua Watt
e16d690e77 bitbake: hashserv: server: Add support for SO_REUSEPORT
SO_REUSEPORT is a socket option that allows multiple servers to listen
on the same TCP port, and the kernel will automatically load balance the
connections between them. This is particularly helpful for the hash
server since it runs in a single thread. To take advantage of a
multi-core server, multiple servers can be started in parallel with this
option (up to 1 per CPU) and the kernel will load balance between them.

(Bitbake rev: d72d5a7decb489e2af0ebc43cfea0ca3e4353e9b)

Signed-off-by: Joshua Watt <JPEWhacker@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2024-05-30 07:38:10 +01:00
Joshua Watt
e598b2d135 bitbake: bitbake-hashclient: Improve ping command line options
Adds a --quiet option to suppress the message for each ping, and report
the median ping time.

(Bitbake rev: 3c85b5e2d9b9c39507ed362aaa115b7f6f155966)

Signed-off-by: Joshua Watt <JPEWhacker@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2024-05-28 23:46:21 +01:00
Joshua Watt
5f602c1bd5 bitbake: bitbake-hashclient: Improve stress statistics reporting
Improves the way statistics are reported for the stress test. This makes
it easier to compare them to the ping test

(Bitbake rev: ce166ae25793c11b0a190c531bef0c296fd74497)

Signed-off-by: Joshua Watt <JPEWhacker@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2024-05-23 11:27:08 +01:00
Joshua Watt
76421d5742 bitbake: bitbake-hashclient: Add ping command
Adds a ping subcommand to bitbake-hashclient which can be useful to
measure connection latency

(Bitbake rev: 337487fdffae92091fc33b2346d46c39db5a130f)

Signed-off-by: Joshua Watt <JPEWhacker@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2024-05-23 11:27:08 +01:00
Michael Opdenacker
fa9689923f bitbake: prserv: add bitbake selftests
Run them with "bitbake-selftest prserv.tests"

(Bitbake rev: 34287fbf3d6be813aa5b767f540e4662f0d8d18d)

Signed-off-by: Michael Opdenacker <michael.opdenacker@bootlin.com>
Cc: Joshua Watt <JPEWhacker@gmail.com>
Cc: Tim Orling <ticotimo@gmail.com>
Cc: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2024-05-21 14:23:43 +01:00
Michael Opdenacker
4cbce9cdf7 bitbake: prserv: add "upstream" server support
Introduce a PRSERVER_UPSTREAM variable that makes the
local PR server connect to an "upstream" one.

This makes it possible to implement local fixes to an
upstream package (revision "x", in a way that gives the local
update priority (revision "x.y").

Update the calculation of the new revisions to support the
case when prior revisions are not integers, but have
an "x.y..." format."

Set the comments in the handle_get_pr() function in serv.py
for details about the calculation of the local revision.

This is done by going on supporting the "history" mode that
wasn't used so far (revisions can return to a previous historical value),
in addition to the default "no history" mode (revisions can never decrease).

Rather than storing the history mode in the database table
itself (i.e. "PRMAIN_hist" and "PRMAIN_nohist"), the history mode
is now passed through the client requests. As a consequence, the
table name is now "PRMAIN", which is incompatible with what
was generated before, but avoids confusion if we kept the "PRMAIN_nohist"
name for both "history" and "no history" modes.

Update the server version to "2.0.0".

(Bitbake rev: 48857ec3e075791bd73d92747c609a0a4fda0e0c)

Signed-off-by: Michael Opdenacker <michael.opdenacker@bootlin.com>
Cc: Joshua Watt <JPEWhacker@gmail.com>
Cc: Tim Orling <ticotimo@gmail.com>
Cc: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2024-05-21 14:23:43 +01:00
Michael Opdenacker
5f99010e41 bitbake: prserv: move code from __init__ to bitbake-prserv
This script was the only user of this code.

(Bitbake rev: 19a5595e3f70d61fd6fa414f9fd5b413a02de37b)

Signed-off-by: Michael Opdenacker <michael.opdenacker@bootlin.com>
Cc: Joshua Watt <JPEWhacker@gmail.com>
Cc: Tim Orling <ticotimo@gmail.com>
Cc: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2024-05-21 14:23:43 +01:00
Richard Purdie
f4885e97a6 bitbake: bitbake: update to version 2.9.1
This allow the use of new siggen API

(Bitbake rev: e53503546990adeab67b6d044fcce59dc5a3f455)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2024-05-08 14:53:01 +01:00
Simone Weiß
f6de2b033d bitbake: bitbake-layers: adapt force option to not use tinfoil
Fixes [YOCTO #15417]

When a layer adds a new dependency after it was added to a conf, it can not be
removed w/o this dependency in the setup. Even the dependent layer can not be
added, as the tinfoil setup will fail.
Adapt --force to not perform the tinfoil at all, the use will be at own risk,
i.e. the added layers might not parse properly afterwards.
This is not merged into the force option with -F as it even changes the loading of
plugins from other layers and is hence even more invasive as force. Instead
force can now be speciefied multiple times and is counted.

(Bitbake rev: 541fa7f582133949563e65f2d43c4b16e873e5c1)

Signed-off-by: Simone Weiß <simone.p.weiss@posteo.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2024-05-08 14:51:43 +01:00
Richard Purdie
ae3bca8492 bitbake: bitbake: Bump to version 2.9.0 development version postrelease
(Bitbake rev: 67a1aa8dbb3cb3a30fa7d697431ebb30323e4f28)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2024-04-16 08:12:27 +01:00
Richard Purdie
3bbe0a45b4 bitbake: bitbake: Bump to version 2.8.0
(Bitbake rev: c86466d51e8ff14e57a734c1eec5bb651fdc73ef)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2024-04-16 07:38:22 +01:00
Michael Opdenacker
d57e17425e bitbake: bitbake-prserv: replace deprecated optparse by argparse
optparse is deprecated since Python 2.7

Note that this is neither supposed to change the options
supported by bitbake-prserv nor the way they are interpreted.

Note that in the "--help" output, long options are now reported
for example as "--host HOST" instead of "--host=HOST" but
both are equivalent anyway, as they already were with optparse.

(Bitbake rev: 434cd00a9e5a8ef32088f1a587005adf910a92eb)

Signed-off-by: Michael Opdenacker <michael.opdenacker@bootlin.com>
Cc: Joshua Watt <JPEWhacker@gmail.com>
Cc: Tim Orling <ticotimo@gmail.com>
Cc: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2024-04-14 06:31:45 +01:00
Michael Opdenacker
4b1ef692a9 bitbake: prserv: use double quotes by default
To aligh with the hashserv code

(Bitbake rev: 7a6999750791659eaffe49aabfbfba9f37f51913)

Signed-off-by: Michael Opdenacker <michael.opdenacker@bootlin.com>
Cc: Joshua Watt <JPEWhacker@gmail.com>
Cc: Tim Orling <ticotimo@gmail.com>
Cc: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2024-04-14 06:31:45 +01:00
Joshua Watt
f2ff622a4c bitbake: bitbake-hashclient: Warn on bad .netrc
If there is an error parsing .netrc, warn the user on stderr

(Bitbake rev: 6366ea8d9c284d10bb8f4129004b55239d9022c0)

Signed-off-by: Joshua Watt <JPEWhacker@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2024-03-26 17:11:15 +00:00
Yang Xu
64057e6b15 bitbake: bitbake-worker: Fix silent hang issue caused by unexpected stdout content
This patch addresses an issue in bitbake-worker where stdout,
reserved for status reporting, is improperly accessed by child processes.

The problem occurs during the execution of parseRecipe,
which calls anonymous functions. If these functions use print-like operations,
they can inadvertently output data to stdout. This unexpected data can cause
the runqueue to hang silently, if the stdout buffer is flushed
before exec_task is executed.

To prevent this, the patch redirects stdout to /dev/null and ensures it is
flushed prior to the execution of exec_task.

(Bitbake rev: 08f3e677d6af27a41a918aaa9da9c1c9b20a0b95)

Signed-off-by: Yang Xu <yang.xu@mediatek.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2024-03-22 16:26:46 +00:00
david d zuhn
ecb1248914 bitbake: bitbake-worker: allow '=' in environment variable values
Limit the split to key & value (2 items) instead of the n items one
can get if there are '=' characters in the value.

Fixes [YOCTO #15447]

(Bitbake rev: 86315961829ab1d137a0265cc246c44d3929e1fb)

Signed-off-by: david d zuhn <david.zuhn@sonos.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2024-03-22 16:26:45 +00:00
Sava Jakovljev
0acdb81ca6 bitbake: bitbake-worker: Fix bug where umask 0 was not being applied to a task
* In the current implementation, "umask" variable is initially set to
  None and overwritten with user-specified value. However, in the worker
  implementation, a faulty if clause would only check whether the
  variable contains a value that evaluates to True, and not whether
  the variable is defined, so the value of 0 would lead to umask not
  being changed.
  This bug makes it impossible to have a task set its umask to value 0,
  for any possible reason it may want to.
  Fix this bug by extending the condition checked in the worker
  implementation.

(Bitbake rev: 19f9df6c750c592316a0fa18165b68636281fe3e)

Signed-off-by: Sava Jakovljev <sjakovljev@outlook.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2024-03-22 16:26:45 +00:00
André Draszik
e6892bc47a bitbake: git-make-shallow: support git's safe.bareRepository
When git is configured with safe.bareRepository=explicit [1], the
git-make-shallow fails miserably. LWN has an article about the
problem that this configuration option addresses and why it is useful
in [2].

It also seems that it is being rolled out in some environments as a
default for users.

In order to allow having this configuration turned on for a user's
environment in general, the fetcher has to be tought to use --git-dir=
for all relevent git operations.

The alternative, implemented here, is to forcibly turn off that option
for all git operations. In the future, we could look into converting
these to using the --git-dir= command line argument instead.

Link: https://git.kernel.org/pub/scm/git/git.git/tree/Documentation/config/safe.txt#n1 [1]
Link: https://lwn.net/Articles/892755/ [2]
(Bitbake rev: 7c63989db4590564516ed150930f4e2fa503e98f)

Signed-off-by: André Draszik <andre.draszik@linaro.org>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2024-02-19 12:00:59 +00:00
Richard Purdie
eca5708b87 bitbake: bitbake: Bump version to 2.7.3 for hashserv changes
(Bitbake rev: c1e0a0b6ddc9667c9d62319bd9ccd4eb8c64c2a6)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2024-02-19 11:58:12 +00:00
Joshua Watt
3bd2c69e70 bitbake: hashserv: Add unihash-exists API
Adds API to check if the server is aware of the existence of a given
unihash. This can be used as an optimization for sstate where a client
can query the hash equivalence server to check if a unihash exists
before querying the sstate cache. If the hash server isn't aware of the
existence of a unihash, then there is very likely not a matching sstate
object, so this should be able to significantly cut down on the number
of negative hits on the sstate cache.

(Bitbake rev: cfe0ac071cfb998e4a1dd263f8860b140843361a)

Signed-off-by: Joshua Watt <JPEWhacker@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2024-02-19 11:58:12 +00:00
Joshua Watt
1effd1014d bitbake: hashserv: Add Unihash Garbage Collection
Adds support for removing unused unihashes from the database. This is
done using a "mark and sweep" style of garbage collection where a
collection is started by marking which unihashes should be kept in the
database, then performing a sweep to remove any unmarked hashes.

(Bitbake rev: 433d4a075a1acfbd2a2913061739353a84bb01ed)

Signed-off-by: Joshua Watt <JPEWhacker@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2024-02-19 11:58:12 +00:00
Paul Gortmaker
324c9fd666 bitbake: hashserv: improve the loglevel error message to be more helpful
Coming from a kernel background, I was thinking along the lines of

	dmesg -n <integer>

for loglevel adjustments.  So I tried various large and small and
even zero number values with no luck before getting frustrated and
opening up the python.

Let us save others the frustration and give a hint what the args it
expects should look like.

(Bitbake rev: df184b2a4e80fca847cfe90644110b74a1af613e)

Signed-off-by: Paul Gortmaker <paulg@kernel.org>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2024-02-19 11:58:12 +00:00
Martin Jansa
58155ea1a3 bitbake: bitbake-diffsigs: fix walking the task dependencies and show better error
* when comparing 2 tmp/stamps/*do_configure.sigdata* I got
  TypeError("filename must be a str or bytes object, or a file")
  but both files I was comparing were
  Zstandard compressed data (v0.8+), Dictionary ID: None
  according to "file"

  with TypeError catched to show which file it failed to open I got better
  error which shows it was trying to read "do_prepare_recipe_sysroot.sigdata"
  file now and after a while you might notice that it's not just the expected
  file, but a dict with 'path', 'sstate', 'time'.

  Fix that in bitbake-diffsigs but keep the TypeError and add OSError
  in case it will eventually walk on file which isn't zstd compressed
  pipecompress throws CompressionError.

ERROR: Failed to open {'path': '5.15.do_prepare_recipe_sysroot.sigdata.99b12a401341a0df7c3553cb00c87a7674295496bd5c25ed71764ee0d0fb8eb8', 'sstate': False, 'time': 1707136354.991718}: filename must be a str or bytes object, or a file
Traceback (most recent call last):
  File "bitbake/bin/bitbake-diffsigs", line 192, in <module>
    output = bb.siggen.compare_sigfiles(options.sigdatafile1, options.sigdatafile2, recursecb, color=color)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "bitbake/lib/bb/siggen.py", line 1039, in compare_sigfiles
    recout = recursecb(dep, a[dep], b[dep])
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "bitbake/bin/bitbake-diffsigs", line 102, in recursecb
    out2 = bb.siggen.compare_sigfiles(hashfiles[hash1], hashfiles[hash2], recursecb, color=color)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "bitbake/lib/bb/siggen.py", line 857, in compare_sigfiles
    raise err
  File "bitbake/lib/bb/siggen.py", line 853, in compare_sigfiles
    with bb.compress.zstd.open(a, "rt", encoding="utf-8", num_threads=1) as f:
         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "bitbake/lib/bb/compress/zstd.py", line 12, in open
    return bb.compress._pipecompress.open_wrap(ZstdFile, *args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "bitbake/lib/bb/compress/_pipecompress.py", line 59, in open_wrap
    raise TypeError("filename must be a str or bytes object, or a file")
TypeError: filename must be a str or bytes object, or a file

* if I replace zstd file with just plaintext it fails with:
  $ echo foo > foo
  $ echo foo > bar
  $ bitbake-diffsigs foo bar
  zstd: /*stdin*\: unsupported format
  ERROR: Process died with 1
  sys:1: ResourceWarning: unclosed file <_io.BufferedReader name='foo'>

  with this change it shows the name of the file which it failed to uncompress:

  $ bitbake-diffsigs foo bar
  zstd: /*stdin*\: unsupported format
  ERROR: Failed to open sigdata file 'foo': Process died with 1
  ERROR: Process died with 1
  sys:1: ResourceWarning: unclosed file <_io.BufferedReader name='foo'>

(Bitbake rev: f3f843a4fd6e0a9e8f6edef5dd3cf1fce29c50ba)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2024-02-08 10:59:32 +00:00
Ross Burton
ff0bc2a284 bitbake: bitbake: Version bump for inherit_defer addition
We've added a new statement, inherit_defer, so bump the version so this
can be checked.

(Bitbake rev: 191e6eb2bceb467c97e315301f1f64722cf0e976)

Signed-off-by: Ross Burton <ross.burton@arm.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2024-01-26 16:25:39 +00:00
Richard Purdie
71c0d311ba bitbake: bitbake: Version bump for find_siginfo chanages
Bump the version to 2.7.1 for the find_siginfo changes.

(Bitbake rev: 03995e16bf7186f5368f772f617d563f4d280641)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2024-01-05 11:59:08 +00:00
Alexander Kanavin
cc85c8eb9d bitbake: bitbake-diffsigs/runqueue: adapt to reworked find_siginfo()
In particular having 'time' explicitly used as a sorting key should make it
more clear how the entries are being sorted.

(Bitbake rev: 5439aca056c84ab4410aaf24bdb68e896191d8e1)

Signed-off-by: Alexander Kanavin <alex@linutronix.de>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2024-01-05 11:59:08 +00:00
Richard Purdie
a41b54ccbd bitbake: bitbake: Post release version bump to 2.7.0
Bump to a development version post release.

(Bitbake rev: 28364c08f36c778a5cb2e3f20ceb052370ef153c)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2024-01-05 11:39:33 +00:00