We have other places in the code where we need to take filemode/mask
information from a bitbake variable and turn it into a real python
number. Turn this internal code into public API in bb.utils and
add some tests for it.
(Bitbake rev: d89e30fb2fb15b09f2cb95c4e5aa9f749ca257ea)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The bitbake worker/server IPC is asynchronous, but tinfoil only has
functionality to wait for a response on the build_targets() call.
Extract the bulk of the "wait for events and handle errors" logic to a
standalone wait_for wrapper, which is the build_targets code without the
extra_events or event_callback arguments (for now).
Then use this to create a build_file_sync() helper that just wraps the
existing build_file() with @wait_for.
(Bitbake rev: bacd125a9da66cd205f6ba2ab17930b976e82150)
Signed-off-by: Ross Burton <ross.burton@arm.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The current default value for BB_PRESSURE_MAX_CPU is 500, which is
really low for most systems. Provide a value of 15000 which limits the
load average without being extremely slow either.
Provide similar values for BB_PRESSURE_MAX_IO and
BB_PRESSURE_MAX_MEMORY. Mention that these should be adjusted depending
on the need, and warn about the potential spam of messages when the
value is too low.
(Bitbake rev: 09baa527dafca4bdf56a9189f6b3f7512886cbfb)
Signed-off-by: Antonin Godard <antonin.godard@bootlin.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
This variable was recently added with 378db0fdd95f ("fetch2/git: allow
overriding default unpack directory with a variable").
(Bitbake rev: a569be0d369cb8b9457beedfeb75b984f751ba5a)
Signed-off-by: Antonin Godard <antonin.godard@bootlin.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
It makes sense to allow the shallow clone mirror tarball name to include the tag
name so that tags can be added to existing urls and the tarballs will be maintained
correctly. The code already allows this to be done easily just by moving the tag
handling code.
(Bitbake rev: 68fce3be14e4dd801661f4ef302d229fb16a04b5)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Currently, tags are fetched as just a name, which works but means they're not
seen as tags by git commands like git describe. Instead, fetch them as refs/tags/XXX
which means such commands then work correctly.
(Bitbake rev: c1f30ad61f5e55beb377451887bbbc5cb569f2e5)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Also, drop an obsolete S = WORKDIR/git note.
(Bitbake rev: 3884b386cda649030f59e3040dd08e6aa2ec50cf)
Signed-off-by: Alexander Kanavin <alex@linutronix.de>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The display window scrolling mechanism we were using was failing in one case
and needed in another. After trying various approaches, this one with
window.scrollTo seems to work and fixes the test failures.
(Bitbake rev: f1175778108a559e1dd0d55cb68f42816c86f393)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
When reviewing proposed fragments to add settings for DISTRO and MACHINE,
RP noted that such fragments only add clutter and overhead, and there's
no need to maintain them as separate files.
Rather when bitbake sees 'fragmentvar/fragmentvalue' it can expand that into
FRAGMENTVAR = "fragmentvalue".
To achieve that, 'addfragments' directive is extended with a parameter
that sets the name of the variable that holds definitions of such
built-in fragments, for example like this:
"machine:MACHINE distro:DISTRO"
Then each enabled fragment name is matched against these definitions and the
respective variable is set, e.g. 'machine/qemuarm' would match
'machine:MACHINE' and result in MACHINE set to 'qemuarm'.
This happens before any fragment files are looked up on disk,
and no such lookup happens if there was a match, which should prevent
possible misuse of the feature. So the builtin fragment definition
is also an allowlist for them.
Please also see the patches for oe-core that show an application of the feature.
(Bitbake rev: 3b9d7bea915dc7f10e845854f1dae325743f9456)
Signed-off-by: Alexander Kanavin <alex@linutronix.de>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
This will allow setting the variable in oe-core such that it matches
S, which in turn will allow dropping S from almost all git:// fetching recipes,
and not having to set it in newly written ones.
(Bitbake rev: 378db0fdd95f8704fccd852452555bfddcad384d)
Signed-off-by: Alexander Kanavin <alex@linutronix.de>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Update version to 2.15.0 for the development series and features needed for
toolchain selection in OE.
(Bitbake rev: c2f29c9475c4b9cdd12af1f8610f2675f8fdd964)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Calling oe-debuginfod in a build failed:
...
$ oe-debuginfod
|Getting sysroot...
|Error: NOTE: Reconnecting to bitbake server...
|NOTE: Retrying server connection (#1)... (18:55:53.009687)
|path-to-build/tmp/work/x86_64-linux/elfutils-native/0.192/recipe-sysroot-native doesn't exist.
|Have you run 'bitbake elfutils-native -caddto_recipe_sysroot'?
...
The script oe-debuginfod calls bitbake-getvar to get sysroot, the
output of bitbake-getvar was mixed with info output of bitbake
...
NOTE: Reconnecting to bitbake server...
NOTE: Retrying server connection (#1)... (18:55:53.009687)
...
Set logger level to logging.WARNING to skip info output
for quiet
(Bitbake rev: 873c524e1a33846df8f34b7c87b298349277b3d5)
Signed-off-by: Hongxu Jia <hongxu.jia@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Add support for automatically promoting class inherits to deferred inherits
by listing them in the BB_DEFER_BBCLASSES variable.
(Bitbake rev: 8e741b2e885a12d119788d04aa4efcd724dd6bfa)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Now that deferred inherits are extension specific, we can pass this
list to an event, which our metadata can use to set class overrides
earlier (as an example).
There are limitations to this, the list of classes is unexpanded and
recursive classes are not visible. There isn't much that can be done
about this, the ones we are interested in would usually be visible
at the top level (such as class extensions).
(Bitbake rev: 205d461c05fc7b4a7c81039af3bc3fd71cbb982c)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Currently deferred inherits get processed once for all class extensions
as a minor speed optimisation. Unfortunately this limits our options for
being able to report deferred classes to our code.
There are two challenges with using our deferred classes in OE at present.
One is that PACKAGECONFIG values don't work well with class overrides like
class-native if there are deferred classes based on PACKAGECONFIG, such
as python support. The second is that toolchain selection is proving
problematic to implement due to interactions between the toolchain deferred
inherit, the class extensions and class overrides being very late.
By changing deferred inherits to be recipe extension specific, we open
the way to generate events and "peek" at where things will end up,
allowing the class overrides to be set earlier.
The class extension code is updated to use a deferred inherit for the
class extension inheriting so that it is still inherited last.
(Bitbake rev: 29277cf4d88eb4dfa9572851177d009eab5afd0c)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Add Walnascar as a supported release manual, and move Styhead to the
outdated release manuals.
(Bitbake rev: 629aff734794568510c30939a4584cee2e0185ec)
Signed-off-by: Antonin Godard <antonin.godard@bootlin.com>
Signed-off-by: Mathieu Dubois-Briand <mathieu.dubois-briand@bootlin.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
AZ_SAS token should be prefixed with a question mark. Add a sanity check for
this and fix the documentation.
[YOCTO #15882]
(Bitbake rev: 22011765202514600314732b97f1bb938e21f585)
Signed-off-by: Robbin Van Damme <robbinvandamme@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The project is switching the way handle our CDN provision of sstate objects,
update the URL accordingly.
(Bitbake rev: 3372524cb961d95993b27fe4a8d794cdb7255e09)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
- Since `ud.path` contains in that case the `PREMIRRORS` prefix path,
this change ensures that a correct symlink is set up.
(Bitbake rev: 37ed18e45aa17406162efc5ee3ddb2d6b33d07b9)
Signed-off-by: Stefan Koch <stefan-koch@siemens.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Some .deb files contain uncompressed data tarballs which do not have an
additional file extension after `.tar`. Add support for such cases while
preserving the existing behaviour.
(Bitbake rev: e3834deb427ceb1d1ae9a96c1e0dec990d7f3805)
Signed-off-by: Isaac True <isaac.true@emlix.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
In scenequeue data generation loop, progress bar update in each cycle cause
a significant loss of speed. Remove progress bar update in for loop. It was
become quite fast (faster 30 times, about) without progress bar update.
(Bitbake rev: f775e53f1196e3e5d0ded277c061c5c268b3b28d)
Signed-off-by: Benjamin Szőke <egyszeregy@freemail.hu>
Signed-off-by: Mathieu Dubois-Briand <mathieu.dubois-briand@bootlin.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The default adapters for sqlite datetime are deprecated as of Python
3.12, so implement our own.
[YOCTO #15333]
(Bitbake rev: 38a1d715bf58acbc9cb21eed413b3542c81cf15a)
Signed-off-by: Joshua Watt <JPEWhacker@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
By switching from print() to bb.plain() to print failing task logs, we
allow them to be saved in BB_CONSOLELOG.
Fixes [YOCTO #15798]: This allows AB reproducibility test to save the
full log of the failing tasks and helps debugging.
(Bitbake rev: d3b3ad32da7c7ebf61814fc807f8667a37aa149b)
Signed-off-by: Yoann Congal <yoann.congal@smile.fr>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Update the fixtures generator table and update the fixtures removing
styhead and adding whinlatter.
(Bitbake rev: 72b1dee84df641cb24bc726f2dfe8f1cf1555247)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Add a test case to ensure git shallow fetch succeeds for SRC_URI
with tag containing slash.
For example, we want to succeed for SRC_URI like below:
SRC_URI = "git://salsa.debian.org/debian/debianutils.git;protocol=https;branch=master;tag=debian/${PV}"
See the following link for more information:
https://bugzilla.yoctoproject.org/show_bug.cgi?id=15862
(Bitbake rev: 919d4cf6e688e67229c46d30c84d523b21936377)
Signed-off-by: Chen Qi <Qi.Chen@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
If a tag contains slash, e.g., debian/5.22, then shallow clone
fails because it's using a wrong ref.
To reproduce the issue, add the following lines in local.conf:
BB_GIT_SHALLOW = "1"
BB_GENERATE_SHALLOW_TARBALLS = "1"
And then run 'bitbake debianutils -c fetch'.
What the original os.path.basename(ref) wanted to do is to remove
the strings such as refs/heads/. So we do it explitly to fix this
issue.
Fixes: [YOCTO #15862]
(Bitbake rev: c6d6999f1ed01e7445b8f177a888038edacf555c)
Signed-off-by: Chen Qi <Qi.Chen@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Add a test case to verify that the gitsm fetcher properly handles
repositories storing objects with LFS.
The test case verifies that LFS objects are fetched on the initial clone
but also ensures that consecutive updates extend the original clone with
any newly referenced LFS objects.
(Bitbake rev: 2a8722ddd155596862029f6ea34e1e92c77e0b7f)
Signed-off-by: Philip Lorenz <philip.lorenz@bmw.de>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Skipping smudging prevents the LFS objects from replacing their
placeholder files when `git submodule update` actually checks out the
target revision in the submodule. Smudging cannot happen earlier as the
clone stored in `.git/modules` is bare.
This should be fine as long as all LFS objects are available in the
download cache (which they are after the other fixes are applied).
(Bitbake rev: d270e33a07c50bb9c08861cf9a6dc51e1fd2d874)
Signed-off-by: Philip Lorenz <philip.lorenz@bmw.de>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The existing code would pass `True` or `False` to the git fetcher. As
the fetcher expects `lfs` to be set to `1` this always lead to LFS
fetching being disabled.
(Bitbake rev: 5e487a5a096400271ed1e29b0df72903f2304e49)
Signed-off-by: Philip Lorenz <philip.lorenz@bmw.de>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Its not clear which exact git-lfs versions failed to work with bare
repositories, however git-lfs 2.13.2 which is shipped by Debian 10 (i.e.
the oldest supported distribution by scarthgap) shows no issue when
fetching into a bare repository. Switch to git-lfs fetch which in turn
eliminates issues seen when using the gitsm fetcher with submodules
utilizing lfs.
In these scenarios, fetching of LFS objects did not actually happen as
the gitsm fetcher parametrizes the to be fetched repositories with
`bareclone=1` which in turn means that the target revision was never
checked out (and therefore no LFS objects were fetched).
(Bitbake rev: 501d3317df5780a5b0e3c2562b26ed276c1dbe90)
Signed-off-by: Philip Lorenz <philip.lorenz@bmw.de>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Its unclear why this function does not operate on the desired source
revision to begin with (which really should be the decider on whether a
particular source revision uses LFS or not). Simplify the decision logic
by always checking the `.gitattributes` file of the target revision.
(Bitbake rev: b3faa0ce5c0a6945f26b5b303a7f38c00d132397)
Signed-off-by: Philip Lorenz <philip.lorenz@bmw.de>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
So far, existence of `git-lfs` was only checked during unpacking. As the
binary is also used in earlier steps also check for its existence there.
Additionally, factor out the LFS existence check into a dedicated
function and call it wherever git-lfs is used for the first time.
(Bitbake rev: 5818367db9b261b7e07c347d38044e6cba8f9727)
Signed-off-by: Philip Lorenz <philip.lorenz@bmw.de>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Avoid multiple import statements for anything that is used more than
once. Additionally, drop no longer used imports.
(Bitbake rev: 7c74310440f4d6ec47cf5bacf597e18308b3bb20)
Signed-off-by: Philip Lorenz <philip.lorenz@bmw.de>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
There's no need to pass `name` when it is no longer used.
(Bitbake rev: b132d35dee643e270e3e6dd536dcc90334a0111c)
Signed-off-by: Philip Lorenz <philip.lorenz@bmw.de>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Replace the now obsolete paragraph of the "Other Functions" section by
a link to the new "Library Functions" document.
(Bitbake rev: 0eecb4b2ee4fe5339340ed0d08783aa03bf051cc)
Signed-off-by: Antonin Godard <antonin.godard@bootlin.com>
Signed-off-by: Mathieu Dubois-Briand <mathieu.dubois-briand@bootlin.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Add a new document to the BitBake user manual that automatically
documents the library functions from their docstrings. The docstrings
can be formatted in reStructuredText.
Here logging utilities and the bb.utils module is documented. Some
members of the utils module were deliberately excluded as their usage is
most likely only internal to BitBake.
Fixes [YOCTO #9612]
(Bitbake rev: 0a711949acc2696f32a61c591ee3ea37041acb91)
Signed-off-by: Antonin Godard <antonin.godard@bootlin.com>
Signed-off-by: Mathieu Dubois-Briand <mathieu.dubois-briand@bootlin.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Format the docstrings of the utils modules to be automatically
documented with the autodoc Sphinx extensions.
(Bitbake rev: 4963bfc6045ad1f49e721edd97766dab1e2d1edc)
Signed-off-by: Antonin Godard <antonin.godard@bootlin.com>
Signed-off-by: Mathieu Dubois-Briand <mathieu.dubois-briand@bootlin.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Format the docstrings of the utils modules to be automatically
documented with the autodoc Sphinx extensions. Sphinx syntax can be used
in those for proper formatting. Cross-referencing with :term: is not
possible in these.
(Bitbake rev: 2fa1c7ad43639c6d25c94b7794bcce5f5ff74e10)
Signed-off-by: Antonin Godard <antonin.godard@bootlin.com>
Signed-off-by: Mathieu Dubois-Briand <mathieu.dubois-briand@bootlin.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
lock_timeout_nocheck() can be interrupted immediately after enterring
the try-block and prior to initializing 'l', for example with a
ctrl-C, the code in finally will still be run and the 'if l' will
fail. Initialize 'l' as False to avoid this possiblity.
(Bitbake rev: 4885cd9d275ba2ab60e5c76aed856c34533cd3ae)
Signed-off-by: Mark Asselstine <mark.asselstine@windriver.com>
Signed-off-by: Mathieu Dubois-Briand <mathieu.dubois-briand@bootlin.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
A lack of whitespace around variable assignment operators makes the
files harder to read.
There is a deeper issue in that a "+" character can sometimes be confused
between the variable name and the assignment operator.
Start showing warnings for such usage so we encourage people to use
consistent whitespace which helps with file readability in general.
(Bitbake rev: 24772dd2ae6c0cd11540a260f15065f906fb0997)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Fix the old Python 2 print statement, replace it by a bb.plain() call
instead. Also replace time by datetime for a slightly simpler example.
[YOCTO #14670]
Reported-by: Robert Berger <pokylinux@reliableembeddedsystems.com>
(Bitbake rev: 874c607f475f3d60677d2720b80a28d0c2e963c6)
Signed-off-by: Antonin Godard <antonin.godard@bootlin.com>
Signed-off-by: Mathieu Dubois-Briand <mathieu.dubois-briand@bootlin.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Adds bb.parse.vardeps bb.parse.excludevardeps function decorators that
can be used to explicitly add or exclude variables from a python
function parsed by bitbake
(Bitbake rev: 030fb3dee067640a3a50f24a53d200bdb5048376)
Signed-off-by: Joshua Watt <JPEWhacker@gmail.com>
Signed-off-by: Mathieu Dubois-Briand <mathieu.dubois-briand@bootlin.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Correct typo Fash -> Fast.
With it also delete some trailing spaces which are unnecessary and
editors are trying to remove when changing anything in the file.
(Bitbake rev: b96040a0a066344527f0bc6ca4d09cda6b9d2f11)
Signed-off-by: Peter Marko <peter.marko@siemens.com>
Signed-off-by: Mathieu Dubois-Briand <mathieu.dubois-briand@bootlin.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Before fast shallow tarballs the shallow tarball with lfs worked only
if the original .../downloads/git2/... directory existed. Once it was
not there, do_unpack on shallow tarball without lfs files failed due to
disabled network.
It was fixed for fast shallow tarballs and this commit fixes the other
scenarios, too.
This can happen when such shallow tarball is put to mirror or if
someone does cleanup to reclaim disk space.
(Bitbake rev: 4e6fb31a88448cdacf4e9f84cf0a8a8035f84d63)
Signed-off-by: Peter Marko <peter.marko@siemens.com>
Signed-off-by: Mathieu Dubois-Briand <mathieu.dubois-briand@bootlin.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Using a shallow clone to simulate an outdated git mirror tarball does
not work in the intended way. A shallow clone already contains the
latest commit which can hide certain fetcher behavior. Simulate an
outdated mirror tarball, as the test titles indicate, by removing the
newer commits from the mirror.
(Bitbake rev: a51ee01f0a586fefd5a4061f4a1ca6cbf81b7046)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
When invoking the original git fetcher after downloading a mirror
tarball, BB_FETCH_PREMIRRORONLY is ignored. This leads to git fetch
commands targeting the upstream source being executed silently. Ensure
setting BB_NO_NETWORK before invoking the original fetcher. While this
was only observed for git, setting this in general for all fetcher
types makes sense at this location.
(Bitbake rev: 1b1321f2b60c0a66159e3f20c6befcb0b3ccc4c7)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
