Currently, files and directories created under ${SSTATE_DIR} when
fetching from an sstate mirror are not created with group write,
unlike when the sstate artifacts are generated locally. That's
inconsistent, and problematic when the local sstate dir is shared
among multiple users.
Wrap the fetching in a bb.utils.umask() context manager, and for simplicity
move the mkdir of SSTATE_DIR inside that.
(From OE-Core rev: a6038553aaef3b88b834a09018c524c4fa41e625)
Signed-off-by: Rasmus Villemoes <ravi@prevas.dk>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Potentially there could be a grace period for fixing recipes (with warnings, etc),
on the other hand, changes to git unpacking destination would already
break various layers, so we might as well make this a hard qa error and drop
the magic at the same time.
I commit to sending fixes for layers included in autobuilder testing
(meta-oe/arm/intel/virt/agl/aws/mingw/etc).
SOURCE_BASEDIR is at the same time adjusted to be calculated relative
to UNPACKDIR (previously it only worked if S was set to
WORKDIR/something/otherthing, and that is no longer working). It is also
no longer removed from the filesystem, as content of unpackdir is managed elsewhere.
(From OE-Core rev: 53e9ea30aaf48292307b4cff6964bead74c69fff)
Signed-off-by: Alexander Kanavin <alex@linutronix.de>
Signed-off-by: Mathieu Dubois-Briand <mathieu.dubois-briand@bootlin.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The checks are fatal, as this avoids rather more cryptic errors further
down the build. Example:
ERROR: gnu-config-native-20240823+git-r0 do_unpack: Recipes that set S = "${WORKDIR}/git" or S = "${UNPACKDIR}/git" should remove that assignment, as S set by bitbake.conf in oe-core now works.
ERROR: perlcross-native-1.6.2-r0 do_unpack: S should be set relative to UNPACKDIR, e.g. replace WORKDIR with UNPACKDIR in "S = ${WORKDIR}/perl-cross-${PV}"
Dropping the S = ${WORKDIR}/git assignment (addressing the first error) can be done
with a single sed command when there is a lot of recipes:
sed -i "/^S = \"\${WORKDIR}\/git\"/d" `find . -name *.bb -o -name *.inc -o -name *.bbclass`
Replacing WORKDIR with UNPACKDIR can be done similarly, but should be done after
the removals:
sed -i "s/^S = \"\${WORKDIR}\//S = \"\${UNPACKDIR}\//g" `find . -name *.bb -o -name *.inc -o -name *.bbclass`
(From OE-Core rev: 46480a5e66747a673041fe4452a0ab14a1736d5e)
Signed-off-by: Alexander Kanavin <alex@linutronix.de>
Signed-off-by: Mathieu Dubois-Briand <mathieu.dubois-briand@bootlin.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
This change implements a toolchain selection mechanism. Selection is
made using a set of variables, primarily PREFERRED_TOOLCHAIN_TARGET which
defaults to gcc.
It uses the familiar name for toolchain e.g. "gcc" which selects GNU
compiler + binutils as default C/C++ toolchain or "clang" which will
use LLVM/Clang Compiler. Layers an add their own toolchain definitions
too.
There are also PREFERRED_TOOLCHAIN_NATIVE and PREFERRED_TOOLCHAIN_SDK
which will ulitmately allow selection of the toolchain used for the
native/cross and nativesdk/crosssdk compilers. This currently isn't
functional but is essential to the patch to ensure things are set
to the existing gcc support in those cases.
Users would most commonly want to set:
PREFERRED_TOOLCHAIN_TARGET ?= "clang"
in local.conf or other distro specific global configuration metadata.
It is also selectable at recipe scope, since not all packages are
buildable with either clang or gcc, a recipe can explicitly require
a given toolchain using the TOOLCAHIN variable, e.g. glibc can not
be built with clang therefore glibc recipe sets:
TOOLCHAIN = "gcc"
The TOOLCHAIN variable is distinct from the user preference so recipes
with specific requirements can be identified. This also allows different
polcies to be be specified for native/SDK cases in the future.
(From OE-Core rev: 45bdedd213aff8df3214b95ef2a8551c0abd93a0)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Use the new deferred class event to set the class overrides earlier.
This improves interaction of the override with PACKAGECONFIG values
that control conditional inherits (such as python support).
This also allows toolchain configuration in an easier and more user
friendly way.
(From OE-Core rev: d58f94d5684332bd4fac3747688558bb261b9c63)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Fixes bug: https://bugzilla.yoctoproject.org/show_bug.cgi?id=15901
The cheeck for debug-deps is looking for the mere existence of the
substring "-dbg" inside of the name of an RDEPENDS package, but it should be
an endswith check. This helps with some eroneous errors in kernel
module names like:
ERROR: linux-xxx do_package_qa: QA Issue: kernel-modules rdepends on kernel-module-g-dbgp-6.12.22-ti [debug-deps]
and
ERROR: QA Issue: kernel-module-mtk-vcodec-dec-6.16.0-rc1-next-20250610-dirty rdepends on kernel-module-mtk-vcodec-dbgfs-6.16.0-rc1-next-20250610-dirty [debug-deps]
(From OE-Core rev: 1b85d84c736a0fa5cb27b8716ca37f181464c85a)
Signed-off-by: Ryan Eatmon <reatmon@ti.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
It's not unheard of for new users to create a recipe called foo-native
that has BBCLASSEXTEND="native" instead of "inherit native". This will
result in a foo-native recipe that is actually a target recipe, and a
foo-native-native recipe for native builds.
Add a test in recipe_qa to verify that recipes called -native inherit
native, and recipes called nativesdk- inherit nativesdk.
As this behaviour is expected, add the new test to the set of tests
required to pass for Yocto Project Compatible status.
(From OE-Core rev: ec2c10a3e85d0772135289fe416d13fa3afca571)
Signed-off-by: Ross Burton <ross.burton@arm.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Currently, pseudo tracks all files referenced within its presence unless
they're listed in an exclusion list. The exclusion list has grown to be
fairly unwieldy.
This patch swaps PSEUDO_IGNORE_PATHS for PSEUDO_INCLUDE_PATHS which in
theory should be easier and more explicit to maintain.
This change does drop many directories from pseudo coverage including
/home and /tmp. There may be adapatations needed for recipes/classes
using pseudo in specific ways.
(From OE-Core rev: 2502da81709f25de499277b28d33c915638c45f6)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Add a new test that checks that no tasks between do_fetch (exclusive)
and do_build (inclusive) are allowed to use the network, with rare
exceptions.
The only exception currently is build-appliance-image's do_image task,
as that currently usese pip to install the required Toaster dependencies.
Note that this will mean layers that have Go-based recipes will fail
unless they're using the gomod fetcher and have a complete list of
modules in the SRC_URI.
(From OE-Core rev: e95b3bd194e294412bc0419c9c74abfc2f37406f)
Signed-off-by: Ross Burton <ross.burton@arm.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Rearrange the class so that the check is a separate function, to make it
neater to add further tests in the future.
(From OE-Core rev: 36658a113811de98249bc1e1b79cfadd405e5391)
Signed-off-by: Ross Burton <ross.burton@arm.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
We call the installer script but if it fails the user doesn't see the
reason, which makes resolving the problem hard. Capture both stdout and
stderr as text, and display it to the user when it fails.
(From OE-Core rev: c67fbe17ce063350181c28f99662004737d8b770)
Signed-off-by: Ross Burton <ross.burton@arm.com>
Signed-off-by: Mathieu Dubois-Briand <mathieu.dubois-briand@bootlin.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Rather than reporting each invalid PACKAGECONFIG with a separate error
message, report them all with one error message.
(From OE-Core rev: bf9366583f53fe2498d7aa9192ebfe6562887cf3)
Signed-off-by: Peter Kjellerstedt <peter.kjellerstedt@axis.com>
Signed-off-by: Mathieu Dubois-Briand <mathieu.dubois-briand@bootlin.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
This makes sure invalid PACKAGECONFIGs are reported also for recipes
that have no do_configure task, e.g., packagegroups.
(From OE-Core rev: d3325c384a7df54c564cae093659cf7b692629f2)
Signed-off-by: Peter Kjellerstedt <peter.kjellerstedt@axis.com>
Signed-off-by: Mathieu Dubois-Briand <mathieu.dubois-briand@bootlin.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
This is needed to build nodejs from meta-oe. Check this early to avoid
an error later in the build.
Fixes [YOCTO #15804]
(From OE-Core rev: 19004950ad5691ef51f7fcf946558bd0de561403)
Signed-off-by: Yoann Congal <yoann.congal@smile.fr>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
This can be use to check for other flags.
(From OE-Core rev: 42d012e8414736d89618422fb6f9bce9e4a2a391)
Signed-off-by: Yoann Congal <yoann.congal@smile.fr>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Since Ubuntu 20.04 is heading to EoL and that we're having other issues
on that platform needing buildtools anyway, we're about to lose the
last python 3.8 platform we were supporting.
Bump the minimum version to 3.9 since there are many developers wanting
access to newer python features.
This likely won't be seen since the bitbake minimum has also been
increased but helps for documentation purposes.
(From OE-Core rev: c3db241c6e97420ca0705d1535a7cdd0c273b761)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Change DEBIAN_MIRROR to point at the canonical server, deb.debian.org.
This is a CDN-backed server using network magic to load balance across
the planet so there's no need to set a slew of regional mirrors.
Also add a more recent snapshot.debian.org from the beginning of 2025.
(From OE-Core rev: 3d95d45836accd29916dd8cb9bfe624d63d6c202)
Signed-off-by: Ross Burton <ross.burton@arm.com>
Signed-off-by: Mathieu Dubois-Briand <mathieu.dubois-briand@bootlin.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
In rare cases BUILDNAME can seemingly be None outside of heartbeat events
which leads to UnboundLocalErrors as bsdir and taskdir aren't defined.
Skip the code in these cases rather than generate tracebacks which cause
bitbake server to exit.
(From OE-Core rev: 0f74d804ba0daf7e8bd6481597740b9d89821414)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
This prevented package_qa sstate from being reusable unless host uid/gid
values would match exactly (and they unfortunately do on the yocto autobuilder
worker machines which all share a 'pokybuild' user).
I noticed this when testing CDN sstate reuse, which otherwise works well.
(From OE-Core rev: 6ea8b4b10b0549c858427a8411bf2a4cd5c0eb7b)
Signed-off-by: Alexander Kanavin <alex@linutronix.de>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Some modules (like Perl's MakeMaker) do not support non ASCII
characters in build folder's path, this would cause build failures
of software that does not support non ASCII.
A sanity check is added to warn the user.
Fixes [YOCTO #15764]
(From OE-Core rev: 2b3be97a0d0d60d026786a4465b24b6f6752ba32)
Signed-off-by: Sofiane HAMAM <sofiane.hamam@smile.fr>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
We don't have many cross recipes that use PACKAGECONFIG but gdb-cross does,
so correctly remap dependencies for that case allowing the gdb recipe to be
simplified.
(From OE-Core rev: c5bb7976f4d6e8559b4b87e4c3f39135dbb40ef8)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
opkg accepts both - and _ in its options but use the help text
preferred format. From https://github.com/yoctoproject/poky/pull/14
from Philipp-Alexander Blum <blum@gessler.de>.
(From OE-Core rev: 4bea64347407dfb1eb46dacffd05af3bd200b419)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
meta-oe has switched from p7zip to 7zip.
p7zip recipe does not exist anymore and p7zip is provided and rprovided
by 7zip recipe.
Use real provider instead of replaced one.
(From OE-Core rev: 5aa516bfa295d5be919459dfe45f452cdec45e81)
Signed-off-by: Peter Marko <peter.marko@siemens.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Recipes are much more readable with whitespace around the assignment operators.
Fix various assignments in OE-Core to show this is definitely the preferred
formatting.
(From OE-Core rev: 30ea609d3357fb3de911f2f6a5e6856c151b976a)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Users reported issues caused by missing the right libstdc++-version-dev.
A new function 'check_cpp_toolchain' added in sanity.bbclass to test linking libstdc++
[YOCTO #15712]
(From OE-Core rev: 611c1a26212dfbfe8d0640d9fefe5df49f7b69b8)
Signed-off-by: Christos Gavros <gavrosc@yahoo.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The idea of the base class dependency is to say "yes, I need a C cross compiler"
and this was never meant to be gcc specific. Looking at the codebase, whilst we
code triplets into this, it does overcomplicate things as there are only ever
limited, "target", "sdk" and the class extended versions like mutlilib.
After much thought, we can simplify this to virtual/cross-cc and virtual/nativesdk-cross-cc.
This lets us remove the "gcc" specific element as well as removing the over
complicated triplet usage.
At the same time, change the much less widely used "g++" variant to "c++" for
similar reasons and remove the triplet from virtual/XXX-binutils too.
Backwards compatibility mappings could be left but are just going to confuse
things in future so we'll just require users to update.
This simplification, whilst disruptive for any toolchain focused layers, will
make improved toolchain selection in the future much easier.
Since we no longer have overlapping variables, some code for that can just
be removed. The class extension code does need to start remapping some variables
but not the crosssdk target recipe names.
This patch is in two pieces, this one handles the renaming with the functional
changes separate in a second for easier review even if this breaks bisection.
(From OE-Core rev: 4ccc3bc8266c327bcc18c9a3faf7536210dfb9f0)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The git commit hashes for the kernel checkout are not reproducible under
certain conditions:
- If the git repository is initialized on an archive (rather than a
git), the initial git commit not only has the current user name set,
it also uses the current system time as committer and author date.
This will affect the initial git hash and thus all subsequent ones.
- The patches applied by the kern-tools have a valid author and date.
However, their committer again depends on the user building the BSP.
This is an issue, for example, if one compiles a kernel with
CONFIG_LOCALVERSION_AUTO enabled where the commit hash lands into the
kernel and thus the package version. This not only makes the package
version non-reproducible, but also leads to version mismatches between
kernel modules built against a fresh kernel checkout and the kernel
retrieved from the sstate cache.
The class uses 'check_git_config' from utils.bbclass, but this only sets
the git user and only if none existed before. Thus it doesn't really
help here.
Since in Git the committer information can be set only from the
environment variables GIT_COMMITTER_NAME, GIT_COMMITTER_EMAIL, and
GIT_COMMITTER_DATE, we introduce a helper function to set those and
apply the author settings in the same way.
As values simply use PATCH_GIT_USER_NAME, PATCH_GIT_USER_EMAIL (from
patch.bbclass) and SOURCE_DATE_EPOCH.
For convenience, put the new helper 'reproducible_git_committer_author'
into utils.bbclass next to 'check_git_config' so others can use it, too.
Using this helper in kernel-yocto.bbclass makes the committer and author
date/name/email for the initial commit reproducible, as well as the
committer name/email for the patches applied with kern-tools.
For debugging purpose, allow disabling the reproducibility features by
setting KERNEL_DEBUG_TIMESTAMPS to "1".
Suggested-by: Felix Klöckner <F.Kloeckner@weinmann-emt.de>
(From OE-Core rev: aab4517b4649917abd519ea85a20fd9d51bf3d99)
Signed-off-by: Enrico Jörns <ejo@pengutronix.de>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
There is the following warning when the do_package_write_rpm task is executed
when bitbake linux-yocto:
warning: %clamp_mtime_to_source_date_epoch is deprecated, please use %build_mtime_policy
After RPM is upgraded to 4.20.0, the %clamp_mtime_to_source_date_epoch macro is
now deprecated in favor of %build_mtime_policy set to clamp_to_source_date_epoch.
(From OE-Core rev: fa9b9e599bbf3ac73c7a9b1d15ccb0e0bbbe4c11)
Signed-off-by: Bin Lan <bin.lan.cn@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
When TMPDIR has length 410, bitbake uninative-tarball will fail at
do_populate_sdk with error: Failed to open backup database:
"TMPDIR/work/x86_64-nativesdk-pokysdk-linux/uninative-tarball/1.0/sdk/image/var/lib/dnf/history.sqlite"
sqlite3 has a hardcode max pathname limit 512, refer [1], and there is a
check, refer [2], pathname + 8 > max path name limit(512) is not
allowed. So the pathname length need to <= 504. While, length of
"TMPDIR/work/x86_64-nativesdk-pokysdk-linux/uninative-tarball/1.0/sdk/image/var/lib/dnf/history.sqlite"
is 505, so error happened.
We had met similar issue [3], and refer the comments, upstream reject
the change of the hardcode length, see [4][5].
This patch try to improve and workaround this issue by lower max TMPDIR
length to 400, at least for distro name length <=13, the build can
success.
[1] https://github.com/sqlite/sqlite/blob/master/src/os_unix.c#L180
[2] https://github.com/sqlite/sqlite/blob/master/src/pager.c#L4777
[3] https://bugzilla.yoctoproject.org/show_bug.cgi?id=12374
[4] https://sqlite.org/cgi/src/tktview/c060923a5422590b3734
[5] https://sqlite.org/forum/forumpost/0b1b8b5116
(From OE-Core rev: da1c0998bdaf5def7ae94f44a33b9a505edc99c1)
Signed-off-by: Changqing Li <changqing.li@windriver.com>
Signed-off-by: Mathieu Dubois-Briand <mathieu.dubois-briand@bootlin.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
So that value "0" can turn them off.
(From OE-Core rev: 1a2afcd0773c5ec5946d1dfc27bde585c52a5724)
Signed-off-by: Robert Yang <liezhi.yang@windriver.com>
Signed-off-by: Mathieu Dubois-Briand <mathieu.dubois-briand@bootlin.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Instead of iterating every datastore value by hand to find tasks, use
the new bb.build.listtasks() function (bitbake 185c4b)
(From OE-Core rev: 466c3ed0d01bc70caa29d5eb8bb99f7d0e6e710c)
Signed-off-by: Ross Burton <ross.burton@arm.com>
Signed-off-by: Mathieu Dubois-Briand <mathieu.dubois-briand@bootlin.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
With this release, as expected, deprecated openpgp support
has been fully removed and replaced with sequoia:
https://github.com/rpm-software-management/rpm-sequoia/
Alas, it's written in rust, and has no recipe, so for now
signing rpm packages has to be disabled.
Remove package name parameter to %prep in source archiving, as it's
been dropped in the new rpm, and wasn't needed to begin with.
Drop 0001-perl-disable-auto-reqs.patch
(files removed upstream).
(From OE-Core rev: 8c15b4577d5e554cc2dd5adfb88b816894b05a9a)
Signed-off-by: Alexander Kanavin <alex@linutronix.de>
Signed-off-by: Mathieu Dubois-Briand <mathieu.dubois-briand@bootlin.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
During handle PACKAGECONFIG, DEPENDS/RDEPENDS/RRECOMMENDS is appended by
calling function appendVar according to the PACKAGECONFIG setting:
PACKAGECONFIG[f1] = "\
--with-f1, \
--without-f1, \
build-deps-for-f1, \
runtime-deps-for-f1, \
runtime-recommends-for-f1, \
packageconfig-conflicts-for-f1"
[snip of base.bbclass]
pkgconfigflags = d.getVarFlags("PACKAGECONFIG") or {}
if pkgconfigflags:
...
appendVar('DEPENDS', extradeps)
appendVar('RDEPENDS:${PN}', extrardeps)
appendVar('RRECOMMENDS:${PN}', extrarrecs)
...
[snip of base.bbclass]
After following commit in meta-oe,
[fe3ba7ac5c cryptsetup: demote udev runtime dependency to recommendation]
'udev lvm2' changed from RDEPENDS to RRECOMMENDS, but RRECOMMENDS is not
handled in this process, and causes following error:
$bitbake lib32-cryptsetup
ERROR: QA Issue: lib32-cryptsetup package lib32-cryptsetup - suspicious values 'udev lvm2' in RRECOMMENDS [multilib]
In layer openembedded-core, we do have very few recipes that have
runtime-recommends-for-f1 in PACKAGECONFIG setting, but f1 is not
enabled in PACKAGECONFIGS, so not report error when multilib is enabled.
(From OE-Core rev: 2f539a1318a80d3ad5d0d43a9ba4070465e402af)
Signed-off-by: Changqing Li <changqing.li@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The check to see if a provider of a given package is listed first
checks for an exact match of the provider name. However, if this match
existed, but didn't match in the task dependencies, it would not
continue to look for other providers of package. This would manifest if
one (non-virtual) recipe package RPROVIDES the name of a package
produced by another recipe.
Fix this, and also clean up the code to make it more readable by using a
function to check if a runtime dependency is in the task dependencies.
In addition, if no provider is found, list all the possible providers
instead of the last one that was looked at.
(From OE-Core rev: f13de6ab616eb1e38960a2296111febe2a9f4a28)
Signed-off-by: Joshua Watt <JPEWhacker@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
These two files are the only ones that are left in meta/lib.
They logically belong to meta/lib/oe, so move them there.
(From OE-Core rev: c65dd0e3e463d6072b9364ac74e1fef0d998068f)
Signed-off-by: Chen Qi <Qi.Chen@windriver.com>
Signed-off-by: Mathieu Dubois-Briand <mathieu.dubois-briand@bootlin.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
insane.bbclass now defines CHECKLAYER_REQUIRED_TESTS list with required
QA checks that are becoming mandatory for Yocto Project Compatible layers.
Update yocto-check-layer.bbclass in order to catch when packages from such
layers try to skip any of the required QA checks.
(From OE-Core rev: 9c3ba88628853b20fb4c98c99cf3fe8349024016)
Signed-off-by: Denys Dmytriyenko <denys@konsulko.com>
Signed-off-by: Mathieu Dubois-Briand <mathieu.dubois-briand@bootlin.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
We'd like to start requiring some set of checks are enabled for Yocto Project
Compatible Status and to pass yocto-check-layer. Start by splitting ERROR_QA
into two sets, the ones we think can be required and the ones we know have
challenges to implement (e.g. with prebuilt binaries).
To change the required list, the YP TSC would need to approve.
(From OE-Core rev: ce2e42ace2d15fb6745437cf0a7f07d28398ca12)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Signed-off-by: Mathieu Dubois-Briand <mathieu.dubois-briand@bootlin.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
TL;DR version:
with this, and the previous compression level changes
I am seeing drastic speedups in package_write_rpm completion times:
webkitgtk goes from 78 seconds to 37 seconds
glibc-locale goes from 399 seconds to 58 seconds (!)
The long version:
rpm uses multithreading for two purposes:
- spawning compressors (which are nowadays themselves
multi-threaded, so the feature is not as useful as it once
was)
- parallel file classification
While the former behaves well on massively parallel CPUs
(it was written and verified here :), the latter was then added
by upstream and only benchmarked on their very old, slow laptop,
apparently:
41f0e214f2
On anything more capable it starts showing pathologic behavior,
presumably from spawning massive amount of very short-lived threads,
and then having to synchronize them. For example classifying glibc-locale
takes
5m20s with 256 threads (default on my machine!)
1m49s with 64 threads
59s with 16 threads
48s with 8 threads
Even a more typical recipe like webkitgtk is affected:
47s with 256 threads
32s with 64 threads
27s with 16 or 8 threads
I have found that the optimal amount is actually four: this also
means that only four compressors are running at a time, but
as they're themselves using threads, and typical recipes are dominated
by just two or three large packages, this does not affect overall
completion time.
(From OE-Core rev: 286d456e71ee2730c197ce394d6be2c7eeced18d)
Signed-off-by: Alexander Kanavin <alex@linutronix.de>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
zstd uses 3 by default (and ZSTD_COMPRESSION_LEVEL is set to that),
while 19 is the highest and slowest.
It's not clear why 19 was picked to begin with, possibly
I copy-pasted it from rpm's examples without thinking:
https://git.yoctoproject.org/poky/commit/?h=master-next&id=4a4d5f78a6962dda5f63e9891825c80a8a87bf66
This brings significant speedups in rpm's compression step:
for example compressing webkitgtk takes 11s instead of 36s.
The rpm size increases from 175648k to 234860k. I think it's
a worthy default tradeoff.
(From OE-Core rev: f1c7d76361425c85d68ad1f61be1e7ff05df4bab)
Signed-off-by: Alexander Kanavin <alex@linutronix.de>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
As sstate_archive_package just calls tar, writing the function in shell
is actually more complex and opaque than the equivalent python.
Don't check for zstd vs pzstd, we have pzstd in HOSTTOOLS so it will
always be available.
(From OE-Core rev: c9ac5d9d94f254292cf3cafdf273dd6b61d3baa7)
Signed-off-by: Ross Burton <ross.burton@arm.com>
Signed-off-by: Mathieu Dubois-Briand <mathieu.dubois-briand@bootlin.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The code was changing the timestamps of the files in the do_package output,
particularly the files added for debug sources. This was to do two things:
a) make do_package sstate more reproducible
b) ensure better hash equivalence matching
Unfortuately the debug source files are hardlinks into the source tree for
efficiency so touching these, touches a lot of files in ${B} and ${S}. This
causes unpredictable effects if compile is run again for example, or could
cause compiling in the install task.
The hash equivalence matching is of key importance but we can mimic that
using clamping of the file timestamps in the depsig output used to generate
the hashes.
This patch drops the global timestamp clamping, instead allowing the files
to retain their creation timestamps into sstate. This makes do_package sstate
slightly less reproducibile. We could clamp the sstate timestamps but that
would lead to two different sets of timestamps depending on whether the
data came from sstate or not. I'd prefer to have consistent code behaviour,
rather than differing behavhour depending on whether data came from sstate
or not.
If we wanted to have reproducibiliy and fix the "corruption" of S/B and have
consistent codepaths, the only other option would be two copies of the
sources, which could end up huge and seems the least desireable option.
This patch therefore drops the timestamp clamping in the sstate files
and tweaks the depsig data generation to clamp the timestamps for do_package
instead since this seems the best compromise.
I validated that rpm/deb/ipk files still generate correctly as before.
(From OE-Core rev: 475759fdab7200488b2a568b2ba1aa31a456d113)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Instead of checking for incompatible licenses in the anonymous python
and setting '_exclude_incompatible-', (re)check all the packages in
populate_packages(). This ensures that all packages are processed, even
dynamically generated ones.
The use of the '_exclude-incompatible-' variable set in base.bbclass has
been the mechanism used for per-packages licenses since it was added as
a feature (although with different names for the variable throughout
history). However, since this misses dynamic packages, calling
oe.license.skip_incompatible_package_licenses() a second time on the
actual final package set is a better solution.
(From OE-Core rev: 1816c5a5e724a7f907c9afe4a47b6699ea081ace)
Signed-off-by: Joshua Watt <JPEWhacker@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Moves the code that skips packages with incompatible licenses to the
library code so that it can be called in other locations
(From OE-Core rev: 86eb409e3c1b30110869ec5a0027ae2d48bbfe7f)
Signed-off-by: Joshua Watt <JPEWhacker@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Moves several of the functions in license.bbclass to be library code
New function dependencies were manually verified using bitbake-dumpsigs
to ensure that bitbake identified the same dependencies even though they
are now in library code (although the new function names mean that the
task hashes still change)
(From OE-Core rev: 0333e04e353991260c5f67a72f80f3ab9dcf526a)
Signed-off-by: Joshua Watt <JPEWhacker@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Currently, if you "bitbake XXX" and XXX depends on something else,
the do_package_qa teask for that something may not run. Users would
generally expect it to have though.
Add in the missing dependency to ensure that do_build does trigger
the right package_qa tasks.
(From OE-Core rev: e0beb64c6d3cf1d649f79a8704fb25cdf83b4a8b)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Bitbake preserve network for non-local uid, refer [1],
so check_userns is not needed for non-local uid
[1] https://git.openembedded.org/bitbake/commit/?id=4eafae7904bae6e5c6bc50356e8a9077f2e207fa
(From OE-Core rev: 808d0cece22bcbee15236717e158da247cbedaf1)
Signed-off-by: Changqing Li <changqing.li@windriver.com>
Signed-off-by: Mathieu Dubois-Briand <mathieu.dubois-briand@bootlin.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
There is a race condition when iterating directories which are being
altered whilst iterating, which is something that can and does happen
when do_package_qa runs at the same time as eg do_package_write_ipkg
(the opkg metadata is written inside the build tree). The race is that
naive code will list a directory contents and then stat() each name to
determine if its a directory or file. The classic failure that we see
is that CONTROL/ is found on a listdir but deleted by the time the stat
happens, so is incorrectly listed as a file (because it is not a
directory).
Since Python 3.5, os.walk() uses scandir() instead of listdir() which
mitigates this race by returning the file type alongside the name, so
a stat is no longer needed to identify the type.
However, cachedpath.walk() was copied from Python before this, so it
uses listdir() and has this race condition. Since I changed insane to
use cachedpath.walk()[1] I inadvertently reintroduced this race.
I believe there's actually no need to use cachedpath.walk() and a
logical fix is to simply use os.walk():
With os.walk() each directory is listed and categorised in a single
os.scandir() as the underlying syscall, getdents64, returns the type.
However, cachedpath.walk() uses os.listdir() which ignores the type
field returned and has to do a stat() on every file to determine the
type.
Thus, we should switch users of cachedpath.walk() to os.walk(): there's
no real gain in what is effectively just a prefetch for the stat cache,
but depending on what the calling code does may result in more stat()
calls than needed.
In the future we may want to redesign cachedpath to reimplement walk so
that it can also cache the DirEntry instances as returned by scandir()
as that will avoid needing to call stat() at all in many cases. However
I believe we should instead use a caching pathlib.Path instance instead.
[1] cad3c8 insane: use oe.cachedpath.CachedPath instead of os.path
(From OE-Core rev: 22e4486d65e4874bf48d89160d69118f318278e8)
Signed-off-by: Ross Burton <ross.burton@arm.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
