There is a long standing bug where FILES_INFO isn't written into pkgdata
with a package suffix. This means if the files are read into the datastore
as intended, the last one "wins".
Fix this to work as intended. Most of the call sites using the data need
to be updated to handle this and the overrides change correctly.
Also fix some other problematic references noticed along the way.
(From OE-Core rev: a1190903e0a61a12c9854c96af918ae8d12c6327)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Company name is properly spelled as "Red Hat".
(From OE-Core rev: cd60f20fe40bba5b80ab604ee34fbf4ec2b0b4a2)
Signed-off-by: Robert P. J. Day <rpjday@crashcourse.ca>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
This reverts part of oe-core eecbe62555, which was a previous attempt
to solve the Y2038 problem. This is now solved centrally in e2fsprogs,
so doesn't need to be dealt with in wic.
We don't revert the commit entirely, to retain the warning if a
filesystem has small inodes.
(From OE-Core rev: 7e8017208bed98b6c90735cb641fc9d7aedf9140)
Signed-off-by: Ross Burton <ross.burton@arm.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Currently lines like below are converted.
e.g.
# IMAGE_INSTALL_append = " A"
But lines without a leading whitespace are not converted.
e.g.
#IMAGE_INSTALL_append = " A"
We should be converting both.
(From OE-Core rev: 1994e3844c1aa6b595c0c18040e4f8240fa04438)
Signed-off-by: Chen Qi <Qi.Chen@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
e.g. VIRTUAL-RUNTIME_com.webos.service.flowmanager_armv4 weren't replaced
with VIRTUAL-RUNTIME_com.webos.service.flowmanager:armv4 or when package
name contains a dot like in:
RDEPENDS:gstreamer1.0-meta-base:remove
(From OE-Core rev: 41bff44bd26c09573eb3139bb6735e5ecfda18b7)
Signed-off-by: Martin Jansa <Martin.Jansa@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
For consistency, use override syntax for these variables as well since
it is more consistent with the rest of the image code. We may be able to use
these as proper overrides in due course.
(From OE-Core rev: 52674c4b1fdf79829095031b2e342d44fb0dc181)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The IMAGE_TYPEDEP variable would make more sense to match the form of the
other image override variables, convert it to use the overrides format.
(From OE-Core rev: 8573f6b2a7af9867da0b21936ffd2cd2a417de1d)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Add task-configure and few more supported values of TARGET_OS override.
(From OE-Core rev: 1172f9593902f28ddd8da47de6bd51cda9a0f86a)
Signed-off-by: Denys Dmytriyenko <denis@denix.org>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
This will allow the use of a bios (e.g. U-Boot) in combination with
a WIC file (from which U-Boot loads the kernel).
(From OE-Core rev: 5fc65c9d18679b62b13fead7b91126c665525884)
Signed-off-by: Matthias Klein <matthias@extraklein.de>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
It is handy to be able to have the script convert a single patch file
and it turns out to be straightforward to make it do this.
(From OE-Core rev: 21df7acc969f47d615d1701ee71f19571de94949)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The automated conversion of OE-Core to use the new override sytax isn't
perfect. This patches some mis-converted lines and some lines which were missed
by the automation.
(From OE-Core rev: 4e9a06b64b43131b731fb59a0305f78a98e27fbd)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
This is the result of automated script conversion:
scripts/contrib/convert-overrides.py <oe-core directory>
converting the metadata to use ":" as the override character instead of "_".
(From OE-Core rev: 42344347be29f0997cc2f7636d9603b1fe1875ae)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* on bigger layer this script takes couple minutes without showing any progress
* add a version to reference it in the conversion commit (to easily figure out
if it's worth re-converting the layer).
(From OE-Core rev: 39edf868deafe79ffd2ea57620af70840ec4c208)
Signed-off-by: Martin Jansa <Martin.Jansa@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
This adds a script I've developed to migrate metadata to use the new override
syntax. It is a bit rough but since its for a single use with validation, it
doesn't need to be perfect. It is run simply as:
scripts/contrib/convert-overrides.py <directory>
It is setup and has been tested to work with OE-Core, Bitbake, yocto-docs,
meta-yocto, meta-gplv2 and meta-mingw. For OE-Core, it converts around 10,100
lines with about 34 manual fixes needed.
For other layers it would need updating for override names and exclusions for
functions/variable names with "append", "prepend" or "remove" in them.
(From OE-Core rev: e8fc67f6f6baf54ccdf548b947c35dea926842c3)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
In order to be compliant with the YP compatible status, a layer also
needs to ensure that all its dependencies are compatible
too. Currently yocto-check-layer only checks the requested layer,
without testing any dependencies.
With this change, all dependencies are also checked by default, so the
summary printed at the end will give a clear picture whether all
dependencies pass the script or not.
Using --no-auto-dependency can be used to skip that.
(From OE-Core rev: 45d59b774b95c91193a8376b83c05291d555e5c8)
Signed-off-by: Nicolas Dechesne <nicolas.dechesne@linaro.org>
Signed-off-by: Alexandre Belloni <alexandre.belloni@bootlin.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
What this function does is really to find a layer, not a 'depends'. We
are using this function to find a dependent layer, but the name is
confusing.
(From OE-Core rev: e9b7690ab30d0e7c07471034f6cb89ccc3168a11)
Signed-off-by: Nicolas Dechesne <nicolas.dechesne@linaro.org>
Signed-off-by: Alexandre Belloni <alexandre.belloni@bootlin.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Split add_layer_dependencies() into 2 parts. First search for layer
dependencies, and then add them to the config. That allows us to
call get_layer_dependencies() independently.
(From OE-Core rev: 08edf928aac3f2daaa0c256d4c21e56e2db72bff)
Signed-off-by: Nicolas Dechesne <nicolas.dechesne@linaro.org>
Signed-off-by: Alexandre Belloni <alexandre.belloni@bootlin.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The first 2 calls to add_layer_dependencies() are here to add all
dependencies for the 'layer under test' and the additional layers
provided by the users.
In both cases, we use misssing_dependencies boolean to indicate if any
dependency is missing. But we then never really use
missing_dependencies. Instead the script is calling
add_layer_dependencies() again (for both the layer under test, and the
additional layers) to detect if there are any missing dependency. As a
result, we are trying to add again all dependencies, and we can see
that from the traces:
INFO: Detected layers:
INFO: meta-aws: LayerType.SOFTWARE, /work/oe/sources/meta-aws
INFO: checklayer: Doesn't have conf/layer.conf file, so ignoring
INFO:
INFO: Setting up for meta-aws(LayerType.SOFTWARE), /work/oe/sources/meta-aws
INFO: Adding layer meta-python
INFO: Adding layer meta-oe
INFO: Adding layer meta-networking
-->
INFO: Adding layer meta-python
INFO: meta-python is already in /work/oe/poky/master/build-checklayer/conf/bblayers.conf
INFO: Adding layer meta-oe
INFO: meta-oe is already in /work/oe/poky/master/build-checklayer/conf/bblayers.conf
INFO: Adding layer meta-networking
INFO: meta-networking is already in /work/oe/poky/master/build-checklayer/conf/bblayers.conf
<--
INFO: Getting initial bitbake variables ...
The code appears more complex than it should, and we can simply
replace the complex if statement by using missing_dependencies, and
avoid duplicating the call to add_layer_dependencies().
(From OE-Core rev: fceb84f7bc472731b8f96ee1ebf0f4485943226c)
Signed-off-by: Nicolas Dechesne <nicolas.dechesne@linaro.org>
Signed-off-by: Alexandre Belloni <alexandre.belloni@bootlin.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
This updates the link to the YP docs
and proposes to access the OE website through https
(From OE-Core rev: 87686233aeffc639c3f412fd5c4898b32b15013b)
Signed-off-by: Michael Opdenacker <michael.opdenacker@bootlin.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Nicolas Dechesne spotted there was duplicate code I had introduced with a
previous fix. Remove the second statement since the earlier one is correct
all that is needed.
(From OE-Core rev: f3e8d9f0e53e73de5498fccce81d049a88f6473b)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Signed-off-by: Alexandre Belloni <alexandre.belloni@bootlin.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
When S is set to a sub-directory of upstream source, the license
checks and the bbappend writing (specifically, setting EXTERNALSRC)
need to operate on that sub-directory.
'devtool modify' already has similar logic, and it was copied from there
and adjusted.
(From OE-Core rev: fd77e356d4507405fde352c8bba7d3842518bbdd)
Signed-off-by: Alexander Kanavin <alex.kanavin@gmail.com>
Signed-off-by: Alexandre Belloni <alexandre.belloni@bootlin.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Options:
-c | --count <amount> dd (transfer) <amount> KiB of data within specified timeout to detect latency.
Must enable -t option.
-t | --timeout <time> timeout in seconds for the <count> amount of data to be transferred.
-l | --log-only run the commands without performing the data transfer.
-h | --help show help
(From OE-Core rev: 302bc6c99226a4d050e4e454afc461a25e127632)
Signed-off-by: Sakib Sajal <sakib.sajal@windriver.com>
Signed-off-by: Alexandre Belloni <alexandre.belloni@bootlin.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The qemu tap device handling is potentially race ridden. We pass the
fd to the main qemu subprocess which is good as it means the lock is held
as long as the qemu process exists. This means we shouldn't unlock it
ourselves though, only close the file. We also can't delete the file
as we have no idea if qemu is still using it. We could try and obtain
an exclusive new lock, then the file would be safe to unlink but it
doesn't seem worth it.
Also fix the same issue in the port lock code.
(From OE-Core rev: 2a87bddabf816d09ec801e33972879e6983627eb)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Commit a2db4fa127 added ${WORKDIR}/deploy-* to
PSEUDO_IGNORE_PATHS. This breaks the --strip mode since ${D} is copied to
deploy-target-stripped. Use the directory devtool-deploy-target-stripped
instead.
[YOCTO #14451]
(From OE-Core rev: 02661f20faf11d0fa2f1874bd423f5d9fa7a31c9)
Signed-off-by: Florian Amstutz <florian.amstutz@scs.ch>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Previously the bootimg-pcbios wic plugin was not respecting
the --label option provided from the wks file. The plugin
was setting the label to 'boot'. With this fix, the --label
option is use. If no option are specified, then the default
is 'boot'.
(From OE-Core rev: 0fd7a73c1bd2486b7a022f0f69bbcb2e0d9cb141)
Signed-off-by: jbouchard <jeanbouch418@gmail.com>
Signed-off-by: Alexandre Belloni <alexandre.belloni@bootlin.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
There's a number of recipes (such as nettle or glib) which were upgraded
incorrectly due to rebasing only the default set of patches in main SRC_URI.
Native-only or musl-only patches were not handled, causing do_patch fails
and overall misery.
Unfortunately, this also necessitates aborting any incomplete rebases as
devtool needs to be able to return to the original branch to complete
the upgrade command.
Also, do not add devtool-patched-%version tags, as they are not used anywhere,
and it's unclear how to extend them to cover multiple override branches.
(From OE-Core rev: 00131cbc969f6a73caffe068fcb983287cbe577f)
(From OE-Core rev: 588d04748dd89b5371ca9144666e97ec82dd6204)
Signed-off-by: Alexander Kanavin <alex.kanavin@gmail.com>
Signed-off-by: Alexandre Belloni <alexandre.belloni@bootlin.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Measure and display the time it takes to copy the rootfs to its new
location (usually in tmpfs) to try to understand whether it sometimes takes
a lot of time on the autobuilders.
(From OE-Core rev: be213114c1349e02ca48fd617a6c3badae8aa101)
Signed-off-by: Alexandre Belloni <alexandre.belloni@bootlin.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Since Linux 5.4 a new read-only filesystem is available, erofs.
Compared to squashfs it offers much better read performance with and
without compression enabled.
It suppports two optional compressors, lz4 and lz4hc.
>From the mkfs.erofs man page:
EROFS is a new enhanced lightweight linux read-only filesystem with
modern designs (eg. no buffer head, reduced metadata, inline
xattrs/data, etc.) for scenarios which need high-performance read-only
requirements, e.g. Android OS for smartphones and LIVECDs.
It also provides fixed-sized output compression support, which improves
storage density, keeps relatively higher compression ratios, which is
more useful to achieve high performance for embedded devices with
limited memory since it has unnoticable memory overhead and page cache
thrashing.
This commit adds support for three new filesystem targets:
erofs: erofs without compression
erofs-lz4: erofs with lz4 compresssion enabled
erofs-lz4hc: erofs with lz4hc compression enabled
(From OE-Core rev: 41dead1ff8ccc49e6cd6e6f5d41a59d164693e0d)
Signed-off-by: Richard Weinberger <richard@nod.at>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Each of the cores is mapped to a thread on the host, this
should speed up things inside qemu which can take advantage of that.
(From OE-Core rev: 56f98d12a79585a4d4c459160f39403e05b620a5)
Signed-off-by: Alexander Kanavin <alex.kanavin@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Dynamic layers may have a conf directory, but don't need (or
want) a conf/layer.conf
This isn't an error, so we can just log it and indicate the
layer is being skipped.
A full layer without a conf file isn't all that useable, so we
aren't letting anything subtle slip through by just logging
it.
(From OE-Core rev: 9841a7c70bc21c5cd3bbea98d5f8654434577f61)
Signed-off-by: Bruce Ashfield <bruce.ashfield@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Incremental build in Docker fails with:
OSError: [Errno 18] Invalid cross-device link
when source and destination are on different overlay filesystems.
Rather than adding fallback code to every call site, use a new wrapper
in bitbake which detects this case and falls back to shutil.move
which is slower but will handtle the overlay docker filesystems correctly.
[YOCTO #14301]
(From OE-Core rev: 656a65b2b84e7d529b89cf5de7eb838f902d84a2)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
There are many Python scripts in oe-core that want to use Tinfoil, and
right now they have to know where they are to work out where BitBake is
likely to be.
This is suboptimal as BitBake could be somewhere else, so this
approach doesn't scale to other layers at all.
Solve this by adding BITBAKEDIR/lib to PYTHONPATH in oe-buildenv-internal,
so that Python has BitBake on its search path once the build system is
configured.
(From OE-Core rev: a48178f6d00e7f97a09f42d5a164204e9dcffa9f)
Signed-off-by: Ross Burton <ross.burton@arm.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
When disk stats don't run frequenctly enough, we see divide by zero
errors. The code already has a fallback path so ensure we use it
for this case too.
[YOCTO #14360]
(From OE-Core rev: b71d30aef5dc2c360432c0dd4147859dd303ea48)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Collect the last 30 lines from the cooker.log
whenever the timeout is exceeded.
(From OE-Core rev: 58f7cd4d6186525f08f3027975530d647cbfa26b)
Signed-off-by: Sakib Sajal <sakib.sajal@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
If iterating a layer with multiple components and auto-adding dependencies
the tests can break since layers are never removed and order isn't guaranteed
to account for that.
Fix this by resetting the layer list back to the original list each time
before auto-adding the dependencies in each case.
This fixes scanning of meta-openembedded in particular where the sublayers
may not be added in order of minimal dependency.
(From OE-Core rev: bf1b467dacf345379cd5d84a1c9b3b0d844d5c91)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
We need to cleanup snapshot files if we make a copy of them to ensure
the tmpfs doesn't run out of space. There is already NFS code needing
this so make it a generic code path.
(From OE-Core rev: a3e0eec5a4785a0c4859455eb10b43aa832e606d)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Particularly:
- nesting subprocess.run() inside subprocess.check_output() does not work at all.
How was this tested?
- -R and -U options can be combined; no need to separate the invocations based
on packaging format
- both exception handlers are unnecessary; we can simply print the hint if
invocation did not succeed
- to run debuginfod from its own sysroot, '-c addto_recipe_sysroot' for elfutils-native
must be executed
(From OE-Core rev: 9e57bf636ec63e74d56f1ac48b5a27c5b80f1877)
Signed-off-by: Alexander Kanavin <alex.kanavin@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
With the previous timeout of 5 seconds, there would be
builds such as:
https://autobuilder.yocto.io/pub/non-release/20210417-13/
which produced 17 files with top output with top running 454 times
and that's a bit too much data to analyze for each run. By
increasing the timeout, we'll find the worse problems
first, fix them and then we can decrease the timeout if needed.
(From OE-Core rev: 92b29a09b4c442597d212337b785afb76129ac7c)
Signed-off-by: Randy MacLeod <Randy.MacLeod@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
image-manifest: script to generate product/image specific BOM
The image-manifest script generates image specific reports based on
an image manifest file. Currently there is data generated by buildhistory,
pkgdata, and license manifest but this data is poorly formated and spread
across multiple text files. This script can generate a single JSON output
file that is machine readable by other tools.
The manifest-info collects package information and stores the information
in a tarball. manifest-info can be configured using a json configuration
file. The default configuration including all possible options can be
dumped using the dump-config subcommand.
image-manifest takes an image manifest file as input to get the runtime
dependencies. As an option image-manifest can also use the build dependency
file, pn-buildlist, to get the build dependencies excluding native
packages.
This script extends the oe-image-manifest script [0] done by Paul Eggleton
[0]
https://github.com/intel/clear-linux-dissector-web/blob/master/layerindex/static/files/oe-image-manifest
------------------------------------------------------
usage: image-manifest [-h] [-d] [-q] <subcommand> ...
Image manifest utility
options:
-h, --help show this help message and exit
-d, --debug Enable debug output
-q, --quiet Print only errors
subcommands:
recipe-info Get recipe info
list-depends List dependencies
list-recipes List recipes producing packages within an image
list-packages List packages within an image
list-layers List included layers
dump-config Dump default config
manifest-info Export recipe info for a manifest
Use image-manifest <subcommand> --help to get help on a specific command
Co-developed-by: Paul Eggleton <bluelightning@bluelightning.org>
(From OE-Core rev: ad8fec9ce1704866df925bda18a240d6889b1ed5)
Signed-off-by: Anders Wallin <anders.wallin@windriver.com>
Signed-off-by: Saul Wold <saul.wold@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Improvements:
- increase width to 512
- pass -c option to show full command-line
(From OE-Core rev: aeae9467af5609c3c7bf8d0379d5546d9797ead5)
Signed-off-by: Sakib Sajal <sakib.sajal@windriver.com>
Signed-off-by: Randy MacLeod <Randy.MacLeod@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
We have a working theory that IO queues on the autobuilder are impacting
runtime testing under qemu, particularly async writes which inice does not
influence. We already pass the snapshot option to qemu which copies the
image and runs out of the copy. Add in the ability to copy the image to
a specificed location which can be a tmpfs. This means that writes to the
image would no longer be blocked by other writes to disk in the system.
Preliminary tests show that this does improve the qemu errors at the expense
of sometimes showing qemu startup timeouts as on a loaded system with a large
test image, it can take longer than 120s to copy the image to tmpfs. Having
a most consistent failure mode for loaded tests is probably desireable though.
(From OE-Core rev: fd1c26ab426c3699ffd8082b83d65a84c8eb8bff)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
