Commit Graph

3289 Commits

Author SHA1 Message Date
Alexandre Belloni
998d23a2c6 runqemu: time the copy to tmpfs
Measure and display the time it takes to copy the rootfs to its new
location (usually in tmpfs) to try to understand whether it sometimes takes
a lot of time on the autobuilders.

(From OE-Core rev: be213114c1349e02ca48fd617a6c3badae8aa101)

Signed-off-by: Alexandre Belloni <alexandre.belloni@bootlin.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2021-06-06 23:16:50 +01:00
Richard Weinberger
63b3c44d27 Add support for erofs filesystems
Since Linux 5.4 a new read-only filesystem is available, erofs.
Compared to squashfs it offers much better read performance with and
without compression enabled.
It suppports two optional compressors, lz4 and lz4hc.

>From the mkfs.erofs man page:
EROFS is a new enhanced lightweight linux read-only filesystem with
modern designs (eg. no buffer head, reduced metadata, inline
xattrs/data, etc.) for scenarios which need high-performance read-only
requirements, e.g. Android OS for smartphones and LIVECDs.
It also provides fixed-sized output compression support, which improves
storage density, keeps relatively higher compression ratios, which is
more useful to achieve high performance for embedded devices with
limited  memory since it has unnoticable memory overhead and page cache
thrashing.

This commit adds support for three new filesystem targets:
erofs: erofs without compression
erofs-lz4: erofs with lz4 compresssion enabled
erofs-lz4hc: erofs with lz4hc compression enabled

(From OE-Core rev: 41dead1ff8ccc49e6cd6e6f5d41a59d164693e0d)

Signed-off-by: Richard Weinberger <richard@nod.at>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2021-05-31 14:31:01 +01:00
Alexander Kanavin
1584108bdc qemu: use 4 cores in qemu guests
Each of the cores is mapped to a thread on the host, this
should speed up things inside qemu which can take advantage of that.

(From OE-Core rev: 56f98d12a79585a4d4c459160f39403e05b620a5)

Signed-off-by: Alexander Kanavin <alex.kanavin@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2021-05-13 18:09:10 +01:00
Bruce Ashfield
2dcd1f2a21 yocto-check-layer: Only note a layer without a conf/layer.conf (versus error)
Dynamic layers may have a conf directory, but don't need (or
want) a conf/layer.conf

This isn't an error, so we can just log it and indicate the
layer is being skipped.

A full layer without a conf file isn't all that useable, so we
aren't letting anything subtle slip through by just logging
it.

(From OE-Core rev: 9841a7c70bc21c5cd3bbea98d5f8654434577f61)

Signed-off-by: Bruce Ashfield <bruce.ashfield@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2021-05-06 16:55:35 +01:00
Devendra Tewari
b71375304f classes/lib/scripts: Use bb.utils.rename() instead of os.rename()
Incremental build in Docker fails with:

OSError: [Errno 18] Invalid cross-device link

when source and destination are on different overlay filesystems.

Rather than adding fallback code to every call site, use a new wrapper
in bitbake which detects this case and falls back to shutil.move
which is slower but will handtle the overlay docker filesystems correctly.

[YOCTO #14301]

(From OE-Core rev: 656a65b2b84e7d529b89cf5de7eb838f902d84a2)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2021-05-06 11:16:23 +01:00
Ross Burton
e3887b0690 oe-buildenv-internal: add BitBake's library to PYTHONPATH
There are many Python scripts in oe-core that want to use Tinfoil, and
right now they have to know where they are to work out where BitBake is
likely to be.

This is suboptimal as BitBake could be somewhere else, so this
approach doesn't scale to other layers at all.

Solve this by adding BITBAKEDIR/lib to PYTHONPATH in oe-buildenv-internal,
so that Python has BitBake on its search path once the build system is
configured.

(From OE-Core rev: a48178f6d00e7f97a09f42d5a164204e9dcffa9f)

Signed-off-by: Ross Burton <ross.burton@arm.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2021-05-06 08:41:26 +01:00
Richard Purdie
86a66606fe pybootchart/draw: Avoid divide by zero error
When disk stats don't run frequenctly enough, we see divide by zero
errors. The code already has a fallback path so ensure we use it
for this case too.

[YOCTO #14360]

(From OE-Core rev: b71d30aef5dc2c360432c0dd4147859dd303ea48)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2021-05-02 23:21:42 +01:00
Sakib Sajal
f8b2e0f600 oe-time-dd-test.sh: collect cooker log when timeout is exceeded
Collect the last 30 lines from the cooker.log
whenever the timeout is exceeded.

(From OE-Core rev: 58f7cd4d6186525f08f3027975530d647cbfa26b)

Signed-off-by: Sakib Sajal <sakib.sajal@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2021-04-27 15:11:47 +01:00
Richard Purdie
bf59f39653 yocto-check-layer: Avoid bug when iterating and autoadding dependencies
If iterating a layer with multiple components and auto-adding dependencies
the tests can break since layers are never removed and order isn't guaranteed
to account for that.

Fix this by resetting the layer list back to the original list each time
before auto-adding the dependencies in each case.

This fixes scanning of meta-openembedded in particular where the sublayers
may not be added in order of minimal dependency.

(From OE-Core rev: bf1b467dacf345379cd5d84a1c9b3b0d844d5c91)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2021-04-24 17:53:05 +01:00
Richard Purdie
d5eb86b3aa runqemu: Ensure we cleanup snapshot files after image run
We need to cleanup snapshot files if we make a copy of them to ensure
the tmpfs doesn't run out of space. There is already NFS code needing
this so make it a generic code path.

(From OE-Core rev: a3e0eec5a4785a0c4859455eb10b43aa832e606d)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2021-04-23 00:08:38 +01:00
Alexander Kanavin
5aa20fe1b9 scripts/oe-debuginfod: correct several issues
Particularly:
- nesting subprocess.run() inside subprocess.check_output() does not work at all.
How was this tested?
- -R and -U options can be combined; no need to separate the invocations based
on packaging format
- both exception handlers are unnecessary; we can simply print the hint if
invocation did not succeed
- to run debuginfod from its own sysroot, '-c addto_recipe_sysroot' for elfutils-native
must be executed

(From OE-Core rev: 9e57bf636ec63e74d56f1ac48b5a27c5b80f1877)

Signed-off-by: Alexander Kanavin <alex.kanavin@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2021-04-20 13:56:48 +01:00
Randy MacLeod
193251b8d0 oe-time-dd-test.sh: increase timeout to 15 sec
With the previous timeout of 5 seconds, there would be
builds such as:
   https://autobuilder.yocto.io/pub/non-release/20210417-13/
which produced 17 files with top output with top running 454 times
and that's a bit too much data to analyze for each run. By
increasing the timeout, we'll find the worse problems
first, fix them and then we can decrease the timeout if needed.

(From OE-Core rev: 92b29a09b4c442597d212337b785afb76129ac7c)

Signed-off-by: Randy MacLeod <Randy.MacLeod@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2021-04-20 13:56:48 +01:00
Anders Wallin
9e0f210179 scripts/contrib/image-manifest: add new script
image-manifest: script to generate product/image specific BOM

The image-manifest script generates image specific reports based on
an image manifest file. Currently there is data generated by buildhistory,
pkgdata, and license manifest but this data is poorly formated and spread
across multiple text files. This script can generate a single JSON output
file that is machine readable by other tools.

The manifest-info collects package information and stores the information
in a tarball. manifest-info can be configured using a json configuration
file. The default configuration including all possible options can be
dumped using the dump-config subcommand.

image-manifest takes an image manifest file as input to get the runtime
dependencies. As an option image-manifest can also use the build dependency
file, pn-buildlist, to get the build dependencies excluding native
packages.

This script extends the oe-image-manifest script [0] done by Paul Eggleton

[0]
https://github.com/intel/clear-linux-dissector-web/blob/master/layerindex/static/files/oe-image-manifest

------------------------------------------------------
usage: image-manifest [-h] [-d] [-q] <subcommand> ...

Image manifest utility

options:
  -h, --help     show this help message and exit
  -d, --debug    Enable debug output
  -q, --quiet    Print only errors

subcommands:
  recipe-info    Get recipe info
  list-depends   List dependencies
  list-recipes   List recipes producing packages within an image
  list-packages  List packages within an image
  list-layers    List included layers
  dump-config    Dump default config
  manifest-info  Export recipe info for a manifest
Use image-manifest <subcommand> --help to get help on a specific command

Co-developed-by: Paul Eggleton <bluelightning@bluelightning.org>
(From OE-Core rev: ad8fec9ce1704866df925bda18a240d6889b1ed5)

Signed-off-by: Anders Wallin <anders.wallin@windriver.com>
Signed-off-by: Saul Wold <saul.wold@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2021-04-18 11:37:25 +01:00
Sakib Sajal
1cb269448b oe-time-dd-test.sh: provide more information from "top"
Improvements:
    - increase width to 512
    - pass -c option to show full command-line

(From OE-Core rev: aeae9467af5609c3c7bf8d0379d5546d9797ead5)

Signed-off-by: Sakib Sajal <sakib.sajal@windriver.com>
Signed-off-by: Randy MacLeod <Randy.MacLeod@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2021-04-16 09:24:48 +01:00
Sakib Sajal
6e8f334c77 oe-time-dd-test.sh: make executable
(From OE-Core rev: d58d5ce00a997646fc7b691e6fd23ebd7f84e3ab)

Signed-off-by: Sakib Sajal <sakib.sajal@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2021-04-16 09:24:48 +01:00
Richard Purdie
232cb7b055 oeqa/runqemu: Support RUNQEMU_TMPFS_DIR as a location to copy snapshot images to
We have a working theory that IO queues on the autobuilder are impacting
runtime testing under qemu, particularly async writes which inice does not
influence. We already pass the snapshot option to qemu which copies the
image and runs out of the copy. Add in the ability to copy the image to
a specificed location which can be a tmpfs. This means that writes to the
image would no longer be blocked by other writes to disk in the system.

Preliminary tests show that this does improve the qemu errors at the expense
of sometimes showing qemu startup timeouts as on a loaded system with a large
test image, it can take longer than 120s to copy the image to tmpfs. Having
a most consistent failure mode for loaded tests is probably desireable though.

(From OE-Core rev: fd1c26ab426c3699ffd8082b83d65a84c8eb8bff)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2021-04-06 09:30:10 +01:00
Alexander Kanavin
379ee98034 runqemu: do not stop processing graphical options after nographic
Some options such as egl-headless are fully compatible with it, so
there is no need to quit.

(From OE-Core rev: 66d11106f9e76d19e397ba3d14c3a22726033567)

Signed-off-by: Alexander Kanavin <alex.kanavin@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2021-04-05 15:29:01 +01:00
Khem Raj
a7e1bbaf6d documentation-audit.sh: Fix typo in specifying LICENSE_FLAGS_WHITELIST
(From OE-Core rev: 410a45639d84a3d69a65133593da32062196dd59)

Signed-off-by: Khem Raj <raj.khem@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2021-03-26 11:34:33 +00:00
Sakib Sajal
4b5a93d190 scripts: add oe-time-dd-test.sh
oe-time-dd-test records how much time it takes to
write <count> number of kilobytes to the filesystem.
It also records the number of processes that are in
running (R), uninterruptible sleep (D) and interruptible
sleep (S) state from the output of "top" command.
The purporse of this script is to find which part of
the build system puts stress on the filesystem io and
log all the processes.

(From OE-Core rev: 26d82f92e5ed2ebdde158d66b86eaf248f56892a)

Signed-off-by: Sakib Sajal <sakib.sajal@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2021-03-23 22:51:25 +00:00
Petr Vorel
596ccd7cb8 scripts/verify-bashisms: Update checkbashisms.pl URL
anonscm.debian.org is not used any more, update the URL.

(From OE-Core rev: 3c89071a2c19a522c1df5919f500b88339ec6294)

Signed-off-by: Petr Vorel <petr.vorel@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2021-03-20 18:54:56 +00:00
Richard Purdie
e9aac299c7 scripts/contrib/list-packageconfig-flags: Upate for tinfoil API changes
Update after tinfoil API changes in bitake for REQUIRED_VERSION.

(From OE-Core rev: 1645c9b093bebf7ced67cbee0009d81d1a215966)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2021-03-15 14:49:09 +00:00
Robert P. J. Day
97edca5997 bitbake-whatchanged: change ending quote to proper period
Pretty sure that trailing quote should be a period; it
appears to work properly.

(From OE-Core rev: 579f9ada19bd174bff0002cd6a731d12a1868252)

Signed-off-by: Robert P. J. Day <rpjday@crashcourse.ca>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2021-03-14 16:33:59 +00:00
Alejandro Hernandez Samaniego
6f3b5a8e24 devtool: Fix do_kernel_configme task
The do_kernel_configme task is no longer part of SRCTREECOVEREDTASKS,
its been removed from the kernel-yocto.bbclass since b72dbb2e4, but
there wasnt a matching patch for devtool for those changes.

This patch enables us to invoke the do_kernel_configme task when
using a devtool workspace, it also prepends a check for an existing
.config file in the source directory and moves it if thats the case,
since when using devtool modify a .config is created and
do_kernel_configme complains about it, this is not the case when
using bitbake since the .config file would be on B instead.

Alowing do_kernel_configme to run also fixes the flow where testing a
new config fragment from devtool workspace isnt added properly
(config queue shows it as ///frg.cfg) and as a side effect it never
gets merged into the final config.

(From OE-Core rev: 08dcc0e68095dcf2a159546a48b29d40c9aabc0b)

Signed-off-by: Alejandro Enedino Hernandez Samaniego <alhe@linux.microsoft.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2021-03-14 16:33:59 +00:00
Jon Mason
fae414bb70 runqemu: modify novga to conform to documentation
Passing 'novga' only adds '-vga none' to the qemu commandline and does
not prevent other vga devices from being added, contrary to the
documentation/help.  Clean-up the vga logic and add the ability to
prevent other vga devices from being added.

(From OE-Core rev: 9f8d049dbbe0b1760979d9f3b745124abfc54c90)

Signed-off-by: Jon Mason <jon.mason@arm.com>
Change-Id: I7ff8f9f3e419ce8bae3f3847d75f9275ca30406a
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2021-03-11 14:00:36 +00:00
Jon Mason
16d1669ff9 runqemu: correct forcing of ttyS0
Some platforms do not use ttyS* for their serial consoles (e.g., qemuarm
and qemuarm64).  The hardcoding of this can cause issues.  Modify
runqemu to use the serial consoles defined in SERIAL_CONSOLES instead of
hardcoding.

(From OE-Core rev: 9dea4cd2f9f46ab3a75562639a22d8f56b4d26af)

Signed-off-by: Jon Mason <jon.mason@arm.com>
Change-Id: I746d56de5669c955c5e29d3ded70c0a4d3171f17
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2021-03-11 14:00:36 +00:00
Jon Mason
bd0ada8994 runqemu: add QB_GRAPHICS
Add a new runqemu field for VGA devices.  Currently, these are being set
in QB_OPT_APPEND, which can make them difficult to override if importing
the config file into another one.

(From OE-Core rev: 695c98b6522be4373806c154a2999eaeef205556)

Signed-off-by: Jon Mason <jon.mason@arm.com>
Change-Id: I8cb9527954c5b06c083c42fe2466cb3338584b7d
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2021-03-11 14:00:36 +00:00
Diego Santa Cruz
aafa7a3f3a sstate-diff-machines.sh: support rpm, deb and tar package types in analysis
The sstate-diff-machines.sh script's --analyze option was only supporting
ipk packages, this adds rpm, deb and tar packages, checking all the used
package types simultaneously.

Without this changing sstate between machines would not report an error
if the difference occured in packages and ipk packaging was not enabled.

(From OE-Core rev: b27bff26a9640dfdfba4fc711a9b28acb4501d2b)

Signed-off-by: Diego Santa Cruz <Diego.SantaCruz@spinetix.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2021-03-10 10:30:39 +00:00
Jon Mason
ca1382756c runqemu: use "raw" instead of "bin" for ovmf
bin is not an acceptable format for qemu and will throw an error is set.
The correct format in this case is raw.

(From OE-Core rev: cd279aeb986b4676ea8ecb1e7b12c7e29e83460a)

Signed-off-by: Jon Mason <jon.mason@arm.com>
Change-Id: I9d78e356d39ecb15bef886c19870b029d9badd65
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2021-03-06 22:39:03 +00:00
Dorinda
5fc2c2ea51 scripts/oe-debuginfod: script that fetches package manager directory
Added a script that fetches the package manager directory and runs the elfutils-native debuginfod on DEPLOY_DIR
Added a check to ensure that PACKAGECONFIG options is set in local.conf

(From OE-Core rev: 410083f93e2c986247cf3a2ff5050847e10cf359)

Signed-off-by: Dorinda Bassey <dorindabassey@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2021-03-02 20:39:35 +00:00
Alexander Kanavin
d53dddd7ca scripts/lib/wic/partition.py: do not set FAT size
Modern dosfstools automatically determines the appropriate size
and will error out if something that doesn't make sense is
supplied on the command line.

(From OE-Core rev: b85a09ea450a5e8f49418f4a930805fbb88dc83b)

Signed-off-by: Alexander Kanavin <alex.kanavin@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2021-03-02 20:39:35 +00:00
Florian Bezdeka
a334cbb12d wic: Warn if an ext filesystem affected by the Y2038 problem is used
We are getting closer and closer to the year 2038 where the 32 bit
time_t overflow will happen. While products (= embedded systems) with an
expected life time of 15 years are still save the situation may change
if your system has to survive the next 20 years.

ext2 and ext3 filesystems are always affected by the time overflow, so
let's warn the user if these filesystems are still being used.

If ext4 is affected depends on the inode size chosen during filesystem
creation. At least 256 bytes are necessary to be safe. As ext4 is
used very often (and partitions may be created small first and extended
later) this might be an issue for many users.

Some filesystems created during CI runs were already affected by the Y2038
problem. By using `--mkfs-extraopts "-T default"` we tell mke2fs not to
auto-detect the usage type based on the filesystem size. mke2fs will use
the default values for tuning parameters instead. The inode size is one
of these parameters.

(From OE-Core rev: eecbe625558406680121d2a7e84917fea45ea9dc)

Signed-off-by: Florian Bezdeka <florian.bezdeka@siemens.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2021-02-26 15:21:21 +00:00
Khem Raj
acfa0e0dce runqemu: Add new option to disable vga emulation
When using nographic, explicitly disabling vga is needed some cases
since some qemu firmware (OpenFirmware ) defaults to std vga, and
when vga is enabled then it disables output to serial and redirects
that to vga which is by design, hwoever we expect the console output
to go to serial when using nographic, therefore its important to
disable vga with nographic especially on qemu based ppc platforms

(From OE-Core rev: eabbdb3c4f78ab23348a6b988f59d9d98b5e2c7c)

Signed-off-by: Khem Raj <raj.khem@gmail.com>
Cc: Alexander Kanavin <alex.kanavin@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2021-02-23 22:35:00 +00:00
Dorinda
9294bc4bb4 oe-pkgdata-util: Check if environment script is initialized
Tinfoil doesn't behave well if environment is not initialized, this check ensures a proper error log if environment is not initialized.

[YOCTO #12096]

(From OE-Core rev: e88073e16f1b4cfd0f97c81a988640a84adad674)

Signed-off-by: Dorinda Bassey <dorindabassey@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2021-02-10 23:48:55 +00:00
Lee Chee Yang
34ea1433fc wic: debug mode to keep tmp directory
files in wic tmp directory can be usefull for debugging, so do not remove
tmp directory when wic create run with debugging mode (-D or --debug).

also update wic.Wic.test_debug_short and wic.Wic.test_debug_long to
check for tmp directory.

[YOCTO#14216]

(From OE-Core rev: a122e2418b67d38f691edcf8dd846c167d6b4fa9)

Signed-off-by: Lee Chee Yang <Chee.Yang.Lee@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2021-02-09 08:56:11 +00:00
Tomasz Dziendzielski
3a220f1e41 devtool: Fix file:// fetcher symlink directory structure
Add relative path if file is under directory.

[YOCTO #13738]

(From OE-Core rev: 19ddacc1b38f9ebb86a9359963ccc3c707f7125e)

Signed-off-by: Tomasz Dziendzielski <tomasz.dziendzielski@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2021-01-23 17:08:54 +00:00
Tomasz Dziendzielski
0ad1537642 devtool: Fix source extraction for gcc shared source
If do_patch task is disabled then prepare do_configure dependencies to
fetch external sources and create symlink to ${S} in devtool workspace.

[YOCTO #13036]

(From OE-Core rev: 9e47319bfe62d289b90f7545a64dbdc1cbde7f1d)

Signed-off-by: Tomasz Dziendzielski <tomasz.dziendzielski@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2021-01-20 22:46:18 +00:00
Paul Barker
da846838fc wic: Optimise fstab modification for ext2/3/4 and msdos partitions
The fix for [Yocto #13994] required the rootfs directory to be copied
(using hardlinks if possible) when modifying the fstab file under wic.

We can optimise this copy away for filesystems where we have the tools
to modify the contents of the partition image after it is created. For
ext2/3/4 filesystems we have the debugfs tool and for msdos/vfat
filesystems we have the mcopy tool. So for any of these filesystems we
skip the modification of the fstab file in the rootfs directory (and
skip the associated copy unless it is otherwise necessary) and update
the contents of fstab directly in the partition image.

(From OE-Core rev: 5fb8ae0e9159597d7eaa9307a3a8543800bf9405)

Signed-off-by: Paul Barker <pbarker@konsulko.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2021-01-20 22:46:18 +00:00
Paul Barker
36575a9493 wic: Copy rootfs dir if fstab needs updating
By default, wic updates the /etc/fstab in the rootfs to include details
of additional partitions described in the selected wks file. If this
modification is performed in place, other tasks which create an image
file from the rootfs directory (e.g. do_image_tar and do_image_ext4)
will pick up the modified fstab file which would not be appropriate for
those images as they do not include the additional partitions described
in the wks file. wic does undo modifications to the fstab file once it
has finished creating the filesystem image, however this leaves open a
race condition if one of the other tasks reads the contents of the fstab
file from the rootfs directory between the point where wic modifies the
fstab file and the point where wic restores the files original content.

This could be solved by adding a lockfile for tasks which use the rootfs
directory to ensure that no other such task is reading the rootfs
directory while do_image_wic is running. This would serialize several
do_image_* tasks and result in slower builds, especially for large
images. Another drawback of this solution is that it is hard to
selectively optimise - adding lockfiles to do_image_* tasks would result
in these tasks always being serialized even if no fstab modification
will take place.

An alternative solution is to copy the rootfs directory when fstab needs
to be modified. The code to do this in wic already exists as it is
needed when including or excluding content in the rootfs. This still
results in an impact on build times but the copy uses hardlinks if
possible (so little data is actually copied) and we can make selective
optimisations to improve things. The rootfs copy will only take place if
fstab modification is required (or if it was already needed to include
or exclude rootfs content). We can also follow up with further
optimisations after this commit. So this second solution is chosen.

Fixes [Yocto #13994]

(From OE-Core rev: ce682a73b7447652f898ce1d1d0416a456df5416)

Signed-off-by: Paul Barker <pbarker@konsulko.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2021-01-20 22:46:18 +00:00
Paul Barker
f85a4a1462 wic: Update pseudo db when excluding content from rootfs
To exclude content from the rootfs, wic makes a copy (using hardlinks if
possible) of the rootfs directory and associated pseudo db, then removes
files & directories as needed. However if these files and directories
are removed using the python functions os.remove and shutil.rmtree, the
copied pseudo db will not be updated correctly. For files copied from
the original rootfs, if hardlinks were used successfully when copying
the rootfs this should mean that the relevant inodes can't be reused and
so the risk of pseudo aborts should be avoided. However, this logic
doesn't apply for directories (as they can't be hardlinked) or for files
added via the '--include-path' argument (as they weren't present in the
original rootfs) and so there remains some risk of inodes being reused
and the pseudo db becoming corrupted.

To fix this, use the 'rm' command under pseudo when removing files &
directories from the copied rootfs to ensure that the copied pseudo db
is updated.

(From OE-Core rev: d5db7e268947f0392c2126137571a44acd29ccd6)

Signed-off-by: Paul Barker <pbarker@konsulko.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2021-01-20 22:46:18 +00:00
Paul Barker
4e6cb084f5 wic: Ensure internal workdir is not reused
If a path is specified for the internal wic working directory using
the -w/--workdir argument then it must not already exist. Re-using a
previous workdir could easily result in rootfs and intermediate files
from a previous build being added to the current image.

(From OE-Core rev: 2e40c8d4109024ff704c5ce40d98050ca7f34dd5)

Signed-off-by: Paul Barker <pbarker@konsulko.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2021-01-20 22:46:18 +00:00
Sinan Kaya
24cd090741 runqemu: Add support for VHD/VHDX rootfs
(From OE-Core rev: fafa2fe6d4d40ab6ba7d39b397d2bde9eb0d6b72)

Signed-off-by: Sinan Kaya <okaya@kernel.org>
Signed-off-by: Anatol Belski <anbelski@linux.microsoft.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2021-01-20 00:45:05 +00:00
Kamel Bouhara
93543e9960 recipetool: create: only add npmsw url if required
Before adding a npmsw fetcher to a recipe we
should first check if the generated shrinkwrap file
contains dependencies.

(From OE-Core rev: ef153ad36d0299e83a03af8f207686d0d8a238b3)

Signed-off-by: Kamel Bouhara <kamel.bouhara@bootlin.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2021-01-16 22:39:36 +00:00
Adrian Herrera
946e62145e scripts: oe-run-native, fix *-native directories
This fixes a crash with "find" when running a native tool and *-native
directories do not exist under the binary directory in the sysroot.
This happened because the directory wildcard was passed as part of the
root directory.
The directory wildcard is now passed by "-name", which returns an empty
result if no matching directory.

(From OE-Core rev: f6c90ed0ad24b7d4f892e22e088b1578824eb1d3)

Signed-off-by: Adrian Herrera <adrian.herrera@arm.com>
Change-Id: Iba7acd8bbd7e0beb4d25c984f6af7a4fd21486e6
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2021-01-15 13:39:25 +00:00
Paul Barker
cf73a72632 wic: Allow exec_native_cmd to run HOSTTOOLS
This allows programs from HOSTTOOLS (e.g. 'install', 'rm', 'mv', etc) to
be more easily executed by wic. Without this change only programs from
an actual *-native recipe built by bitbake can be executed by wic.

(From OE-Core rev: 8eb186acdecfbb3151c9a0ab148358e3fe5cce39)

Signed-off-by: Paul Barker <pbarker@konsulko.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2021-01-13 23:13:45 +00:00
Paul Barker
1b816d180b wic: Add workdir argument
This allows the path for the temporary workdir used by wic to be set
when running wic from bitbake or directly from the command line.

(From OE-Core rev: 2e7314ac3a3cab89de93c932e8efc75de0feb0a7)

Signed-off-by: Paul Barker <pbarker@konsulko.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2021-01-13 23:13:45 +00:00
Lee Chee Yang
3fa6a47aeb wic/direct/kparser: ensure fsuuid for vfat and msdos align with format
vfat/msdos filesystem should have fsuuid in format 0xYYYYYYYY where "0x"
in front follow with 8 hexadecimal number in uppercase. In wic, when using
custom fsuuid for vfat/msdos partition in wks, it is able to set the value
in any length, with or without leading "0x". This can cause fsuuid
missaligned when fstab updates, fstab expect exactly 10 character
fsuuid for vfat/msdos partition and all in uppercase.

if custom fsuuid for vfat/msdos is set, check the length and format,
error if it exceed the format size. Amend it so it is align with format
0xYYYYYYYY. This is done before image create and fstab update to ensure the
fsuuid are same in all followup process. if custom fsuuid length less than
expected, fill in "0".

[YOCTO #14161]

(From OE-Core rev: d9686ae511ef10a504becfd81bfe296b788e1456)

Signed-off-by: Lee Chee Yang <chee.yang.lee@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2021-01-01 23:15:13 +00:00
Alexander Kanavin
f1efa4792f devtool: gitsm:// should be handled same as git:// in upgrades
(From OE-Core rev: e7076f1742bb83e76aa4e90aad861546536166cf)

Signed-off-by: Alexander Kanavin <alex.kanavin@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2020-12-31 11:53:26 +00:00
Peter Kjellerstedt
e12072c7e0 wic: Pass canonicalized paths in PSEUDO_IGNORE_PATHS
(From OE-Core rev: 55618cbd58d6784a82e773f323723be6f722232f)

Signed-off-by: Peter Kjellerstedt <peter.kjellerstedt@axis.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2020-12-20 00:03:04 +00:00
Diego Sueiro
75b74f6b8d wic: Introduce empty plugin to create unformatted empty partitions
The empty wic plugin is used to create unformatted empty partitions for wic
images.

To use it you must pass "empty" as argument for the "--source" parameter in
the wks file. For example:
    part foo --source empty --ondisk sda --size="1024" --align 1024

Also adds a selftest for this plugin where the 'Fstype' column from 'wic
ls' should be empty for the second partition as listed in
test_empty_plugin.wks.

(From OE-Core rev: 77d174fc80663403ef76c5b808aafc1117d3545c)

Signed-off-by: Diego Sueiro <diego.sueiro@arm.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2020-12-15 22:51:56 +00:00
Chaitanya Vadrevu
ea5880f9f9 isoimage-isohybrid.py: Support adding files/dirs
Add "payload_dir" parameter so user can add other files/dirs
to the ISO.

(From OE-Core rev: bbe84a33c81066fee96dd28a4009d8404b51e642)

Signed-off-by: Chaitanya Vadrevu <chaitanya.vadrevu@ni.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2020-11-08 14:03:20 +00:00