Commit Graph

1531 Commits

Author SHA1 Message Date
Stefan Herbrechtsmeier
d8655d3e15 recipetool: Change default paramter fallback_licenses of function split_pkg_licenses from None to []
(From OE-Core rev: ab06305bf9a8ee2250aae8043e37119df2fbfc15)

Signed-off-by: Stefan Herbrechtsmeier <stefan.herbrechtsmeier@weidmueller.com>
Signed-off-by: Alexandre Belloni <alexandre.belloni@bootlin.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2021-10-14 11:48:45 +01:00
Stefan Herbrechtsmeier
f70715b54a recipetool: Skip common source files in guess_license
(From OE-Core rev: a930d39787dd77e10dfa7b7297af5fa04ca731ea)

Signed-off-by: Stefan Herbrechtsmeier <stefan.herbrechtsmeier@weidmueller.com>
Signed-off-by: Alexandre Belloni <alexandre.belloni@bootlin.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2021-10-14 11:48:45 +01:00
Stefan Herbrechtsmeier
17798d21e0 recipetool: Move license md5sums into CSV files
(From OE-Core rev: 631f6b7d1856e0e4f2313703fa1162d05e5deea6)

Signed-off-by: Stefan Herbrechtsmeier <stefan.herbrechtsmeier@weidmueller.com>
Signed-off-by: Alexandre Belloni <alexandre.belloni@bootlin.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2021-10-14 11:48:45 +01:00
Kristian Klausen
b8c0f073f6 wic/bootimg-efi: Add Unified Kernel Image option
"A unified kernel image is a single EFI PE executable combining an EFI
stub loader, a kernel image, an initramfs image, and the kernel command
line.

[...]

Images of this type have the advantage that all metadata and payload
that makes up the boot entry is monopolized in a single PE file that can
be signed cryptographically as one for the purpose of EFI
SecureBoot."[1]

This commit adds a create-unified-kernel-image=true option to the
bootimg-efi plugin for creating a Unified Kernel Image[1] and installing
it into $BOOT/EFI/Linux/ with a .efi extension per the the Boot Loader
Specification[1][2]. This is useful for implementing Secure Boot.

systemd-boot is the only mainstream bootloader implementing the
specification, but GRUB should be able to boot the EFI binary, this
commit however doesn't implement the necessary changes to the GRUB
config generation logic to boot the Unified Kernel Image.

[1] https://systemd.io/BOOT_LOADER_SPECIFICATION/#type-2-efi-unified-kernel-images
[2] https://systemd.io/BOOT_LOADER_SPECIFICATION/

(From OE-Core rev: b0573f240525df561ddef6e47cb285b217d38487)

Signed-off-by: Kristian Klausen <kristian@klausen.dk>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2021-10-04 15:03:53 +01:00
Chandana kalluri
327ea9d446 scriptutils.py: Add check before deleting path
Add a check before deleting path when using recipetool commands to avoid the following type of errors:

Traceback (most recent call last):
   File "<workdir>/sources/core/scripts/lib/scriptutils.py", line 218, in fetch_url
    shutil.rmtree(path)
  File "/usr/local/lib/python3.7/shutil.py", line 476, in rmtree
    onerror(os.lstat, path, sys.exc_info())
  File "/usr/local/lib/python3.7/shutil.py", line 474, in rmtree
    orig_st = os.lstat(path)
FileNotFoundError: [Errno 2] No such file or directory: '<workdir>/build/tmp/work/recipetool-usg7o81n/work/recipe-sysroot'
ERROR: Command 'script -e -q -c "recipetool --color=always create --devtool -o /tmp/devtool5sq_op37 'file:///<SRCTREE>'  -x <workdir>/build/workspace/sources/devtoolsrcxc1b9zjq -N test" /dev/null' failed

(From OE-Core rev: b6aa8b47e023004ffd6958d1cec18c2d9c95d77b)

Signed-off-by: Sai Hari Chandana Kalluri <chandana.kalluri@xilinx.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2021-10-01 14:51:45 +01:00
Markus Volk
37b4fa7a10 wic:direct.py: ignore invalid mountpoints during fstab update
wic fstab-update creates invalid entries for partitons that are not supposed to
be mounted from userspace eg u-boot partitions.

The following lines were added to fstab on a rock-pi-4:
/dev/mmcblk1p1 loader1 vfat defaults 0 0
/dev/mmcblk1p2 reserved1 vfat defaults 0 0
/dev/mmcblk1p3 reserved2 vfat defaults 0 0
/dev/mmcblk1p4 loader2 vfat defaults 0 0
/dev/mmcblk1p5 atf vfat defaults 0 0
/dev/mmcblk1p6 /boot vfat defaults 0 0

With this patch only valid entries should be added
/dev/mmcblk1p6 /boot vfat defaults 0 0

(From OE-Core rev: 7aa678ce804c21dc1dc51b9be442671bc33c4041)

Signed-off-by: MarkusVolk <f_l_k@t-online.de>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2021-09-26 16:35:00 +01:00
Richard Purdie
d19571cefc meta: Drop useless class
This class was added by me back in 2007 and has changed one for a whitespace issue
since. It only has two lines and neither are particularly useful, one was replaced
by the nopackages class, the other adding recursive dependencies also is now
mainly problematic adding tons of unneeded dependencies. The name is hard to
understand and the class doesn't have a clear purpose. Drop it.

Remove the references in devtool (which may be the one reason to keep it around
but in my view still not worth it).

(From OE-Core rev: 6e2cbfc561dac89bf9183d24d90e52f7d9117826)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2021-09-23 13:49:11 +01:00
Alexander Kanavin
2697d3984f wic: keep rootfs_size as integer
The corrected line accidentally converted it to float,
which causes problems later on with python 3.10:

|   File "/home/alex/development/poky/scripts/lib/wic/partition.py", line 278, in prepare_rootfs_ext
|     os.ftruncate(sparse.fileno(), rootfs_size * 1024)
| TypeError: 'float' object cannot be interpreted as an integer

(From OE-Core rev: d1d260dd2d196d10379ed9e238bcb34f39f3a3b7)

Signed-off-by: Alexander Kanavin <alex@linutronix.de>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2021-09-19 11:33:14 +01:00
Alexander Kanavin
5128defbc3 meta: correct collections vs collections.abc deprecation
collections.abc is available since 3.3 and doesn't need special handling.

(From OE-Core rev: 01152c9410ba00274c8415a5d914dc33dfd0cf46)

Signed-off-by: Alexander Kanavin <alex@linutronix.de>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2021-09-19 11:33:14 +01:00
Kristian Klausen
73384ac936 wic: Add extra-space argument
This allows extra space to be added after the last partition and is
especially useful when free space is needed for ex: adding partitions on
first boot with ex: systemd-repart[1] and the image is tested in QEMU.

[1] https://www.freedesktop.org/software/systemd/man/systemd-repart.html

(From OE-Core rev: f81b188bcf5aa18746fd622eb7b5c0dcb0b5c93d)

Signed-off-by: Kristian Klausen <kristian@klausen.dk>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2021-09-06 09:49:50 +01:00
Ross Burton
2d30e89a15 recipetool/create_buildsys_python: treat BSD as BSD-3-Clause
The PyPI license classifiers include "OSI Approved: BSD", which does not
specify which of the many variations of BSD license it actually means.

The generic "BSD" license in the oe-core set is actually BSD-3-Clause.
>From a random sample of ten PyPI modules that use "BSD", they are all
BSD-3-Clause.  As we expect the recipe maintainer to verify the license
anyway, and this matches the previous license text, explicitly set the
license to BSD-3-Clause.

(From OE-Core rev: a879fff9af31e45b1acc3f19a3c2a7eaf6319ad4)

Signed-off-by: Ross Burton <ross.burton@arm.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2021-09-04 08:44:11 +01:00
Daniel Gomez
6fdbc5144f wic: Add --no-fstab-update part option
When embedding a rootfs image (e.g. 'rootfs-dir') as a partition we
might want to keep the stock fstab for that image. In such a case, use
this option to not update the fstab and use the stock one instead.

This option allows you to specify which partitions get the fstab
updated and which get the stock fstab.

The option matches the argument you can pass to wic itself where the
same action is performed but for all the partitions.

Example:
    part /export --source rootfs --rootfs-dir=hockeycam-image
--fstype=ext4 --label export --align 1024 --no-fstab-update

    part / --source rootfs --fstype=ext4 --label rootfs --align 1024

(From OE-Core rev: ab4c95af8ecd15dc136194ab761afae756db5803)

Signed-off-by: Daniel Gomez <daniel@qtec.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2021-08-20 08:53:03 +01:00
Richard Purdie
dd6b55d70c package/scripts: Fix FILES_INFO handling
There is a long standing bug where FILES_INFO isn't written into pkgdata
with a package suffix. This means if the files are read into the datastore
as intended, the last one "wins".

Fix this to work as intended. Most of the call sites using the data need
to be updated to handle this and the overrides change correctly.

Also fix some other problematic references noticed along the way.

(From OE-Core rev: a1190903e0a61a12c9854c96af918ae8d12c6327)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2021-08-17 09:53:04 +01:00
Robert P. J. Day
166296f4bc scripts/lib/wic/help/py: "Redhat" -> "Red Hat"
Company name is properly spelled as "Red Hat".

(From OE-Core rev: cd60f20fe40bba5b80ab604ee34fbf4ec2b0b4a2)

Signed-off-by: Robert P. J. Day <rpjday@crashcourse.ca>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2021-08-17 09:53:04 +01:00
Ross Burton
c1b7a9cd64 wic: don't forcibly pass -T default
This reverts part of oe-core eecbe62555, which was a previous attempt
to solve the Y2038 problem.  This is now solved centrally in e2fsprogs,
so doesn't need to be dealt with in wic.

We don't revert the commit entirely, to retain the warning if a
filesystem has small inodes.

(From OE-Core rev: 7e8017208bed98b6c90735cb641fc9d7aedf9140)

Signed-off-by: Ross Burton <ross.burton@arm.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2021-08-12 06:26:15 +01:00
Alexander Kanavin
41753ff4f7 devtool: print a warning on upgrades if PREFERRED_VERSION is set
(From OE-Core rev: 0b0f53eed0aadbf45d9eead96ebf7725cc7447e6)

Signed-off-by: Alexander Kanavin <alex.kanavin@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2021-08-02 15:44:11 +01:00
Richard Purdie
65133015b1 meta: Manual override fixes
The automated conversion of OE-Core to use the new override sytax isn't
perfect. This patches some mis-converted lines and some lines which were missed
by the automation.

(From OE-Core rev: 4e9a06b64b43131b731fb59a0305f78a98e27fbd)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2021-08-02 15:44:10 +01:00
Richard Purdie
bb6ddc3691 Convert to new override syntax
This is the result of automated script conversion:

scripts/contrib/convert-overrides.py <oe-core directory>

converting the metadata to use ":" as the override character instead of "_".

(From OE-Core rev: 42344347be29f0997cc2f7636d9603b1fe1875ae)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2021-08-02 15:44:10 +01:00
Nicolas Dechesne
8dbef94dce checklayer: rename _find_layer_depends
What this function does is really to find a layer, not a 'depends'. We
are using this function to find a dependent layer, but the name is
confusing.

(From OE-Core rev: e9b7690ab30d0e7c07471034f6cb89ccc3168a11)

Signed-off-by: Nicolas Dechesne <nicolas.dechesne@linaro.org>
Signed-off-by: Alexandre Belloni <alexandre.belloni@bootlin.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2021-07-28 23:47:00 +01:00
Nicolas Dechesne
4f452a8be1 checklayer: new function get_layer_dependencies()
Split add_layer_dependencies() into 2 parts. First search for layer
dependencies, and then add them to the config. That allows us to
call get_layer_dependencies() independently.

(From OE-Core rev: 08edf928aac3f2daaa0c256d4c21e56e2db72bff)

Signed-off-by: Nicolas Dechesne <nicolas.dechesne@linaro.org>
Signed-off-by: Alexandre Belloni <alexandre.belloni@bootlin.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2021-07-28 23:47:00 +01:00
Alexander Kanavin
59cb4c7c3c devtool: correctly handle non-standard source tree locations in upgrades
When S is set to a sub-directory of upstream source, the license
checks and the bbappend writing (specifically, setting EXTERNALSRC)
need to operate on that sub-directory.

'devtool modify' already has similar logic, and it was copied from there
and adjusted.

(From OE-Core rev: fd77e356d4507405fde352c8bba7d3842518bbdd)

Signed-off-by: Alexander Kanavin <alex.kanavin@gmail.com>
Signed-off-by: Alexandre Belloni <alexandre.belloni@bootlin.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2021-07-16 21:23:43 +01:00
Florian Amstutz
73efc71b2c devtool: deploy-target: Fix preserving attributes when using --strip
Commit a2db4fa127 added ${WORKDIR}/deploy-* to
PSEUDO_IGNORE_PATHS. This breaks the --strip mode since ${D} is copied to
deploy-target-stripped. Use the directory devtool-deploy-target-stripped
instead.

[YOCTO #14451]

(From OE-Core rev: 02661f20faf11d0fa2f1874bd423f5d9fa7a31c9)

Signed-off-by: Florian Amstutz <florian.amstutz@scs.ch>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2021-07-02 07:24:21 +01:00
jbouchard
a79b5f0f21 Use the label provided when formating a dos partition
Previously the bootimg-pcbios wic plugin was not respecting
the --label option provided from the wks file. The plugin
was setting the label to 'boot'. With this fix, the --label
option is use. If no option are specified, then the default
is 'boot'.

(From OE-Core rev: 0fd7a73c1bd2486b7a022f0f69bbcb2e0d9cb141)

Signed-off-by: jbouchard <jeanbouch418@gmail.com>
Signed-off-by: Alexandre Belloni <alexandre.belloni@bootlin.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2021-06-29 22:18:56 +01:00
Alexander Kanavin
c341eea9e9 devtool upgrade: rebase override-only patches as well
There's a number of recipes (such as nettle or glib) which were upgraded
incorrectly due to rebasing only the default set of patches in main SRC_URI.
Native-only or musl-only patches were not handled, causing do_patch fails
and overall misery.

Unfortunately, this also necessitates aborting any incomplete rebases as
devtool needs to be able to return to the original branch to complete
the upgrade command.

Also, do not add devtool-patched-%version tags, as they are not used anywhere,
and it's unclear how to extend them to cover multiple override branches.

(From OE-Core rev: 00131cbc969f6a73caffe068fcb983287cbe577f)

(From OE-Core rev: 588d04748dd89b5371ca9144666e97ec82dd6204)

Signed-off-by: Alexander Kanavin <alex.kanavin@gmail.com>
Signed-off-by: Alexandre Belloni <alexandre.belloni@bootlin.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2021-06-26 11:27:16 +01:00
Richard Weinberger
63b3c44d27 Add support for erofs filesystems
Since Linux 5.4 a new read-only filesystem is available, erofs.
Compared to squashfs it offers much better read performance with and
without compression enabled.
It suppports two optional compressors, lz4 and lz4hc.

>From the mkfs.erofs man page:
EROFS is a new enhanced lightweight linux read-only filesystem with
modern designs (eg. no buffer head, reduced metadata, inline
xattrs/data, etc.) for scenarios which need high-performance read-only
requirements, e.g. Android OS for smartphones and LIVECDs.
It also provides fixed-sized output compression support, which improves
storage density, keeps relatively higher compression ratios, which is
more useful to achieve high performance for embedded devices with
limited  memory since it has unnoticable memory overhead and page cache
thrashing.

This commit adds support for three new filesystem targets:
erofs: erofs without compression
erofs-lz4: erofs with lz4 compresssion enabled
erofs-lz4hc: erofs with lz4hc compression enabled

(From OE-Core rev: 41dead1ff8ccc49e6cd6e6f5d41a59d164693e0d)

Signed-off-by: Richard Weinberger <richard@nod.at>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2021-05-31 14:31:01 +01:00
Devendra Tewari
b71375304f classes/lib/scripts: Use bb.utils.rename() instead of os.rename()
Incremental build in Docker fails with:

OSError: [Errno 18] Invalid cross-device link

when source and destination are on different overlay filesystems.

Rather than adding fallback code to every call site, use a new wrapper
in bitbake which detects this case and falls back to shutil.move
which is slower but will handtle the overlay docker filesystems correctly.

[YOCTO #14301]

(From OE-Core rev: 656a65b2b84e7d529b89cf5de7eb838f902d84a2)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2021-05-06 11:16:23 +01:00
Alejandro Hernandez Samaniego
6f3b5a8e24 devtool: Fix do_kernel_configme task
The do_kernel_configme task is no longer part of SRCTREECOVEREDTASKS,
its been removed from the kernel-yocto.bbclass since b72dbb2e4, but
there wasnt a matching patch for devtool for those changes.

This patch enables us to invoke the do_kernel_configme task when
using a devtool workspace, it also prepends a check for an existing
.config file in the source directory and moves it if thats the case,
since when using devtool modify a .config is created and
do_kernel_configme complains about it, this is not the case when
using bitbake since the .config file would be on B instead.

Alowing do_kernel_configme to run also fixes the flow where testing a
new config fragment from devtool workspace isnt added properly
(config queue shows it as ///frg.cfg) and as a side effect it never
gets merged into the final config.

(From OE-Core rev: 08dcc0e68095dcf2a159546a48b29d40c9aabc0b)

Signed-off-by: Alejandro Enedino Hernandez Samaniego <alhe@linux.microsoft.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2021-03-14 16:33:59 +00:00
Alexander Kanavin
d53dddd7ca scripts/lib/wic/partition.py: do not set FAT size
Modern dosfstools automatically determines the appropriate size
and will error out if something that doesn't make sense is
supplied on the command line.

(From OE-Core rev: b85a09ea450a5e8f49418f4a930805fbb88dc83b)

Signed-off-by: Alexander Kanavin <alex.kanavin@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2021-03-02 20:39:35 +00:00
Florian Bezdeka
a334cbb12d wic: Warn if an ext filesystem affected by the Y2038 problem is used
We are getting closer and closer to the year 2038 where the 32 bit
time_t overflow will happen. While products (= embedded systems) with an
expected life time of 15 years are still save the situation may change
if your system has to survive the next 20 years.

ext2 and ext3 filesystems are always affected by the time overflow, so
let's warn the user if these filesystems are still being used.

If ext4 is affected depends on the inode size chosen during filesystem
creation. At least 256 bytes are necessary to be safe. As ext4 is
used very often (and partitions may be created small first and extended
later) this might be an issue for many users.

Some filesystems created during CI runs were already affected by the Y2038
problem. By using `--mkfs-extraopts "-T default"` we tell mke2fs not to
auto-detect the usage type based on the filesystem size. mke2fs will use
the default values for tuning parameters instead. The inode size is one
of these parameters.

(From OE-Core rev: eecbe625558406680121d2a7e84917fea45ea9dc)

Signed-off-by: Florian Bezdeka <florian.bezdeka@siemens.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2021-02-26 15:21:21 +00:00
Lee Chee Yang
34ea1433fc wic: debug mode to keep tmp directory
files in wic tmp directory can be usefull for debugging, so do not remove
tmp directory when wic create run with debugging mode (-D or --debug).

also update wic.Wic.test_debug_short and wic.Wic.test_debug_long to
check for tmp directory.

[YOCTO#14216]

(From OE-Core rev: a122e2418b67d38f691edcf8dd846c167d6b4fa9)

Signed-off-by: Lee Chee Yang <Chee.Yang.Lee@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2021-02-09 08:56:11 +00:00
Tomasz Dziendzielski
3a220f1e41 devtool: Fix file:// fetcher symlink directory structure
Add relative path if file is under directory.

[YOCTO #13738]

(From OE-Core rev: 19ddacc1b38f9ebb86a9359963ccc3c707f7125e)

Signed-off-by: Tomasz Dziendzielski <tomasz.dziendzielski@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2021-01-23 17:08:54 +00:00
Tomasz Dziendzielski
0ad1537642 devtool: Fix source extraction for gcc shared source
If do_patch task is disabled then prepare do_configure dependencies to
fetch external sources and create symlink to ${S} in devtool workspace.

[YOCTO #13036]

(From OE-Core rev: 9e47319bfe62d289b90f7545a64dbdc1cbde7f1d)

Signed-off-by: Tomasz Dziendzielski <tomasz.dziendzielski@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2021-01-20 22:46:18 +00:00
Paul Barker
da846838fc wic: Optimise fstab modification for ext2/3/4 and msdos partitions
The fix for [Yocto #13994] required the rootfs directory to be copied
(using hardlinks if possible) when modifying the fstab file under wic.

We can optimise this copy away for filesystems where we have the tools
to modify the contents of the partition image after it is created. For
ext2/3/4 filesystems we have the debugfs tool and for msdos/vfat
filesystems we have the mcopy tool. So for any of these filesystems we
skip the modification of the fstab file in the rootfs directory (and
skip the associated copy unless it is otherwise necessary) and update
the contents of fstab directly in the partition image.

(From OE-Core rev: 5fb8ae0e9159597d7eaa9307a3a8543800bf9405)

Signed-off-by: Paul Barker <pbarker@konsulko.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2021-01-20 22:46:18 +00:00
Paul Barker
36575a9493 wic: Copy rootfs dir if fstab needs updating
By default, wic updates the /etc/fstab in the rootfs to include details
of additional partitions described in the selected wks file. If this
modification is performed in place, other tasks which create an image
file from the rootfs directory (e.g. do_image_tar and do_image_ext4)
will pick up the modified fstab file which would not be appropriate for
those images as they do not include the additional partitions described
in the wks file. wic does undo modifications to the fstab file once it
has finished creating the filesystem image, however this leaves open a
race condition if one of the other tasks reads the contents of the fstab
file from the rootfs directory between the point where wic modifies the
fstab file and the point where wic restores the files original content.

This could be solved by adding a lockfile for tasks which use the rootfs
directory to ensure that no other such task is reading the rootfs
directory while do_image_wic is running. This would serialize several
do_image_* tasks and result in slower builds, especially for large
images. Another drawback of this solution is that it is hard to
selectively optimise - adding lockfiles to do_image_* tasks would result
in these tasks always being serialized even if no fstab modification
will take place.

An alternative solution is to copy the rootfs directory when fstab needs
to be modified. The code to do this in wic already exists as it is
needed when including or excluding content in the rootfs. This still
results in an impact on build times but the copy uses hardlinks if
possible (so little data is actually copied) and we can make selective
optimisations to improve things. The rootfs copy will only take place if
fstab modification is required (or if it was already needed to include
or exclude rootfs content). We can also follow up with further
optimisations after this commit. So this second solution is chosen.

Fixes [Yocto #13994]

(From OE-Core rev: ce682a73b7447652f898ce1d1d0416a456df5416)

Signed-off-by: Paul Barker <pbarker@konsulko.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2021-01-20 22:46:18 +00:00
Paul Barker
f85a4a1462 wic: Update pseudo db when excluding content from rootfs
To exclude content from the rootfs, wic makes a copy (using hardlinks if
possible) of the rootfs directory and associated pseudo db, then removes
files & directories as needed. However if these files and directories
are removed using the python functions os.remove and shutil.rmtree, the
copied pseudo db will not be updated correctly. For files copied from
the original rootfs, if hardlinks were used successfully when copying
the rootfs this should mean that the relevant inodes can't be reused and
so the risk of pseudo aborts should be avoided. However, this logic
doesn't apply for directories (as they can't be hardlinked) or for files
added via the '--include-path' argument (as they weren't present in the
original rootfs) and so there remains some risk of inodes being reused
and the pseudo db becoming corrupted.

To fix this, use the 'rm' command under pseudo when removing files &
directories from the copied rootfs to ensure that the copied pseudo db
is updated.

(From OE-Core rev: d5db7e268947f0392c2126137571a44acd29ccd6)

Signed-off-by: Paul Barker <pbarker@konsulko.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2021-01-20 22:46:18 +00:00
Paul Barker
4e6cb084f5 wic: Ensure internal workdir is not reused
If a path is specified for the internal wic working directory using
the -w/--workdir argument then it must not already exist. Re-using a
previous workdir could easily result in rootfs and intermediate files
from a previous build being added to the current image.

(From OE-Core rev: 2e40c8d4109024ff704c5ce40d98050ca7f34dd5)

Signed-off-by: Paul Barker <pbarker@konsulko.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2021-01-20 22:46:18 +00:00
Kamel Bouhara
93543e9960 recipetool: create: only add npmsw url if required
Before adding a npmsw fetcher to a recipe we
should first check if the generated shrinkwrap file
contains dependencies.

(From OE-Core rev: ef153ad36d0299e83a03af8f207686d0d8a238b3)

Signed-off-by: Kamel Bouhara <kamel.bouhara@bootlin.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2021-01-16 22:39:36 +00:00
Paul Barker
cf73a72632 wic: Allow exec_native_cmd to run HOSTTOOLS
This allows programs from HOSTTOOLS (e.g. 'install', 'rm', 'mv', etc) to
be more easily executed by wic. Without this change only programs from
an actual *-native recipe built by bitbake can be executed by wic.

(From OE-Core rev: 8eb186acdecfbb3151c9a0ab148358e3fe5cce39)

Signed-off-by: Paul Barker <pbarker@konsulko.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2021-01-13 23:13:45 +00:00
Paul Barker
1b816d180b wic: Add workdir argument
This allows the path for the temporary workdir used by wic to be set
when running wic from bitbake or directly from the command line.

(From OE-Core rev: 2e7314ac3a3cab89de93c932e8efc75de0feb0a7)

Signed-off-by: Paul Barker <pbarker@konsulko.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2021-01-13 23:13:45 +00:00
Lee Chee Yang
3fa6a47aeb wic/direct/kparser: ensure fsuuid for vfat and msdos align with format
vfat/msdos filesystem should have fsuuid in format 0xYYYYYYYY where "0x"
in front follow with 8 hexadecimal number in uppercase. In wic, when using
custom fsuuid for vfat/msdos partition in wks, it is able to set the value
in any length, with or without leading "0x". This can cause fsuuid
missaligned when fstab updates, fstab expect exactly 10 character
fsuuid for vfat/msdos partition and all in uppercase.

if custom fsuuid for vfat/msdos is set, check the length and format,
error if it exceed the format size. Amend it so it is align with format
0xYYYYYYYY. This is done before image create and fstab update to ensure the
fsuuid are same in all followup process. if custom fsuuid length less than
expected, fill in "0".

[YOCTO #14161]

(From OE-Core rev: d9686ae511ef10a504becfd81bfe296b788e1456)

Signed-off-by: Lee Chee Yang <chee.yang.lee@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2021-01-01 23:15:13 +00:00
Alexander Kanavin
f1efa4792f devtool: gitsm:// should be handled same as git:// in upgrades
(From OE-Core rev: e7076f1742bb83e76aa4e90aad861546536166cf)

Signed-off-by: Alexander Kanavin <alex.kanavin@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2020-12-31 11:53:26 +00:00
Peter Kjellerstedt
e12072c7e0 wic: Pass canonicalized paths in PSEUDO_IGNORE_PATHS
(From OE-Core rev: 55618cbd58d6784a82e773f323723be6f722232f)

Signed-off-by: Peter Kjellerstedt <peter.kjellerstedt@axis.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2020-12-20 00:03:04 +00:00
Diego Sueiro
75b74f6b8d wic: Introduce empty plugin to create unformatted empty partitions
The empty wic plugin is used to create unformatted empty partitions for wic
images.

To use it you must pass "empty" as argument for the "--source" parameter in
the wks file. For example:
    part foo --source empty --ondisk sda --size="1024" --align 1024

Also adds a selftest for this plugin where the 'Fstype' column from 'wic
ls' should be empty for the second partition as listed in
test_empty_plugin.wks.

(From OE-Core rev: 77d174fc80663403ef76c5b808aafc1117d3545c)

Signed-off-by: Diego Sueiro <diego.sueiro@arm.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2020-12-15 22:51:56 +00:00
Chaitanya Vadrevu
ea5880f9f9 isoimage-isohybrid.py: Support adding files/dirs
Add "payload_dir" parameter so user can add other files/dirs
to the ISO.

(From OE-Core rev: bbe84a33c81066fee96dd28a4009d8404b51e642)

Signed-off-by: Chaitanya Vadrevu <chaitanya.vadrevu@ni.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2020-11-08 14:03:20 +00:00
Roland Hieber
a6c9ee99b3 devtool: make sure .git/info exists before writing to .git/info/excludes
If nothing else is specified, 'git init' uses its default repository
template from the install location (e.g. /usr/share/git-core/templates),
which already includes an info/ subdirectory. However, when setting
init.templateDir to a different template path in ~/.gitconfig, this
isn't necessarily the case, and it can lead to setup_git_repo() failing
with stack traces like:

    File: '.../scripts/lib/devtool/__init__.py', lineno: 234, function: setup_git_repo
         0230:        pass
         0231:    if 'singletask.lock\n' not in excludes:
         0232:        excludes.append('singletask.lock\n')
         0233:    bb.warn("try writing excludefile")
     *** 0234:    with open(excludefile, 'w') as f:
         0235:        for line in excludes:
         0236:            f.write(line)
         0237:
         0238:    bb.process.run('git checkout -b %s' % devbranch, cwd=repodir)
    Exception: FileNotFoundError: [Errno 2] No such file or directory: '.../devtooltmp-6m36b181/workdir/foobar-1.0.1/.git/info/exclude'

Fix this edge case by creating the .git/info/ directory first.

Fixes: 334ba846c795fc0d8c73 (2018-02-01, "devtool: set up git repos so that singletask.lock is ignored")
(From OE-Core rev: 148a23e4d5ceaf655ccacb52deca4ba501f12975)

Signed-off-by: Roland Hieber <rhi@pengutronix.de>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2020-10-17 12:34:29 +01:00
Ross Burton
197cec77f5 devtool: remove unused variable
This variable isn't used at all.

(From OE-Core rev: 9676ee24d190b144cb50be6ebde3990644a6df2a)

Signed-off-by: Ross Burton <ross.burton@arm.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2020-10-08 08:08:32 +01:00
Richard Purdie
c194e5fac6 wic: Handle new PSEUDO_IGNORE_PATHS variable
Adjust wic to correctly handle the new PSEUDO_IGNORE_PATH variable and avoid
inode corruption issues.

(From OE-Core rev: 13500f5234361385c365c7c35e83f99435500481)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2020-10-07 11:18:38 +01:00
Khairul Rohaizzat Jamaluddin
85574ce0cf wic/bootimg-efi: IMAGE_EFI_BOOT_FILES variable added to separate bootimg-efi and bootimg-partition
Due to recent changes in bootimg-efi to include IMAGE_BOOT_FILES,
when both bootimg-partition and bootimg-efi occur in a single .wks
and IMAGE_BOOT_FILES are defined, files listed in IMAGE_BOOT_FILES
will be duplicated in both partition.
Since IMAGE_BOOT_FILES are crucial for bootimg-partition, but
optional for bootimg-efi, hence allowing bootimg-efi to have the option
to ignore it.

The new variable, IMAGE_EFI_BOOT_FILES, was added to help handle this
issue. Its basic usage is the same as IMAGE_BOOT_FILES.
Usage example:
        ${IMGDEPLOYDIR}/${IMAGE_BASENAME}-${MACHINE}.ext4;rootfs.img \

This commit is also squashed with the updated testcase to cover for
this change.

[YOCTO #14011]

(From OE-Core rev: 945339e06b273df1935cfd784f548ef57e0b7f4c)

Signed-off-by: Khairul Rohaizzat Jamaluddin <khairul.rohaizzat.jamaluddin@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2020-09-15 11:53:28 +01:00
Michael Tretter
fdc6ef9673 devtool: deploy-target: Fix size calculation for hard links
If a package contains hard links to a file, the file size is added for
each hard link instead of once for the file. Therefore, the calculated
size may be much larger than the actual package size.

For example, the mesa-megadriver package contains several hard links to
the same library.

Keep track of the inode numbers when listing the files that are
installed and use the actual size only for the first occurrence of an
inode. All further hard links to the same inode are added to the file
list, but accounted with size 0.

All file names need to be added to the file list, because the list is
used for preserving the files/hard links on the target.

(From OE-Core rev: 17e92572278980d1a7f06de9d72c68baf57698f1)

Signed-off-by: Michael Tretter <m.tretter@pengutronix.de>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2020-09-02 16:00:46 +01:00
Vijai Kumar K
435ec90a67 wic: misc: Add /bin to the list of searchpaths
/bin is also a valid path where one can find executables. Add
that to the search path.

(From OE-Core rev: ca0a6025351cb2135e87cecf828633cf12aa34c6)

Signed-off-by: Vijai Kumar K <vijaikumar.kanagarajan@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2020-09-02 16:00:46 +01:00