This patches makes locating the file database containing the file and
folder usernames and permissions more reliable.
In addition to locating it relative to the partition directory, we also
try to locate it relative to the IMAGE_ROOTFS.
Prior to this patch, the database was not found when using
--rootfs-dir=${IMAGE_ROOTFS}/<x> in the WIC script, leading to erronous file
permissions and ownership.
(From OE-Core rev: d730fbc26993c2187c908e57a465536a508e9ee2)
Signed-off-by: Felix Moessbauer <felix.moessbauer@siemens.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
(cherry picked from commit 09e18ee246da8b56f446c4db548fb9c7e895142b)
Signed-off-by: Anuj Mittal <anuj.mittal@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
When embedding a rootfs image (e.g. 'rootfs-dir') as a partition we
might want to keep the stock fstab for that image. In such a case, use
this option to not update the fstab and use the stock one instead.
This option allows you to specify which partitions get the fstab
updated and which get the stock fstab.
The option matches the argument you can pass to wic itself where the
same action is performed but for all the partitions.
Example:
part /export --source rootfs --rootfs-dir=hockeycam-image
--fstype=ext4 --label export --align 1024 --no-fstab-update
part / --source rootfs --fstype=ext4 --label rootfs --align 1024
(From OE-Core rev: ab4c95af8ecd15dc136194ab761afae756db5803)
Signed-off-by: Daniel Gomez <daniel@qtec.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The fix for [Yocto #13994] required the rootfs directory to be copied
(using hardlinks if possible) when modifying the fstab file under wic.
We can optimise this copy away for filesystems where we have the tools
to modify the contents of the partition image after it is created. For
ext2/3/4 filesystems we have the debugfs tool and for msdos/vfat
filesystems we have the mcopy tool. So for any of these filesystems we
skip the modification of the fstab file in the rootfs directory (and
skip the associated copy unless it is otherwise necessary) and update
the contents of fstab directly in the partition image.
(From OE-Core rev: 5fb8ae0e9159597d7eaa9307a3a8543800bf9405)
Signed-off-by: Paul Barker <pbarker@konsulko.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
By default, wic updates the /etc/fstab in the rootfs to include details
of additional partitions described in the selected wks file. If this
modification is performed in place, other tasks which create an image
file from the rootfs directory (e.g. do_image_tar and do_image_ext4)
will pick up the modified fstab file which would not be appropriate for
those images as they do not include the additional partitions described
in the wks file. wic does undo modifications to the fstab file once it
has finished creating the filesystem image, however this leaves open a
race condition if one of the other tasks reads the contents of the fstab
file from the rootfs directory between the point where wic modifies the
fstab file and the point where wic restores the files original content.
This could be solved by adding a lockfile for tasks which use the rootfs
directory to ensure that no other such task is reading the rootfs
directory while do_image_wic is running. This would serialize several
do_image_* tasks and result in slower builds, especially for large
images. Another drawback of this solution is that it is hard to
selectively optimise - adding lockfiles to do_image_* tasks would result
in these tasks always being serialized even if no fstab modification
will take place.
An alternative solution is to copy the rootfs directory when fstab needs
to be modified. The code to do this in wic already exists as it is
needed when including or excluding content in the rootfs. This still
results in an impact on build times but the copy uses hardlinks if
possible (so little data is actually copied) and we can make selective
optimisations to improve things. The rootfs copy will only take place if
fstab modification is required (or if it was already needed to include
or exclude rootfs content). We can also follow up with further
optimisations after this commit. So this second solution is chosen.
Fixes [Yocto #13994]
(From OE-Core rev: ce682a73b7447652f898ce1d1d0416a456df5416)
Signed-off-by: Paul Barker <pbarker@konsulko.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
To exclude content from the rootfs, wic makes a copy (using hardlinks if
possible) of the rootfs directory and associated pseudo db, then removes
files & directories as needed. However if these files and directories
are removed using the python functions os.remove and shutil.rmtree, the
copied pseudo db will not be updated correctly. For files copied from
the original rootfs, if hardlinks were used successfully when copying
the rootfs this should mean that the relevant inodes can't be reused and
so the risk of pseudo aborts should be avoided. However, this logic
doesn't apply for directories (as they can't be hardlinked) or for files
added via the '--include-path' argument (as they weren't present in the
original rootfs) and so there remains some risk of inodes being reused
and the pseudo db becoming corrupted.
To fix this, use the 'rm' command under pseudo when removing files &
directories from the copied rootfs to ensure that the copied pseudo db
is updated.
(From OE-Core rev: d5db7e268947f0392c2126137571a44acd29ccd6)
Signed-off-by: Paul Barker <pbarker@konsulko.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Combine all the common path validation in a function to avoid code
duplication.
Cc: Paul Barker <pbarker@konsulko.com>
(From OE-Core rev: ebd12b10d17db0b4176b0188407d7e9b8420eab1)
Signed-off-by: Ricardo Ribalda Delgado <ricardo@ribalda.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Allow specifying an optional destination to include-path and make the
option aware of permissions and owners.
It is very useful for making a partition that contains the rootfs for a
host and a target Eg:
/ -> Roofs for the host
/export/ -> Rootfs for the target (which will netboot)
Although today we support making a partition for "/export" this might
not be compatible with some upgrade systems, or we might be limited by
the number of partitions.
With this patch we can use something like:
part / --source rootfs --fstype=ext4 --include-path core-image-minimal-mtdutils export/ --include-path hello
on the .wks file.
Cc: Paul Barker <pbarker@konsulko.com>
(From OE-Core rev: e8c21c6ebaebde88151697381bdb2452f1171090)
Signed-off-by: Ricardo Ribalda Delgado <ricardo@ribalda.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
If an excuded path does not exist, continue without an error.
This allows to seamleasly reuse .wks among different projects.
Eg:
part / --source rootfs --fstype=ext4 --exclude-path=opt/private_keys
Where /opt/private_keys in only populated by some of the image.bb files.
Cc: Paul Barker <pbarker@konsulko.com>
(From OE-Core rev: 2dbd692b8e563cf991fb4ae1ef6129fda0d7e3c4)
Signed-off-by: Ricardo Ribalda Delgado <ricardo@ribalda.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
This option allows to specify which part of a rootfs is going to be
included, the same way the -C argument on tar.
Thanks to this option we can make sure the permissions and usernames
on the target partition are respected, and also simplify the creation of
splitted partitons, not neeting to invoke external vars or using .wks.in
files. Eg:
part / --source rootfs --ondisk sda --fstype=ext4 --exclude-path=etc/
part /etc --source rootfs --fstype=ext4 --change-directory=etc
Cc: Paul Barker <pbarker@konsulko.com>
(From OE-Core rev: 2265d089a58e1f78f26d623ee667c420cb1c3bd4)
Signed-off-by: Ricardo Ribalda Delgado <ricardo@ribalda.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
When parameters include_path or exclude_path are passed to the rootfs
plugin, it will copy the partition content into a folder and make all
the modifications there.
This is done using copyhardlinktree(), which does not take into
consideration the content of the pseudo folder, which contains the
information about the right permissions and ownership of the folders.
This results in a rootfs owned by the user that is running the wic
command (usually UID 1000), which makes some rootfs unbootable.
This bug can be easily triggerd with the following .wks
part / --source rootfs --fstype=ext4 --exclude-path=home
And this sequence:
$ wic create test-permissions -e core-image-minimal -o test/
$ sudo mount test/test-permissions-202004080823-sda.direct.p1 /mnt
$ ls -la /mnt/etc/shadow
To fix this we copy the content of the pseudo folders to the new folder
and modify the pseudo database using the "pseudo -B" command.
If the rootfs is not a rootfs generated by bitbake a warning is shown
making the user aware that the permissions on the target might not match
what he expects.
WARNING: /tmp/test/../pseudo folder does not exist. Usernames and permissions will be invalid
Cc: Paul Barker <pbarker@konsulko.com>
(From OE-Core rev: fd739c15cdba221d9d497d3402b7d929c0be2ca4)
Signed-off-by: Ricardo Ribalda Delgado <ricardo@ribalda.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
This reverts commit 36993eea89d1c011397b7692b9b8d61b499d0171.
After discussion on the mailing list it was felt these changes
were not ready yet.
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
This reverts commit efdcf94801f6abe8e4099e324d9a3deccd8d4384.
After discussion on the mailing list it was felt these
changes were not ready yet.
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
This option adds the content of a rootfs on a specific location on the
rootfs.
It is very useful for making a partition that contains the rootfs for a
host and a target Eg:
/ -> Roofs for the host
/export/ -> Rootfs for the target (which will netboot)
Although today we support making a partition for "/export" this might
not be compatible with some upgrade systems, or we might be limited by
the number of partitions.
With this patch we can use something like:
part / --source rootfs --embed-rootfs target-image /export --embed-rootfs target-image2 /export2
on the .wks file.
(From OE-Core rev: efdcf94801f6abe8e4099e324d9a3deccd8d4384)
Signed-off-by: Ricardo Ribalda Delgado <ricardo@ribalda.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
When parameters include_path or exclude_path are passed to the rootfs
plugin, it will copy the partition content into a folder and make all
the modifications there.
This is done using copyhardlinktree(), which does not take into
consideration the content of the pseudo folder, which contains the
information about the right permissions and ownership of the folders.
This results in a rootfs owned by the user that is running the wic
command (usually UID 1000), which makes some rootfs unbootable.
To fix this we copy the content of the pseudo folders to the new folder
and modify the pseudo database using the "pseudo -B" command.
(From OE-Core rev: 36993eea89d1c011397b7692b9b8d61b499d0171)
Signed-off-by: Ricardo Ribalda Delgado <ricardo@ribalda.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
This option adds the contents of the given path to a partition built
with the rootfs source plugin. The path is relative to the directory in
which wic is running not the rootfs itself so use of an absolute path
is recommended. This option is most useful when multiple copies of the
rootfs are added to an image and it is required to add extra content to
only one of these copies. This option only has an effect with the
rootfs source plugin.
(From OE-Core rev: d4cd27a9837426e809190548a83c6c7c76505114)
Signed-off-by: Paul Barker <pbarker@konsulko.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
This adds SPDX license headers in place of the wide assortment of things
currently in our script headers. We default to GPL-2.0-only except for the
oeqa code where it was clearly submitted and marked as MIT on the most part
or some scripts which had the "or later" GPL versioning.
The patch also drops other obsolete bits of file headers where they were
encoountered such as editor modelines, obsolete maintainer information or
the phrase "All rights reserved" which is now obsolete and not required in
copyright headers (in this case its actually confusing for licensing as all
rights were not reserved).
More work is needed for OE-Core but this takes care of the bulk of the scripts
and meta/lib directories.
The top level LICENSE files are tweaked to match the new structure and the
SPDX naming.
(From OE-Core rev: f8c9c511b5f1b7dbd45b77f345cb6c048ae6763e)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Used unique suffix (line number from .wks file) for the
copied rootfs directory to avoid possible conflicts.
(From OE-Core rev: 01efc234a8caab67ed3138ab2de9bbd82ce97b44)
Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
wic gets rootfs paths from partition object property
'rootfs_dir' and shows them in final report.
rootfs plugin sets this property to the temporary path,
which causes temporary paths appearing in the report.
Changed the code to prevent storing temporary rootfs path
in part.rootfs_dir. This should fix the report.
(From OE-Core rev: 28d2d7d6f79df08431187c7debaab2a3fa516671)
Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Using relative paths can cause copyhardlinktree API to fail as
it changes current directory when working. Converted all paths
to absolute paths using os.path.realpath.
(From OE-Core rev: a1c83cebe986e211dfc31be5cbd748f53fc298df)
Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Moved misc.py from wic/utils/ to wic/
Removed wic/utils directory
(From OE-Core rev: df906f3caa0721756f5ed48fa657e62e05ae2aa3)
Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
This was overlooked when msger was removed in 28014087b8.
(From OE-Core rev: b561292c4e7b5578172066f82b6518b5bda53f42)
Signed-off-by: Kristian Amlie <kristian.amlie@mender.io>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Removed unused exceptions from error.py
Moved definition of WicError to lib/wic/__init__.py
(From OE-Core rev: 15442d072bb6d93bd9b941726f93262503053da5)
Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Replaced sys.exit with raising WicError in wic plugins.
(From OE-Core rev: 92e8c81c941597eb2b4b61d5c28833e4826888f8)
Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
It will omit the given path from the resulting partition, and if the
given path ends in a slash, it will only delete the content, and keep
the directory.
Since mkfs only accepts whole directories as input, we need to copy
the rootfs directory to the workdir so that we can selectively delete
files from it.
Since we want to use the copyhardlinktree() function, we need to put
the generic oe lib in the module search path.
(From OE-Core rev: 6602392db3d391d926dead49fcc54326015cfe35)
Signed-off-by: Kristian Amlie <kristian.amlie@mender.io>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Got rid of get_rootfs and set_rootfs java-like getter and
setter. Renamed rootfs to rootfs_dir to be consistent with
the name of kickstart parameter --rootfs-dir.
(From OE-Core rev: 51ec52a62ce49d1a1a83489379990f78cfe849f9)
Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Wic gets bitbake variables by parsing output of 'bitbake -e' command.
This implementation improves this procedure as it runs 'bitbake -e' only
when API is called and does it only once, i.e. in a "lazy" way. As parsing
results are cached 'bitbake -e' is run only once and results are parsed
only once per requested set of variables.
get_bitbake_var became the only API call. It replaces find_artifacts,
find_artifact, find_bitbake_env_lines, get_bitbake_env_lines,
set_bitbake_env_lines and get_line_val calls making API much more clear.
(From OE-Core rev: 3abe23bd217315246ec2d98dc9c390b85cfe6a92)
Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Docstings from plugin classes are used as part of
output of 'wic help plugins'. Adding them makes help page
more informative.
(From OE-Core rev: d4414b45c81ab539f75a7bb16fc6412c30dfc45f)
Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Fixed wrong name for the first argument in class methods.
Pylint complains about the issue this way:
Class method should have 'cls' as first argument
(From OE-Core rev: cd7c72d6d5a5110173d0d3f60f2991b1dc30fc26)
Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Here is what PEP8(Style Guide for Python Code) says about this:
Wildcard imports (from <module> import *) should be avoided, as they
make it unclear which names are present in the namespace, confusing
both readers and many automated tools.
(From OE-Core rev: 13416c1941f5dc8abcdb0073f2104a89eae2d6f1)
Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
As well as any other stray instances of mic in the codebase that can
be removed.
We don't really need to carry around legacy naming, and the history is
in git.
(From OE-Core rev: 598b120406dc1d2b7e377bd1ab6f0acbef034b22)
Signed-off-by: Tom Zanussi <tom.zanussi@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
