The dependencies of do_package_write_* tasks are either going to be packaging
tools needed to build the packages, or, native tools needed at postinst
time. Now we've formalised this dependency pattern, drop the hardcoded
list and work based on the rule. The package creation tools are usually
the same tools needed at rootfs/postinst time anyway so the difference is
moot.
(From OE-Core rev: 8082c6aabf838a2cc5253d2bb1bd8867f2e1ba6a)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
It turns out that this check cannot work. We don't have the information
to know whether an sstate package is really needed at this point in the
execution, so we check the availability for things that we won't
actually end up needing later on. Thus we can't fail if some of these
aren't found or we'll get needless failures.
This check was intended to give earlier more accurate errors when sstate
artifacts failed to download, but that's not practical so we'll rely
solely on the task execution check that was added within the runqueue.
This reverts most of commit 9e711b54487c3141d7264b8cf0d74f9465020190 (we
still need to allow BB_SETSCENE_ENFORCE through from the external
environment since the eSDK relies upon that.)
(From OE-Core rev: ff29ac6901d04487312f554d9e62250a18729c6c)
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The code wasn't working properly, tweak so that it works as expected and
the grep expression includes the right patterns. Not sure this code has ever
worked prior to this.
(From OE-Core rev: 6a46a974611c5262efefc21a11adb736c0fb206a)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* Fix not decoding output from grep ("Matched in b'manifest...')
* Fix showing "Matched in b''" if no match (show "not matched to any
task" instead)
* Drop the filtering out of .populate-sysroot from matched manifest
names - it should have been .populate_sysroot so it doesn't work, and
in any case the value of removing the task name is questionable given
that we aren't removing it for any other task, and that the rest of
the filename isn't only the task name, we might as well have the whole
thing. At least then you can do a find on that exact name without
wildcards and find it.
* Fix indenting of file list entries and indent "matched in" further
underneath
* Minor punctuation fixes
(From OE-Core rev: 4675ce43496898fccbac738835d7e92b1cca648e)
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
If a recipe installs a file or directory whose name contains square
brackets [ ] that form a valid glob expression and that file then they
won't be correctly removed from the sysroot, because we pass each path
in the sstate manifest to our oe.path.remove() function which calls
glob.glob() on the path passed into it and the expression won't
actually match the original filename. Since we don't expect to put any
wildcarded expressions in the sstate manifests, and we already have a
try...except around this, we can actually use os.remove() here instead.
Similarly, when we pass existing file paths to "grep" looking through
the manifests, we don't want those paths to be treated as regexes - so
use grep's -F command line switch.
Fixes [YOCTO #10836].
(From OE-Core rev: fd8a57861024fc82e15a2a4ec8c20ed0ebb242f6)
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
getVarFlag() now defaults to expanding by default, thus remove the
True option from getVarFlag() calls with a regex search and
replace.
Search made with the following regex:
getVarFlag ?\(( ?[^,()]*, ?[^,()]*), True\)
(From OE-Core rev: 2dea9e490a98377010b3d4118d054814c317a735)
Signed-off-by: Joshua Lock <joshua.g.lock@intel.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
getVar() now defaults to expanding by default, thus remove the True
option from getVar() calls with a regex search and replace.
Search made with the following regex: getVar ?\(( ?[^,()]*), True\)
(From OE-Core rev: 7c552996597faaee2fbee185b250c0ee30ea3b5f)
Signed-off-by: Joshua Lock <joshua.g.lock@intel.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
We don't remove sigdata files, we also shouldn't remove sigbasedata files
as this hinders debugging.
(From OE-Core rev: 1ebd85f8dfe45b92c0137547c05e013e340f9cec)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The checkstatus function fires an event to notify bitbake UI about
the progress of the task, this function is implemented using ThreadPool
and is causing event lose when multiple threads tries to fire an event
(writes over socket/fd).
[YOCTO #10330]
(From OE-Core rev: 6e0bb9d141438c0051c32b0d3a247915b71ccb82)
Signed-off-by: Aníbal Limón <anibal.limon@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
This sets a good example and avoids unnecessarily contributing to
perceived complexity and cargo culting.
Motivating quote below:
< kergoth> the *original* intent was for the function/task to error via
whatever appropriate means, bb.fatal, whatever, and
funcfailed was what you'd catch if you were calling
exec_func/exec_task. that is, it's what those functions
raise, not what metadata functions should be raising
< kergoth> it didn't end up being used that way
< kergoth> but there's really never a reason to raise it yourself
FuncFailed.__init__ takes a 'name' argument rather than a 'msg'
argument, which also shows that the original purpose got lost.
(From OE-Core rev: 33611b69c221cf875eba1c7cb599c256825ae470)
Signed-off-by: Ulf Magnusson <ulfalizer@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
There are some issues in sstate which can't be handled by file removal
alone. Currently there is no way to execute a command against sstate and
doing so is potentially problematic for things like dependencies. This
patch adds a mechanism where any "postrm" script is executed if its present
allowing some openjade/sgml issues to be resolved.
[YOCTO #8273]
(From OE-Core rev: 2268efd0cd3ddb40870c4c424d10444ba86d2849)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Without this a whitelisted path containing /../ breaks the test for a file
allowed to be provided by more than one recipe.
Noticed when local.conf contains:
DEPLOY_DIR = "${TOPDIR}/../deploy"
|ERROR: core-image-minimal-1.0-r0 do_image_complete: The recipe
| core-image-minimal is trying to install files into a shared area when those
| files already exist. Those files and their manifest location are:
| .../poky/deploy/images/qemux86/README_-_DO_NOT_DELETE_FILES_IN_THIS_DIRECTORY.txt
| Matched in b'manifest-qemux86-linux-yocto.deploy'
(From OE-Core rev: d82acf20541463b14a811788d28fb1db3539885b)
Signed-off-by: Max Krummenacher <max.krummenacher@toradex.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
SSTATE_SKIP_CREATION variable will be used to skip creation of
sstate .tgz files. It makes sense for image creation tasks as
tarring images and keeping them in sstate would consume a lot of
disk space.
(From OE-Core rev: 7e821ccd221916ae8482b9113df2de704f4a99a4)
Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Add a meta-recipe to bring the toolchain into the extensible SDK. This
was modelled on meta-ide-support but some adjustments were needed to the
dependency validation function in sstate.bbclass to ensure that all of
the toolchain gets installed into the sysroot. With this, after
installing a minimal eSDK you only need to run the following after
sourcing the environment setup script to get the toolchain:
devtool sdk-install meta-extsdk-toolchain
Addresses [YOCTO #9257].
(From OE-Core rev: 8110806b1b5534ae830a4fdd1a5293c86a712d0b)
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Currently PV is defined in meta/conf/bitbake.conf as a python
expression: "${@bb.parse.BBHandler.vars_from_file(d.getVar('FILE',
False),d)[1] or '1.0'}". As FILE is whitelisted it causes PV to
not depend on it. This causes sstate code to not detect that
PV changes when recipe filename changes.
Making PV to explicitly depend on PV variable value overrides default
behaviour. Instead of depending on python expression bitbake depends
on evaluated value of PV variable, which should fix the above
mentioned issue.
[YOCTO #9806]
(From OE-Core rev: 918646ca803d56004fb0ab7c21e86cc9cb14513d)
Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
If BB_SETSCENE_ENFORCE is set to "1" and an sstate package fails to
download outside of the whitelist specified by
BB_SETSCENE_ENFORCE_WHITELIST, then fail immediately so you can tell
that the problem was caused by failing to restore the task from sstate.
Part of the implementation of [YOCTO #9367].
(From OE-Core rev: 9e711b54487c3141d7264b8cf0d74f9465020190)
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Show progress through bitbake's standard terminal UI when checking for
shared state object availability, since this can take some time if there
are a large number of tasks to be executed and/or the network connection
is slow.
Part of the implementation for [YOCTO #5853].
(From OE-Core rev: 1a064385d6921ec90b33c9064dafaab11a36267c)
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The syntax for octal values changed in python3, adapt to it.
(From OE-Core rev: 737a095fcde773a36e0fee1f27b74aaa88062386)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Previously the sstate was all downloaded to the same directory and then
symlinks were added in the directories that pointed to the siginfo and
sstate in the parent directory.
This change makes it so that now the files are just downloaded to the
correct location without the need for symlinks.
(From OE-Core rev: 55d25ed6b30ed7105d3b6421fbf2a03cea009a59)
Signed-off-by: Randy Witt <randy.e.witt@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
This avoids 'basehash changed' errors with python 3 but could break
build determinism in general.
(From OE-Core rev: 488f88fc12eea9788e5db1de354f0118e2c58878)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Add the SIGGEN_LOCKEDSIGS_TASKSIG_CHECK and
SIGGEN_LOCKEDSIGS_SSTATE_EXISTS_CHECK variables to replace
SIGGEN_LOCKEDSIGS_CHECK_LEVEL.
SIGGEN_LOCKEDSIGS_TASKSIG_CHECK will no control whether there is a
warning or error if a task's hash in the locked signature file doesn't match
the computed hash from the current metadata.
SIGGEN_LOCKEDSIGS_SSTATE_EXISTS_CHECK will control whther there is a
warning or error if a task that supports sstate is in the locked
signature file, but no sstate exists for the task.
Previously you could only have warning/errors for both controlled by
SIGGEN_LOCKEDSIGS_CHECK_LEVEL. This was an issue in the extensible sdk,
because we know sstate won't exist for certain items in the reverse
dependencies list for tasks. However, we still want to error if task
signatures don't match.
[YOCTO #9195]
(From OE-Core rev: 0fe2a5e5ffd01e926d0f3d4c78ad9910296e2d1a)
Signed-off-by: Randy Witt <randy.e.witt@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
At present, if a recipe is built which creates users/groups via
useradd.bbclass, those users/groups are not removed from sysroot
when the recipe/package is cleaned using clean/cleansstate/cleanall
or when a recipe is rebuild and 'unstaged' from the the sysroot.
The "userdel_sysroot_sstate()" provides that functionality.
[YOCTO #9262]
(From OE-Core rev: b5304ce438666a7418746f4ddd32703ae3188089)
Signed-off-by: Maxin B. John <maxin.john@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
uninative needs to adjust NATIVELSBSTRING fairly late in the
configuration parsing process but the sstate code encodes it into
variables. Since this string doesn't vary on a per recipe basis, we
defer its expansion until usage time.
(From OE-Core rev: fb680ab8b3cf7aba9b579403b1aeb96d30762320)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
(From OE-Core rev: a6426479f6348524a904ba28c16c1e3c8793cb08)
Signed-off-by: Ming Liu <peter.x.liu@external.atlascopco.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
do_cleansstate wasn't cleaning do_populate_lic sstate objects in the
native/cross case since the wildcard path entry wasn't being cleared
at the same time as the path extra prefix. Fix by clearing it at the
same time.
(From OE-Core rev: af72c507f5e20acf64b7431cee989af9908ef199)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
This variable is only used by populate_lic tasks (gcc-source overrides it)
and refers to BPN. In recipes like gcc, where there are multiple variants,
it is resulting in sstate objects which encode PN (they install into a PN subdir)
but the sstate object reflects BPN. This leads to corruption between then and
eventually, warnings from image builds like:
WARNING: The license listed GPLv3 was not in the licenses collected for recipe gcc
WARNING: The license listed GPL-3.0-with-GCC-exception was not in the licenses collected for recipe gcc
WARNING: The license listed GPLv3 was not in the licenses collected for recipe gcc
WARNING: The license listed NCSA was not in the licenses collected for recipe gcc-sanitizers
WARNING: The license listed MIT was not in the licenses collected for recipe gcc-sanitizers
WARNING: The license listed GPL-3.0-with-GCC-exception was not in the licenses collected for recipe gcc
WARNING: The license listed GPLv3 was not in the licenses collected for recipe gcc
WARNING: The license listed NCSA was not in the licenses collected for recipe gcc-sanitizers
By referring to PN, as used by license.bbclass, this issue is resolved.
(From OE-Core rev: dd4a319efcfa1174ba0f48fd15cb446280c426e4)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
In http://git.yoctoproject.org/cgit.cgi/poky/commit/?id=2d89cff42af2bb0049224bfaaebaa2b21966169f
we added a workaround for dealing with lack of time sync between build
machines and their users.
This has turned out to cause problems for people who rely on timestamps
being preserved in sstate output.
Since our autobuilders are all in time sync with ntp, revert the commit.
[YOCTO #8996]
(From OE-Core rev: 004a8bffdb501a99530dab2bc1c5f3d44b97b8f7)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
useradd has sstate [depends] for both do_package and do_populate_sysroot
yet the dependency validation code only covers do_package.
Add coverage of populate_sysroot, else the order inversion that [depends]
creates means unexpected installation of users of useradd.bbclass (e.g.
avahi do_populate_sysroot) in cases where it shouldn't be (e.g.
libnss-mdns -c packagedata).
The code needs to move above the other populate_sysroot intercept code
since there are specific cases we need to cover before that code.
The result of this change is more optimal installation of sstate objects
in common usage scenarios.
(From OE-Core rev: 21368b4a7934b3a199e61c1072416fd5c091e5ff)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
When cleaning old builds from the sysroots, also print the sysroot architecture.
(From OE-Core rev: 98a98a4bf71f74b5c27291ee9a2ac14006377e49)
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
If there are lots of changes between the previous build and the build about to
start bitbake will potentially print pages of:
DEBUG: Stamp $BUILD/stamps/corei7-64-poky-linux/libdrm/2.4.66-r0 is not reachable, removing related manifests
Instead of spamming the console with this list, write the list of manifests only
to the debug log and simply write a count to the console. This way the user
doesn't get spammed but still knows what is happening if bitbake appears to
stall with heavy I/O.
(From OE-Core rev: e777b541c76dad293d1c214e46c00f8f78fe0539)
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
If we "bitbake X -c packagedata" and the packagedata comes from sstate, we
don't need any of the tasks dependencies. This is similar to the
populate_lic case, we only care about the end result.
Therefore short circuit the dependencies so packagedata doesn't pull in
any other dependencies.
(From OE-Core rev: 76aa4f49db7d32bbd35703c64470fdfc63f403a4)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
It was observed that do_fetch dependencies (e.g. subversion-native of tremor)
were being installed even when sstate was available and hence no fetch was
needed. This turned out to be due to the recursive nature of the rootfs
dependencies which include populate_lic.
We can explicitly whitelist these dependencies as being ones we don't
need to follow when installing sstate (the code defaults to being paranoid).
[YOCTO #8746]
(From OE-Core rev: 06c891f0fa6c7712eae233dd18612f1ab1889c45)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Maintaining an sstate mirror is complicated at the moment as only the main
sstate tgz files have their timestamp updated when they're used.
This causes problems as the siginfo files can be removed from the server
due to inactivity but the sstate fetch code tries to fetch them leading
to confusing warnings and reduced debug capability.
This change ensures we touch any sig/siginfo files present and should
help ageing of the objects on sstate mirrors. It doesn't handle the intermediate
task siginfo files but those are never fetched by the current sstate code
so are an order of magnitude less problematic.
(From OE-Core rev: 80b3974081c4a8c604e23982a6db8fb32c616058)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The package feed signing code supports the user providing the path to the gpg
binary and an alternative gpg 'home' (usually ~/.gnupg), which are useful for
both deployment and QA purposes.
Factor out the gpg command line construction to a function which can fetch both
of these variables, and also use pipes.quote() to sanitise the arguments when
used in a shell context.
[ YOCTO #8559 ]
(From OE-Core rev: 6daf138822bbbc46960121d3b76b42eaf19e7c0e)
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
To avoid races between the sstate tasks/hooks using ${B} as the cwd, and other
tasks such as cmake_do_configure which deletes and re-creates ${B}, ensure that
all sstate hooks are run in the right directory, and run the prefunc/postfunc in WORKDIR.
(From OE-Core rev: 1d3bde02641f4b40030cf7e305ee3d7c2faabe29)
(From OE-Core rev: a026dbfa0ce4ee769ae2c714f27f4c4eaa4a0c73)
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
It is useful in a few different contexts to see which files have been
written out by an sstate task; break out a function that lets us get the
path to the manifest file easily.
(From OE-Core rev: 090196dd2d8f4306b34b239e78c39d37cc86034c)
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Only fetch remote signatures if verification has been enabled, as otherwise the
fetcher throws errors that sstate.bbclass can't ignore.
[ YOCTO #8265 ]
(From OE-Core rev: aa8ba5c4af2636dbc2d0c1ad9c230e8aa26d9962)
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
To provide some element of integrity to sstate archives, allow sstate archives
to be GPG signed with a specified key (detached signature to a sidecar .sig
file), and verify the signatures when sstate archives are unpacked.
(From OE-Core rev: 237b6c51b42b0c64434dc45685e10f757ac939c2)
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* Rename from gtk-update-icon-cache to gtk-icon-utils to better
reflect the content. Fix references in other recipes and classes
* Upgrade to GTK+ 3.16.4
* Add gtk-encode-symbolic-svg binary: it is used by icon themes
(e.g. Adwaita) to generate png versions of svg icons.
* Depend on librsvg-native for gtk-encode-symbolic-svg
* Add a patch that removes Gdk dependency from gtk-encode-symbolic-svg:
this way the native build stays slim.
(From OE-Core rev: 5f4fe36f735ff8b97c5112b16bd90e794053573d)
Signed-off-by: Jussi Kukkonen <jussi.kukkonen@intel.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Update sstate_create_package and sstate_unpack_package to remove
redundant initial mkdir and cd commands. The working directory is
now setup correctly before the shell functions are called.
(From OE-Core rev: fd2d50d46e38d86829f88b5884a6fde6c38f155e)
Signed-off-by: Andre McCurdy <armccurdy@gmail.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Use FetcherConnectionCache to improve times when do checkstatus over
sstate resources.
[YOCTO #7796]
(From OE-Core rev: e6f66370c6ce15aca18ef64491bab3dc92b80c57)
Signed-off-by: Aníbal Limón <anibal.limon@linux.intel.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Add init/end helper functions for ThreadWorker also pass ThreadWorker
as first argument to init/end/func functions this enables per-thread
storage handling.
classes/sstate.bbclass: Add thread_worker argument to checkstatus
function.
(From OE-Core rev: 08c50d62b520c8405f034e3d7adeea89e06226ee)
Signed-off-by: Aníbal Limón <anibal.limon@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
In order to fix Thread leakage caused by not call join() in Threads,
Pass num_tasks in ThreadPool for add all the tasks into a Queue this
enable catch of Queue.Empty exception and exit the threads.
classes/sstate.bbclass: Change checkstatus function to match new
ThreadPool operation.
(From OE-Core rev: 524d92ed7b53bef933527095e82f378b934f25ef)
Signed-off-by: Aníbal Limón <anibal.limon@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Rather than just use d.getVar(X), use the more explict d.getVar(X, False)
since at some point in the future, having the default of expansion would
be nice. This is the first step towards that.
This patch was mostly made using the command:
sed -e 's:\(getVar([^,()]*\)\s*):\1, False):g' -i `grep -ril getVar *`
(From OE-Core rev: ab7c1d239b122c8e549e8112c88fd46c9e2b061b)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
"Incremental builds do not work well when renaming recipes or changing
architecture" is a long standing issue which causes people considerable
pain. We've struggled for a long time to come up with a way to
generically address the problem.
There are additional issues where removal of a layer caused data to
continue to exist and additionally, changing DISTRO_FEATURES also caused
problems in an existing TMPDIR.
This patch attempts to address this by adding a mapping between stamp
files and manifests. After parsing we can easily tell which stamp files
are still reachable, if any manifest has a stamp that can no longer be
reached, we can remove it. Since this code ties this to the sstate
architecture list, it will not remove data from other than the current
MACHINE (and its active architectures). It does not clean the sstate
cache so if another build activates something which was cleaned, it
should reinstall from sstate.
We can also go one step further, depending on the setting of
SSTATE_PRUNE_OBSOLETEWORKDIR, workdirs which are no longer active can
also be removed. This avoids the buildup of many old copies of data in
WORKDIR for example when versions are upgraded.
The one thing which may surprise people with this change is if you
remove a layer, data added by that layer will be "uninstalled" before
the next build continues. I believe this is a feature and a good thing
to do though.
This code is safe with existing builds. If something isn't in the new
index it simply isn't removed. Since changes to the sstate code trigger
a rebuild, after this merges, we can assume the code will start to
detect changes from that point onwards.
[YOCTO #4102]
(From OE-Core rev: 4ea39427eedeadd51439a62fa015c86be30c3445)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The build can pause whilst remote sstate mirrors are checked for
sstate objects. Inform the user this is happening.
(From OE-Core rev: 4f4cfe8a4a0c602d6e11ea9bb75b52241deda9d6)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Currently the urls are checked serially which is a performance bottleneck when
looking at http:// urls in particular. This adds code to check the url status in
parallel, mirroring the way we do this elsewhere.
We need the datastore for the fetcher so we use threads, not multiprocess.
(From OE-Core rev: 77c4865bbde4cd2a061cf333f9ad798afc6de0ef)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Changing TMPDIR and rebuilding an image was resulting in rebuilds of
kernels due to dependencies on the shared_workdir task. If installed
from sstate, nothing needs this task so add it to the whitelisted
task patterns.
After this change, the kernel does not rebuild when a new TMPDIR and
hot sstate cache is used.
(From OE-Core rev: 624f575298c74272b69560735cb6f13f5a7761c1)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
In some cases we want to test the availability of siginfo files, in some
cases we do not and really want the .tgz files (which may or may
not be present too). This makes adds a parameter to the function to allow
this.
(From OE-Core rev: e8867b97a772f8e64fe209436c15d382b2175ca0)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
FILESPATH was only being overridden in one fetch location, it should be
equally handled in both.
Also use SSTATE_DIR as FILESPATH so that mirror urls which do remapping
can search the local SSTATE_DIR for other paths.
Also ensure that MIRRORS is removed in both locations, previously
it was only unset in one but both codepaths should be consistent.
(From OE-Core rev: ab6bebddbdefec323e284b6438d9c57b3d8a2cc3)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Without this patch the source archiver class is not allowed to
archive the same source archive for different builds.
(From OE-Core rev: 37284185912d496a974d507bde7208d93eefc29d)
Signed-off-by: Adrian <adrian.freihofer@gmail.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Commit e9672387 split one long line into a multi-line string, but in
the process white space between words was lost. This results in badly
formatted output when this message is printed.
(From OE-Core rev: b145374c0a498de0160a9b81f50ce0066ab14862)
Signed-off-by: Peter Urbanec <openembedded-devel@urbanec.net>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
For some recipes that inhrient cmake, the ${B} may be removed by
cmake_do_configure() while sstate_hardcode_path() running, this
causes build errors:
Exception: OSError: [Errno 2] No such file or directory: \
'/path/to/build'
The function sstate_hardcode_path() called command:
$SSTATE_SCAN_CMD which extended as "find ${SSTATE_BUILDDIR} ..."
So the proper function dirs could be ${SSTATE_BUILDDIR}.
(From OE-Core rev: a949943e622b08485fc7632a0a743bc009079c67)
Signed-off-by: Wenzong Fan <wenzong.fan@windriver.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The too long line would cause "git send-email" report errors:
patch contains a line longer than 998 characters
Though we can use "--no-validate" to force the send.
(From OE-Core rev: e96723879eb3352a5bdea7b3e1a576edf9550e5a)
Signed-off-by: Robert Yang <liezhi.yang@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The code:
bb.fatal("foo1")
bb.fatal("foo2")
Would make the second one not work, use bb.error for first one to fix
the problem.
(From OE-Core rev: 33a87187a8520e190bcade76cc965aa58faaa85a)
Signed-off-by: Robert Yang <liezhi.yang@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The packagegroup allarch rpm files for multilib can overwrite each other since
they are in theory indentical (in contrast to the other backends). We therefore
need to whitelist this to avoid build failures now this overwrite failure
is fatal.
(From OE-Core rev: d59ade0ca2cf629937434fa423dfbf35ce1209fc)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Update the sstate file's timestamps after it is installed, it will be
very useful for removing the old sstate file, especially, it's not easy
to remove when use the shared SSTATE_DIR, we can easily remove them with
this change, for example:
$ find state-cache -type f -ctime +10 -exec rm -f {} \;
Will remove the sstate file which isn't used by recent 10 days.
We can use the -atime, but it is not always available, for example,
when mounted with "-o noatime".
The touch is a very light weight action, and the
scripts/sstate-cache-management.sh also requires this.
(From OE-Core rev: bbee747466a6947319cff2ffd676abf9432c16ae)
Signed-off-by: Robert Yang <liezhi.yang@windriver.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
When files overlap in the sysroot, something bad usually happened. We've had
two independent cases recently where a couple of months after one of these
warnings was shown, builds failed due to corruption.
This change moves the warning to become a fatal error. The complaint I've had
about this is that we need to tell the user what happened and more importantly
how to recover from it. If we could recover from it, great but the trouble is
we simply don't know what happened.
As a compromise, we can document several of the possible scenarios in the error
message. We don't normally go to this level of detail however in this case, I'm
lacking other viable alternatives.
I do believe it is important to stop as corruption occurs rather than letting the
build contunue into territory that is not deterministic amongst other things.
The complex message is followed by a simpler one in case the long message is too
much for the user.
(From OE-Core rev: 179ac7de03977b6e440409eddb2166819e07286a)
(From OE-Core rev: 4b503f25f1ef8f554d3c76d88399db379dc818cc)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
I've been giving things some thought, specifically why sstate doesn't
get used more and why we have people requesting external toolchains. I'm
guessing the issue is that people don't like how often sstate can change
and the lack of an easy way to lock it down.
Locking it down is actually quite easy so patch implements some basics
of how you can do this (for example to a specific toolchain). With an
addition like this to local.conf (or wherever):
SIGGEN_LOCKEDSIGS = "\
gcc-cross:do_populate_sysroot:a8d91b35b98e1494957a2ddaf4598956 \
eglibc:do_populate_sysroot:13e8c68553dc61f9d67564f13b9b2d67 \
eglibc:do_packagedata:bfca0db1782c719d373f8636282596ee \
gcc-cross:do_packagedata:4b601ff4f67601395ee49c46701122f6 \
"
the code at the end of the email will force the hashes to those values
for the recipes mentioned. The system would then find and use those
specific objects from the sstate cache instead of trying to build
anything.
Obviously this is a little simplistic, you might need to put an override
against this to only apply those revisions for a specific architecture
for example. You'd also probably want to put code in the sstate hash
validation code to ensure it really did install these from sstate since
if it didn't you'd want to abort the build.
This patch also implements support to add to bitbake -S which dumps the
locked sstate checksums for each task into a ready prepared include file
locked-sigs.inc (currently placed into cwd). There is a function,
bb.parse.siggen.dump_lockedsigs() which can be called to trigger the
same functionality from task space.
A warning is added to sstate.bbclass through a call back into the siggen
class to warn if objects are not used from the locked cache. The
SIGGEN_ENFORCE_LOCKEDSIGS variable controls whether this is just a warning
or a fatal error.
A script is provided to generate sstate directory from a locked-sigs file.
(From OE-Core rev: 7e14784f2493a19c6bfe3ec3f05a5cf9797a2f22)
(From OE-Core rev: 884d4fa3e77cf32836f14a113c11489076f4a84d)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The "grep -e (x|y)" doesn't work, for example:
$ echo xy | grep -e '(x|y)'
No output
We can use "grep -E" (extended regexp) or "grep -e x -e y" to fix it.
It only affected the cross recipes.
(From OE-Core rev: 62722de6d0ec00608eacc2cb0396362aced00047)
Signed-off-by: Robert Yang <liezhi.yang@windriver.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The use of [ and && here means $? is reset and the exit 1 error
interception wasn't working, leading to "file changed as we read it"
errors from sstate_create_package when heavily using hardlinks.
Fix this by placing $? into a variable.
(From OE-Core rev: 6e51f900b76b06c09a3d6927f8db7398e2c035ed)
(From OE-Core rev: 75040a098e11927e6872e3a2a6286fe3ed0c7f47)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
In some cases we do either need to add extra sstate manipulation
functions, or change the existing modification functions. This patch
parametrizes it to SSTATEPOSTCREATEFUNCS after sstate_create_package
(From OE-Core rev: ed5fb9c69fcf0f6e8e98f72ac753c66119468ea5)
Signed-off-by: Hongxu Jia <hongxu.jia@windriver.com>
Signed-off-by: Saul Wold <sgw@linux.intel.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
In some cases we do either need to add extra sstate manipulation
functions, or change the existing modification functions. This patch
parametrises them to SSTATECREATEFUNCS and SSTATEPOSTUNPACKFUNCS and
abstracts the "hardcoded path" functions into separate functions using
these new variables.
We may use this new functionality to improve binary relocating using
patchelf for example, this at least lets us have the hooks to be able to
experiment.
(From OE-Core rev: 9d659c6f20fa4a141b491c62a3ef0dfb1f896d9c)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
If we have done do_package_qa_setscene we do not also need
do_package_setscene. This means we can again for example rebuild an image
without needing do_package sstate packages to be present.
(From OE-Core rev: a0f584ac3d5a94dec121b684206ecd40c968f7fc)
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Currently if you try and switch between linux/darwin/mingw SDK builds in the
same TMPDIR, things break. This is due to sstate not reflecting the SDK_OS in
the manifest names.
Since they are different, reflect this in the manifest naming and allow
this to work.
(From OE-Core rev: 7a0e1233ba196797f5f9bf862685dfae5511e751)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
When the preconfigure task was added to gcc, it wasn't added to sstate's list
of possible shared work tasks. This meant that diffsigs wasn't able to find
the preconfigure sigdata/info file since it has inconsistent naming.
This adds the task name to the list. Ideally this list would be autogenerated
or not even required, right now its a sanity test that the shared work
code works as intended so is best left as is.
(From OE-Core rev: 72032f6dd6724663a3417b1d1b666d9a63fcbfdb)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
With the renaming of the cross packages, its no longer possible to use
endswith("-cross") and similar to detect cross packages. Replace these
references with other techniques.
This resolves certain build from sstate failures which were due to the
system believing cross packages were target packages and therefore
dependency handling was altered.
(From OE-Core rev: 91edf4cac223298e50a4b8e59dd19f1b272e3418)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Signed-off-by: Saul Wold <sgw@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
This allows them to co-exist together in the native sysroot, with one
set of cross tools per target architecture.
(From OE-Core rev: a2c5509520d5c3e082f55844e6545d0309565f8f)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
We recently added multiple provider warnings. These could get issued
from -cross recipes due to the way these currently overlap in the
native sysroot. Filter out these warnings for now, until such times
as we improve the cross recipes so they don't have an overlapping
namespace.
(From OE-Core rev: d62f8d428831cc62b395f4c764c1e876353ddede)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
A while ago we stopped supporting sstate names which were different from the
taskname. This patch finishes cleaning up some code remnants from that
which were causing data duplication and confusion.
(From OE-Core rev: 5a19863e389d28d8db4a86e409c1daa3c6b46eff)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Looking at the code, its clear 'task' is meant not to have the do_ prefix,
however its also clear it can be left in through some code paths.
One result of this can be files not being cleaned from the sysroot correctly.
Fix this.
(From OE-Core rev: 3c912102fd9ea6b360f1af209da21e8f27b845c0)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
This is a patch to update the missed sstate event with
info about the sstate files locations that were found.
It's needed as to display the found file in the toaster ui.
Also fixes a bug where a setscene task may have appeared in the
missed list even if it was found in a sstate mirror.
(From OE-Core rev: ad66cd521d3e661dd57c5aa02c204585101984f3)
Signed-off-by: Alexandru DAMIAN <alexandru.damian@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Toaster needs to record the attempts to restore
setscene tasks that don't have a sstate file.
We build a list of tasks for which we can't find an
sstate file, and if we're running under Toaster data
collection, we send it off with a MetadataEvent.
(From OE-Core rev: 109ae6c5c981610ab0d63d2c83dcd50b2e93276b)
Signed-off-by: Alexandru DAMIAN <alexandru.damian@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
This was stomping over the value added in buildhistory.bbclass, leading
to task signatures *still* changing when buildhistory was added to
INHERIT.
Fixes [YOCTO #5897].
(From OE-Core rev: f7a0aa412a921a41d301be072cedcda2662ffbd7)
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Saul Wold <sgw@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Bitbake can't interpret the python execution of these functions so
include the dependency manually.
[YOCTO #4507]
(From OE-Core rev: 39c53d26a1ea0859ca30456b6af0a28b2ea16769)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
I have a feeling this code exists from the time before we had proper
coverage of one sstate task by another task. At that time it was a
"poor" persons version of that idea, we now have much better
code internal to bitbake which handles this.
Worse, this code actually breaks certain rebuild scenarios,
e.g.:
bitbake libtool-cross
bitbake libtool-cross -c cleansstate
rm tmp -rf
bitbake libtool-cross
would fail as binutils-cross wasn't installed from sstate.
The easiest fix is to remove the obsolete/broken code.
[YOCTO #5773]
(From OE-Core rev: ccad07f35fb7f959e24fd50d04c7d10dd5cf20d0)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The code here that deletes stamps was also deleting the taint files; so
forcing an sstate task with -f would force it to execute and then
because the taint file was deleted in the process, the next execution
would simply restore the output from sstate again. We need to exclude
the taint files just like we did in bb.build.make_stamp().
Fixes [YOCTO #5805].
(From OE-Core rev: 4708859e5627488251dc4250d45cb5f4e9736b8a)
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
After the change to allow target recipes to depend on native recipes, the
native checksums becomes all the more critical. Add to this that we're now
accounting for pre/postfuncs and we have a cache reuse issue since the
distro LSB string is getting coded in when it shouldn't be.
This excludes that string and allows one set of native sstate to share
checksums with another set from a different host distro. They're separated
into different directories so this is fine for our use cases.
(From OE-Core rev: 1fa7d4331d994b9eeb6f973d1a1f04cb4df92c13)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Now that bitbake is recursing into pre/postfuncs, we need to ensure
the dependencies of these functions is correct. We don't want dependencies
on MACHINE or other related variables. This patch adds in appropriate
variable exclusions to achieve this.
(From OE-Core rev: 8461283a648d7c5affd51971ebd9b35a8a4c625f)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
There is a potential problem if we don't remove the previous version's
stamp, for example:
The depend chain is:
libtool-native -> autoconf-native -> m4-native
We have two m4-native: 1.4.9 and 1.4.7
1) Clean all of them to make a fresh build so that we can reproduce the
problem
$ bitbake m4-native autoconf-native libtool-native -ccleansstate
2) Build libtool-native so that the m4-native_1.4.17 will be built
$ bitbake libtool-native
3) Set PREFERRED_VERSION_m4-native = "1.4.9" and build again
$ bitbake libtool-native
4) Set PREFERRED_VERSION_m4-native = "1.4.17" and build again
$ bitbake libtool-native -ccleansstate && bitbake libtool-native
Then the build will fail:
[snip]
| m4: unrecognized option '--gnu'
| Try `m4 --help' for more information.
| autom4te: m4 failed with exit status: 1
[snip]
The is because when we change m4-native to 1.4.17 and build
libtool-native again:
5) libtool-native depends on autoconf-native, and autoconf-native's
version isn't change, so it can remove the current stamp and mirror
the sstate (the one depends on m4-native_1.4.9) from the SSTATE_DIR
correctly.
6) The mirrored autoconf-native depends on m4-native_1.4.17's
do_populate_sysroot, and the stamp is already there (which is made
by step 2), so it would do nothing, but this is incorrect, since
the one that really in the sysroot is m4-native_1.4.9, then the
error happens.
Remove previous version's stamp in sstate_clean() will fix the problem.
[YOCTO #5422]
(From OE-Core rev: 4659d29b1040349116549644e45035a5b37d9311)
Signed-off-by: Robert Yang <liezhi.yang@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
do_package doesn't exist and are noexec anyway for native/cross/crosssdk
packages.
(From OE-Core rev: 1028ac813fa9803ebfff6bcfa7f8b67012609b27)
Signed-off-by: Ming Liu <ming.liu@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
This patch mainly aims to add optimisation for crosssdk setscene dependency
validating which we haven't handled in current logic, and which I think we
could have as we've already implemented to native/cross, although there
are albeit not many crossdk tasks, we could still get some performance
enhancement.
(From OE-Core rev: 1094983ff87a8b745a5bc7bfe9514433ee3c4ad2)
Signed-off-by: Ming Liu <ming.liu@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The way the "all arch" PKGSPEC is enabled, it causes corruption
of the datastore of sstate operations against other tasks.
Data store copies are cheap and allow us to use that trick, resetting
to a clean copy of the data afterwards.
(From OE-Core rev: afaf16100efa79a275a2f4b9f2caa80decfdeb81)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Like fetch, unpack and patch, populate_lic doesn't vary between different
archs so we should mark it as such. This means better sstate cache reuse
with fewer duplicate files as well as less confusing sstate debugging.
sstatesig also needs to account for the fact BPN is used for sstate files
in these cases.
(From OE-Core rev: 3d59d0bed756f64d0092caa3892239c779c4a341)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Currently the code has problems differentiating between "gcc-cross" and "gcc-cross-initial"
sstate files. We could add in a ton of special casing but tests show this isn't scaling
well. Using a more unique separator resolves the issue.
The choice of which separator to use is a hard one. We need something which isn't commonly
used in PN, PV, PR, *_OS and *_ARCH which rules out '-', '_' and it needs to work ok with
webservers/http which makes ';' and '%' harder.
The change also sets SSTATE_SWSPEC globally since writing out differently named siginfo
files for the fetch/unpack/patch tasks is a waste of diskspace, the hashes match for
all PN in the majority of cases and if they don't, its not a big issue as the hash is
different. This makes the results from sstate debugging more understandable.
(From OE-Core rev: 6f823a23c5f1d0ffa0a27db1c1bc1907de788505)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
When originally developed, it was thought a task may have more than one associated
sstate archive. The way the code has grown that idea is now not possible or needed.
We can therefore assume one sstate archive per task and drop the crazy name
mapping code. Simpler is better in this case.
The downside is that various sstate archives will change name so this forces a cache
rebuild. Given the other sstate changes going in at this time, this isn't really
a bad thing as things would rebuild anyway.
(From OE-Core rev: 5afe86a6854b21692fd97c5fc7fab50dbc068acb)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
For a shared workdir, any one of the fetch/unpack/patch tasks may run yet the
PN and architecture fields in SSTATE_PKGSPEC may differ. This makes looking up
the appropriate siginfo file near impossible.
I've tried several different ways of resolving this and this is the neatest
solution I could find, its still rather ugly. I believe the usefulness of
better sstate debugging outweighs the ugliness of the code.
This patch also changes the sstate_checkhashes() code to look for siginfo
files rather than the actual sstate packages themselves. This means the
function can be used in other contexts to find info files for tasks that
may not have sstate data. It is assumed that sstate mirrors will have both
files available. This is done to allow bitbake to query whether tasks have
matching signatures in sstate directories or not.
(From OE-Core rev: 068e4289b597699cbff2dfde44ba833af4535281)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Currently siginfo files are only written for sstate tasks. In order to be truly
debuggable, its helpful to have the siginfo for intermediate tasks. This
adds that functionality so the extra siginfo files are written out too.
This will be used to add better sstate debugging in future changes.
(From OE-Core rev: 04d108cd16f5ad8f92a62ea537d1330fee712470)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
sstate_package creates hardlink from sysroot to SSTATE_BUILDDIR, and
sstate_create_package stores SSTATE_BUILDDIR into a archive file by
tar, these two task can be run simultaneously for different packages,
and make a hardlink for a file will lead to the change of the links
number of file, and if tar is reading this file, it will fail with exit
code 1, and report "file changed as we read it":
DEBUG: Executing shell function sstate_create_package
tar: x86_64-linux/usr/share/aclocal/xorg-macros.m4: file changed as we read it
4b3e353a5[sstate.bbclass: fix parallel building issue] tries to use the
tar parameter --ignore-failed-read to fix, but it does not work, and
tar parameter --warning=no-file-changed can close the warning, but can
not change the exit code. so close shell immediate exit, only fail
if tar returns not 1 and 0.
Exit codes of tar:
http://www.gnu.org/software/tar/manual/html_section/Synopsis.html
(From OE-Core rev: fad604b719e00b03e09da5fdb485e72332275b4a)
Signed-off-by: Roy Li <rongqing.li@windriver.com>
Signed-off-by: Saul Wold <sgw@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
sstate_package creates hardlink from sysroot to SSTATE_BUILDDIR, then
sstate_create_package will store SSTATE_BUILDDIR into a archive file by
tar, but once other packages install the same file into sysroot, the
creating the archive file will fail with below error:
DEBUG: Executing shell function sstate_create_package
tar: x86_64-linux/usr/share/aclocal/xorg-macros.m4: file changed as we read it
This kind of error is harmless, use --ignore-failed-read to ignore it.
The error in tar occurs when the timestamp of the file changes and this
can happen when the number of symlinks change. The file will be included
in the archive.
[YOCTO #5122]
(From OE-Core rev: 4b3e353a532c7b68b0bb86df4a2fcc44f8bb3ef2)
Signed-off-by: Roy.Li <rongqing.li@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
These have been deprecated for a long time, convert the remaining
references to the correct modules and prepare for removal of the
compatibility support from bitbake.
(From OE-Core rev: 6a39835af2b2b3c7797fe05479341d71a3f3aaf6)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
ant reported on irc that the sstate absolute to relative symlink creation
code wasn't working in klibc. He was correct although the level of breakage is
rather surprising since it only worked for one level of symlink (usr/include) with
everything else being broken.
The reason is probably that nothing really uses absolute paths, we use relative
paths where at all possible already. Nothing in the target sysroot should use
absolute paths for a start. In this regard, the klibc-dev package is broken and
needs fixing. It will currently break when building for one machine, then switching
to another of the same TUNE_PKGARCH and installing from sstate but that is a
separate issue.
This patch fixes the symlink creation code by firstly passing in the correct
value we need (where the symlink will end up) and seccondly, actually using it.
I've also tweaked the debug message to contain appropriate information and got
right of the double "//" value the existing code created in favour of the form
'./..' which looks neater.
(From OE-Core rev: 9b05c65450526522d7358d0c0901b594de546748)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
BB_NO_NETWORK disables any fetching, however if we're using an external
sstate cache, we may want to be able to fetch those objects even if we are
not fetching the upstream sources. Denote this situation by setting
SSTATE_MIRROR_ALLOW_NETWORK in local.conf. When it is found, for sstate
cache fetches, mask off BB_NO_NETWORK for the local function.
(From OE-Core rev: ed585cad2e1fdc323c05fa82055a071bcf98d1bc)
Signed-off-by: Joe MacDonald <joe.macdonald@windriver.com>
Signed-off-by: Saul Wold <sgw@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
This avoids unnecessary duplication of setup. The only visible change in
behavior will be the case if siginfo exists and the archive does not, in which
case it'll redownload both, but this doesn't seem unresasonable to me,
particularly since the archive is downloaded first, making this case
particularly unlikely.
(From OE-Core rev: aa4991c307d4bbdd06c3cbf8448240b74c5e01c4)
Signed-off-by: Christopher Larson <chris_larson@mentor.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The default FILESPATH isn't really of use, as we don't expect to find sstate
archives buried in layer recipe directories, and the default MIRRORS is
intended for use for fetching SRC_URI, not sstate.
(From OE-Core rev: 46402b2f5b69004751f6663d435bedae0ad9dab1)
Signed-off-by: Christopher Larson <chris_larson@mentor.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
