Currently the urls are checked serially which is a performance bottleneck when
looking at http:// urls in particular. This adds code to check the url status in
parallel, mirroring the way we do this elsewhere.
We need the datastore for the fetcher so we use threads, not multiprocess.
(From OE-Core rev: 77c4865bbde4cd2a061cf333f9ad798afc6de0ef)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Changing TMPDIR and rebuilding an image was resulting in rebuilds of
kernels due to dependencies on the shared_workdir task. If installed
from sstate, nothing needs this task so add it to the whitelisted
task patterns.
After this change, the kernel does not rebuild when a new TMPDIR and
hot sstate cache is used.
(From OE-Core rev: 624f575298c74272b69560735cb6f13f5a7761c1)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
In some cases we want to test the availability of siginfo files, in some
cases we do not and really want the .tgz files (which may or may
not be present too). This makes adds a parameter to the function to allow
this.
(From OE-Core rev: e8867b97a772f8e64fe209436c15d382b2175ca0)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
FILESPATH was only being overridden in one fetch location, it should be
equally handled in both.
Also use SSTATE_DIR as FILESPATH so that mirror urls which do remapping
can search the local SSTATE_DIR for other paths.
Also ensure that MIRRORS is removed in both locations, previously
it was only unset in one but both codepaths should be consistent.
(From OE-Core rev: ab6bebddbdefec323e284b6438d9c57b3d8a2cc3)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Without this patch the source archiver class is not allowed to
archive the same source archive for different builds.
(From OE-Core rev: 37284185912d496a974d507bde7208d93eefc29d)
Signed-off-by: Adrian <adrian.freihofer@gmail.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Commit e9672387 split one long line into a multi-line string, but in
the process white space between words was lost. This results in badly
formatted output when this message is printed.
(From OE-Core rev: b145374c0a498de0160a9b81f50ce0066ab14862)
Signed-off-by: Peter Urbanec <openembedded-devel@urbanec.net>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
For some recipes that inhrient cmake, the ${B} may be removed by
cmake_do_configure() while sstate_hardcode_path() running, this
causes build errors:
Exception: OSError: [Errno 2] No such file or directory: \
'/path/to/build'
The function sstate_hardcode_path() called command:
$SSTATE_SCAN_CMD which extended as "find ${SSTATE_BUILDDIR} ..."
So the proper function dirs could be ${SSTATE_BUILDDIR}.
(From OE-Core rev: a949943e622b08485fc7632a0a743bc009079c67)
Signed-off-by: Wenzong Fan <wenzong.fan@windriver.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The too long line would cause "git send-email" report errors:
patch contains a line longer than 998 characters
Though we can use "--no-validate" to force the send.
(From OE-Core rev: e96723879eb3352a5bdea7b3e1a576edf9550e5a)
Signed-off-by: Robert Yang <liezhi.yang@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The code:
bb.fatal("foo1")
bb.fatal("foo2")
Would make the second one not work, use bb.error for first one to fix
the problem.
(From OE-Core rev: 33a87187a8520e190bcade76cc965aa58faaa85a)
Signed-off-by: Robert Yang <liezhi.yang@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The packagegroup allarch rpm files for multilib can overwrite each other since
they are in theory indentical (in contrast to the other backends). We therefore
need to whitelist this to avoid build failures now this overwrite failure
is fatal.
(From OE-Core rev: d59ade0ca2cf629937434fa423dfbf35ce1209fc)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Update the sstate file's timestamps after it is installed, it will be
very useful for removing the old sstate file, especially, it's not easy
to remove when use the shared SSTATE_DIR, we can easily remove them with
this change, for example:
$ find state-cache -type f -ctime +10 -exec rm -f {} \;
Will remove the sstate file which isn't used by recent 10 days.
We can use the -atime, but it is not always available, for example,
when mounted with "-o noatime".
The touch is a very light weight action, and the
scripts/sstate-cache-management.sh also requires this.
(From OE-Core rev: bbee747466a6947319cff2ffd676abf9432c16ae)
Signed-off-by: Robert Yang <liezhi.yang@windriver.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
When files overlap in the sysroot, something bad usually happened. We've had
two independent cases recently where a couple of months after one of these
warnings was shown, builds failed due to corruption.
This change moves the warning to become a fatal error. The complaint I've had
about this is that we need to tell the user what happened and more importantly
how to recover from it. If we could recover from it, great but the trouble is
we simply don't know what happened.
As a compromise, we can document several of the possible scenarios in the error
message. We don't normally go to this level of detail however in this case, I'm
lacking other viable alternatives.
I do believe it is important to stop as corruption occurs rather than letting the
build contunue into territory that is not deterministic amongst other things.
The complex message is followed by a simpler one in case the long message is too
much for the user.
(From OE-Core rev: 179ac7de03977b6e440409eddb2166819e07286a)
(From OE-Core rev: 4b503f25f1ef8f554d3c76d88399db379dc818cc)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
I've been giving things some thought, specifically why sstate doesn't
get used more and why we have people requesting external toolchains. I'm
guessing the issue is that people don't like how often sstate can change
and the lack of an easy way to lock it down.
Locking it down is actually quite easy so patch implements some basics
of how you can do this (for example to a specific toolchain). With an
addition like this to local.conf (or wherever):
SIGGEN_LOCKEDSIGS = "\
gcc-cross:do_populate_sysroot:a8d91b35b98e1494957a2ddaf4598956 \
eglibc:do_populate_sysroot:13e8c68553dc61f9d67564f13b9b2d67 \
eglibc:do_packagedata:bfca0db1782c719d373f8636282596ee \
gcc-cross:do_packagedata:4b601ff4f67601395ee49c46701122f6 \
"
the code at the end of the email will force the hashes to those values
for the recipes mentioned. The system would then find and use those
specific objects from the sstate cache instead of trying to build
anything.
Obviously this is a little simplistic, you might need to put an override
against this to only apply those revisions for a specific architecture
for example. You'd also probably want to put code in the sstate hash
validation code to ensure it really did install these from sstate since
if it didn't you'd want to abort the build.
This patch also implements support to add to bitbake -S which dumps the
locked sstate checksums for each task into a ready prepared include file
locked-sigs.inc (currently placed into cwd). There is a function,
bb.parse.siggen.dump_lockedsigs() which can be called to trigger the
same functionality from task space.
A warning is added to sstate.bbclass through a call back into the siggen
class to warn if objects are not used from the locked cache. The
SIGGEN_ENFORCE_LOCKEDSIGS variable controls whether this is just a warning
or a fatal error.
A script is provided to generate sstate directory from a locked-sigs file.
(From OE-Core rev: 7e14784f2493a19c6bfe3ec3f05a5cf9797a2f22)
(From OE-Core rev: 884d4fa3e77cf32836f14a113c11489076f4a84d)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The "grep -e (x|y)" doesn't work, for example:
$ echo xy | grep -e '(x|y)'
No output
We can use "grep -E" (extended regexp) or "grep -e x -e y" to fix it.
It only affected the cross recipes.
(From OE-Core rev: 62722de6d0ec00608eacc2cb0396362aced00047)
Signed-off-by: Robert Yang <liezhi.yang@windriver.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The use of [ and && here means $? is reset and the exit 1 error
interception wasn't working, leading to "file changed as we read it"
errors from sstate_create_package when heavily using hardlinks.
Fix this by placing $? into a variable.
(From OE-Core rev: 6e51f900b76b06c09a3d6927f8db7398e2c035ed)
(From OE-Core rev: 75040a098e11927e6872e3a2a6286fe3ed0c7f47)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
In some cases we do either need to add extra sstate manipulation
functions, or change the existing modification functions. This patch
parametrizes it to SSTATEPOSTCREATEFUNCS after sstate_create_package
(From OE-Core rev: ed5fb9c69fcf0f6e8e98f72ac753c66119468ea5)
Signed-off-by: Hongxu Jia <hongxu.jia@windriver.com>
Signed-off-by: Saul Wold <sgw@linux.intel.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
In some cases we do either need to add extra sstate manipulation
functions, or change the existing modification functions. This patch
parametrises them to SSTATECREATEFUNCS and SSTATEPOSTUNPACKFUNCS and
abstracts the "hardcoded path" functions into separate functions using
these new variables.
We may use this new functionality to improve binary relocating using
patchelf for example, this at least lets us have the hooks to be able to
experiment.
(From OE-Core rev: 9d659c6f20fa4a141b491c62a3ef0dfb1f896d9c)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
If we have done do_package_qa_setscene we do not also need
do_package_setscene. This means we can again for example rebuild an image
without needing do_package sstate packages to be present.
(From OE-Core rev: a0f584ac3d5a94dec121b684206ecd40c968f7fc)
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Currently if you try and switch between linux/darwin/mingw SDK builds in the
same TMPDIR, things break. This is due to sstate not reflecting the SDK_OS in
the manifest names.
Since they are different, reflect this in the manifest naming and allow
this to work.
(From OE-Core rev: 7a0e1233ba196797f5f9bf862685dfae5511e751)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
When the preconfigure task was added to gcc, it wasn't added to sstate's list
of possible shared work tasks. This meant that diffsigs wasn't able to find
the preconfigure sigdata/info file since it has inconsistent naming.
This adds the task name to the list. Ideally this list would be autogenerated
or not even required, right now its a sanity test that the shared work
code works as intended so is best left as is.
(From OE-Core rev: 72032f6dd6724663a3417b1d1b666d9a63fcbfdb)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
With the renaming of the cross packages, its no longer possible to use
endswith("-cross") and similar to detect cross packages. Replace these
references with other techniques.
This resolves certain build from sstate failures which were due to the
system believing cross packages were target packages and therefore
dependency handling was altered.
(From OE-Core rev: 91edf4cac223298e50a4b8e59dd19f1b272e3418)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Signed-off-by: Saul Wold <sgw@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
This allows them to co-exist together in the native sysroot, with one
set of cross tools per target architecture.
(From OE-Core rev: a2c5509520d5c3e082f55844e6545d0309565f8f)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
We recently added multiple provider warnings. These could get issued
from -cross recipes due to the way these currently overlap in the
native sysroot. Filter out these warnings for now, until such times
as we improve the cross recipes so they don't have an overlapping
namespace.
(From OE-Core rev: d62f8d428831cc62b395f4c764c1e876353ddede)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
A while ago we stopped supporting sstate names which were different from the
taskname. This patch finishes cleaning up some code remnants from that
which were causing data duplication and confusion.
(From OE-Core rev: 5a19863e389d28d8db4a86e409c1daa3c6b46eff)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Looking at the code, its clear 'task' is meant not to have the do_ prefix,
however its also clear it can be left in through some code paths.
One result of this can be files not being cleaned from the sysroot correctly.
Fix this.
(From OE-Core rev: 3c912102fd9ea6b360f1af209da21e8f27b845c0)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
This is a patch to update the missed sstate event with
info about the sstate files locations that were found.
It's needed as to display the found file in the toaster ui.
Also fixes a bug where a setscene task may have appeared in the
missed list even if it was found in a sstate mirror.
(From OE-Core rev: ad66cd521d3e661dd57c5aa02c204585101984f3)
Signed-off-by: Alexandru DAMIAN <alexandru.damian@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Toaster needs to record the attempts to restore
setscene tasks that don't have a sstate file.
We build a list of tasks for which we can't find an
sstate file, and if we're running under Toaster data
collection, we send it off with a MetadataEvent.
(From OE-Core rev: 109ae6c5c981610ab0d63d2c83dcd50b2e93276b)
Signed-off-by: Alexandru DAMIAN <alexandru.damian@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
This was stomping over the value added in buildhistory.bbclass, leading
to task signatures *still* changing when buildhistory was added to
INHERIT.
Fixes [YOCTO #5897].
(From OE-Core rev: f7a0aa412a921a41d301be072cedcda2662ffbd7)
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Saul Wold <sgw@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Bitbake can't interpret the python execution of these functions so
include the dependency manually.
[YOCTO #4507]
(From OE-Core rev: 39c53d26a1ea0859ca30456b6af0a28b2ea16769)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
I have a feeling this code exists from the time before we had proper
coverage of one sstate task by another task. At that time it was a
"poor" persons version of that idea, we now have much better
code internal to bitbake which handles this.
Worse, this code actually breaks certain rebuild scenarios,
e.g.:
bitbake libtool-cross
bitbake libtool-cross -c cleansstate
rm tmp -rf
bitbake libtool-cross
would fail as binutils-cross wasn't installed from sstate.
The easiest fix is to remove the obsolete/broken code.
[YOCTO #5773]
(From OE-Core rev: ccad07f35fb7f959e24fd50d04c7d10dd5cf20d0)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The code here that deletes stamps was also deleting the taint files; so
forcing an sstate task with -f would force it to execute and then
because the taint file was deleted in the process, the next execution
would simply restore the output from sstate again. We need to exclude
the taint files just like we did in bb.build.make_stamp().
Fixes [YOCTO #5805].
(From OE-Core rev: 4708859e5627488251dc4250d45cb5f4e9736b8a)
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
After the change to allow target recipes to depend on native recipes, the
native checksums becomes all the more critical. Add to this that we're now
accounting for pre/postfuncs and we have a cache reuse issue since the
distro LSB string is getting coded in when it shouldn't be.
This excludes that string and allows one set of native sstate to share
checksums with another set from a different host distro. They're separated
into different directories so this is fine for our use cases.
(From OE-Core rev: 1fa7d4331d994b9eeb6f973d1a1f04cb4df92c13)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Now that bitbake is recursing into pre/postfuncs, we need to ensure
the dependencies of these functions is correct. We don't want dependencies
on MACHINE or other related variables. This patch adds in appropriate
variable exclusions to achieve this.
(From OE-Core rev: 8461283a648d7c5affd51971ebd9b35a8a4c625f)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
There is a potential problem if we don't remove the previous version's
stamp, for example:
The depend chain is:
libtool-native -> autoconf-native -> m4-native
We have two m4-native: 1.4.9 and 1.4.7
1) Clean all of them to make a fresh build so that we can reproduce the
problem
$ bitbake m4-native autoconf-native libtool-native -ccleansstate
2) Build libtool-native so that the m4-native_1.4.17 will be built
$ bitbake libtool-native
3) Set PREFERRED_VERSION_m4-native = "1.4.9" and build again
$ bitbake libtool-native
4) Set PREFERRED_VERSION_m4-native = "1.4.17" and build again
$ bitbake libtool-native -ccleansstate && bitbake libtool-native
Then the build will fail:
[snip]
| m4: unrecognized option '--gnu'
| Try `m4 --help' for more information.
| autom4te: m4 failed with exit status: 1
[snip]
The is because when we change m4-native to 1.4.17 and build
libtool-native again:
5) libtool-native depends on autoconf-native, and autoconf-native's
version isn't change, so it can remove the current stamp and mirror
the sstate (the one depends on m4-native_1.4.9) from the SSTATE_DIR
correctly.
6) The mirrored autoconf-native depends on m4-native_1.4.17's
do_populate_sysroot, and the stamp is already there (which is made
by step 2), so it would do nothing, but this is incorrect, since
the one that really in the sysroot is m4-native_1.4.9, then the
error happens.
Remove previous version's stamp in sstate_clean() will fix the problem.
[YOCTO #5422]
(From OE-Core rev: 4659d29b1040349116549644e45035a5b37d9311)
Signed-off-by: Robert Yang <liezhi.yang@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
do_package doesn't exist and are noexec anyway for native/cross/crosssdk
packages.
(From OE-Core rev: 1028ac813fa9803ebfff6bcfa7f8b67012609b27)
Signed-off-by: Ming Liu <ming.liu@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
This patch mainly aims to add optimisation for crosssdk setscene dependency
validating which we haven't handled in current logic, and which I think we
could have as we've already implemented to native/cross, although there
are albeit not many crossdk tasks, we could still get some performance
enhancement.
(From OE-Core rev: 1094983ff87a8b745a5bc7bfe9514433ee3c4ad2)
Signed-off-by: Ming Liu <ming.liu@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The way the "all arch" PKGSPEC is enabled, it causes corruption
of the datastore of sstate operations against other tasks.
Data store copies are cheap and allow us to use that trick, resetting
to a clean copy of the data afterwards.
(From OE-Core rev: afaf16100efa79a275a2f4b9f2caa80decfdeb81)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Like fetch, unpack and patch, populate_lic doesn't vary between different
archs so we should mark it as such. This means better sstate cache reuse
with fewer duplicate files as well as less confusing sstate debugging.
sstatesig also needs to account for the fact BPN is used for sstate files
in these cases.
(From OE-Core rev: 3d59d0bed756f64d0092caa3892239c779c4a341)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Currently the code has problems differentiating between "gcc-cross" and "gcc-cross-initial"
sstate files. We could add in a ton of special casing but tests show this isn't scaling
well. Using a more unique separator resolves the issue.
The choice of which separator to use is a hard one. We need something which isn't commonly
used in PN, PV, PR, *_OS and *_ARCH which rules out '-', '_' and it needs to work ok with
webservers/http which makes ';' and '%' harder.
The change also sets SSTATE_SWSPEC globally since writing out differently named siginfo
files for the fetch/unpack/patch tasks is a waste of diskspace, the hashes match for
all PN in the majority of cases and if they don't, its not a big issue as the hash is
different. This makes the results from sstate debugging more understandable.
(From OE-Core rev: 6f823a23c5f1d0ffa0a27db1c1bc1907de788505)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
When originally developed, it was thought a task may have more than one associated
sstate archive. The way the code has grown that idea is now not possible or needed.
We can therefore assume one sstate archive per task and drop the crazy name
mapping code. Simpler is better in this case.
The downside is that various sstate archives will change name so this forces a cache
rebuild. Given the other sstate changes going in at this time, this isn't really
a bad thing as things would rebuild anyway.
(From OE-Core rev: 5afe86a6854b21692fd97c5fc7fab50dbc068acb)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
For a shared workdir, any one of the fetch/unpack/patch tasks may run yet the
PN and architecture fields in SSTATE_PKGSPEC may differ. This makes looking up
the appropriate siginfo file near impossible.
I've tried several different ways of resolving this and this is the neatest
solution I could find, its still rather ugly. I believe the usefulness of
better sstate debugging outweighs the ugliness of the code.
This patch also changes the sstate_checkhashes() code to look for siginfo
files rather than the actual sstate packages themselves. This means the
function can be used in other contexts to find info files for tasks that
may not have sstate data. It is assumed that sstate mirrors will have both
files available. This is done to allow bitbake to query whether tasks have
matching signatures in sstate directories or not.
(From OE-Core rev: 068e4289b597699cbff2dfde44ba833af4535281)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Currently siginfo files are only written for sstate tasks. In order to be truly
debuggable, its helpful to have the siginfo for intermediate tasks. This
adds that functionality so the extra siginfo files are written out too.
This will be used to add better sstate debugging in future changes.
(From OE-Core rev: 04d108cd16f5ad8f92a62ea537d1330fee712470)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
sstate_package creates hardlink from sysroot to SSTATE_BUILDDIR, and
sstate_create_package stores SSTATE_BUILDDIR into a archive file by
tar, these two task can be run simultaneously for different packages,
and make a hardlink for a file will lead to the change of the links
number of file, and if tar is reading this file, it will fail with exit
code 1, and report "file changed as we read it":
DEBUG: Executing shell function sstate_create_package
tar: x86_64-linux/usr/share/aclocal/xorg-macros.m4: file changed as we read it
4b3e353a5[sstate.bbclass: fix parallel building issue] tries to use the
tar parameter --ignore-failed-read to fix, but it does not work, and
tar parameter --warning=no-file-changed can close the warning, but can
not change the exit code. so close shell immediate exit, only fail
if tar returns not 1 and 0.
Exit codes of tar:
http://www.gnu.org/software/tar/manual/html_section/Synopsis.html
(From OE-Core rev: fad604b719e00b03e09da5fdb485e72332275b4a)
Signed-off-by: Roy Li <rongqing.li@windriver.com>
Signed-off-by: Saul Wold <sgw@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
sstate_package creates hardlink from sysroot to SSTATE_BUILDDIR, then
sstate_create_package will store SSTATE_BUILDDIR into a archive file by
tar, but once other packages install the same file into sysroot, the
creating the archive file will fail with below error:
DEBUG: Executing shell function sstate_create_package
tar: x86_64-linux/usr/share/aclocal/xorg-macros.m4: file changed as we read it
This kind of error is harmless, use --ignore-failed-read to ignore it.
The error in tar occurs when the timestamp of the file changes and this
can happen when the number of symlinks change. The file will be included
in the archive.
[YOCTO #5122]
(From OE-Core rev: 4b3e353a532c7b68b0bb86df4a2fcc44f8bb3ef2)
Signed-off-by: Roy.Li <rongqing.li@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
These have been deprecated for a long time, convert the remaining
references to the correct modules and prepare for removal of the
compatibility support from bitbake.
(From OE-Core rev: 6a39835af2b2b3c7797fe05479341d71a3f3aaf6)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
ant reported on irc that the sstate absolute to relative symlink creation
code wasn't working in klibc. He was correct although the level of breakage is
rather surprising since it only worked for one level of symlink (usr/include) with
everything else being broken.
The reason is probably that nothing really uses absolute paths, we use relative
paths where at all possible already. Nothing in the target sysroot should use
absolute paths for a start. In this regard, the klibc-dev package is broken and
needs fixing. It will currently break when building for one machine, then switching
to another of the same TUNE_PKGARCH and installing from sstate but that is a
separate issue.
This patch fixes the symlink creation code by firstly passing in the correct
value we need (where the symlink will end up) and seccondly, actually using it.
I've also tweaked the debug message to contain appropriate information and got
right of the double "//" value the existing code created in favour of the form
'./..' which looks neater.
(From OE-Core rev: 9b05c65450526522d7358d0c0901b594de546748)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
BB_NO_NETWORK disables any fetching, however if we're using an external
sstate cache, we may want to be able to fetch those objects even if we are
not fetching the upstream sources. Denote this situation by setting
SSTATE_MIRROR_ALLOW_NETWORK in local.conf. When it is found, for sstate
cache fetches, mask off BB_NO_NETWORK for the local function.
(From OE-Core rev: ed585cad2e1fdc323c05fa82055a071bcf98d1bc)
Signed-off-by: Joe MacDonald <joe.macdonald@windriver.com>
Signed-off-by: Saul Wold <sgw@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
This avoids unnecessary duplication of setup. The only visible change in
behavior will be the case if siginfo exists and the archive does not, in which
case it'll redownload both, but this doesn't seem unresasonable to me,
particularly since the archive is downloaded first, making this case
particularly unlikely.
(From OE-Core rev: aa4991c307d4bbdd06c3cbf8448240b74c5e01c4)
Signed-off-by: Christopher Larson <chris_larson@mentor.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The default FILESPATH isn't really of use, as we don't expect to find sstate
archives buried in layer recipe directories, and the default MIRRORS is
intended for use for fetching SRC_URI, not sstate.
(From OE-Core rev: 46402b2f5b69004751f6663d435bedae0ad9dab1)
Signed-off-by: Christopher Larson <chris_larson@mentor.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The do_package_setscene task needs to depend on fakeroot in order to correctly
install its files.
We can whitelist the dependency in the sstate handling code for some
performance improvements since we only need this if we're installing the
package from sstate.
Also use an append operator in base.bbclass for clarity.
(From OE-Core rev: 0810ea2a72bdea67a3d8002c4e12fb20f45cf1d5)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
sstate has a logic to fixup hardcoded paths in scripts,
but it misses in some specific cases, so add
EXTRA_STAGING_FIXMES to the fixup hardcoded paths
mechanism, so that we can specify what hardcoded paths
need to be fixed in a recipe, e.g.
EXTRA_STAGING_FIXMES = "STAGING_BINDIR_TOOLCHAIN"
(From OE-Core rev: 2e840db56c45b4c63fded55f4ed763b7099284b9)
Signed-off-by: Jackie Huang <jackie.huang@windriver.com>
Signed-off-by: Saul Wold <sgw@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
In python 3 print is a function call. In some cases bb.note is a more
appropriate call to make.
(From OE-Core rev: 754874e68d1037c76fc4cc7d35d3fa12946d00f8)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Make hard links for staging files instead of copy to save the disk space
(3G will be saved for a core-image-sato build), and it doesn't affect
much on the build time.
The following directories are affected:
1) The sysroot
2) The DEPLOY_DIR
3) The pkgdata
[YOCTO #4372]
(From OE-Core rev: 5853e0f482b22258c909268fe71673a29e31989b)
Signed-off-by: Robert Yang <liezhi.yang@windriver.com>
Signed-off-by: Saul Wold <sgw@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Otherwise we might end up creating directories under sstate-cache with whatever
random umask has been selected for the task that we're trying to package. This
would be a bad thing since it might result in losing group write access for
newly created dirs, and/or losing group read access for the sstate files
themselves.
(From OE-Core rev: d8c4f442c41bf3ac5e064630657cd3fa1b5c43b1)
Signed-off-by: Phil Blundell <philb@gnu.org>
Signed-off-by: Saul Wold <sgw@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
If no files have been staged we want to continue without error instead
of showing a traceback.
Fixes [YOCTO #4056].
(From OE-Core rev: ca36be708e54c0c86535bc8512295c76c48f6cf5)
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Under some conditions (ACLs enabled, NFS) mv can interactively prompt
before overwriting files. Avoid hanging builds in that case by using
-f which should be harmless in other cases.
(From OE-Core rev: b1a085db9d8ad2a3117af6f50e510bc9c2f9407b)
Signed-off-by: Jeff Polk <jeff.polk@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
In case filenames have spaces, execution of the function
sstater_install will hang, so the print parameter %s must be
enclosed with quotes.
(From OE-Core rev: 545d7aa26dfefdc927e0f4e2cc37398ef2c63fa6)
Signed-off-by: Leonardo Sandoval <leonardo.sandoval@freescale.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
When checking the dependencies in setscene_depvalid(), make sure we also
consider those dependencies needed when running the postinstalls on
host.
[YOCTO #3918]
(From OE-Core rev: 8de0616825ed1b238b3486077af6897834bcb62d)
Signed-off-by: Laurentiu Palcu <laurentiu.palcu@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
This directory is cleaned upon completion however if a previous build
crashes, it can lead to corrpution, hence ensure its clean at the start
too.
(From OE-Core rev: 8ef0e59d5a7da3671d1ad9a54fe068ed78f928d5)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Add a function which copys a tree as a set of hardlinks to the original
files, then use this in sstate to reduce some of the overhead of sstate
package creation since the file isn't actually copied.
(From OE-Core rev: 8e373e69acac853213a62afb8bbdf0adc0c5045a)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The same log message gets output multiple times in the log which look
confusing and is rather pointless. Move the log message to the correct
level.
(From OE-Core rev: 3917409004a830e7ad0646f05ad7421385cbd1de)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
useradd.bbclass adds sstate dependencies on base-passwd, shadow-native and
shadow-sysroot. Due to the way these are injected, they interact badly with
the other dependency validation logic and end up pulling in dependencies we
don't actually need. This patch adds code to optimise those cases out.
(From OE-Core rev: 784ca68fcca4ffb34390d55d9343570cfdf0305f)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Currently, do_rootfs has a dependency on all the do_package output being present
due to its usage of the pkgdata directories. This means that if you run:
bitbake xxxx-image -c rootfs
you end up having to fetch and unpack all the do_package data which is usually
large and inefficient. It also means rm_work has to leave all the do_package
data lying around so rootfs works.
This patch splits the actual creation of the pkgdata directory off into a separate
task, "packagedata" which happens immediately after do_package. We can then remap
the dependencies so this task is depended upon, not do_package. Sstate can then be
programmed not to require do_package at the appropriate times.
Whilst this patch doesn't do so, it opens the possibility of rm_work wiping
out the do_package output from WORKDIR as long as it also removed the do_package
stamp (both normal and setscene variants) and allowing more space savings
with rm_work which has been regularly requested.
(From OE-Core rev: 6107ee294afde395e39d084c33e8e94013c625a9)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Specify dirs in which the shell function "sstate_create_package" and
"sstate_unpack_package" are executed and don't use ${B} as default dirs
to avoid possible race with task do_rootfs at deb image creation time.
[YOCTO #3674]
(From OE-Core rev: ccef1cf783669a4683eda9d4b44dbe6bcf426259)
Signed-off-by: Hongxu Jia <hongxu.jia@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
If you change a machine to a different package architecture, you will see sstate
errors about overwriting files as the code stands today. Instead it should clean
out the files safely and correctly. This patch changes the naming of stamp-extra-info
manifest files to avoid this problem. It will potentially trigger warnings during
builds in existing TMPDIRs until the system adjusts to the new naming, these are
harmless.
[YOCTO #3521]
(From OE-Core rev: 2cc8ee57f8148844bb7bcd4aaf34f6891cf3d410)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The code was written before SSTATETASKNAMES existed. Since it
does exist, lets simply the code.
(From OE-Core rev: 9817e2efdb94395655d711f5eadedcd249c8cffe)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
This would be useful for doing siginfo compares to understand why a build
is not reusing something when using SSTATE_MIRROR. No error will be reported
if it fails to find the .siginfo file
[YOCTO #2898]
[RP: Small tweaks]
(From OE-Core rev: 6d86690330f0d43839b904fced4b4b02cb27b8c6)
Signed-off-by: Saul Wold <sgw@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* without this patch:
Python 2.7.3
>>> sstate_search_cmd = "grep -rl /OE/jansa-test/shr-core/tmp-eglibc/pkgdata/armv7a-vfp-neon-oe-linux-gnueabi/runtime-reverse/vim-common /OE/jansa-test/shr-core/tmp-eglibc/sstate-control --exclude=master.list | sed -e 's:^.*/::' -e 's:\.populate-sysroot::'"
>>> cmd_array = sstate_search_cmd.split(' ')
>>> search_output = subprocess.Popen(cmd_array, stdout=subprocess.PIPE).communicate()[0]
grep: |: No such file or directory
grep: sed: No such file or directory
* Adding shell=True and using cmd string instead of array makes it work:
>>> search_output = subprocess.Popen(sstate_search_cmd, shell=True, stdout=subprocess.PIPE).communicate()[0]
>>> print search_output
manifest-armv7a-vfp-neon-gvim.package
manifest-armv7a-vfp-neon-vim-tiny.package
manifest-armv7a-vfp-neon-vim.package
But still isn't 100% reliable, I guess it's caused by some other package
being removed from sstate while grep is already running.
So sometimes grep can show error on STDERR
>>> search_output = subprocess.Popen(sstate_search_cmd, shell=True, stdout=subprocess.PIPE).communicate()[0]
grep: /OE/jansa-test/shr-core/tmp-eglibc/sstate-control/manifest-armv7a-vfp-neon-systemtap.package: No such file or directory
(From OE-Core rev: d84f7d7a12b4271f7b2bfde9fb356d750abff15d)
Signed-off-by: Martin Jansa <Martin.Jansa@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The value of subprocess.Popen().communicate()[0] is a string.
Checking for '!= None' will always match causing bogus warnings
regarding already staged files.
(From OE-Core rev: acdd76482efc3caf80e9e0b7359be7ca724ae09a)
Signed-off-by: Enrico Scholz <enrico.scholz@sigma-chemnitz.de>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Currently there is some odd behaviour of the packagegroup class in relation
to sstate since it sets PACKAGE_ARCH = "all" but does not use the allarch class
leading to it being undetected by sstate.
Previously it was not possible to use allarch as the recipe couldn't "undo"
settings made by the allarch class. Since this no longer happens when
PACKAGE_ARCH != all, we can use the allarch class.
This patch also fixes up one case we need to preserve TRANSLATED_TARGET_ARCH
and ensures sstate only assumes allarch when PACKAGE_ARCH is "all".
(From OE-Core rev: 591fa7c1ab9e9ff75fdce602c77ecdeda3a255d9)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Clean up and clarify the populate_sysroot task dependencies. Target sysroot
packages do need their dependencies installed, as do some target/cross
relationships. We can whitelist the *-initial dependencies as these are
never needed indirectly.
(From OE-Core rev: eeec307917234d97be2674beeadef71599fb1487)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
For example gcc-cross depends on linux-libc-headers and needs it to be present
to build/work correctly.
(From OE-Core rev: 43ce7a1d86bf82d976ad241057a4207b1a340b3b)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
This is a first attempt at logic to determine when a sstate dependency needs
to be installed and when it does not. Its a start at the logic and errs on the
side of caution, as it gets wider testing, we can refine the logic as needed.
This code should allow a significant performance speedup to certain workflows, for
example "bitbake xxx-image -c rootfs" will not populate the target sysroot.
(From OE-Core rev: b43faba37816817edc5240a139361d16e07c6131)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The recent preveeding sstate directory layout fixes made the code do what it
was originally intended to do, as can be clearly seen from the code.
Unfortunately this changed the contents and layout of the sstate files
themselves since the bug was leading to a directory prefix being missing.
This is now resulting in chaotic messages on the console since things
are getting confused with the two different layouts. The simplest way to
resolve this is to bump the version number, hence moving the new layout
into its own new namespace.
Its worth noting that whilst the failure messages are scary, the failure
mode is relatively harmless since it will just fall back to building the
data rather than installing from sstate.
Usually I'd give more notice of a change like this but under the
circumstances, I'm just going to push this in to resolve the failures
people are seeing. Initially I thought the problem was limited to
some of the -cross packages and therefore of low impact but that is
clearly not the case.
(From OE-Core rev: b53ea6687b6201c8c5ab5cb0d2a845ef7e7b2abe)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Fix missing parameter to endswith and pass paths through normpath to remove
any duplicate "/" characters which would corrupt other calls like basename.
(From OE-Core rev: 172a74c540378149eec493c37c030e9f42f9603d)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The manifest file can become corrupted if sstate-inputdirs and sstate-outputdirs
don't have matching endings. This patch ensures that even if set incorrectly,
the code functions as intended, thereby handling manifest corruption safely.
(From OE-Core rev: 0109a3623a19f9ae289952a4f054e53c3eca4eaa)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Present the manifest file that contains the matches for
files being installed to a location that already contains
that file. This will help to determine which is the correct
recipe to fix when this occurs.
[YOCTO #3191]
(From OE-Core rev: 56268f6e4ed1fc11143173bb1717a8be78c728a5)
Signed-off-by: Saul Wold <sgw@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Now do_package isn't machine specific, we're only left with do_populate_sysroot as a
machine specific task. This change marks only the machine specific manifests as machine
specific, defaulting to PACKAGE_ARCH for everything else.
This means we do less work where there are multiple machines using the same
core package architecture and we can start to clean up the sstate duplicate files
whitelist.
(From OE-Core rev: febeaf3d1b8917b660c7279b008d8b03337568e9)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
We've noticed failures on the project autobuilders where a shared sstate
directory is used across multiple builders and the clocks become skewed.
Most of the time this causes harmless building but if this happens where
an environment is changed (make install vs make in qt4-x11-free for example),
the build can fail.
This avoids modification times in the future and should make builds safer
in shared environments sstate was designed for.
(From OE-Core rev: 8f1bdb4f4afd7f5f4c121be8ba82f4675f73e300)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
This avoids errors where gcc/binutils get installed to the native sysroot
in the same location for multiple package architectures. Ultimately making
these native recipes with ${PACKAGE_ARCH} appended to PN will resolve this
but hide the warnings until this gets sorted out.
Also hide the python and docbook catalog warnings since they're known about,
nothing to worry about and we'll aim to clean them up properly in the 1.4 cycle.
(From OE-Core rev: 5bae58a5b59c04d8947f4842f19837a914c29b52)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Without this, path components like // could break comparisions with the whitelist leading
to warnings being displayed to the user unintentionally.
(From OE-Core rev: d3c46ca56fab2f07bf16b61514f30765543a8747)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
do_package is a machine specific task at the moment due to packagedata. This means
do_package tasks and their dependencies rerun between different machines
with various duplicate file installations. There are plans to fix this but they're
too invasive before release.
This patch relaxes the whitelist for sstate duplicate file detection to account
for this. Post-release, we re-enable stricter settings once do_package is not
machine specific.
(From OE-Core rev: c858259ce1881c6284f1fc2790c225c81e4a751e)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Signed-off-by: Saul Wold <sgw@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
This was added to allow detection of duplicate files being installed by sstate.
There is a much simpler way, just check if the file already exists. This
effectively uses the kernel VFS as the cache which is much more efficient.
This resolves a significant performance bottleneck (lock contention on a
single file) when running builds that are just being generated from sstate
cache files.
(From OE-Core rev: 603daf343ad3f18c8adb799e3625ae2a18d94f56)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
We need to split this variable before using it. Otherwise a single "/"
character in the list whitelists every overlapping sysroot file which
was not the intention making the whole thing useless.
We'll start seeing warnings about overlapping files now this is working
correctly after this patch.
(From OE-Core rev: 9e31c748327e92b809330f4ad7b6aaecb2edf559)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Appending to EXTRA_SSTATEMAPS is better than just hardcoding a value. Also
add a comment about why this is necessary.
(From OE-Core rev: d4f4a57b8d564d57256017d937ed2eabf94c36ae)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
I've not idea why this got left in but as per the comment, it needs fixing
and we shouldn't have hardcoded mappings like this. Lets remove it
and dynamically generate the data instead.
[YOCTO #3039]
(From OE-Core rev: 2df064ad46c1510fa8a401c22db4ab3278c3c807)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
This fixes builds where master.list doesn't already exist. This
change was meant to be part of the previous sstate commit but ended
up separated.
(From OE-Core rev: c2109b765b24a7ffe4781257ad3fe4641a3b2a49)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
We have a potential problem where two sstate packages try and touch the same
file. This adds code which will print a warning whenever this happens.
The implementation does but by maintaining a master file list and comparing
file accesses against this. There are a number of places we have duplicate
accesses which are harmless, mostly in the deploy directory so these
are whitelisted.
For now the code prints warnings, this could be strengthened in future to
become error messages. Whilst working on this code on and off over the past
few months various issues were uncovered, some serious.
[YOCTO #238]
(From OE-Core rev: 1f1b6f93d6b7aa8c9bd9bb5b1826997812e36932)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
It was pointed out we have a number of weird indentations in the python functions.
This patch cleans up 3, 7 and other weird indentations for the core bbclass files.
It also fixes some wierd (odd) shell function indentation which my searches picked up.
(From OE-Core rev: 8385d6d74624000d68814f4e3266d47bc8885942)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
This patch adds the lsb name and revision to the path used for sstate files.
This means that reuse of sstate files between different distributions is restricted
by default. The behaviour can be configured using mirror urls, for example:
SSTATE_MIRRORS = "file://Ubuntu-11.10/(.*) file://Ubuntu/\1 \n"
would map Ubuntu 11.10 to a more generic "Ubuntu" named sstate feed.
Usually, more modern distros have increased libc versions for example
so whilst more older native/cross sstate packages will usually work on newer
distros, the opposite is not true. This patch allows development of policy
to better handle this although no default policy is currently being used.
(From OE-Core rev: 9360fc318cec5c873b17f59b817dd6312c81ee27)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Currently all sstate files are placed into one directory. This does not scale and
causes a variety of filesystem issues. This patch adds a two character subdirectory
to the layout (based on the first two characters of the hash) so that files
can be split into several directories.
This should help performance of sstate in most cases.
(From OE-Core rev: 5d45c7b025f6635b2232d7bf92b7c1aba350396b)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
allarch sstate packages could be marked as machine or package_arch specific. This
change ensures they are not.
(From OE-Core rev: f3104240ad5bb542c339ee29b2672523ad3ae50c)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
There was a bug showing up where the crosssdk recipes were being installed as machine
specific in the sstate-control directory. This turned out to be due to the architecture
fields used by sstate being set incorrectly. The problem is that the crosssdk inherits
the cross class. It therefore needs to be listed in this if statement block before
the cross check, not after.
This should resolve some package architecture issue of crosssdk sstate files.
(From OE-Core rev: 79fe28e6033273f9632ca7549d1599d3fd1463ef)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Signed-off-by: Saul Wold <sgw@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Replace os.system with subprocess.call since the older function would
fail (more or less) silently if the executed program cannot be found
More info:
http://docs.python.org/library/subprocess.html#subprocess-replacements
[YOCTO #2454]
(From OE-Core rev: a07d03cc6f67c88feb9813ae7deb6e4a93552dfe)
Signed-off-by: Robert Yang <liezhi.yang@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Jason Wessel noticed that a package without any fixmepath entries would
generate a sed warning about no input files. This patch resolves that
by ensuring that an empty fixmepath file never gets written into the
sstate archive. Also we avoid a second message by only doing xargs if
we got input.
(From OE-Core rev: 326563d5a897ae2dba7cfd8d73579d3d979d72c8)
Signed-off-by: Mark Hatle <mark.hatle@windriver.com>
Signed-off-by: Jason Wessel <jason.wessel@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
In order to make sharing sstate files easier chmod them with 0664 permissions so that they are readable and writable by the user and any other members of their group.
(From OE-Core rev: dae9ad8a0ba0343e3083694cdcb20f0d02927ad0)
Signed-off-by: Joshua Lock <josh@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The fixmepath file that is generated contains a list of all of the files
that need their paths fixed. In the previous version the fixmepath was
generated to include all of the files that sed may have changed. In the
new version, we first grep the files to see if they contain a path that
needs to be changed, only then do we perform the sed operation on those files.
This results in a modest performance increate in the creation of the sstate
file. The following numbers include the do_package and do_populate_sysroot
tasks on the perl recipe.
Before the change:
real 4m23.018s
user 1m57.067s
sys 1m33.327s
After the change:
real 4m13.083s
user 1m54.062s
sys 1m26.064s
However, a more significnt performance gain is felt during the
extraction/install of sstate cache files, as the fixmepaths file now has a
significantly smaller list of files to modify.
Before the change:
real 0m39.798s
user 0m11.158s
sys 0m12.642s
After the change:
real 0m25.511s
user 0m8.408s
sys 0m5.077s
(All numbers above were recorded with a cold filesystem cache on a machine
with 12 GB of ram.)
(From OE-Core rev: 46067264bedeff8248a2b2441733420fe6651f84)
Signed-off-by: Mark Hatle <mark.hatle@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
In a pathological case, lots of files to process, the sstate_installpkg
performance was very poor. It interated over each file and ran 3
individual sed commands per file. Changing this to keep iterating
but running only a single command took about 1/3 time time.
However, when looking at the corresponding sstate_hardcode_path
function, it was clear we could optimize this further.
Using the same encoding logic to specify only the minimumal sed
operation necessary, and using xargs to avoid the os.system call the
install step was able to be performed in 13% of the original time.
Example timing numbers for perl:
3m7s original code
1m20s single sed, but interating
0m26s using xargs and limited sed
(From OE-Core rev: d9f655753fbdc8cbd8e705577430fed4f23732b3)
Signed-off-by: Mark Hatle <mark.hatle@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Reluctantly revert this since it breaks the tar-native workaround we have
for old versions of tar :(
This reverts commit 01218e29f9.
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
There is currently a race window when creating sstate packages since we don't
atomically write the files to SSTATE_DIR. This change ensures we do so by writing
to a temporary file and then doing an atomic move.
(From OE-Core rev: 52bf113e786a57123a9da98f64442afbc2f1471e)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
This resolves issues related to pigz-native when installing from sstate that people
have been seeing. It also gives us a way to solve issues like the gzip-native race
during sstate package creation covered in Yocto #1774.
(From OE-Core rev: 518dffe20178f5969dddccb17d6ab347afb72beb)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* also fixes replacing paths for perl where cmd line was probably
too long for os.system(cmd) (it had 560410 characters because a lot of
files from sstate_scan_cmd).
* also print those 2 commands so we can find them in log.do_package
(From OE-Core rev: 94c52d68fc2ce258bcc5b0978ac73413480a1a93)
Signed-off-by: Martin Jansa <Martin.Jansa@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* e.g. mysql5 has mysql_config not mysql-config
(From OE-Core rev: aebdffc93ba8446bbd4e20263f0f75d4d4460c47)
Signed-off-by: Martin Jansa <Martin.Jansa@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The sstate_hardcode_path() function triggered large numbers of exec()
calls when processing packages with large numbers of file relocations
(e.g. perl). This patch optimises those calls into longer single commands
which make the code significantly more efficient.
This reduced the do_package time for perl by 2 minutes (from 4.75 minutes)
for me.
(From OE-Core rev: 4159ff761cf29a03aeb56c7686a4e7af02b52219)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
We process all files in the native/cross cases for finding and
fixing relocation issues. In the target case we've only processed
.la and binconfig files. Since there are other files which are
in need of this processing, this change allows recipes to specify
files that may be outside the normal set. This means hardcoded
paths that need to be fixmepathed to work correctly are handled
and addresses some sstate relocation bugs that have been seen.
Based on a patch from Saul Wold <sgw@linux.intel.com>
(From OE-Core rev: 6ffdcd9120b572fa41659029c3bda7bf00ebcb77)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Complete the bb.data.getVar/setVar replacements with accesses
directly to the data store object.
(From OE-Core rev: 2864ff6a4b3c3f9b3bbb6d2597243cc5d3715939)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Without this change we can end up looking for <stamp>.${MACHINE}
instead of the expected expanded value.
(From OE-Core rev: 9f743b5033177216fe0e1d3e43ba831f356df08e)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
sstate was being a little too ethusiastic about removing stamp files and
was removing stamp files for other machines when it shouldn't have been.
This patch teaches sstate about machine specific stamp extensions and
allows it to only remove the current task's stampfiles.
Based on a patch from Phil Blundell <philb@gnu.org> with some tweaks
from me.
(From OE-Core rev: 5e9488495401399d39fcb5012b86c313b6caca73)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
If several sstate packages are decompressing at the same time, they can
conflict over the "fixmepath". If two packages try to write to this at
the same time it results in an error. To avoid this we remove the file
once we've finished processing it.
(From OE-Core rev: c08d7093bf654ffedb155c5627972e9ecfeb0b60)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Currently its hard to figure out if/when sstate is checking for possible
packages to speed up builds. This patch adds level 2 debug output which
better indicates what files are being searched for an why.
[YOCTO #1259]
(From OE-Core rev: a752b23767189f2678367e47ca0b41f49c56a631)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
This patch removes the variables BASE_PACKAGE_ARCH, BASEPKG_HOST_SYS,
BASEPKG_TARGET_SYS and also removes the immediate assignments in
several core classes as these are no longer required.
This should make it clearer what some of the core variables do and
simplfy some overly complex and confusing class code.
(From OE-Core rev: d5521be2dcbaf213c140b0d12a4176380874426b)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
We need to check if we know of the task type, before we attempt
to process it. In order to reproduce the problem build with:
PACKAGE_CLASSES = "package_ipk"
Then change it to:
PACKAGE_CLASSES = "package_rpm"
Build again -- and then try bitbake -c cleansstate <recipe>
(From OE-Core rev: 8870ba9d261d3cacbe5d1219fdd95840c05ecf9e)
Signed-off-by: Mark Hatle <mark.hatle@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Distributions can then override this variable if needed
useful e.g. when building uclibc/eglibc both using
same tmpdir
(From OE-Core rev: 839e6e7f3717cc69cf8fc5b1c51bac63ec00655e)
Signed-off-by: Khem Raj <raj.khem@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Fix [YOCTO #964]
A recent commit 25a6e5f9(sstate: use only unique set of SSTATETASK) breaks
the ordered mapping between SSTATETASKS and SSTATETASKNAMES. As a result,
in sstate_cleanall, the line
taskname = tasks[namemap.index(name)]
gets an incorrect result, and "bitbake -c cleanall" doesn't really remove
the files populalted by do_populate_sysroot.
(From OE-Core rev: 2f6505f0e795b6c8cad641a6918739c3faac1f99)
Signed-off-by: Dexuan Cui <dexuan.cui@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* otherwise strange error like this:
ERROR: Logfile of failure stored in: /OE/shr-core/tmp/work/armv7a-oe-linux-gnueabi/libtool-cross-2.4-r1/temp/log.do_package_write_ipk.25551
Log data follows:
| ERROR: Package already staged (/OE/shr-core/tmp/sstate-control/manifest-nokia900-libtool-cross.deploy-ipk)?!
| ERROR: Function 'sstate_task_postfunc' failed
NOTE: package libtool-cross-2.4-r1: task do_package_write_ipk: Failed
ERROR: Task 11 (/OE/shr-core/openembedded-core/meta/recipes-devtools/libtool/libtool-cross_2.4.bb, do_package_write_ipk) failed with exit code '1'
is shown in this case with package_ipk twice in INHERIT
* Thanks to Richard for fix
(From OE-Core rev: f2fe5e840b8aa0558b5462ef2c7517b2f14ec2ea)
Signed-off-by: Martin Jansa <Martin.Jansa@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
For safety, always delete the stamp files in sstate_clean regardless of
whether the manifest file exists or not.
(From OE-Core rev: f781c35da9a11eefdb06bda72ca89753df863efa)
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Avoid deleting stamp files whose names contain the current task's name as a
substring. This will be especially important for example if do_package_write
is ever made an sstate task (as it would previously have deleted the stamps
here for do_package_write_ipk etc.)
(From OE-Core rev: ea743ea30e2289733d27979e8ec921648342da0e)
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
When doing builds using sstate cache, there was no way to run
the equivalent of a pkg_postinst function. This is needed by
the SGML-related documentation recipes to properly update the
catalog files when new DTDs and stylesheets are installed.
SSTATEPOSTINSTFUNCS is a new variable you can set to function(s)
in your recipe to run after install is completed from sstate.
Thanks to Richard Purdie for suggesting this solution.
Signed-off-by: Scott Garman <scott.a.garman@intel.com>
Current sstate's fetch code doesn't reflect latest fetcher changes, so old
fetch style cause exception and fail silently.
[BUGID #708] got fixed.
Another issue is "import xxx" in python function from sstate.bbclass can only
sit in the head of the function, else have UnboundLocalError: "local variable
XXX referenced before assignment".
Signed-off-by: Zhai Edwin <edwin.zhai@intel.com>
