Commit Graph

77 Commits

Author SHA1 Message Date
Ross Burton
1fb99950a2 oe/utils: allow naming threads in ThreadedPool
When looking at logs involving thread pools it is useful if the threads
can be named.

(From OE-Core rev: 18342945b021608794d83ecf567afd43f4379b24)

Signed-off-by: Ross Burton <ross.burton@arm.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2021-11-25 21:55:10 +00:00
Ross Burton
f59811aeeb oe/utils: log exceptions in ThreadedWorker functions
If the function a ThreadedWorker is executing raises an exception, don't
use print() as that mostly disappears.  Instead, output it to the logger.

This is done using bb.mainlogger.debug directly instead of bb.debug() as
this allows us to pass the exception instance directly, which is then
incorporated into the log stream.

(From OE-Core rev: 2f1ea25c222b344dd8b784b2bc73a6540ab30274)

Signed-off-by: Ross Burton <ross.burton@arm.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2021-10-18 13:48:16 +01:00
Andres Beltran
157462fc2a buildhistory: Label packages providing per-file dependencies in depends.dot
Currently, depends.dot includes per-file dependencies but not the packages
providing those files. This makes it hard to obtain all package
dependencies by just looking at depends.dot.

Parse the RPROVIDES and FILERPROVIDES fields from pkgdata to map each of
their values to the package providing the component. Include runtime
packages as dependencies in depends.dot, together with the component
provided by the package as a label.

(From OE-Core rev: 2ba33093017574bbe29eeba699eb90628614d03a)

Signed-off-by: Andres Beltran <abeltran@linux.microsoft.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2021-09-01 14:06:18 +01:00
Ross Burton
3743234dab lib/oe/utils: add directory size function
For the purpose of image construction using du on a rootfs directory isn't
entirely satisfactory.  Bare "du" will report the actual disk usage so file
systems which can compress the data will report less than the actual space
required.  Using "du --apparent-size" will report the actual space used, but as
this simply sums the bytes used for content across an entire file system can
result in significant under-reporting due to block size overhead.

Attempt to solve these problems by implementing our own function to calculate
how large a rootfs will be.  This function handles hardlinks correctly but
rounds up all sizes to multiples of the block size (currently, 4KB is the
hard-coded block size).

(From OE-Core rev: 6ca53ad7b26ee2b4e6d2c121c6f6d6eed7f6b56f)

Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2021-03-28 22:28:26 +01:00
Tomasz Dziendzielski
ca608639f8 lib/oe/utils: Return empty string in parallel_make
In cmake.bbclass we set CMAKE_BUILD_PARALLEL_LEVEL using parallel_make
function and if PARALLEL_MAKE is set to empty string then this variable
is exported as "None" causing cmake to fail with:
"'CMAKE_BUILD_PARALLEL_LEVEL' environment variable
invalid number 'None' given."

(From OE-Core rev: 2f790ded554a52ac18d1c28002142f9c62abec8b)

Signed-off-by: Tomasz Dziendzielski <tomasz.dziendzielski@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2021-01-16 22:39:35 +00:00
Ross Burton
c2a11e18c8 utils: respect scheduler affinity in cpu_count()
cpu_count() returns multiprocessing.cpu_count() but that is simply returns
os.cpu_count() so we could use that directly.

However this returns the number of CPUs on the host, not the number of
usable CPUs on the host. If the user is using scheduler affinity then
the number of usable CPUs may be less, so when determining how many cores
we can use check the affinity instead.

(From OE-Core rev: e2e8ccbe410b2f38bcd9525982b2261cf71aab60)

Signed-off-by: Ross Burton <ross.burton@arm.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2020-09-30 15:01:51 +01:00
Tim Orling
d5d634f682 lib/oe/utils.py: add get_host_compiler_version()
Add helper function to get the host compiler and version.
Do not assume compiler is gcc.

NOTE: cannot set env to d.getVar("PATH") as that does not contain
the session PATH which was set by environment-setup-... which
breaks the install-buildtools use-case

(From OE-Core rev: 88712929354ff9c876bb1e48b6f15c33af5f2bbc)

Signed-off-by: Tim Orling <timothy.t.orling@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2020-04-01 11:44:24 +01:00
Charles-Antoine Couret
ad366e7383 utils: fix gcc 10 version detection
Utils can not detect GCC 10 correctly due to wrong regex.
It generates this error "ERROR: Can't get compiler version from gcc  --version output"

Sub-version numbers should be 1 or more digits instead of 1 only.

(From OE-Core rev: 186fe4a3d390a52b87282c3e694ce3251e45ee78)

Signed-off-by: Charles-Antoine Couret <charles-antoine.couret@mind.be>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2020-03-29 12:05:31 +01:00
André Draszik
0d7f24ce89 lib/oe/utils: allow to set a lower bound on returned cpu_count()
This will be needed for making xz compression more deterministic,
as xz archives are created differently in single- vs multi-threaded
modes.

This means that due to bitbake's default of using as many threads
as there are cores in the system, files compressed with xz
will be different if built on a multi-core system compared to
single-core systems.

Allowing cpu_count() here to return a lower bound, will allow
forcing xz to always use multi-threaded operation.

(From OE-Core rev: c6dda204e15c5b27559f7617b18a7b3ce604e2f8)

Signed-off-by: André Draszik <git@andred.net>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2020-03-06 08:19:17 +00:00
Daniel McGregor
a26efa2875 cmake: prefer CMAKE_BUILD_PARALLEL_LEVEL
cmake 3.12 introduced this environment variable. Prefer it to passing
PARALLEL_MAKE and PARALLEL_MAKEINST on the cmake command line, because
it gets passed to second stage cmake invocations while command-line
arguments do not (for example, multi-stage clang builds)

(From OE-Core rev: cdd44c93f02bb8cc2fa773e13c8ce36e3da23921)

Signed-off-by: Daniel McGregor <daniel.mcgregor@vecima.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2020-02-04 15:56:29 +00:00
Richard Purdie
546c32753b classes/lib: Remove bb.build.FuncFailed
Whilst seemingly a good idea, this exception doesn't really serve any purpose
that bb.fatal() doesn't cover. Wrapping exceptions within exceptions isn't
pythonic.

Its not used in many places, lets clean up those and remove usage of it
entirely. It may ultimately be dropped form bitbake entirely.

(From OE-Core rev: efe87ce4b2154c6f1c591ed9d8f770c229b044ad)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-08-06 11:24:26 +01:00
Richard Purdie
ffae400179 meta/lib+scripts: Convert to SPDX license headers
This adds SPDX license headers in place of the wide assortment of things
currently in our script headers. We default to GPL-2.0-only except for the
oeqa code where it was clearly submitted and marked as MIT on the most part
or some scripts which had the "or later" GPL versioning.

The patch also drops other obsolete bits of file headers where they were
encoountered such as editor modelines, obsolete maintainer information or
the phrase "All rights reserved" which is now obsolete and not required in
copyright headers (in this case its actually confusing for licensing as all
rights were not reserved).

More work is needed for OE-Core but this takes care of the bulk of the scripts
and meta/lib directories.

The top level LICENSE files are tweaked to match the new structure and the
SPDX naming.

(From OE-Core rev: f8c9c511b5f1b7dbd45b77f345cb6c048ae6763e)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-05-09 16:31:55 +01:00
Richard Purdie
244cbcce0e utils/multiprocess_launch: Improve failing subprocess output
Output before this patch:

ERROR: bash-4.4.18-r0 do_package_write_ipk: Fatal errors occurred in subprocesses:
Command 'PATH="X" opkg-build -Z xz -a "--memlimit=50% --threads=88" Foobar /media/build1/poky/build/nodistro-glibc/work/core2-64-oe-linux/bash/4.4.18-r0/deploy-ipks/core2-64' returned non-zero exit status 1.: Traceback (most recent call last):
  File "/media/build1/poky/meta/lib/oe/utils.py", line 272, in run
    ret = self._target(*self._args, **self._kwargs)
  File "/media/build1/poky/meta/classes/package_ipk.bbclass", line 230, in ipk_write_pkg
    shell=True)
  File "/usr/lib/python3.6/subprocess.py", line 336, in check_output
    **kwargs).stdout
  File "/usr/lib/python3.6/subprocess.py", line 418, in run
    output=stdout, stderr=stderr)
subprocess.CalledProcessError: Command 'PATH="X" opkg-build -Z xz -a "--memlimit=50% --threads=88" Foobar /media/build1/poky/build/nodistro-glibc/work/core2-64-oe-linux/bash/4.4.18-r0/deploy-ipks/core2-64' returned non-zero exit status 1.

Note how stdout/stderr from the failing command isn't shown.

After this patch:

ERROR: bash-4.4.18-r0 do_package_write_ipk: Fatal errors occurred in subprocesses:
Command 'PATH="X" opkg-build -Z xz -a "--memlimit=50% --threads=88" Foobar /media/build1/poky/build/nodistro-glibc/work/core2-64-oe-linux/bash/4.4.18-r0/deploy-ipks/core2-64' returned non-zero exit status 1.
Subprocess output:Foobar
*** Error: Package name Foobar contains illegal characters, (other than [a-z0-9.+-])

opkg-build: Please fix the above errors and try again.

We suddenly get a much more usable error message. The traceback is supressed
as its distracting from the real problem in this case.

Ideally python itself would handle this but it doesn't so we have to
wrap the exception. We already do this in bitbake itself for the same reason.

(From OE-Core rev: 09276dc76a8bda237b0b0b6d117a1980ae9dbfcc)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-04-26 10:09:08 +01:00
Andre Rosa
0dd6823875 lib/oe/utils: Make prune_suffix prune a suffix
... instead of replacing a substring that could happen more than once and not only when it ends with it. Do the same for the prefix.

See related https://github.com/openembedded/bitbake/pull/24 . There it stops replacing sufixes once first one is matched but not here.

(From OE-Core rev: 610ac84170f8a91cc3321edfc336a9e39f24ebe3)

Signed-off-by: Andre Rosa <andre.rosa@lge.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-04-09 13:44:39 +01:00
Enrico Scholz via Openembedded-core
1891a852b2 utils.py: added sh_quote() function
This function is a wrapper around "shlex.quote()" and can be used in
"${@...}" context where shlex (or pipes, which provides similar
functionality) is unavailable.

(From OE-Core rev: 127141f5023a7e3fc3963dc7d76cfce9067a9e8a)

Signed-off-by: Enrico Scholz <enrico.scholz@sigma-chemnitz.de>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-04-04 22:57:32 +01:00
Richard Purdie
f3e56f1b57 lib/oe/utils: Fix hang in multiprocess_launch()
If large results values are returned by the subprocesses, we can hit a deadlock
where the subprocess is trying to write data back to the parent, the pipe is full
and the parent is waiting for the child to exit.

Avoid this by calling the update() method which would trigger reading a result
from the child, avoiding the deadlock. The issue is described in
https://bugs.python.org/issue8426

(From OE-Core rev: 0035e8066ecbbff94d6a1994a9f72b1368d660d2)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-02-17 10:19:59 +00:00
Ross Burton
cdd0683e5e utils: add -src to system package name blacklist
oe.utils.packages_filter_out_system() returns PACKAGES after removing "system"
packages but it doesn't handle ${PN}-src as generated by
PACKAGE_DEBUG_SPLIT_STYLE=debug-with-srcpkg.

(From OE-Core rev: 162632d3d1e40c83ed9c5d49a026edf3912860a0)

Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-02-15 08:17:49 +00:00
Richard Purdie
cd4b8a8553 meta: Fix Deprecated warnings from regexs
Fix handling of escape characters in regexs and hence fix python
Deprecation warnings which will be problematic in python 3.8.

Note that some show up as:

"""
meta/classes/package.bbclass:1293: DeprecationWarning: invalid escape sequence \.  

"""

where the problem isn't on 1293 in package.bbclass but in some _prepend to a
package.bbclass function in a different file like mesa.inc, often from
do_package_split() calls.

(From OE-Core rev: 4b1c0c7d5525fc4cea9e0f02ec54e92a6fbc6199)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-01-16 15:35:07 +00:00
Robert Yang
0c98ff2c31 ccache.bbclass: Refactor it to make it more reliable
The previous ccache.bbclass has the following problems:
- It uses host's ccache for native recipes, but this may not work on some
  hosts, for example, it nerver works on my Ubuntu 14.04.4, there are always
  build failures (m4-native failed at do_configure, and others will also be
  failed if I disable CCACHE for m4-native)

- native/nativesdk/cross/crosssdk recipes use host's ccache, but target uses
  ccache-native, this may confuse user.

- The target recipes may use both host's ccache and ccache-native, this may
  cause unexpected problems and be hard to debug. This is because ccache-native is
  in SIGGEN_EXCLUDE_SAFE_RECIPE_DEPS, so ccache-native may not be present when
  rebuild target recipes, and then it would use hosttools/ccache, but the
  previous ccache files were generated by ccache-native.

- Target recipes can't use ccache when no ccache is installed on the host:
  CCACHE = "${@bb.utils.which(d.getVar('PATH'), 'ccache') and 'ccache '}"

After refactored:
All types recipes (native, target and others) will use ccache-native except
ccache-native itself, host's cache won't be used any more. It is more
reliable now, which will work everywhere when ccache-native can be built.

And now we need use "CCACHE_DISABLE = '1'" to disable ccache for the recipe
rather than "CCACHE = ''" since we set CCACHE in anonymous function, and
d.getVar('CCACHE') works after "CCACHE ??=" which is set in bitbake.conf, so we
can't check whether CCACHE is set or not in anonymous function since it is
always set. Use CCACHE_DISABLE to disable it would be more clear.

(From OE-Core rev: b25271b65262f70d849a4861da216c9be6c54d53)

Signed-off-by: Robert Yang <liezhi.yang@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-01-14 11:35:55 +00:00
Robert Yang
66c2638844 lib/oe/utils: Set stderr for host_gcc_version()
Fixed:
$ ln -s /usr/bin/ccache /folk/lyang1/bin/gcc
$ rm -fr tmp/hosttools/ && bitbake -p
[snip]
ERROR: Error running gcc  --version:

It didn't print the error message, now it is:
ERROR: Error running gcc  --version: ccache: error: Could not find compiler "gcc" in PATH

For the error itself, it is because ccache is not in my HOSTTOOLS, so this is
an expected error.

(From OE-Core rev: 91955caae584b4f75118e04411851b1a3d783fec)

Signed-off-by: Robert Yang <liezhi.yang@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2018-12-14 17:10:59 +00:00
Richard Purdie
d8d5199e66 lib/oe/utils: Improve multiprocess_lauch exception handling
We've seen a cryptic:

"ERROR: Fatal errors occurred in subprocesses, tracebacks printed above"

message from oe-selftest with no other traceback information. Improve the
traceback logging to try and give a better indication of any errors that is
ocurring.

(From OE-Core rev: 521dd3d00979a27b6932e58d5497de68abac26e1)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2018-12-05 12:37:02 +00:00
grygorii tertychnyi via Openembedded-core
bc8f229366 lib/oe/utils: add eol to format_pkg_list()
Append '\n' to the non-empty formatted string before return. If you
write it to the (manifest) file, it will ensure file ends with a newline.

Many GNU utilities have problems processing the last line of a file
if it is not '\n' terminated. E.g. if the last line is not terminated
by a newline character, then "read" will read it but return false,
leaving the broken partial line in the read variable(s).
It can also break or adversely affect some text processing tools,
that operate on the file.

(From OE-Core rev: ee4d0c879713ba50dc6cc3300f44647faebee2e0)

Signed-off-by: grygorii tertychnyi <gtertych@cisco.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2018-10-11 23:06:35 +01:00
Robert Yang
285b9a0d8e oe/utils.py: Add vartrue()
It can be used to simplify code like:
"${@['iffalse', 'iftrue'][var]}"

(From OE-Core rev: fc5a5af7bc3619f575988a75efc0c4fe15478b2d)

Signed-off-by: Robert Yang <liezhi.yang@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2018-09-13 07:42:28 +01:00
Richard Purdie
7a76bb5304 populate_base_sdk: Stop running gcc --version all the time
Running 'gcc --version' for every image recipe is slow and increases parsing
time/resource usage for no good reason. Only compute the value in when we're
really running the task/function.

(From OE-Core rev: bf49316bb9913b7c89de64d6a194be31aa66e16b)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2018-09-05 18:00:25 +01:00
Richard Purdie
5875571f03 lib/oe/utils: Fix get_multilib_datastore to work for original tune
Currently the original datastore returned by this function doesn't
always work as the tune isn't set back to the original. Fix it
to work like all_multilib_tune_list() in utils.bbclass and correct
the data returned.

(From OE-Core rev: 4e1dc858fbf671ef27089a2b9bcdc965fe19d698)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2018-08-23 17:58:59 +01:00
Richard Purdie
34c2bd84ee lib/oe/utils: Drop now unused multiprocess_exec
(From OE-Core rev: f3b30def2cd1c9ede7630489c3949a45b6eba6ed)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2018-07-24 11:52:27 +01:00
Richard Purdie
6d66b57409 utils: Add multiprocess_launch API and testcase
The current methods of spawning processes for parallel execution have
issues around collection of results or exceptions.

Take the code from package_ipk/deb, make it generic, add a results
collection mechanism, fix the exception handling and for it into a
standard library function.

Also add a test case which tests both the success and failure modes
of operation to stop this functionality regressiing again.

In particular, compared to multiprocess_exec, this fork off the parent
approach means we can pass in the datastore and functions work in the
same scope as the parent. This removes some of the complexities
found trying to scale multiprocess_exec to wider use.

(From OE-Core rev: 88f0c214e593a45566df5131bda4c946f5ccc8c2)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2018-07-24 11:52:27 +01:00
Richard Purdie
e9e1107c61 lib/oe/utils: Improve get_multilib_datastore
Currently this function assumes that no multilib is applied and that
we're applying a multilib. This means if we're in multilib context
and want the non-multilib context we can't obtain it (and no other
function exists for this either).

Improve the function to allow this to be requested.

(From OE-Core rev: 295c5a3d19834a2fac255346d0a373449cfdd776)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2018-07-02 21:37:47 +01:00
Joshua Watt
c4172f3f76 utils.py: Fix parallel_make limit
parallel_make_argument() was incorrectly taking the maximum of the limit
and the calculated value instead of the minimum.

(From OE-Core rev: 45205be547967c84dff96403c3a6825a62e3ca6a)

Signed-off-by: Joshua Watt <JPEWhacker@gmail.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2018-02-24 10:31:45 +00:00
Joshua Watt
feded5fe27 utils.py: add parallel make helpers
The code to extract the integer number of parallel build threads and
construct a new argument from them has started to be copied in multiple
locations, so create two new helper utilities to aid recipes.

The first helper (parallel_make()) extracts the integer number of
parallel build threads from PARALLEL_MAKE. The second
(parallel_make_argument()) does the same and then puts the result back
into a format string, optionally clamping it to some maximum value.

Additionally, rework the oe-core recipes that were manually doing this
to use the new helper utilities.

(From OE-Core rev: ccd1142d22b31ed85d8823b1bc9e11ccfd72b61f)

Signed-off-by: Joshua Watt <JPEWhacker@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2018-02-16 18:05:40 +00:00
Richard Purdie
41ea1be73c utils: Make get_multilib_datastore available from lib/oe
Currently we can't access this function from lib/oe as its a class function.
Move it to allow such access.

(From OE-Core rev: b241a666f2867ffa425f6d43763d7c3c17941dcf)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2018-01-04 12:57:43 +00:00
Ross Burton
4469acdf1d lib/oe/utils: remove param_bool()
This function is not used by any classes or recipes that I can find, so lets
delete it.

(From OE-Core rev: a7cd9d1183be603777fc9c8c448281fe01224f7b)

Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-12-02 11:25:34 +00:00
Richard Purdie
627d9a71df lib/oe/utils: Handle exceptions in multiprocess_exec
Currently exceptions that happen in pool commands are ignored. Any errors
would be printed on the console but everything else is silent.

Switch to use pool.map_async which allows for an error_callback which
we can use to detect exceptions and make sure these errors are handled.

(From OE-Core rev: 7f2f9b3ff011b340b5d23bb7c47b12c357dc9f02)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-09-02 00:52:10 +01:00
Patrick Ohly
baeafe6dcc utils.py: helper function for optional include files
The main intention is to provide easy-to-use and read helper functions
for including files only when certain distro features are
set. Functionally they are the same as bb.utils.contains and
bb.utils.contains_any.

Distro features are part of the base configuration and thus safe to
use for conditional inclusion in recipes and bbappends, in contrast to
recipe variables which might still change during parsing. Therefore
the check is limited to DISTRO_FEATURES. This is the reason for having
this in OE-core instead of bitbake.

Default values are set so that no redundant parameters have to be
passed for conditional includes. As a secondary usage, the functions
can also be used in boolean checks.

(From OE-Core rev: 13024ce5aae453769b546d5fbe533443aec3d6fd)

Signed-off-by: Patrick Ohly <patrick.ohly@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-06-12 15:08:32 +01:00
Richard Purdie
eeb30764e9 lib/oe/utils: Add build_depends_string function
This is useful when manipulating depends strings for task [depends]
flags and is slightly easier to parse than some inline python.

(From OE-Core rev: 7b05ea65a8db8a27b2a5579675775ee34ceb63c2)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-01-20 11:53:49 +00:00
Richard Purdie
022a8b58c8 meta/scripts: Various getVar/getVarFlag expansion parameter fixes
There were a few straggling expansion parameter removals left for
getVar/getVarFlag where the odd whitespace meant they were missed
on previous passes. There were also some plain broken ussages such
as:

d.getVar('ALTERNATIVE_TARGET', old_name, True)
path = d.getVar('PATH', d, True)
d.getVar('IMAGE_ROOTFS', 'True')

which I've corrected (they happend to work by luck).

(From OE-Core rev: 688f7a64917a5ce5cbe12f8e5da4d47e265d240f)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-01-09 13:39:11 +00:00
Richard Purdie
d77a135e18 lib/oe/utils: Drop python2 compatibility code
We've moved to python3, we don't need this compatibility code which just makes
the code less readable.

(From OE-Core rev: 425afe2484707640ac71194885fdb263e95e9950)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-12-22 12:36:40 +00:00
Ross Burton
1969854ec3 utils: Always use datastore's PATH for host_gcc_version
BUILD_CC may reference something like ccache and expect this to come from
ccache-native, we at least have some selftests which assume this. Modify the
code to use PATH when runnig BUILD_CC to ensure the tests continue to work
as expected.

(From OE-Core rev: f3e753372baac43d0921186340cf260df056de20)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-12-16 10:23:23 +00:00
Joshua Lock
c4e2c59088 meta: remove True option to getVar calls
getVar() now defaults to expanding by default, thus remove the True
option from getVar() calls with a regex search and replace.

Search made with the following regex: getVar ?\(( ?[^,()]*), True\)

(From OE-Core rev: 7c552996597faaee2fbee185b250c0ee30ea3b5f)

Signed-off-by: Joshua Lock <joshua.g.lock@intel.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-12-16 10:23:23 +00:00
Ed Bartosh
c21cec8488 uninative: rebuild uninative for gcc 4.8 and 4.9
Some c++ libraries fail to build if uninative is built
with gcc 5.x and host gcc version is either 4.8 or 4.9.

The issue should be solved by making separate uninative sstate
directory structure sstate-cache/universal-<gcc version> for host gcc
versions 4.8 and 4.9. This causes rebuilds of uninative if host gcc
is either 4.8 or 4.9 and it doesn't match gcc version used to build
uninative.

[YOCTO #10441]

(From OE-Core rev: d36f41e5658bbbb6080ee833027879c119edf3e0)

Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-12-16 10:23:23 +00:00
Robert Yang
b8e2bab119 meta: cleanup d.getVar(var, 1)
(From OE-Core rev: 79fe476be233015c1c90e9c3fb4572267b5551d1)

Signed-off-by: Robert Yang <liezhi.yang@windriver.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-09-14 22:22:07 +01:00
Joshua Lock
4518058942 image: add do_image_qa task to run QA checks on the constructed image
This task runs all functions in IMAGE_QA_COMMANDS after the image
construction has completed in order to validate the resulting image.

Image sanity checks should either be Python functions which raise
bb.build.FuncFailed on failure or shell functions with return a
non-zero exit code.

Python functions may instead raise an oe.utils.ImageQAFailed
Exception which takes an extra argument, a description of the
failure.

   python image_check_python_ok () {
       if True:
           raise bb.build.FuncFailed('This check always fails')
       else:
           bb.note("Nothing to see here")
   }

   image_check_shell_ok () {
       if true
           exit 1
       else
           exit 0
       fi
   }

[YOCTO #9448]

(From OE-Core rev: c9bef2ecf1a30159d11781184829f41844a58c13)

Signed-off-by: Joshua Lock <joshua.g.lock@intel.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-07-20 10:28:52 +01:00
Richard Purdie
3b39971748 classes/lib: Complete transition to python3
This patch contains all the other misc pieces of the transition to
python3 which didn't make sense to be broken into individual patches.

(From OE-Core rev: fcd6b38bab8517d83e1ed48eef1bca9a9a190f57)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-06-02 08:24:01 +01:00
Richard Purdie
44e9a0d2fa classes/lib: Update to explictly create lists where needed
Iterators now return views, not lists in python3. Where we need
lists, handle this explicitly.

(From OE-Core rev: caebd862bac7eed725e0f0321bf50793671b5312)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-06-02 08:24:00 +01:00
Richard Purdie
3ddde5f32a meta: Update to modern exception syntax
Update older exception syntax to modern one required by python 3.
Compatible with python 2.7.

(From OE-Core rev: d13f0ac614f1d1e2ef2c8ddc71cbfcf76a8dc3f2)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-05-21 22:26:33 +01:00
Richard Purdie
ffe9604be2 meta/selftest/scripts: Use print function for python3 compatibility
Used print function instead of print statement to make
the code work in python 3.

[Changes from both Ed and Richard]

(From OE-Core rev: ced1995694c394d92cb82fb9c25a33dc027a3b69)

Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-05-21 22:26:32 +01:00
Richard Purdie
0b84897ea8 insane/prelink: Handle nonstandard library paths
Prelink contains some hardcoded assumptions about the path layout of
the target system. Unfortunately if the system doesn't match, prelink
doesn't work. This breaks:

a) prelink of those images
b) the unsafe-references-in-binaries QA test (which uses prelink-rtld)

One way to work around this is to construct an ld.so.conf file which
lists the library paths in question. We do this in sanity QA check and
in the rootfs prelink code, being careful not to trample any existing
target ld.so.conf.

There is an additional problem that $LIB references in RPATHs won't be
handled correctly, I've not see any system use these in reality though
so this change at least improves things.

(From OE-Core rev: 7fd1d7e639c2ed7e0699937a5cb245c187b7c811)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-03-07 00:11:39 +00:00
Mariano Lopez
03075f671c lib/oe/utils: Add function format_pkg_list()
The class PkgsList returns a dictionary with all the installed
packages, because the data structure is a dictionary there is
needed to format the data in order to write to a file.

The function format_pkg_list returns a formated sting with all
packages installed. The output will depend on the requested format
when calling the function.

[YOCTO #7427]

(From OE-Core rev: 25725e6e5fff8017aaf3a6fcd9b1b893c22630b5)

Signed-off-by: Mariano Lopez <mariano.lopez@linux.intel.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-01-20 17:07:15 +00:00
Aníbal Limón
615b351bf4 oe/utils.py: Add support for init/end helper functions in ThreadWorker.
Add init/end helper functions for ThreadWorker also pass ThreadWorker
as first argument to init/end/func functions this enables per-thread
storage handling.

classes/sstate.bbclass: Add thread_worker argument to checkstatus
function.

(From OE-Core rev: 08c50d62b520c8405f034e3d7adeea89e06226ee)

Signed-off-by: Aníbal Limón <anibal.limon@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2015-06-26 09:27:30 +01:00
Aníbal Limón
3fa32158c4 oe/utils.py: Fix thread leakage in ThreadPool
In order to fix Thread leakage caused by not call join() in Threads,

Pass num_tasks in ThreadPool for add all the tasks into a Queue this
enable catch of Queue.Empty exception and exit the threads.

classes/sstate.bbclass: Change checkstatus function to match new
ThreadPool operation.

(From OE-Core rev: 524d92ed7b53bef933527095e82f378b934f25ef)

Signed-off-by: Aníbal Limón <anibal.limon@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2015-06-26 09:27:30 +01:00