Commit Graph

89 Commits

Author SHA1 Message Date
Richard Purdie
d5d35a27b4 lib/oe: Move vardepexclude entries alongside functions
Now we have decorators that can do this, move the variable dependencies
exclusions alongside the code that needs them for maintainability.

(From OE-Core rev: e522169c5f95de6fc74b43672573700d8eb8e082)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2025-06-16 17:57:29 +01:00
Richard Purdie
430ed4a884 lib/package/utils: Improve multiprocess_launch argument passing
The current code for multiple argument passing is horrible. Tweak the
multiprocess_launch function to only convert to a tuple if it isn't already
one, which means we can then use function arguments in a standard way.

(From OE-Core rev: 7c99f90079e722764ebdc30e8d0e781454b3a51a)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2025-02-28 07:49:18 +00:00
Peter Kjellerstedt
b0eb2b9b36 image.bbclass: Drop support for ImageQAFailed exceptions in image_qa
After commit 905e224849fbbed1719e0add231b00e2d570b3b4 (image_qa: fix
error handling), any unexpected exceptions in do_image_qa() would result
in a variable being set, but never used, effectively hiding the error.

Since image_qa now calls oe.qa.exit_if_errors(), remove the support for
oe.utils.ImageQAFailed and instead rely on the called functions to call
oe.qa.handle_error() themselves. This matches what do_package_qa() does.

Also update the description of do_image_qa() to explain that the called
functions are expected to call oe.qa.handle_error() themselves.

[ YOCTO #15601 ]

(From OE-Core rev: 0c3e111c965af2bc56533633c376b70b7fa5e1de)

Signed-off-by: Peter Kjellerstedt <peter.kjellerstedt@axis.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2024-09-30 17:07:17 +01:00
Louis Rannou
48a898fdfb image_qa: fix error handling
Make ImageQAFailed inherit BBHandledException so exceptions raised in tests are
catched when the actual test function is executed by bb.utils.better_exec.

Change the do_image_qa tasks so errors are handled with oe.qa.handle_error. Add
some comment to explain this requires to list the test in ERROR_QA or WARN_QA.

[YOCTO #14807]
https://bugzilla.yoctoproject.org/show_bug.cgi?id=14807

(From OE-Core rev: 905e224849fbbed1719e0add231b00e2d570b3b4)

Signed-off-by: Louis Rannou <louis.rannou@non.se.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2024-09-17 12:16:01 +01:00
Richard Purdie
3d35d09ff2 sstate: Ensure sstate searches update file mtime
Commands like "bitbake XXX -S printdiff" search for sstate files but don't download
them. This means that local files aren't touched as the download code would do, meaning
the sstate cleanup scripts can delete them. This can then lead to obtuse build failures.

Have the search code touch local files in the same way as the main code paths would to
avoid these files disappearing.

Move the function to a common touch() function in lib/oe instead of duplicating code.

(From OE-Core rev: a27fc0bd5706ab5b9c68a0271fcf57377a678cdf)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2023-11-08 10:56:35 +00:00
Adrian Freihofer
40fdebda20 lib/oe/utils: Refactor to make multiprocess_launch callable without d
This is a preparation for making the strip_execs function callable from
devtool without going via tinfoil and a bitbake server process.

(From OE-Core rev: af8ee73cdef90b83556a7ac5e139a08108706486)

Signed-off-by: Adrian Freihofer <adrian.freihofer@siemens.com>
Signed-off-by: Alexandre Belloni <alexandre.belloni@bootlin.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2023-09-12 12:44:47 +01:00
Richard Purdie
6fd8af0d30 classes: Drop ';' delimiter from ROOTFS/IMAGE*COMMAND variables
Originally these were shell functions but they have long since been processed by
bb.build.exec_func(). Since we no longer need shell syntax, we can drop the ';'
delimiters and just use a space separated string.

This cleans up the variable and quietly removes any stray ';' that do happen to
still make it in.

(From OE-Core rev: c3365dfd9ddd7fbe70b62e0f11166e57a8ca6f73)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2023-09-09 22:14:41 +01:00
Richard Purdie
ce08cf4825 lib: Add copyright statements to files without one
Where there isn't a copyright statement, add one to make it explicit.
Also add license identifiers as MIT if there isn't one.

(From OE-Core rev: bb731d1f3d2a1d50ec0aed864dbca54cf795b040)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2022-08-12 12:00:43 +01:00
Jose Quaresma
4161dbbbd6 oe/utils: remove the ThreadedPool
The ThreadedPool in OE-core is mainly because python2
doesn't have threaded pools but python2 is dead for
some time now and python3 have a ThreadPoolExecutor.

The only local in OE-core where this ThreadedPool
is in use is on the sstate.bbclass that is ported
to the python3 ThreadPoolExecutor.

(From OE-Core rev: aa8fd5e7c2a1125895accfd55ce9320819a10959)

Signed-off-by: Jose Quaresma <quaresma.jose@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2022-07-15 17:51:34 +01:00
Saul Wold
d9e500f83d meta/scripts: Improve internal variable naming
Update internal variable names to improve the terms used.

(From OE-Core rev: f408068e5d7998ae165f3002e51bc54b380b8099)

Signed-off-by: Saul Wold <saul.wold@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2022-03-10 08:00:28 +00:00
Richard Purdie
a242274d98 prelink: Drop support for it
Prelink is being dropped by glibc in 2.36. It already causes issues with
binary corruption, has a number of open bugs and is of questionable benefit
without disabling load address randomization and PIE executables.

We disabled it by default a while back but left people able to use it.
We would be unable to maintain it alone without glibc support so remove
the remaining pieces.

(From OE-Core rev: 23c0be78106f1d1e2bb9c724174a1bb8c56c2469)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2022-02-05 12:26:36 +00:00
Ross Burton
c6f23f1f0f oe/utils: by default cap cpu_count() to 64 cores
Larger systems may have large numbers of cores, but beyond a certain
point they can't all be used for compiling: whilst purely
compute-intensive jobs can be parallelised to hundreds of cores,
operations such as compressing (needs lots of RAM) or compiling (lots of
I/O) don't scale linearly.

For example, the Marvel ThunderX2 has 32 cores, each capable of
executing four threads, and can be configured with two sockets, making
256 CPUs according to Linux. Zstd using 256 threads has been seen to
fail to allocate memory during even small recipes such as iso-codes.

Add a default cap of 64 CPUs to the cpu_count() method so that extreme
parallisation is limited.  64 is high enough that meaningful gains
beyond it are unlikely, but high enough that most systems won't be
effected.

(From OE-Core rev: 765d0f25ce48636b1838a5968e2dc15de2127428)

Signed-off-by: Ross Burton <ross.burton@arm.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2021-11-25 21:55:10 +00:00
Ross Burton
1fb99950a2 oe/utils: allow naming threads in ThreadedPool
When looking at logs involving thread pools it is useful if the threads
can be named.

(From OE-Core rev: 18342945b021608794d83ecf567afd43f4379b24)

Signed-off-by: Ross Burton <ross.burton@arm.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2021-11-25 21:55:10 +00:00
Ross Burton
f59811aeeb oe/utils: log exceptions in ThreadedWorker functions
If the function a ThreadedWorker is executing raises an exception, don't
use print() as that mostly disappears.  Instead, output it to the logger.

This is done using bb.mainlogger.debug directly instead of bb.debug() as
this allows us to pass the exception instance directly, which is then
incorporated into the log stream.

(From OE-Core rev: 2f1ea25c222b344dd8b784b2bc73a6540ab30274)

Signed-off-by: Ross Burton <ross.burton@arm.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2021-10-18 13:48:16 +01:00
Andres Beltran
157462fc2a buildhistory: Label packages providing per-file dependencies in depends.dot
Currently, depends.dot includes per-file dependencies but not the packages
providing those files. This makes it hard to obtain all package
dependencies by just looking at depends.dot.

Parse the RPROVIDES and FILERPROVIDES fields from pkgdata to map each of
their values to the package providing the component. Include runtime
packages as dependencies in depends.dot, together with the component
provided by the package as a label.

(From OE-Core rev: 2ba33093017574bbe29eeba699eb90628614d03a)

Signed-off-by: Andres Beltran <abeltran@linux.microsoft.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2021-09-01 14:06:18 +01:00
Ross Burton
3743234dab lib/oe/utils: add directory size function
For the purpose of image construction using du on a rootfs directory isn't
entirely satisfactory.  Bare "du" will report the actual disk usage so file
systems which can compress the data will report less than the actual space
required.  Using "du --apparent-size" will report the actual space used, but as
this simply sums the bytes used for content across an entire file system can
result in significant under-reporting due to block size overhead.

Attempt to solve these problems by implementing our own function to calculate
how large a rootfs will be.  This function handles hardlinks correctly but
rounds up all sizes to multiples of the block size (currently, 4KB is the
hard-coded block size).

(From OE-Core rev: 6ca53ad7b26ee2b4e6d2c121c6f6d6eed7f6b56f)

Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2021-03-28 22:28:26 +01:00
Tomasz Dziendzielski
ca608639f8 lib/oe/utils: Return empty string in parallel_make
In cmake.bbclass we set CMAKE_BUILD_PARALLEL_LEVEL using parallel_make
function and if PARALLEL_MAKE is set to empty string then this variable
is exported as "None" causing cmake to fail with:
"'CMAKE_BUILD_PARALLEL_LEVEL' environment variable
invalid number 'None' given."

(From OE-Core rev: 2f790ded554a52ac18d1c28002142f9c62abec8b)

Signed-off-by: Tomasz Dziendzielski <tomasz.dziendzielski@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2021-01-16 22:39:35 +00:00
Ross Burton
c2a11e18c8 utils: respect scheduler affinity in cpu_count()
cpu_count() returns multiprocessing.cpu_count() but that is simply returns
os.cpu_count() so we could use that directly.

However this returns the number of CPUs on the host, not the number of
usable CPUs on the host. If the user is using scheduler affinity then
the number of usable CPUs may be less, so when determining how many cores
we can use check the affinity instead.

(From OE-Core rev: e2e8ccbe410b2f38bcd9525982b2261cf71aab60)

Signed-off-by: Ross Burton <ross.burton@arm.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2020-09-30 15:01:51 +01:00
Tim Orling
d5d634f682 lib/oe/utils.py: add get_host_compiler_version()
Add helper function to get the host compiler and version.
Do not assume compiler is gcc.

NOTE: cannot set env to d.getVar("PATH") as that does not contain
the session PATH which was set by environment-setup-... which
breaks the install-buildtools use-case

(From OE-Core rev: 88712929354ff9c876bb1e48b6f15c33af5f2bbc)

Signed-off-by: Tim Orling <timothy.t.orling@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2020-04-01 11:44:24 +01:00
Charles-Antoine Couret
ad366e7383 utils: fix gcc 10 version detection
Utils can not detect GCC 10 correctly due to wrong regex.
It generates this error "ERROR: Can't get compiler version from gcc  --version output"

Sub-version numbers should be 1 or more digits instead of 1 only.

(From OE-Core rev: 186fe4a3d390a52b87282c3e694ce3251e45ee78)

Signed-off-by: Charles-Antoine Couret <charles-antoine.couret@mind.be>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2020-03-29 12:05:31 +01:00
André Draszik
0d7f24ce89 lib/oe/utils: allow to set a lower bound on returned cpu_count()
This will be needed for making xz compression more deterministic,
as xz archives are created differently in single- vs multi-threaded
modes.

This means that due to bitbake's default of using as many threads
as there are cores in the system, files compressed with xz
will be different if built on a multi-core system compared to
single-core systems.

Allowing cpu_count() here to return a lower bound, will allow
forcing xz to always use multi-threaded operation.

(From OE-Core rev: c6dda204e15c5b27559f7617b18a7b3ce604e2f8)

Signed-off-by: André Draszik <git@andred.net>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2020-03-06 08:19:17 +00:00
Daniel McGregor
a26efa2875 cmake: prefer CMAKE_BUILD_PARALLEL_LEVEL
cmake 3.12 introduced this environment variable. Prefer it to passing
PARALLEL_MAKE and PARALLEL_MAKEINST on the cmake command line, because
it gets passed to second stage cmake invocations while command-line
arguments do not (for example, multi-stage clang builds)

(From OE-Core rev: cdd44c93f02bb8cc2fa773e13c8ce36e3da23921)

Signed-off-by: Daniel McGregor <daniel.mcgregor@vecima.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2020-02-04 15:56:29 +00:00
Richard Purdie
546c32753b classes/lib: Remove bb.build.FuncFailed
Whilst seemingly a good idea, this exception doesn't really serve any purpose
that bb.fatal() doesn't cover. Wrapping exceptions within exceptions isn't
pythonic.

Its not used in many places, lets clean up those and remove usage of it
entirely. It may ultimately be dropped form bitbake entirely.

(From OE-Core rev: efe87ce4b2154c6f1c591ed9d8f770c229b044ad)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-08-06 11:24:26 +01:00
Richard Purdie
ffae400179 meta/lib+scripts: Convert to SPDX license headers
This adds SPDX license headers in place of the wide assortment of things
currently in our script headers. We default to GPL-2.0-only except for the
oeqa code where it was clearly submitted and marked as MIT on the most part
or some scripts which had the "or later" GPL versioning.

The patch also drops other obsolete bits of file headers where they were
encoountered such as editor modelines, obsolete maintainer information or
the phrase "All rights reserved" which is now obsolete and not required in
copyright headers (in this case its actually confusing for licensing as all
rights were not reserved).

More work is needed for OE-Core but this takes care of the bulk of the scripts
and meta/lib directories.

The top level LICENSE files are tweaked to match the new structure and the
SPDX naming.

(From OE-Core rev: f8c9c511b5f1b7dbd45b77f345cb6c048ae6763e)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-05-09 16:31:55 +01:00
Richard Purdie
244cbcce0e utils/multiprocess_launch: Improve failing subprocess output
Output before this patch:

ERROR: bash-4.4.18-r0 do_package_write_ipk: Fatal errors occurred in subprocesses:
Command 'PATH="X" opkg-build -Z xz -a "--memlimit=50% --threads=88" Foobar /media/build1/poky/build/nodistro-glibc/work/core2-64-oe-linux/bash/4.4.18-r0/deploy-ipks/core2-64' returned non-zero exit status 1.: Traceback (most recent call last):
  File "/media/build1/poky/meta/lib/oe/utils.py", line 272, in run
    ret = self._target(*self._args, **self._kwargs)
  File "/media/build1/poky/meta/classes/package_ipk.bbclass", line 230, in ipk_write_pkg
    shell=True)
  File "/usr/lib/python3.6/subprocess.py", line 336, in check_output
    **kwargs).stdout
  File "/usr/lib/python3.6/subprocess.py", line 418, in run
    output=stdout, stderr=stderr)
subprocess.CalledProcessError: Command 'PATH="X" opkg-build -Z xz -a "--memlimit=50% --threads=88" Foobar /media/build1/poky/build/nodistro-glibc/work/core2-64-oe-linux/bash/4.4.18-r0/deploy-ipks/core2-64' returned non-zero exit status 1.

Note how stdout/stderr from the failing command isn't shown.

After this patch:

ERROR: bash-4.4.18-r0 do_package_write_ipk: Fatal errors occurred in subprocesses:
Command 'PATH="X" opkg-build -Z xz -a "--memlimit=50% --threads=88" Foobar /media/build1/poky/build/nodistro-glibc/work/core2-64-oe-linux/bash/4.4.18-r0/deploy-ipks/core2-64' returned non-zero exit status 1.
Subprocess output:Foobar
*** Error: Package name Foobar contains illegal characters, (other than [a-z0-9.+-])

opkg-build: Please fix the above errors and try again.

We suddenly get a much more usable error message. The traceback is supressed
as its distracting from the real problem in this case.

Ideally python itself would handle this but it doesn't so we have to
wrap the exception. We already do this in bitbake itself for the same reason.

(From OE-Core rev: 09276dc76a8bda237b0b0b6d117a1980ae9dbfcc)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-04-26 10:09:08 +01:00
Andre Rosa
0dd6823875 lib/oe/utils: Make prune_suffix prune a suffix
... instead of replacing a substring that could happen more than once and not only when it ends with it. Do the same for the prefix.

See related https://github.com/openembedded/bitbake/pull/24 . There it stops replacing sufixes once first one is matched but not here.

(From OE-Core rev: 610ac84170f8a91cc3321edfc336a9e39f24ebe3)

Signed-off-by: Andre Rosa <andre.rosa@lge.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-04-09 13:44:39 +01:00
Enrico Scholz via Openembedded-core
1891a852b2 utils.py: added sh_quote() function
This function is a wrapper around "shlex.quote()" and can be used in
"${@...}" context where shlex (or pipes, which provides similar
functionality) is unavailable.

(From OE-Core rev: 127141f5023a7e3fc3963dc7d76cfce9067a9e8a)

Signed-off-by: Enrico Scholz <enrico.scholz@sigma-chemnitz.de>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-04-04 22:57:32 +01:00
Richard Purdie
f3e56f1b57 lib/oe/utils: Fix hang in multiprocess_launch()
If large results values are returned by the subprocesses, we can hit a deadlock
where the subprocess is trying to write data back to the parent, the pipe is full
and the parent is waiting for the child to exit.

Avoid this by calling the update() method which would trigger reading a result
from the child, avoiding the deadlock. The issue is described in
https://bugs.python.org/issue8426

(From OE-Core rev: 0035e8066ecbbff94d6a1994a9f72b1368d660d2)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-02-17 10:19:59 +00:00
Ross Burton
cdd0683e5e utils: add -src to system package name blacklist
oe.utils.packages_filter_out_system() returns PACKAGES after removing "system"
packages but it doesn't handle ${PN}-src as generated by
PACKAGE_DEBUG_SPLIT_STYLE=debug-with-srcpkg.

(From OE-Core rev: 162632d3d1e40c83ed9c5d49a026edf3912860a0)

Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-02-15 08:17:49 +00:00
Richard Purdie
cd4b8a8553 meta: Fix Deprecated warnings from regexs
Fix handling of escape characters in regexs and hence fix python
Deprecation warnings which will be problematic in python 3.8.

Note that some show up as:

"""
meta/classes/package.bbclass:1293: DeprecationWarning: invalid escape sequence \.  

"""

where the problem isn't on 1293 in package.bbclass but in some _prepend to a
package.bbclass function in a different file like mesa.inc, often from
do_package_split() calls.

(From OE-Core rev: 4b1c0c7d5525fc4cea9e0f02ec54e92a6fbc6199)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-01-16 15:35:07 +00:00
Robert Yang
0c98ff2c31 ccache.bbclass: Refactor it to make it more reliable
The previous ccache.bbclass has the following problems:
- It uses host's ccache for native recipes, but this may not work on some
  hosts, for example, it nerver works on my Ubuntu 14.04.4, there are always
  build failures (m4-native failed at do_configure, and others will also be
  failed if I disable CCACHE for m4-native)

- native/nativesdk/cross/crosssdk recipes use host's ccache, but target uses
  ccache-native, this may confuse user.

- The target recipes may use both host's ccache and ccache-native, this may
  cause unexpected problems and be hard to debug. This is because ccache-native is
  in SIGGEN_EXCLUDE_SAFE_RECIPE_DEPS, so ccache-native may not be present when
  rebuild target recipes, and then it would use hosttools/ccache, but the
  previous ccache files were generated by ccache-native.

- Target recipes can't use ccache when no ccache is installed on the host:
  CCACHE = "${@bb.utils.which(d.getVar('PATH'), 'ccache') and 'ccache '}"

After refactored:
All types recipes (native, target and others) will use ccache-native except
ccache-native itself, host's cache won't be used any more. It is more
reliable now, which will work everywhere when ccache-native can be built.

And now we need use "CCACHE_DISABLE = '1'" to disable ccache for the recipe
rather than "CCACHE = ''" since we set CCACHE in anonymous function, and
d.getVar('CCACHE') works after "CCACHE ??=" which is set in bitbake.conf, so we
can't check whether CCACHE is set or not in anonymous function since it is
always set. Use CCACHE_DISABLE to disable it would be more clear.

(From OE-Core rev: b25271b65262f70d849a4861da216c9be6c54d53)

Signed-off-by: Robert Yang <liezhi.yang@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-01-14 11:35:55 +00:00
Robert Yang
66c2638844 lib/oe/utils: Set stderr for host_gcc_version()
Fixed:
$ ln -s /usr/bin/ccache /folk/lyang1/bin/gcc
$ rm -fr tmp/hosttools/ && bitbake -p
[snip]
ERROR: Error running gcc  --version:

It didn't print the error message, now it is:
ERROR: Error running gcc  --version: ccache: error: Could not find compiler "gcc" in PATH

For the error itself, it is because ccache is not in my HOSTTOOLS, so this is
an expected error.

(From OE-Core rev: 91955caae584b4f75118e04411851b1a3d783fec)

Signed-off-by: Robert Yang <liezhi.yang@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2018-12-14 17:10:59 +00:00
Richard Purdie
d8d5199e66 lib/oe/utils: Improve multiprocess_lauch exception handling
We've seen a cryptic:

"ERROR: Fatal errors occurred in subprocesses, tracebacks printed above"

message from oe-selftest with no other traceback information. Improve the
traceback logging to try and give a better indication of any errors that is
ocurring.

(From OE-Core rev: 521dd3d00979a27b6932e58d5497de68abac26e1)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2018-12-05 12:37:02 +00:00
grygorii tertychnyi via Openembedded-core
bc8f229366 lib/oe/utils: add eol to format_pkg_list()
Append '\n' to the non-empty formatted string before return. If you
write it to the (manifest) file, it will ensure file ends with a newline.

Many GNU utilities have problems processing the last line of a file
if it is not '\n' terminated. E.g. if the last line is not terminated
by a newline character, then "read" will read it but return false,
leaving the broken partial line in the read variable(s).
It can also break or adversely affect some text processing tools,
that operate on the file.

(From OE-Core rev: ee4d0c879713ba50dc6cc3300f44647faebee2e0)

Signed-off-by: grygorii tertychnyi <gtertych@cisco.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2018-10-11 23:06:35 +01:00
Robert Yang
285b9a0d8e oe/utils.py: Add vartrue()
It can be used to simplify code like:
"${@['iffalse', 'iftrue'][var]}"

(From OE-Core rev: fc5a5af7bc3619f575988a75efc0c4fe15478b2d)

Signed-off-by: Robert Yang <liezhi.yang@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2018-09-13 07:42:28 +01:00
Richard Purdie
7a76bb5304 populate_base_sdk: Stop running gcc --version all the time
Running 'gcc --version' for every image recipe is slow and increases parsing
time/resource usage for no good reason. Only compute the value in when we're
really running the task/function.

(From OE-Core rev: bf49316bb9913b7c89de64d6a194be31aa66e16b)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2018-09-05 18:00:25 +01:00
Richard Purdie
5875571f03 lib/oe/utils: Fix get_multilib_datastore to work for original tune
Currently the original datastore returned by this function doesn't
always work as the tune isn't set back to the original. Fix it
to work like all_multilib_tune_list() in utils.bbclass and correct
the data returned.

(From OE-Core rev: 4e1dc858fbf671ef27089a2b9bcdc965fe19d698)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2018-08-23 17:58:59 +01:00
Richard Purdie
34c2bd84ee lib/oe/utils: Drop now unused multiprocess_exec
(From OE-Core rev: f3b30def2cd1c9ede7630489c3949a45b6eba6ed)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2018-07-24 11:52:27 +01:00
Richard Purdie
6d66b57409 utils: Add multiprocess_launch API and testcase
The current methods of spawning processes for parallel execution have
issues around collection of results or exceptions.

Take the code from package_ipk/deb, make it generic, add a results
collection mechanism, fix the exception handling and for it into a
standard library function.

Also add a test case which tests both the success and failure modes
of operation to stop this functionality regressiing again.

In particular, compared to multiprocess_exec, this fork off the parent
approach means we can pass in the datastore and functions work in the
same scope as the parent. This removes some of the complexities
found trying to scale multiprocess_exec to wider use.

(From OE-Core rev: 88f0c214e593a45566df5131bda4c946f5ccc8c2)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2018-07-24 11:52:27 +01:00
Richard Purdie
e9e1107c61 lib/oe/utils: Improve get_multilib_datastore
Currently this function assumes that no multilib is applied and that
we're applying a multilib. This means if we're in multilib context
and want the non-multilib context we can't obtain it (and no other
function exists for this either).

Improve the function to allow this to be requested.

(From OE-Core rev: 295c5a3d19834a2fac255346d0a373449cfdd776)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2018-07-02 21:37:47 +01:00
Joshua Watt
c4172f3f76 utils.py: Fix parallel_make limit
parallel_make_argument() was incorrectly taking the maximum of the limit
and the calculated value instead of the minimum.

(From OE-Core rev: 45205be547967c84dff96403c3a6825a62e3ca6a)

Signed-off-by: Joshua Watt <JPEWhacker@gmail.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2018-02-24 10:31:45 +00:00
Joshua Watt
feded5fe27 utils.py: add parallel make helpers
The code to extract the integer number of parallel build threads and
construct a new argument from them has started to be copied in multiple
locations, so create two new helper utilities to aid recipes.

The first helper (parallel_make()) extracts the integer number of
parallel build threads from PARALLEL_MAKE. The second
(parallel_make_argument()) does the same and then puts the result back
into a format string, optionally clamping it to some maximum value.

Additionally, rework the oe-core recipes that were manually doing this
to use the new helper utilities.

(From OE-Core rev: ccd1142d22b31ed85d8823b1bc9e11ccfd72b61f)

Signed-off-by: Joshua Watt <JPEWhacker@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2018-02-16 18:05:40 +00:00
Richard Purdie
41ea1be73c utils: Make get_multilib_datastore available from lib/oe
Currently we can't access this function from lib/oe as its a class function.
Move it to allow such access.

(From OE-Core rev: b241a666f2867ffa425f6d43763d7c3c17941dcf)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2018-01-04 12:57:43 +00:00
Ross Burton
4469acdf1d lib/oe/utils: remove param_bool()
This function is not used by any classes or recipes that I can find, so lets
delete it.

(From OE-Core rev: a7cd9d1183be603777fc9c8c448281fe01224f7b)

Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-12-02 11:25:34 +00:00
Richard Purdie
627d9a71df lib/oe/utils: Handle exceptions in multiprocess_exec
Currently exceptions that happen in pool commands are ignored. Any errors
would be printed on the console but everything else is silent.

Switch to use pool.map_async which allows for an error_callback which
we can use to detect exceptions and make sure these errors are handled.

(From OE-Core rev: 7f2f9b3ff011b340b5d23bb7c47b12c357dc9f02)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-09-02 00:52:10 +01:00
Patrick Ohly
baeafe6dcc utils.py: helper function for optional include files
The main intention is to provide easy-to-use and read helper functions
for including files only when certain distro features are
set. Functionally they are the same as bb.utils.contains and
bb.utils.contains_any.

Distro features are part of the base configuration and thus safe to
use for conditional inclusion in recipes and bbappends, in contrast to
recipe variables which might still change during parsing. Therefore
the check is limited to DISTRO_FEATURES. This is the reason for having
this in OE-core instead of bitbake.

Default values are set so that no redundant parameters have to be
passed for conditional includes. As a secondary usage, the functions
can also be used in boolean checks.

(From OE-Core rev: 13024ce5aae453769b546d5fbe533443aec3d6fd)

Signed-off-by: Patrick Ohly <patrick.ohly@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-06-12 15:08:32 +01:00
Richard Purdie
eeb30764e9 lib/oe/utils: Add build_depends_string function
This is useful when manipulating depends strings for task [depends]
flags and is slightly easier to parse than some inline python.

(From OE-Core rev: 7b05ea65a8db8a27b2a5579675775ee34ceb63c2)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-01-20 11:53:49 +00:00
Richard Purdie
022a8b58c8 meta/scripts: Various getVar/getVarFlag expansion parameter fixes
There were a few straggling expansion parameter removals left for
getVar/getVarFlag where the odd whitespace meant they were missed
on previous passes. There were also some plain broken ussages such
as:

d.getVar('ALTERNATIVE_TARGET', old_name, True)
path = d.getVar('PATH', d, True)
d.getVar('IMAGE_ROOTFS', 'True')

which I've corrected (they happend to work by luck).

(From OE-Core rev: 688f7a64917a5ce5cbe12f8e5da4d47e265d240f)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-01-09 13:39:11 +00:00
Richard Purdie
d77a135e18 lib/oe/utils: Drop python2 compatibility code
We've moved to python3, we don't need this compatibility code which just makes
the code less readable.

(From OE-Core rev: 425afe2484707640ac71194885fdb263e95e9950)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-12-22 12:36:40 +00:00
Ross Burton
1969854ec3 utils: Always use datastore's PATH for host_gcc_version
BUILD_CC may reference something like ccache and expect this to come from
ccache-native, we at least have some selftests which assume this. Modify the
code to use PATH when runnig BUILD_CC to ensure the tests continue to work
as expected.

(From OE-Core rev: f3e753372baac43d0921186340cf260df056de20)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-12-16 10:23:23 +00:00