Commit Graph

4347 Commits

Author SHA1 Message Date
Richard Purdie
d41af446fe bitbake: data_smart: Allow colon in variable expansion regex
Now that ":" is a valid character in variable key names, it needs to be
allowed by the variable expansion code too, to match.

(Bitbake rev: 019251649a38754d5877759d13b664e28dea77de)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2021-07-29 23:21:24 +01:00
Paul Barker
261da52339 bitbake: parse/ast: Substitute '~' when naming anonymous functions
When parsing an anonymous python function, bitbake generates a name for
the function based on the full path to the file in which it was found.
As not all characters which are valid in file paths are valid in Python
function names we have a translation table. However, this translation
table was missing an entry for '~'.

(Bitbake rev: b201c0b284e25c20685d9d206797c4cc650eeca0)

Signed-off-by: Paul Barker <paul@pbarker.dev>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2021-07-29 23:21:24 +01:00
Joshua Watt
a83ea01b99 bitbake: bitbake: asyncrpc: Catch early SIGTERM
If the SIGTERM signal is sent to an asyncrpc server before it has
installed the SIGTERM handler in the main loop, it may miss the signal
which will can cause the calling process to wait forever on the join().
To resolve this, the calling process should mask of SIGTERM before
forking the server process and the server should unmask the signal only
after the handler is installed. To simplify the usage of the server, an
new helper function called serve_as_process() is added to do this
automatically and correctly.

Thanks: Scott Murray <scott.murray@konsulko.com> for helping debug
(Bitbake rev: ef2865efa98ad20823267364f2159d8d8c931400)

Signed-off-by: Joshua Watt <JPEWhacker@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2021-07-29 23:21:24 +01:00
Paul Barker
445c5b9324 bitbake: asyncrpc: Set timeout when waiting for reply from server
(Bitbake rev: b31f1853d7fcb8b8f8885ca513a0021a5d0301e6)

Signed-off-by: Paul Barker <pbarker@konsulko.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2021-07-29 23:21:24 +01:00
Paul Barker
57ec655acc bitbake: asyncrpc: Fix bad message error in client
If there is an issue with the format of the reply given by the server then we
should print this reply line in the error message. Printing the message which
the client sent doesn't illuminate anything here.

(Bitbake rev: bd8f8d7b055da15cd7bdd0b383061852a0f54cb7)

Signed-off-by: Paul Barker <pbarker@konsulko.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2021-07-29 23:21:24 +01:00
Joshua Watt
75f491e5e2 bitbake: server: Fix early parsing errors preventing zombie bitbake
If the client process never sends cooker data, the server timeout will
be 0.0, not None. This will prevent the server from exiting, as it is
waiting for a new client. In particular, the client will disconnect with
a bad "INHERIT" line, such as:

    INHERIT += "this-class-does-not-exist"

Instead of checking explicitly for None, check for a false value, which
means either 0.0 or None.

(Bitbake rev: 13e2855bff6a6ead6dbd33c5be4b988aafcd4afa)

Signed-off-by: Joshua Watt <JPEWhacker@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2021-07-20 19:00:32 +01:00
Joshua Watt
8410987884 bitbake: bitbake: Add piping compression library
Adds a library that implements file-like objects (similar to
gzip.GzipFile) that can stream to arbitrary compression programs. This
is utilized to implement a LZ4 and zstd compression API.

(Bitbake rev: 61c3acd058ea018696bd284b3922d0b458838d05)

Signed-off-by: Joshua Watt <JPEWhacker@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2021-07-20 18:59:18 +01:00
Richard Purdie
75fad23fc0 bitbake: data_smart/parse: Allow ':' characters in variable/function names
It is becomming increasingly clear we need to find a way to show what
is/is not an override in our syntax. We need to do this in a way which
is clear to users, readable and in a way we can transition to.

The most effective way I've found to this is to use the ":" charater
to directly replace "_" where an override is being specified. This
includes "append", "prepend" and "remove" which are effectively special
override directives.

This patch simply adds the character to the parser so bitbake accepts
the value but maps it back to "_" internally so there is no behaviour
change.

This change is simple enough it could potentially be backported to older
version of bitbake meaning layers using the new syntax/markup could
work with older releases. Even if other no other changes are accepted
at this time and we don't backport, it does set us on a path where at
some point in future we could
require a more explict syntax.

I've tested this patch by converting oe-core/meta-yocto to the new
syntax for overrides (9000+ changes) and then seeing that builds
continue to work with this patch.

(Bitbake rev: 0dbbb4547cb2570d2ce607e9a53459df3c0ac284)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2021-07-20 18:59:18 +01:00
Adam Romanek
5dce2f3da2 bitbake: fetch2/s3: allow to use credentials from environment variables
Previously access to AWS S3 was expected to be preconfigured and
credentials to be stored in ~/.aws/credentials. With this change
one can use Bitbake s3 fetcher without AWS credentials stored
permanently as above, just with them exported as the following
environment variables:
 - AWS_ACCESS_KEY_ID,
 - AWS_SECRET_ACCESS_KEY.
 - AWS_DEFAULT_REGION.

(Bitbake rev: 01825699044c42e87e485e6c64cc1dd9b6f87f48)

Signed-off-by: Adam Romanek <romanek.adam@gmail.com>
Signed-off-by: Damian Wrobel <dwrobel@ertelnet.rybnik.pl>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2021-07-11 23:01:25 +01:00
Denys Dmytriyenko
6b25d75ca8 bitbake: providers: replace newly added logger.warn() with logger.warning()
Commits https://git.openembedded.org/bitbake/commit/?id=78cd63285713fde59506eb2e71a7b7ee59a594ff
and https://git.openembedded.org/bitbake/commit/?id=5cbf6d95fc1009e78e7d0745a49e0bf418b37abb
added few calls to logger.warn(), which is deprecated and instead
should use logger.warning():

https://lists.openembedded.org/g/bitbake-devel/topic/82742194#12377

(Bitbake rev: a28ba2d31cd3aa557d4977e9376c5d01cd863e9a)

Signed-off-by: Denys Dmytriyenko <denis@denix.org>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2021-07-10 23:11:33 +01:00
Adam Romanek
6cd8032e35 bitbake: fetch/git: run gc in foreground to avoid race with tar
It looks like git gc can interrupts the package creation when
BB_GENERATE_MIRRORS_TARBALL is in use.

Log excerpts:

tar -czf TOPDIR/../downloads/git2_bitbucket.name-hidden.git.tar.gz . failed with exit code 1, output:
tar: ./objects/pack/pack-89a1d76f6c08f53172ef1d02ff851d90564362c4.pack: file changed as we read it
tar: ./objects/pack/pack-b4a48ada355d333630fdf6b4f67205b7c264dc2c.idx: File removed before we read it

Auto packing the repository in background for optimum performance.
See "git help gc" for manual housekeeping.

(Bitbake rev: a8d8cb847063862d1a7998963dd8b767ff73d877)

Signed-off-by: Adam Romanek <romanek.adam@gmail.com>
Signed-off-by: Damian Wrobel <dwrobel@ertelnet.rybnik.pl>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2021-07-10 22:43:41 +01:00
Chris Laplante
66f22c0bd1 bitbake: build: warn on setting noexec/nostamp/fakeroot flag to any value besides '1'
Currently setting those flags to even the empty string "" causes it to be set,
which is contrary to the documentation. In a future version of BitBake, we'd
like to change the behavior so that setting the flag to "" does not set it.
This will allow conditionally setting noexec, using variable expansion or
inline Python.

I found no places in poky or meta-openembedded where this warning would trigger.

[YOCTO #13808]

(Bitbake rev: 1e7655c4f765ba7b4791c4cca048a69bf8d9c93d)

Signed-off-by: Chris Laplante <chris.laplante@agilent.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2021-06-06 23:18:26 +01:00
Paul Barker
287ee25145 bitbake: asyncrpc: Reduce verbosity
(Bitbake rev: cc803609167b4c399ab768d9e131d618c086a4f2)

Signed-off-by: Paul Barker <pbarker@konsulko.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2021-06-06 23:18:26 +01:00
Paul Barker
cba7d8f361 bitbake: asyncrpc: Add ping method
This method is needed to support startup of the prservice. As it is so
generic we can add it to the common asyncrpc module.

(Bitbake rev: 25ccd697ea76f66b813be2296866b2d3405b079c)

Signed-off-by: Paul Barker <pbarker@konsulko.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2021-06-06 23:18:26 +01:00
Scott Weaver
cbb8744735 bitbake: fetch2: add check for empty SRC_URI hash string
No error was being reported when the hash string was set to empty.
For example: SRC_URI[md5sum] = ""

On a related note (not a bug):
Because whitespace in the string will result in a checksum mismatch, the error
message was updated to make it a little clearer why the error was thrown.
For example: SRC_URI[md5sum] = " " or
SRC_URI[md5sum] = " 209f8326f5137d8817a6276d9577a2f1"

Now creates a message like this:
File: '/home/scott/yocto-cache/downloads/rsync-3.2.3.tar.gz' has md5
checksum '209f8326f5137d8817a6276d9577a2f1' when ' 209f8326f5137d8817a6276d9577a2f1' was expected

[YOCTO #14232]

(Bitbake rev: a13510d0028e234ea2f4744b0d0c38558395c70f)

Signed-off-by: Scott Weaver <weaverjs@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2021-06-06 23:18:26 +01:00
Richard Purdie
2f12a20935 bitbake: server/process: Handle error in heartbeat funciton in OOM case
We've seen cases where an OOM error causes bitbake server to hang:

9171 02:21:09.127810 Command Completed
Traceback (most recent call last):
  File "/home/pokybuild/yocto-worker/qemux86/build/bitbake/bin/bitbake-server", line 51, in <module>
    bb.server.process.execServer(lockfd, readypipeinfd, lockname, sockname, timeout, xmlrpcinterface)
  File "/home/pokybuild/yocto-worker/qemux86/build/bitbake/lib/bb/server/process.py", line 550, in execServer
    server.run()
  File "/home/pokybuild/yocto-worker/qemux86/build/bitbake/lib/bb/server/process.py", line 108, in run
    ret = self.main()
  File "/home/pokybuild/yocto-worker/qemux86/build/bitbake/lib/bb/server/process.py", line 242, in main
    ready = self.idle_commands(.1, fds)
  File "/home/pokybuild/yocto-worker/qemux86/build/bitbake/lib/bb/server/process.py", line 370, in idle_commands
    bb.event.fire(heartbeat, self.cooker.data)
  File "/home/pokybuild/yocto-worker/qemux86/build/bitbake/lib/bb/event.py", line 216, in fire
    fire_class_handlers(event, d)
  File "/home/pokybuild/yocto-worker/qemux86/build/bitbake/lib/bb/event.py", line 123, in fire_class_handlers
    execute_handler(name, handler, event, d)
  File "/home/pokybuild/yocto-worker/qemux86/build/bitbake/lib/bb/event.py", line 93, in execute_handler
    ret = handler(event)
  File "/home/pokybuild/yocto-worker/qemux86/build/meta/classes/buildstats.bbclass", line 182, in defaultrun_buildstats
    write_host_data(os.path.join(bsdir, "host_stats"), e, d, "interval")
  File "/home/pokybuild/yocto-worker/qemux86/build/meta/classes/buildstats.bbclass", line 160, in write_host_data
    output = subprocess.check_output(c.split(), stderr=subprocess.STDOUT, timeout=limit).decode('utf-8')
  File "/usr/lib/python3.6/subprocess.py", line 356, in check_output
    **kwargs).stdout
  File "/usr/lib/python3.6/subprocess.py", line 423, in run
    with Popen(*popenargs, **kwargs) as process:
  File "/usr/lib/python3.6/subprocess.py", line 729, in __init__
    restore_signals, start_new_session)
  File "/usr/lib/python3.6/subprocess.py", line 1295, in _execute_child
    restore_signals, start_new_session, preexec_fn)
OSError: [Errno 12] Cannot allocate memory

We need to wrap the calls in the same high level wrapper as idle function calls
and trigger an exit upon an unhandled exception.

(Bitbake rev: 74042b5b89d5a170013fc1a327ce3a6530fbf7d5)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2021-05-18 23:53:15 +01:00
Alexander Kanavin
37792f1410 bitbake: fetch2/wget: when checking latest versions, consider all numerical directories
Previously the regex was maching x.y, but wasn't matching x, which is a problem
e.g. here:
https://download.gnome.org/sources/epiphany/
(the new gnome version scheme adds 40-series at the end).

(Bitbake rev: c03101576f447263ea38e8464210d3a3a2c27226)

Signed-off-by: Alexander Kanavin <alex.kanavin@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2021-05-14 07:57:57 +01:00
Richard Purdie
49da1ed2dc bitbake: bitbake: Switch to post release version number 1.51.0
(Bitbake rev: 97a64d12f70eb02f1d35b4ffefb291b80ca8c425)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2021-05-06 11:08:08 +01:00
Harald Brinkmann
e155a81ab6 bitbake: fetch/svn: Fix parsing revision of SVN repos with redirects
svn was printing a message when encountering HTTP redirects.
This confused the revision parser.

(Bitbake rev: a944a335f8f4c4fe5df55f3d7d8e757bd2835146)

Signed-off-by: Harald Brinkmann <Harald.Brinkmann@detectomat.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2021-05-06 11:08:08 +01:00
Devendra Tewari
581233a798 bitbake: lib/bb: Add bb.utils.rename() helper function and use for renaming
os.rename can fail for example an incremental build in Docker fails with:

OSError: [Errno 18] Invalid cross-device link

when source and destination are on different overlay filesystems.

Rather than trying to fix every call site, add a wrapper in bb.utils
for renames. We can then handle cross device failures and
fall back to shutil.move. The reason os.rename is still used is
because shutil.move is too slow for speed sensitive sections of code.

[YOCTO #14301]

(Bitbake rev: c5c4e49574ab2a65e06298a0a77bb98b041cf56b)

Signed-off-by: Devendra Tewari <devendra.tewari@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2021-05-06 11:04:59 +01:00
Richard Purdie
93aaa5e994 bitbake: runqueue: Handle deferred task rehashing in multiconfig builds
If the hash of a task changes and that hash is a deferred task (e.g. a multiconfig
build), we need to ensure that the hash change propagates through to all the tasks
else the build will run multiple copies of the task, sometimes with oddly differing
results as the outhashes of native tasks built in differing locations can confuse
things.

(Bitbake rev: 2db571324f755edc4981deecbcfdf0aaa5a97627)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2021-05-01 22:51:06 +01:00
Richard Purdie
76891afd76 bitbake: runqueue: Fix multiconfig deferred task sstate validity caching issue
We were testing the validity of deferred tasks setscene status "up front" which
is very unlikely to succeed and leads to cache invalidation issues. With the
change to rebuild the deferred task list, this status becomes out of sync. The
result was tasks being executed when they should not have been leading to extra
work for the build unnecessarily.

Instead, don't process validity status for deferred tasks and assume their
data will become available. If it doesn't, this will now result in a build
error as the setscene task will fail and the main task will run instead.

In theory we could try and track the state changes in the deferred list and
re-test validity then but I'm not sure it is worth the effort when the other
code path and errors in setscene tasks will give a pretty good idea of what
is happening anyway.

(Bitbake rev: edcafac13b3b241b6687419e59018d21811507a1)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2021-05-01 22:51:06 +01:00
Joshua Watt
f3e1a668fb bitbake: knotty: Re-enable command line logging levels
The "-l" command line options to enable specific logging domains wasn't
working with the switch to structured logging because they were only
being used to set the legacy logging domains. Fix this by implementing
the logic to parse the user options into the logging configuration.

(Bitbake rev: 005fc7a8c588d0b0bca382469645cbf481ad8e30)

Signed-off-by: Joshua Watt <JPEWhacker@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2021-05-01 22:51:06 +01:00
Przemyslaw Gorszkowski
6da327c788 bitbake: fetch/s3: Add progress handler for S3 cp command
Adds progress support for fetchers from S3.

(Bitbake rev: 90d31b2d5a81e5f41fe95907c78fd2f5f36e39ee)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2021-05-01 22:51:06 +01:00
Przemyslaw Gorszkowski
a854068b52 bitbake: progress: LineFilterProgressHandler - Handle parsing line which ends with CR only
S3 commands need to handle different CR only line endings, update the handler
to cope with this.

(Bitbake rev: 3f7b9c1b429a4c68240e80832a8ef93ee210e5ff)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2021-05-01 22:51:06 +01:00
Paul Barker
244b044fd6 bitbake: asyncrpc: Common implementation of RPC using json & asyncio
The hashserv module implements a flexible RPC mechanism based on sending
json formatted messages over unix or tcp sockets and uses Python's
asyncio features to build an efficient message loop on both the client
and server side. Much of this implementation is not specific to the
hash equivalency service and can be extracted into a new module for
easy re-use elsewhere in bitbake.

(Bitbake rev: 4105ffd967fa86154ad67366aaf0f898abf78d14)

Signed-off-by: Paul Barker <pbarker@konsulko.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2021-04-27 15:12:57 +01:00
Paul Barker
1023671823 bitbake: hashserv: Use generic ConnectionError
The Python built-in ConnectionError type can be used instead of a custom
HashConnectionError type. This will make code refactoring simpler.

(Bitbake rev: 8a796c3d6d99cfa8ef7aff0ae55bb0f23bbbeae1)

Signed-off-by: Paul Barker <pbarker@konsulko.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2021-04-27 15:12:57 +01:00
Niels Avonds
d12a06cd49 bitbake: fetch/gitsm: Fix crash when using git LFS and submodules
Gitsm fetcher crashes when cloning a repository that contains LFS files.
This happens because the unpack method is called during download, but the
submodules have not been downloaded yet at this point.

This issue was introduced in this
commit: 977b7268bf

[YOCTO #14283]

(Bitbake rev: 26caedc4d2e9b5a0f1d57f9291754a7f6c5e437e)

Signed-off-by: Niels Avonds <niels@codebits.be>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2021-04-20 13:57:50 +01:00
Ross Burton
01066a584a bitbake: bitbake-server: ensure server timeout is a float
bitbake-server is spawned by process.py and passes the arguments it is
given to ProcessServer.  There's some type confusion here:

bitbake-server is called with a string representation of the timeout,
which may be None.  If the timeout is not set, pass 0 instead of None.

Inside bitbake-server a ProcessServer is created which expects the
timeout to be a float not a string, so always float() the value.

[ YOCTO #14350 ]

(Bitbake rev: c93ae1f861208f6d39fd15c84fbcd0e2b54331f5)

Signed-off-by: Ross Burton <ross.burton@arm.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2021-04-20 13:57:50 +01:00
Richard Purdie
d08d362cf9 bitbake: tinfoil/data_smart: Allow variable history emit() to function remotely
We can't access the emit() function of varhistory currently as the datastore parameter
isn't handled correctly, nor is the output stream. Add a custom wrapper for this
function which handles the two details correctly.

(Bitbake rev: ba0fa084ccd2b1ade96425d158fd31e49e42f286)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2021-04-18 11:38:22 +01:00
Henning Schild
46654e14a5 bitbake: tests/fetch: add tests for local and remote "noshared" git fetching
(Bitbake rev: e0267fe43bda208856af939b17e39beb9e5586c3)

Signed-off-by: Henning Schild <henning.schild@siemens.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2021-04-18 11:38:22 +01:00
Henning Schild
d274cf0cf8 bitbake: tests/fetch: deduplicate local git testing code
Purely cosmetic change that probably improves the code.

(Bitbake rev: 9c0733f0062f3cf19514c891cc06c9a8e0db429b)

Signed-off-by: Henning Schild <henning.schild@siemens.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2021-04-18 11:38:22 +01:00
Henning Schild
2cb6ce7788 bitbake: fetch/git: add support for disabling shared clones on unpack
By default the unpacker will create a "shared" clone when cloning from
the DL_DIR to the WORKDIR. This patch introduces an option to control
that behaviour.

Imagine some recipe steps are executed in a namespace that is different
from the one your downloader and unpacker ran in. (chroot) Because a
"shared" clone has an absolute reference to its "alternate" you now
have to make that "alternate" visible in that new namespace (chroot) at
the exact place.

With this patch you can unpack "noshared" and get a stand-alone copy.
This copy will also work if the "alternate" is not visible or existant.

The switch is a global bitbake switch and will affect all git urls.
Build systems that need "noshared" most likely need it for everything
they do with git.

(Bitbake rev: 6ae6f1865d5e666ebc670f70b7401a7b41648102)

Signed-off-by: Henning Schild <henning.schild@siemens.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2021-04-18 11:38:22 +01:00
Mikko Rapeli
33650ffdc7 bitbake: bitbake: tests/fetch: remove write protected files too
For some reason several git-annex files in Debian 10 buster
are read-only and removing them with "rm -rf" fails.

Fixes test failures like:

$ bitbake-selftest
...
rm: cannot remove '/tmp/tmpwmfn4w64/download/git2/tmp.tmpwmfn4w64.gitsource/annex/objects/f87/4d5/SHA256E-s0--e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855/SHA256E-s0--e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855': Permission denied
rm: cannot remove '/tmp/tmpwmfn4w64/download/git2/tmp.tmpwmfn4w64.gitsource/annex/objects/f87/4d5/SHA256E-s0--e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855/SHA256E-s0--e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855': Permission denied
EE..................................ssss.sssssssssssssss.sssss.......................................................................................................
======================================================================
ERROR: test_shallow_annex (bb.tests.fetch.GitShallowTest)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/builder/src/base/poky/bitbake/lib/bb/tests/fetch.py", line 1773, in test_shallow_annex
    fetcher, ud = self.fetch_shallow(uri)
  File "/home/builder/src/base/poky/bitbake/lib/bb/tests/fetch.py", line 1541, in fetch_shallow
    bb.utils.remove(ud.clonedir, recurse=True)
  File "/home/builder/src/base/poky/bitbake/lib/bb/utils.py", line 700, in remove
    subprocess.check_call(cmd + ['rm', '-rf'] + glob.glob(path))
  File "/usr/lib/python3.7/subprocess.py", line 347, in check_call
    raise CalledProcessError(retcode, cmd)
subprocess.CalledProcessError: Command '['rm', '-rf', '/tmp/tmpwmfn4w64/download/git2/tmp.tmpwmfn4w64.gitsource']' returned non-zero exit status 1.

Also, one "chmod" call was failing since the .git/annex subdirectory doesn't exist so just chmod
the whole temporary directory which should cover any directory name differences between
different git-annex versions. Fixes tests failing after chmod call:

Running 'export PSEUDO_DISABLED=1; unset _PYTHON_SYSCONFIGDATA_NAME; chmod u+w -R /tmp/tmpwmfn4w64/git//.git/annex' in /tmp/tmpwmfn4w64/git/

(Bitbake rev: 7729ef2983c72867e99fad82d671069ba5cb32b2)

Signed-off-by: Mikko Rapeli <mikko.rapeli@bmw.de>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2021-04-18 11:38:22 +01:00
Mikko Rapeli
490fb73e34 bitbake: bitbake: tests/fetch: fix test execution without .gitconfig
A CI user validating changes does not have any git push rights or
even a .gitconfig file so fix tests so that they run
by setting the user.name and user.email for the repo before
committing changes.

Fixes errors like:

ERROR: test_that_unpack_throws_an_error_when_the_git_clone_nor_shallow_tarball_exist (bb.tests.fetch.GitShallowTest)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/builder/src/base/poky/bitbake/lib/bb/tests/fetch.py", line 2055, in test_that_unpack_throws_an_error_when_the_git_clone_nor_shallow_tarball_exist
    self.add_empty_file('a')
  File "/home/builder/src/base/poky/bitbake/lib/bb/tests/fetch.py", line 1562, in add_empty_file
    self.git(['commit', '-m', msg, path], cwd)
  File "/home/builder/src/base/poky/bitbake/lib/bb/tests/fetch.py", line 1553, in git
    return bb.process.run(cmd, cwd=cwd)[0]
  File "/home/builder/src/base/poky/bitbake/lib/bb/process.py", line 184, in run
    raise ExecutionError(cmd, pipe.returncode, stdout, stderr)
bb.process.ExecutionError: Execution of 'git commit -m a a' failed with exit code 128:

*** Please tell me who you are.

Run

  git config --global user.email "you@example.com"
  git config --global user.name "Your Name"

to set your account's default identity.
Omit --global to set the identity only in this repository.

(Bitbake rev: 57c0811f1ee19b6619f4840a39e01e3cb98c34c4)

Signed-off-by: Mikko Rapeli <mikko.rapeli@bmw.de>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2021-04-18 11:38:22 +01:00
Richard Purdie
7ea37b9291 bitbake: runqueue: Fix deferred task issues
In a multiconfig situation there are circumstances where firstly, tasks
are deferred when they shouldn't be, then later, tasks can end up as
both covered and not covered.

This patch fixes two related issues. Firstly, the stamp validity checking
is done up front in the build and not reevaulated. When rebuilding the
deferred task list after scenequeue hash change updates, we need therefore
need to check if a task was in notcovered *or* covered when deciding to
defer it. This avoids strange logs like:

NOTE: Running setscene task X of Y (mc:initrfs_guest:/A/alsa-state.bb:do_deploy_source_date_epoch_setscene)
NOTE: Deferring mc:initrfs_guest:/A/alsa-state.bb:do_deploy_source_date_epoch after mc:host:/A/alsa-state.bb:do_deploy_source_date_epoch

where tasks have run but are then deferred.

Since we're recalculating the whole list, we also need to clear it before
iterating to rebuild it. By ensuring covered tasks aren't added to the
deferred queue, the covered + notcovered issue should also be avoided.
in the task deadlock forcing code.

[YOCTO #14342]

(Bitbake rev: 3c8717fb9ee1114dd80fc1ad22ee6c9e312bdac7)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2021-04-18 11:38:22 +01:00
Richard Purdie
ca07d22405 bitbake: bitbake: Update version to 1.50.0 stable release series
(Bitbake rev: e70b925ba98fd4fedf3940d141a4210c953087ca)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2021-04-09 14:20:01 +01:00
Richard Purdie
9982b2c0f2 bitbake: runqueue: Further fixes for confused setscene tasks
There is further evidence of tasks ending up being "covered" and "notcovered"
which shouldn't happen and is bad. The code that caused this problem last
time appears to have issues where stamps for tasks already exist.

Split out the setscene stamp checking code to a separate function and
use this when checking "hard dependencies" (like pseudo-native) so
that if the stamps exist and it will be "covered", it is not put on
the notcovered list.

(Bitbake rev: a1848a481e36b729c8e4130c394b1d462d4b488a)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2021-04-06 09:31:24 +01:00
Michael Opdenacker
62880ffb17 bitbake: doc: Update links to documentation
(Bitbake rev: ed8e1fd4cf9d5ac8a8203638add99d686b4b3521)

Signed-off-by: Michael Opdenacker <michael.opdenacker@bootlin.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2021-03-23 22:51:25 +00:00
Richard Purdie
86ec266d91 bitbake: runqueue/event: Add an event for notifying of stale setscene tasks
Use the new functionality in build.py to identify stale setscene tasks
and send an event to the metadata listing them. The metadata then
has the option of performing cleanup operations if it thinks that
appropriate.

(Bitbake rev: ef8c980a3ae92c168b7ca16a4d19cd38a9574761)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2021-03-23 22:51:25 +00:00
Richard Purdie
ab55f8bfec bitbake: build: Add find_stale_stamps function
Add a new function which compares the stamp filename we want (including
taskhash) with what is in the stamp directory (using the clean mask).
This tells us which stamp files are stale and are due to be rerun.

(Bitbake rev: b126a53882d202e4df0f9661303355c9fe9ec80e)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2021-03-23 22:51:25 +00:00
Tomasz Dziendzielski
a9098e6423 bitbake: codeparser: Fix TypeError in bitbake debug mode
Commit 75f87db413 fixed the confusion between bitbake and python logger
but in codeparser still old method of setting debug level was used
causing TypeError, because debug level value was incorrectly returned
and assigned to event.msg.
| File "./bitbake/lib/bb/ui/knotty.py", line 660, in main
|   event.msg = event.fn + ': ' + event.msg
| TypeError: can only concatenate str (not "int") to str

[YOCTO #14298]

(Bitbake rev: bec9ea7ab28a8dfad1a6010ca5c6ec691754748d)

Signed-off-by: Tomasz Dziendzielski <tomasz.dziendzielski@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2021-03-15 21:47:42 +00:00
Charlie Davies
cb3cbd8bb2 bitbake: bitbake: cooker: Add REQUIRED_VERSION checks
Add logic to check that if REQUIRED_VERSION has been set that the cooker
class method findBestProviders properly handles the case where the
REQUIRED_VERSION has not been found.

Fixes [YOCTO #10096]

(Bitbake rev: 5df201d746f26154213e6d15d2721cd35b38b59e)

Signed-off-by: Charlie Davies <charles.davies@whitetree.xyz>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2021-03-15 14:36:52 +00:00
Charlie Davies
889a7ce8de bitbake: bitbake: lib: add Required Version to Bitbake --show-versions command
This commit adds a new column to the Bitbake --show-versions command
called Required Version. This column will display any packages which
have a REQUIRED_VERSION successfully set.

Fixes [YOCTO #10096]

(Bitbake rev: 90c7d1815e41243323d32b9dbb865757a922578a)

Signed-off-by: Charlie Davies <charles.davies@whitetree.xyz>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2021-03-15 14:36:52 +00:00
Charlie Davies
78fcf6831a bitbake: bitbake: providers: check for REQUIRED_VERSION in _filterProviders
Before the REQUIRED_VERSION variable was introduced the
PREFERRED_VERSION variable allowed for a fallback to the next most
suitable version.

Since REQUIRED_VERSION does not allow a fallback to a different version
implement a check in the _filterProviders function to make sure that
if a requested REQUIRED_VERSION is not found then the function returns
no eligible providers have been found.

Fixes [YOCTO #10096]

(Bitbake rev: c41386b78aa53e0bf081cd973c950b88126670a7)

Signed-off-by: Charlie Davies <charles.davies@whitetree.xyz>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2021-03-15 14:36:51 +00:00
Charlie Davies
3752782cb7 bitbake: bitbake: providers: introduce logic for REQUIRED_VERSION variable
This commit adds checks during the findPreferredProvider function for a
new variable REQUIRED_VERSION. This can be set, in exactly the same
manner as PREFERRED_VERSION, on a per package basis to enforce the use
of a particular version of a package.

REQUIRED_VERSION is similar in behaviour to PREFERRED_VERSION except if
the version specified by REQUIRED_VERSION is not found an error occurs
and the execution of Bitbake stops.

Fixes [YOCTO #10096]

(Bitbake rev: 5cbf6d95fc1009e78e7d0745a49e0bf418b37abb)

Signed-off-by: Charlie Davies <charles.davies@whitetree.xyz>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2021-03-15 14:36:51 +00:00
Charlie Davies
70f90dcfd7 bitbake: bitbake: providers: consistent single line formatting between functions
(Bitbake rev: e8c74ac8e589ee44d862b71fd63dd75ab303d3b5)

Signed-off-by: Charlie Davies <charles.davies@whitetree.xyz>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2021-03-15 14:36:51 +00:00
Charlie Davies
04bb096170 bitbake: bitbake: providers: remove unneeded logging call
The logging line for when no eligible providers are found only adds
unnecessary noise and is not required. As the stack unwinds better
logging occurs which satisfactorily describes the no eligible providers
error condition.

(Bitbake rev: 933c9df919f7947f515d843199cce2621cf8195f)

Signed-off-by: Charlie Davies <charles.davies@whitetree.xyz>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2021-03-15 14:36:51 +00:00
Richard Purdie
cca5433baf bitbake: tests/color: Fix event register to pass the datastore
Adding the "if d is None" to the event handling code meant some of
these tests stopped working. The reason is that len(d) was zero but
not equal to None.

Passing the data object to the register() function in the test
correctly registers the event handler and avoids the problem, it
just happened to work previously, incorrectly.

(Bitbake rev: 426eb83c6668d82a2ebaca6c672db131e37c11da)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2021-03-12 15:28:57 +00:00
Richard Purdie
22d7989e8f bitbake: event: Fix another 'if d' test to test for None explictly
This should have been fixed in the previous patch, well spotted Chris!

(Bitbake rev: c3f6fee42bfa23f23f167cb29f0cfa05ac2fa197)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2021-03-11 21:38:32 +00:00