In case a recipe specifies a git SRC_URI along with revision and tag, but only the
revision is present in the local clone without the tag (because it was tagged after
it was cloned), then unpacking fails with the following error:
... rev-list -n 1 1.0 failed with exit code 128, output:\nfatal: ambiguous argument \'1.0\': unknown revision or path not in the working tree
This happens because the during the download step only the revision's presence is
verified to decide if the repository needs to be updated.
To avoid this, check also if the tag is present in the local repository, when the "tag"
tag is specified.
(Bitbake rev: 546b347b4d3d82c01ecc99f45296f66e44638adc)
Signed-off-by: Gyorgy Sarvari <skandigraun@gmail.com>
Signed-off-by: Mathieu Dubois-Briand <mathieu.dubois-briand@bootlin.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Refactoring prints, print() functions were eliminated from all loops and it uses
"\n".join(...) in a single print() call for better performance.
(Bitbake rev: c32c3d9b83818661e12f3e437563ab4e1fa05e15)
Signed-off-by: Benjamin Szőke <egyszeregy@freemail.hu>
Signed-off-by: Mathieu Dubois-Briand <mathieu.dubois-briand@bootlin.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Refresh footer in 40 Hz to avoid heavy print() flooding but keep it fluent for human eyes.
(Bitbake rev: c36efdf642d858c6997819744d00a3c1965c6417)
Signed-off-by: Benjamin Szőke <egyszeregy@freemail.hu>
Signed-off-by: Mathieu Dubois-Briand <mathieu.dubois-briand@bootlin.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Optimize printing in footer update with use a StringIO buffer and it
prints content to terminal in a single call in every cycle.
(Bitbake rev: 32ba622d78f20b231f30f848379b4bbc3d7414da)
Signed-off-by: Benjamin Szőke <egyszeregy@freemail.hu>
Signed-off-by: Mathieu Dubois-Briand <mathieu.dubois-briand@bootlin.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Introduce self._fd_console as a dedicated attribute of self._handle_resize().
(Bitbake rev: f8c76eb89d52b1c28407f0b52dfe4318faa47cd2)
Signed-off-by: Benjamin Szőke <egyszeregy@freemail.hu>
Signed-off-by: Mathieu Dubois-Briand <mathieu.dubois-briand@bootlin.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Prior to commit aa84a900e ("cooker: Ensure delays are accurately transfered to
the idle loops from runqueue"), this was necessary. But now retval is returned
directly.
(Bitbake rev: c2eb4336fe10b1bf8bbc6291c32ca362840f39e1)
Signed-off-by: Chris Laplante <chris.laplante@agilent.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Marks bb.utils.explode_deps as callable from filter functions
(Bitbake rev: bb07003641e76de994482f7835a432f20297af96)
Signed-off-by: Joshua Watt <JPEWhacker@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Adds a new setVarFilter() API to the data store allowing filters to be
applied to variables.
Note that filters are applied to the non-override part of the variable name
so a filter set against RDEPENDS would apply against RDEPENDS:${PN} and
friends.
The filter function is applied before returning the final variable value.
(Bitbake rev: a9471c10d1de039474ddb4738abd286b928d82f4)
Signed-off-by: Joshua Watt <JPEWhacker@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Add the python API for applying filters to a string and being able to
register functions as filters.
Filter functions are pure functions where an input is translated into
an output and there are no external data accesses. This means translations
can be cached as they won't change.
(Bitbake rev: 7d25d7511ca14213eea78ee739d260295cfa4045)
Signed-off-by: Joshua Watt <JPEWhacker@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
When parsing failures occur, trap KeyError to avoid these kinds
of tracebacks (from a parsing error in meta-ti).
bb.parse.ParseError: ParseError at /srv/pokybuild/yocto-worker/check-layer-nightly/build/meta-ti/meta-ti-bsp/recipes-kernel/linux/linux-ti-mainline_git.bb:6: Could not inherit file classes/${KERNEL_BASE_CLASS}.bbclass
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/srv/pokybuild/yocto-worker/check-layer-nightly/build/lib/bb/command.py", line 123, in runAsyncCommand
self.cooker.updateCache()
File "/srv/pokybuild/yocto-worker/check-layer-nightly/build/lib/bb/cooker.py", line 1639, in updateCache
if not self.parser.parse_next():
File "/srv/pokybuild/yocto-worker/check-layer-nightly/build/lib/bb/cooker.py", line 2314, in parse_next
self.shutdown(clean=False, eventmsg=str(exc))
File "/srv/pokybuild/yocto-worker/check-layer-nightly/build/lib/bb/cooker.py", line 2209, in shutdown
read_results()
File "/srv/pokybuild/yocto-worker/check-layer-nightly/build/lib/bb/cooker.py", line 2195, in read_results
self.result_queue.get(timeout=0.25)
File "/usr/lib/python3.10/multiprocessing/queues.py", line 122, in get
return _ForkingPickler.loads(res)
File "/srv/pokybuild/yocto-worker/check-layer-nightly/build/lib/bb/cache.py", line 338, in __setstate__
setattr(self, key, self._restore(state[key], pid))
File "/srv/pokybuild/yocto-worker/check-layer-nightly/build/lib/bb/cache.py", line 318, in _restore
ret[dep] = map[mapnum]
KeyError: 156
(Bitbake rev: 750c68ee7ee3f3d4518348e4c948243504880770)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Fixes [YOCTO 15917]
When fetching a git repository the .gitattributes file is scanned, checking if LFS
support is required for the repository. This scan checks if the word "lfs" is present
in the file, however the used regex doesn't account for comments, which makes some
repositories[1] be to misidentified as requiring LFS support (which fails fetching, in case
lfs support isn't installed on the build host).
To avoid it, change the used regex to ignore lines starting with "#".
[1]: e.g. https://github.com/MicrochipTech/cryptoauthlib
(Bitbake rev: 7917a758fc328747116c7899e689171bd0efc883)
Signed-off-by: Gyorgy Sarvari <skandigraun@gmail.com>
Signed-off-by: Mathieu Dubois-Briand <mathieu.dubois-briand@bootlin.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
When recreating the uri in wget's checkstatus method, we only use the
scheme, netloc and path. This completely strips the query parameters
from the final URI and potentially breaks the checking functionality
from certain fetchers.
This is the case for the Azure storage fetcher, as it requires a SAS
token that is formatted as a series of query parameters. The error
manifests itself when using a private storage account as a PREMIRROR or
SSTATE_MIRROR (since regular SRC_URI won't run the checkstatus).
This problem is present in scarthgap, but wasn't in kirkstone.
CC: Mathieu Dubois-Briand <mathieu.dubois-briand@bootlin.com>
(Bitbake rev: 096301250455e2a83bdd818a56317c62436c9981)
Signed-off-by: Philippe-Alexandre Mathieu <pamathieu@poum.ca>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
This API is no longer used and bitbake has moved beyond Python 2.7.3 as
the minimum version, so remove it.
(Bitbake rev: 0eb7b5dd512ed8d8b77b5779858b9fbd99edb4a4)
Signed-off-by: Joshua Watt <JPEWhacker@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Python 3.14 changes the default multiprocessing context from "fork" to
"forkserver"; however bitbake heavily relies on "fork" to efficiently
pass data to the child processes. As such, make "fork" context in the bb
namespace and use it in place of the normal multiprocessing module.
Note that multiprocessing contexts were added in Python 3.4, so this
should be safe to use even before Python 3.14
[YOCTO #15858]
(Bitbake rev: 62be9113d98fccb347c6aa0a10d5c4ee2857f8b6)
Signed-off-by: Joshua Watt <JPEWhacker@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The upstream servers are having issues so switch to our own shadow copy
of the repo.
(Bitbake rev: e910c7cd24fd366d6756641cd599c4efeb492e2a)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Use a more logical name for the profile reports and put each report
into a separate file since people struggle to discover them currently.
(Bitbake rev: a8145c84e0899285a5e6a809f1515118b002b106)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Running "time bitbake -pP idle" with a valid cache shows around 800,000
calls to enum creation from python's signal.py. We don't care about this
overhead and it adversely affects cache load time quite badly.
Try and use _signal directly, falling back to signal, which avoids
this overhead we don't need and makes cache loading much faster.
(Bitbake rev: ee5fce67ce35b025c68aa61e2e758903269ee346)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Starting with python 3.12, profiling now stays enabled over threads yet
you can't extract the profile data in the threads themselves, which makes it
difficult to use for our use case.
Our main loop starts the idle loop which starts the parsing threads and this
means we can't profile in the main loop and the parsing threads or the idle
loop at the same time due to this.
Add options to the commandline so you can specify which piece of bitbake
you want to enable profiling for. This allows some profiling with python 3.12
onwards rather than crashing.
(Bitbake rev: 09f29a4968841ee5070f70277ba8c253bb14f017)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
It is possible for multple bitbake threads to empty ui_queue in parallel
leading to duplicate console messages and much confusion when debuging.
Use the lock to extract the queue data which means only one thread will
processing, removing the duplicate out of order messages.
(Bitbake rev: 945095602e40d54efb8de494218f4a2b25c9969f)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
We have code duplication in the way we handle profiling of code sections.
Create a common function in utils which covers this.
The main loop and idle loop profile files were also reversed. Fix this and the naming,
removing a couple of unused variables containing the profile log names in the process too.
(Bitbake rev: b4f6bae97ac9607420fc49fd4c9e957d89c9a5f3)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
If the server is quickly stopped, we see tracebacks in the locks
due to the file not existing. Hide these as they're not errors.
(Bitbake rev: a7e1a07e9ef7e6f6a1bcaf567d7916a8ee1ef087)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The test behavior did not change visibly though.
"bitbake-selftest bb.tests.runqueue" passes completely, just like before.
(Bitbake rev: 1751aed08f8472f20fcfbadbb09d35f951904952)
Signed-off-by: Gyorgy Sarvari <skandigraun@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Instead of pre-partitioning which jobs will go to which parser
processes, pass the list of all jobs to all the parser processes
(efficiently via fork()), then used a shared counter of the next index
in the list that needs to be processed. This allows the parser processes
to run independently of needing to be feed by the parent process, and
load balances them much better.
(Bitbake rev: 373c4ddaf0e8128cc4f7d47aefa9860bd477a00f)
Signed-off-by: Joshua Watt <JPEWhacker@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
If parsing ends early and unexpectedly, add some internal values
to better understand why/how it failed.
(Bitbake rev: 775f9720a17c9f3d6815d42c733ab5aaaa53749c)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
We sometimes see hangs in parsing during automated testing. It appears that
SIGINT was sent to the underlying processes which see KeyboardInterrupt but
they're stuck trying to write into the results pipe. The SIGINT was probably
from some kind of parsing failure which doens't happen often, hence the hang
being rare (in the incompatible license selftests from OE).
This patch:
* sets a flag to indicate exit upon SIGINT so the exit is more graceful
and a defined exit path
* empties the results queue after we send the quit event
* empties the results queue after the SIGINT for good measure
* increases the 0.5s timeout to 2s since we now have some very slow to
parse recipes due to class extensions (ptests)
This should hopefully make the parsing failure codepaths more robust.
(Bitbake rev: 5b533370595f83b87e480bace3e0b42c9ba61e22)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
We have other places in the code where we need to take filemode/mask
information from a bitbake variable and turn it into a real python
number. Turn this internal code into public API in bb.utils and
add some tests for it.
(Bitbake rev: d89e30fb2fb15b09f2cb95c4e5aa9f749ca257ea)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The bitbake worker/server IPC is asynchronous, but tinfoil only has
functionality to wait for a response on the build_targets() call.
Extract the bulk of the "wait for events and handle errors" logic to a
standalone wait_for wrapper, which is the build_targets code without the
extra_events or event_callback arguments (for now).
Then use this to create a build_file_sync() helper that just wraps the
existing build_file() with @wait_for.
(Bitbake rev: bacd125a9da66cd205f6ba2ab17930b976e82150)
Signed-off-by: Ross Burton <ross.burton@arm.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
It makes sense to allow the shallow clone mirror tarball name to include the tag
name so that tags can be added to existing urls and the tarballs will be maintained
correctly. The code already allows this to be done easily just by moving the tag
handling code.
(Bitbake rev: 68fce3be14e4dd801661f4ef302d229fb16a04b5)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Currently, tags are fetched as just a name, which works but means they're not
seen as tags by git commands like git describe. Instead, fetch them as refs/tags/XXX
which means such commands then work correctly.
(Bitbake rev: c1f30ad61f5e55beb377451887bbbc5cb569f2e5)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The display window scrolling mechanism we were using was failing in one case
and needed in another. After trying various approaches, this one with
window.scrollTo seems to work and fixes the test failures.
(Bitbake rev: f1175778108a559e1dd0d55cb68f42816c86f393)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
When reviewing proposed fragments to add settings for DISTRO and MACHINE,
RP noted that such fragments only add clutter and overhead, and there's
no need to maintain them as separate files.
Rather when bitbake sees 'fragmentvar/fragmentvalue' it can expand that into
FRAGMENTVAR = "fragmentvalue".
To achieve that, 'addfragments' directive is extended with a parameter
that sets the name of the variable that holds definitions of such
built-in fragments, for example like this:
"machine:MACHINE distro:DISTRO"
Then each enabled fragment name is matched against these definitions and the
respective variable is set, e.g. 'machine/qemuarm' would match
'machine:MACHINE' and result in MACHINE set to 'qemuarm'.
This happens before any fragment files are looked up on disk,
and no such lookup happens if there was a match, which should prevent
possible misuse of the feature. So the builtin fragment definition
is also an allowlist for them.
Please also see the patches for oe-core that show an application of the feature.
(Bitbake rev: 3b9d7bea915dc7f10e845854f1dae325743f9456)
Signed-off-by: Alexander Kanavin <alex@linutronix.de>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
This will allow setting the variable in oe-core such that it matches
S, which in turn will allow dropping S from almost all git:// fetching recipes,
and not having to set it in newly written ones.
(Bitbake rev: 378db0fdd95f8704fccd852452555bfddcad384d)
Signed-off-by: Alexander Kanavin <alex@linutronix.de>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Update version to 2.15.0 for the development series and features needed for
toolchain selection in OE.
(Bitbake rev: c2f29c9475c4b9cdd12af1f8610f2675f8fdd964)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Add support for automatically promoting class inherits to deferred inherits
by listing them in the BB_DEFER_BBCLASSES variable.
(Bitbake rev: 8e741b2e885a12d119788d04aa4efcd724dd6bfa)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Now that deferred inherits are extension specific, we can pass this
list to an event, which our metadata can use to set class overrides
earlier (as an example).
There are limitations to this, the list of classes is unexpanded and
recursive classes are not visible. There isn't much that can be done
about this, the ones we are interested in would usually be visible
at the top level (such as class extensions).
(Bitbake rev: 205d461c05fc7b4a7c81039af3bc3fd71cbb982c)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Currently deferred inherits get processed once for all class extensions
as a minor speed optimisation. Unfortunately this limits our options for
being able to report deferred classes to our code.
There are two challenges with using our deferred classes in OE at present.
One is that PACKAGECONFIG values don't work well with class overrides like
class-native if there are deferred classes based on PACKAGECONFIG, such
as python support. The second is that toolchain selection is proving
problematic to implement due to interactions between the toolchain deferred
inherit, the class extensions and class overrides being very late.
By changing deferred inherits to be recipe extension specific, we open
the way to generate events and "peek" at where things will end up,
allowing the class overrides to be set earlier.
The class extension code is updated to use a deferred inherit for the
class extension inheriting so that it is still inherited last.
(Bitbake rev: 29277cf4d88eb4dfa9572851177d009eab5afd0c)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
AZ_SAS token should be prefixed with a question mark. Add a sanity check for
this and fix the documentation.
[YOCTO #15882]
(Bitbake rev: 22011765202514600314732b97f1bb938e21f585)
Signed-off-by: Robbin Van Damme <robbinvandamme@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The project is switching the way handle our CDN provision of sstate objects,
update the URL accordingly.
(Bitbake rev: 3372524cb961d95993b27fe4a8d794cdb7255e09)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
- Since `ud.path` contains in that case the `PREMIRRORS` prefix path,
this change ensures that a correct symlink is set up.
(Bitbake rev: 37ed18e45aa17406162efc5ee3ddb2d6b33d07b9)
Signed-off-by: Stefan Koch <stefan-koch@siemens.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Some .deb files contain uncompressed data tarballs which do not have an
additional file extension after `.tar`. Add support for such cases while
preserving the existing behaviour.
(Bitbake rev: e3834deb427ceb1d1ae9a96c1e0dec990d7f3805)
Signed-off-by: Isaac True <isaac.true@emlix.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
In scenequeue data generation loop, progress bar update in each cycle cause
a significant loss of speed. Remove progress bar update in for loop. It was
become quite fast (faster 30 times, about) without progress bar update.
(Bitbake rev: f775e53f1196e3e5d0ded277c061c5c268b3b28d)
Signed-off-by: Benjamin Szőke <egyszeregy@freemail.hu>
Signed-off-by: Mathieu Dubois-Briand <mathieu.dubois-briand@bootlin.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The default adapters for sqlite datetime are deprecated as of Python
3.12, so implement our own.
[YOCTO #15333]
(Bitbake rev: 38a1d715bf58acbc9cb21eed413b3542c81cf15a)
Signed-off-by: Joshua Watt <JPEWhacker@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
