The project is switching the way handle our CDN provision of sstate objects,
update the URL accordingly.
(Bitbake rev: 74c28e14a9b5e2ff908a03f93c189efa6f56b0ca)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
(cherry picked from commit 3372524cb961d95993b27fe4a8d794cdb7255e09)
Signed-off-by: Steve Sakoman <steve@sakoman.com>
For example the miniupnpd recipe has a SRC_URI like this:
http://miniupnp.tuxfamily.org/files/download.php?file=${BP}.tar.gz
In this case the path is /files/download.php, which isn't useful when
the latest_upstream logic bails early if there is no version in the path.
The logic now also checks in the downloadfilename, so add a test that
this works as expected.
(Bitbake rev: fffbf5d5e1c8556cddf0794e0b303bb0106747a0)
Signed-off-by: Ross Burton <ross.burton@arm.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Some code paths in latest_versionstring() need PV to be set correctly.
(Bitbake rev: 0a9f90ff658e09feda63b398ec35715a65ff6193)
Signed-off-by: Ross Burton <ross.burton@arm.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Use a named tuple so the test can access named members instead of just
accessing the data via index, which is harder to understand.
(Bitbake rev: 4b15652c84b06f0506c757e2647875a9b1cc7bfe)
Signed-off-by: Ross Burton <ross.burton@arm.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Marking the test iterations as subtests means that when one fails, it
can identify clearly which iteration has failed.
(Bitbake rev: 52c55e681332d7cdbe06f3c9d9c8d77cb0cb93f6)
Signed-off-by: Ross Burton <ross.burton@arm.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
It's best practise to use long-form arguments in scripts where the
conciseness of short arguments is less useful than in an interactive
terminal.
(Bitbake rev: 54039bc9d169871af6d36578df4c21bff296f6de)
Signed-off-by: Ross Burton <ross.burton@arm.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
If the target file exists, we pass --continue so that wget will try to
resume the download instead of starting from the beginning. However if
the file doesn't exist --continue has no effect, so there's no need to
conditionalise the use of the argument.
(Bitbake rev: a5ee50d4d2b2e6c00abe1551382afd9799345dbf)
Signed-off-by: Ross Burton <ross.burton@arm.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
latest_versionstring() currently looks at just the end of the URI when
guessing what the filename to look for is, but this doesn't work if the
URL filename is not simple.
For example, miniupnpd has a SRC_URI of:
http://miniupnp.tuxfamily.org/files/download.php?file=${BP}.tar.gz;downloadfilename=${BP}.tar.gz
The filename component of this is "download.php", which causes the
heuristics in latest_versionstring() to exit early.
Instead, if the downloadfilename is set then use that, as it's often the
actual filename that we're after.
(Bitbake rev: 2d5f135e997d13fabda0ad266fd5c928ee33f487)
Signed-off-by: Ross Burton <ross.burton@arm.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Build the 'downloadfilename' parameter by replacing path separators in
the module path like the git fetcher builds the mirror tar ball name.
Copy the downloaded file in the fetcher's unpack method like the crate
fetcher instead of calling the base fetcher's unpack method.
(Bitbake rev: 7762cea087597019460d66b04268757bd46befdf)
Signed-off-by: Christian Lindeberg <christian.lindeberg@axis.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
After the fetcher revisions changes, we need a new version marker to
match this with in OE-Core.
(Bitbake rev: 8cc976e2792fdde3900729f3b09dd18ab640b5e8)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
This reverts commit 004cfdec1c865f2351bbac99acb3d63bfef9d380.
Now we have python 3.9 as a minimum we can do this.
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
We used to use this for bare clones where a single git url could handle multiple
revisions (which implied multiple branches).
We don't use this any more and I doubt we'd want to go back to it. If we
remove it, we can simplfy the looping in the code which seems desireable.
This patch does change the warning for missing branch parameters to a
error. The message has hinted about that for long enough.
Some test cases are removed since they are no longer needed.
(Bitbake rev: 2515fbd10824005fa7f34e87706000c079920366)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Since Ubuntu 20.04 is heading to EoL and that we're having other issues
on that platform needing buildtools anyway, we're about to lose the
last python 3.8 platform we were supporting.
Bump the minimum version to 3.9 since there are many developers wanting
access to newer python features.
(Bitbake rev: 0d2e682d00dfc2ec776b3a89f25202db98024895)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Add tests for git tag verification in both standard and shallow clones.
(Bitbake rev: f47127066d67e2ad80974fa1e7c0fcc7409161af)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
If we want to validate a tag when using shallow clones, we need to ensure
the tag is imported to the clone. Adding to extra_refs allows this to happen
ensuring tag validation in shallow clones works.
(Bitbake rev: 03945475886b8d31c7eccc80fe594ab5306bcf5d)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Currently bitbake disallows tag parameters along with revision parameters.
This isn't great since quite often, we'd like to verify that a given revision
does match some tag. At the same time we don't want to or need to access
the network to verify this, which normally a tag would require.
Rework the code so that tag and revisions can both be specified together.
Verify that any tag specified matches the revision in use at unpack time.
This means we can start requiring people to put tags in git SRC_URIs
when revisions are used, making review a little easier that it isn't
some random revision.
The test that is dropped looks like a different test but the comment
is a copy and paste error. The SRCREV/rev mismatch test remains, this
removes the rev and tag set test.
(Bitbake rev: d591d7633fe8d739ec00395920e44910b0b77e27)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
finalizeData changes the value of variables, so, it needs to have data
tracking enabled to get a full and accurate variable value history.
Fixes [YOCTO #15652]
(Bitbake rev: 228979b506f7d0e24c643440bcc7becf684da283)
Signed-off-by: Yoann Congal <yoann.congal@smile.fr>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
A new context manager Tinfoil._data_tracked_if_enabled() is introduced to
replace the following structure:
if self.tracking:
self.run_command('enableDataTracking')
# Code that need data tracking
if self.tracking:
self.run_command('disableDataTracking')
(Bitbake rev: 0fea4555d2143c6b23a79d3d5cf791103a68141b)
Signed-off-by: Yoann Congal <yoann.congal@smile.fr>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
This fixes a bug where escapes in the url path would not be properly
restored for the git commands in the git fetcher. For example, a
space which is encoded as '%20' was not properly encoded before the
clone command.
e.g.
SRC_URI="git://git.openembedded.org/bitbake%20example/bitbake;protocol=https"
resulted in
git clone 'https://git.openembedded.org/bitbake example/bitbake'
instead of
git clone 'https://git.openembedded.org/bitbake%20example/bitbake'
(Bitbake rev: be48024253b93215bb110cd1d05925e789ec9680)
Signed-off-by: Patrik Nordvall <patrik.nordvall95@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Changing module files was changing the tash hashes but it was not invalidating
the parse cache, leading to tashhash mismatch errors during builds.
Add information from modulecode_deps to the configuration hash used for
cache invalidation to avoid this and trigger reparses when function library
code changes.
[YOCTO #15795]
(Bitbake rev: a121db3d8d28420c36369237b8bb11c2d0aaf5f7)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Implements the `gc-mark-stream` command to allow for marking equivalence entries
in batch, by making use of stream mode communication to the server.
The aim of this is to improve efficiency by reducing the impact of latency when
marking a high volume of hash entries.
Example usage of the new `gc-mark-stream` command:
```
$ cat << HASHES | \
./bin/bitbake-hashclient --address "ws://localhost:8688/ws" gc-mark-stream "alive"
unihash f37918cc02eb5a520b1aff86faacbc0a38124646
unihash af36b199320e611fbb16f1f277d3ee1d619ca58b
taskhash a1117c1f5a7c9ab2f5a39cc6fe5e6152169d09c0 method oe.sstatesig.OEOuthashBasic
HASHES
```
(Bitbake rev: c84715f28cd36666ea07a179d91b8c32ea0df8e7)
Signed-off-by: Alexander Marques <c137.marques@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
We're fairly careful in bitbake about how we handle fork() calls and believe our code
to be safe. The upstream deprecation warning is problematic as it can appear in log
output as a WARNING, breaking tests. It also tends to alarm users.
Hide the warning for now to avoids the test failures.
(Bitbake rev: c636bd629896f56e5f3d4030da3d1f130590afc6)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Document the include_all directive, which can be used to include
multiple files present in the same location in different layers.
(Bitbake rev: bd41f09c7fa67f917246a740fd4068e4588a1dcd)
Signed-off-by: Antonin Godard <antonin.godard@bootlin.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The command complete event triggers the UI to start new commands.
We should send this once we've cleaned up existing commands since
on very slow systems, this could otherwise cause failures. There
is a 30s timeout on incomming commands but slow systems have been
known to hit this on poorly configured I setups.
(Bitbake rev: d202298b3d989c39bc532b69c410e2e78b082a11)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
We have a "parsing started" event in the cooker deamon log but we don't
currently log the corresponding "parsing complete" event which is
confusing. Add this so that the logs are more logical.
(Bitbake rev: 1aa491c1f1211bf9faab712c321b66629fb7be66)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
If a custom python module is added thru BBIMPORTS and it
uses typing(Any,Tuple,Union...), codeparser will fail because
inspect.py raises TypeError exception if the object is a
built-in module, class, or function.
(Bitbake rev: 0ecfd0b8540220633e71d24cd73cc5306863ae3c)
Signed-off-by: Pedro Silva Ferreira <Pedro.Silva.Ferreira@criticaltechworks.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
A regular expression "URL" in PREMIRRORS and MIRRORS may contain a ? as
part of the regular expression. Make sure this does not cause
problems.
(Bitbake rev: 5af7fe4473cd7e75d4eb7f8b93c499bd157ff156)
Signed-off-by: Peter Kjellerstedt <peter.kjellerstedt@axis.com>
Signed-off-by: Stefan Herbrechtsmeier <stefan.herbrechtsmeier@weidmueller.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
This partial reverts commit a5d569c94700f04b8193c6bccae5af619931b00f
which changes decodeurl() to use the URI class to parse the URL instead
of doing it itself. While reusing code is generally a good idea, using
urllib.parse.urlparse() (which the URI class does) to parse the regular
expression "URLs" that are used in PREMIRRORS and MIRRORS does not work.
A regular expression URL containing https?://... would be silently
ignored, while a URL using a negative lookahead such as
git://(?!internal\.git\.server).*/.* would result in a cryptic error:
Exception: re.error: missing ), unterminated subpattern at position 0
The problem is that urllib.parse.urlparse() treats the ? as the start of
URL parameters and thus stops parsing whatever part of the URL it was
parsing. Restore the old function and use it in the PREMIRRORS and
MIRRORS code.
(Bitbake rev: f8a7712754e6d0199a0d227fca288307b935368d)
Signed-off-by: Peter Kjellerstedt <peter.kjellerstedt@axis.com>
Signed-off-by: Stefan Herbrechtsmeier <stefan.herbrechtsmeier@weidmueller.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
At the end of this function, self.overrides is cleared, which means we'll
construct a new self.overrides after this call. And a new self.overrides
will give out different expand_cache, so the old expand_cache should also
be cleared to avoid any wrong value.
Currently, there's a problem revealed by recent recipe specific virtual
provider patch. If we enable multilib and set "OVERRIDES:prepend" in local.conf,
things don't work.
Here's the error message:
ERROR: Nothing PROVIDES 'lib32-gcc-cross-x86_64'
Below are reproduce steps:
1. Add in local.conf the following lines:
MACHINE ?= "qemux86-64"
require conf/multilib.conf
MULTILIBS ?= "multilib:lib32"
DEFAULTTUNE:virtclass-multilib-lib32 ?= "core2-32"
OVERRIDES:prepend = "some-override:"
(Note that using :append and :remove also reproduces the issue.)
2. bitbake -n lib32-sysstat
(bitbake -n core-image-minimal also reproduces the issue)
The expandWithRefs calls getVar, which fills expand_cache. So when setting
OVERRIDES:prepend, this will fill the expand_cache. When overridevars are updated,
if we don't clear expand_cache, we'll retrieve wrong values.
Previously, things happened to work because there's a call to expand
PREFERRED_PROVIDER_virtual/${TARGET_PREFIX}gcc, which in turn expands
TARGET_VENDOR. Now what we expand is PREFERRED_PROVIDER_ virtual/cross-cc,
so the problem is revealed.
(Bitbake rev: 842e94234c9b9faf4e902721f516071f643e49c9)
Signed-off-by: Chen Qi <Qi.Chen@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
When var matches __setvar_regexp__, which has the value as below,
__setvar_regexp__ = re.compile(r'(?P<base>.*?)(?P<keyword>:append|:prepend|:remove)(:(?P<add>[^A-Z]*))?$')
the keyword will not be __doc__ or __module__ and var will always
contain ":".
(Bitbake rev: 0af396917fe176df7b1ed0bd7c7267507ed12e03)
Signed-off-by: Chen Qi <Qi.Chen@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The non-greedy modifier suffix ? is not necessary here because } is
not in the character set [a-zA-Z0-9\-_+./~:].
(Bitbake rev: 80735719d0002c02b704a49604ddd82c461203a2)
Signed-off-by: Chen Qi <Qi.Chen@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Previously if the websockets module was not installed
bitbake would only print a warning and continue, resulting in a degraded user
experience due to inability to use the configured sstate server.
Let's consider that as fatal misconfiguration, so that users can address
the issue properly and not wonder why builds are taking forever.
(Bitbake rev: cfba2a9fca9dfa3b05ec9040fe0cb8143ac04af7)
Signed-off-by: Alexander Kanavin <alex@linutronix.de>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
- Address the absence of an initial full bare clone
- Utilize the initial shallow clone
- Modify existing test cases for this behavior
- Remove incompatible test cases
(Bitbake rev: 599fedacd7782dcb52825c22200f35344c102548)
Signed-off-by: Stefan Koch <stefan-koch@siemens.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
When `ud.shallow == 1`:
- Prefer an initial shallow clone over an initial full bare clone,
while still utilizing any already existing full bare clones.
- If the Git error "Server does not allow request for unadvertised object"
occurs, the initial full bare clone is fetched automatically.
This may happen if the Git server does not allow the request
or if the Git client has issues with this functionality,
especially with the Git client from Ubuntu 20.04.
This improves:
- Resolve timeout issues during initial clones on slow internet connections
by reducing the amount of data transferred.
- Eliminate the need to use an HTTPS tarball `SRC_URI`
to reduce data transfer.
- Allow SSH-based authentication (e.g. cert and agent-based) when
using non-public repos, so additional HTTPS tokens may not be required.
(Bitbake rev: 457288b2fda86fd00cdcaefac616129b0029e1f9)
Signed-off-by: Stefan Koch <stefan-koch@siemens.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
After the fixture changes, the tests need to be tweaked unfotunately.
(Bitbake rev: 708abd1a8060684127acc7ce4142f05865005750)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
We need to validate the toaster fixtures information against our current release
data to ensure it is correct when we release. Add a script we can use to
do this which the autobuilder can run to validate things.
[YOCTO #15516]
(Bitbake rev: 5b2d79ed505bbfa2fb2d355935e75199b7f2c37e)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Use :setfiletype instead of :set filetype. The former only sets the
'filetype' option if it has not been set before, which makes it possible
to override the syntax of certain *.inc files in autocommands from e.g.
.vimrc or modelines. All other ftdetect plugins in upstream vim also use
:setfiletype for this reason.
The detection for bitbake *.inc files is now upstream since Vim 9.0
patch 0055 [1]. If we're running an earlier Vim, use the detection
heuristic from upstream [2] to overwrite the filetype explicitely if we
find bitbake code. But don't always assuming that *.inc files are
bitbake files so as not to break Perl, PHP, Assembly, Povray, etc.
[1]: https://github.com/vim/vim/commit/fa49eb482729
[2]: https://github.com/vim/vim/blob/fb49e3cde79d/runtime/autoload/dist/ft.vim#L715
(Bitbake rev: e8efbba5d7bb4b685ed0a9b970e042ad99be8afb)
Signed-off-by: Roland Hieber <rhi@pengutronix.de>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
In order to avoid a potentially confusing backtrace, check that the mcdepend
is valid when we add it.
Add a test case to ensure invalid configurations are caught and trigger an
error.
[RP: Reworked test case to simplify and improve code]
(Bitbake rev: ff523497270f37b484b44a4445c2194791bcb6ff)
Signed-off-by: Mark Hatle <mark.hatle@amd.com>
Signed-off-by: Mark Hatle <mark.hatle@kernel.crashing.org>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
We've been seeing intermittent failures on Ubuntu 22.04 in oe-selftest which
were problematic to debug. The failure was inside lock_timeout and once that was
identified and the backtrace obtained, the problem becomes clearer:
File "X/bitbake/lib/bb/server/process.py", line 466, in idle_thread_internal
retval = function(self, data, False)
File "X/bitbake/lib/bb/command.py", line 123, in runAsyncCommand
self.cooker.updateCache()
File "X/bitbake/lib/bb/cooker.py", line 1629, in updateCache
self.parser = CookerParser(self, mcfilelist, total_masked)
File "X/bitbake/lib/bb/cooker.py", line 2141, in __init__
self.bb_caches = bb.cache.MulticonfigCache(self.cfgbuilder, self.cfghash, cooker.caches_array)
File "X/bitbake/lib/bb/cache.py", line 772, in __init__
loaded += c.prepare_cache(progress)
File "X/bitbake/lib/bb/cache.py", line 435, in prepare_cache
loaded = self.load_cachefile(progress)
File "X/bitbake/lib/bb/cache.py", line 516, in load_cachefile
progress(cachefile.tell() + previous_progress)
File "X/bitbake/lib/bb/cache.py", line 751, in progress
bb.event.fire(bb.event.CacheLoadProgress(current_progress, cachesize),
File "X/bitbake/lib/bb/event.py", line 234, in fire
fire_ui_handlers(event, d)
File "X/bitbake/lib/bb/event.py", line 210, in fire_ui_handlers
_ui_handlers[h].event.send(event)
File "X/bitbake/lib/bb/cooker.py", line 117, in send
str_event = codecs.encode(pickle.dumps(event), \'base64\').decode(\'utf-8\')
File "/usr/lib/python3.10/asyncio/sslproto.py", line 320, in __del__
_warn(f"unclosed transport {self!r}", ResourceWarning, source=self)
File "/usr/lib/python3.10/warnings.py", line 109, in _showwarnmsg
sw(msg.message, msg.category, msg.filename, msg.lineno,
File "X/bitbake/lib/bb/main.py", line 113, in _showwarning
warnlog.warning(s)
File "/usr/lib/python3.10/logging/__init__.py", line 1489, in warning
self._log(WARNING, msg, args, **kwargs)
File "/usr/lib/python3.10/logging/__init__.py", line 1624, in _log
self.handle(record)
File "/usr/lib/python3.10/logging/__init__.py", line 1634, in handle
self.callHandlers(record)
File "/usr/lib/python3.10/logging/__init__.py", line 1696, in callHandlers
hdlr.handle(record)
File "/usr/lib/python3.10/logging/__init__.py", line 968, in handle
self.emit(record)
File "X/bitbake/lib/bb/event.py", line 778, in emit
fire(record, None)
File "X/bitbake/lib/bb/event.py", line 234, in fire
fire_ui_handlers(event, d)
File "X/bitbake/lib/bb/event.py", line 197, in fire_ui_handlers
with bb.utils.lock_timeout(_thread_lock):
File "/usr/lib/python3.10/contextlib.py", line 135, in __enter__
return next(self.gen)
File "X/bitbake/lib/bb/utils.py", line 1888, in lock_timeout
bb.server.process.serverlog("Couldn\'t get the lock for 5 mins, timed out, exiting. %s" % traceback.format_stack())
or put in simpler terms, whilst sending an event(), an unrelated warning
message happens to be triggered from asyncio:
/usr/lib/python3.10/asyncio/sslproto.py:320: ResourceWarning: unclosed transport <asyncio.sslproto._SSLProtocolTransport object at 0x7f0e797d3100>
which triggers a second event() which can't be sent as we're already
in the critcal section and already hold the lock.
That warning is due to the version of asyncio used on Ubuntu 22.04 with
python 3.10 and that comined with timing issues explains why we don't
see it on other python versions or distros.
We can't handle the second event as the lock is there to serialise the
events. Instead, we queue the event and then process the queue later.
Add a new version of lock_timeout which allows us to handle the situation
more gracefully.
(Bitbake rev: 2c590ff1aff89d23b25ce808650f200013a1e6af)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
We never want to exit whilst holding these locks as it deadlocks all python
threads. Add signal blocking around the lock critical part so a signal
shouldn't cause such an exit.
(Bitbake rev: a097755c671e2b530dea6200a94b39fa9dca246c)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
We should really try and take the lock in the try/finally block so that
in some rare cases such as badly timed interrupt/signal, we always release
the lock.
(Bitbake rev: a9eb8bf7174b6962b5ba07192fe95b8c7112d9d2)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
