This alias is intended for managing specific local configs and it
is prompted by adding support for config fragments (in a separate
commit to oe-core); after some deliberation I concluded there should be
a separate tool, as bitbake-layers is already somewhat over-stuffed,
and this will give space for more build/conf/* operations in the future
that anyone can come up with (such as tweaking site-specific items
in site.conf etc.)
The alias completely reuses existing code via symlink and
the difference is in where it looks for plugins.
(Bitbake rev: ba90fe673aa87cb0cda9b2e465ebe2063551f527)
Signed-off-by: Alexander Kanavin <alex@linutronix.de>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Use the new listtasks() function (introduced in 185c4b) to avoid
accessing a private variable.
Also use assertSequenceEqual() directly as we don't really care about
the actual type returned, only that it's a sequence.
(Bitbake rev: 452289efc6d3608ceca321286fd5bf417a6e1bed)
Signed-off-by: Ross Burton <ross.burton@arm.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
It was suggested that using flags in fragment content to
mark fragment-specific metadata (such as descriptions and
dependencies) is prone to quiet regressions when fragments
are renamed or moved, and it clutters the fragment content.
With this change allowed fragment metadata variables must
be explicitly listed in a variable that is given to addfragment
directive, and parser will add the flag containing the fragment name
to them.
(Bitbake rev: ed9a3ca9426500511feb77f41a146953dbfe9af7)
Signed-off-by: Alexander Kanavin <alex@linutronix.de>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
It takes two parameters:
- location prefix for fragments
- name of variable that holds the list of enabled fragments, each of them prefixed
by layer id
Implementation of this directive essentially expands the fragment list
obtained from the variable into absolute fragment paths and hands them to the
implementation of 'require'.
(Bitbake rev: f687746703e7b096c5480668fd4f49bd4951182b)
Signed-off-by: Alexander Kanavin <alex@linutronix.de>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
This is beneficial for config fragments, as their names
are specified via flags, and those names can include slashes:
BB_CONF_FRAGMENT_SUMMARY[init/systemd] = "This fragment enables systemd as an init manager"
(Bitbake rev: 80805988492b35593067230f68782a5687c8f557)
Signed-off-by: Alexander Kanavin <alex@linutronix.de>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
enum was introduced in Python 3.4
(Bitbake rev: 35b71a94f8757fcca830f972a42edab1dd000c16)
Signed-off-by: Chris Laplante <chris.laplante@agilent.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
When the user forgets to set MACHINE, bitbake just exits without
printing anything.
This is because BB_CONSOLELOG ends up with an unexpanded '${MACHINE}', which
bb.utils.mkdirhier tries to report using bb.fatal. But bb.fatal utilizes the
very logging infrastructure that this code was trying to setup.
(Bitbake rev: 7d3f3655b2f610f76898c84b8b97ef2e26529c41)
Signed-off-by: Chris Laplante <chris.laplante@agilent.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Analysis shows that "bitbake core-image-ptest-all" spends a lot of
time in scenequeue_updatecounters and much of it is rebuilding a set
which doens't change. Reorder the code to avoid that performance
glitch.
(Bitbake rev: 923c19b9713e398d8e66e6d4422dfd4c18a03486)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Rather than looping through things we looped through on the previous execution,
start looping where we left off for setscene processing. This gives speed
improvements depending on the kind of build being executed.
(Bitbake rev: 00f4d932e3af0eeb333339cbe942010fc76dee0f)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
There have been complaints about the performance of large multiconfig builds
for a while. The key missing data point was that the builds needed to have large
overlaps in sstate objects. This can be simulated by building the same things with
just different TMPDIRs. In runqueue/bitbake terms this equates to large numbers of
deferred tasks.
The issue is that the expensive checks in the setscene loop were hit every time
through runqueue's execute function before the check on deferred tasks. This leads
to task execution starvation as that only happens once per iteration.
Move the skip check earlier in the function which speeds things up enormously
and should improve performance of such builds for users.
(Bitbake rev: 9c6c506757f2b3e28c8b20513b45da6b4659c95f)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
do_unpack generates a 'Bitbake Fetcher Error: UnpackError' when
uncompressing a single 'file.zst', zstd command line returns the error
'unexpected end of file', and the input file 'file.zst' becomes empty.
The command line is 'zstd --decompress --stdout file.zst > file.zst'.
So the issue is that the input and output filenames are the same.
Fix the output filename when using zstd for a single file.
(Bitbake rev: 5881ca0e0d53937cd25a57ff682a3b7577b0b7f8)
Signed-off-by: Pascal Eberhard <pascal.eberhard@se.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Removes the code in bitbake to show custom backtrace formatting for
exceptions. In particular, the bitbake exception code prints function
arguments, which while helpful is a security problem when passwords and
other secrets can be passed as function arguments.
As it turns out, the handling of the custom serialized exception stack
frames was pretty much made obsolete by d7db75020ed ("event/msg: Pass
formatted exceptions"), which changed the events to pass a preformatted
stacktrack list of strings, but the passing of the serialized data was
never removed.
Change all the code to use the python traceback API to format exceptions
instead of the custom code; conveniently traceback.format_exception()
also returns a list of stack trace strings, so it can be used as a drop
in replacement for bb.exception.format_exception()
(Bitbake rev: 2cda75a185aaf8f657f072dac34f8cef9d75f63a)
Signed-off-by: Joshua Watt <JPEWhacker@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
There's no easy way to list all of the tasks in a recipe, you can either
look at __BBTASKS (internal variable, shouldn't be used) or iterate all
items in the datastore looking for variables with the 'task' flag set
(which is slow).
Solve this problem by adding a bb.build.listtasks() function that
returns an immutable copy of the __BBTASSK variable.
(Bitbake rev: 185c4b803962b20ba65a7d885dfe1a14e68736ef)
Signed-off-by: Ross Burton <ross.burton@arm.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
This was added in 2.7.2. Since using inherit_defer is safer that inherit
when inheriting conditionally, move the instructions about that in
inherit_defer.
Fixes [YOCTO #15640].
Reported-by: Yoann Congal <yoann.congal@smile.fr>
(Bitbake rev: eb10df5a9619e243e28e0f4cd6122c24ed668f52)
Signed-off-by: Antonin Godard <antonin.godard@bootlin.com>
Signed-off-by: Mathieu Dubois-Briand <mathieu.dubois-briand@bootlin.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
When the recipe provided by (-r, --recipe) is not found
tinfoil raises an exception that is not catched for
readability, example:
Traceback (most recent call last):
File "/.../poky/bitbake/bin/bitbake-getvar", line 45, in <module>
d = tinfoil.parse_recipe(args.recipe)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/.../poky/bitbake/lib/bb/tinfoil.py", line 633, in parse_recipe
fn = self.get_recipe_file(pn)
^^^^^^^^^^^^^^^^^^^^^^^^
File "/.../poky/bitbake/lib/bb/tinfoil.py", line 550, in get_recipe_file
raise bb.providers.NoProvider('Unable to find any recipe file matching "%s"' % pn)
bb.providers.NoProvider: Unable to find any recipe file matching "aaa"
(Bitbake rev: 06aa6c292813a28c84736193b550fb2d18884d43)
Signed-off-by: Talel BELHAJSALEM <bhstalel@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Testing shows the worst case CDN response time can be up to 100s. The wget fetcher
is used for accessing sstate from the CDN so increase our timeouts there to match
our worst case repsonse times.
(Bitbake rev: 7aca591529e115bc277f93811d8c586630acc8c3)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The default pipe size is 64kb on builds, which can be inefficient
for larger log files from workers. Increase the pipe size to 512kb
since build systems have decent amounts of memory and this is a more
efficient way of batching the data.
Tweak the default read sizes to match the pipe size for efficiency.
Since the contstant is only present in python 3.10 onwards, add
some compatibility code.
(Bitbake rev: 69c14e46600ba5ae9703f67704ab2548875ae6d7)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
If there are large amounts of data being transferred to the cooker
from the worker, recreating the bytearray becomes inefficient as it
happens for every pipesize block of data, defaulting to 64kb.
Instead we can use the deletion API for bytearrays to make this more
efficient and avoid the object recreation.
We noticed this with a strace ptest image taking days to complete the
build after having 6GB of data in the testimage log. Whilst there are
other issues there, making this code more efficient doesn't hurt.
(Bitbake rev: a4a72b7edb368f352784c856a647236a887010dd)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
DANGLINGAPPENDS_WARNONLY is a very poorly designed interface and
is commonly abused. The challenge is that once it is set, by any
layer, it applies everywhere.
Some layers rely on this to get notification they need to update
bbappend files and having the behaviour change from inclusion of an
antisocial layer is not good.
In addition, showing warnings as an accepted thing on the console
devalues them and trains the user to ignore them. I want to steer us
away from this mindset.
We could extend the functionality and make it apply only to certain
layers, or only to certain appends but then we've basically re-invented
BBMASK.
Given all the above, we should drop support for BB_DANGLINGAPPENDS_WARNONLY
and direct anyone with issues to BBMASK instead.
https://lists.openembedded.org/g/openembedded-architecture/message/2029
[YOCTO #14870]
(Bitbake rev: fca9c9e3cb6f8e9f99bf51dc5e8a8d83f4c84c69)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Create a new tinfoil API "finalizeData" which does key expantion and
runs the anonymous functions to allow bitbake-getvar to have the same
output as "bitbake -e".
(Bitbake rev: 63db64ca98564420a378f14c337074ae06da8c63)
Signed-off-by: Yoann Congal <yoann.congal@smile.fr>
Signed-off-by: Mathieu Dubois-Briand <mathieu.dubois-briand@bootlin.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
In commit 9257c48c3e36 ("Group and reorder options in bitbake help"),
help output was changed, update the doc to match the new output.
(Bitbake rev: 7ef357f1c6fcd50c9d22190ba39d1cfd82070c79)
Signed-off-by: Yoann Congal <yoann.congal@smile.fr>
Reviewed-by: Quentin Schulz <quentin.schulz@cherry.de>
Reviewed-by: Antonin Godard <antonin.godard@bootlin.com>
Signed-off-by: Mathieu Dubois-Briand <mathieu.dubois-briand@bootlin.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Fixes what looked like a type since parseConfiguration.needconfig is
already set just after parseConfiguration() definition.
(Bitbake rev: d371df029ece3b9e600a89d08337c437a8ddbf63)
Signed-off-by: Yoann Congal <yoann.congal@smile.fr>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The idle and main loops have socket select calls to know when to execute.
This means we can increase the normal timeout frequency since this is
just a fall back and have some small efficiency gains.
(Bitbake rev: 8d8e17af8619c976819170c9d5d9a686a666c317)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
If there are no idle functions present, don't sent heartbeat events. These
are only meant to happen while builds are active.
(Bitbake rev: 9a2d5e63b07c3912838781776c61f0f1ac9640e1)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Keeping this code separate just makes the code harder to understand,
merge them.
(Bitbake rev: e5ac26a0e1779df1da3277bf48899c8f7642f1f8)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
We're seeing some failures on hosts where slow "idle" loop iterations are
causing bitbake server timeouts. These seem to happen particularly in the
dump_signatures() function within runqueue.
That isn't entirely surprising since it creates a pool of threads to execute
work an at best can take around 10s to execture and return control backto the
main loop. On a slow system, it is understandable this can take longer,
particularly as these functions are creating large chunks of IO.
Since the work is being done in threads, we can launch them, return to idle
and check on the results periodically as they complete.
This should hopefully address some of the remaining timeout issues we see on
the autobuilder in oe-selftest sstate tests.
(Bitbake rev: e66f1b643b4b77404ba31f2704cda5af9bf00a57)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
lz4c has been considered deprecated by upstream since at least 2018 [1]
and has been disabled by default recently [2]. openSUSE Tumbleweed is
no longer packaging the deprecated version and others will likely
follow.
Luckily, we're not using any legacy-specific options and, going back as
far as Ubuntu 16.04/Fedora 25, both lz4 and lz4cli are installed as part
of the same package
1. https://github.com/lz4/lz4/pull/553
2. https://github.com/lz4/lz4/pull/1479
(Bitbake rev: 907472034b344e4eb73cfd43059a413469f52e1c)
Signed-off-by: Justin Bronder <jsbronder@cold-front.org>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
So that we can compare the different pn-buildlist easily.
(Bitbake rev: 529043117a7c62feb45bc891658a412cc8dd7e3f)
Signed-off-by: Robert Yang <liezhi.yang@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
GNOME gitlab has occasional downtime which impacts bitbake-selftest
and causes autobuilder failures. Switch to our own mirror for test
purposes to avoid those issues.
(Bitbake rev: 91e268b11ed683bd197026f9b36001f6d54ee05c)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
GNOME gitlab has occasional downtime which impacts bitbake-selftest
and causes autobuilder failures. Switch to our own mirror for test
purposes to avoid those issues.
(Bitbake rev: 008808755ed6cfeb6c41273e69ce718f0833c26c)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The file may be a symlink when mirror from local disk, and the symlink will be
broken when the linked file is removed, use os.path.islink() to check and
remove the broken symlink.
(Bitbake rev: d0deb5fe6e7a84b07b49ef685b209729d2fdc780)
Signed-off-by: Robert Yang <liezhi.yang@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Fixed:
BB_GIT_SHALLOW = "1"
BB_GENERATE_SHALLOW_TARBALLS = "1"
$ bitbake utfcpp -cfetch && bitbake utfcpp -ccleanall
The downloads/gitsmshallow_github.com.nemtrif.*.tar.gz won't be cleaned without
this fix.
(Bitbake rev: bab7a8970a0237a9d24217685a595e76a1336c07)
Signed-off-by: Robert Yang <liezhi.yang@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Fixed:
$ bitbake utfcpp -cfetch && bitbake utfcpp -ccleanall
The downloads/git2/github.com.nemtrif.ftest won't be cleaned without this fix.
(Bitbake rev: 79f25fc5c1b8d0e08540f4aa07875309f5325f47)
Signed-off-by: Robert Yang <liezhi.yang@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The tmpdir such as downloads/tmplp3cnemv won't be removed without this fix.
(Bitbake rev: 2ba8d3214759142afc11f0a88d80eb30a8bcde3a)
Signed-off-by: Robert Yang <liezhi.yang@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
There are 14 lines can be removed, and can make it easy to maintain.
(Bitbake rev: 0ea2c1ac079d63349407a69172ff80cd9acc7252)
Signed-off-by: Robert Yang <liezhi.yang@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Libgcc is required as a runtime dependency. Without it, we get the
following error:
OSError: Error loading shared library libgcc_s.so.1: No such file or
directory
(Bitbake rev: 95e61f3dacacb3a001d9f0e2db4c4a2960d96640)
Signed-off-by: Jesse Riemens <riemensjesse@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Adds an epilog to the help text that indicates the possible options for
the server address
(Bitbake rev: b6b703fce02057212ad11b1d1286c6178c533bad)
Signed-off-by: Joshua Watt <JPEWhacker@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
We really don't want to be skipping a test like this, drop the exception handling
code as in my testing we thankfully no longer seem to be seeing it. If we do again
we need to fix the underlying issue.
(Bitbake rev: b67547f890dc7f347ec05a97cbfb22d3703998cc)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Fix the exception handling in the cancel builds function.
This involved adding WebDriverException which sometimes occurs but
also correcting the other exception handlers to continue to increment
the timeout to avoid test hangs.
(Bitbake rev: e111a2bd4f7a8a4dc2c63e94e91ac6cacca95af8)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Most of the tests that click on buttons need the DOM to stablise, including
any running JQuery code before the test can proceed. Add calls to do this
whenever we're about to click on an element.
(Bitbake rev: 0eb206b355248e2a874a62baec30025652f2a5a8)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Switch the clickable() calls to use the new element_clickable() function
which accepts a finder labmda function. This means if the element doesn't
yet exist, the code can rebuild the query and try again once a small
amount of time has elapsed.
There were a ton of timing related races around these element interactions
and this seemed to be the most robust way to address the issues.
The change also makes some of the elements slightly more specific so
the code can work effectively.
(Bitbake rev: 38643aadbb5a960004b886cf7709beaf2fc96652)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
If the element never exists, the timeout is never incremented and the test
hangs indefinitely. Fix the exception handling to avoid that and allow
the timeout to happen.
(Bitbake rev: 9eabe923d457bbce65227da4cd71c275c32108e6)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Fixes for the browser tests which:
* Add/improve waiting for specific elements
* Waiting for elements to be interactable, not just visible
* Hiding notification windows
* Using more specific asserts to obtain improved error messages
(Bitbake rev: 0c3f9bc2b9516d2af391f2e081280eb7fa62a05e)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Add a variant of the clickable function that accepts an element parameter.
(Bitbake rev: a34b2451a0e3baaf3b2fd2eb351847ff6a5ee033)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Our own Wait() class allows exception handling which this form of wrapper
does not. Switch the code to use our Wait() class to allow retrying upon
encountering those exceptions (such as an element not being present yet).
The displayed and visible test is what Selenium would be doing internally,
there is no JS reprensetation of clickable directly.
(Bitbake rev: 8266a01b750b3758badeee8fb3a1acfa72c17a93)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The code modifying the build object requires direct database access
which is potentially problematic. It was being run on a database which
was being reset after changes anyway so the code effectively did nothing.
Remove it as it is unneeded.
(Bitbake rev: 2e3fa624eb3972aef91d877337479eeb0f3dd365)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Drop the remaining poll parameters from the helpers code along
with the remaining sleep call since the tests no longer depend
on this.
This has the nice benefit of significantly speeding up the toaster
test runs (45 minutes down to 12 minutes overall).
If a parameter is needed, it should be the timeout, not the polling
frequency.
(Bitbake rev: 6de912e4f278ffd694fb2258482081dc3bc61c7a)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Drop the poll parameters and make the waits much more specific for
the requirements of the tests. This includes looping waiting for
a list of layer elements as that code was previously particularly
fragile.
(Bitbake rev: cf6b8e8aa5484110a41377ba42b3fdd9d6efd877)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Switch this test module to use the common project creation code which contains
race fixes. That code requires the database access wrapper be dropped and
we no longer have ordering constraints.
There is one test that does require database access. Move this to a separate class
and allow database access there. Use ordering constraints to allow them to run
after the main code. They depend on the project creation from the other class which
isn't ideal but good enough for now.
(Bitbake rev: fa10ba2a8749415d8f06cfc15c228c6eb7df1bcf)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
In some cases we want to wait until some element is not visible.
Add such a function helper.
(Bitbake rev: cede6519d25112037264550126b109903220b58c)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
These changes:
* Drop the poll parameters
* Add waits where needed or make them more specific
* Add calls to close notification boxes since they can obscure
other page elements
* In one case wait for a "Loading" message to be replaced
(Bitbake rev: 2bde969486a4761fa0e6f9590c9fa08d385666b1)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The project creation test is well covered in another module so just
drop this one as there isn't any extra value added here.
(Bitbake rev: d26414f191c73fb5330d10d97e240150a3fa1ee8)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Switch this test module to use the common project creation code which contains
race fixes. That code requires the database access wrapper be dropped and
we no longer have ordering constraints.
There are two tests that do require database access. Move these to a separate class
and allow database access there. Use ordering constraints to allow them to run
after the main code. They depend on the project creation from the other class which
isn't ideal but good enough for now.
(Bitbake rev: e441bfe98ac41d48692ffbaeec90a9c780337fa4)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Drop poll calls within the module, adding wait calls for elements of the
page where testing showed it was needed.
(Bitbake rev: 26746eae31f126ff87e678da66dcc976e412c89b)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The project page selection code is race prone. Create a common function to resolve
the race issue and use it from all the call sites rather than duplicate code.
(Bitbake rev: f2bd615b97a6ff3944fa9c1d89a0ea996a12943d)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Drop the remaining poll values and the sleep within the main wait function
as the races this was working around now seem to be resolved with the
preceeding patches.
(Bitbake rev: 91e54cbacc57316a4f2ca766afac81e7f3a54718)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
In my testing I wasn't able to see issues needing the poll parameter in the
browser tests. Drop those.
(Bitbake rev: 3be5e7c78129abf059476416279718156fc666b5)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Rather than hardcoded poll and sleep values, we want an overall long timeout
and to drop all the other 'sleeping' code.
To facilitate this, increase the overall timeout so we can then start to
remove the other values.
(Bitbake rev: 69a479383f9a8cae6600ce51b12b62be3a15ea38)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The tests would race and potentially fail as the elements on the
page being waited for were not specific enough.
Add suitable elements to wait for and drop the remainder of the poll
parameters since these are no longer needed since we no longer need
to sleep for things to appear.
(Bitbake rev: 9db2fd8534610ad8db90a7a696fa7b748b497d15)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The 'fail' calls are useful for find() failures but not for wrapping existing asserts
where they hide more useful error messages. Rework the code flow to give more readable
error reporting.
(Bitbake rev: 1a7a9af418874d3112aa6f148b30e5a4d3d176f3)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Switch this test module to use the common project creation code which contains
race fixes. That code requires the database access wrapper be dropped and
we no longer have ordering constraints.
(Bitbake rev: 653233a1c798d71800b41d3321203a48aa0337d8)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Switch this test module to use the common project creation code which contains
race fixes. That code requires the database access wrapper be dropped and
we no longer have ordering constraints.
(Bitbake rev: f015d96e28d89fd0aef76ff127ebde8613125a66)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Tweak the function name and return the created project ID.
Allow the release_name test to be skipped with a value of None.
Add an extra wait call to ensure the create project button is ready
before clicking it.
(Bitbake rev: d20c6e3c67298c5db33b220a2e4a554e6f2c4418)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The test project creation is needed by mutliple tests. Move it to the
base class as a helper function for reuse.
(Bitbake rev: abe483b98b38e95b4ad42629a67fb5d243c94094)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Mixing database access and access via a running server is fraught with
danger and problems. The "django_db" marker means the transactions are
dropped at the end of the test but the transactions made via the webapi
remain so the database ends up confused at best.
Drop the database accesses and use the server API. This means slightly
abusing the typeahead to get lists of projects in the database.
Add code to delete a project if it already exists. This allows tests
to re-run against an existing database. Deletion is done using the
server API but this means handling CSRF tokens.
Add requests module requirement to requirements file since the project
creation code now uses requests.
(Bitbake rev: 738270c53a08ddc95400de70f3dd8c08b2940182)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
This eliminates the last usage of 'fake mozilla' in bitbake, and
it's then truthful everywhere about presenting itself, or wget
(when that is used).
I understand this will make people nervous so I want to provide
an extended decription.
1. How was this tested?
- bitbake-selftest -k FetchCheckStatusTest
(tests a few hardcoded URIs, all passed)
- bitbake -k -c checkuri world
(runs checkstatus() over all recipes in oe-core, and all passed again -
this hopefully goes a long way to reassure everyone that hosts around
the world and various CDNs typically do not have a problem with user-agent
strings they haven't seen before or bitbake user-agent specifically)
2. What about that removed cloudflare comment?
I digged into git history, and I think it is not fully accurate. First, 'fake
mozilla' agent is used only for checkstatus() - in actual fetching with wget
it is not. And that has not been a problem for anyone.
Second, here's how the comment occured. Usage of 'fake mozilla' was introduced here:
https://git.yoctoproject.org/poky/commit/?h=master&id=ab26fdae9e5ae56bb84196698d3fa4fd568fe903
At that point it did not have to be specifically 'mozilla', the commit message
indicates that any User-Agent would have been ok. Mozilla was simply copied
from upstream version check for convenience.
Later on, the string was updated to a more recent Mozilla:
https://git.yoctoproject.org/poky/commit/?h=master&id=9f123238261a68e37cec634782e9320633cac5d4
The claim in the added comment become something else: that User-Agent *must* a browser,
without evidence or tests. Even though it demonstrably doesn't have to be - wget is ok.
3. What if someone has a server that is ok with wget agent, but not ok with bitbake agent?
Please see point one. It's not impossible but I think it's highly unlikely. I do think
we should rather tell servers the truth, and learn where the actual issues are. Then
we can consider options - whether that would be pretending to be wget, or allowing user-agent
to be configured. We should also add such servers to bitbake-selftest so we know what they
are.
(Bitbake rev: 234f9e810494394527f59fdf22eb86435d046d53)
Signed-off-by: Alexander Kanavin <alex@linutronix.de>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
This has been set to various mozilla versions from this point onwards without
explanation or specific examples:
https://git.yoctoproject.org/poky/commit/?h=master-next&id=ba06cc8943d021f406c3d292ce415b77319ca5c6
However, sourceforge has recently banned the current 'fake mozilla' version
we've been supplying, resulting in broken version checks:
alex@Zen2:/srv/storage/alex/yocto/build-test-sourceforge$ devtool check-upgrade-status gptfdisk
...
--2024-10-12 11:53:23-- https://sourceforge.net/projects/gptfdisk/files/gptfdisk/
Resolving sourceforge.net (sourceforge.net)... 172.64.150.145, 104.18.37.111, 2606:4700:4400::6812:256f, ...
Connecting to sourceforge.net (sourceforge.net)|172.64.150.145|:443... connected.
HTTP request sent, awaiting response... 403 Forbidden
2024-10-12 11:53:24 ERROR 403: Forbidden.
gptfdisk 1.0.10 UNKNOWN_BROKEN Alexander Kanavin <alex.kanavin@gmail.com>
On the other hand, using wget's default user-agent works just fine:
--2024-10-12 12:16:06-- https://sourceforge.net/projects/gptfdisk/files/gptfdisk/
Resolving sourceforge.net (sourceforge.net)... 104.18.37.111, 172.64.150.145, 2606:4700:4400::6812:256f, ...
Connecting to sourceforge.net (sourceforge.net)|104.18.37.111|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: unspecified [text/html]
(Bitbake rev: 138117dfc05248dcb48707c838c8c1cc09f12d07)
Signed-off-by: Alexander Kanavin <alex@linutronix.de>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
assetTrue is a poor choice for "x in y" since assertIn gives much more
useful output upon failure.
Change such inserts to assertIn or assertEqual to make errors easier
to debug.
(Bitbake rev: dde78e0ff8af872fdc5cdf5354174fc713141102)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
GitHub Codespaces allow users to use GitHub-hosted pre-configured Docker
containers to work on GitHub repositories in VS Code within the browser
or using a locally running instance of VS Code [1].
Users can define access permissions for Codespaces containers, allowing
them to clone or work with other private repositories [2].
To do this, a git credential helper is injected into the container at
/.codespaces/bin/gitcredential_github.sh, which contains the following
lines:
#!/bin/sh
echo protocol=https
echo host=github.com
echo path=
echo username=PersonalAccessToken
echo password=$GITHUB_TOKEN
The `GITHUB_TOKEN` environment variable is automatically set when the
Codespaces container is created. If authorized by the user, it has the
specified permissions. It is thus required to be able to fetch private
git repos from a Codespace container, so add it to `FETCH_EXPORT_VARS`
to make it available for invocations of git.
[1] https://docs.github.com/en/codespaces/overview
[2] https://docs.github.com/en/codespaces/managing-your-codespaces/managing-repository-access-for-your-codespaces
(Bitbake rev: 71ca83bdd541f70737a3a85f05e40c222611af2f)
Signed-off-by: Oliver Kästner <git@oliver-kaestner.de>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
On faster machines these tests are failing as the tests are running
before the page has rendered. Add appripriate wait calls.
(Bitbake rev: c3a425ac3ccafa7b06b319c6a525773d04a2ddac)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The WebDriverException can occur if elements are not present in the DOM
yet. Catch and handle this. It is our most frequently occuring failure
in automated testing now.
(Bitbake rev: 006173cbd32116ff1cea59b2c99eead807be39bb)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
We should release locks in the opposite order to the order we locked
them in to avoid potentially problematic behaviour. This hasn't been
observed with our current usage and is just a preventative improvement.
(Bitbake rev: 4fbef87adaca2df8c1e877aec5588c2e5466f252)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Instead of PRSERVER_UPSTREAM.
The intended variable name is PRSERV_UPSTREAM, as
already used in lib/prserv/serv.py, an consistently
with the PRSERV_HOST variable name.
(Bitbake rev: b0c277f16f9fae51914024c1daecd5d3e4fac5c2)
Signed-off-by: Michael Opdenacker <michael.opdenacker@rootcommit.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Prevents the following warning in bitbake-cookerdaemon.log:
2386729 10:54:16.195427 Exiting (socket: True)
2386729 10:54:16.201065 Exiting as we could obtain the lock
sys:1: ResourceWarning: unclosed file <_io.TextIOWrapper name='/home/laplante/main_yocto/build/bitbake-cookerdaemon.log' mode='a+' encoding='UTF-8'>
(Bitbake rev: 8dbf1ec8139d9dd7f52c1773cccbe7696b3ec1b4)
Signed-off-by: Chris Laplante <chris.laplante@agilent.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
THe test fails on faster systems as we need to wait. Fix that and
a typo.
(Bitbake rev: 749b561196b0b1b388051b345fee2efb6ce68ff7)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Using a hardcoded database ID for a recipe is a bad idea and causes tests
to fail when we change data/versions.
Remove magic numbers.
(Bitbake rev: 6e3a958666ae4821cfa1d43c064a2294e6881638)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Consistently use meta-poky as the test target, instead of a magic number
ID which can vary.
Fix incorrect comments.
Also ensure the viewport is scrolled to the top when changing tabs.
This breaks if other layers are selected like openembedded-core due
to the number of recipes.
(Bitbake rev: 0bcb7b4a8d8e89647aa6179cc61e5eb55d433585)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
"False is not True" is not a helpful error messgae. "XXX not in YYY" is
much more helpful. Tweak the test to give more meaninful failure messages
using assertIn rather than assertTrue.
(Bitbake rev: 6292b5a81ce1f3cf35980d0f669a1b52f3c56695)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
When the fixtures were updated for the styhead release, the test cases
were not updated to match. Fix this.
(Bitbake rev: d40bd9d35fa3850e77a1e349dab2308a54d70d60)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
In python 3 we should always have named tuples in this context. Drop
the obsolete exception handling.
(Bitbake rev: c11c9f18dad1cf4cb21e3bd873338ecc95652062)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
It was never a great solution to persisting data and there are much better
ones now. The last user has been replaced so drop the code and tests.
(Bitbake rev: 681a7516e9f7027e0be6f489c54a7a5e19fa9f06)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The sqlite connection handling is causing problems with python 3.13. The
connection can be closed at gc time which causing warnings and those
can appear at 'random' points and break output, causing weird failures
in different tinfoil tools and other tests.
Using sqlite as an IPC was never a great idea so drop that usage entirely
and just use the standard cache mechanism we already have for other
situations.
(Bitbake rev: fdc55bb649cb77456d0ac48a9600ef289a52af18)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
>From re on python 3.13 onwards: "Passing count and flags as positional arguments is deprecated.
In future Python versions they will be keyword-only parameters."
Avoid the warning.
(Bitbake rev: 016d872dc3c36792af5c84da1d35a5c86996d795)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The COW tests started failing on python 3.13. Looks like it is time
to fix the FIXME and drop the magic numbers!
(Bitbake rev: 2e6608cec508b3b9bab3530f83e70665ff638182)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The pipes module is removed in python 3.13. It was already using the
quote function from shlex so use that directly instead.
The module already imports shlex too so it is an easy substitution.
(Bitbake rev: 9ef4f2a30127455562e38e869793a723eed6c07c)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
When copying files as part of the unpack we currently use cp -p, which
is a shortcut for --preserve=mode,ownership,timestamps.
We do want to preserve timestamps, because some fetchers set these
explicitly.
We don't care about ownership. If the files are owned by us then they
ill remain owned by us, and if they're not then the attempt to change
ownership will be silently ignored. In a shared DL_DIR where files have
group ownership this group access isn't relevant in the single-user build
tree.
We do want to preserve executable bits in the mode, but cp always does
this. The difference between --preserve=mode and no --preserve is that
the mode isn't preserved exactly (no sticky bits, no suid, umask is
applied) but this also isn't a relevant difference in a build tree.
Also expand the arguments to be clearer about what options are being
passed.
The impetus for this is that coreutils 9.4 includes a change in gnulib[1]
and will now try to preserve permission-based xattrs if asked to preserve
the mode. This can result in cp failing when copying a file from a
NFSv4 server with ACLs stored in xattrs to a non-NFS directory where
those xattrs cannot be written:
cp: preserving permissions for ‘./jquery-3.7.1.js’: Operation not supported
The error comes from the kernel refusing to write a system.nfs4_acl
xattr to a file on ext4.
This situation doesn't appear on all systems with coreutils 9.4, at the
time of writing it fails on Ubuntu 24.04 onwards but not Fedora 40. This
is because /etc/xattr.conf is used to determine which xattrs describe
permissions, and Fedora 40 has removed the NFSv4 attributes[2].
Also, use long-form options to make the cp command clearer.
[1] https://github.com/coreutils/gnulib/commit/eb6a8a4dfb
[2] https://src.fedoraproject.org/rpms/attr/blob/rawhide/f/0003-attr-2.4.48-xattr-conf-nfs4-acls.patch
[ YOCTO #15596 ]
(Bitbake rev: 2f35dac0c821ab231459922ed98e1b2cc599ca9a)
Signed-off-by: Ross Burton <ross.burton@arm.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Add test cases for a module path consisting of only a hostname and add
checksum verification of the unpacked go.mod files.
(Bitbake rev: 9380859b59923dee17469348f472a22e11be1779)
Signed-off-by: Christian Lindeberg <christian.lindeberg@axis.com>
Signed-off-by: Alexandre Belloni <alexandre.belloni@bootlin.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Add styhead. Also fix a typo in the usage instructions.
(Bitbake rev: d3c84198771b7f79aa84dc73061d8ca071fe18f3)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
When calculating the module name for a gomod URI with only a hostname,
e.g.:
gomod://go.opencensus.io;version=v0.24.0;sha256sum=203a767d7f8e7c1ebe5588220ad168d1e15b14ae70a636de7ca9a4a88a7e0d0c
the non-existing path would actually be treated as "/", which resulted
in a trailing slash being added to the module name preventing the unpack
method from correctly locating the go.mod file.
(Bitbake rev: f0e02e1de4d649e647e4ab61341042dd38d0eeb0)
Signed-off-by: Peter Kjellerstedt <peter.kjellerstedt@axis.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
This reverts commit a4f42e396e2942fde94b8b4944487c1c45f7a295.
Unfortunately configparser is a bit more strict and fails to parse e.g.:
e104dd7f27/.gitmodules
[submodule "components/bt/controller/lib_esp32"]
path = components/bt/controller/lib_esp32
url = ../../espressif/esp32-bt-lib.git
The url is using 8 spaces while path 1 tab causing:
Exception: configparser.NoOptionError: No option 'url' in section: 'submodule "components/bt/controller/lib_esp32"'
It was fixed in:
62ca8e2fb4 (diff-fe7afb5c9c916e521401d3fcfb4277d5071798c3baf83baf11d6071742823584)
but mcuboot is using a bit older esp-idf revision in:
https://github.com/ATmobica/mcuboot/blame/main/.gitmodules
and mcuboot is then used as submodule in:
https://github.com/project-chip/connectedhomeip/blob/master/.gitmodules
so it might take a while for the fix to be propagated everywhere.
Not sure how common these issues are, but configparser parses
"url = ../../espressif/esp32-bt-lib.git" as 2nd line of "path"
value (because it's indented differently) while git submodule
and old gitsm implementation parses it as separate path and
url keys.
(Bitbake rev: d9eb2650256292a524a0c0bb1c0562d87bedcbc4)
Signed-off-by: Martin Jansa <martin.jansa@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Previously, incrementing "0.9" would result in "0.1.0", which
generally gets recognised as a lower version number. Even more
surprising, incrementing "0.99" returned "0.1.0.0".
This is due to the behaviour of the list function on a string
object; it adds each character as an element in a new list,
causing the new string '10' to become the list [ '1', '0' ].
Instead of converting a string to a list, add the string to a
new list, and concatenate it with the existing list slice. And
provide test cases for "0.9" -> "0.10" and related edge cases.
(Bitbake rev: 96ddeefa88ff4c37e9ea096726a7cdca5b5b4572)
Signed-off-by: Dan McGregor <dan.mcgregor@usask.ca>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
.gitmodules is basically ini-style, so use configparser instead of manually
parsing by hand.
(Bitbake rev: a4f42e396e2942fde94b8b4944487c1c45f7a295)
Signed-off-by: Ross Burton <ross.burton@arm.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Add a go module fetcher for downloading module dependencies to the
module cache directly from a git repository. The fetcher can be used
with the go-mod class in OE-Core.
A module dependency can be specified with:
SRC_URI += "gomodgit://golang.org/x/net;version=v0.9.0;srcrev=..."
(Bitbake rev: 29ff38ccf0d5389a5bee81e252a78548361a9d7c)
Signed-off-by: Christian Lindeberg <christian.lindeberg@axis.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Add a go module fetcher for downloading module dependencies to the
module cache from a module proxy. The fetcher can be used with the
go-mod class in OE-Core.
A module dependency can be specified with:
SRC_URI += "gomod://golang.org/x/net;version=v0.9.0;sha256sum=..."
(Bitbake rev: 5ff4694bf305e266ebf0abab5d9745c6b6d07d67)
Signed-off-by: Christian Lindeberg <christian.lindeberg@axis.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
There is a subtle issue with full pathnames being included in the
file checksums since the sorting might be different depending upon
how layers are being setup causing hash mismatches for recipes appeneded
from other layers with differing directory layouts.
Avoid this by filtering out to the path basename which is what is written
into the sig data anyway later in the code.
(Bitbake rev: 83acc21cdfdb410082c0871ac7693d29a7c5627d)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Allow the metadata to define code visitor expressions which mean that
custom dependencies can be handled in function libraries.
An example is the qa.handle_error function in OE which can set something
like:
"""
def handle_error_visitorcode(name, args):
execs = set()
contains = {}
warn = None
if isinstance(args[0], ast.Constant) and isinstance(args[0].value, str):
for i in ["ERROR_QA", "WARN_QA"]:
if i not in contains:
contains[i] = set()
contains[i].add(args[0].value)
else:
warn = args[0]
execs.add(name)
return contains, execs, warn
handle_error.visitorcode = handle_error_visitorcode
"""
Meaning that it can have contains optimisations on ERROR and WARN_QA
instead of hard dependencies.
One drawback to this solution is the parsing order. Functions with
visitorcode need to be defined before anything else references them
or the visitor code will not function for the earlier references.
(Bitbake rev: 5bd0c65c217394cde4c8e382eba6cf7f4b909c97)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The '@types/ramda' [1] npm package has recently gained a dependency on
the 'types-ramda' [2] npm package. Both have the same version number.
The name mangling results in the tarballs of both packages sharing the same
name, but different contents.
Fix that by accepting '@' as valid character in the package name,
resulting in one package named @types-ramda and one called types-ramda.
[1]: https://www.npmjs.com/package/@types/ramda
[2]: https://www.npmjs.com/package/types-ramda
(Bitbake rev: 7c9573cb6ea2081bc585eb65267f3124fd4d7e43)
Signed-off-by: Leonard Göhrs <l.goehrs@pengutronix.de>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The changes of 1ab1d36c0af6fc58a974106b61ff4d37da6cb229 added calls to "gsutil stat" to avoid unhandled exceptions, however:
- in the case of checkstatus() this is redundant with the call to self.gcp_client.bucket(ud.host).blob(path).exists() which already returns True/False
and does not throw an exception in case the file does not exist.
- Also the call to gsutil stat is much slower than using the python client to call exists() so we should not replace the call to exists() with a call to gsutil stat.
- I think the intent of calling check_network_access in checkstatus() was to error-out in case the error is disabled. We can rather change the string "gsutil stat" to something else to make the code more readable.
- add a try/except block in download() instead of the extra call to gsutil
[RP: Tweak to avoid import until needed so google module isn't required for everyone]
(Bitbake rev: dd120f630e9ddadad95fe83728418335a14d3c3b)
Signed-off-by: Etienne Cordonnier <ecordonnier@snap.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
We fetch npm dependencies from the npm-shrinkwrap.json file. They can
point to a package on the NPM registry with a version field, or to a
git/http/file URL with the resolved field. Such packages are allowed not
to declare a registry version field because they may not have been
published to the NPM registry. The previous implementation refuses to
fetch such packages and throws an error.
The resolved field contains the exact source, including the revision,
wich we can use to pass as SRC_URI to the git/http/file fetcher. The
integrity field is also mandatory for HTTP tarballs which will ensure
reproducibility. So even if the version field is not present, we are
still fetching a precise revision of the package.
Another commit published along this stack is also required in the npm
class to support these packages.
v5:
- improve commit message
v3:
- Split bitbake npmsw.py modification in another commit
Co-authored-by: Tanguy Raufflet <tanguy.raufflet@savoirfairelinux.com>
(Bitbake rev: 209982b5a3efc8081e65b4326bf9b64eef7f0ba0)
Signed-off-by: Tanguy Raufflet <tanguy.raufflet@savoirfairelinux.com>
Signed-off-by: Enguerrand de Ribaucourt <enguerrand.de-ribaucourt@savoirfairelinux.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The NPM package.json documentation[1] states that git URLs may contain
a commit-ish suffix to specify a specific revision. When running
`npm install`, this revision will be looked for on any branch of the
repository.
The bitbake implementation however translates the URL stored in
package.json into a git URL to be fetch by the bitbake git fetcher. The
bitbake fetcher git.py, enforces the branch to be master by default. If
the revision specified in the package.json is not on the master branch,
the fetch will fail while the package.json is valid.
To fix this, append the ";nobranch=1" suffix to the revision in the git
URL to be fetched. This will make the bitbake git fetcher ignore the
branch and respect the behavior of `npm install``.
This can be tested with the following command:
$ devtool add --npm-dev https://github.com/seapath/cockpit-cluster-dashboard.git -B version
Which points to a project which has a package.json with a git URL:
```json
"devDependencies": {
"cockpit-repo": "git+https://github.com/cockpit-project/cockpit.git#d34cabacb8e5e1e028c7eea3d6e3b606d862b8ac"
}
```
In this repo, the specified revision is on the "main" branch, which
would fail without this fix.
[1] https://docs.npmjs.com/cli/v10/configuring-npm/package-json#git-urls-as-dependencies
Co-authored-by: Tanguy Raufflet <tanguy.raufflet@savoirfairelinux.com>
(Bitbake rev: 37a35adf7882f231c13643dbf9168497c6a242a1)
Signed-off-by: Tanguy Raufflet <tanguy.raufflet@savoirfairelinux.com>
Signed-off-by: Enguerrand de Ribaucourt <enguerrand.de-ribaucourt@savoirfairelinux.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
When tasks fail, it's very frustrating to have to scroll up to find the
log path(s). Many of us have the muscle memory to navigate to the 'temp'
directories under tmp/work/, but new users do not.
This change enhances the final summary to include log paths (reported
via bb.build.TaskFailed events). Here's an example:
NOTE: Tasks Summary: Attempted 856 tasks of which 853 didn't need to be rerun and 3 failed.
Summary: 3 tasks failed:
virtual:native:/home/chris/repos/poky/meta/recipes-core/ncurses/ncurses_6.5.bb:do_fetch
log: /home/chris/repos/poky/build/tmp/work/x86_64-linux/ncurses-native/6.5/temp/log.do_fetch.1253462
/home/chris/repos/poky/meta/recipes-core/ncurses/ncurses_6.5.bb:do_fetch
log: /home/chris/repos/poky/build/tmp/work/core2-64-poky-linux/ncurses/6.5/temp/log.do_fetch.1253466
virtual:nativesdk:/home/chris/repos/poky/meta/recipes-core/ncurses/ncurses_6.5.bb:do_fetch
log: /home/chris/repos/poky/build/tmp/work/x86_64-nativesdk-pokysdk-linux/nativesdk-ncurses/6.5/temp/log.do_fetch.1253467
Summary: There were 3 WARNING messages.
Summary: There were 6 ERROR messages, returning a non-zero exit code.
Each log is rendered as a clickable hyperlink in the terminal. See
https://gist.github.com/egmontkob/eb114294efbcd5adb1944c9f3cb5feda
(Bitbake rev: 2852a478ab03a482989c3a7e247860ab4f0e9f3e)
Signed-off-by: Chris Laplante <chris.laplante@agilent.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
When getting errors during build, they would be printed using repr(), which
doesnt have a lot of context in some cases.
For example FileNotFoundError(2, "file or directory not found"), would be
printed, without the path of the file not found.
This changes the build logging to use str() instead, which according to
the spec is fore human readable strings, whereas repr() is for string
representations that can be be used as valid python.
(Bitbake rev: 2a97024b8b9245ec47deace011a7560a25491207)
Signed-off-by: Troels Dalsgaard Hoffmeyer <tdah@bang-olufsen.dk>
Signed-off-by: Alexandre Belloni <alexandre.belloni@bootlin.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
"addtask do_XXX before YYY after ZZZ "
where YYY or ZZZ is missing the "do_" prefix don't work as expected. Ajust the
code so that it doesn't just silently do the wrong thing but works as expected.
Expand a test case to cover this.
(Bitbake rev: 21670b9bb8936ec44aedff26163948bbc2ceb44a)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The recent addtask improvement to handle comments complicated the regex significantly
and there are already a number of corner cases in that code which aren't handled well.
Instead of trying to complicate the regex further, switch to code logic instead. This
means the following cases are now handled:
* addtask with multiple task names
* addtask with multiple before constraints
* addtask with multiple after constraints
The testcase is updated to match the improvements.
(Bitbake rev: 417016b83c21fca7616b2ee768d5d08e1edd1e06)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Within tinfoil, the user can write to the configuration data but it won't
cause the data_hash checksum to be re-written, meaning cached parsing
data can be reused when it would now be incorrect.
Abstract out the data_hash code and add it to the invalidateCaches
command, called by tinfoil.modified_files() meaning that tinfoil can
instruct bitbake to update the caches and re-parse if necessary.
Also move the data_hash entirely into databuilder and drop the copy
in cooker as obsolete and not needed.
(Bitbake rev: d9ee77829f693ce75348fa64f406fcecfe4989aa)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
I can't spot any users of this function and it is poking at variables
inside cooker that could and are about to change so drop it.
(Bitbake rev: 52491808706e9e58b5e6b59d2d792353d77c8b66)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Technically our syntax would allow for comments after an addtask/deltask.
Currently these get silently processed in various ways by the code which
is bad.
Tweak the regex to drop any comments and add test cases to ensure this
continues to work in the future.
[YOCTO #15250]
(Bitbake rev: 64f8796e55a8826ffec0d76b993c8256713f67a3)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
We missed on adding Scarthgap to the list of user manuals, so let's fix
this oversight.
(Bitbake rev: 2f12db7b7b03c18de6257a9886c493535f0cb5a2)
Signed-off-by: Quentin Schulz <quentin.schulz@cherry.de>
Signed-off-by: Alexandre Belloni <alexandre.belloni@bootlin.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
We missed on adding nanbield to the release manuals. It's now EOL so
let's add it directly to the oudated release manuals section.
(Bitbake rev: b891878a7f08b15ee5d6d037d99fbc769cc905e4)
Signed-off-by: Quentin Schulz <quentin.schulz@cherry.de>
Signed-off-by: Alexandre Belloni <alexandre.belloni@bootlin.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Fixed:
- BBMULTICONFIG = "qemux86-64 qemuarm64" and more than 70 layers in BBLAYERS
$ bitbake -p -P
Check profile.log.processed, the record() cost more than 20 seconds, it is less
than 1 second when multiconfig is not enabled, and there would be the following
error when more muticonfigs are enabled:
Timeout while waiting for a reply from the bitbake server
Don't change the type of loginfo['detail'] or re-assign it can make record()
back to less than 1 second, this won't affect COW since loginfo is a mutable
type.
The time mainly affected by two factors:
1) The number of enabled layers, nearly 1 second added per layer when the
number is larger than 50.
2) The global var such as USER_CLASSES, about 1 ~ 2 seconds added per layer
when the layers number is larger than 50.
(Bitbake rev: 0596aa0d5b0e4ed3db11b5bd560f1d3439963a41)
Signed-off-by: Robert Yang <liezhi.yang@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Enforce the default remote name to "origin", as assumed in numerous
places.
This prevents build failures in case the system/user configuration sets
this to a different value.
(Bitbake rev: 1d7360031164f04887c792fb0b2dd86c6ccfcc23)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Update the test cases since the implementation is changed:
* test_shallow_multi_one_uri()
The a_branch and v0.0 had the same revision, and it required fetch a_branch
and remove histories of v0.0 which were conflicted, and bitbake reported:
fatal: no commits selected for shallow requests
Make a_branch and v0.0 have different revs to fix the problem.
And now the 'rev^' is not needed, so update self.assertRevCount() as well.
* test_shallow_multi_one_uri_depths()
Update self.assertRevCount(), now git only fetches the required revs.
* test_shallow_fetch_missing_revs()
The command is:
$ git fetch --shallow-exclude=v0.0 master
But master and v0.0 uses the same revision, so there is no commit to fetch.
* test_shallow_fetch_missing_revs_fails()
Two unneeded committs are not fetched now:
- rev^
- One not specified or required tag.
So update self.assertRevCount()
(Bitbake rev: 48eff9d9a660ad6b9bd8b53a7dcec600ef42b1d1)
Signed-off-by: Robert Yang <liezhi.yang@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
This patch can make the following settings much more faster:
BB_GIT_SHALLOW = "1"
BB_GENERATE_MIRROR_TARBALLS = "1"
* The previous implementation was:
- Make a full clone for the repo from local ud.clonedir
- Use git-make-shallow to remove unneeded revs
It was very slow for recipes which have a lot of SRC_URIs, for example
vulkan-samples and docker-compose, the docker-compose can't be done after 5
hours.
$ bitbake vulkan-samples -cfetch
Before: 12 minutes
Now: 2 minutes
$ bitbake docker-compose -cfetch
Before: More than 300 minutes
Now: 15 minutes
* The patch uses git shallow fetch to fetch the repo from local
ud.clonedir:
- For BB_GIT_SHALLOW_DEPTH: git fetch --depth <depth> rev
- For BB_GIT_SHALLOW_REVS: git fetch --shallow-exclude=<revs> rev
Then the git repo will be shallow, and git-make-shallow is not needed any
more.
And git shallow fetch will download less commits than before since it doesn't
need "rev^" to parse the dependencies, the previous code always need 'rev^'.
(Bitbake rev: a5a569c075224fe41707cfa9123c442d1fda2fbf)
Signed-off-by: Robert Yang <liezhi.yang@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* to avoid wrong http response in checkstatus function:
in wget checkstatus() we are using 'HTTPConnectionCache' to share connections
1. state_file1(exists on http server) use shared connection <shared1> to send request
2. http_server recieved request of state_file1, but delayed by some reason to sent respone
3. state_file1 checkstatus() failed by timeout and drop shared connection <shared1>
4. state_file2(not exists on http server) get shared connection <shared1> and send request
5. http_server finally send 200 response for state_file1
6. state_file2 recived 200 response and thought it was exists on http_server
(Bitbake rev: bf6d0282ab88b4edc4b9e58184cd76cce965abbd)
Signed-off-by: y75zhang <yang-mark.zhang@nokia-sbell.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Layer cleanup similar to
https://git.openembedded.org/openembedded-core/commit/?id=26c74fd10614582e177437608908eb43688ab510
It was probably not found before beacause of the extra "d" parameter.
That seem to be a bug as getVar does not support that.
(Bitbake rev: 720189b810995c5737853458b7eb3779ca0df37e)
Signed-off-by: Peter Marko <peter.marko@siemens.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
This fixes the following error when calling 'bitbake-dumpsig' or
'bitbake-diffsigs' when having only a single sigfile available:
| Traceback (most recent call last):
| File "[..]/poky/bitbake/bin/bitbake-dumpsig", line 171, in <module>
| files = find_siginfo_task(tinfoil, options.taskargs[0], options.taskargs[1])
| ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
| File "[..]/poky/bitbake/bin/bitbake-dumpsig", line 83, in find_siginfo_task
| sig2 = latestsigs[1]
| ~~~~~~~~~~^^^
| IndexError: list index out of range
Handle this by adding (and returning) the path for the second sigfile
only if one is found. This way it will work for both diffsigs and
dumpsig use case.
The calling argparse code already deals with find_siginfo_task()
returning only a single file.
For 'bitbake-dumpsig' it will just dump the single sigfile, for
'bitbake-diffsigs' it will emit a proper error message again:
| ERROR: Only one matching sigdata file found for the specified task (systemd configure)
(Bitbake rev: 25057d33e9131f3214a06bbb316c916c744f8f03)
Signed-off-by: Enrico Jörns <ejo@pengutronix.de>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
If modules do something like "from glob import glob" then we end up
checksumming the glob code. That leads to bugs as the code can change
between different python versions for example, leading to checksum
instability.
We should ignore functions not from the current file as implemented
by this change.
(Bitbake rev: 1e6f862864539d6f6a0bea3e4479e0dd40ff3091)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Currently, if a pylib function changes contents, the taskhash remains
unchanged since we assume the functions have stable output. This is
probably a poor assumption so take the code of the function into account
in the taskhashes. This avoids certain frustrating build failures we've
been seeing in automated testing.
To make this work we have to add an extra entry to the python code parsing
cache so that we can store the hashed function contents for efficiency as
in the python module case, that isn't used as the key to the cache.
The cache version changes since we're adding data to the cache.
(Bitbake rev: b2c3438ebe62793ebabe2c282534893908d520b4)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The bb_cache.dat might be an invalid symlink when error happens, then
os.path.exists(symlink) would return False for it, the invalid symlink
wouldn't be removed and os.symlink can't update it any more.
Use os.path.islink(symlink) can fix the problem.
(Bitbake rev: 1387d7b9ee3f270488f89b29f36f9f240e44accc)
Signed-off-by: Robert Yang <liezhi.yang@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* avoids long trace when BB_HASHSERVE points to non-existent socket
file, e.g.:
BB_HASHSERVE = "unix:///OE/no-socket.sock"
or when running the build before starting the bin/bitbake-hashserv.
* now it shows just warnings like it did in kirkstone
many of them, e.g. 6 just for rebuilding zlib-native, but better than long trace
for nonexistent socket file:
WARNING: zlib-native-1.3.1-r0 do_create_spdx: Error contacting Hash Equivalence Server unix:///OE/no-socket.sock: [Errno 2] No such file or directory
for existing file, but before starting bin/bitbake-hashserv:
WARNING: zlib-native-1.3.1-r0 do_create_spdx: Error contacting Hash Equivalence Server unix:///OE/hashserv.sock: [Errno 111] Connection refused
ERROR: An uncaught exception occurred in runqueue############################################################### | ETA: 0:00:00
Traceback (most recent call last):
File "/OE/build/oe-core/bitbake/lib/hashserv/__init__.py", line 80, in create_client(addr='unix:///OE/no-socket.sock', username=None, password=None):
if typ == ADDR_TYPE_UNIX:
> c.connect_unix(*a)
elif typ == ADDR_TYPE_WS:
File "/OE/build/oe-core/bitbake/lib/bb/asyncrpc/client.py", line 241, in Client.connect_unix(path='/OE/no-socket.sock'):
self.loop.run_until_complete(self.client.connect_unix(path))
> self.loop.run_until_complete(self.client.connect())
File "/usr/lib/python3.12/asyncio/base_events.py", line 687, in _UnixSelectorEventLoop.run_until_complete(future=<Task finished name='Task-6' coro=<AsyncClient.connect() done, defined at /OE/build/oe-core/bitbake/lib/bb/asyncrpc/client.
py:150> exception=FileNotFoundError(2, 'No such file or directory')>):
> return future.result()
File "/OE/build/oe-core/bitbake/lib/bb/asyncrpc/client.py", line 152, in AsyncClient.connect():
if self.socket is None:
> self.socket = await self._connect_sock()
await self.setup_connection()
File "/OE/build/oe-core/bitbake/lib/bb/asyncrpc/client.py", line 85, in connect_sock:
sock = socket.socket(socket.AF_UNIX, socket.SOCK_STREAM, 0)
> sock.connect(os.path.basename(path))
finally:
FileNotFoundError: [Errno 2] No such file or directory
ERROR: Running idle function
Traceback (most recent call last):
File "/OE/build/oe-core/bitbake/lib/hashserv/__init__.py", line 80, in create_client(addr='unix:///OE/no-socket.sock', username=None, password=None):
if typ == ADDR_TYPE_UNIX:
> c.connect_unix(*a)
elif typ == ADDR_TYPE_WS:
File "/OE/build/oe-core/bitbake/lib/bb/asyncrpc/client.py", line 241, in Client.connect_unix(path='/OE/no-socket.sock'):
self.loop.run_until_complete(self.client.connect_unix(path))
> self.loop.run_until_complete(self.client.connect())
File "/usr/lib/python3.12/asyncio/base_events.py", line 687, in _UnixSelectorEventLoop.run_until_complete(future=<Task finished name='Task-6' coro=<AsyncClient.connect() done, defined at /OE/build/oe-core/bitbake/lib/bb/asyncrpc/client.
py:150> exception=FileNotFoundError(2, 'No such file or directory')>):
> return future.result()
File "/OE/build/oe-core/bitbake/lib/bb/asyncrpc/client.py", line 152, in AsyncClient.connect():
if self.socket is None:
> self.socket = await self._connect_sock()
await self.setup_connection()
File "/OE/build/oe-core/bitbake/lib/bb/asyncrpc/client.py", line 85, in connect_sock:
sock = socket.socket(socket.AF_UNIX, socket.SOCK_STREAM, 0)
> sock.connect(os.path.basename(path))
finally:
FileNotFoundError: [Errno 2] No such file or directory
Summary: There were 2 ERROR messages, returning a non-zero exit code.
1605616 09:29:05.369352 Parse cache valid
1605616 09:30:14.500863 Registering idle function <function BBCooker.buildTargets.<locals>.buildTargetsIdle at 0x7f43988c09a0>
1605616 09:30:14.500927 Removing idle function <bound method Command.runAsyncCommand of <bb.command.Command object at 0x7f43a961c350>>
1605616 09:30:14.573274 Exception Traceback (most recent call last):
File "/OE/build/oe-core/bitbake/lib/bb/server/process.py", line 435, in idle_thread_internal
retval = function(self, data, False)
^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/OE/build/oe-core/bitbake/lib/bb/cooker.py", line 1487, in buildTargetsIdle
retval = rq.execute_runqueue()
^^^^^^^^^^^^^^^^^^^^^
File "/OE/build/oe-core/bitbake/lib/bb/runqueue.py", line 1651, in execute_runqueue
return self._execute_runqueue()
^^^^^^^^^^^^^^^^^^^^^^^^
File "/OE/build/oe-core/bitbake/lib/bb/runqueue.py", line 1567, in _execute_runqueue
if self.rqdata.prepare() == 0:
^^^^^^^^^^^^^^^^^^^^^
File "/OE/build/oe-core/bitbake/lib/bb/runqueue.py", line 1290, in prepare
unihashes = bb.parse.siggen.get_unihashes(ready)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/OE/build/oe-core/bitbake/lib/bb/siggen.py", line 713, in get_unihashes
with self.client() as client:
File "/usr/lib/python3.12/contextlib.py", line 137, in __enter__
return next(self.gen)
^^^^^^^^^^^^^^
File "/OE/build/oe-core/bitbake/lib/bb/siggen.py", line 595, in client
self._client = hashserv.create_client(self.server, **self.get_hashserv_creds())
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/OE/build/oe-core/bitbake/lib/hashserv/__init__.py", line 88, in create_client
raise e
File "/OE/build/oe-core/bitbake/lib/hashserv/__init__.py", line 80, in create_client
c.connect_unix(*a)
File "/OE/build/oe-core/bitbake/lib/bb/asyncrpc/client.py", line 241, in connect_unix
self.loop.run_until_complete(self.client.connect())
File "/usr/lib/python3.12/asyncio/base_events.py", line 687, in run_until_complete
return future.result()
^^^^^^^^^^^^^^^
File "/OE/build/oe-core/bitbake/lib/bb/asyncrpc/client.py", line 152, in connect
self.socket = await self._connect_sock()
^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/OE/build/oe-core/bitbake/lib/bb/asyncrpc/client.py", line 85, in connect_sock
sock.connect(os.path.basename(path))
FileNotFoundError: [Errno 2] No such file or directory
broke the idle_thread, exiting
1605616 09:30:14.673756 Exiting (socket: True)
1605616 09:30:14.683153 Exiting as we could obtain the lock
sys:1: ResourceWarning: unclosed file <_io.TextIOWrapper name='/OE/build/oe-core/bitbake-cookerdaemon.log' mode='a+' encoding='UTF-8'>
sys:1: ResourceWarning: unclosed <socket.socket fd=17, family=1, type=1, proto=0>
ResourceWarning: Enable tracemalloc to get the object allocation traceback
(Bitbake rev: 550c86969e5a137ffef61b08a520a4855232fb1c)
Signed-off-by: Martin Jansa <martin.jansa@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The test_single_layer_page test case consistently fails. It is not obvious why
but if we change the argument in the following from 8 to 7 it passes.
url = reverse("layerdetails", args=(TestProjectPage.project_id, 8))
E selenium.common.exceptions.TimeoutException: Message: An element matching "#change-notification" should be visible
=========================== short test summary info ============================
FAILED ../bitbake/lib/toaster/tests/functional/test_project_page.py::TestProjectPage::test_single_layer_page
(Bitbake rev: c7e12145d8ea641925e3c06ba4f11c2dae66288a)
Signed-off-by: Tim Orling <tim.orling@konsulko.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
In line with changes in gen_fixtures.py:
* Add projectscarthgap
- Add Scarthgap to slot 1.
* Move Kirkstone down to slot 4
* Drop projectdunfell
- Drop EOL Dunfell from slot 5
(Bitbake rev: a4ae788f95d8e54713528374a9171c636aa747c5)
Signed-off-by: Tim Orling <tim.orling@konsulko.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
gen_fixtures.py:
* Add Scarthgap to slot #1
* Drop EOL Mickledore
* Move Kirkstone to lower slot
* Drop optional slot for EOL Dunfell
Refresh oe-core.xml and poky.xml
(Bitbake rev: 11c7214a292cd296eed5490b6726e672f9179131)
Signed-off-by: Tim Orling <tim.orling@konsulko.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Some distributions (namely Fedora Core 40) have started replacing
wget with wget2. There are some changes to wget2 that make it
incompatible with wget:
1. ftp/ftps is not supported anymore
2. progress 'dot' is not yet supported
3. Relative paths in -P and -O are not correctly dealt with
Item 1: Is already dealt with since Scarthgap by only adding the
option --passive-ftp when the URL specifies ftp/sftp. While that
won't help if ftp/sftp is actually required it at least does
not break http/https downloads.
Item 2: While not supported it at least does not break the operation.
Item 3: If there are relative path components in -P or -O then wget2
only deals with them correctly if there is one, and only one, relative
path component at the beginning of the path:
-P ./downloads works
-P ../downloads works
-P ../../downloads does not work
-P ./../downloads does not work
-P /home/user/downloads/../downloads does not work
In cases where there are more than one relative path component at
the beginning of the path and/or one or more reltaive path
component somewhere in the middle or end of the path, wget2 aborts
with the message Internal error: Unexpected relative path: '<path>')
Such can happen if DL_DIR includes relative path components e.g.
DL_DIR = "${TOPDIR}/../../downloads".
This patch canonicalizes DL_DIR before it is passed to wget.
(Bitbake rev: 3e4208952b086adc510e78c1c5f9cf4550d79dc9)
Signed-off-by: Rudolf J Streif <rudolf.streif@ibeeto.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The save comes with an IO overhead which can slow down the rehash loop in bitbake
a lot. We only needed to do this when recipes were doing unihash cache copying. Now
they aren't doing that, drop this IO pain point.
(Bitbake rev: dfc15ef99302dea22a051c9eb8398ffd5cf1fc20)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The code in OE-Core using this has been replaced with something more fit
for purpose. Drop these function calls as they were never a great idea in the
first place and cause IO slowdown for runqueue needing to sync the cache.
(Bitbake rev: 2c8fa57778c4bd2a5c48a60b701ac57de4289cb2)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Upstream changed their urls, update our test to match.
(Bitbake rev: dc391b86540ec5e0a0f1d811c776a22d443b1c06)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The default connection timeout for websockets is 10 seconds, so use the
provided client timeout instead (which defaults to 30 seconds).
(Bitbake rev: 23681775e5941e54ebead469addf708fca1e6beb)
Signed-off-by: Joshua Watt <JPEWhacker@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
On Fedora40, "localhost" sometimes resolves to ::1 and sometimes to 127.0.0.1
and python only binds to one of the addresses, leading to test failures.
Use 127.0.0.1 explicitly to avoid problems of the name resolution, we're trying
to test things other than the host networking.
(Bitbake rev: 9adc6da42618f41bf0d6b558d62b2f3c13bedd61)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
When we intercepted the file download to a temp file, we broke the
exist/size checks which need to happen before the rename. Correct
the ordering.
For some reason, python 3.12 exposes this problem in the selftests
differently to previous versions.
(Bitbake rev: c56bd9a9280378bc64c6a7fe6d7b70847e0b9e6d)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
When switching from normal mode to stream mode, skip calling
self._set_mode() again because this will cause a recursion into the
_set_mode() function and causes problems.
Also cleanup some of the error checking during this process
This bug affected when a client would attempt to switch from one stream
mode to another, and meant that the server would get an invalid message
from the client. This would cause the server to disconnect the client,
and the client would then reconnect in normal mode which was the mode it
wanted anyway and thus it would carry on without any errors. This made
the bug not visible on the client side, but resulting in a lot of
backtrace JSON decoding exceptions in the server logs.
(Bitbake rev: 1826bc41ab3369ac40034c5eaf698748b769b881)
Signed-off-by: Joshua Watt <JPEWhacker@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Similar to looking up unihashes, use the batch API when checking if a
unihash exists to speed up lookups
(Bitbake rev: 0ac521ff37b578f7487bca0eccc7dc9e5974991b)
Signed-off-by: Joshua Watt <JPEWhacker@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Batching support on the client side has proven to be a much more
effective way of dealing with server latency than multiple client
connections and is also much nicer on the server, so drop the client
pool support from asyncrpc and the hash server
(Bitbake rev: 6f80560f1c7010d09fe5448fdde616aef8468102)
Signed-off-by: Joshua Watt <JPEWhacker@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Drops support for client pools, since batching support in the client
code has proven to be much more effective
(Bitbake rev: 85dafaf8e070459f7de7bfb37300d8b60a27002e)
Signed-off-by: Joshua Watt <JPEWhacker@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
cooker: We can call multiprocessing close() unconditionally and tweak a
comment give 3.8 is now the minimum version.
lib/bb: We can drop the logger addition code only needed before 3.6
asyncrpc/hashserv: Since the minimum version is 3.8, we can drop the
conditional code.
(Bitbake rev: 16f4386400f88ba50605307961c248bef09895c1)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
We've seeing profiling tracebacks when parse errors occur during
profiling. Try and avoid these but not processing invalid profiles.
(Bitbake rev: 171bd9dd575307fbd61b5179ad86131d76add067)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
It makes sense to switch to a more recent version and keep up to date
with upstream changes and things like new python version support.
(Bitbake rev: f5462156036e71911c66d07dbf3303cde862785b)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
SO_REUSEPORT is a socket option that allows multiple servers to listen
on the same TCP port, and the kernel will automatically load balance the
connections between them. This is particularly helpful for the hash
server since it runs in a single thread. To take advantage of a
multi-core server, multiple servers can be started in parallel with this
option (up to 1 per CPU) and the kernel will load balance between them.
(Bitbake rev: d72d5a7decb489e2af0ebc43cfea0ca3e4353e9b)
Signed-off-by: Joshua Watt <JPEWhacker@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
