This is set at the start of the loop anyway so it does nothing. Drop
the pointless code.
(Bitbake rev: dcf78788daa177bf5c438f33b3c9f7ced7aea8ab)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
(cherry picked from commit e6a3173c9cdf349ccbd4cf612868f92cce8717c8)
Signed-off-by: Steve Sakoman <steve@sakoman.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
(Bitbake rev: 1e5c5efa6fe9175b27b21cd7a5b833bd3af55238)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
(cherry picked from commit 7254eb6b3e8ef504ef2274541dcc55f1d42238c6)
Signed-off-by: Steve Sakoman <steve@sakoman.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
(Bitbake rev: 00f10fe95393728e94339f3512171ebab96f1900)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
(cherry picked from commit 625565087d8c9e7a6a79b0b4f3e5be2d77d5f100)
Signed-off-by: Steve Sakoman <steve@sakoman.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
(Bitbake rev: f9b563dc02535b594975ac0f5dab8e12d11a6ce0)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
(cherry picked from commit 42809f6acb79e39042e81d54c28efb92b7481e44)
Signed-off-by: Steve Sakoman <steve@sakoman.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
(Bitbake rev: 43bb92a2728832eeb1a207809e21404c86ff1710)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
(cherry picked from commit 879f17ecd5ba09e217cef74f6a51339b145e8ef5)
Signed-off-by: Steve Sakoman <steve@sakoman.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The function has a loop where the variable is never used which I was going
to fix but the entire function never seems to be called so remove it entirely.
(Bitbake rev: d739799a1e68dc2ad0414d4ae7d9e079cedcee3c)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
(cherry picked from commit 3bcb20f025907f4e88bbe3d14f5638d5f01010cb)
Signed-off-by: Steve Sakoman <steve@sakoman.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
(Bitbake rev: 609d50a6ccb6f794a3ba6d73a820927eba68891c)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
(cherry picked from commit 140929b404ee1e2f5e0e1a3a1d3aa49fb3759ade)
Signed-off-by: Steve Sakoman <steve@sakoman.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
(Bitbake rev: c6df0e68ef52c10be9e8773e660c3bb5c9d1e915)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
(cherry picked from commit 1c811ad6f10560e7a7fb6830cf83707551ba04bd)
Signed-off-by: Steve Sakoman <steve@sakoman.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
(Bitbake rev: bf221631e4b591c80aadd26c2328586b6a4c9eac)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
(cherry picked from commit aca0ff85109f4b0f3c201c02c3f59cad7ee2e787)
Signed-off-by: Steve Sakoman <steve@sakoman.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
(Bitbake rev: 65413c1cecc7e667b04d73b95de9159c688f7ce3)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
(cherry picked from commit 543315e6463f15ca7ab2b4ef3e8ed41bb4207ccf)
Signed-off-by: Steve Sakoman <steve@sakoman.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Fix an issue where two tests have the same name with one overwriting the
other.
(Bitbake rev: 38842a8150f44f5d31d9bd2b0b6ec0502acc971d)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
(cherry picked from commit da812d938fd79e2cc7bdf355ccf5b0f9ead684c4)
Signed-off-by: Steve Sakoman <steve@sakoman.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
During parsing, Python raises
RuntimeError: dictionary changed size during iteration
in getRuntimeProviders, if you happen to have a recipe
with an explicit RDEPENDS on a dynamic package containing a '+'
character, such as 'gtk+3-locale-en'.
This is because we're using the modified pattern as the
key into the packages_dynamic dict to append to rproviders,
and since that key doesn't exist, the dict is getting modified
to add a new, empty, entry for it. So even without the runtime
error, we'd be generating an incorrect result.
Fix this by using a local variable for modifying the pattern
and using the original key to retrieve the value on a match.
(Bitbake rev: 69d3b86449be23b07f794e302f6e18f3a2c46424)
Signed-off-by: Matt Madison <matt@madison.systems>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
(cherry picked from commit 07de375c3e57f17ab7b47569186f24ecd9896825)
Signed-off-by: Steve Sakoman <steve@sakoman.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
logger.debug was giving an integer value (2) as event message, causing
knotty to crash when running with debug enabled.
bitbake/lib/bb/ui/knotty.py", line 685, in main
event.msg = taskinfo['title'] + ': ' + event.msg
TypeError: can only concatenate str (not "int") to str
Same issue also happens in the original code that was taken from
oe-core (openembedded-core/meta/lib/crate.py honister) / meta-rust.
(Bitbake rev: c212b0f3b542efa19f15782421196b7f4b64b0b9)
Signed-off-by: Ricardo Salveti <ricardo@foundries.io>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Use bitwise operators to manipulate the received event mask in
_ProcessEvent.
Also minor clarification & clean up of the related comments.
(Bitbake rev: 2ab60c7be124d928d304ab1fb73f0dbff29964ae)
Signed-off-by: Peter Kjellerstedt <peter.kjellerstedt@axis.com>
Signed-off-by: Alexandre Belloni <alexandre.belloni@bootlin.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
In `runtaskhashes`, the keys contain the absolute paths to the recipe. When
working with shared sstate caches (where these absolute paths can be different)
we see that compare_sigfiles does not identifiy a changed hash of a dependent
task as "changed", but instead as "removed"&"added", preventing the function
from recursing and continuing the comparison.
By calling `clean_basepaths` before comparing the `runtaskhashes` dicts, we
avoid this.
(Bitbake rev: 7358378b90b68111779e6ae72948e5e7a3de00a9)
Signed-off-by: Adriaan Schmidt <adriaan.schmidt@siemens.com>
Signed-off-by: Alexandre Belloni <alexandre.belloni@bootlin.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Currently, if you use one of the functions from EXPORT_FUNCTIONS,
the meaning of cleandirs and fakeroot are lost. This leads to the function
changing in behaviour depending upon it's caller context. This isn't intended
so add mapping for the cleandirs and fakeroot flags too.
This does break devtool in OE-Core and there is a separate fix for that.
[YOCTO #8621]
(Bitbake rev: b074f4aff00923acc5bf6649d204d541a79fd2b6)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
GIT_SSH_COMMAND is more convinient to use if arguments have to be passed
and the user doesn't want to create a wrapper script around ssh.
(Bitbake rev: 5e746cb9d26ce87d6c9d52d9022122081a9811c5)
Signed-off-by: Pavel Zhukov <pavel.zhukov@huawei.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Change the owner information in the mirror tarballs generated using
BB_GENERATE_MIRROR_TARBALLS="1". This is an extension of commit
0178ab83, which used the original pokybuild:user information, but failed
to clean up the numerical user and group ids. Now set the more canonical
values of oe:oe and 0:0.
(Bitbake rev: 37437115d3fb1a9f5d8ed7356a0fc01a408e4f8c)
Signed-off-by: Olaf Mandel <o.mandel@menlosystems.com>
CC: Marek Vasut <marex@denx.de>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
If there are spaces in the URI filenames it can break the code.
We already solved this issue once somewhere else in the code so
use the same regex trick here as well.
We should ultimately refactor this code but at least fix the issue
for now.
(Bitbake rev: 57e2fc4d7f60afea4d4b2c84761324dd99e74a87)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
While the progress bar is good at conveying information about how much
of a task has executed, the elapsed time of the task is still very
much relevant to show.
(Bitbake rev: 41eeb4f34fb2306303f7688ec5e0ae965a573aa4)
Signed-off-by: Peter Kjellerstedt <peter.kjellerstedt@axis.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
This was only meant to be added for the handle_contains function in
a previous commit, fix it.
(Bitbake rev: 7399be398df39bc29e1b5eaac23b29cfae017abd)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The kernel inotify code can set more than one of the bits in the mask,
fsnotify_change() in linux/fsnotify.h is quite clear that IN_ATTRIB,
IN_MODIFY and IN_ACCESS can arrive together. We don't care about two
of these from a bitbake perspective but it probably explains why in
real world builds, we've seen:
pyinotify.ProcessEventError: Unknown mask 0x00000006
This module code assumes only one mask bit can be present. Since we
don't care about two of these events, just mask them out for now.
The "upstream" code is unmainained since 2015.
(Bitbake rev: 7fb93c2ce6dacd9b53fc3a227133a3493e6a6a1d)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The recent inotify changes can cause entire build trees to be monitored
which is suboptimal for performance.
Rather than trying increasingly convoluted tricks to try and handle add/removed
directories, rebuild the inotify watch when we reparse the configuration or
metadata.
(Bitbake rev: 3df322a200c28b45af1f2c92478c85eb7d20c38b)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The python gc can trigger whilst we're holding the event stream lock
and when cleaning up objects, they can trigger warnings. This translates
into a new event which would then need the lock and we can deadlock.
Disable gc whilst we hold that lock to avoid this unfortunate and
problematic situation.
(Bitbake rev: 96a6303949cefd469bcf5ed250ff512271354357)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Adding bb.utils.filter('WARN_QA', 'patch-fuzz', d) when WARN_QA is in
BB_BASEHASH_IGNORE_VARS or in vardepsexclude should not add a dependency
on WARN_QA.
Fix it and add some tests.
(Bitbake rev: 6aecc2fe51a52020f6f13be08449e18d42e7a6b5)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
We're still seeing issues with unclosed asyncio event loops. At the
init site, make sure any existing one is closed first to try and avoid
this.
(Bitbake rev: 78dee3c03c75a27531fcff26f9298fce2519bdde)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
In some cases we'd never have setup a siggen so don't error in that case.
(Bitbake rev: bbaaf2cf7b5a9339d3790610e622020c19d52f5a)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Ensure any exiting hash server connection is terminated before we start
a new bitbake session. This avoids errors seen with memory resident bitbake
when the asyncio event loop isn't closed correctly.
(Bitbake rev: 42ff9de77f24e2a0bec48a14b64c4b538e00b4af)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
When memory resident bitbake is active and we re-parse, the old module
configuration is present which can lead to strange errors. Reset this
when reparsing so the state is consistent. This fixes memory resident
bitbake errors.
(Bitbake rev: 951942c3c284ec2c62e730e145688033190af9b2)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The previous fix for inotify wasn't quite correct as we need to modify
bbseen before calling add_filewatch(). We also need to ensure the parse
mtime cache is cleared when directories are added/removed. There was also
a typo in the original fix and the wrong watcher was being changed. Fix
the various issues which improves memory resident bitbake testing results.
(Bitbake rev: 66cadd6be58bce5f7a56556cf92efd8159fb0b0e)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
- Replace hyphens by em dashes when necessary
See https://www.grammarly.com/blog/hyphens-and-dashes/
- No uppercase after em dashes
- No uppercase after colons if what follows is not a
complete sentence.
- Fix spacing before colons ":"
- Replace em-dashes with colons for consistency in a section
(Bitbake rev: 72230d6a9976b3bfca1f1e6fb09736fec195e2fe)
(Bitbake rev: f1c4ac816e927f490fb9852c12aa408e8c9403b1)
Signed-off-by: Michael Opdenacker <michael.opdenacker@bootlin.com>
Signed-off-by: Alexandre Belloni <alexandre.belloni@bootlin.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The "force" option to parser shutdown was often the cause of lockups and
there is no good reason we should have two different behaviours.
Change and unify the codepaths to always:
* Wait for longer for a controlled shutdown of a process (0.5s). Usually
it will be much faster if it has finished so the delay doesn't really matter.
* Send processes a SIGINT
* Failing that, send a SIGTERM
* Call .close() if available to release zombies
This means we no longer need the "force" parameter to the function so it is removed.
(Bitbake rev: de88c62ef873e9fce78ba162f5311d846de96f2b)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
If a parser process is terminated while holding a write lock, then it
will lead to a deadlock (see
https://docs.python.org/3/library/multiprocessing.html#multiprocessing.Process.terminate).
With SIGTERM, we don't want to terminate holding the lock. We also don't
want a SIGINT to cause a partial write to the event stream.
I tried using signal masks to avoid this but it doesn't work, see
https://bugs.python.org/issue47139
Instead, add a signal handler and catch the calls around the critical section.
We also need a thread lock to ensure other threads in the same process don't
handle the signal until all the threads are not in the lock.
(Bitbake rev: a40efaa5556a188dfe46c8d060adde37dc400dcd)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Not sure why this is so convoluted but we should simplify it!
(Bitbake rev: 6195343c46ba9d2685fc2d42366922f88ff3f369)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
If an exception occurs in the parsing process, ensure the cleanup
is called via all codepaths using a try/finally.
(Bitbake rev: e1ba2a69f1fc02f01a851bce20b1badf0b991f03)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
When parsing, the parser isn't servicing the main loop so a Ctrl+C in the
UI won't be seen on the cooker/server side. Fix this by returning when queue
timeouts occur. This helps where there is a hung or slow parsing thread.
(Bitbake rev: a2cde38311a51112dca5e7bb4e7feaf4e6a281b3)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
We shouldn't be generating exception inside a generator. Rearrange the
code to improve the handling of this. Also fix the misconverted code
from when multiconfig was added and pass the exception as "result".
(Bitbake rev: ae89e23696de2f27c00ae00922933395171de5d5)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
This parameter is now required by the git fetcher module
(Bitbake rev: d61b349581c219e7f9d50f683177184fa473cb83)
Signed-off-by: Michael Opdenacker <michael.opdenacker@bootlin.com>
Signed-off-by: Alexandre Belloni <alexandre.belloni@bootlin.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
When using BB_GENERATE_MIRROR_TARBALLS="1" to generate mirror tarballs
of git repositories, they leaked local information: username, group and
time of the last fetch. Remove all these by setting fixed information:
* uname = pokybuild
* gname = users
* mtime = committer time of newest commit in repo
The username and group value were taken from the archives available on
the downloads.yoctoproject.org mirror. The modification time is chosen
so it still retains some relationship to the contents of the archive.
(Bitbake rev: 0178ab83e6312e97e528aa8c5e12105f5165d896)
Signed-off-by: Olaf Mandel <o.mandel@menlosystems.com>
Signed-off-by: Alexandre Belloni <alexandre.belloni@bootlin.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Thanks to great debugging from pavel@zhukoff.net we had a simpler reproducer
for the corruption see in oe-selftest when using BB_SERVER_TIMEOUT=60, i.e.
with bitbake in memory resident mode. This was effectively:
oe-selftest -r devtool.DevtoolUpgradeTests.test_devtool_upgrade devtool.DevtoolUpgradeTests.test_devtool_upgrade_git -j 1 -K
The issue is that if directories are removed (such as workspace), if
they are added again, we don't have the watches in place any more. This
patch adds some slightly paranoid checks to ensure we do the correct things
for directory additions and removals (we track directories, not files
specifically to avoid running out of watches).
[YOCTO #14023]
(Bitbake rev: 2c414f659d793d732041614caedd773959eb4f27)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
This debug is useful but the cooker shutdown or post_serve() may have cleanup
left so run after those.
(Bitbake rev: 1463fc0448d1a6a7265806a4a8b165b610dfb43f)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
When we have a client connection, we should close that connection when reset()
is called on the siggen. Add the missing function.
(Bitbake rev: 770b4ea81b6126b0830e51649c40f7a46c64132a)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
We have shutdown functions within the async client code but the siggen
doesn't currently call them. We notice on python 3.10 (such as on fedora35)
that at exit, there is a stray asyncio process left behind. Usually this
doesn't cause problems but it could potentially be a cause of a hang.
For general cleanliness and completness, add in hooks to call the exit handler.
(Bitbake rev: 9ee3fb95330003878fbd64b3ce8897aad96fcd0f)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
There are some commands where we want to see the events returned so allow
the caller to request this. This also allows us to fix an infamous bug in
the tinfoil testsuite in OE-Core.
(Bitbake rev: 0e8421c41d97d5d50a553d70c8f775d521f1a199)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
If the path to bitbake.lock is in a deep directory, bitbake will hang. The
reason was that the max file length limiting code (to 255 chars) was including
the directory name and it should only act on the filename within the directory.
Fix it to just use the base filename.
[YOCTO #14766]
(Bitbake rev: 89d70e7b71eecfe06592202f326e566c579ba01d)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
I'd never spotted this until it was pointed out but the task isn't dependent,
it is a dependency. Fix this confusing reference.
(Bitbake rev: 93395559c9dda734a3dec9ffd9bb2156a71a2c17)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The keep alive timeout is excessively long at 83 minutes (5000 seconds),
reduce this to 10 minutes: this should be long enough that it rarely
triggers in normal builds, but when it does it has useful information.
(Bitbake rev: 2e47346b95b09d7ab8f0603e2d62cfb549dc1f5c)
(Bitbake rev: dcf52157d3635925491783be656c6b76d1efe1a4)
Signed-off-by: Ross Burton <ross.burton@arm.com>
Signed-off-by: Alexandre Belloni <alexandre.belloni@bootlin.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
In interactive bitbake sessions it is obvious what tasks are running
when one of them hangs or otherwise takes a long time. However, in
non-interactive sessions (such as automated builds) bitbake just prints
a message saying that it is "still alive" with no clues as to what tasks
are active still.
By simply listing the active tasks when printing the keep alive message,
we don't need to parse the bitbake log to identify which of the tasks
is still active and has presumably hung.
(Bitbake rev: f9f57fb7d2c8a13df1eb9d5b9766f15e229dcf97)
(Bitbake rev: 30f6c3f175617beea8e8bb75dcf255611e3fc2fd)
Signed-off-by: Ross Burton <ross.burton@arm.com>
Signed-off-by: Alexandre Belloni <alexandre.belloni@bootlin.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Adds a Dockerfile for building the PR service in a container.
Based on the hash equivalence server container [hashserv].
Use the following environment variables to configure the PR service
container:
- DBFILE: database filename. Default:
/var/lib/bbprserv/prserv.sqlite3.
- LOGFILE: log filename. Default: /var/lib/bbprserv/prserv.log.
- LOGLEVEL: logging level. Default: DEBUG.
- HOST: ip address to bind. Default: 0.0.0.0.
- PORT: port number. Default: 8585.
- DBMODE: database mode. Default: Empty (RW).
Note: DBMODE in RO: "--read-only".
[hashserv]: contrib/hashserv/Dockerfile
(Bitbake rev: a5497428b539e8598263924f63a9df8fe1ea70f9)
(Bitbake rev: 1d05abd92da56e284fcd904cf32bd12485903f10)
Signed-off-by: Daniel Gomez <daniel@qtec.com>
Signed-off-by: Alexandre Belloni <alexandre.belloni@bootlin.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Force a sync point for end-of build event handler force
the build's outcome status commit, to resolve a race
condition with the build completion takedown.
[YOCTO #14765]
(Bitbake rev: f76c13536e19d2401365c0fc240ef8dbcb3ee7e8)
Signed-off-by: David Reyna <David.Reyna@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Add a polling check on tracebacks in a build's log. This
can for example indicate that bitbake crashed, which would
stop the event stream that Toaster normally uses to detect
build errors.
[YOCTO #14085]
(Bitbake rev: 32b1c0b3477e359d2e2a61a23a294e317e417f95)
Signed-off-by: David Reyna <David.Reyna@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Creating the new log instances triggers a lot of python logging overhead
in a commonly called function (about 600,000 for parsing OE-Core).
We only need the log functionality if we're parsing, not if we just hit
from the cache. Therefore defer the log setup overhead until we know it
is a cache miss.
Whilst this complicates the code slightly, the performance gain is worth
it as for parsing OE-Core we drop 60 million funciton calls (from 225
overall).
(Bitbake rev: ac868167ad854f9bb32dcb2e63528870547805a7)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
We can call getVarFlags() instead of the multiple getVarFlag calls
which is a little more efficient. This reduces the number of overall
function calls so is good but probably isn't much faster (or slower).
(Bitbake rev: 505a4dd34e822bdf902d9b348dbcdf5b2c94e784)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The code tries to expand missing entities when they're encountered in
python expressions. Looking at traces, these are often things which
would not be in the data store such as "bb".
Optimise to skip the data store queries for things we know will never
be there. The expansion cache usually covers these but skipping
entirely cuts a few million function calls parsing OE-Core.
(Bitbake rev: 1ae712635a2eef3ecbf541e1f7cc2eeb2f3799c9)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
With the upcoming LTS, it is time we changed the bitbake version so
move to 2.0.
(Bitbake rev: 9a13bf8e20b1841ec99281d45be4c4fc1118438c)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
By alphabetical order, to get the same order as in the HTML
output, sorted thanks to the ":sorted:" directive.
(Bitbake rev: 8b97af89316a4407dd07649e8aadd495edd86ee4)
Signed-off-by: Michael Opdenacker <michael.opdenacker@bootlin.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
This template is not referenced anywhere and therefore not used.
It also refers to the obsolete "build mode" in the old docs.
(Bitbake rev: f89a35155e3b8d12ac609a165c63ed206751b8da)
Signed-off-by: Tim Orling <tim.orling@konsulko.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
connection.features.autocommits_when_autocommit_is_off
was deprecated in 3.0 and is no longer present in 3.2
(Bitbake rev: cc0f526fb4298349d0eea44c8f35d3dd226cc8d6)
Signed-off-by: Tim Orling <tim.orling@konsulko.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Follow the default guidelines from Django 3.2 setting DEFAULT_AUTO_FIELD
to django.db.models.BigAutoField which leads to these migrations.
(Bitbake rev: ccfdcf5ed6b5d2024c268bace30e53753d1f4da4)
Signed-off-by: Tim Orling <tim.orling@konsulko.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
New in Django 3.2.
Silence warnings by adding:
DEFAULT_AUTO_FIELD = 'django.db.models.BigAutoField'
in toastermain/settings.py
(Bitbake rev: cbc5a8e339a8b5eb4cdd89ad898af432f05cadfb)
Signed-off-by: Tim Orling <tim.orling@konsulko.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Django 3.2 no longer has django.utils.six module.
(Bitbake rev: ae03c1f05e37f70e72a2ac08f7c718b8fc4fc36e)
Signed-off-by: Tim Orling <tim.orling@konsulko.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Update docs links from www.yoctoproject.org to the docs.yoctoproject.org
equivalents.
(Bitbake rev: 74194962eb294c9bd1192ddd7935d57258712ee2)
Signed-off-by: Tim Orling <tim.orling@konsulko.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Django 2.2.x goes EOL April 2022.
Switch the to current LTS Django 3.2.x supported until April 2024.
(Bitbake rev: 23a15c5703d5ef4190921f9bb0273e43b1de489c)
Signed-off-by: Tim Orling <tim.orling@konsulko.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Fix a bad patch merge and move the file to the correct path.
(Bitbake rev: d173f45a88cc9460734e7cc0b87952c0ea0ff270)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Add script to safely generate the fixture files from a table
[YOCTO #14759]
(Bitbake rev: ec5e9fff8f71ba81bf5e9207abcd3011d63d86d9)
Signed-off-by: David Reyna <David.Reyna@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
[YOCTO #14757]
(Bitbake rev: 49308d1db4f7af81e31db21dc8954947de5976c7)
Signed-off-by: David Reyna <David.Reyna@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
After the override syntax change, toaster isn't working correctly. This
is because it uses IMAGE_INSTALL_append instead of IMAGE_INSTALL:append.
This tweaks the code accordingly to fix this.
I have a suspicion that exiting toaster databases may struggle with
the change and there are some migration steps missing for the whole
overrides syntax change step.
(Bitbake rev: befc93c3e03a9b1a2b630cd5badcda756fe6cd28)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
bitbake -e is not showing variable history correctly for all variables.
The issue was triggered by the change to knotty to default to enabling
variable tracking in the base datastore but that exposed another issue.
The real problem is that calling reset() inside cooker reverts to the
data tracking enabled by the UI for the base datastore, then turns
off tracking. In the case of the environment printing code, it needs
it to be left on.
Tweak the code to ensure data store tracking really is enabled.
The code here is clearly a bit of a mess but this at least fixes a
clear regression until more invasive improvements can be made.
(Bitbake rev: 9596bffc9903f023d1cc97ce531ddb1cf010c4d7)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* Use max() to clamp progress to >= 0.
* Be consistent when evaluating self.quiet (treat it as a boolean).
(Bitbake rev: 160f71372ff93894d9314619e9d3b547c1f3cda3)
Signed-off-by: Peter Kjellerstedt <peter.kjellerstedt@axis.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
In commit 8055ec36 (knotty: Improve setscene task display) the setscene
tasks got their own line in the task output. However, the progress bar
code does not handle newlines in its widgets, so the length of the
setscene line was included when calculating how much space is available
for the progress bar of the running tasks, making it much too short.
Instead of trying to teach the progress bar code to handle newlines,
separate the output of the setscene tasks from the progress bar for the
real tasks.
(Bitbake rev: a41f7792f17acdba8c7ea83b79e413ae6a49da68)
Signed-off-by: Peter Kjellerstedt <peter.kjellerstedt@axis.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Use pluralise() to correct the grammar, and drop the colon at the end if
runnning in quiet mode.
(Bitbake rev: 57396289f935892390c11afe95f3eede28fd80e7)
Signed-off-by: Peter Kjellerstedt <peter.kjellerstedt@axis.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The "Parsing halted due to errors" is shown when user has old-style
variables requiring renaming. Make it clear that it isn't an additional
error, but a result of earlier errors.
(Bitbake rev: e81b6ae61dd8b5d0f77146263d79dfe66c66b060)
Signed-off-by: Marta Rybczynska <marta.rybczynska@huawei.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
This avoids backtraces when starting toaster or using bitbake in
remote mode.
(Bitbake rev: bf723f2cb5d288ca730e4f029110b36380420a01)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Add links corresponding to recent releases
Align the title of sections with yocto-docs' releases.rst
(Bitbake rev: 5700317975e115aa31186c8cd3eddca16ec387fb)
Signed-off-by: Michael Opdenacker <michael.opdenacker@bootlin.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
(Bitbake rev: ad2307823a96e91873d804aa9d8f11c9ff39bec0)
Signed-off-by: Michael Opdenacker <michael.opdenacker@bootlin.com>
Reported-by: Peter Kjellerstedt <peter.kjellerstedt@axis.com>
Reported-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Now defunct.
(Bitbake rev: 2c7bff686c1f5eb80c89cb7f9f12287e8bb5770f)
Signed-off-by: Michael Opdenacker <michael.opdenacker@bootlin.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
There is general agreement the current display of the total setscene and
executed full tasks is confusing. Fixing it is tricky to be clear to new
users whilst not wasting screen real estate for experienced users and not
compromisng features like the progress bar.
As a compromise, move the setscene total to the previous line making
the status display a two line summary for interactive terminals. This
makes it clear what two of the numbers represent (setscene task) whilst
separating this out from the other items.
Also tweak the non-interactive output to use a comma as a separator,
the use of the forward slash was a poor choice.
(Bitbake rev: 8055ec360507e6a678ee5c4018ec1ab7f5a9cce5)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Currently the footer code doesn't quite handle multiline items correct.
Fix this to do so.
(Bitbake rev: 8eeccf73185d986c6abd6426b1d1e72da3a982df)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
This implements support for sstate mirrors using ssh as transport
protocol.
(Bitbake rev: 0a3b5b3de7bcb1c5c3748cba42d394cc484e966b)
Signed-off-by: Daniel Wagenknecht <dwagenknecht@emlix.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Fix absolute paths and paths containing the ":" character. Both is
necessary for supporting sstate mirrors via ssh (not implemented yet).
(Bitbake rev: df5505a1ba15524c3a185360d687854300aef342)
Signed-off-by: Daniel Wagenknecht <dwagenknecht@emlix.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Support URLs like ssh://HOST/PATH. They were previously not recognized
due to a missing @ sign.
(Bitbake rev: a2aa18bd27dac8902e52b466cb7118f71367d3dc)
Signed-off-by: Daniel Wagenknecht <dwagenknecht@emlix.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
hashlib.md5() is not permitted on a FIPS enabled host system. This is due
to md5 not being an approved hash algorithm.
Instead use:
hashlib.new('MD5', usedforsecurity=False)
This is allowed, as it's clear the hash is used for a non-security purpose.
Note: utils.py version should never be used to verify file integrity, but
instead be used to identify if the file may have changed. sha256 should be
used for integrity purposes.
(Bitbake rev: af866dd077867cba0129757bfcc689551445e9d7)
Signed-off-by: Mark Hatle <mark.hatle@xilinx.com>
Signed-off-by: Mark Hatle <mark.hatle@kernel.crashing.org>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
File "scripts/lib/checklayer/__init__.py", line 49, in _get_layer_collections
ldata.expandVarref('LAYERDIR')
File "build/bitbake/lib/bb/data_smart.py", line 1007, in expandVarref
if referrervalue and ref in referrervalue:
TypeError: argument of type 'bool' is not iterable
We inject True values as an internal sentinel to the datastore, fix
this codepath to handle it.
(Bitbake rev: 3b88562d87ac94725c1a683c859c2a6a3287d173)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Until now, if a renamed variable was found, bitbake exited
immediately if it was in a class, but continued after an error
message if the variable was in a recipe. This was caused by
cookerdata.py CookerDataBuilder::parseBaseConfiguration checking
a different DataSmart instance than the variable was set in.
To solve the issue, add a special variable and set it when we
find a renamed variable. Check for it in ast.py and bail out
if needed.
(Bitbake rev: d12400600e30549c88dc9e7883dc3d63b1dc1117)
Signed-off-by: Marta Rybczynska <marta.rybczynska@huawei.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
In line with the inclusive language migration defined at:
https://wiki.yoctoproject.org/wiki/Inclusive_language
replace the remaining usage of "abort" in documentation, error
messages, and comments with halt/fail/exit as appropriate.
A couple of external Javascript API calls in Toaster remain, as
they cannot currently be changed.
(Bitbake rev: bc27762bf3ffb4a20b58eace5302438c4a526626)
Signed-off-by: Scott Murray <scott.murray@konsulko.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
In line with the inclusive language migration defined at:
https://wiki.yoctoproject.org/wiki/Inclusive_language
replace the "ABORT" action in BB_DISKMON_DIRS entries with "HALT".
In order to ease migration, code has been added to warn users to
update their configurations if the old name is used, as opposed to
to throwing an error.
(Bitbake rev: 11dc65dc077398ff9818060769c99c0090291186)
Signed-off-by: Scott Murray <scott.murray@konsulko.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
In line with the inclusive language migration defined at:
https://wiki.yoctoproject.org/wiki/Inclusive_language
replace the use of "abort" with "halt" in code related to handling
task failure.
(Bitbake rev: 831fb7f2329a3cd95b71e9c85d7d7f0d717f947f)
Signed-off-by: Scott Murray <scott.murray@konsulko.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
In line with the inclusive language migration defined at:
https://wiki.yoctoproject.org/wiki/Inclusive_language
replace the remaining non-backwards-compatibility related usage
of "blacklist"/"whitelist" with "allowed"/"disallowed" in a few
places as appropriate.
(Bitbake rev: f579fb8c23d2919d25641fa4234f8a1e9c06a922)
Signed-off-by: Scott Murray <scott.murray@konsulko.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Add support for custom strings in BB_RENAMED_VARIABLES and use this to show
that BB_STAMP_WHITELIST and BB_STAMP_POLICY are no longer supported.
(Bitbake rev: 0914011f7647571ab125bbddcd7d68e3da47226a)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
In line with the inclusive language migration defined at:
https://wiki.yoctoproject.org/wiki/Inclusive_language
rename:
MULTI_PROVIDER_WHITELIST -> BB_MULTI_PROVIDER_ALLOWED
(Bitbake rev: a09546b725fda13c0279638c7c904110da7bf6cd)
(Bitbake rev: d035435c1a4951a45481867cf932faa4a6f8f936)
Signed-off-by: Scott Murray <scott.murray@konsulko.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
In line with the inclusive language migration defined at:
https://wiki.yoctoproject.org/wiki/Inclusive_language
rename:
BB_SETSCENE_ENFORCE_WHITELIST -> BB_SETSCENE_ENFORCE_IGNORE_TASKS
(Bitbake rev: 2e243ac06581c4de8c6e697dfba460ca017d067c)
(Bitbake rev: f8f7b80a0df4646247e58238a52a7d85a37116d4)
Signed-off-by: Scott Murray <scott.murray@konsulko.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
In line with the inclusive language migration defined at:
https://wiki.yoctoproject.org/wiki/Inclusive_language
rename:
BB_HASHCONFIG_WHITELIST -> BB_HASHCONFIG_IGNORE_VARS
(Bitbake rev: f344246be73d626c215f867718e45fd6cddc2aaf)
(Bitbake rev: 371deb3fe8510aadf4455810d7c5243d374e6532)
Signed-off-by: Scott Murray <scott.murray@konsulko.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
In line with the inclusive language migration defined at:
https://wiki.yoctoproject.org/wiki/Inclusive_language
rename:
BB_ENV_WHITELIST -> BB_ENV_PASSTHROUGH
BB_ENV_EXTRAWHITE -> BB_ENV_PASSTHROUGH_ADDITIONS
(Bitbake rev: fe60627839d4280cf0117ed1afbfccdff1181b6a)
(Bitbake rev: 87104b6a167188921da157c7dba45938849fb22a)
Signed-off-by: Scott Murray <scott.murray@konsulko.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
In line with the inclusive language migration defined at:
https://wiki.yoctoproject.org/wiki/Inclusive_language
rename:
BB_HASHBASE_WHITELIST -> BB_BASEHASH_IGNORE_VARS
BB_HASHTASK_WHITELIST -> BB_TASKHASH_IGNORE_TASKS
the derived code variables basewhitelist and taskwhitelist have been
renamed to basehash_ignore_vars and taskhash_ignore_tasks,
respectively.
[RP: Added compatibility mapping code and tweaked naming]
(Bitbake rev: efaafc9ec2e8c0475e3fb27e877a1c0a5532a0e5)
Signed-off-by: Scott Murray <scott.murray@konsulko.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
This change adds support for improving the user experience when variables
are renamed. This isn't as simple as it might first appear since some
bitbake variables are used through the environment before the datastore
exists, some are bitbake variables which we know about straight away
and some are metadata defined which we don't know about until later.
This patch adds support for handling these different cases, allowing a list
of bitbake renamed variables to be defined in bitbake itself and allows this
to be extended through the metadata using BB_RENAMED_VARIABLES.
In order to give the best feedback to the user, we default to turning on
variable history tracking in the base data store from knotty, which allows
filename and line number information to be shown.
(Bitbake rev: bd50a5d5e4b4fa90844464396887ebdff0d4e5f7)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
If a python function exec_func() calls a shell task, the logging wasn't working
correctly in all cases since the exception was turned into a BBHandledException()
and the logfile piece was lost which is handled at the top task level.
The easiest way to avoid this is to allow the ExecutionError exceptions to
be raised to a higher level, we don't need the traceback for them.
(Bitbake rev: 7cae11f558f9ff5fd05ef23b789aaef92fb5a327)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The changing of the environment inside the wget fetcher can race if
threading is used, such as with sstate in OE-Core. Abstract the function
so the environment can be correct before the function is called, removing
the race since the enviroment is then no longer changed.
(Bitbake rev: c73bb6023c73f003a160bb02aa4da1b580b86c23)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
BB_ORIGENV is used as a fallback environment block. It is repeatedly
accessed inside the loop. Since it is a loop invariant, move it out of
the loop.
(Bitbake rev: 346a1a6c76d40458d7b4c116147ec4d371bee74a)
Signed-off-by: Zygmunt Krynicki <zygmunt.krynicki@huawei.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
File "/home/pokybuild/yocto-worker/qemuarm/build/build/tmp/work/qemuarm-poky-linux-gnueabi/core-image-sato/1.0-r0/testsdkext/layers/build/bitbake/lib/bb/fetch2/wget.py", line 328, in checkstatus
with bb.utils.environment(**newenv):
File "/home/pokybuild/yocto-worker/qemuarm/build/build/tmp/work/qemuarm-poky-linux-gnueabi/core-image-sato/1.0-r0/testsdkext/buildtools/sysroots/x86_64-pokysdk-linux/usr/lib/python3.10/contextlib.py", line 142, in __exit__
next(self.gen)
File "/home/pokybuild/yocto-worker/qemuarm/build/build/tmp/work/qemuarm-poky-linux-gnueabi/core-image-sato/1.0-r0/testsdkext/layers/build/bitbake/lib/bb/utils.py", line 1737, in environment
del os.environ[var]
File "/home/pokybuild/yocto-worker/qemuarm/build/build/tmp/work/qemuarm-poky-linux-gnueabi/core-image-sato/1.0-r0/testsdkext/buildtools/sysroots/x86_64-pokysdk-linux/usr/lib/python3.10/os.py", line 695, in __delitem__
raise KeyError(key) from None
It is possible the wrapped function might change the environment which can lead
to KeyError issues like this. The key may no longer be in the original environment
so handle that case correctly and avoid the tracebacks.
(Bitbake rev: ae4ce4c21998bb2a12a1e1f0b8e0af8d218b87e2)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
There are some cases where the metadata needs to check if a variable
has any overrides set, even if they are currently inactive. That code
currently pokes into datastore internals. Add API instead to replace
and avoid that.
(Bitbake rev: c885e5542dcf760b8fc5881e385abb4a10020874)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Loggers created with a logging configuration file automatically get the
once filter applied to ensure that warnonce()/erroronce() only appear a
single time. The same filter needs to also be added to
bb.msg.logger_create() to ensure that manually created loggers have the
same behavior
(Bitbake rev: bd93f7baff9d56de37a1af474189e0888b82d475)
Signed-off-by: Joshua Watt <JPEWhacker@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
This adds a log level and logging function call to use it where the
warning or error will only be displayed once, regardless of how many
times the message is logged.
This has to be done either in the cooker or on the UI side. I've opted
for the UI side since display control is really a UI issue but it uses
a common library filter function to enable it which can be reused
elsewhere.
The knotty message displayed as the build summary is tweaked to
make sense when the numbers won't match since it will still count
the number of times it was logged and this is probably helpful
for debugging in some cases so I've deliberately left it that way.
(Bitbake rev: 7bd40e3003a043e3cb7efc276681054b563b5e7b)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Signed-off-by: Joshua Watt <JPEWhacker@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
When a recipe raises a BBHandledException, it means the error was already
shown to the user. Adding an additional one here isn't helpful. What is
helpful is to mention that parsing was halted.
Tweak the code to do this with improves the messages the user sees
and helps understand what happened.
(Bitbake rev: fdf6ebc8b603fcfd3ed7c64baf486a4adabd25be)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The overrides warning message is meant to show filename and line
numbers but the variable names are incorrect and this wasn't working.
Fix it.
(Bitbake rev: 551c1cb20fc9b9d0dab5d830182c2bf626e72845)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Export ClientError and ServerError classes from bb.asyncrpc.serv as
required by lib/hashserv/server.py module.
Error:
Client ('192.168.2.184', 51142) connected
Error from client: module 'bb.asyncrpc' has no attribute 'ClientError'
Traceback (most recent call last):
File "/opt/bbhashserv/lib/bb/asyncrpc/serv.py", line 192, in handle_client
await client.process_requests()
File "/opt/bbhashserv/lib/hashserv/server.py", line 200, in process_requests
await super().process_requests()
File "/opt/bbhashserv/lib/bb/asyncrpc/serv.py", line 72, in process_requests
await self.dispatch_message(d)
File "/opt/bbhashserv/lib/hashserv/server.py", line 217, in dispatch_message
raise bb.asyncrpc.ClientError("Unrecognized command %r" % msg)
AttributeError: module 'bb.asyncrpc' has no attribute 'ClientError'
(Bitbake rev: 8aca1d341500af8bbe788801c2df3da9bbfacaed)
Signed-off-by: Daniel Gomez <daniel@qtec.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
When building the Hash Equivalence server Dockerfile, some
dependencies are missing in order to run the hash server properly:
Traceback errors:
Traceback (most recent call last):
File "/opt/bbhashserv/bin/bitbake-hashserv", line 61, in <module>
ret = main()
File "/opt/bbhashserv/bin/bitbake-hashserv", line 54, in main
server = hashserv.create_server(args.bind, args.database, upstream=args.upstream, read_only=args.read_only)
File "/opt/bbhashserv/lib/hashserv/__init__.py", line 120, in create_server
from . import server
File "/opt/bbhashserv/lib/hashserv/server.py", line 14, in <module>
import bb.asyncrpc
ModuleNotFoundError: No module named 'bb'
Traceback (most recent call last):
File "/opt/bbhashserv/bin/bitbake-hashserv", line 61, in <module>
ret = main()
File "/opt/bbhashserv/bin/bitbake-hashserv", line 54, in main
server = hashserv.create_server(args.bind, args.database, upstream=args.upstream, read_only=args.read_only)
File "/opt/bbhashserv/lib/hashserv/__init__.py", line 120, in create_server
from . import server
File "/opt/bbhashserv/lib/hashserv/server.py", line 14, in <module>
import bb.asyncrpc
File "/opt/bbhashserv/lib/bb/__init__.py", line 128, in <module>
from bb import fetch2 as fetch
File "/opt/bbhashserv/lib/bb/fetch2/__init__.py", line 1930, in <module>
from . import git
File "/opt/bbhashserv/lib/bb/fetch2/git.py", line 70, in <module>
import bb.progress
File "/opt/bbhashserv/lib/bb/progress.py", line 14, in <module>
import bb.build
File "/opt/bbhashserv/lib/bb/build.py", line 27, in <module>
from bb import data, event, utils
File "/opt/bbhashserv/lib/bb/data.py", line 36, in <module>
from bb import data_smart
File "/opt/bbhashserv/lib/bb/data_smart.py", line 23, in <module>
import bb, bb.codeparser
File "/opt/bbhashserv/lib/bb/codeparser.py", line 26, in <module>
import codegen
ModuleNotFoundError: No module named 'codegen'
Traceback (most recent call last):
File "/opt/bbhashserv/bin/bitbake-hashserv", line 61, in <module>
ret = main()
File "/opt/bbhashserv/bin/bitbake-hashserv", line 54, in main
server = hashserv.create_server(args.bind, args.database, upstream=args.upstream, read_only=args.read_only)
File "/opt/bbhashserv/lib/hashserv/__init__.py", line 120, in create_server
from . import server
File "/opt/bbhashserv/lib/hashserv/server.py", line 14, in <module>
import bb.asyncrpc
File "/opt/bbhashserv/lib/bb/__init__.py", line 128, in <module>
from bb import fetch2 as fetch
File "/opt/bbhashserv/lib/bb/fetch2/__init__.py", line 1930, in <module>
from . import git
File "/opt/bbhashserv/lib/bb/fetch2/git.py", line 70, in <module>
import bb.progress
File "/opt/bbhashserv/lib/bb/progress.py", line 14, in <module>
import bb.build
File "/opt/bbhashserv/lib/bb/build.py", line 27, in <module>
from bb import data, event, utils
File "/opt/bbhashserv/lib/bb/data.py", line 36, in <module>
from bb import data_smart
File "/opt/bbhashserv/lib/bb/data_smart.py", line 23, in <module>
import bb, bb.codeparser
File "/opt/bbhashserv/lib/bb/codeparser.py", line 32, in <module>
from bb.pysh import pyshyacc, pyshlex
File "/opt/bbhashserv/lib/bb/pysh/pyshyacc.py", line 13, in <module>
import bb.pysh.pyshlex as pyshlex
File "/opt/bbhashserv/lib/bb/pysh/pyshlex.py", line 17, in <module>
from ply import lex
ModuleNotFoundError: No module named 'ply'
Traceback (most recent call last):
File "/opt/bbhashserv/bin/bitbake-hashserv", line 61, in <module>
ret = main()
File "/opt/bbhashserv/bin/bitbake-hashserv", line 54, in main
server = hashserv.create_server(args.bind, args.database, upstream=args.upstream, read_only=args.read_only)
File "/opt/bbhashserv/lib/hashserv/__init__.py", line 120, in create_server
from . import server
File "/opt/bbhashserv/lib/hashserv/server.py", line 14, in <module>
import bb.asyncrpc
File "/opt/bbhashserv/lib/bb/__init__.py", line 128, in <module>
from bb import fetch2 as fetch
File "/opt/bbhashserv/lib/bb/fetch2/__init__.py", line 1935, in <module>
from . import wget
File "/opt/bbhashserv/lib/bb/fetch2/wget.py", line 30, in <module>
from bs4 import BeautifulSoup
ModuleNotFoundError: No module named 'bs4'
(Bitbake rev: 0bd637acfcba5a44230c291889d2a5ff571cb8c6)
Signed-off-by: Daniel Gomez <daniel@qtec.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Where a git url uses a tag instead of a full source revision, breakage
can currently occur in builds. Issues include:
* the revision being looked up in multiple tasks (fetch and unpack)
* the risk a different revision may be obtained in those tasks
* that some tasks may not be allowed to access the network
* that a revision may not be consistent throughout a given build
* rerunning a specific task may given inconsistent results
To fix this, stop the workers from cleaning out the source revision store. This
should only be done in the cooker itself (based on current policy).
Also, where the code "sees" an upstream access, mark the recipe as not to be
cached. The reparse re-triggers the upstream lookup by the server.
Add a test to ensure that if get_srcrev isn't called, the user is told they're
using a configuration that is known to break.
(Bitbake rev: 4b5eed1626709ef3dc06b32fd55d40a2a6edd179)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
We started to require the branch parameter for git urls. This updates
the tests to match and quieten some noise.
(Bitbake rev: e5144887f2b6ae266f7b52151005619480e80ec6)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Previously npm files that specify downloadfilename= in the SRC_URI
would be downloaded to the root of ${DL_DIR} rather than in the
${DL_DIR}/npm2 directory where all other npm files are downloaded.
This should make it simpler when setting up and configuring a
premirror with the downloaded npm packages.
(Bitbake rev: 73fa376d6502ab6f2cccfb25a1193d9b1c3c3bc8)
Signed-off-by: Peter Kjellerstedt <peter.kjellerstedt@axis.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The solution implementated in commit 96c30007 (fetch2: fix
downloadfilename issue with premirror) missed two corner cases. The
first is if the basename of the original URI also appears somewhere
else in the replacement URI, in which case it would also be replaced.
The second is if the basename of the original URI partially matches
the basename of the replacement URI, in which case the inital part of
the basename from the replacement URI would be left behind.
The second case caused test_npm_premirrors_with_specified_filename to
fail.
The solution is to prefix the basename with a slash when matching to
avoid partial matches, and only replace the basename at the end of the
URI.
This also adds two test cases that test for these problems. Before
this they would give the following errors:
- ['file:///mirror/example/1.0.0/some-example-1.0.0.tgz;downloadfilename=some-example-1.0.0.tgz']
+ ['file:///mirror/some-example-1.0.0.tgz/1.0.0/some-example-1.0.0.tgz;downloadfilename=some-example-1.0.0.tgz']
? +++++ ++++++++++
- ['file:///mirror/some-example-1.0.0.tgz;downloadfilename=some-example-1.0.0.tgz']
+ ['file:///mirror/some-some-example-1.0.0.tgz;downloadfilename=some-example-1.0.0.tgz']
? +++++
(Bitbake rev: 5924c6f007519cd8ea6cc8b316814d17b43048ca)
Signed-off-by: Peter Kjellerstedt <peter.kjellerstedt@axis.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
There are two totally opposite use cases for how a premirror is
expected to behave in combination with specifying a downloadfilename=
parameter in the SRC_URI. On the one hand there is the expectation
that it works like any other mirror, which means the premirror is
expected to contain a file with the original name specified in the
SRC_URI. On the other hand there is the expectation that one can use
the artefacts downloaded by bitbake in ${DL_DIR} as a premirror, in
which case it is expected to contain a file with the name from the
downloadfilename= parameter.
The latter case has been how downloaded files have been handled until
commit 8a3ff9f3 (fetch2: fix premirror URI when downloadfilename
defined), where the fetcher was changed to store files as per the
first case. This is also when the test_npm_premirrors test case was
modified in commit 5ba191a0 (tests/fetch: add and fix npm tests) to
expect the first case.
The above change was later reverted in commit 96c30007 (fetch2: fix
downloadfilename issue with premirror). However the
test_npm_premirrors test case was not updated to match, and has been
failing ever since. This has probably gone unnoticed because the npm
related test cases require that npm is installed on the host.
This commit updates test_npm_premirrors to expect that premirrors use
the filenames specified by downloadfilename= as this matches the
current fetcher implementation and also is the most likely use case
for premirrors. It also tries to mimic how one typically might setup
the premirror directory by simply copying the download directory.
(Bitbake rev: 9e913ade70474aaeb928814d4763e7105569d63a)
Signed-off-by: Peter Kjellerstedt <peter.kjellerstedt@axis.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
If they are set, assume they are correct.
(Bitbake rev: a3a3a40b96c29051d97d0c96eb01784cbd1f5420)
Signed-off-by: Peter Kjellerstedt <peter.kjellerstedt@axis.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
This unifies the three separate implementations of git() from
GitMakeShallowTest, GitShallowTest and GitLfsTest into one
implementation in FetcherTest. It also makes use of this in
FetcherLocalTest and FetcherNetworkTest.
(Bitbake rev: 8c8b418c3e6f8ab2535ad405e8006ee1012f3484)
Signed-off-by: Peter Kjellerstedt <peter.kjellerstedt@axis.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
This changes:
test_foo .. skipped 'Network tests being skipped'
test_foo .. skipped 'npm not installed, tests being skipped'
to:
test_foo .. skipped 'network test'
test_foo .. skipped 'npm not installed'
to avoid double "skipped" in the output.
(Bitbake rev: e551328a4595c220b94ab5002d978ddb4deeebcf)
Signed-off-by: Peter Kjellerstedt <peter.kjellerstedt@axis.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Since commit 044fb04d (fetch2: Allow whitespace only mirror entries)
there is no need to separate the entries in mirror variables with
"\n".
(Bitbake rev: ed22b078aae25e804ca5edf062e2c905d3e49426)
Signed-off-by: Peter Kjellerstedt <peter.kjellerstedt@axis.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Both test_crate_url and test_crate_url_multi require Internet access.
(Bitbake rev: 34e9daed6a7f766bb6e27c7e02c94b545f4791d8)
Signed-off-by: Peter Kjellerstedt <peter.kjellerstedt@axis.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Since commit 044fb04d (fetch2: Allow whitespace only mirror entries)
there is no need to separate the entries in PREMIRRORS with "\n".
(Bitbake rev: 341746c7f7b8e04d3fad51ec1b9e462ced355bfc)
Signed-off-by: Peter Kjellerstedt <peter.kjellerstedt@axis.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The NIS can't work when network is dissable, so preserve network for it, the
error is like:
do_ypcall: clnt_call: RPC: Unable to send; errno = Network is unreachable
Note, enable nscd on the build machine might be a solution, but that isn't
reliable since it depends on whether the network function has been cached or
not.
(Bitbake rev: 4eafae7904bae6e5c6bc50356e8a9077f2e207fa)
Signed-off-by: Robert Yang <liezhi.yang@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
This imports the crate fetcher from OE-Core to resolve various module issues
and adds some very very basic tests of that new fetcher.
(Bitbake rev: 1f06f326fa8b47e2a4dce756d57a9369a2225201)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
If an unexpected exception occurs in a setscene task, it is currently hidden
from the user and not recorded in any logs. This isn't helpful to debug
such failures.
Change the code so that even in the "silent" or "quiet" task case (setscene
tasks), a warning is shown with the traceback unless it was an "handled"
exception. This means the failing function can show it's own warning/error
instead if it wants to and then raise a handled event.
(Bitbake rev: 41dcdc61eb40def8c14a42e8d7bb9ce5a34afa57)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
These variables were removed from the codebase, update the docs to match.
(Bitbake rev: 32180d5057c818a69987aada482e82acf3c72ef2)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The URL expected by git fetcher when the SSH protocol is used differs
from the one given by Git servers to be used with the `git clone`
command.
Add a note making this specificity known to users.
Cc: Quentin Schulz <foss+yocto@0leil.net>
(Bitbake rev: 81c09d434aed13fcfc4ba426ae6a0b849447eb5f)
Signed-off-by: Quentin Schulz <quentin.schulz@theobroma-systems.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The different stamp policies were poor versions of the siggen code and task
hashes, predating it and being used by packaged staging. They had many
limitations, hence their replacement. I'm not aware of any users of that
code any more so I believe it and the assoicated stamp whitelist variable
can simply be removed.
(Bitbake rev: 98407efc8c670abd71d3fa88ec3776ee9b5c38f3)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
This patch changes behaviour. After this change any task which does not
have the network flag set will have networking disabled on systems that
support that (kernel version dependent).
Add a "network" task specific flag which then triggers networking to
be enabled for this task, it is otherwise disabled.
This needs to happen before we enter the fakeroot environment of the task
due to the need for the real uid/gid which we save in the parent process.
(Bitbake rev: 0746b6a2a32fec4c18bf1a52b1454ca4c04bf543)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Add a function which uses the unshare glibc call to disable networking
in the current process. This doesn't work on older distros/kernels
but will on more recent ones so for now we simply ignore the cases we
can't execute on. uid/gid can be passed in externally so this can
work with pseudo/fakeroot contexts.
(Bitbake rev: 9d6341df611a1725090444f6f8eb0244aed08213)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
