Clean up the whitespace and comments style, keep pylint happy.
(Bitbake rev: ee59fdaae68543ade03cacfdbbf14fdc7e469412)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Fix a typo in a variable which would affect proxy handling spotted by
pylint.
(Bitbake rev: 19f4ca0119e33df64da629253d5002fcdb385f4f)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
I wonder how this used to work for anybody.
(Bitbake rev: 9481e9cd688f635a7fe2ac60d5826ac26f0cb9ba)
Signed-off-by: Stefan Klug <stefan.klug@baslerweb.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Previously, this would happen:
======================================================================
ERROR: test_vercmpstring (bb.tests.utils.VerCmpString)
----------------------------------------------------------------------
Traceback (most recent call last):
File "/home/alexander/development/poky/bitbake/lib/bb/tests/utils.py", line 45, in test_vercmpstring
result = bb.utils.vercmp_string('1.', '1.1')
File "/home/alexander/development/poky/bitbake/lib/bb/utils.py", line 143, in vercmp_string
return vercmp(ta, tb)
File "/home/alexander/development/poky/bitbake/lib/bb/utils.py", line 135, in vercmp
r = vercmp_part(va, vb)
File "/home/alexander/development/poky/bitbake/lib/bb/utils.py", line 124, in vercmp_part
elif ca < cb:
TypeError: '<' not supported between instances of 'NoneType' and 'int'
----------------------------------------------------------------------
(Bitbake rev: fef56d28c3efec4876c379898cbc4d4c65303aee)
Signed-off-by: Alexander Kanavin <alex.kanavin@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
There were a couple of problems with the multiconfig dependency resolution:
- the "if mc" condition triggering this code wasn't correct, it needs
to be "if more than one multiconfig" configured
- after adding providers we need to call add_unresolved again
and rebuild mcdeps within the "while new" loop
By fixing these issues we allow various other combinations of multiconfig
builds to work which previously didn't.
[YOCTO #13090]
[YOCTO #13130]
(Bitbake rev: 4359b037de578095db2595f119dfb8e3340e1414)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
There was already a fix to ignore some multiconfig dependencies but its
'opposite' case wasn't covered. Cover that combination to so as to avoid
tracebacks in multiconfig builds.
[YOCTO #13090]
[YOCTO #13130]
(Bitbake rev: c883dfe378af9dfc192a8e392e84325d68648806)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The consumers of BB_TASKDEPDATA in OE metadata can't cope with multiconfig
dependencies. The choice is either to start adding code to each of them to
filter out multiconfig dependencies, or do this at source.
After consideration we've decided to do this at source as doing otherwise
is code duplication and error prone and in any case we've looked at, they
don't make sense.
[YOCTO #13090]
[YOCTO #13130]
(Bitbake rev: 531dcd221a10853f45cc057b52bb2d5083e0ee42)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Currently the mechanism for breaking out of the dependnecy loop analysis
code is broken and doesn't work leading to bitbake appearing to hang.
Add in a custom exception for this purpose and fix the code to exit
as intended, fixing the hang and making the dependency loop code
usable again.
(Bitbake rev: 8756e4ade67c16e35269ea0659e10b9ebaa6117f)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Changes FAKEROOTCMD so that it can accept additional arguments to pass
to the fakeroot implementation instead of being treated as a simple
command
(Bitbake rev: 4fa51afb56b090cf1f746842acd602c9536715d5)
Signed-off-by: Joshua Watt <JPEWhacker@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
It shouldn't work when there are duplicated BBFILE_COLLECTIONS.
(Bitbake rev: a1f251e5ab859d6d4a2cb908408d4ddcab5a5de1)
Signed-off-by: Robert Yang <liezhi.yang@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Previously this information was simply discarded, which in some cases
resulted in generic, unhelpful failures.
With this change the user would see what the exception is, but
without ugly tracebacks or bulky, irrelevant information from the log file.
(Bitbake rev: 8da9242702fbef8cf3156b95a1076802e0f653c7)
Signed-off-by: Alexander Kanavin <alex.kanavin@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
getVar() has been defaulting to expanding by default for
a long time (2016), thus remove the True option from
getVar() calls with a regex search and replace.
Search & replace made using the following command:
sed -e 's|\(d\.getVar \?\)( \?\([^,()]*\), \?True)|\1(\2)|g' \
-i $(git grep -E 'getVar ?\( ?([^,()]*), ?True\)' \
| cut -d':' -f1 \
| sort -u)
(Bitbake rev: 3bba0dbd524cf72176a765957adff544ae5c255a)
Signed-off-by: André Draszik <andre.draszik@jci.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The intent of the code was to catch one kind of error, it was actually swallowing
all exceptions and looping indefinitely. Fix it to work as intended.
This explains some mystery hangs we've been seeing.
(Bitbake rev: d73dbc3580faa1225d95ae4cefac4879ca3c1b2f)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Fixed:
- Add the following lines to conf/local.conf:
FOO = "BAR1"
FOO_append = "\
BAR2"
$ bitbake -e | grep '^FOO'
FOO="BAR1BAR2"
The leading spaces in the second line have been removed.
- But if add the previous two lines to base.bbclass:
$ bitbake -e | grep '^FOO'
FOO="BAR1 BAR2"
The leading spaces in the second line are preserved, this is inconsistent, now
fix ConfHandler to preserve leading spaces.
[YOCTO #12380]
(Bitbake rev: 8c3bc15a7b5e0a81d7b6c9d3fe43fbff63207156)
Signed-off-by: Robert Yang <liezhi.yang@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Move the submodule processing outside of the if statement to avoid any
optimizations that may happen.
Update the test cases to include the additional case, and split the other
test cases into individual tests to make it easier to figure out what
the failure may be.
(Bitbake rev: 0ec98c01ae50f95c9c74acf53013ac59e0e72b08)
Signed-off-by: Mark Hatle <mark.hatle@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
If the submodule is in a subdirectory, it needs to have that structure
preserved. This means the unpack path needs to be in the 'dirname' of the
final path -- since the unpack directory name is specified in the URI.
Additional specific test cases were added to ensure this is working properly
based on two recent error reports.
(Bitbake rev: 8c8ecec2a722bc2885e2648d41ac8df07bdf660d)
Signed-off-by: Mark Hatle <mark.hatle@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Prior code happened to work with relative URLs, when the code was recently
restructured it caused all relative urls to no longer work. Restore the prior
code flow for relative support and better comment why that code is there.
(Bitbake rev: 14bb383af0ca98e0e04ec217e537b6a899f3658e)
Signed-off-by: Mark Hatle <mark.hatle@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
We don't want references including shell parameter expansion modifiers
(i.e. `:-`, `#`, `%%`, etc) to be added to our vardeps, so add a test to
ensure this.
YOCTO #12987
(Bitbake rev: be022085fe1ea1b9a9d519f0455883e2da363d2c)
Signed-off-by: Christopher Larson <chris_larson@mentor.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
This aligns the behavior of expansion with the recipe parser, only
attempting to expand references to valid variable names. This avoids
adding references for things like `${foo#${TOPDIR}}` to our vardeps
without imposing much additional processing overhead beyond the change
to the expansion regexp.
YOCTO #12987
(Bitbake rev: df2ac65370aa86cdbc1574fdede25e3519410e45)
Signed-off-by: Christopher Larson <chris_larson@mentor.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The update_submodules and unpack_submodules functions were nearly indentical,
so we made a common function where the different behavior could be passed
in by the download and unpack users. The new function is process_submodules.
Moved the parse_gitmodules function under the new process_submodules, since
there are no external callers.
Refactor the file relative path processing to the URL translation code.
We also add a warning to the translation if a relative ssh URL has been
detected. Since this can cause a problem.
In the case of a relative URL that does not work after being translated,
it should be possible to use the MIRROR functions to manual translate the
generated relative URL into one that works properly.
Remove 'git config' processing on download contents. It turns out this is not
necessary since all of the later components work using the git fetcher.
Limit the 'git submodule update' call to only when unpacking a non-bare
repository. Submodules are always loaded as bare, so this prevents
intermediate unpacks from being attempted.
Finally, the test cases were updated and the new commit ids in the test
repository were updates as well.
(Bitbake rev: 610dbee5634677f5055e2b36a3043cd197fb8c51)
Signed-off-by: Mark Hatle <mark.hatle@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
A custom shallow submodule is no longer necessary, as the regular git
fetcher is used and shallow handling works with the same code.
The only general difference between the regular change is simply declaring a
clone as shallow, when appropriate.
This also removes a potential race condition in copying repositories
vs cloning them.
The gitsm shallow fetcher test was revised to verify that the submodule
is shallow cloned along with the primary repository.
The first step of this change was to be sure to clean the gitsubmodule download
directory, as was previously done with the may gitsource directory.
Additional test components were added to verify commit counts, and an
obsolete (and likely incorrect) test for the .git/modules directory to be
empty was also removed.
(Bitbake rev: f9cc4684dcf4281acc557cda8cb35602354ac3d6)
Signed-off-by: Mark Hatle <mark.hatle@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Greatly simply the unpack rule by copying the general functionality of
update_submodules as unpack_submodules. This will recursively construct
a set of urls and unpack them using the standard system behaviors.
The overall code may be slightly bigger, but this ensures that all of the
standard locks are inplace, ensuring the code doesn't change out from
under the unpack function. (This could have happened before due to using
'cp' instead of further unpacks on submodules. This may still happen in
shallow clones.)
(Bitbake rev: 7d7ee630f1c65e7dd234f945edf5e3b3bcb0fc30)
Signed-off-by: Mark Hatle <mark.hatle@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
It was reported that a race condition on a shared download directory could
occur with the gitsm fetcher, the result happened with a call to
git config
that occured within the update_submodules. Since the fetch is locked by the
upper level, it was probably the prior need_update(...) function causing this
because of some old code.
The gitsm class inherits the git class. The need_update was overridding the
version in gitsm, so that it forceably checked the submodules.
It's clear we can optimize the code by only updating if the primary repository
needs updating. Since we don't care if the submodule repository has changed
because if the primary hasn't, references to the submodule won't change.
(Bitbake rev: 346338667edca1f58ace769ad417548da2b8d981)
Signed-off-by: Mark Hatle <mark.hatle@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
In order to test the ssh processing in gitsm, we add an alternative
testcase that can be downloaded from git.yoctoproject.org. However,
this test case requries (read) access, via ssh, to git.yoctoproject.org.
(Bitbake rev: c8554cdc1287620fe8e8960561e614567879a010)
Signed-off-by: Mark Hatle <mark.hatle@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The following appear to be the git supported formats:
proto://user:pass@host/path (URI format)
user@host:path (SSH format)
/path or ./path or ../path (local file format)
We adjust the parsing to find out if we have a URI format or not.
When we are NOT in URI format, we do our best to determine SSH or
file format by looking for a ':' in the overall string. If we find
a ':' we assume SSH format and adjust accordingly.
Note, in SSH format we simply replace the ':' with a '/' when constructing
the URL. However, if the original path was ":/...", we don't want '//' so
we deal with this corner case as well.
(Bitbake rev: dcac05e7dc6b0c5f8e63d36ad105b7eab41f0016)
Signed-off-by: Mark Hatle <mark.hatle@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
It is possible for a submodule to be defined in the .gitmodules file, but
never initialized in the repository itself. This shows itself when searching
for the defined module hash you will get back a empty value.
Similarly we need to identify and skip defined but not initialized submodules
during the unpack stages as well.
Thanks to raphael.lisicki@siemens.com for their help is figuring out how
to resolve this issue.
Additionally a problem was found where, while unlikely, it may be possible
for the wrong revision to have been searched using ls-tree. This has been
resolved in the update_submodules function by keeping the correct revision
along with the submodule path.
(Bitbake rev: 49e1ff920143dcd4e7f73933d7ab8a84b8ffa1a3)
Signed-off-by: Mark Hatle <mark.hatle@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Fix handling of escape characters in regexs and hence fix python
Deprecation warnings which will be problematic in python 3.8.
(Bitbake rev: c1fcc46e2498ddd41425d8756754f814d682aba3)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The unihash should be fetched using the task filename that includes the
multiconfig prefixes.
[YOCTO #13124]
(Bitbake rev: 5e7f4e77e27bceaf6c68137cacb4f8d7d7de49dd)
Signed-off-by: Joshua Watt <JPEWhacker@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Similarly to the codeparser change, change to sha256 hashes due to worries
over collisions. The main impact of this change is slightly slower parsing
time as well as longer sstate file names.
(Bitbake rev: 66f1b766997d53b4375fdd25719b1175f3828903)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
We've reports of hash collision with codeparser. Looking at the way collision
problems occur with md5 and the way our function templating works, I can believe
we may run into issues.
This patch therefore switches to sha256.
Performance wise, parse time could appear to rise by 4s in 374s
Before:
384329 in 2.966s (md5)
After:
349743 in 2.340s (sha256)
34723 in 1.245s (md5)
since we still have md5 used elsewhere in the code, something we should look at
next (using sha256 everywhere is around 5.3s in total)
Unfortunately this does nearly double the size of the codeparser cache file
due to the hash size change.
(Bitbake rev: 4bed7a97198176503fe8c72d8dd7c36b28fc9dd1)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
We have two choices, split the recipes amongst the parsing threads in
blocks ahead of time, or have a queue which parsers pull from when idle.
The optimum approach depends on how similar the pieces are. For the single
recipe reparse case, there is currently a significant wait for the feeder
thread to start (around 0.25s in a 2s command).
Its possible splitting into blocks in advance may be unluckly for some other
workloads but experimentally it seems to work better overall for me at least.
(Bitbake rev: ae79868861568d673a70472e85a4bde9e2d84a8f)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
There used to be many bugs in multiprocessing and we implemented our own
feeder process to avoid them. Now that we have python 3.x, these are fixed
and just using the standard Queue mechanism appears to work fine. We can
therefore drop the unneeded code and simplify.
(Bitbake rev: b2d39fc37fcf3c81a562ec1ef4f8b4c1a493fc57)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Add a test for x86_64 in overrides where is was being incorrectly handled.
There was a previous fix (3a3be518536acc868c7eeb3c1111ad1b321480b7) but
this ensures we don't regress.
(Bitbake rev: 08c314eac231ac9292c8c95f1c5a6fc2023fe749)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Under heavy load with process delays, sqlite can issues a "locking
protocol" error (SQLITE_PROTOCOL). Unfortunately, it is impossible to
distinguish between actual locking protocol errors and this race
condition, so they best that can be done is to retry the operation when
the error is detected.
[YOCTO #13108]
(Bitbake rev: 93cd15644f9d12b38abea276fee7b5bade0276df)
Signed-off-by: Joshua Watt <JPEWhacker@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
If the unique hash is being used to track task dependencies, the hash
validation function needs to know about it in order to properly validate
the hash.
[YOCTO #13030]
(Bitbake rev: 9a529bb2658a4046dafbf32e1eb503d84e64e947)
Signed-off-by: Joshua Watt <JPEWhacker@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The unique hash is now passed to the task in the BB_UNIHASH variable
[YOCTO #13030]
(Bitbake rev: aab80b099f6f259e4b57cba2c26dd385d07c5947)
Signed-off-by: Joshua Watt <JPEWhacker@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Requests the task unique hash from siggen and tracks it
[YOCTO #13030]
(Bitbake rev: 1ecc47f0831b35c8c92b37a81cef4e43ff9f67b2)
Signed-off-by: Joshua Watt <JPEWhacker@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Abstracts the function to get the unique hash for a task. This hash is
used as in place of the taskhash for the purpose of determine how other
tasks depend on this one. Unless overridden, the taskhash is the same as
the unique hash, preserving the original behavior.
[YOCTO #13030]
(Bitbake rev: d0065b34cea81fcadff14b0383779e9559d94508)
Signed-off-by: Joshua Watt <JPEWhacker@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
When an indirect multiconfig dependency exists, such as:
A depends on B, B has a multiconfig dependency to C,and our build
target is A, the multiconfig dependency to C is not processed on
time, hence no providers are added for it, causing an exception in
the runqueue because the dependency does exist in it.
Call add_unresolved() for all available multiconfigs before processing
providers for multiconfig dependencies, detecting mcdepends on time so
providers for them can be added correctly.
(Bitbake rev: 8a6bc7584ad61b4de98af92a86066602006262f9)
Signed-off-by: Alejandro Enedino Hernandez Samaniego <alejandr@xilinx.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
If the bitbake server recieved multiple connections, it currently closes
ones it can't handle (while its dealing with another). This is rather
antisocial behaviour which causes clients to quickly run through their
retries and abort.
Instead, queue any other connections until the current one is closed. This
way the client can decide when it wants to stop waiting for the server. If the
client is gone by the time we handle it, we handle that gracefully.
This also fixes a number of bugs in the connection handling where connections
which did drop early were badly handled causing tracebacks in the logs.
Also, handle queue incomming connections in a loop to ensure that the main
client handling doesn't starve that piece of the system.
This code was stress tested by running 50 connection attempts in parallel at
once, ensuring the code correctly handled them.
(Bitbake rev: 02845a561b38658ac3edf5cc9c34625ed860d34f)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Now that we set a timeout for the socket, it can return EWOULDBLOCK
if a signal or other event happens to wake up even if we don't timeout.
If this happens, retry the connection, else we simply see it quickly
loop through the retries and abort the connection in a very short
interval.
(Bitbake rev: d5b0a9a302ac0eafa4f797ac15ea77db87e82b3c)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The current bitbake output makes it hard to know which retry is being
attempted. Add this information to the output to make it clearer.
(Bitbake rev: 0774e6e03d27adb7aca6fa9c47ab6ad426c937de)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The current value of 2 seconds has shown to be short in
wider testing.
(Bitbake rev: 8a1f2fcf35d61d83bbafa8fa3ae215fd5f51728b)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Adds a test suite for testing the persistent data cache
[YOCTO #13030]
(Bitbake rev: 96a4155049e834af17069d981cc2215e50d18c1a)
Signed-off-by: Joshua Watt <JPEWhacker@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
We're seeing hangs in oe-selftest where server startup and shutdown are
racing. The assumption was a connect would timeout however no timeout is
set which can leave processes hanging. Set a short timeout for
the connection to avoid this.
(Bitbake rev: f02114cb70e8f6f1d32e19c02b758fe0aadecd19)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
With aarch64 hosts coming into use, set the syscall number to
avoid ioprio warnings on that platform.
(Bitbake rev: 5eaf9e7b26f09f5f106e1c3c6976d517b289450a)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
We're seeing problems due to the way x86-64 is handled (or not handled)
as an override. Relax the containts on overrides from being lowercase
to being lowercase or numeric. This fixes problem where MACHINE=qemux86
would work but MACHINE=qemux86-64 would fail the same tests.
(Bitbake rev: 3a3be518536acc868c7eeb3c1111ad1b321480b7)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Fix some further python3 warnings about unescaped regexs.
(Bitbake rev: 8667605d016e82add95638fcb15c2bbc1b489ecc)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The p_error() is used for printing errors when parse shell code, but it can't
the EOF error correctly
- Add the following lines to quilt.inc
do_configure_prepend () {
find ${s} -name "*.in" -exec sed -i -e "1s,^#\!.*@perl@ -w$,#\! @perl@\nuse warnings;," {} \;
if [ hello ]; then
}
- Before the patch:
$ rm -fr cache/ tmp/cache/; bitbake -p
[snip]
WARNING: /path/to/quilt/quilt-native_0.65.bb: Error during finalise of /path/to/quilt/quilt-native_0.65.bb
[snip]
bb.pysh.sherrors.ShellSyntaxError: None
followed by:
We can see that this isn't easy to debug, let p_error() check wheter it is EOF
and print appropriate errors can improve the error message. And don't let
codeparser.py except pyshlex.NeedMore (in fact, it never worked since p_error()
only raise ShellSyntaxError), but make it print the last 5 lines which might be
useful for debuging.
- After the patch
$ rm -fr cache/ tmp/cache/; bitbake -p
[snip]
ERROR: /path/to/quilt/quilt_0.65.bb: Error during parse shell code, the last 5 lines are:
find /path/to/quilt/0.65-r0/quilt-0.65 -name "*.in" -exec sed -i -e "1s,^#\!.*@PERL@ -w$,#\! @PERL@\nuse warnings;," {} \;
if [ hello ]; then
autotools_do_configure
sed -e 's,^COMPAT_SYMLINKS.*:=.*,COMPAT_SYMLINKS :=,' -i /path/to/quilt/0.65-r0/quilt-0.65/Makefile
[snip]
File "/path/to/bb/pysh/pyshyacc.py", line 649, in p_error(p=None):
w('Unexpected EOF')
> raise sherrors.ShellSyntaxError(''.join(msg))
bb.pysh.sherrors.ShellSyntaxError: Unexpected EOF
(Bitbake rev: 44790597951638e32eb1672de2e40bd5a603326b)
Signed-off-by: Robert Yang <liezhi.yang@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
>>> from Set import Set
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
ImportError: No module named 'Set'
So remove related code.
(Bitbake rev: 118e9d8894e64d102543562314b318676006e710)
Signed-off-by: Robert Yang <liezhi.yang@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The only two that we need are ShellSyntaxError and ShellError, others are not
used, so remove them.
(Bitbake rev: abee2ff5ee71bf8c3221cd9d3cd4243ebb4ca273)
Signed-off-by: Robert Yang <liezhi.yang@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
We don't use these modules in bitbake, so remove them.
And pysh's author declared that pysh is no longer maintained:
http://pysh.sourceforge.net/
(Bitbake rev: 9cc4f12d2aef366a593c7977d3818a3002b1d8bf)
Signed-off-by: Robert Yang <liezhi.yang@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The configuration of the sqlite database can timeout due to locking
under heavy load and should be subject to the same retry logic as the
other statements.
[YOCTO #13069]
(Bitbake rev: 5a2a95b0396e39662968690b3065d2f88167a71c)
Signed-off-by: Joshua Watt <JPEWhacker@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
We've already tweaked remove() for speed and not to error if it
races for deletion. Therefore use this for prunedir() which was
starting to show the same bug reports.
[YOCTO #13003]
(Bitbake rev: 47f359f9b017f8d99d0bb2161ac0dcefcbd915de)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
os.errno used to happen to work but is invalid. Correct to use errno.
[YOCTO #13068]
(Bitbake rev: 03aae96a8859409b1ce7e3d5c1197371fa96bd14)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
This could clearly never have worked since the python3 migration
but as its in an error path, it doesn't get tested/used much.
(Bitbake rev: 704f27dc28d50a6dc02c8b64274ee4ecb3058c4a)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
We're seeing slow startup in bitbake, add some timeing debug messages so
the logs are more useful for debugging when its slow.
(Bitbake rev: 8d1fc115b8a176009f1f3a8ce840b422e7e0b45e)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
On loaded production systems we've seen bitbake server take over
40s to start up. Increase the timeout to 90s which tries to avoid
failures in selftests.
The delays come from setting up the inotify watches (31s) so can't
really be avoided.
After 5s delay we now warn the user we're waiting for 90s so the
interactive exeperience shouldn't be much changed and its very
unlikely the user would see that anyway.
(Bitbake rev: 7a4e105093c3080c1087cd06abc2883852a23e28)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Remove a deprecated warning and stop using our own deprecated API!
(Bitbake rev: 83ece2c6f4b000e906fec9148f25bd1dff66cfb0)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The mechanism used to get the hash for a stamp file is split out so that
it can be overridden by derived classes
[YOCTO #13030]
(Bitbake rev: ce241534d19b2f1c51dbdb3b92419676d234e464)
Signed-off-by: Joshua Watt <JPEWhacker@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Pass the task hash as a parameter to the 'runtask' message instead of
passing the entire dictionary of hashes when the worker is setup. This
is possible less efficient, but prevents the worker taskhashes from
being out of sync with the runqueue in the event that the taskhashes in
the runqueue change.
[YOCTO #13030]
(Bitbake rev: 1e86d8c1bec7ea5d016a5ad2097f999362e29033)
Signed-off-by: Joshua Watt <JPEWhacker@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Turns off the shared cache. It isn't a significant factor in performance
(now that WAL is enabled), and is a really bad idea to have enabled in
processes that fork() (as bitbake it prone to do).
[YOCTO #13030]
(Bitbake rev: 71b427bf01374973a971c10cb64024c8ef2a11eb)
Signed-off-by: Joshua Watt <JPEWhacker@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Enabling the write ahead log improves database reliability, speeds up
writes (since they mostly happen sequentially), and speeds up readers
(since they are no longer blocked by most write operations). The
persistent database is very read heavy, so the auto-checkpoint size is
reduced from the default (usually 1000) to 100 so that reads remain
fast.
[YOCTO #13030]
(Bitbake rev: 79100fa67539f9654af9bf6d3e6842eb5c12e989)
Signed-off-by: Joshua Watt <JPEWhacker@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Constructs the "key" column in the persistent database as a non-NULL
primary key. This significantly speeds up lookup operations in large
databases.
[YOCTO #13030]
(Bitbake rev: f5ba7775cfcb90401522d977cc66fe0f5aeb7a66)
Signed-off-by: Joshua Watt <JPEWhacker@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The original implementation of persistent data executed all SQL
statements via sqlite3.Connection.execute(). Behind the scenes, this
function created a sqlite3 Cursor object, executed the statement, then
returned the cursor. However, the implementation did not account for
this and failed to close the cursor object when it was done. The cursor
would eventually be closed when the garbage collector got around to
destroying it. However, sqlite has a limit on the number of cursors that
can exist at any given time, and once this limit is reached it will
block a query to wait for a cursor to be destroyed. Under heavy database
queries, this can result in Python deadlocking with itself, since the
SQL query will block waiting for a free cursor, but Python can no longer
run garbage collection (as it is blocked) to free one.
This restructures the SQLTable class to use two decorators to aid in
performing actions correctly. The first decorator (@retry) wraps a
member function in the retry logic that automatically restarts the
function in the event that the database is locked.
The second decorator (@transaction) wraps the function so that it occurs
in a database transaction, which will automatically COMMIT the changes
on success and ROLLBACK on failure. This function additionally creates
an explicit cursor, passes it to the wrapped function, and cleans it up
when the function is finished.
Note that it is still possible to leak cursors when iterating. This is
much less frequent, but can still be mitigated by wrapping the iteration
in a `with` statement:
with db.iteritems() as it:
for (k, v) in it:
...
As a side effect, since most statements are wrapped in a transaction,
setting the isolation_level when the connection is created is no longer
necessary.
[YOCTO #13030]
(Bitbake rev: e8b9d3f534ef404780be23b601d5a4bb9cec928a)
Signed-off-by: Joshua Watt <JPEWhacker@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Small tweak to ensure these items are printed into the log since there
is other logging code which looks for the header and this makes it clearer
the server did start but is slow somewhere in startup.
(Bitbake rev: 7c0b84cac892744225fa0462f918ea9a79b356cb)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Its possible the read may return a smaller number of characters. Remove
the possibility by using a single character to signal the server is ready.
(Bitbake rev: 767c9596d129d21ddf5d2e00c55f0a0525c641dc)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
This simplifies the code and makes it easier to read but has the
same functionality.
(Bitbake rev: a0b9cfaf2d03fd047a79d32e668001718d02c4bf)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The previous check was in data.py which only can check code like "python
funcname()" in the dependency chain, but there are 3 kinds of python functions:
- python()
- def py_funcname()
- python funcname()
Add the checking to BBHandler to check and warn for all of them.
The warning looks like:
WARNING: /path/to/recipes-core/busybox/busybox_1.29.2.bb: python should use 4 spaces indentation, but found tabs in busybox.inc, line 75
(Bitbake rev: 0cdc5b81fc1f5e5281a525a657e420ebc3bb9e90)
Signed-off-by: Robert Yang <liezhi.yang@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The ${DL_DIR}/svn directory is used by BitBake to keep checked-out SVN
repositories from which tarballs are generated. These repositories were
protected from concurrent update with a lock on the tarballs. However,
the tarballs are specific to the SRCREV and module checked out (many
tarballs can come from the same repository), meaning a repository could
be modified concurrently if two recipes checked out two different
SRCREVs or modules from it in parallel. This caused errors like the
following:
ERROR: Fetcher failure: Fetch command failed with exit code 1, output:
svn: E155004: Run 'svn cleanup' to remove locks (type 'svn help cleanup' for details)
svn: E155004: Working copy '/home/foo/downloads/svn/repo/trunk' locked.
svn: E155004: '/home/foo/downloads/svn/repo/trunk' is already locked.
Fix it by adding a per-repository lock that's independent of the module
and SRCREV.
(Bitbake rev: 3f1f183a17bf3580da8a4ffd6dab30b62c2654a8)
Signed-off-by: Ulf Magnusson <Ulf.Magnusson@bmw.de>
Signed-off-by: Michael Ho <Michael.Ho@bmw.de>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Currently if there are no setscene tasks, the disk monitor isn't started.
Move the startup code to somewhere to ensure it always is started. This
issue would partially explain occasional selftest failures.
(Bitbake rev: 5ba83ee25c1c9cba349edb68a22476b1d5fca6ce)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
10 log lines may not capture any full traceback, increase the number of
lines to 60 which covers most tracebacks.
(Bitbake rev: b8538c2088c199157404712a60a6740c926ce312)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
We're seeing issues where the server doesn't start with no logs as to why. Allow
the server to print the last 60 log lines just in case this shows us something useful
about what is failing.
(Bitbake rev: c8c80b404e38fe96f65d6314cd95f4069319f3d6)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Also use assetIn instead of assertTrue which aids debugging failures.
(Bitbake rev: 19dee675bb9ad012d28e1e57a888931355a831cb)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
This code checks to see if shallow is either disabled or the tarball is
missing, but the else block tries to print the tarball filename, and
this attribute doesn't exist at all when shallow is disabled. Handle the
two cases separately to give sane errors for both cases without the
exception:
Exception: AttributeError: 'FetchData' object has no attribute 'fullshallow'
(Bitbake rev: bdbb558342ebb4e64384c9838d2485d9299d91a6)
Signed-off-by: Christopher Larson <chris_larson@mentor.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
When multiconfig is enabled the cooker adds providers
for all the targets to be built on all the multiconfig
variables that were set, regardless if there is a dependency
to it or not.
This causes an issue when a certain target is incompatible
with one or more of the multiconfigs, e.g. the target is not
in COMPATIBLE_MACHINE for one of the MACHINEs being built,
causing the cooker to error out since no providers can be
found for that certain target on that multiconfig.
This patch modifies the behavior to only look for PROVIDERS
for a target on the multiconfig that was selected to be built,
PROVIDERS are then looked for in other multiconfigs only when
there is a defined dependency to them.
[YOCTO #12985]
(Bitbake rev: f2106a3a767542359fdde238abcf5fe35ab3a144)
Signed-off-by: Alejandro Enedino Hernandez Samaniego <alejandr@xilinx.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
shrinkwrap resolved relative URL can start with http. For example,
"resolved: http-proxy/-/http-proxy-${PV}.tgz" is still relative URL
to npm registry, but starts with http.
Current if statement compares the startswith 'resolved' to 'http',
which makes impossible to use npm download. Condtional comparison
now strictly checks for "http://" and "https://"
(Bitbake rev: f76075aa1a5159fd4d62949cb588346888b9fe60)
Signed-off-by: Parthiban Nallathambi <pn@denx.de>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Fixed:
- Add a comment in base.bbclass:
def oe_import(d):
import sys
# Comment
bbpath = d.getVar("BBPATH").split(":")
[snip]
Note, '# Comment' is started with '#', it is legal in python's syntax
(though maybe not a good style), but bitbake reported errors:
$ bitbake -p
ERROR: ParseError at /path/to/base.bbclass:20: unparsed line: ' bbpath = d.getVar("BBPATH").split(":")'
This error report would mislead people, the real problem is that '# Comment'
is not supported, but it reports the next line, this may make it hard to debug
the code are complicated.
We can make __python_func_regexp__ handle '^#' to fix the problem, since it
already can handle blank line "^$" in a python function, so it would be pretty
safe to handle "^#" as well.
(Bitbake rev: 79e62eef1c93f742bf71e9f25db57fdd2ffedd02)
Signed-off-by: Robert Yang <liezhi.yang@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
[YOCTO #12898]
There might be no bitbake-cookerdaemon.log, print a message for debugging.
(Bitbake rev: 4adc582d2df7fdb9e51c4ebb5e66bbd21165b4dc)
Signed-off-by: Robert Yang <liezhi.yang@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Fixed:
Add the following two lines to conf/local.conf:
FOO = "${@foo = 5}"
HOSTTOOLS += "${FOO}"
* Before the patch
$ bitbake -p
Check the first lines of bitbake bitbake-cookerdaemon.log
[snip]
File "/buildarea1/lyang1/poky/bitbake/lib/bb/data_smart.py", line 125, in python_sub
codeobj = compile(code.strip(), self.varname or "<expansion>", "eval")
File "FOO", line 1
[snip]
There isn't a file named 'FOO', but a variable name.
* After the patch
$ bitbake -p
[snip]
File "/buildarea1/lyang1/poky/bitbake/lib/bb/data_smart.py", line 129, in python_sub
codeobj = compile(code.strip(), varname, "eval")
File "Var <FOO>", line 1
foo = 5
(Bitbake rev: 540b546be55e0f5f5d91695956da3a7732b2f90a)
Signed-off-by: Robert Yang <liezhi.yang@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
This can make it print clearer errors, for exmaple:
Add Runtime_error to 'def oe_import(d)"
16 def oe_import(d):
17 import sys
18 Runtime_error
[snip]
* Before the patch:
$ bitbake -p
ERROR: Unable to parse /buildarea1/lyang1/poky/bitbake/lib/bb/data_smart.py
Traceback (most recent call last):
File "/buildarea1/lyang1/poky/bitbake/lib/bb/data_smart.py", line 430, in DataSmart.expandWithRefs(s='${@oe_import(d)}', varname='OE_IMPORTED[:=]'):
except Exception as exc:
> raise ExpansionError(varname, s, exc) from exc
bb.data_smart.ExpansionError: Failure expanding variable OE_IMPORTED[:=], expression was ${@oe_import(d)} which triggered exception NameError: name 'Runtime_error' is not defined
This error message has two problems:
- "Unable to parse data_smart.py": This isn't the real cause.
- It pionts to "raise ExpansionError(varname, s, exc) from exc" which isn't clear enough.
* After the patch:
$ bitbake -p
ERROR: Unable to parse OE_IMPORTED[:=]
Traceback (most recent call last):
File "OE_IMPORTED[:=]", line 1, in <module>
File "/buildarea1/lyang1/poky/meta/classes/base.bbclass", line 18, in oe_import(d=<bb.data_smart.DataSmart object at 0x7f9257e7a0b8>):
import sys
> Runtime_error
bb.data_smart.ExpansionError: Failure expanding variable OE_IMPORTED[:=], expression was ${@oe_import(d)} which triggered exception NameError: name 'Runtime_error' is not defined
This one is more clearer than before.
(Bitbake rev: c0fe524c1aeccb24ddd2e1f7bf235c00fdbf79a7)
Signed-off-by: Robert Yang <liezhi.yang@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Fixed:
- Define an error anonymous function in base.bbclass:
15
16 python() {
17 Compile error
18 }
$ bitbake -p
ERROR: Error in compiling python function in /buildarea1/lyang1/poky/meta/classes/base.bbclass, line 18:
The code lines resulting in this error were:
0001:def __anon_18__buildarea1_lyang1_poky_meta_classes_base_bbclass(d):
*** 0002: Compile error
0003:
SyntaxError: invalid syntax (base.bbclass, line 18)
The lineno should be 17, but it reported 18, this would mislead people a lot
when there more lines.
- Now fix it to:
ERROR: Error in compiling python function in /buildarea1/lyang1/poky/meta/classes/base.bbclass, line 17:
The code lines resulting in this error were:
0001:def __anon_18__buildarea1_lyang1_poky_meta_classes_base_bbclass(d):
*** 0002: Compile error
0003:
SyntaxError: invalid syntax (base.bbclass, line 17)
This is because the anonymous function is constructed by:
text = "def %s(d):\n" % (funcname) + text
The len(self.body) doesn't include the "def " line, the length of the function
should be "len(self.body) + 1", so we need pass "self.lineno - (len(self.body) + 1)"
which is the same as 'self.lineno - len(self.body) - 1' to
bb.methodpool.insert_method() as we already had done to named function. Otherwise, the
lineno is wrong, and would cause other problems such as report which line is
wrong, but the line is not what we want since it reports incorrect line.
(Bitbake rev: 7466c8765fcc792e5ea3daefda3c5895e782d6c4)
Signed-off-by: Robert Yang <liezhi.yang@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Fixed:
- Add an error line in base.bbclass, e.g.:
15
16 def oe_import(d):
17 import sys
18 Compile error
19 bbpath = d.getVar("BBPATH").split(":")
[snip]
Note the "Compile error" line, I added it for reporting errors.
$ bitbake -p
ERROR: Error in compiling python function in /buildarea1/lyang1/poky/meta/classes/base.bbclass, line 15:
The code lines resulting in this error were:
0014: import oe.data
0015: for toimport in oe.data.typed_value("OE_IMPORTS", d):
0016: imported = __import__(toimport)
0017: inject(toimport.split(".", 1)[0], imported)
*** 0018:
0019: return ""
0020:
SyntaxError: invalid syntax (base.bbclass, line 18)
There are 2 problems:
- The "line 15" is incorrect, it is a blank line, not the error line.
- The "*** 0018" points to incorrect position.
These two problems would mislead people a lot sometimes.
- Now fix it to:
$ bitbake -p
ERROR: Error in compiling python function in /buildarea1/lyang1/poky/meta/classes/base.bbclass, line 18:
The code lines resulting in this error were:
0001:def oe_import(d):
0002: import sys
*** 0003: Compile error
0004: bbpath = d.getVar("BBPATH").split(":")
[snip]
SyntaxError: invalid syntax (base.bbclass, line 18)
Please see comments in the code for more details on how it is fixed.
(Bitbake rev: bbb3d87d171da38fd8e9bce011d109fba28a75c0)
Signed-off-by: Robert Yang <liezhi.yang@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The colors specified for use with bitbake-diffsigs were adapted for a
dark theme, e.g., by setting the background color to black, which made
it look very bad when used with a light theme.
To make it look good both with a dark or a light theme, it is better
to drop the background color. It is also better to leave out the color
altogether for the title and just use bold. Finally, dropping bold for
the red and green texts indicating removed/added values better matches
other colorized diff implementations as, e.g., git diff.
(Bitbake rev: f1a2c23520832ee91e85338c1ad8af1fec0d0b19)
Signed-off-by: Peter Kjellerstedt <peter.kjellerstedt@axis.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The imp module is deprecated, port the code over to use importlib.
bitbake/lib/bb/utils.py:30: DeprecationWarning: the imp module is deprecated in favour of importlib; see the module's documentation for alternative uses
import imp
(Bitbake rev: 3c2cb35588e91fbd7b136e5e2c78eeb77e126c84)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Traceback (most recent call last):
File "/home/pokybuild/yocto-worker/nightly-oe-selftest/build/bitbake/lib/bb/main.py", line 464, in setup_bitbake
server_connection = bb.server.process.connectProcessServer(sockname, featureset)
File "/home/pokybuild/yocto-worker/nightly-oe-selftest/build/bitbake/lib/bb/server/process.py", line 490, in connectProcessServer
if command_chan_recv:
UnboundLocalError: local variable 'command_chan_recv' referenced before assignment
(Bitbake rev: ad79fadd855f5c10242ed17e9e0f3eb0274f26d2)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Use % operator to ensure that the dependency is substituted into the error
message correctly.
(Bitbake rev: 8e587386b0b3a0ed11edf71dfbe9fb508f60d0b3)
Signed-off-by: Mike Crowe <mac@mcrowe.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Support users who want to specify a custom ssh to Git using an environment
variable. (This lets build systems work without requiring users to
configure Git explicitly.)
(Bitbake rev: db46fb2755a4b1033ab60051ce511cc9dd7e34fb)
Signed-off-by: Mike Crowe <mac@mcrowe.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Git does not require the module and target path to be the same in the
.gitmodules file. This incorrect assumption was being made previously
causing various unpack failures.
An example .gitmodule showing this issue:
[submodule "plugins/WaveShaper/Libs/inih"]
path = plugins/wolf-shaper/Libs/inih
url = https://github.com/pdesaulniers/inih.git
The unpack function also needed to work in a loop on the overall
submodules_queue. Before it could have missed items that were not in the
primary repository.
(Bitbake rev: 5a7009c204f4d2254e3b2d83ad9319ac23f1cf4d)
Signed-off-by: Mark Hatle <mark.hatle@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Submodules by definition refer to a specific commit, not branch. If we don't
ignore the branch, then any commits on a submodule on a branch different then
the original module will trigger a failure that the commit is not on the
branch.
(Bitbake rev: fdc1dbf96f153b496de52acd8263366a1ff303ad)
Signed-off-by: Mark Hatle <mark.hatle@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
If a variable has a _remove applied to it but that variable is in turn
'renamed' through OVERRIDES, the removal gets lost with the current code.
TEST = "foo"
TEST_someval = "bar"
TEST_someval_remove = "bar"
OVERRIDES = "someval"
currently gives "bar" for TEST but should give "".
This fixes the code to track the removal and adds a test case to ensure this
doesn't regress again.
(Bitbake rev: 8f55010c18057be040f073d8bcb4c5c2c311d809)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
