Commit Graph

3602 Commits

Author SHA1 Message Date
Andre Rosa
20d0282c60 bitbake: Replace deprecated git branch parameter "--set-upstream"
Since 2017-08-17 (git version 2.14.1.473.g3ec7d702a) using deprecated
git branch parameter "--set-upstream" causes a fetcher error. Replace
it by "--set-upstream-to".

https://git.kernel.org/pub/scm/git/git.git/commit/?id=52668846ea2d41ffbd87cda7cb8e492dea9f2c4d
says, it's deprecated since 2012-08-30 so hopefully all still supported
host distributions have new enough git to support "--set-upstream-to".

ERROR: PACKAGE do_unpack: Fetcher failure: ...;
git -c core.fsyncobjectfiles=0 branch --set-upstream master origin/master failed with exit code 128, output:
fatal: the '--set-upstream' option is no longer supported. Please use '--track' or '--set-upstream-to' instead.

ERROR: PACKAGE do_unpack: Function failed: base_do_unpack

(Bitbake rev: 2ab50074c1a6c56a8a178755de108447d7b7acaf)

Signed-off-by: Andre Rosa <andre.rosa@lge.com>
Signed-off-by: Martin Jansa <Martin.Jansa@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-09-11 17:19:37 +01:00
Paul Eggleton
c697226168 bitbake: cooker: clear extra config settings and remote datastores on client disconnect
When the UI disconnects, we can throw away any server-side remote
datastores we created in response to calls from the UI, and we *must*
drop everything in extraconfigdata or it will taint any future
operations.

Dropping extraconfigdata upon disconnect fixes taskhash mismatch errors
when running devtool.DevtoolTests.test_devtool_update_recipe_local_files
within oe-selftest with BB_SERVER_TIMEOUT=100 in OpenEmbedded.

(Bitbake rev: 1ca2eec459424892391f060442ef38cf28d6a54a)

Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-09-05 15:00:49 +01:00
Mark Hatle
2b2606003b bitbake: cooker.py: Fix layer priority processing
If you have a layer with a blank BBFILE_PATTERN the layer was ignored
when processing the list of layers with priorities.  This list is not
only used for processing recipes, but also by additional programs such
as bitbake-layers show-layers.

Without this change, a layer that provides configuration or classes
only does now show up in show-layers, which is used by the
yocto-compat-layer.py script.  This causes a failures in the compatibility
check.

(Bitbake rev: a0eaf8c0f228f984bafff09e4e9739f758dc1a9b)

Signed-off-by: Mark Hatle <mark.hatle@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-09-05 15:00:49 +01:00
Paul Eggleton
f16cab3e74 bitbake: siggen: move reset() definition to base SignatureGenerator class
If we're implementing reset() in SignatureGenerator at all (and we need
to for a basic non-OE BitBake setup where that is the default signature
generator), then we need it to be clearing out the internal values
properly.

(Bitbake rev: 13f52d38fdbcb84c2a0c46f85baa44b22d53fdc1)

Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-09-05 15:00:49 +01:00
Richard Purdie
201fd94185 bitbake: cooker: Ensure parseConfiguration clears parsecache_valid
BB_SERVER_TIMEOUT=100 oe-selftest -r bblayers

was failing and highlighted that since parseConfiguation clears data
structures, it needs to also clear parsecache_valid as it no longer
contains correct data.

(Bitbake rev: 7234f33a7eb38ad51a8345f6689bc26e29f29f92)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-09-02 00:52:10 +01:00
Richard Purdie
2355a9b6f0 bitbake: cookerdata/taskdata/runqueue: Drop remaining tryaltconfigs code and commandline option
I can't actually see how this was working, nothing connected the commandline option
to the data in TaskData(). Drop the remaining pieces of this option, it was a relic
from a decade ago and we want deterministic builds, not random tries until something
might work.

(Bitbake rev: 767c7ba8fc76ec667ac1567de1c971c3575f2ecd)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-09-02 00:52:10 +01:00
Caio Marcelo de Oliveira Filho
49f530c218 bitbake: siggen: Add reset to SignatureGenerator
Fix failure after commit "cooker/siggen: Reset siggen when reparsing"
(e4c6ca9440f63761560b49bbe12654441f54687e) when executing without
specifying a BB_SIGNATURE_HANDLER.

(Bitbake rev: 2a78c2d09aea0323632bbc927f370f1d3c9c249e)

Signed-off-by: Caio Marcelo de Oliveira Filho <caio.oliveira@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-08-31 23:30:04 +01:00
Richard Purdie
fcac678c6c bitbake: cooker: Ensure buildFile doesn't have lasting side effects
BB_SERVER_TIMEOUT=100 oe-selftest -r devtool.DevtoolTests.test_devtool_build_image

fails, the reason is that internally, the limited_deps flag is set in
the bitbake server and then never cleared. This causes the sysroots to
be setup incorrectly (as per the limited dependency case) and builds
break. There is also potential for corruption of recipecaches.

Add shutdown/cleanup code to ensure these effects don't 'stick'.

This bug is particularly nasty as you can destroy TMPDIR with large
sysroots in build work directories which are prone to break.

Also ensure mtime cache is cleared (to match buildTargets) and
that no lasting changes are made to siggen either which ensures:

BB_SERVER_TIMEOUT=100 oe-selftest -r devtool.DevtoolTests.test_devtool_upgrade_git devtool.DevtoolTests.test_devtool_virtual_kernel_modify

works.

(Bitbake rev: 0a7ee8c8378bba9877c260b1aee782878f1935b4)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-08-31 23:30:04 +01:00
Paul Eggleton
29c8eb4759 bitbake: fetch2/npm: add noverify parameter to skip lockdown/shrinkwrap
When fetching source for the first time within scripts such as
OpenEmbedded's recipetool, we don't want to be showing warnings about
NPM_SHRINKWRAP or NPM_LOCKDOWN not being set since there's no way we
could have set them in advance. Previously we were using
ud.ignore_checksums to suppress these but since we are now using a more
standard task-based path to fetch the source, we need to disable these
through the metadata. Look for a "noverify" parameter set on the npm URL
and skip the checks if it is set to "1".

(Bitbake rev: 8c4b35d1e4d31bae9fddd129d5ba230acb72c3bb)

Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-08-31 23:30:04 +01:00
Paul Eggleton
1524222c84 bitbake: cooker: ensure we can run buildFileInternal() after cache is populated
If you run some other operations that result in the cache being
populated, and then call buildFileInternal(), then you can end up in a
situation where the cache already contains information about the recipe.
For example in OE this can now happen when you use devtool upgrade.
Normally this doesn't cause any problems, unless you have a non-absolute
path in BBLAYERS - in buildFileInternal() we are calling matchfile() which
will convert the filename to absolute, but later when taskdata goes to find
the providers of the recipe it finds the non-absolute path, sets up the
task information using this and then the runqueue can't find any tasks
matching the absolute path. To fix this, back out the optimisation I did
earlier in bitbake rev ba53e067a2d448dd63b4ca252557ce98aa8e6321 to avoid
calling parseConfiguration() again, which is unfortunate but does result
in the cached information being that causes the problem being cleared
out.

This fixes "Task do_unpack does not exist for target ..." running
devtool upgrade within intel-iot-refkit.

(Bitbake rev: f120355eaec4571ba6d60fc5f7ae9e1f31d846d1)

Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-08-31 23:30:04 +01:00
Paul Eggleton
8f716abaa4 bitbake: tinfoil: ensure variable history tracking works when parsing a recipe
If you set tracking=True when creating the tinfoil object, that ensures
history is collected for the main datastore, but at the end of parsing
the configuration, history tracking gets turned off to save time with
the result that we don't collect history for any recipes we parse.
Enable tracking when we parse a recipe (and disable it afterwards if we
enabled it) in order to fix this.

This fixes functionality in OE's devtool that relies upon variable
history (such as devtool upgrade updating PV when it's set within a
recipe).

(Bitbake rev: cc8b4c81bb589fb70774a0151f87a8d277f40f06)

Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-08-31 23:30:04 +01:00
Paul Eggleton
a3971620dc bitbake: tinfoil: ensure log lines get printed when tasks fail
If a task fails during build_targets(), we need to print out the log
lines as knotty does or the user will be missing information about the
failure.

(This should get some deeper refactoring, but now isn't the time for
that.)

(Bitbake rev: 24879df071d4803db3d39ae1d5cad852daa92f28)

Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-08-31 23:30:03 +01:00
Paul Eggleton
6ef0a56770 bitbake: tinfoil: fix log message doubling when config_only=False
With config_only=False we launch the UI and it sets up a logger, whereas
when config_only=True we don't, with the result that with True we are
seeing log messages from both our logger and the one set up by the UI.
Suppress our loggers with config_only=True to avoid this.

Fixes [YOCTO #11275] (again).

(Bitbake rev: b5e3b28b7c982dd8a3991d727f25710dbf58bb80)

Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-08-31 23:30:03 +01:00
Richard Purdie
260a22b241 bitbake: ui/knotty: Send updateConfig early
If for example you run:

bitbake -r somefile.inc
rm somefile.inc
bitbake -e

bitbake will crash with an error about not being able to find somefile.inc. This
is because it tries to reparse the base config for the early getVariable requests
before it sees the updated missing -r option.

Send the updateConfig command earlier to avoid this.

(Bitbake rev: a38164620ebdc770690c5f39ff9ed69d3f82719e)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-08-31 23:30:03 +01:00
Richard Purdie
0b331bc4c9 bitbake: cookerdata: Avoid tracebacks from early reset() calls
cooker.reset() can be called before we've actually setup the datastore. Gracefully
handle this case instead of the current traceback+exit.

(Bitbake rev: 8fd30ca6d271c125a8ea03ef0c5d7ab176900701)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-08-31 23:30:03 +01:00
Richard Purdie
2e41f733fa bitbake: cooker: Allow changes in PRSERV_HOST to be responded to
When we reload the configuration, PRSERV_HOST can change. Therefore
restart the PR Server depending on the new configuration at reparse.

Note that the server has to be started from the right process, it
can't be in the UI which shuts down as that shutdown triggers a shutdown
of its children and the PR Server shuts down too. This is why we need
pre_serve() which ensures its executed in the right context.

(Bitbake rev: 971272e84f4efe7ebd0037e164ba54f013a2a34e)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-08-31 23:30:03 +01:00
Richard Purdie
984da85479 bitbake: cooker: Change to consistent prefile/postfile handling
Currently the original prefile and postfile passed when starting bitbake
server are 'sticky'. With the new memory resident model this doesn't make
sense as the server the system is started with isn't special.

This patch changes the code so the prefile/postfile are used if specified
on the commandline and not used otherwise. This makes the behaviour much
more predictable and expected and as an added bonus simplifies the code.

(Bitbake rev: 638d366234fad78f283d3a13a12b07cb0ccbe914)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-08-31 23:30:03 +01:00
Richard Purdie
5759039a89 bitbake: cooker: Handle datastore tracking for showEnvironment server side only
The current passing of "tracking" backwards and forwards, client to server
is ugly and complex and error prone. Instead, set this during showEnvironment
commands triggering a reset there if/as required.

(Bitbake rev: 9dc7f384db0479569ff93a76a623d5395fecaf47)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-08-31 23:30:03 +01:00
Richard Purdie
313bf30db7 bitbake: prserv/cooker: Drop unused param
Drop pointless unused function parameter.

(Bitbake rev: 8104b33656de0b619943bd7a9884eb650ccafbf4)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-08-31 23:30:03 +01:00
Richard Purdie
92903ced3a bitbake: command: Fix some bugs identified by BB_SERVER_TIMEOUT
The 'needconfig' flag was meant to be the default as most commands do need
it and the "False" cases were the exception. The code was written backwards
with a default False value. Invert this to match the intent, resulting in
the config being reparsed if metadata has changed.

Also ensure the second level configuration is parsed for the getLayerPriorities
command as otherwise it can return stale info.

With these changes:

BB_SERVER_TIMEOUT=100 oe-selftest -r bblayers.BitbakeLayers.test_bitbakelayers_add_remove

passes instead of fails.

(Bitbake rev: af3c8928a69f204d5ced02c947485990ac04a776)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-08-31 23:30:03 +01:00
Richard Purdie
4152c47243 bitbake: main: Attempt to gain bitbake.lock rather than just waiting
Rather than just waiting for 5s, try and get the lockfile. If we gain
the lock, we know we're ready to retry and can skip any remaining timeout.

(Bitbake rev: 8a60106c6f7d586c793b965c5e9460b6016fab15)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-08-24 13:48:40 +01:00
Richard Purdie
2b453483d4 bitbake: process: Clean up connection retry logic
Its possible for a connection to connect to the server as its shutting down
but before its removed the socket file. This patch:

a) Removes the socket file earlier to avoid connections.
b) Handles EOFError in initial connections gracefully. These occur if the
   socket is closed during the server shutdown.
c) Ensure duplicate events aren't shown on the console. This makes debugging
   these issues very very confusing.

With these changes the backtrace that was concerning users is hidden and the
server works as expected with a reconnect when it catches it in a bad state.

(Bitbake rev: f45196cf84669723382730944dddc7eaf50826f2)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-08-24 13:48:40 +01:00
Paul Eggleton
2f9f417de8 bitbake: fetch2: don't mandate path element in encodeurl()
URLs do not have to have a path; currently our npm URLs don't, so
encodeurl() needs to handle if the path element isn't specified. This
fixes errors using OpenEmbedded's devtool add / recipetool create on an
npm URL after OE-Core revision ecca596b75cfda2f798a0bdde75f4f774e23a95b
that uses decodeurl() and encodeurl() to change URL parameter values.

(Bitbake rev: d5cab2dbf5682d2fd08e58316a3bf39a10f63df2)

Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-08-23 13:48:50 +01:00
Jussi Kukkonen
ff57794b51 bitbake: bitbake: ui: Sort 'Dependent tasks' in taskexp
The underlying model is already sorted for use in the other view,
add a sorting model for the 'Dependent Tasks' view.

(Bitbake rev: 27ca94c33234f0ef9753f8285213dde2871a3fcf)

Signed-off-by: Jussi Kukkonen <jussi.kukkonen@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-08-23 12:07:42 +01:00
Richard Purdie
ae874ef553 bitbake: process: Ensure we call select() to know which fds to read
There is an interesting bug in the current code where a sync command
is not seen until the current async command completes, by which time
the UI may have shut down.

The reason is that if there are idle commands, we may not end up sleeping
in the select call at all, partiularly under heavy load like parsing.

Fix this by calling select with a zero timeout so that we see active
fds and know to read from them. This fixes various problems toaster was
having with the recent server changes.

[YOCTO #11898]

(Bitbake rev: bbcce58e824b2793abf50efa52db158ae16e23e4)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-08-22 19:50:25 +01:00
Richard Purdie
1bfa9ca6c6 bitbake: process: Increase server startup timeout
We're seeing the server fail to start within 8s on heavily loaded
autobuilders so increase this timeout to 30s which should be more
than enough time.

(Bitbake rev: 8d4c120ec46d6d7a54947c64d33e18cb60b60505)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-08-16 00:04:38 +01:00
Richard Purdie
43b392536d bitbake: process: Improve client disconnects
There have been cases where the server could loop indefinitely and incorrectly
handle client disconnects. In the EOFError case, ensure a full disconnect
happens in the alternative disconnect path to avoid this.

(Bitbake rev: 5e267f14bb0155889615f567a920af4a37eb3c6b)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-08-16 00:04:38 +01:00
Richard Purdie
034bff2ca4 bitbake: cooker: Ignore common bitbake files for the parse cache invalidation
Writes to the cookerdaemon log and/or the lockfile were meaning the parser
cache was always being invalidated and reparsed. This is unnecessary so
spot accesses to these two common cases and ignore the files from a reparse
perspective.

This doesn't remove many sources of reparse but does improve several
common cases.

(Bitbake rev: 218e4b6418992588312b8ef5949b84ef43263d1a)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-08-13 09:28:14 +01:00
Richard Purdie
a0f729d31d bitbake: cooker: Improve inotify overflow handling
Add a proper function for clearing the mtime cache. Clean up the inotify
event overflow case to err on the side of caution and clear any potentially
now out of sync caches.

(Bitbake rev: ec60459fe2ba16966544eebff43b061abb7ed3ba)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-08-13 09:28:14 +01:00
Richard Purdie
019f518287 bitbake: cooker: Ensure we handle inotify before running the next command
The inotify watch events are handled "at idle" which could in theory mean
a command could run before any preceeding inotify events have been processed.
This leads to a theoretical race window where those events may have a
signficicant effect on the command.

Add a mechanism to allow us to ensure all pending events are processed before
running commands.

(Bitbake rev: bf76cd7e5881adf264b8ba64e27a5b6ca9df4fde)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-08-13 09:28:14 +01:00
Richard Purdie
0cb62f8ca7 bitbake: cooker: Track directories searched for bbappend/bb files
Some of the directories searched by BBFILES are not currently being added
to the inotify watch list. This can mean that added append files are not
noticed leading to misleading metadata results when using
BB_SERVER_TIMEOUT != 0.

We use glob to expand the BBFILES references and without writing our own
implentation, figuring out the directories it searches is hard. We use
some horrible hacks here to intecept the listdir calls, I'm open
to better ways to handle this but this does ensure we have the right
watches set.

(Bitbake rev: 4d508d35a224e3a25d2d59c8415ab7985964b14f)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-08-13 09:28:14 +01:00
Richard Purdie
801b0d29d4 bitbake: server/process: Ensure we don't loop on client EOFError
The server currently crashes if we hit an EOFError due to controllersock
still being in ready and the continue meaning ready isn't re-evaluated.
Setting the value to False can mean the shutdown code doesn't handle the
situation cleanly.

Clear ready to avoid the crash/loop instead and handle any OSError whilst
we're in here.

(Bitbake rev: 2bc47e887c3b41417edaa89a8708c223fd2085de)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-08-13 09:28:14 +01:00
Richard Purdie
ea9fff4bd0 bitbake: cooker/siggen: Reset siggen when reparsing
If we don't do this, we get basehash mismatch errors occurring from the reparse
which would then set bitbake's error exit code.

This for example would cause oe-selftest -r bbtests.BitbakeTests.test_bbappend_order
to fail with a non-zero BB_SERVER_TIMEOUT.

(Bitbake rev: e4c6ca9440f63761560b49bbe12654441f54687e)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-08-13 09:28:14 +01:00
Robert Yang
803b97a32e bitbake: main: Handle BB_SERVER_TIMEOUT = -1 for no server timeout
Make BB_SERVER_TIMEOUT = -1 mean no unload forever.

(Bitbake rev: 923d5567be6fd9969ca74c166f36817ec09305e3)

Signed-off-by: Robert Yang <liezhi.yang@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-08-09 09:24:56 +01:00
Robert Yang
11bde5de83 bitbake: process: Fix disconnect when BB_SERVER_TIMEOUT
Fixed:
$ export BB_SERVER_TIMEOUT=10000
$ bitbake --server-only
$ bitbake --status-only
[snip]
  File "/buildarea/lyang1/poky/bitbake/lib/bb/server/process.py", line 472, in recvfds
    msg, ancdata, flags, addr = sock.recvmsg(1, socket.CMSG_LEN(bytes_size))
OSError: [Errno 9] Bad file descriptor

And:
$ export BB_SERVER_TIMEOUT=10000
$ bitbake --server-only -B localhost:-1
$ bitbake --status-only # Everything is fine in first run
$ bitbake --status-only
[snip]
  File "/buildarea/lyang1/poky/bitbake/lib/bb/server/process.py", line 472, in recvfds
    msg, ancdata, flags, addr = sock.recvmsg(1, socket.CMSG_LEN(bytes_size))
OSError: [Errno 9] Bad file descriptor

This was because self.controllersock was not set to False, so it still ran
sock.recvmsg() when sock was closed.

And also need set command_channel to Flase, otherwise the
self.command_channel.get() will always run when EOF, and cause infinite loop.

(Bitbake rev: 7b739a38601b053d9bea4df2c0b44a952ab670c4)

Signed-off-by: Robert Yang <liezhi.yang@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-08-09 09:24:56 +01:00
Robert Yang
896243f4fe bitbake: main: Remove unneeded float()
There is already a type=float, so the float() is not needed, which also makes
the error clearer:

$ export BB_SERVER_TIMEOUT=10000AA
With float():
$ bitbake quilt-native
[snip]
ValueError: could not convert string to float: '10000AA'

Without float():
$ bitbake quilt-native
[snip]
optparse.OptionValueError: option --idle-timeout: invalid floating-point value: '10000AA'

The second one tells clearly where is wrong.

(Bitbake rev: 7a4ea1e6a10f20d80009a78f4f0aebf8f90095fd)

Signed-off-by: Robert Yang <liezhi.yang@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-08-09 09:24:56 +01:00
Paul Eggleton
46e2251625 bitbake: Revert "tinfoil: fix duplication of log messages"
In combination with the recent server reworking, this change actually
prevents messages sent from tasks from being logged properly. This will
of course give us the duplicated messages back, and I really hate to do
that effectively a second time, but that's better than seeing no error
at all in the case of a failure - we'll have to find the proper way of
avoiding the duplication that doesn't result in some messages going
missing.

This reverts commit 8a5bae76f91f2411187c638a42fa3c762052cf11.

(Bitbake rev: 645c8dd15762516ae5ab64a1df47fadb95d072d1)

Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-08-09 09:24:56 +01:00
Richard Purdie
9ed748a542 bitbake: daemonize: Always print any remaning UI events at exit
If there are events in the UI queue we want to print them regardless of
whether we're handling an exception or something like SystemExit.

This improves error messages for some other failure cases where currently
no logging would get printed and leave the user confused as to what went
wrong.

(Bitbake rev: 76c4f2c20216719736766e8ae7d089ccd061b71b)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-08-03 11:14:13 +01:00
Richard Purdie
66b8c16659 bitbake: cooker: Ensure all tasks are shown in task dependency tree
"bitbake -g m4-native -c do_unpack" doesn't list any dependencies in
task-depends.dot. This is because no header describing the task was
added unless a task has dependencies. Tweak the code to fix this.

[YOCTO #10893]

(Bitbake rev: e2fbe728769f09ef4178e2ef4d69a2f60ffe3ebb)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-08-03 11:14:13 +01:00
Richard Purdie
914a53cc94 bitbake: daemonize: clean up code
This was originally based on some other code but its mostly misleading
comments now. Massively clean this up and accept its now a total fork.
There are no funciton changes here, just cleanup.

(Bitbake rev: 3daa9001269e5b9156d0eb3299d2628d2d219fff)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-07-31 15:13:53 +01:00
Richard Purdie
8625f83e38 bitbake: daemonize: Ensure child process exits safely
When we create the child, if an exception occurred it was transfering
back into the parent context. We don't want to do that us use a try/finally
to ensure we exit.

We need to ensure a traceback is printed and any queued UI messages which
may not have made it to the client UI at this point.

(Bitbake rev: dec1d2c26f6cb3ffeb44beaab0129cd531a6d08b)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-07-31 15:13:53 +01:00
Richard Purdie
3010e4e7cf bitbake: process: Add some extra server startup logs
We have cases where the server is being started but we're not seeing any messages
from it. Add some earlier logging so we can try and better understand where
issues may be occurring.

(Bitbake rev: 10a6a48c0c5a2fe2051e90143e66075356853971)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-07-31 15:13:53 +01:00
Richard Purdie
129a52725f bitbake: process: Reorder server command processing and handle EOFError
If the connection control socket and the command channel close together,
we can race and hit EOFError exceptions before we close the channel.

Reorder the code to handle this in the correct order and ignore the
EOFError exceptions as they mean the client is disconnecting and shouldn't
terminate the server.

(Bitbake rev: 974281ed72d8366baa16ee85f7e93970689b5956)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-07-31 15:13:53 +01:00
Ross Burton
780e9fb877 bitbake: wget: fix FusionForge workaround
My previous assertion about FusionForge appears to have been wrong, or
FusionForge has changed behaviour, or both.

FusionForge now mandates that downloads have the Accept header set, despite that
header being optional, and returns a 406 Not Acceptable error if it isn't set.
As we were pretending that 406 was actually 405 (Moved) and tried to handle it as a
redirect this results in an infinite loop until Python kills the recursion.

Delete the handling of 406 as 405, and pass Accept: */* in the headers.

(Bitbake rev: bb70ae0c9aac5ec688026d23a64ac0cac1947187)

Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-07-31 15:13:53 +01:00
Richard Purdie
3f71378fde bitbake: runqueue: Tweak debug message to make it more readable/diffable
Having this as one huge long line isn't easy to manipulate, split it into
multiple lines for ease of debugging issues.

(Bitbake rev: 5753fe81194f75fbcf4ccdc733cc585d02794cb1)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-07-31 15:13:53 +01:00
Ross Burton
6986198da9 bitbake: fetch2: fix checkstatus fallback to MIRRORS
The checkstatus() code was expecting checkstatus to throw exceptions if it
failed, but in general it should return False.

(Bitbake rev: 57be5cc6228518e60f564570a39cebbeb6cf564e)

Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-07-31 15:13:52 +01:00
Christopher Larson
e75639f85a bitbake: tests/fetch: fix GitShallowTest.test_bitbake
`git fetch --tags` seems to interact badly with `mirror=fetch`, resulting in
the regular branches not being fetched, so drop the unnecessary `--tags`. This
fixes this unit test failure: `bb.fetch2.FetchError: Fetcher failure: Unable
to resolve 'master' in upstream git repository in git ls-remote output for
/tmp/tmp4ag_mgmn/gitsource`

[YOCTO #11698]

(Bitbake rev: 2d0203fae08c5ff8dc3e9afaa9a819abc4a1af6f)

Signed-off-by: Christopher Larson <chris_larson@mentor.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-07-30 08:43:36 +01:00
Richard Purdie
e4c9a92cb4 bitbake: main: Always return 0 for bitbake --kill-server
If the server isn't running return 0 as to do otherwise complicates
scripts which do cleanup of bitbake servers which would potentially
be memory resident.

(Bitbake rev: 0f85c54385f325f3ebbf911b35bbac11b8863b6b)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-07-30 08:43:36 +01:00
Richard Purdie
5f835043f2 bitbake: process: Clean up server communication timeout errors
This timeout path was commonly hit due to errors starting the server. Now we
have a better way to handle that, the retry logic can be improved and cleaned
up. This patch:

* Makes the timeout 5s rather than intervals of 1s with a message. Paul
  noted some commands can take around 1s to run on a server which has just
  been started on a loaded system.
* Allows a broke connection to exit immediately rather than retrying something
  which will never work.
* Drops the Ctrl+C masking, we shouldn't need that anymore and any issues
  would be better handled in other ways.

This should make things clearer and less confusing for users and is much cleaner
code too.

(Bitbake rev: 8633b7cd03cfaba3e0359aa5da22fc76b66768c7)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-07-30 08:43:36 +01:00
Richard Purdie
07cee9dea1 bitbake: process: Don't leak open pipes upon reconnection
If we reconnect to the server, stop leaking pipes and clean up
after ourselves.

(Bitbake rev: f41e4e971e807157be68cf4496580494b8b60643)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-07-30 08:43:36 +01:00