Commit Graph

70 Commits

Author SHA1 Message Date
Robert Yang
afbc3f4427 bitbake: bitbake: cooker: don't stop file notifier when cooker is shutdown
It should be live/exited with server rather than cooker, fixed:
$ bitbake --server-only -T -1
Set MACHINE = "qemux86" in conf/local.conf
$ bitbake quilt
Set MACHINE = "qemuppc" in conf/local.conf
$ bitbake quilt
[snip]
ERROR: When reparsing /workspace1/lyang1/poky/meta/recipes-connectivity/openssl/openssl_1.0.2m.bb.do_package, the basehash value changed from c216f7f4fdd3cf4a0b10b975a636426c to d5a8e9431ab261381752d7a64c7b2fa9. The metadata is not deterministic and this needs to be fixed.
[snip]

This is because the server doesn't know local.conf is changed since the
notifiers are stopped, so it doesn't reparse, and then we would get the errors,
let the notifiers live/exited with server can fix the problem.

(Bitbake rev: a6a641cb9c5f3abe901b150da915372e295383d7)

Signed-off-by: Robert Yang <liezhi.yang@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2018-02-14 15:26:03 +00:00
Richard Purdie
7a822506c0 bitbake: server/process: Increase runCommand timeout
We were seeing cases where we could hit the 5s timeout on large/fast machines
running many different tasks at once. Increase this to 30s since the main
connection timeout path should no longer hit this slow path.

[YOCTO #12116]

(Bitbake rev: 131d4b8a5834781a93ed41e2967d8dcd4d80f29a)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-09-25 14:14:17 +01:00
Paul Eggleton
e2c25f4bd0 bitbake: server/process: ensure server failure log is limited to current session
Printing the last 10 lines of bitbake-cookerdaemon.log when the server
fails to start can sometimes result in printing the output from a
previous run, which could lead the user completely down the wrong path
in terms of the cause of the failure. Use a known start text containing
the time which we can then look for when scanning through the log, and
then grab the last 10 lines of that part instead.

Fixes [YOCTO #11903].

(Bitbake rev: 567f2cf1bc455b4f3cfb1cbd7f25145360b05a62)

Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-09-12 23:55:29 +01:00
Richard Purdie
28949d3f80 bitbake: process/cooker: Improve readypipe handling
Issues in start are not being correctly detected by the current readypipe
code. Change it to use specific "ready" or "fail" messages to correctly
determine the correct failure mode and avoid bitbake seeming to hang
(it does currently timeout eventually).

[YOCTO #12062]

(Bitbake rev: 60d4791e3dd05729d2a2adf6f3b203c80d466a73)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-09-12 23:55:29 +01:00
Richard Purdie
984da85479 bitbake: cooker: Change to consistent prefile/postfile handling
Currently the original prefile and postfile passed when starting bitbake
server are 'sticky'. With the new memory resident model this doesn't make
sense as the server the system is started with isn't special.

This patch changes the code so the prefile/postfile are used if specified
on the commandline and not used otherwise. This makes the behaviour much
more predictable and expected and as an added bonus simplifies the code.

(Bitbake rev: 638d366234fad78f283d3a13a12b07cb0ccbe914)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-08-31 23:30:03 +01:00
Richard Purdie
2b453483d4 bitbake: process: Clean up connection retry logic
Its possible for a connection to connect to the server as its shutting down
but before its removed the socket file. This patch:

a) Removes the socket file earlier to avoid connections.
b) Handles EOFError in initial connections gracefully. These occur if the
   socket is closed during the server shutdown.
c) Ensure duplicate events aren't shown on the console. This makes debugging
   these issues very very confusing.

With these changes the backtrace that was concerning users is hidden and the
server works as expected with a reconnect when it catches it in a bad state.

(Bitbake rev: f45196cf84669723382730944dddc7eaf50826f2)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-08-24 13:48:40 +01:00
Richard Purdie
ae874ef553 bitbake: process: Ensure we call select() to know which fds to read
There is an interesting bug in the current code where a sync command
is not seen until the current async command completes, by which time
the UI may have shut down.

The reason is that if there are idle commands, we may not end up sleeping
in the select call at all, partiularly under heavy load like parsing.

Fix this by calling select with a zero timeout so that we see active
fds and know to read from them. This fixes various problems toaster was
having with the recent server changes.

[YOCTO #11898]

(Bitbake rev: bbcce58e824b2793abf50efa52db158ae16e23e4)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-08-22 19:50:25 +01:00
Richard Purdie
1bfa9ca6c6 bitbake: process: Increase server startup timeout
We're seeing the server fail to start within 8s on heavily loaded
autobuilders so increase this timeout to 30s which should be more
than enough time.

(Bitbake rev: 8d4c120ec46d6d7a54947c64d33e18cb60b60505)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-08-16 00:04:38 +01:00
Richard Purdie
43b392536d bitbake: process: Improve client disconnects
There have been cases where the server could loop indefinitely and incorrectly
handle client disconnects. In the EOFError case, ensure a full disconnect
happens in the alternative disconnect path to avoid this.

(Bitbake rev: 5e267f14bb0155889615f567a920af4a37eb3c6b)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-08-16 00:04:38 +01:00
Richard Purdie
801b0d29d4 bitbake: server/process: Ensure we don't loop on client EOFError
The server currently crashes if we hit an EOFError due to controllersock
still being in ready and the continue meaning ready isn't re-evaluated.
Setting the value to False can mean the shutdown code doesn't handle the
situation cleanly.

Clear ready to avoid the crash/loop instead and handle any OSError whilst
we're in here.

(Bitbake rev: 2bc47e887c3b41417edaa89a8708c223fd2085de)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-08-13 09:28:14 +01:00
Robert Yang
803b97a32e bitbake: main: Handle BB_SERVER_TIMEOUT = -1 for no server timeout
Make BB_SERVER_TIMEOUT = -1 mean no unload forever.

(Bitbake rev: 923d5567be6fd9969ca74c166f36817ec09305e3)

Signed-off-by: Robert Yang <liezhi.yang@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-08-09 09:24:56 +01:00
Robert Yang
11bde5de83 bitbake: process: Fix disconnect when BB_SERVER_TIMEOUT
Fixed:
$ export BB_SERVER_TIMEOUT=10000
$ bitbake --server-only
$ bitbake --status-only
[snip]
  File "/buildarea/lyang1/poky/bitbake/lib/bb/server/process.py", line 472, in recvfds
    msg, ancdata, flags, addr = sock.recvmsg(1, socket.CMSG_LEN(bytes_size))
OSError: [Errno 9] Bad file descriptor

And:
$ export BB_SERVER_TIMEOUT=10000
$ bitbake --server-only -B localhost:-1
$ bitbake --status-only # Everything is fine in first run
$ bitbake --status-only
[snip]
  File "/buildarea/lyang1/poky/bitbake/lib/bb/server/process.py", line 472, in recvfds
    msg, ancdata, flags, addr = sock.recvmsg(1, socket.CMSG_LEN(bytes_size))
OSError: [Errno 9] Bad file descriptor

This was because self.controllersock was not set to False, so it still ran
sock.recvmsg() when sock was closed.

And also need set command_channel to Flase, otherwise the
self.command_channel.get() will always run when EOF, and cause infinite loop.

(Bitbake rev: 7b739a38601b053d9bea4df2c0b44a952ab670c4)

Signed-off-by: Robert Yang <liezhi.yang@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-08-09 09:24:56 +01:00
Richard Purdie
3010e4e7cf bitbake: process: Add some extra server startup logs
We have cases where the server is being started but we're not seeing any messages
from it. Add some earlier logging so we can try and better understand where
issues may be occurring.

(Bitbake rev: 10a6a48c0c5a2fe2051e90143e66075356853971)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-07-31 15:13:53 +01:00
Richard Purdie
129a52725f bitbake: process: Reorder server command processing and handle EOFError
If the connection control socket and the command channel close together,
we can race and hit EOFError exceptions before we close the channel.

Reorder the code to handle this in the correct order and ignore the
EOFError exceptions as they mean the client is disconnecting and shouldn't
terminate the server.

(Bitbake rev: 974281ed72d8366baa16ee85f7e93970689b5956)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-07-31 15:13:53 +01:00
Richard Purdie
5f835043f2 bitbake: process: Clean up server communication timeout errors
This timeout path was commonly hit due to errors starting the server. Now we
have a better way to handle that, the retry logic can be improved and cleaned
up. This patch:

* Makes the timeout 5s rather than intervals of 1s with a message. Paul
  noted some commands can take around 1s to run on a server which has just
  been started on a loaded system.
* Allows a broke connection to exit immediately rather than retrying something
  which will never work.
* Drops the Ctrl+C masking, we shouldn't need that anymore and any issues
  would be better handled in other ways.

This should make things clearer and less confusing for users and is much cleaner
code too.

(Bitbake rev: 8633b7cd03cfaba3e0359aa5da22fc76b66768c7)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-07-30 08:43:36 +01:00
Richard Purdie
07cee9dea1 bitbake: process: Don't leak open pipes upon reconnection
If we reconnect to the server, stop leaking pipes and clean up
after ourselves.

(Bitbake rev: f41e4e971e807157be68cf4496580494b8b60643)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-07-30 08:43:36 +01:00
Richard Purdie
b2c39835bb bitbake: process/cooker: Allow UI process to know if the cooker was started successfully
Currently if the server fails to start, the user sees no error message and
the server will be repeatedly attempted to be started until some longer
timeouts expire. There are error messages in the cookerdeamon log but
nobody thinks to look there.

Add in a pipe which can be used to tell the starting process whether the cooker
did actually start or not. If it fails to start, no further attempts can be
made and if present, the log file can be shown to the user.

[YOCTO #11834]

(Bitbake rev: 57000d44beb1aeba37dfc70782b0d6418943acc5)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-07-30 08:43:36 +01:00
Richard Purdie
dd3a59d354 bitbake: process: Move socket keep alive into BitBakeProcessServerConnection
This cleans up the socket keep alive into better class structured code
and adds cleanup of the open file descriptors upon shutdown.

(Bitbake rev: 77fd3a3a29a569e212374b27aea742ddbaafcdd5)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-07-30 08:43:36 +01:00
Richard Purdie
76b6cdf414 bitbake: process: Allow BBUIEventQueue to exit cleanly
Currently the monitoring thread exits with some error code or runs indefinitely. Allow
closure of the pipe its monitoring to have the thread exit cleanly/silently.

(Bitbake rev: 930d077637928213e13a07c78fee3bf7a8c37ebf)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-07-30 08:43:36 +01:00
Richard Purdie
8b7aa5bfd6 bitbake: process: Ensure ConnectionReader/Writer have fileno() and close() methods
Expose the underlying close() and fileno() methods which allow connection
monitoring and cleanup.

(Bitbake rev: f79187f4ebfad7969be47b429995e7f7a3e33c1e)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-07-30 08:43:36 +01:00
Richard Purdie
33b73b2d79 bitbake: process: Change timeout warning to a note
The warning message currently shown can occur more frequently than previously
if a previous bitbake server is shutting down and we're reconnecting to a new
server. Change it to a note message to match the higher level connection
logging retry messages and so as not to interfer with selftests.

(Bitbake rev: b7514340cd6a2753eb217b059229bb279c3849ec)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-07-24 16:57:14 +01:00
Richard Purdie
76ecfa5f69 bitbake: cooker/process: Drop server_main function
Now that there is only one server, this abstraction is no longer needed
and causes indrection/confusion. The server shutdown is also broken with
the cooker post_server calls happening too late, leading to "lock held"
warnings in the logs if PRServ is enabled.

Remove the abstraction and put the shutdown calls in the right order
with respect to the locking.

(Bitbake rev: c0ddde7cf680225127d6285685652b905ed176c3)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-07-24 16:57:14 +01:00
Robert Yang
47ce5bb034 bitbake: server/process.py: fix self.bitbake_lock.write()
There is no global var "configuration", so the old code hang at
self.bitbake_lock.write(), and nothing wrote to bitbake.lock. I didn't
figure out why it hang (but not print errors).

Reproducer:
$ bitbake -B localhost:-1 world -k

Check bitbake.log, there was nothing, now fixed.

(Bitbake rev: dbdd9010663cd8dcb328e1b6fb40e3a777293bc5)

Signed-off-by: Robert Yang <liezhi.yang@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-07-21 08:41:11 +01:00
Richard Purdie
4602408c69 bitbake: server: Rework the server API so process and xmlrpc servers coexist
This changes the way bitbake server works quite radically. Now, the
server is always a process based server with the option of starting
an XMLRPC listener on a specific inferface/port.

Behind the scenes this is done with a "bitbake.sock" file alongside
the bitbake.lock file. If we can obtain the lock, we know we need
to start a server. The server always listens on the socket and UIs
can then connect to this. UIs connect by sending a set of three file
descriptors over the domain socket, one for sending commands, one for
receiving command results and the other for receiving events.

These changes meant we can throw away all the horrid server abstraction
code, the plugable transport option to bitbake and the code becomes
much more readable and debuggable. It also likely removes a ton of
ways you could hang the UI/cooker in weird ways due to all the race
conditions that existed with previous processes.

Changes:

 * The foreground option for bitbake-server was dropped. Just tail
   the log if you really want this, the codepaths were complicated enough
   without adding one for this.
 * BBSERVER="autodetect" was dropped. The server will autostart and
   autoconnect in process mode. You have to specify an xmlrpc server
   address since that can't be autodetected. I can't see a use case
   for autodetect now.
 * The transport/servetype option to bitbake was dropped.
 * A BB_SERVER_TIMEOUT variable is added which allows the server
   to stay resident for a period of time after the last client
   disconnects before unloading. This is used if the -T/--idle-timeout
   option is not passed to bitbake.

This change is invasive and may well introduce new issues however I
believe the codebase is in a much better position for further
development and debugging.

(Bitbake rev: 72a3dbe13a23588e24c0baca6d58c35cdeba3f63)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-07-21 08:41:11 +01:00
Richard Purdie
577b75086e bitbake: server: Remove base classes and inline code
In preparation for rewriting this code, expand the relatively useless
base classes into the code itself.

(Bitbake rev: a1c6151420d86bac658c08ae714647062edd6ef2)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-07-21 08:41:11 +01:00
Richard Purdie
dd71707d5a bitbake: event/command: Allow UI to request the UI eventhander ID
The UI may want to change its event mask however to do this, it needs the
event handler's ID. Tweak the code to allow this to be stored and add
a command to query it.

Use the new command in the process server backend.

(Bitbake rev: f8cf2cb58b80ce74f756a11a9773b6b0e78d51ee)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-07-21 08:41:11 +01:00
Aníbal Limón
5a5c6c7fb6 bitbake: bb/server/process: Handle EINTR on idle_commands select
If a signal is sent like SIGWINCH the select could be interrupted
so ignore the InterruptError like in XMLRPC server [1].

[1]
http://git.yoctoproject.org/cgit/cgit.cgi/poky/tree/bitbake/lib/bb/server/xmlrpc.py#n307

(Bitbake rev: 96bb174325493764718c61c12c943c37b882cd61)

Signed-off-by: Aníbal Limón <anibal.limon@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-07-18 22:32:02 +01:00
Richard Purdie
35846a3961 bitbake: event: Queue offline events for the UI
Messages printed when no UI is connected (e.g. memres) are currently lost.
Use the existing queue mechanism to queue these until a UI attaches, then
replay them. This isn't ideal but better than the current situation of
losing them entirely.

(Bitbake rev: cb241fb8544dfb05646dbae1a1b04e17878a466c)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-07-08 13:29:54 +01:00
Richard Purdie
9245c3b87f bitbake: server/process: Fix waitEvent() calls with 0 timeout
You might think Queue.Queue.get(True, 0) would return an event immediately
if present and otherwise return. It doesn't, it immediately "times out"
and returns with nothing from the queue.

The behaviour we want is not to wait but return anything present which is
what .get(False) does so map to this.

This fixes some odd behaviour observed in some of the tinfoil selftests.

(Bitbake rev: 412bfab8721ea317898a1974f6a7a0d0bea763df)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-07-08 13:29:54 +01:00
Aníbal Limón
509d26f95b bitbake: bb/server/process.py: ProcessEventQueue add close of _writer pipe
Call explicity close in _writer to avoid fd leakage because isn't
called on Queue.close()

[YOCTO #10873]

(Bitbake rev: 12dfac442d2d5674198485fbeb708a01d215c576)

Signed-off-by: Aníbal Limón <anibal.limon@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-01-23 22:38:49 +00:00
Joshua Lock
97f83e911e bitbake: bitbake: remove True option to getVar calls (take 2)
getVar() now defaults to expanding by default, thus remove the True
option from getVar() calls with a regex search and replace.

Search made with the following regex: getVar ?\(( ?[^,()]*), True\)

(a follow on patch to fix up a few recent introductions)

(Bitbake rev: f9961fd5beb31d5ab9656a5be59f7ab3effef2f0)

Signed-off-by: Joshua Lock <joshua.g.lock@intel.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-12-16 10:23:24 +00:00
Paul Eggleton
6688f6a115 bitbake: server/process: don't change UI process signal handler on terminate
On terminating the connection to the server, we were disabling SIGINT -
and this is executed on the UI side. I'm not sure whether the intention
here was to undo the SIGINT disabling we did in the server, and it was
just a mistake that it disabled rather than restored and it's run on the
wrong side, or whether we wanted to stop the user from breaking out of
the shutdown code - the commit message provides no clues either way.
Regardless, we do not want to permanently disable Ctrl+C here - it's
legitimate to terminate the connection to the server and then
re-establish it within the same process; at least currently, devtool
modify by virtue of using tinfoil in two separate parts of the code does
this, and the result of this disabling is that during the second tinfoil
usage we can potentially be parsing all recipes without the ability to
easily interrupt the process.

(Bitbake rev: 58c60a951229dcbd8253863fb24228d046c23f6e)

Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-12-14 12:25:07 +00:00
Patrick Ohly
083365143e bitbake: cooker process: fire heartbeat event at regular time intervals
The intended usage is for recording current system statistics from
/proc in buildstats.bbclass during a build and for improving the
BB_DISKMON_DIRS implementation.

All other existing hooks are less suitable because they trigger at
unpredictable rates: too often can be handled by doing rate-limiting
in the event handler, but not often enough (for example, when there is
only one long-running task) cannot because the handler does not get
called at all.

The implementation of the new heartbeat event hooks into the cooker
process event queue. The process already wakes up every 0.1s, which is
often enough for the intentionally coarse 1s delay between
heartbeats. That value was chosen to keep the overhead low while still
being frequent enough for the intended usage.

If necessary, BB_HEARTBEAT_EVENT can be set to a float specifying
the delay in seconds between these heartbeat events.

(Bitbake rev: 7cf22ea057d28c54bd98dc1ab7a43402a29ff1f5)

Signed-off-by: Patrick Ohly <patrick.ohly@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-12-07 10:42:22 +00:00
Richard Purdie
0f2c59367a bitbake: bitbake: Convert to python 3
Various misc changes to convert bitbake to python3 which don't warrant
separation into separate commits.

(Bitbake rev: d0f904d407f57998419bd9c305ce53e5eaa36b24)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-06-02 08:24:02 +01:00
Richard Purdie
bfc21fd9b2 bitbake: server/process: Fix missing log messages issue
Currently if the server dies, its possible that log messages are never
displayed which is particularly problematic if one of those messages
is the exception and backtrace the server died with.

Rather than having the event queue exit as soon as the server disappears,
we should pop events from the queue until its empty before exiting.

This patch tweaks that code so that even if the server is dead and we're
going to exit, we return any events left in the pipe. This makes
debugging certain failures much easier.

(Bitbake rev: 29f6ade68fb2b506a23a7eb3a00cdcffa291b362)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-05-14 23:05:15 +01:00
Ed Bartosh
b58e5b1fb3 bitbake: bitbake: xmlrpc: set single use mode differently
Currently xmlrpc server implicitly sets itself into single use mode
when bitbake server is started with anonymous port (0) or no port is
provided in command line. In this mode bitbake shuts down xmlrpc server
after build is done. This assumption is incorrect in some cases.
For example Toaster uses bitbake in this mode and expects xmlrpc server
to stay in memory.

Till recent changes single use mode was always unset due to the bug.
When the bug was fixed it broke toaster builds as Toaster couldn't
communicate with bitbake server in single use mode.

Reimplemented logic of setting single use mode. The mode is explicity
set when --server-only command line parameter is not provided to bitbake.
It doesn't depend on the port number anymore.

[YOCTO #9275]
[YOCTO #9240]
[YOCTO #9252]

(Bitbake rev: afc0dd5c532684f6201b1e12bbf4c226ea19062d)

Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com>
Signed-off-by: Elliot Smith <elliot.smith@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-03-24 12:31:50 +00:00
Lucas Dutra Nunes
d53413d3a8 bitbake: server/process: Try connecting 4 times before giving up
Instead of trying one time with a timeout of 20 seconds try 4 times with
a timeout of 5 seconds, to account for a slow server start.

(Bitbake rev: 4a7fe63126dd8177baa5ad21e59e0bebeea8c596)

Signed-off-by: Lucas Dutra Nunes <ldnunes@ossystems.com.br>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-03-09 22:47:29 +00:00
Richard Purdie
5375e6431c bitbake: bitbake: Set process names to be meaninful
This means that when you view the process tree, the processes
have meaningful names, aiding debugging:

$ pstree -p 30021
bash(30021)───KnottyUI(115579)───Cooker(115590)─┬─PRServ(115592)───{PRServ Handler}(115593)
                                                ├─Worker(115630)───bash:sleep(115631)───run.do_sleep.11(115633)───sleep(115634)
                                                └─{ProcessEQueue}(115591)

$ pstree -p 30021
bash(30021)───KnottyUI(117319)───Cooker(117330)─┬─Cooker(117335)
                                                ├─PRServ(117332)───{PRServ Handler}(117333)
                                                ├─Parser-1:2(117336)
                                                └─{ProcessEQueue}(117331)

Applies to parse threads, PR Server, cooker, the workers and execution
threads, working within the 16 character limit as best we can.

Needed to tweak the bitbake-worker magic values to tell the
workers apart.

(Bitbake rev: 539726a3b2202249a3f148d99e08909cb61902a5)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-01-30 11:43:56 +00:00
Richard Purdie
384533bdb4 bitbake: server/process: Handle SIGTERM more gracefully
Currently if you send a SIGTERM to the bitbake UI process, the system basically
hangs if tasks are executing. This is because the server process doesn't
actually try any kind of shutdown before exiting.

This patch trys executing a stateForceShutdown command first, which is
enough to stop any active tasks before the system exits.

I also noticed that terminate can execute multiple times, once at SIGTERM
from the handler and once from the real exit. Double execution leads to
stack traces and potential hangs (writes to dead pipes), so ensure
the code only can run once.

With these fixes, bitbake much more correctly deals with SIGTERM to the
UI process.

(Bitbake rev: 1032ddddbe3241da02ebb3608a1c40f9123b9e80)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2015-09-09 14:27:55 +01:00
Richard Purdie
4b9de767f0 bitbake: event/server: Add _uiready flag to handle missing error messages
If you start and suspend a bitbake execution so the bitbake lock is held,
then try and run "bitbake -w '' X", you will see bitbake return an error exit
code but print no message about what happened at all.

The reason is that the -w option creates a "UI" which swallows the messages. The
code which handles this exit failure mode thinks a UI has printed the messages
and therefore doesn't do so.

This adds in an extra parameter to the UI registration code so that we
can figure out whether its a primary UI or not and base decisions on whether
to display information on that instead. This fixes the error shown above and
some bizarre failures on the Yocto Project Autobuilder.

[YOCTO #8239]

(Bitbake rev: d1d60a68c2de40c2984d5040d14251c1be121b0b)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2015-09-04 16:23:56 +01:00
Paul Eggleton
715d857174 bitbake: Fix default function parameter assignment to a list
With python you should not assign a list as the default value of a
function parameter - because a list is mutable, the result will be that
the first time a value is passed it will actually modify the default.
Reference:

http://docs.python-guide.org/en/latest/writing/gotchas/#mutable-default-arguments

(Bitbake rev: 7859f7388f2e3f675d0e1527cfde18625f36f637)

Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2015-08-19 18:05:44 +01:00
Richard Purdie
2ce9055e4f bitbake: server/process: Don't log BBHandledException
If we see a BBHandledException in the idle handler, the understanding
is the system handled it, printing a log and traceback is just confusing.

Therefore only print these in the cases where its an unknown/unhandled
exception.

(Bitbake rev: d88ecc2bc44dce8fd92ca3a2c0fd4124a5e464fa)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2015-06-23 11:57:58 +01:00
Richard Purdie
0467d53017 bitbake: cooker/server: Fix up 100% CPU usage at idle
The recent inotify changes are causing a 100% cpu usage issue in the
idle handlers. To avoid this, we update the idle functions to optionally
report a float value which is the delay before the function needs to be
called again. 1 second is fine for the inotify handler, in reality its
more like 0.1s due to the default idle function sleep.

This reverts performance regressions of 1.5 minutes on a kernel build
and ~5-6 minutes on a image from scratch.

(Bitbake rev: 0e0ba408c2dce14a0fabd3fdf61d8465a031495b)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2015-03-10 10:37:28 +00:00
Richard Purdie
793f2f2e69 bitbake: server/process: Fix select call
There was a report that bitbake -e | less would use 100% cpu when it shouldn't
really. The issue appears to be a bogus file descriptor in the select call. We
shouldn't be blocking if there is event data pending to a *reader* from server
context.

[YOCTO #7138]

(Bitbake rev: 8f166e1a0f3574ae7d1e917a8bb403b87bad15bf)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2015-01-23 11:33:54 +00:00
Richard Purdie
83ebcb1f0c bitbake: process: Ensure abnormal exits set an error level
(Bitbake rev: 8f5c1cdae1ee6ce04ae0d04d0b95bd80efbf7534)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2014-09-02 18:10:37 +01:00
Richard Purdie
d33cc89e53 bitbake: process: Further improve robustness against server shutdown
Currently, if an exception occurs in an event handler, the server shuts
down but the UI simply hangs. This happens in two places, firstly waiting
for events and secondly, sending events to a server which no longer exists.

The latter does time out, the former does not. These patches improve
both code sections to check if the main server process is alive and if not,
trigger things to shut down gracefully. This avoids the timeout in the
command sending case too.

This resolves various cases where the UI would simply hang indefintely.

(Bitbake rev: ac418e1112ff5f9c3157569316902f7a27fba4b4)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2014-08-23 09:31:42 +01:00
Richard Purdie
db50630948 bitbake: process: Deal with infinite looping of the server
Currently if an exception occurs, we just run the idle handler again and
again, usually looping indefintely. Chances are the exception that occurred
will keep occuring and this is not a good place to be.

This was breaking the autobuilders with gigabytes of logs.

At least improve things so the cooker shuts down gracefully when this happens.
Some trace of the original problem may still be present on the console too!

(Bitbake rev: 1f28d8d3311262427938180435b68f0a35c2b330)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2014-08-23 09:31:42 +01:00
Richard Purdie
324969e9e2 bitbake: server/process: Optimise latency when finishing idle functions
When idle functions finish, its likely we have some other work
to do, so don't sleep in the select call but instead, skip it.
This removes small amounts of latency in common commands.

(Bitbake rev: 069d6538f83b607cb46c6fe21bf6c596e8b99242)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2014-03-10 11:10:02 -07:00
Richard Purdie
5dcc20b954 bitbake: server/process: Drop unnecessary exit delay
When the server exits, we no longer appear to need this delay. This
is likely due to improvements in the various exit codepaths. There
is therefore no longer any point in taking the latency hit.

(Bitbake rev: 8e75ee29ae07e13f23525c5c6045fbf6cdbe7675)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2014-03-10 11:10:01 -07:00
Richard Purdie
efdc1598d2 bitbake: server/process: Use a pipe for quit events instead of Event()
Its not possible to notice the change of status of an Event() in
the select call we sleep in. It would be possible in python 3.3 but
for now use a pipe instead. This removes small latency when bitbake
commands finish since the system doesn't sit in the select call.

(Debugging these kind of issues is apparent by setting a long sleep
for the select call)

(Bitbake rev: def28239b0f0d5f1cf13214b263114a5328538b7)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2014-03-10 11:10:01 -07:00