The intent of the code was to catch one kind of error, it was actually swallowing
all exceptions and looping indefinitely. Fix it to work as intended.
This explains some mystery hangs we've been seeing.
(Bitbake rev: d73dbc3580faa1225d95ae4cefac4879ca3c1b2f)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
If the bitbake server recieved multiple connections, it currently closes
ones it can't handle (while its dealing with another). This is rather
antisocial behaviour which causes clients to quickly run through their
retries and abort.
Instead, queue any other connections until the current one is closed. This
way the client can decide when it wants to stop waiting for the server. If the
client is gone by the time we handle it, we handle that gracefully.
This also fixes a number of bugs in the connection handling where connections
which did drop early were badly handled causing tracebacks in the logs.
Also, handle queue incomming connections in a loop to ensure that the main
client handling doesn't starve that piece of the system.
This code was stress tested by running 50 connection attempts in parallel at
once, ensuring the code correctly handled them.
(Bitbake rev: 02845a561b38658ac3edf5cc9c34625ed860d34f)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Now that we set a timeout for the socket, it can return EWOULDBLOCK
if a signal or other event happens to wake up even if we don't timeout.
If this happens, retry the connection, else we simply see it quickly
loop through the retries and abort the connection in a very short
interval.
(Bitbake rev: d5b0a9a302ac0eafa4f797ac15ea77db87e82b3c)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The current value of 2 seconds has shown to be short in
wider testing.
(Bitbake rev: 8a1f2fcf35d61d83bbafa8fa3ae215fd5f51728b)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
We're seeing hangs in oe-selftest where server startup and shutdown are
racing. The assumption was a connect would timeout however no timeout is
set which can leave processes hanging. Set a short timeout for
the connection to avoid this.
(Bitbake rev: f02114cb70e8f6f1d32e19c02b758fe0aadecd19)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
On loaded production systems we've seen bitbake server take over
40s to start up. Increase the timeout to 90s which tries to avoid
failures in selftests.
The delays come from setting up the inotify watches (31s) so can't
really be avoided.
After 5s delay we now warn the user we're waiting for 90s so the
interactive exeperience shouldn't be much changed and its very
unlikely the user would see that anyway.
(Bitbake rev: 7a4e105093c3080c1087cd06abc2883852a23e28)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Small tweak to ensure these items are printed into the log since there
is other logging code which looks for the header and this makes it clearer
the server did start but is slow somewhere in startup.
(Bitbake rev: 7c0b84cac892744225fa0462f918ea9a79b356cb)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Its possible the read may return a smaller number of characters. Remove
the possibility by using a single character to signal the server is ready.
(Bitbake rev: 767c9596d129d21ddf5d2e00c55f0a0525c641dc)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
This simplifies the code and makes it easier to read but has the
same functionality.
(Bitbake rev: a0b9cfaf2d03fd047a79d32e668001718d02c4bf)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
10 log lines may not capture any full traceback, increase the number of
lines to 60 which covers most tracebacks.
(Bitbake rev: b8538c2088c199157404712a60a6740c926ce312)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
We're seeing issues where the server doesn't start with no logs as to why. Allow
the server to print the last 60 log lines just in case this shows us something useful
about what is failing.
(Bitbake rev: c8c80b404e38fe96f65d6314cd95f4069319f3d6)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
[YOCTO #12898]
There might be no bitbake-cookerdaemon.log, print a message for debugging.
(Bitbake rev: 4adc582d2df7fdb9e51c4ebb5e66bbd21165b4dc)
Signed-off-by: Robert Yang <liezhi.yang@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Traceback (most recent call last):
File "/home/pokybuild/yocto-worker/nightly-oe-selftest/build/bitbake/lib/bb/main.py", line 464, in setup_bitbake
server_connection = bb.server.process.connectProcessServer(sockname, featureset)
File "/home/pokybuild/yocto-worker/nightly-oe-selftest/build/bitbake/lib/bb/server/process.py", line 490, in connectProcessServer
if command_chan_recv:
UnboundLocalError: local variable 'command_chan_recv' referenced before assignment
(Bitbake rev: ad79fadd855f5c10242ed17e9e0f3eb0274f26d2)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
There were various problems in the server startup loggin:
a) stdout/stderr were not being flushed before forking which could potentially
duplicate output
b) there were separate buffers for stdout/stderr leading to confusing logs
where the entries could be reordered. This was particularly confusing
due to the separator the logs use to idendify new messages
c) an fd wasn't being closed during server startup meaning if the
server failed to start, the closed fd wasn't detected as it was held
open by the other reference
d) If the pipe was detected as being closed, the code incorrectly retried
server startup
e) The event code would remap stdout/stderr without flushing them, leading
to lose log messages
(Bitbake rev: 0594faa0b52ce5dbd948d836d88617d38d9862d1)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Currently the bitbake-cookerdaemon.log is placed into cwd. This seems like a
bad idea, we can place it in the build directory alongside the lockfile that
represents the server instead.
(Bitbake rev: 1620dbc48ffb2a882371cf9174a7b12648befc8a)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
It should be live/exited with server rather than cooker, fixed:
$ bitbake --server-only -T -1
Set MACHINE = "qemux86" in conf/local.conf
$ bitbake quilt
Set MACHINE = "qemuppc" in conf/local.conf
$ bitbake quilt
[snip]
ERROR: When reparsing /workspace1/lyang1/poky/meta/recipes-connectivity/openssl/openssl_1.0.2m.bb.do_package, the basehash value changed from c216f7f4fdd3cf4a0b10b975a636426c to d5a8e9431ab261381752d7a64c7b2fa9. The metadata is not deterministic and this needs to be fixed.
[snip]
This is because the server doesn't know local.conf is changed since the
notifiers are stopped, so it doesn't reparse, and then we would get the errors,
let the notifiers live/exited with server can fix the problem.
(Bitbake rev: a6a641cb9c5f3abe901b150da915372e295383d7)
Signed-off-by: Robert Yang <liezhi.yang@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
We were seeing cases where we could hit the 5s timeout on large/fast machines
running many different tasks at once. Increase this to 30s since the main
connection timeout path should no longer hit this slow path.
[YOCTO #12116]
(Bitbake rev: 131d4b8a5834781a93ed41e2967d8dcd4d80f29a)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Printing the last 10 lines of bitbake-cookerdaemon.log when the server
fails to start can sometimes result in printing the output from a
previous run, which could lead the user completely down the wrong path
in terms of the cause of the failure. Use a known start text containing
the time which we can then look for when scanning through the log, and
then grab the last 10 lines of that part instead.
Fixes [YOCTO #11903].
(Bitbake rev: 567f2cf1bc455b4f3cfb1cbd7f25145360b05a62)
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Issues in start are not being correctly detected by the current readypipe
code. Change it to use specific "ready" or "fail" messages to correctly
determine the correct failure mode and avoid bitbake seeming to hang
(it does currently timeout eventually).
[YOCTO #12062]
(Bitbake rev: 60d4791e3dd05729d2a2adf6f3b203c80d466a73)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Currently the original prefile and postfile passed when starting bitbake
server are 'sticky'. With the new memory resident model this doesn't make
sense as the server the system is started with isn't special.
This patch changes the code so the prefile/postfile are used if specified
on the commandline and not used otherwise. This makes the behaviour much
more predictable and expected and as an added bonus simplifies the code.
(Bitbake rev: 638d366234fad78f283d3a13a12b07cb0ccbe914)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Its possible for a connection to connect to the server as its shutting down
but before its removed the socket file. This patch:
a) Removes the socket file earlier to avoid connections.
b) Handles EOFError in initial connections gracefully. These occur if the
socket is closed during the server shutdown.
c) Ensure duplicate events aren't shown on the console. This makes debugging
these issues very very confusing.
With these changes the backtrace that was concerning users is hidden and the
server works as expected with a reconnect when it catches it in a bad state.
(Bitbake rev: f45196cf84669723382730944dddc7eaf50826f2)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
There is an interesting bug in the current code where a sync command
is not seen until the current async command completes, by which time
the UI may have shut down.
The reason is that if there are idle commands, we may not end up sleeping
in the select call at all, partiularly under heavy load like parsing.
Fix this by calling select with a zero timeout so that we see active
fds and know to read from them. This fixes various problems toaster was
having with the recent server changes.
[YOCTO #11898]
(Bitbake rev: bbcce58e824b2793abf50efa52db158ae16e23e4)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
We're seeing the server fail to start within 8s on heavily loaded
autobuilders so increase this timeout to 30s which should be more
than enough time.
(Bitbake rev: 8d4c120ec46d6d7a54947c64d33e18cb60b60505)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
There have been cases where the server could loop indefinitely and incorrectly
handle client disconnects. In the EOFError case, ensure a full disconnect
happens in the alternative disconnect path to avoid this.
(Bitbake rev: 5e267f14bb0155889615f567a920af4a37eb3c6b)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The server currently crashes if we hit an EOFError due to controllersock
still being in ready and the continue meaning ready isn't re-evaluated.
Setting the value to False can mean the shutdown code doesn't handle the
situation cleanly.
Clear ready to avoid the crash/loop instead and handle any OSError whilst
we're in here.
(Bitbake rev: 2bc47e887c3b41417edaa89a8708c223fd2085de)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Make BB_SERVER_TIMEOUT = -1 mean no unload forever.
(Bitbake rev: 923d5567be6fd9969ca74c166f36817ec09305e3)
Signed-off-by: Robert Yang <liezhi.yang@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Fixed:
$ export BB_SERVER_TIMEOUT=10000
$ bitbake --server-only
$ bitbake --status-only
[snip]
File "/buildarea/lyang1/poky/bitbake/lib/bb/server/process.py", line 472, in recvfds
msg, ancdata, flags, addr = sock.recvmsg(1, socket.CMSG_LEN(bytes_size))
OSError: [Errno 9] Bad file descriptor
And:
$ export BB_SERVER_TIMEOUT=10000
$ bitbake --server-only -B localhost:-1
$ bitbake --status-only # Everything is fine in first run
$ bitbake --status-only
[snip]
File "/buildarea/lyang1/poky/bitbake/lib/bb/server/process.py", line 472, in recvfds
msg, ancdata, flags, addr = sock.recvmsg(1, socket.CMSG_LEN(bytes_size))
OSError: [Errno 9] Bad file descriptor
This was because self.controllersock was not set to False, so it still ran
sock.recvmsg() when sock was closed.
And also need set command_channel to Flase, otherwise the
self.command_channel.get() will always run when EOF, and cause infinite loop.
(Bitbake rev: 7b739a38601b053d9bea4df2c0b44a952ab670c4)
Signed-off-by: Robert Yang <liezhi.yang@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
We have cases where the server is being started but we're not seeing any messages
from it. Add some earlier logging so we can try and better understand where
issues may be occurring.
(Bitbake rev: 10a6a48c0c5a2fe2051e90143e66075356853971)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
If the connection control socket and the command channel close together,
we can race and hit EOFError exceptions before we close the channel.
Reorder the code to handle this in the correct order and ignore the
EOFError exceptions as they mean the client is disconnecting and shouldn't
terminate the server.
(Bitbake rev: 974281ed72d8366baa16ee85f7e93970689b5956)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
This timeout path was commonly hit due to errors starting the server. Now we
have a better way to handle that, the retry logic can be improved and cleaned
up. This patch:
* Makes the timeout 5s rather than intervals of 1s with a message. Paul
noted some commands can take around 1s to run on a server which has just
been started on a loaded system.
* Allows a broke connection to exit immediately rather than retrying something
which will never work.
* Drops the Ctrl+C masking, we shouldn't need that anymore and any issues
would be better handled in other ways.
This should make things clearer and less confusing for users and is much cleaner
code too.
(Bitbake rev: 8633b7cd03cfaba3e0359aa5da22fc76b66768c7)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
If we reconnect to the server, stop leaking pipes and clean up
after ourselves.
(Bitbake rev: f41e4e971e807157be68cf4496580494b8b60643)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Currently if the server fails to start, the user sees no error message and
the server will be repeatedly attempted to be started until some longer
timeouts expire. There are error messages in the cookerdeamon log but
nobody thinks to look there.
Add in a pipe which can be used to tell the starting process whether the cooker
did actually start or not. If it fails to start, no further attempts can be
made and if present, the log file can be shown to the user.
[YOCTO #11834]
(Bitbake rev: 57000d44beb1aeba37dfc70782b0d6418943acc5)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
This cleans up the socket keep alive into better class structured code
and adds cleanup of the open file descriptors upon shutdown.
(Bitbake rev: 77fd3a3a29a569e212374b27aea742ddbaafcdd5)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Currently the monitoring thread exits with some error code or runs indefinitely. Allow
closure of the pipe its monitoring to have the thread exit cleanly/silently.
(Bitbake rev: 930d077637928213e13a07c78fee3bf7a8c37ebf)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Expose the underlying close() and fileno() methods which allow connection
monitoring and cleanup.
(Bitbake rev: f79187f4ebfad7969be47b429995e7f7a3e33c1e)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The warning message currently shown can occur more frequently than previously
if a previous bitbake server is shutting down and we're reconnecting to a new
server. Change it to a note message to match the higher level connection
logging retry messages and so as not to interfer with selftests.
(Bitbake rev: b7514340cd6a2753eb217b059229bb279c3849ec)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Now that there is only one server, this abstraction is no longer needed
and causes indrection/confusion. The server shutdown is also broken with
the cooker post_server calls happening too late, leading to "lock held"
warnings in the logs if PRServ is enabled.
Remove the abstraction and put the shutdown calls in the right order
with respect to the locking.
(Bitbake rev: c0ddde7cf680225127d6285685652b905ed176c3)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
There is no global var "configuration", so the old code hang at
self.bitbake_lock.write(), and nothing wrote to bitbake.lock. I didn't
figure out why it hang (but not print errors).
Reproducer:
$ bitbake -B localhost:-1 world -k
Check bitbake.log, there was nothing, now fixed.
(Bitbake rev: dbdd9010663cd8dcb328e1b6fb40e3a777293bc5)
Signed-off-by: Robert Yang <liezhi.yang@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
This changes the way bitbake server works quite radically. Now, the
server is always a process based server with the option of starting
an XMLRPC listener on a specific inferface/port.
Behind the scenes this is done with a "bitbake.sock" file alongside
the bitbake.lock file. If we can obtain the lock, we know we need
to start a server. The server always listens on the socket and UIs
can then connect to this. UIs connect by sending a set of three file
descriptors over the domain socket, one for sending commands, one for
receiving command results and the other for receiving events.
These changes meant we can throw away all the horrid server abstraction
code, the plugable transport option to bitbake and the code becomes
much more readable and debuggable. It also likely removes a ton of
ways you could hang the UI/cooker in weird ways due to all the race
conditions that existed with previous processes.
Changes:
* The foreground option for bitbake-server was dropped. Just tail
the log if you really want this, the codepaths were complicated enough
without adding one for this.
* BBSERVER="autodetect" was dropped. The server will autostart and
autoconnect in process mode. You have to specify an xmlrpc server
address since that can't be autodetected. I can't see a use case
for autodetect now.
* The transport/servetype option to bitbake was dropped.
* A BB_SERVER_TIMEOUT variable is added which allows the server
to stay resident for a period of time after the last client
disconnects before unloading. This is used if the -T/--idle-timeout
option is not passed to bitbake.
This change is invasive and may well introduce new issues however I
believe the codebase is in a much better position for further
development and debugging.
(Bitbake rev: 72a3dbe13a23588e24c0baca6d58c35cdeba3f63)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
In preparation for rewriting this code, expand the relatively useless
base classes into the code itself.
(Bitbake rev: a1c6151420d86bac658c08ae714647062edd6ef2)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The UI may want to change its event mask however to do this, it needs the
event handler's ID. Tweak the code to allow this to be stored and add
a command to query it.
Use the new command in the process server backend.
(Bitbake rev: f8cf2cb58b80ce74f756a11a9773b6b0e78d51ee)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
If a signal is sent like SIGWINCH the select could be interrupted
so ignore the InterruptError like in XMLRPC server [1].
[1]
http://git.yoctoproject.org/cgit/cgit.cgi/poky/tree/bitbake/lib/bb/server/xmlrpc.py#n307
(Bitbake rev: 96bb174325493764718c61c12c943c37b882cd61)
Signed-off-by: Aníbal Limón <anibal.limon@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Messages printed when no UI is connected (e.g. memres) are currently lost.
Use the existing queue mechanism to queue these until a UI attaches, then
replay them. This isn't ideal but better than the current situation of
losing them entirely.
(Bitbake rev: cb241fb8544dfb05646dbae1a1b04e17878a466c)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
You might think Queue.Queue.get(True, 0) would return an event immediately
if present and otherwise return. It doesn't, it immediately "times out"
and returns with nothing from the queue.
The behaviour we want is not to wait but return anything present which is
what .get(False) does so map to this.
This fixes some odd behaviour observed in some of the tinfoil selftests.
(Bitbake rev: 412bfab8721ea317898a1974f6a7a0d0bea763df)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Call explicity close in _writer to avoid fd leakage because isn't
called on Queue.close()
[YOCTO #10873]
(Bitbake rev: 12dfac442d2d5674198485fbeb708a01d215c576)
Signed-off-by: Aníbal Limón <anibal.limon@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
getVar() now defaults to expanding by default, thus remove the True
option from getVar() calls with a regex search and replace.
Search made with the following regex: getVar ?\(( ?[^,()]*), True\)
(a follow on patch to fix up a few recent introductions)
(Bitbake rev: f9961fd5beb31d5ab9656a5be59f7ab3effef2f0)
Signed-off-by: Joshua Lock <joshua.g.lock@intel.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
On terminating the connection to the server, we were disabling SIGINT -
and this is executed on the UI side. I'm not sure whether the intention
here was to undo the SIGINT disabling we did in the server, and it was
just a mistake that it disabled rather than restored and it's run on the
wrong side, or whether we wanted to stop the user from breaking out of
the shutdown code - the commit message provides no clues either way.
Regardless, we do not want to permanently disable Ctrl+C here - it's
legitimate to terminate the connection to the server and then
re-establish it within the same process; at least currently, devtool
modify by virtue of using tinfoil in two separate parts of the code does
this, and the result of this disabling is that during the second tinfoil
usage we can potentially be parsing all recipes without the ability to
easily interrupt the process.
(Bitbake rev: 58c60a951229dcbd8253863fb24228d046c23f6e)
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The intended usage is for recording current system statistics from
/proc in buildstats.bbclass during a build and for improving the
BB_DISKMON_DIRS implementation.
All other existing hooks are less suitable because they trigger at
unpredictable rates: too often can be handled by doing rate-limiting
in the event handler, but not often enough (for example, when there is
only one long-running task) cannot because the handler does not get
called at all.
The implementation of the new heartbeat event hooks into the cooker
process event queue. The process already wakes up every 0.1s, which is
often enough for the intentionally coarse 1s delay between
heartbeats. That value was chosen to keep the overhead low while still
being frequent enough for the intended usage.
If necessary, BB_HEARTBEAT_EVENT can be set to a float specifying
the delay in seconds between these heartbeat events.
(Bitbake rev: 7cf22ea057d28c54bd98dc1ab7a43402a29ff1f5)
Signed-off-by: Patrick Ohly <patrick.ohly@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
