Commit Graph

139 Commits

Author SHA1 Message Date
Richard Purdie
98f1b3d6ae bitbake: server/process: Add bitbake.sock race handling
We've seen cases where the bitbake.sock file appears to disappear but the
server continues to hold bitbake.lock. The most likely explaination is
that some previous build directory was moved out the way, a server there
kept running, eventually exited and removed the sock file from the wrong
directory.

To guard against this, save the inode information for the sock file and check
it before deleting the file. The new code isn't entirely race free but should
guard against what is a rare but annoying potential issue.

(Bitbake rev: b02ebbffdae27e564450446bf84c4e98d094ee4a)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2022-12-21 14:15:26 +00:00
Frank de Brabander
3c6753d03d bitbake: process: log odd unlink events with bitbake.sock
Log when the socket file already exists and is removed before
recreating a new socket.

Log when unlinking the socket file failed.

(Bitbake rev: cfd7c9899f988bab6d9fe7bbfbdb60603fb5ed34)

Signed-off-by: Frank de Brabander <debrabander@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2022-12-17 23:50:13 +00:00
Richard Purdie
77a9c9b66b bitbake: main/process: Add extra sockname debugging
We're struggling to understand how bitbake.sock can sometimes disappear
in live builds when we can't see where it could have been deleted.
This causes connection failures to the server and failed builds.

Add some extra debugging around the server log and client retry
log messages to give more information for the next time this issue
occurs.

(Bitbake rev: 376a516dc8c96727fd042ada65f803013601ee2d)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2022-12-13 23:26:28 +00:00
Richard Purdie
a9505a86fd bitbake: main/server: Add lockfile debugging upon server retry
We keep seeing server issues where the lockfile is present but we can't
connect to it. Reuse the lockfile debugging code from the server to
dump better information to the console from the client side when we
run into this issue. Whilst not pretty, this might give us a chance
of being able to debug the problems further.

(Bitbake rev: 22685460b5ecb1aeb4ff3436088ecdacb43044d7)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2022-12-09 13:22:11 +00:00
Richard Purdie
16bc168084 bitbake: server: Ensure cooker profiling works
The previous cleanups meant that when the cooker was started, profiling
was always disabled as configuration was sent to the server later and this
was too late to profile the main loop.

Pass the "profile" option over the server commandline so that we can
profile cooker itself again, the setting can now take effect early enough.

(Bitbake rev: c97c1f1c127ef3f8fbbd1b4e187ab58bfb0a73e5)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2022-11-20 08:31:28 +00:00
Richard Purdie
4a241d0cfb bitbake: server/process: Fix logging issues where only the first message was displayed
I realised only the first logging message was being displayed in a given
parsing process. The reason turned out to be the UI handler failing
with a "pop from empty list". The default handler was then lost and
no further messages were processed.

Fix this by catching the exception correctly in the connection writer code.

(Bitbake rev: d3e64f64525187f1409531a0bd99df576e627f7f)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2022-06-25 21:14:07 +01:00
Richard Purdie
6c9a516e76 bitbake: server/process: Avoid tracebacks at exit
In theory this should have been worked around but is still occurring. Add
it to the list of things to ignore when bitbake is shutting down.

Traceback (most recent call last):
  File "/usr/lib64/python3.9/threading.py", line 973, in _bootstrap_inner
    self.run()
  File "/home/pokybuild/yocto-worker/oe-selftest-fedora/build/bitbake/lib/bb/server/process.py", line 698, in startCallbackHandler
    event = self.reader.get()
  File "/home/pokybuild/yocto-worker/oe-selftest-fedora/build/bitbake/lib/bb/server/process.py", line 722, in get
    res = self.reader.recv_bytes()
  File "/usr/lib64/python3.9/multiprocessing/connection.py", line 221, in recv_bytes
    buf = self._recv_bytes(maxlength)
  File "/usr/lib64/python3.9/multiprocessing/connection.py", line 426, in _recv_bytes
    return self._recv(size)
  File "/usr/lib64/python3.9/multiprocessing/connection.py", line 384, in _recv
    chunk = read(handle, remaining)
TypeError: an integer is required (got type NoneType)'

(Bitbake rev: 7a28ac4fe478bee1e52e84412da9626495f9c6c7)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2022-06-10 13:40:02 +01:00
Richard Purdie
ce592bc9ac bitbake: server/process: Remove daemonic thread usage
We're seeing UI deadlocks occasionally and this is possibly due to the
use of a daemonic thread in the UI event queue processing. This thread
could terminate holding a threading Lock() which would cause issues
for the process when exitting.

Change the shutdown process to handle this more cleanly.

(Bitbake rev: f5ad8349a5dbff9824a89f5708cfd011d61888c9)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2022-06-08 21:53:15 +01:00
Richard Purdie
5940020cfb bitbake: server/process: Avoid risk of exception deadlocks
The open coded lock acquire/release in the UI event handler doesn't
cover the case an exception occurs and if one did, it could deadlock
the code. Switch to use 'with' statements which would handle this
possibility.

We have seen deadlocks in the UI at exit this so this removes a
possible cause.

(Bitbake rev: bd12792f28efd2f03510653ec947ebf961315272)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2022-06-08 21:53:15 +01:00
Richard Purdie
a2e0ed2cbf bitbake: server/process: Drop unused import
(Bitbake rev: 543315e6463f15ca7ab2b4ef3e8ed41bb4207ccf)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2022-04-21 21:00:35 +01:00
Richard Purdie
496cbc01ca bitbake: server/process: Disable gc around critical section
The python gc can trigger whilst we're holding the event stream lock
and when cleaning up objects, they can trigger warnings. This translates
into a new event which would then need the lock and we can deadlock.

Disable gc whilst we hold that lock to avoid this unfortunate and
problematic situation.

(Bitbake rev: 96a6303949cefd469bcf5ed250ff512271354357)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2022-04-03 17:51:26 +01:00
Richard Purdie
928bcb10a4 bitbake: cooker/process: Fix signal handling lockups
If a parser process is terminated while holding a write lock, then it
will lead to a deadlock (see
https://docs.python.org/3/library/multiprocessing.html#multiprocessing.Process.terminate).

With SIGTERM, we don't want to terminate holding the lock. We also don't
want a SIGINT to cause a partial write to the event stream.

I tried using signal masks to avoid this but it doesn't work, see
https://bugs.python.org/issue47139

Instead, add a signal handler and catch the calls around the critical section.
We also need a thread lock to ensure other threads in the same process don't
handle the signal until all the threads are not in the lock.

(Bitbake rev: a40efaa5556a188dfe46c8d060adde37dc400dcd)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2022-03-30 13:05:03 +01:00
Peter Kjellerstedt
279e754d86 bitbake: server/process: Correct a typo in a comment
(Bitbake rev: b4a157b2fe2fb481ffa40e0f32659d05dd6320c2)

Signed-off-by: Peter Kjellerstedt <peter.kjellerstedt@axis.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2022-03-28 13:33:28 +01:00
Richard Purdie
cff6c1a18d bitbake: server/process: Move threads left debug to after cooker shutdown
This debug is useful but the cooker shutdown or post_serve() may have cleanup
left so run after those.

(Bitbake rev: 1463fc0448d1a6a7265806a4a8b165b610dfb43f)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2022-03-26 09:27:43 +00:00
Richard Purdie
34e4eebc32 bitbake: lib/bb: Fix string concatination potential performance issues
Python scales badly when concatinating strings in loops. Most of these
references aren't problematic but at least one (in data.py) is probably
a performance issue as the issue is compounded as strings become large.

The way to handle this in python is to create lists which don't reconstruct
all the objects when appending to them. We may as well fix all the references
since it stops them being copy/pasted into something problematic in the future.

This patch was based on issues highligthted by a report from AWS Codeguru.

(Bitbake rev: d654139a833127b16274dca0ccbbab7e3bb33ed0)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2021-11-03 10:12:42 +00:00
Alexander Kanavin
fbbc0f7461 bitbake: bitbake: correct deprecation warning in process.py
(Bitbake rev: aff52fe21a0b27f6302555c1e52a864550eb46ce)

Signed-off-by: Alexander Kanavin <alex@linutronix.de>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2021-09-17 07:26:24 +01:00
Martin Jansa
b978f7c3a0 bitbake: cooker/process: Fix typos in exiting message
(Bitbake rev: 1ff1ea3880d293b14ce0fc65e3bc4c938d587a2f)

Signed-off-by: Martin Jansa <Martin.Jansa@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2021-09-01 18:56:25 +01:00
Richard Purdie
dc7ef0f896 bitbake: process: Improve traceback error reporting from main loop
Currently the code can just show nothing as the exception if there was a double
fault, which in this code path is quite likely. This leads to an error log
which effectively says "it failed" with no information about how.

Improve things so we get a nice verbose traceback left in the logs/output
which is preferable to no logs.

(Bitbake rev: e5782b71647d1eb6de53bde7bc4f6019a5589f21)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2021-08-06 06:34:58 +01:00
Joshua Watt
75f491e5e2 bitbake: server: Fix early parsing errors preventing zombie bitbake
If the client process never sends cooker data, the server timeout will
be 0.0, not None. This will prevent the server from exiting, as it is
waiting for a new client. In particular, the client will disconnect with
a bad "INHERIT" line, such as:

    INHERIT += "this-class-does-not-exist"

Instead of checking explicitly for None, check for a false value, which
means either 0.0 or None.

(Bitbake rev: 13e2855bff6a6ead6dbd33c5be4b988aafcd4afa)

Signed-off-by: Joshua Watt <JPEWhacker@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2021-07-20 19:00:32 +01:00
Richard Purdie
2f12a20935 bitbake: server/process: Handle error in heartbeat funciton in OOM case
We've seen cases where an OOM error causes bitbake server to hang:

9171 02:21:09.127810 Command Completed
Traceback (most recent call last):
  File "/home/pokybuild/yocto-worker/qemux86/build/bitbake/bin/bitbake-server", line 51, in <module>
    bb.server.process.execServer(lockfd, readypipeinfd, lockname, sockname, timeout, xmlrpcinterface)
  File "/home/pokybuild/yocto-worker/qemux86/build/bitbake/lib/bb/server/process.py", line 550, in execServer
    server.run()
  File "/home/pokybuild/yocto-worker/qemux86/build/bitbake/lib/bb/server/process.py", line 108, in run
    ret = self.main()
  File "/home/pokybuild/yocto-worker/qemux86/build/bitbake/lib/bb/server/process.py", line 242, in main
    ready = self.idle_commands(.1, fds)
  File "/home/pokybuild/yocto-worker/qemux86/build/bitbake/lib/bb/server/process.py", line 370, in idle_commands
    bb.event.fire(heartbeat, self.cooker.data)
  File "/home/pokybuild/yocto-worker/qemux86/build/bitbake/lib/bb/event.py", line 216, in fire
    fire_class_handlers(event, d)
  File "/home/pokybuild/yocto-worker/qemux86/build/bitbake/lib/bb/event.py", line 123, in fire_class_handlers
    execute_handler(name, handler, event, d)
  File "/home/pokybuild/yocto-worker/qemux86/build/bitbake/lib/bb/event.py", line 93, in execute_handler
    ret = handler(event)
  File "/home/pokybuild/yocto-worker/qemux86/build/meta/classes/buildstats.bbclass", line 182, in defaultrun_buildstats
    write_host_data(os.path.join(bsdir, "host_stats"), e, d, "interval")
  File "/home/pokybuild/yocto-worker/qemux86/build/meta/classes/buildstats.bbclass", line 160, in write_host_data
    output = subprocess.check_output(c.split(), stderr=subprocess.STDOUT, timeout=limit).decode('utf-8')
  File "/usr/lib/python3.6/subprocess.py", line 356, in check_output
    **kwargs).stdout
  File "/usr/lib/python3.6/subprocess.py", line 423, in run
    with Popen(*popenargs, **kwargs) as process:
  File "/usr/lib/python3.6/subprocess.py", line 729, in __init__
    restore_signals, start_new_session)
  File "/usr/lib/python3.6/subprocess.py", line 1295, in _execute_child
    restore_signals, start_new_session, preexec_fn)
OSError: [Errno 12] Cannot allocate memory

We need to wrap the calls in the same high level wrapper as idle function calls
and trigger an exit upon an unhandled exception.

(Bitbake rev: 74042b5b89d5a170013fc1a327ce3a6530fbf7d5)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2021-05-18 23:53:15 +01:00
Ross Burton
01066a584a bitbake: bitbake-server: ensure server timeout is a float
bitbake-server is spawned by process.py and passes the arguments it is
given to ProcessServer.  There's some type confusion here:

bitbake-server is called with a string representation of the timeout,
which may be None.  If the timeout is not set, pass 0 instead of None.

Inside bitbake-server a ProcessServer is created which expects the
timeout to be a float not a string, so always float() the value.

[ YOCTO #14350 ]

(Bitbake rev: c93ae1f861208f6d39fd15c84fbcd0e2b54331f5)

Signed-off-by: Ross Burton <ross.burton@arm.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2021-04-20 13:57:50 +01:00
Richard Purdie
0c0b236b4c bitbake: process: Show command exceptions in the server log as well
There are autobuilder logs where the server commands are failing
but we have no debug info in the server log. Improve this to try and
understand what is failing.

(Bitbake rev: 04d3a79226c9ea448b22f4efbab33876a72c9bdb)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2020-10-11 13:44:26 +01:00
Richard Purdie
ef21d08424 bitbake: server/process: Note when commands complete in logs
Its hard to tell from the server logs whether commands complete or not
(or how long they take). Add extra info to allow more debugging of
server timeouts.

(Bitbake rev: 56285ada585ec1481449522282b335bcb5a2671e)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2020-09-05 11:45:18 +01:00
Richard Purdie
707f99642a bitbake: server/process: Prefix the log data with pid/time information
Knowing which process printed which messages and the timestamp of the
message is useful for debugging, so add this. Ensure the log parsing
isn't affected by using search() instead of match().

(Bitbake rev: 1d043666710df1fa9d9586fd974c0371dd1514b0)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2020-09-05 11:45:18 +01:00
Richard Purdie
f71587695c bitbake: server/process: Ensure we don't keep looping if some other server is started
Showing "leftover process" messages when a new server has started and is being
used by some UI is horrible. Compare the PID data from the lockfile to
avoid this (and the ton of confusing log data it generates).

Also, move the time.sleep() call to be after the first lock attempt, which
reduces noise in the logs significantly.

(Bitbake rev: ce1897a31afb5a14997bc3d2f459b90d43eecb7d)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2020-09-05 11:45:18 +01:00
Richard Purdie
175e5f2702 bitbake: server/process: Don't show tracebacks if the lockfile is removed
lsof/fuser error if the file doesn't exist. It can be deleted by something
else so ignore this if it happens and loop.

(Bitbake rev: b100d22ce37b7548b50e59a71802bcc903acd6ea)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2020-09-05 11:45:16 +01:00
Richard Purdie
f6a40d457c bitbake: server/process: Ensure logging is flushed
The cookerlog output goes to a file and its misleading to look at it and
not have it up to date with what the cooker is actually doing. Ensure
written data is flushed.

Ultimately this should be using python's logging but that is for another
day, we need simple fixes right now.

(Bitbake rev: d95c3dd2b8ac50423976a7baf0a51e9580871761)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2020-09-05 11:45:16 +01:00
Richard Purdie
341c5ca419 bitbake: process: Avoid printing binary strings for leftover processes
The binary string printed into the output is ugly, parse this so the
linebreaks come out in the logs and make them much more readable (I
was misssing the information initially despite looking for it).

(Bitbake rev: c2dd8bb434d5738fedf9019651074b90affff3b2)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2020-09-05 11:45:16 +01:00
Mark Hatle
4cb33dae15 bitbake: process.py: Handle SystemExit exception to eliminate backtrace
With an invalid layer, the desired error output should be:

  ERROR: The following layer directories do not exist:
  ERROR:    /this_path_does_not_exist
  ERROR: Please check BBLAYERS in .../build-invalid-layer/conf/bblayers.conf

Instead we were met with a backtrace:

Traceback (most recent call last):
  File "/scratch1/fray/xilinx/poky/bitbake/bin/bitbake", line 36, in <module>
    cookerdata.CookerConfiguration()))
...
  File "/scratch1/fray/xilinx/poky/bitbake/lib/bb/cookerdata.py", line 267, in parseBaseConfiguration
    self.data = self.parseConfigurationFiles(self.prefiles, self.postfiles)
  File "/scratch1/fray/xilinx/poky/bitbake/lib/bb/cookerdata.py", line 358, in parseConfigurationFiles
    sys.exit(1)
SystemExit: 1

(Bitbake rev: 3a2503c785a5cd9dca0dc68c3aec31b4bec7684b)

Signed-off-by: Mark Hatle <mark.hatle@kernel.crashing.org>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2020-09-05 11:45:16 +01:00
Richard Purdie
a6709152e9 bitbake: process/knotty: Improve early exception handling
The new server startup code means exceptions can happen when we aren't
setup to show them to the user correctly, leading to ugly tracebacks.

Add in some special case handling of BBHandledException to at least
ensure that common case doesn't traceback and the user sees meaningful
output.

In the future, the logging setup can likely be improved, as can the way
runCommand handles exceptions, they all should likely become real
exceptions again on the UI side.

[YOCTO #14022]
[YOCTO #14033]

(Bitbake rev: 6059d0e77f60ddb679049bd34478f41b1ab7995d)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2020-09-02 16:01:05 +01:00
Richard Purdie
2b3bd34f9b bitbake: server/process: Fix typo in code causing tracebacks
(Bitbake rev: 14caa3d4e5615252b9453162183980044d896d2f)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2020-08-26 11:28:54 +01:00
Richard Purdie
8ecde1aa8f bitbake: process: Avoid bb.utils.timeout
I have a suspicion based on process traces that the flock() call is no
longer interrupted by SIGALRM and hence the timeout doesn't work. We
were relying on EINTR triggering around syscalls but python is likely
protecting us from that in modern versions.

Re-implement this code with a different mechanism which doesn't have
that potential issue.

(Bitbake rev: 8eb52afdfd4c3e6478d4f8cc56e99def3f1c924c)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2020-08-26 09:05:38 +01:00
Richard Purdie
392b2cf529 bitbake: server/process: Use sys.executable for bitbake-server
Using sys.executable ensures we're using the same python binary for the server
as the client, which is probably advisable. "bitbake-server" is left as the process
name as its more distinctive in process listings.

(Bitbake rev: 387a339b330e8122a62a148249beb3f064dd4e3d)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2020-08-26 09:05:38 +01:00
Frazer Clews
abc6f864b9 bitbake: lib: fix most undefined code picked up by pylint
Correctly import, and inherit functions, and variables.
Also fix some typos and remove some Python 2 code that isn't recognised.

(Bitbake rev: b0c807be5c2170c9481c1a04d4c11972135d7dc5)

Signed-off-by: Frazer Clews <frazerleslieclews@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2020-08-25 18:14:53 +01:00
Richard Purdie
7002d67de0 bitbake: server/process: Add bitbake-server and exec() a new server process
Trying to have a new python process forked off an original doesn't work
out well and ends up having race issues. To avoid this, exec() a new
bitbake server process. This starts with a fresh python interpreter
and resolves various atexit and other multiprocessing issues once
and for all.

(Bitbake rev: 9501dd6fdd7a7c25cbfa4464cf881fcf8c049ce2)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2020-08-25 18:14:53 +01:00
Richard Purdie
6bab132879 bitbake: server/process: Log extra threads at exit
Dump info into the logs if there are extra threads left at process exit
time.

(Bitbake rev: 1c9496797b753e67351bd5cb98ef2b8e9435d51e)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2020-08-25 18:14:53 +01:00
Richard Purdie
b9bbb5c7b7 bitbake: main/server/process: Drop configuration object passing
The first thing the UIs do is update the server config from the UI. We
can just rely upon that and start the server with a standard config,
removing the need to pass the confusing configuration object around
as well as configParams, which contains a similar copy of some of the
data.

This makes memory resident bitbake work the same way as the normal
mode, removing the opportunity for some class of bugs.

The xmlrpcinterface and server_timeout values are passed in at server
startup time now and there no longer a second option in the
configuration which is effective ignored once the server starts.

(Bitbake rev: 783a03330802e83c525c55522e3ee2a933bded3a)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2020-08-25 18:14:53 +01:00
Richard Purdie
a1c956ab4c bitbake: server/process: Move the socket code to server process only
The sock object isn't used client side so we can just created it in
the server process and save passing around the fd/object.

(Bitbake rev: ee5d2c92dcce89ccb701e028ffc6419eb315f5ce)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2020-08-25 18:14:53 +01:00
Richard Purdie
0bcc00ac51 bitbake: cooker: Defer configuration init to after UI connection
Currently we end up parsing the base configuration multiple times as
initially, the right settings haven't come from the UI. We can defer
this until later in startup using runCommand as a trigger.

The advantage to doing this is improved startup times and ultimately
we should be able to avoid the double parse of the base configuration.

(Bitbake rev: 3caa43b665604475d2c87ba505efb0b9fca9c2e9)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2020-08-25 18:14:53 +01:00
Richard Purdie
2976f43c22 bitbake: server/process: Add extra logfile flushing
There is the possibility data is being lost from the logfile due to
data buffering. Add in a couple of extra flush calls to ensure data
is being written out before the lock file is dropped.

Possible fix for [YOCTO #14000]?

(Bitbake rev: 596ea229a87d26a8e970c7ee77179519ad081fef)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2020-08-13 08:07:00 +01:00
Richard Purdie
b4919c7ad7 bitbake: server/process: Pass timeout/xmlrpc parameters directly to the server
Further cleanup, just pass these settings directly.

(Bitbake rev: ac2284357f1fc7044dac9c146fad218fc9906412)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2020-08-13 08:07:00 +01:00
Richard Purdie
a3448ad15e bitbake: server/process: Simplfy idle callback handler function
Having the idle callbacks abstracted via the configuration object
makes no sense. Its like this for historical reasons from the
multiple server backends but we don't need this now so simplfy.

(Bitbake rev: e56c49717355c9493b07d5fc80981a95ad8a0ec8)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2020-08-13 08:07:00 +01:00
Richard Purdie
0c91113c07 bitbake: server/process: Remove pointless process forking
We already call bb.daemonize.createDaemon() in BitBakeServer so the extra
multiprocessing.Process() appears to be totally unneeded and just an extra layer
of forking which confuses things. Remove it.

(Bitbake rev: d214e55c45f9733b3289138feec0ae3361a4a48b)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2020-08-13 08:07:00 +01:00
Richard Purdie
4cc76db0a0 bitbake: server/process: Account for xmlrpc connections
UI control can happen via the xmlrpc connection. Account for this when timing
out UI connections. This was causing issues for toaster on systems where it
couldn't parse the metadata within the timeout.

(Bitbake rev: fa85a8263971c25e67fa3b421c686a90e46acd87)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2020-07-28 12:40:11 +01:00
Richard Purdie
449e07c197 bitbake: server/process: Fix UI first connection tracking
We're only meant to be doing UI connection timeouts on the first connection
but haveui changes for each connection. We need to add a specific variable
to track this correctly and get the intended behaviour.

(Bitbake rev: e7c387c2e2fb2cc3ca1dc9d2029362909c326d72)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2020-07-28 12:40:11 +01:00
Richard Purdie
cc985986f9 bitbake: server/process: Fix note reference -> info
Its bb.note or logger.info, this avoids a backtrace.

(Bitbake rev: 82c534ca1a1313de067b0d79c79857e89fa2764a)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2020-07-21 16:57:43 +01:00
Richard Purdie
b0e2e954f2 bitbake: server/process: Ensure UI-less servers don't sit in infinite loops
If server startup is broken for some reason (e.g. lockfile issues)
and no UI connection is made, the server will just sit inifinitely
waiting.

Add a timeout upon startup in the non-memory resident case so that
such infinite waits are avoided. In the memory resident case, the
server wouldn't have shut down in the first place or will timeout
according to configuration.

Since any race may mean the socket file is no longer present, ensure
the unlink doesn't fault upon exit, thus ensuring any hashequiv or
PRServ is removed from memory, allowing all processes to exit
cleanly in such scenarios.

(Bitbake rev: 39888b750df12478e8bdea6727cca112dce1df85)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2020-07-13 12:53:22 +01:00
Richard Purdie
ab8ebfd35a bitbake: server/process: Fix a rare lockfile race
We're seeing rare occasional races on the autobuilder as if two server
processes have the lockfile at the same time. We need to be extremely
careful this does not happen.

I think there is a potential race in this shutdown code since we delete
the lockfile, then call unlockfile() which also tries to delete it.

This means we may remove a lock file now held by another process if we're
unlucky. Since unlockfile removes the lockfile when it can, just rely on
that and remove any possible race window.

An example cooker-deamonlog:

 --- Starting bitbake server pid 2266 at 2020-07-11 06:17:18.210777 ---
Started bitbake server pid 2266
Entering server connection loop
Accepting [<socket.socket fd=20, family=AddressFamily.AF_UNIX, type=SocketKind.SOCK_STREAM, proto=0, laddr=bitbake.sock>] ([])
Processing Client
Connecting Client
Running command ['setFeatures', [2]]
Running command ['updateConfig', XXX]
Running command ['getVariable', 'BBINCLUDELOGS']
Running command ['getVariable', 'BBINCLUDELOGS_LINES']
Running command ['getSetVariable', 'BB_CONSOLELOG']
Running command ['getSetVariable', 'BB_LOGCONFIG']
Running command ['getUIHandlerNum']
Running command ['setEventMask', XXXX]
Running command ['getVariable', 'BB_DEFAULT_TASK']
Running command ['setConfig', 'cmd', 'build']
Running command ['getVariable', 'BBTARGETS']
Running command ['parseFiles']
 --- Starting bitbake server pid 8252 at 2020-07-11 06:17:28.584514 ---
Started bitbake server pid 8252
 --- Starting bitbake server pid 13278 at 2020-07-11 06:17:31.330635 ---
Started bitbake server pid 13278
Running command ['dataStoreConnectorCmd', 0, 'getVar', ('BBMULTICONFIG',), {}]
Running command ['getRecipes', '']
Running command ['clientComplete']
Processing Client
Disconnecting Client
No timeout, exiting.
Exiting

where it looks like there are two server processes running which should not be.
In that build there was a process left sitting in memory with its bitbake.sock file
missing but holding the lock (not sure why it wouldn't timeout/exit).

(Bitbake rev: e1a7c1821483031b224a1570bfe834da755219cc)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2020-07-12 22:38:34 +01:00
Richard Purdie
968fcf4989 bitbake: server/process: Increase timeout for commands
We're running into this timeout on loaded autobuilders in situations
where things should otherwise succeed. Log a note in these
cases and continue to try for longer.

(Bitbake rev: e567743e70f426786ae54dcb5ab550748d9266e4)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2020-07-08 12:12:45 +01:00
Robert Yang
2cb6a75b5a bitbake: bitbake: server/process: Handle BBHandledException to avoid unexpected exceptions
The parseBaseConfiguration() raises bb.BBHandledException(), but
BitBakeServer() didn't handle it, so we always got unexpected exceptions when
there were errors. For example:

=== Case 1:
* Add "print "hello"' in base.bbclass' def oe_import() function
def oe_import(d):
    print "hello"
[snip]

$ bitbake -p
ERROR: Unable to start bitbake server (None)
ERROR: Last 60 lines of server log for this session (/buildarea1/lyang1/test_hy/bitbake-cookerdaemon.log):
  File "/buildarea1/lyang1/poky/meta/classes/base.bbclass", line 21
    print "hello"
                ^
SyntaxError: Missing parentheses in call to 'print'

<The first exception>
During handling of the above exception, another exception occurred:
<Tracebacks>

<The second exception>
During handling of the above exception, another exception occurred:
<Tracebacks>

<The third exception>
During handling of the above exception, another exception occurred:
<Tracebacks>
[snip]

Now it looks like:
$ bitbake -p
ERROR: Unable to start bitbake server (None)
ERROR: Server log for this session (/buildarea1/lyang1/test_hy/bitbake-cookerdaemon.log):
ERROR: Error in compiling python function in /buildarea1/lyang1/poky/meta/classes/base.bbclass, line 21:

The code lines resulting in this error were:
     0001:def oe_import(d):
 *** 0002:    print "hello"
     0003:    import sys
     0004:
     0005:    bbpath = d.getVar("BBPATH").split(":")
     0006:    sys.path[0:0] = [os.path.join(dir, "lib") for dir in bbpath]
SyntaxError: Missing parentheses in call to 'print' (base.bbclass, line 21)

=== Case 2:
* Add 'HOSTTOOLS += "hello"' to conf/local.conf:
$ bitbake -p
ERROR: Unable to start bitbake server (None)
ERROR: Server log for this session (/buildarea1/lyang1/test_hy/bitbake-cookerdaemon.log):
<Tracebacks>
[snip]
During handling of the above exception, another exception occurred:
[snip]
<Tracebacks>
ERROR: The following required tools (as specified by HOSTTOOLS) appear to be unavailable in PATH, please install them in order to proceed:
  hello

The error message is printed by bb.fatal() which raises bb.BBHandledException(),
but BitBakeServer() doesn't handle it, so we got it.

Now it looks like:
ERROR: Unable to start bitbake server (None)
ERROR: Server log for this session (/buildarea1/lyang1/test_hy/bitbake-cookerdaemon.log):
ERROR: The following required tools (as specified by HOSTTOOLS) appear to be unavailable in PATH, please install them in order to proceed:
  hello

No unexpected exceptions anymore.

[YOCTO #13267]

(Bitbake rev: 6e6865e6371dbd31a136eae64cc5b1fa5f5bee33)

Signed-off-by: Robert Yang <liezhi.yang@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-08-16 10:05:32 +01:00