While hacking on this I got an Exception. It's better to define
variables also in python.
Signe:-off-by: Adrian Freihofer <adrian.freihofer@siemens.com>
(From OE-Core rev: 1ea225a86cdee4ed932ede509d3351d5aecae497)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
After the commit ad522ea6a6 ("runqemu: Let qemuparams override default
settings"), the order of the two "-serial" parameters when running the
qemu have been switched. The effect of this is that the logging thread
will use ttyS1 (of course can't capture the kernel boot message anymore),
and the test command will run on the ttyS0. So the output of the test
command may be mangled by the kernel message (such as call trace), and
let the test command produce a fake timeout error message. We can't fix
it by just adjusting the order of the threadport and serverport, since
it will break some machines such as qemuarm64 which use the virtio
serial. So using the tcpserial to setup both the threadport and
serverport.
[YOCTO Bug 13309]
(From OE-Core rev: 9f2005dee41b1ef5a0d1f7b69bcd6c8352dac016)
Signed-off-by: Kevin Hao <kexin.hao@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
This adds SPDX license headers in place of the wide assortment of things
currently in our script headers. We default to GPL-2.0-only except for the
oeqa code where it was clearly submitted and marked as MIT on the most part
or some scripts which had the "or later" GPL versioning.
The patch also drops other obsolete bits of file headers where they were
encoountered such as editor modelines, obsolete maintainer information or
the phrase "All rights reserved" which is now obsolete and not required in
copyright headers (in this case its actually confusing for licensing as all
rights were not reserved).
More work is needed for OE-Core but this takes care of the bulk of the scripts
and meta/lib directories.
The top level LICENSE files are tweaked to match the new structure and the
SPDX naming.
(From OE-Core rev: f8c9c511b5f1b7dbd45b77f345cb6c048ae6763e)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
self.msg is a str(), bootlog is b'' so this code clearly doesn't work. Add
in a decode since its being used as a string.
(From OE-Core rev: e7546759d3102b576974bbab52594796f084721e)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The 5s timeout for non-kvm is too short, especially when the load is high,
which leads to unexpected errors, so set timeout to 60s by default.
(From OE-Core rev: 8197d0f638a760fc03062c7a9009117d083d7ead)
Signed-off-by: Robert Yang <liezhi.yang@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
This is useful for debugging.
(From OE-Core rev: da527d73f5eeae0f29b5f99aab757491d3f87ec7)
Signed-off-by: Robert Yang <liezhi.yang@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Reorder the shutdown/teardown to avoid:
File "/home/pokybuild/yocto-worker/oe-selftest-ubuntu/build/meta/lib/oeqa/utils/qemurunner.py", line 224, in launch
op = self.getOutput(output)
File "/home/pokybuild/yocto-worker/oe-selftest-ubuntu/build/meta/lib/oeqa/utils/qemurunner.py", line 90, in getOutput
fl = fcntl.fcntl(o, fcntl.F_GETFL)
ValueError: I/O operation on closed file
(From OE-Core rev: 8e7d756862d2a8d62f3c87497d6d65ddb3c1b962)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
If runqemu fails it would leak an unclosed socket and file. Ensure we
close these in all cases to remove the resource warning.
(From OE-Core rev: ed80e46ccbc8fe8e9148d80723152066fa00ba28)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Enable qemurunner for slirp. Retrieved the ip & port from host machine
to connect to qemu from host machine.
[YOCTO#10713]
(From OE-Core rev: 1db6a6fc9cde28d0a29bcf6d24a8bfbe51d120b1)
Signed-off-by: Yeoh Ee Peng <ee.peng.yeoh@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The qemu shutdown can race with the signal handler removal leading to
confusing tracebacks on slower/loaded systems.
Remove the signal handler first before shutting down.
(From OE-Core rev: 1b63ce7bb83134f84bf07d1075d5ca0d5466ed3d)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
[YOCTO #12493]
Fixed:
- qemu started with pid 10000
- qemu exited unexpectedly
- The pid 10000 is re-used by another different process.
The is_alive() returned True in such a case because both qemu_pidfile and
/proc/10000 exist, but it's another process, this patch fixed the problem.
(From OE-Core rev: 78033af85698b4e112ae78e3140b241b1b3c06a3)
Signed-off-by: Robert Yang <liezhi.yang@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The warn method is deprecated. We should use the documented warning instead.
Quoting from the python's official doc:
"""
Note: There is an obsolete method warn which is functionally identical to warning.
As warn is deprecated, please do not use it - use warning instead.
"""
(From OE-Core rev: f467fd277eb77336097cfc0f5f329bdc8d0f70cb)
Signed-off-by: Chen Qi <Qi.Chen@windriver.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Minor grammar correction along with making the term 'login banner'
consistent throughout to make searching logs easier.
(From OE-Core rev: 54e739485faba8d80f5af15407905aaeda4767f4)
Signed-off-by: Mark Asselstine <mark.asselstine@windriver.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The current logging always assumes the boot timeout has expired yet
there is a second reason we might have ended up in a position where no
login banner was found, that being a socket disconnect. Add logging
for the disconnect case and make the timeout expiration conditional on
the timeout being exhausted.
(From OE-Core rev: 49403368ccf3e469ac111afa259a38cc11e0b688)
Signed-off-by: Mark Asselstine <mark.asselstine@windriver.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
When debugging issues when timeouts are involved it is always best to
have wall clock times included. This helps give confidence that the
timeout is in fact run down at the right rate and that no unexpected
events were the true cause of a premature running down of the
timeout. Having these times in old logs also helps when debugging
issues as we have a historic record as to what is a 'typical' time to
complete an action.
In addition to adding the wall clock times the time to 'login' is now
printed making it consistent with the time to 'qemu pid'.
(From OE-Core rev: 4a6364309547d77d1d7a94c48f7c51ceee2b5d1a)
Signed-off-by: Mark Asselstine <mark.asselstine@windriver.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
I have concerns that bad timing of the flow of data from the logger
might corrupt the output due to the way binary strings are handled
in qemurunner.
This simplifies the code to do the same thing it did before but much
more safely.
(From OE-Core rev: 1e87283e92a2765bb5d54d17138b208bc395953b)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The logger handling in oeqa was confused at best. This patch:
a) Passes in a logger through various qemu runner pieces
b) Uses that logger consistently in the code
c) Creates a logger for QemuRunner outside the bitbake namespace
meaning we don't conflict with the tinfoil logging changes
The result of this is more consistency. For runtime tests in testimage,
the logs always contain the debug info, nothing is shwon on the console.
For the oe-selftests, logs are intercepted and only shown if the test
fails.
(From OE-Core rev: 4ff678137a55b93c9ba2cbffda34335ba859f704)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Bitbake logs info messages to the console. These messages are really
there as debugging information. At the debug level, they will be shown
in failure logs and in the task logs but not on the console which
is what we want in this case.
(From OE-Core rev: e05b14ad5a3c5ac1656b6fffdcf7e58250638e96)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
If we don't remove the handler we end up with duplicate log messages
which is undesireable.
(From OE-Core rev: 4155a58aebe9c730be9ff1f68150f34d1cd472e5)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
There are cases where the 'while loop' waiting for login prompt fails
and the bootlog variable does not get populated, thus use the the new
qemurunner member (self.msg) which stores all output coming from the qemu
process.
[YOCTO #12113]
(From OE-Core rev: 39ffa0f3779305c5e8ef86fe4572e961c5912021)
Signed-off-by: Leonardo Sandoval <leonardo.sandoval.gonzalez@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
We need QEMU PID in order to access "/proc/<qemupid>/cmdline"
Having a valid QEMU PID does not mean we can access the proc entry
immediately, we need to wait for the /proc/<qemupid> to appear
before we can access it.
(From OE-Core rev: d2d069fa9910d1c7a94c898355a63fca03ec5ad8)
Signed-off-by: Juro Bystricky <juro.bystricky@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Ask QEMU to store the QEMU process PID in a file, this way we don't
have to parse running processes and analyze descendents.
This is done via QEMU command line argument "-pidfile".
[YOCTO #12001]
(From OE-Core rev: 67612dcd2a8a1aa1d683dddb0bd2f592886ff020)
Signed-off-by: Juro Bystricky <juro.bystricky@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Python unittest intercepts stdout and stderr however qemurunner sets
up a streamhandler before that interception occurs, hence the messages
spam the unittest output.
By moving the logging init to the class init time, we use the unittest
stdout/stderr and this means unittest can only show the log output upon
failure. This cleans up the selftest and testimage output whilst still
showing logging upon failure.
(From OE-Core rev: 9099cecc727fe0ae5f1559582426d30ba7a9f4d3)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
We're seeing timeouts in the autobuilder testing code. Increase these
timeouts to 120, print the length of time we're having to wait, change
the error messages to really be errors and don't print empty logs, its
not helpful, print a message about the empty log instead.
(From OE-Core rev: b87a33d9c86cc1d2ea196259020e1d3dc712ccef)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
That a utility function permanently changes the process environment is
bad style and leads to subtle, hard to debug problems.
For example, we had one oe-selftest which used runqemu() with an
override for DEPLOY_DIR_IMAGE. Another test then just called runCmd()
and ended up passing the wrong DEPLOY_DIR_IMAGE set earlier in
os.environ.
The approach used here is to pass the desired environment dict to the
launch() method as a new, optional parameter, which then gets passed
on to subproject.Popen(). The modified env variables do not get
logged, as before.
[YOCTO #11443]
(From OE-Core rev: cab20f3b2fe668a63c58b44f2ad797fed74226fe)
Signed-off-by: Patrick Ohly <patrick.ohly@intel.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
By default, QemuRunner avoids modifying the image files that it boots
into by enabling the qemu snapshot mode. However, some tests may want
to test changes that must persists across reboots, so this mode
should be optional.
This can be combined by copying the image file to a temporary location
first and then booting with that copy. It's also useful when testing
with additional drives attached to a virtual machine.
QemuTinyRunner doesn't use the snapshot parameter and therefore ignores
the new parameter.
Long term, a better way of passing these various configuration
parameters should be used, and perhaps QemuRunner and QemuTinyRunner
can be merged into one again to avoid code duplication. But for now
the patch follows the exiting style.
Also beware that QemuTarget.start() now acts in two different modes
(with or without explicit launch command), and depending on that mode
parameters like discard_writes must be ignored, i.e. not get passed to
launch().
(From OE-Core rev: 969d079a33a57f5a8f7af86d7bab04d35ab07584)
Signed-off-by: Patrick Ohly <patrick.ohly@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Sometimes, the OEQA utility code aborts with:
...
File ".../meta/lib/oeqa/utils/qemurunner.py", line 131, in start
return self.launch(launch_cmd, qemuparams=qemuparams, get_ip=get_ip, extra_bootparams=extra_bootparams)
File ".../meta/lib/oeqa/utils/qemurunner.py", line 259, in launch
sread, swrite, serror = select.select(socklist, [], [], 5)
InterruptedError: [Errno 4] Interrupted system call
strace shows that this is because of a SIGWINCH:
Connection from 127.0.0.1:52668
select(21, [20], [], [], {5, 0}) = ? ERESTARTNOHAND (To be restarted if no handler)
--- SIGWINCH {si_signo=SIGWINCH, si_code=SI_KERNEL} ---
This is related to some special conditions:
* whether qemu opens a graphical console window (enabled in Poky by default)
* where that window gets opened
* whether the window manager changes the size of the shell window (mine
is a tiling window manager and reorders and resizes windows automatically)
Ignoring the interrupted system calls avoids the problem. Code elsewhere (for example,
run() in ssh.py) already does the same thing.
(From OE-Core rev: 44fe106baf5fd5aebe26c5f28004e2b18d839b7c)
Signed-off-by: Patrick Ohly <patrick.ohly@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The launch() doesn't need runqemuparams, we need handle it in start().
(From OE-Core rev: 16400f2b8bffc4cae0263bb597522071299b46ca)
Signed-off-by: Chen Qi <Qi.Chen@windriver.com>
Signed-off-by: Robert Yang <liezhi.yang@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Modify the following files to test runqemu:
targetcontrol.py
utils/commands.py
utils/qemurunner.py
We need simulate how "runqemu" works in command line, so when test
"runqemu", the targetcontrol.py, utils/commands.py and
utils/qemurunner.py don't have to find the rootfs or set env vars.
[YOCTO #10249]
(From OE-Core rev: 9305d816bdf8837ea3a407091cb7f24a9a3ae8dc)
Signed-off-by: Robert Yang <liezhi.yang@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Some commands might need to run longer than the default timeout of
five seconds. If that occurred, run_serial() returned with a status
code of zero (sic!) and no other indication of what went wrong.
Now the timeout is configurable (with five still the default) and
an explicit warning ("<<< run_serial(): command timed out after 5 seconds without output >>>")
gets appended at the end of the data returned to the caller.
While at it, the logic for checking for the timeout was updated a bit
because both implementations could overshoot the timeout when entering
select() right before the final deadline.
(From OE-Core rev: accf0362f964cc9d6330b6e52e83d748d890521f)
Signed-off-by: Patrick Ohly <patrick.ohly@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Configured guest network interface through serial connection
when kernel is not run by qemu.
This should make it possible to test wic images with testimage.
[YOCTO #10833]
(From OE-Core rev: 2032d9be26b539bf867622c0090fb4696209eba9)
Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Parsed runqemu output to get guest network configuration
if it's not present in runqemu command line.
[YOCTO #10833]
(From OE-Core rev: d4d7ed48c1cff1351ddc2f60bcfa153c373a8ab8)
Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
qemu output can contain control characters. This cause qemurunner
API to crash when decoding the output to utf-8:
Traceback (most recent call last):
File "/usr/lib64/python3.4/threading.py", line 911, in _bootstrap_inner
self.run()
File "meta/lib/oeqa/utils/qemurunner.py", line 472, in run
threading.Thread.run(self)
File "/usr/lib64/python3.4/threading.py", line 859, in run
self._target(*self._args, **self._kwargs)
File "meta/lib/oeqa/utils/qemurunner.py", line 465, in threadtarget
self.eventloop()
File "meta/lib/oeqa/utils/qemurunner.py", line 526, in eventloop
self.logfunc(data)
File "meta/lib/oeqa/utils/qemurunner.py", line 77, in log
msg = msg.decode("utf-8")
UnicodeDecodeError: 'utf-8' codec can't decode byte 0xda in position 0:
unexpected end of data
Added errors='ignore' to decode call to fix this.
(From OE-Core rev: 4a46dd5190d97fdcb6297a0c1d8c824d425c4c51)
Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Added possibility to pass additional runqemu parameters
down the stack of APIs:
commands.runqemu -> QemuTarget.start -> QemuRunner.start
This will be used to pass ovmf parameter in testing of
efi wic images under qemu.
(From OE-Core rev: 5aa4b5a10fb8191cd3453d09701c8beeff9a952f)
Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
When runqemu fails, qemu-system process would keep running
and won't be killed, setpgrp() was used when runqemu was
a shell script but it seems it doesn't work always with python.
This would kill qemu-system explicity and to avoid leaving
it behind.
(From OE-Core rev: 9d2b1aa1bcfb2f1933a8eeb9470b4174d5da2f0d)
Signed-off-by: Mariano Lopez <mariano.lopez@linux.intel.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
This adds the missing sys module used by the child process
to exit. It seems the exception was cached in testimage and
selftest. It seems nobody noticed this because the module
is only used for sys.exit().
(From OE-Core rev: 66f66d1d763ff7bbaab9e8fcdf7fc882f2dfbb13)
Signed-off-by: Mariano Lopez <mariano.lopez@linux.intel.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
StreamHandler was added due missing log information on the console in
oe-selftest with Qemu Runner
(From OE-Core rev: a4e2df151af781edbcb6b0e17b51b5ed226bf77f)
Signed-off-by: Francisco Pedraza <francisco.j.pedraza.gonzalez@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
This logs the launch command that was used for runqemu while running -c
testimage. This way, if I'd like to easily launch qemu manually in
order to debug a failed test, I know what commmand was run to create
the qemu instance.
(From OE-Core rev: 34aa20c6f323bbf7ad53beb643126e4e03634708)
Signed-off-by: bavery <brian.avery@intel.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Rather than copying images, use the snapshot option to qemu. This fixes a regression
caused by the recent runqemu changes where the wrong images were being testes since
the image is copied without the qemuboot.conf file. This means the latest image is
found by runqemu rather than the specified one, leading to various confused testing
results.
It could be fixed by copying more files but use snapshot mode instead.
(From OE-Core rev: eab91997d415b0e690b3482749a32087e6a8b00a)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
There is no runqemu-internal any more.
(From OE-Core rev: 14bacf7203ab7a638b67eb143225d8c75bbb703d)
Signed-off-by: Robert Yang <liezhi.yang@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Using kvm can provide significant speedups when running qemux86* machines
on an x86* host. Enabled by using the new QEMU_USE_KVM variable.
[YOCTO #9298]
(From OE-Core rev: ebac2c8d1fcd09ebce0659a4abb445e4f1c18571)
Signed-off-by: Bill Randle <william.c.randle@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Add ability to specify extra_bootargs (kernel cmdline) in order to enable systemd
debug log in images that enables systemd init.
[YOCTO #9299]
(From OE-Core rev: 09d62551c289b5607341a4f9c46eecd6390ad774)
Signed-off-by: Aníbal Limón <anibal.limon@linux.intel.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Since the stream can contain invalid binary characters (e.g. from
ppc's bootloader) use surrogateescape decoding to ensure we do process
the character stream, else it can hang/timeout.
(From OE-Core rev: 28a0030430d4cfcaf5dfc3e71bda07cdbfbbf4a7)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
This patch contains all the other misc pieces of the transition to
python3 which didn't make sense to be broken into individual patches.
(From OE-Core rev: fcd6b38bab8517d83e1ed48eef1bca9a9a190f57)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Converted str to bytes before sending to socket.
Converted bytes to str after receiving from socket.
This should fix TypeError: 'str' does not support the buffer interface
for qemurunner.run_serial method.
(From OE-Core rev: 210e290c9251839dc74e3aabdcea3655dd707a50)
Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Iterators now return views, not lists in python3. Where we need
lists, handle this explicitly.
(From OE-Core rev: caebd862bac7eed725e0f0321bf50793671b5312)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
