testimage.bbclass starts qemu with the first image type found in
the IMAGE_FSTYPES list. It's weird: this ['wic', 'tar'] works but
this ['tar'. 'wic'] does not. If QB_DEFAULT_FSTYPE is defined,
this fstype is booted.
(From OE-Core rev: aedb6bf9b6ccf37f69372642bc4c5dbbca92d0d9)
Signed-off-by: Adrian Freihofer <adrian.freihofer@siemens.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Some CI pipelines might perform further processing of the test output
(for instance, to plot some metrics into a chart). However, Since `thud`
we switched away from the XML-based jUnit reporting, and at the same
time we lost the ability of collecting the stdout and stderr of the
various tests.
We now restore this functionality by adding `stdout` and `stderr` keys
to the JSON reports. This behavior is off by default; in order to enable
it, one must set the `TESTREPORT_FULLLOGS` variable in the bitbake
configuration.
(From OE-Core rev: fd0048630ece5b21efb3a79e97046be0ab2a1514)
Signed-off-by: Alberto Mardegan <amardegan@luxoft.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
This is particularly useful when setting up GL tests.
(From OE-Core rev: 167a46775059b782c6f82ce8c5a47b27262e95d4)
Signed-off-by: Alexander Kanavin <alex.kanavin@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
There are failures when multiple users run oe-selftest on the same
host:
PermissionError: [Errno 13] Permission denied: '/tmp/oe-saved-
tests/201812250324_qemu'
This is because /tmp/oe-saved-tests was created by user A, while user B tries
to write data in it, then the error will happen. This patch can fix the
problem.
Move the dumped data to ${LOG_DIR}/runtime-hostdump/ rather than
/tmp/oe-saved-tests/ to fix the problem.
(From OE-Core rev: e219fe5329599cd6c3682f521eaee3852a2c8980)
Signed-off-by: Robert Yang <liezhi.yang@windriver.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Fixed:
MACHINE = "qemux86"
QEMU_USE_KVM = "qemux86"
IMAGE_CLASSES += "testimage"
$ oe-selftest -r runqemu.RunqemuTests.test_boot_rootfs
[snip]
File "/buildarea1/lyang1/poky/meta/lib/oe/types.py", line 122, in boolean
raise ValueError("Invalid boolean value '%s'" % value)
ValueError: Invalid boolean value 'qemux86'
Now QEMU_USE_KVM can only be boolean, can not contain MACHINE any more, kvm
will be enabled if target_arch == build_arch or both of them are x86 archs.
(From OE-Core rev: 7c1a8a624cad8d967635c6cb5f99cf655bde3d44)
Signed-off-by: Robert Yang <liezhi.yang@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
When use simpleremote target the flash/boot process is executed
manually, the IMAGE_FSTYPES validation is only needed when execute
testimage against qemu.
The supported_fstypes comes from oeqa.core.target.qemu module.
(From OE-Core rev: e7dc5963adbacc091fe8943119262166977623ad)
Signed-off-by: Aníbal Limón <anibal.limon@linaro.org>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Boot parameter systemd.log_target=console affects command journalctl's
output and causes oe selftest case test_systemd_boot_time fail to pass.
| Error at obtaining the boot time from journalctl
| RESULTS:
| RESULTS - systemd.SystemdJournalTests.test_systemd_boot_time - Testcase -1: SKIPPED (0.74s)
systemd.log_target=console was introduced by oe-core commit a0bb649 and
work with parameter systemd.log_level to enable systemd debug.
systemd.log_level has been removed already, so remove systemd.log_target
too to make case test_systemd_boot_time pass.
(From OE-Core rev: caa776bdcf8ea34c857f45970370bf771075f4bc)
Signed-off-by: Kai Kang <kai.kang@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Add a variable called TEST_QEMUPARAMS in testimage.bbclass to make it
possible to pass parameters to qemu. This can be useful for e.g.
increasing the amount of RAM available during testimage runs.
(From OE-Core rev: 1a9163f5779d233c884c8fd50e0812eabab4fdf3)
Signed-off-by: Erik Botö <erik.boto@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Now that the hangs in httpservice are fixed we can let these tests
auto skip as appropriate.
(From OE-Core rev: 42a0d70291d551578e21f590fcb85ca72a78ccb5)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Use our own lsb function instead as used elsewhere by the codebase.
(From OE-Core rev: acac45a6fd604d28ef7c23d67482af3d7e8bcfe3)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Now the test markup of the development tools tests is complete, this
can be further tweaked to auto run the correct tests.
(From OE-Core rev: bd4f8d12fe1f9f2643ee9e68fa2bb981134294fb)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Now that the tests have correct markup to automatically determine which images
they can run against, clean up the default test suites logic to be simpler
and not image specific.
Some cleanup of the compiler tests still needs to be completed but this
is a good first step.
The only downside to this is more noise during testing as we now see
many skipped messages for simple images like core-image-minimal.
The auto type is being removed since it currently breaks badly due to the
socat mandatory inclusion from the meta-selftest layer which is a problem
which needs to be addressed seperately.
(From OE-Core rev: 4966bc33845752eb0aeae54b72e8ba0146a7ed52)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Tweak the preceeding commit to:
* Add STARTTIME to the identifier to make it unique
* Log DISTRO
* Use LOG_DIR
* Store the layer config in a more natural json format
* Drop '_' function prefixes
(From OE-Core rev: dda94cfa708cf3c6e5408d6dbcece360b4e8cd27)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
As part of the solution to replace Testopia to store testresult,
OEQA testimage need to output testresult into json files, where
these json testresult files will be stored into git repository
by the future test-case-management tools.
By default, json testresult file will be written to "oeqa"
directory under the "WORKDIR" directory.
To configure multiple instances of bitbake to write json testresult
to a single testresult file at custom directory, user will define
the variable "OEQA_JSON_RESULT_DIR" with the custom directory for
json testresult.
(From OE-Core rev: 5d135d4769b6bb60d575eb6ed196367f9e077cc4)
Signed-off-by: Yeoh Ee Peng <ee.peng.yeoh@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Ptest has dependencies on other tests so contain them in a
variable. This allows layers to not worry about maintaining the depends
themselves.
(From OE-Core rev: c5c5ad9030ec7ddaa6387e2ba52e9e37b9ea76a2)
Signed-off-by: Armin Kuster <akuster@mvista.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Currently only the dnf package manager tests are automatically added to the
list of tests to run. Improve the code to handle automatic addition of
the apt and opkg tests too.
(From OE-Core rev: c9169e4d1fa67289d5beb0e1f8fb5c54153c4242)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
If the opkg/apt tests are run without a package-index they will fail.
Trigger this here for now as a dependency until the code can be improved.
(From OE-Core rev: 7805eec0f522da2d2b6a08c8a07a9243e26c014a)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The current qemuall override overwrites other dependencies unintentionally.
Tweak the code to avoid this by appending to the variable.
(From OE-Core rev: fc8df45e6b27dd7a26b7bf88a06ffa831fcf075c)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
QEMU_USE_KVM can either be a boolean or a whitespace separated list
of kvm supported machines.
For the 'intel-corei7-64' machine, defined in meta-intel, kvm could not be
used as the 'x86' substring is not part of its machine name.
By changing the order of this 'or' statement and setting
the 'QEMU_USE_KVM' variable to 'intel-corei7-64', it is possible to run the
'testimage' task with kvm support successfully.
(From OE-Core rev: a22789253aa653dc50fb159b40910248c2f98dd4)
Signed-off-by: Emmanuel Roullit <emmanuel.roullit@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The testimage changes break the autobuilder. The plan is to revert these hacks
but due to various changes happening with the autobuilder, we need these for now
to keep things working until we can move to the new autobuilder codebase.
(From OE-Core rev: 309a02931779f32d1139cc1169a039cbe4638706)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The TEST_IMAGE interface has never particularly worked and image testing currently
gets configured manually.
This reworks the interface to better serve the needs of its users, replacing it
with TESTIMAGE_AUTO.
This does away with the need for the separate class due to better bitbake APIs for
changing tasks.
TESTIMAGE_AUTO will automatically boot under qemu any image that is built. It
also adds in dependencies so that any SDK for which testing is requested will
automatically be built first.
The code in bitbake.conf was error prone (e.g. testsdk wasn't considered), this
improves it to standardise on IMAGE_CLASSES as the standard configuration mechanism.
(From OE-Core rev: b34d44f3dfea8254826a46701a4fe3769a900434)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
testimage-auto is expected to run testimage task's codes automatically.
But in fact, it's currently missing some codes, including testimage_sanity
and create_rpm_index.
This leads to the problem of unexpected runtime failure of test_dnf_makecache.
The error message is as below.
RESULTS - dnf.DnfRepoTest.test_dnf_makecache - Testcase 1744: ERROR
This error is caused by the fact that create_rpm_index is not executed
before running the tests.
There's no reason why such codes should not be in testimage_main, so
move them into it.
(From OE-Core rev: fa7ba486ded13907f63f9300f66350ba2835a3f7)
Signed-off-by: Chen Qi <Qi.Chen@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Having 'auto' in TEST_SUITES will also run the 'dnf' test cases,
so also check it to determine whether to create rpm index or not.
This is to fix the following error when TEST_SUITES = "auto".
RESULTS - dnf.DnfRepoTest.test_dnf_makecache - Testcase 1744: ERROR
(From OE-Core rev: c1f4177848d25a9121f2a85da655ee414cd424b1)
Signed-off-by: Chen Qi <Qi.Chen@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The current behaviour of TEST_SUITES is very confusing.
setting: TEST_SUITES = ""
result: Execute all test cases.
setting: TEST_SUITES = "some_case_not_exist"
result: Error out with 'Empty test suite' message.
The expected behaviour of TEST_SUITES should be:
1. when 'auto' is in it, execute as many test cases as possible
2. when no valid test case is specified, error out and give user message
The first one is implemented by a previous patch.
The second one is fixed in this patch.
Also add debug message to show test cases to be executed. This is
for easier debugging.
(From OE-Core rev: 909568821fbad8a6a7034b10a2dc349a210fdfc6)
Signed-off-by: Chen Qi <Qi.Chen@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Add test cases to test “exclude” and “installroot“ options, also modify
the logic of filtering packages on the feed to have all the packages
needed by the tests.
[YOCTO #10744]
(From OE-Core rev: 1121806603c6f621d084b692216f3f616a0768dc)
Signed-off-by: Jose Perez Carranza <jose.perez.carranza@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
If no ptest packages are installed in the image, the test does nothing;
if ptest packages are installed in the image, then they should be
run without user having to enable that manually.
(From OE-Core rev: f57feab2727dca916744deb64825f3beaf07961d)
Signed-off-by: Alexander Kanavin <alexander.kanavin@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
So that both QEMU_USE_KVM = "True" and "1" will work.
[YOCTO #12343]
(From OE-Core rev: f28890ae2eb8a9cb2da39588e696a4b0c501bdf0)
Signed-off-by: Robert Yang <liezhi.yang@windriver.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Currently, the fact an error message is shown means the rest of the
task logs are suppressed. In this case we don't want that as it hides
the real errors and useful information. Therefore override this behaviour.
(From OE-Core rev: 7058ac23c5ce32fbdc4be6c05f85550e1fd9fe77)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
I have no idea why we didn't do this but it means the code has nowhere
to log to unless we do this. This means we can then use the logger
to log data to the task logs.
(From OE-Core rev: e241fb128840889acff79dbf3fae7bfaafcb5357)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
It's very important to add IMAGE_CLASSES += "testimage" in local.conf firstly,
otherwise the var like TEST_LOG_DIR (defined in testimage.bbclass) will not be
in testdata.json.
[YOCTO #11547]
(From OE-Core rev: 5663ed989f0af5b1c61c74288ec421cbca2261e7)
Signed-off-by: Robert Yang <liezhi.yang@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
buildiptables has been replaced with buildlzip
(From OE-Core rev: 57cd682e05f82b63090b0cbb4a1d8feb3bfbff23)
Signed-off-by: Khem Raj <raj.khem@gmail.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Is better to log the summary at end to see in an easy way
the actual result of the test run.
[YOCTO #11622]
(From OE-Core rev: 4e3ab36e8c90abc740cce1ba31faf6595116e1e2)
Signed-off-by: Aníbal Limón <anibal.limon@linux.intel.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
With the new filters by <module>.[class].[testname] the modules
arg is expected to be a list so use split into TEST_SUITES variable.
(From OE-Core rev: b588c5c65e55a31d61c86553158eef409cd71086)
Signed-off-by: Aníbal Limón <anibal.limon@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Those methods are used to write in the log the results so
it makes sense to have defined at OETestResult because
is a format of the result itself.
[YOCTO #11450]
(From OE-Core rev: 33a783f59ed4e232f41f8b09dfa7955f2ddc2f80)
Signed-off-by: Aníbal Limón <anibal.limon@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
qemu-native-helper has an additional task that needs to be run in order
for testimage to work. This task is usually run by default in a full
build but there are use cases where it might be skipped. This commit
adds the dependency explicitly.
Also, this commit adds a try/catch error message to make it clearer what
you need to do if you try to run testimage before you have built or
downloaded the image artifacts.
[YOCTO #11375]
(From OE-Core rev: 6e019537b9eb3af482e474a8cb248fe7312f4b58)
Signed-off-by: brian avery <brian.avery@intel.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Previously, QEMU_USE_KVM=True enabled the use of kvm only when "x86"
was in the MACHINE name. That is too limiting, because for example
intel-corei7-64 can also use kvm but it wasn't possible to enable that
without changing OE-core.
That traditional usage is still supported. In addition, QEMU_USE_KVM
can be set to a list of space-separated MACHINE names for which kvm is
to be enabled.
(From OE-Core rev: d5421dd00b9cf785fa77e77c6c739e8bd8822fa3)
Signed-off-by: Patrick Ohly <patrick.ohly@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
We currently fetch,configure,build, and install cvs as our test for the
sdk. cvs unfortunately, requires a default editor in order to run. The
change in 94790a8254 that checks to see if you have something like vi
installed is fragile since you may have a different default editor. This
patch switches from using cvs as a test to using cpio. cpio also uses
autotools so the functionality tested is equivalent.
(From OE-Core rev: 233d36b0382a8b2e430c3377e50885d1a0c3ba21)
Signed-off-by: brian avery <brian.avery@intel.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
This causes MBs of log messages which overload the standard 200kb syslog limit
used by busybox syslog by default. This means some tests which rely on messages
being detected in syslog, e.g.
rpm.RpmInstallRemoveTest.test_check_rpm_install_removal_log_file_size
fail (AssertionError: 54 not greater than or equal to 80 : Cound not find sufficient
amount of rpm entries in /var/log/messages, found 54 entries).
We enabled this to aid debugging of some systemd race issues, those are
now resolved so we can disable this. Leave the log level parameter as a
comment to save others having to look it up if they need debugging.
(From OE-Core rev: 84d4daffaebcb4816d597671b287329e8a472f59)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The idea on getTarget is to use kwargs to send custom variables
to different targets, instead of this, a new variable was added
(just used for custom targets) and this broke testexport. So
in order to fix it, just add the custom variable to kwargs.
This fixes the use of getTarget() in testexport class that was
introduced in 1dc8010afd71fe46fb28bb86fb7c07a5fbd3d7cf.
(From OE-Core rev: cf138029a1f18f991fc7a28c81d85884942e9d56)
Signed-off-by: Mariano Lopez <mariano.lopez@linux.intel.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
If for some reason (most likely incorrect TEST_SUITES var)
you end with an empty test suite, testimage will start the
target (qemu by default) and will run an empty suite, so
better have a check before starting the target.
[YOCTO #10979]
(From OE-Core rev: 8345f834a080737f03bee3f0c615dd9654401d93)
Signed-off-by: Mariano Lopez <mariano.lopez@linux.intel.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
This patch implements custom target loading for testimage, currently
missing due to major changes to the test framework.
Custom targets can be defined in various meta-layers, so we
need an extra path information in order to find them.
Any other additional info is retrieved as usual via the variables
TEST_TARGET and TEST_SUITES
(From OE-Core rev: 1dc8010afd71fe46fb28bb86fb7c07a5fbd3d7cf)
Signed-off-by: Juro Bystricky <juro.bystricky@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Debian based distros has a builtin syslog module so when
try to load tests using unittest it references the builtin
module instead of runtime/cases.
[YOCTO #10964]
(From OE-Core rev: 9923e3cdb58c2b3c54ec5fe99b2cec4cdc9fff92)
Signed-off-by: Aníbal Limón <anibal.limon@linux.intel.com>
Signed-off-by: Mariano Lopez <mariano.lopez@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
There are some missing suites on sato-sdk and lsb-sdk images so add it.
The createrepo-native needs to be built before run testimage in
order to create a repository from install packages.
The DL_DIR data also needs to be updated from the current bitbake
environment instead of use the value from testdata file.
[YOCTO #10964]
(From OE-Core rev: 42f3026b39519ba98ae3153329622ce670d57c09)
Signed-off-by: Mariano Lopez <mariano.lopez@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
