For now, even if we have specified to skip the whole module/class via
command line, e.g., `oe-selftest -R gotoolchain', the class setup method
is still run. This at least results in unnecessary builds, and at worst
results in ERROR, if the setup method fails.
So improve the skipping mechanism to avoid class setup method to run
when specified to skip.
(From OE-Core rev: b0b79bf65f5e5e65958090a4a88622b42df896bf)
Signed-off-by: Chen Qi <Qi.Chen@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Currently `oe-selftest -R a' will skip 'archiver' tests. This is
not expected. Fix it so that the '-R' should be followed by actual
module/class/test names.
(From OE-Core rev: de3b070fc2ddd0b63a324679ec5adbe30142fc22)
Signed-off-by: Chen Qi <Qi.Chen@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The current codes to skip test cases are logically correct, but they
do not work correctly in reality. It does skip the tests as the command
line argument specifies, but the related information is wrong.
e.g.
$ oe-selftest -R archiver bblayers runtime_test
2019-06-17 09:24:53,764 - oe-selftest - WARNING - meta-selftest layer not found in BBLAYERS, adding it
2019-06-17 09:25:06,309 - oe-selftest - INFO - Adding layer libraries:
2019-06-17 09:25:06,310 - oe-selftest - INFO - /buildarea5/chenqi/SWAT/poky/meta/lib
2019-06-17 09:25:06,310 - oe-selftest - INFO - /buildarea5/chenqi/SWAT/poky/meta-yocto-bsp/lib
2019-06-17 09:25:06,310 - oe-selftest - INFO - /buildarea5/chenqi/SWAT/poky/meta-selftest/lib
2019-06-17 09:25:06,312 - oe-selftest - INFO - Running bitbake -e to test the configuration is valid/parsable
2019-06-17 09:25:10,521 - oe-selftest - INFO - Adding: "include selftest.inc" in /buildarea5/chenqi/SWAT/poky/build-selftest/conf/local.conf
2019-06-17 09:25:10,521 - oe-selftest - INFO - Adding: "include bblayers.inc" in bblayers.conf
2019-06-17 09:25:10,522 - oe-selftest - INFO - test_archiver_allows_to_filter_on_recipe_name (archiver.Archiver)
2019-06-17 09:25:10,522 - oe-selftest - INFO - ... skipped 'Skip by the command line argument "runtime_test"'
2019-06-17 09:25:10,522 - oe-selftest - INFO - Skip by the command line argument "runtime_test"
2019-06-17 09:25:10,523 - oe-selftest - INFO - test_archiver_filters_by_type (archiver.Archiver)
2019-06-17 09:25:10,523 - oe-selftest - INFO - ... skipped 'Skip by the command line argument "runtime_test"'
2019-06-17 09:25:10,523 - oe-selftest - INFO - Skip by the command line argument "runtime_test"
The archiver.Archiver.xxx tests should be skipped by 'archiver' command line
argument, not 'runtime_test'.
Change to use a function generator to achieve the desired effect. After the change,
the effect is as follows.
$ oe-selftest -R archiver bblayers runtime_test
2019-06-17 09:19:06,223 - oe-selftest - WARNING - meta-selftest layer not found in BBLAYERS, adding it
2019-06-17 09:19:19,598 - oe-selftest - INFO - Adding layer libraries:
2019-06-17 09:19:19,599 - oe-selftest - INFO - /buildarea5/chenqi/SWAT/poky/meta/lib
2019-06-17 09:19:19,599 - oe-selftest - INFO - /buildarea5/chenqi/SWAT/poky/meta-yocto-bsp/lib
2019-06-17 09:19:19,599 - oe-selftest - INFO - /buildarea5/chenqi/SWAT/poky/meta-selftest/lib
2019-06-17 09:19:19,602 - oe-selftest - INFO - Running bitbake -e to test the configuration is valid/parsable
2019-06-17 09:19:24,368 - oe-selftest - INFO - Adding: "include selftest.inc" in /buildarea5/chenqi/SWAT/poky/build-selftest/conf/local.conf
2019-06-17 09:19:24,368 - oe-selftest - INFO - Adding: "include bblayers.inc" in bblayers.conf
2019-06-17 09:19:24,369 - oe-selftest - INFO - test_archiver_allows_to_filter_on_recipe_name (archiver.Archiver)
2019-06-17 09:19:24,369 - oe-selftest - INFO - ... skipped 'Skip by the command line argument "archiver"'
2019-06-17 09:19:24,369 - oe-selftest - INFO - Skip by the command line argument "archiver"
2019-06-17 09:19:24,369 - oe-selftest - INFO - test_archiver_filters_by_type (archiver.Archiver)
2019-06-17 09:19:24,370 - oe-selftest - INFO - ... skipped 'Skip by the command line argument "archiver"'
2019-06-17 09:19:24,370 - oe-selftest - INFO - Skip by the command line argument "archiver"
2019-06-17 09:19:24,370 - oe-selftest - INFO - test_archiver_filters_by_type_and_name (archiver.Archiver)
2019-06-17 09:19:24,370 - oe-selftest - INFO - ... skipped 'Skip by the command line argument "archiver"'
2019-06-17 09:19:24,371 - oe-selftest - INFO - Skip by the command line argument "archiver"
2019-06-17 09:19:24,371 - oe-selftest - INFO - test_archiver_srpm_mode (archiver.Archiver)
2019-06-17 09:19:24,371 - oe-selftest - INFO - ... skipped 'Skip by the command line argument "archiver"'
2019-06-17 09:19:24,372 - oe-selftest - INFO - Skip by the command line argument "archiver"
2019-06-17 09:19:24,372 - oe-selftest - INFO - test_bitbakelayers_add_remove (bblayers.BitbakeLayers)
2019-06-17 09:19:24,373 - oe-selftest - INFO - ... skipped 'Skip by the command line argument "bblayers"'
2019-06-17 09:19:24,373 - oe-selftest - INFO - Skip by the command line argument "bblayers"
2019-06-17 09:19:24,373 - oe-selftest - INFO - test_bitbakelayers_createlayer (bblayers.BitbakeLayers)
2019-06-17 09:19:24,373 - oe-selftest - INFO - ... skipped 'Skip by the command line argument "bblayers"'
2019-06-17 09:19:24,374 - oe-selftest - INFO - Skip by the command line argument "bblayers"
[snip]
(From OE-Core rev: 50004f431a71c71af4acf0f25403fee5a8447eab)
Signed-off-by: Chen Qi <Qi.Chen@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
We continued to see encoding problems with ssh commands run in oeqa. After much
research the conclusion was we should use ignore the errors since some occasional bad
locale encoding is better than the unicode decoding issues we were seeing which crashed
large parts of tests.
(From OE-Core rev: b3bae8aba5b897c1afcc8bf4ad8929251812d2b5)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Currently setupClass errors were not being mapped back to the failing tests
and they were hence being marked as UNKNOWN and the test statistics were
inaccurate.
This is because whilst the errors were being encoded into the test results
stream, the decoder doesn't cope with an error outside a testStart event.
We patch in an addError handler to the outsideTest parser so that this
does get handled in a way similar to the non-concurrent case.
It would be nice if we didn't have to do this but there doesn't seem
to be any other way to fix this other than forking subunit.
We also make a minor change so another of our changes can cope with
tests without a start time.
(From OE-Core rev: 8f7352ed9c1a3e82689b842b7f87e469ebf2e48f)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The string format for subunit setupClass/setupModule failures is slightly
different, tweak the regex to correctly handle both cases.
(From OE-Core rev: 425fbba9348420df72de5c6292f50acaf2023d0b)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Instead of showing:
RESULTS - ptest.PtestRunnerTest.test_ptestrunner - Testcase 1600: UNKNOWN (32.30s)
map unexpectedSuccesses to PASSED and improve the way they're displayed. We
expect/allow ptest runner to fail but if it passes we should handle it correctly.
(From OE-Core rev: 7885939ca3dea9704185e93469fe515c17eb8017)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
This adds SPDX license headers in place of the wide assortment of things
currently in our script headers. We default to GPL-2.0-only except for the
oeqa code where it was clearly submitted and marked as MIT on the most part
or some scripts which had the "or later" GPL versioning.
The patch also drops other obsolete bits of file headers where they were
encoountered such as editor modelines, obsolete maintainer information or
the phrase "All rights reserved" which is now obsolete and not required in
copyright headers (in this case its actually confusing for licensing as all
rights were not reserved).
More work is needed for OE-Core but this takes care of the bulk of the scripts
and meta/lib directories.
The top level LICENSE files are tweaked to match the new structure and the
SPDX naming.
(From OE-Core rev: f8c9c511b5f1b7dbd45b77f345cb6c048ae6763e)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
These IDs refer to testopia which we're no longer using. We would now use the test
names to definitively reference tests and the IDs can be dropped, along with their
supporting code.
(From OE-Core rev: 8e2d0575e4e7036b5f60e632f377a8ab2b96ead8)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
This code really needs to be rewritten to not split potential
multibyte characters, for now work around it to avoid exceptions like:
File "/home/pokybuild/yocto-worker/qa-extras2/build/meta/lib/oeqa/core/target/ssh.py", line 211, in run
data = reader.read(1024, 4096)
File "/usr/lib64/python3.6/codecs.py", line 503, in read
newchars, decodedbytes = self.decode(data, self.errors)
UnicodeDecodeError: 'utf-8' codec can't decode byte 0x92 in position 0: invalid start byte
(From OE-Core rev: 17e87510378f2729208a8262695f28e1efe5eb4c)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Some CI pipelines might perform further processing of the test output
(for instance, to plot some metrics into a chart). However, Since `thud`
we switched away from the XML-based jUnit reporting, and at the same
time we lost the ability of collecting the stdout and stderr of the
various tests.
We now restore this functionality by adding `stdout` and `stderr` keys
to the JSON reports. This behavior is off by default; in order to enable
it, one must set the `TESTREPORT_FULLLOGS` variable in the bitbake
configuration.
(From OE-Core rev: fd0048630ece5b21efb3a79e97046be0ab2a1514)
Signed-off-by: Alberto Mardegan <amardegan@luxoft.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
This is particularly useful when setting up GL tests.
(From OE-Core rev: 167a46775059b782c6f82ce8c5a47b27262e95d4)
Signed-off-by: Alexander Kanavin <alex.kanavin@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The previous code assumed builddir and meta-selftest are in the same dir, but
this isn't always true, builddir can be anywhere, use get_test_layer() to
locate meta-selftest can fix the problem.
(From OE-Core rev: 56d2493a9adfcc47ae7e265439e05ff42cdbbbbf)
Signed-off-by: Robert Yang <liezhi.yang@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Enable qemu for slirp. Initialize Qemurunner with slirp. Setup ip
and port attribute to enable connection with qemu running with slirp.
[YOCTO#10713]
(From OE-Core rev: 815e05ac4abb42238d6fdeb15d7426759e059ce1)
Signed-off-by: Yeoh Ee Peng <ee.peng.yeoh@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Qemu do not use the ip input from external. It will
retrieve ip from QemuRunner instance and assign
ip value.
(From OE-Core rev: 14d99dc6c39c963ba3e0d9a30274846bd5369210)
Signed-off-by: Yeoh Ee Peng <ee.peng.yeoh@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Avoid an unclosed file per thread warning when running selftests concurrently
by closing the result stream.
(From OE-Core rev: 33a4a076e8aa72a872807332501e7f5ae1cee0e2)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Clean up the warning:
meta/lib/oeqa/core/loader.py:27: DeprecationWarning: inspect.getargspec() is deprecated, use inspect.signature() or inspect.getfullargspec()
_failed_test_args = inspect.getargspec(unittest.loader._make_failed_test).args
(From OE-Core rev: d2deb66830be2d44532fea3d5db763b57778252a)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
We want to see failures/errors listed last since this is the most easily
visible part of the log on consoles or autobuilder output and makes
human processing easier rather than having to scroll up and scan for
a single failure.
(From OE-Core rev: 7954b19020c28a4120bc1671aa81b9e1e2b05fa2)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Its useful to have the counts of success/failure/error/skipped at the end of the
results to allow for easier human reading of what happened.
(From OE-Core rev: 080d8900d470a8e7f929b0c5c2765ad461744fbb)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
This allows the ptest results from ptest-runner, run in an image to be
transferred over to the resulting json results output.
Each test is given a pass/skip/fail so individual results can be monitored
and the raw log output from the ptest-runner is also dumped into the
results json file as this means after the fact debugging becomes much easier.
Currently the log output is not split up per test but that would make a good
future enhancement.
I attempted to implement this as python subTests however it failed as the
output was too confusing, subTests don't support any kind of log
output handling, subTest successes aren't logged and it was making things
far more complex than they needed to be.
We mark ptest-runner as "EXPECTEDFAILURE" since its unlikely every ptest
will pass currently and we don't want that to fail the whole image test run.
Its assumed there would be later analysis of the json output to determine
regressions. We do have to change the test runner code so that
'unexpectedsuccess' is not a failure.
Also, the test names are manipuated to remove spaces and brackets with
"_" used as a replacement and any duplicate occurrences truncated.
(From OE-Core rev: a13e088942e2a3c3521e98954a394e61a15234e8)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
There is no point in adding empty log entries to the json result files, only
add them if there is log data.
(From OE-Core rev: da6aef0b6bd4655805f7f743858d4e04341f75ed)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
As part of the solution to replace Testopia to store testresult,
OEQA need to output testresult into single json file, where json
testresult file will be stored in git repository by the future
test-case-management tools.
The json testresult file will store more than one set of results,
where each set of results was uniquely identified by the result_id.
The result_id would be like "runtime-qemux86-core-image-sato", where
it was a runtime test with target machine equal to qemux86 and running
on core-image-sato image. The json testresult file will only store
the latest test content for a given result_id. The json testresult
file contains the configuration (eg. COMMIT, BRANCH, MACHINE, IMAGE),
result (eg. PASSED, FAILED, ERROR), test log, and result_id.
Based on the destination json testresult file directory provided,
it could have multiple instances of bitbake trying to write json
testresult to a single testresult file, using locking a lockfile
alongside the results file directory to prevent races.
Also the library class inside this patch will be reused by the future
test-case-management tools to write json testresult for manual test
case executed.
(From OE-Core rev: 00e03b5004f1eb6d59295544b3a8620504278f51)
Signed-off-by: Yeoh Ee Peng <ee.peng.yeoh@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Refactor the original _getDetailsNotPassed method to return
testresult details (test status and log), which will be reused
by future OEQA code to write json testresult.
Take the opportunity to consolidate and simplify the logic used
to gather test status and log within the TestResult instance.
(From OE-Core rev: 79ee7d1c371a86edeb61c99679985118da657e5d)
Signed-off-by: Yeoh Ee Peng <ee.peng.yeoh@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
skipIfInDataVar will skip a test if a value is in a certain variable.
(From OE-Core rev: 10b935c713748346aea6c36c2f41e0ae6c320821)
Signed-off-by: Richard Leitner <richard.leitner@skidata.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Its a pain to have to wait until oe-selftest finishes to see the
failures for example.
(From OE-Core rev: 4c499a1b10a0c2647b6a753b8f9cd934ae4ad0da)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Periodically we'd see:
NOTE: core-image-sato-1.0-r0 do_testsdk: ======================================================================
NOTE: core-image-sato-1.0-r0 do_testsdk: ERROR: broken-runner
NOTE: core-image-sato-1.0-r0 do_testsdk: ----------------------------------------------------------------------
NOTE: core-image-sato-1.0-r0 do_testsdk: testtools.testresult.real._StringException: Traceback (most recent call last):
File "/home/pokybuild/yocto-autobuilder/yocto-worker/nightly-mips/build/meta/lib/oeqa/core/utils/concurrencytest.py", line 122, in _run_test
test.run(process_result)
File "/usr/lib/python3.6/site-packages/subunit/__init__.py", line 1194, in run
protocol = TestProtocolServer(result, self._passthrough, self._forward)
File "/usr/lib/python3.6/site-packages/subunit/__init__.py", line 514, in __init__
stream = stream.buffer
AttributeError: '_io.StringIO' object has no attribute 'buffer'
which seems to occur if a result arrives before all the runner threads
have started. The runner's result handling changes sys.stdout to a buffer
temporarily which can be seen in other threads and it can sometimes fail.
Since the tests are running in a separate process we don't need this buffer
handling in the concurrent case so only set when not parallelising. The
concurrent class handle setting buffer mode internally.
(From OE-Core rev: e2c66aa0f74d19fbc7e56af1fb440ad6473d3a8e)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Without this, error output such as that in the teardown can be lost
and processes may recieve signals they're not expecting causing other
strange errors.
(From OE-Core rev: 1e3f44737a15feb3128ba7fc0dbe896dd8782e07)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
oe-selftest -r <invalid test name>
currently triggers a traceback. Ensure this doesn't happen and the user gets a
sensible error message.
(From OE-Core rev: f2042bf3638ed4edfb167e7f7d4be6da60997ead)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Checking if the dependency had any failure is unreliable, for example
if the underlying data doesn't get transferred and the list is empty,
success of the dependency is assumed.
Since we now have success data available, change the code to use it.
(From OE-Core rev: 4abba4c30d5a6163a968a119395a679e5e281ab4)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
This isn't present on modern distros by default and doesn't work with
testtools, needing multiple code paths in the code. Remove it in favour
of finding a better replacement for results collection/analysis.
(From OE-Core rev: 8001d933a8dc86004db014777f094d718086687d)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
There doesn't appear to be any reason we need this _results indirection
any more so remove it.
(From OE-Core rev: b618261811c48ff3b98eab1b340a8cd09ef183c6)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
We have several options for parallel processing in oeqa, parallel
execution of modules, threading and mulitple processes for the runners.
After much experimentation is appears the most scalable and least
invasive approach is multiple processes using concurrenttestsuite
from testtools. This means we can drop the current threading code
which is only used by the sdk test execution.
oeqa/decorator/depends: Remove threading code
Revert "oeqa/sdk: Enable usage of OEQA thread mode"
This reverts commit adc434c063.
Revert "oeqa/core/tests: Add tests of OEQA Threaded mode"
This reverts commit a4eef558c9.
Revert "oeqa/core/decorator/oetimeout: Add support for OEQA threaded mode"
This reverts commit d3d4ba902d.
(From OE-Core rev: a98ab5e560e73b6988512fbae5cefe9e42ceed53)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
This allows oe-selftest to take a -j option which specifies how much test
parallelisation to use. Currently this is "module" based with each module
being split and run in a separate build directory. Further splitting could
be done but this seems a good compromise between test setup and parallelism.
You need python-testtools and python-subunit installed to use this but only
when the -j option is specified.
See notes posted to the openedmbedded-architecture list for more details
about the design choices here.
Some of this functionality may make more sense in the oeqa core ultimately.
(From OE-Core rev: 326ababfd620ae5ea29bf486b9d68ba3d60cad30)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Currently some tests run in buffer mode and some don't. Those that don't can
corrupt stdout/stderr. Switch to using buffer mode everywhere so we're consistent.
If there is useful output on stdout/stderr, it will be displayed if the test
fails.
(From OE-Core rev: 978548c0abde2cb94c2782538552f39bdf2bf630)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The current code assumes if something isn't a failure of some
kind, it was a pass. When test case IDs weren't matching, this lead
to very confusing output where things would fail, then be listed as
passing.
This adds code to track successes, ensuring we don't end up in this
position again with unmatched entries being listed as UNKNOWN.
(From OE-Core rev: 4374c296d8963e4f6a1aa7bef7983ad0a1c2fcff)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
We can directly compare the test case IDs rather than representations,
then if we're using subunit to split the tests, the comparisions still
work as intended.
(From OE-Core rev: 72e5f46f75454ba4c445c65c1cbc616a9e72fc6e)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
When running testimage task for core-image-sato-sdk, the following
error appeared.
UnicodeDecodeError: 'utf-8' codec can't decode byte 0x82 at position 0: invalid start byte
Checking the codes, I found it's caused by setting a 1024 limit for the
read method of the StreamReader object.
Comments from the manual:
"""
The chars argument indicates the number of decoded code points or bytes to
return. The read() method will never return more data than requested, but
it might return less, if there is not enough available.
"""
When running `systemctl status --full' on target, this error occurs.
This patch increase the bytes limit to 4096 to fix the error.
(From OE-Core rev: f1fad60ae3be4450aca6058d5665fb10a9148b44)
Signed-off-by: Chen Qi <Qi.Chen@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The var might not be set, resulting in unexpected error.
RESULTS - multilib.MultilibTest.test_check_multilib_libc - Testcase 1593: ERROR
The above error is due to MULTILIBS being not set, which is the default
for OE. This patch fixes this problem.
Also, the debugging message in skipIfNotInDataVar is currently confusing.
Instead of
DEBUG: Checking if 'MULTILIBS' value is in 'multilib:lib32' to run the test
it should be
DEBUG: Checking if 'MULTILIBS' value contains 'multilib:lib32' to run the test
This patch also fixes it.
(From OE-Core rev: 3f5c678664a2bba43d99508779dc2ce227cf52a2)
Signed-off-by: Chen Qi <Qi.Chen@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Use 'cls' instead of 'obj' to better reflect that registerDecorator
actually serves as a class decorator.
(From OE-Core rev: e06e4c859e8be5225d80806a2ebe175f0b152fe1)
Signed-off-by: Chen Qi <Qi.Chen@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
OETestFilter is a subclass of OETestDecorator. It wants to make
use of @abstractmethod decorator. But such decorator requires
metaclass to be ABCMeta to have effect. So add it now to achieve
the designed behaviour.
Comments from python's manual:
"""
Using this decorator requires that the class's metaclass is ABCMeta
or is derived from it.
"""
(From OE-Core rev: 28c4fafb2322ea8c37bcd7710f22f46ef552a902)
Signed-off-by: Chen Qi <Qi.Chen@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
In previous OEQA, having 'auto' in TEST_SUITES results in executing
as many test cases as possible.
This behaviour is broken for now. From the codes in core/loader.py,
I can see that it tries to use another keyword 'all'. But in fact,
it does not work.
I've checked the current manual. The manual says using 'auto'.
Below is the current information in manual.
"""
Alternatively, you can provide the "auto" option to have all applicable
tests run against the image.
TEST_SUITES_append = " auto"
"""
So we should restore this behaviour. This patch does so.
Also, output warning message is some module is named as 'auto', as this
is a reserved keyword.
(From OE-Core rev: a65460a063a958cc887c756db5f7ab18e3f5a8c1)
Signed-off-by: Chen Qi <Qi.Chen@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Some test cases (eSDK.oeSDK*, runtime_test/*) does not match
with current regex, fix it accept all.
[YOCTO #12385]
(From OE-Core rev: 1ecf48fd286a77078451b67879a44f9c9dc7a894)
Signed-off-by: Leonardo Sandoval <leonardo.sandoval.gonzalez@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Since we are depercating the ELF image type, we should not test it
here either.
(From OE-Core rev: c0c10ae9f7e206ee156a68ddbed73c8820c37824)
Signed-off-by: Saul Wold <sgw@linux.intel.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The value could be False in which case we should pass that through.
(From OE-Core rev: d0a3379bbcbcd8153bd59ccdb56d40fff7ad6c6b)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
This ensures the console is kept clear of confusing output but that
the main logs contain good debugging information.
(From OE-Core rev: caeb5dcfbd3c1d71f8e0eb78b3dd45d5ce349d25)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The logger handling in oeqa was confused at best. This patch:
a) Passes in a logger through various qemu runner pieces
b) Uses that logger consistently in the code
c) Creates a logger for QemuRunner outside the bitbake namespace
meaning we don't conflict with the tinfoil logging changes
The result of this is more consistency. For runtime tests in testimage,
the logs always contain the debug info, nothing is shwon on the console.
For the oe-selftests, logs are intercepted and only shown if the test
fails.
(From OE-Core rev: 4ff678137a55b93c9ba2cbffda34335ba859f704)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
In general we don't need to see the output of runqemu however if it fails
we do. Use the buffer option that already exists in TestResult but allow
us to trigger it on a per test basis.
(From OE-Core rev: 86bef95bccbe9dad3dc3445ea365439861966179)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
For test modules in a package, e.g., oelib.license, running
`oe-selftest -r oelib.license` or `oe-selftest -r
oelib.license.TestSimpleCombinations` would fail with a message that
the specified test cases could not be found. This was due to the
parsing in _built_modules_dict(), which failed to distinguish between
<package>.<module>.<class> and <module>.<class>.<testcase> and treated
both cases as the latter.
(From OE-Core rev: 80db3d999ae26d298d9d5418a32b11a4f27af9d5)
Signed-off-by: Peter Kjellerstedt <peter.kjellerstedt@axis.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
