This can avoid UnicodeDecodeError error.
(From OE-Core rev: baa78420d8d8e716935852c9c7b749af0161395a)
Signed-off-by: Robert Yang <liezhi.yang@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
There might be wild strings when read from target (especially when
reading ptest results), replace the errors to avoid breaking the test.
Fixed: (Not always happen)
$ bitbake core-image-sato -ctestimage
[snip]
status, output = self.target.run('ptest-runner', 0)
File "/buildarea/lyang1/poky/meta/lib/oeqa/core/target/ssh.py", line 84, in run
status, output = self._run(sshCmd, processTimeout, True)
File "/buildarea/lyang1/poky/meta/lib/oeqa/core/target/ssh.py", line 55, in _run
status, output = SSHCall(command, self.logger, timeout)
File "/buildarea/lyang1/poky/meta/lib/oeqa/core/target/ssh.py", line 258, in SSHCall
run()
File "/buildarea/lyang1/poky/meta/lib/oeqa/core/target/ssh.py", line 236, in run
output = process.communicate()[0].decode("utf-8")
UnicodeDecodeError: 'utf-8' codec can't decode byte 0xc3 in position 4906: invalid continuation byte
[YOCTO #11547]
(From OE-Core rev: d0d2f892f0bed6adb5ec6fb59d64efcc97c83e19)
Signed-off-by: Robert Yang <liezhi.yang@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
A new command line argument (-R, which is the oposite of current -r) that allows
to run all test cases except the ones indicated through the command line.
Some command line examples:
* Run all except the distro test case:
$ oe-selftest -R distrodata
* Run all except the archiver test case and a single bblayers unit test
$ oe-selftest -R archiver bblayers.BitbakeLayers.test_bitbakelayers_add_remove
[YOCTO #11847]
(From OE-Core rev: e40eeaa790b95d9c25832405c0b0d5b3a0d0292b)
Signed-off-by: Leonardo Sandoval <leonardo.sandoval.gonzalez@linux.intel.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Since OEQA framework uses Python logging functionality to report test
results there is a class that wraps PyUnit writes into logging commands
(OEStreamLogger), so don't buffer the actual test execution to have
insight of what is currently executing.
This fix will change a little the test output format adding an '\n'
previous the test result, for example:
From:
test_nonmatching_checksum (lic_checksum.LicenseTests) ... ok
To:
test_nonmatching_checksum (lic_checksum.LicenseTests)
... ok
This is because the new line added by the PyUnit StreamLogger because
currently we don't have a manner to identify when a test execution
starts at report level (write msg).
[YOCTO #11827]
(From OE-Core rev: 4c2276469f58a88f864eb374c00dbbaace702de4)
Signed-off-by: Aníbal Limón <anibal.limon@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
If some test module/case is specified to run and isn't found the OEQA
framework didn't notice it, so complete the implementation using
modules_required and validate for the test case prescense.
Raise an exception when the test module/case required isn't found.
[YOCTO #11645]
(From OE-Core rev: e50b415aaaa1581473f85f0a8afa278b5f95129b)
Signed-off-by: Aníbal Limón <anibal.limon@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Python versions has different features from branches 3.4.x, 3.5.x and
3.6.x, i expected in wrong mode that was incremental for example changes
in 3.4.4 be in 3.5.x but that's not true.
The _make_failed_test internal method differs and is only available in
certain versions >= 3.4.4 and in 3.5.x and 3.6.x branches but not
realeses have been made including it.
So to avoid futher problems inspect the _make_failed_test and generates
function definition according what parameters are needed, the unique
supossition is that exception argument is always passed.
Related to,
http://git.yoctoproject.org/cgit/cgit.cgi/poky/commit/?id=d8380d098a290510b442a7abd2dd5a50cabf5844
(From OE-Core rev: b4740d2b325a80bcecc5e56dff9add9081fcd31b)
Signed-off-by: Aníbal Limón <anibal.limon@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Python unittest change the signature of the _make_failed_test
after python 3.4.4 don't pass the method name.
(From OE-Core rev: 767b68e6ca22512ff80e6fbc42154f3f0c2206c0)
Signed-off-by: Aníbal Limón <anibal.limon@linux.intel.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The small module name was added to support run a whole suite that
has more that 3 levels in the test case name, but this broke the
behaviour for use a full test case name.
[YOCTO #11632]
(From OE-Core rev: 9ab20ceb5801bee8dd8b218b3928720da5e1d403)
Signed-off-by: Aníbal Limón <anibal.limon@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The details of a test failure is upper on the unittest output
so don't log twice the actual failure.
[YOCTO #11622]
(From OE-Core rev: 0f2e81c2a4458ad0ec6bab2710952ac2c2bbf1af)
Signed-off-by: Aníbal Limón <anibal.limon@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Currently there was a restriction to only execute tests that's
inherits from OETestCase but in some circunstancies the features
from the OEQA framework isn't needed so we need to support
basic unittests.
[YOCTO #10828]
(From OE-Core rev: baac26f1b36e89e07637b738dd31ec7356f05a02)
Signed-off-by: Aníbal Limón <anibal.limon@linux.intel.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Our filtering allows to specify which tests to run using,
<module_name>.[test_class].[test_name]
But the module name logic was restricted to only accept one level,
for example: runtime_test vs oelib.types, to support multiple
submodules use only the first part for filtering.
This allows to run the whole tests in a module with more than tree
levels.
Due to the ambiguity on the test filtering options with test cases
with more than tree levels the supported sytnax is,
<module>
or
<module>.[submoduleN].[test_class].[test_name]
[YOCTO #11632]
(From OE-Core rev: cf2ee12b007e5570959ccfbb643159b21d90426e)
Signed-off-by: Aníbal Limón <anibal.limon@linux.intel.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Is better to log the summary at end to see in an easy way
the actual result of the test run.
[YOCTO #11622]
(From OE-Core rev: 4e3ab36e8c90abc740cce1ba31faf6595116e1e2)
Signed-off-by: Aníbal Limón <anibal.limon@linux.intel.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
After Python 3.5 the init method contains initialization of
_loading_packages set to avoid infinite loops in recursive
loading.
Fix,
Traceback (most recent call last):
File "/home/pokybuild/yocto-autobuilder/yocto-worker/nightly-oe-selftest/build/scripts/oe-selftest", line 70, in <module>
ret = main()
File "/home/pokybuild/yocto-autobuilder/yocto-worker/nightly-oe-selftest/build/scripts/oe-selftest", line 57, in main
results = args.func(logger, args)
File "/home/pokybuild/yocto-autobuilder/yocto-worker/nightly-oe-selftest/build/meta/lib/oeqa/selftest/context.py", line 214, in run
rc = self._internal_run(logger, args)
File "/home/pokybuild/yocto-autobuilder/yocto-worker/nightly-oe-selftest/build/meta/lib/oeqa/selftest/context.py", line 175, in _internal_run
self.tc.loadTests(self.module_paths, **self.tc_kwargs['load'])
File "/home/pokybuild/yocto-autobuilder/yocto-worker/nightly-oe-selftest/build/meta/lib/oeqa/core/context.py", line 51, in loadTests
self.suites = self.loader.discover()
File "/home/pokybuild/yocto-autobuilder/yocto-worker/nightly-oe-selftest/build/meta/lib/oeqa/core/loader.py", line 268, in discover
pattern='*.py', top_level_dir=path)
File "/usr/lib/python3.5/unittest/loader.py", line 341, in discover
tests = list(self._find_tests(start_dir, pattern))
File "/usr/lib/python3.5/unittest/loader.py", line 398, in _find_tests
full_path, pattern, namespace)
File "/usr/lib/python3.5/unittest/loader.py", line 473, in _find_test_path
self._loading_packages.add(name)
AttributeError: 'OETestLoader' object has no attribute '_loading_packages'
(From OE-Core rev: 17b832ba05d640570a773adbff2fa4e9ff83fce3)
Signed-off-by: Anibal Limon <anibal.limon@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
In order to maintain compatibility with oe-selftest, the
load_test_components needs to be re-used, so the script
executor needs to pass to only load components supported
by certain script (oe-test, oe-selftest).
(From OE-Core rev: d6b78ae711b93b4059690320cb8d821aaadd1684)
Signed-off-by: Aníbal Limón <anibal.limon@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
This pre runner will serve to allow Test components executes code
previously of the run a suite.
(From OE-Core rev: bafb7e221d40d7a87a02cec8a97d98eec7c23438)
Signed-off-by: Leonardo Sandoval <leonardo.sandoval.gonzalez@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
By default, the xml runner class prints out the docstring for every
unit test but it order to keep the same format as the standard
runner, avoid docstring output setting descriptions to False.
(From OE-Core rev: 05b16219698fbd30ec76697e1b3be6d31e8f8878)
Signed-off-by: Leonardo Sandoval <leonardo.sandoval.gonzalez@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The OETestLoader.modules must be an iterable object, otherwise
checking presence of specific modules with the 'in' operator fails
with the following command/error:
% oe-test core
.. - core - INFO - Running bitbake -p
Traceback (most recent call last):
File "..poky2/scripts/oe-test", line 108, in <module>
ret = main()
File "..poky2/scripts/oe-test", line 93, in main
results = args.func(logger, args)
File "..poky2/meta/lib/oeqa/core/context.py", line 235, in run
self.tc.loadTests(self.module_paths, **self.tc_kwargs['load'])
File "..poky2/meta/lib/oeqa/core/context.py", line 53, in loadTests
self.suites = self.loader.discover()
File "..poky2/meta/lib/oeqa/core/loader.py", line 204, in discover
pattern='*.py', top_level_dir=path)
File "/usr/lib64/python3.4/unittest/loader.py", line 275, in discover
tests = list(self._find_tests(start_dir, pattern))
File "/usr/lib64/python3.4/unittest/loader.py", line 339, in _find_tests
yield self.loadTestsFromModule(module)
File "..poky2/meta/lib/oeqa/core/loader.py", line 266, in loadTestsFromModule
and module.__name__ in self.modules \
TypeError: argument of type 'NoneType' is not iterable
(From OE-Core rev: 2bb02cd3d6ec6c2c22ab6b567e2985015655fae6)
Signed-off-by: Leonardo Sandoval <leonardo.sandoval.gonzalez@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
This will enable only run certain module tests and filter by
class and test name.
(From OE-Core rev: 0f7531803aa6ffef2b7c5be9acee3efe7d2974ef)
Signed-off-by: Aníbal Limón <anibal.limon@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
A common operation is to list tests, currently only selftest
support it, this changes enables this functionality into the
core framework.
(From OE-Core rev: 7e803f1a855d3091a772b13efd3cc8e9c0c766e9)
Signed-off-by: Aníbal Limón <anibal.limon@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Add needed tests to validate the OEQA Threaded mode, the remaining
parts are tested by the OEQA without Threaded mode.
- test_loader.py: Add a test to validate rules when creating the
list of test suites.
- test_decorators.py: Add oetimeout test because the threaded mode
uses Timer instead of signal.
[YOCTO #11450]
(From OE-Core rev: fb9d91ca34c1b5d3e0034f5135e71f964fca5b82)
Signed-off-by: Aníbal Limón <anibal.limon@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
In python signals are only allowed to delivery into the main thread,
to support the threading mode test if the runner is threaded and
use threading.Timer instead.
There are some considerations like SIGALRM interrupts the execution
after N seconds but the Timer only starts a Thread to notice the
timeout and the exception will be raised when the test run ends.
[YOCTO #11450]
(From OE-Core rev: 8ab201612e22493dc2509ba339a8f07ade611a54)
Signed-off-by: Aníbal Limón <anibal.limon@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The _skipTestDependency needs to know if the thread mode is
enabled because the _results are by thread.
[YOCTO #11450]
(From OE-Core rev: 257d9ef4531052ba8507771b58884c7d2b64143d)
Signed-off-by: Aníbal Limón <anibal.limon@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
This class sets the {loader,runner}Class to the threaded versions and
overrides loadTests method to be able specify the process_num.
[YOCTO #11450]
(From OE-Core rev: 3ec0d3825aa31219e0ccabc1f158d3a2f8adbda2)
Signed-off-by: Aníbal Limón <anibal.limon@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The OETestRunnerThreaded overrides the run method of OETestRunner
it recieves a list of suites to be executed by a ThreadPool.
The new run method handles the ThreadPool creation and the
OETestResultThreaded fill.
[YOCTO #11450]
(From OE-Core rev: 48b7a407d692e6c49c41b16f2bd11e8c3f47a421)
Signed-off-by: Aníbal Limón <anibal.limon@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The OETestResultThreadedInternal extends OETestResult to stores
results by Thread.
The OETestResultThreaded is a simple class that provides the
implementation of interfaces needed by outside like wasSuccesful,
stop, logSummary, logDetails.
[YOCTO #11450]
(From OE-Core rev: 8e71844fc4dd3fcc8a19f9d4c25aafb09c5525fe)
Signed-off-by: Aníbal Limón <anibal.limon@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
This method is to assign results into the TestContext, create
an internal one to support change implementation in Thread version.
[YOCTO #11450]
(From OE-Core rev: 8a6a9cb816d78e4cf71b79c35b579918d31053f2)
Signed-off-by: Aníbal Limón <anibal.limon@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The OEStreamLoggerThreaded overrides OEStreamLogger to redirect
the PyUnit output to a logger.
Instead of log every line when comes the OEStreamLoggerThreaded
will buffer the PyUnit output and write everything at end of every
suite execution to don't have mixed suite outputs.
[YOCTO #11450]
(From OE-Core rev: 87d3e5b70c52e5c7439afe4af5aa002522043e81)
Signed-off-by: Aníbal Limón <anibal.limon@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The threded module will implement Thread support into the OEQA
framework.
The OETestLoaderThreaded overrides discover to return a list of
suites every suite will be run by a separate Thread.
Some design considerations are:
- All the tests of a certain module needs to be run at one thread
because unittest framework needs to keep track of setUp{Module,
Class,} and tearDown{Module, Class,}.
- Tests that depends on other needs to be run at the same thread
because OEQA framework look at the status of dependant test to
decide if skip or not, this constraint can be change in the future
but a sync mechanishm is needed to implement between the Threads.
[YOCTO #11450]
(From OE-Core rev: b4ae730ee1d6003ff005148f741270e2878de484)
Signed-off-by: Aníbal Limón <anibal.limon@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Those methods are used to write in the log the results so
it makes sense to have defined at OETestResult because
is a format of the result itself.
[YOCTO #11450]
(From OE-Core rev: 33a783f59ed4e232f41f8b09dfa7955f2ddc2f80)
Signed-off-by: Aníbal Limón <anibal.limon@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
I override this method before for keep track of results and forget
to remove it, now isn't need.
(From OE-Core rev: 63606ffaaac1b84ddcad8a1c1006f8110050e20e)
Signed-off-by: Aníbal Limón <anibal.limon@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The OEStreamLogger class is used for redirect PyUnit output
to a certain logger so there is not need to expose at level
of OETestContext because only OETestRunner needs to know.
[YOCTO #11450]
(From OE-Core rev: 8787fba3df8acd9d2438669d20b1a5060caa9022)
Signed-off-by: Aníbal Limón <anibal.limon@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
It was found a case (see the bugzilla entry) where two runners were running at the same
second, creating identical test result folders, so one of them (the second runner)
was not able to create the folder because the other has already created it,
raising the following exception (many text was removed from log)
NOTE: Executing RunQueue Tasks
NOTE: Running task 1 of 2 (/home/pokybuild/yocto-autobuilder/yocto-worker/nightly-mips/build/meta/recipes-core/images/core-image-minimal.bb:do_testsdkext)
NOTE: Running task 2 of 2 (/home/pokybuild/yocto-autobuilder/yocto-worker/nightly-mips/build/meta/recipes-sato/images/core-image-sato.bb:do_testsdkext)
NOTE: recipe core-image-sato-1.0-r0: task do_testsdkext: Started
NOTE: recipe core-image-minimal-1.0-r0: task do_testsdkext: Started
.
.
The stack trace of python calls that resulted in this exception/failure was:
File: 'exec_python_func() autogenerated', lineno: 2, function: <module>
0001:
*** 0002:do_testsdkext(d)
0003:
File: '/home/pokybuild/yocto-autobuilder/yocto-worker/nightly-mips/build/meta/classes/testsdk.bbclass', lineno: 188, function: do_testsdkext
0184:
0185:testsdkext_main[vardepsexclude] =+ "BB_ORIGENV"
0186:
0187:python do_testsdkext() {
*** 0188: testsdkext_main(d)
0189:}
0190:addtask testsdkext
0191:do_testsdkext[nostamp] = "1"
0192:
File: '/home/pokybuild/yocto-autobuilder/yocto-worker/nightly-mips/build/meta/classes/testsdk.bbclass', lineno: 171, function: testsdkext_main
0167: except Exception as e:
0168: import traceback
0169: bb.fatal("Loading tests failed:\n%s" % traceback.format_exc())
0170:
*** 0171: result = tc.runTests()
.
.
File: '/usr/lib64/python3.5/os.py', lineno: 241, function: makedirs
0237: cdir = bytes(curdir, 'ASCII')
0238: if tail == cdir: # xxx/newdir/. exists if xxx/newdir exists
0239: return
0240: try:
*** 0241: mkdir(name, mode)
0242: except OSError:
0243: # Cannot rely on checking for EEXIST, since the operating system
0244: # could give priority to other errors like EACCES or EROFS
0245: if not exist_ok or not path.isdir(name):
Exception: FileExistsError: [Errno 17] File exists: '/home/pokybuild/yocto-autobuilder/yocto-worker/nightly-mips/build/build/TestResults_20170409130114'
[YOCTO #11318]
(From OE-Core rev: 10aa09d39c1b70ce2a88f59601f1f8dbc96ed817)
Signed-off-by: Leonardo Sandoval <leonardo.sandoval.gonzalez@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Underscore modules are meant to be run only when manually added to the test
suite, so far another mechanisms are in place to make this happen with
runtime, sdk, and esdk (mostly in test* bbclasses).
This will add such functionality in the core framework so other specific
frameworks can take use this without adding something else.
[YOCTO #10980]
(From OE-Core rev: 2c6eac774768aa610a8b3784483b9e90fb629c2d)
Signed-off-by: Mariano Lopez <mariano.lopez@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
If importing a test with the same name as a built-in module,
it will silently import the built-in and check for tests in
built-in module. This happened with syslog module in debian
based machines, so add a raise to avoid this behavior.
[YOCTO #10978]
(From OE-Core rev: d9548f981448307b042807373e469f0d0b110bfe)
Signed-off-by: Mariano Lopez <mariano.lopez@linux.intel.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
With this we get the class that is actually having the problem,
not just a TypeError with an unknown class causing the error.
(From OE-Core rev: d6ff4891376417504018af27e8e729a412feeeea)
Signed-off-by: Mariano Lopez <mariano.lopez@linux.intel.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The idea on getTarget is to use kwargs to send custom variables
to different targets, instead of this, a new variable was added
(just used for custom targets) and this broke testexport. So
in order to fix it, just add the custom variable to kwargs.
This fixes the use of getTarget() in testexport class that was
introduced in 1dc8010afd71fe46fb28bb86fb7c07a5fbd3d7cf.
(From OE-Core rev: cf138029a1f18f991fc7a28c81d85884942e9d56)
Signed-off-by: Mariano Lopez <mariano.lopez@linux.intel.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Without this an exception would occur
(From OE-Core rev: d1bda068d4bc0ad3ff81ab1f104dc12cedd20b3f)
Signed-off-by: Juro Bystricky <juro.bystricky@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Add 'elf' as also supported by OEQemuTarget.
(From OE-Core rev: c1ec42ee9beb1603012db5cf990cdbf1596b2a3d)
Signed-off-by: Juro Bystricky <juro.bystricky@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Change conditional to avoid losing stack traces when log
into the logger. The logic only needs to don't log empty lines.
(From OE-Core rev: 545dd180cc27a69732f2332565038bd06d01c014)
Signed-off-by: Aníbal Limón <anibal.limon@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The run-tests option is optional so if isn't specified set
to None instead of crash on split().
(From OE-Core rev: 877732406cad2ee4bf11293dc8ccddc2998ffbaf)
Signed-off-by: Aníbal Limón <anibal.limon@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
This is a complementary fix to commit 7c552996:
[ meta: remove True option to getVar calls ]
it intended to remove all True option to getVar calls, but there are
still some remaining.
Search made with the following regex: getVar ?\((.*), True\)
(From OE-Core rev: 87d03ffe03d6f01e360bfd51714be96e62506e0a)
Signed-off-by: Ming Liu <peter.x.liu@external.atlascopco.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
With the change to the new framework data store dependecy was
removed, instead a new file is generated and used in testimage.
When testing builds from the autobuilders the test data values
are from the autobuilder, including the paths.
Some tests require paths to current environment in order to run,
this commit will update such paths and fix the error of running
images donwloaded from autobuilders.
[YOCTO #10964]
(From OE-Core rev: 26ad5105fc2ce03b7ee8ecc6911fd40a52bd573a)
Signed-off-by: Mariano Lopez <mariano.lopez@linux.intel.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
This allows to use the package install feature with
the new OEQA framework.
[YOCTO #10234]
(From OE-Core rev: 077dc19445574457769eb4f231de97e8059cb75e)
Signed-off-by: Mariano Lopez <mariano.lopez@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
This will add functions to get module file path from a test
case or a complete suite.
[YOCTO #10234]
(From OE-Core rev: 11dd49cb9faaf6588f045083ab5bd6679e2eafc1)
Signed-off-by: Mariano Lopez <mariano.lopez@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
This adds qemu target to be used in testimage.
It uses the current QemuRunner class in order
to boot and control qemu.
[YOCTO #10231]
(From OE-Core rev: 44d4e9d0bb31fbc28d8c1fad8860ff56e5ae043a)
Signed-off-by: Mariano Lopez <mariano.lopez@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
This add the option to select what tests to run in the
<module>[.<class>[.<test>]] format.
Currently it just support modules
(From OE-Core rev: 0050565d71cee5a0cc22660b2398e4334d4b6719)
Signed-off-by: Mariano Lopez <mariano.lopez@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
skipIfNotDataVar will skip a test if a variable doesn't have certain value.
skipIfNotInDataVar will skip a test if a value is not in a certain variable.
(From OE-Core rev: a81045f4e2b740173237f5ae4e80e2bc0b287faa)
Signed-off-by: Mariano Lopez <mariano.lopez@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
This adds a new decorator to check if image under tests has
certain DISTRO_FEATURE or IMAGE_FEATURE.
[YOCTO #10234]
(From OE-Core rev: 8740803d0696a0e97b72210a56f4fbd3135826ed)
Signed-off-by: Mariano Lopez <mariano.lopez@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
With this commit now it is possible to add targets with SSH for testing.
Most of it was imported for existing code, with improvements in log
handling.
[YOCTO #10234]
(From OE-Core rev: 3bc13548df4adb85f09467d200530a9c9f60da04)
Signed-off-by: Mariano Lopez <mariano.lopez@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The README has an introduction and explains how to run the test suite
and creates a new Test component.
(From OE-Core rev: 9d474172c47695be1a61538f5b87ca8d9db25fa7)
Signed-off-by: Aníbal Limón <anibal.limon@linux.intel.com>
Signed-off-by: Mariano Lopez <mariano.lopez@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Serves as an first input of how to the OEQA framework works.
(From OE-Core rev: 115f80adf1b230c5d0392e7833e9aeb274642bcb)
Signed-off-by: Aníbal Limón <anibal.limon@linux.intel.com>
Signed-off-by: Mariano Lopez <mariano.lopez@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The OETestContextExecutor class supports to use oe-test for run core
test component also is a base class for the other test components
(runtime, sdk, selftest).
Te principal functionality is to support cmdline parsing and execution
of OETestContext, the test components could extend the common options
to provide specific ones. The common options between test components
are test data file, output log and test cases path's to scan.
Also it initializes the logger to be passed to the whole OEQA framework.
[YOCTO #10230]
(From OE-Core rev: 039deafa5f2c8fab31b8373b39f8bc219377b893)
Signed-off-by: Aníbal Limón <anibal.limon@linux.intel.com>
Signed-off-by: Mariano Lopez <mariano.lopez@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
This test suite covers the current functionality for the OEQA
framework.
For run certain test suite,
$ cd meta/lib/oeqa/core/tests
$ ./test_data.py
(From OE-Core rev: 7d7d0dc3736fc12ae7848de2785f0066e6470cd1)
Signed-off-by: Aníbal Limón <anibal.limon@linux.intel.com>
Signed-off-by: Mariano Lopez <mariano.lopez@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The OETestDataDepends decorator skips a test case if a variable
isn't into test data (d).
The skipIfDataVar decorator skips a test case if a variable
has certain value.
(From OE-Core rev: 7dc519d20e835ee7693c31903e164c4bc0e5e598)
Signed-off-by: Aníbal Limón <anibal.limon@linux.intel.com>
Signed-off-by: Mariano Lopez <mariano.lopez@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The OETimeout provides support for specify certain timeout
in seconds for a test case, if the timeout is reach the SIGALRM
is sent and an exception is raised to notify the timeout.
[YOCTO #10235]
(From OE-Core rev: 1bf66a370361912e9950d7ff45e382c93622a169)
Signed-off-by: Mariano Lopez <mariano.lopez@linux.intel.com>
Signed-off-by: Aníbal Limón <anibal.limon@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
These two decorators stores certain TAG or ID for the test case
also provides support for filtering in loading step.
[YOCTO #10236]
(From OE-Core rev: 047af4ce864bbf98e2617b348ae9ccb77ac52871)
Signed-off-by: Mariano Lopez <mariano.lopez@linux.intel.com>
Signed-off-by: Aníbal Limón <anibal.limon@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The OETestDepends decorator could be used over test cases to
define some dependency between them.
At loading time sorting the tests to grauntee that a test case
executes before also raise an exception if found a circular
dependency between test cases.
At before test case run reviews if the dependency if meet, in the
case of don't it skips the test case run.
(From OE-Core rev: 2385bd3c8a7c012fd1cad5465ec7d34675552c75)
Signed-off-by: Aníbal Limón <anibal.limon@linux.intel.com>
Signed-off-by: Mariano Lopez <mariano.lopez@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
loader: Implements OETestLoader handling OETestDecorator
and filtering support when load tests. The OETestLoader is
responsible to set custom methods, attrs of the OEQA
frameowork.
[YOCTO #10231]
[YOCTO #10317]
[YOCTO #10353]
decorator: Add base class OETestDecorator to provide a common
way to define decorators to be used over OETestCase's, every
decorator has a method to be called when loading tests and
before test execution starts. Special decorators could be
implemented for filter tests on loading phase.
context: Provides HIGH level API for loadTests and runTests
of certain test component (i.e. runtime, sdk, selftest).
[YOCTO #10230]
(From OE-Core rev: 275ef03b77ef5f23b75cb01c55206d1ab0261342)
Signed-off-by: Aníbal Limón <anibal.limon@linux.intel.com>
Signed-off-by: Mariano Lopez <mariano.lopez@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
misc: Functions for transform object to other types.
path: Functions for path handling.
test: Functions for operations related to test cases and suites.
[YOCTO #10232]
(From OE-Core rev: 102d04ccca3ca89d41b76a8c44e0ca0f436b7004)
Signed-off-by: Mariano Lopez <mariano.lopez@linux.intel.com>
Signed-off-by: Aníbal Limón <anibal.limon@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
case: Defines OETestCase base class that provides custom
methods/attrs defined by the framework.
Every OETestCase instance contains a reference to the test
data (d), the test context (tc) and the logger.
Also implements _oe{SetUp,TearDown}Class for make special
handling of OEQA decorators and validations.
runner: Defines OETestRunner/OETestResult with support for RAW
and XML result logs.
exception: Custom exceptions related to the OEQA framework based
on class OEQAException.
[YOCTO #10230]
[YOCTO #10233]
(From OE-Core rev: c466086ccc4d4bb02d578a821cfb945945bfd529)
Signed-off-by: Aníbal Limón <anibal.limon@linux.intel.com>
Signed-off-by: Mariano Lopez <mariano.lopez@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
