Commit Graph

49 Commits

Author SHA1 Message Date
Christos Gavros
de51bc3cd9 oeqa/selftest: add a newline in local.conf (newbuilddir)
If the build-st/conf/local.conf does not end with a newline
when is generated then add one

[YOCTO #15734]

CC: Yoann Congal <yoann.congal@smile.fr>
CC: Randy MacLeod <randy.macleod@windriver.com>
CC: Alexander Kanavin <alex.kanavin@gmail.com>
(From OE-Core rev: d2fcd9e880126bc33be2ef14e678cc1aa72683c3)

Signed-off-by: Christos Gavros <gavrosc@yahoo.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2025-02-27 10:55:17 +00:00
Oleksandr Hnatiuk
7e3d21d0b2 oeqa/selftest: Fix failure when configuration contains BBLAYERS:append
When used with `--newbuilddir` option, it replaces relative paths in
BBLAYERS variable with absolute paths by evaluating the final value of
the variable, converting paths and saving result by assignment
(`BBLAYERS =`) at the end of bblayers.conf.

This breaks tests when bblayers.conf contains BBLAYERS:append because
:append statements are evaluated after all assignments and we end up
with the appended layer added twice - first by evaluating final value
of the variable, then again by bitbake evaluating BBLAYERS:append.

The error is as follows:
AssertionError: Command 'bitbake  -e' returned non-zero exit status 1:
ERROR: Found duplicated BBFILE_COLLECTIONS 'layer-name', check bblayers.conf or layer.conf to fix it.

(From OE-Core rev: 516c8e8f722ed0eb85a715a23b5e68d05abe9275)

Signed-off-by: Oleksandr Hnatiuk <ohnatiuk@cisco.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2024-12-12 12:59:35 +00:00
Yoann Congal
61186bc98a oeqa/selftest: Only rewrite envvars paths that absolutely point to builddir
When building the new selftest builddir, paths in environment variables
are rewritten to point to the new buildir, but users can have
environment variables that point outside of the build dir using
relative paths from builddir. We must not rewrite those.

Check this by verifying that the absolute path still contains the
builddir.

Fixes [YOCTO #15241]

(From OE-Core rev: c5e70500caffcd0518899cc6eba23a38bc3be108)

Signed-off-by: Yoann Congal <yoann.congal@smile.fr>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2024-09-12 16:15:09 +01:00
Julien Stephan
e9939fe2c6 oeqa: selftest: context: run tests serially if testtools/subunit modules are not found
If testtools and/or subunit modules are not found we get the following backtrace
(example for testtools):

  NOTE: Starting bitbake server...
  Traceback (most recent call last):
    File "<..>/poky/scripts/oe-selftest", line 60, in
  <module>
      ret = main()
    File "<..>/poky/scripts/oe-selftest", line 47, in main
      results = args.func(logger, args)
    File "<..>/poky/meta/lib/oeqa/selftest/context.py",
  line 391, in run
      rc = self._internal_run(logger, args)
    File "<..>/poky/meta/lib/oeqa/selftest/context.py",
  line 377, in _internal_run
      rc = self.tc.runTests(**self.tc_kwargs['run'])
    File "<..>/poky/meta/lib/oeqa/selftest/context.py",
  line 161, in runTests
      return super(OESelftestTestContext, self).runTests(processes, skips)
    File "<..>/poky/meta/lib/oeqa/core/context.py", line
  91, in runTests
      result = self.runner.run(self.prepareSuite(self.suites, processes))
    File "<..>/poky/meta/lib/oeqa/selftest/context.py",
  line 154, in prepareSuite
      from oeqa.core.utils.concurrencytest import ConcurrentTestSuite
    File
  "<..>/poky/meta/lib/oeqa/core/utils/concurrencytest.py",
  line 22, in <module>
      import testtools
  ModuleNotFoundError: No module named 'testtools'

Fix this by adding a custom callback on -j/--num-processes parameter to
check testtools and subunit modules. Fallback to serial testing if
missing. This strategy is already used in sdk/context.py

(From OE-Core rev: 35284404473b2c2d9f69594582868ed66ef3525e)

Signed-off-by: Julien Stephan <jstephan@baylibre.com>
Signed-off-by: Alexandre Belloni <alexandre.belloni@bootlin.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2024-04-23 13:40:24 +01:00
Richard Purdie
e598223e0b oeqa/selftest: Drop machines support
The machines option to oe-selftest isn't used in our CI and is never likely to
be, we focus and execute testing explictly. The YOCTO #15247 is about how
this code doesn't interact well with build directory cleanup and at this point
I think we should just remove the option/code.

(From OE-Core rev: 815d04a2007e1154b69f1a027c8677ea86935354)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2023-11-05 10:57:56 +00:00
Richard Purdie
7e6514b28b oeqa: Streamline oe-selftest startup time
"bitbake -e" executions from get_bb_var calls are slow and slow down oe-selftest
startup. Rationalise the code to avoid them and minimise the number of "parsing"
locations we use by caching key variables and passing them around more.

This was particularly problematic with oe-selftest -j usage since it would
have multiple bitbake -e executions per process making parallel usage
particularly slow.

(From OE-Core rev: 3689cadeb07d76e66f97d890e844f899f69666fe)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2023-09-22 07:45:17 +01:00
Richard Purdie
0eec01dee1 oeqa/selftest: Fix broken symlink removal handling
The test above this removal correctly looks at symlinks however to
remove a symlink we should call unlink(), not remove(). This avoids
some build failures/tracebacks.

(From OE-Core rev: dbdb6e73b0f52bc5f9429aca47802d51edbbc834)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2023-09-22 07:45:17 +01:00
Mikko Rapeli
de863bc1fa oeqa/selftest/context.py: check git command return values
Don't ignore return values from the git command lines. If something goes
wrong, fail the test right away.

(From OE-Core rev: dfc178a70d6fa60e89d4716f05d68e2c72c6ecd3)

Signed-off-by: Mikko Rapeli <mikko.rapeli@linaro.org>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2023-09-14 21:57:16 +01:00
Richard Purdie
a8f3ebc4e0 Revert "oeqa selftest context.py: fix git commands and set branch name"
We have autobuilder workers which don't have the -b option unfortunately.

This reverts commit 5ec557467dda29309e25102b507bb919275bedbb.

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2023-09-07 23:26:14 +01:00
Mikko Rapeli
d95c3d3b18 oeqa selftest context.py: fix git commands and set branch name
Check return values of each git command and set initial branch name
to avoid this warning on every test run:

NOTE: Starting bitbake server...
hint: Using 'master' as the name for the initial branch. This default branch name
hint: is subject to change. To configure the initial branch name to use in all
hint: of your new repositories, which will suppress this warning, call:
hint:
hint:   git config --global init.defaultBranch <name>
hint:
hint: Names commonly chosen instead of 'master' are 'main', 'trunk' and
hint: 'development'. The just-created branch can be renamed via this command:
hint:
hint:   git branch -m <name>

(From OE-Core rev: 5ec557467dda29309e25102b507bb919275bedbb)

Signed-off-by: Mikko Rapeli <mikko.rapeli@linaro.org>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2023-09-07 14:36:30 +01:00
Mikko Rapeli
4bbcafa914 oeqa selftest context.py: remove warning from missing meta-selftest
It's not a warning but a handled case and layer gets added
automatically. Very few build configs have this layer enabled
by default.

(From OE-Core rev: 9a2493ea83f0b30578a819de05108502aaadc7f5)

Signed-off-by: Mikko Rapeli <mikko.rapeli@linaro.org>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2023-09-07 14:36:30 +01:00
Mikko Rapeli
72d34fa0ab oeqa selftest context.py: whitespace fix
(From OE-Core rev: 8aed3ca405cb3cc5d346eb8da8b3307d93f75d5b)

Signed-off-by: Mikko Rapeli <mikko.rapeli@linaro.org>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2023-09-07 14:36:30 +01:00
Thomas Roos
67e22b9fc9 oeqa/selftest: Use SSTATE_DIR of parent build dir
This will configure the build-st/conf/local.conf with the SSTATE_DIR
of the parent build dir to speed up the build, test, dev process.

(From OE-Core rev: 71dca60ddcf4370ff7e91de6ddf8a7d4cf76db50)

Signed-off-by: Thomas Roos <throos@amazon.de>
Signed-off-by: Luca Ceresoli <luca.ceresoli@bootlin.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2023-04-20 11:56:05 +01:00
Jose Quaresma
f5f465ff57 oeqa/selftest: OESelftestTestContext: convert relative to full path when newbuilddir is provided
Relative paths in BBLAYERS only works when the new build dir are on the same
ascending directory node:
 .
 ├── build
 ├── build-st

It works because they share the same ascending relative directory node.

So use the full path when the argument newbuilddir is provided
to make the oe-selftest work everywere regardless of the location chosen.

(From OE-Core rev: 2e022c1977bc1006c00a87e08a2dca5b69db4801)

Signed-off-by: Jose Quaresma <jose.quaresma@foundries.io>
Signed-off-by: Alexandre Belloni <alexandre.belloni@bootlin.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2023-03-28 22:28:45 +01:00
Jose Quaresma
4147b7e76d oeqs/selftest: OESelftestTestContext: replace the os.environ after subprocess.check_output
No function change but is needed to simplify the next patch.

(From OE-Core rev: 193d8f0d6383f54c61f90d58eeb8f41580c2360a)

Signed-off-by: Jose Quaresma <jose.quaresma@foundries.io>
Signed-off-by: Alexandre Belloni <alexandre.belloni@bootlin.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2023-03-28 22:28:45 +01:00
Alexis Lothoré
e557cd92d6 scripts/oe-selftest: append metadata to tests results
Many stored results TEST_TYPE are set to "oeselftest", however all those
tests are not run with the same sets of parameters, so those tests results may
not be comparable.

Attach relevant parameters as tests metadata to allow identifying tests
configuration so we can compare tests only when they are run with the same
parameters.

(From OE-Core rev: 406ec0a87e80d1ee0dd1d246adfe9507c10450ff)

Signed-off-by: Alexis Lothoré <alexis.lothore@bootlin.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2023-02-26 11:59:52 +00:00
Chen Qi
f27e91305b selftest: allow '-R' and '-r' be used together
For now, there's no easy way to skip some specified tests when
using 'oe-selftest -r'. This is because '-R' and '-r' are in the
mutually exclusive group. Looking at the codes, the codes have already
allowed running specified tests and skipping specified tests at
the same time. So move '-R' out of the mutually exclusive group
and change its help message.

After this change, the following command could be used.
  oe-selftest -r A -R A.B.C

This does introduce a behavior regression.
Previous 'oe-selftest -R xxx' needs to now be changed to
'oe-selftest -a -R xxx'.

(From OE-Core rev: df9a2b69748d8a24c3390f812225231e9e9acb66)

Signed-off-by: Chen Qi <Qi.Chen@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2022-12-26 11:19:40 +00:00
Richard Purdie
f4886717e5 oeqa: Update cleanup code to wait for hashserv exit
We sometimes see exceptions from code seeing the hashserv DB files
being removed at directory cleanup time. Add a check to ensure the
hashserv has written the data base journal (and hence likely exited)
before cleaning up.

This will hopefully avoid errors like:

Traceback (most recent call last):
  File "[...]/meta/lib/oeqa/sdk/buildtools-cases/build.py", line 30, in test_libc
    delay = delay - 1
  File "/usr/lib/python3.6/tempfile.py", line 948, in __exit__
    self.cleanup()
  File "/usr/lib/python3.6/tempfile.py", line 952, in cleanup
    _rmtree(self.name)
  File "/usr/lib/python3.6/shutil.py", line 486, in rmtree
    _rmtree_safe_fd(fd, path, onerror)
  File "/usr/lib/python3.6/shutil.py", line 424, in _rmtree_safe_fd
    _rmtree_safe_fd(dirfd, fullname, onerror)
  File "/usr/lib/python3.6/shutil.py", line 444, in _rmtree_safe_fd
    onerror(os.unlink, fullname, sys.exc_info())
  File "/usr/lib/python3.6/shutil.py", line 442, in _rmtree_safe_fd
    os.unlink(name, dir_fd=topfd)
FileNotFoundError: [Errno 2] No such file or directory: 'hashserv.db-wal'

(From OE-Core rev: 0b07d9add687d78495176cda0f3011c10ffa4d4b)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Signed-off-by: Alexandre Belloni <alexandre.belloni@bootlin.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2021-10-16 17:41:59 +01:00
Paul Barker
9efd66ad10 selftest: Add argument to keep build dir
The oe-selftest code already keeps the selftest build directory in place
if any tests failed. By default the build directory is deleted if all
tests pass but there may be cases where it's desirable to keep this
directory around, for example to compare intermediate files between
passing and failing test runs.

(From OE-Core rev: 67aa7069dbe8f5f5f186eb67708ece5c4bd42976)

Signed-off-by: Paul Barker <pbarker@konsulko.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2021-01-08 10:10:15 +00:00
Richard Purdie
e9ec164efa oeqa/concurrencytest: Improve builddir path manipulations
Its possible some patterns may cause problems with the current path
manipulations, make a small tweak to try and avoid potential pathname
overlap issues.

(From OE-Core rev: 889005dc17d3e3b8eadee907ee2c05b8ff613285)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2020-09-08 16:43:40 +01:00
Richard Purdie
79856aaf93 oeqa/selftest: Clean up separate builddir in success case when non-threaded
If oe-selftest is run without -j, the separate build directory "build-st"
isn't cleaned up afterwards. Mirror the behaviour of the -j option to
handle this the same way, only preserve upon failure.

To do this, the remove function needs to be moved to the selftest
context module so that it can be accessed without requiring the
testtools and subunit modules the -j option requires.

A dummy wrapper class is used to wrap the tests and clean up afterwards.

[YOCTO #13953]

(From OE-Core rev: 1b376ade430d40d3cfe9c18f200c764d622710e5)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2020-06-28 08:36:00 +01:00
Paul Barker
9800884deb oe-selftest: Recursively patch test case paths
This ensures that builddir is updated correctly to point to the new
selftest build directory when we're given a list of test suites instead
of a list of test cases.

(From OE-Core rev: 56e211d0f3c6cb84f5982e5de00faeeed69c1912)

Signed-off-by: Paul Barker <pbarker@konsulko.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2020-06-04 13:27:32 +01:00
Paul Barker
18a69b1cd2 oe-selftest: Support verbose log output
(From OE-Core rev: 6a607da9a6aa05f86b0e77cccd066b9f286cfada)

Signed-off-by: Paul Barker <pbarker@konsulko.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2020-06-04 13:27:32 +01:00
Paul Barker
05882158de oe-selftest: Allow overriding the build directory used for tests
This may be useful if the parent directory of the original builddir is
not writable, on a lower performance drive, etc.

(From OE-Core rev: dab719e5f1964fd48e9a35e6b9db1508b5414540)

Signed-off-by: Paul Barker <pbarker@konsulko.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2020-06-04 13:27:32 +01:00
Richard Purdie
f930e2cadb oeqa/selftest: Ensure buildtools in environment variables isn't replaced
This avoids the seeing broken replacements like:
oe-selftest-centos/build/build-st-926tools/sysroots/x86_64-pokysdk-linux/etc/ssl/certs/ca-certificates.crt
which understandably break builds.

(From OE-Core rev: 04ee0e8b95cd8ed890374e0007f976684206b630)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2020-03-10 23:20:34 +00:00
Richard Purdie
bf745476ea oeqa/selftest: Drop 'backup' code and SIGTERM handler
Now selftest is using its own copied build directory, we can stop worrying about
copying files around as backup, and drop the SIGTERM handler to try and restore
them, simplifying the code.

(From OE-Core rev: b8ea8a910267fee4bb9e57f24ba829064e22d016)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2020-02-21 09:39:00 +00:00
Richard Purdie
c569157d9d oeqa/selftest: Standardise seperate builddir for concurrent and non-concurrent selftest
Currently oe-selftest reuses the current build directory and the concurrent
version run with -j does not.

Standardise and use a separate new build directory in both cases. This will lead
to simpler code and more reliable user run tests.

(From OE-Core rev: 50238b0717b04e0a1fa69d618e8c8aa8445a80b0)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2020-02-21 09:39:00 +00:00
Richard Purdie
dba8c1d5ef selftest/context: Avoid tracebacks from tests using multiprocessing
We can see tracebacks where the SIGTERM handler catches things
it shouldn't. Avoid exit(1) unless we're the process that
it was intended for.

[YOCTO #13664]

(From OE-Core rev: d9c62ffac611310efd47ed6397d31dccb72fe868)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2020-02-21 09:39:00 +00:00
Nathan Rossi
4a10fa34ca oeqa/selftest/context.py: For -t/-T use append argparse action
Use the 'append' action of argparse instead of nargs. This changes the
behaviour of the option from "-t foo bar -r" to "-t foo -t bar -r".

Additionally rename the long form options to be consistent with
behaviour, such that they specifying a single tag at a time.

(From OE-Core rev: ffe9e4303fa9799d2e8af9188853a262e15af226)

Signed-off-by: Nathan Rossi <nathan@nathanrossi.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-09-11 07:30:13 +01:00
Nathan Rossi
405d98e2bd oeqa/selftest/context.py: Change -t/-T args to be optional
Change the -t/-T args to be optional so that they can be used together
with the existing -r/-a/... args to run a more flexible filtering of
test tags.

(From OE-Core rev: 55ee27bb07113a45da18711b5509764f62be4d75)

Signed-off-by: Nathan Rossi <nathan@nathanrossi.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-09-07 21:56:43 +01:00
Nathan Rossi
d4dd54fa50 oeqa/selftest: Add test run filtering based on test tags
Add '--run-only-tags' for running tests which match any of the provided
tags, and '--run-exclude-tags' for running all tests except those that
have any of the provided tags.

(From OE-Core rev: 0dc3caf21c4519ef16c2ac99b93c03e23aab61d9)

Signed-off-by: Nathan Rossi <nathan@nathanrossi.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-09-07 21:56:43 +01:00
Richard Purdie
ffae400179 meta/lib+scripts: Convert to SPDX license headers
This adds SPDX license headers in place of the wide assortment of things
currently in our script headers. We default to GPL-2.0-only except for the
oeqa code where it was clearly submitted and marked as MIT on the most part
or some scripts which had the "or later" GPL versioning.

The patch also drops other obsolete bits of file headers where they were
encoountered such as editor modelines, obsolete maintainer information or
the phrase "All rights reserved" which is now obsolete and not required in
copyright headers (in this case its actually confusing for licensing as all
rights were not reserved).

More work is needed for OE-Core but this takes care of the bulk of the scripts
and meta/lib directories.

The top level LICENSE files are tweaked to match the new structure and the
SPDX naming.

(From OE-Core rev: f8c9c511b5f1b7dbd45b77f345cb6c048ae6763e)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-05-09 16:31:55 +01:00
Chen Qi
d3a94e5b9b oeqa/selftest/context: ensure log directory exists
Ensure log directory exists to avoid the following error.

  FileNotFoundError: [Errno 2] No such file or directory: '/.../build-selftest/tmp/log/oe-selftest-results-20181207043431.log'

(From OE-Core rev: 6c41de48db76087fee596d9440d8f05346ab1094)

Signed-off-by: Chen Qi <Qi.Chen@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2018-12-08 17:17:01 +00:00
Richard Purdie
28cbaf1de1 oeqa/selftest/context: Improve log file handling
The existing logfile is simply placed in the current directory. Since the test
changes cwd to BUILDDIR, the symlink to the log can be placed in an invalid
directory. We also see trackbacks if the symlink is invalid.

Improve things by:

* Placing logs in LOG_DIR (or BUILDDIR if unset).
* Using a full path to the log meaning the log and link are placed in the same directory.
* Using lexists instead of exists so invalid symlinks are handled correctly.

(From OE-Core rev: 750ece11bed0e62a11e0003d1d16a81f7c219761)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2018-12-01 11:38:36 +00:00
Richard Purdie
99f0e68973 testimage/testsdk/selftest: Avoid platform.distro_identifier deprecation warnings
Use our own lsb function instead as used elsewhere by the codebase.

(From OE-Core rev: acac45a6fd604d28ef7c23d67482af3d7e8bcfe3)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2018-11-16 11:46:07 +00:00
Richard Purdie
3e9505274b oeqa/selftest/context: Replace deprecated imp module usage
Avoid the warning:

meta/lib/oeqa/selftest/context.py:8: DeprecationWarning: the imp module is deprecated in favour of importlib; see the module's documentation for alternative uses
  import imp

In this case importlib is a direct replacement.

(From OE-Core rev: db7a60c36a2d3eefc61ae6e1ede01680dc932035)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2018-11-16 11:46:07 +00:00
Yeoh Ee Peng
b1c3b40146 oeqa/selftest: Standardize json logging output directory
Currently sdk & sdkext will output json file to LOG_DIR, while
selftest will output json file to TOPDIR/log.

Standardize selftest json output file to LOG_DIR.

(From OE-Core rev: 2012d6b076fc833864a0254d56d78536314a6799)

Signed-off-by: Yeoh Ee Peng <ee.peng.yeoh@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2018-11-09 15:27:34 +00:00
Richard Purdie
1418c0ea24 oeqa/selftest: Improvements to the json logging
Tweak the preceeding commit to:

* Add STARTTIME to the identifier to make it unique
* Use LOG_DIR
* Store the layer config in a more natural json format
* Drop '_' function prefixes

(From OE-Core rev: 173f59acf9722e2ef27fdd49c20f7d3d664917eb)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2018-10-29 17:26:47 +00:00
Yeoh Ee Peng
d89e06083e oeqa/selftest/context: write testresult to json files
As part of the solution to replace Testopia to store testresult,
OEQA selftest need to output testresult into json files, where
these json testresult files will be stored into git repository
by the future test-case-management tools.

By default, json testresult file will be written to "oeqa"
directory under the oe-selftest log directory.

To configure multiple instances of bitbake to write json testresult
to a single testresult file at custom directory, user will define
the variable "OEQA_JSON_RESULT_DIR" with the custom directory for
json testresult.

(From OE-Core rev: 10697165c832e3dbb2913b6215164ea75e23ec23)

Signed-off-by: Yeoh Ee Peng <ee.peng.yeoh@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2018-10-29 17:26:47 +00:00
Richard Purdie
899a3694de selftest: Replace bitbake -p with bitbake -e
Parsing all the recipes is annoying when trying to re-execute oe-selftest
and also unnecessary as its really just a sanity check. When the tests were
originally being developed the guard was useful but less so now.

Replace it with bitbake -e which is fast and checks the basic configuration
is valid.

(From OE-Core rev: acec5180b8d2731002979179e08439b615631e70)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2018-08-15 09:44:33 +01:00
Anuj Mittal
9f7923c7e0 oeqa/selftest: check if rm_work is enabled
rm_work if enabled leads to some tests failing that rely on artifacts
being present. Check if rm_work.bbclass is included and show an error
and exit if it is.

Fixes [YOCTO #12694]

(From OE-Core rev: dde7e2f590834aa8034f1371954f9b4fbc7a60b7)

Signed-off-by: Anuj Mittal <anuj.mittal@intel.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2018-07-30 12:44:35 +01:00
Chen Qi
869e501544 logging: use warning instead warn
The warn method is deprecated. We should use the documented warning instead.

Quoting from the python's official doc:
"""
Note: There is an obsolete method warn which is functionally identical to warning.
      As warn is deprecated, please do not use it - use warning instead.
"""

(From OE-Core rev: f467fd277eb77336097cfc0f5f329bdc8d0f70cb)

Signed-off-by: Chen Qi <Qi.Chen@windriver.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2018-07-26 13:16:40 +01:00
Richard Purdie
ebd97e728a oeqa: Add selftest parallelisation support
This allows oe-selftest to take a -j option which specifies how much test
parallelisation to use. Currently this is "module" based with each module
being split and run in a separate build directory. Further splitting could
be done but this seems a good compromise between test setup and parallelism.

You need python-testtools and python-subunit installed to use this but only
when the -j option is specified.

See notes posted to the openedmbedded-architecture list for more details
about the design choices here.

Some of this functionality may make more sense in the oeqa core ultimately.

(From OE-Core rev: 326ababfd620ae5ea29bf486b9d68ba3d60cad30)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2018-07-18 10:18:41 +01:00
Leonardo Sandoval
5ec91f9a32 context: Include a command line argument to run all except certain tests
A new command line argument (-R, which is the oposite of current -r) that allows
to run all test cases except the ones indicated through the command line.

Some command line examples:

* Run all except the distro test case:
$ oe-selftest -R distrodata

* Run all except the archiver test case and a single bblayers unit test
$ oe-selftest -R archiver bblayers.BitbakeLayers.test_bitbakelayers_add_remove

[YOCTO #11847]

(From OE-Core rev: e40eeaa790b95d9c25832405c0b0d5b3a0d0292b)

Signed-off-by: Leonardo Sandoval <leonardo.sandoval.gonzalez@linux.intel.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-08-13 09:27:37 +01:00
Aníbal Limón
2d50f153b5 oeqa/{core,selftest}: Add support to validate if a specified test case isn't found
If some test module/case is specified to run and isn't found the OEQA
framework didn't notice it, so complete the implementation using
modules_required and validate for the test case prescense.

Raise an exception when the test module/case required isn't found.

[YOCTO #11645]

(From OE-Core rev: e50b415aaaa1581473f85f0a8afa278b5f95129b)

Signed-off-by: Aníbal Limón <anibal.limon@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-07-30 08:46:19 +01:00
Aníbal Limón
ad734fd64a oeqa/selftest/{context,case}: Handle KeyboardInterrupt/SIGINT and SIGTERM
In order to avoid corrupt local.conf and bblayers.conf adds
signal handler for SIGTERM and use try/finally (KeyboardIntrrupt) block
to restore previously backuped configuration.

[YOCTO #11650]

(From OE-Core rev: 9419c81e69d2facc82e39c846466670c09e6b444)

Signed-off-by: Aníbal Limón <anibal.limon@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-07-21 08:44:25 +01:00
Aníbal Limón
7a19bbcf81 oeqa/selftest/context: Reload testlayer_path when meta-selftest isn't added
When add meta-selftest by the script the testlayer_path needs to be
reloaded to avoid None value.

(From OE-Core rev: d1120849e12c46aa46b02905a3f43c0853bd11a0)

Signed-off-by: Aníbal Limón <anibal.limon@linux.intel.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-06-23 11:44:13 +01:00
Aníbal Limón
e9a6fd9e97 oeqa: Change the order to logDetails and logSummary
Is better to log the summary at end to see in an easy way
the actual result of the test run.

[YOCTO #11622]

(From OE-Core rev: 4e3ab36e8c90abc740cce1ba31faf6595116e1e2)

Signed-off-by: Aníbal Limón <anibal.limon@linux.intel.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-06-09 17:12:15 +01:00
Leonardo Sandoval
10c512b60d scripts/oe-selftest: Migrate to new framework into oeqa.selftest.context
The new OEQA framework aims to re-use code into the different Test
components.

The previous oe-selftest implements it-self loading, run, and list test
cases in a non-standard way (unittest base) and other functionalities
like logging that is now on oeqa core. This ends on a compact oe-selftest
script.

All needed command line options was migrated but there are some of them
pending of implementation and others deprecated.

Deprecated options:

list-tags: The tag functionality into the old oeqa framework isn't
    work, the selftest doesn't has tag decorators.
{run, list}-tests-by: Ambiguos options it accepts all the posibilites module,
    class, name, id or tag.

Remaining to implement:

coverage: It enables covrage reports over a test run, currently isn't on
    on use and some bugs [1], i filed a bug to add support to OEQA core module in
    this way other Test components could enable it.
repository: It push XML results into a git repository and isn't in use,
    i filed a bug to implement this into OEQA core module. [2]

[1] https://bugzilla.yoctoproject.org/show_bug.cgi?id=11582#c0
[2] https://bugzilla.yoctoproject.org/show_bug.cgi?id=11583#c0

(From OE-Core rev: 3b2a20eee4a39f40287bf67545839eaa09fc892d)

Signed-off-by: Leonardo Sandoval <leonardo.sandoval.gonzalez@linux.intel.com>
Signed-off-by: Aníbal Limón <anibal.limon@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-06-06 19:02:43 +01:00