With the change to the new framework data store dependecy was
removed, instead a new file is generated and used in testimage.
When testing builds from the autobuilders the test data values
are from the autobuilder, including the paths.
Some tests require paths to current environment in order to run,
this commit will update such paths and fix the error of running
images donwloaded from autobuilders.
[YOCTO #10964]
(From OE-Core rev: 26ad5105fc2ce03b7ee8ecc6911fd40a52bd573a)
Signed-off-by: Mariano Lopez <mariano.lopez@linux.intel.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
This adds systemd test automatically when an image
is build with systemd DISTRO_FEATURE
[YOCTO #10935]
(From OE-Core rev: 2e6ba167bee7c7b42b1e0e96eb5a7be2eb0fd2c0)
Signed-off-by: Mariano Lopez <mariano.lopez@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
This allows to use the package install feature with
the new OEQA framework.
[YOCTO #10234]
(From OE-Core rev: 077dc19445574457769eb4f231de97e8059cb75e)
Signed-off-by: Mariano Lopez <mariano.lopez@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
testimage support the installation of packages without a package
manager in the target. This adds support for package extraction
required to support the installation feature.
[YOCTO #10234]
(From OE-Core rev: 8c7335290cb00ed0683241249297ca573ebd353a)
Signed-off-by: Mariano Lopez <mariano.lopez@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
This migrates the smart test from the old framework to
the new one. This has its own commit because smart
test was using bb and oe libraries that are available
when exporting the test cases to run in a different host.
Because of the removal of bb and oe libraries index and
packages feeds creation will be managed in testimage bbclass.
[YOCTO #10234]
(From OE-Core rev: 8d64ac4208e8dcb8a6fde6ea2959c9b3edfe2172)
Signed-off-by: Mariano Lopez <mariano.lopez@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
This migrates testimage class to use the new framework. Most of
the code added here is to get rid off the data store dependency.
[YOCTO #10234]
(From OE-Core rev: 2aa5a4954d7610f31875ba7e655f25f8892517b6)
Signed-off-by: Mariano Lopez <mariano.lopez@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
TargetBuildProject was refactored to avoid bitbake dependency so
the instance don't allow to pass data store anymore.
classes/testimage: Export proxies before run tests
The TargetBuildProject based tests download archives from network.
(From OE-Core rev: e275f29de500a338a02402ecc570405309963b35)
Signed-off-by: Aníbal Limón <anibal.limon@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
When runqemu fails, qemu-system process would keep running
and won't be killed, setpgrp() was used when runqemu was
a shell script but it seems it doesn't work always with python.
This would kill qemu-system explicity and to avoid leaving
it behind.
(From OE-Core rev: 9d2b1aa1bcfb2f1933a8eeb9470b4174d5da2f0d)
Signed-off-by: Mariano Lopez <mariano.lopez@linux.intel.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
getVar() now defaults to expanding by default, thus remove the True
option from getVar() calls with a regex search and replace.
Search made with the following regex: getVar ?\(( ?[^,()]*), True\)
(From OE-Core rev: 7c552996597faaee2fbee185b250c0ee30ea3b5f)
Signed-off-by: Joshua Lock <joshua.g.lock@intel.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
It's not uncommon for qemumips[64] builds on the Yocto Project
autobuilder to fail during Sanity Tests after a very long timeout
period. This is due to the MIPS emulation in QEMU being slow and
some of the build tests taking a very long time on MIPS machines.
This patch works around this slowness by disabling the more
complex build tests for QEMU MIPS machines.
[YOCTO #10340]
(From OE-Core rev: 4a1c04c0d509b2cda9b2ccd5a80523c05fa279c6)
Signed-off-by: Joshua Lock <joshua.g.lock@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
This sets a good example and avoids unnecessarily contributing to
perceived complexity and cargo culting.
Motivating quote below:
< kergoth> the *original* intent was for the function/task to error via
whatever appropriate means, bb.fatal, whatever, and
funcfailed was what you'd catch if you were calling
exec_func/exec_task. that is, it's what those functions
raise, not what metadata functions should be raising
< kergoth> it didn't end up being used that way
< kergoth> but there's really never a reason to raise it yourself
FuncFailed.__init__ takes a 'name' argument rather than a 'msg'
argument, which also shows that the original purpose got lost.
(From OE-Core rev: 5f8eb6726a492d259bfe25b0bbce2333c9505504)
Signed-off-by: Ulf Magnusson <ulfalizer@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The feature to install packages in the target requires to
build the package manager. It would fail, with very obtuse
errors, if a test requires to install something and the
package manager hasn't been build. This will add the package
manager as dependency for testimage.
[YOCTO #10260]
(From OE-Core rev: cf548fd85297585cc5688eda45ee332200bbd4b7)
Signed-off-by: Mariano Lopez <mariano.lopez@linux.intel.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The smart tests need createrepo to be in the native sysroot. It needs
to be one of the depends for testimage.
(From OE-Core rev: 2f1a44d33d9ecceef69d11ef92c5369314cb3bf5)
Signed-off-by: bavery <brian.avery@intel.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
In order to get more information about systemd boot process to
be able to debug random failures due to high I/O.
[YOCTO #9299]
(From OE-Core rev: a0bb64973e767c3b8e0bae18ee84ed92693922f0)
Signed-off-by: Aníbal Limón <anibal.limon@linux.intel.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Use IMAGE_CLASSES which is only seen by image recipe.
(From OE-Core rev: 7be8f1a9dad4512c3a979ad744e223edb38fccc6)
Signed-off-by: Robert Yang <liezhi.yang@windriver.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Also delete the removal of suduko for qemumips, as galculator builds fine on
that hardware now.
(From OE-Core rev: 4a81b3f669073455c9b2ee1514c43b96df9f7faa)
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Exctraction of RPMs needs cpio, not all distros include cpio by
default, so we need to build it.
[YOCTO #8694]
(From OE-Core rev: 95cd427b3887b087533fba11c67ef9bc173f9aa5)
(From OE-Core rev: 5a4c73bd3f2bbba2ad5413367fa7ca2f625ffdd7)
Signed-off-by: Mariano Lopez <mariano.lopez@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Add the functionality to install/unistall packages in the
DUTs without the use of the package manager. This is possible
with the extraction introduced in package manager class.
testimage and testexport bbclasses has been modified in order
to support this new feature.
[YOCTO #8694]
(From OE-Core rev: b7111d9e9d64d21f57729d1ac1865aea6e54cc8b)
Signed-off-by: Mariano Lopez <mariano.lopez@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Exported tests now have their own class, so
the code in testimage is not needed anymore.
(From OE-Core rev: bd4dbe4f5ba334690fcb6f26eb06c793ac026b55)
Signed-off-by: Mariano Lopez <mariano.lopez@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Modify existing default test suites to reuse the new
generic test suites.
Related to [YOCTO #8410]
(From OE-Core rev: 1da1bbc88b1ec1476107003f47f7b240d0794165)
Signed-off-by: Alex Franco <alejandro.franco@linux.intel.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Add generic test suites MINTESTSUITE, NETTESTSUITE, DEVTESTSUITE to
group common test suites.
Add DEFAULT_TEST_SUITES_pn- variables for:
* core-image-minimal-dev
* core-image-full-cmdline
* core-image-x11
* core-image-lsb
* core-image-lsb-dev
New default test suites use generic test suites.
Existing default test suites not modified.
[YOCTO #8410]
(From OE-Core rev: 8d194c6300964a393b6b69b7d06c0a729a5f8b5f)
Signed-off-by: Alex Franco <alejandro.franco@linux.intel.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
With new structure of TestContext now holds suite and variable
that contains unittest instances, it can't be exported using
JSON causing and exception.
Adds the suite variable for avoid export it.
(From OE-Core rev: cecc7ec2bcb28d7d8a3277fb097efbbe13adff1c)
Signed-off-by: Aníbal Limón <anibal.limon@linux.intel.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Method's for loadTests and runTests make sense to define
inside TestContext because it can be different around
Image, SDK, SDKExt.
(From OE-Core rev: 03af7b99e3ce36ce3e29dc31e33d2cc74eb14849)
Signed-off-by: Aníbal Limón <limon.anibal@gmail.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
In order to provide better abstraction move functions to get the test
suite inside the TestContext.
(From OE-Core rev: 552285688441330440bd1bc138aeaea87549024a)
Signed-off-by: Aníbal Limón <limon.anibal@gmail.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Move anonymous duplicated class TestContext from testimage/testsdk to
oeqa/oetest now we have two new classes ImageTestContext and
SDKTestContext with common code in TestContext class.
(From OE-Core rev: 593f2fdf6ee94c5f91761a669048da3598cbe3fa)
Signed-off-by: Aníbal Limón <limon.anibal@gmail.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
For compatibility adds default inherit of the new class testsdk
for now, we need to review the code base.
(From OE-Core rev: a9861f6a0fd10c2a56129f2f0a7cdd806912290f)
Signed-off-by: Aníbal Limón <anibal.limon@linux.intel.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Moves all the testsdk code from testimage in order to have it's own
class because new tests will be added for extensible SDK.
The old paths for store logs "${WORKDIR}/testimage" and sdk
"${WORKDIR}/testimage-sdk" was maintained for compatibility may be
change to point testsdk after review the codebase.
The dependency of QEMU was removed because isn't needed.
(From OE-Core rev: b30edc18866865ec757b3fd86eb84de530720acf)
Signed-off-by: Aníbal Limón <anibal.limon@linux.intel.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Test lists functions can be used in other parts so modularize it and
move to oeqa/oetest.py library.
Testimage class was updated to meet the new sign of the functions.
(From OE-Core rev: c9f771533af70e7ccb1e7064e58926cfaee7367a)
Signed-off-by: Aníbal Limón <anibal.limon@linux.intel.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
This sets the scene for removing the default False for expansion
from getVarFlag. This would later allow True to become the default.
On the most part this is an automatic translation with:
sed -e 's:\(\.getVarFlag([^,()]*, [^,()]*\)):\1, True):g' -i `grep -ril getVar *`
In this case, the default was False, but True was used since in most
cases here expansion would be expected.
(From OE-Core rev: 42a10788e89b07b14a150ced07113566cf99fcdd)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Now that x11vnc won't be in Sato images we don't want to test it.
(From OE-Core rev: 90ff7219332d9651fcdfc895d61fb73a46ffde78)
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
If tests are organized in folders then it will copy the whole
folder to the export location. This should keep the original
directory structure for exported oeqa/runtime.
(From OE-Core rev: 8fe45badd3dfe44fa5161229324b6439594123fd)
Signed-off-by: Lucian Musat <george.l.musat@intel.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
This has been obsoleted by parselogs.
(From OE-Core rev: 05132fd236835db71d3e763b4d6ce01fcf14741a)
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
This change just add two more commands to
the host dumps to get more information
related to the IO.
[YOCTO #8412]
(From OE-Core rev: 9c9fd32105bd1a5e1c01f9dd5d1a56ceef12554e)
Signed-off-by: Mariano Lopez <mariano.lopez@linux.intel.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Just like we have in oe-selftest, you can add
<filename>.<Class>.<testname> in TEST_SUITES in order to
run just that test
(From OE-Core rev: 3bf5204315fedc586fdf641583cfdb2c1a78c5c0)
Signed-off-by: Lucian Musat <george.l.musat@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
In the json file the whole datastore is serialized which contains
absolute path to the oeqa folder. This breaks the functionality
when trying to run the tests from other machines.
(From OE-Core rev: 94057ccc70aa3fe29d774f571a65f56fd1285d4c)
Signed-off-by: Lucian Musat <george.l.musat@intel.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The SIGTERM introduced another break into runexported test,
the handler is in the class TestContext while exporting it
will try to convert the singnal handler to JSON, and this
will throw an exception.
The SIGTERM handler is not needed in the exported test, so this
change will remove the handler before exporting the test. Also
when a test is exported the target is not deployed/started
anymore, it is not necessary.
[YOCTO #8239]
(From OE-Core rev: 94ab9892e87f159840432fdbfb3873d4496d4786)
Signed-off-by: Mariano Lopez <mariano.lopez@linux.intel.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Sometimes an "image" may not actually have a manifest file such as when
using a "baremetal kernel". This change allows for a user to set a
IMAGE_NO_MANIFEST flag to 1 in order to inform the code that there is no
corresponding manifest that should exist for an image.
(From OE-Core rev: 78f5c3cb3971c31a950deb5cab8992f3ba577440)
Signed-off-by: Randy Witt <randy.e.witt@linux.intel.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
On qemumips, some tests are slow to the point of the autobuilder appearing
hung. We therefore specifically exclude buildsudoku but to do this, we need
to list the tests we expect to work on core-image-lsb-sdk.
(From OE-Core rev: cf03765705c6cd26c1a904b62b9aca8f914f44aa)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Currently if the SDK fails to extract, no error is shown and the test is marked
as passed! Clearly this is incorrect, fix it to correctly raise an error.
(From OE-Core rev: fb2235a21e45fa1a47c3b7a9a6a72c515ef10dd1)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
In the current state if a SIGTERM is sent to
the testimage worker, the worker will exit but
runqemu and qemu won't exit and the processes
need to be killed manually to free the
bitbake lock.
This allows to catch the SIGTERM signal in
testimage, this way it is possible to stop
runqemu and qemu and allow to free the bitbake lock.
Also this allows to skip the rest of the tests
when running the tests in qemu or real hardware.
This also solves minimal breaks in the setup of the
runtime test when checking if qemu is alive.
[YOCTO #8239]
(From OE-Core rev: 2694d2f17d597b44fcc7aed5f6836081fa88a6b3)
Signed-off-by: Benjamin Esquivel <benjamin.esquivel@linux.intel.com>
Signed-off-by: Mariano Lopez <mariano.lopez@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
If a test case is decorate by oeqa.utils.decorators.tag, this case will
by add a tag, testrunner will filter these tags by TEST_SUITES_TAGS
[YOCTO #7849]
(From OE-Core rev: 085589b1018ba4d950baf7bcfb499be02c1b29fc)
Signed-off-by: zjh <junhuix.zhang@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Currently it is not possible to run a exported test,
but this patch will allow to use the HosDumper class
when running a exported test, otherwise the HostDumper
class will break runexpored test.
[YOCTO #8118]
(From OE-Core rev: 25814dbdd12db0499e4d81876873158e1b8a4fcc)
Signed-off-by: Mariano Lopez <mariano.lopez@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
This allows to have datastore variables in the dump
commands and will get the data when a new instance
it's created.
Also this remove special cases from the commands.
[YOCTO #8118]
(From OE-Core rev: 384927eb8d52bc5f14c63c8421aa62ee859587f0)
Signed-off-by: Mariano Lopez <mariano.lopez@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
It makes sense to separate the dump commands from the
oeRuntimeTest class, this way it can be used in all
the test context.
These are the changes included in this patch:
- Created classes: BaseDumper, HostDumper, TargetDumper
- Create an instance of HostDumper in imagetest.bbclass
and add it to TestContext class, this way any class
that have access to the TestContext would be able
to dump logs from the host
- Create an instance of TargetDumper in QemuTarget
class after get the runner, this way it is
accessible during the tests.
[YOCTO #8118]
(From OE-Core rev: ad10af6be343b5425fde43055263b0744c161cb3)
Signed-off-by: Mariano Lopez <mariano.lopez@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
This patch modify three files altought two of them
are minimal modifications. This version includes
the changes proposed by Paul.
testimage.bbclass:
Create new vars for easy modification of the dump
directory and commands to be run on host and target
when a test fails
TESTIMAGE_DUMP_DIR: Directory to save the dumps
testimage_dump_target: Commands to run on target
testimage_dump_host: Commands to run on host
oetest.py:
- Allow to use the vars defined in testimage class
- Now able to run commands in the host and dump the
results
- Fix an issue with the condition where to run the
dump commands (Before it run the commands every
test after a failure, now it runs the commands only
in tests that failed)
- Fix the output to stdout
[YOCTO #8118]
(From OE-Core rev: 26fe645457633f90bb5ddbb12f5f7b9ca4a06cc5)
(From OE-Core rev: 7b4fbbf979ed22434b8e3f83ae145139bb0d9fc7)
Signed-off-by: Mariano Lopez <mariano.lopez@linux.intel.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The "bitbake meta-toolchain" comment is dated and incorrect, fix to
point at -c populate_sdk.
If a toolchain contains multiple environment files, iterate them and run
the sdk test suite on each on in turn rather than giving a fatal error and
giving up.
Also improve the debug output so that rather than PN, it also show the
toolchain tarball under test, and the environment file name its using.
Also enable the accidentally disabled cleanup code.
(From OE-Core rev: 44c8b1bd58397db85e6f7bb9a57f0d57d2c69ad5)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Allow the list of TEST_SUITES to be read from a list of manifest files
in the TEST_SUITES_MANIFEST variable.
[YOCTO #7848]
(From OE-Core rev: c270148b3f0e87bbaafbcffbca4ee7b965be6569)
Signed-off-by: zjh <junhuix.zhang@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
If a TC name start with "oeqa", it is thought as a full TC path. if not,
follow original logic (assume under oeqa.runtime)
For example:
TEST_SUITES = "oeqa.runtime.pnp.get_memory_size oeqa.runtime.sanity.reboot"
will run these two case when bitbake -c testimage
[YOCTO #7834]
(From OE-Core rev: 2c3fe7351edd954230b14443f82914a3f70e8a40)
Signed-off-by: zjh <junhuix.zhang@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The qemuarm64 machine doesn't have graphics so don't test xorg/vnc
as they won't work.
[YOCTO #7103]
(From OE-Core rev: 27d17c662aa103dc728f14e78cac772546a9e5dc)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The default test list only works for rpm packaging. This fixes it for
deb and ipk too.
(From OE-Core rev: 210c8926405fcf695ec00f5768f29ba198320d6a)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Currently TEST_SUITES is used for both target image and sdk versions which
can be confusing. This introduces TEST_SUITES_SDK for the sdk version of
the code so that the different test sets can be specified independently.
(From OE-Core rev: ffd84177c68a6c86e654a9ba2512c299b40ec5e9)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Fix SDK_MANIFEST -> SDK_TARGET_MANIFEST and add support for host
version too which is useful in SDK QA tests.
(From OE-Core rev: df91dd8d064dc3e59c7f057d3f869500a233a76f)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
TEST_SUITES="auto" is useful to run all suitable tests without
having to hard-code the list. However, it did not take test
dependencies into account, which can be an issue for tests
which really depend on some other test to run first.
To fix this, modules get loaded in the order determined by
TESTS_SUITES, but then get re-ordered based on dependencies
derived from @skipUnlessPassed before running them. The original
order is used to break ties when there are no dependencies, so
reordering only occurs when really necessary.
@skipUnlessPassed gets extended such that it makes the test name
a method depends on available for inspection by the test loader
in oetest.py.
Unfortunately Python's unittest offers no API to inspect tests
in a TestSuite, so the code has to rely on implementation details
to find all tests. The worst that can happen when the implementation
changes is that tests are not found and reordering does not happen.
(From OE-Core rev: 6e4543a35836c572b23b9f8162b19d1e038d3ed2)
Signed-off-by: Patrick Ohly <patrick.ohly@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
In order to use hasFeature and hasPackage methods in sdk test modules, we need specific fields to be set in the TestContext object.
Adding pkgmanifest, imagefeatures and distrofeatures to the TestContext.
(From OE-Core rev: cd342b399b2d78724032cdd7042968d3238cd548)
Signed-off-by: Corneliu Stoicescu <corneliux.stoicescu@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
- add new testsdk task for meta-toolchain testing.
- enable the get_tests_list method to work with sdk tests.
- add default TEST_SUITES value for meta-toolchain package
NOTE: Original patch made by: Richard Purdie <richard.purdie@linuxfoundation.org>
(From OE-Core rev: b78bc50904d53d5091729de481b99cc3ac4aaa1e)
Signed-off-by: Corneliu Stoicescu <corneliux.stoicescu@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
If we couldn't start the target, it doesn't make sense to try and stop
it here since logically it shouldn't now be in any kind of "started"
state. (It's the start function's job to clean up after itself if it
fails - to that end, fix up the QemuTarget class so that it does.)
(From OE-Core rev: 819ebddae6b78120e5e082423793ff988419b5c4)
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
For machines other than qemu it will not be okay to run
multiple testimage tasks in parallel.
(From OE-Core rev: 9560305a9e28316438cb57421afc7877890c4b76)
Signed-off-by: Stefan Stanacar <stefanx.stanacar@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Sometimes we may forget to actually build the image
we want to test (when testimage task is called manually).
Instead of an ugly traceback we should fail nicely.
The manifest is written after the rootfs so this ensures
the image was actually built.
(From OE-Core rev: 85c8dd4170a88a5d7f3d9ca181e75720302727c5)
Signed-off-by: Stefan Stanacar <stefanx.stanacar@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Add the ability to export the tests so that they can run independently of
the build system, as is required if you want to be able to hand the test
execution off to a scheduler.
Booting/deployment of the target is still handled by the build system,
as before, only the execution of the tests happens outside of the build system.
Tests exported are the ones defined in TEST_SUITES.
No tests have been changed as interesting parts of the data store have been
exported and tests can continue to query them as before. Small adjustments were made
for a couple of oeqa modules though.
[YOCTO #5613]
(From OE-Core rev: 155dd52e0f707e06f50756584a50f744ba6b7844)
Signed-off-by: Stefan Stanacar <stefanx.stanacar@intel.com>
Signed-off-by: Saul Wold <sgw@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Run a python script on the target
1)checks the output.
2)Call os.system method create a testfile
(From OE-Core rev: 4465c9368b0c37a3a2c41b68f65de08690a8179b)
Signed-off-by: Saul Wold <sgw@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Makes it much easier to figure out where a syntax error is.
(From OE-Core rev: 0c30a25c3d5f7fb1087dff45e01595329620235f)
Signed-off-by: Stefan Stanacar <stefanx.stanacar@intel.com>
Signed-off-by: Saul Wold <sgw@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
This patch makes the necessary changes for using the targetcontrol.py module
so that one can run the same tests on a qemu instance or a remote machine
based on the value of TEST_TARGET variable: "qemu" or "simpleremote".
The default value is "qemu" which starts a qemu instance and it's the
with what we currently have.
With "simpleremote", the remote machine must be up with network and ssh
and you need to set TEST_TARGET_IP with the IP address of the remote machine
(it can still be a qemu instance that was manually started).
Basically testimage.bbclass now does something along the lines of:
- load tests -> deploy (prepare) / start target -> run tests.
There were a couple of changes necessary for tests and
also some cleanups/renames that were needed to adjust this change. (use
ip everywhere when refering to target and server_ip when refering to host/build machine)
Also two unnecessary and unsed methods were dropped from sshcontrol.
[ YOCTO #5554 ]
(From OE-Core rev: a7820350fa3271d78ed7476e02f4aef593be1125)
Signed-off-by: Stefan Stanacar <stefanx.stanacar@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Builds a simple Hello World module on target.
Added to the defaults for core-image-sato-sdk.
(From OE-Core rev: c61c3dee162aa1f5bf31b2a09d8b916dc1712056)
Signed-off-by: Stefan Stanacar <stefanx.stanacar@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The BB_ORIGENV variable isn't picklable and causes failures when generating the task
signature for the autorun image mode. We don't want to depend on its contents anyway
so lets exclude it.
(From OE-Core rev: 3f5f9d2681d41e44dade5d3a5a4bff3fd38f5506)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Adds info about tests run even if they passed (in case of errors log will
be printed anyway). Also some style change for qemu object.
(From OE-Core rev: 4adcf960be9c7477e314788d42b80753c2aaf572)
Signed-off-by: Stefan Stanacar <stefanx.stanacar@intel.com>
Signed-off-by: Saul Wold <sgw@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
This allows a clean seperation between image outputs from different
machines, and makes it possible to have convenience symlinks to make
the output ready to deploy.
This did require some surgery in runqemu; if explicit paths to the image
and kernel are not supplied then DEPLOY_DIR_IMAGE needs to be determined
from bitbake or set in the environment. However the script does try to
avoid requiring it unless it really is needed. Corresponding changes
were made in the automated testing code as well.
Based on an RFC patch by Koen Kooi <koen@dominion.thruhere.net>
(From OE-Core rev: 7e90261aec61f79680b5eaeaf5b18c7b795412a4)
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
A layer can add tests in lib/oeqa/runtime (provided it extends BBPATH as
normal) and enable them with TEST_SUITES_append = " testname". Test
module names shouldn't collide though.
(From OE-Core rev: e1e347a2d509303e1c566450b0f2b485d3d6629f)
Signed-off-by: Stefan Stanacar <stefanx.stanacar@intel.com>
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Saul Wold <sgw@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
While I can't reproduce on local builds, sometimes images fail
to boot on AB (which runs many builds at once). Assuming
there isn't something weird going on, let's just give it more time.
(From OE-Core rev: db38e10701cd2392a57e559573b715fd6daf6e2a)
Signed-off-by: Stefan Stanacar <stefanx.stanacar@intel.com>
Signed-off-by: Saul Wold <sgw@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Some new tests were added, safe to have them in the defaults
for sato-sdk and sato. Not all of the new tests are here though,
either because they aren't applicable to default images or take too long.
(like build iptables/cvs/sudoky ones, they can be enabled
in local.conf and a special target on AB setups.). Also reorder them a bit.
(From OE-Core rev: caa18a99ec002e4e87e32cae8a2d28bb0e32c5a6)
Signed-off-by: Stefan Stanacar <stefanx.stanacar@intel.com>
Signed-off-by: Saul Wold <sgw@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The full name for the log is confusing when there are multiple files.
Also move the ssh log path stuff where it's needed.
(From OE-Core rev: 1dbef61a0776ec6c9ac9209442bb4c346e706d7d)
Signed-off-by: Stefan Stanacar <stefanx.stanacar@intel.com>
Signed-off-by: Saul Wold <sgw@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Adds some comments to testimage.bbclass and the files it calls,
just to give an ideea of what it does.
(From OE-Core rev: 8bbb7116cf02466dfc59a17dc7bb51287aeea55b)
Signed-off-by: Stefan Stanacar <stefanx.stanacar@intel.com>
Signed-off-by: Saul Wold <sgw@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Make a copy of the rootfs and test that.
We can now drop the snapshot option.
(From OE-Core rev: ba58f1fe8fb7a0e3ff9320dfc108235d484da6a1)
Signed-off-by: Stefan Stanacar <stefanx.stanacar@intel.com>
Signed-off-by: Saul Wold <sgw@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Add tests for free space and syslog.
Changed in v2:
- limit df's output to /
- syslog: fix restart in case of systemd
(From OE-Core rev: 1b39d57e7b5c9b69d565cf4d188ebc2f14e66ae6)
Signed-off-by: Alexandru Palalau <alexandru.palalau@intel.com>
Signed-off-by: Stefan Stanacar <stefanx.stanacar@intel.com>
Signed-off-by: Saul Wold <sgw@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
On a loaded host (such as an AB setup where multiple builds run)
the 200 seconds timeout might not be enough to reach the login prompt.
Also make it configurable so we can set it from local.conf/auto.conf
(From OE-Core rev: e828571683333e6f18ed56a84e114b5e54975bc7)
Signed-off-by: Stefan Stanacar <stefanx.stanacar@intel.com>
Signed-off-by: Saul Wold <sgw@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Replacement class for imagetest-qemu.bbclass. It launches a qemu instance and
runs test modules defined in TEST_SUITES.
(From OE-Core rev: e0e32b978e5af128d7ff4ee2686777b49f919e27)
Signed-off-by: Stefan Stanacar <stefanx.stanacar@intel.com>
Signed-off-by: Radu Moisan <radu.moisan@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
