If python3 git module is not installed, we fall back to some git
command to get info. A previous commit (b8d22ed6) accidently deleted
the return statement, causing errors like below.
Exception: UnboundLocalError: local variable 'InvalidGitRepositoryError' referenced before assignment
(From OE-Core rev: cd8aedc408afa4f8589f79e4d85befc7cac76ac6)
Signed-off-by: Chen Qi <Qi.Chen@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Now that we have a packagegroup that can be used to replace the overall
functionality of busybox as base-utils, add a test that we can continue
to build a fairly complex image without busybox being available. This
also doubles as documentation-by-test of how to do this.
(From OE-Core rev: 6319a59c1d30eeb8ad4871d43641e3469fb543ba)
Signed-off-by: Tom Rini <trini@konsulko.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Two identical classes exist for SystemdbootTarget,
introduced in commit f9a61d3400ad
("gummiboot: Remove/change gummiboot references with systemd-boot")
poky commit 2dce2648e3
Remove the duplicated copy.
(From OE-Core rev: a59562c7ddbda1a266f0ea22ab78fb86de4861a3)
Signed-off-by: André Draszik <andre.draszik@jci.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Note that the tests require that the host machine has a X display,
has mesa development files installed and is able to create OpenGL contexts.
(From OE-Core rev: 2868e8dfb9e62b49cd06f6c2d010405079d3a71c)
Signed-off-by: Alexander Kanavin <alex.kanavin@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Add a test for a multiconfig build which mixes tiny and musl builds
along with using the mcextend class to combine and package multiple images
into another image. This gives the multiconfig a decent test in a scenario
users may use.
(From OE-Core rev: 0c7fa15a7350808242754944243f01155bc6784c)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Currently if python3-git isn't installed we can get odd behaviours when the
commit_count is absent. Avoid this set of bugs by adding a fallback here.
(From OE-Core rev: b8d22ed681141ce360d742a96cec5f2925a20222)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The parent code currently assumed that any parent branch is locally
checked out which may not be the case.
Use the local branch by default but fall back to the origin. This
also means removing the later saftey check as the branch may not exist
locally.
This fixes the autobuilder resulttool test pushing code.
(From OE-Core rev: 0ee636bc89d93282cb3d3cd747b3dd046f6e5a9c)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
This is a combined patch of the various tweaks and improvements I
made to resulttool:
* Avoid subprocess.run() as its a python 3.6 feature and we
have autobuilder workers with 3.5.
* Avoid python keywords as variable names
* Simplify dict accesses using .get()
* Rename resultsutils -> resultutils to match the resultstool ->
resulttool rename
* Formalised the handling of "file_name" to "TESTSERIES" which the code
will now add into the json configuration data if its not present, based
on the directory name.
* When we don't have failed test cases, print something saying so
instead of an empty table
* Tweak the table headers in the report to be more readable (reference
"Test Series" instead if file_id and ID instead of results_id)
* Improve/simplify the max string length handling
* Merge the counts and percentage data into one table in the report
since printing two reports of the same data confuses the user
* Removed the confusing header in the regression report
* Show matches, then regressions, then unmatched runs in the regression
report, also remove chatting unneeded output
* Try harder to "pair" up matching configurations to reduce noise in
the regressions report
* Abstracted the "mapping" table concept used to pairing in the
regression code to general code in resultutils
* Created multiple mappings for results analysis, results storage and
'flattening' results data in a merge
* Simplify the merge command to take a source and a destination,
letting the destination be a directory or a file, removing the need for
an output directory parameter
* Add the 'IMAGE_PKGTYPE' and 'DISTRO' config options to the regression
mappings
* Have the store command place the testresults files in a layout from
the mapping, making commits into the git repo for results storage more
useful for simple comparison purposes
* Set the oe-git-archive tag format appropriately for oeqa results
storage (and simplify the commit messages closer to their defaults)
* Fix oe-git-archive to use the commit/branch data from the results file
* Cleaned up the command option help to match other changes
* Follow the model of git branch/tag processing used by oe-build-perf-report
and use that to read the data using git show to avoid branch change
* Add ptest summary to the report command
* Update the tests to match the above changes
(From OE-Core rev: ff2c029b568f70aa9960dde04ddd207829812ea0)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
OEQA outputs test results into json files and these files were
archived by Autobuilder during QA releases. Example: each oe-selftest
run by Autobuilder for different host distro generate a
testresults.json file.
These scripts were developed as a test result tools to manage
these testresults.json file.
Using the "store" operation, user can store multiple testresults.json
files as well as the pre-configured directories used to hold those files.
Using the "merge" operation, user can merge multiple testresults.json
files to a target file.
Using the "report" operation, user can view the test result summary
for all available testresults.json files inside a ordinary directory
or a git repository.
Using the "regression-file" operation, user can perform regression
analysis on testresults.json files specified. Using the "regression-dir"
and "regression-git" operations, user can perform regression analysis
on directory and git accordingly.
These resulttool operations expect the testresults.json file to use
the json format below.
{
"<testresult_1>": {
"configuration": {
"<config_name_1>": "<config_value_1>",
"<config_name_2>": "<config_value_2>",
...
"<config_name_n>": "<config_value_n>",
},
"result": {
"<testcase_namespace_1>": {
"status": "<PASSED or FAILED or ERROR or SKIPPED>",
"log": "<failure or error logging>"
},
"<testcase_namespace_2>": {
"status": "<PASSED or FAILED or ERROR or SKIPPED>",
"log": "<failure or error logging>"
},
...
"<testcase_namespace_n>": {
"status": "<PASSED or FAILED or ERROR or SKIPPED>",
"log": "<failure or error logging>"
},
}
},
...
"<testresult_n>": {
"configuration": {
"<config_name_1>": "<config_value_1>",
"<config_name_2>": "<config_value_2>",
...
"<config_name_n>": "<config_value_n>",
},
"result": {
"<testcase_namespace_1>": {
"status": "<PASSED or FAILED or ERROR or SKIPPED>",
"log": "<failure or error logging>"
},
"<testcase_namespace_2>": {
"status": "<PASSED or FAILED or ERROR or SKIPPED>",
"log": "<failure or error logging>"
},
...
"<testcase_namespace_n>": {
"status": "<PASSED or FAILED or ERROR or SKIPPED>",
"log": "<failure or error logging>"
},
}
},
}
To use these scripts, first source oe environment, then run the
entry point script to look for help.
$ resulttool
To store test result from oeqa automated tests, execute the below
$ resulttool store <source_dir> <git_branch>
To merge multiple testresults.json files, execute the below
$ resulttool merge <base_result_file> <target_result_file>
To report test report, execute the below
$ resulttool report <source_dir>
To perform regression file analysis, execute the below
$ resulttool regression-file <base_result_file> <target_result_file>
To perform regression dir analysis, execute the below
$ resulttool regression-dir <base_result_dir> <target_result_dir>
To perform regression git analysis, execute the below
$ resulttool regression-git <source_dir> <base_branch> <target_branch>
[YOCTO# 13012]
[YOCTO# 12654]
(From OE-Core rev: 78a322d7be402a5b9b5abf26ad35670a8535408a)
Signed-off-by: Yeoh Ee Peng <ee.peng.yeoh@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
These functions can be reused by the resulttool code so move to the common
function library for this purpose.
(From OE-Core rev: c66f848938c04e133259c5b6903dc592866ab385)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
During the test logrotate.LogrotateTest.test_1_logrotate_setup,
there is below logic:
# mkdir $HOME/logrotate_dir
# sed -i "s#wtmp {#wtmp {\n olddir $HOME/logrotate_dir#" /etc/logrotate.d/wtmp
After all logrotate.LogrotateTest finished, only cleanup
$HOME/logrotate_dir as below, but don't restore
the config file /etc/logrotate.d/wtmp.
[snip]
def tearDownClass(cls):
cls.tc.target.run('rm -rf $HOME/logrotate_dir')
[snip]
That's to say, there is one additional line added
to /etc/logrotate.d/wtmp and will make the logrotate
service start failed when run systemd.SystemdBasicTests.test_systemd_failed
Take an example as below when run test as root:
# cat /etc/logrotate.d/wtmp
# no packages own wtmp -- we'll rotate it here
/var/log/wtmp {
olddir /root/logrotate_dir
missingok
monthly
create 0664 root utmp
minsize 1M
rotate 1
}
# ls /root/logrotate_dir
ls: cannot access '/root/logrotate_dir': No such file or directory
# systemctl start logrotate
Job for logrotate.service failed because the control process exited with error code.
See "systemctl status logrotate.service" and "journalctl -xe" for details.
# systemctl status logrotate
logrotate.service - Rotate log files
Loaded: loaded (/lib/systemd/system/logrotate.service; static; vendor preset>
Active: failed (Result: exit-code) since Wed 2019-02-13 03:35:19 UTC; 7s ago
Docs: man:logrotate(8)
man:logrotate.conf(5)
Process: 540 ExecStart=/usr/sbin/logrotate /etc/logrotate.conf (code=exited, status=1/FAILURE)
Main PID: 540 (code=exited, status=1/FAILURE)
Feb 13 03:35:18 qemumips systemd[1]: Starting Rotate log files...
Feb 13 03:35:19 qemumips logrotate[540]: error: wtmp:9 error verifying olddir path /root/logrotate_dir: No such file or directory
Feb 13 03:35:19 qemumips logrotate[540]: error: found error in file wtmp, skipping
Feb 13 03:35:19 qemumips systemd[1]: logrotate.service: Main process exited, code=exited, status=1/FAILURE
Feb 13 03:35:19 qemumips systemd[1]: logrotate.service: Failed with result 'exit-code'.
Feb 13 03:35:19 qemumips systemd[1]: Failed to start Rotate log files.
Add the logic to restore /etc/logrotate.d/wtmp to
make the cleanup complete to fix the above issue.
(From OE-Core rev: a2db9320d97d12d87524ff16a329f9c38a8da33f)
Signed-off-by: Mingli Yu <Mingli.Yu@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
This turns the core of the script into a library function. Ultimately this
will let us call that code with custom 'keywords' rather than relying
on the data parsed from bitbake metadata which can't be used when archiving
historical results.
(From OE-Core rev: 4820ca2b0850e29b04a4fd5659a6e9837d6714d0)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
QA team found that 10 manual bsp test cases from Testopia for
Beaglebone, EdgeRouter, and MPC need to be up streamed
[YOCTO #12650]
(From OE-Core rev: f47410899eb97cfd7db428f45b7ba354c609b4d5)
Signed-off-by: Mazliana <mazliana.mohamad@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Fix another load of regex escape sequence warnings for newer
python versions.
(From OE-Core rev: bd2c125bb9c362b6122e99dfdf4e1cfe12c26a90)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
With newest Python, the following is printed:
PyGIDeprecationWarning: GObject.markup_escape_text is deprecated; use GLib.markup_escape_text instead)
(From OE-Core rev: 7b4302d84e6c9833089bd575c1083fe826082387)
Signed-off-by: Alexander Kanavin <alex.kanavin@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Currently processed being killed by the OOM killer may not be spotted by
ptest-runner. After we complete the tests, check the logs and report if there
were any. This ensures the user is aware of OOM conditions affecting the
ptest results.
(From OE-Core rev: 20a441d53817f80e0ce1597e77f6e794422ac49a)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Sometimes we meet the following failure for the test_lookup_recipe
test case.
AssertionError: 'zlib\nbusybox is in the RPROVIDES of target-sdk-provid[32 chars]ummy' != 'zlib\nbusybox'
zlib
+ busybox- busybox is in the RPROVIDES of target-sdk-provides-dummy:
- target-sdk-provides-dummy
This is because target-sdk-provides-dummy rprovides busybox.
So clean things up to avoid failure.
(From OE-Core rev: b3001770df6640549270361bfaa449cb3e79a0b7)
Signed-off-by: Chen Qi <Qi.Chen@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Get rid of further unneeded code complications:
* value mappings we could just direct use
* ftools when we can write files easily ourself
* test result status filtering we don't use
* variable overwriting module imports
(From OE-Core rev: d6065f136f6d353c3054cc3f440a4e259509f876)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Merge the results handling into the ptest log parser as a seperate
method.
Drop the weird "pass.skip.fail." prefix to the results filename, its
just bizarre.
Drop the code turning a list into a regex then searching the regex for
an item, "x in y" is perfectly capable.
Use a dict, sort the keys as needed and drop the list sorting code.
(From OE-Core rev: f317800e950b4a37b4034133bc52e0c47f04dc29)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Allow parsing of the ptest duration, exit code and timeout keywords
from the logs, returning data on each section.
Also include the logs broken out per section.
(From OE-Core rev: a9a67dccaa5be0f06eedcab46dcff7cbf9202850)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Now we have a dedicated ptest parser, merge in the remaining ptest
specific pieces to further clarify and simplify the code, moving to
a point where we can consider extending/enhancing it.
(From OE-Core rev: 05991bb5bc8018275d03fdeecee3d5a757840c7c)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Rename the paster to be ptest specific and apply some further cleanups
to the code to simplify and clarify what its doing.
(From OE-Core rev: 45a5886f1ec458d4c306b8d68fd31d568bc36b47)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
logparser is only used by ptest. Its slightly overcomplicated as it was
intended to be reusable but wasn't. Simplify it as a dedicated parser is
likely to me more readable and maintainable.
(From OE-Core rev: c7478345b2b4a85cb1fec40e762633871f0e94cb)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Some tests end up without a section, avoid tracebacks trying to use
None as a string in that case.
(From OE-Core rev: 86fb5d898a29761f120c2eaa538a32cf2e078487)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Manual test step for SDK is updated. Previously toolchain was "poky-glibc-x86_64-core-image-sato-sdk<type-arch>-toolchain-<release-version>.sh"
But toochain for core-image-sato-sdk is not available in releases after 2.1, hence changed it to "poky-glibc-x86_64-core-image-sato-<type-arch>-toolchain-<release-version>.sh
Other less significant update is to add exepcted result in intermediate test steps.
(From OE-Core rev: f5083ba27e667789c32dcd00bc1b883f4b76bd4f)
Signed-off-by: sangeeta jain <sangeeta.jain@intel.com>
"poky-glibc-x86_64-core-image-sato-<type-arch>-toolchain-<release-version>.sh"
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Added json file for compliance test cases to the manual runtime tests
(From OE-Core rev: 946f37d57f7b23c7875089ed98be9148ccc05890)
Signed-off-by: Sudhir Sharma <sudhirx.sharma@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Changes the default PACKAGE_DEBUG_SPLIT_STYLE to generate separate
source and debug packages. SDKIMAGE_FEATURES is updated to include the
source packages so that there is not change for the SDK contents.
[YOCTO #12931]
(From OE-Core rev: 8df14b678e44cc749b361224af05ccbcfa9ae9b5)
Signed-off-by: Joshua Watt <JPEWhacker@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
This is particularly useful when setting up GL tests.
(From OE-Core rev: 167a46775059b782c6f82ce8c5a47b27262e95d4)
Signed-off-by: Alexander Kanavin <alex.kanavin@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
When executing eSDK test case, the following error appears.
WARNING: attempting to use the extensible SDK in an environment
set up to run bitbake - this may lead to unexpected
results. Please source this script in a new shell session
instead.
FileExistsError: [Errno 17] File exists: '/.../tmp/work/qemux86_64-poky-linux/core-image-minimal/1.0-r0/selftest-esdk-fcuyzsqu/tmp/sysroots/x86_64/bin/pigz' -> '/.../tmp/hosttools/pigz'
So unset these two vars to avoid messing things up.
(From OE-Core rev: bc07d825ce8bb3b337623c232fef61f2781c82af)
Signed-off-by: Chen Qi <Qi.Chen@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
When removing the temporary directory, it's possible that bitbake.lock
file is removed by bitbake during the cleanup. And this leads to the
following error.
FileNotFoundError: [Errno 2] No such file or directory: 'bitbake.lock'
So add a check to remove this file before cleaning up the temporary
directory.
(From OE-Core rev: 984f56b37bd0014e5bf9509fc8ed181973e61773)
Signed-off-by: Chen Qi <Qi.Chen@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
A new commit was introduced to kernel, which caused kprobe sample case
failed on arm.
[kernel commit: e46daee53bb50b, ARM: 8806/1: kprobes: Fix false positive
with FORTIFY_SOURCE]
And according to author, the case just works on x86 and powerpc.
So we just promise that the case can pass on x86 and powerpc.
(From OE-Core rev: 1a5b14bc3fe29618beea016ab2b32f55c19dd6ad)
Signed-off-by: Hongzhi.Song <hongzhi.song@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
This test case boots the image in qemu and checks for mounted partitions.
But the outputs of mount are different between sysvinit and systemd:
sysvinit:
/dev/root /\r\n/dev/sda1 /boot\r\n/dev/sda3 /media\r\n/dev/sda4 /mnt
systemd:
/dev/sda1 /boot\r\n/dev/sda2 /\r\n/dev/sda3 /media\r\n/dev/sda4 /mnt
So check mounted partitions by egrep rather than check output of runqemu.
(From OE-Core rev: 8e3311d45ec68d74e934e966f67cf5102634981c)
Signed-off-by: Kai Kang <kai.kang@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
When using linux-yocto-dev, we will have one more amba error message
for qemuarm, so ignore it too.
(From OE-Core rev: 37cb293cf0f4194ec923d3bc4ebb9645f07536c0)
Signed-off-by: Chen Qi <Qi.Chen@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Now we can compile m4-native rather than target m4, this can save a lot of
build time.
(From OE-Core rev: 8e33cef647f4e8a104da61aecab4ac6c6b9bfd3b)
Signed-off-by: Robert Yang <liezhi.yang@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
There are failures when multiple users run oe-selftest on the same
host:
PermissionError: [Errno 13] Permission denied: '/tmp/oe-saved-
tests/201812250324_qemu'
This is because /tmp/oe-saved-tests was created by user A, while user B tries
to write data in it, then the error will happen. This patch can fix the
problem.
Move the dumped data to ${LOG_DIR}/runtime-hostdump/ rather than
/tmp/oe-saved-tests/ to fix the problem.
(From OE-Core rev: e219fe5329599cd6c3682f521eaee3852a2c8980)
Signed-off-by: Robert Yang <liezhi.yang@windriver.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
And fix the reported upstream check failures.
(From OE-Core rev: 63af0edcd954fac530ba17b04e5df5837cddc0a4)
Signed-off-by: Alexander Kanavin <alex.kanavin@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Unify style with the other tests.
(From OE-Core rev: 545cde27b13f9d68211fd3a671182203ac47756d)
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Use a different STAMPS_DIR and SSTATE_DIR in test_yocto_source_mirror.
Otherwise, when executing `oe-selftest -a', we will get a lot of failures
due to do_unpack failure.
(From OE-Core rev: f8502c68163ec288cf6fb6754829b455501b4a05)
Signed-off-by: Chen Qi <Qi.Chen@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
(From OE-Core rev: 564de3681353fe8e203425388e8be9703a89d2da)
Signed-off-by: Robert Yang <liezhi.yang@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Now QEMU_USE_KVM can only be boolean, can not contain MACHINE any more.
(From OE-Core rev: 37e5035a2ba27595eeef22062580a5b04a73ba15)
Signed-off-by: Robert Yang <liezhi.yang@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Fixed:
MACHINE = "qemux86"
QEMU_USE_KVM = "qemux86"
IMAGE_CLASSES += "testimage"
$ oe-selftest -r runqemu.RunqemuTests.test_boot_rootfs
[snip]
File "/buildarea1/lyang1/poky/meta/lib/oe/types.py", line 122, in boolean
raise ValueError("Invalid boolean value '%s'" % value)
ValueError: Invalid boolean value 'qemux86'
Now QEMU_USE_KVM can only be boolean, can not contain MACHINE any more, kvm
will be enabled if target_arch == build_arch or both of them are x86 archs.
(From OE-Core rev: 7c1a8a624cad8d967635c6cb5f99cf655bde3d44)
Signed-off-by: Robert Yang <liezhi.yang@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The 5s timeout for non-kvm is too short, especially when the load is high,
which leads to unexpected errors, so set timeout to 60s by default.
(From OE-Core rev: 8197d0f638a760fc03062c7a9009117d083d7ead)
Signed-off-by: Robert Yang <liezhi.yang@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
This is useful for debugging.
(From OE-Core rev: da527d73f5eeae0f29b5f99aab757491d3f87ec7)
Signed-off-by: Robert Yang <liezhi.yang@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The previous code assumed builddir and meta-selftest are in the same dir, but
this isn't always true, builddir can be anywhere, use get_test_layer() to
locate meta-selftest can fix the problem.
(From OE-Core rev: 56d2493a9adfcc47ae7e265439e05ff42cdbbbbf)
Signed-off-by: Robert Yang <liezhi.yang@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Rather than use the obsolete do_checkpkg function, use the new recipeutils
function which uses tinfoil to get the data rather than needing csv file
manipulation.
(From OE-Core rev: 3f3f80b00cd999f1b2aef8f5c0ce0900aa4dcbcb)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Use tinfoil to enumerate recipes and get the value of RECIPE_MAINTAINER
to make it a bit more reliable in the face of do_checkpkg issues we are
currently seeing on the Yocto Project autobuilder. This also makes it a
little less painful to re-execute test_maintainers() since you don't
have to wait for bitbake -c checkpkg to complete every time.
Note that the new test has been written in such a way that it will still
function if RECIPE_MAINTAINER values are ever moved to the recipes.
Also, the test still currently fails as there are recipes that don't
have an assigned maintainer.
(From OE-Core rev: 47282a2f6f12acebf58961ea9410cfbc335d560b)
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Don't use the helper class as it gets in the way more than it helps, exercise
the out-of-tree paths, and verify the installed files match the expected
architecture.
(From OE-Core rev: 6d666b0413336de2e556b2722c5be97ae5cd40ad)
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Don't use the helper class as it gets in the way more than it helps, exercise
the out-of-tree paths, and verify the installed files match the expected
architecture.
(From OE-Core rev: 920ae8c6537c2469f21ab9439587fd094ecc40f6)
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Drop redundant imports and variables, and use os.makedirs() instead of
bb.utils.mkdirhier().
(From OE-Core rev: 2de9b1e611e5047afb540f98756994925c22e446)
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Use oeqa.utils.subprocesstweak to monkey-patch the subprocess exception so that
any output is shown, and remove any explicit try/catch handling that would have
hidden this.
(From OE-Core rev: 55964b33b561397287779ee474170790dfd03e85)
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
For example, instead of saying "3 != 62", say "Binary was x86-64 but expected
i586".
(From OE-Core rev: 9ab94cea589fca4394ec1fd8dc06b23fd8e990b9)
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
(From OE-Core rev: 7c2c9b876334cc7357f818c3b3744fc97987e38f)
Signed-off-by: Ola x Nilsson <olani@axis.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Expecting 1s accuracy on a 2s timeout on a heavily loaded system has proven to be
unreliable. Update this to a 5s timeout with a 3s delta which should be achievable.
(From OE-Core rev: 5feecb639d49d72d8a6abc589b937e07ee72f252)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Ensure log directory exists to avoid the following error.
FileNotFoundError: [Errno 2] No such file or directory: '/.../build-selftest/tmp/log/oe-selftest-results-20181207043431.log'
(From OE-Core rev: 6c41de48db76087fee596d9440d8f05346ab1094)
Signed-off-by: Chen Qi <Qi.Chen@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
INFO - ======================================================================
INFO - ERROR: setUpClass (eSDK.oeSDKExtSelfTest)
INFO - ----------------------------------------------------------------------
INFO - Traceback (most recent call last):
File "/home/pokybuild/yocto-worker/oe-selftest-debian/build/meta/lib/oeqa/core/case.py", line 32, in _oeSetUpClass
clss.setUpClassMethod()
File "/home/pokybuild/yocto-worker/oe-selftest-debian/build/meta/lib/oeqa/selftest/cases/eSDK.py", line 76, in setUpClass
cls.tmpdirobj = tempfile.TemporaryDirectory(prefix="selftest-esdk-", dir=bb_vars["WORKDIR"])
File "/usr/lib/python3.5/tempfile.py", line 929, in __init__
self.name = mkdtemp(suffix, prefix, dir)
File "/usr/lib/python3.5/tempfile.py", line 507, in mkdtemp
_os.mkdir(file, 0o700)
FileNotFoundError: [Errno 2] No such file or directory: '/home/pokybuild/yocto-worker/oe-selftest-debian/build/build/tmp/work/qemux86_64-poky-linux/core-image-minimal/1.0-r0/selftest-esdk-q7ln84gc'
(From OE-Core rev: 48719bc4d108df7e357e7f22f9f356cd72b3ebbd)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2018-12-06 23:19:24,564 - oe-selftest - INFO - Traceback (most recent call last):
File "/media/build1/poky-sumo/meta/lib/oeqa/core/case.py", line 32, in _oeSetUpClass
clss.setUpClassMethod()
File "/media/build1/poky-sumo/meta/lib/oeqa/selftest/cases/eSDK.py", line 78, in setUpClass
cls.tmpdir_eSDKQA = cls.tempdirobj.name
AttributeError: type object 'oeSDKExtSelfTest' has no attribute 'tempdirobj'
(From OE-Core rev: c6de27c2f6f598849dcb8036ee849f449ba7f327)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
We've seen issues with rootfs size calculations and we've seen systems
like opensuse which have btrfs mounted on /tmp causing selftest failures.
(From OE-Core rev: 61be3cd748d1b7321a1fc4cfe84efa9b26a6aee0)
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
If we need to create a temporary directory in targetbuild or buildproject use
tempfile.TemporaryDirectory so that when the test case is finished, the
directory is deleted.
Also synchronise the logic and don't possibly store the temporary directory in
self.tmpdir as nothing uses that.
(From OE-Core rev: db0e658097130d146752785d0d45f46a3e0bad71)
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Reorder the shutdown/teardown to avoid:
File "/home/pokybuild/yocto-worker/oe-selftest-ubuntu/build/meta/lib/oeqa/utils/qemurunner.py", line 224, in launch
op = self.getOutput(output)
File "/home/pokybuild/yocto-worker/oe-selftest-ubuntu/build/meta/lib/oeqa/utils/qemurunner.py", line 90, in getOutput
fl = fcntl.fcntl(o, fcntl.F_GETFL)
ValueError: I/O operation on closed file
(From OE-Core rev: 8e7d756862d2a8d62f3c87497d6d65ddb3c1b962)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
assertTrue doesn't give good debug information when things fail. Update
several to use assertIn which gives information upon failure, for the
others print the log information upon failure.
(From OE-Core rev: c29cb75d5ce6b0873a934f4709b0c8824f7164d3)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
We've seen a cryptic:
"ERROR: Fatal errors occurred in subprocesses, tracebacks printed above"
message from oe-selftest with no other traceback information. Improve the
traceback logging to try and give a better indication of any errors that is
ocurring.
(From OE-Core rev: 521dd3d00979a27b6932e58d5497de68abac26e1)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
setUp() is used to populate a directory of temporary files, and deleted in
__del__. However setUp() is called once *per test* so __del__ would only be
able to remove the last directory created.
Fix the code by using the natural counterpart to setUp, tearDown(), to clean up.
(From OE-Core rev: 68b4723e6fb11d171869185bccf28f32f6284c18)
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Add a Python 2 form to exercise that if present, and fix the setUp() so it
actually looks for a package that exists (nativesdk-python3 is a virtual
package, the interpretter is in nativesdk-python3-core).
(From OE-Core rev: d286c2ad3eec24978557e16a8fa599476791109f)
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Add some tests for functions in meta/lib/oe/recipeutils.py, in
particular for a few issues I've just fixed. I haven't added tests for
all of the functions - some of them are already being tested via devtool
in any case.
(From OE-Core rev: 72d0cb3f8f1e69aeef93ea0bc90db3e8b8d6f94c)
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The existing logfile is simply placed in the current directory. Since the test
changes cwd to BUILDDIR, the symlink to the log can be placed in an invalid
directory. We also see trackbacks if the symlink is invalid.
Improve things by:
* Placing logs in LOG_DIR (or BUILDDIR if unset).
* Using a full path to the log meaning the log and link are placed in the same directory.
* Using lexists instead of exists so invalid symlinks are handled correctly.
(From OE-Core rev: 750ece11bed0e62a11e0003d1d16a81f7c219761)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Heartbeat events default to once a second and we need to ensure we have
enough time in the task to see them.
Add a nostamp delay task 5s long so we can have a consistently timed
task which doesn't need cleanup or have unneeded dependencies. This
ensures we should deterministically see the disk moinitor events
regardless of the state of the build. This is done in a way which
doesn't corrupt build state or need cleanup and is efficient.
(From OE-Core rev: ecc49ee8986929e2429d948000a0ca588fe63959)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
This test occisionally fails as m4 doesn't recompile, meaning the logfile test
then doesn't find mention of ccache.
To ensure m4 does recompile, clean m4 before force compiling it.
(Reading the test is confusing due to the test cleanup also involving a clean)
(From OE-Core rev: 6e0b9214a0d57ed45a5df0ba5c9887a9045b89b1)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
If runqemu fails it would leak an unclosed socket and file. Ensure we
close these in all cases to remove the resource warning.
(From OE-Core rev: ed80e46ccbc8fe8e9148d80723152066fa00ba28)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Each time a runqemu() fails, the log handler would be left behind meaning
messages from any subsequent run would be duplicated (or worse/more).
This ensures we remove the handler regardless and means we no longer
have the duplication.
(From OE-Core rev: 532984708436bdfa3a8cac2c684a425eb249bad0)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Rather than just referring the user to the logs containing the failure, print
them on the console. This aids debugging with oe-selftest with parallelisation
as the logs may otherwise be lost.
(From OE-Core rev: 36a018e245a232f520ff946f152cc875927a6fb4)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The current failure mode doesn't show us what the logs actually looked like
and later cleans can lose them. Show the whole log in case of failure
to aid debugging intermittent problems on the autobuilder.
(From OE-Core rev: 7c3a0dc5978cea898b1ca51decf4d6e7cf9d519f)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
This avoids problems where shutil.remove will error with:
File "/usr/lib/python3.5/shutil.py", line 436, in _rmtree_safe_fd
os.unlink(name, dir_fd=topfd)
FileNotFoundError: [Errno 2] No such file or directory: 'S.gpg-agent.extra'
when there are races over file deletion (gpg agent may be slow to exit).
We already worked around speed and race issues in bb.utils.
(From OE-Core rev: 00a8fd5b93a5c19ce0b7498e2bc653ce8ad58aaf)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
This should speed the test up signficiantly without any loss of functionality
for the purposes of the test.
(From OE-Core rev: 3dde0b749643575878bfbca2f8d2d9ec30bad166)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
We ideally don't want to use gpg from the host. This is straightforward for package
management but not for sstate.
For sstate, create a second build directory to run the test in using gnupg-native
from the original build directory.
(From OE-Core rev: 10afa94c3f0d7eb7524a26deda86949073d55fde)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Raising an assertionError in the class setup isn't a particuarly good way to
indicate gpg isn't installed. Instead skip the tests if the required binary
isn't present. For the signing tests we do require it to be present and can't
use a prebuilt one.
(From OE-Core rev: 2d486af97e51b9daa9c40482c31d637c9ab4ae79)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Enable qemu for slirp. Initialize Qemurunner with slirp. Setup ip
and port attribute to enable connection with qemu running with slirp.
[YOCTO#10713]
(From OE-Core rev: 815e05ac4abb42238d6fdeb15d7426759e059ce1)
Signed-off-by: Yeoh Ee Peng <ee.peng.yeoh@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Enable qemurunner for slirp. Retrieved the ip & port from host machine
to connect to qemu from host machine.
[YOCTO#10713]
(From OE-Core rev: 1db6a6fc9cde28d0a29bcf6d24a8bfbe51d120b1)
Signed-off-by: Yeoh Ee Peng <ee.peng.yeoh@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Qemu do not use the ip input from external. It will
retrieve ip from QemuRunner instance and assign
ip value.
(From OE-Core rev: 14d99dc6c39c963ba3e0d9a30274846bd5369210)
Signed-off-by: Yeoh Ee Peng <ee.peng.yeoh@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Splits the SDK test implementation into configurable Python classes. The
classes used for the normal and extensible SDKs are
${TESTSDK_CLASS_NAME} and ${TESTSDKEXT_CLASS_NAME} respectively.
This allows SDK machines to override the classes used to implement the
tests. For the traditional SDK, a common "run()" function is provided by
the class (oeqa.sdk.testsdk.TestSDK), with several hook member functions
that can be overridden in child classes, making it easier to have
consistent behavior. The extensible SDK class
(oeqa.sdkext.testsdk.TestSDKEXT) also has a common "run()" function, but
no hooks have yet been added as there is not currently a known use case
for create derived classes.
These changes should be purely organizational; no functional changes
have been made to either the standard SDK or extensible SDK tests.
[YOCTO #13020]
(From OE-Core rev: a06d53928b22d5f88276023c4d57b206db2f27f9)
Signed-off-by: Joshua Watt <JPEWhacker@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
testimage.bbclass installs a SIGTERM handler which conflicts with the
use of multiprocessing here. This is paritcularly problematic if the http
service is terminated before its started and hence before its had a chance
to reset the default signal handler (as the code was written).
Instead, temporarily remove testimage's handler whilst forking the http process
which means the correct handler is installed and won't deadlock.
Also take the opportunity to add in some log messages about the server start
and shutdown so that future debugging is easier and its clearer what the code
is doing.
(From OE-Core rev: 0762b6021b87ceb1f37952f3a6d64a36e99ae6a5)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Use our own lsb function instead as used elsewhere by the codebase.
(From OE-Core rev: acac45a6fd604d28ef7c23d67482af3d7e8bcfe3)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Avoid an unclosed file per thread warning when running selftests concurrently
by closing the result stream.
(From OE-Core rev: 33a4a076e8aa72a872807332501e7f5ae1cee0e2)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Avoid the warning:
meta/lib/oeqa/selftest/context.py:8: DeprecationWarning: the imp module is deprecated in favour of importlib; see the module's documentation for alternative uses
import imp
In this case importlib is a direct replacement.
(From OE-Core rev: db7a60c36a2d3eefc61ae6e1ede01680dc932035)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Clean up the warning:
meta/lib/oeqa/core/loader.py:27: DeprecationWarning: inspect.getargspec() is deprecated, use inspect.signature() or inspect.getfullargspec()
_failed_test_args = inspect.getargspec(unittest.loader._make_failed_test).args
(From OE-Core rev: d2deb66830be2d44532fea3d5db763b57778252a)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
