Commit Graph

621 Commits

Author SHA1 Message Date
Ross Burton
696cf580a5 oeqa/recipetool: change the CMake test to use taglib
The current test builds Navit, which uses GTK+ 2.  As GTK+ 2 is being removed
from oe-core, change the test to build taglib instead.

(From OE-Core rev: f627bce009dd275df807ffbc53b880c2490559ae)

Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-07-11 09:32:50 +01:00
Richard Purdie
2a6094ca8c oeqa/bbtests: Tweak test bitbake output pattern matching
The output from bitbake will change slightly soon due to runqueue changes,
adpat the test now to account for both the old and new cases.

(From OE-Core rev: 78fcea74517de4793cc0ecc97bce7f5c7dcd44c0)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-07-09 23:30:44 +01:00
Chen Qi
5fc7930064 devtool.py: track to clean devtool.conf in test_create_workspace
`devtool create-workspace' would create devtool.conf, so track to
clean it up. Otherwise, this devtool.conf file would mess things up.

e.g.
oe-selftest -r devtool && oe-selftest -r devtool -j 2
AssertionError: '/PROJ_DIR/build-selftest-st-15753/workspace/conf/layer.conf' does not exist : Workspace directory not created

This is because the devtool.conf is also copied to build-selftest-st-xxxx
directory, resulting in devtool to create and use workspace specified
in this file.

(From OE-Core rev: a0a96db23686a38235fb0373e75e8b2951216c18)

Signed-off-by: Chen Qi <Qi.Chen@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-07-03 17:00:57 +01:00
Joshua Watt
17268ae841 oeqa: Test bitbake --skip-setsecene
Tests the bitbake --skip-setscene option to verify that it correctly
reuses tasks previously restored from sstate, but doesn't execute new
sstate tasks

(From OE-Core rev: af10c620b1496bf5224d0628a92cb42e9cd362a8)

Signed-off-by: Joshua Watt <JPEWhacker@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-07-03 17:00:57 +01:00
Robert Yang
15cdacb8c0 oeqa: Cleanup /usr/bin/env python
(From OE-Core rev: eb79217b8761816a21c8f7bed3c5379c1b9230ea)

Signed-off-by: Robert Yang <liezhi.yang@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-06-30 22:40:52 +01:00
Ross Burton
bb75a12425 oeqa/buildoptions: check that Fortran code actually cross-compiles
Don't just test that we can build the cross-compiler, but test that it actually
can cross-compile some Fortran.

The quadmath dependency is now handled in gcc-runtime and isn't needed in this
test (as per local.conf.sample.extended changes).

There's also no need to build libgfortran explicitly, as fortran-helloworld depends on it.

(From OE-Core rev: 4b43c655ed5ff8b9d2662730526811220b21ff8c)

Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-06-27 12:20:36 +01:00
Chen Qi
f25d8604ea manifest.py: fix test_SDK_manifest_entries
TOOLCHAIN_OUTPUTNAME could be overridden. So use this variable directly
instead of its default value ${SDK_NAME}-toolchain-${SDK_VERSION}.

(From OE-Core rev: 35a13468f551fff6305ba1ebbcb5dd28e99ec050)

Signed-off-by: Chen Qi <Qi.Chen@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-06-19 12:46:43 +01:00
Oleksandr Kravchuk
ca9c738053 selftest/devtool: fix URI to MarkupSafe package
PyPi packages are now hosted at files.pythonhosted.org.

[YOCTO #13243]

(From OE-Core rev: 384e2c3765c77c29527a43f2e0bffbc41170eeb6)

Signed-off-by: Oleksandr Kravchuk <open.source@oleksandr-kravchuk.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-06-14 22:48:22 +01:00
Chen Qi
d9b0adbce8 oescripts.py: avoid error when cairo module is not available
When running 'oe-selftest -r oescripts', the following error appeared.

    cls.skipTest('Python module cairo is not present')
    TypeError: skipTest() missing 1 required positional argument: 'reason'

This is because the host does not have the cairo python module installed.
Fix this problem by using unittest's SkipTest exception.

(From OE-Core rev: 785bb108ceb9dc137ec8d75d887b6a3869cbfb9c)

Signed-off-by: Chen Qi <Qi.Chen@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-06-14 22:48:22 +01:00
Quentin Schulz
e3d0dd9163 selftests: add tests for INCOMPATIBLE_LICENSE
One bug went unnoticed without these selftests: an INCOMPATIBLE_LICENSE
with a non-SPDX license for a package with that non-SPDX license wasn't
enforcing the denial of build for said package. See
4b6ce4604c ("meta: license: fix non-SPDX
license being removed from INCOMPATIBLE_LICENSE")

While adding a test for that particular case, let's add a few more so
that we cover a handful more use cases of INCOMPATIBLE_LICENSE.

(From OE-Core rev: 6c2ca52218c196e7ccf6b3275bffc3e3a04193c6)

Signed-off-by: Quentin Schulz <quentin.schulz@streamunlimited.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-06-12 10:54:40 +01:00
Maciej Pijanowski
bb59bcd016 recipetool: add python3 support
Add support for generating python3 recipes using the recipetool / devtool.
Drop python2 support at the same time.

Tested with:

oe-selftest -r recipetool.RecipetoolTest

[YOCTO #13264]

(From OE-Core rev: d8b2f58974482b3b1ccc65c5f93104d0d7ba87bc)

Signed-off-by: Maciej Pijanowski <maciej.pijanowski@3mdeb.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-06-10 17:38:10 +01:00
Joshua Watt
e5217b6c10 oeqa: Add reproducible build selftest
Adds an initial test for reproducible builds to the OE selftest. This
initial test builds core-image-minimal using sstate, then does a clean
build without sstate in another build directory, and finally does a
binary comparison of the resulting package files between the two builds.

The test is currently always skipped since it doesn't pass yet, but it
can easily be enabled locally

(From OE-Core rev: 2e591bdf93ec9e59b900562263dfe8e72b163baa)

Signed-off-by: Joshua Watt <JPEWHacker@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-06-10 14:46:38 +01:00
Richard Purdie
1334b064d3 gpg_sign/selftest: Fix secmem parameter handling
We keep seeing "cannot allocate memory" errors from rpm when signing packages
on the autobuilder. The following were tried:

* checking locked memory use (isn't hitting limits)
* Restricting RPM_GPG_SIGN_CHUNK to 1
* Limiting to 10 parallel do_package_write_rpm tasks
* Allowing unlimied memory overcommit
* Disabling rpm parallel compression

and the test still failed. Further invetigation showed that the --auto-expand-secmem
wasn't being passed to gpg-agent which meant the secmem couldn't be expanded hence the
errors when there was pressure on the agent.

The reason this happens is that some of the early gpg commands can start the agent
without the option and it sticks around in memory so a version with the correct
option may or may not get started.

We therefore add the option to all the key gpg calls.

(From OE-Core rev: c7e131a76e522503df55e211dd261829feacfa28)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-06-07 09:11:49 +01:00
Jon Mason
cb153e8da8 resulttool: modify to be multi-machine
Currently, the code will sum all of the different machine results into a
single report of the tests results.  This can lead to confusion as to
which machine may be experiencing issues.  Modify the code to store the
results in a per machine basis and report them accordingly.

(From OE-Core rev: 16d4031ea5df8a4ddfdb937d35464c09e1abd10e)

Signed-off-by: Jon Mason <jdmason@kudzu.us>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-06-04 09:09:42 +01:00
Alistair Francis
ebedc1511d gdb: Upgrade from 8.2.1 to 8.3
Bump GDB and change the oeqa selftest expression to allow ptests to
pass.

(From OE-Core rev: f31986ed9f82c48834b2f25979c93697228acbf7)

Signed-off-by: Alistair Francis <alistair.francis@wdc.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-05-21 12:56:33 +01:00
Richard Purdie
5a16dee75f oeqa/selftest: Automate manual pybootchart tests
Automate the current manual pybootchart tests. This includes a check
for the cairo dependency, skipping the test if appropriate.

Based on original patch from Armin Kuster <akuster808@gmail.com>

(From OE-Core rev: ff5370a381a4996b7da56aaaa7055f7a1786c823)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-05-12 09:04:26 +01:00
Richard Purdie
ffae400179 meta/lib+scripts: Convert to SPDX license headers
This adds SPDX license headers in place of the wide assortment of things
currently in our script headers. We default to GPL-2.0-only except for the
oeqa code where it was clearly submitted and marked as MIT on the most part
or some scripts which had the "or later" GPL versioning.

The patch also drops other obsolete bits of file headers where they were
encoountered such as editor modelines, obsolete maintainer information or
the phrase "All rights reserved" which is now obsolete and not required in
copyright headers (in this case its actually confusing for licensing as all
rights were not reserved).

More work is needed for OE-Core but this takes care of the bulk of the scripts
and meta/lib directories.

The top level LICENSE files are tweaked to match the new structure and the
SPDX naming.

(From OE-Core rev: f8c9c511b5f1b7dbd45b77f345cb6c048ae6763e)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-05-09 16:31:55 +01:00
Richard Purdie
c7592b0147 oeqa: Drop OETestID
These IDs refer to testopia which we're no longer using. We would now use the test
names to definitively reference tests and the IDs can be dropped, along with their
supporting code.

(From OE-Core rev: 8e2d0575e4e7036b5f60e632f377a8ab2b96ead8)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-05-09 16:31:55 +01:00
Tom Rini
648cd455dc imagefeatures: Add test to blacklist building busybox
Now that we have a packagegroup that can be used to replace the overall
functionality of busybox as base-utils, add a test that we can continue
to build a fairly complex image without busybox being available.  This
also doubles as documentation-by-test of how to do this.

(From OE-Core rev: 6319a59c1d30eeb8ad4871d43641e3469fb543ba)

Signed-off-by: Tom Rini <trini@konsulko.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-03-06 23:12:27 +00:00
Alexander Kanavin
d3c7d1a036 selftest: add tests for virgl GL acceleration
Note that the tests require that the host machine has a X display,
has mesa development files installed and is able to create OpenGL contexts.

(From OE-Core rev: 2868e8dfb9e62b49cd06f6c2d010405079d3a71c)

Signed-off-by: Alexander Kanavin <alex.kanavin@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-03-04 14:27:06 +00:00
Richard Purdie
d9018a3d9c selftest: Add multiconfig test
Add a test for a multiconfig build which mixes tiny and musl builds
along with using the mcextend class to combine and package multiple images
into another image. This gives the multiconfig a decent test in a scenario
users may use.

(From OE-Core rev: 0c7fa15a7350808242754944243f01155bc6784c)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-03-04 14:27:06 +00:00
Richard Purdie
47eb3d00e9 resulttool: Improvements to allow integration to the autobuilder
This is a combined patch of the various tweaks and improvements I
made to resulttool:

* Avoid subprocess.run() as its a python 3.6 feature and we
  have autobuilder workers with 3.5.

* Avoid python keywords as variable names

* Simplify dict accesses using .get()

* Rename resultsutils -> resultutils to match the resultstool ->
  resulttool rename

* Formalised the handling of "file_name" to "TESTSERIES" which the code
  will now add into the json configuration data if its not present, based
  on the directory name.

* When we don't have failed test cases, print something saying so
  instead of an empty table

* Tweak the table headers in the report to be more readable (reference
  "Test Series" instead if file_id and ID instead of results_id)

* Improve/simplify the max string length handling

* Merge the counts and percentage data into one table in the report
  since printing two reports of the same data confuses the user

* Removed the confusing header in the regression report

* Show matches, then regressions, then unmatched runs in the regression
  report, also remove chatting unneeded output

* Try harder to "pair" up matching configurations to reduce noise in
  the regressions report

* Abstracted the "mapping" table concept used to pairing in the
  regression code to general code in resultutils

* Created multiple mappings for results analysis, results storage and
  'flattening' results data in a merge

* Simplify the merge command to take a source and a destination,
  letting the destination be a directory or a file, removing the need for
  an output directory parameter

* Add the 'IMAGE_PKGTYPE' and 'DISTRO' config options to the regression
  mappings

* Have the store command place the testresults files in a layout from
  the mapping, making commits into the git repo for results storage more
  useful for simple comparison purposes

* Set the oe-git-archive tag format appropriately for oeqa results
  storage (and simplify the commit messages closer to their defaults)

* Fix oe-git-archive to use the commit/branch data from the results file

* Cleaned up the command option help to match other changes

* Follow the model of git branch/tag processing used by oe-build-perf-report
  and use that to read the data using git show to avoid branch change

* Add ptest summary to the report command

* Update the tests to match the above changes

(From OE-Core rev: ff2c029b568f70aa9960dde04ddd207829812ea0)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-02-21 12:34:00 +00:00
Yeoh Ee Peng
1fd5ebdb06 resulttool: enable merge, store, report and regression analysis
OEQA outputs test results into json files and these files were
archived by Autobuilder during QA releases. Example: each oe-selftest
run by Autobuilder for different host distro generate a
testresults.json file.

These scripts were developed as a test result tools to manage
these testresults.json file.

Using the "store" operation, user can store multiple testresults.json
files as well as the pre-configured directories used to hold those files.

Using the "merge" operation, user can merge multiple testresults.json
files to a target file.

Using the "report" operation, user can view the test result summary
for all available testresults.json files inside a ordinary directory
or a git repository.

Using the "regression-file" operation, user can perform regression
analysis on testresults.json files specified. Using the "regression-dir"
and "regression-git" operations, user can perform regression analysis
on directory and git accordingly.

These resulttool operations expect the testresults.json file to use
the json format below.
{
    "<testresult_1>": {
        "configuration": {
            "<config_name_1>": "<config_value_1>",
            "<config_name_2>": "<config_value_2>",
            ...
            "<config_name_n>": "<config_value_n>",
        },
        "result": {
            "<testcase_namespace_1>": {
                "status": "<PASSED or FAILED or ERROR or SKIPPED>",
                "log": "<failure or error logging>"
            },
            "<testcase_namespace_2>": {
                "status": "<PASSED or FAILED or ERROR or SKIPPED>",
                "log": "<failure or error logging>"
            },
            ...
            "<testcase_namespace_n>": {
                "status": "<PASSED or FAILED or ERROR or SKIPPED>",
                "log": "<failure or error logging>"
            },
        }
    },
    ...
    "<testresult_n>": {
        "configuration": {
            "<config_name_1>": "<config_value_1>",
            "<config_name_2>": "<config_value_2>",
            ...
            "<config_name_n>": "<config_value_n>",
        },
        "result": {
            "<testcase_namespace_1>": {
                "status": "<PASSED or FAILED or ERROR or SKIPPED>",
                "log": "<failure or error logging>"
            },
            "<testcase_namespace_2>": {
                "status": "<PASSED or FAILED or ERROR or SKIPPED>",
                "log": "<failure or error logging>"
            },
            ...
            "<testcase_namespace_n>": {
                "status": "<PASSED or FAILED or ERROR or SKIPPED>",
                "log": "<failure or error logging>"
            },
        }
    },
}

To use these scripts, first source oe environment, then run the
entry point script to look for help.
    $ resulttool

To store test result from oeqa automated tests, execute the below
    $ resulttool store <source_dir> <git_branch>

To merge multiple testresults.json files, execute the below
    $ resulttool merge <base_result_file> <target_result_file>

To report test report, execute the below
    $ resulttool report <source_dir>

To perform regression file analysis, execute the below
    $ resulttool regression-file <base_result_file> <target_result_file>

To perform regression dir analysis, execute the below
    $ resulttool regression-dir <base_result_dir> <target_result_dir>

To perform regression git analysis, execute the below
    $ resulttool regression-git <source_dir> <base_branch> <target_branch>

[YOCTO# 13012]
[YOCTO# 12654]

(From OE-Core rev: 78a322d7be402a5b9b5abf26ad35670a8535408a)

Signed-off-by: Yeoh Ee Peng <ee.peng.yeoh@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-02-21 12:34:00 +00:00
Richard Purdie
e1e38c9bd1 lib/oe,oeqa/selftest: Fix DeprecationWarning: invalid escape sequence
Fix another load of regex escape sequence warnings for newer
python versions.

(From OE-Core rev: bd2c125bb9c362b6122e99dfdf4e1cfe12c26a90)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-02-16 08:19:34 +00:00
Chen Qi
b68a42596d pkgdata.py: avoid target-sdk-dummy-provides to mess things up
Sometimes we meet the following failure for the test_lookup_recipe
test case.

  AssertionError: 'zlib\nbusybox is in the RPROVIDES of target-sdk-provid[32 chars]ummy' != 'zlib\nbusybox'
    zlib
  + busybox- busybox is in the RPROVIDES of target-sdk-provides-dummy:
  - target-sdk-provides-dummy

This is because target-sdk-provides-dummy rprovides busybox.

So clean things up to avoid failure.

(From OE-Core rev: b3001770df6640549270361bfaa449cb3e79a0b7)

Signed-off-by: Chen Qi <Qi.Chen@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-01-31 23:23:26 +00:00
Joshua Watt
ba3aa53112 Change default debug split to make separate source packages
Changes the default PACKAGE_DEBUG_SPLIT_STYLE to generate separate
source and debug packages. SDKIMAGE_FEATURES is updated to include the
source packages so that there is not change for the SDK contents.

[YOCTO #12931]

(From OE-Core rev: 8df14b678e44cc749b361224af05ccbcfa9ae9b5)

Signed-off-by: Joshua Watt <JPEWhacker@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-01-28 23:14:09 +00:00
Chen Qi
9a9124e50c eSDK.py: unset BBPATH and BUILDDIR to avoid eSDK failure
When executing eSDK test case, the following error appears.

  WARNING: attempting to use the extensible SDK in an environment
  	   set up to run bitbake - this may lead to unexpected
	   results. Please source this script in a new shell session
	   instead.

  FileExistsError: [Errno 17] File exists: '/.../tmp/work/qemux86_64-poky-linux/core-image-minimal/1.0-r0/selftest-esdk-fcuyzsqu/tmp/sysroots/x86_64/bin/pigz' -> '/.../tmp/hosttools/pigz'

So unset these two vars to avoid messing things up.

(From OE-Core rev: bc07d825ce8bb3b337623c232fef61f2781c82af)

Signed-off-by: Chen Qi <Qi.Chen@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-01-24 17:45:25 +00:00
Chen Qi
15c54f56d6 eSDK.py: avoid error in tearDownClass due to race condistion
When removing the temporary directory, it's possible that bitbake.lock
file is removed by bitbake during the cleanup. And this leads to the
following error.

  FileNotFoundError: [Errno 2] No such file or directory: 'bitbake.lock'

So add a check to remove this file before cleaning up the temporary
directory.

(From OE-Core rev: 984f56b37bd0014e5bf9509fc8ed181973e61773)

Signed-off-by: Chen Qi <Qi.Chen@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-01-24 17:45:25 +00:00
Kai Kang
83ade722e1 selftest/wic: update test case test_qemu
This test case boots the image in qemu and checks for mounted partitions.
But the outputs of mount are different between sysvinit and systemd:

sysvinit:
/dev/root /\r\n/dev/sda1 /boot\r\n/dev/sda3 /media\r\n/dev/sda4 /mnt
systemd:
/dev/sda1 /boot\r\n/dev/sda2 /\r\n/dev/sda3 /media\r\n/dev/sda4 /mnt

So check mounted partitions by egrep rather than check output of runqemu.

(From OE-Core rev: 8e3311d45ec68d74e934e966f67cf5102634981c)

Signed-off-by: Kai Kang <kai.kang@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-01-16 15:35:07 +00:00
Robert Yang
227aafa1cf oeqa/selftest/cases: Update test_ccache_tool
Now we can compile m4-native rather than target m4, this can save a lot of
build time.

(From OE-Core rev: 8e33cef647f4e8a104da61aecab4ac6c6b9bfd3b)

Signed-off-by: Robert Yang <liezhi.yang@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-01-14 11:35:55 +00:00
Alexander Kanavin
eca87bf2b4 selftest/distrodata: un-break the upstream version check test
And fix the reported upstream check failures.

(From OE-Core rev: 63af0edcd954fac530ba17b04e5df5837cddc0a4)

Signed-off-by: Alexander Kanavin <alex.kanavin@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-01-11 10:39:08 +00:00
Chen Qi
89dd30049b buildoptions.py: use different STAMPS_DIR and SSTATE_DIR
Use a different STAMPS_DIR and SSTATE_DIR in test_yocto_source_mirror.
Otherwise, when executing `oe-selftest -a', we will get a lot of failures
due to do_unpack failure.

(From OE-Core rev: f8502c68163ec288cf6fb6754829b455501b4a05)

Signed-off-by: Chen Qi <Qi.Chen@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-01-08 11:16:44 +00:00
Robert Yang
c75b8f33bf oeqa/selftest/runqemu: Enable kvm when QEMU_USE_KVM is set
(From OE-Core rev: 564de3681353fe8e203425388e8be9703a89d2da)

Signed-off-by: Robert Yang <liezhi.yang@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-01-08 11:16:44 +00:00
Richard Purdie
b9c9018220 oeqa/selftest/distrodata: Drop now unneeded distrodata inherit
(From OE-Core rev: 1e03261d5e5e43ddda279750493f79aa865810c5)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2018-12-18 16:19:34 +00:00
Richard Purdie
7d7340bf31 oeqa/selftest/distrodata: Port to use the new recipeutils.get_recipe_upgrade_status() function
Rather than use the obsolete do_checkpkg function, use the new recipeutils
function which uses tinfoil to get the data rather than needing csv file
manipulation.

(From OE-Core rev: 3f3f80b00cd999f1b2aef8f5c0ce0900aa4dcbcb)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2018-12-15 11:48:07 +00:00
Paul Eggleton
d2ba6dfab3 oe-selftest: distrodata: change test_maintainers() to use tinfoil
Use tinfoil to enumerate recipes and get the value of RECIPE_MAINTAINER
to make it a bit more reliable in the face of do_checkpkg issues we are
currently seeing on the Yocto Project autobuilder. This also makes it a
little less painful to re-execute test_maintainers() since you don't
have to wait for bitbake -c checkpkg to complete every time.

Note that the new test has been written in such a way that it will still
function if RECIPE_MAINTAINER values are ever moved to the recipes.
Also, the test still currently fails as there are recipes that don't
have an assigned maintainer.

(From OE-Core rev: 47282a2f6f12acebf58961ea9410cfbc335d560b)

Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2018-12-14 17:10:59 +00:00
Ola x Nilsson
e798efcc5b selftest/package: Correct format arguments in test_gdb_hardlink_debug
(From OE-Core rev: 7c2c9b876334cc7357f818c3b3744fc97987e38f)

Signed-off-by: Ola x Nilsson <olani@axis.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2018-12-13 16:32:21 +00:00
Richard Purdie
559fe1a4d4 oeqa/selftest/runcmd: Increase timeout delta
Expecting 1s accuracy on a 2s timeout on a heavily loaded system has proven to be
unreliable. Update this to a 5s timeout with a 3s delta which should be achievable.

(From OE-Core rev: 5feecb639d49d72d8a6abc589b937e07ee72f252)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2018-12-08 17:17:01 +00:00
Chen Qi
d3a94e5b9b oeqa/selftest/context: ensure log directory exists
Ensure log directory exists to avoid the following error.

  FileNotFoundError: [Errno 2] No such file or directory: '/.../build-selftest/tmp/log/oe-selftest-results-20181207043431.log'

(From OE-Core rev: 6c41de48db76087fee596d9440d8f05346ab1094)

Signed-off-by: Chen Qi <Qi.Chen@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2018-12-08 17:17:01 +00:00
Richard Purdie
0985ac745f oeqa/selftest/esdk: Ensure parent directory exists
INFO - ======================================================================
INFO - ERROR: setUpClass (eSDK.oeSDKExtSelfTest)
INFO - ----------------------------------------------------------------------
INFO - Traceback (most recent call last):
  File "/home/pokybuild/yocto-worker/oe-selftest-debian/build/meta/lib/oeqa/core/case.py", line 32, in _oeSetUpClass
    clss.setUpClassMethod()
  File "/home/pokybuild/yocto-worker/oe-selftest-debian/build/meta/lib/oeqa/selftest/cases/eSDK.py", line 76, in setUpClass
    cls.tmpdirobj = tempfile.TemporaryDirectory(prefix="selftest-esdk-", dir=bb_vars["WORKDIR"])
  File "/usr/lib/python3.5/tempfile.py", line 929, in __init__
    self.name = mkdtemp(suffix, prefix, dir)
  File "/usr/lib/python3.5/tempfile.py", line 507, in mkdtemp
    _os.mkdir(file, 0o700)
FileNotFoundError: [Errno 2] No such file or directory: '/home/pokybuild/yocto-worker/oe-selftest-debian/build/build/tmp/work/qemux86_64-poky-linux/core-image-minimal/1.0-r0/selftest-esdk-q7ln84gc'

(From OE-Core rev: 48719bc4d108df7e357e7f22f9f356cd72b3ebbd)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2018-12-08 17:17:01 +00:00
Richard Purdie
6f2ef620d9 oeqa/selftest/esdk: Fix typo causing test failure
2018-12-06 23:19:24,564 - oe-selftest - INFO - Traceback (most recent call last):
  File "/media/build1/poky-sumo/meta/lib/oeqa/core/case.py", line 32, in _oeSetUpClass
    clss.setUpClassMethod()
  File "/media/build1/poky-sumo/meta/lib/oeqa/selftest/cases/eSDK.py", line 78, in setUpClass
    cls.tmpdir_eSDKQA = cls.tempdirobj.name
AttributeError: type object 'oeSDKExtSelfTest' has no attribute 'tempdirobj'

(From OE-Core rev: c6de27c2f6f598849dcb8036ee849f449ba7f327)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2018-12-06 23:27:50 +00:00
Ross Burton
13dd62344b oeqa/selftest/esdk: run selftest inside workdir not /tmp
We've seen issues with rootfs size calculations and we've seen systems
like opensuse which have btrfs mounted on /tmp causing selftest failures.

(From OE-Core rev: 61be3cd748d1b7321a1fc4cfe84efa9b26a6aee0)

Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2018-12-05 12:37:02 +00:00
Richard Purdie
f24df3f8b2 oeqa/selftest/runqemu: Improve testcase failure handling
assertTrue doesn't give good debug information when things fail. Update
several to use assertIn which gives information upon failure, for the
others print the log information upon failure.

(From OE-Core rev: c29cb75d5ce6b0873a934f4709b0c8824f7164d3)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2018-12-05 12:37:02 +00:00
Richard Purdie
d8d5199e66 lib/oe/utils: Improve multiprocess_lauch exception handling
We've seen a cryptic:

"ERROR: Fatal errors occurred in subprocesses, tracebacks printed above"

message from oe-selftest with no other traceback information. Improve the
traceback logging to try and give a better indication of any errors that is
ocurring.

(From OE-Core rev: 521dd3d00979a27b6932e58d5497de68abac26e1)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2018-12-05 12:37:02 +00:00
Ross Burton
367df47d37 oeqa/oelib/path: don't leak temporary directories
setUp() is used to populate a directory of temporary files, and deleted in
__del__.  However setUp() is called once *per test* so __del__ would only be
able to remove the last directory created.

Fix the code by using the natural counterpart to setUp, tearDown(), to clean up.

(From OE-Core rev: 68b4723e6fb11d171869185bccf28f32f6284c18)

Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2018-12-05 12:37:02 +00:00
Paul Eggleton
5cd79e8147 oe-selftest: add some tests for recipeutils module
Add some tests for functions in meta/lib/oe/recipeutils.py, in
particular for a few issues I've just fixed. I haven't added tests for
all of the functions - some of them are already being tested via devtool
in any case.

(From OE-Core rev: 72d0cb3f8f1e69aeef93ea0bc90db3e8b8d6f94c)

Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2018-12-03 12:20:00 +00:00
Richard Purdie
28cbaf1de1 oeqa/selftest/context: Improve log file handling
The existing logfile is simply placed in the current directory. Since the test
changes cwd to BUILDDIR, the symlink to the log can be placed in an invalid
directory. We also see trackbacks if the symlink is invalid.

Improve things by:

* Placing logs in LOG_DIR (or BUILDDIR if unset).
* Using a full path to the log meaning the log and link are placed in the same directory.
* Using lexists instead of exists so invalid symlinks are handled correctly.

(From OE-Core rev: 750ece11bed0e62a11e0003d1d16a81f7c219761)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2018-12-01 11:38:36 +00:00
Richard Purdie
5c7f5e2d3b oeqa/selftest/buildoptions: Ensure diskmon tests run consistently
Heartbeat events default to once a second and we need to ensure we have
enough time in the task to see them.

Add a nostamp delay task 5s long so we can have a consistently timed
task which doesn't need cleanup or have unneeded dependencies. This
ensures we should deterministically see the disk moinitor events
regardless of the state of the build. This is done in a way which
doesn't corrupt build state or need cleanup and is efficient.

(From OE-Core rev: ecc49ee8986929e2429d948000a0ca588fe63959)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2018-12-01 11:38:36 +00:00
Richard Purdie
91f838e036 oeqa/selftest/buildoptions: Improve ccache test
This test occisionally fails as m4 doesn't recompile, meaning the logfile test
then doesn't find mention of ccache.

To ensure m4 does recompile, clean m4 before force compiling it.

(Reading the test is confusing due to the test cleanup also involving a clean)

(From OE-Core rev: 6e0b9214a0d57ed45a5df0ba5c9887a9045b89b1)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2018-12-01 11:38:36 +00:00
Richard Purdie
e865e4e703 oeqa/selftest/buildoptions: Improve ccache test failure output
The current failure mode doesn't show us what the logs actually looked like
and later cleans can lose them. Show the whole log in case of failure
to aid debugging intermittent problems on the autobuilder.

(From OE-Core rev: 7c3a0dc5978cea898b1ca51decf4d6e7cf9d519f)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2018-11-27 22:13:11 +00:00