Commit Graph

68 Commits

Author SHA1 Message Date
Alexis Lothoré
198110b1b9 scripts/resulttool: make additional info more compact
Since "matched" and "improved" tests are not as important as regressions,
reduce the place they take in the regression report:

- merge "matched" and "improved" tests, while removing the label
- add a single line of additional info per pair

Those changes make the "Matches and improvements" look like the following
sample:

oeselftest_almalinux-9.2_qemux86-64_20230910083156
oeselftest_almalinux-8.8_qemux86-64_20231018010951
    -> +7 test(s) present

oeselftest_almalinux-9.2_qemux86-64_20230911010538
oeselftest_debian-11_qemux86-64_20231017150459

oeselftest_debian-11_qemux86-64_20230910012927
oeselftest_debian-11_qemux86-64_20231017151319
    -> +7 test(s) present
[...]

(From OE-Core rev: 6de4426d9a7da67deed7d3a3918892fb56238ff3)

Signed-off-by: Alexis Lothoré <alexis.lothore@bootlin.com>
Signed-off-by: Alexandre Belloni <alexandre.belloni@bootlin.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2023-10-23 10:49:19 +01:00
Alexis Lothoré
faef63eab6 scripts/resulttool: rearrange regressions report order
Regressions reports currently reports matching pairs and improved pairs
first, then regressions.

Change order to print regressions first, which is the most valuable info in
the report, and then print improvements and matches at the bottom.

(From OE-Core rev: 599267467430e70fa4dc8ba6b2a8b126bf6da359)

Signed-off-by: Alexis Lothoré <alexis.lothore@bootlin.com>
Signed-off-by: Alexandre Belloni <alexandre.belloni@bootlin.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2023-10-23 10:49:19 +01:00
Alexis Lothoré
c304fcbe05 scripts/resulttool: limit the number of changes displayed per test
Most of the changes list generated in regression reports fall in one
of the two following categories:
- there is only a few (<10) changes listed and the info is
  valuable/relevant
- the list is huge (> 100 ? 1000 ?) and basically tells us that the whole
  tests category suffers the same status (test missing, test failing, test
  skipped, etc)

Prevent those huge, worthless lists by limiting the output for each test
group:
- current default limit is arbitrarily set to 50
- limit can still be overriden with a new "-l"/"--limit" flag, either with
  custom value, or with 0 to print the whole lists of changes
- limit is applied per test family: currently it distinguishes only types
  of ptests, but it can be adapted to other kind of tests

(From OE-Core rev: cec118406f3ad81cb4709f6e6ae1cef65799658e)

Signed-off-by: Alexis Lothoré <alexis.lothore@bootlin.com>
Signed-off-by: Alexandre Belloni <alexandre.belloni@bootlin.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2023-10-23 10:49:19 +01:00
Richard Purdie
8a9cbf62d3 resulttool/report: Avoid divide by zero
Avoid a divide by zero traceback if unfortunate test counts are encountered.

(From OE-Core rev: c5aeea53dfacb53dedb8445cb3523dc3a8cb6dca)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2023-08-15 08:15:25 +01:00
Alexis Lothoré
572d820da9 scripts/resulttool: define custom string for "not found" test results
Whenever a test result is present in base test result but absent from
target test results, we have an entry in regression report looking like the
following one:

    ptestresult.apr.testfile: PASSED -> None

The "None" status may be a bit confusing, so replace it with a custom
string which really states what "None" means in this context:

    ptestresult.apr.testfile: PASSED -> No matching test result

(From OE-Core rev: b5f6da6e13f3484f51dfa07f6b3500aea7a21825)

Signed-off-by: Alexis Lothoré <alexis.lothore@bootlin.com>
Signed-off-by: Alexandre Belloni <alexandre.belloni@bootlin.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2023-08-04 11:44:28 +01:00
Alexis Lothoré
6de3817c62 scripts/resulttool: allow to replace test raw status with custom string
Add a STATUS_STRINGS dictionnary matching raw statuses to custom strings.
Whenever a regression must be reported, raw status is searched in the
custom statuses dict (key search is case insensitive). If no custom string
is found, raw status is kept and used in regression report

(From OE-Core rev: 9d22bfc9d0c4092dba1af0ee11a4c51b7b270786)

Signed-off-by: Alexis Lothoré <alexis.lothore@bootlin.com>
Signed-off-by: Alexandre Belloni <alexandre.belloni@bootlin.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2023-08-04 11:44:28 +01:00
Alexis Lothoré
ae1d378693 scripts/resulttool: add mention about new detected tests
Some regression reports show a lot of "PASSED->None" transitions. When such
big lot of identical transitions are observed, it could be that tests are
now failing, but it could also be that some tests has been renamed.

To detect such case, add a log in regression report to report the number of
new tests (i.e: tests that are present in target results but not in base
result). This new log also allows to know about newly added tests bases

(From OE-Core rev: 01b5cefd07e01c7407bc663842b8a8d502358a6d)

Signed-off-by: Alexis Lothoré <alexis.lothore@bootlin.com>
Signed-off-by: Alexandre Belloni <alexandre.belloni@bootlin.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2023-07-25 15:27:33 +01:00
Michael Halstead
312a2a68fd resulttool/resultutils: allow index generation despite corrupt json
non-release indexes will continue to generate when test output is
corrupted.

(From OE-Core rev: 1a9157684a6bff8406c9bb470cb2e16ee006bbe9)

Signed-off-by: Michael Halstead <mhalstead@linuxfoundation.org>
Signed-off-by: Alexandre Belloni <alexandre.belloni@bootlin.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2023-07-19 19:37:38 +01:00
Richard Purdie
bedf823032 resulttool: Improve overlapping ptest result reporting
With the move to a ptest per image, the multiple ptest log scenario
is much more likely. Tweak the handling to only warn if there are
overlapping files.

(From OE-Core rev: 8391f1668fcbe932ba846299b0ec22de09e06dd4)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2023-03-16 22:52:15 +00:00
Ross Burton
9db9f8c575 resulttool: add log --list-ptest
Add a convenience argument to the log subcommand to list all of the
ptest logs in a testresults file.

(From OE-Core rev: cd2d7adf02005d46f19c93a40db1e10ce01ac261)

Signed-off-by: Ross Burton <ross.burton@arm.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2023-03-14 17:13:11 +00:00
Ross Burton
37ab2ecbb7 lib/resulttool: fix typo breaking resulttool log --ptest
ptestresult_get_log() looked for a key called 'ptestresuls.sections',
which should be 'ptestresult.sections'

(From OE-Core rev: 7c8c9f7283e54bf8b1521fbaad7dceb66a8fcdbb)

Signed-off-by: Ross Burton <ross.burton@arm.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2023-03-14 17:13:11 +00:00
Alexis Lothoré
1eae53e277 scripts/resulttool: do not count newly passing tests as regressions
resulttool regression module simply compare a base test status to a target test
result status. This approach raises many false positives since all XXX -> PASS
transitions (XXX being any status different from PASS) are flagged as
regression.

- Do not list XXX -> PASS transitions in regression  report, instead count them
  and print a summary of "newly passing tests"
- If an inspected pair has only "newly passing tests", do not print detailed
  list and print it as "Improvement" instead of "Regression"

Updated output example looks like the following:
[...]
Improvement: oeselftest_fedora-37_qemux86-64_20230127010225
             oeselftest_ubuntu-22.04_qemux86-64_20230226120516
             (+1 test(s) passing)
[...]
Match:       oeselftest_almalinux-8.7_qemuarm64_20230127015830
             oeselftest_almalinux-8.7_qemuarm64_20230227015258
[...]
Regression:  oeselftest_almalinux-9.1_qemumips_20230127000217
             oeselftest_opensuseleap-15.4_qemumips_20230226130046
    ptestresult.glibc-user.debug/tst-read-chk-cancel: PASS -> None
    ptestresult.glibc-user.nptl/tst-mutexpi4: PASS -> FAIL
    ptestresult.glibc-user.nptl/tst-mutexpi5a: PASS -> FAIL
    Additionally, 44 previously failing test(s) is/are now passing

(From OE-Core rev: c335f96f687c73fde443ac330ca3e17113794d9e)

Signed-off-by: Alexis Lothoré <alexis.lothore@bootlin.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2023-03-12 23:39:13 +00:00
Alexis Lothoré
1feffaf41a scripts/resulttool: fix ptests results containing a non reproducible path
Some ptests results may be wrongly sampled, resulting in some part of the error
being propagated in test name. For example:

"ptestresult.binutils-ld.in testcase /home/pokybuild/yocto-worker/qemumips/build/build-st-1666126/tmp/work/mips32r2-poky-linux/binutils-cross-testsuite/2.40-r0/git/ld/testsuite/ld-ctf/ctf.exp"
"ptestresult.gcc.Couldn't create remote directory /tmp/runtest.455781 on ssh"
"ptestresult.gcc-libstdc++-v3.Couldn't create remote directory /tmp/runtest.3814266 on target"

While the root errors must be fixed in corresponding ptests packages for those
tests, the test results history must still be usable even with this issue

Add new filters to detect if temporary test directories (build-st-*,
/tmp/runtime.*) are present in name. If so, truncate test name. As a side
effect, it will aggregate multiple failing errors into one, but the regression
will still be raised

(From OE-Core rev: 601eecfddd26bfe2954835a73ed1116bb520235f)

Signed-off-by: Alexis Lothoré <alexis.lothore@bootlin.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2023-03-12 23:39:13 +00:00
Alexis Lothoré
e8048a5b66 scripts/resulttool: call fixup_ptest_names in regression_common
ptests names not only need to be fixed for regression based on git testresults
but also for testsresults provided "manually"

Move ptests naming fixup in regression_common to share the fixup between both
regression use cases

(From OE-Core rev: f772ccd108dc3d618db9d479d672c0f3edd203ca)

Signed-off-by: Alexis Lothoré <alexis.lothore@bootlin.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2023-03-12 23:39:12 +00:00
Richard Purdie
c45d58f003 resulttool/regression: Improve matching of poor ptest test names
Some test case naming is poor and contains random strings, particularly
lttng/babeltrace but also curl. Truncating the test names works since they
contain file and line number identifiers which allows us to match them
without the random components, or in the case or curl, test IDs.

Going forward we may be able to improve the test names but this
tweak allows historical test results to work in reports.

(From OE-Core rev: 541a2e2683531355e678fd93524a0c4a8c43a8ff)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2023-02-26 11:59:52 +00:00
Richard Purdie
401d022a26 resulttool/regression: Ensure LTP results are only compared against other LTP runs
If a test result contains LTP test results, it should only be compared with
other runs containing LTP test results.

(From OE-Core rev: 4dbbf2f4a85620a08dc2fa65095dc17fe6c530f8)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2023-02-26 11:59:52 +00:00
Alexis Lothoré
383cd86595 scripts/resulttool/regression: add metadata filtering for oeselftest
When generating regression reports, many false positive can be observed since
some tests results are compared while the corresponding tests sets are not the
same, as it can be seen for example for oeselftest tests (oeselftest is run
multiple time but with different parameters, resulting in different tests sets)

Add a filtering mechanism in resulttool regression module to enable a better
matching between tests. The METADATA_MATCH_TABLE defines that when the TEST_TYPE
is "oeselftest", then resulttool should filter pairs based on
OESELFTEST_METADATA appended to test configuration. If metadata is absent from
test results, in order to keep compatibility with older results, add a
"guessing" mechanism to generate the missing OESELFTEST_METADATA. The guessed
data is tightly coupled to the autobuilder configuration, where all oe-selftest
executions are described

(From OE-Core rev: 94ab7c2b892bf292dd86619ca9c63ddd7bf53f3c)

Signed-off-by: Alexis Lothoré <alexis.lothore@bootlin.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2023-02-26 11:59:52 +00:00
Alexis Lothoré
88345ecf47 scripts/resulttool/regression: remove unused import
(From OE-Core rev: 4d6a74fcee1479562bf66360f20d528107c70887)

Signed-off-by: Alexis Lothoré <alexis.lothore@bootlin.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2023-02-26 11:59:52 +00:00
Richard Purdie
f08831c5a9 resulttool/log: Add ability to dump ltp logs as well as ptest
Currently only ptest logs are accessible with the log command, this
adds support so the ltp logs can be extracted too.

(From OE-Core rev: 64a2121a875ce128959ee0a62e310d5f91f87b0d)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2020-05-27 10:22:31 +01:00
Richard Purdie
7a6739398a resulttool/report: Remove leftover debugging
I've long since wondered why there was some odd output in result reports,
remove the leftover debug which was causing it.

(From OE-Core rev: 66e96bf70753933714ff8edcc13a1f35a052656f)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2020-05-27 10:22:31 +01:00
Richard Purdie
51e9c2fd73 resulttool/resultutils: Fix unicode error handling
This error handling didn't work as expected since upon failure it would
inject bytestreams back into the code leading to tracebacks.

Instead, ignore the decode errors. Fixes:

Traceback (most recent call last):
  File "/home/pokybuild/yocto-worker/a-full/build/scripts/resulttool", line 78, in <module>
    sys.exit(main())
  File "/home/pokybuild/yocto-worker/a-full/build/scripts/resulttool", line 72, in main
    ret = args.func(args, logger)
  File "/home/pokybuild/yocto-worker/a-full/build/scripts/lib/resulttool/store.py", line 70, in store
    resultutils.save_resultsdata(results, tempdir, ptestlogs=True)
  File "/home/pokybuild/yocto-worker/a-full/build/scripts/lib/resulttool/resultutils.py", line 178, in save_resultsdata
    f.write(sectionlog)
TypeError: write() argument must be str, not bytes

(From OE-Core rev: 3aa0d0fd0bbf8c9c13719251a2fd1e85a22c7336)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2020-02-15 17:26:01 +00:00
Yeoh Ee Peng
0b032d39fa scripts/lib/resulttool/report: Enable report selected test case result
Enable reporting selected test case result given the user provided
the selected test case id. If both test result id and test case id
were provided, report the selected test case result from the
selected test result id.

(From OE-Core rev: 7161310ce32d6e0c397d0132808d556bdc80d183)

Signed-off-by: Yeoh Ee Peng <ee.peng.yeoh@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2020-02-13 12:19:14 +00:00
Yeoh Ee Peng
f2d5800563 resulttool/store.py: Enable add extra test environment data
Enable the option to add extra test environment data to the
configuration of each test result (as optional).

Example of optional test environment data include:
	- custom packages included for runtime test
	- detail machine specification used as target
	- detail host environment used for bitbake

(From OE-Core rev: 3cc887a72c5726024c3699e6a3deb72fad4992a9)

Signed-off-by: Yeoh Ee Peng <ee.peng.yeoh@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-11-14 13:20:59 +00:00
Yeoh Ee Peng
610b02a2fb scripts/resulttool/report: Add total statistic to test result.
Add total passed, failed, and skipped statistic to test result.

(From OE-Core rev: 7ad74f99b6105f23bcf930712627aac279c6af10)

Signed-off-by: Yeoh Ee Peng <ee.peng.yeoh@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-11-14 13:20:59 +00:00
Yeoh Ee Peng
565b05b0e4 scripts/resulttool/report: Enable output raw test results
In case of debugging, report user need to acccess the raw
test result. Instead of going back to source file/directory/URL
to manually pull out the raw result, provide alternative
way to let report showing raw test results by providing
the result id (optional).

(From OE-Core rev: 8b2485bc0d0fa2f72b37e8bafbfa663d456ea98c)

Signed-off-by: Yeoh Ee Peng <ee.peng.yeoh@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-11-14 13:20:59 +00:00
Yeoh Ee Peng
3adefee2a1 scripts/resulttool/report: Enable report to use regression_map
By default, report will use the store_map to generate the key
to reference each result set. In some situation when using store_map
with multiple set of tests sharing similar test configurations,
the report will only showing partial result set for results
that having identical result_id (use of multiconfig to run tests
where it generate identical result_id).

Enable report to have the option to use the regression_map (optional)
instead of the default store_map, where it will take larger
set of configurations to generate the key to reference each
result set, this will prevent the report from only showing
partial result set.

(From OE-Core rev: 8577abce6feb1f529537da554d39e86a5b05ca2d)

Signed-off-by: Yeoh Ee Peng <ee.peng.yeoh@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-11-14 13:20:59 +00:00
Nathan Rossi
ec1104fd7f oeqa/core/case.py: Encode binary data of log
Do not decode the log content into a string only to re-encode it as
binary data again. Some logs might un-intentionally contain bytes that
do not decode as utf-8, as such preserve the log file content as it was
on disk.

Handle the decoding on the resulttool side, but also handle the failure
to decode the data.

(From OE-Core rev: 20531dc0b8f76a6e37cc856f36cd94077b6aba50)

Signed-off-by: Nathan Rossi <nathan@nathanrossi.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-09-30 16:55:21 +01:00
Nathan Rossi
e08f657220 oeqa/core/case.py: Add OEPTestResultTestCase for ptestresult helpers
Add the OEPTestResultTestCase class as a mix-in class to provide helper
functions for interacting with ptestresults within the extraresults
object generated by the test case.

This class also provides default compression of log text and log files.

Also add support to resulttool for decoding/decompressing log files
embedded in the test results.

(From OE-Core rev: 06cba9883a5964320969301fd05eeb6bec3e786d)

Signed-off-by: Nathan Rossi <nathan@nathanrossi.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-09-16 23:02:43 +01:00
Nathan Rossi
05e583ae84 resulttool: Handle multiple series containing ptestresults
Handle multiple results series having ptestresults content. The contents
are merged on a per-result basis where duplicates are ignored (with a
warning message printed). The 'ptestresults.sections' collection is also
merged on a per-suite basis.

(From OE-Core rev: 47edd51970ed0c33edbe04fd72abd1cfc6ecd3d1)

Signed-off-by: Nathan Rossi <nathan@nathanrossi.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-09-16 23:02:43 +01:00
Joshua Watt
bff47225c6 resulttool: Add reproducible log extraction
Adds an argument to the log subcommand to extract the raw logs from the
reproducible selftest.

To prevent ambiguity, the "--raw" argument has been renamed
"--raw-ptest", although the old "--raw" argument is kept around for
compatibility.

[YOCTO #13324]

(From OE-Core rev: 7a4ebb361ff1efc22e7dafadfa60c98bc8a79ed4)

Signed-off-by: Joshua Watt <JPEWhacker@gmail.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-09-06 14:58:10 +01:00
Nathan Rossi
6e48f9aa23 scripts/lib/resulttool/report.py: Add more result types
Add additional result types into the dictionary to handle dejagnu style
test results. These include PASS, FAIL, XPASS, XFAIL, UNSUPPORTED,
UNTESTED, UNRESOLVED and ERROR.

(From OE-Core rev: 684fc36402a23760b203f4761f284043031c799c)

Signed-off-by: Nathan Rossi <nathan@nathanrossi.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-08-28 16:44:09 +01:00
Jon Mason
87138c4c2b resulttool: Prevent multiple results for the same test
Currently, if a test occurs multiple times over different series, the
code will sum these.  This can lead to confusion over the desired
results.  Change the code to report the redundant tests and skip adding
an additional values.

(From OE-Core rev: caeaa0648dff64c56a33f52e45e82bfab6719e3e)

Signed-off-by: Jon Mason <jdmason@kudzu.us>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-08-21 21:52:59 +01:00
Yeoh Ee Peng
99c821eaaa resulttool/merge: Enable control TESTSERIES and extra configurations
Current QA team need to merge test result files from multiple sources.
Adding TESTSERIES configuration too early will have negative
implication to report and regression. Enable control to add TESTSERIES
when needed. Also enable adding EXECUTED_BY configuration when
needed.

(From OE-Core rev: 651d8d371e78e77599d56681228d5782664f7743)

Signed-off-by: Yeoh Ee Peng <ee.peng.yeoh@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-06-12 10:54:40 +01:00
Yeoh Ee Peng
41f10ab3b1 resulttool/store: Enable add EXECUTED_BY config to results
Current results stored does not have information needed to trace who
executed the tests. Enable store to add EXECUTED_BY configuration
to results file in order to track who executed the tests.

(From OE-Core rev: dca2a57d54163a2e63b06e2f140fea3bd49cef0d)

Signed-off-by: Yeoh Ee Peng <ee.peng.yeoh@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-06-12 10:54:40 +01:00
Yeoh Ee Peng
588f455082 resulttool/resultutils: Enable add extra configurations to results
Current resultutils library always add "TESTSERIES" configuration
to results. Enhance this to allow control of adding "TESTSERIES"
configuration as well as allow adding extra configurations
when needed.

(From OE-Core rev: 443c0acc14ef2451b10878fc83dd11b46805daf0)

Signed-off-by: Yeoh Ee Peng <ee.peng.yeoh@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-06-12 10:54:40 +01:00
Jon Mason
b6ca6ac564 resulttool: Remove prints if no tests occur
Printing the lack of a test is not necessary (per feedback).  Remove
this from the template to quieten it.

(From OE-Core rev: b1fe6ae66360e160eeaeafe456536f335a0eab60)

Signed-off-by: Jon Mason <jdmason@kudzu.us>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-06-04 23:09:24 +01:00
sangeeta jain
d9cb1c472c resulttool/manualexecution: Enable creation of test case configuration
Allow the creation of test case configuration file based on user inputs.
Where this testcase configuration file will be used by the the manual
execution to run selected test cases for a module rather than compulsory
run all test cases in manual json file.

(From OE-Core rev: 73d2a747c17779da0ca972da776b3cf02c2e1cbc)

Signed-off-by: sangeeta jain <sangeeta.jain@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-06-04 09:09:42 +01:00
Jon Mason
cb153e8da8 resulttool: modify to be multi-machine
Currently, the code will sum all of the different machine results into a
single report of the tests results.  This can lead to confusion as to
which machine may be experiencing issues.  Modify the code to store the
results in a per machine basis and report them accordingly.

(From OE-Core rev: 16d4031ea5df8a4ddfdb937d35464c09e1abd10e)

Signed-off-by: Jon Mason <jdmason@kudzu.us>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-06-04 09:09:42 +01:00
Richard Purdie
ffae400179 meta/lib+scripts: Convert to SPDX license headers
This adds SPDX license headers in place of the wide assortment of things
currently in our script headers. We default to GPL-2.0-only except for the
oeqa code where it was clearly submitted and marked as MIT on the most part
or some scripts which had the "or later" GPL versioning.

The patch also drops other obsolete bits of file headers where they were
encoountered such as editor modelines, obsolete maintainer information or
the phrase "All rights reserved" which is now obsolete and not required in
copyright headers (in this case its actually confusing for licensing as all
rights were not reserved).

More work is needed for OE-Core but this takes care of the bulk of the scripts
and meta/lib directories.

The top level LICENSE files are tweaked to match the new structure and the
SPDX naming.

(From OE-Core rev: f8c9c511b5f1b7dbd45b77f345cb6c048ae6763e)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-05-09 16:31:55 +01:00
Joshua Watt
9a97ea263c resulttool: Add option to dump all ptest logs
Adds an option to dump all the ptest logs to individual files in a
specified directory. If multiple test runs are present, the
'--prepend-run' argument will create separate directories for each test
run under the target directory and put the logs there to prevent each
test run from clobbering the others.

[YOCTO #13331]

(From OE-Core rev: f262b9e7482e38f04dfa162f0aaf65b17e86f8b2)

Signed-off-by: Joshua Watt <JPEWhacker@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-05-08 22:56:45 +01:00
Armin Kuster
12d8f77a88 resulttool: add LTP compliance section
(From OE-Core rev: a680d7d15fafbecf4edce9a29cc4eda16c11fd94)

Signed-off-by: Armin Kuster <akuster808@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-04-29 14:16:30 +01:00
Armin Kuster
9c82085036 resulttool: add ltp test support
(From OE-Core rev: 966795aa2c6960aca11a04e87f415256faf26957)

Signed-off-by: Armin Kuster <akuster808@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-04-29 14:16:30 +01:00
sangeeta jain
ebeecd60c2 resulttool/manualexecution: Enable test case configuration option
Current manualexecution required user to exceute all test cases defined inside a "modulename.json" file in oeqa/manual

There are cases when all test cases all not required to run for a module on specific DUT.

Enable manualexecution to have the optional feature where it will use pre-defined json format test case configuration file
where user will be able to select test cases from the "modulename.json" instead of running all of them. This will help
in reducing testing time and reporting unneccesary skip or failures.

Example pre-defined json format test case configuration file (for build-applince):

{
    "testcases" : [
        "build-appliance.build-appliance.Create_core-image-sato-sdk_using_build_appliance",
        "build-appliance.build-appliance.Build_a_image_without_error_(added_recipe)"
    ]
}

(From OE-Core rev: 3950c28a34f94b5907d37b579bdaee5a59794652)

Signed-off-by: sangeeta jain <sangeeta.jain@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-04-26 10:09:08 +01:00
Richard Purdie
d0d822d173 resultool/resultutils: Fix module import error
Fix AttributeError: module 'urllib' has no attribute 'request' when
using remote http urls.

(From OE-Core rev: 9acc47236e2a9ac7f97f80040f5e508f23f01611)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-04-26 10:09:07 +01:00
Joshua Watt
e567c3761b resulttool: Add log subcommand
Adds a subcommand for dumping various logs from test results

(From OE-Core rev: 454b8d2cdc6f79a51e610dae92c22352850c3f7c)

Signed-off-by: Joshua Watt <JPEWhacker@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-04-25 00:05:20 +01:00
Joshua Watt
3326a902db resulttool: Load results from URL
Adds support for resulttool to load JSON files directly from a http://
or https:// URL

(From OE-Core rev: 235bcf0c504e7ea253ccfb28d600898117c64c1f)

Signed-off-by: Joshua Watt <JPEWhacker@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-04-25 00:05:20 +01:00
Yeoh Ee Peng
1576ce540b resulttool/manualexecution: Refactor and remove duplicate code
Remove duplicate codes. Replace unnecessary class variables with
local variables. Rename variables and arguments with simple and
standard name.

(From OE-Core rev: 6bef61d36f3328fad003c0dc9c010d3f76ba96d8)

Signed-off-by: Yeoh Ee Peng <ee.peng.yeoh@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-04-12 09:29:06 +01:00
Yeoh Ee Peng
974d786d08 resulttool/manualexecution: Enable creation of configuration option file
Allow the creation of configuration option file based on user inputs.
Where this configuration option file will be used by the the manual
execution to display options for configuration rather than user
need to inputs configuration manually.

(From OE-Core rev: 23c7a037a85a825aa088bfac3a144c14087a9f49)

Signed-off-by: Yeoh Ee Peng <ee.peng.yeoh@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-04-10 13:46:17 +01:00
Yeoh Ee Peng
8d70c77678 resulttool/manualexecution: Enable configuration options selection
Current manualexecution required user to input configuration manually
where there were inconsistent inputs and human typo issues.

Enable manualexecution to have the optional feature where it
will use pre-compiled configuration options file where user will
be able to select configuration from the pre-compiled list instead
of manual key-in the configuration. This will eliminate human error.

Expect the pre-compiled configuration options file in json format below

{
    "bsps-hw": {
        "IMAGE_BASENAME": {
            "1": "core-image-sato-sdk"
        },
        "MACHINE": {
            "1": "beaglebone-yocto",
            "2": "edgerouter",
            "3": "mpc8315e-rdb",
            "4": "genericx86",
            "5": "genericx86-64"
        }
    },
    "bsps-qemu": {
        "IMAGE_BASENAME": {
            "1": "core-image-sato-sdk"
        },
        "MACHINE": {
            "1": "qemuarm",
            "2": "qemuarm64",
            "3": "qemumips",
            "4": "qemumips64",
            "5": "qemuppc",
            "6": "qemux86",
            "7": "qemux86-64"
        }
    }
}

(From OE-Core rev: 1af7e04223b415a64e100a9f91d60f5e9b0f789a)

Signed-off-by: Yeoh Ee Peng <ee.peng.yeoh@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-04-10 13:46:17 +01:00
Yeoh Ee Peng
996d1f4518 resulttool/manualexecution: Refactor and simplify codebase
Simplify and removed unnecessary codes.
Refactor to allow pythonic loop.

(From OE-Core rev: 84c6a992e1114685194f6e8a554bce7753c090cc)

Signed-off-by: Yeoh Ee Peng <ee.peng.yeoh@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-04-04 22:57:32 +01:00