Commit Graph

50 Commits

Author SHA1 Message Date
Richard Purdie
f08831c5a9 resulttool/log: Add ability to dump ltp logs as well as ptest
Currently only ptest logs are accessible with the log command, this
adds support so the ltp logs can be extracted too.

(From OE-Core rev: 64a2121a875ce128959ee0a62e310d5f91f87b0d)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2020-05-27 10:22:31 +01:00
Richard Purdie
7a6739398a resulttool/report: Remove leftover debugging
I've long since wondered why there was some odd output in result reports,
remove the leftover debug which was causing it.

(From OE-Core rev: 66e96bf70753933714ff8edcc13a1f35a052656f)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2020-05-27 10:22:31 +01:00
Richard Purdie
51e9c2fd73 resulttool/resultutils: Fix unicode error handling
This error handling didn't work as expected since upon failure it would
inject bytestreams back into the code leading to tracebacks.

Instead, ignore the decode errors. Fixes:

Traceback (most recent call last):
  File "/home/pokybuild/yocto-worker/a-full/build/scripts/resulttool", line 78, in <module>
    sys.exit(main())
  File "/home/pokybuild/yocto-worker/a-full/build/scripts/resulttool", line 72, in main
    ret = args.func(args, logger)
  File "/home/pokybuild/yocto-worker/a-full/build/scripts/lib/resulttool/store.py", line 70, in store
    resultutils.save_resultsdata(results, tempdir, ptestlogs=True)
  File "/home/pokybuild/yocto-worker/a-full/build/scripts/lib/resulttool/resultutils.py", line 178, in save_resultsdata
    f.write(sectionlog)
TypeError: write() argument must be str, not bytes

(From OE-Core rev: 3aa0d0fd0bbf8c9c13719251a2fd1e85a22c7336)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2020-02-15 17:26:01 +00:00
Yeoh Ee Peng
0b032d39fa scripts/lib/resulttool/report: Enable report selected test case result
Enable reporting selected test case result given the user provided
the selected test case id. If both test result id and test case id
were provided, report the selected test case result from the
selected test result id.

(From OE-Core rev: 7161310ce32d6e0c397d0132808d556bdc80d183)

Signed-off-by: Yeoh Ee Peng <ee.peng.yeoh@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2020-02-13 12:19:14 +00:00
Yeoh Ee Peng
f2d5800563 resulttool/store.py: Enable add extra test environment data
Enable the option to add extra test environment data to the
configuration of each test result (as optional).

Example of optional test environment data include:
	- custom packages included for runtime test
	- detail machine specification used as target
	- detail host environment used for bitbake

(From OE-Core rev: 3cc887a72c5726024c3699e6a3deb72fad4992a9)

Signed-off-by: Yeoh Ee Peng <ee.peng.yeoh@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-11-14 13:20:59 +00:00
Yeoh Ee Peng
610b02a2fb scripts/resulttool/report: Add total statistic to test result.
Add total passed, failed, and skipped statistic to test result.

(From OE-Core rev: 7ad74f99b6105f23bcf930712627aac279c6af10)

Signed-off-by: Yeoh Ee Peng <ee.peng.yeoh@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-11-14 13:20:59 +00:00
Yeoh Ee Peng
565b05b0e4 scripts/resulttool/report: Enable output raw test results
In case of debugging, report user need to acccess the raw
test result. Instead of going back to source file/directory/URL
to manually pull out the raw result, provide alternative
way to let report showing raw test results by providing
the result id (optional).

(From OE-Core rev: 8b2485bc0d0fa2f72b37e8bafbfa663d456ea98c)

Signed-off-by: Yeoh Ee Peng <ee.peng.yeoh@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-11-14 13:20:59 +00:00
Yeoh Ee Peng
3adefee2a1 scripts/resulttool/report: Enable report to use regression_map
By default, report will use the store_map to generate the key
to reference each result set. In some situation when using store_map
with multiple set of tests sharing similar test configurations,
the report will only showing partial result set for results
that having identical result_id (use of multiconfig to run tests
where it generate identical result_id).

Enable report to have the option to use the regression_map (optional)
instead of the default store_map, where it will take larger
set of configurations to generate the key to reference each
result set, this will prevent the report from only showing
partial result set.

(From OE-Core rev: 8577abce6feb1f529537da554d39e86a5b05ca2d)

Signed-off-by: Yeoh Ee Peng <ee.peng.yeoh@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-11-14 13:20:59 +00:00
Nathan Rossi
ec1104fd7f oeqa/core/case.py: Encode binary data of log
Do not decode the log content into a string only to re-encode it as
binary data again. Some logs might un-intentionally contain bytes that
do not decode as utf-8, as such preserve the log file content as it was
on disk.

Handle the decoding on the resulttool side, but also handle the failure
to decode the data.

(From OE-Core rev: 20531dc0b8f76a6e37cc856f36cd94077b6aba50)

Signed-off-by: Nathan Rossi <nathan@nathanrossi.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-09-30 16:55:21 +01:00
Nathan Rossi
e08f657220 oeqa/core/case.py: Add OEPTestResultTestCase for ptestresult helpers
Add the OEPTestResultTestCase class as a mix-in class to provide helper
functions for interacting with ptestresults within the extraresults
object generated by the test case.

This class also provides default compression of log text and log files.

Also add support to resulttool for decoding/decompressing log files
embedded in the test results.

(From OE-Core rev: 06cba9883a5964320969301fd05eeb6bec3e786d)

Signed-off-by: Nathan Rossi <nathan@nathanrossi.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-09-16 23:02:43 +01:00
Nathan Rossi
05e583ae84 resulttool: Handle multiple series containing ptestresults
Handle multiple results series having ptestresults content. The contents
are merged on a per-result basis where duplicates are ignored (with a
warning message printed). The 'ptestresults.sections' collection is also
merged on a per-suite basis.

(From OE-Core rev: 47edd51970ed0c33edbe04fd72abd1cfc6ecd3d1)

Signed-off-by: Nathan Rossi <nathan@nathanrossi.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-09-16 23:02:43 +01:00
Joshua Watt
bff47225c6 resulttool: Add reproducible log extraction
Adds an argument to the log subcommand to extract the raw logs from the
reproducible selftest.

To prevent ambiguity, the "--raw" argument has been renamed
"--raw-ptest", although the old "--raw" argument is kept around for
compatibility.

[YOCTO #13324]

(From OE-Core rev: 7a4ebb361ff1efc22e7dafadfa60c98bc8a79ed4)

Signed-off-by: Joshua Watt <JPEWhacker@gmail.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-09-06 14:58:10 +01:00
Nathan Rossi
6e48f9aa23 scripts/lib/resulttool/report.py: Add more result types
Add additional result types into the dictionary to handle dejagnu style
test results. These include PASS, FAIL, XPASS, XFAIL, UNSUPPORTED,
UNTESTED, UNRESOLVED and ERROR.

(From OE-Core rev: 684fc36402a23760b203f4761f284043031c799c)

Signed-off-by: Nathan Rossi <nathan@nathanrossi.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-08-28 16:44:09 +01:00
Jon Mason
87138c4c2b resulttool: Prevent multiple results for the same test
Currently, if a test occurs multiple times over different series, the
code will sum these.  This can lead to confusion over the desired
results.  Change the code to report the redundant tests and skip adding
an additional values.

(From OE-Core rev: caeaa0648dff64c56a33f52e45e82bfab6719e3e)

Signed-off-by: Jon Mason <jdmason@kudzu.us>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-08-21 21:52:59 +01:00
Yeoh Ee Peng
99c821eaaa resulttool/merge: Enable control TESTSERIES and extra configurations
Current QA team need to merge test result files from multiple sources.
Adding TESTSERIES configuration too early will have negative
implication to report and regression. Enable control to add TESTSERIES
when needed. Also enable adding EXECUTED_BY configuration when
needed.

(From OE-Core rev: 651d8d371e78e77599d56681228d5782664f7743)

Signed-off-by: Yeoh Ee Peng <ee.peng.yeoh@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-06-12 10:54:40 +01:00
Yeoh Ee Peng
41f10ab3b1 resulttool/store: Enable add EXECUTED_BY config to results
Current results stored does not have information needed to trace who
executed the tests. Enable store to add EXECUTED_BY configuration
to results file in order to track who executed the tests.

(From OE-Core rev: dca2a57d54163a2e63b06e2f140fea3bd49cef0d)

Signed-off-by: Yeoh Ee Peng <ee.peng.yeoh@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-06-12 10:54:40 +01:00
Yeoh Ee Peng
588f455082 resulttool/resultutils: Enable add extra configurations to results
Current resultutils library always add "TESTSERIES" configuration
to results. Enhance this to allow control of adding "TESTSERIES"
configuration as well as allow adding extra configurations
when needed.

(From OE-Core rev: 443c0acc14ef2451b10878fc83dd11b46805daf0)

Signed-off-by: Yeoh Ee Peng <ee.peng.yeoh@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-06-12 10:54:40 +01:00
Jon Mason
b6ca6ac564 resulttool: Remove prints if no tests occur
Printing the lack of a test is not necessary (per feedback).  Remove
this from the template to quieten it.

(From OE-Core rev: b1fe6ae66360e160eeaeafe456536f335a0eab60)

Signed-off-by: Jon Mason <jdmason@kudzu.us>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-06-04 23:09:24 +01:00
sangeeta jain
d9cb1c472c resulttool/manualexecution: Enable creation of test case configuration
Allow the creation of test case configuration file based on user inputs.
Where this testcase configuration file will be used by the the manual
execution to run selected test cases for a module rather than compulsory
run all test cases in manual json file.

(From OE-Core rev: 73d2a747c17779da0ca972da776b3cf02c2e1cbc)

Signed-off-by: sangeeta jain <sangeeta.jain@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-06-04 09:09:42 +01:00
Jon Mason
cb153e8da8 resulttool: modify to be multi-machine
Currently, the code will sum all of the different machine results into a
single report of the tests results.  This can lead to confusion as to
which machine may be experiencing issues.  Modify the code to store the
results in a per machine basis and report them accordingly.

(From OE-Core rev: 16d4031ea5df8a4ddfdb937d35464c09e1abd10e)

Signed-off-by: Jon Mason <jdmason@kudzu.us>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-06-04 09:09:42 +01:00
Richard Purdie
ffae400179 meta/lib+scripts: Convert to SPDX license headers
This adds SPDX license headers in place of the wide assortment of things
currently in our script headers. We default to GPL-2.0-only except for the
oeqa code where it was clearly submitted and marked as MIT on the most part
or some scripts which had the "or later" GPL versioning.

The patch also drops other obsolete bits of file headers where they were
encoountered such as editor modelines, obsolete maintainer information or
the phrase "All rights reserved" which is now obsolete and not required in
copyright headers (in this case its actually confusing for licensing as all
rights were not reserved).

More work is needed for OE-Core but this takes care of the bulk of the scripts
and meta/lib directories.

The top level LICENSE files are tweaked to match the new structure and the
SPDX naming.

(From OE-Core rev: f8c9c511b5f1b7dbd45b77f345cb6c048ae6763e)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-05-09 16:31:55 +01:00
Joshua Watt
9a97ea263c resulttool: Add option to dump all ptest logs
Adds an option to dump all the ptest logs to individual files in a
specified directory. If multiple test runs are present, the
'--prepend-run' argument will create separate directories for each test
run under the target directory and put the logs there to prevent each
test run from clobbering the others.

[YOCTO #13331]

(From OE-Core rev: f262b9e7482e38f04dfa162f0aaf65b17e86f8b2)

Signed-off-by: Joshua Watt <JPEWhacker@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-05-08 22:56:45 +01:00
Armin Kuster
12d8f77a88 resulttool: add LTP compliance section
(From OE-Core rev: a680d7d15fafbecf4edce9a29cc4eda16c11fd94)

Signed-off-by: Armin Kuster <akuster808@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-04-29 14:16:30 +01:00
Armin Kuster
9c82085036 resulttool: add ltp test support
(From OE-Core rev: 966795aa2c6960aca11a04e87f415256faf26957)

Signed-off-by: Armin Kuster <akuster808@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-04-29 14:16:30 +01:00
sangeeta jain
ebeecd60c2 resulttool/manualexecution: Enable test case configuration option
Current manualexecution required user to exceute all test cases defined inside a "modulename.json" file in oeqa/manual

There are cases when all test cases all not required to run for a module on specific DUT.

Enable manualexecution to have the optional feature where it will use pre-defined json format test case configuration file
where user will be able to select test cases from the "modulename.json" instead of running all of them. This will help
in reducing testing time and reporting unneccesary skip or failures.

Example pre-defined json format test case configuration file (for build-applince):

{
    "testcases" : [
        "build-appliance.build-appliance.Create_core-image-sato-sdk_using_build_appliance",
        "build-appliance.build-appliance.Build_a_image_without_error_(added_recipe)"
    ]
}

(From OE-Core rev: 3950c28a34f94b5907d37b579bdaee5a59794652)

Signed-off-by: sangeeta jain <sangeeta.jain@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-04-26 10:09:08 +01:00
Richard Purdie
d0d822d173 resultool/resultutils: Fix module import error
Fix AttributeError: module 'urllib' has no attribute 'request' when
using remote http urls.

(From OE-Core rev: 9acc47236e2a9ac7f97f80040f5e508f23f01611)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-04-26 10:09:07 +01:00
Joshua Watt
e567c3761b resulttool: Add log subcommand
Adds a subcommand for dumping various logs from test results

(From OE-Core rev: 454b8d2cdc6f79a51e610dae92c22352850c3f7c)

Signed-off-by: Joshua Watt <JPEWhacker@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-04-25 00:05:20 +01:00
Joshua Watt
3326a902db resulttool: Load results from URL
Adds support for resulttool to load JSON files directly from a http://
or https:// URL

(From OE-Core rev: 235bcf0c504e7ea253ccfb28d600898117c64c1f)

Signed-off-by: Joshua Watt <JPEWhacker@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-04-25 00:05:20 +01:00
Yeoh Ee Peng
1576ce540b resulttool/manualexecution: Refactor and remove duplicate code
Remove duplicate codes. Replace unnecessary class variables with
local variables. Rename variables and arguments with simple and
standard name.

(From OE-Core rev: 6bef61d36f3328fad003c0dc9c010d3f76ba96d8)

Signed-off-by: Yeoh Ee Peng <ee.peng.yeoh@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-04-12 09:29:06 +01:00
Yeoh Ee Peng
974d786d08 resulttool/manualexecution: Enable creation of configuration option file
Allow the creation of configuration option file based on user inputs.
Where this configuration option file will be used by the the manual
execution to display options for configuration rather than user
need to inputs configuration manually.

(From OE-Core rev: 23c7a037a85a825aa088bfac3a144c14087a9f49)

Signed-off-by: Yeoh Ee Peng <ee.peng.yeoh@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-04-10 13:46:17 +01:00
Yeoh Ee Peng
8d70c77678 resulttool/manualexecution: Enable configuration options selection
Current manualexecution required user to input configuration manually
where there were inconsistent inputs and human typo issues.

Enable manualexecution to have the optional feature where it
will use pre-compiled configuration options file where user will
be able to select configuration from the pre-compiled list instead
of manual key-in the configuration. This will eliminate human error.

Expect the pre-compiled configuration options file in json format below

{
    "bsps-hw": {
        "IMAGE_BASENAME": {
            "1": "core-image-sato-sdk"
        },
        "MACHINE": {
            "1": "beaglebone-yocto",
            "2": "edgerouter",
            "3": "mpc8315e-rdb",
            "4": "genericx86",
            "5": "genericx86-64"
        }
    },
    "bsps-qemu": {
        "IMAGE_BASENAME": {
            "1": "core-image-sato-sdk"
        },
        "MACHINE": {
            "1": "qemuarm",
            "2": "qemuarm64",
            "3": "qemumips",
            "4": "qemumips64",
            "5": "qemuppc",
            "6": "qemux86",
            "7": "qemux86-64"
        }
    }
}

(From OE-Core rev: 1af7e04223b415a64e100a9f91d60f5e9b0f789a)

Signed-off-by: Yeoh Ee Peng <ee.peng.yeoh@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-04-10 13:46:17 +01:00
Yeoh Ee Peng
996d1f4518 resulttool/manualexecution: Refactor and simplify codebase
Simplify and removed unnecessary codes.
Refactor to allow pythonic loop.

(From OE-Core rev: 84c6a992e1114685194f6e8a554bce7753c090cc)

Signed-off-by: Yeoh Ee Peng <ee.peng.yeoh@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-04-04 22:57:32 +01:00
Yeoh Ee Peng
f1e470d8fa resulttool/manualexecution: Fixed step sorted by integer
Currently the manual execution display step by sorting
the step as string, where steps were not being sorted
correctly when there are more than 9 steps.

Fixed the step sorting by sorting step as integer.

(From OE-Core rev: 192c255d9f597b3526a5b94fcf3eedc4527189ed)

Signed-off-by: Yeoh Ee Peng <ee.peng.yeoh@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-04-04 22:57:32 +01:00
Yeoh Ee Peng
6401dfb6fa resulttool/manualexecution: Enable display full steps without press enter
Current manualexecution required pressing enter button to show each step
information, where this was wasting execution time. Enable display
full steps without needing to any press enter button.

(From OE-Core rev: e44c9a018c13208fef0dcef4479ce71977628cd4)

Signed-off-by: Mazliana <mazliana.mohamad@intel.com>
Signed-off-by: Yeoh Ee Peng <ee.peng.yeoh@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-04-04 22:57:32 +01:00
Yeoh Ee Peng
8262db1c27 resulttool/manualexecution: Standardize input check
Current input checking does not match the standard input practiced
by QA team. Change the input checking to match the standard
input practiced by the QA team.

(From OE-Core rev: 630a93c740359a65249a198f314f15040042f8d3)

Signed-off-by: Yeoh Ee Peng <ee.peng.yeoh@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-04-04 22:57:32 +01:00
Richard Purdie
00faa62c4c resulttool: Allow extraction of ptest data
Rather than simply discarding the ptest data, change the code to discard
it when writing out the new testresult files, or optionally either preserve
it, or write it as seperate discrete logs.

This means the autobuilder should start writing out individual ptest log
files as well as allowing ueers to extract these manually.

(From OE-Core rev: a1e0944bf260ef50dd7dfcb10db248fdd7f45bc9)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-04-04 22:57:32 +01:00
Richard Purdie
803d404bdf resulttool: Allow store to work on single files
Store operations using a single file as a source weren't working as the os.walk
command didn't like being given a single file. Fix the store operation to
work for single files.

(From OE-Core rev: eb5bbe613d8c7cbcd8b74d8ac0073c2217970410)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-04-04 22:57:32 +01:00
Mazliana
2c49965485 resulttool/manualexecution: To output right test case id
We found that manualexecution does not capture test suite values
correctly if there are more than one test suite in test cases.
After verification has made we found out we should retrieved
full test cases value <test_module.test_suite.test_case> from
oeqa/manual/ json file rather than split it them into new
variables test_suite and test_cases.

(From OE-Core rev: db08ec5f48badd5be2fa24cc0d89a4386ab44a59)

Signed-off-by: Mazliana <mazliana.mohamad@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-03-18 22:59:58 +00:00
Yeoh Ee Peng
456a03bc74 resulttool/report: Enable roll-up report for a commit
Enable roll-up all test results belong to a commit
and to provide a roll-up report.

(From OE-Core rev: a98a75a3db1fe9d327fd21919fb8ea09e76fb1e3)

Signed-off-by: Yeoh Ee Peng <ee.peng.yeoh@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-03-18 11:12:25 +00:00
Yeoh Ee Peng
975a1416e7 scripts/resulttool: Enable manual result store and regression
To enable store for testresults.json file from manualexecution,
add layers metadata to configuration and add "manual" map to
resultutils.store_map.

To enable regression for manual, add "manual" map to
resultutils.regression_map. Also added compulsory configurations
('MACHINE', 'IMAGE_BASENAME') to manualexecution.

(From OE-Core rev: 7521834f8557565d3898605257c7fd9505ea22bb)

Signed-off-by: Yeoh Ee Peng <ee.peng.yeoh@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-03-07 21:55:29 +00:00
Yeoh Ee Peng
21b3b83a33 resulttool/regression: Ensure regressoin results are sorted
Sorted regression results to provide friendly viewing of report.

(From OE-Core rev: 59d827889dd8b790a7c9323c69565d6bba3041bd)

Signed-off-by: Yeoh Ee Peng <ee.peng.yeoh@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-02-28 17:47:10 +00:00
Richard Purdie
80d29f1cb1 resulttool/store: Fix missing variable causing testresult corruption
(From OE-Core rev: a89846cca457de10249881ab695f0f883bfb6f40)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-02-28 13:17:03 +00:00
Richard Purdie
2268bf470c resulttool/report: Ensure ptest results are sorted
(From OE-Core rev: 3ec9a6288793f384bf108d7194cf50147485fa28)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-02-27 18:04:14 +00:00
Richard Purdie
f11237c0aa resulttool/report: Ensure test suites with no results show up on the report
ptest suites with no results don't show up on the reports even though we have
a duration for them. Fix this so the fact they report no tests is visible.

(From OE-Core rev: 58268151704246a81ec8dae46c26346023057554)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-02-27 18:04:14 +00:00
Richard Purdie
43f793b78d resulttool/report: Handle missing metadata sections more cleanly
Currently some older results files cause the code to give tracebacks.
Handle these missing sections more cleanly.

(From OE-Core rev: f778c191dbd5740173b3be07f4c1655d85a07bb2)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-02-27 18:04:14 +00:00
Richard Purdie
d8f276b2cb resulttool/store: Handle results files for multiple revisions
Currently we cant store results if the results files span multiple
different build revisons. Remove this limitation by iterating.

(From OE-Core rev: 4da12c00963b02508056b87ce9b972528ce3a1be)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-02-27 18:04:14 +00:00
Richard Purdie
49b134f30a resulttool/resultutils: Avoids tracebacks for missing logs
(From OE-Core rev: ec020699086cc56bfa37bab38065472da52b3d0f)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-02-25 10:43:06 +00:00
Richard Purdie
47eb3d00e9 resulttool: Improvements to allow integration to the autobuilder
This is a combined patch of the various tweaks and improvements I
made to resulttool:

* Avoid subprocess.run() as its a python 3.6 feature and we
  have autobuilder workers with 3.5.

* Avoid python keywords as variable names

* Simplify dict accesses using .get()

* Rename resultsutils -> resultutils to match the resultstool ->
  resulttool rename

* Formalised the handling of "file_name" to "TESTSERIES" which the code
  will now add into the json configuration data if its not present, based
  on the directory name.

* When we don't have failed test cases, print something saying so
  instead of an empty table

* Tweak the table headers in the report to be more readable (reference
  "Test Series" instead if file_id and ID instead of results_id)

* Improve/simplify the max string length handling

* Merge the counts and percentage data into one table in the report
  since printing two reports of the same data confuses the user

* Removed the confusing header in the regression report

* Show matches, then regressions, then unmatched runs in the regression
  report, also remove chatting unneeded output

* Try harder to "pair" up matching configurations to reduce noise in
  the regressions report

* Abstracted the "mapping" table concept used to pairing in the
  regression code to general code in resultutils

* Created multiple mappings for results analysis, results storage and
  'flattening' results data in a merge

* Simplify the merge command to take a source and a destination,
  letting the destination be a directory or a file, removing the need for
  an output directory parameter

* Add the 'IMAGE_PKGTYPE' and 'DISTRO' config options to the regression
  mappings

* Have the store command place the testresults files in a layout from
  the mapping, making commits into the git repo for results storage more
  useful for simple comparison purposes

* Set the oe-git-archive tag format appropriately for oeqa results
  storage (and simplify the commit messages closer to their defaults)

* Fix oe-git-archive to use the commit/branch data from the results file

* Cleaned up the command option help to match other changes

* Follow the model of git branch/tag processing used by oe-build-perf-report
  and use that to read the data using git show to avoid branch change

* Add ptest summary to the report command

* Update the tests to match the above changes

(From OE-Core rev: ff2c029b568f70aa9960dde04ddd207829812ea0)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-02-21 12:34:00 +00:00
Mazliana
beed7523b6 scripts/resulttool: enable manual execution and result creation
Integrated “manualexecution” operation to resulttool scripts.
Manual execution script is a helper script to execute all manual
test cases in baseline command, which consists of user guideline
steps and the expected results. The last step will ask user to
provide their input to execute result. The input options are
passed/failed/blocked/skipped status. The result given will be
written in testresults.json including log error from the user
input and configuration if there is any.The output test result
for json file is created by using OEQA library.

The configuration part is manually key-in by the user. The system
allow user to specify how many configuration they want to add and
they need to define the required configuration name and value pair.
In QA perspective, "configuration" means the test environments and
parameters used during QA setup before testing can be carry out.
Example of configurations: image used for boot up, host machine
distro used, poky configurations, etc.

The purpose of adding the configuration is to standardize the
output test result format between automation and manual execution.

To use these scripts, first source oe environment, then run the
entry point script to look for help.
        $ resulttool

To execute manual test cases, execute the below
        $ resulttool manualexecution <manualjsonfile>

By default testresults.json store in <build_dir>/tmp/log/manual/

[YOCTO #12651]

(From OE-Core rev: f24dc9e87085a8fe5410feee10c7a3591fe9d816)

Signed-off-by: Mazliana <mazliana.mohamad@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-02-21 12:34:00 +00:00
Yeoh Ee Peng
1fd5ebdb06 resulttool: enable merge, store, report and regression analysis
OEQA outputs test results into json files and these files were
archived by Autobuilder during QA releases. Example: each oe-selftest
run by Autobuilder for different host distro generate a
testresults.json file.

These scripts were developed as a test result tools to manage
these testresults.json file.

Using the "store" operation, user can store multiple testresults.json
files as well as the pre-configured directories used to hold those files.

Using the "merge" operation, user can merge multiple testresults.json
files to a target file.

Using the "report" operation, user can view the test result summary
for all available testresults.json files inside a ordinary directory
or a git repository.

Using the "regression-file" operation, user can perform regression
analysis on testresults.json files specified. Using the "regression-dir"
and "regression-git" operations, user can perform regression analysis
on directory and git accordingly.

These resulttool operations expect the testresults.json file to use
the json format below.
{
    "<testresult_1>": {
        "configuration": {
            "<config_name_1>": "<config_value_1>",
            "<config_name_2>": "<config_value_2>",
            ...
            "<config_name_n>": "<config_value_n>",
        },
        "result": {
            "<testcase_namespace_1>": {
                "status": "<PASSED or FAILED or ERROR or SKIPPED>",
                "log": "<failure or error logging>"
            },
            "<testcase_namespace_2>": {
                "status": "<PASSED or FAILED or ERROR or SKIPPED>",
                "log": "<failure or error logging>"
            },
            ...
            "<testcase_namespace_n>": {
                "status": "<PASSED or FAILED or ERROR or SKIPPED>",
                "log": "<failure or error logging>"
            },
        }
    },
    ...
    "<testresult_n>": {
        "configuration": {
            "<config_name_1>": "<config_value_1>",
            "<config_name_2>": "<config_value_2>",
            ...
            "<config_name_n>": "<config_value_n>",
        },
        "result": {
            "<testcase_namespace_1>": {
                "status": "<PASSED or FAILED or ERROR or SKIPPED>",
                "log": "<failure or error logging>"
            },
            "<testcase_namespace_2>": {
                "status": "<PASSED or FAILED or ERROR or SKIPPED>",
                "log": "<failure or error logging>"
            },
            ...
            "<testcase_namespace_n>": {
                "status": "<PASSED or FAILED or ERROR or SKIPPED>",
                "log": "<failure or error logging>"
            },
        }
    },
}

To use these scripts, first source oe environment, then run the
entry point script to look for help.
    $ resulttool

To store test result from oeqa automated tests, execute the below
    $ resulttool store <source_dir> <git_branch>

To merge multiple testresults.json files, execute the below
    $ resulttool merge <base_result_file> <target_result_file>

To report test report, execute the below
    $ resulttool report <source_dir>

To perform regression file analysis, execute the below
    $ resulttool regression-file <base_result_file> <target_result_file>

To perform regression dir analysis, execute the below
    $ resulttool regression-dir <base_result_dir> <target_result_dir>

To perform regression git analysis, execute the below
    $ resulttool regression-git <source_dir> <base_branch> <target_branch>

[YOCTO# 13012]
[YOCTO# 12654]

(From OE-Core rev: 78a322d7be402a5b9b5abf26ad35670a8535408a)

Signed-off-by: Yeoh Ee Peng <ee.peng.yeoh@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-02-21 12:34:00 +00:00