Avoid a divide by zero traceback if unfortunate test counts are encountered.
(From OE-Core rev: b95c6a5278d44fddfbaea45cc78324f1e099187c)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
(cherry picked from commit c5aeea53dfacb53dedb8445cb3523dc3a8cb6dca)
Signed-off-by: Steve Sakoman <steve@sakoman.com>
non-release indexes will continue to generate when test output is
corrupted.
(From OE-Core rev: 9467528e89d44a016a4c1e509a3a7da56ea20f74)
Signed-off-by: Michael Halstead <mhalstead@linuxfoundation.org>
Signed-off-by: Alexandre Belloni <alexandre.belloni@bootlin.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
(cherry picked from commit 1a9157684a6bff8406c9bb470cb2e16ee006bbe9)
Signed-off-by: Steve Sakoman <steve@sakoman.com>
ptestresult_get_log() looked for a key called 'ptestresuls.sections',
which should be 'ptestresult.sections'
(From OE-Core rev: 213ed03f5a15be639a97dca6f5bbf810f7ac2eb5)
Signed-off-by: Ross Burton <ross.burton@arm.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
(cherry picked from commit 7c8c9f7283e54bf8b1521fbaad7dceb66a8fcdbb)
Signed-off-by: Steve Sakoman <steve@sakoman.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Currently only ptest logs are accessible with the log command, this
adds support so the ltp logs can be extracted too.
(From OE-Core rev: 64a2121a875ce128959ee0a62e310d5f91f87b0d)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
I've long since wondered why there was some odd output in result reports,
remove the leftover debug which was causing it.
(From OE-Core rev: 66e96bf70753933714ff8edcc13a1f35a052656f)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
This error handling didn't work as expected since upon failure it would
inject bytestreams back into the code leading to tracebacks.
Instead, ignore the decode errors. Fixes:
Traceback (most recent call last):
File "/home/pokybuild/yocto-worker/a-full/build/scripts/resulttool", line 78, in <module>
sys.exit(main())
File "/home/pokybuild/yocto-worker/a-full/build/scripts/resulttool", line 72, in main
ret = args.func(args, logger)
File "/home/pokybuild/yocto-worker/a-full/build/scripts/lib/resulttool/store.py", line 70, in store
resultutils.save_resultsdata(results, tempdir, ptestlogs=True)
File "/home/pokybuild/yocto-worker/a-full/build/scripts/lib/resulttool/resultutils.py", line 178, in save_resultsdata
f.write(sectionlog)
TypeError: write() argument must be str, not bytes
(From OE-Core rev: 3aa0d0fd0bbf8c9c13719251a2fd1e85a22c7336)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Enable reporting selected test case result given the user provided
the selected test case id. If both test result id and test case id
were provided, report the selected test case result from the
selected test result id.
(From OE-Core rev: 7161310ce32d6e0c397d0132808d556bdc80d183)
Signed-off-by: Yeoh Ee Peng <ee.peng.yeoh@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Enable the option to add extra test environment data to the
configuration of each test result (as optional).
Example of optional test environment data include:
- custom packages included for runtime test
- detail machine specification used as target
- detail host environment used for bitbake
(From OE-Core rev: 3cc887a72c5726024c3699e6a3deb72fad4992a9)
Signed-off-by: Yeoh Ee Peng <ee.peng.yeoh@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Add total passed, failed, and skipped statistic to test result.
(From OE-Core rev: 7ad74f99b6105f23bcf930712627aac279c6af10)
Signed-off-by: Yeoh Ee Peng <ee.peng.yeoh@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
In case of debugging, report user need to acccess the raw
test result. Instead of going back to source file/directory/URL
to manually pull out the raw result, provide alternative
way to let report showing raw test results by providing
the result id (optional).
(From OE-Core rev: 8b2485bc0d0fa2f72b37e8bafbfa663d456ea98c)
Signed-off-by: Yeoh Ee Peng <ee.peng.yeoh@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
By default, report will use the store_map to generate the key
to reference each result set. In some situation when using store_map
with multiple set of tests sharing similar test configurations,
the report will only showing partial result set for results
that having identical result_id (use of multiconfig to run tests
where it generate identical result_id).
Enable report to have the option to use the regression_map (optional)
instead of the default store_map, where it will take larger
set of configurations to generate the key to reference each
result set, this will prevent the report from only showing
partial result set.
(From OE-Core rev: 8577abce6feb1f529537da554d39e86a5b05ca2d)
Signed-off-by: Yeoh Ee Peng <ee.peng.yeoh@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Do not decode the log content into a string only to re-encode it as
binary data again. Some logs might un-intentionally contain bytes that
do not decode as utf-8, as such preserve the log file content as it was
on disk.
Handle the decoding on the resulttool side, but also handle the failure
to decode the data.
(From OE-Core rev: 20531dc0b8f76a6e37cc856f36cd94077b6aba50)
Signed-off-by: Nathan Rossi <nathan@nathanrossi.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Add the OEPTestResultTestCase class as a mix-in class to provide helper
functions for interacting with ptestresults within the extraresults
object generated by the test case.
This class also provides default compression of log text and log files.
Also add support to resulttool for decoding/decompressing log files
embedded in the test results.
(From OE-Core rev: 06cba9883a5964320969301fd05eeb6bec3e786d)
Signed-off-by: Nathan Rossi <nathan@nathanrossi.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Handle multiple results series having ptestresults content. The contents
are merged on a per-result basis where duplicates are ignored (with a
warning message printed). The 'ptestresults.sections' collection is also
merged on a per-suite basis.
(From OE-Core rev: 47edd51970ed0c33edbe04fd72abd1cfc6ecd3d1)
Signed-off-by: Nathan Rossi <nathan@nathanrossi.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Adds an argument to the log subcommand to extract the raw logs from the
reproducible selftest.
To prevent ambiguity, the "--raw" argument has been renamed
"--raw-ptest", although the old "--raw" argument is kept around for
compatibility.
[YOCTO #13324]
(From OE-Core rev: 7a4ebb361ff1efc22e7dafadfa60c98bc8a79ed4)
Signed-off-by: Joshua Watt <JPEWhacker@gmail.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Add additional result types into the dictionary to handle dejagnu style
test results. These include PASS, FAIL, XPASS, XFAIL, UNSUPPORTED,
UNTESTED, UNRESOLVED and ERROR.
(From OE-Core rev: 684fc36402a23760b203f4761f284043031c799c)
Signed-off-by: Nathan Rossi <nathan@nathanrossi.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Currently, if a test occurs multiple times over different series, the
code will sum these. This can lead to confusion over the desired
results. Change the code to report the redundant tests and skip adding
an additional values.
(From OE-Core rev: caeaa0648dff64c56a33f52e45e82bfab6719e3e)
Signed-off-by: Jon Mason <jdmason@kudzu.us>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Current QA team need to merge test result files from multiple sources.
Adding TESTSERIES configuration too early will have negative
implication to report and regression. Enable control to add TESTSERIES
when needed. Also enable adding EXECUTED_BY configuration when
needed.
(From OE-Core rev: 651d8d371e78e77599d56681228d5782664f7743)
Signed-off-by: Yeoh Ee Peng <ee.peng.yeoh@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Current results stored does not have information needed to trace who
executed the tests. Enable store to add EXECUTED_BY configuration
to results file in order to track who executed the tests.
(From OE-Core rev: dca2a57d54163a2e63b06e2f140fea3bd49cef0d)
Signed-off-by: Yeoh Ee Peng <ee.peng.yeoh@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Current resultutils library always add "TESTSERIES" configuration
to results. Enhance this to allow control of adding "TESTSERIES"
configuration as well as allow adding extra configurations
when needed.
(From OE-Core rev: 443c0acc14ef2451b10878fc83dd11b46805daf0)
Signed-off-by: Yeoh Ee Peng <ee.peng.yeoh@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Printing the lack of a test is not necessary (per feedback). Remove
this from the template to quieten it.
(From OE-Core rev: b1fe6ae66360e160eeaeafe456536f335a0eab60)
Signed-off-by: Jon Mason <jdmason@kudzu.us>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Allow the creation of test case configuration file based on user inputs.
Where this testcase configuration file will be used by the the manual
execution to run selected test cases for a module rather than compulsory
run all test cases in manual json file.
(From OE-Core rev: 73d2a747c17779da0ca972da776b3cf02c2e1cbc)
Signed-off-by: sangeeta jain <sangeeta.jain@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Currently, the code will sum all of the different machine results into a
single report of the tests results. This can lead to confusion as to
which machine may be experiencing issues. Modify the code to store the
results in a per machine basis and report them accordingly.
(From OE-Core rev: 16d4031ea5df8a4ddfdb937d35464c09e1abd10e)
Signed-off-by: Jon Mason <jdmason@kudzu.us>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
This adds SPDX license headers in place of the wide assortment of things
currently in our script headers. We default to GPL-2.0-only except for the
oeqa code where it was clearly submitted and marked as MIT on the most part
or some scripts which had the "or later" GPL versioning.
The patch also drops other obsolete bits of file headers where they were
encoountered such as editor modelines, obsolete maintainer information or
the phrase "All rights reserved" which is now obsolete and not required in
copyright headers (in this case its actually confusing for licensing as all
rights were not reserved).
More work is needed for OE-Core but this takes care of the bulk of the scripts
and meta/lib directories.
The top level LICENSE files are tweaked to match the new structure and the
SPDX naming.
(From OE-Core rev: f8c9c511b5f1b7dbd45b77f345cb6c048ae6763e)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Adds an option to dump all the ptest logs to individual files in a
specified directory. If multiple test runs are present, the
'--prepend-run' argument will create separate directories for each test
run under the target directory and put the logs there to prevent each
test run from clobbering the others.
[YOCTO #13331]
(From OE-Core rev: f262b9e7482e38f04dfa162f0aaf65b17e86f8b2)
Signed-off-by: Joshua Watt <JPEWhacker@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Current manualexecution required user to exceute all test cases defined inside a "modulename.json" file in oeqa/manual
There are cases when all test cases all not required to run for a module on specific DUT.
Enable manualexecution to have the optional feature where it will use pre-defined json format test case configuration file
where user will be able to select test cases from the "modulename.json" instead of running all of them. This will help
in reducing testing time and reporting unneccesary skip or failures.
Example pre-defined json format test case configuration file (for build-applince):
{
"testcases" : [
"build-appliance.build-appliance.Create_core-image-sato-sdk_using_build_appliance",
"build-appliance.build-appliance.Build_a_image_without_error_(added_recipe)"
]
}
(From OE-Core rev: 3950c28a34f94b5907d37b579bdaee5a59794652)
Signed-off-by: sangeeta jain <sangeeta.jain@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Fix AttributeError: module 'urllib' has no attribute 'request' when
using remote http urls.
(From OE-Core rev: 9acc47236e2a9ac7f97f80040f5e508f23f01611)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Adds a subcommand for dumping various logs from test results
(From OE-Core rev: 454b8d2cdc6f79a51e610dae92c22352850c3f7c)
Signed-off-by: Joshua Watt <JPEWhacker@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Adds support for resulttool to load JSON files directly from a http://
or https:// URL
(From OE-Core rev: 235bcf0c504e7ea253ccfb28d600898117c64c1f)
Signed-off-by: Joshua Watt <JPEWhacker@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Remove duplicate codes. Replace unnecessary class variables with
local variables. Rename variables and arguments with simple and
standard name.
(From OE-Core rev: 6bef61d36f3328fad003c0dc9c010d3f76ba96d8)
Signed-off-by: Yeoh Ee Peng <ee.peng.yeoh@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Allow the creation of configuration option file based on user inputs.
Where this configuration option file will be used by the the manual
execution to display options for configuration rather than user
need to inputs configuration manually.
(From OE-Core rev: 23c7a037a85a825aa088bfac3a144c14087a9f49)
Signed-off-by: Yeoh Ee Peng <ee.peng.yeoh@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Current manualexecution required user to input configuration manually
where there were inconsistent inputs and human typo issues.
Enable manualexecution to have the optional feature where it
will use pre-compiled configuration options file where user will
be able to select configuration from the pre-compiled list instead
of manual key-in the configuration. This will eliminate human error.
Expect the pre-compiled configuration options file in json format below
{
"bsps-hw": {
"IMAGE_BASENAME": {
"1": "core-image-sato-sdk"
},
"MACHINE": {
"1": "beaglebone-yocto",
"2": "edgerouter",
"3": "mpc8315e-rdb",
"4": "genericx86",
"5": "genericx86-64"
}
},
"bsps-qemu": {
"IMAGE_BASENAME": {
"1": "core-image-sato-sdk"
},
"MACHINE": {
"1": "qemuarm",
"2": "qemuarm64",
"3": "qemumips",
"4": "qemumips64",
"5": "qemuppc",
"6": "qemux86",
"7": "qemux86-64"
}
}
}
(From OE-Core rev: 1af7e04223b415a64e100a9f91d60f5e9b0f789a)
Signed-off-by: Yeoh Ee Peng <ee.peng.yeoh@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Currently the manual execution display step by sorting
the step as string, where steps were not being sorted
correctly when there are more than 9 steps.
Fixed the step sorting by sorting step as integer.
(From OE-Core rev: 192c255d9f597b3526a5b94fcf3eedc4527189ed)
Signed-off-by: Yeoh Ee Peng <ee.peng.yeoh@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Current manualexecution required pressing enter button to show each step
information, where this was wasting execution time. Enable display
full steps without needing to any press enter button.
(From OE-Core rev: e44c9a018c13208fef0dcef4479ce71977628cd4)
Signed-off-by: Mazliana <mazliana.mohamad@intel.com>
Signed-off-by: Yeoh Ee Peng <ee.peng.yeoh@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Current input checking does not match the standard input practiced
by QA team. Change the input checking to match the standard
input practiced by the QA team.
(From OE-Core rev: 630a93c740359a65249a198f314f15040042f8d3)
Signed-off-by: Yeoh Ee Peng <ee.peng.yeoh@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Rather than simply discarding the ptest data, change the code to discard
it when writing out the new testresult files, or optionally either preserve
it, or write it as seperate discrete logs.
This means the autobuilder should start writing out individual ptest log
files as well as allowing ueers to extract these manually.
(From OE-Core rev: a1e0944bf260ef50dd7dfcb10db248fdd7f45bc9)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Store operations using a single file as a source weren't working as the os.walk
command didn't like being given a single file. Fix the store operation to
work for single files.
(From OE-Core rev: eb5bbe613d8c7cbcd8b74d8ac0073c2217970410)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
We found that manualexecution does not capture test suite values
correctly if there are more than one test suite in test cases.
After verification has made we found out we should retrieved
full test cases value <test_module.test_suite.test_case> from
oeqa/manual/ json file rather than split it them into new
variables test_suite and test_cases.
(From OE-Core rev: db08ec5f48badd5be2fa24cc0d89a4386ab44a59)
Signed-off-by: Mazliana <mazliana.mohamad@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Enable roll-up all test results belong to a commit
and to provide a roll-up report.
(From OE-Core rev: a98a75a3db1fe9d327fd21919fb8ea09e76fb1e3)
Signed-off-by: Yeoh Ee Peng <ee.peng.yeoh@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
To enable store for testresults.json file from manualexecution,
add layers metadata to configuration and add "manual" map to
resultutils.store_map.
To enable regression for manual, add "manual" map to
resultutils.regression_map. Also added compulsory configurations
('MACHINE', 'IMAGE_BASENAME') to manualexecution.
(From OE-Core rev: 7521834f8557565d3898605257c7fd9505ea22bb)
Signed-off-by: Yeoh Ee Peng <ee.peng.yeoh@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Sorted regression results to provide friendly viewing of report.
(From OE-Core rev: 59d827889dd8b790a7c9323c69565d6bba3041bd)
Signed-off-by: Yeoh Ee Peng <ee.peng.yeoh@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
ptest suites with no results don't show up on the reports even though we have
a duration for them. Fix this so the fact they report no tests is visible.
(From OE-Core rev: 58268151704246a81ec8dae46c26346023057554)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Currently some older results files cause the code to give tracebacks.
Handle these missing sections more cleanly.
(From OE-Core rev: f778c191dbd5740173b3be07f4c1655d85a07bb2)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Currently we cant store results if the results files span multiple
different build revisons. Remove this limitation by iterating.
(From OE-Core rev: 4da12c00963b02508056b87ce9b972528ce3a1be)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
