The LTP test reporting appears to be a little fragile so I tried to make
it more reliable.
Primarily this is done by not passing -p to runltp, which results in
machine-readable logfiles instead of human-readable. These are easier
to parse and have more context in, so we can also report correctly
skipped tests.
(From OE-Core rev: d585c6062fcf452e7288f6f8fb540fd92cbf5ea2)
Signed-off-by: Ross Burton <ross.burton@arm.com>
Signed-off-by: Alexandre Belloni <alexandre.belloni@bootlin.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Occasionally we see:
File "/media/build/poky/meta/lib/oeqa/runtime/cases/ptest.py", line 27, in test_ptestrunner_expectfail
self.do_ptestrunner()
File "/media/build/poky/meta/lib/oeqa/runtime/cases/ptest.py", line 77, in do_ptestrunner
results, sections = parser.parse(ptest_runner_log)
File "/media/build/poky/meta/lib/oeqa/utils/logparser.py", line 80, in parse
self.results[current_section['name']][result.group(1).strip()] = t
KeyError: 'No-section'
which occurs when there are "results" outside the main log section. The strace
tests do then upon failure as they dump logs there.
Add code to avoid the tracebacks and just make them warnings.
(From OE-Core rev: d9bf95d8cfb123f9d992fd2a95099bdcece97be8)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Ensure that even if a ptests results section is empty, the log parser adds that
empty section. Then ensure that empty sections trigger warnings.
This means if a ptest suddently stops returning any results, we notice and see
warnings about it. This has gone unnoticed on the autobuilder far too many times
so is very much worth highlighting as a regression. We shouldn't have empty ptests.
(From OE-Core rev: 5ad0cf57b41ec7f44647a03bc568d0b24906cc8d)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Where there isn't a copyright statement, add one to make it explicit.
Also add license identifiers as MIT if there isn't one.
(From OE-Core rev: bb731d1f3d2a1d50ec0aed864dbca54cf795b040)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The ltp compliancy parser is rewritten to actually
match the logs: they seem to be unstructured, test case names
are not printed and the only indication of failure is appearance of
FAIL[ED] somewhere.
(From OE-Core rev: 52766561dbfee625c89393905a85e10d85f69c6c)
Signed-off-by: Alexander Kanavin <alex@linutronix.de>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
I spotted this by noticing ptests started to unexpectedly pass,
which shouldn't happen yet.
(From OE-Core rev: 1cd6e9a8fdfef927916c6cea2371fc0430b1ffdf)
Signed-off-by: Alexander Kanavin <alex.kanavin@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
On the autobuilder a ptest log with 2.1 million lines took around 18 hours to
process. This is clearly crazy.
We can tweak the processing code to:
a) Stop repeatedly joining large strings together (append to a list instead)
b) Use one startswith expression instead of multiple re.search() operations
With this change it takes 18 hours down to around 12s.
[YOCTO #13696]
(From OE-Core rev: 01c9f40b979e36a53e789a6bedd89b6d9557dce3)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
(From OE-Core rev: eb79217b8761816a21c8f7bed3c5379c1b9230ea)
Signed-off-by: Robert Yang <liezhi.yang@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The output format for Python and GLib both can be of this form:
FAIL: foobar (Segmentation fault)
In this case the test is called foobar not foobar_segmentation_fault.
(From OE-Core rev: 0219e9bd0273661b4b70df97e5762f77b3ac3e8c)
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
This adds SPDX license headers in place of the wide assortment of things
currently in our script headers. We default to GPL-2.0-only except for the
oeqa code where it was clearly submitted and marked as MIT on the most part
or some scripts which had the "or later" GPL versioning.
The patch also drops other obsolete bits of file headers where they were
encoountered such as editor modelines, obsolete maintainer information or
the phrase "All rights reserved" which is now obsolete and not required in
copyright headers (in this case its actually confusing for licensing as all
rights were not reserved).
More work is needed for OE-Core but this takes care of the bulk of the scripts
and meta/lib directories.
The top level LICENSE files are tweaked to match the new structure and the
SPDX naming.
(From OE-Core rev: f8c9c511b5f1b7dbd45b77f345cb6c048ae6763e)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Get rid of further unneeded code complications:
* value mappings we could just direct use
* ftools when we can write files easily ourself
* test result status filtering we don't use
* variable overwriting module imports
(From OE-Core rev: d6065f136f6d353c3054cc3f440a4e259509f876)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Merge the results handling into the ptest log parser as a seperate
method.
Drop the weird "pass.skip.fail." prefix to the results filename, its
just bizarre.
Drop the code turning a list into a regex then searching the regex for
an item, "x in y" is perfectly capable.
Use a dict, sort the keys as needed and drop the list sorting code.
(From OE-Core rev: f317800e950b4a37b4034133bc52e0c47f04dc29)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Allow parsing of the ptest duration, exit code and timeout keywords
from the logs, returning data on each section.
Also include the logs broken out per section.
(From OE-Core rev: a9a67dccaa5be0f06eedcab46dcff7cbf9202850)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Now we have a dedicated ptest parser, merge in the remaining ptest
specific pieces to further clarify and simplify the code, moving to
a point where we can consider extending/enhancing it.
(From OE-Core rev: 05991bb5bc8018275d03fdeecee3d5a757840c7c)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Rename the paster to be ptest specific and apply some further cleanups
to the code to simplify and clarify what its doing.
(From OE-Core rev: 45a5886f1ec458d4c306b8d68fd31d568bc36b47)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
logparser is only used by ptest. Its slightly overcomplicated as it was
intended to be reusable but wasn't. Simplify it as a dedicated parser is
likely to me more readable and maintainable.
(From OE-Core rev: c7478345b2b4a85cb1fec40e762633871f0e94cb)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The section might be None or '', so use "if section" for it.
Fixed:
File "/buildarea/lyang1/poky/meta/lib/oeqa/utils/logparser.py", line 113, in log_as_files
prefix += section
TypeError: Can't convert 'NoneType' object to str implicitly
[YOCTO #11547]
(From OE-Core rev: 320ea01f9eb33df462534bf08488ff6ada9bbe7b)
Signed-off-by: Robert Yang <liezhi.yang@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Some test cases maybe skipped, let's parse it.
[YOCTO #11547]
(From OE-Core rev: f23c0f6a5aa93d38a5ea9d450f0accff0a5cc869)
Signed-off-by: Robert Yang <liezhi.yang@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
This patch contains all the other misc pieces of the transition to
python3 which didn't make sense to be broken into individual patches.
(From OE-Core rev: fcd6b38bab8517d83e1ed48eef1bca9a9a190f57)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
A module for parsing results based logs like ptest, compliance and performance.
Supports breaking the logs into multiple sections and also provides a result object to use the parser with.
The parser is initialized with the regex required to identify results and section statements in the target log file.
(From OE-Core rev: 72308d030fc98982e88b121b052cd2438f6b031c)
Signed-off-by: Corneliu Stoicescu <corneliux.stoicescu@intel.com>
Signed-off-by: Lucian Musat <georgex.l.musat@intel.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
