mirror of git://git.yoctoproject.org/poky.git synced 2025-07-19 21:09:03 +02:00

Richard Purdie 47eb3d00e9 resulttool: Improvements to allow integration to the autobuilder

This is a combined patch of the various tweaks and improvements I
made to resulttool:

* Avoid subprocess.run() as its a python 3.6 feature and we
  have autobuilder workers with 3.5.

* Avoid python keywords as variable names

* Simplify dict accesses using .get()

* Rename resultsutils -> resultutils to match the resultstool ->
  resulttool rename

* Formalised the handling of "file_name" to "TESTSERIES" which the code
  will now add into the json configuration data if its not present, based
  on the directory name.

* When we don't have failed test cases, print something saying so
  instead of an empty table

* Tweak the table headers in the report to be more readable (reference
  "Test Series" instead if file_id and ID instead of results_id)

* Improve/simplify the max string length handling

* Merge the counts and percentage data into one table in the report
  since printing two reports of the same data confuses the user

* Removed the confusing header in the regression report

* Show matches, then regressions, then unmatched runs in the regression
  report, also remove chatting unneeded output

* Try harder to "pair" up matching configurations to reduce noise in
  the regressions report

* Abstracted the "mapping" table concept used to pairing in the
  regression code to general code in resultutils

* Created multiple mappings for results analysis, results storage and
  'flattening' results data in a merge

* Simplify the merge command to take a source and a destination,
  letting the destination be a directory or a file, removing the need for
  an output directory parameter

* Add the 'IMAGE_PKGTYPE' and 'DISTRO' config options to the regression
  mappings

* Have the store command place the testresults files in a layout from
  the mapping, making commits into the git repo for results storage more
  useful for simple comparison purposes

* Set the oe-git-archive tag format appropriately for oeqa results
  storage (and simplify the commit messages closer to their defaults)

* Fix oe-git-archive to use the commit/branch data from the results file

* Cleaned up the command option help to match other changes

* Follow the model of git branch/tag processing used by oe-build-perf-report
  and use that to read the data using git show to avoid branch change

* Add ptest summary to the report command

* Update the tests to match the above changes

(From OE-Core rev: ff2c029b568f70aa9960dde04ddd207829812ea0)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>

2019-02-21 12:34:00 +00:00

3.4 KiB

Executable File

Raw Blame History

#!/usr/bin/env python3

test results tool - tool for manipulating OEQA test result json files

(merge results, summarise results, regression analysis, generate manual test results file)

To look for help information.

$ resulttool

To store test results from oeqa automated tests, execute the below

$ resulttool store <source_dir> <git_branch>

To merge test results, execute the below

$ resulttool merge <base_result_file> <target_result_file>

To report test report, execute the below

$ resulttool report <source_dir>

To perform regression file analysis, execute the below

$ resulttool regression-file <base_result_file> <target_result_file>

To execute manual test cases, execute the below

$ resulttool manualexecution

By default testresults.json for manualexecution store in /tmp/log/manual/

Copyright (c) 2019, Intel Corporation.

This program is free software; you can redistribute it and/or modify it

under the terms and conditions of the GNU General Public License,

version 2, as published by the Free Software Foundation.

This program is distributed in the hope it will be useful, but WITHOUT

ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or

FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for

more details.

import os import sys import argparse import logging script_path = os.path.dirname(os.path.realpath(file)) lib_path = script_path + '/lib' sys.path = sys.path + [lib_path] import argparse_oe import scriptutils import resulttool.merge import resulttool.store import resulttool.regression import resulttool.report import resulttool.manualexecution logger = scriptutils.logger_create('resulttool')

def _validate_user_input_arguments(args): if hasattr(args, "source_dir"): if not os.path.isdir(args.source_dir): logger.error('source_dir argument need to be a directory : %s' % args.source_dir) return False return True

def main(): parser = argparse_oe.ArgumentParser(description="OEQA test result manipulation tool.", epilog="Use %(prog)s --help to get help on a specific command") parser.add_argument('-d', '--debug', help='enable debug output', action='store_true') parser.add_argument('-q', '--quiet', help='print only errors', action='store_true') subparsers = parser.add_subparsers(dest="subparser_name", title='subcommands', metavar='') subparsers.required = True subparsers.add_subparser_group('manualexecution', 'manual testcases', 300) resulttool.manualexecution.register_commands(subparsers) subparsers.add_subparser_group('setup', 'setup', 200) resulttool.merge.register_commands(subparsers) resulttool.store.register_commands(subparsers) subparsers.add_subparser_group('analysis', 'analysis', 100) resulttool.regression.register_commands(subparsers) resulttool.report.register_commands(subparsers)

args = parser.parse_args()
if args.debug:
    logger.setLevel(logging.DEBUG)
elif args.quiet:
    logger.setLevel(logging.ERROR)

if not _validate_user_input_arguments(args):
    return -1

try:
    ret = args.func(args, logger)
except argparse_oe.ArgumentUsageError as ae:
    parser.error_subcommand(ae.message, ae.subcommand)
return ret

if name == "main": sys.exit(main())

3.4 KiB Executable File Raw Blame History