The autobuilder isn't just autobuilder.yoctoproject.org, as the new
cluster is valkyrie.yoctoproject.org. Generalise the regex to match on
any buildbot URLs with "yocto" in the hostname, as we also have machines
using *.yocto.io.
Signed-off-by: Ross Burton <ross.burton@arm.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Cache files were getting spread all over. Use a more consistent directory
name to make it clear they are cache files.
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Add a cache to improve performance and allow old index entries to be reused.
Also add locking so only one copy of the script can run at once.
Fix the path to the index template to be more stable in different execution environments.
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The ptest extraction should really happen separately to the index generation,
move it to do so.
Add code to only consider directories changed in the last 7 days to make
the script faster.
Add an indexing target to config.json to trigger the script.
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
There's a chance that run-cvecheck and run-patchmetrics fail to push
because the remote repository has changed. Try to resolve this by pulling
and rebasing immediately before the push, and failing if we can't rebase.
[ YOCTO #15529 ]
Signed-off-by: Ross Burton <ross.burton@arm.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Fixes [YOCTO 15503]
When running autobuilder onm a testing or stable branch, the following
exception may occur:
Traceback (most recent call last):
File "/home/pokybuild/yocto-worker/a-full/yocto-autobuilder-helper/scripts/send_qa_email.py", line 278, in <module>
send_qa_email()
File "/home/pokybuild/yocto-worker/a-full/yocto-autobuilder-helper/scripts/send_qa_email.py", line 207, in send_qa_email
generate_regression_report(querytool, targetrepodir, regression_base, regression_target, tempdir, args.results_dir, log)
File "/home/pokybuild/yocto-worker/a-full/yocto-autobuilder-helper/scripts/send_qa_email.py", line 93, in generate_regression_report
regreport = subprocess.check_output([querytool, "regression-report", base, target, '-t', resultdir])
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/lib64/python3.11/subprocess.py", line 466, in check_output
return run(*popenargs, stdout=PIPE, timeout=timeout, check=True,
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/lib64/python3.11/subprocess.py", line 548, in run
with Popen(*popenargs, **kwargs) as process:
^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/lib64/python3.11/subprocess.py", line 1026, in __init__
self._execute_child(args, executable, preexec_fn, close_fds,
File "/usr/lib64/python3.11/subprocess.py", line 1883, in _execute_child
self.pid = _fork_exec(
^^^^^^^^^^^
TypeError: expected str, bytes or os.PathLike object, not NoneType
This issue is due to base and target revision being None, but subprocess
module do not tolerate arguments being None,
Prevent the script from even trying to generate a regression report if we
are missing some info about revisions to compare.
Cc: Steve Sakoman <steve@sakoman.com>
Signed-off-by: Alexis Lothoré <alexis.lothore@bootlin.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Commit 1dc77f5908 ("scripts: send_qa_email: add dry-run mode") updated
the get_regression_base_and_target to allow it to receive a test result
url. Unfortunately, this function's tests have not been updated
accordingly, so they currently fail because of the missing parameter.
Add a default url in test case and update the function call with this
default value.
Signed-off-by: Alexis Lothoré <alexis.lothore@bootlin.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
When send_qa_email is executed with python >= 3.12, the following warnings
are emitted:
/home/alexis/src/yocto_ci/yocto-autobuilder-helper/./scripts/send_qa_email.py:22: SyntaxWarning: invalid escape sequence '\d'
p = re.compile('\d{8}-\d+')
/home/alexis/src/yocto_ci/yocto-autobuilder-helper/./scripts/send_qa_email.py:67: SyntaxWarning: invalid escape sequence '\/'
tested_revision = re.match('refs\/tags\/.*\/\d+-g([a-f0-9]+)\/\d', latest_test_tag).group(1)
This warning has been introduced to replace the DeprecationWarning
initially raised on such escape code in string regex. Python 3.12 changelog
([1]) states that raw string must be used in this case. Update both
send_qa_email and its tests file.
[1] https://docs.python.org/dev/whatsnew/3.12.html#other-language-changes
Signed-off-by: Alexis Lothoré <alexis.lothore@bootlin.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
- Added pico.css to make CSS improvements
- Added filters to the table for better readability of the test results. Filters were added for build type, branch and date
- Added pagination, which improved the performance of the website
- The html index template has been added as a separate file for better maintainability
Co-Authored-By: Ninette Adhikari <13760198+ninetteadhikari@users.noreply.github.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
This was removed from meta-arm as it has been obsoleted by genericarm64.
Signed-off-by: Ross Burton <ross.burton@arm.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
A build of "stable/dunfell-nut" caused incorrect data to be added to the metrics
repo and resulted in script failures. Handle this situation better.
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
the two html files, index.html and index-full.html,
got merged so we only need the json file with the complete set of data.
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The default sorting was based on the list of seen times, but it should
be based on the latest seen time.
Signed-off-by: Ross Burton <ross.burton@arm.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Add an argument parser so that the use of the cache or verbose logging
can be enabled/disabled without having to edit the source code.
Signed-off-by: Ross Burton <ross.burton@arm.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
This script generates sorting keys so that 20240201-4 sorts before 20240201-20,
but the code assumes that there is only one hyphen when it splits.
However, there is now a patchstatus-meta-oe directory, which causes the
script to throw an exception.
Before padding, use a regex to check that the key is of the format we expect,
that is two integers separated by a hyphen.
Signed-off-by: Ross Burton <ross.burton@arm.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
This report visualises the AB-INT bugs over time, to help find trends
such as bugs which are no longer occuring, or bugs which should be fixed
urgently.
Signed-off-by: Ross Burton <ross.burton@arm.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Add meta-exein yocto-check-layer testing. Due to HOSTTOOLS issues, we
need to had a horrible hack. v2 compatibility doesn't test for HOSTTOOLS
changes although perhaps the next version might.
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
This is useful when using large sets of layers to
verify that the right layers have been added in
the right order.
Signed-off-by: Ross Burton <ross.burton@arm.com>
Extract the logging of auto.conf to a new log_file_contents() function,
and instead of calling it before _every_ call of bitbake, only show it when
actually writing the auto.conf.
Signed-off-by: Ross Burton <ross.burton@arm.com>
It is often helpful to know how many CVEs are open against a given recipe.
Add a summary table of this to the end of the CVE listing.
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Add a dry-run mode to be able to run send_qa_email locally but disabling
any output (no commit or tag pushed upstream, no mail sent). This eases
all release-related debugging. This dry-run mode is enabled by the
following changes:
- add a -d/--dry-run parameter to send_qa_email
- update test_results url to allow cloning test_results repository without
having its public key registered upstream
- skip test results storage
- do not erase test results temp dir
- skip email sending (but still generate it)
Signed-off-by: Alexis Lothoré <alexis.lothore@bootlin.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
generate_regression_reports is currently protect in a try/catch block to
prevent it from canceling QA email generation when encountering an issue,
but get_regression_base_and_target is not.
Make sure that get_regression_base_and_target can not prevent QA email from
being generated by adding it to the try/catch block protecting
send_qa_email. While doing so, make sure to preserve the exitcode variable
to make sure that the step is still marked as fail in autobuilder to make
sure the error does not go silent. However the variable is not needed as
global anymore since it is now used in a single function.
Signed-off-by: Alexis Lothoré <alexis.lothore@bootlin.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
send_qa_email currently fails when dealing with release version starting a
new major, for example 5.0. This has been observed for example when trying
to generate 5.0_M1.rc1
This specific versioning makes previous tag computation method fall through
last branch which currently expects that the current release tag indeed
exists (5.0_M1), which is true when checking regression reports a
posteriori, but not in an autobuilder run (tag is added only when the
release has been "validated")
Fix tag computation for this case by getting previous release tag with git
ls-remote, instead of relying on git describe with a possibly non-existing
tag. While doing so, add a few tests about this specific case.
Signed-off-by: Alexis Lothoré <alexis.lothore@bootlin.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
send_qa_email currently deals with tags in a "reproducible" way: despite
new versions being released on different branches, the computation of the
"reference" version for a specific input version always remain the same.
This behavior does not match perfectly real expectations: if at the some
point we get version 4.3.1 as a comparison reference for a regression
report, and 4.3.2 is released some time later, we want the next comparision
to be done against 4.3.2.
Start introducing this new behavior by allowing the tests to check returned
version against regex patterns instead of static strings, so we can for
example use wildcards on the "micro" version
Signed-off-by: Alexis Lothoré <alexis.lothore@bootlin.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Use intermediate variables for test input/output, and print expected
output in subtest to get more info when a test fails
Signed-off-by: Alexis Lothoré <alexis.lothore@bootlin.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
