It would be useful to know the list of releases in each release series
so add this information to releases.json. Also add milestone informaiton
for the current development branch.
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
For autobuilder use, it will be more helpful to print warnings for the issues found
which will show up in the autobuilder UI in this format. We can then save the error
exit code for actual script failures which should improve usability from the autobuilder
perspective.
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Add scripts/yocto-supported-distros to either:
- List the supported distros on the autobuilder (prints the workers for
one or more release).
- With --compare, get the supported distro from poky.conf
(SANITY_TESTED_DISTROS), mangle the worker names to make them match the
lsb_release distro strings, and return 1 in case of difference (and
print the differences). Return 0 in case of 1 to 1 match.
The aim of this script is to make maintaining the poky.conf file and the
workers easier.
The release-from-env flag can be used to get the current release from
METADATA_BRANCH.
The --config-from-web flag can be used to get the current autobuilder
config.py by fetching it from the web repo.
Signed-off-by: Antonin Godard <antonin.godard@bootlin.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
rsvg-convert, which we use to convert SVG images to PDFs/PNGs, would
always print out the following error:
Fontconfig error: Cannot load default config file: No such file: (null)
As a result some SVGs are oddly rendered because it doesn't find the
correct fonts.
This can be easily fixed by setting and exporting the FONTCONFIG_PATH
variable which must point to a directory containing a fonts.conf file.
Since we build on multiple distros, we may not always find this file in
/etc/fonts (installed there by default on many distros). Additionally,
we can also install custom fonts using tlmgr and use them thanks to that
file.
Add a custom fonts.conf that adds /texlive/texmf-dist/fonts to the list
of directories to look for fonts. This files is just based on the
original fonts.conf. Also add the nimbus15 package to the list of tlmgr
installed packages so that we make sure we always use the same font for
rendering images (assuming the SVG files use the "Nimbus Sans L" font).
Signed-off-by: Antonin Godard <antonin.godard@bootlin.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
On https://www.yoctoproject.org/development/releases, the "Full Release
List" can mislead users into thinking the list represents all of the
Yocto release, whereas these are only EOL releases. Remove this list and
merge everything in "Previous Release List".
[YOCTO #15339]
Signed-off-by: Antonin Godard <antonin.godard@bootlin.com>
We were using this one temporarily to use the new Sphinx version and
rsvg-tools. Update the URL now that 5.1.1 is released.
Signed-off-by: Antonin Godard <antonin.godard@bootlin.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Poky and OE-Core commits are being stored into the same branch of the test
results repo which causes confusion. Filter these to poky and ignore the
OE-Core result for now as the least bad option we have to avoid issues.
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Add a modified 5.1 buildtools tarball for docs builds which has support for svg -> png
conversion using rsvg-convert.
Older releases need older docs buildtools. Add support for this in the build
script for docs, installing both and choosing depending on how old the main
release branch is. For now I've put the changeover at scarthgap, it may be
possible to make it older. I know kirkstone fails with the new tools.
Bitbake doesn't need this since it's docs work fine with both versions.
The actual docs build is done in a subshell so we can choose the environment.
We may need to improve the selection logic in future since some branches may
start to build with the newer tools after backports.
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Tweak the tex install command to include latexmk which is needed
by the docs build process.
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
To build the latexpdf output for the docs, we need Xetex (which can handle
international characters). Rather than require it on all autobuilder workers,
along with dependencies, we install a copy onto NFS which the workers can
use. The full install is 6GB so just install the pieces we've found we
need via tlmgr on top of a small install. This script creates the install
if it isn't present.
For local docs builds, we assume the user has the tools available from
the host OS. It is unlikely most users using this script would have the
NAS path present so this seems like a reasonable compromise.
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The autobuilder isn't just autobuilder.yoctoproject.org, as the new
cluster is valkyrie.yoctoproject.org. Generalise the regex to match on
any buildbot URLs with "yocto" in the hostname, as we also have machines
using *.yocto.io.
Signed-off-by: Ross Burton <ross.burton@arm.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Cache files were getting spread all over. Use a more consistent directory
name to make it clear they are cache files.
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Add a cache to improve performance and allow old index entries to be reused.
Also add locking so only one copy of the script can run at once.
Fix the path to the index template to be more stable in different execution environments.
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The ptest extraction should really happen separately to the index generation,
move it to do so.
Add code to only consider directories changed in the last 7 days to make
the script faster.
Add an indexing target to config.json to trigger the script.
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
There's a chance that run-cvecheck and run-patchmetrics fail to push
because the remote repository has changed. Try to resolve this by pulling
and rebasing immediately before the push, and failing if we can't rebase.
[ YOCTO #15529 ]
Signed-off-by: Ross Burton <ross.burton@arm.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Fixes [YOCTO 15503]
When running autobuilder onm a testing or stable branch, the following
exception may occur:
Traceback (most recent call last):
File "/home/pokybuild/yocto-worker/a-full/yocto-autobuilder-helper/scripts/send_qa_email.py", line 278, in <module>
send_qa_email()
File "/home/pokybuild/yocto-worker/a-full/yocto-autobuilder-helper/scripts/send_qa_email.py", line 207, in send_qa_email
generate_regression_report(querytool, targetrepodir, regression_base, regression_target, tempdir, args.results_dir, log)
File "/home/pokybuild/yocto-worker/a-full/yocto-autobuilder-helper/scripts/send_qa_email.py", line 93, in generate_regression_report
regreport = subprocess.check_output([querytool, "regression-report", base, target, '-t', resultdir])
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/lib64/python3.11/subprocess.py", line 466, in check_output
return run(*popenargs, stdout=PIPE, timeout=timeout, check=True,
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/lib64/python3.11/subprocess.py", line 548, in run
with Popen(*popenargs, **kwargs) as process:
^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/lib64/python3.11/subprocess.py", line 1026, in __init__
self._execute_child(args, executable, preexec_fn, close_fds,
File "/usr/lib64/python3.11/subprocess.py", line 1883, in _execute_child
self.pid = _fork_exec(
^^^^^^^^^^^
TypeError: expected str, bytes or os.PathLike object, not NoneType
This issue is due to base and target revision being None, but subprocess
module do not tolerate arguments being None,
Prevent the script from even trying to generate a regression report if we
are missing some info about revisions to compare.
Cc: Steve Sakoman <steve@sakoman.com>
Signed-off-by: Alexis Lothoré <alexis.lothore@bootlin.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Commit 1dc77f5908 ("scripts: send_qa_email: add dry-run mode") updated
the get_regression_base_and_target to allow it to receive a test result
url. Unfortunately, this function's tests have not been updated
accordingly, so they currently fail because of the missing parameter.
Add a default url in test case and update the function call with this
default value.
Signed-off-by: Alexis Lothoré <alexis.lothore@bootlin.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
When send_qa_email is executed with python >= 3.12, the following warnings
are emitted:
/home/alexis/src/yocto_ci/yocto-autobuilder-helper/./scripts/send_qa_email.py:22: SyntaxWarning: invalid escape sequence '\d'
p = re.compile('\d{8}-\d+')
/home/alexis/src/yocto_ci/yocto-autobuilder-helper/./scripts/send_qa_email.py:67: SyntaxWarning: invalid escape sequence '\/'
tested_revision = re.match('refs\/tags\/.*\/\d+-g([a-f0-9]+)\/\d', latest_test_tag).group(1)
This warning has been introduced to replace the DeprecationWarning
initially raised on such escape code in string regex. Python 3.12 changelog
([1]) states that raw string must be used in this case. Update both
send_qa_email and its tests file.
[1] https://docs.python.org/dev/whatsnew/3.12.html#other-language-changes
Signed-off-by: Alexis Lothoré <alexis.lothore@bootlin.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
- Added pico.css to make CSS improvements
- Added filters to the table for better readability of the test results. Filters were added for build type, branch and date
- Added pagination, which improved the performance of the website
- The html index template has been added as a separate file for better maintainability
Co-Authored-By: Ninette Adhikari <13760198+ninetteadhikari@users.noreply.github.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
This was removed from meta-arm as it has been obsoleted by genericarm64.
Signed-off-by: Ross Burton <ross.burton@arm.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
A build of "stable/dunfell-nut" caused incorrect data to be added to the metrics
repo and resulted in script failures. Handle this situation better.
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
the two html files, index.html and index-full.html,
got merged so we only need the json file with the complete set of data.
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The default sorting was based on the list of seen times, but it should
be based on the latest seen time.
Signed-off-by: Ross Burton <ross.burton@arm.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Add an argument parser so that the use of the cache or verbose logging
can be enabled/disabled without having to edit the source code.
Signed-off-by: Ross Burton <ross.burton@arm.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
