Commit Graph

531 Commits

Author SHA1 Message Date
Richard Purdie
b666351385 scripts/dashboard: Improve index page
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2025-03-05 18:11:11 +00:00
Richard Purdie
c6b8047092 scripts/run-dashboard-index: Correct rsync command
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2025-03-05 16:50:43 +00:00
Ross Burton
b75ea57cd5 scripts/run-dashboard-index: add ability to override the destination for testing
Signed-off-by: Ross Burton <ross.burton@arm.com>
2025-03-05 16:41:57 +00:00
Ross Burton
c8f84c27c7 dashboard: add new bug triage page
Add a replacement for the the existing Bug Triage page on the wiki[1].

This uses client-side JavaScript to parse the releases.json and query
Bugzilla, so shouldn't need updating very often.

[1] https://wiki.yoctoproject.org/wiki/Bug_Triage

Signed-off-by: Ross Burton <ross.burton@arm.com>
2025-03-05 16:40:15 +00:00
Antonin Godard
80a2e370ae scripts/yocto-supported-distros: use urllib instead of requests
The requests module is not available to all workers, use urllib instead.

Signed-off-by: Antonin Godard <antonin.godard@bootlin.com>
2025-03-05 14:47:24 +00:00
Richard Purdie
e84649fd17 scripts/run-config: Simplify toolchain test builds as dropped from a-quick
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2025-03-05 14:41:32 +00:00
Richard Purdie
ef5e2143a3 scripts/dashboard: Update index page
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2025-03-05 10:39:36 +00:00
Antonin Godard
f72aed5f5c scripts/yocto-supported-distros: ignore missing on AB for stable branches
Add a stable_release boolean argument to _compare(). If True just
ignore missing workers on the AB, because we want to keep the full list
of tested releases in poky.conf for stable branches.

Signed-off-by: Antonin Godard <antonin.godard@bootlin.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2025-03-05 10:25:52 +00:00
Antonin Godard
dd78eda374 scripts/yocto-supported-distros: run bitbake-getvar with --quiet
On older releases the bitbake server output is confusing the script.

Signed-off-by: Antonin Godard <antonin.godard@bootlin.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2025-03-05 10:25:52 +00:00
Richard Purdie
3659743a57 scripts/run-toaster-tests: Add check of toaster fixtures against current release data
Add a check of the fixture data to try and ensure we don't release with
incorrect configuration for toaster.

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2025-03-05 10:24:55 +00:00
Richard Purdie
b990e819aa config.json/utils: Disable buildtools and poky repo for indexing
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2025-03-05 10:24:55 +00:00
Richard Purdie
cea5f73aac setup-config: Add support for removevars
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2025-03-05 10:24:55 +00:00
Michael Halstead
6301c006a6 scripts/release-parser.py: changed exceptions after tag clean up
Signed-off-by: Michael Halstead <mhalstead@linuxfoundation.org>
2025-03-05 10:24:55 +00:00
Michael Halstead
a1582ba785 scripts/layer-parser.py: remove EOL layers
Remove caching and optimize to run on the autobuilder from scripts/run-dashboard-index

Signed-off-by: Michael Halstead <mhalstead@linuxfoundation.org>
2025-03-05 10:24:55 +00:00
Richard Purdie
6a7b1e8dc3 list-yp-compatible-layers: Update to current release branches
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2025-03-05 10:24:55 +00:00
Richard Purdie
5c0f4e14b1 scripts/run-dashboard-index: Ensure failures show an error
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2025-03-05 10:24:55 +00:00
Michael Halstead
56ce05ac27 scripts/run-dashboard-index: Generate parsed-layers.json from the dashboard script
Signed-off-by: Michael Halstead <mhalstead@linuxfoundation.org>
2025-03-05 10:24:55 +00:00
Richard Purdie
2f325f5909 scripts/run-dashboard-index: Generate releases.json from the dashboard script
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2025-03-05 10:24:55 +00:00
Richard Purdie
8dc7382252 scripts/run-dashboard-index: Add basic index page for new dashboard site
Add a basic index page for the dashboard site and tag onto the indexing job

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2025-03-05 10:24:40 +00:00
Richard Purdie
0cd37de01e scripts/release-parser: Add list of release version to release series information
It would be useful to know the list of releases in each release series
so add this information to releases.json. Also add milestone informaiton
for the current development branch.

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2025-03-04 21:59:20 +00:00
Richard Purdie
6d1ea392e5 scripts/yocto-supported-distros: Better integrate for autobuilder use
For autobuilder use, it will be more helpful to print warnings for the issues found
which will show up in the autobuilder UI in this format. We can then save the error
exit code for actual script failures which should improve usability from the autobuilder
perspective.

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2025-03-04 15:17:41 +00:00
Antonin Godard
68a8ce4e33 scripts: add yocto-supported-distros to list or compare supported distros
Add scripts/yocto-supported-distros to either:

- List the supported distros on the autobuilder (prints the workers for
  one or more release).

- With --compare, get the supported distro from poky.conf
  (SANITY_TESTED_DISTROS), mangle the worker names to make them match the
  lsb_release distro strings, and return 1 in case of difference (and
  print the differences). Return 0 in case of 1 to 1 match.

The aim of this script is to make maintaining the poky.conf file and the
workers easier.

The release-from-env flag can be used to get the current release from
METADATA_BRANCH.

The --config-from-web flag can be used to get the current autobuilder
config.py by fetching it from the web repo.

Signed-off-by: Antonin Godard <antonin.godard@bootlin.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2025-03-04 13:52:55 +00:00
Antonin Godard
c68e7b7b95 run-docs-build: add a custom fonts.conf to fix rsvg-convert errors
rsvg-convert, which we use to convert SVG images to PDFs/PNGs, would
always print out the following error:

  Fontconfig error: Cannot load default config file: No such file: (null)

As a result some SVGs are oddly rendered because it doesn't find the
correct fonts.

This can be easily fixed by setting and exporting the FONTCONFIG_PATH
variable which must point to a directory containing a fonts.conf file.
Since we build on multiple distros, we may not always find this file in
/etc/fonts (installed there by default on many distros). Additionally,
we can also install custom fonts using tlmgr and use them thanks to that
file.

Add a custom fonts.conf that adds /texlive/texmf-dist/fonts to the list
of directories to look for fonts. This files is just based on the
original fonts.conf. Also add the nimbus15 package to the list of tlmgr
installed packages so that we make sure we always use the same font for
rendering images (assuming the SVG files use the "Nimbus Sans L" font).

Signed-off-by: Antonin Godard <antonin.godard@bootlin.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2025-02-20 15:12:49 +00:00
Antonin Godard
abae49b0f6 scripts/release-parser.py: merge full list with previous list
On https://www.yoctoproject.org/development/releases, the "Full Release
List" can mislead users into thinking the list represents all of the
Yocto release, whereas these are only EOL releases. Remove this list and
merge everything in "Previous Release List".

[YOCTO #15339]

Signed-off-by: Antonin Godard <antonin.godard@bootlin.com>
2025-01-23 11:35:37 -10:00
Michael Halstead
384d37a957 scripts/release-parser: remove Dunfell from active releases
Signed-off-by: Michael Halstead <mhalstead@linuxfoundation.org>
2024-12-26 08:51:00 -10:00
Richard Purdie
e39db7b08e Revert "scripts/run-docs-build: update buildtools-docs url"
This reverts commit 440a0b2d8f.
2024-12-26 10:19:04 +00:00
Mathieu Dubois-Briand
78806a4f4c prepare-shared-repos: Allow to tag poky git
Signed-off-by: Mathieu Dubois-Briand <mathieu.dubois-briand@bootlin.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2024-12-17 15:22:36 +00:00
Antonin Godard
440a0b2d8f scripts/run-docs-build: update buildtools-docs url
We were using this one temporarily to use the new Sphinx version and
rsvg-tools. Update the URL now that 5.1.1 is released.

Signed-off-by: Antonin Godard <antonin.godard@bootlin.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2024-12-16 15:34:33 +00:00
Richard Purdie
89d58d91af send_qa_email: Add directory to save logs too
The logs take up too much space in the git repo, save them separately.

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2024-11-25 16:02:25 +00:00
Richard Purdie
001b65c6bd send_qa_email: Filter to specific revision to avoid OE-Core commits
Poky and OE-Core commits are being stored into the same branch of the test
results repo which causes confusion. Filter these to poky and ignore the
OE-Core result for now as the least bad option we have to avoid issues.

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2024-11-14 11:45:32 +00:00
Richard Purdie
79abfc9a43 scripts/run-docs-build: Add support for multiple buildtools and add modified 5.1 option
Add a modified 5.1 buildtools tarball for docs builds which has support for svg -> png
conversion using rsvg-convert.

Older releases need older docs buildtools. Add support for this in the build
script for docs, installing both and choosing depending on how old the main
release branch is. For now I've put the changeover at scarthgap, it may be
possible to make it older. I know kirkstone fails with the new tools.

Bitbake doesn't need this since it's docs work fine with both versions.

The actual docs build is done in a subshell so we can choose the environment.
We may need to improve the selection logic in future since some branches may
start to build with the newer tools after backports.

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2024-10-31 14:13:43 +00:00
Richard Purdie
b2f7653b8b scripts/run-docs-build: Add latexmk to tex install
Tweak the tex install command to include latexmk which is needed
by the docs build process.

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2024-10-31 14:12:40 +00:00
Richard Purdie
33e9270998 scripts/run-docs-build: Fix missing fi typo
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2024-10-31 09:43:07 +00:00
Richard Purdie
bb8ba61312 scripts/run-docs-build: Add support for installing texlive for pdf generation
To build the latexpdf output for the docs, we need Xetex (which can handle
international characters). Rather than require it on all autobuilder workers,
along with dependencies, we install a copy onto NFS which the workers can
use. The full install is 6GB so just install the pieces we've found we
need via tlmgr on top of a small install. This script creates the install
if it isn't present.

For local docs builds, we assume the user has the tools available from
the host OS. It is unlikely most users using this script would have the
NAS path present so this seems like a reasonable compromise.

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2024-10-30 16:35:15 +00:00
Ross Burton
f2ff264492 scripts/abint: generalise autobuilder URL regex
The autobuilder isn't just autobuilder.yoctoproject.org, as the new
cluster is valkyrie.yoctoproject.org.  Generalise the regex to match on
any buildbot URLs with "yocto" in the hostname, as we also have machines
using *.yocto.io.

Signed-off-by: Ross Burton <ross.burton@arm.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2024-10-23 11:38:22 +01:00
Michael Halstead
51d842b5e0 scripts/release-parser.py: Update for 5.1 release and Walnascar branch
Signed-off-by: Michael Halstead <mhalstead@linuxfoundation.org>
2024-10-16 15:30:17 -07:00
Mathieu Dubois-Briand
bcf51897f1 scripts/run-docs-build: Fix built branches list
Only build contrib branches when explicitly requested.

Signed-off-by: Mathieu Dubois-Briand <mathieu.dubois-briand@bootlin.com>
2024-10-16 11:06:42 +01:00
Richard Purdie
27196fdf41 scripts/run-patchmetrics: Avoid races over directory creation
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2024-10-09 23:20:46 +01:00
Richard Purdie
028cada341 auh: Update urls for new autobuilder address
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2024-10-09 23:14:08 +01:00
Richard Purdie
cca455446c scripts/run-docs-build: Use the 'cluster-downloads-cache' location for cache files
The docs build uses some cached tarballs, use the standarised location for these.

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2024-10-09 23:06:08 +01:00
Richard Purdie
c79cc99309 scripts/utils: Use a consistent 'cluster-downloads-cache' location for cache files
Cache files were getting spread all over. Use a more consistent directory
name to make it clear they are cache files.

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2024-10-09 21:19:17 +01:00
Michael Halstead
b093850639 scripts/release-parser: Update release generator for 5.1
Comment out future stable branch for easy updating.

Signed-off-by: Michael Halstead <mhalstead@linuxfoundation.org>
2024-08-22 06:09:56 -07:00
Richard Purdie
19f4fa0ac5 scripts/collect-results: Handle artefact collection/separation
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2024-07-17 14:44:51 +01:00
Richard Purdie
0646d5df44 scripts/archive_buildstats.py: Only create directories if we copy files
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2024-07-14 13:13:31 +01:00
Richard Purdie
faf9deb130 scripts/collect-results: Only create directories if we copy files
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2024-07-14 13:12:24 +01:00
Richard Purdie
3c44f8da3c collect-results: Fix dir creation and function indentation
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2024-07-13 23:49:10 +01:00
Richard Purdie
78ca7fb97b archive_buildstats: Only create dirs if we have data
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2024-07-13 11:42:30 +01:00
Richard Purdie
b1e2db6640 collect-results: Only create dirs if we have data
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2024-07-13 11:39:07 +01:00
Richard Purdie
9ada77ab57 scripts/generate-testresult-index.py: Add back data.json generation
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2024-07-12 23:11:51 +01:00
Richard Purdie
da2c03a6cb scripts/generate-ptest-logs: Fix typo
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2024-07-12 18:04:21 +01:00