Commit Graph

542 Commits

Author SHA1 Message Date
Richard Purdie
e90fc2078c scripts/run-config: Allow a specific step to disable buildtools
We need buildtools (and the poky repo) for resulttool in some indexing steps. We don't
want buildtools in the dashboard step as we need semver from the host.

Rework the config to allow this and fix failures on the infrastructure.

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2025-03-10 11:51:27 +00:00
Michael Halstead
daac0b0326 scripts/release-parser.py: display end of stable release period.
Signed-off-by: Michael Halstead <mhalstead@linuxfoundation.org>
2025-03-07 10:01:48 -08:00
Antonin Godard
5b9fefac38 scripts/yocto-supported-distros: check METADATA_BRANCH for master
METADATA_BRANCH is the only way for us to check whether we're on master.
Check this variable first, and return the value of
LAYERSERIES_COMPAT_core if we're not on master.

Signed-off-by: Antonin Godard <antonin.godard@bootlin.com>
2025-03-07 14:40:16 +00:00
Antonin Godard
4e7389cbdb scripts/yocto-supported-distros: use LAYERSERIES_COMPAT_core to get current release
Using METADATA_BRANCH is less accurate than getting the current release
from LAYERSERIES_COMPAT_core, which should always be equal to the
release we want to compare for.

The code the exit if the current release was not found is left there
just in case.

Signed-off-by: Antonin Godard <antonin.godard@bootlin.com>
2025-03-07 10:31:43 +00:00
Richard Purdie
2ea53506e2 dashboard: Add additional urls to index page
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2025-03-06 12:13:15 +00:00
Richard Purdie
28cc1c8b88 dashboard: Update to new gitstats urls
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2025-03-05 23:19:05 +00:00
Richard Purdie
32b06ff9b0 run-gitstats: Drop yocto-docs for now, not mirrored
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2025-03-05 22:31:05 +00:00
Richard Purdie
7120bbbefa scripts/run-gitstats: Hide stdout since it is too verbose
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2025-03-05 22:07:37 +00:00
Richard Purdie
a516c15ba7 scripts/run-gitstats: Add script to generate gitstats data
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2025-03-05 22:01:30 +00:00
Richard Purdie
010b3adda4 dashboard: Add additional urls to index page
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2025-03-05 22:01:30 +00:00
Richard Purdie
cbdc73dd1d scripts/dashboard: Separate weekly status report
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2025-03-05 18:12:57 +00:00
Richard Purdie
b666351385 scripts/dashboard: Improve index page
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2025-03-05 18:11:11 +00:00
Richard Purdie
c6b8047092 scripts/run-dashboard-index: Correct rsync command
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2025-03-05 16:50:43 +00:00
Ross Burton
b75ea57cd5 scripts/run-dashboard-index: add ability to override the destination for testing
Signed-off-by: Ross Burton <ross.burton@arm.com>
2025-03-05 16:41:57 +00:00
Ross Burton
c8f84c27c7 dashboard: add new bug triage page
Add a replacement for the the existing Bug Triage page on the wiki[1].

This uses client-side JavaScript to parse the releases.json and query
Bugzilla, so shouldn't need updating very often.

[1] https://wiki.yoctoproject.org/wiki/Bug_Triage

Signed-off-by: Ross Burton <ross.burton@arm.com>
2025-03-05 16:40:15 +00:00
Antonin Godard
80a2e370ae scripts/yocto-supported-distros: use urllib instead of requests
The requests module is not available to all workers, use urllib instead.

Signed-off-by: Antonin Godard <antonin.godard@bootlin.com>
2025-03-05 14:47:24 +00:00
Richard Purdie
e84649fd17 scripts/run-config: Simplify toolchain test builds as dropped from a-quick
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2025-03-05 14:41:32 +00:00
Richard Purdie
ef5e2143a3 scripts/dashboard: Update index page
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2025-03-05 10:39:36 +00:00
Antonin Godard
f72aed5f5c scripts/yocto-supported-distros: ignore missing on AB for stable branches
Add a stable_release boolean argument to _compare(). If True just
ignore missing workers on the AB, because we want to keep the full list
of tested releases in poky.conf for stable branches.

Signed-off-by: Antonin Godard <antonin.godard@bootlin.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2025-03-05 10:25:52 +00:00
Antonin Godard
dd78eda374 scripts/yocto-supported-distros: run bitbake-getvar with --quiet
On older releases the bitbake server output is confusing the script.

Signed-off-by: Antonin Godard <antonin.godard@bootlin.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2025-03-05 10:25:52 +00:00
Richard Purdie
3659743a57 scripts/run-toaster-tests: Add check of toaster fixtures against current release data
Add a check of the fixture data to try and ensure we don't release with
incorrect configuration for toaster.

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2025-03-05 10:24:55 +00:00
Richard Purdie
b990e819aa config.json/utils: Disable buildtools and poky repo for indexing
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2025-03-05 10:24:55 +00:00
Richard Purdie
cea5f73aac setup-config: Add support for removevars
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2025-03-05 10:24:55 +00:00
Michael Halstead
6301c006a6 scripts/release-parser.py: changed exceptions after tag clean up
Signed-off-by: Michael Halstead <mhalstead@linuxfoundation.org>
2025-03-05 10:24:55 +00:00
Michael Halstead
a1582ba785 scripts/layer-parser.py: remove EOL layers
Remove caching and optimize to run on the autobuilder from scripts/run-dashboard-index

Signed-off-by: Michael Halstead <mhalstead@linuxfoundation.org>
2025-03-05 10:24:55 +00:00
Richard Purdie
6a7b1e8dc3 list-yp-compatible-layers: Update to current release branches
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2025-03-05 10:24:55 +00:00
Richard Purdie
5c0f4e14b1 scripts/run-dashboard-index: Ensure failures show an error
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2025-03-05 10:24:55 +00:00
Michael Halstead
56ce05ac27 scripts/run-dashboard-index: Generate parsed-layers.json from the dashboard script
Signed-off-by: Michael Halstead <mhalstead@linuxfoundation.org>
2025-03-05 10:24:55 +00:00
Richard Purdie
2f325f5909 scripts/run-dashboard-index: Generate releases.json from the dashboard script
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2025-03-05 10:24:55 +00:00
Richard Purdie
8dc7382252 scripts/run-dashboard-index: Add basic index page for new dashboard site
Add a basic index page for the dashboard site and tag onto the indexing job

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2025-03-05 10:24:40 +00:00
Richard Purdie
0cd37de01e scripts/release-parser: Add list of release version to release series information
It would be useful to know the list of releases in each release series
so add this information to releases.json. Also add milestone informaiton
for the current development branch.

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2025-03-04 21:59:20 +00:00
Richard Purdie
6d1ea392e5 scripts/yocto-supported-distros: Better integrate for autobuilder use
For autobuilder use, it will be more helpful to print warnings for the issues found
which will show up in the autobuilder UI in this format. We can then save the error
exit code for actual script failures which should improve usability from the autobuilder
perspective.

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2025-03-04 15:17:41 +00:00
Antonin Godard
68a8ce4e33 scripts: add yocto-supported-distros to list or compare supported distros
Add scripts/yocto-supported-distros to either:

- List the supported distros on the autobuilder (prints the workers for
  one or more release).

- With --compare, get the supported distro from poky.conf
  (SANITY_TESTED_DISTROS), mangle the worker names to make them match the
  lsb_release distro strings, and return 1 in case of difference (and
  print the differences). Return 0 in case of 1 to 1 match.

The aim of this script is to make maintaining the poky.conf file and the
workers easier.

The release-from-env flag can be used to get the current release from
METADATA_BRANCH.

The --config-from-web flag can be used to get the current autobuilder
config.py by fetching it from the web repo.

Signed-off-by: Antonin Godard <antonin.godard@bootlin.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2025-03-04 13:52:55 +00:00
Antonin Godard
c68e7b7b95 run-docs-build: add a custom fonts.conf to fix rsvg-convert errors
rsvg-convert, which we use to convert SVG images to PDFs/PNGs, would
always print out the following error:

  Fontconfig error: Cannot load default config file: No such file: (null)

As a result some SVGs are oddly rendered because it doesn't find the
correct fonts.

This can be easily fixed by setting and exporting the FONTCONFIG_PATH
variable which must point to a directory containing a fonts.conf file.
Since we build on multiple distros, we may not always find this file in
/etc/fonts (installed there by default on many distros). Additionally,
we can also install custom fonts using tlmgr and use them thanks to that
file.

Add a custom fonts.conf that adds /texlive/texmf-dist/fonts to the list
of directories to look for fonts. This files is just based on the
original fonts.conf. Also add the nimbus15 package to the list of tlmgr
installed packages so that we make sure we always use the same font for
rendering images (assuming the SVG files use the "Nimbus Sans L" font).

Signed-off-by: Antonin Godard <antonin.godard@bootlin.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2025-02-20 15:12:49 +00:00
Antonin Godard
abae49b0f6 scripts/release-parser.py: merge full list with previous list
On https://www.yoctoproject.org/development/releases, the "Full Release
List" can mislead users into thinking the list represents all of the
Yocto release, whereas these are only EOL releases. Remove this list and
merge everything in "Previous Release List".

[YOCTO #15339]

Signed-off-by: Antonin Godard <antonin.godard@bootlin.com>
2025-01-23 11:35:37 -10:00
Michael Halstead
384d37a957 scripts/release-parser: remove Dunfell from active releases
Signed-off-by: Michael Halstead <mhalstead@linuxfoundation.org>
2024-12-26 08:51:00 -10:00
Richard Purdie
e39db7b08e Revert "scripts/run-docs-build: update buildtools-docs url"
This reverts commit 440a0b2d8f.
2024-12-26 10:19:04 +00:00
Mathieu Dubois-Briand
78806a4f4c prepare-shared-repos: Allow to tag poky git
Signed-off-by: Mathieu Dubois-Briand <mathieu.dubois-briand@bootlin.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2024-12-17 15:22:36 +00:00
Antonin Godard
440a0b2d8f scripts/run-docs-build: update buildtools-docs url
We were using this one temporarily to use the new Sphinx version and
rsvg-tools. Update the URL now that 5.1.1 is released.

Signed-off-by: Antonin Godard <antonin.godard@bootlin.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2024-12-16 15:34:33 +00:00
Richard Purdie
89d58d91af send_qa_email: Add directory to save logs too
The logs take up too much space in the git repo, save them separately.

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2024-11-25 16:02:25 +00:00
Richard Purdie
001b65c6bd send_qa_email: Filter to specific revision to avoid OE-Core commits
Poky and OE-Core commits are being stored into the same branch of the test
results repo which causes confusion. Filter these to poky and ignore the
OE-Core result for now as the least bad option we have to avoid issues.

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2024-11-14 11:45:32 +00:00
Richard Purdie
79abfc9a43 scripts/run-docs-build: Add support for multiple buildtools and add modified 5.1 option
Add a modified 5.1 buildtools tarball for docs builds which has support for svg -> png
conversion using rsvg-convert.

Older releases need older docs buildtools. Add support for this in the build
script for docs, installing both and choosing depending on how old the main
release branch is. For now I've put the changeover at scarthgap, it may be
possible to make it older. I know kirkstone fails with the new tools.

Bitbake doesn't need this since it's docs work fine with both versions.

The actual docs build is done in a subshell so we can choose the environment.
We may need to improve the selection logic in future since some branches may
start to build with the newer tools after backports.

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2024-10-31 14:13:43 +00:00
Richard Purdie
b2f7653b8b scripts/run-docs-build: Add latexmk to tex install
Tweak the tex install command to include latexmk which is needed
by the docs build process.

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2024-10-31 14:12:40 +00:00
Richard Purdie
33e9270998 scripts/run-docs-build: Fix missing fi typo
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2024-10-31 09:43:07 +00:00
Richard Purdie
bb8ba61312 scripts/run-docs-build: Add support for installing texlive for pdf generation
To build the latexpdf output for the docs, we need Xetex (which can handle
international characters). Rather than require it on all autobuilder workers,
along with dependencies, we install a copy onto NFS which the workers can
use. The full install is 6GB so just install the pieces we've found we
need via tlmgr on top of a small install. This script creates the install
if it isn't present.

For local docs builds, we assume the user has the tools available from
the host OS. It is unlikely most users using this script would have the
NAS path present so this seems like a reasonable compromise.

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2024-10-30 16:35:15 +00:00
Ross Burton
f2ff264492 scripts/abint: generalise autobuilder URL regex
The autobuilder isn't just autobuilder.yoctoproject.org, as the new
cluster is valkyrie.yoctoproject.org.  Generalise the regex to match on
any buildbot URLs with "yocto" in the hostname, as we also have machines
using *.yocto.io.

Signed-off-by: Ross Burton <ross.burton@arm.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2024-10-23 11:38:22 +01:00
Michael Halstead
51d842b5e0 scripts/release-parser.py: Update for 5.1 release and Walnascar branch
Signed-off-by: Michael Halstead <mhalstead@linuxfoundation.org>
2024-10-16 15:30:17 -07:00
Mathieu Dubois-Briand
bcf51897f1 scripts/run-docs-build: Fix built branches list
Only build contrib branches when explicitly requested.

Signed-off-by: Mathieu Dubois-Briand <mathieu.dubois-briand@bootlin.com>
2024-10-16 11:06:42 +01:00
Richard Purdie
27196fdf41 scripts/run-patchmetrics: Avoid races over directory creation
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2024-10-09 23:20:46 +01:00
Richard Purdie
028cada341 auh: Update urls for new autobuilder address
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2024-10-09 23:14:08 +01:00