send_qa_email currently fails when dealing with release version starting a
new major, for example 5.0. This has been observed for example when trying
to generate 5.0_M1.rc1
This specific versioning makes previous tag computation method fall through
last branch which currently expects that the current release tag indeed
exists (5.0_M1), which is true when checking regression reports a
posteriori, but not in an autobuilder run (tag is added only when the
release has been "validated")
Fix tag computation for this case by getting previous release tag with git
ls-remote, instead of relying on git describe with a possibly non-existing
tag. While doing so, add a few tests about this specific case.
Signed-off-by: Alexis Lothoré <alexis.lothore@bootlin.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
send_qa_email currently deals with tags in a "reproducible" way: despite
new versions being released on different branches, the computation of the
"reference" version for a specific input version always remain the same.
This behavior does not match perfectly real expectations: if at the some
point we get version 4.3.1 as a comparison reference for a regression
report, and 4.3.2 is released some time later, we want the next comparision
to be done against 4.3.2.
Start introducing this new behavior by allowing the tests to check returned
version against regex patterns instead of static strings, so we can for
example use wildcards on the "micro" version
Signed-off-by: Alexis Lothoré <alexis.lothore@bootlin.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Use intermediate variables for test input/output, and print expected
output in subtest to get more info when a test fails
Signed-off-by: Alexis Lothoré <alexis.lothore@bootlin.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Update run-toaster-test to include BUILDDIR and TOASTER_TEST_USE_SSTATE_MIRROR which will be use in the build test in toaster
Signed-off-by: Marlon Rodriguez Garcia <marlon.rodriguez-garcia@savoirfairelinux.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Strip quotes from directory variables as they caused path errors.
Add environment variable for the toaster artifact directory.
Migrate from tox and django tools to using pytest.
Install python module requirements from the script as this is no
longer handled by tox.
Signed-off-by: Alexander Lussier-Cullen <alexander.lussier-cullen@savoirfairelinux.com>
CC: Richard Purdie <richard.purdie@linuxfoundation.org>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Pass the toaster test environment SSTATE_DIR and DL_DIR for faster
builds and TOASTER_DJANGO_TMPDIR to remove problematic temp files
from the root level '/tmp' directory.
Signed-off-by: Alexander Lussier-Cullen <alexander.lussier-cullen@savoirfairelinux.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Sort versions as if we used semantic versioning.
Include updates for the 4.3 release as well.
Signed-off-by: Michael Halstead <mhalstead@linuxfoundation.org>
Add a runner that installs the patchtest dependencies in a Python venv
and then starts patchtest's selftests.
Signed-off-by: Trevor Gamblin <tgamblin@baylibre.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Add a toaster test builder that runs the tox test suite using a new
run-toaster-tests script.
Signed-off-by: Alexander Lussier-Cullen <alexander.lussier-cullen@savoirfairelinux.com>
CC: richard.purdie@linuxfoundation.org
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Do another git-pull in the metrics repository before updating the
metrics, in case other metrics jobs running in parallel have updated the
repositories since they were cloned. There will always be possibility
of racing metrics jobs, but this should reduce the chance of it
happening.
An alternative would be to commit and then rebase before pushing, but I
fear that a git-merge could produce invalid JSON and we'd have to
manually fix up the repository. In my opinion, a wasted metrics run is
preferable to potentially corrupted repositories.
Signed-off-by: Ross Burton <ross.burton@arm.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Do another git-pull in the metrics repository before updating the
metrics, in case other metrics jobs running in parallel have updated the
repositories since they were cloned. There will always be possibility
of racing metrics jobs, but this should reduce the chance of it
happening.
An alternative would be to commit and then rebase before pushing, but I
fear that a git-merge could produce invalid JSON and we'd have to
manually fix up the repository. In my opinion, a wasted metrics run is
preferable to potentially corrupted repositories.
[RP: Moved to after the bitbke invocation]
Signed-off-by: Ross Burton <ross.burton@arm.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
setup-auh and run-auh were doing what the AB config.json does:
* creating repo checkouts: Now use NEEDREPOS
* configuring bitbake env: Now use extravars
This refactoring is needed to prepare adding AUH meta-oe support.
Signed-off-by: Yoann Congal <yoann.congal@smile.fr>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Rewrite the scripts that gather the metrics to be more generic.
Extract the metrics repository cloning out so that we don't have to
repeatedly clone it.
Make the scripts parse their arguments using getopt and be more specific
about what they're passed. In particular, this means that for the patch
review run we pass the _repository_ that we're scanning so we can do git
operations on it, and the base of the _layers_ (either a layer, or a
directory containing layers) so we know what to scan.
Be more clever when identifying what commits we need to analyse for
patch review: instead of iterating through a set randomly, we can keep
the revision list sorted and the checkout operations are a lot faster.
Remove the commit/file count metric addition as patchreview itself does
that now.
Add an explicit --push option so it's easy to test the scripts in
isolation without pushing.
Signed-off-by: Ross Burton <ross.burton@arm.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
This is a handy helper script to query the Buildbot REST API to
create a list of Yocto Project Compatible 2.0 layers, as defined by
successful running of the "check-layers*" jobs.
The script only considers the 'jobs' ("Builders" in Buildbot speak) you
ask it to query.
The script looks at the latest -n successful builds for a branch that
matches from a list. The number of builds approach was used because
the property_list cannot be queried for "branch" in the "builders" API.
It looks for a "step" name pattern via a regex, with which all currently
running "check-layer" and "check-layer-nightly" jobs conform.
Once a branch has successfully been queried, it is removed from the list
in further iterations. The intent is to only find the latest passing
list of compatible layers, for each branch.
The output is a yp_compatible_layers.json file which can
be pretty printed with:
cat yp_compatible_layers.json | jq
Signed-off-by: Tim Orling <tim.orling@konsulko.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Rather than running multiple checkouts, lets move this to the autobuilder
to handle and have it trigger the builds with the right checkouts.
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
We only monitor the master branch for patch metrics as we can't really make
improvements to release branches.
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
To prepare for splitting things up, only clone the metrics repo if it isn't present.
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
It has been observed that some regression reports generation may failed
when the comparision base is a branch (e.g master) because we can not find
any test results associated to the branch HEAD. This is especially true for
branches which often change, because not all revisions on those branch are
subject to CI tests.
To fix that, whenever we are not dealing with a release, parse the latest
tested revision in test results repository on target branch in order to
guess the corresponding revision in poky repository, so we are sure that
revisions passed to yocto_testresults_query are indeed tested and
regression report can be generated
Signed-off-by: Alexis Lothoré <alexis.lothore@bootlin.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Test results repository url is used at least twice, so define a constant
holding the url instead of hardcoding it multiple times
Signed-off-by: Alexis Lothoré <alexis.lothore@bootlin.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
There are some inversions in words used to describe elements of comparison
for regression reporting: the main function of send_qa_email starts using
"base" to talk about the target revision and "compare" to talk about the
reference against which it is compared. Then later in the script, the
"base" is used as "base of comparison"/reference revision, while the
"target" branch/revision appears. This becomes quite confusing when we need
to update the script
Re-align wording to avoid confusion:
- always use "target" to talk about current branch/revision of interest
(the newest)
- always use "base" to talk about the reference branch/revision (the
oldest), against which we want to compare the target revision
Signed-off-by: Alexis Lothoré <alexis.lothore@bootlin.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Whilst the script needs to continue if we fail to generate a regression report,
set the exit code accordingly so our CI can flag the issue.
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
It is hard to tell which section of the code specific error messages
come from at present. Add more headers to the output so we can at
least tell which section the messages are from. It also adds some
timing information.
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Update getcomparisonbranch unit tests by removing BUILD_HISTORY_DIRECTPUSH
entry in fake configuration
Signed-off-by: Alexis Lothoré <alexis.lothore@bootlin.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
It has been observed that when a new release branch is created, it is quite
easy to forget to update the BUILD_HISTORY_DIRECTPUSH variable, which leads
to failures in autobuilder like test results not being pushed.
Replace the BUILD_HISTORY_DIRECTPUSH usage with a hardcoded condition which
validates any branch in poky representing a "main" branch, i.e all branches
not ending in "-next"
Signed-off-by: Alexis Lothoré <alexis.lothore@bootlin.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Tests results push command depends on basebranch and comparebranch
variables, which are computed based on config.json content. If this file is
not in sync with current release branch, tests results will be properly
stored in git directory but not pushed onto test results server. Since we
are able to detect this scenario, print at least a warning, without
breaking current build since it could be a release
Signed-off-by: Alexis Lothoré <alexis.lothore@bootlin.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
As for other scripts in yocto-autobuilder-helper or oecore, use python
logger class instead of raw print calls to allow log level distinction
Signed-off-by: Alexis Lothoré <alexis.lothore@bootlin.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
edgerouter is no longer part of meta-yocto so we removed it from the
autobuilder configuration as well.
Signed-off-by: Michael Halstead <mhalstead@linuxfoundation.org>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
AUH itself already has an option to include the link into
its email reports; the option just needs to be enabled.
[YOCTO #15103]
Signed-off-by: Alexander Kanavin <alex.kanavin@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
When available, expose tesresult-regressions-report.txt on non-release web page,
as it is done for many other artifacts currently
Signed-off-by: Alexis Lothoré <alexis.lothore@bootlin.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Some nightly builders are configured in yocto-autobuilder2 to run master builds.
Those build parameters currently skip all branches of
get_regression_base_and_target, which then return None, while the caller
expects a base and target tuple
Set default behaviour to return previous tag as comparison base and passed
branch as target for such builds
Signed-off-by: Alexis Lothoré <alexis.lothore@bootlin.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
When the test assert is about a tag in Poky, the result will not be the same
depending on existing tags at the time of running tests.
Add a LAST_TAG marker to loosen constraints but still allow to tests for general
cases (e.g. : test that tag-depending tests does not return None)
Signed-off-by: Alexis Lothoré <alexis.lothore@bootlin.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
d6018b891a broke regression reporting for testing
branches (e.g: master-next in poky, ross/mut in poky-contrib) by ignoring the comparebranch returned by
utils.getcomparison branch
Fix regression reporting for those branches by using comparebranch again. The
fix also refactor/add a intermediary step to guess base and target for
regression reporting, to isolate a bit the logic and make it easier later to add
multiple base/target couples
Signed-off-by: Alexis Lothoré <alexis.lothore@bootlin.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
is_non_release_version has an inverted logic which makes its reuse quite
confusing
Transform it as is_release_version and let caller do the negation if needed
Signed-off-by: Alexis Lothoré <alexis.lothore@bootlin.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
A new tool named yocto_testresults_query.py has been integrated in poky as a
thin wrapper between send-qa-email and resulttool. The new tool is in charge of
converting tags/branches names to SHA1 revisions and to call resulttool with
those revisions
Remove any code related to tag/branches conversions to SHA1 and use
yocto_testresults_qery.py instead
Signed-off-by: Alexis Lothoré <alexis.lothore@bootlin.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Minor typo observed when cheking the "Prepared shared repository" step logs in
autobuilder web interface:
====================================================================================================
Intially fetching repo poky (1675810261.1)
====================================================================================================
Signed-off-by: Alexis Lothoré <alexis.lothore@bootlin.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Current regression reports do not contain information about versions compared
when generating reports. While it is still possible to get the information by
searching the autobuilder log, it is not convenient. Moreover, future
developments will allow to generate multiple reports (with different bases for
comparison) in a single build.
As a consequence, embed target and base revisions in the report header
Signed-off-by: Alexis Lothoré <alexis.lothore@bootlin.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Since we are now force-fetching base revisions and target revisions for
regression report generation, we can make testresults clone even more "shallow"
to increase clone speed in CI pipelines
Signed-off-by: Alexis Lothoré <alexis.lothore@bootlin.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
If we try to run send-qa-email to simulate past releases (for example, for
development or debugging), the execution will very likely fail because the
target revision to examine (ie: the poky revision) is too old, and as a
consequence is not contained in the testresults shallow clone anymore (because
testsresults history keeps moving forward as builds are triggered on
autobuilder). As a consequence, force-fetch the "target" revision, as it is
already done for the "base" revision
Signed-off-by: Alexis Lothoré <alexis.lothore@bootlin.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The "previous version" computation bring many edge cases depending on the
version under release. Add a basic test suite to validate currently implemented
computation strategy and to prevent mistakes when ediiting it in the future
Signed-off-by: Alexis Lothoré <alexis.lothore@bootlin.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Instead of only generating regressions reports against HEAD of relevant branch, compute
most relevant tag (ie : release) against which we can check for regressions. General rules
introduced are the following :
- milestone release is checked against previous milestone if possible, otherwise
against major release
- point release is checked against previous point release if possible,
otherwise against major release
- major release is checked against previous major release
- a non release build is checked against base branch
Examples :
- 4.1.2.rc1 is checked against yocto-4.1.1
- 4.1.2 is checked against yocto-4.1.1
- 4.1.1.rc1 is checked against yocto-4.1
- 4.1.1 is checked against yocto-4.1
- 4.1 is checked against yocto-4.0
- 4.1.rc4 is checked against yocto-4.0
- 4.1_M2.rc1 is checked against 4.1_M1
- 4.1_M2 is checked against 4.1_M1
- 4.1_M1.rc1 is checked against yocto-4.0
- 4.1_M1 is checked against yocto-4.0
Signed-off-by: Alexis Lothoré <alexis.lothore@bootlin.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
This wrapping allows importing send_qa_email.py in another script without
executing the QA emailing routine. This is useful for unit testing the script
internal functions
Signed-off-by: Alexis Lothoré <alexis.lothore@bootlin.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Merely renaming send-qa-email to send_qa_email.py would imply to update
yocto-autobuilder2 scripts, which can be daunting. In order to preserve
compatibility with current configuration, add a symlink pointing to the
new script
Signed-off-by: Alexis Lothoré <alexis.lothore@bootlin.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
This rename fixes send-qa-email regarding python guidelines, which allows to
import the script content in another script (for unit testing purpose for
example)
Signed-off-by: Alexis Lothoré <alexis.lothore@bootlin.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The pub/non-release directories have aged out and no longer work.
Switch to a stable location and bump the version to 4.1.2 as well.
Signed-off-by: Michael Halstead <mhalstead@linuxfoundation.org>
This will be particularly useful for testing Y2038 issues,
as that is the only 32 bit target that can be run at native speed
on the AB cluster.
Whether a-full should get fast ptests or full ptests can be decided
after collecting AB execution times for both.
Signed-off-by: Alexander Kanavin <alex.kanavin@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
It happened that the git repositories were dirty and resulted in
incorrect files being used. Let's use git clean -ffdx to force a
completely clean git repositories before and after checking out a branch
so that nothing is left from or to another branch build. Also git reset
--hard to undo changes to tracked files.
Cc: Quentin Schulz <foss+yocto@0leil.net>
Reported-by: Peter Kjellerstedt <peter.kjellerstedt@axis.com> #https://lore.kernel.org/yocto-docs/e50abe3c777e4a23a752a3ec25ad0b2a@axis.com/
Signed-off-by: Quentin Schulz <quentin.schulz@theobroma-systems.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
We disabled meta-gplv2 for master, ensure the repo isn't used in the
collection or QA scripts.
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
With the recent template config changes to oe-core its content is no longer
backwards compatible.
'-f' is to guarantee future compatibility with build setups where the file
is not present.
Signed-off-by: Alexander Kanavin <alex.kanavin@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Archive the buildstats of every failed build and 1% of random builds. Convert
the time-stamped buildstats directory to a compressed tarball using the
hostname as a prefix (to the file name only) so that one can identify the
source machine. Move these tarballs to the directory:
testresults/<build_name>/buildstats/
The archiving is performed during the "collect results" step.
Signed-off-by: Aryaman Gupta <aryaman.gupta@windriver.com>
Signed-off-by: Randy MacLeod <Randy.MacLeod@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Master branch of the yocto-docs now autogenerates the releases.rst file
and the file therefore does not exist anymore.
Signed-off-by: Quentin Schulz <foss+yocto@0leil.net>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
`git checkout <pathspec>` without <tree-ish> only undo the changes made
to pathspec compared to HEAD. It is not guaranteed that HEAD is on
latest commit of master branch from upstream when running this script,
therefore let's add the <tree-ish> argument to specify which branch to
use to make sure set_versions.py is the latest, otherwise getlatest
might return the wrong information.
Cc: Quentin Schulz <foss+yocto@0leil.net>
Signed-off-by: Quentin Schulz <quentin.schulz@theobroma-systems.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
We may not have fetched the master revision if a different one was being
built so using origin/master works consistently in all cases.
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* Fix a syntax error causing the script to fail
* Skip symlinks files (symlink directories were already skipped)
* Drop unneeded whitespace
* Abstract path join to a variable
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Add a script which adds banners to the old docs and outdated dunfell docs
so that the user knows they're using old documentation.
Signed-off-by: Abongwa Bonalais Amahnui <abongwabonalais@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
For some reason, the old docs tarball includes many eclipse
subdirectories which are just cluttering the docs website up.
Therefore, let's just not extract eclipse directories from the tarball.
Cc: Quentin Schulz <foss+yocto@0leil.net>
Signed-off-by: Quentin Schulz <quentin.schulz@theobroma-systems.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Keeping all the previous json files in tree has too much overhead
so drop the old ones. Honister doesn't support cve-check with json.
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
When generating the chart specific data, remove elements which aren't
used to reuse the filesize which improves browser loading speed.
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
These are not part of our standard testing but allow them to
be triggered manually to assess their status.
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The intent here is to use the latest release branch name instead of a tagged version.
For the 4.0 case, this means the full migration guide will be shown instead of
the reduced information in the 4.0 tag.
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Since the latest tag already has a specific handling in the forloop,
let's just move the symlink creation inside the forloop.
Cc: Quentin Schulz <foss+yocto@0leil.net>
Signed-off-by: Quentin Schulz <quentin.schulz@theobroma-systems.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Currently, the master branch is the default branch being displayed by
default when reaching the Yocto docs website. When big changes are
implemented for the next release, these are shown immediately to the
user, even though there is currently no release available for those
changes. This is an issue when e.g. behaviors are changed, new features
are added, variables get renamed or some syntaxes change because the
user might try to use things that aren't available to them yet.
I believe more people are using released version of Yocto
Project/Bitbake than people working on latest master. So let's make the
default version of the docs the latest tag (in terms of version number,
not date of tagging) to avoid too much confusion.
The master branch of the docs is now available at /dev subpath.
Cc: Quentin Schulz <foss+yocto@0leil.net>
Signed-off-by: Quentin Schulz <quentin.schulz@theobroma-systems.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
set_versions.py and switchers.js.in need to be up-to-date so that a
consistent behavior is kept between different branches and tags of the
documentation.
Right now, kirkstone branch is lagging behind master and therefore does
not have the latest changes from master (e.g. the new obsolete
algorithm, which obviously isn't an issue right now, but will be in two
years). Using master version for those scripts also lightens the
maintenance burden.
Cc: Quentin Schulz <foss+yocto@0leil.net>
Signed-off-by: Quentin Schulz <quentin.schulz@theobroma-systems.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The rsync only makes sense for the Autobuilder as most people don't have
access to docs@docs.yoctoproject.org. Therefore, to allow for easier
contribution to this script, let's allow to skip the rsync so the script
can continue to run.
Cc: Quentin Schulz <foss+yocto@0leil.net>
Signed-off-by: Quentin Schulz <quentin.schulz@theobroma-systems.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Currently, the path to buildtools script and docbook tarball are
hardcoded to work on Yocto Project Autobuilder. However, this makes it
harder to contribute to this script because it is very unlikely those
paths exist on a developer PC.
Instead, let's allow to override variables by using the environment and
make the current hardcoded values the default ones.
Cc: Quentin Schulz <foss+yocto@0leil.net>
Signed-off-by: Quentin Schulz <quentin.schulz@theobroma-systems.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The script is written in such a way that absolute paths are expected.
Instead of failing weirdly at some point in the script, let's just make
all paths passed to this script absolute by calling realpath on them.
Cc: Quentin Schulz <foss+yocto@0leil.net>
Signed-off-by: Quentin Schulz <quentin.schulz@theobroma-systems.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
This uses the set_versions.py script from the docs master branch to handle
creation of the swictchers.js version information as well as the data in
poky.yaml.
This allows patches to those sections of the docs to be dropped.
It is assumed the patch to use set_versions is applied to the docs transitions
branch so that it's switchers.js files no longer need to be tweaked.
This does lead to user visible changes on the website:
- Older versioned released docs gain their current version and the
latest version listed in the switcher rather than being unlisted
- The list of releases is normally filtered down to our active ones
(hardknott, honister and dev right now)
- 3.3's doc references to gatesgath are corrected to hardknott
- Docs for unversioned release branches (not linked on the website) now
use 3.1.999 versioning instead of confusing them with the last relased version
- 3.1.13 refers to 3.1.13 instead of 3.1.12
all of which seem to be reasonable improvements.
The big advantage of these changes is that with a single change to the script
in the master branch, the right thing should happen for all the current docs
and at release time, only a docs rebuild should need to be triggered after tags
are pushed.
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Except patching which is specific to tags and yocto- tag prefix
stripping, the logic is identical, so let's merge both loops together.
Cc: Quentin Schuls <foss+yocto@0leil.net>
Signed-off-by: Quentin Schulz <quentin.schulz@theobroma-systems.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The commit that introduced Sphinx support in yocto-docs is
01dd5af7954e24552aca022917669b27bb0541ed. Any tag containing this commit
is buildable by sphinx.
Dunfell tags don't all have Sphinx support. However, all tags containing
the introducing commit c25fe058b88b893b0d146f3ed27320b47cdec236 are
buildable by sphinx.
Therefore, let's just list all tags which contains either of those two
commits instead of the complex series of pipes and shell commands.
Cc: Quentin Schulz <foss+yocto@0leil.net>
Signed-off-by: Quentin Schulz <quentin.schulz@theobroma-systems.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Since commit 01dd5af7954e24552aca022917669b27bb0541ed, all later
releases of yocto-docs can be built with Sphinx. Instead of manually
updating this list, let's have git return the list of remote branches
which contains the commit.
dunfell branch was initially released without Sphinx support but was
later patched, hence why it's explicitly listed.
Cc: Quentin Schulz <foss+yocto@0leil.net>
Signed-off-by: Quentin Schulz <quentin.schulz@theobroma-systems.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
master, master-next and transition only differ from other branches by
their output directory name. Let's put everything in common and only
have a check on whether the branch is master, master-next or transition
and modify the output dir in those cases.
Cc: Quentin Schulz <foss+yocto@0leil.net>
Signed-off-by: Quentin Schulz <quentin.schulz@theobroma-systems.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Since commit 84ccba0f4aff91528f764523fe1205a354c889ed, docs of all later
releases can be built with Sphinx. Instead of manually updating this
list, let's have git return the list of remote branches which contains
this commit.
1.46 branch was initially released without Sphinx support but was later
patched, hence why it's explicitly listed.
Cc: Quentin Schulz <foss+yocto@0leil.net>
Signed-off-by: Quentin Schulz <quentin.schulz@theobroma-systems.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
master and master-next only differ from other branches by their output
directory name. Let's put everything in common and only have a check on
whether the branch is master or master-next and modify the output dir in
those cases.
Cc: Quentin Schulz <foss+yocto@0leil.net>
Signed-off-by: Quentin Schulz <quentin.schulz@theobroma-systems.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The patch was meant as a quick temporary work-around to have the docs
built and published.
Now that releases where -W flag is set (turning warnings into errors)
are appropriately patched to make those warnings disappear (on Sphinx
v3.2.1 which is the one used on the builder), this patch can be reverted
so that next time a warning appears the doc building will fail but will
not destroy the doc website (because of commit
6a4e6ef18d "scripts: run-docs-build: make
the script fail hard ASAP when there's an error", since rsync will not
be run if any error happens before).
This reverts commit 931d409b25.
Cc: Quentin Schulz <foss+yocto@0leil.net>
Signed-off-by: Quentin Schulz <quentin.schulz@theobroma-systems.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The patch was missing from yocto-3.4.1 release and only made it to
yocto-3.4.2 unfortunately so let's backport it.
Cc: Quentin Schulz <foss+yocto@0leil.net>
Signed-off-by: Quentin Schulz <quentin.schulz@theobroma-systems.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Each Yocto Project release is validated for a specific Bitbake release.
By mistake, most release docs are pointing at the development branch of
Bitbake documentation instead of the one that was validated for the
Yocto Project release.
This fixes the issue by patching the conf.py of impacted releases.
Cc: Quentin Schulz <foss+yocto@0leil.net>
Signed-off-by: Quentin Schulz <quentin.schulz@theobroma-systems.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
There's no need to keep a list of tags requiring to be patched since the
tag is part of the path where patches are stored.
Therefore, let's only check if there's a patch directory for a given tag
and if so, apply all patches in there.
Cc: Quentin Schulz <foss+yocto@0leil.net>
Signed-off-by: Quentin Schulz <quentin.schulz@theobroma-systems.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
This approach didn't work reliably so switch to text based emails
pointing at the website version with graphs and data.
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Current setup only allows sending mail via localhost on alma workers.
Signed-off-by: Alexander Kanavin <alex.kanavin@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Add patch for version 3.3.5 and move patches into a dedicated
subdirectrory to keep the scripts directory tidy.
Signed-off-by: Michael Halstead <mhalstead@linuxfoundation.org>
When current_version and all_versions are not updated before the release
build the tag will not build working docs. Make docs builds pull in
needed updates after the fact.
Signed-off-by: Michael Halstead <mhalstead@linuxfoundation.org>
Default flags for sphinx-build contain -W which turns warnings into
errors. This is really helpful during debugging and could be used for
continuous integration though we do not have such a thing separate from
the continuous delivery in place currently. Nowadays, the docs files
served at docs.yoctoproject.com are actually removed before being
updated from the newly built docs. If the `html` target, a dependency of
`publish` target is failing, the docs aren't copied over to the `final`
temporary directory. Therefore it'll be missing in the rsync upload to
docs.yoctoproject.org.
Instead, let's disable the turning of warnings into errors so that the
`html` make target can finish successfully and the `publish` make target
to finish successfully too.
Cc: Quentin Schulz <foss+yocto@0leil.net>
Signed-off-by: Quentin Schulz <quentin.schulz@theobroma-systems.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
There are some intermittent issues with the script not publishing all
versions. So let's go extreme and fail the script if any error happens:
- a command returns a non-zero code, even if piped,
- a variable is used uninitialized,
This also makes the script print each and every command being run so we
have a better idea where the script struggles.
Cc: Quentin Schulz <foss+yocto@0leil.net>
Signed-off-by: Quentin Schulz <quentin.schulz@theobroma-systems.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
A wrong path was given given the working directory.
Also revert the changes with "git reset --hard" to
have a clean state before further branch switches.
Signed-off-by: Michael Opdenacker <michael.opdenacker@bootlin.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
This is done regularly by the overall AB maintenance tooling, and erasing
build artifacts makes it difficult to find out what went wrong
when it does.
Signed-off-by: Alexander Kanavin <alex.kanavin@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Without this we'd see failures in the perf builds since it would be able to track
history correctly for commit counts.
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
This allows all versions of Bitbake and Yocto Project manuals
to see the manuals for the latest versions.
This also simplifies the release process, not having to update the
releases.rst file for all releases every time a new release is made.
Note that such synchronization is already done for the
switchers.js file (but in a different way). This way, advertised
releases are in sync with switchers.js.
Signed-off-by: Michael Opdenacker <michael.opdenacker@bootlin.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Publish the generic-arm64 binaries in a dedicated meta-arm/ directory so
it is clear this isn't from the core layers.
Signed-off-by: Ross Burton <ross.burton@arm.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
