Commit Graph

308 Commits

Author SHA1 Message Date
Michael Opdenacker
ce75f076e8 run-docs-build: fix checkout of releases.rst from master
A wrong path was given given the working directory.

Also revert the changes with "git reset --hard" to
have a clean state before further branch switches.

Signed-off-by: Michael Opdenacker <michael.opdenacker@bootlin.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2022-01-25 16:54:39 +00:00
Richard Purdie
aa378bfa84 prepare-shared-repos: Fix none published builds after disabling limited depth
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2022-01-24 18:05:23 +00:00
Richard Purdie
1e0b8d9936 prepare-shared-repos: Disable shallow clone speedup as it breaks release builds
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2022-01-21 17:19:47 +00:00
Ross Burton
fa3cf565d6 Add qemuarm64-alt to the build matrix
Signed-off-by: Ross Burton <ross.burton@arm.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2022-01-12 18:25:49 +00:00
Alexander Kanavin
5b2651e502 run-auh: do not clean up build directory
This is done regularly by the overall AB maintenance tooling, and erasing
build artifacts makes it difficult to find out what went wrong
when it does.

Signed-off-by: Alexander Kanavin <alex.kanavin@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2021-12-12 23:00:20 +00:00
Richard Purdie
68b4b7090d prepare-shared-repos: Use full depth for release builds
Without this we'd see failures in the perf builds since it would be able to track
history correctly for commit counts.

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2021-12-12 11:54:37 +00:00
Michael Opdenacker
86d7ed6fe9 scripts/run-docs-build: make all versions list releases known to master
This allows all versions of Bitbake and Yocto Project manuals
to see the manuals for the latest versions.

This also simplifies the release process, not having to update the
releases.rst file for all releases every time a new release is made.

Note that such synchronization is already done for the
switchers.js file (but in a different way). This way, advertised
releases are in sync with switchers.js.

Signed-off-by: Michael Opdenacker <michael.opdenacker@bootlin.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2021-12-08 22:00:05 +00:00
Ross Burton
e13059fe73 publish-artefacts: publish meta-arm/generic-arm64 binaries
Publish the generic-arm64 binaries in a dedicated meta-arm/ directory so
it is clear this isn't from the core layers.

Signed-off-by: Ross Burton <ross.burton@arm.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2021-11-25 15:23:37 +00:00
Michael Opdenacker
13457b72f4 scripts/run-docs-build: add comments
Signed-off-by: Michael Opdenacker <michael.opdenacker@bootlin.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2021-11-24 21:14:21 +00:00
Richard Purdie
e420545f62 utils: Limit fetch optimisation to master for now
The depth code doesn't work if the branch doesn't exist in the main repo
mirror and only the remote.

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2021-11-01 14:15:54 +00:00
Richard Purdie
aff49e938e shared-repos: Use tar instead of rsync for speed
The rysnc of 20,000 files (650MB) onto the nas is slow taking ~3 minutes
at idle and worse at load. This is due to the number of files which
is a pain point for NFS. This piece of the build is also a bottleneck
since the rest of a build depends on it happening.

If we switch to zstd compressed tar, it takes 2.49s. Other compression
methods were much slower but zstd seems 'accptable' and speeds things
up too.

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2021-10-31 17:37:34 +00:00
Richard Purdie
5dc0559cc8 prepare-shared-repo/utils: Limit HEAD clones to shallow depth to save time/space
By not syncing all the history is is possible to save some time/space
in the checkout process since we never use this data. This reduces data
from 650MB to 400MB or with the tarball, 416MB to 55MB.

The logic for the commands needs to be tweaked to handle this and as
written it can't work in non-HEAD revision case but that isn't a commonly
used situation.

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2021-10-31 13:03:22 +00:00
Richard Purdie
8e996a95a8 prepare-shared-repos: Make it clear when rsync starts in logs
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2021-10-30 16:35:29 +01:00
Richard Purdie
298a105758 scripts/prepare-shared-repos: Use tmpfs for speed
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2021-10-30 15:03:38 +01:00
Quentin Schulz
2b05713022 scripts: run-docs-build: patch yocto-3.3 and 3.4 releases
Both releases are missing an important patch that changes the displayed
version from dev to the appropriate release number.

This is confusing to the user and probably breaks some assumptions in
some scripts.

Ideally, the tags should have been moved with those patches applied to
their respective branch but that is not a git best practice so we're
stuck with this "hack" instead.

3.3.x releases aren't impacted as they got the patch applied.

Signed-off-by: Quentin Schulz <quentin.schulz@theobroma-systems.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2021-10-28 23:48:08 +01:00
Quentin Schulz
ed331e7405 scripts: run-docs-build: build latest BB and YP releases
honister and 1.52 Bitbake branch were recently released, so let's build
those too.

Signed-off-by: Quentin Schulz <quentin.schulz@theobroma-systems.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2021-10-28 23:48:08 +01:00
Michael Halstead
772f4b3e17 auh: update the from address to valid domain
The auh.yoctoproject.org domain is no longer used and cannot be
validated.

Signed-off-by: Michael Halstead <mhalstead@linuxfoundation.org>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2021-10-25 19:47:31 +01:00
Alexander Kanavin
9928e6ebe8 scripts/collect-results: publish everything in tmp/log/oeqa/
In addition to the testresult json, testimage class now also
provides the testimage task log and qemu console output log
which can be useful for debugging test failures or
even checking qemu test runs when failures did not happen.

Rather than duplicate specific file/folder names, let's copy all
that is available, and define what is published in the testimage
class itself (with appropriate folder structure if/when needed).
At the moment there's just three files, and they are copied into
folders named after image names, so there's no clutter or risk
of mixing them up with unrelated logs.

[YOCTO #14518]

Signed-off-by: Alexander Kanavin <alex@linutronix.de>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2021-10-14 22:23:15 +01:00
Alexander Kanavin
d544e8ddfb auh-config: adjust to new override syntax
Signed-off-by: Alexander Kanavin <alex.kanavin@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2021-08-16 10:34:38 +01:00
Alexander Kanavin
86e23dff9b auh-config: enable systemd only for glibc
It is not guaranteed to build or work otherwise, and causes
update failures in unrelated recipes when building for musl for instance.

Signed-off-by: Alexander Kanavin <alex.kanavin@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2021-08-16 10:34:38 +01:00
Richard Purdie
6301960da7 scripts/setup-config: Update to use new overrides syntax
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2021-07-30 23:34:30 +01:00
sakib.sajal@windriver.com
0194623371 scripts/generate-testresult-index.py: update index to list host data
Host data is collected on failure by default. List the collected files
in the index.

Signed-off-by: Sakib Sajal <sakib.sajal@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2021-07-12 22:36:20 +01:00
sakib.sajal@windriver.com
681b5a6f92 summarize_top_output.py: summarize data logged on failure
Signed-off-by: Sakib Sajal <sakib.sajal@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2021-07-12 22:36:20 +01:00
sakib.sajal@windriver.com
3ba08a6c45 collect-results: collect all host_stats files
data collection on failure is enabled by default.
Collect all host stats files created.

Signed-off-by: Sakib Sajal <sakib.sajal@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2021-07-12 22:36:20 +01:00
sakib.sajal@windriver.com
bbe9741d01 summarize_top_output.py: add script, use it and publish summary
summarize_top_output.py is used to summarize the top
output that is captured during autobuilder intermittent
failures.

Use the script to summarize the host top output and
publish the summary that is created instead of
the raw logfile.

Signed-off-by: Sakib Sajal <sakib.sajal@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2021-06-24 14:24:54 +01:00
Nicolas Dechesne
a731359179 scripts/run-docs-build: add 'current' link when deploying docs
Make sure that https://docs.yoctoproject.org/current will always link
to the most recent release documentation.

Signed-off-by: Nicolas Dechesne <nicolas.dechesne@linaro.org>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2021-05-25 17:23:04 +01:00
Yi Fan Yu
0d938352d7 scripts/utils.py: Add reporting for yocto-check-layer
The default behavior is to look for a bitbake command,
which fails and produces a confusing output of [-1:].

[YOCTO #14208]

Signed-off-by: Yi Fan Yu <yifan.yu@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2021-04-25 17:02:28 +01:00
sakib.sajal@windriver.com
041ccae3d3 generate-testresult-index.py: publish host data collected by collect-data template
Add a column "Host Data" to publish the host data collected.

Signed-off-by: Sakib Sajal <sakib.sajal@windriver.com>
Signed-off-by: Randy MacLeod <Randy.MacLeod@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2021-04-15 14:54:02 +01:00
sakib.sajal@windriver.com
7593f93869 collect-results: collect "host_stats" files generated by collect-data template
collect-data template generates host_stats files which are collected.
All files are published, file names are determined by the contents:
    - Files that contain "top" output are named host_stats_<number>_top.txt
    - host_stats_<number>.txt otherwise

Signed-off-by: Sakib Sajal <sakib.sajal@windriver.com>
Signed-off-by: Randy MacLeod <Randy.MacLeod@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2021-04-15 14:54:02 +01:00
Michael Halstead
89fd213eb4 run-docs-build: build from tags dynamically instead of static list
Build Sphinx docs for all versions newer than yocto-3.1.5 inclusive.
Integrate suggestions from Quentin Schulz and Nicolas Dechesne.

Signed-off-by: Michael Halstead <mhalstead@linuxfoundation.org>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2021-04-13 15:13:42 +01:00
Michael Halstead
8b8528f565 run-docs-build: add 3.2.3 release to docs build
Signed-off-by: Michael Halstead <mhalstead@linuxfoundation.org>
Reviewed-by: Nicolas Dechesne <nicolas.dechesne@linaro.org>
2021-04-06 11:30:20 -07:00
Nicolas Dechesne
35fcb5fa67 scripts/run-docs-build: build hardknott docs
The branches exist now, let's publish the corresponding docs.

Signed-off-by: Nicolas Dechesne <nicolas.dechesne@linaro.org>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2021-04-06 09:24:56 +01:00
Nicolas Dechesne
f2816b929f scripts/run-docs-build: publish docs for 3.1.6 and 3.2.2
Fixes: #14308

Signed-off-by: Nicolas Dechesne <nicolas.dechesne@linaro.org>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2021-03-25 21:28:08 +00:00
Nicolas Dechesne
1c44c0e0fd scripts/run-docs-build: add for loop when publishing tags/branches
Use for loop to avoid repeating the same pattern over and over.

Signed-off-by: Nicolas Dechesne <nicolas.dechesne@linaro.org>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2021-03-25 21:28:08 +00:00
Michael Halstead
f8c2faaf20 send-qa-email: Save the QA email in case it doesn't reach the mailing lists.
Signed-off-by: Michael Halstead <mhalstead@linuxfoundation.org>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2021-02-23 10:24:01 +00:00
Richard Purdie
c21732937c scripts/run-config: Disable output buffering
Line buffering (bufsize=1) is unavailable with binary mode so use unbuffered
mode instead. This fixes python runtime warnings.

[YOCTO #14093]

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2021-01-31 11:21:47 +00:00
Richard Purdie
241ae71c18 scripts/run-docs-build: Generate 3.2 docs as well
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2021-01-22 22:31:55 +00:00
Richard Purdie
9a34cc8120 scripts/run-docs-build: Generate 3.1.5 and 3.2.1 docs (at least for now)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2021-01-22 22:24:00 +00:00
Nicolas Dechesne
698743bdd6 run-docs-build: delete files on the server when publishing
run-docs-build rebuilds the entire doc website, so when we rsync the
output, we need to delete files no longer needed, this is especially
important when we move/rename pages, to avoid stale content.

Using rsync with "wildcards" was a problem in case a file/folder is
removed in the output dir, it won't be deleted.

Signed-off-by: Nicolas Dechesne <nicolas.dechesne@linaro.org>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2021-01-22 22:09:26 +00:00
Richard Purdie
56db25b673 run-docs-build: Enable sphinx docs for dunfell
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2021-01-04 10:57:59 +00:00
Richard Purdie
bceb63fb79 config.json/run-config: Add support for shortnames and descriptions
Learn from the previous experiments and add meaninful shortnames and
descriptions to work around the 50 char name limit.

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2020-11-20 23:32:26 +00:00
Richard Purdie
6e2825564c scripts/shared-repo-unpack: Add flush call to update the output more regularly before buildtools
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2020-11-20 23:32:26 +00:00
Richard Purdie
2bb4804243 scripts/run-config: Remove redundant boilerplate json
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2020-11-20 23:32:26 +00:00
Richard Purdie
ce63e8f024 config.json/run-config: Add human readable descriptions of steps
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2020-11-20 23:32:26 +00:00
Richard Purdie
b1bc4d64c2 scripts/run-config: Ensure logging to both logfile and stdout
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2020-11-20 23:32:26 +00:00
Richard Purdie
4a4c888f66 scripts/run-config: Improve logfile naming
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2020-11-20 23:32:26 +00:00
Richard Purdie
23d65680f8 run-config: Adapt to two pass execution
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2020-11-20 23:32:26 +00:00
Richard Purdie
e183db413f scripts/run-config: If target is present default to 1 step
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2020-11-20 23:32:20 +00:00
Richard Purdie
afb6c5a887 scripts/run-config: Ensure stepnum has a value when there are no steps
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2020-11-20 23:24:22 +00:00
Richard Purdie
d9c77dd000 scripts: Add target present check script
Add a script which from the exit code determines if a build target is present
in the configuration. The autobuilder can use this to determine whether to
execute the target at all or whether to skip it.

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2020-11-20 17:28:51 +00:00