Commit Graph

393 Commits

Author SHA1 Message Date
Quentin Schulz
ed331e7405 scripts: run-docs-build: build latest BB and YP releases
honister and 1.52 Bitbake branch were recently released, so let's build
those too.

Signed-off-by: Quentin Schulz <quentin.schulz@theobroma-systems.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2021-10-28 23:48:08 +01:00
Michael Halstead
772f4b3e17 auh: update the from address to valid domain
The auh.yoctoproject.org domain is no longer used and cannot be
validated.

Signed-off-by: Michael Halstead <mhalstead@linuxfoundation.org>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2021-10-25 19:47:31 +01:00
Alexander Kanavin
9928e6ebe8 scripts/collect-results: publish everything in tmp/log/oeqa/
In addition to the testresult json, testimage class now also
provides the testimage task log and qemu console output log
which can be useful for debugging test failures or
even checking qemu test runs when failures did not happen.

Rather than duplicate specific file/folder names, let's copy all
that is available, and define what is published in the testimage
class itself (with appropriate folder structure if/when needed).
At the moment there's just three files, and they are copied into
folders named after image names, so there's no clutter or risk
of mixing them up with unrelated logs.

[YOCTO #14518]

Signed-off-by: Alexander Kanavin <alex@linutronix.de>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2021-10-14 22:23:15 +01:00
Alexander Kanavin
d544e8ddfb auh-config: adjust to new override syntax
Signed-off-by: Alexander Kanavin <alex.kanavin@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2021-08-16 10:34:38 +01:00
Alexander Kanavin
86e23dff9b auh-config: enable systemd only for glibc
It is not guaranteed to build or work otherwise, and causes
update failures in unrelated recipes when building for musl for instance.

Signed-off-by: Alexander Kanavin <alex.kanavin@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2021-08-16 10:34:38 +01:00
Richard Purdie
6301960da7 scripts/setup-config: Update to use new overrides syntax
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2021-07-30 23:34:30 +01:00
sakib.sajal@windriver.com
0194623371 scripts/generate-testresult-index.py: update index to list host data
Host data is collected on failure by default. List the collected files
in the index.

Signed-off-by: Sakib Sajal <sakib.sajal@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2021-07-12 22:36:20 +01:00
sakib.sajal@windriver.com
681b5a6f92 summarize_top_output.py: summarize data logged on failure
Signed-off-by: Sakib Sajal <sakib.sajal@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2021-07-12 22:36:20 +01:00
sakib.sajal@windriver.com
3ba08a6c45 collect-results: collect all host_stats files
data collection on failure is enabled by default.
Collect all host stats files created.

Signed-off-by: Sakib Sajal <sakib.sajal@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2021-07-12 22:36:20 +01:00
sakib.sajal@windriver.com
bbe9741d01 summarize_top_output.py: add script, use it and publish summary
summarize_top_output.py is used to summarize the top
output that is captured during autobuilder intermittent
failures.

Use the script to summarize the host top output and
publish the summary that is created instead of
the raw logfile.

Signed-off-by: Sakib Sajal <sakib.sajal@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2021-06-24 14:24:54 +01:00
Nicolas Dechesne
a731359179 scripts/run-docs-build: add 'current' link when deploying docs
Make sure that https://docs.yoctoproject.org/current will always link
to the most recent release documentation.

Signed-off-by: Nicolas Dechesne <nicolas.dechesne@linaro.org>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2021-05-25 17:23:04 +01:00
Yi Fan Yu
0d938352d7 scripts/utils.py: Add reporting for yocto-check-layer
The default behavior is to look for a bitbake command,
which fails and produces a confusing output of [-1:].

[YOCTO #14208]

Signed-off-by: Yi Fan Yu <yifan.yu@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2021-04-25 17:02:28 +01:00
sakib.sajal@windriver.com
041ccae3d3 generate-testresult-index.py: publish host data collected by collect-data template
Add a column "Host Data" to publish the host data collected.

Signed-off-by: Sakib Sajal <sakib.sajal@windriver.com>
Signed-off-by: Randy MacLeod <Randy.MacLeod@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2021-04-15 14:54:02 +01:00
sakib.sajal@windriver.com
7593f93869 collect-results: collect "host_stats" files generated by collect-data template
collect-data template generates host_stats files which are collected.
All files are published, file names are determined by the contents:
    - Files that contain "top" output are named host_stats_<number>_top.txt
    - host_stats_<number>.txt otherwise

Signed-off-by: Sakib Sajal <sakib.sajal@windriver.com>
Signed-off-by: Randy MacLeod <Randy.MacLeod@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2021-04-15 14:54:02 +01:00
Michael Halstead
89fd213eb4 run-docs-build: build from tags dynamically instead of static list
Build Sphinx docs for all versions newer than yocto-3.1.5 inclusive.
Integrate suggestions from Quentin Schulz and Nicolas Dechesne.

Signed-off-by: Michael Halstead <mhalstead@linuxfoundation.org>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2021-04-13 15:13:42 +01:00
Michael Halstead
8b8528f565 run-docs-build: add 3.2.3 release to docs build
Signed-off-by: Michael Halstead <mhalstead@linuxfoundation.org>
Reviewed-by: Nicolas Dechesne <nicolas.dechesne@linaro.org>
2021-04-06 11:30:20 -07:00
Nicolas Dechesne
35fcb5fa67 scripts/run-docs-build: build hardknott docs
The branches exist now, let's publish the corresponding docs.

Signed-off-by: Nicolas Dechesne <nicolas.dechesne@linaro.org>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2021-04-06 09:24:56 +01:00
Nicolas Dechesne
f2816b929f scripts/run-docs-build: publish docs for 3.1.6 and 3.2.2
Fixes: #14308

Signed-off-by: Nicolas Dechesne <nicolas.dechesne@linaro.org>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2021-03-25 21:28:08 +00:00
Nicolas Dechesne
1c44c0e0fd scripts/run-docs-build: add for loop when publishing tags/branches
Use for loop to avoid repeating the same pattern over and over.

Signed-off-by: Nicolas Dechesne <nicolas.dechesne@linaro.org>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2021-03-25 21:28:08 +00:00
Michael Halstead
f8c2faaf20 send-qa-email: Save the QA email in case it doesn't reach the mailing lists.
Signed-off-by: Michael Halstead <mhalstead@linuxfoundation.org>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2021-02-23 10:24:01 +00:00
Richard Purdie
c21732937c scripts/run-config: Disable output buffering
Line buffering (bufsize=1) is unavailable with binary mode so use unbuffered
mode instead. This fixes python runtime warnings.

[YOCTO #14093]

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2021-01-31 11:21:47 +00:00
Richard Purdie
241ae71c18 scripts/run-docs-build: Generate 3.2 docs as well
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2021-01-22 22:31:55 +00:00
Richard Purdie
9a34cc8120 scripts/run-docs-build: Generate 3.1.5 and 3.2.1 docs (at least for now)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2021-01-22 22:24:00 +00:00
Nicolas Dechesne
698743bdd6 run-docs-build: delete files on the server when publishing
run-docs-build rebuilds the entire doc website, so when we rsync the
output, we need to delete files no longer needed, this is especially
important when we move/rename pages, to avoid stale content.

Using rsync with "wildcards" was a problem in case a file/folder is
removed in the output dir, it won't be deleted.

Signed-off-by: Nicolas Dechesne <nicolas.dechesne@linaro.org>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2021-01-22 22:09:26 +00:00
Richard Purdie
56db25b673 run-docs-build: Enable sphinx docs for dunfell
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2021-01-04 10:57:59 +00:00
Richard Purdie
bceb63fb79 config.json/run-config: Add support for shortnames and descriptions
Learn from the previous experiments and add meaninful shortnames and
descriptions to work around the 50 char name limit.

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2020-11-20 23:32:26 +00:00
Richard Purdie
6e2825564c scripts/shared-repo-unpack: Add flush call to update the output more regularly before buildtools
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2020-11-20 23:32:26 +00:00
Richard Purdie
2bb4804243 scripts/run-config: Remove redundant boilerplate json
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2020-11-20 23:32:26 +00:00
Richard Purdie
ce63e8f024 config.json/run-config: Add human readable descriptions of steps
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2020-11-20 23:32:26 +00:00
Richard Purdie
b1bc4d64c2 scripts/run-config: Ensure logging to both logfile and stdout
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2020-11-20 23:32:26 +00:00
Richard Purdie
4a4c888f66 scripts/run-config: Improve logfile naming
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2020-11-20 23:32:26 +00:00
Richard Purdie
23d65680f8 run-config: Adapt to two pass execution
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2020-11-20 23:32:26 +00:00
Richard Purdie
e183db413f scripts/run-config: If target is present default to 1 step
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2020-11-20 23:32:20 +00:00
Richard Purdie
afb6c5a887 scripts/run-config: Ensure stepnum has a value when there are no steps
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2020-11-20 23:24:22 +00:00
Richard Purdie
d9c77dd000 scripts: Add target present check script
Add a script which from the exit code determines if a build target is present
in the configuration. The autobuilder can use this to determine whether to
execute the target at all or whether to skip it.

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2020-11-20 17:28:51 +00:00
Richard Purdie
290e1bc2ee scripts/run-config: Don't execute steps that don't exist!
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2020-11-20 17:27:15 +00:00
Alexander Kanavin
2c3f66ba37 auh-config: add non-default distro features
This adds systemd and pam related recipes to upstream checks and devtool-driven updates.

Signed-off-by: Alexander Kanavin <alex.kanavin@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2020-11-17 21:56:47 +00:00
Richard Purdie
354b43fece scripts/run-docs-build: Further improve docs rsync flags
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2020-11-09 17:18:35 +00:00
Richard Purdie
b8c8ddac4a scripts/run-docs-build: Drop time rsync flag as we don't need that
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2020-11-09 16:55:42 +00:00
Richard Purdie
7958928739 scripts/run-docs-build: Improve docs rsync flags
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2020-11-09 16:26:14 +00:00
Richard Purdie
078d72fe0e scripts/run-docs-build: Fix typo in previous commit
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2020-11-09 16:03:12 +00:00
Richard Purdie
14d98490e9 scripts/run-docs-build: make target change needs different artefact copying
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2020-11-09 15:03:42 +00:00
Richard Purdie
c5860c6ef2 scripts/run-docs-build: Enable docs builds, remove dry run option to rsync
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2020-11-09 10:51:48 +00:00
Richard Purdie
2f423c5a07 scripts/run-docs-build: Use an archive copy of the docbook archives for efficiency
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2020-11-09 10:51:02 +00:00
Richard Purdie
761c27ad68 scripts/run-docs-build: Use make publissh target instead of html
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2020-11-09 10:48:01 +00:00
Richard Purdie
19b288977b scripts/run-docs-build: Ensure potentially stale _build is cleaned before build
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2020-11-06 15:05:26 +00:00
Richard Purdie
fba3c36389 scripts/run-docs-build: Publish gatesgarth to the correct location
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2020-11-06 15:04:23 +00:00
Richard Purdie
5d64a5d569 scripts/run-docs-build: Ensure switchers update affects gatesgarth too
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2020-11-06 15:04:03 +00:00
Richard Purdie
8288b27ac2 scripts/run-docs-build: Fix rsync paths and drop continue mode for wget
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2020-11-06 14:28:56 +00:00
Richard Purdie
f4a7a7a644 scripts/run-docs-build: First pass at building the various documentation branches
The script is suboptimal in many ways but is a start and gives us something to test
and improve upon.

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2020-11-06 14:08:08 +00:00
Richard Purdie
590b045b34 scripts: Add run-docs-build as a wrapper around documentation building
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2020-11-05 12:59:27 +00:00
Richard Purdie
4b8eab92ee scripts/generate-testresult-index.py: Use bulma css to improve the look of the index
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2020-10-21 21:42:32 +01:00
Richard Purdie
ab485e89bd scripts/generate-testresult-index: Reorder buildhistory to improve display
Also, use the artefacts directory in the main link, not direct to testresults.

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2020-10-21 21:08:25 +01:00
Richard Purdie
9f50544d00 scripts/generate-testresult-index: Improve index to list test reports, ptest and buildperf separately
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2020-10-21 21:00:20 +01:00
Richard Purdie
098de71cbd scripts/generate-testresult-index.py: Ensure we're not always rerunning resulttool
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2020-10-21 19:30:41 +01:00
Richard Purdie
83c2052781 scripts/generate-testresult-index: Ensure backwards compatibility with older layout
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2020-10-21 19:23:49 +01:00
Richard Purdie
6d6c45cbda scripts/generate-testresult-index: Update after 'posttrigger' renaming broke the index generation
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2020-10-21 19:17:58 +01:00
Richard Purdie
3634672674 scripts/send-qa-email: Fix buildtools handling
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2020-09-24 12:27:58 +01:00
Richard Purdie
eb295f62f1 utils: Teach buildtools script handling about unset
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2020-09-23 08:53:19 +01:00
Alexander Kanavin
0f8534c747 auh: correct the SMTP server in config file
Michael Halstead has confirmed this.

Signed-off-by: Alexander Kanavin <alex.kanavin@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2020-09-17 12:31:52 +01:00
Richard Purdie
e47a9a26f5 scripts/send-qa-email,utils: Use buildtools if present
resulttool has python version requirements so use buildtools if present.

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2020-09-17 07:44:46 +01:00
Richard Purdie
838be1a00c scripts: Add runqemu-renice.c for renicing runqemu
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2020-09-05 14:59:14 +01:00
Michael Halstead
01c16fa772 scripts/send-qa-email: Update to dedicated push capable domain
Signed-off-by: Michael Halstead <mhalstead@linuxfoundation.org>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2020-09-01 18:14:31 +01:00
Richard Purdie
a677305551 scripts/utils: Ensuring new logging is flushed from buffers
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2020-08-24 18:31:10 +01:00
Richard Purdie
f3892fac1e scripts/utils: Add more logging for fetchgitrepo
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2020-08-24 18:27:56 +01:00
Quentin Schulz
7571e44fc1 scripts: run-auh: update upgrade-helper name
Since https://lists.yoctoproject.org/g/yocto/message/50282 was merged in
autoupgrade-helper git repo, the python script isn't called
upgradehelper.py anymore but upgrade-helper.py for consistency sake.

Let's update the run-auh script so that it's still working.

Signed-off-by: Quentin Schulz <foss@0leil.net>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2020-08-17 11:38:32 +01:00
Richard Purdie
4c9358ac78 scripts/utils: Ensure buildtools tarball hash matches any supplied value
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2020-07-14 22:42:25 +01:00
Alexander Kanavin
d88728431a auh-config: send AUH update emails to oe-core list as well
Sadly, many maintainers do not act on AUH mails in a timely manner
(or do not act at all). Sending these emails to oe-core list as well
will allow other people to pick them up, and ease up the path
to doing recipe version maintenance.

Also, recently the amount of outdate recipes has been greatly reduced,
so there should not be too many of those emails each month.

Signed-off-by: Alexander Kanavin <alex.kanavin@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2020-06-30 17:33:41 +01:00
Vineela
3aefa5a763 scripts/utils.py: Convert md5sum to sha256sum for layer tarballs
Signed-off-by: Vineela Tummalapalli <vineela.tummalapalli@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2020-06-26 21:33:42 +01:00
Richard Purdie
eb86b7fd11 send-qa-email: Fix stable branch test result history issues
On stable release branch builds, it will clone "master" since comparebranch
is None. It will then fail to push since there is already a dunfell
branch upstream with changes on which it needs to rebase on top of.
The change to use shallow clones significantly increases this problem.

To fix this, fall back to cloning basebranch by name, then master
if it doesn't exist.

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2020-06-09 16:51:23 +01:00
Tummalapalli, Vineela
835ac325a4 publish-artefacts: Improve sha256 checksum handling
The existing checksum files were encoding full paths. This patch changes
this to use just the filename which allows relocation of the directories
and is more appropriate for sharing over http.

Signed-off-by: Vineela Tummalapalli <vineela.tummalapalli@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2020-05-27 07:55:06 +01:00
Alexander Kanavin
cfa62a586a scripts: add a pair of scripts to set up and run Auto Upgrade Helper
This allows automating its setup and execution on all autobuilder worker machines;
previously there was a static setup on a dedicated machine, which wasn't
great from maintenance perspective.

To use:

scripts/setup-auh target_dir
scripts/run-auh target_dir

(run-auh can be run several times in a directory that
was previously set up)

Signed-off-by: Alexander Kanavin <alex.kanavin@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2020-05-17 16:57:25 +01:00
Richard Purdie
80c53d03ea scripts/publish-artefacts: Convert md5sum to sha256sum
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2020-05-05 13:29:01 +01:00
Richard Purdie
033ae6ffbd scripts/run-config: Share buildtools setup code with shared-repo-unpack
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2020-04-27 19:05:11 +01:00
Richard Purdie
0a7bf5191e scripts/shared-repo-unpack: Accept workername parameter
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2020-04-27 17:09:03 +01:00
Richard Purdie
aa81bd61c5 scripts/run-config: Add extra debug about buildtools to show when active
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2020-04-27 17:07:28 +01:00
Steve Sakoman
00df7cbc02 scripts/send-qa-email: fix bug in git push logic for yocto-testresults
We were mistakenly doing a force push if the branch was in either
BUILD_HISTORY_FORKPUSH or BUILD_HISTORY_DIRECTPUSH.

Now we force push for branches in BUILD_HISTORY_FORKPUSH, regular push
for branches in BUILD_HISTORY_DIRECTPUSH, and no push if the branch is
in neither list.

Signed-off-by: Steve Sakoman <steve@sakoman.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2020-04-23 14:43:36 +01:00
Steve Sakoman
e79f6ddafd scripts/utils.py: Fix confusing naming of getcomparisonbranch() return value
This code was copy/pasted but in this case the variables should
be called "compare", not base.

Signed-off-by: Steve Sakoman <steve@sakoman.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2020-04-23 14:42:38 +01:00
Richard Purdie
a1ea9f6e60 scripts/run-config: Resolve path early for buildtools
The buildtools path contains "../" which may not resolve correctly by the
shell if the final directory doesn't exist. Avoid this by using abspath()
on the expression straightaway.

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2020-04-07 08:33:51 +01:00
Richard Purdie
a9f4f21ee9 run-config: Fix lockfile loop handling
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2020-03-12 18:13:42 +00:00
Richard Purdie
531cdb168e scripts/run-config: Fix BB_LOGFILE missing /
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2020-03-12 12:44:49 +00:00
Richard Purdie
c41c1d0c74 scripts/run-config: Ensure racing for the lockfile doesn't traceback
Traceback (most recent call last):
  File "yocto-autobuilder-helper/scripts/run-config", line 116, in <module>
    fcntl.flock(fileno, fcntl.LOCK_EX)
OSError: [Errno 22] Invalid argument

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2020-03-12 12:41:32 +00:00
Richard Purdie
1622cb7b09 scripts/run-config: Add support for local filesystem buildtools urls
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2020-03-12 11:58:13 +00:00
Richard Purdie
0902ededff scripts/run-config: Handle quoting issues in buildtools environment files
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2020-03-12 11:58:05 +00:00
Richard Purdie
8e7a052f07 scripts/run-config: Handle multiple builds racing with lockfile for buildtools
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2020-03-12 11:57:55 +00:00
Richard Purdie
37955b4b8d send-qa-email: Limit results clone to shallow depth to avoid timeouts
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2020-03-10 22:22:36 +00:00
Richard Purdie
91bd587bf3 scripts/run-config: Set BB_LOGCONFIG on autobuilder
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2020-03-09 22:43:49 +00:00
Richard Purdie
d4a9248fd0 scripts/run-config: Fix buildtools path
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2020-03-07 10:02:20 +00:00
Richard Purdie
fbdaa169c3 run-config: Add support for wrapping builds in a buildtools tarball
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2020-03-06 15:51:42 +00:00
Richard Purdie
17386f43c2 run-config: Add workername parameter
Allow the workername to be passed into the script. This is to allow future
buildtools tarball support based upon the worker.

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2020-03-06 14:37:38 +00:00
Richard Purdie
ee6fc35569 build-perf/send-qa-email: Fallback to master as a comparision branch
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2020-03-01 17:44:34 +00:00
Richard Purdie
89633facc4 run-config: Preface log output with auto.conf settings to aid debugging
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2020-02-19 16:47:09 +00:00
Richard Purdie
1b78039d41 scripts/send-qa-email: Allow force pushing of basebranch based comparisions
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2020-02-08 15:25:42 +00:00
Richard Purdie
5f423e3a26 scripts/send-qa-email: Handle older git clients
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2020-02-08 14:32:10 +00:00
akuster
32218185dd yocto-autobuilder-helper: remove mpc8315 from configs
Signed-off-by: Armin Kuster <akuster808@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2020-02-02 10:38:30 +00:00
Richard Purdie
8bae4b2955 run-config/upload-error-reports: Handle builddir not existing
This fixes failures on the buildperf workers which do early cleanup of build
directories.

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-12-12 13:02:57 +00:00
Richard Purdie
79dd2f94aa send-qa-email/utils: Improve new branch regression script handling
There are several issues:

* New branches don't currently have git regression history
* The regression tool errors if there isn't anything to compare against

To fix this, create a branch with history and only generate a regression
report if there are commits to compare against.

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-11-16 16:25:43 +00:00
Richard Purdie
014a1c4b89 scripts/publish-artefact: Ensure ptest images are published
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-10-31 11:14:24 +00:00
Richard Purdie
aa5f3f7b39 build-perf-test-wrapper: Cannot clean up helper at this point
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-10-04 08:39:56 +01:00
Richard Purdie
488882ac9a build-perf-test-wrapper: Ensure worker checkout and sstate is cleaned up
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-10-03 16:49:56 +01:00
Richard Purdie
f54b2f5104 publish-artefacts: Update to use -alt naming for release output
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-10-02 10:44:51 +01:00
Richard Purdie
28ee9be07d config.py: Update -lsb -> -alt naming
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-10-02 10:44:00 +01:00
Richard Purdie
81b3587a90 run-config: Fix selftest tag filtering command
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-09-07 16:50:32 +01:00
Richard Purdie
68c19d067d config.json/run-config: Add configuration for toolchain testsuites
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-09-06 17:55:24 +01:00
Richard Purdie
718fe955c2 prepare-shared-repos: Really fix shell globbing
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-09-06 15:57:56 +01:00
Richard Purdie
e99f772fc4 prepare-shared-repos: UAllow shell globing to fix failure
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-09-06 15:45:19 +01:00
Richard Purdie
554a2114ef prepare-shared-repos: Use temporary non-NFS directory for layer clones for speed
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-09-06 15:37:43 +01:00
Richard Purdie
21ebf7df51 scripts/run-config: Add --build-type commandline option
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-09-06 14:24:04 +01:00
Thomas Goodwin
417c7d33db layer-config, shared-repo-unpack: Sub-repos in NEEDREPOS
The previous fixes requires the user to set "no-layer-add"
for a repo and then use ADDLAYER to insert the sub-repos
(e.g., meta-openmbedded/meta-oe) as a two-part process.
This means that you would also have to specify that flag
if a repo that is a layer with dependencies is in the
list so that it can be inserted in the correct order later
via ADDLAYER to avoid parsing problems.  This fix allows
for specifying a NEEDREPOS with the subdirectory of the
target layer (e.g., meta-openembedded/meta-oe) so that
there is no need for the "no-layer-add" followed by
ADDLAYER combination.  The entire meta-openembedded
repo would be moved into place, and the sublayer added
to bblayers.conf.

Signed-off-by: Thomas Goodwin <btgoodwin@geontech.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-08-02 17:34:48 +01:00
Thomas Goodwin
2aeb77f3f2 shared-repo-unpack: suppressing exception, returning exit code
The preceeding errors that would be shown should be enough to
identify the problem since it would have been a problem
running bitbake-layers add-layer, indicating a misconfiguration
of config.json.

Signed-off-by: Thomas Goodwin <btgoodwin@geontech.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-08-02 17:34:48 +01:00
Thomas Goodwin
43c2a3acc3 layer-config: fixing silent failures from always exiting '0'
The return value from bitbakecmd was not being returned when
errors occurred which allowed shared-repo-unpack to succeed
despite the failure.  This fix changes to check_call and a
try-catch when attempting to add repos that fail for whatever
reason during add-layer, like a missing conf/layer.conf at
the top level or a previously-added layer breaks parsing
because of missing dependencies.

Signed-off-by: Thomas Goodwin <btgoodwin@geontech.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-08-02 17:34:48 +01:00
Richard Purdie
7689893bb3 generate-testresult-index: Handle build performance without reports
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-06-05 16:55:12 +01:00
Richard Purdie
1b3e3ced9f scripts: Add script to generate autobuilder results index
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-06-05 16:43:24 +01:00
Richard Purdie
becdbd3729 utils: Fix incorrect module usage
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-04-12 18:21:51 +01:00
Richard Purdie
68f351ed83 run-config: Fix uninitalized variable problem
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-04-12 18:17:16 +01:00
Richard Purdie
f9bdcf888f config.json/utils: Allow buildhistory to be configurable and disable for oe-selftest in qemu-arch
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-04-12 16:23:54 +01:00
Richard Purdie
a93a79945f scripts/collect-results: Ensure selftest doesn't trigger buildhistory results collection
Affects sumo since that doesn't have oe-selftest parallelism. thud/master create
these in a different location so don't see the issue.

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-03-25 10:07:21 +00:00
Richard Purdie
372282eb6b scripts/publish-artefacts: Ensure aarch64 uninative is published
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-03-19 15:24:33 +00:00
Richard Purdie
520ea457e8 send-qa-email: Fix typo causing wrong report contents to be written
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-02-25 09:34:22 +00:00
Richard Purdie
2deb018441 send-qa-email: Be specific about which test result branches to push
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-02-23 22:53:50 +00:00
Richard Purdie
f19aa61cd6 buildhistory-init/collect-results: Use a tag to make buildhistory data consistent
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-02-23 17:23:52 +00:00
Richard Purdie
a0963045de scripts/utils: Simplify buildhistory path, no need for such deep paths
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-02-22 00:07:37 +00:00
Richard Purdie
e67a6aab27 send-qa-report: Add handling of results files with resulttool
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-02-21 17:19:02 +00:00
Richard Purdie
13eb8eb282 scripts/send-qa-email: Add optional -R option for test results directory
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-02-21 17:13:16 +00:00
Richard Purdie
5130ab9f21 utils: Move getcomparisonbranch() to common function library
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-02-21 17:13:16 +00:00
Richard Purdie
e1564ff86c collect-results: Generate buildhistory-diff output
Since we now have functional buildhistory repository functionality,
generate buildhistory-diff output for builds where its enabled.

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-02-21 13:50:26 +00:00
Richard Purdie
e5310650b0 scripts/utils: Ensure list filtering code doesn't match appends/prepends
We want all INHERIT += to apply for example

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-02-12 14:18:09 +00:00
Richard Purdie
700f297cd5 build-perf-test-wrapper: Print URL data is published to in output
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-02-12 12:47:35 +00:00
Richard Purdie
5a2eddd6bd setup-config/utils: Improve extravars handling
Currently there was no way to override an existing extra vars
assignment, which meant BB_NUMBER_THREADS was impossible to change.

This adds code to allow the higher override of a value to override
the other settings which makes the code function more as a user
would expect.

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-02-11 11:05:49 +00:00
Richard Purdie
28ad5cdcc9 build-perf-test-wrapper: Place auto.conf into the correct location
Currently it was being placed a directory level too high and hence
was being ignored.

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-02-11 11:03:52 +00:00
Richard Purdie
e1b505f006 build-perf-test-wrapper: Fix branch comparision code
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-02-10 08:50:26 +00:00
Richard Purdie
3dfa90c4ba build-perf-test-wrapper: Fix to use full commit hashes and short repo names
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-02-10 08:18:00 +00:00
Richard Purdie
5e62bc3f9c build-perf-test-wrapper: Fix missing module reference
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-02-09 21:18:03 +00:00
Richard Purdie
72b22dc785 config.json/run-config/build-perf-test-wrapper: Allow build performance branch comparisions
This ensures that release branch performance build tests work on the correct
branch and that comparisions between mut+master and master-next+master
start to work correctly. It also ensures reports for the current
revision are generated even if later revisions are built.

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-02-09 17:24:16 +00:00
Richard Purdie
07d4e99b6d build-perf-test-wrapper: Fix missing variable
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-02-08 23:14:57 +00:00
Richard Purdie
9a62bbf6b4 build-perf-test-wrapper: Add HTML link to published text report
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-02-08 14:02:55 +00:00
Richard Purdie
6fbc1321a0 config.json/run-config: Ensure build-perf publish-dir contains the target name
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-02-05 22:22:46 +00:00
Richard Purdie
d4f649024d build-perf-test-wrapper: Fix report path names
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-02-04 20:40:20 +00:00
Richard Purdie
1d8d6b70fa build-perf-test-wrapper: Further variable cleanup
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-02-04 16:12:50 +00:00
Richard Purdie
c865dbb73e build-perf-test-wrapper: Clean up path option handling and directory creation
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-02-04 15:25:21 +00:00
Richard Purdie
ca730319a9 build-perf-test-wrapper: Handle the case where /bin/sh is dash
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-02-04 15:10:31 +00:00
Richard Purdie
72ee261692 build-perf-test-wrapper: Fix path joining issues
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-02-03 22:13:18 +00:00
Richard Purdie
852ebb4889 build-perf-test-wrapper: Fix report path names to account for build directory
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-02-03 15:38:22 +00:00
Richard Purdie
4fda3f2816 build-perf-test-wrapper: Ensure scripts within OE-Core are available within PATH
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-02-03 11:29:07 +00:00
Richard Purdie
e0770f4b55 scripts: Convert build-perf-test-wrapper from shell to python
The shell script can't access the config.json data easily, convert
from shell to python so that we prepare the way for further enhancements.

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-02-02 11:42:52 +00:00
Richard Purdie
efeb042a77 config.json/build-perf-test-wrapper.sh: Add support for publishing to the standard results location
This makes the build performance code behave similarly to the other
results publishing code using the standard location.

Also tweak the report names so they sort.

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-02-01 15:18:57 +00:00
Richard Purdie
b480d77271 scripts/run-config: Allow command failures to be non-fatal
We need the final logging to show which command steps contain errors so print
failing commands but allow the code to continue and show the final log output.

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-01-31 12:57:54 +00:00
Richard Purdie
81382c6c5d scripts/run-config: Always run bitbake with -k option
This ensures the builds get as far as they can when there is failure rather
that stopping at the first sign of one. This means caches are hotter and
allows for faster respins, at the cost of the failure being lost in longer
logs.

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-01-31 12:53:00 +00:00
Richard Purdie
69b8ac6c27 build-perf-test-wrapper: Ensure download_dir exists
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-01-22 23:09:26 +00:00
Richard Purdie
6e078513b7 build-perf-test-wrapper: Remove code we don't need
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-01-22 23:02:13 +00:00