Commit Graph

550 Commits

Author SHA1 Message Date
Richard Purdie
5f423e3a26 scripts/send-qa-email: Handle older git clients
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2020-02-08 14:32:10 +00:00
akuster
32218185dd yocto-autobuilder-helper: remove mpc8315 from configs
Signed-off-by: Armin Kuster <akuster808@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2020-02-02 10:38:30 +00:00
Richard Purdie
8bae4b2955 run-config/upload-error-reports: Handle builddir not existing
This fixes failures on the buildperf workers which do early cleanup of build
directories.

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-12-12 13:02:57 +00:00
Richard Purdie
79dd2f94aa send-qa-email/utils: Improve new branch regression script handling
There are several issues:

* New branches don't currently have git regression history
* The regression tool errors if there isn't anything to compare against

To fix this, create a branch with history and only generate a regression
report if there are commits to compare against.

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-11-16 16:25:43 +00:00
Richard Purdie
014a1c4b89 scripts/publish-artefact: Ensure ptest images are published
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-10-31 11:14:24 +00:00
Richard Purdie
aa5f3f7b39 build-perf-test-wrapper: Cannot clean up helper at this point
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-10-04 08:39:56 +01:00
Richard Purdie
488882ac9a build-perf-test-wrapper: Ensure worker checkout and sstate is cleaned up
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-10-03 16:49:56 +01:00
Richard Purdie
f54b2f5104 publish-artefacts: Update to use -alt naming for release output
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-10-02 10:44:51 +01:00
Richard Purdie
28ee9be07d config.py: Update -lsb -> -alt naming
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-10-02 10:44:00 +01:00
Richard Purdie
81b3587a90 run-config: Fix selftest tag filtering command
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-09-07 16:50:32 +01:00
Richard Purdie
68c19d067d config.json/run-config: Add configuration for toolchain testsuites
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-09-06 17:55:24 +01:00
Richard Purdie
718fe955c2 prepare-shared-repos: Really fix shell globbing
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-09-06 15:57:56 +01:00
Richard Purdie
e99f772fc4 prepare-shared-repos: UAllow shell globing to fix failure
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-09-06 15:45:19 +01:00
Richard Purdie
554a2114ef prepare-shared-repos: Use temporary non-NFS directory for layer clones for speed
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-09-06 15:37:43 +01:00
Richard Purdie
21ebf7df51 scripts/run-config: Add --build-type commandline option
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-09-06 14:24:04 +01:00
Thomas Goodwin
417c7d33db layer-config, shared-repo-unpack: Sub-repos in NEEDREPOS
The previous fixes requires the user to set "no-layer-add"
for a repo and then use ADDLAYER to insert the sub-repos
(e.g., meta-openmbedded/meta-oe) as a two-part process.
This means that you would also have to specify that flag
if a repo that is a layer with dependencies is in the
list so that it can be inserted in the correct order later
via ADDLAYER to avoid parsing problems.  This fix allows
for specifying a NEEDREPOS with the subdirectory of the
target layer (e.g., meta-openembedded/meta-oe) so that
there is no need for the "no-layer-add" followed by
ADDLAYER combination.  The entire meta-openembedded
repo would be moved into place, and the sublayer added
to bblayers.conf.

Signed-off-by: Thomas Goodwin <btgoodwin@geontech.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-08-02 17:34:48 +01:00
Thomas Goodwin
2aeb77f3f2 shared-repo-unpack: suppressing exception, returning exit code
The preceeding errors that would be shown should be enough to
identify the problem since it would have been a problem
running bitbake-layers add-layer, indicating a misconfiguration
of config.json.

Signed-off-by: Thomas Goodwin <btgoodwin@geontech.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-08-02 17:34:48 +01:00
Thomas Goodwin
43c2a3acc3 layer-config: fixing silent failures from always exiting '0'
The return value from bitbakecmd was not being returned when
errors occurred which allowed shared-repo-unpack to succeed
despite the failure.  This fix changes to check_call and a
try-catch when attempting to add repos that fail for whatever
reason during add-layer, like a missing conf/layer.conf at
the top level or a previously-added layer breaks parsing
because of missing dependencies.

Signed-off-by: Thomas Goodwin <btgoodwin@geontech.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-08-02 17:34:48 +01:00
Richard Purdie
7689893bb3 generate-testresult-index: Handle build performance without reports
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-06-05 16:55:12 +01:00
Richard Purdie
1b3e3ced9f scripts: Add script to generate autobuilder results index
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-06-05 16:43:24 +01:00
Richard Purdie
becdbd3729 utils: Fix incorrect module usage
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-04-12 18:21:51 +01:00
Richard Purdie
68f351ed83 run-config: Fix uninitalized variable problem
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-04-12 18:17:16 +01:00
Richard Purdie
f9bdcf888f config.json/utils: Allow buildhistory to be configurable and disable for oe-selftest in qemu-arch
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-04-12 16:23:54 +01:00
Richard Purdie
a93a79945f scripts/collect-results: Ensure selftest doesn't trigger buildhistory results collection
Affects sumo since that doesn't have oe-selftest parallelism. thud/master create
these in a different location so don't see the issue.

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-03-25 10:07:21 +00:00
Richard Purdie
372282eb6b scripts/publish-artefacts: Ensure aarch64 uninative is published
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-03-19 15:24:33 +00:00
Richard Purdie
520ea457e8 send-qa-email: Fix typo causing wrong report contents to be written
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-02-25 09:34:22 +00:00
Richard Purdie
2deb018441 send-qa-email: Be specific about which test result branches to push
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-02-23 22:53:50 +00:00
Richard Purdie
f19aa61cd6 buildhistory-init/collect-results: Use a tag to make buildhistory data consistent
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-02-23 17:23:52 +00:00
Richard Purdie
a0963045de scripts/utils: Simplify buildhistory path, no need for such deep paths
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-02-22 00:07:37 +00:00
Richard Purdie
e67a6aab27 send-qa-report: Add handling of results files with resulttool
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-02-21 17:19:02 +00:00
Richard Purdie
13eb8eb282 scripts/send-qa-email: Add optional -R option for test results directory
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-02-21 17:13:16 +00:00
Richard Purdie
5130ab9f21 utils: Move getcomparisonbranch() to common function library
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-02-21 17:13:16 +00:00
Richard Purdie
e1564ff86c collect-results: Generate buildhistory-diff output
Since we now have functional buildhistory repository functionality,
generate buildhistory-diff output for builds where its enabled.

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-02-21 13:50:26 +00:00
Richard Purdie
e5310650b0 scripts/utils: Ensure list filtering code doesn't match appends/prepends
We want all INHERIT += to apply for example

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-02-12 14:18:09 +00:00
Richard Purdie
700f297cd5 build-perf-test-wrapper: Print URL data is published to in output
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-02-12 12:47:35 +00:00
Richard Purdie
5a2eddd6bd setup-config/utils: Improve extravars handling
Currently there was no way to override an existing extra vars
assignment, which meant BB_NUMBER_THREADS was impossible to change.

This adds code to allow the higher override of a value to override
the other settings which makes the code function more as a user
would expect.

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-02-11 11:05:49 +00:00
Richard Purdie
28ad5cdcc9 build-perf-test-wrapper: Place auto.conf into the correct location
Currently it was being placed a directory level too high and hence
was being ignored.

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-02-11 11:03:52 +00:00
Richard Purdie
e1b505f006 build-perf-test-wrapper: Fix branch comparision code
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-02-10 08:50:26 +00:00
Richard Purdie
3dfa90c4ba build-perf-test-wrapper: Fix to use full commit hashes and short repo names
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-02-10 08:18:00 +00:00
Richard Purdie
5e62bc3f9c build-perf-test-wrapper: Fix missing module reference
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-02-09 21:18:03 +00:00
Richard Purdie
72b22dc785 config.json/run-config/build-perf-test-wrapper: Allow build performance branch comparisions
This ensures that release branch performance build tests work on the correct
branch and that comparisions between mut+master and master-next+master
start to work correctly. It also ensures reports for the current
revision are generated even if later revisions are built.

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-02-09 17:24:16 +00:00
Richard Purdie
07d4e99b6d build-perf-test-wrapper: Fix missing variable
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-02-08 23:14:57 +00:00
Richard Purdie
9a62bbf6b4 build-perf-test-wrapper: Add HTML link to published text report
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-02-08 14:02:55 +00:00
Richard Purdie
6fbc1321a0 config.json/run-config: Ensure build-perf publish-dir contains the target name
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-02-05 22:22:46 +00:00
Richard Purdie
d4f649024d build-perf-test-wrapper: Fix report path names
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-02-04 20:40:20 +00:00
Richard Purdie
1d8d6b70fa build-perf-test-wrapper: Further variable cleanup
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-02-04 16:12:50 +00:00
Richard Purdie
c865dbb73e build-perf-test-wrapper: Clean up path option handling and directory creation
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-02-04 15:25:21 +00:00
Richard Purdie
ca730319a9 build-perf-test-wrapper: Handle the case where /bin/sh is dash
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-02-04 15:10:31 +00:00
Richard Purdie
72ee261692 build-perf-test-wrapper: Fix path joining issues
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-02-03 22:13:18 +00:00
Richard Purdie
852ebb4889 build-perf-test-wrapper: Fix report path names to account for build directory
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-02-03 15:38:22 +00:00
Richard Purdie
4fda3f2816 build-perf-test-wrapper: Ensure scripts within OE-Core are available within PATH
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-02-03 11:29:07 +00:00
Richard Purdie
e0770f4b55 scripts: Convert build-perf-test-wrapper from shell to python
The shell script can't access the config.json data easily, convert
from shell to python so that we prepare the way for further enhancements.

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-02-02 11:42:52 +00:00
Richard Purdie
efeb042a77 config.json/build-perf-test-wrapper.sh: Add support for publishing to the standard results location
This makes the build performance code behave similarly to the other
results publishing code using the standard location.

Also tweak the report names so they sort.

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-02-01 15:18:57 +00:00
Richard Purdie
b480d77271 scripts/run-config: Allow command failures to be non-fatal
We need the final logging to show which command steps contain errors so print
failing commands but allow the code to continue and show the final log output.

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-01-31 12:57:54 +00:00
Richard Purdie
81382c6c5d scripts/run-config: Always run bitbake with -k option
This ensures the builds get as far as they can when there is failure rather
that stopping at the first sign of one. This means caches are hotter and
allows for faster respins, at the cost of the failure being lost in longer
logs.

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-01-31 12:53:00 +00:00
Richard Purdie
69b8ac6c27 build-perf-test-wrapper: Ensure download_dir exists
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-01-22 23:09:26 +00:00
Richard Purdie
6e078513b7 build-perf-test-wrapper: Remove code we don't need
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-01-22 23:02:13 +00:00
Richard Purdie
f022d37f70 build-perf-test-wrapper/report-email: Tweak for ab-helper usage
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-01-22 23:01:57 +00:00
Richard Purdie
fbe172bbdf scripts: Import build-perf scripts from oe-core
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-01-22 22:58:11 +00:00
Richard Purdie
d36deb8543 run-config: Ensure log directory (builddir) exists
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2018-12-26 11:34:03 +00:00
Richard Purdie
d6049a972f publish-artefacts: Fix an incorrect target directory for qemux86-64-lsb
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2018-11-21 17:38:38 +00:00
Richard Purdie
a1e1525178 config.json/publish-artefacts: Split qemu machines from real hardware and drop nightly prefix+rename
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2018-11-21 15:32:38 +00:00
Richard Purdie
dfd07f06f8 scripts/run-config: Ensure we rename the build directory before exitting when failing
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2018-11-21 11:47:46 +00:00
Richard Purdie
e9b855fe8c scripts: Add results collection script and option in run-config 2018-11-20 23:32:09 +00:00
Richard Purdie
10b47e4928 run-config: Fix missing / typo
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2018-11-20 15:24:42 +00:00
Richard Purdie
37b3943ca7 scripts/run-config: Rename any completed build directory so that other builds can't reference paths within it
This should further protect us against contamination between different build directories
as old build directory paths should no longer exist.

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2018-11-20 14:52:00 +00:00
Richard Purdie
9e181c3ebb run-config: Remove successful build directories straight away
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2018-11-20 14:50:35 +00:00
Richard Purdie
19bd8ebfee setup-config: Also set HELPERBUILDDIR
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2018-11-11 09:26:51 +00:00
Richard Purdie
3c57c77a77 run-config: Fix typo
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2018-11-10 14:45:26 +00:00
Richard Purdie
885d79c393 config.json/run-config: Add HELPERBUILDDIR and fix sstate reuse for nightly-bringup
There is no current mechanism to reference the 'builddir' within the config.json,
so add HELPERBUILDDIR for this purpose.

Use this to set SSTATEDIR for nightly-bringup, meaning sstate arefacts should be
shared for use between the different oe-selftest threads, speeding up builds.

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2018-11-10 14:42:07 +00:00
Richard Purdie
7f6a3b7fba scripts/utils: Fix string/float combination error
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2018-11-09 14:17:10 +00:00
Richard Purdie
dabf9d6e20 scripts/utils: Fix variable name conflict
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2018-11-09 14:14:34 +00:00
Richard Purdie
8ce219dda5 run-config/utils: Allow headers to show the difference in time since the previous header
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2018-11-09 14:06:52 +00:00
Richard Purdie
df2e005030 scripts/setup-config: Allow SSTATEDIR to unset
The SSTATEDIR was accidentally using getconfiglist instead of getconfigvar which
meant it couldn't be unset. Fix to use var so the values in config.json work
as intended.

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2018-11-08 21:22:01 +00:00
Richard Purdie
478eb6d78c scripts/run-config: Export SCRIPTSDIR to avoid scripts having to mess with paths
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2018-11-08 20:45:56 +00:00
Richard Purdie
5fe0e21b8a scripts/send-qa-email: Handle systems with sendmail in sbin
Many of the distros we use have sendmail in /sbin or /usr/sbin
which may not be in the path for non-interactive users. Handle
this in the script to avoid failures where tests work but the
real QA email fails.

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2018-11-05 16:57:37 +00:00
Richard Purdie
e23d0b3ab2 scripts/setup-config: Ensure PACKAGE_CLASSES is set
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2018-10-10 11:40:41 +01:00
Richard Purdie
b27ef9a53d run-config: Renumber the log files to match step numbers
Also print a summary of the logfiles which contain failures.

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2018-09-05 14:11:33 +01:00
Aaron Chan
842def0599 publish-artefacts: Add deployment BSP support on x86_64
Add in the support to publish images into the designated path.
BSP packages of the previous build will be cleaned up before
new BSP packages are copied over. This ensures the previous
image will not be retained and causes conflicts before image
is loaded into x86_64 (MTURBOT64) hardware.

Signed-off-by: Aaron Chan <aaron.chun.yew.chan@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2018-08-30 16:21:51 +01:00
Paul Eggleton
d7d1a2b345 scripts: use argparse for command line parsing
[RP: Minor tweaks made to adpat to buildbot and option naming

Fixed shared-repo-unpack for publishing when no cache-dir supplied]

Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2018-08-30 14:36:07 +01:00
Richard Purdie
c41c2e0a5a run-config: Add a new option to control echoing of log output
The new buildbot code can directly monitor the logfiles itself so
we need to be able to optionally stop echoing onto the console.

Add missing flush() calls and print the command being used and logfile
so its clear what is happening on stdout too.

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2018-08-30 13:25:05 +01:00
Richard Purdie
1cb54ada86 scripts/checkvnc: Add checkvnc script from old autobuilder codebase
Update the scripts to correctly reference the new script.

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2018-08-30 13:22:16 +01:00
Aaron Chan
0be30e2b54 run-config: Reverse the oder of remove layers
This patch fixes the "Collection Error during parsing layer conf"
when a parent layer is accidentally added before depend layers.
When removing layers from bblayers, we should not be following
the same sequence of adding the layers, it should be done in a
reversed order. This is an assumption that the layers have their
dependent layers added before the parent layers are added.
In general, the parent layer require to be remove first before
its child dependencies layers.

Signed-off-by: Aaron Chan <aaron.chun.yew.chan@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2018-08-08 11:15:20 +01:00
Richard Purdie
a49b11979e run-config: Add a test mode using an enviornmental variable
Testing this code is becoming increasingly problematic. This allows
ABHELPERTEST to be set from the environment and means the script will
still create directories and configuration files but it will not execute
anything, just print what would be executed.

This allows faster validation of changes to the configurations/code.

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2018-07-06 16:01:24 +01:00
Richard Purdie
89b5f50dd2 utils: Fix previous commit
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2018-07-06 15:55:02 +01:00
Richard Purdie
579b732755 utils: Filter reponame to only use base repository name, not a full url
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2018-07-06 15:48:30 +01:00
Richard Purdie
621143cee4 scripts: Various buildhistory code fixes/improvements
* Remove the code disabling it
* Move to a non-shared directory by default (but still support absolute pathnames
  if wanted)
* Use shallow clones by default for speed
* Allow force pushing 'fork' mode branches

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2018-07-06 15:17:41 +01:00
Richard Purdie
004d403b6a upload-error-reports: Shorted name since server has a 50 char limit
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2018-07-04 10:45:29 +01:00
Richard Purdie
696eacc94a utils: Ensure unspecified variables aren't expanded badly
We want ${BUILDDIR} to stay as ${BUILDDIR}, not become BUILDDIR
for example.

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2018-07-03 14:57:56 +01:00
Richard Purdie
42416bdc9c utils: Ensure we don't infinitely recurse for variable expansion
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2018-07-03 14:08:40 +01:00
Richard Purdie
94cc0afb1c utils: Enhance loadconfig to better merge dicts
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2018-06-21 00:04:11 +01:00
Richard Purdie
b7cf6d8209 utils: Allow customisation using ABHELPER_JSON from the environment
Usage is documented in README and an example, local-example.json is
included.

Also clean up the parameter to loadconfig() as its actually not needed.

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2018-06-20 23:16:10 +01:00
Richard Purdie
0ccbfc0e05 utils: Ensure list entries also are expanded
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2018-06-20 23:07:09 +01:00
Richard Purdie
547a3793fc utils: Fix for python 3.5 and before, remove debug
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2018-06-20 22:14:46 +01:00
Richard Purdie
d6253df2bc scripts: Convert to use data expansion pretty much everywhere
We want to be able to use data expansion/variable substitution
in most variables so convert to use the utils API for this.

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2018-06-20 16:21:16 +01:00
Richard Purdie
d2c0dcd1d9 utils: Add data expansion handling/API
The code clearly won't scale without some kind of variable substitution/
variable expansion mechanism, so add one.

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2018-06-20 16:20:08 +01:00
Richard Purdie
fcd4a3891f scripts/send-qa-email: Handle form enable/disable of email
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2018-06-10 23:17:51 +01:00
Richard Purdie
001353ea6f send-qa-email: Ensure we don't run when not publishing
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2018-06-10 14:10:29 +01:00
Richard Purdie
51054804fc send-qa-email: Resolve revisions into full hashes
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2018-05-16 16:03:42 +01:00
Richard Purdie
477d511aa0 send-qa-email: Allow execution (chmod)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2018-05-14 17:00:42 +01:00
Richard Purdie
25d574946a config.json: Add QA/publish info and script to mail QA
Add a script which allows QA to be emailed upon build completion
for QA test runs.

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2018-05-14 13:04:50 +01:00
Ross Burton
b3eb077391 pre-commit-hook: add a pre-commit hook to verify the JSON
Signed-off-by: Ross Burton <ross.burton@intel.com>
2018-05-11 11:36:06 +01:00
Richard Purdie
74de885ad7 scripts/run-config: Add more detail when running EXTRA*CMDS
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2018-05-09 15:44:38 +01:00
Richard Purdie
7175195f7a publish-artefacts: Fix eclipse artefact publish dir
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2018-03-10 05:38:20 -08:00
Richard Purdie
fde1417b5d utils: Ensure updates on the remote branch are applied for revison==HEAD
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2018-03-09 11:19:01 -08:00
Richard Purdie
c8e4cc8483 utils: Ensure fetchgitrepo fetches revisions as well as tags
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2018-03-01 23:29:59 +00:00
Richard Purdie
ef9b1c8d47 run-config: Add eclipe build support
This means making the configuration generation conditional, the error
reporting conditional and allowing non bitbake commands.

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2018-03-01 21:41:30 +00:00
Richard Purdie
59607f3a19 Add gplv3 check from existing AB code
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2018-02-28 17:37:04 +00:00
Richard Purdie
a37a62a15c utils: Fix typo
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2018-02-07 11:29:38 +00:00
Richard Purdie
0a6549a913 utils: Add timestamp to log headers
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2018-02-07 10:59:18 +00:00
Richard Purdie
53618a59cb utils: Handle oe-selftest in error reports
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2018-02-07 10:39:43 +00:00
Richard Purdie
a979873199 run-config: Remove layers after EXTRACMDS to fix oe-selftest
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2018-02-07 10:35:22 +00:00
Richard Purdie
d8f1e3ec27 run-config: Replace tee usage
tee hides the exit code of the pipeline. Instead use subprocess to write out the logfile.

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2018-02-06 12:58:41 +00:00
Richard Purdie
eb0f56cece utils: Need to flush *after* the header, not before
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2018-02-05 23:25:12 +00:00
Richard Purdie
a7ca61715d upload-error-reports: Fix parameter contamination problem
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2018-02-05 22:45:08 +00:00
Richard Purdie
9948821070 utils: Ensure header flushes stdout/stderr first to order output correctly
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2018-02-05 22:33:21 +00:00
Richard Purdie
7d31d6cb84 run-config: Handle build directory being removed/recreated
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2018-02-05 22:32:57 +00:00
Richard Purdie
cf9cfd5f3a publish-artefacts: Fix another typo
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2018-02-05 22:29:48 +00:00
Richard Purdie
08410f5257 run-config: Fix step numbering printed
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2018-02-05 22:20:42 +00:00
Richard Purdie
cece09f2cf publish-artefacts: Fix recursive function reference and typos
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2018-02-05 22:20:27 +00:00
Richard Purdie
193ea5a9fe utils: Ensure revision has no whitespace
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2018-02-05 18:00:06 +00:00
Richard Purdie
477b75ac3c utils: Add missing output decoding
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2018-02-05 17:58:17 +00:00
Richard Purdie
a71ea84970 Add support for publishing the layer tarballs
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2018-02-05 17:40:49 +00:00
Richard Purdie
38af6b0f6c run-config: Add missing parameter documentation
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2018-02-05 16:38:39 +00:00
Richard Purdie
297219ecc0 run-config: Clean up stepnum a bit
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2018-02-05 16:36:56 +00:00
Richard Purdie
f66abd6270 run-config: Print step x of y headers and summary of the number of failures
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2018-02-05 16:34:51 +00:00
Richard Purdie
4239708b85 Add error reporting server handling
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2018-02-05 13:16:41 +00:00
Richard Purdie
d9137f05ff publish-artefacts: Fix syntax error
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2018-01-31 11:46:53 +00:00
Richard Purdie
2808a9bc72 run-config: Add publish artefact support
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2018-01-31 10:12:52 +00:00
Richard Purdie
943395f925 layer-config: Add configuration for calling the init scripts (fixes eclipse build)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2018-01-30 14:13:22 +00:00
Richard Purdie
6e000b6eb6 layer-config: No longer need to use 'topdir'
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2018-01-30 13:40:43 +00:00
Richard Purdie
a28efd463b shared-repo-unpack: Drop now unneeded directory deletion
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2018-01-30 13:40:16 +00:00
Richard Purdie
f91610defd shared-repo-unpack: Allow shared to be unspecified
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2018-01-29 17:52:27 +00:00
Richard Purdie
aad15671f7 shared-repos-unpack/layer-config: Use topdir and repos subdirs for checkouts/builds
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2018-01-25 10:15:58 +00:00
Joshua Lock
28cf55e5ab shared-repo-unpack: fix mv calls
As the mv commands use a filename wildcard we need to tell check_call to
execute through the shell so that the wildcard is expanded, rather than
treat as a file name.

Signed-off-by: Joshua Lock <joshua.g.lock@intel.com>
2018-01-24 22:20:12 +00:00
Richard Purdie
2591cb364b Add initial publish-artefacts script
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2018-01-24 16:51:04 +00:00
Joshua Lock
8a18530eca shared-repo-unpack: make moving old repos more resilient
Only move the directory contents if they exist, try to move dot
files after moving non dot files.

Signed-off-by: Joshua Lock <joshua.g.lock@intel.com>
2018-01-24 16:22:41 +00:00
Joshua Lock
d3ef90083b shared-repo-unpack: move existing dir's contents, not dir itself
Signed-off-by: Joshua Lock <joshua.g.lock@intel.com>
2018-01-24 14:29:51 +00:00
Richard Purdie
c6a0df6db6 shared-repo-pack: Add missing imports
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2018-01-24 14:11:05 +00:00
Richard Purdie
f92af9cb9a shared-repo-pack: Move out any existing build directory to the trash delection handler
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2018-01-24 14:05:20 +00:00
Richard Purdie
98f101ee93 shared-repo-unpack: Call layer-config
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2018-01-23 16:57:45 +00:00
Richard Purdie
838339775d shared-repo-unpack: Correct the rsync command
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2018-01-23 16:11:36 +00:00
Richard Purdie
c431561542 Add layer-config script to setup layers correctly
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2018-01-23 16:11:18 +00:00
Richard Purdie
376ad36d4d Move buildhistory init to run-config (and factor common functionality to utils)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2018-01-23 13:49:38 +00:00
Richard Purdie
f79fab19a1 utils/shared-repo-unpack: Create common mkdir function and fix shared-repo-unpack to create directory to rsync to
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2018-01-23 13:09:53 +00:00
Richard Purdie
806957b392 scripts: Fix help text and program name
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2018-01-23 13:09:53 +00:00
Joshua Lock
316e0629f9 utils: fix variable name in fetchgitrepo
Signed-off-by: Joshua Lock <joshua.g.lock@intel.com>
2018-01-22 20:14:30 +00:00
Richard Purdie
0ffecfbb25 Add repo filtering functionality per target
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2018-01-22 16:25:13 +00:00
Richard Purdie
89fbb72693 shared-repo-unpack: Fix missing variable
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2018-01-22 16:13:55 +00:00
Richard Purdie
7f70683c57 prepare-shared-repos/shared-repo-unpack: Allow future option to filter repos and implement more functionality
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2018-01-22 16:12:34 +00:00
Richard Purdie
6bf85dbbdc Add initial repo handling scripts
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2018-01-22 15:20:37 +00:00
Richard Purdie
45b14f6c80 utils: Create helper for loading config.json
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2018-01-22 15:20:00 +00:00
Richard Purdie
b47ea0eaa6 run-config: Also use templating support (move to utils)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2018-01-20 13:41:33 +00:00
Richard Purdie
40eb3f0820 run-config: Fix return value handling
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2018-01-20 13:40:01 +00:00
Richard Purdie
ea38889fc3 config.json/setup-config: Add templating support to remove error prone configuration duplication
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2018-01-20 13:01:06 +00:00
Richard Purdie
db8cf29820 run-config: Add support for adding/removing layers and extra commands in steps
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2018-01-20 11:14:58 +00:00
Richard Purdie
9996fff768 run-config: Ensure stdout/stderr is flushed and fix DISPLAY
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2018-01-19 12:26:00 +00:00
Richard Purdie
0fbdf0c27f run-config: Fix typo
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2018-01-18 23:02:48 +00:00
Richard Purdie
4ceb9e49d4 run-config: Fix off by one error
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2018-01-18 22:29:46 +00:00
Richard Purdie
307ce0c502 run-config: Print error message/exit code upon failure
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2018-01-18 18:30:34 +00:00
Richard Purdie
8cd1d0ccb8 Handle build-appliance srcrev
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2018-01-18 18:14:06 +00:00
Richard Purdie
0a045ed92d run-config: Cleanup/fix commands
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2018-01-18 17:52:30 +00:00
Richard Purdie
4433714dfe setup-config: Also use printheader from utils (and cleanup utils extra \n)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2018-01-18 17:12:52 +00:00
Richard Purdie
94460edf3b run-config: Use subprocess directly so we see the output
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2018-01-18 17:10:02 +00:00
Richard Purdie
94b4144c14 utils: Add printheader function
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2018-01-18 15:27:27 +00:00
Richard Purdie
a18dd64923 run-config: Complete the code
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2018-01-18 15:27:14 +00:00
Richard Purdie
1a5e7e1cb8 add run-config script to execute targets
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2018-01-18 15:09:20 +00:00
Richard Purdie
337f0c12f9 setup-config: Split useful functions to utils.py
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2018-01-18 14:53:59 +00:00
Richard Purdie
3374739122 Only run buildhistory-init on first step
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2018-01-17 23:32:56 +00:00
Richard Purdie
d3efe9c894 indent print output to aid readability
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2018-01-17 23:27:35 +00:00
Richard Purdie
cb4b520530 Fix sdk-extras mirror url
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2018-01-17 23:26:41 +00:00
Richard Purdie
59bd6f009b Improve sstate mirror handling for release
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2018-01-17 23:25:26 +00:00
Richard Purdie
bf3986819f Handle buildinfo
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2018-01-17 22:58:29 +00:00
Richard Purdie
a2c18b4537 Finish buildhistory support
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2018-01-17 22:44:28 +00:00
Richard Purdie
88973c3c99 Add sdk-extra handling, DL_DIR and SSTATE_DIR
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2018-01-17 18:24:20 +00:00
Richard Purdie
8e3081ffd7 Handle PRSERV
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2018-01-17 17:47:32 +00:00
Joshua Lock
1db6f2f787 setup-config: disable version-going-backwards error when using buildhistory
The version-going-backwards ERROR_QA option should be disabled when using
buildhistory.

Signed-off-by: Joshua Lock <joshua.g.lock@intel.com>
2018-01-17 14:00:40 +00:00
Richard Purdie
8deb8008d4 setup-config: Add handling for MACHINE and SDKMACHINE
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2018-01-16 21:47:48 +00:00
Richard Purdie
0495bf45cc setup-config: Clean up stepnum handling and drop debug
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2018-01-15 23:42:13 +00:00
Richard Purdie
885d2d8c91 setup-config: Handle step variable overrides
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2018-01-15 21:43:38 +00:00
Richard Purdie
c4999fdb69 Add getconfiglist and use for hanfling extravars
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2018-01-15 17:28:33 +00:00
Richard Purdie
f1f2190025 Allow extravars to be specified
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2018-01-15 17:20:21 +00:00
Joshua Lock
0c49fbbc80 setup-config: don't strip off "nightly-" prefix from target
This ultimately makes the code more complex as not all buildsets on the
Yocto Project autobuilder are named with that prefix.

Signed-off-by: Joshua Lock <joshua.g.lock@intel.com>
2018-01-15 17:07:17 +00:00
Richard Purdie
43b255cae8 Add USE_ERROR_REPORT
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2018-01-15 17:04:24 +00:00
Joshua Lock
bb3a281db6 setup-config: add a 5th parameter to take the repo name
This will likely be useful in several places and is require for buildhistory
configuration.

Signed-off-by: Joshua Lock <joshua.g.lock@intel.com>
2018-01-15 16:47:06 +00:00
Richard Purdie
22f5aea386 Handle DISTRO setup
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2018-01-15 16:38:23 +00:00
Joshua Lock
19ebd0492a setup-config: fix another os.remove() call
The remove method is in the top-level os module, not os.path - I missed
this call last time...

Signed-off-by: Joshua Lock <joshua.g.lock@intel.com>
2018-01-15 16:36:13 +00:00
Joshua Lock
c4fd38e466 setup-config: fix os.remove() call
The remove method is in the top-level os module, not os.path

Signed-off-by: Joshua Lock <joshua.g.lock@intel.com>
2018-01-15 16:22:34 +00:00
Joshua Lock
15613c6708 setup-config: fix last commit, import errno module
Signed-off-by: Joshua Lock <joshua.g.lock@intel.com>
2018-01-15 15:33:36 +00:00
Joshua Lock
fb6405431f setup-config: don't fail if autoconf's parent dir already exists
Signed-off-by: Joshua Lock <joshua.g.lock@intel.com>
2018-01-15 15:31:08 +00:00
Joshua Lock
f3e37a8614 setup-config: remove stray :
Signed-off-by: Joshua Lock <joshua.g.lock@intel.com>
2018-01-15 15:27:30 +00:00
Joshua Lock
dccd73ef47 setup-config: fix a couple of missing : after if expressions
Signed-off-by: Joshua Lock <joshua.g.lock@intel.com>
2018-01-15 15:22:29 +00:00
Richard Purdie
dc2ec763af setup-config: Add branch information to build and generate auto.conf
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2018-01-15 15:09:43 +00:00
Richard Purdie
c3eda8bbfa Further development of the file
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2018-01-15 14:51:19 +00:00
Richard Purdie
c3ff43bcbf Further development of the file
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2018-01-15 14:47:55 +00:00
Richard Purdie
3081270545 Add buildhistory-reset and further cleanup
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2018-01-15 14:03:33 +00:00
Richard Purdie
bb67d8aa84 setup-config: Start to build up variables to add to configuration
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2018-01-15 13:56:00 +00:00
Richard Purdie
d19e7f2802 Continue to fill out buildhistory code
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2018-01-15 13:45:29 +00:00
Richard Purdie
70db7ea161 setup-config: Convert to python, handle paths issues and add basic json config
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2018-01-15 10:30:55 +00:00
Richard Purdie
09d2ec62dc Initial population with sample code
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2018-01-15 10:16:13 +00:00