Commit Graph

21 Commits

Author SHA1 Message Date
Alexis Lothoré
ff074831d7 scripts/prepare-shared-repos: fix minor typo in autobuilder logs
Minor typo observed when cheking the "Prepared shared repository" step logs in
autobuilder web interface:

====================================================================================================
Intially fetching repo poky (1675810261.1)
====================================================================================================

Signed-off-by: Alexis Lothoré <alexis.lothore@bootlin.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2023-02-15 11:31:10 +00:00
Richard Purdie
effb9b0fcd scripts: Ensure gplv2 revision isn't listed and repos aren't used
We disabled meta-gplv2 for master, ensure the repo isn't used in the
collection or QA scripts.

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2022-09-26 13:01:37 +01:00
Richard Purdie
8702fdbb72 Clarify license and copyright information
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2022-05-17 10:29:45 +01:00
Richard Purdie
aa378bfa84 prepare-shared-repos: Fix none published builds after disabling limited depth
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2022-01-24 18:05:23 +00:00
Richard Purdie
1e0b8d9936 prepare-shared-repos: Disable shallow clone speedup as it breaks release builds
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2022-01-21 17:19:47 +00:00
Richard Purdie
68b4b7090d prepare-shared-repos: Use full depth for release builds
Without this we'd see failures in the perf builds since it would be able to track
history correctly for commit counts.

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2021-12-12 11:54:37 +00:00
Richard Purdie
aff49e938e shared-repos: Use tar instead of rsync for speed
The rysnc of 20,000 files (650MB) onto the nas is slow taking ~3 minutes
at idle and worse at load. This is due to the number of files which
is a pain point for NFS. This piece of the build is also a bottleneck
since the rest of a build depends on it happening.

If we switch to zstd compressed tar, it takes 2.49s. Other compression
methods were much slower but zstd seems 'accptable' and speeds things
up too.

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2021-10-31 17:37:34 +00:00
Richard Purdie
5dc0559cc8 prepare-shared-repo/utils: Limit HEAD clones to shallow depth to save time/space
By not syncing all the history is is possible to save some time/space
in the checkout process since we never use this data. This reduces data
from 650MB to 400MB or with the tarball, 416MB to 55MB.

The logic for the commands needs to be tweaked to handle this and as
written it can't work in non-HEAD revision case but that isn't a commonly
used situation.

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2021-10-31 13:03:22 +00:00
Richard Purdie
8e996a95a8 prepare-shared-repos: Make it clear when rsync starts in logs
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2021-10-30 16:35:29 +01:00
Richard Purdie
298a105758 scripts/prepare-shared-repos: Use tmpfs for speed
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2021-10-30 15:03:38 +01:00
Richard Purdie
718fe955c2 prepare-shared-repos: Really fix shell globbing
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-09-06 15:57:56 +01:00
Richard Purdie
e99f772fc4 prepare-shared-repos: UAllow shell globing to fix failure
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-09-06 15:45:19 +01:00
Richard Purdie
554a2114ef prepare-shared-repos: Use temporary non-NFS directory for layer clones for speed
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-09-06 15:37:43 +01:00
Paul Eggleton
d7d1a2b345 scripts: use argparse for command line parsing
[RP: Minor tweaks made to adpat to buildbot and option naming

Fixed shared-repo-unpack for publishing when no cache-dir supplied]

Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2018-08-30 14:36:07 +01:00
Richard Purdie
b7cf6d8209 utils: Allow customisation using ABHELPER_JSON from the environment
Usage is documented in README and an example, local-example.json is
included.

Also clean up the parameter to loadconfig() as its actually not needed.

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2018-06-20 23:16:10 +01:00
Richard Purdie
d6253df2bc scripts: Convert to use data expansion pretty much everywhere
We want to be able to use data expansion/variable substitution
in most variables so convert to use the utils API for this.

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2018-06-20 16:21:16 +01:00
Richard Purdie
a71ea84970 Add support for publishing the layer tarballs
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2018-02-05 17:40:49 +00:00
Richard Purdie
806957b392 scripts: Fix help text and program name
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2018-01-23 13:09:53 +00:00
Richard Purdie
0ffecfbb25 Add repo filtering functionality per target
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2018-01-22 16:25:13 +00:00
Richard Purdie
7f70683c57 prepare-shared-repos/shared-repo-unpack: Allow future option to filter repos and implement more functionality
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2018-01-22 16:12:34 +00:00
Richard Purdie
6bf85dbbdc Add initial repo handling scripts
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2018-01-22 15:20:37 +00:00