This writes the metrics to a new yocto-metrics-meta-oe repository, and
write the results to patchmetrics-meta-oe.
There's a great/ugly (delete to taste) hack to hide CVEs from oe-core,
so this only lists issues in meta-oe.
Signed-off-by: Ross Burton <ross.burton@arm.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Rewrite the scripts that gather the metrics to be more generic.
Extract the metrics repository cloning out so that we don't have to
repeatedly clone it.
Make the scripts parse their arguments using getopt and be more specific
about what they're passed. In particular, this means that for the patch
review run we pass the _repository_ that we're scanning so we can do git
operations on it, and the base of the _layers_ (either a layer, or a
directory containing layers) so we know what to scan.
Be more clever when identifying what commits we need to analyse for
patch review: instead of iterating through a set randomly, we can keep
the revision list sorted and the checkout operations are a lot faster.
Remove the commit/file count metric addition as patchreview itself does
that now.
Add an explicit --push option so it's easy to test the scripts in
isolation without pushing.
Signed-off-by: Ross Burton <ross.burton@arm.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
This is a handy helper script to query the Buildbot REST API to
create a list of Yocto Project Compatible 2.0 layers, as defined by
successful running of the "check-layers*" jobs.
The script only considers the 'jobs' ("Builders" in Buildbot speak) you
ask it to query.
The script looks at the latest -n successful builds for a branch that
matches from a list. The number of builds approach was used because
the property_list cannot be queried for "branch" in the "builders" API.
It looks for a "step" name pattern via a regex, with which all currently
running "check-layer" and "check-layer-nightly" jobs conform.
Once a branch has successfully been queried, it is removed from the list
in further iterations. The intent is to only find the latest passing
list of compatible layers, for each branch.
The output is a yp_compatible_layers.json file which can
be pretty printed with:
cat yp_compatible_layers.json | jq
Signed-off-by: Tim Orling <tim.orling@konsulko.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
This is now done with a dedicated switch, where previously the file was always written
out, creating often unneeded clutter.
Signed-off-by: Alexander Kanavin <alex.kanavin@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Those tests are currently run as the last step in a-full/a-quick,
but it's useful to also trigger them separately, after perhaps
executing other builders that populate the mirrors (such as
qemux86-64 and such).
Signed-off-by: Alexander Kanavin <alex.kanavin@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Add a SECURITY.md file with hints for security researchers and other
parties who might report potential security vulnerabilities.
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Some recipes depend on DISTRO_FEATURES, commercial licensing or compiler
config like fortran. Set these things so that we get wider soruce mirror
coverage and fewer warnings.
'commercial' license usage is ok here since we're not building and then
shipping any binaries or using it, only mirroring the source code.
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Rather than running multiple checkouts, lets move this to the autobuilder
to handle and have it trigger the builds with the right checkouts.
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
We only monitor the master branch for patch metrics as we can't really make
improvements to release branches.
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
To prepare for splitting things up, only clone the metrics repo if it isn't present.
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Currently, the tests for meta-poky and meta-yocto-bsp were malfunctioning.
We need to a) set a distro other than poky and b) avoid the isolation wrapper
since it will use the default environment and reset to the poky distro and
default bblayers.conf which will disable the layer test.
With those two fixes, the layer checks work correctly.
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Now that the _qemutiny test case actually works, we can ensure that all
of the tested poky-tiny machines actually at least boot.
This is a very low bar, but it's better than nothing.
Signed-off-by: Ross Burton <ross.burton@arm.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
This way explicitly listing the test can be avoided, especially when more tests
of the same kind will be added (such as te CDN sstate cache mirror test).
Signed-off-by: Alexander Kanavin <alex.kanavin@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Inspired from trigger-build and trigger-build-post-trigger
The branch must be selected on build configuration.
Signed-off-by: David Pierret <david.pierret@smile.fr>
Reviewed-by: Yoann Congal <yoann.congal@smile.fr>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
For the reproducible-meta-oe builder, workaround the bug #15241 [1], by
passing BB_LOGCONFIG through "readlink -f" to avoid relative reference
to the main build dir.
Also, switch from BBPATH to BUILDDIR to reference the main build dir.
[1]: https://bugzilla.yoctoproject.org/show_bug.cgi?id=15241
Signed-off-by: Yoann Congal <yoann.congal@smile.fr>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Fix meta-filesystem build issue due to hidden dependencies (not set by
LAYERDEPENDS) smbnetfs(meta-filesystem) depends on
samba(meta-networking) that depends on python3-dnspython (meta-python)
Create new build-st dir for each layer to avoid chain of
failure between steps
Signed-off-by: Fabien Thomas <fabien.thomas@smile.fr>
Reviewed-by: Yoann Congal <yoann.congal@smile.fr>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
- Distro has been changed to Poky so exclude Poky's layers from world to
limit reproducibility tests to the targeted layer only.
- Execution on autobuilder shows that addlayer steps often fail :
reorder layer in ADDLAYER list regarding their dependencies.
- Fix shortid building issues by setting USERADD_U(G)ID_TABLES to
passwd and group files. These files must exist in meta-openmbedded.
- Shorten the steps shortnames to improve readability on UI.
- Add a directory level for output reports, for convenience.
[RP: Tweak shortnames and fix parsing issues]
Signed-off-by: Fabien Thomas <fabien.thomas@smile.fr>
Reviewed-by: Yoann Congal <yoann.congal@smile.fr>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
git:// is less performant than https:// and ssh://. We are using ssh://
to ensure we have the very latest commits. The public mirrors may have a
30 second delay while they sync.
Signed-off-by: Michael Halstead <mhalstead@linuxfoundation.org>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The purpose of this new builder is to report the reproducibility status
of all meta-openembedded recipes layer by layer. It use the same
reproducible selftest than OE-Core but setting only world as target,
and excluding all oecore and other openembedded layer recipes.
Also, the report output directory is split by layers.
Signed-off-by: Fabien Thomas <fabien.thomas@smile.fr>
Reviewed-by: Yoann Congal <yoann.congal@smile.fr>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
It has been observed that some regression reports generation may failed
when the comparision base is a branch (e.g master) because we can not find
any test results associated to the branch HEAD. This is especially true for
branches which often change, because not all revisions on those branch are
subject to CI tests.
To fix that, whenever we are not dealing with a release, parse the latest
tested revision in test results repository on target branch in order to
guess the corresponding revision in poky repository, so we are sure that
revisions passed to yocto_testresults_query are indeed tested and
regression report can be generated
Signed-off-by: Alexis Lothoré <alexis.lothore@bootlin.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Test results repository url is used at least twice, so define a constant
holding the url instead of hardcoding it multiple times
Signed-off-by: Alexis Lothoré <alexis.lothore@bootlin.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
There are some inversions in words used to describe elements of comparison
for regression reporting: the main function of send_qa_email starts using
"base" to talk about the target revision and "compare" to talk about the
reference against which it is compared. Then later in the script, the
"base" is used as "base of comparison"/reference revision, while the
"target" branch/revision appears. This becomes quite confusing when we need
to update the script
Re-align wording to avoid confusion:
- always use "target" to talk about current branch/revision of interest
(the newest)
- always use "base" to talk about the reference branch/revision (the
oldest), against which we want to compare the target revision
Signed-off-by: Alexis Lothoré <alexis.lothore@bootlin.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Whilst the script needs to continue if we fail to generate a regression report,
set the exit code accordingly so our CI can flag the issue.
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
It is hard to tell which section of the code specific error messages
come from at present. Add more headers to the output so we can at
least tell which section the messages are from. It also adds some
timing information.
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The i686 SDKMACHINE target makes little sense now, reduce it's usage substantially to
match use in reality and make x86_64 the default.
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
SDKMACHINE defaults to i686 for historical reasons but it should no longer be the main
thing we test. Start to improve things by setting things by everywhere explicitly so
we can then rationalise to what makes sense.
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Introduce the concept of "qemu lite" and reduce the testing on these platforms,
mainly with the removal of core-imaege-sato-sdk.
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
We want to promote generic-arm64 to meta-yocto-bsp in the future, so
start testing this more comprehensively.
Signed-off-by: Ross Burton <ross.burton@arm.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Update getcomparisonbranch unit tests by removing BUILD_HISTORY_DIRECTPUSH
entry in fake configuration
Signed-off-by: Alexis Lothoré <alexis.lothore@bootlin.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Now that BUILD_HISTORY_DIRECTPUSH has been replaced by a hardcoded
condition, remove it from config.json
Signed-off-by: Alexis Lothoré <alexis.lothore@bootlin.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
It has been observed that when a new release branch is created, it is quite
easy to forget to update the BUILD_HISTORY_DIRECTPUSH variable, which leads
to failures in autobuilder like test results not being pushed.
Replace the BUILD_HISTORY_DIRECTPUSH usage with a hardcoded condition which
validates any branch in poky representing a "main" branch, i.e all branches
not ending in "-next"
Signed-off-by: Alexis Lothoré <alexis.lothore@bootlin.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
[YOCTO #14952] -- https://bugzilla.yoctoproject.org/show_bug.cgi?id=14952
Help avoid contention by adding a check to ensure files and directories
are at least 60s old when attempting to delete them from the trashdir,
in case bitbake or another process is still actively using them.
Signed-off-by: Trevor Gamblin <tgamblin@baylibre.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Now that mickledore is released, builds are executed on mickeldore release
branch. If not properly described in config.json, it will be considered a
"work" branch, and as a consequence test results will not be pushed onto
test results git repository
Add mickeldore entry in config.json to fix test results storage
Signed-off-by: Alexis Lothoré <alexis.lothore@bootlin.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Tests results push command depends on basebranch and comparebranch
variables, which are computed based on config.json content. If this file is
not in sync with current release branch, tests results will be properly
stored in git directory but not pushed onto test results server. Since we
are able to detect this scenario, print at least a warning, without
breaking current build since it could be a release
Signed-off-by: Alexis Lothoré <alexis.lothore@bootlin.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
As for other scripts in yocto-autobuilder-helper or oecore, use python
logger class instead of raw print calls to allow log level distinction
Signed-off-by: Alexis Lothoré <alexis.lothore@bootlin.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
