Do another git-pull in the metrics repository before updating the
metrics, in case other metrics jobs running in parallel have updated the
repositories since they were cloned. There will always be possibility
of racing metrics jobs, but this should reduce the chance of it
happening.
An alternative would be to commit and then rebase before pushing, but I
fear that a git-merge could produce invalid JSON and we'd have to
manually fix up the repository. In my opinion, a wasted metrics run is
preferable to potentially corrupted repositories.
[RP: Moved to after the bitbke invocation]
Signed-off-by: Ross Burton <ross.burton@arm.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
setup-auh and run-auh were doing what the AB config.json does:
* creating repo checkouts: Now use NEEDREPOS
* configuring bitbake env: Now use extravars
This refactoring is needed to prepare adding AUH meta-oe support.
Signed-off-by: Yoann Congal <yoann.congal@smile.fr>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Rewrite the scripts that gather the metrics to be more generic.
Extract the metrics repository cloning out so that we don't have to
repeatedly clone it.
Make the scripts parse their arguments using getopt and be more specific
about what they're passed. In particular, this means that for the patch
review run we pass the _repository_ that we're scanning so we can do git
operations on it, and the base of the _layers_ (either a layer, or a
directory containing layers) so we know what to scan.
Be more clever when identifying what commits we need to analyse for
patch review: instead of iterating through a set randomly, we can keep
the revision list sorted and the checkout operations are a lot faster.
Remove the commit/file count metric addition as patchreview itself does
that now.
Add an explicit --push option so it's easy to test the scripts in
isolation without pushing.
Signed-off-by: Ross Burton <ross.burton@arm.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
This is a handy helper script to query the Buildbot REST API to
create a list of Yocto Project Compatible 2.0 layers, as defined by
successful running of the "check-layers*" jobs.
The script only considers the 'jobs' ("Builders" in Buildbot speak) you
ask it to query.
The script looks at the latest -n successful builds for a branch that
matches from a list. The number of builds approach was used because
the property_list cannot be queried for "branch" in the "builders" API.
It looks for a "step" name pattern via a regex, with which all currently
running "check-layer" and "check-layer-nightly" jobs conform.
Once a branch has successfully been queried, it is removed from the list
in further iterations. The intent is to only find the latest passing
list of compatible layers, for each branch.
The output is a yp_compatible_layers.json file which can
be pretty printed with:
cat yp_compatible_layers.json | jq
Signed-off-by: Tim Orling <tim.orling@konsulko.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Rather than running multiple checkouts, lets move this to the autobuilder
to handle and have it trigger the builds with the right checkouts.
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
We only monitor the master branch for patch metrics as we can't really make
improvements to release branches.
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
To prepare for splitting things up, only clone the metrics repo if it isn't present.
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
It has been observed that some regression reports generation may failed
when the comparision base is a branch (e.g master) because we can not find
any test results associated to the branch HEAD. This is especially true for
branches which often change, because not all revisions on those branch are
subject to CI tests.
To fix that, whenever we are not dealing with a release, parse the latest
tested revision in test results repository on target branch in order to
guess the corresponding revision in poky repository, so we are sure that
revisions passed to yocto_testresults_query are indeed tested and
regression report can be generated
Signed-off-by: Alexis Lothoré <alexis.lothore@bootlin.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Test results repository url is used at least twice, so define a constant
holding the url instead of hardcoding it multiple times
Signed-off-by: Alexis Lothoré <alexis.lothore@bootlin.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
There are some inversions in words used to describe elements of comparison
for regression reporting: the main function of send_qa_email starts using
"base" to talk about the target revision and "compare" to talk about the
reference against which it is compared. Then later in the script, the
"base" is used as "base of comparison"/reference revision, while the
"target" branch/revision appears. This becomes quite confusing when we need
to update the script
Re-align wording to avoid confusion:
- always use "target" to talk about current branch/revision of interest
(the newest)
- always use "base" to talk about the reference branch/revision (the
oldest), against which we want to compare the target revision
Signed-off-by: Alexis Lothoré <alexis.lothore@bootlin.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Whilst the script needs to continue if we fail to generate a regression report,
set the exit code accordingly so our CI can flag the issue.
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
It is hard to tell which section of the code specific error messages
come from at present. Add more headers to the output so we can at
least tell which section the messages are from. It also adds some
timing information.
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Update getcomparisonbranch unit tests by removing BUILD_HISTORY_DIRECTPUSH
entry in fake configuration
Signed-off-by: Alexis Lothoré <alexis.lothore@bootlin.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
It has been observed that when a new release branch is created, it is quite
easy to forget to update the BUILD_HISTORY_DIRECTPUSH variable, which leads
to failures in autobuilder like test results not being pushed.
Replace the BUILD_HISTORY_DIRECTPUSH usage with a hardcoded condition which
validates any branch in poky representing a "main" branch, i.e all branches
not ending in "-next"
Signed-off-by: Alexis Lothoré <alexis.lothore@bootlin.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Tests results push command depends on basebranch and comparebranch
variables, which are computed based on config.json content. If this file is
not in sync with current release branch, tests results will be properly
stored in git directory but not pushed onto test results server. Since we
are able to detect this scenario, print at least a warning, without
breaking current build since it could be a release
Signed-off-by: Alexis Lothoré <alexis.lothore@bootlin.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
As for other scripts in yocto-autobuilder-helper or oecore, use python
logger class instead of raw print calls to allow log level distinction
Signed-off-by: Alexis Lothoré <alexis.lothore@bootlin.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
edgerouter is no longer part of meta-yocto so we removed it from the
autobuilder configuration as well.
Signed-off-by: Michael Halstead <mhalstead@linuxfoundation.org>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
AUH itself already has an option to include the link into
its email reports; the option just needs to be enabled.
[YOCTO #15103]
Signed-off-by: Alexander Kanavin <alex.kanavin@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
When available, expose tesresult-regressions-report.txt on non-release web page,
as it is done for many other artifacts currently
Signed-off-by: Alexis Lothoré <alexis.lothore@bootlin.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Some nightly builders are configured in yocto-autobuilder2 to run master builds.
Those build parameters currently skip all branches of
get_regression_base_and_target, which then return None, while the caller
expects a base and target tuple
Set default behaviour to return previous tag as comparison base and passed
branch as target for such builds
Signed-off-by: Alexis Lothoré <alexis.lothore@bootlin.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
When the test assert is about a tag in Poky, the result will not be the same
depending on existing tags at the time of running tests.
Add a LAST_TAG marker to loosen constraints but still allow to tests for general
cases (e.g. : test that tag-depending tests does not return None)
Signed-off-by: Alexis Lothoré <alexis.lothore@bootlin.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
d6018b891a broke regression reporting for testing
branches (e.g: master-next in poky, ross/mut in poky-contrib) by ignoring the comparebranch returned by
utils.getcomparison branch
Fix regression reporting for those branches by using comparebranch again. The
fix also refactor/add a intermediary step to guess base and target for
regression reporting, to isolate a bit the logic and make it easier later to add
multiple base/target couples
Signed-off-by: Alexis Lothoré <alexis.lothore@bootlin.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
is_non_release_version has an inverted logic which makes its reuse quite
confusing
Transform it as is_release_version and let caller do the negation if needed
Signed-off-by: Alexis Lothoré <alexis.lothore@bootlin.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
A new tool named yocto_testresults_query.py has been integrated in poky as a
thin wrapper between send-qa-email and resulttool. The new tool is in charge of
converting tags/branches names to SHA1 revisions and to call resulttool with
those revisions
Remove any code related to tag/branches conversions to SHA1 and use
yocto_testresults_qery.py instead
Signed-off-by: Alexis Lothoré <alexis.lothore@bootlin.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Minor typo observed when cheking the "Prepared shared repository" step logs in
autobuilder web interface:
====================================================================================================
Intially fetching repo poky (1675810261.1)
====================================================================================================
Signed-off-by: Alexis Lothoré <alexis.lothore@bootlin.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Current regression reports do not contain information about versions compared
when generating reports. While it is still possible to get the information by
searching the autobuilder log, it is not convenient. Moreover, future
developments will allow to generate multiple reports (with different bases for
comparison) in a single build.
As a consequence, embed target and base revisions in the report header
Signed-off-by: Alexis Lothoré <alexis.lothore@bootlin.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Since we are now force-fetching base revisions and target revisions for
regression report generation, we can make testresults clone even more "shallow"
to increase clone speed in CI pipelines
Signed-off-by: Alexis Lothoré <alexis.lothore@bootlin.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
If we try to run send-qa-email to simulate past releases (for example, for
development or debugging), the execution will very likely fail because the
target revision to examine (ie: the poky revision) is too old, and as a
consequence is not contained in the testresults shallow clone anymore (because
testsresults history keeps moving forward as builds are triggered on
autobuilder). As a consequence, force-fetch the "target" revision, as it is
already done for the "base" revision
Signed-off-by: Alexis Lothoré <alexis.lothore@bootlin.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
