Commit 1dc77f5908 ("scripts: send_qa_email: add dry-run mode") updated
the get_regression_base_and_target to allow it to receive a test result
url. Unfortunately, this function's tests have not been updated
accordingly, so they currently fail because of the missing parameter.
Add a default url in test case and update the function call with this
default value.
Signed-off-by: Alexis Lothoré <alexis.lothore@bootlin.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
When send_qa_email is executed with python >= 3.12, the following warnings
are emitted:
/home/alexis/src/yocto_ci/yocto-autobuilder-helper/./scripts/send_qa_email.py:22: SyntaxWarning: invalid escape sequence '\d'
p = re.compile('\d{8}-\d+')
/home/alexis/src/yocto_ci/yocto-autobuilder-helper/./scripts/send_qa_email.py:67: SyntaxWarning: invalid escape sequence '\/'
tested_revision = re.match('refs\/tags\/.*\/\d+-g([a-f0-9]+)\/\d', latest_test_tag).group(1)
This warning has been introduced to replace the DeprecationWarning
initially raised on such escape code in string regex. Python 3.12 changelog
([1]) states that raw string must be used in this case. Update both
send_qa_email and its tests file.
[1] https://docs.python.org/dev/whatsnew/3.12.html#other-language-changes
Signed-off-by: Alexis Lothoré <alexis.lothore@bootlin.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
send_qa_email currently fails when dealing with release version starting a
new major, for example 5.0. This has been observed for example when trying
to generate 5.0_M1.rc1
This specific versioning makes previous tag computation method fall through
last branch which currently expects that the current release tag indeed
exists (5.0_M1), which is true when checking regression reports a
posteriori, but not in an autobuilder run (tag is added only when the
release has been "validated")
Fix tag computation for this case by getting previous release tag with git
ls-remote, instead of relying on git describe with a possibly non-existing
tag. While doing so, add a few tests about this specific case.
Signed-off-by: Alexis Lothoré <alexis.lothore@bootlin.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
send_qa_email currently deals with tags in a "reproducible" way: despite
new versions being released on different branches, the computation of the
"reference" version for a specific input version always remain the same.
This behavior does not match perfectly real expectations: if at the some
point we get version 4.3.1 as a comparison reference for a regression
report, and 4.3.2 is released some time later, we want the next comparision
to be done against 4.3.2.
Start introducing this new behavior by allowing the tests to check returned
version against regex patterns instead of static strings, so we can for
example use wildcards on the "micro" version
Signed-off-by: Alexis Lothoré <alexis.lothore@bootlin.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Use intermediate variables for test input/output, and print expected
output in subtest to get more info when a test fails
Signed-off-by: Alexis Lothoré <alexis.lothore@bootlin.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
It has been observed that some regression reports generation may failed
when the comparision base is a branch (e.g master) because we can not find
any test results associated to the branch HEAD. This is especially true for
branches which often change, because not all revisions on those branch are
subject to CI tests.
To fix that, whenever we are not dealing with a release, parse the latest
tested revision in test results repository on target branch in order to
guess the corresponding revision in poky repository, so we are sure that
revisions passed to yocto_testresults_query are indeed tested and
regression report can be generated
Signed-off-by: Alexis Lothoré <alexis.lothore@bootlin.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
There are some inversions in words used to describe elements of comparison
for regression reporting: the main function of send_qa_email starts using
"base" to talk about the target revision and "compare" to talk about the
reference against which it is compared. Then later in the script, the
"base" is used as "base of comparison"/reference revision, while the
"target" branch/revision appears. This becomes quite confusing when we need
to update the script
Re-align wording to avoid confusion:
- always use "target" to talk about current branch/revision of interest
(the newest)
- always use "base" to talk about the reference branch/revision (the
oldest), against which we want to compare the target revision
Signed-off-by: Alexis Lothoré <alexis.lothore@bootlin.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Some nightly builders are configured in yocto-autobuilder2 to run master builds.
Those build parameters currently skip all branches of
get_regression_base_and_target, which then return None, while the caller
expects a base and target tuple
Set default behaviour to return previous tag as comparison base and passed
branch as target for such builds
Signed-off-by: Alexis Lothoré <alexis.lothore@bootlin.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
When the test assert is about a tag in Poky, the result will not be the same
depending on existing tags at the time of running tests.
Add a LAST_TAG marker to loosen constraints but still allow to tests for general
cases (e.g. : test that tag-depending tests does not return None)
Signed-off-by: Alexis Lothoré <alexis.lothore@bootlin.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
A new tool named yocto_testresults_query.py has been integrated in poky as a
thin wrapper between send-qa-email and resulttool. The new tool is in charge of
converting tags/branches names to SHA1 revisions and to call resulttool with
those revisions
Remove any code related to tag/branches conversions to SHA1 and use
yocto_testresults_qery.py instead
Signed-off-by: Alexis Lothoré <alexis.lothore@bootlin.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The "previous version" computation bring many edge cases depending on the
version under release. Add a basic test suite to validate currently implemented
computation strategy and to prevent mistakes when ediiting it in the future
Signed-off-by: Alexis Lothoré <alexis.lothore@bootlin.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
