When giving jobs to workers, we want to prioritise idle ones and only
give them to the busiest ones when there are no others free.
This isn't entirely straight forward as there is no "idle" attribute
for workers but we can use a query and work it out.
Includes fixes from Mathieu Dubois-Briand <mathieu.dubois-briand@bootlin.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Make sure tags created on a-full/a-quick builds are unique, even across
different controller instances.
Signed-off-by: Mathieu Dubois-Briand <mathieu.dubois-briand@bootlin.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
In the current form when forcing a docs build, the different repo
parameters do nothing and are not propagated. The properties from
props_for_builder() are propagated, but weren't used in the factory
definition.
So use FixedParameters in the codebases definition: this removes the
useless entries in the form (we cannot simply remove the codebases arg
or it triggers another error "No sourcestamp found in build for
codebase"). Then in config.py, populate the "docs" entry in
buildertorepos, so that these show up in the form. Finally, in the docs
factory build, use Interpolate to get the property (I had issues using
util.Property directly).
Note: in latest buildbot version I hit a known bug
https://github.com/buildbot/buildbot/issues/8274. It shouldn't affect us
since we use an older version of Buildbot, but leaving this here just in
case.
Signed-off-by: Antonin Godard <antonin.godard@bootlin.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The introduction of codebases for docs builder changed the git clone
steps internal names, as they are now suffixed with the codebase name.
This new name for the yocto-autobuilder-helper seems to be too long for
the database used on our autobuilder and triggers an exception. Use a
shorter codebase name to fix this issue.
Signed-off-by: Mathieu Dubois-Briand <mathieu.dubois-briand@bootlin.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
While most of the builds on the autobuilder rely on the
shared-repo-unpack helper script and do not use the Buildbot codebases,
the "docs" builder use Buildbot traditional way to setup sources. As a
consequence, we have to use Buildbot codebases, otherwise the same
branch name is used for all cloned repos.
Add the codebase specifications for all three repos used by the docs
builder. Also remove them from the repos list, as these fields are not
used.
Signed-off-by: Mathieu Dubois-Briand <mathieu.dubois-briand@bootlin.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Helper branches with names like contrib/kirkstone-nut don't work with
worker restrictions currently. Try and fix this.
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Simplfy the space command to directly use findmnt and output in bytes to simplify the expression.
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
We've been seeing increasing problems with disk space issues breaking builds.
Increase the check from 100GB to 200GB
Monitor /tmp for 10GB free as well
Refactor the code to make adding other mount point checks easy
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
This enables faster cleanup of cancelled builds, reducing chance
of disk space running out.
[YOCTO #15205]
Signed-off-by: Alexander Kanavin <alex.kanavin@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
This avoids OE-Core builds reseting to an OE-Core revision, cleaning up the
console display.
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Try and avoid errors like:
/bin/sh: 1: cd: can't cd to /poky
0631ddde57c27c25f5729bc0204476f8c7ff826f
for builds without a parent trigger and hence a shared checkout directory.
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Practise is showing 60GB can still let the builders run out of space. Increase
to 100GB and also delay things more to allow better recovery, 2 minutes isn't
a long time when dealing with file deletion.
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The way the canStartBuild code was written, it inserted a delay between
each build starting of 2 minutes unconditionally. We only want to do this
if the worker had run out of space so tweak the code accordingly.
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The bonus is applied to the start time and needs to make the build
appear as if it started earlier than it did since earlier builds are
prioritised. This means the bonus should be subtracted, not added.
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Modify the builder_bonuses calculation so that it provides variable
bonus time based on the builder, rather than a constant value.
Signed-off-by: Trevor Gamblin <trevor.gamblin@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
We need a way to limit the builds for when a given worker has less than
a certain amount of disk space available. This implements a
canStartBuild method based on the example in the Buildbot docs and
blocks a build if the worker has less than 60GB of disk space available.
Unlike the example code, we want the stdout of the command so that we
can calculate the amount of disk space, rather than just relying on the
remote command's return code.
Docs: https://docs.buildbot.net/latest/manual/customization.html#canstartbuild-functions
[YOCTO #14591]
Signed-off-by: Trevor Gamblin <trevor.gamblin@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The previous idea whilst looking good meant those builders were prioritised
over all others meaning multiple builds running in parallel which would
all be partly complete.
After discussion with upstream, add a bonus weighting mechanism to the
default sorting (which is based upon start time) so that our priority
builds should start first but builds started earlier should still be
prioritised.
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
This prioritizeBuilders function sorts builders by the length of their
associated worker lists, so that builders that can only be assigned to a
small number of workers are assigned to those workers before other
builds that don't have specific needs when resources are limited. An
example might be when a slot is available on an Ubuntu-based worker, and
"oe-selftest-ubuntu" and "genericx86-64" build requests exist in the
queue. Since oe-selftest-ubuntu requires an Ubuntu-based worker and
genericx86-64 does not, genericx86-64 will be assigned a higher value
(lower priority) so that oe-selftest-ubuntu is assigned to that worker
first.
[YOCTO #13644]
Signed-off-by: Trevor Gamblin <trevor.gamblin@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
For new tasks, hide errors/warnings if not present, don't monitor steyXY
logfiles as they're no longer needed and ensure description is provided
by our code as shellCommand doesn't handle it.
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Rather than running all the work in a single run-config command, split this into
multiple steps. This allows for separate result code reporting, timestamps
for indvidual steps and is generally much cleaner. The naming and so on will need
further work but this is a start.
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Since clobber is crossing two file systems in this case the worker
can delete the files faster than it can move them
Also fixes a occasional race that results in build failures
Signed-off-by: Steve Sakoman <steve@sakoman.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
