We've been seeing increasing problems with disk space issues breaking builds.
Increase the check from 100GB to 200GB
Monitor /tmp for 10GB free as well
Refactor the code to make adding other mount point checks easy
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
This enables faster cleanup of cancelled builds, reducing chance
of disk space running out.
[YOCTO #15205]
Signed-off-by: Alexander Kanavin <alex.kanavin@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
This avoids OE-Core builds reseting to an OE-Core revision, cleaning up the
console display.
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Try and avoid errors like:
/bin/sh: 1: cd: can't cd to /poky
0631ddde57c27c25f5729bc0204476f8c7ff826f
for builds without a parent trigger and hence a shared checkout directory.
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Practise is showing 60GB can still let the builders run out of space. Increase
to 100GB and also delay things more to allow better recovery, 2 minutes isn't
a long time when dealing with file deletion.
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The way the canStartBuild code was written, it inserted a delay between
each build starting of 2 minutes unconditionally. We only want to do this
if the worker had run out of space so tweak the code accordingly.
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The bonus is applied to the start time and needs to make the build
appear as if it started earlier than it did since earlier builds are
prioritised. This means the bonus should be subtracted, not added.
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Modify the builder_bonuses calculation so that it provides variable
bonus time based on the builder, rather than a constant value.
Signed-off-by: Trevor Gamblin <trevor.gamblin@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
We need a way to limit the builds for when a given worker has less than
a certain amount of disk space available. This implements a
canStartBuild method based on the example in the Buildbot docs and
blocks a build if the worker has less than 60GB of disk space available.
Unlike the example code, we want the stdout of the command so that we
can calculate the amount of disk space, rather than just relying on the
remote command's return code.
Docs: https://docs.buildbot.net/latest/manual/customization.html#canstartbuild-functions
[YOCTO #14591]
Signed-off-by: Trevor Gamblin <trevor.gamblin@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The previous idea whilst looking good meant those builders were prioritised
over all others meaning multiple builds running in parallel which would
all be partly complete.
After discussion with upstream, add a bonus weighting mechanism to the
default sorting (which is based upon start time) so that our priority
builds should start first but builds started earlier should still be
prioritised.
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
This prioritizeBuilders function sorts builders by the length of their
associated worker lists, so that builders that can only be assigned to a
small number of workers are assigned to those workers before other
builds that don't have specific needs when resources are limited. An
example might be when a slot is available on an Ubuntu-based worker, and
"oe-selftest-ubuntu" and "genericx86-64" build requests exist in the
queue. Since oe-selftest-ubuntu requires an Ubuntu-based worker and
genericx86-64 does not, genericx86-64 will be assigned a higher value
(lower priority) so that oe-selftest-ubuntu is assigned to that worker
first.
[YOCTO #13644]
Signed-off-by: Trevor Gamblin <trevor.gamblin@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
For new tasks, hide errors/warnings if not present, don't monitor steyXY
logfiles as they're no longer needed and ensure description is provided
by our code as shellCommand doesn't handle it.
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Rather than running all the work in a single run-config command, split this into
multiple steps. This allows for separate result code reporting, timestamps
for indvidual steps and is generally much cleaner. The naming and so on will need
further work but this is a start.
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Since clobber is crossing two file systems in this case the worker
can delete the files faster than it can move them
Also fixes a occasional race that results in build failures
Signed-off-by: Steve Sakoman <steve@sakoman.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
This means the source mirror check is run a significant time after the source mirror
update allowing the mirroring code to have time to take effect rather than showing
errors on first build runs.
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
nextWorker can block a request if there is no worker available.
_getNextUnclaimedBuildRequest will always return the first request
which then will always fail to find worker, and this will block the queue
We therefore randomise the build requests queue with nextBuild to avoid
blocking.
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The readme suggests exporting this variable from the
environment to expose multiple configuration JSON files
but the intended behavior doesn't work without adding the
variable to extra_env.
Signed-off-by: Thomas Goodwin <btgoodwin@geontech.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
This reduces unnecessary properties being set in builds where we don't need
them by filtering to the list of repos the build really needs.
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
