builders: Add prioritizeBuilders

This prioritizeBuilders function sorts builders by the length of their
associated worker lists, so that builders that can only be assigned to a
small number of workers are assigned to those workers before other
builds that don't have specific needs when resources are limited. An
example might be when a slot is available on an Ubuntu-based worker, and
"oe-selftest-ubuntu" and "genericx86-64" build requests exist in the
queue. Since oe-selftest-ubuntu requires an Ubuntu-based worker and
genericx86-64 does not, genericx86-64 will be assigned a higher value
(lower priority) so that oe-selftest-ubuntu is assigned to that worker
first.

[YOCTO #13644]

Signed-off-by: Trevor Gamblin <trevor.gamblin@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>

builders.py

@@ -135,6 +135,19 @@ for builder in config.subbuilders:
                                        workernames=workers, nextWorker=nextWorker, nextBuild=nextBuild,
                                        factory=f, env=extra_env))
 
+# prioritize assigning builders to available workers based on the length
+# of the worker lists they are associated with. Builders that have fewer
+# valid worker options should always be assigned first
+def prioritizeBuilders(buildmaster, builders):
+    # re-use the builder_to_workers list
+    builder_to_workers = config.builder_to_workers
+
+    # sort builders by the length of their worker lists. Since not all 
+    # builders are explicitly listed in builder_to_workers, make sure to
+    # default to the len() of the "default" value
+    builders.sort(key=lambda b: len(builder_to_workers.get(b.name)) if b.name in builder_to_workers.keys() else len(builder_to_workers.get("default")))
+    return builders
+
 def create_parent_builder_factory(buildername, waitname):
     factory = util.BuildFactory()
     # NOTE: Assumes that yocto-autobuilder repo has been cloned to home

master.cfg

@@ -88,6 +88,12 @@ c['www'] = www.www
 # These items are specific to an individual AB deployment
 c['workers'] = workers.workers
 
+# This enables our prioritizeBuilders function from builders.py.
+# Builders such as buildperf-centos7, buildperf-ubuntu1604,
+# oe-selftest-*, and reproducible-* will be assigned (if possible)
+# before other builders since their possible worker lists are smaller
+c['prioritizeBuilders'] = builders.prioritizeBuilders
+
 c['title'] = "Yocto Autobuilder"
 c['titleURL'] = "https://autobuilder.yoctoproject.org/main/"
 # visible location for internal web server