MIRRORS/poky
1
0
Fork 0
mirror of git://git.yoctoproject.org/poky.git synced 2025-07-19 21:09:03 +02:00
Code Issues Actions Packages Projects Releases Wiki Activity

oe/utils: by default cap cpu_count() to 64 cores

Browse Source 
Larger systems may have large numbers of cores, but beyond a certain
point they can't all be used for compiling: whilst purely
compute-intensive jobs can be parallelised to hundreds of cores,
operations such as compressing (needs lots of RAM) or compiling (lots of
I/O) don't scale linearly.

For example, the Marvel ThunderX2 has 32 cores, each capable of
executing four threads, and can be configured with two sockets, making
256 CPUs according to Linux. Zstd using 256 threads has been seen to
fail to allocate memory during even small recipes such as iso-codes.

Add a default cap of 64 CPUs to the cpu_count() method so that extreme
parallisation is limited.  64 is high enough that meaningful gains
beyond it are unlikely, but high enough that most systems won't be
effected.

(From OE-Core rev: 765d0f25ce48636b1838a5968e2dc15de2127428)

Signed-off-by: Ross Burton <ross.burton@arm.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
This commit is contained in:
Ross Burton 2021-11-24 17:15:29 +00:00 committed by Richard Purdie
parent 317511ce1f
commit c6f23f1f0f
1 changed files with 2 additions and 2 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

4
meta/lib/oe/utils.py
View File

@ -248,9 +248,9 @@ def trim_version(version, num_parts=2):
trimmed = ".".join(parts[:num_parts]) trimmed = ".".join(parts[:num_parts])
return trimmed return trimmed
def cpu_count(at_least=1): def cpu_count(at_least=1, at_most=64):
cpus = len(os.sched_getaffinity(0)) cpus = len(os.sched_getaffinity(0))
return max(cpus, at_least) return max(min(cpus, at_most), at_least)
def execute_pre_post_process(d, cmds): def execute_pre_post_process(d, cmds):
if cmds is None: if cmds is None: