poky/meta/classes-global
Alexander Kanavin 697eeef71b package_rpm: restrict rpm to 4 threads
TL;DR version:

with this, and the previous compression level changes
I am seeing drastic speedups in package_write_rpm completion times:

webkitgtk goes from 78 seconds to 37 seconds
glibc-locale goes from 399 seconds to 58 seconds (!)

The long version:

rpm uses multithreading for two purposes:

- spawning compressors (which are nowadays themselves
multi-threaded, so the feature is not as useful as it once
was)
- parallel file classification

While the former behaves well on massively parallel CPUs
(it was written and verified here :), the latter was then added
by upstream and only benchmarked on their very old, slow laptop,
apparently:
41f0e214f2

On anything more capable it starts showing pathologic behavior,
presumably from spawning massive amount of very short-lived threads,
and then having to synchronize them. For example classifying glibc-locale
takes
5m20s with 256 threads (default on my machine!)
1m49s with 64 threads
59s with 16 threads
48s with 8 threads

Even a more typical recipe like webkitgtk is affected:
47s with 256 threads
32s with 64 threads
27s with 16 or 8 threads

I have found that the optimal amount is actually four: this also
means that only four compressors are running at a time, but
as they're themselves using threads, and typical recipes are dominated
by just two or three large packages, this does not affect overall
completion time.

(From OE-Core rev: 286d456e71ee2730c197ce394d6be2c7eeced18d)

Signed-off-by: Alexander Kanavin <alex@linutronix.de>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2024-11-21 12:16:28 +00:00
..
base.bbclass lib/package: Check incompatible licenses at packaging time 2024-10-25 15:37:10 +01:00
buildstats.bbclass classes: Update classes to match new bitbake class scope functionality 2022-08-12 15:27:17 +01:00
debian.bbclass package/package_write: Improve packagedata code location 2023-11-08 10:56:35 +00:00
devshell.bbclass devshell: Do not add scripts/git-intercept to PATH 2023-02-10 09:25:44 +00:00
insane.bbclass insane: Ensure package_qa tasks run in builds when expected 2024-10-24 15:18:54 +01:00
license.bbclass classes-global/license: Move functions to library code 2024-10-25 15:37:10 +01:00
logging.bbclass classes: Update classes to match new bitbake class scope functionality 2022-08-12 15:27:17 +01:00
mirrors.bbclass binutils-cross-testsuite: Rename to binutils-testsuite 2024-06-27 13:03:34 +01:00
package_deb.bbclass package/package_write: Improve packagedata code location 2023-11-08 10:56:35 +00:00
package_ipk.bbclass ipk: Switch to using zstd compression 2023-12-23 08:46:00 +00:00
package_pkgdata.bbclass classes: Update classes to match new bitbake class scope functionality 2022-08-12 15:27:17 +01:00
package_rpm.bbclass package_rpm: restrict rpm to 4 threads 2024-11-21 12:16:28 +00:00
package.bbclass lib/package: Check incompatible licenses at packaging time 2024-10-25 15:37:10 +01:00
packagedata.bbclass classes: Update classes to match new bitbake class scope functionality 2022-08-12 15:27:17 +01:00
patch.bbclass patch.bbclass: Make use of oe.patch.GitApplyTree.commitIgnored() 2024-02-19 16:03:22 +00:00
retain.bbclass classes: add new retain class for retaining build results 2024-08-03 07:56:10 +01:00
sanity.bbclass sanity.bbclass: skip check_userns for non-local uid 2024-10-22 12:28:18 +01:00
sstate.bbclass sstate: rewrite sstate_archive_package in python 2024-11-18 22:09:03 +00:00
staging.bbclass classes-global/staging: Exclude do_create_spdx from automatic sysroot extension 2024-07-13 23:28:31 +01:00
uninative.bbclass uninative: Add pthread linking workaround 2024-03-19 15:25:12 +00:00
utility-tasks.bbclass classes: Update classes to match new bitbake class scope functionality 2022-08-12 15:27:17 +01:00
utils.bbclass utils.bbclass: Use objdump instead of readelf to compute SONAME 2024-07-01 13:56:02 +01:00