Willem de Bruijn 9ef5d4748d bpf: Adjust free target to avoid global starvation of LRU map ... [ Upstream commit d4adf1c9ee ] BPF_MAP_TYPE_LRU_HASH can recycle most recent elements well before the map is full, due to percpu reservations and force shrink before neighbor stealing. Once a CPU is unable to borrow from the global map, it will once steal one elem from a neighbor and after that each time flush this one element to the global list and immediately recycle it. Batch value LOCAL_FREE_TARGET (128) will exhaust a 10K element map with 79 CPUs. CPU 79 will observe this behavior even while its neighbors hold 78 * 127 + 1 * 15 == 9921 free elements (99%). CPUs need not be active concurrently. The issue can appear with affinity migration, e.g., irqbalance. Each CPU can reserve and then hold onto its 128 elements indefinitely. Avoid global list exhaustion by limiting aggregate percpu caches to half of map size, by adjusting LOCAL_FREE_TARGET based on cpu count. This change has no effect on sufficiently large tables. Similar to LOCAL_NR_SCANS and lru->nr_scans, introduce a map variable lru->free_target. The extra field fits in a hole in struct bpf_lru. The cacheline is already warm where read in the hot path. The field is only accessed with the lru lock held. Tested-by: Anton Protopopov <a.s.protopopov@gmail.com> Signed-off-by: Willem de Bruijn <willemb@google.com> Acked-by: Stanislav Fomichev <sdf@fomichev.me> Link: https://lore.kernel.org/r/20250618215803.3587312-1-willemdebruijn.kernel@gmail.com Signed-off-by: Alexei Starovoitov <ast@kernel.org> Signed-off-by: Sasha Levin <sashal@kernel.org>		2025-07-17 18:37:22 +02:00
..
libbpf	docs/bpf: Add missing BPF program types to docs	2024-09-12 10:56:41 -07:00
standardization	bpf, docs: Address comments from IETF Area Directors	2024-06-23 09:10:26 -07:00
bpf_design_QA.rst	docs/bpf: Add documentation for new instructions	2023-07-27 18:54:17 -07:00
bpf_devel_QA.rst	bpf: Add namespace to BPF internal symbols	2025-05-02 07:59:04 +02:00
bpf_iterators.rst	bpf, docs: BPF Iterator Document	2023-06-22 19:44:22 +02:00
bpf_licensing.rst	bpf: Document BPF licensing.	2021-09-22 23:13:13 +02:00
bpf_prog_run.rst	Documentation/bpf: Add documentation for BPF_PROG_RUN	2022-03-09 14:19:22 -08:00
btf.rst	docs/bpf: Add constant values for linkages	2024-09-11 13:47:04 -07:00
clang-notes.rst	bpf, docs: Explain helper functions	2023-03-10 13:02:00 -08:00
classic_vs_extended.rst	bpf, docs: Split the comparism to classic BPF from instruction-set.rst	2021-12-30 16:31:44 -08:00
cpumasks.rst	bpf: Add bpf_cpumask_weight() kfunc	2023-12-09 21:37:33 -08:00
drgn.rst	bpf: Document bpf_inspect drgn tool	2020-03-26 00:08:36 +01:00
faq.rst	bpf, docs: Fix ordering of bpf documentation	2021-11-17 23:24:01 +01:00
fs_kfuncs.rst	Documentation/bpf: Add documentation for filesystem kfuncs	2023-12-01 16:21:03 -08:00
graph_ds_impl.rst	Documentation: Fix typos	2023-08-18 11:29:03 -06:00
helpers.rst	bpf, docs: Fix ordering of bpf documentation	2021-11-17 23:24:01 +01:00
index.rst	Documentation/bpf: Add documentation for filesystem kfuncs	2023-12-01 16:21:03 -08:00
kfuncs.rst	bpf: treewide: Annotate BPF kfuncs in BTF	2024-01-31 20:40:56 -08:00
linux-notes.rst	bpf, docs: Move linux-notes.rst to root bpf docs tree	2023-08-30 16:35:44 +02:00
llvm_reloc.rst	docs/bpf: Fix "file doesn't exist" warnings in {llvm_reloc,btf}.rst	2023-09-01 15:56:48 +02:00
map_array.rst	docs/bpf: Fix sphinx warnings in BPF map docs	2022-11-24 01:05:04 +01:00
map_bloom_filter.rst	docs/bpf: Document BPF_MAP_TYPE_BLOOM_FILTER	2022-11-23 22:47:32 +01:00
map_cgroup_storage.rst	Remove duplicate words inside documentation	2022-09-27 13:21:43 -06:00
map_cgrp_storage.rst	docs/bpf: Add documentation for new cgroup local storage	2022-10-25 23:19:20 -07:00
map_cpumap.rst	docs/bpf: Fix sphinx warnings for cpumap	2022-11-23 22:38:53 +01:00
map_devmap.rst	docs/bpf: Fix sphinx warnings for devmap	2022-11-23 22:40:27 +01:00
map_hash.rst	bpf: Adjust free target to avoid global starvation of LRU map	2025-07-17 18:37:22 +02:00
map_lpm_trie.rst	bpf: Replace bpf_lpm_trie_key 0-length array with flexible array	2024-02-29 22:52:43 +01:00
map_lru_hash_update.dot	bpf: Adjust free target to avoid global starvation of LRU map	2025-07-17 18:37:22 +02:00
map_of_maps.rst	docs/bpf: Fix sphinx warnings in BPF map docs	2022-11-24 01:05:04 +01:00
map_queue_stack.rst	docs/bpf: Fix sphinx warnings in BPF map docs	2022-11-24 01:05:04 +01:00
map_sk_storage.rst	docs/bpf: Reword docs for BPF_MAP_TYPE_SK_STORAGE	2022-12-14 18:35:41 +01:00
map_sockmap.rst	sock: Remove ->sendpage*() in favour of sendmsg(MSG_SPLICE_PAGES)	2023-06-24 15:50:13 -07:00
map_xskmap.rst	Documentation: bpf: correct spelling	2023-01-28 12:22:20 -08:00
maps.rst	bpf, docs: Fix final bpf docs build failure	2023-03-02 22:36:10 -08:00
other.rst	bpf, documentation: Add graph documentation for non-owning refs	2023-02-13 19:40:53 -08:00
prog_cgroup_sockopt.rst	bpf: Document EFAULT changes for sockopt	2023-05-13 16:55:46 -07:00
prog_cgroup_sysctl.rst	bpf: Document BPF_PROG_TYPE_CGROUP_SYSCTL	2019-04-18 16:05:36 -07:00
prog_flow_dissector.rst	docs/bpf: update out-of-date doc in BPF flow dissector	2023-09-13 14:22:57 -07:00
prog_lsm.rst	lsm: move hook comments docs to security/security.c	2023-04-28 11:58:34 -04:00
prog_sk_lookup.rst	bpf: sk_lookup: Add user documentation	2020-08-24 14:46:50 -07:00
programs.rst	docs/bpf: Add table of BPF program types to libbpf docs	2022-11-23 13:31:20 -08:00
redirect.rst	bpf, docs: DEVMAPs and XDP_REDIRECT	2022-11-18 23:16:31 +01:00
ringbuf.rst	Documentation: bpf: correct spelling	2023-01-28 12:22:20 -08:00
s390.rst	bpf: Add s390 testing documentation	2019-10-30 16:25:31 +01:00
syscall_api.rst	bpf, docs: Fix ordering of bpf documentation	2021-11-17 23:24:01 +01:00
test_debug.rst	bpf, docs: Fix ordering of bpf documentation	2021-11-17 23:24:01 +01:00
verifier.rst	docs/bpf: Fix a typo in verifier.rst	2024-08-29 12:19:30 -07:00