MIRRORS/linux-yocto
1
0
Fork 0
mirror of git://git.yoctoproject.org/linux-yocto.git synced 2025-10-23 07:23:12 +02:00
Code Issues Actions Packages Projects Releases Wiki Activity

gup: optimize longterm pin_user_pages() for large folio

Browse Source 
commit a03db236ae upstream.

In the current implementation of longterm pin_user_pages(), we invoke
collect_longterm_unpinnable_folios().  This function iterates through the
list to check whether each folio belongs to the "longterm_unpinnabled"
category.  The folios in this list essentially correspond to a contiguous
region of userspace addresses, with each folio representing a physical
address in increments of PAGESIZE.

If this userspace address range is mapped with large folio, we can
optimize the performance of function collect_longterm_unpinnable_folios()
by reducing the using of READ_ONCE() invoked in
pofs_get_folio()->page_folio()->_compound_head().

Also, we can simplify the logic of collect_longterm_unpinnable_folios().
Instead of comparing with prev_folio after calling pofs_get_folio(), we
can check whether the next page is within the same folio.

The performance test results, based on v6.15, obtained through the
gup_test tool from the kernel source tree are as follows.  We achieve an
improvement of over 66% for large folio with pagesize=2M.  For small
folio, we have only observed a very slight degradation in performance.

Without this patch:

    [root@localhost ~] ./gup_test -HL -m 8192 -n 512
    TAP version 13
    1..1
    # PIN_LONGTERM_BENCHMARK: Time: get:14391 put:10858 us#
    ok 1 ioctl status 0
    # Totals: pass:1 fail:0 xfail:0 xpass:0 skip:0 error:0
    [root@localhost ~]# ./gup_test -LT -m 8192 -n 512
    TAP version 13
    1..1
    # PIN_LONGTERM_BENCHMARK: Time: get:130538 put:31676 us#
    ok 1 ioctl status 0
    # Totals: pass:1 fail:0 xfail:0 xpass:0 skip:0 error:0

With this patch:

    [root@localhost ~] ./gup_test -HL -m 8192 -n 512
    TAP version 13
    1..1
    # PIN_LONGTERM_BENCHMARK: Time: get:4867 put:10516 us#
    ok 1 ioctl status 0
    # Totals: pass:1 fail:0 xfail:0 xpass:0 skip:0 error:0
    [root@localhost ~]# ./gup_test -LT -m 8192 -n 512
    TAP version 13
    1..1
    # PIN_LONGTERM_BENCHMARK: Time: get:131798 put:31328 us#
    ok 1 ioctl status 0
    # Totals: pass:1 fail:0 xfail:0 xpass:0 skip:0 error:0

[lizhe.67@bytedance.com: whitespace fix, per David]
  Link: https://lkml.kernel.org/r/20250606091917.91384-1-lizhe.67@bytedance.com
Link: https://lkml.kernel.org/r/20250606023742.58344-1-lizhe.67@bytedance.com
Signed-off-by: Li Zhe <lizhe.67@bytedance.com>
Cc: David Hildenbrand <david@redhat.com>
Cc: Dev Jain <dev.jain@arm.com>
Cc: Jason Gunthorpe <jgg@ziepe.ca>
Cc: John Hubbard <jhubbard@nvidia.com>
Cc: Muchun Song <muchun.song@linux.dev>
Cc: Peter Xu <peterx@redhat.com>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
Signed-off-by: Greg Kroah-Hartman <gregkh@linuxfoundation.org>
This commit is contained in:
Li Zhe 2025-06-06 10:37:42 +08:00 committed by Greg Kroah-Hartman
parent 3958f9ec72
commit 163843e8c8
1 changed files with 30 additions and 8 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

38
mm/gup.c
View File

@ -2300,6 +2300,31 @@ static void pofs_unpin(struct pages_or_folios *pofs)
unpin_user_pages(pofs->pages, pofs->nr_entries); unpin_user_pages(pofs->pages, pofs->nr_entries);
} }
static struct folio *pofs_next_folio(struct folio *folio,
struct pages_or_folios *pofs, long *index_ptr)
{
long i = *index_ptr + 1;
if (!pofs->has_folios && folio_test_large(folio)) {
const unsigned long start_pfn = folio_pfn(folio);
const unsigned long end_pfn = start_pfn + folio_nr_pages(folio);
for (; i < pofs->nr_entries; i++) {
unsigned long pfn = page_to_pfn(pofs->pages[i]);
/* Is this page part of this folio? */
if (pfn < start_pfn || pfn >= end_pfn)
break;
}
}
if (unlikely(i == pofs->nr_entries))
return NULL;
*index_ptr = i;
return pofs_get_folio(pofs, i);
}
/* /*
* Returns the number of collected folios. Return value is always >= 0. * Returns the number of collected folios. Return value is always >= 0.
*/ */
@ -2307,16 +2332,13 @@ static unsigned long collect_longterm_unpinnable_folios(
struct list_head *movable_folio_list, struct list_head *movable_folio_list,
struct pages_or_folios *pofs) struct pages_or_folios *pofs)
{ {
unsigned long i, collected = 0; unsigned long collected = 0;
struct folio *prev_folio = NULL;
bool drain_allow = true; bool drain_allow = true;
struct folio *folio;
long i = 0;
for (i = 0; i < pofs->nr_entries; i++) { for (folio = pofs_get_folio(pofs, i); folio;
struct folio *folio = pofs_get_folio(pofs, i); folio = pofs_next_folio(folio, pofs, &i)) {
if (folio == prev_folio)
continue;
prev_folio = folio;
if (folio_is_longterm_pinnable(folio)) if (folio_is_longterm_pinnable(folio))
continue; continue;