mm/mglru: don't use stack pointers when walking PUD and PMD tables

MGLRU passes stack copies of page-table entries in walk_pud_range() and
walk_pmd_range() when walking to the next level. These copies can become
stale when khugepaged or MADV_DONTNEED (CONFIG_PT_RECLAIM) removes a PTE
table concurrently.

This pattern dates back to commit bd74fdaea1 ("mm: multi-gen LRU:
support page table walks"). Likely, earlier locking masked the latent
bug. Later changes exposed it: commit 1d65b771bc ("mm/khugepaged:
retract_page_tables() without mmap or vma lock") allowed concurrent
retraction, and commit cf4a65539c ("memcg-v1: no need for memcg locking
for MGLRU") dropped the walker's outer RCU protection.

The fix is simple, so just fix the original commit so we don't have to
speculate. To fix it, pass the actual PMD/PUD pointers. This restores the
RCU-protected lookup and validation against the live PMD. Note that the PUD
handling is required as well to handle folded page tables correctly in all
cases.

Fix generated by an LLM and sent by Josef Bacik in private.

[ david: handle walk_pud_range() as well, clarify+simplify description,
         adjust Fixes: ]

Fixes: bd74fdaea1 ("mm: multi-gen LRU: support page table walks")
Cc: [email protected]
Signed-off-by: Josef Bacik <[email protected]>
Reviewed-by: Rik van Riel <[email protected]>
Reviewed-by: Qi Zheng <[email protected]>
Reviewed-by: Dev Jain <[email protected]>
Reviewed-by: Lorenzo Stoakes (ARM) <[email protected]>
Reviewed-by: Kairui Song <[email protected]>
Tested-by: Muhammad Usama Anjum <[email protected]>
Reviewed-by: Harry Yoo (Meta) <[email protected]>
Reviewed-by: Barry Song <[email protected]>
Reviewed-by: Baolin Wang <[email protected]>
Reviewed-by: Pedro Falcato <[email protected]>
Reviewed-by: Kiryl Shutsemau (Meta) <[email protected]>
Reviewed-by: SJ Park <[email protected]>
Link: https://patch.msgid.link/[email protected]
Reviewed-by: Baoquan He <[email protected]>
Signed-off-by: David Hildenbrand (Arm) <[email protected]>
This commit is contained in:
Josef Bacik
2026-11-02 11:21:28 +01:00
committed by David Hildenbrand (Arm)
parent 59511d630a
commit 65a9bf08a3
+2 -2
View File
@@ -3911,7 +3911,7 @@ restart:
walk->mm_stats[MM_NONLEAF_FOUND]++;
if (!walk_pte_range(&val, addr, next, args))
if (!walk_pte_range(pmd + i, addr, next, args))
continue;
walk->mm_stats[MM_NONLEAF_ADDED]++;
@@ -3947,7 +3947,7 @@ restart:
if (!pud_present(val) || WARN_ON_ONCE(pud_leaf(val)))
continue;
walk_pmd_range(&val, addr, next, args);
walk_pmd_range(pud + i, addr, next, args);
if (need_resched() || walk->batched >= MAX_LRU_BATCH) {
end = (addr | ~PUD_MASK) + 1;