mirror of
https://github.com/torvalds/linux.git
synced 2026-11-20 13:03:48 +00:00
page_ref_add_unless_zero() reads the refcount and then updates it with a
cmpxchg. A lockless get, such as a page cache lookup or a PFN scanner,
can read a non-zero refcount from a page that is then freed and reused as
a hugetlb tail page. If HVO remaps its struct page to the shared tail
page in between, the cmpxchg writes to read-only vmemmap:
CPU0 (lockless get) CPU1
c = atomic_read(&page->_refcount)
page freed
page allocated as a hugetlb tail
HVO remaps the struct page RO
try_cmpxchg(&page->_refcount, &c, c + 1)
-> write fault on read-only vmemmap
The refcount is zero by then and the compare fails, but a compare-and-swap
instruction needs write access whether or not it succeeds. The kernel
oopses when hugetlb pages are allocated at runtime with HVO enabled while
a lockless get races with it.
Commit bd225530a4 ("mm/hugetlb_vmemmap: fix race with speculative PFN
walkers") closed the window with RCU on both sides. The RCU was removed
together with fake heads, on the wrong belief that it was only needed for
them.
Map the shared tail page writable instead of restoring the RCU. Every
struct page on it is a tail with a zero refcount, so a late cmpxchg
fails and leaves the page unchanged. RCU would put rcu_read_lock()
back around every get and synchronize_rcu() back into hugetlb
allocation and freeing.
The cost is the protection against stray writes to tail struct pages. The
shared page is shared by all hugetlb folios of the same order in the zone,
so such a write now corrupts all of them instead of oopsing.
Bootmem folios optimized when the memmap is set up keep their tails
read-only. Nothing does a lockless get on them before hugetlb takes them
over.
Link: https://lore.kernel.org/[email protected]
Fixes: 32c440d67e ("mm: drop fake head checks")
Fixes: 01b1d0ffb6 ("hugetlb: remove VMEMMAP_SYNCHRONIZE_RCU")
Signed-off-by: Kiryl Shutsemau (Meta) <[email protected]>
Signed-off-by: Andrew Morton <[email protected]>
Reported-by: David Hildenbrand (Arm) <[email protected]>
Closes: https://lore.kernel.org/r/[email protected]
Acked-by: Muchun Song <[email protected]>
Reviewed-by: Oscar Salvador <[email protected]>
Cc: Peter Zijlstra <[email protected]>
Cc: Frank van der Linden <[email protected]>
Cc: Matthew Wilcox (Oracle) <[email protected]>
Cc: Lorenzo Stoakes <[email protected]>
Cc: Liam R. Howlett <[email protected]>
Cc: Vlastimil Babka <[email protected]>
Cc: Mike Rapoport <[email protected]>
Cc: Suren Baghdasaryan <[email protected]>
Cc: Michal Hocko <[email protected]>
Cc: Jonathan Corbet <[email protected]>
Cc: Shuah Khan <[email protected]>
Cc: Randy Dunlap <[email protected]>
Cc: <[email protected]>