mirror of
https://github.com/torvalds/linux.git
synced 2026-11-20 13:03:48 +00:00
Merge tag 'mm-stable-2026-10-19-21-33' of git://git.kernel.org/pub/scm/linux/kernel/git/akpm/mm
Pull MM updates from Andrew Morton:
- "selftests/mm: improve khugepaged coverage" (Kiryl Shutsemau):
Give khugepaged a proper workout. Extend the tests to all supported
mTHP orders, check exactly which windows collapsed, and make scan
completion deterministic. Race collapse against faults, pinning, fork
and remap; verify that the contents survive.
Add coverage for whole-page-table zaps and memory pressure too.
- "selftests/mm: separate GUP microbenchmarking from functional testing"
(Sarthak Sharma):
Untangle GUP benchmarking from correctness testing. Move the
benchmark into tools/mm, add a standalone harness-based GUP selftest,
and move shared helpers into tools/lib/mm. Give each tool one job
instead of mixing timing measurements with correctness checks.
Cover the mapping, pinning and batch-size combinations explicitly.
- "mm: Switch device DAX to section-based vmemmap optimization" (Muchun
Song):
Apply section-based vmemmap optimization to device DAX. Rework the
backing metadata layout to save memory while preserving the handling
of device memory hotplug and its different alignment and lifetime
requirements.
- "mm: Introduce section-based vmemmap optimization for HugeTLB"
(Muchun Song):
Cut the memory consumed by HugeTLB's struct page array with
section-based vmemmap optimization. Arrange the metadata so large
huge pages need fewer backing pages, while keeping initialization,
teardown and the various hugepage configurations straight.
- "mm/huge_memory: clean up and decouple the anon and file split
helpers" (Kairui Song):
Separate anonymous and file-backed folio splitting, which have rather
different locking and bookkeeping requirements. Move the work into
separate helpers, drop dead checks and tidy swap-cache handling. Fix
error paths, including one that could leave a folio frozen.
Keep the swap cache cases separate from file-cache splitting.
- "mm: Unconditional per-VMA locks and cleanups" (Dave Hansen):
Make per-VMA locking unconditional and clean up the old conditional
code around it. This removes configuration-dependent plumbing and
lets callers depend on one locking model rather than carrying both
cases along.
- "memcg: remove the v1 soft limit" (Shakeel Butt):
Remove the old cgroup v1 memory soft-limit machinery. This has long
since been superseded by the newer memory-control interfaces; hauling
around a separate reclaim scheme for it no longer buys us much.
- "read proc/pid/smaps_rollup under per-vma lock" (Suren Baghdasaryan):
Read /proc/PID/smaps_rollup using per-VMA locks where possible,
rather than holding mmap_lock across the entire walk. Retain the
contention fallback and add tearing tests. This considerably reduces
latency when another thread is busy changing its mappings.
- "mm/damon: introduce data access-as-a-data attribute" (SJ Park):
Teach DAMON to expose memory access observations as data attributes,
making them available to the probe and filtering machinery. Extend
the interfaces, documentation and tests so the new information can be
used rather than merely collected.
- "Remove PG_private by using page/folio->private checks instead" (Zi
Yan):
Get rid of PG_private and use the folio's private pointer to say
whether private data is attached. Convert the filesystem users,
including erofs, f2fs and md, and clean up the old accessors and
flags. There are a few special cases, notably swapcache and hugetlb,
which need explicit care.
- "mm: make userland page table freeing RCU-safe" (Lorenzo Stoakes):
Make freeing userspace page tables safe against RCU readers. Sort out
the page-table lifetime and locking rules so walkers cannot stumble
over freed tables while mappings are being torn down.
- "make unused huge shrinker memcg aware" (Qi Zheng):
Make shmem's unused-huge-folio shrinker aware of memory cgroups, so
reclaim can find those folios under the right memcg. Wire up the
per-memcg tracking and its lifetime rules, and give the shrinklist
its own lockdep class to avoid spurious dependency reports.
- "mm: Drop pxd_ERROR()" (Anshuman Khandual):
Remove the old pxd_ERROR() page-table checks and their
architecture-specific baggage. Simplify the common page-table
interfaces and callers now that these checks no longer serve a
useful purpose.
- "memcg: group struct fields by access pattern" (Shakeel Butt):
Rearrange memcg's hot and cold fields according to how they're
actually accessed. Keep frequently updated counters and their
neighboring state from fighting over cachelines and reduce needless
cacheline traffic.
- "mm/damon/core: fix the size charged for a filter-trimmed region"
(Donggeun Yoo):
Fix DAMOS quota accounting when a filter trims a monitoring region.
Charge only the part left after filtering, not the original region,
and add KUnit coverage. Previously the excluded memory could eat into
the quota and make schemes run more slowly than requested.
- "mm/damon: add kunit and selftests for probes and probe weights" (SJ
Park):
Add more tests for DAMON probes and their weights, including
validation and the results reported to userspace. This is mostly
about catching regressions in the newer probe infrastructure before
they become harder to diagnose.
- "mm: memcontrol: constify the read side of the read side of the memcg
API" (Tal Zussman):
Mark the read-only parts of the memcg interface const, from folio and
lruvec accessors to counters and swap helpers. Most of this is
mechanical, but it makes the contracts clearer and lets the compiler
catch accidental writes.
- "mm/damon/vaddr: support {prep,apply}_probes" (SJ Park):
Support the new prepare/apply probe operations in DAMON's
virtual-address backend. This brings vaddr monitoring into line with
the probe infrastructure and makes the added probe types useful there
too.
- "mm/mglru: clean up folio counters and flag usage" (Kairui Song):
Simplify MGLRU's folio counters and flag handling. Remove redundant
accounting paths and make the state transitions easier to follow
without changing the intended reclaim policy.
- "arch, mm/execmem: resolve confusion about set_direct_map_valid_noflush()"
(Mike Rapoport (Microsoft)):
Clarify and straighten out the set_direct_map_valid_noflush()
interface across architectures and execmem. Make the requested
mapping changes and the callers' expectations agree, rather than
relying on subtly different architecture behavior.
- "mm: zswap: free cold writeback folios promptly" (Alexandre Ghiti):
Free zswap writeback folios as soon as their disk write completes.
They are cold temporary buffers, so putting them on the LRU only
makes reclaim rediscover them later. Extend swap-cache dropbehind and
let zswap keep these folios off the LRU altogether.
- "mm: gup: cleanup gup_fast call chain" (Aristeu Rozanski):
Clean up the GUP-fast call chain and simplify how the fast-path
helpers pass their arguments and results around. Keep the page-pin
behavior the same while making the call sequence easier to maintain.
- "mm: optimize zone-device memmap initialization" (Li Zhe):
Speed up initialization of zone-device struct pages. Avoid redundant
work when setting up large device-memory ranges, where touching every
descriptor can get expensive.
- "mm/zswap: shrink zswap_entry via a pool id" (Jianyue Wu):
Reduce the size of zswap_entry by storing a compact pool identifier
instead of a full pointer. Keep the pool lookup and lifetime handling
correct while saving per-entry metadata.
- "mm/damon: support access monitoring of hugetlb-backed memory"
(Krishna Iyer):
Let DAMON monitor hugetlb-backed memory too. Extend the relevant
access checks and add coverage so those mappings no longer fall
through a hole in the monitoring machinery.
- "mm/execmem: fixes and cleanups for the ROX cache" (Mike Rapoport):
Tidy execmem's read-only-executable cache and fix its handling of the
awkward allocation and release cases. The aim is to make the cache's
ownership and permission transitions more predictable.
- "zsmallc: remove old object read API" (Sergey Senozhatsky):
Drop zsmalloc's old object-read API and move its remaining users to
the supported access mechanism. This removes a duplicate interface
which we no longer need.
- "mm/mempolicy: stop copying state in the interleave paths" (Gregory
Price):
Stop copying mempolicy state in the interleave paths when a reference
to the existing policy will do. This avoids extra work and makes the
ownership rules less obscure.
- "mm/mglru: speed up inc_min_seq() and fix cold/hot inversions" (Barry
Song):
Make MGLRU's generation advancement cheaper and fix cases where cold
and hot folios end up in the wrong order. Adjust the counter and list
handling so reclaim doesn't keep passing over the better eviction
candidates.
- "mm: Fix MF_DELAYED handling on memory failure" (Lisa Wang):
Treat MF_DELAYED as a partially completed memory-failure operation,
not an outright failure. Propagate that result correctly through
shmem and the generic handler, fix the extra-reference accounting,
and add tests for the resulting SIGBUS behavior.
- "tools/mm/page_owner_sort: fix --sort, add module filter, improve
usage" (Ye Liu):
Fix page_owner_sort's --sort handling, add module-based filtering and
make the usage output more helpful. This makes the tool less
frustrating when tracking down who is holding on to pages.
- "mm/damon: move damos filter range arguments validation to core" (SJ
Park):
Move DAMOS filter range validation into the core, where all callers
can share it. Also close the sysfs read-twice races which could pass
inconsistent ranges, and add tests for invalid input.
- "mm/damon: misc cleanups" (Zenghui Yu):
Clean up a collection of small DAMON rough edges in the core,
interfaces and tests. No large behavior change here, just simpler
code and fewer opportunities for confusion.
- "mm/damon: cleanup code, add test cases, and update guidances in
docs" (SJ Park):
Clean up DAMON's core and interfaces, add missing test cases and
bring the documentation into line with the code. These changes should
make the newer facilities easier to understand and maintain.
- "mm/damon: introduce hugepage_size probe filter" (SJ Park):
Add a DAMON probe filter for hugepage size. Let users select a range
of folio sizes through sysfs, validate the range and document the new
interface, so monitoring can concentrate on memory backed by the
desired hugepage sizes.
- "selftests/mm: pagemap_ioctl test fixes and cleanups" (Zenghui Yu):
Fix some pagemap ioctl test assumptions and clean up their setup and
reporting. Make failures point to the operation that went wrong
rather than to test harness bookkeeping.
- "mm: avoid large folio splits when swap is unavailable" (Johannes
Weiner):
Avoid splitting large folios merely to attempt swap when no swap
space is available. There is little point paying for an expensive
split when reclaim cannot actually swap the resulting pages out.
- "mm: workingset: fix the shadow node budget under MGLRU" (Hui Zhu):
Fix the workingset shadow-node budget with MGLRU enabled. Keep the
shadow entries within the intended limits so refault tracking does
not quietly consume more metadata than it should.
- "selftests/mm: TAP output and global-state fixes" (Song Hu):
Fix selftests which produce misleading TAP output or fail to restore
global settings after a run. A test ought to leave the system as it
found it, particularly when a whole suite follows behind.
- "mm: remove page_swap_entry()" (Tal Zussman):
Remove page_swap_entry() and convert its remaining users to the folio
and swap-entry helpers. Lose an old page-level interface and make the
swap paths a little more consistent.
- "mm: refactor zonelist constructors and iterators" (Gregory Price):
Refactor the zonelist construction and walk helpers to reduce
duplicated logic. Keep the allocation ordering rules intact while
making it clearer how zones are selected.
- "mm: kmemleak: batch the struct page scan" (Breno Leitao):
Scan adjacent struct pages in batches instead of taking kmemleak_lock
once for every single page. This preserves scan coverage and the
lock-hold bound, while cutting overhead substantially on debug
kernels.
- "mm/damon: misc improvements in tests and documents" (Zenghui Yu):
Fix some DAMON selftests which leave kdamond running or fail under
Python safe-path mode, and improve the KUnit coverage and
documentation. Nothing especially glamorous, but these are worth
doing.
- "mm/damon: introduce probe_hits_wsum DAMOS core filter" (SJ Park):
Add a DAMOS core filter using weighted probe-hit counts. This gives
schemes another way to select regions based on what the monitoring
probes observed.
- "mm/mglru: fix ineffective memory protection for non-kswapd reclaim"
(Ridong Chen):
Fix MGLRU's memory protection handling for reclaim outside kswapd.
The protection settings weren't having the intended effect in these
paths, leaving protected memory more exposed than requested.
- "mm/damon: improve readability, clarity and test coverage" (SJ Park):
Simplify a handful of DAMON core helpers, clarify their behavior and
expand test coverage. There are no intended userspace-visible changes
in this batch.
- "mm: replace PF_KCOMPACTD/PF_KSWAPD with kthread_func()" (Kefeng
Wang):
Stop identifying reclaim and compaction workers through the PF_KSWAPD
and PF_KCOMPACTD task flags. Use their kthread functions instead and
remove the redundant flag plumbing.
- "mm/hugetlb: fix surplus accounting and availability checks during
demotion" (Longlong Xia):
Fix HugeTLB surplus accounting and availability checks when demoting
huge pages. Keep the counters and reservation decisions consistent as
one huge page turns into several smaller ones.
- "mm/damon: add kunit tests for probe_hits handling and probe params
validation" (Jason Angelov):
Add KUnit tests for DAMON probe-hit handling and invalid probe
parameters. These cover edge cases that were too easy to miss in
ordinary monitoring tests.
- "mm/damon: fix misc bugs in kunit, quota goals and sysfs refresh_ms"
(SJ Park):
Fix a few DAMON bugs in KUnit tests, quota-goal handling and the
sysfs refresh interval. Small fixes, but the incorrect results and
control behavior were real.
- "mm: zswap: optimize zswap invalidate and store" (Kefeng Wang):
Simplify zswap's invalidate and store paths to avoid avoidable
lookups and work. Keep the entry lifetime rules intact while making
those frequent operations cheaper.
- "mm: replace custom ratelimiting logic" (Pedro Falcato):
Replace hand-rolled MM rate limiting with the common kernel helpers.
There is no need to maintain another local version of the same
mechanism.
- "mm/vmalloc: minor cleanups" (Ye Liu):
Tidy some vmalloc initialization and reporting code, and pull
duplicated vmap-area insertion work into a helper. No functional
changes are intended.
- "mm/mglru: clean up isolate_folios for readability and clarity"
(Ridong Chen):
Clean up MGLRU's isolate_folios() so the decisions are easier to read
and verify. Keep reclaim behavior unchanged while removing
unnecessary twists in the control flow.
- "Catch automatic storage in IDA and Maple Tree definitions" (Yury
Norov):
Catch IDA and maple-tree definitions placed in automatic storage
where the lifetime rules are wrong or easy to get wrong. Add checks
so such mistakes are harder to slip through unnoticed.
- "mm, swap: some random fixes and cleanups" (Kemeng Shi):
Fix and simplify a few unrelated swap paths, removing redundant work
and tidying up error handling. A small collection of things which
were better fixed than left lying around.
- "mm/page_io: folio conversion cleanups" (Tal Zussman):
Finish some page-to-folio conversions in swap I/O, removing obsolete
wrappers and using folio operations consistently. The I/O behavior is
intended to remain the same.
- "mm/swap: skip empty clusters in the swapoff scan" (Youngjun Park):
Skip empty swap clusters during swapoff instead of walking slots
which have nothing to release. This avoids pointless work on sparsely
used swap devices.
- "mm: trace: decode architecture-specific VMA flags" (Meijing Zhao):
Teach the VMA trace output to decode architecture-specific flags,
rather than leaving them as unexplained bits. That should make
tracing a good deal more useful when debugging unusual mappings.
- "mm, swap: don't spin on a bad swap entry" (Breno Leitao):
Stop repeatedly retrying a swap entry which is already known to be
bad. Make the error path terminate cleanly instead of wasting CPU
chasing an entry that cannot be used.
- "mm/page_isolation: fix UBSAN shift-out-of-bounds in
isolate_single_pageblock" (Qi Xi):
Fix an out-of-range shift in isolate_single_pageblock() and tighten
its associated boundary handling. UBSAN found the issue; the result
should be more robust pageblock isolation.
- "mm: khugepaged: fix tracepoint UAF" (Vernon Yang):
Fix a use-after-free in khugepaged tracepoint handling. Make sure the
information recorded by tracing stays valid for as long as the trace
event needs it.
- "mm/damon: fix DAMOS bugs in core, paddr and vaddr" (SJ Park):
Fix several DAMOS problems across the core and physical- and
virtual-address backends. The changes close error cases and bring the
implementations back into agreement.
- "mm/hugetlb_cgroup: move the per-node usage along with the folio"
(Hongfu Li):
Move HugeTLB per-node usage accounting when folios migrate between
nodes or are reparented to another cgroup. Otherwise numa_stat can
retain stale charges and even underflow when the folio is freed.
- "docs/core-api: memory-allocation: add k[mz]alloc_obj() and clarify
kmalloc" (Mike Rapoport):
Document the typed k[mz]alloc_obj() allocation helpers and clear up
some kmalloc guidance. Make it easier for callers to choose the right
interface.
- "selftests/mm: Validate selections and scope memfd_secret setup"
(Tianyi Chen):
Check selftest selections before running and confine memfd_secret
setup to the cases that actually need it. Avoid confusing failures
caused by the test harness itself.
- "mm/damon: move zero damos quota target_value handling to the core
layer" (SJ Park):
Handle a zero DAMOS quota goal target in the core rather than making
each user of the interface invent its own rule. Keep quota decisions
consistent across callers.
- "mm/damon: Introduce a huge page collapsing mechanism using auto
tuning" (Asier Gutierrez):
Add an auto-tuned DAMON mechanism for collapsing huge pages. Use
monitoring feedback to decide when collapse is worthwhile rather than
forcing one fixed policy onto every workload.
- "kmemleak: fix stale documentation and raise the verbose default"
(Breno Leitao):
Bring the kmemleak documentation up to date and require a third
confirming scan before verbose kernels print a suspected leak. The
extra scan helps suppress transient false positives in noisy debug
builds.
- "mm/memory: improve insert_pages() error handling" (Avi Weiss):
Tighten insert_pages() error handling so partial failures are
reported and cleaned up consistently. The caller should not have to
guess how far the operation got.
- "mm/damon: allow NULL or unstarted damon_ctx parameter for
damon_call()" (SJ Park):
Make damon_call() handle NULL and not-yet-started contexts safely.
Simplify the rules for callers and avoid an indefinite wait on an
invalid state.
- "mm/damon: introduce pgidle_set probe filter type" (SJ Park):
Add a pgidle_set DAMON probe filter, giving monitoring users another
way to select pages by their idle-tracking state. Wire it through the
filter interfaces and tests.
- "mm/damon/core: remove unused helper functions" (Cheng-Han Wu):
Remove DAMON core helpers which no longer have callers. Keep the API
surface smaller rather than retaining dead code just in case.
- "mm: stop calling pmd_folio() on special PMDs" (Gregory Price):
Avoid calling pmd_folio() on special PMD mappings that have no struct
page. The old assumption could fault in procfs walkers; use the
normal-folio checks and handle those mappings properly.
- "mm: remove three unused helpers from mm.h" (Tal Zussman):
Remove three unused helpers from mm.h. Fewer obsolete interfaces for
the next person to stumble over.
- "mm/damon: improvements in efficiency, error handling, documents"
(Liew Rui Yan):
Skip needless DAMOS quota work once the quota is full, propagate a
missing sysfs error and tidy a couple of comments. The ignored error
could make a monitoring-start request appear successful when it
wasn't.
- "docs/mm/damon/design: add explanation of nr_snapshots" (Liew Rui
Yan):
Explain DAMON's nr_snapshots accounting in the design documentation
and straighten out the associated terminology. Make it clearer what
users should expect from that counter.
- "selftests/cgroup: fixes for test_zswap on single core VM" (Wilson
Felipe Pereira):
Fix two test_zswap failures on single-CPU VMs. Wait for the exiting
child's cgroup to become unpopulated before reconfiguring it, and
avoid another test assumption which breaks under constrained
scheduling.
- "docs/ksm: fix advisor documentation and comment" (Jinjiang Tu):
Correct the KSM advisor documentation and a misleading code comment.
Keep the description aligned with how the advisor actually works.
- "Fix bugs in HugeTLB allocation when mem_cgroup_charge_hugetlb()
fails" (Ackerley Tng):
Fix HugeTLB allocation cleanup when the memcg charge fails. Make sure
failure exits undo what was allocated rather than leaving stale
accounting or resources behind.
- "mm, memcg: fix memory.peak reset clobbering other fds' watermark"
(Ridong Chen):
Fix memory.peak reset handling so resetting one file descriptor
doesn't overwrite the watermark kept for another. Keep those per-open
measurements independent as intended.
- "mm: reject zone device folios in more folio walkers" (Gregory
Price):
Reject zone-device folios in page walkers that cannot safely handle
them. Extend the existing checks rather than assuming every folio has
ordinary system-memory backing.
- "percpu: remove code with no effect" (Sang-Heon Jeon):
Remove a few pieces of percpu code which no longer do anything. There
is no reason to keep maintaining dead paths.
* tag 'mm-stable-2026-10-19-21-33' of git://git.kernel.org/pub/scm/linux/kernel/git/akpm/mm: (539 commits)
mm: shmem: assign a lock class to the unused huge shrinklist
mm: vmscan: put rotation-missed folios at the LRU tail
mm: kmemleak: scan the struct page array in MAX_SCAN_SIZE batches
mm: kmemleak: move the struct page scan into a helper
mm/damon: document that a zero sample_interval is accepted
mm/damon: fix typos in comments
mm/damon/sysfs: propagate damon_call() error in turn_damon_on
mm/damon/core: skip quota score setup when the quota is full
mm/damon/tests/core-kunit: test the size charged for a filter-trimmed region
mm/damon/core: charge only the part of a region the filter left
mm: disallow raw PFN mappings of huge/shared zeropage
selftests/mm: zap whole PTE tables in the khugepaged race harness
selftests/mm: add memory-pressure threads to the khugepaged race harness
selftests/mm: race the collapse of windows with holes
selftests/mm: add khugepaged race harness
selftests/mm: check that one khugepaged pass collapses one window
selftests/mm: run every supported collapse order by default
selftests/mm: cover a shared-source collapse write race
selftests/mm: parameterize the mixed-source collapse case by source order
selftests/mm: add order-parameterized khugepaged collapse cases
...
This commit is contained in:
@@ -3,7 +3,7 @@ Date: Mar 2022
|
||||
Contact: SJ Park <[email protected]>
|
||||
Description: Interface for Data Access MONitoring (DAMON). Contains files
|
||||
for controlling DAMON. For more details on DAMON itself,
|
||||
please refer to Documentation/admin-guide/mm/damon/index.rst.
|
||||
please refer to Documentation/mm/damon/index.rst.
|
||||
|
||||
What: /sys/kernel/mm/damon/admin/
|
||||
Date: Mar 2022
|
||||
@@ -173,6 +173,19 @@ Contact: SJ Park <[email protected]>
|
||||
Description: Writing to and reading from this file sets and gets the
|
||||
per-probe attribute weight.
|
||||
|
||||
What: /sys/kernel/mm/damon/admin/kdamonds/<K>/contexts/<C>/monitoring_attrs/probes/<P>/preps/nr_preps
|
||||
Date: Jun 2026
|
||||
Contact: SJ Park <[email protected]>
|
||||
Description: Writing a number 'N' to this file creates the number of
|
||||
directories for each DAMON probing preparation action named '0'
|
||||
to 'N-1' under the preps/ directory.
|
||||
|
||||
What: /sys/kernel/mm/damon/admin/kdamonds/<K>/contexts/<C>/monitoring_attrs/probes/<P>/preps/<R>/prep_action
|
||||
Date: Jun 2026
|
||||
Contact: SJ Park <[email protected]>
|
||||
Description: Writing to and reading from this file sets and gets the probing
|
||||
preparation action.
|
||||
|
||||
What: /sys/kernel/mm/damon/admin/kdamonds/<K>/contexts/<C>/monitoring_attrs/probes/<P>/filters/nr_filters
|
||||
Date: May 2026
|
||||
Contact: SJ Park <[email protected]>
|
||||
@@ -193,6 +206,20 @@ Description: If 'memcg' is written to the 'type' file, writing to and
|
||||
reading from this file sets and gets the path to the memory
|
||||
cgroup of the interest.
|
||||
|
||||
What: /sys/kernel/mm/damon/admin/kdamonds/<K>/contexts/<C>/monitoring_attrs/probes/<P>/filters/<F>/min
|
||||
Date: Sep 2026
|
||||
Contact: SJ Park <[email protected]>
|
||||
Description: If 'hugepage_size' is written to the 'type' file, writing to and
|
||||
reading from this file sets and gets the minimum size of the
|
||||
huge page of the interest.
|
||||
|
||||
What: /sys/kernel/mm/damon/admin/kdamonds/<K>/contexts/<C>/monitoring_attrs/probes/<P>/filters/<F>/max
|
||||
Date: Sep 2026
|
||||
Contact: SJ Park <[email protected]>
|
||||
Description: If 'hugepage_size' is written to the 'type' file, writing to and
|
||||
reading from this file sets and gets the maximum size of the
|
||||
huge page of the interest.
|
||||
|
||||
What: /sys/kernel/mm/damon/admin/kdamonds/<K>/contexts/<C>/monitoring_attrs/probes/<P>/filters/<F>/matching
|
||||
Date: May 2026
|
||||
Contact: SJ Park <[email protected]>
|
||||
|
||||
@@ -1321,6 +1321,10 @@ All memory amounts are in bytes. If a value which is not aligned to
|
||||
PAGE_SIZE is written, the value may be rounded up to the closest
|
||||
PAGE_SIZE multiple when read back.
|
||||
|
||||
For the limit files described below, an empty or all-whitespace
|
||||
write is accepted and sets the limit to 0. To disable a limit,
|
||||
write "max"; to set it to zero explicitly, write "0".
|
||||
|
||||
memory.current
|
||||
A read-only single value file which exists on non-root
|
||||
cgroups.
|
||||
|
||||
@@ -325,7 +325,7 @@ NR_FREE_PAGES
|
||||
On linux-2.6.21 or later, the number of free pages is in
|
||||
vm_stat[NR_FREE_PAGES]. Used to get the number of free pages.
|
||||
|
||||
PG_lru|PG_private|PG_swapcache|PG_swapbacked|PG_hwpoison|PG_head_mask
|
||||
PG_lru|PG_swapcache|PG_swapbacked|PG_hwpoison|PG_head_mask
|
||||
--------------------------------------------------------------------------
|
||||
|
||||
Page attributes. These flags are used to filter various unnecessary for
|
||||
|
||||
@@ -74,8 +74,11 @@ comma (",").
|
||||
│ │ │ │ │ │ nr_regions/min,max
|
||||
│ │ │ │ │ │ :ref:`probes <damon_usage_sysfs_probes>`/nr_probes
|
||||
│ │ │ │ │ │ │ 0/weight
|
||||
│ │ │ │ │ │ │ │ preps/nr_preps
|
||||
│ │ │ │ │ │ │ │ │ 0/prep_action
|
||||
│ │ │ │ │ │ │ │ │ ...
|
||||
│ │ │ │ │ │ │ │ filters/nr_filters
|
||||
│ │ │ │ │ │ │ │ │ 0/type,matching,allow,path
|
||||
│ │ │ │ │ │ │ │ │ 0/type,matching,allow,path,min,max
|
||||
│ │ │ │ │ │ │ │ │ ...
|
||||
│ │ │ │ │ │ │ ...
|
||||
│ │ │ │ │ :ref:`targets <sysfs_targets>`/nr_targets
|
||||
@@ -283,9 +286,18 @@ In the beginning, this directory has only one file, ``nr_probes``. Writing a
|
||||
number (``N``) to the file creates the number of child directories named ``0``
|
||||
to ``N-1``. Each directory represents each monitoring probe.
|
||||
|
||||
In each probe directory, one directory, ``filters`` exists. The directory
|
||||
contains files for installing filters for the probe, that is used to determine
|
||||
the data attribute for the probe.
|
||||
In each probe directory, two directories, ``preps`` and ``filters`` exist. The
|
||||
directories contain files for installing probing preparation actions and
|
||||
filters for the probe, that are used to determine the data attribute for the
|
||||
probe.
|
||||
|
||||
In the beginning, ``preps`` directory has only one file, ``nr_preps``.
|
||||
Writing a number (``N``) to the file creates the number of child directories
|
||||
named ``0`` to ``N-1``. Each directory represents each preparation action.
|
||||
Each directory has one file, ``prep_action``. The preparation action can be
|
||||
selected by writing the name of the action to the ``prep_action`` file. Refer
|
||||
to the :ref:`design doc <damon_design_data_attrs_monitoring>` for the list of
|
||||
supported actions.
|
||||
|
||||
Each probe directory also contains ``weight`` file. Reading from and writing
|
||||
to the file gets and sets the :ref:`attributes-only monitoring
|
||||
@@ -296,6 +308,8 @@ Writing a number (``N``) to the file creates the number of child directories
|
||||
named ``0`` to ``N-1``. Each directory represents each filter and works in a
|
||||
way similar to that for :ref:`DAMOS filter <sysfs_filters>`. When the filter
|
||||
``type`` is ``memcg``, ``path`` file acts as ``memcg_path`` for :ref:`DAMOS
|
||||
filter <sysfs_filters>`. When the filter ``type`` is ``hugepage_size``,
|
||||
``min`` and ``max`` files act as files of the same names for :ref:`DAMOS
|
||||
filter <sysfs_filters>`.
|
||||
|
||||
.. _sysfs_targets:
|
||||
@@ -539,6 +553,10 @@ and ``damon_target_idx``. To ``type`` file, you can write the type of the
|
||||
filter. Refer to :ref:`the design doc <damon_design_damos_filters>` for
|
||||
available type names, their meaning and on what layer those are handled.
|
||||
|
||||
For ``probe_hits_wsum`` type, you can specify the minimum and maximum probe
|
||||
hits weighted sum value for the filter to ``min`` and ``max`` files,
|
||||
respectively.
|
||||
|
||||
For ``memcg`` type, you can specify the memory cgroup of the interest by
|
||||
writing the path of the memory cgroup from the cgroups mount point to
|
||||
``memcg_path`` file. For ``addr`` type, you can specify the start and end
|
||||
|
||||
@@ -174,7 +174,7 @@ advisor_mode
|
||||
The section about ``advisor`` explains in detail how the scan time
|
||||
advisor works.
|
||||
|
||||
adivsor_max_cpu
|
||||
advisor_max_cpu
|
||||
specifies the upper limit of the cpu percent usage of the ksmd
|
||||
background thread. The default is 70.
|
||||
|
||||
@@ -183,10 +183,10 @@ advisor_target_scan_time
|
||||
pages. The default value is 200 seconds.
|
||||
|
||||
advisor_min_pages_to_scan
|
||||
specifies the lower limit of the ``pages_to_scan`` parameter of the
|
||||
scan time advisor. The default is 500.
|
||||
specifies the initial value of the ``pages_to_scan`` parameter of
|
||||
the scan time advisor. The default is 500.
|
||||
|
||||
adivsor_max_pages_to_scan
|
||||
advisor_max_pages_to_scan
|
||||
specifies the upper limit of the ``pages_to_scan`` parameter of the
|
||||
scan time advisor. The default is 30000.
|
||||
|
||||
|
||||
@@ -19,82 +19,28 @@ With 1G PUD level mapping, we require 16384 struct pages and a single 64K
|
||||
vmemmap page can contain 1024 struct pages (64K/sizeof(struct page)). Hence we
|
||||
require 16 64K pages in vmemmap to map the struct page for 1G PUD level mapping.
|
||||
|
||||
Here's how things look like on device-dax after the sections are populated::
|
||||
+-----------+ ---virt_to_page---> +-----------+ mapping to +-----------+
|
||||
| | | 0 | -------------> | 0 |
|
||||
| | +-----------+ +-----------+
|
||||
| | | 1 | -------------> | 1 |
|
||||
| | +-----------+ +-----------+
|
||||
| | | 2 | ----------------^ ^ ^ ^ ^ ^
|
||||
| | +-----------+ | | | | |
|
||||
| | | 3 | ------------------+ | | | |
|
||||
| | +-----------+ | | | |
|
||||
| | | 4 | --------------------+ | | |
|
||||
| PUD | +-----------+ | | |
|
||||
| level | | . | ----------------------+ | |
|
||||
| mapping | +-----------+ | |
|
||||
| | | . | ------------------------+ |
|
||||
| | +-----------+ |
|
||||
| | | 15 | --------------------------+
|
||||
| | +-----------+
|
||||
| |
|
||||
| |
|
||||
| |
|
||||
+-----------+
|
||||
|
||||
|
||||
With 4K page size, 2M PMD level mapping requires 512 struct pages and a single
|
||||
4K vmemmap page contains 64 struct pages(4K/sizeof(struct page)). Hence we
|
||||
require 8 4K pages in vmemmap to map the struct page for 2M pmd level mapping.
|
||||
|
||||
Here's how things look like on device-dax after the sections are populated::
|
||||
Here's how things look on device-dax after vmemmap-optimized sections are
|
||||
populated. ``N`` is the number of vmemmap pages required by the DAX mapping
|
||||
above::
|
||||
|
||||
+-----------+ ---virt_to_page---> +-----------+ mapping to +-----------+
|
||||
| | | 0 | -------------> | 0 |
|
||||
| | +-----------+ +-----------+
|
||||
| | | 1 | -------------> | 1 |
|
||||
| | +-----------+ +-----------+
|
||||
| | | 2 | ----------------^ ^ ^ ^ ^ ^
|
||||
| | +-----------+ | | | | |
|
||||
| | | 3 | ------------------+ | | | |
|
||||
| | +-----------+ | | | |
|
||||
| | | 4 | --------------------+ | | |
|
||||
| PMD | +-----------+ | | |
|
||||
| level | | 5 | ----------------------+ | |
|
||||
| mapping | +-----------+ | |
|
||||
| | | 6 | ------------------------+ |
|
||||
| | +-----------+ |
|
||||
| | | 7 | --------------------------+
|
||||
| | +-----------+
|
||||
| |
|
||||
| |
|
||||
| |
|
||||
+-----------+
|
||||
|
||||
With 1G PUD level mapping, we require 262144 struct pages and a single 4K
|
||||
vmemmap page can contain 64 struct pages (4K/sizeof(struct page)). Hence we
|
||||
require 4096 4K pages in vmemmap to map the struct pages for 1G PUD level
|
||||
mapping.
|
||||
|
||||
Here's how things look like on device-dax after the sections are populated::
|
||||
|
||||
+-----------+ ---virt_to_page---> +-----------+ mapping to +-----------+
|
||||
| | | 0 | -------------> | 0 |
|
||||
| | +-----------+ +-----------+
|
||||
| | | 1 | -------------> | 1 |
|
||||
| | +-----------+ +-----------+
|
||||
| | | 2 | ----------------^ ^ ^ ^ ^ ^
|
||||
| | +-----------+ | | | | |
|
||||
| | | 3 | ------------------+ | | | |
|
||||
| | +-----------+ | | | |
|
||||
| | | 4 | --------------------+ | | |
|
||||
| PUD | +-----------+ | | |
|
||||
| level | | . | ----------------------+ | |
|
||||
| mapping | +-----------+ | |
|
||||
| | | . | ------------------------+ |
|
||||
| | +-----------+ |
|
||||
| | | 4095 | --------------------------+
|
||||
| | +-----------+
|
||||
Device DAX vmemmap pages (N pages) backing page frames
|
||||
+-----------+ ---virt_to_page---> +-----------+ mapping to +-------------+
|
||||
| | | 0 | -------------> | 0 |
|
||||
| | +-----------+ +-------------+
|
||||
| | | 1 | ------+
|
||||
| | +-----------+ |
|
||||
| | | 2 | ------+
|
||||
| | +-----------+ |
|
||||
| | | . | ------+ +-------------+
|
||||
| PMD/PUD | +-----------+ | | A single, |
|
||||
| level | | . | ------+------> | per-zone |
|
||||
| mapping | +-----------+ | | shared tail |
|
||||
| | | N - 1 | ------+ | page |
|
||||
| | +-----------+ +-------------+
|
||||
| |
|
||||
| |
|
||||
| |
|
||||
|
||||
@@ -19,10 +19,18 @@ Diversity of the allocation APIs combined with the numerous GFP flags
|
||||
makes the question "How should I allocate memory?" not that easy to
|
||||
answer, although very likely you should use
|
||||
|
||||
::
|
||||
|
||||
kzalloc_obj(<VAR_OR_TYPE>);
|
||||
|
||||
if you need memory for an object and
|
||||
|
||||
::
|
||||
|
||||
kzalloc(<size>, GFP_KERNEL);
|
||||
|
||||
if you need memory for a buffer.
|
||||
|
||||
Of course there are cases when other allocation APIs and different GFP
|
||||
flags must be used.
|
||||
|
||||
@@ -139,10 +147,13 @@ allocate memory for an array, there are kmalloc_array() and kcalloc()
|
||||
helpers. The helpers struct_size(), array_size() and array3_size() can
|
||||
be used to safely calculate object sizes without overflowing.
|
||||
|
||||
The maximal size of a chunk that can be allocated with `kmalloc` is
|
||||
limited. The actual limit depends on the hardware and the kernel
|
||||
configuration, but it is a good practice to use `kmalloc` for objects
|
||||
smaller than page size.
|
||||
Since 7.0 there are type aware kmalloc-family helpers that let you safely and
|
||||
conveniently allocate a single object or arrays of objects with kzalloc_obj()
|
||||
and kmalloc_obj() and their array versions kzalloc_objs() and
|
||||
kmalloc_objs(). These helpers only need the type of the object that should be
|
||||
allocated and the count of elements in the array for the array versions.
|
||||
|
||||
As of v7.2, vast majority of the memory allocations use kzalloc_obj().
|
||||
|
||||
The address of a chunk allocated with `kmalloc` is aligned to at least
|
||||
ARCH_KMALLOC_MINALIGN bytes. For sizes which are a power of two, the
|
||||
@@ -154,9 +165,16 @@ Chunks allocated with kmalloc() can be resized with krealloc(). Similarly
|
||||
to kmalloc_array(): a helper for resizing arrays is provided in the form of
|
||||
krealloc_array().
|
||||
|
||||
For large allocations you can use vmalloc() and vzalloc(), or directly
|
||||
request pages from the page allocator. The memory allocated by `vmalloc`
|
||||
and related functions is not physically contiguous.
|
||||
`kmalloc` always allocates physically contiguous memory and the maximal size of
|
||||
a chunk that can be allocated with `kmalloc` is limited by `KMALLOC_MAX_SIZE`,
|
||||
which matches the page allocator's MAX_PAGE_ORDER limit.
|
||||
|
||||
Internally, the slab allocator differentiates allocations of different orders
|
||||
and delegates larger allocations to the page allocator, but for the users of
|
||||
`kmalloc` family it is entirely transparent.
|
||||
|
||||
For large allocations that do not require physically contiguous memory you can
|
||||
use vmalloc() and vzalloc() family.
|
||||
|
||||
If you are not sure whether the allocation size is too large for
|
||||
`kmalloc`, it is possible to use kvmalloc() and its derivatives. It will
|
||||
|
||||
@@ -52,6 +52,15 @@ Virtually Contiguous Mappings
|
||||
.. kernel-doc:: mm/vmalloc.c
|
||||
:export:
|
||||
|
||||
Kernel Page Table Permissions
|
||||
=============================
|
||||
|
||||
.. kernel-doc:: include/linux/set_memory.h
|
||||
:doc: Kernel page table permissions
|
||||
|
||||
.. kernel-doc:: include/linux/set_memory.h
|
||||
:internal:
|
||||
|
||||
File Mapping and Page Cache
|
||||
===========================
|
||||
|
||||
|
||||
@@ -226,15 +226,6 @@ will be pinned longterm, and whose data will be accessed.
|
||||
|
||||
Unit testing
|
||||
============
|
||||
This file::
|
||||
|
||||
tools/testing/selftests/mm/gup_test.c
|
||||
|
||||
has the following new calls to exercise the new pin*() wrapper functions:
|
||||
|
||||
* PIN_FAST_BENCHMARK (./gup_test -a)
|
||||
* PIN_BASIC_TEST (./gup_test -b)
|
||||
|
||||
You can monitor how many total dma-pinned pages have been acquired and released
|
||||
since the system was booted, via two new /proc/vmstat entries: ::
|
||||
|
||||
|
||||
@@ -66,9 +66,14 @@ Memory scanning parameters can be modified at run-time by writing to the
|
||||
Kmemleak can also be disabled at boot-time by passing ``kmemleak=off`` on
|
||||
the kernel command line.
|
||||
|
||||
Memory may be allocated or freed before kmemleak is initialised and
|
||||
these actions are stored in an early log buffer. The size of this buffer
|
||||
is configured via the CONFIG_DEBUG_KMEMLEAK_MEM_POOL_SIZE option.
|
||||
Memory may be allocated or freed before kmemleak is initialised, so a
|
||||
static pool of metadata objects is used to track those allocations. Once
|
||||
kmemleak is fully initialised the pool becomes an emergency reserve, used
|
||||
whenever a metadata object cannot be allocated from the slab. The number
|
||||
of objects in the pool is configured via the
|
||||
CONFIG_DEBUG_KMEMLEAK_MEM_POOL_SIZE option. Exhausting it at run time
|
||||
prints "Cannot allocate a kmemleak_object structure" and disables
|
||||
kmemleak.
|
||||
|
||||
If CONFIG_DEBUG_KMEMLEAK_DEFAULT_OFF are enabled, the kmemleak is
|
||||
disabled by default. Passing ``kmemleak=on`` on the kernel command
|
||||
@@ -185,7 +190,8 @@ reported by kmemleak because values found during the memory scanning
|
||||
point to such objects. To reduce the number of false negatives, kmemleak
|
||||
provides the kmemleak_ignore, kmemleak_scan_area, kmemleak_no_scan and
|
||||
kmemleak_erase functions (see above). The task stacks also increase the
|
||||
amount of false negatives and their scanning is not enabled by default.
|
||||
amount of false negatives and their scanning is enabled by default; it
|
||||
can be turned off with ``stack=off``.
|
||||
|
||||
The false positives are objects wrongly reported as being memory leaks
|
||||
(orphan). For objects known not to be leaks, kmemleak provides the
|
||||
@@ -195,12 +201,12 @@ longer be scanned.
|
||||
|
||||
Some of the reported leaks are only transient, especially on SMP
|
||||
systems, because of pointers temporarily stored in CPU registers or
|
||||
stacks. Kmemleak defines MSECS_MIN_AGE (defaulting to 1000) representing
|
||||
stacks. Kmemleak defines MSECS_MIN_AGE (defaulting to 5000) representing
|
||||
the minimum age of an object to be reported as a memory leak.
|
||||
|
||||
The ``min_unref_scans`` module parameter requires an object to be seen
|
||||
unreferenced in that many consecutive scans before it is reported. It
|
||||
defaults to 2 when CONFIG_DEBUG_KMEMLEAK_VERBOSE is enabled, where the
|
||||
defaults to 3 when CONFIG_DEBUG_KMEMLEAK_VERBOSE is enabled, where the
|
||||
periodic scan thread confirms a leak on its own, and to 1 otherwise. A
|
||||
value of 1 preserves the historical behaviour; higher values filter the
|
||||
transient false positives described above, at the cost of delaying genuine
|
||||
@@ -212,8 +218,10 @@ Limitations and Drawbacks
|
||||
-------------------------
|
||||
|
||||
The main drawback is the reduced performance of memory allocation and
|
||||
freeing. To avoid other penalties, the memory scanning is only performed
|
||||
when the /sys/kernel/debug/kmemleak file is read. Anyway, this tool is
|
||||
freeing. To avoid other penalties, the memory scanning is performed by a
|
||||
periodic thread rather than on every allocation. Reading the
|
||||
/sys/kernel/debug/kmemleak file only lists the objects found by the last
|
||||
scan; writing ``scan`` to it triggers a new one. Anyway, this tool is
|
||||
intended for debugging purposes where the performance might not be the
|
||||
most important requirement.
|
||||
|
||||
|
||||
@@ -649,8 +649,8 @@ Writeback.
|
||||
|
||||
The first can be used independently to the others. The VM can try to
|
||||
release clean pages in order to reuse them. To do this it can call
|
||||
->release_folio on clean folios with the private
|
||||
flag set. Clean pages without PagePrivate and with no external references
|
||||
->release_folio on clean folios with folio->private set. Clean pages
|
||||
without folio->private set and with no external references
|
||||
will be released without notice being given to the address_space.
|
||||
|
||||
To achieve this functionality, pages need to be placed on an LRU with
|
||||
@@ -674,7 +674,7 @@ filemap_fdatawait_range, to wait for all writeback to complete.
|
||||
|
||||
An address_space handler may attach extra information to a page,
|
||||
typically using the 'private' field in the 'struct page'. If such
|
||||
information is attached, the PG_Private flag should be set. This will
|
||||
information is attached, non-NULL 'private' field will
|
||||
cause various VM routines to make extra calls into the address_space
|
||||
handler to deal with that data.
|
||||
|
||||
|
||||
@@ -249,7 +249,7 @@ For each ``aggregation interval``, it compares the access frequencies
|
||||
sum of the two regions' sizes is smaller than the size of total regions divided
|
||||
by the ``minimum number of regions``, DAMON merges the two regions. If the
|
||||
resulting number of total regions is still higher than ``maximum number of
|
||||
regions``, it repeats the merging with increasing access frequenceis difference
|
||||
regions``, it repeats the merging with increasing access frequencies difference
|
||||
threshold until the upper-limit of the number of regions is met, or the
|
||||
threshold becomes higher than possible maximum value (``aggregation interval``
|
||||
divided by ``sampling interval``). Then, after it reports and clears the
|
||||
@@ -293,8 +293,15 @@ registration is made by specifying a probe per attribute. Each of the probe
|
||||
specifies a rule to determine if a given memory region has the related
|
||||
attribute. The rule is constructed with multiple filters. The filters work
|
||||
same to :ref:`DAMOS filters <damon_design_damos_filters>` except the supported
|
||||
filter types. Currently only ``anon`` and ``memcg`` filter types are supported
|
||||
for data attributes monitoring.
|
||||
filter types. Currently below filter types are supported.
|
||||
|
||||
- ``anon``: Same to that for DAMOS filters.
|
||||
- ``memcg``: Same to that for DAMOS filters.
|
||||
- ``pgidle_unset``: Matches if the page for the memory is marked as not
|
||||
access-idle.
|
||||
- ``pgidle_set``: Matches if the page for the memory is marked as access-idle.
|
||||
- ``hugepage_size``: Matches if the page for the memory is a part of a hugepage
|
||||
of a given size range.
|
||||
|
||||
If such probes are registered, DAMON executes the probes for each region's
|
||||
sampling memory when it does the access :ref:`sampling
|
||||
@@ -305,6 +312,13 @@ Users can therefore know how much of a given DAMON region has a specific data
|
||||
attribute by reading the per-region per-probe probe hits counter after each
|
||||
aggregation interval.
|
||||
|
||||
Users can optionally register probing preparation actions per probe. If such
|
||||
actions are registered, DAMON applies the actions to each region's sampling
|
||||
memory before starting the next sampling interval. Currently only one action,
|
||||
``set_pgidle`` is supported. The action marks the page for the probing target
|
||||
memory as access-idle. This can be useful to be used together with
|
||||
``pgidle_unset`` or ``pgidle_set`` probe filter.
|
||||
|
||||
This is a sampling based mechanism. Hence, it is lightweight but the output
|
||||
may include some measurement errors. The output should be used with good
|
||||
understanding of statistics.
|
||||
@@ -431,7 +445,7 @@ users to set the aimed amount of access events to observe via DAMON within
|
||||
given time interval. The target can be specified by the user as a ratio of
|
||||
DAMON-observed access events to the theoretical maximum amount of the events
|
||||
(``access_bp``) that measured within a given number of aggregations
|
||||
(``aggrs``).
|
||||
(``aggrs``). The ratio is in basis point (bp or 1/10,000).
|
||||
|
||||
The DAMON-observed access events are calculated in byte granularity based on
|
||||
DAMON :ref:`region assumption <damon_design_region_based_sampling>`. For
|
||||
@@ -682,9 +696,11 @@ There are two such tuning algorithms that users can select as they need.
|
||||
This is the default selection. If unsure, use this.
|
||||
- ``temporal``: More straightforward algorithm. Tries to achieve the goal as
|
||||
fast as possible, using maximum allowed quota, but only for a temporal short
|
||||
time. When the quota is under-achieved, this algorithm keeps tuning quota to
|
||||
a maximum allowed one. Once the quota is [over]-achieved, this sets the
|
||||
quota zero. Useful for deterministic control required environments.
|
||||
time. When the goal is under-achieved, this algorithm keeps tuning quota to
|
||||
a maximum allowed one. Once the goal is [over-]achieved, this sets the
|
||||
quota zero. Useful for deterministic control required environments. Note
|
||||
that the zero quota is a valid quota, and therefore ``qt_exceeds`` :ref:`stat
|
||||
<damon_design_damos_stat>` will keep increasing in this case.
|
||||
|
||||
The goal can be specified with five parameters, namely ``target_metric``,
|
||||
``target_value``, ``current_value``, ``nid`` and ``path``. The auto-tuning
|
||||
@@ -701,7 +717,8 @@ mechanism tries to make ``current_value`` of ``target_metric`` be same to
|
||||
in microseconds that measured from last quota reset to next quota reset.
|
||||
DAMOS does the measurement on its own, so only ``target_value`` need to be
|
||||
set by users at the initial time. In other words, DAMOS does self-feedback.
|
||||
- ``node_mem_used_bp``: Specific NUMA node's used memory ratio in bp (1/10,000).
|
||||
- ``node_mem_used_bp``: Specific NUMA node's used memory ratio in basis point
|
||||
(bp or 1/10,000).
|
||||
- ``node_mem_free_bp``: Specific NUMA node's free memory ratio in bp (1/10,000).
|
||||
- ``node_memcg_used_bp``: Specific cgroup's node used memory ratio for a
|
||||
specific NUMA node, in bp (1/10,000).
|
||||
@@ -713,6 +730,8 @@ mechanism tries to make ``current_value`` of ``target_metric`` be same to
|
||||
bp (1/10,000).
|
||||
- ``node_eligible_mem_bp``: Scheme target access pattern-eligible memory ratio
|
||||
of a node in bp (1/10,000).
|
||||
- ``hugepage_mem_bp``: Total huge page to total used memory ratio in bp
|
||||
(1/10,000).
|
||||
|
||||
``nid`` is optionally required for ``node_mem_used_bp``, ``node_mem_free_bp``,
|
||||
``node_memcg_used_bp``, ``node_memcg_free_bp`` and ``node_eligible_mem_bp`` to
|
||||
@@ -809,6 +828,9 @@ Below ``type`` of filters are currently supported.
|
||||
- Applied to pages that belonging to a given address range.
|
||||
- target
|
||||
- Applied to pages that belonging to a given DAMON monitoring target.
|
||||
- probe_hits_wsum
|
||||
- Matches to monitoring regions having a given range of :ref:`probe
|
||||
hits weighted sum <damon_design_attrs_only_monitoring>` value.
|
||||
- Operations layer handled, supported by only ``paddr`` operations set.
|
||||
- anon
|
||||
- Applied to pages that containing data that not stored in files.
|
||||
@@ -846,8 +868,8 @@ scheme's execution.
|
||||
- ``nr_applied``: Total number of regions that the scheme is applied.
|
||||
- ``sz_applied``: Total size of regions that the scheme is applied.
|
||||
- ``qt_exceeds``: Total number of times the quota of the scheme has exceeded.
|
||||
- ``nr_snapshots``: Total number of DAMON snapshots that the scheme is tried to
|
||||
be applied.
|
||||
- ``nr_snapshots``: Total number of DAMON snapshots that the scheme is
|
||||
completely tried to be applied.
|
||||
- ``max_nr_snapshots``: Upper limit of ``nr_snapshots``.
|
||||
|
||||
"A scheme is tried to be applied to a region" means DAMOS core logic determined
|
||||
@@ -871,11 +893,12 @@ action is ``pageout`` while all pages of the region are unreclaimable, applying
|
||||
the action to the region will fail.
|
||||
|
||||
Unlike normal stats, ``max_nr_snapshots`` is set by users. If it is set as
|
||||
non-zero and ``nr_snapshots`` be same to or greater than ``nr_snapshots``, the
|
||||
scheme is deactivated.
|
||||
non-zero and ``nr_snapshots`` equals or is greater than ``max_nr_snapshots``,
|
||||
the scheme is deactivated. Note that, unlike watermarks, even if a scheme's
|
||||
``nr_snapshots`` reaches ``max_nr_snapshots``, monitoring will not stop.
|
||||
|
||||
To know how user-space can read the stats via :ref:`DAMON sysfs interface
|
||||
<sysfs_interface>`, refer to :ref:s`stats <sysfs_stats>` part of the
|
||||
<sysfs_interface>`, refer to :ref:`stats <sysfs_schemes_stats>` part of the
|
||||
documentation.
|
||||
|
||||
Regions Walking
|
||||
|
||||
@@ -106,18 +106,9 @@ AI Review
|
||||
|
||||
For patches that are publicly posted to DAMON mailing list
|
||||
([email protected]), AI reviews of the patches will be available at
|
||||
sashiko.dev. The reviews could also be sent as mails to the author of the
|
||||
patch.
|
||||
sashiko.dev. The reviews will also be sent as replies to the author of the
|
||||
patch and the mailing list.
|
||||
|
||||
Patch authors are encouraged to check the AI reviews and share their opinions.
|
||||
The sharing could be done as a reply to the mail thread. Consider reducing the
|
||||
recipients list for such sharing, since some people are not really interested
|
||||
in AI reviews. As a rule of thumb, drop [email protected] and individuals
|
||||
except DAMON maintainer.
|
||||
|
||||
`hkml` also provides a `feature
|
||||
<https://github.com/sjp38/hackermail/blob/master/USAGE.md#forwarding-sashikodev-statuscomments-to-mailing-list>`_
|
||||
for such sharing. Please feel free to use the feature.
|
||||
|
||||
It is only an optional recommendation. DAMON maintainer could also ask any
|
||||
question about the AI reviews, though.
|
||||
Patch authors are encouraged to check the AI reviews and share their opinions
|
||||
by replying on the mail thread. It is only an optional recommendation. DAMON
|
||||
maintainer could also ask any question about the AI reviews, though.
|
||||
|
||||
@@ -25,6 +25,7 @@ see the :doc:`admin guide <../admin-guide/mm/index>`.
|
||||
|
||||
physical_memory
|
||||
page_tables
|
||||
kernel-page-tables
|
||||
process_addrs
|
||||
bootmem
|
||||
page_allocation
|
||||
|
||||
@@ -0,0 +1,410 @@
|
||||
.. SPDX-License-Identifier: GPL-2.0
|
||||
|
||||
==================
|
||||
Kernel Page Tables
|
||||
==================
|
||||
|
||||
Introduction
|
||||
============
|
||||
|
||||
The kernel page tables are created early during boot and, unlike the page
|
||||
tables of user processes, most of them remain static throughout the system
|
||||
lifetime.
|
||||
|
||||
Every architecture has a direct map (also called linear map) that maps the
|
||||
physical memory at a fixed offset, so that a physical address can be
|
||||
translated to a kernel virtual address with simple arithmetic. On most
|
||||
architectures the direct map is a part of the kernel page tables, with a few
|
||||
exceptions described in the `Direct map`_ section below.
|
||||
|
||||
On 32-bit systems with high memory the direct map covers only a part of the
|
||||
physical memory, see Documentation/mm/highmem.rst.
|
||||
|
||||
The vmalloc area, present on every architecture with an MMU, is used for
|
||||
allocations of virtually contiguous memory whose backing pages are not
|
||||
necessarily physically contiguous, and for mapping of the device memory. Its
|
||||
page tables are created and torn down at runtime, see
|
||||
Documentation/mm/vmalloc.rst.
|
||||
|
||||
Architectures that use the `SPARSEMEM_VMEMMAP` memory model reserve a range of
|
||||
kernel address space for the memory map, so that `struct page` objects appear
|
||||
as a virtually contiguous array indexed by the page frame number, see
|
||||
Documentation/mm/memory-model.rst.
|
||||
|
||||
Besides these, the kernel image may be mapped in a dedicated part of the
|
||||
kernel address space rather than accessed through the direct map. In that
|
||||
case its mapping is an alias of the direct map of the physical memory the
|
||||
image occupies.
|
||||
|
||||
The rest of the kernel address space is architecture specific. For instance,
|
||||
x86 has a region for the EFI runtime services and s390 has a region for the
|
||||
code that has to run in the 31-bit addressing mode.
|
||||
|
||||
Direct map
|
||||
==========
|
||||
|
||||
On most architectures the direct map is an ordinary part of the kernel page
|
||||
tables. It is created early during boot with the largest pages the hardware
|
||||
and the kernel configuration allow.
|
||||
|
||||
Several architectures are different.
|
||||
|
||||
MIPS
|
||||
----
|
||||
|
||||
MIPS does not map the physical memory with page tables at all. Instead, a
|
||||
part of the kernel virtual address space is a window into the physical
|
||||
address space: the hardware translates the addresses that fall into that
|
||||
window by a fixed transformation of the address bits, without walking the
|
||||
page tables and without using the TLB. The memory attributes, such as
|
||||
cacheability and the privilege level required to access the memory, are a
|
||||
property of the window rather than of an individual page.
|
||||
|
||||
There are no page table entries describing the direct map, so its properties
|
||||
cannot be changed for an individual page. On 32-bit systems the window covers
|
||||
only 512 MiB of the physical memory, so everything above that is high memory.
|
||||
|
||||
LoongArch
|
||||
---------
|
||||
|
||||
Like MIPS, LoongArch maps the physical memory with a hardware window. The
|
||||
window covers 256 TiB of the physical address space on 64-bit and 512 MiB on
|
||||
32-bit systems, and everything above that is high memory.
|
||||
|
||||
The window occupies the lower part of the kernel address space. The upper
|
||||
part, which includes the vmalloc area, is mapped with kernel page tables.
|
||||
|
||||
PowerPC with the hash MMU
|
||||
-------------------------
|
||||
|
||||
On 64-bit PowerPC systems with the hash MMU the direct map does not exist in
|
||||
the Linux page tables. It is installed into the hardware hash page table early
|
||||
during boot.
|
||||
|
||||
Modifying such mappings requires updating the hash page table directly, and
|
||||
the hash MMU code implements this only for the kernel image permissions,
|
||||
`debug_pagealloc` and KFENCE.
|
||||
|
||||
With the radix MMU the direct map is a part of the ordinary kernel page
|
||||
tables.
|
||||
|
||||
Modifying the kernel page tables
|
||||
================================
|
||||
|
||||
Except for the vmalloc area, the kernel page tables are mostly static. Still,
|
||||
there are cases when the permissions of existing kernel mappings have to be
|
||||
updated, for instance when a module is loaded and its text becomes read-only
|
||||
and executable, or when a page is temporarily removed from the direct map to
|
||||
reduce its exposure.
|
||||
|
||||
There are two families of functions for this, both declared in
|
||||
`include/linux/set_memory.h`:
|
||||
|
||||
* `set_memory_*()` change permissions of an arbitrary kernel mapping. They
|
||||
take a kernel virtual address and the number of pages.
|
||||
|
||||
* `set_direct_map_*()` change permissions of the direct mapping for the range
|
||||
of page frames starting at the page represented by a `struct page`. They take
|
||||
a `struct page` pointer and the number of pages.
|
||||
|
||||
Architectures that implement `set_memory()` select `CONFIG_ARCH_HAS_SET_MEMORY`
|
||||
|
||||
Architectures that implement `set_direct_map()` select
|
||||
`CONFIG_ARCH_HAS_SET_DIRECT_MAP`.
|
||||
|
||||
Common semantics
|
||||
----------------
|
||||
|
||||
Ranges
|
||||
~~~~~~
|
||||
|
||||
The `set_memory()` functions expect a range described by a start address and a
|
||||
number of pages. The address must be page aligned and the entire range must be
|
||||
covered by page table entries the architecture knows how to update.
|
||||
|
||||
The entries may be marked as not present, set_memory_p() and set_memory_valid()
|
||||
exist exactly to bring such a mapping back.
|
||||
|
||||
When a range does not qualify, an architecture will usually say so by returning
|
||||
an error and sometimes by a WARN()ing as well, unless it prefers to keep it to
|
||||
itself and return success, see `Architecture specific differences`_.
|
||||
|
||||
The `set_direct_map()` functions expect a range described by the first
|
||||
`struct page` and a number of pages, and they update the direct map starting
|
||||
at that page.
|
||||
|
||||
The pages that follow the first one are updated regardless of what they are, so
|
||||
the caller has to make sure that the range does not extend beyond the memory it
|
||||
owns.
|
||||
|
||||
Some architectures cannot split a large mapping, and they reject a range that
|
||||
is a part of one, see `Architecture specific differences`_.
|
||||
|
||||
Calling `set_memory()` with the number of pages set to zero is a no-op that
|
||||
returns success, except on arm64, where doing nothing to the wrong address is
|
||||
still an error.
|
||||
|
||||
Aliases
|
||||
~~~~~~~
|
||||
|
||||
A physical page may be mapped several times, for instance in the direct map
|
||||
and in the vmalloc area, and the permissions of these mappings may differ.
|
||||
|
||||
Whether the direct map alias is updated by a `set_memory()` call, and
|
||||
which permission bits make it there, is entirely up to the architecture, and
|
||||
there is not much agreement between them, see
|
||||
`Architecture specific differences`_.
|
||||
|
||||
Relying on that is a gamble; code that needs the direct map alias to change
|
||||
should say so with the `set_direct_map()` APIs.
|
||||
|
||||
Failures and partial updates
|
||||
~~~~~~~~~~~~~~~~~~~~~~~~~~~~
|
||||
|
||||
Both families of functions return 0 on success and a negative error code on
|
||||
failure. The most common failures are `-EINVAL` for a range that cannot be
|
||||
handled and `-ENOMEM` when splitting a large mapping fails to allocate a page
|
||||
table.
|
||||
|
||||
Some of the range checks happen upfront, so their failure leaves the page
|
||||
tables unchanged.
|
||||
|
||||
**The update is not atomic and there is no rollback.**
|
||||
|
||||
The architecture implementations walk the range and update the page tables as
|
||||
they go, and they stop at the first entry that cannot be updated. When an error
|
||||
is returned, an arbitrary prefix of the range may have been updated already,
|
||||
and the same is true for the direct map alias when the architecture updates it.
|
||||
|
||||
For example, when a range spans two large mappings and splitting the second
|
||||
one fails because there is no memory for a page table, the first one is
|
||||
already split and updated by the time the error is returned.
|
||||
|
||||
None of the APIs inform the caller where in the range they failed, so reverting
|
||||
such a partial update is possible in principle but unreliable in practice.
|
||||
|
||||
The caller may try to restore the original permissions over the entire range,
|
||||
but that revert goes through the very code that has just failed, which does
|
||||
not inspire much confidence.
|
||||
|
||||
The callers should therefore be prepared to give up on the memory in question:
|
||||
leak it or panic, but never return it to the allocator before the permissions
|
||||
are restored and never assume that the requested permissions are in effect.
|
||||
Neither option is appealing, but both beat handing out a page whose
|
||||
permissions nobody knows.
|
||||
|
||||
TLB flushing
|
||||
~~~~~~~~~~~~
|
||||
|
||||
The `set_memory()` functions flush the TLB for the affected range before
|
||||
they return, so that the new permissions are in effect for every CPU.
|
||||
|
||||
The `set_direct_map()` functions have `_noflush` in their names because when
|
||||
they were first introduced on x86, the intention was that the TLB flushing
|
||||
could be optimized by letting the caller handle it.
|
||||
|
||||
For example, vfree() batches the TLB flushes for the areas allocated with
|
||||
`VM_FLUSH_RESET_PERMS`, folding the flush of the direct map into the flush it
|
||||
has to do for the vmalloc mapping anyway.
|
||||
|
||||
Some architectures flush the TLB in the `_noflush` functions anyway, so the
|
||||
name is best read as a suggestion. It does not make the flush by the caller
|
||||
unnecessary, it only makes it more expensive.
|
||||
|
||||
A caller that changes the permissions to more restrictive ones must flush the
|
||||
TLB itself.
|
||||
|
||||
Context
|
||||
~~~~~~~
|
||||
|
||||
Architectures use different locking mechanisms to synchronize kernel page table
|
||||
updates, and both families of functions may sleep, for instance when they
|
||||
allocate memory to split a large mapping.
|
||||
|
||||
The caller cannot presume it is safe to call these APIs from an atomic context.
|
||||
|
||||
The `set_direct_map()` functions must not be called for high memory pages,
|
||||
which have no direct map alias to update.
|
||||
|
||||
Unimplemented APIs silently succeed
|
||||
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
|
||||
|
||||
When an architecture does not implement these APIs, the generic stubs in
|
||||
`include/linux/set_memory.h` return 0, that is, they report success for work
|
||||
they have no intention of doing.
|
||||
|
||||
The same happens inside several architecture implementations. The arm64
|
||||
`set_direct_map()` functions return 0 when can_set_direct_map() is false, and
|
||||
the LoongArch `set_memory()` functions return 0 for the addresses in its
|
||||
windowed direct mapping, which is not backed by page tables at all.
|
||||
|
||||
Returning success without actually updating the page tables is a deliberate
|
||||
trade-off that keeps the callers free of `#ifdef`\ s, but they have to realize:
|
||||
|
||||
**a return value of 0 does not imply that the permissions were actually
|
||||
changed.**
|
||||
|
||||
For best-effort hardening that is good enough. When correctness or security
|
||||
depends on the permissions, the caller has to make sure the architecture
|
||||
really implements what it needs. For instance, secretmem depends on
|
||||
`CONFIG_ARCH_HAS_SET_DIRECT_MAP` and calls can_set_direct_map() at runtime.
|
||||
|
||||
Architecture specific differences
|
||||
=================================
|
||||
|
||||
The APIs are implemented by seven architectures and, beyond the common
|
||||
semantics described above, their behaviour differs in several respects.
|
||||
|
||||
Which of the APIs are implemented:
|
||||
|
||||
========= ===================== =========================
|
||||
Arch `ARCH_HAS_SET_MEMORY` `ARCH_HAS_SET_DIRECT_MAP`
|
||||
========= ===================== =========================
|
||||
arm yes no
|
||||
arm64 yes yes
|
||||
loongarch yes yes
|
||||
powerpc yes no
|
||||
riscv yes (MMU only) yes (MMU only)
|
||||
s390 yes yes
|
||||
x86 yes yes
|
||||
========= ===================== =========================
|
||||
|
||||
Only set_memory_ro(), set_memory_rw(), set_memory_x() and set_memory_nx() are
|
||||
available everywhere, and even these are not universal: some architectures
|
||||
restrict the ranges that can be modified, for instance arm64 rejects direct map
|
||||
addresses and accepts only addresses in vmalloc space. The architectures that
|
||||
may run on hardware without an execute permission bit, like x86 and s390,
|
||||
silently skip the update of the executable bit.
|
||||
|
||||
set_memory_rox() has a generic implementation that calls set_memory_ro() and
|
||||
set_memory_x() in turn; PowerPC, s390 and x86 override it with a single-pass
|
||||
version.
|
||||
|
||||
Several architectures define additional APIs. Some of those may have identical
|
||||
semantics but different names. For example, making a mapping present or not
|
||||
present is spelled differently: set_memory_p() and set_memory_np() on x86 and
|
||||
PowerPC, set_memory_valid() on arm and arm64.
|
||||
|
||||
The direct map and the kernel image are normally mapped with the largest
|
||||
possible pages, and changing the permissions of a single page inside such a
|
||||
mapping requires splitting it, which not every architecture can do.
|
||||
|
||||
arm
|
||||
---
|
||||
|
||||
* Does not implement `set_direct_map()`.
|
||||
* Provides set_memory_valid().
|
||||
* set_memory_ro(), set_memory_rw(), set_memory_x() and set_memory_nx() accept
|
||||
only vmalloc and module addresses.
|
||||
* set_memory_valid() accepts any address.
|
||||
* Does not update mapping aliases.
|
||||
|
||||
arm64
|
||||
-----
|
||||
|
||||
* Provides set_memory_valid().
|
||||
* Provides the memory encryption helpers, which are effective only when the
|
||||
kernel runs as a confidential guest.
|
||||
* set_memory_ro(), set_memory_rw(), set_memory_x() and set_memory_nx() accept
|
||||
only vmalloc and module addresses:
|
||||
|
||||
- the range must fit in the VM area that contains its start
|
||||
- the VM area must have `VM_ALLOC` set and `VM_ALLOW_HUGE_VMAP` clear
|
||||
|
||||
* set_memory_valid() accepts any address.
|
||||
* The encryption helpers accept only direct map addresses.
|
||||
* The `set_memory()` functions propagate the read-only and the read-write
|
||||
changes to the direct map alias when `rodata=full` is in effect.
|
||||
* Splits leaf mappings before the update on the hardware that supports it.
|
||||
The split itself may be partial when it fails midway, but the permissions are
|
||||
left untouched in that case.
|
||||
Without support for splitting large mappings, a range that covers a leaf
|
||||
entry only partially fails with a WARN()ing and `-EINVAL`.
|
||||
If a range spans one or more full leaf entries and a partial leaf entry, the
|
||||
permissions of the full leaf entries are updated before the failure.
|
||||
* The `set_direct_map()` functions return 0 without doing anything when the
|
||||
direct map cannot be modified, see can_set_direct_map().
|
||||
* Skips the TLB flush in `set_memory()` when the update only turns an invalid
|
||||
mapping into a valid one.
|
||||
* Does not flush TLB in `set_direct_map()`.
|
||||
|
||||
LoongArch
|
||||
---------
|
||||
|
||||
* Accepts only the addresses above the hardware window and silently returns
|
||||
success for the rest, see `Direct map`_.
|
||||
* Does not update mapping aliases.
|
||||
* Does not split anything: a leaf entry is updated as a whole, which changes
|
||||
the permissions of the entire large mapping.
|
||||
* Flushes the TLB in `set_direct_map()`.
|
||||
|
||||
PowerPC
|
||||
-------
|
||||
|
||||
* Does not implement `set_direct_map()`.
|
||||
* Provides set_memory_np() and set_memory_p().
|
||||
* Rejects huge vmalloc mappings.
|
||||
* With the hash MMU on 64-bit systems accepts nothing but the vmalloc and the
|
||||
I/O regions.
|
||||
* With the radix MMU accepts direct map addresses, but still cannot split a
|
||||
large mapping.
|
||||
* Does not update mapping aliases.
|
||||
|
||||
riscv
|
||||
-----
|
||||
|
||||
* Implements both APIs only when the MMU is enabled.
|
||||
* Provides set_memory_rw_nx().
|
||||
* The `set_memory()` functions accept any mapped kernel address, including the
|
||||
direct map, but a vmalloc range must have the `pages` array of its VM area
|
||||
populated, which rules out vmap() and ioremap() mappings.
|
||||
* On 64-bit systems the `set_memory()` functions update the direct map alias of
|
||||
a vmalloc range, including the executable bit.
|
||||
* Does not split vmalloc ranges: a leaf entry is updated as a whole, which
|
||||
changes the permissions of the entire large mapping.
|
||||
* Splits the direct map on 64-bit systems.
|
||||
* Flushes the TLB in `set_direct_map()`.
|
||||
|
||||
s390
|
||||
----
|
||||
|
||||
* Provides set_memory_4k(), set_memory_rwnx() and the
|
||||
`__set_memory_*(start, end)` variants that take a range rather than a page
|
||||
count.
|
||||
* The `set_memory()` functions accept any mapped kernel address, including the
|
||||
direct map.
|
||||
* Skips the update of the executable bit when the hardware has no support for
|
||||
it.
|
||||
* The `set_memory()` functions propagate only the read-only and read-write
|
||||
changes to the direct map alias of a `VM_ALLOC` area, and deliberately not
|
||||
the executable bit.
|
||||
* Splits leaf PUD and PMD entries when the range is not aligned to them or when
|
||||
set_memory_4k() is requested.
|
||||
* Updates the page table entries with instructions that invalidate the
|
||||
corresponding TLB entries, so no separate flush is needed anywhere.
|
||||
|
||||
x86
|
||||
---
|
||||
|
||||
* Provides the largest set of operations on top of the common ones:
|
||||
|
||||
- the cache attribute helpers: set_memory_uc(), set_memory_wc(),
|
||||
set_memory_wb()
|
||||
- presence control: set_memory_np() and set_memory_p()
|
||||
- set_memory_4k()
|
||||
- set_memory_global() and set_memory_nonglobal()
|
||||
- the array variants that operate on `struct page` arrays or arrays of
|
||||
virtual addresses
|
||||
- memory encryption: set_memory_encrypted() and set_memory_decrypted()
|
||||
|
||||
* The `set_memory()` functions accept any mapped kernel address, including the
|
||||
direct map, and silently succeed for the unmapped holes inside it.
|
||||
* Does nothing in set_memory_x() and set_memory_nx() when the CPU has no
|
||||
execute permission bit.
|
||||
* The `set_memory()` functions apply the change to the direct map alias and,
|
||||
for the kernel image, to the high kernel mapping. The NX bit is never
|
||||
propagated, so that the direct map stays non-executable.
|
||||
* Splits large mappings on demand and can collapse them back when the
|
||||
permissions become uniform again.
|
||||
* Does not flush the TLB in `set_direct_map()`, but splitting a large
|
||||
mapping flushes it anyway.
|
||||
@@ -78,7 +78,7 @@ The frequency of such scans is defined by
|
||||
Reference
|
||||
---------
|
||||
.. kernel-doc:: mm/ksm.c
|
||||
:functions: mm_slot ksm_scan stable_node rmap_item
|
||||
:functions: ksm_mm_slot ksm_scan ksm_stable_node ksm_rmap_item
|
||||
|
||||
--
|
||||
Izik Eidus,
|
||||
|
||||
@@ -199,6 +199,7 @@ Usage
|
||||
-p Sort by pid.
|
||||
-P Sort by tgid.
|
||||
-n Sort by task command name.
|
||||
-M Sort by module name.
|
||||
-r Sort by memory release time.
|
||||
-s Sort by stack trace.
|
||||
-t Sort by times (default).
|
||||
@@ -240,8 +241,10 @@ Usage
|
||||
group ID numbers appear in <tgidlist>.
|
||||
--name <cmdlist> Select by task command name. This selects the blocks whose
|
||||
task command name appear in <cmdlist>.
|
||||
--module <modlist> Select by module name. This selects the blocks whose
|
||||
module name appear in <modlist>.
|
||||
|
||||
<pidlist>, <tgidlist>, <cmdlist> are single arguments in the form of a comma-separated list,
|
||||
<pidlist>, <tgidlist>, <cmdlist>, <modlist> are single arguments in the form of a comma-separated list,
|
||||
which offers a way to specify individual selecting rules.
|
||||
|
||||
|
||||
@@ -249,6 +252,7 @@ Usage
|
||||
./page_owner_sort <input> <output> --pid=1
|
||||
./page_owner_sort <input> <output> --tgid=1,2,3
|
||||
./page_owner_sort <input> <output> --name name1,name2
|
||||
./page_owner_sort <input> <output> --module xfs,ext4
|
||||
|
||||
STANDARD FORMAT SPECIFIERS
|
||||
==========================
|
||||
@@ -265,6 +269,7 @@ STANDARD FORMAT SPECIFIERS
|
||||
ft free_ts timestamp of the page when it was released
|
||||
at alloc_ts timestamp of the page when it was allocated
|
||||
ator allocator memory allocator for pages
|
||||
mod module kernel module name
|
||||
|
||||
For --cull option:
|
||||
|
||||
@@ -275,6 +280,7 @@ STANDARD FORMAT SPECIFIERS
|
||||
f free whether the page has been released or not
|
||||
st stacktrace stack trace of the page allocation
|
||||
ator allocator memory allocator for pages
|
||||
mod module kernel module name
|
||||
|
||||
Filtering page_owner output
|
||||
============================
|
||||
|
||||
@@ -537,6 +537,17 @@ We establish basic locking rules when interacting with page tables:
|
||||
* When changing a page table entry the page table lock for that page table
|
||||
**must** be held, except if you can safely assume nobody can access the page
|
||||
tables concurrently (such as on invocation of :c:func:`!free_pgtables`).
|
||||
* Page tables may be *walked* under RCU alone, as page tables are freed only
|
||||
after an RCU grace period has elapsed. However, any entry found must be
|
||||
revalidated after the page table lock is taken (such as the
|
||||
:c:func:`!pmd_same` recheck performed by :c:func:`!pte_offset_map_lock`)
|
||||
before it is acted upon. Changing an entry requires the page table lock
|
||||
and one of the locks that excludes teardown (any one of the mmap, VMA or
|
||||
rmap locks).
|
||||
* When traversing page tables under RCU alone it is important to take care
|
||||
when operating upon leaf entries - if the value is operated upon (for
|
||||
instance getting the folio associated with a PTE) an appropriate lock must
|
||||
be taken to prevent concurrent modification.
|
||||
* Reads from and writes to page table entries must be *appropriately*
|
||||
atomic. See the section on atomicity below for details.
|
||||
* Populating previously empty entries requires that the mmap or VMA locks are
|
||||
|
||||
@@ -1,4 +1,3 @@
|
||||
|
||||
.. SPDX-License-Identifier: GPL-2.0
|
||||
|
||||
=========================================
|
||||
@@ -192,32 +191,7 @@ to 4 on HugeTLB pages.
|
||||
|
||||
There's no remapping of vmemmap given that device-dax memory is not part of
|
||||
System RAM ranges initialized at boot. Thus the tail page deduplication
|
||||
happens at a later stage when we populate the sections. HugeTLB reuses the
|
||||
the head vmemmap page representing, whereas device-dax reuses the tail
|
||||
vmemmap page. This results in only half of the savings compared to HugeTLB.
|
||||
happens at a later stage when we populate the sections.
|
||||
|
||||
Deduplicated tail pages are not mapped read-only.
|
||||
|
||||
Here's how things look like on device-dax after the sections are populated::
|
||||
|
||||
+-----------+ ---virt_to_page---> +-----------+ mapping to +-----------+
|
||||
| | | 0 | -------------> | 0 |
|
||||
| | +-----------+ +-----------+
|
||||
| | | 1 | -------------> | 1 |
|
||||
| | +-----------+ +-----------+
|
||||
| | | 2 | ----------------^ ^ ^ ^ ^ ^
|
||||
| | +-----------+ | | | | |
|
||||
| | | 3 | ------------------+ | | | |
|
||||
| | +-----------+ | | | |
|
||||
| | | 4 | --------------------+ | | |
|
||||
| PMD | +-----------+ | | |
|
||||
| level | | 5 | ----------------------+ | |
|
||||
| mapping | +-----------+ | |
|
||||
| | | 6 | ------------------------+ |
|
||||
| | +-----------+ |
|
||||
| | | 7 | --------------------------+
|
||||
| | +-----------+
|
||||
| |
|
||||
| |
|
||||
| |
|
||||
+-----------+
|
||||
Deduplicated tail pages are not mapped read-only. The mapping layout is the same
|
||||
as HugeTLB.
|
||||
|
||||
+8
-3
@@ -12168,6 +12168,7 @@ F: Documentation/mm/hugetlbfs_reserv.rst
|
||||
F: Documentation/mm/vmemmap_dedup.rst
|
||||
F: fs/hugetlbfs/
|
||||
F: include/linux/hugetlb.h
|
||||
F: include/linux/vmemmap-optimization.h
|
||||
F: include/trace/events/hugetlbfs.h
|
||||
F: mm/hugetlb.c
|
||||
F: mm/hugetlb_cgroup.c
|
||||
@@ -17264,8 +17265,9 @@ T: git git://git.kernel.org/pub/scm/linux/kernel/git/akpm/mm
|
||||
F: mm/gup.c
|
||||
F: mm/gup_test.c
|
||||
F: mm/gup_test.h
|
||||
F: tools/mm/gup_bench.c
|
||||
F: tools/testing/selftests/mm/gup.c
|
||||
F: tools/testing/selftests/mm/gup_longterm.c
|
||||
F: tools/testing/selftests/mm/gup_test.c
|
||||
|
||||
MEMORY MANAGEMENT - KSM (Kernel Samepage Merging)
|
||||
M: Andrew Morton <[email protected]>
|
||||
@@ -17342,8 +17344,10 @@ F: Documentation/ABI/testing/sysfs-kernel-mm
|
||||
F: Documentation/ABI/testing/sysfs-kernel-mm-cma
|
||||
F: Documentation/ABI/testing/sysfs-kernel-mm-numa
|
||||
F: Documentation/admin-guide/mm/
|
||||
F: Documentation/core-api/memory-allocation.rst
|
||||
F: Documentation/core-api/mm-api.rst
|
||||
F: Documentation/mm/
|
||||
F: drivers/char/mem.c
|
||||
F: mm/char-mem.c
|
||||
F: include/linux/cma.h
|
||||
F: include/linux/dmapool.h
|
||||
F: include/linux/ioremap.h
|
||||
@@ -17361,6 +17365,7 @@ F: mm/mapping_dirty_helpers.c
|
||||
F: mm/page_idle.c
|
||||
F: mm/pgalloc-track.h
|
||||
F: mm/process_vm_access.c
|
||||
F: tools/lib/mm/
|
||||
F: tools/testing/selftests/mm/
|
||||
|
||||
MEMORY MANAGEMENT - NUMA MEMBLOCKS AND NUMA EMULATION
|
||||
@@ -17567,7 +17572,7 @@ L: [email protected]
|
||||
S: Maintained
|
||||
W: http://www.linux-mm.org
|
||||
T: git git://git.kernel.org/pub/scm/linux/kernel/git/akpm/mm
|
||||
F: drivers/char/mem.c
|
||||
F: mm/char-mem.c
|
||||
F: include/trace/events/mmap.h
|
||||
F: fs/proc/task_mmu.c
|
||||
F: fs/proc/task_nommu.c
|
||||
|
||||
@@ -526,13 +526,6 @@ config HAVE_ARCH_JUMP_LABEL
|
||||
config HAVE_ARCH_JUMP_LABEL_RELATIVE
|
||||
bool
|
||||
|
||||
config MMU_GATHER_TABLE_FREE
|
||||
bool
|
||||
|
||||
config MMU_GATHER_RCU_TABLE_FREE
|
||||
bool
|
||||
select MMU_GATHER_TABLE_FREE
|
||||
|
||||
config MMU_GATHER_PAGE_SIZE
|
||||
bool
|
||||
|
||||
@@ -548,7 +541,6 @@ config MMU_GATHER_MERGE_VMAS
|
||||
|
||||
config MMU_GATHER_NO_GATHER
|
||||
bool
|
||||
depends on MMU_GATHER_TABLE_FREE
|
||||
|
||||
config ARCH_WANT_IRQS_OFF_ACTIVATE_MM
|
||||
bool
|
||||
|
||||
@@ -71,7 +71,6 @@ config ALPHA
|
||||
select ARCH_STACKWALK
|
||||
select CPU_NO_EFFICIENT_FFS if !ALPHA_EV67
|
||||
select MMU_GATHER_NO_RANGE
|
||||
select MMU_GATHER_RCU_TABLE_FREE
|
||||
select SPARSE_IRQ
|
||||
select SPARSEMEM_EXTREME if SPARSEMEM
|
||||
select SYSCTL_EXCEPTION_TRACE
|
||||
|
||||
@@ -428,13 +428,6 @@ static inline pte_t pte_swp_clear_exclusive(pte_t pte)
|
||||
return pte;
|
||||
}
|
||||
|
||||
#define pte_ERROR(e) \
|
||||
printk("%s:%d: bad pte %016lx.\n", __FILE__, __LINE__, pte_val(e))
|
||||
#define pmd_ERROR(e) \
|
||||
printk("%s:%d: bad pmd %016lx.\n", __FILE__, __LINE__, pmd_val(e))
|
||||
#define pgd_ERROR(e) \
|
||||
printk("%s:%d: bad pgd %016lx.\n", __FILE__, __LINE__, pgd_val(e))
|
||||
|
||||
extern void paging_init(void);
|
||||
|
||||
/* We have our own get_unmapped_area */
|
||||
|
||||
@@ -72,7 +72,7 @@ static inline void p4d_populate(struct mm_struct *mm, p4d_t *p4dp, pud_t *pudp)
|
||||
set_p4d(p4dp, __p4d((unsigned long)pudp));
|
||||
}
|
||||
|
||||
#define __pud_free_tlb(tlb, pmd, addr) pud_free((tlb)->mm, pmd)
|
||||
#define __pud_free_tlb(tlb, pmd, addr) tlb_remove_ptdesc((tlb), virt_to_ptdesc(pmd))
|
||||
|
||||
#endif
|
||||
|
||||
@@ -83,10 +83,10 @@ static inline void pud_populate(struct mm_struct *mm, pud_t *pudp, pmd_t *pmdp)
|
||||
set_pud(pudp, __pud((unsigned long)pmdp));
|
||||
}
|
||||
|
||||
#define __pmd_free_tlb(tlb, pmd, addr) pmd_free((tlb)->mm, pmd)
|
||||
#define __pmd_free_tlb(tlb, pmd, addr) tlb_remove_ptdesc((tlb), virt_to_ptdesc(pmd))
|
||||
|
||||
#endif
|
||||
|
||||
#define __pte_free_tlb(tlb, pte, addr) pte_free((tlb)->mm, pte)
|
||||
#define __pte_free_tlb(tlb, pte, addr) tlb_remove_ptdesc((tlb), page_ptdesc(pte))
|
||||
|
||||
#endif /* _ASM_ARC_PGALLOC_H */
|
||||
|
||||
@@ -98,8 +98,6 @@
|
||||
/*
|
||||
* 1st level paging: pgd
|
||||
*/
|
||||
#define pgd_ERROR(e) \
|
||||
pr_crit("%s:%d: bad pgd %08lx.\n", __FILE__, __LINE__, pgd_val(e))
|
||||
|
||||
#if CONFIG_PGTABLE_LEVELS > 3
|
||||
|
||||
@@ -115,9 +113,6 @@
|
||||
/*
|
||||
* 2nd level paging: pud
|
||||
*/
|
||||
#define pud_ERROR(e) \
|
||||
pr_crit("%s:%d: bad pud %08lx.\n", __FILE__, __LINE__, pud_val(e))
|
||||
|
||||
#endif
|
||||
|
||||
#if CONFIG_PGTABLE_LEVELS > 2
|
||||
@@ -137,9 +132,6 @@
|
||||
/*
|
||||
* 3rd level paging: pmd
|
||||
*/
|
||||
#define pmd_ERROR(e) \
|
||||
pr_crit("%s:%d: bad pmd %08lx.\n", __FILE__, __LINE__, pmd_val(e))
|
||||
|
||||
#define pmd_pfn(pmd) ((pmd_val(pmd) & PMD_MASK) >> PAGE_SHIFT)
|
||||
#define pfn_pmd(pfn,prot) __pmd(((pfn) << PAGE_SHIFT) | pgprot_val(prot))
|
||||
|
||||
@@ -165,9 +157,6 @@
|
||||
/*
|
||||
* 4th level paging: pte
|
||||
*/
|
||||
#define pte_ERROR(e) \
|
||||
pr_crit("%s:%d: bad pte %08lx.\n", __FILE__, __LINE__, pte_val(e))
|
||||
|
||||
#define PFN_PTE_SHIFT PAGE_SHIFT
|
||||
#define pte_none(x) (!pte_val(x))
|
||||
#define pte_present(x) (pte_val(x) & _PAGE_PRESENT)
|
||||
|
||||
@@ -42,7 +42,6 @@ config ARM
|
||||
select ARCH_SUPPORTS_ATOMIC_RMW
|
||||
select ARCH_SUPPORTS_CFI
|
||||
select ARCH_SUPPORTS_HUGETLBFS if ARM_LPAE
|
||||
select ARCH_SUPPORTS_PER_VMA_LOCK
|
||||
select ARCH_SUPPORTS_RT
|
||||
select ARCH_USE_BUILTIN_BSWAP
|
||||
select ARCH_USE_CMPXCHG_LOCKREF
|
||||
@@ -135,7 +134,6 @@ config ARM
|
||||
select HAVE_PERF_REGS
|
||||
select HAVE_PERF_USER_STACK_DUMP
|
||||
select HAVE_POSIX_CPU_TIMERS_TASK_WORK
|
||||
select MMU_GATHER_RCU_TABLE_FREE if SMP && ARM_LPAE
|
||||
select HAVE_REGS_AND_STACK_ACCESS_API
|
||||
select HAVE_RSEQ
|
||||
select HAVE_RUST if CPU_LITTLE_ENDIAN && CPU_32v7 && !KASAN
|
||||
|
||||
@@ -44,13 +44,6 @@
|
||||
#define LIBRARY_TEXT_START 0x0c000000
|
||||
|
||||
#ifndef __ASSEMBLY__
|
||||
extern void __pte_error(const char *file, int line, pte_t);
|
||||
extern void __pmd_error(const char *file, int line, pmd_t);
|
||||
extern void __pgd_error(const char *file, int line, pgd_t);
|
||||
|
||||
#define pte_ERROR(pte) __pte_error(__FILE__, __LINE__, pte)
|
||||
#define pmd_ERROR(pmd) __pmd_error(__FILE__, __LINE__, pmd)
|
||||
#define pgd_ERROR(pgd) __pgd_error(__FILE__, __LINE__, pgd)
|
||||
|
||||
/*
|
||||
* This is the lowest virtual address we can permit any user space
|
||||
|
||||
@@ -753,23 +753,6 @@ void __readwrite_bug(const char *fn)
|
||||
}
|
||||
EXPORT_SYMBOL(__readwrite_bug);
|
||||
|
||||
#ifdef CONFIG_MMU
|
||||
void __pte_error(const char *file, int line, pte_t pte)
|
||||
{
|
||||
pr_err("%s:%d: bad pte %08llx.\n", file, line, (long long)pte_val(pte));
|
||||
}
|
||||
|
||||
void __pmd_error(const char *file, int line, pmd_t pmd)
|
||||
{
|
||||
pr_err("%s:%d: bad pmd %08llx.\n", file, line, (long long)pmd_val(pmd));
|
||||
}
|
||||
|
||||
void __pgd_error(const char *file, int line, pgd_t pgd)
|
||||
{
|
||||
pr_err("%s:%d: bad pgd %08llx.\n", file, line, (long long)pgd_val(pgd));
|
||||
}
|
||||
#endif
|
||||
|
||||
asmlinkage void __div0(void)
|
||||
{
|
||||
pr_err("Division by zero in kernel.\n");
|
||||
|
||||
@@ -81,7 +81,6 @@ config ARM64
|
||||
select ARCH_HAS_PTE_PROTNONE
|
||||
select ARCH_SUPPORTS_NUMA_BALANCING
|
||||
select ARCH_SUPPORTS_PAGE_TABLE_CHECK
|
||||
select ARCH_SUPPORTS_PER_VMA_LOCK
|
||||
select ARCH_SUPPORTS_HUGE_PFNMAP if TRANSPARENT_HUGEPAGE
|
||||
select ARCH_SUPPORTS_RMM
|
||||
select ARCH_SUPPORTS_RT
|
||||
@@ -223,7 +222,6 @@ config ARM64
|
||||
select HAVE_RELIABLE_STACKTRACE
|
||||
select HAVE_POSIX_CPU_TIMERS_TASK_WORK
|
||||
select HAVE_FUNCTION_ARG_ACCESS_API
|
||||
select MMU_GATHER_RCU_TABLE_FREE
|
||||
select HAVE_RSEQ
|
||||
select HAVE_RUST
|
||||
select HAVE_STACKPROTECTOR
|
||||
|
||||
@@ -23,9 +23,7 @@ unsigned long mte_copy_tags_from_user(void *to, const void __user *from,
|
||||
unsigned long n);
|
||||
unsigned long mte_copy_tags_to_user(void __user *to, void *from,
|
||||
unsigned long n);
|
||||
int mte_save_tags(struct page *page);
|
||||
void mte_save_page_tags(const void *page_addr, void *tag_storage);
|
||||
void mte_restore_tags(swp_entry_t entry, struct page *page);
|
||||
void mte_restore_page_tags(void *page_addr, const void *tag_storage);
|
||||
void mte_invalidate_tags(int type, pgoff_t offset);
|
||||
void mte_invalidate_tags_area(int type);
|
||||
|
||||
@@ -107,9 +107,6 @@ static inline void arch_leave_lazy_mmu_mode(void)
|
||||
__flush_tlb_range(vma, address, address + PMD_SIZE, PMD_SIZE, 2, \
|
||||
TLBF_NOBROADCAST | TLBF_NONOTIFY | TLBF_NOWALKCACHE)
|
||||
|
||||
#define pte_ERROR(e) \
|
||||
pr_err("%s:%d: bad pte %016llx.\n", __FILE__, __LINE__, pte_val(e))
|
||||
|
||||
#ifdef CONFIG_ARM64_PA_BITS_52
|
||||
static inline phys_addr_t __pte_to_phys(pte_t pte)
|
||||
{
|
||||
@@ -866,9 +863,6 @@ static inline unsigned long pmd_page_vaddr(pmd_t pmd)
|
||||
|
||||
#if CONFIG_PGTABLE_LEVELS > 2
|
||||
|
||||
#define pmd_ERROR(e) \
|
||||
pr_err("%s:%d: bad pmd %016llx.\n", __FILE__, __LINE__, pmd_val(e))
|
||||
|
||||
#define pud_none(pud) (!pud_val(pud))
|
||||
#define pud_bad(pud) ((pud_val(pud) & PUD_TYPE_MASK) != \
|
||||
PUD_TYPE_TABLE)
|
||||
@@ -960,9 +954,6 @@ static inline bool mm_pud_folded(const struct mm_struct *mm)
|
||||
}
|
||||
#define mm_pud_folded mm_pud_folded
|
||||
|
||||
#define pud_ERROR(e) \
|
||||
pr_err("%s:%d: bad pud %016llx.\n", __FILE__, __LINE__, pud_val(e))
|
||||
|
||||
#define p4d_none(p4d) (pgtable_l4_enabled() && !p4d_val(p4d))
|
||||
#define p4d_bad(p4d) (pgtable_l4_enabled() && \
|
||||
((p4d_val(p4d) & P4D_TYPE_MASK) != \
|
||||
@@ -1088,9 +1079,6 @@ static inline bool mm_p4d_folded(const struct mm_struct *mm)
|
||||
}
|
||||
#define mm_p4d_folded mm_p4d_folded
|
||||
|
||||
#define p4d_ERROR(e) \
|
||||
pr_err("%s:%d: bad p4d %016llx.\n", __FILE__, __LINE__, p4d_val(e))
|
||||
|
||||
#define pgd_none(pgd) (pgtable_l5_enabled() && !pgd_val(pgd))
|
||||
#define pgd_bad(pgd) (pgtable_l5_enabled() && \
|
||||
((pgd_val(pgd) & PGD_TYPE_MASK) != \
|
||||
@@ -1217,9 +1205,6 @@ p4d_t *p4d_offset_lockless_folded(pgd_t *pgdp, pgd_t pgd, unsigned long addr)
|
||||
|
||||
#endif /* CONFIG_PGTABLE_LEVELS > 4 */
|
||||
|
||||
#define pgd_ERROR(e) \
|
||||
pr_err("%s:%d: bad pgd %016llx.\n", __FILE__, __LINE__, pgd_val(e))
|
||||
|
||||
#define pgd_set_fixmap(addr) ((pgd_t *)set_fixmap_offset(FIX_PGD, addr))
|
||||
#define pgd_clear_fixmap() clear_fixmap(FIX_PGD)
|
||||
|
||||
|
||||
@@ -11,9 +11,8 @@ bool can_set_direct_map(void);
|
||||
|
||||
int set_memory_valid(unsigned long addr, int numpages, int enable);
|
||||
|
||||
int set_direct_map_invalid_noflush(struct page *page);
|
||||
int set_direct_map_default_noflush(struct page *page);
|
||||
int set_direct_map_valid_noflush(struct page *page, unsigned nr, bool valid);
|
||||
int set_direct_map_invalid_noflush(struct page *page, unsigned int numpages);
|
||||
int set_direct_map_default_noflush(struct page *page, unsigned int numpages);
|
||||
bool kernel_page_present(struct page *page);
|
||||
|
||||
int set_memory_encrypted(unsigned long addr, int numpages);
|
||||
|
||||
+14
-18
@@ -20,22 +20,17 @@ void mte_free_tag_storage(char *storage)
|
||||
kfree(storage);
|
||||
}
|
||||
|
||||
int mte_save_tags(struct page *page)
|
||||
static int mte_save_tags(swp_entry_t entry, struct page *page)
|
||||
{
|
||||
void *tag_storage, *ret;
|
||||
|
||||
if (!page_mte_tagged(page))
|
||||
return 0;
|
||||
|
||||
tag_storage = mte_allocate_tag_storage();
|
||||
if (!tag_storage)
|
||||
return -ENOMEM;
|
||||
|
||||
mte_save_page_tags(page_address(page), tag_storage);
|
||||
|
||||
/* lookup the swap entry.val from the page */
|
||||
ret = xa_store(&mte_pages, page_swap_entry(page).val, tag_storage,
|
||||
GFP_KERNEL);
|
||||
ret = xa_store(&mte_pages, entry.val, tag_storage, GFP_KERNEL);
|
||||
if (WARN(xa_is_err(ret), "Failed to store MTE tags")) {
|
||||
mte_free_tag_storage(tag_storage);
|
||||
return xa_err(ret);
|
||||
@@ -47,7 +42,7 @@ int mte_save_tags(struct page *page)
|
||||
return 0;
|
||||
}
|
||||
|
||||
void mte_restore_tags(swp_entry_t entry, struct page *page)
|
||||
static void mte_restore_tags(swp_entry_t entry, struct page *page)
|
||||
{
|
||||
void *tags = xa_load(&mte_pages, entry.val);
|
||||
|
||||
@@ -68,13 +63,6 @@ void mte_invalidate_tags(int type, pgoff_t offset)
|
||||
mte_free_tag_storage(tags);
|
||||
}
|
||||
|
||||
static inline void __mte_invalidate_tags(struct page *page)
|
||||
{
|
||||
swp_entry_t entry = page_swap_entry(page);
|
||||
|
||||
mte_invalidate_tags(swp_type(entry), swp_offset(entry));
|
||||
}
|
||||
|
||||
void mte_invalidate_tags_area(int type)
|
||||
{
|
||||
swp_entry_t entry = swp_entry(type, 0);
|
||||
@@ -102,15 +90,23 @@ int arch_prepare_to_swap(struct folio *folio)
|
||||
nr = folio_nr_pages(folio);
|
||||
|
||||
for (i = 0; i < nr; i++) {
|
||||
err = mte_save_tags(folio_page(folio, i));
|
||||
struct page *page = folio_page(folio, i);
|
||||
|
||||
if (!page_mte_tagged(page))
|
||||
continue;
|
||||
|
||||
err = mte_save_tags(folio_swap_entry(folio, i), page);
|
||||
if (err)
|
||||
goto out;
|
||||
}
|
||||
return 0;
|
||||
|
||||
out:
|
||||
while (i--)
|
||||
__mte_invalidate_tags(folio_page(folio, i));
|
||||
while (i--) {
|
||||
swp_entry_t swap = folio_swap_entry(folio, i);
|
||||
|
||||
mte_invalidate_tags(swp_type(swap), swp_offset(swap));
|
||||
}
|
||||
return err;
|
||||
}
|
||||
|
||||
|
||||
@@ -251,7 +251,7 @@ int set_memory_valid(unsigned long addr, int numpages, int enable)
|
||||
__pgprot(PTE_PRESENT_VALID_KERNEL));
|
||||
}
|
||||
|
||||
int set_direct_map_invalid_noflush(struct page *page)
|
||||
int set_direct_map_invalid_noflush(struct page *page, unsigned int numpages)
|
||||
{
|
||||
pgprot_t clear_mask = __pgprot(PTE_PRESENT_VALID_KERNEL);
|
||||
pgprot_t set_mask = __pgprot(PTE_PRESENT_INVALID);
|
||||
@@ -260,10 +260,10 @@ int set_direct_map_invalid_noflush(struct page *page)
|
||||
return 0;
|
||||
|
||||
return update_range_prot((unsigned long)page_address(page),
|
||||
PAGE_SIZE, set_mask, clear_mask);
|
||||
PAGE_SIZE * numpages, set_mask, clear_mask);
|
||||
}
|
||||
|
||||
int set_direct_map_default_noflush(struct page *page)
|
||||
int set_direct_map_default_noflush(struct page *page, unsigned int numpages)
|
||||
{
|
||||
pgprot_t set_mask = __pgprot(PTE_PRESENT_VALID_KERNEL | PTE_WRITE);
|
||||
pgprot_t clear_mask = __pgprot(PTE_PRESENT_INVALID | PTE_RDONLY);
|
||||
@@ -272,7 +272,7 @@ int set_direct_map_default_noflush(struct page *page)
|
||||
return 0;
|
||||
|
||||
return update_range_prot((unsigned long)page_address(page),
|
||||
PAGE_SIZE, set_mask, clear_mask);
|
||||
PAGE_SIZE * numpages, set_mask, clear_mask);
|
||||
}
|
||||
|
||||
int __set_memory_enc_dec(unsigned long addr, int numpages, bool encrypt)
|
||||
@@ -319,23 +319,7 @@ int __set_memory_enc_dec(unsigned long addr, int numpages, bool encrypt)
|
||||
__pgprot(PTE_PRESENT_INVALID));
|
||||
}
|
||||
|
||||
int set_direct_map_valid_noflush(struct page *page, unsigned nr, bool valid)
|
||||
{
|
||||
unsigned long addr = (unsigned long)page_address(page);
|
||||
|
||||
if (!can_set_direct_map())
|
||||
return 0;
|
||||
|
||||
return set_memory_valid(addr, nr, valid);
|
||||
}
|
||||
|
||||
#ifdef CONFIG_DEBUG_PAGEALLOC
|
||||
/*
|
||||
* This is - apart from the return value - doing the same
|
||||
* thing as the new set_direct_map_valid_noflush() function.
|
||||
*
|
||||
* Unify? Explain the conceptual differences?
|
||||
*/
|
||||
void __kernel_map_pages(struct page *page, int numpages, int enable)
|
||||
{
|
||||
if (!can_set_direct_map())
|
||||
|
||||
@@ -23,10 +23,6 @@
|
||||
#define PTRS_PER_PMD 1
|
||||
#define PTRS_PER_PTE (PAGE_SIZE / sizeof(pte_t))
|
||||
|
||||
#define pte_ERROR(e) \
|
||||
pr_err("%s:%d: bad pte %08lx.\n", __FILE__, __LINE__, (e).pte_low)
|
||||
#define pgd_ERROR(e) \
|
||||
pr_err("%s:%d: bad pgd %08lx.\n", __FILE__, __LINE__, pgd_val(e))
|
||||
|
||||
#define PFN_PTE_SHIFT PAGE_SHIFT
|
||||
#define pmd_pfn(pmd) (pmd_phys(pmd) >> PAGE_SHIFT)
|
||||
|
||||
@@ -94,9 +94,6 @@
|
||||
#endif
|
||||
|
||||
/* Any bigger and the PTE disappears. */
|
||||
#define pgd_ERROR(e) \
|
||||
printk(KERN_ERR "%s:%d: bad pgd %08lx.\n", __FILE__, __LINE__,\
|
||||
pgd_val(e))
|
||||
|
||||
/*
|
||||
* Page Protection Constants. Includes (in this variant) cache attributes.
|
||||
|
||||
@@ -69,7 +69,6 @@ config LOONGARCH
|
||||
select ARCH_SUPPORTS_MSEAL_SYSTEM_MAPPINGS
|
||||
select ARCH_HAS_PTE_PROTNONE if 64BIT
|
||||
select ARCH_SUPPORTS_NUMA_BALANCING if NUMA
|
||||
select ARCH_SUPPORTS_PER_VMA_LOCK
|
||||
select ARCH_SUPPORTS_RT
|
||||
select ARCH_SUPPORTS_SCHED_SMT if SMP
|
||||
select ARCH_SUPPORTS_SCHED_MC if SMP
|
||||
@@ -188,7 +187,6 @@ config LOONGARCH
|
||||
select IRQ_LOONGARCH_CPU
|
||||
select LOCK_MM_AND_FIND_VMA
|
||||
select MMU_GATHER_MERGE_VMAS if MMU
|
||||
select MMU_GATHER_RCU_TABLE_FREE
|
||||
select MODULES_USE_ELF_RELA if MODULES
|
||||
select NEED_PER_CPU_EMBED_FIRST_CHUNK
|
||||
select NEED_PER_CPU_PAGE_FIRST_CHUNK
|
||||
|
||||
@@ -72,6 +72,7 @@
|
||||
|
||||
#include <linux/mm_types.h>
|
||||
#include <linux/mmzone.h>
|
||||
#include <linux/vmemmap-optimization.h>
|
||||
#include <asm/fixmap.h>
|
||||
#include <asm/sparsemem.h>
|
||||
|
||||
@@ -135,19 +136,6 @@ struct vm_area_struct;
|
||||
#define ptep_get(ptep) READ_ONCE(*(ptep))
|
||||
#define pmdp_get(pmdp) READ_ONCE(*(pmdp))
|
||||
|
||||
#define pte_ERROR(e) \
|
||||
pr_err("%s:%d: bad pte %016lx.\n", __FILE__, __LINE__, pte_val(e))
|
||||
#ifndef __PAGETABLE_PMD_FOLDED
|
||||
#define pmd_ERROR(e) \
|
||||
pr_err("%s:%d: bad pmd %016lx.\n", __FILE__, __LINE__, pmd_val(e))
|
||||
#endif
|
||||
#ifndef __PAGETABLE_PUD_FOLDED
|
||||
#define pud_ERROR(e) \
|
||||
pr_err("%s:%d: bad pud %016lx.\n", __FILE__, __LINE__, pud_val(e))
|
||||
#endif
|
||||
#define pgd_ERROR(e) \
|
||||
pr_err("%s:%d: bad pgd %016lx.\n", __FILE__, __LINE__, pgd_val(e))
|
||||
|
||||
extern pte_t invalid_pte_table[PTRS_PER_PTE];
|
||||
|
||||
#ifndef __PAGETABLE_PUD_FOLDED
|
||||
|
||||
@@ -15,8 +15,7 @@ int set_memory_ro(unsigned long addr, int numpages);
|
||||
int set_memory_rw(unsigned long addr, int numpages);
|
||||
|
||||
bool kernel_page_present(struct page *page);
|
||||
int set_direct_map_default_noflush(struct page *page);
|
||||
int set_direct_map_invalid_noflush(struct page *page);
|
||||
int set_direct_map_valid_noflush(struct page *page, unsigned nr, bool valid);
|
||||
int set_direct_map_default_noflush(struct page *page, unsigned int nr);
|
||||
int set_direct_map_invalid_noflush(struct page *page, unsigned int nr);
|
||||
|
||||
#endif /* _ASM_LOONGARCH_SET_MEMORY_H */
|
||||
|
||||
@@ -197,13 +197,15 @@ void __init __set_fixmap(enum fixed_addresses idx,
|
||||
phys_addr_t phys, pgprot_t flags)
|
||||
{
|
||||
unsigned long addr = __fix_to_virt(idx);
|
||||
char str[PTVAL_STR_MAX];
|
||||
pte_t *ptep;
|
||||
|
||||
BUG_ON(idx <= FIX_HOLE || idx >= __end_of_fixed_addresses);
|
||||
|
||||
ptep = populate_kernel_pte(addr);
|
||||
if (!pte_none(ptep_get(ptep))) {
|
||||
pte_ERROR(*ptep);
|
||||
ptval_to_str(str, pte_val(*ptep));
|
||||
pr_err("unexpected set PTE at %lx in %s: %s\n", addr, __func__, str);
|
||||
return;
|
||||
}
|
||||
|
||||
|
||||
@@ -198,41 +198,22 @@ bool kernel_page_present(struct page *page)
|
||||
return pte_present(ptep_get(pte));
|
||||
}
|
||||
|
||||
int set_direct_map_default_noflush(struct page *page)
|
||||
int set_direct_map_default_noflush(struct page *page, unsigned int nr)
|
||||
{
|
||||
unsigned long addr = (unsigned long)page_address(page);
|
||||
|
||||
if (addr < vm_map_base)
|
||||
return 0;
|
||||
|
||||
return __set_memory(addr, 1, PAGE_KERNEL, __pgprot(0));
|
||||
return __set_memory(addr, nr, PAGE_KERNEL, __pgprot(0));
|
||||
}
|
||||
|
||||
int set_direct_map_invalid_noflush(struct page *page)
|
||||
int set_direct_map_invalid_noflush(struct page *page, unsigned int nr)
|
||||
{
|
||||
unsigned long addr = (unsigned long)page_address(page);
|
||||
|
||||
if (addr < vm_map_base)
|
||||
return 0;
|
||||
|
||||
return __set_memory(addr, 1, __pgprot(0), __pgprot(_PAGE_PRESENT | _PAGE_VALID));
|
||||
}
|
||||
|
||||
int set_direct_map_valid_noflush(struct page *page, unsigned nr, bool valid)
|
||||
{
|
||||
unsigned long addr = (unsigned long)page_address(page);
|
||||
pgprot_t set, clear;
|
||||
|
||||
if (addr < vm_map_base)
|
||||
return 0;
|
||||
|
||||
if (valid) {
|
||||
set = PAGE_KERNEL;
|
||||
clear = __pgprot(0);
|
||||
} else {
|
||||
set = __pgprot(0);
|
||||
clear = __pgprot(_PAGE_PRESENT | _PAGE_VALID);
|
||||
}
|
||||
|
||||
return __set_memory(addr, nr, set, clear);
|
||||
return __set_memory(addr, nr, __pgprot(0), __pgprot(_PAGE_PRESENT | _PAGE_VALID));
|
||||
}
|
||||
|
||||
@@ -29,6 +29,7 @@ config M68K
|
||||
select HAVE_ARCH_LIBGCC_H
|
||||
select HAVE_ARCH_SECCOMP
|
||||
select HAVE_ARCH_SECCOMP_FILTER
|
||||
select HAVE_ARCH_TLB_REMOVE_TABLE if MMU_MOTOROLA
|
||||
select HAVE_ASM_MODVERSIONS
|
||||
select HAVE_DEBUG_BUGVERBOSE
|
||||
select HAVE_EFFICIENT_UNALIGNED_ACCESS if !CPU_HAS_NO_UNALIGNED
|
||||
|
||||
@@ -39,10 +39,7 @@ extern inline pmd_t *pmd_alloc_kernel(pgd_t *pgd, unsigned long address)
|
||||
static inline void __pte_free_tlb(struct mmu_gather *tlb, pgtable_t pgtable,
|
||||
unsigned long address)
|
||||
{
|
||||
struct ptdesc *ptdesc = virt_to_ptdesc(pgtable);
|
||||
|
||||
pagetable_dtor(ptdesc);
|
||||
pagetable_free(ptdesc);
|
||||
tlb_remove_ptdesc(tlb, virt_to_ptdesc(pgtable));
|
||||
}
|
||||
|
||||
static inline pgtable_t pte_alloc_one(struct mm_struct *mm)
|
||||
|
||||
@@ -137,12 +137,6 @@ static inline int pmd_bad2(pmd_t *pmd) { return 0; }
|
||||
#define pmd_present(pmd) (!pmd_none2(&(pmd)))
|
||||
static inline void pmd_clear(pmd_t *pmdp) { pmd_val(*pmdp) = 0; }
|
||||
|
||||
#define pte_ERROR(e) \
|
||||
printk(KERN_ERR "%s:%d: bad pte %08lx.\n", \
|
||||
__FILE__, __LINE__, pte_val(e))
|
||||
#define pgd_ERROR(e) \
|
||||
printk(KERN_ERR "%s:%d: bad pgd %08lx.\n", \
|
||||
__FILE__, __LINE__, pgd_val(e))
|
||||
|
||||
/*
|
||||
* The following only work if pte_present() is true.
|
||||
|
||||
@@ -17,6 +17,7 @@ enum m68k_table_types {
|
||||
extern void init_pointer_table(void *table, int type);
|
||||
extern void *get_pointer_table(struct mm_struct *mm, int type);
|
||||
extern int free_pointer_table(void *table, int type);
|
||||
extern void __tlb_remove_table(void *table);
|
||||
|
||||
/*
|
||||
* Allocate and free page tables. The xxx_kernel() versions are
|
||||
@@ -47,7 +48,7 @@ static inline void pte_free(struct mm_struct *mm, pgtable_t pgtable)
|
||||
static inline void __pte_free_tlb(struct mmu_gather *tlb, pgtable_t pgtable,
|
||||
unsigned long address)
|
||||
{
|
||||
free_pointer_table(pgtable, TABLE_PTE);
|
||||
tlb_remove_table(tlb, (void *)((unsigned long)pgtable | TABLE_PTE));
|
||||
}
|
||||
|
||||
|
||||
@@ -61,10 +62,10 @@ static inline int pmd_free(struct mm_struct *mm, pmd_t *pmd)
|
||||
return free_pointer_table(pmd, TABLE_PMD);
|
||||
}
|
||||
|
||||
static inline int __pmd_free_tlb(struct mmu_gather *tlb, pmd_t *pmd,
|
||||
unsigned long address)
|
||||
static inline void __pmd_free_tlb(struct mmu_gather *tlb, pmd_t *pmd,
|
||||
unsigned long address)
|
||||
{
|
||||
return free_pointer_table(pmd, TABLE_PMD);
|
||||
tlb_remove_table(tlb, (void *)((unsigned long)pmd | TABLE_PMD));
|
||||
}
|
||||
|
||||
|
||||
|
||||
@@ -131,14 +131,6 @@ static inline void pud_set(pud_t *pudp, pmd_t *pmdp)
|
||||
#define pud_clear(pudp) ({ pud_val(*pudp) = 0; })
|
||||
#define pud_page(pud) (mem_map + ((unsigned long)(__va(pud_val(pud)) - PAGE_OFFSET) >> PAGE_SHIFT))
|
||||
|
||||
#define pte_ERROR(e) \
|
||||
printk("%s:%d: bad pte %08lx.\n", __FILE__, __LINE__, pte_val(e))
|
||||
#define pmd_ERROR(e) \
|
||||
printk("%s:%d: bad pmd %08lx.\n", __FILE__, __LINE__, pmd_val(e))
|
||||
#define pgd_ERROR(e) \
|
||||
printk("%s:%d: bad pgd %08lx.\n", __FILE__, __LINE__, pgd_val(e))
|
||||
|
||||
|
||||
/*
|
||||
* The following only work if pte_present() is true.
|
||||
* Undefined behaviour if not..
|
||||
|
||||
@@ -119,13 +119,6 @@ static inline int pmd_present2 (pmd_t *pmd) { return pmd_val (*pmd) & SUN3_PMD_V
|
||||
#define pmd_present(pmd) (!pmd_none2(&(pmd)))
|
||||
static inline void pmd_clear (pmd_t *pmdp) { pmd_val (*pmdp) = 0; }
|
||||
|
||||
|
||||
#define pte_ERROR(e) \
|
||||
pr_err("%s:%d: bad pte %08lx.\n", __FILE__, __LINE__, pte_val(e))
|
||||
#define pgd_ERROR(e) \
|
||||
pr_err("%s:%d: bad pgd %08lx.\n", __FILE__, __LINE__, pgd_val(e))
|
||||
|
||||
|
||||
/*
|
||||
* The following only work if pte_present() is true.
|
||||
* Undefined behaviour if not...
|
||||
|
||||
+77
-42
@@ -20,6 +20,7 @@
|
||||
#include <linux/init.h>
|
||||
#include <linux/memblock.h>
|
||||
#include <linux/gfp.h>
|
||||
#include <linux/cleanup.h>
|
||||
|
||||
#include <asm/setup.h>
|
||||
#include <linux/uaccess.h>
|
||||
@@ -103,6 +104,8 @@ static struct list_head ptable_list[3] = {
|
||||
LIST_HEAD_INIT(ptable_list[2]),
|
||||
};
|
||||
|
||||
static DEFINE_SPINLOCK(ptable_lock);
|
||||
|
||||
#define PD_PTABLE(ptdesc) ((ptable_desc *)&(virt_to_ptdesc((void *)(ptdesc))->pt_list))
|
||||
#define PD_PTDESC(ptable) (list_entry(ptable, struct ptdesc, pt_list))
|
||||
#define PD_MARKBITS(dp) (*(unsigned int *)&PD_PTDESC(dp)->pt_index)
|
||||
@@ -139,52 +142,65 @@ void __init init_pointer_table(void *table, int type)
|
||||
return;
|
||||
}
|
||||
|
||||
/*
|
||||
* For a pointer table for a user process address space, a
|
||||
* table is taken from a ptdesc allocated for the purpose. Each
|
||||
* ptdesc can hold 8 pointer tables. The ptdesc is remapped in
|
||||
* virtual address space to be noncacheable.
|
||||
*/
|
||||
static void *add_pointer_table(struct mm_struct *mm, int type)
|
||||
{
|
||||
struct ptdesc *ptdesc;
|
||||
ptable_desc *new;
|
||||
void *pt_addr;
|
||||
|
||||
ptdesc = pagetable_alloc(GFP_KERNEL | __GFP_ZERO, 0);
|
||||
if (!ptdesc)
|
||||
return NULL;
|
||||
|
||||
pt_addr = ptdesc_address(ptdesc);
|
||||
|
||||
switch (type) {
|
||||
case TABLE_PTE:
|
||||
/*
|
||||
* m68k doesn't have SPLIT_PTE_PTLOCKS for not having
|
||||
* SMP.
|
||||
*/
|
||||
pagetable_pte_ctor(mm, ptdesc);
|
||||
break;
|
||||
case TABLE_PMD:
|
||||
pagetable_pmd_ctor(mm, ptdesc);
|
||||
break;
|
||||
case TABLE_PGD:
|
||||
pagetable_pgd_ctor(ptdesc);
|
||||
break;
|
||||
}
|
||||
|
||||
mmu_page_ctor(pt_addr);
|
||||
|
||||
new = PD_PTABLE(pt_addr);
|
||||
|
||||
PD_MARKBITS(new) = ptable_mask(type) - 1;
|
||||
scoped_guard(spinlock_bh, &ptable_lock)
|
||||
list_add(new, &ptable_list[type]);
|
||||
|
||||
return (pmd_t *)pt_addr;
|
||||
}
|
||||
|
||||
void *get_pointer_table(struct mm_struct *mm, int type)
|
||||
{
|
||||
ptable_desc *dp = ptable_list[type].next;
|
||||
unsigned int mask = list_empty(&ptable_list[type]) ? 0 : PD_MARKBITS(dp);
|
||||
unsigned int tmp, off;
|
||||
unsigned long mask;
|
||||
ptable_desc *dp;
|
||||
void *ret;
|
||||
|
||||
spin_lock_bh(&ptable_lock);
|
||||
dp = ptable_list[type].next;
|
||||
mask = list_empty(&ptable_list[type]) ? 0 : PD_MARKBITS(dp);
|
||||
|
||||
/*
|
||||
* For a pointer table for a user process address space, a
|
||||
* table is taken from a ptdesc allocated for the purpose. Each
|
||||
* ptdesc can hold 8 pointer tables. The ptdesc is remapped in
|
||||
* virtual address space to be noncacheable.
|
||||
*/
|
||||
if (mask == 0) {
|
||||
struct ptdesc *ptdesc;
|
||||
ptable_desc *new;
|
||||
void *pt_addr;
|
||||
|
||||
ptdesc = pagetable_alloc(GFP_KERNEL | __GFP_ZERO, 0);
|
||||
if (!ptdesc)
|
||||
return NULL;
|
||||
|
||||
pt_addr = ptdesc_address(ptdesc);
|
||||
|
||||
switch (type) {
|
||||
case TABLE_PTE:
|
||||
/*
|
||||
* m68k doesn't have SPLIT_PTE_PTLOCKS for not having
|
||||
* SMP.
|
||||
*/
|
||||
pagetable_pte_ctor(mm, ptdesc);
|
||||
break;
|
||||
case TABLE_PMD:
|
||||
pagetable_pmd_ctor(mm, ptdesc);
|
||||
break;
|
||||
case TABLE_PGD:
|
||||
pagetable_pgd_ctor(ptdesc);
|
||||
break;
|
||||
}
|
||||
|
||||
mmu_page_ctor(pt_addr);
|
||||
|
||||
new = PD_PTABLE(pt_addr);
|
||||
PD_MARKBITS(new) = ptable_mask(type) - 1;
|
||||
list_add_tail(new, dp);
|
||||
|
||||
return (pmd_t *)pt_addr;
|
||||
spin_unlock_bh(&ptable_lock);
|
||||
return add_pointer_table(mm, type);
|
||||
}
|
||||
|
||||
for (tmp = 1, off = 0; (mask & tmp) == 0; tmp <<= 1, off += ptable_size(type))
|
||||
@@ -194,7 +210,10 @@ void *get_pointer_table(struct mm_struct *mm, int type)
|
||||
/* move to end of list */
|
||||
list_move_tail(dp, &ptable_list[type]);
|
||||
}
|
||||
return ptdesc_address(PD_PTDESC(dp)) + off;
|
||||
|
||||
ret = ptdesc_address(PD_PTDESC(dp)) + off;
|
||||
spin_unlock_bh(&ptable_lock);
|
||||
return ret;
|
||||
}
|
||||
|
||||
int free_pointer_table(void *table, int type)
|
||||
@@ -204,6 +223,8 @@ int free_pointer_table(void *table, int type)
|
||||
unsigned long pt_addr = ptable & PAGE_MASK;
|
||||
unsigned int mask = 1U << ((ptable - pt_addr)/ptable_size(type));
|
||||
|
||||
spin_lock_bh(&ptable_lock);
|
||||
|
||||
dp = PD_PTABLE(pt_addr);
|
||||
if (PD_MARKBITS (dp) & mask)
|
||||
panic ("table already free!");
|
||||
@@ -213,6 +234,8 @@ int free_pointer_table(void *table, int type)
|
||||
if (PD_MARKBITS(dp) == ptable_mask(type)) {
|
||||
/* all tables in ptdesc are free, free ptdesc */
|
||||
list_del(dp);
|
||||
spin_unlock_bh(&ptable_lock);
|
||||
|
||||
mmu_page_dtor((void *)pt_addr);
|
||||
pagetable_dtor_free(virt_to_ptdesc((void *)pt_addr));
|
||||
return 1;
|
||||
@@ -223,9 +246,21 @@ int free_pointer_table(void *table, int type)
|
||||
*/
|
||||
list_move(dp, &ptable_list[type]);
|
||||
}
|
||||
|
||||
spin_unlock_bh(&ptable_lock);
|
||||
return 0;
|
||||
}
|
||||
|
||||
void __tlb_remove_table(void *table)
|
||||
{
|
||||
/* The bottom 2 bits are used to encode page table type. */
|
||||
const unsigned long encoded = (unsigned long)table;
|
||||
void *addr = (void *)(encoded & ~3UL);
|
||||
const int type = encoded & 3;
|
||||
|
||||
free_pointer_table(addr, type);
|
||||
}
|
||||
|
||||
/* size of memory already mapped in head.S */
|
||||
extern __initdata unsigned long m68k_init_mapped_size;
|
||||
|
||||
|
||||
@@ -25,7 +25,7 @@ extern void __bad_pte(pmd_t *pmd);
|
||||
|
||||
extern pte_t *pte_alloc_one_kernel(struct mm_struct *mm);
|
||||
|
||||
#define __pte_free_tlb(tlb, pte, addr) pte_free((tlb)->mm, (pte))
|
||||
#define __pte_free_tlb(tlb, pte, addr) tlb_remove_ptdesc((tlb), page_ptdesc(pte))
|
||||
|
||||
#define pmd_populate(mm, pmd, pte) \
|
||||
(pmd_val(*(pmd)) = (unsigned long)page_address(pte))
|
||||
|
||||
@@ -103,13 +103,6 @@ extern pte_t *va_to_pte(unsigned long address);
|
||||
#define USER_PGD_PTRS (PAGE_OFFSET >> PGDIR_SHIFT)
|
||||
#define KERNEL_PGD_PTRS (PTRS_PER_PGD-USER_PGD_PTRS)
|
||||
|
||||
#define pte_ERROR(e) \
|
||||
printk(KERN_ERR "%s:%d: bad pte "PTE_FMT".\n", \
|
||||
__FILE__, __LINE__, pte_val(e))
|
||||
#define pgd_ERROR(e) \
|
||||
printk(KERN_ERR "%s:%d: bad pgd %08lx.\n", \
|
||||
__FILE__, __LINE__, pgd_val(e))
|
||||
|
||||
/*
|
||||
* Bits in a linux-style PTE. These match the bits in the
|
||||
* (hardware-defined) PTE as closely as possible.
|
||||
|
||||
@@ -97,7 +97,6 @@ config MIPS
|
||||
select IRQ_FORCED_THREADING
|
||||
select ISA if EISA
|
||||
select LOCK_MM_AND_FIND_VMA
|
||||
select MMU_GATHER_RCU_TABLE_FREE
|
||||
select MODULES_USE_ELF_REL if MODULES
|
||||
select MODULES_USE_ELF_RELA if MODULES && 64BIT
|
||||
select PERF_USE_VMALLOC
|
||||
|
||||
@@ -104,16 +104,6 @@ extern int add_temporary_entry(unsigned long entrylo0, unsigned long entrylo1,
|
||||
# define VMALLOC_END (FIXADDR_START-2*PAGE_SIZE)
|
||||
#endif
|
||||
|
||||
#ifdef CONFIG_PHYS_ADDR_T_64BIT
|
||||
#define pte_ERROR(e) \
|
||||
printk("%s:%d: bad pte %016Lx.\n", __FILE__, __LINE__, pte_val(e))
|
||||
#else
|
||||
#define pte_ERROR(e) \
|
||||
printk("%s:%d: bad pte %08lx.\n", __FILE__, __LINE__, pte_val(e))
|
||||
#endif
|
||||
#define pgd_ERROR(e) \
|
||||
printk("%s:%d: bad pgd %08lx.\n", __FILE__, __LINE__, pgd_val(e))
|
||||
|
||||
extern void load_pgd(unsigned long pg_dir);
|
||||
|
||||
extern pte_t invalid_pte_table[PTRS_PER_PTE];
|
||||
|
||||
@@ -151,19 +151,6 @@
|
||||
#define MODULES_END (FIXADDR_START-2*PAGE_SIZE)
|
||||
#endif
|
||||
|
||||
#define pte_ERROR(e) \
|
||||
printk("%s:%d: bad pte %016lx.\n", __FILE__, __LINE__, pte_val(e))
|
||||
#ifndef __PAGETABLE_PMD_FOLDED
|
||||
#define pmd_ERROR(e) \
|
||||
printk("%s:%d: bad pmd %016lx.\n", __FILE__, __LINE__, pmd_val(e))
|
||||
#endif
|
||||
#ifndef __PAGETABLE_PUD_FOLDED
|
||||
#define pud_ERROR(e) \
|
||||
printk("%s:%d: bad pud %016lx.\n", __FILE__, __LINE__, pud_val(e))
|
||||
#endif
|
||||
#define pgd_ERROR(e) \
|
||||
printk("%s:%d: bad pgd %016lx.\n", __FILE__, __LINE__, pgd_val(e))
|
||||
|
||||
extern pte_t invalid_pte_table[PTRS_PER_PTE];
|
||||
|
||||
#ifndef __PAGETABLE_PUD_FOLDED
|
||||
|
||||
@@ -223,13 +223,6 @@ static inline unsigned long pmd_page_vaddr(pmd_t pmd)
|
||||
return pmd_val(pmd);
|
||||
}
|
||||
|
||||
#define pte_ERROR(e) \
|
||||
pr_err("%s:%d: bad pte %08lx.\n", \
|
||||
__FILE__, __LINE__, pte_val(e))
|
||||
#define pgd_ERROR(e) \
|
||||
pr_err("%s:%d: bad pgd %08lx.\n", \
|
||||
__FILE__, __LINE__, pgd_val(e))
|
||||
|
||||
/*
|
||||
* Encode/decode swap entries and swap PTEs. Swap PTEs are all PTEs that
|
||||
* are !pte_none() && !pte_present().
|
||||
|
||||
@@ -338,13 +338,6 @@ static inline unsigned long pmd_page_vaddr(pmd_t pmd)
|
||||
#define pte_pfn(x) ((unsigned long)(((x).pte)) >> PAGE_SHIFT)
|
||||
#define pfn_pte(pfn, prot) __pte((((pfn) << PAGE_SHIFT)) | pgprot_val(prot))
|
||||
|
||||
#define pte_ERROR(e) \
|
||||
printk(KERN_ERR "%s:%d: bad pte %p(%08lx).\n", \
|
||||
__FILE__, __LINE__, &(e), pte_val(e))
|
||||
#define pgd_ERROR(e) \
|
||||
printk(KERN_ERR "%s:%d: bad pgd %p(%08lx).\n", \
|
||||
__FILE__, __LINE__, &(e), pgd_val(e))
|
||||
|
||||
extern pgd_t swapper_pg_dir[PTRS_PER_PGD]; /* defined in head.S */
|
||||
|
||||
struct vm_area_struct;
|
||||
|
||||
@@ -80,7 +80,6 @@ config PARISC
|
||||
select GENERIC_CLOCKEVENTS
|
||||
select CPU_NO_EFFICIENT_FFS
|
||||
select THREAD_INFO_IN_TASK
|
||||
select MMU_GATHER_RCU_TABLE_FREE
|
||||
select NEED_DMA_MAP_STATE
|
||||
select NEED_SG_DMA_LENGTH
|
||||
select HAVE_ARCH_KGDB
|
||||
|
||||
@@ -75,15 +75,6 @@ extern void __update_cache(pte_t pte);
|
||||
|
||||
#endif /* !__ASSEMBLER__ */
|
||||
|
||||
#define pte_ERROR(e) \
|
||||
printk("%s:%d: bad pte %08lx.\n", __FILE__, __LINE__, pte_val(e))
|
||||
#if CONFIG_PGTABLE_LEVELS == 3
|
||||
#define pmd_ERROR(e) \
|
||||
printk("%s:%d: bad pmd %08lx.\n", __FILE__, __LINE__, (unsigned long)pmd_val(e))
|
||||
#endif
|
||||
#define pgd_ERROR(e) \
|
||||
printk("%s:%d: bad pgd %08lx.\n", __FILE__, __LINE__, (unsigned long)pgd_val(e))
|
||||
|
||||
/* This is the size of the initially mapped kernel memory */
|
||||
#if defined(CONFIG_64BIT) || defined(CONFIG_KALLSYMS)
|
||||
#define KERNEL_INITIAL_ORDER 26 /* 1<<26 = 64MB */
|
||||
|
||||
@@ -160,8 +160,7 @@ static inline void unmap_uncached_pte(pmd_t * pmd, unsigned long vaddr,
|
||||
if (pmd_none(*pmd))
|
||||
return;
|
||||
if (pmd_bad(*pmd)) {
|
||||
pmd_ERROR(*pmd);
|
||||
pmd_clear(pmd);
|
||||
pmd_clear_bad(pmd);
|
||||
return;
|
||||
}
|
||||
pte = pte_offset_kernel(pmd, vaddr);
|
||||
@@ -196,8 +195,7 @@ static inline void unmap_uncached_pmd(pgd_t * dir, unsigned long vaddr,
|
||||
if (pgd_none(*dir))
|
||||
return;
|
||||
if (pgd_bad(*dir)) {
|
||||
pgd_ERROR(*dir);
|
||||
pgd_clear(dir);
|
||||
pgd_clear_bad(dir);
|
||||
return;
|
||||
}
|
||||
pmd = pmd_offset(pud_offset(p4d_offset(dir, vaddr), vaddr), vaddr);
|
||||
|
||||
@@ -305,7 +305,6 @@ config PPC
|
||||
select KASAN_VMALLOC if KASAN && EXECMEM
|
||||
select LOCK_MM_AND_FIND_VMA
|
||||
select MMU_GATHER_PAGE_SIZE
|
||||
select MMU_GATHER_RCU_TABLE_FREE
|
||||
select HAVE_ARCH_TLB_REMOVE_TABLE
|
||||
select MMU_GATHER_MERGE_VMAS
|
||||
select MMU_LAZY_TLB_SHOOTDOWN if PPC_BOOK3S_64
|
||||
|
||||
@@ -203,8 +203,6 @@ void unmap_kernel_page(unsigned long va);
|
||||
/* Bits to mask out from a PGD to get to the PUD page */
|
||||
#define PGD_MASKED_BITS 0
|
||||
|
||||
#define pgd_ERROR(e) \
|
||||
pr_err("%s:%d: bad pgd %08lx.\n", __FILE__, __LINE__, pgd_val(e))
|
||||
/*
|
||||
* Bits in a linux-style PTE. These match the bits in the
|
||||
* (hardware-defined) PowerPC PTE as closely as possible.
|
||||
|
||||
@@ -991,13 +991,6 @@ static inline pmd_t *pud_pgtable(pud_t pud)
|
||||
return (pmd_t *)__va(pud_val(pud) & ~PUD_MASKED_BITS);
|
||||
}
|
||||
|
||||
#define pmd_ERROR(e) \
|
||||
pr_err("%s:%d: bad pmd %08lx.\n", __FILE__, __LINE__, pmd_val(e))
|
||||
#define pud_ERROR(e) \
|
||||
pr_err("%s:%d: bad pud %08lx.\n", __FILE__, __LINE__, pud_val(e))
|
||||
#define pgd_ERROR(e) \
|
||||
pr_err("%s:%d: bad pgd %08lx.\n", __FILE__, __LINE__, pgd_val(e))
|
||||
|
||||
static inline int map_kernel_page(unsigned long ea, unsigned long pa, pgprot_t prot)
|
||||
{
|
||||
if (radix_enabled()) {
|
||||
|
||||
@@ -51,8 +51,6 @@
|
||||
|
||||
#define USER_PTRS_PER_PGD (TASK_SIZE / PGDIR_SIZE)
|
||||
|
||||
#define pgd_ERROR(e) \
|
||||
pr_err("%s:%d: bad pgd %08llx.\n", __FILE__, __LINE__, (unsigned long long)pgd_val(e))
|
||||
|
||||
/*
|
||||
* This is the bottom of the PKMAP area with HIGHMEM or an arbitrary
|
||||
|
||||
@@ -82,9 +82,6 @@ extern struct page *p4d_page(p4d_t p4d);
|
||||
|
||||
#endif /* !__ASSEMBLER__ */
|
||||
|
||||
#define pud_ERROR(e) \
|
||||
pr_err("%s:%d: bad pud %08lx.\n", __FILE__, __LINE__, pud_val(e))
|
||||
|
||||
/*
|
||||
* On all 4K setups, remap_4k_pfn() equates to remap_pfn_range() */
|
||||
#define remap_4k_pfn(vma, addr, pfn, prot) \
|
||||
|
||||
@@ -159,11 +159,6 @@ static inline void huge_ptep_set_wrprotect(struct mm_struct *mm,
|
||||
__young; \
|
||||
})
|
||||
|
||||
#define pmd_ERROR(e) \
|
||||
pr_err("%s:%d: bad pmd %08lx.\n", __FILE__, __LINE__, pmd_val(e))
|
||||
#define pgd_ERROR(e) \
|
||||
pr_err("%s:%d: bad pgd %08lx.\n", __FILE__, __LINE__, pgd_val(e))
|
||||
|
||||
/*
|
||||
* Encode/decode swap entries and swap PTEs. Swap PTEs are all PTEs that
|
||||
* are !pte_none() && !pte_present().
|
||||
|
||||
@@ -19,6 +19,7 @@
|
||||
#include <linux/string_helpers.h>
|
||||
#include <linux/memory.h>
|
||||
#include <linux/kfence.h>
|
||||
#include <linux/vmemmap-optimization.h>
|
||||
|
||||
#include <asm/pgalloc.h>
|
||||
#include <asm/mmu_context.h>
|
||||
@@ -1041,13 +1042,17 @@ static pte_t * __meminit radix__vmemmap_pte_populate(pmd_t *pmdp, unsigned long
|
||||
/*
|
||||
* When a PTE/PMD entry is freed from the init_mm
|
||||
* there's a free_pages() call to this page allocated
|
||||
* above. Thus this get_page() is paired with the
|
||||
* above. Thus this try_get_page() is paired with the
|
||||
* put_page_testzero() on the freeing path.
|
||||
* This can only called by certain ZONE_DEVICE path,
|
||||
* and through vmemmap_populate_compound_pages() when
|
||||
* slab is available.
|
||||
*
|
||||
* Use try_get_page() to prevent the shared page refcount
|
||||
* from overflowing.
|
||||
*/
|
||||
get_page(reuse);
|
||||
if (!try_get_page(reuse))
|
||||
return NULL;
|
||||
p = page_to_virt(reuse);
|
||||
pr_debug("Tail page reuse vmemmap mapping\n");
|
||||
}
|
||||
@@ -1217,92 +1222,6 @@ base_mapping:
|
||||
return 0;
|
||||
}
|
||||
|
||||
static pte_t * __meminit radix__vmemmap_populate_address(unsigned long addr, int node,
|
||||
struct vmem_altmap *altmap,
|
||||
struct page *reuse)
|
||||
{
|
||||
pgd_t *pgd;
|
||||
p4d_t *p4d;
|
||||
pud_t *pud;
|
||||
pmd_t *pmd;
|
||||
pte_t *pte;
|
||||
|
||||
pgd = pgd_offset_k(addr);
|
||||
p4d = p4d_offset(pgd, addr);
|
||||
pud = vmemmap_pud_alloc(p4d, node, addr);
|
||||
if (!pud)
|
||||
return NULL;
|
||||
pmd = vmemmap_pmd_alloc(pud, node, addr);
|
||||
if (!pmd)
|
||||
return NULL;
|
||||
if (pmd_leaf(*pmd))
|
||||
/*
|
||||
* The second page is mapped as a hugepage due to a nearby request.
|
||||
* Force our mapping to page size without deduplication
|
||||
*/
|
||||
return NULL;
|
||||
pte = vmemmap_pte_alloc(pmd, node, addr);
|
||||
if (!pte)
|
||||
return NULL;
|
||||
radix__vmemmap_pte_populate(pmd, addr, node, NULL, NULL);
|
||||
vmemmap_verify(pte, node, addr, addr + PAGE_SIZE);
|
||||
|
||||
return pte;
|
||||
}
|
||||
|
||||
static pte_t * __meminit vmemmap_compound_tail_page(unsigned long addr,
|
||||
unsigned long pfn_offset, int node)
|
||||
{
|
||||
pgd_t *pgd;
|
||||
p4d_t *p4d;
|
||||
pud_t *pud;
|
||||
pmd_t *pmd;
|
||||
pte_t *pte;
|
||||
unsigned long map_addr;
|
||||
|
||||
/* the second vmemmap page which we use for duplication */
|
||||
map_addr = addr - pfn_offset * sizeof(struct page) + PAGE_SIZE;
|
||||
pgd = pgd_offset_k(map_addr);
|
||||
p4d = p4d_offset(pgd, map_addr);
|
||||
pud = vmemmap_pud_alloc(p4d, node, map_addr);
|
||||
if (!pud)
|
||||
return NULL;
|
||||
pmd = vmemmap_pmd_alloc(pud, node, map_addr);
|
||||
if (!pmd)
|
||||
return NULL;
|
||||
if (pmd_leaf(*pmd))
|
||||
/*
|
||||
* The second page is mapped as a hugepage due to a nearby request.
|
||||
* Force our mapping to page size without deduplication
|
||||
*/
|
||||
return NULL;
|
||||
pte = vmemmap_pte_alloc(pmd, node, map_addr);
|
||||
if (!pte)
|
||||
return NULL;
|
||||
/*
|
||||
* Check if there exist a mapping to the left
|
||||
*/
|
||||
if (pte_none(*pte)) {
|
||||
/*
|
||||
* Populate the head page vmemmap page.
|
||||
* It can fall in different pmd, hence
|
||||
* vmemmap_populate_address()
|
||||
*/
|
||||
pte = radix__vmemmap_populate_address(map_addr - PAGE_SIZE, node, NULL, NULL);
|
||||
if (!pte)
|
||||
return NULL;
|
||||
/*
|
||||
* Populate the tail pages vmemmap page
|
||||
*/
|
||||
pte = radix__vmemmap_pte_populate(pmd, map_addr, node, NULL, NULL);
|
||||
if (!pte)
|
||||
return NULL;
|
||||
vmemmap_verify(pte, node, map_addr, map_addr + PAGE_SIZE);
|
||||
return pte;
|
||||
}
|
||||
return pte;
|
||||
}
|
||||
|
||||
int __meminit vmemmap_populate_compound_pages(unsigned long start_pfn,
|
||||
unsigned long start,
|
||||
unsigned long end, int node,
|
||||
@@ -1320,6 +1239,12 @@ int __meminit vmemmap_populate_compound_pages(unsigned long start_pfn,
|
||||
pud_t *pud;
|
||||
pmd_t *pmd;
|
||||
pte_t *pte;
|
||||
struct page *tail_page;
|
||||
unsigned int order = pfn_to_section_compound_order(start_pfn);
|
||||
|
||||
tail_page = vmemmap_shared_tail_page(order, device_zone(node));
|
||||
if (!tail_page)
|
||||
return -ENOMEM;
|
||||
|
||||
for (addr = start; addr < end; addr = next) {
|
||||
|
||||
@@ -1343,16 +1268,15 @@ int __meminit vmemmap_populate_compound_pages(unsigned long start_pfn,
|
||||
if (!pte_none(*pte)) {
|
||||
/*
|
||||
* This could be because we already have a compound
|
||||
* page whose VMEMMAP_RESERVE_NR pages were mapped and
|
||||
* page whose retained vmemmap page was mapped and
|
||||
* this request fall in those pages.
|
||||
*/
|
||||
next = addr + PAGE_SIZE;
|
||||
continue;
|
||||
} else {
|
||||
unsigned long nr_pages = pgmap_vmemmap_nr(pgmap);
|
||||
unsigned long nr_pages = 1UL << order;
|
||||
unsigned long addr_pfn = page_to_pfn((struct page *)addr);
|
||||
unsigned long pfn_offset = addr_pfn - ALIGN_DOWN(addr_pfn, nr_pages);
|
||||
pte_t *tail_page_pte;
|
||||
|
||||
/*
|
||||
* if the address is aligned to huge page size it is the
|
||||
@@ -1365,35 +1289,11 @@ int __meminit vmemmap_populate_compound_pages(unsigned long start_pfn,
|
||||
return -ENOMEM;
|
||||
vmemmap_verify(pte, node, addr, addr + PAGE_SIZE);
|
||||
|
||||
/*
|
||||
* Populate the tail pages vmemmap page
|
||||
* It can fall in different pmd, hence
|
||||
* vmemmap_populate_address()
|
||||
*/
|
||||
pte = radix__vmemmap_populate_address(addr + PAGE_SIZE, node, NULL, NULL);
|
||||
if (!pte)
|
||||
return -ENOMEM;
|
||||
|
||||
next = addr + 2 * PAGE_SIZE;
|
||||
continue;
|
||||
}
|
||||
/*
|
||||
* get the 2nd mapping details
|
||||
* Also create it if that doesn't exist
|
||||
*/
|
||||
tail_page_pte = vmemmap_compound_tail_page(addr, pfn_offset, node);
|
||||
if (!tail_page_pte) {
|
||||
|
||||
pte = radix__vmemmap_pte_populate(pmd, addr, node, NULL, NULL);
|
||||
if (!pte)
|
||||
return -ENOMEM;
|
||||
vmemmap_verify(pte, node, addr, addr + PAGE_SIZE);
|
||||
|
||||
next = addr + PAGE_SIZE;
|
||||
continue;
|
||||
}
|
||||
|
||||
pte = radix__vmemmap_pte_populate(pmd, addr, node, NULL, pte_page(*tail_page_pte));
|
||||
pte = radix__vmemmap_pte_populate(pmd, addr, node, NULL, tail_page);
|
||||
if (!pte)
|
||||
return -ENOMEM;
|
||||
vmemmap_verify(pte, node, addr, addr + PAGE_SIZE);
|
||||
|
||||
@@ -17,7 +17,6 @@ config PPC_POWERNV
|
||||
select PPC_DOORBELL
|
||||
select MMU_NOTIFIER
|
||||
select FORCE_SMP
|
||||
select ARCH_SUPPORTS_PER_VMA_LOCK
|
||||
select PPC_RADIX_BROADCAST_TLBIE if PPC_RADIX_MMU
|
||||
default y
|
||||
|
||||
|
||||
@@ -23,7 +23,6 @@ config PPC_PSERIES
|
||||
select HOTPLUG_CPU
|
||||
select FORCE_SMP
|
||||
select SWIOTLB
|
||||
select ARCH_SUPPORTS_PER_VMA_LOCK
|
||||
select PPC_RADIX_BROADCAST_TLBIE if PPC_RADIX_MMU
|
||||
default y
|
||||
|
||||
|
||||
@@ -72,7 +72,6 @@ config RISCV
|
||||
select ARCH_SUPPORTS_LTO_CLANG_THIN
|
||||
select ARCH_SUPPORTS_MSEAL_SYSTEM_MAPPINGS if 64BIT && MMU
|
||||
select ARCH_SUPPORTS_PAGE_TABLE_CHECK if MMU
|
||||
select ARCH_SUPPORTS_PER_VMA_LOCK if MMU
|
||||
select ARCH_HAS_PTE_PROTNONE if MMU
|
||||
select ARCH_SUPPORTS_RT
|
||||
select ARCH_SUPPORTS_SHADOW_CALL_STACK if HAVE_SHADOW_CALL_STACK
|
||||
@@ -208,7 +207,6 @@ config RISCV
|
||||
select IRQ_FORCED_THREADING
|
||||
select KASAN_VMALLOC if KASAN
|
||||
select LOCK_MM_AND_FIND_VMA
|
||||
select MMU_GATHER_RCU_TABLE_FREE if SMP && MMU
|
||||
select MODULES_USE_ELF_RELA if MODULES
|
||||
select OF
|
||||
select OF_EARLY_FLATTREE
|
||||
|
||||
@@ -76,12 +76,6 @@ typedef struct page *pgtable_t;
|
||||
#define __pgd(x) ((pgd_t) { (x) })
|
||||
#define __pgprot(x) ((pgprot_t) { (x) })
|
||||
|
||||
#ifdef CONFIG_64BIT
|
||||
#define PTE_FMT "%016lx"
|
||||
#else
|
||||
#define PTE_FMT "%08lx"
|
||||
#endif
|
||||
|
||||
#if defined(CONFIG_64BIT) && defined(CONFIG_MMU)
|
||||
/*
|
||||
* We override this value as its generic definition uses __pa too early in
|
||||
|
||||
@@ -264,15 +264,6 @@ static inline unsigned long _pmd_pfn(pmd_t pmd)
|
||||
return __page_val_to_pfn(pmd_val(pmd));
|
||||
}
|
||||
|
||||
#define pmd_ERROR(e) \
|
||||
pr_err("%s:%d: bad pmd %016lx.\n", __FILE__, __LINE__, pmd_val(e))
|
||||
|
||||
#define pud_ERROR(e) \
|
||||
pr_err("%s:%d: bad pud %016lx.\n", __FILE__, __LINE__, pud_val(e))
|
||||
|
||||
#define p4d_ERROR(e) \
|
||||
pr_err("%s:%d: bad p4d %016lx.\n", __FILE__, __LINE__, p4d_val(e))
|
||||
|
||||
static inline void set_p4d(p4d_t *p4dp, p4d_t p4d)
|
||||
{
|
||||
if (pgtable_l4_enabled)
|
||||
|
||||
@@ -556,10 +556,6 @@ static inline pte_t pte_modify(pte_t pte, pgprot_t newprot)
|
||||
return __pte((pte_val(pte) & _PAGE_CHG_MASK) | newprot_val);
|
||||
}
|
||||
|
||||
#define pgd_ERROR(e) \
|
||||
pr_err("%s:%d: bad pgd " PTE_FMT ".\n", __FILE__, __LINE__, pgd_val(e))
|
||||
|
||||
|
||||
/* Commit new configuration to MMU hardware */
|
||||
static inline void update_mmu_cache_range(struct vm_fault *vmf,
|
||||
struct vm_area_struct *vma, unsigned long address,
|
||||
|
||||
@@ -40,9 +40,8 @@ static inline int set_kernel_memory(char *startp, char *endp,
|
||||
}
|
||||
#endif
|
||||
|
||||
int set_direct_map_invalid_noflush(struct page *page);
|
||||
int set_direct_map_default_noflush(struct page *page);
|
||||
int set_direct_map_valid_noflush(struct page *page, unsigned nr, bool valid);
|
||||
int set_direct_map_invalid_noflush(struct page *page, unsigned int nr);
|
||||
int set_direct_map_default_noflush(struct page *page, unsigned int nr);
|
||||
bool kernel_page_present(struct page *page);
|
||||
|
||||
#endif /* __ASSEMBLER__ */
|
||||
|
||||
@@ -22,6 +22,7 @@
|
||||
#include <linux/hugetlb.h>
|
||||
#include <linux/kfence.h>
|
||||
#include <linux/execmem.h>
|
||||
#include <linux/vmemmap-optimization.h>
|
||||
|
||||
#include <asm/alternative.h>
|
||||
#include <asm/fixmap.h>
|
||||
|
||||
@@ -374,33 +374,18 @@ int set_memory_nx(unsigned long addr, int numpages)
|
||||
return __set_memory(addr, numpages, __pgprot(0), __pgprot(_PAGE_EXEC));
|
||||
}
|
||||
|
||||
int set_direct_map_invalid_noflush(struct page *page)
|
||||
int set_direct_map_invalid_noflush(struct page *page, unsigned int nr)
|
||||
{
|
||||
return __set_memory((unsigned long)page_address(page), 1,
|
||||
return __set_memory((unsigned long)page_address(page), nr,
|
||||
__pgprot(0), __pgprot(_PAGE_PRESENT));
|
||||
}
|
||||
|
||||
int set_direct_map_default_noflush(struct page *page)
|
||||
int set_direct_map_default_noflush(struct page *page, unsigned int nr)
|
||||
{
|
||||
return __set_memory((unsigned long)page_address(page), 1,
|
||||
return __set_memory((unsigned long)page_address(page), nr,
|
||||
PAGE_KERNEL, __pgprot(_PAGE_EXEC));
|
||||
}
|
||||
|
||||
int set_direct_map_valid_noflush(struct page *page, unsigned nr, bool valid)
|
||||
{
|
||||
pgprot_t set, clear;
|
||||
|
||||
if (valid) {
|
||||
set = PAGE_KERNEL;
|
||||
clear = __pgprot(_PAGE_EXEC);
|
||||
} else {
|
||||
set = __pgprot(0);
|
||||
clear = __pgprot(_PAGE_PRESENT);
|
||||
}
|
||||
|
||||
return __set_memory((unsigned long)page_address(page), nr, set, clear);
|
||||
}
|
||||
|
||||
#ifdef CONFIG_DEBUG_PAGEALLOC
|
||||
static int debug_pagealloc_set_page(pte_t *pte, unsigned long addr, void *data)
|
||||
{
|
||||
|
||||
@@ -157,7 +157,6 @@ config S390
|
||||
select ARCH_HAS_PTE_PROTNONE
|
||||
select ARCH_SUPPORTS_NUMA_BALANCING
|
||||
select ARCH_SUPPORTS_PAGE_TABLE_CHECK
|
||||
select ARCH_SUPPORTS_PER_VMA_LOCK
|
||||
select ARCH_USES_CFI_GENERIC_LLVM_PASS if CC_IS_CLANG
|
||||
select ARCH_USE_BUILTIN_BSWAP
|
||||
select ARCH_USE_CMPXCHG_LOCKREF
|
||||
@@ -268,7 +267,6 @@ config S390
|
||||
select LOCK_MM_AND_FIND_VMA
|
||||
select MMU_GATHER_MERGE_VMAS
|
||||
select MMU_GATHER_NO_GATHER
|
||||
select MMU_GATHER_RCU_TABLE_FREE
|
||||
select MODULES_USE_ELF_RELA
|
||||
select NEED_DMA_MAP_STATE if PCI
|
||||
select NEED_PER_CPU_EMBED_FIRST_CHUNK
|
||||
|
||||
@@ -68,17 +68,6 @@ extern unsigned long zero_page_mask;
|
||||
|
||||
/* TODO: s390 cannot support io_remap_pfn_range... */
|
||||
|
||||
#define pte_ERROR(e) \
|
||||
pr_err("%s:%d: bad pte %016lx.\n", __FILE__, __LINE__, pte_val(e))
|
||||
#define pmd_ERROR(e) \
|
||||
pr_err("%s:%d: bad pmd %016lx.\n", __FILE__, __LINE__, pmd_val(e))
|
||||
#define pud_ERROR(e) \
|
||||
pr_err("%s:%d: bad pud %016lx.\n", __FILE__, __LINE__, pud_val(e))
|
||||
#define p4d_ERROR(e) \
|
||||
pr_err("%s:%d: bad p4d %016lx.\n", __FILE__, __LINE__, p4d_val(e))
|
||||
#define pgd_ERROR(e) \
|
||||
pr_err("%s:%d: bad pgd %016lx.\n", __FILE__, __LINE__, pgd_val(e))
|
||||
|
||||
/*
|
||||
* The vmalloc and module area will always be on the topmost area of the
|
||||
* kernel mapping. 512GB are reserved for vmalloc by default.
|
||||
|
||||
@@ -60,9 +60,8 @@ __SET_MEMORY_FUNC(set_memory_rox, SET_MEMORY_RO | SET_MEMORY_X)
|
||||
__SET_MEMORY_FUNC(set_memory_rwnx, SET_MEMORY_RW | SET_MEMORY_NX)
|
||||
__SET_MEMORY_FUNC(set_memory_4k, SET_MEMORY_4K)
|
||||
|
||||
int set_direct_map_invalid_noflush(struct page *page);
|
||||
int set_direct_map_default_noflush(struct page *page);
|
||||
int set_direct_map_valid_noflush(struct page *page, unsigned nr, bool valid);
|
||||
int set_direct_map_invalid_noflush(struct page *page, unsigned int nr);
|
||||
int set_direct_map_default_noflush(struct page *page, unsigned int nr);
|
||||
bool kernel_page_present(struct page *page);
|
||||
|
||||
#endif
|
||||
|
||||
+4
-16
@@ -382,26 +382,14 @@ out:
|
||||
return rc;
|
||||
}
|
||||
|
||||
int set_direct_map_invalid_noflush(struct page *page)
|
||||
int set_direct_map_invalid_noflush(struct page *page, unsigned int nr)
|
||||
{
|
||||
return __set_memory((unsigned long)page_to_virt(page), 1, SET_MEMORY_INV);
|
||||
return __set_memory((unsigned long)page_to_virt(page), nr, SET_MEMORY_INV);
|
||||
}
|
||||
|
||||
int set_direct_map_default_noflush(struct page *page)
|
||||
int set_direct_map_default_noflush(struct page *page, unsigned int nr)
|
||||
{
|
||||
return __set_memory((unsigned long)page_to_virt(page), 1, SET_MEMORY_DEF);
|
||||
}
|
||||
|
||||
int set_direct_map_valid_noflush(struct page *page, unsigned nr, bool valid)
|
||||
{
|
||||
unsigned long flags;
|
||||
|
||||
if (valid)
|
||||
flags = SET_MEMORY_DEF;
|
||||
else
|
||||
flags = SET_MEMORY_INV;
|
||||
|
||||
return __set_memory((unsigned long)page_to_virt(page), nr, flags);
|
||||
return __set_memory((unsigned long)page_to_virt(page), nr, SET_MEMORY_DEF);
|
||||
}
|
||||
|
||||
bool kernel_page_present(struct page *page)
|
||||
|
||||
@@ -33,6 +33,7 @@ config SUPERH
|
||||
select HAVE_ARCH_AUDITSYSCALL
|
||||
select HAVE_ARCH_KGDB
|
||||
select HAVE_ARCH_SECCOMP_FILTER
|
||||
select HAVE_ARCH_TLB_REMOVE_TABLE if X2TLB
|
||||
select HAVE_ARCH_TRACEHOOK
|
||||
select HAVE_DEBUG_BUGVERBOSE
|
||||
select HAVE_DEBUG_KMEMLEAK
|
||||
|
||||
@@ -17,7 +17,11 @@ extern void pgd_free(struct mm_struct *mm, pgd_t *pgd);
|
||||
extern void pud_populate(struct mm_struct *mm, pud_t *pudp, pmd_t *pmd);
|
||||
extern pmd_t *pmd_alloc_one(struct mm_struct *mm, unsigned long address);
|
||||
extern void pmd_free(struct mm_struct *mm, pmd_t *pmd);
|
||||
#define __pmd_free_tlb(tlb, pmdp, addr) pmd_free((tlb)->mm, (pmdp))
|
||||
extern void __tlb_remove_table(void *table);
|
||||
|
||||
/* PMDs are slab-allocated, tag so they are freed correctly. */
|
||||
#define __pmd_free_tlb(tlb, pmdp, addr) \
|
||||
tlb_remove_table((tlb), (void *)((unsigned long)(pmdp) | 1))
|
||||
#endif
|
||||
|
||||
static inline void pmd_populate_kernel(struct mm_struct *mm, pmd_t *pmd,
|
||||
|
||||
@@ -25,9 +25,6 @@
|
||||
|
||||
#define PTRS_PER_PMD ((1 << PGDIR_SHIFT) / PMD_SIZE)
|
||||
|
||||
#define pmd_ERROR(e) \
|
||||
printk("%s:%d: bad pmd %016llx.\n", __FILE__, __LINE__, pmd_val(e))
|
||||
|
||||
typedef union {
|
||||
struct {
|
||||
unsigned long pmd_low;
|
||||
|
||||
@@ -400,19 +400,6 @@ static inline unsigned long pmd_page_vaddr(pmd_t pmd)
|
||||
#define pmd_pfn(pmd) (__pa(pmd_val(pmd)) >> PAGE_SHIFT)
|
||||
#define pmd_page(pmd) (virt_to_page(pmd_val(pmd)))
|
||||
|
||||
#ifdef CONFIG_X2TLB
|
||||
#define pte_ERROR(e) \
|
||||
printk("%s:%d: bad pte %p(%08lx%08lx).\n", __FILE__, __LINE__, \
|
||||
&(e), (e).pte_high, (e).pte_low)
|
||||
#define pgd_ERROR(e) \
|
||||
printk("%s:%d: bad pgd %016llx.\n", __FILE__, __LINE__, pgd_val(e))
|
||||
#else
|
||||
#define pte_ERROR(e) \
|
||||
printk("%s:%d: bad pte %08lx.\n", __FILE__, __LINE__, pte_val(e))
|
||||
#define pgd_ERROR(e) \
|
||||
printk("%s:%d: bad pgd %08lx.\n", __FILE__, __LINE__, pgd_val(e))
|
||||
#endif
|
||||
|
||||
/*
|
||||
* Encode/decode swap entries and swap PTEs. Swap PTEs are all PTEs that
|
||||
* are !pte_none() && !pte_present().
|
||||
|
||||
+9
-7
@@ -52,26 +52,24 @@ static pte_t *__get_pte_phys(unsigned long addr)
|
||||
pmd_t *pmd;
|
||||
|
||||
pgd = pgd_offset_k(addr);
|
||||
if (pgd_none(*pgd)) {
|
||||
pgd_ERROR(*pgd);
|
||||
if (pgd_none(*pgd))
|
||||
return NULL;
|
||||
}
|
||||
|
||||
p4d = p4d_alloc(NULL, pgd, addr);
|
||||
if (unlikely(!p4d)) {
|
||||
p4d_ERROR(*p4d);
|
||||
pr_err("allocating p4d table failed\n");
|
||||
return NULL;
|
||||
}
|
||||
|
||||
pud = pud_alloc(NULL, p4d, addr);
|
||||
if (unlikely(!pud)) {
|
||||
pud_ERROR(*pud);
|
||||
pr_err("allocating pud table failed\n");
|
||||
return NULL;
|
||||
}
|
||||
|
||||
pmd = pmd_alloc(NULL, pud, addr);
|
||||
if (unlikely(!pmd)) {
|
||||
pmd_ERROR(*pmd);
|
||||
pr_err("allocating pmd table failed\n");
|
||||
return NULL;
|
||||
}
|
||||
|
||||
@@ -84,7 +82,11 @@ static void set_pte_phys(unsigned long addr, unsigned long phys, pgprot_t prot)
|
||||
|
||||
pte = __get_pte_phys(addr);
|
||||
if (!pte_none(*pte)) {
|
||||
pte_ERROR(*pte);
|
||||
char str[PTVAL_STR_MAX];
|
||||
|
||||
ptval_to_str(str, pte_val(*pte));
|
||||
pr_err("unexpected set PTE at %lx in %s: bad pte %p(%s).\n",
|
||||
addr, __func__, pte, str);
|
||||
return;
|
||||
}
|
||||
|
||||
|
||||
@@ -56,4 +56,24 @@ void pmd_free(struct mm_struct *mm, pmd_t *pmd)
|
||||
{
|
||||
kmem_cache_free(pmd_cachep, pmd);
|
||||
}
|
||||
|
||||
static void __tlb_remove_table_slab(void *table)
|
||||
{
|
||||
kmem_cache_free(pmd_cachep, table);
|
||||
}
|
||||
|
||||
static void __tlb_remove_table_pgtable(void *table)
|
||||
{
|
||||
pagetable_dtor_free(table);
|
||||
}
|
||||
|
||||
void __tlb_remove_table(void *table)
|
||||
{
|
||||
const unsigned long addr = (unsigned long)table;
|
||||
|
||||
if (addr & 1)
|
||||
__tlb_remove_table_slab((void *)(addr & ~1UL));
|
||||
else
|
||||
__tlb_remove_table_pgtable(table);
|
||||
}
|
||||
#endif /* PAGETABLE_LEVELS > 2 */
|
||||
|
||||
+2
-2
@@ -64,6 +64,7 @@ config SPARC32
|
||||
select HAVE_UID16
|
||||
select HAVE_PAGE_SIZE_4KB
|
||||
select LOCK_MM_AND_FIND_VMA
|
||||
select HAVE_ARCH_TLB_REMOVE_TABLE
|
||||
select OLD_SIGACTION
|
||||
select ZONE_DMA
|
||||
|
||||
@@ -75,8 +76,7 @@ config SPARC64
|
||||
select HAVE_FUNCTION_GRAPH_TRACER
|
||||
select HAVE_KRETPROBES
|
||||
select HAVE_KPROBES
|
||||
select MMU_GATHER_RCU_TABLE_FREE if SMP
|
||||
select HAVE_ARCH_TLB_REMOVE_TABLE if SMP
|
||||
select HAVE_ARCH_TLB_REMOVE_TABLE
|
||||
select MMU_GATHER_MERGE_VMAS
|
||||
select MMU_GATHER_NO_FLUSH_CACHE
|
||||
select HAVE_ARCH_TRANSPARENT_HUGEPAGE
|
||||
|
||||
@@ -48,7 +48,9 @@ static inline void free_pmd_fast(pmd_t * pmd)
|
||||
}
|
||||
|
||||
#define pmd_free(mm, pmd) free_pmd_fast(pmd)
|
||||
#define __pmd_free_tlb(tlb, pmd, addr) pmd_free((tlb)->mm, pmd)
|
||||
|
||||
#define __pmd_free_tlb(tlb, pmd, addr) \
|
||||
tlb_remove_table((tlb), (void *)((unsigned long)(pmd) | 1UL))
|
||||
|
||||
#define pmd_populate(mm, pmd, pte) pmd_set(pmd, pte)
|
||||
|
||||
@@ -72,6 +74,7 @@ static inline void free_pte_fast(pte_t *pte)
|
||||
#define pte_free_kernel(mm, pte) free_pte_fast(pte)
|
||||
|
||||
void pte_free(struct mm_struct * mm, pgtable_t pte);
|
||||
#define __pte_free_tlb(tlb, pte, addr) pte_free((tlb)->mm, pte)
|
||||
void __tlb_remove_table(void *table);
|
||||
#define __pte_free_tlb(tlb, pte, addr) tlb_remove_table((tlb), (void *)(pte))
|
||||
|
||||
#endif /* _SPARC_PGALLOC_H */
|
||||
|
||||
@@ -74,8 +74,6 @@ void pte_free_defer(struct mm_struct *mm, pgtable_t pgtable);
|
||||
|
||||
void pgtable_free(void *table, bool is_page);
|
||||
|
||||
#ifdef CONFIG_SMP
|
||||
|
||||
struct mmu_gather;
|
||||
void tlb_remove_table(struct mmu_gather *, void *);
|
||||
|
||||
@@ -96,12 +94,6 @@ static inline void __tlb_remove_table(void *_table)
|
||||
is_page = true;
|
||||
pgtable_free(table, is_page);
|
||||
}
|
||||
#else /* CONFIG_SMP */
|
||||
static inline void pgtable_free_tlb(struct mmu_gather *tlb, void *table, bool is_page)
|
||||
{
|
||||
pgtable_free(table, is_page);
|
||||
}
|
||||
#endif /* !CONFIG_SMP */
|
||||
|
||||
static inline void __pte_free_tlb(struct mmu_gather *tlb, pte_t *pte,
|
||||
unsigned long address)
|
||||
|
||||
@@ -40,9 +40,6 @@ void load_mmu(void);
|
||||
unsigned long calc_highpages(void);
|
||||
unsigned long __init bootmem_init(unsigned long *pages_avail);
|
||||
|
||||
#define pte_ERROR(e) __builtin_trap()
|
||||
#define pmd_ERROR(e) __builtin_trap()
|
||||
#define pgd_ERROR(e) __builtin_trap()
|
||||
|
||||
#define PTRS_PER_PTE 64
|
||||
#define PTRS_PER_PMD 64
|
||||
|
||||
@@ -96,16 +96,6 @@ bool kern_addr_valid(unsigned long addr);
|
||||
#define PTRS_PER_PUD (1UL << PUD_BITS)
|
||||
#define PTRS_PER_PGD (1UL << PGDIR_BITS)
|
||||
|
||||
#define pmd_ERROR(e) \
|
||||
pr_err("%s:%d: bad pmd %p(%016lx) seen at (%pS)\n", \
|
||||
__FILE__, __LINE__, &(e), pmd_val(e), __builtin_return_address(0))
|
||||
#define pud_ERROR(e) \
|
||||
pr_err("%s:%d: bad pud %p(%016lx) seen at (%pS)\n", \
|
||||
__FILE__, __LINE__, &(e), pud_val(e), __builtin_return_address(0))
|
||||
#define pgd_ERROR(e) \
|
||||
pr_err("%s:%d: bad pgd %p(%016lx) seen at (%pS)\n", \
|
||||
__FILE__, __LINE__, &(e), pgd_val(e), __builtin_return_address(0))
|
||||
|
||||
#endif /* !(__ASSEMBLER__) */
|
||||
|
||||
/* PTE bits which are the same in SUN4U and SUN4V format. */
|
||||
|
||||
@@ -29,9 +29,7 @@ void flush_tlb_pending(void);
|
||||
* and therefore we don't need a TLBI when freeing page-table pages.
|
||||
*/
|
||||
|
||||
#ifdef CONFIG_MMU_GATHER_RCU_TABLE_FREE
|
||||
#define tlb_needs_table_invalidate() (false)
|
||||
#endif
|
||||
|
||||
#include <asm-generic/tlb.h>
|
||||
|
||||
|
||||
@@ -22,8 +22,6 @@
|
||||
* @align: requested alignment
|
||||
*
|
||||
* Returns offset in the map or -1 if out of space.
|
||||
*
|
||||
* Not safe to call from an interrupt (uses spin_lock).
|
||||
*/
|
||||
int bit_map_string_get(struct bit_map *t, int len, int align)
|
||||
{
|
||||
@@ -31,6 +29,7 @@ int bit_map_string_get(struct bit_map *t, int len, int align)
|
||||
int off_new;
|
||||
int align1;
|
||||
int i, color;
|
||||
unsigned long flags;
|
||||
|
||||
if (t->num_colors) {
|
||||
/* align is overloaded to be the page color */
|
||||
@@ -50,7 +49,7 @@ int bit_map_string_get(struct bit_map *t, int len, int align)
|
||||
BUG();
|
||||
color &= align1;
|
||||
|
||||
spin_lock(&t->lock);
|
||||
spin_lock_irqsave(&t->lock, flags);
|
||||
if (len < t->last_size)
|
||||
offset = t->first_free;
|
||||
else
|
||||
@@ -64,7 +63,7 @@ int bit_map_string_get(struct bit_map *t, int len, int align)
|
||||
if (offset >= t->size)
|
||||
offset = 0;
|
||||
if (count + len > t->size) {
|
||||
spin_unlock(&t->lock);
|
||||
spin_unlock_irqrestore(&t->lock, flags);
|
||||
/* P3 */ printk(KERN_ERR
|
||||
"bitmap out: size %d used %d off %d len %d align %d count %d\n",
|
||||
t->size, t->used, offset, len, align, count);
|
||||
@@ -90,7 +89,7 @@ int bit_map_string_get(struct bit_map *t, int len, int align)
|
||||
t->last_off = 0;
|
||||
t->used += len;
|
||||
t->last_size = len;
|
||||
spin_unlock(&t->lock);
|
||||
spin_unlock_irqrestore(&t->lock, flags);
|
||||
return offset;
|
||||
}
|
||||
}
|
||||
@@ -103,10 +102,11 @@ int bit_map_string_get(struct bit_map *t, int len, int align)
|
||||
void bit_map_clear(struct bit_map *t, int offset, int len)
|
||||
{
|
||||
int i;
|
||||
unsigned long flags;
|
||||
|
||||
if (t->used < len)
|
||||
BUG(); /* Much too late to do any good, but alas... */
|
||||
spin_lock(&t->lock);
|
||||
spin_lock_irqsave(&t->lock, flags);
|
||||
for (i = 0; i < len; i++) {
|
||||
if (test_bit(offset + i, t->map) == 0)
|
||||
BUG();
|
||||
@@ -115,7 +115,7 @@ void bit_map_clear(struct bit_map *t, int offset, int len)
|
||||
if (offset < t->first_free)
|
||||
t->first_free = offset;
|
||||
t->used -= len;
|
||||
spin_unlock(&t->lock);
|
||||
spin_unlock_irqrestore(&t->lock, flags);
|
||||
}
|
||||
|
||||
void bit_map_init(struct bit_map *t, unsigned long *map, int size)
|
||||
|
||||
+27
-5
@@ -340,38 +340,60 @@ pgd_t *get_pgd_fast(void)
|
||||
* Alignments up to the page size are the same for physical and virtual
|
||||
* addresses of the nocache area.
|
||||
*/
|
||||
|
||||
static DEFINE_SPINLOCK(pte_page_lock);
|
||||
|
||||
pgtable_t pte_alloc_one(struct mm_struct *mm)
|
||||
{
|
||||
unsigned long flags;
|
||||
pte_t *ptep;
|
||||
struct page *page;
|
||||
|
||||
if (!(ptep = pte_alloc_one_kernel(mm)))
|
||||
return NULL;
|
||||
page = pfn_to_page(__nocache_pa((unsigned long)ptep) >> PAGE_SHIFT);
|
||||
spin_lock(&mm->page_table_lock);
|
||||
spin_lock_irqsave(&pte_page_lock, flags);
|
||||
if (page_ref_inc_return(page) == 2 &&
|
||||
!pagetable_pte_ctor(mm, page_ptdesc(page))) {
|
||||
page_ref_dec(page);
|
||||
ptep = NULL;
|
||||
}
|
||||
spin_unlock(&mm->page_table_lock);
|
||||
spin_unlock_irqrestore(&pte_page_lock, flags);
|
||||
|
||||
return ptep;
|
||||
}
|
||||
|
||||
void pte_free(struct mm_struct *mm, pgtable_t ptep)
|
||||
static void __pte_free(pgtable_t ptep)
|
||||
{
|
||||
struct page *page;
|
||||
unsigned long flags;
|
||||
|
||||
page = pfn_to_page(__nocache_pa((unsigned long)ptep) >> PAGE_SHIFT);
|
||||
spin_lock(&mm->page_table_lock);
|
||||
spin_lock_irqsave(&pte_page_lock, flags);
|
||||
if (page_ref_dec_return(page) == 1)
|
||||
pagetable_dtor(page_ptdesc(page));
|
||||
spin_unlock(&mm->page_table_lock);
|
||||
spin_unlock_irqrestore(&pte_page_lock, flags);
|
||||
|
||||
srmmu_free_nocache(ptep, SRMMU_PTE_TABLE_SIZE);
|
||||
}
|
||||
|
||||
void pte_free(struct mm_struct *mm, pgtable_t ptep)
|
||||
{
|
||||
__pte_free(ptep);
|
||||
}
|
||||
|
||||
void __tlb_remove_table(void *table)
|
||||
{
|
||||
const unsigned long encoded = (unsigned long)table;
|
||||
const unsigned long addr = encoded & ~1UL;
|
||||
const bool is_pmd = encoded & 1;
|
||||
|
||||
if (is_pmd)
|
||||
free_pmd_fast((pmd_t *)addr);
|
||||
else
|
||||
__pte_free((pgtable_t)addr);
|
||||
}
|
||||
|
||||
/* context handling - a dynamically sized pool is used */
|
||||
#define NO_CONTEXT -1
|
||||
|
||||
|
||||
@@ -45,7 +45,6 @@ config UML
|
||||
select HAVE_SYSCALL_TRACEPOINTS
|
||||
select THREAD_INFO_IN_TASK
|
||||
select SPARSE_IRQ
|
||||
select MMU_GATHER_RCU_TABLE_FREE if MMU
|
||||
select UACCESS_MEMCPY if !MMU
|
||||
select GENERIC_STRNLEN_USER if !MMU
|
||||
select GENERIC_STRNCPY_FROM_USER if !MMU
|
||||
|
||||
@@ -24,13 +24,6 @@
|
||||
#define USER_PTRS_PER_PGD ((TASK_SIZE + (PGDIR_SIZE - 1)) / PGDIR_SIZE)
|
||||
#define PTRS_PER_PGD 1024
|
||||
|
||||
#define pte_ERROR(e) \
|
||||
printk("%s:%d: bad pte %p(%08lx).\n", __FILE__, __LINE__, &(e), \
|
||||
pte_val(e))
|
||||
#define pgd_ERROR(e) \
|
||||
printk("%s:%d: bad pgd %p(%08lx).\n", __FILE__, __LINE__, &(e), \
|
||||
pgd_val(e))
|
||||
|
||||
static inline int pgd_needsync(pgd_t pgd) { return 0; }
|
||||
static inline void pgd_mkuptodate(pgd_t pgd) { }
|
||||
|
||||
|
||||
@@ -42,19 +42,6 @@
|
||||
|
||||
#define USER_PTRS_PER_PGD ((TASK_SIZE + (PGDIR_SIZE - 1)) / PGDIR_SIZE)
|
||||
|
||||
#define pte_ERROR(e) \
|
||||
printk("%s:%d: bad pte %p(%016lx).\n", __FILE__, __LINE__, &(e), \
|
||||
pte_val(e))
|
||||
#define pmd_ERROR(e) \
|
||||
printk("%s:%d: bad pmd %p(%016lx).\n", __FILE__, __LINE__, &(e), \
|
||||
pmd_val(e))
|
||||
#define pud_ERROR(e) \
|
||||
printk("%s:%d: bad pud %p(%016lx).\n", __FILE__, __LINE__, &(e), \
|
||||
pud_val(e))
|
||||
#define pgd_ERROR(e) \
|
||||
printk("%s:%d: bad pgd %p(%016lx).\n", __FILE__, __LINE__, &(e), \
|
||||
pgd_val(e))
|
||||
|
||||
#define pud_none(x) (!(pud_val(x) & ~_PAGE_NEEDSYNC))
|
||||
#define pud_bad(x) ((pud_val(x) & (~PAGE_MASK & ~_PAGE_USER)) != _KERNPG_TABLE)
|
||||
#define pud_present(x) (pud_val(x) & _PAGE_PRESENT)
|
||||
|
||||
@@ -27,7 +27,6 @@ config X86_64
|
||||
select ARCH_HAS_GIGANTIC_PAGE
|
||||
select ARCH_SUPPORTS_MSEAL_SYSTEM_MAPPINGS
|
||||
select ARCH_SUPPORTS_INT128 if CC_HAS_INT128
|
||||
select ARCH_SUPPORTS_PER_VMA_LOCK
|
||||
select ARCH_SUPPORTS_HUGE_PFNMAP if TRANSPARENT_HUGEPAGE
|
||||
select HAVE_ARCH_SOFT_DIRTY
|
||||
select MODULES_USE_ELF_RELA
|
||||
@@ -150,7 +149,6 @@ config X86
|
||||
select ARCH_WANT_LD_ORPHAN_WARN
|
||||
select ARCH_WANT_OPTIMIZE_DAX_VMEMMAP if X86_64
|
||||
select ARCH_WANT_OPTIMIZE_HUGETLB_VMEMMAP if X86_64
|
||||
select ARCH_WANT_HUGETLB_VMEMMAP_PREINIT if X86_64
|
||||
select ARCH_WANTS_THP_SWAP if X86_64
|
||||
select ARCH_HAS_PARANOID_L1D_FLUSH
|
||||
select ARCH_WANT_IRQS_OFF_ACTIVATE_MM
|
||||
@@ -285,7 +283,6 @@ config X86
|
||||
select HAVE_PERF_REGS
|
||||
select HAVE_PERF_USER_STACK_DUMP
|
||||
select ASYNC_KERNEL_PGTABLE_FREE if IOMMU_SVA
|
||||
select MMU_GATHER_RCU_TABLE_FREE
|
||||
select MMU_GATHER_MERGE_VMAS
|
||||
select HAVE_POSIX_CPU_TIMERS_TASK_WORK
|
||||
select HAVE_REGS_AND_STACK_ACCESS_API
|
||||
@@ -1848,7 +1845,6 @@ config X86_USER_SHADOW_STACK
|
||||
bool "X86 userspace shadow stack"
|
||||
depends on AS_WRUSS
|
||||
depends on X86_64
|
||||
depends on PER_VMA_LOCK
|
||||
select ARCH_USES_HIGH_VMA_FLAGS
|
||||
select ARCH_HAS_USER_SHADOW_STACK
|
||||
select X86_CET
|
||||
|
||||
@@ -11,7 +11,7 @@
|
||||
#undef CONFIG_PGTABLE_LEVELS
|
||||
#undef CONFIG_ILLEGAL_POINTER_VALUE
|
||||
#undef CONFIG_SPARSEMEM_VMEMMAP
|
||||
#undef CONFIG_HUGETLB_PAGE_OPTIMIZE_VMEMMAP
|
||||
#undef CONFIG_VMEMMAP_OPTIMIZATION
|
||||
#undef CONFIG_NR_CPUS
|
||||
#undef CONFIG_PARAVIRT_XXL
|
||||
|
||||
|
||||
@@ -66,9 +66,6 @@ static struct pmu bts_pmu;
|
||||
|
||||
static int buf_nr_pages(struct page *page)
|
||||
{
|
||||
if (!PagePrivate(page))
|
||||
return 1;
|
||||
|
||||
return 1 << page_private(page);
|
||||
}
|
||||
|
||||
|
||||
@@ -781,8 +781,7 @@ static int topa_insert_pages(struct pt_buffer *buf, int cpu, gfp_t gfp)
|
||||
struct page *p;
|
||||
|
||||
p = virt_to_page(buf->data_pages[buf->nr_pages]);
|
||||
if (PagePrivate(p))
|
||||
order = page_private(p);
|
||||
order = page_private(p);
|
||||
|
||||
if (topa_table_full(topa)) {
|
||||
topa = topa_alloc(cpu, gfp);
|
||||
@@ -1296,8 +1295,7 @@ static int pt_buffer_try_single(struct pt_buffer *buf, int nr_pages)
|
||||
if (!intel_pt_validate_hw_cap(PT_CAP_single_range_output))
|
||||
goto out;
|
||||
|
||||
if (PagePrivate(p))
|
||||
order = page_private(p);
|
||||
order = page_private(p);
|
||||
|
||||
if (1 << order != nr_pages)
|
||||
goto out;
|
||||
|
||||
@@ -2,11 +2,6 @@
|
||||
#ifndef _ASM_X86_PGTABLE_2LEVEL_H
|
||||
#define _ASM_X86_PGTABLE_2LEVEL_H
|
||||
|
||||
#define pte_ERROR(e) \
|
||||
pr_err("%s:%d: bad pte %08lx\n", __FILE__, __LINE__, (e).pte_low)
|
||||
#define pgd_ERROR(e) \
|
||||
pr_err("%s:%d: bad pgd %08lx\n", __FILE__, __LINE__, pgd_val(e))
|
||||
|
||||
/*
|
||||
* Certain architectures need to do special things when PTEs
|
||||
* within a page table are directly modified. Thus, the following
|
||||
|
||||
Some files were not shown because too many files have changed in this diff Show More
Reference in New Issue
Block a user