Previously, every asid_free() immediately flushed TLB autorefill
entries on all cores via IPI. This is expensive for workloads that
rapidly create and destroy memory domains (e.g. SOF DP tasks), since
every domain teardown triggers a system-wide TLB invalidation even
when freed ASIDs may never be reused before fresh ones are available.
Add a second bitarray (asid_dirty) where freed ASIDs are placed
without touching the TLB. asid_alloc() uses sys_bitarray_alloc() on
asid_used to find a free ASID; when none remain, performs a single
TLB flush and XORs dirty bits back into asid_used (clearing the
in-use bits for reclaimed ASIDs), amortizing the flush cost across
many alloc/free cycles.
Signed-off-by: Jyri Sarha <jyri.sarha@linux.intel.com>
Add CONFIG_XTENSA_MMU_PTEVADDR_60000000 as a new PTEVADDR base address
choice in the Kconfig.
The PTEVADDR auto-refill mechanism places a self-referencing entry in
each memory domain's L1 page table at position:
(CONFIG_XTENSA_MMU_PTEVADDR >> 22) + ASID
With the default of 0x20000000 (L1 base position 128), ASIDs above 127
place the self-reference at L1 positions 256+, which on platforms like
intel_adsp collides with the uncached memory alias region at VA
0x40000000 (L1 position 256).
Add ASID_PTEVADDR_MAX = 256 - (PTEVADDR >> 22) - 1 and define
ASID_LAST_USER as MIN(XTENSA_MMU_SHARED_ASID - 1, ASID_PTEVADDR_MAX).
ASIDs beyond this limit are marked in-use during bitmap initialization.
Setting PTEVADDR to 0x60000000 (L1 base position 384) places all
possible self-reference entries at L1 positions 384-639, avoiding
collisions with both the uncached alias region (L1 256-383) and cached
SRAM (L1 640+).
For PTEVADDR 0x20000000: max usable ASID = 127.
For PTEVADDR 0x60000000: max usable ASID = 254 (no cap).
Signed-off-by: Jyri Sarha <jyri.sarha@linux.intel.com>
Previously, ASIDs were allocated monotonically and never reused. When a
memory domain was destroyed in arch_mem_domain_deinit(), its ASID was
lost. On platforms where short-lived memory domains are created
frequently (e.g. SOF DP tasks that each get their own domain), the
8-bit ASID space would eventually be exhausted.
Replace the monotonic counter with a bitmap-based allocator using
sys_bitarray covering the full 256-entry ASID space. Bit indices map
directly to ASID values with no translation: set bit (1) = in use or
reserved, clear bit (0) = free. Reserved ASIDs (0-3 and 255) are
marked in-use during initialization so they can never be allocated.
asid_alloc() uses sys_bitarray_alloc() to find a free ASID.
asid_free() releases the ASID and invalidates TLB autorefill entries
on all cores to prevent stale mappings from the previous holder.
Signed-off-by: Jyri Sarha <jyri.sarha@linux.intel.com>
The Xtensa implementation validated the whole maxsize range with
arch_buffer_validate() before calling strnlen(), which breaks the
k_usermode_string_nlen() contract in two ways:
- With maxsize == 0 the string must not be touched at all, but the
validation probed the MPU/MMU region containing the pointer anyway
(alignment rounds a zero size up to a full region), wrongly
failing with -1 for inaccessible addresses.
- The validation required user mode accessibility, while the API
contract explicitly does not guarantee it: user access is checked
by the syscall marshalling layer via K_SYSCALL_MEMORY_READ() once
the length is known. Kernel mode callers measuring kernel strings
(e.g. a buffer on the kernel stack) were wrongly rejected.
Return early for maxsize == 0, and check kernel mode readability
with the new xtensa_buffer_is_kernel_readable() (MPU and MMU
variants) instead of arch_buffer_validate().
On the MMU side, extract a shared mem_buffer_validate() that takes a
ring parameter, so arch_buffer_validate() and the new
xtensa_buffer_is_kernel_readable() share the same logic without
duplication.
Fixeszephyrproject-rtos/zephyr#113722
Signed-off-by: Hongquan Li <hongquan.li@processmission.com>
When __arch_mem_map() fails to map a page, arch_mem_map() should
panic instead of continuing mapping the remaining pages.
Otherwise, there would be hole in the memory map which results
in confusing errors.
Signed-off-by: Daniel Leung <daniel.leung@intel.com>
When a memory domain is freed on Xtensa, it also has to be removed
from the global domain list. Leaving it on the list can cause
use-after-free exceptions.
Signed-off-by: Guennadi Liakhovetski <guennadi.liakhovetski@linux.intel.com>
This supports de-initialization of memory domains to release
allocated page tables back to the pool.
Signed-off-by: Daniel Leung <daniel.leung@intel.com>
When a L2 table is no longer being used, we should set all PTEs
in the table to be illegal PTEs. This is simply a precautious
so that any stray references to the L2 table would not result
in incorrect permissions being applied.
Signed-off-by: Daniel Leung <daniel.leung@intel.com>
This reverts commit 8c02dde437.
For some unknown reasons, xt-clang emits two copies of
z_xt_init_pc if xtensa_mmu_init_paging() is in the same file
as xtensa_mmu_init() and xtensa_mmu_reinit(). So had to
revert the change.
Fixes#103055
Signed-off-by: Daniel Leung <daniel.leung@intel.com>
When using LLEXT, instruction TLB multi-hit becomes a reality
as the same memory space can be occupied by different modules
with different permissions. The ITLB cache may still contain
entries of the unloaded module. So we need to manually
invalidate any cached ITLB corresponding to the exception
address so the TLB associated with the newly loaded module
can be used.
Signed-off-by: Daniel Leung <daniel.leung@intel.com>
There is no need to invalidate the whole auto-refilled data TLB
cache when DTLB multi-hit exception is raised. Now it only
invalidates the TLB entries corresponding to the one causing
the DTLB multi-hit. This allows other non-related TLB entries
to remain in the cache so they don't need to be reloaded.
Signed-off-by: Daniel Leung <daniel.leung@intel.com>
Since we have split source files for page table related stuff
and MMU related stuff, move the MMU initialization functions
from the page table source file into MMU source file.
Signed-off-by: Daniel Leung <daniel.leung@intel.com>
This adds doxygen doc to the page table source file as
we are missing quite a bit of documentation there.
Signed-off-by: Daniel Leung <daniel.leung@intel.com>
If there are not enough free L2 tables to map all predefined
memory regions at boot, halt the system in case assertion is
not enabled. Without all the needed memory regions mapped,
it is very unlikely that anything will run properly.
Signed-off-by: Daniel Leung <daniel.leung@intel.com>
Adds some debug logs when we are allocating page tables.
This provides a more visible way of seeing whether
we need to have more free tables.
Signed-off-by: Daniel Leung <daniel.leung@intel.com>
Add an assertions to halt the system if L2 table allocation
fails when we need to duplicate an existing L2 table, as it is
a must-have and must-success operation.
Signed-off-by: Daniel Leung <daniel.leung@intel.com>
Use void * cleanly: avoid needless type-casts and use void * for
generic pointers instead of uint8_t *.
Signed-off-by: Guennadi Liakhovetski <guennadi.liakhovetski@linux.intel.com>
During L2 page table duplication, there are cases where we
should copy the table over instead of restoring the PTEs.
Add the ability to copy the table over.
Signed-off-by: Daniel Leung <daniel.leung@intel.com>
This adds a spinlock to guard page table counter updates.
For some reasons, without this, xt-clang generated code would
result in incorrect counter update possibly due to memory
access ordering (though GCC generated code works fine without
the spin lock).
Signed-off-by: Daniel Leung <daniel.leung@intel.com>
This removes the following kconfigs:
() CONFIG_XTENSA_RPO_CACHE
() CONFIG_XTENSA_CACHED_REGION
() CONFIG_XTENSA_UNCACHED_REGION
The associated feature is actually SoC specific and not general
architecture feature. Since the only user of these have its own
implementation, we can now safely remove these.
Signed-off-by: Daniel Leung <daniel.leung@intel.com>
This removes CONFIG_XTENSA_MMU_DOUBLE_MAP and its associated
code. The double mapping of a page into both cached and
uncached memory regions should be done in SoC and application,
as it gives finer control on what actually is being mapped.
For example, if a page only needs to mapped in cached region,
there is no need to map it in the uncached region. This may
save the need to allocate a L2 page table for it, and saves
some memory.
Signed-off-by: Daniel Leung <daniel.leung@intel.com>
Adds kconfig CONFIG_XTENSA_MMU_USE_DEFAULT_MAPPINGS for arch
code to initialize the page tables with a few default mappings
covering text, data, BSS and RODATA. Disabling this allows
the SoC to customize and fine-tune the mappings if needed.
Signed-off-by: Daniel Leung <daniel.leung@intel.com>
Set PPN (Physical Page Number) in L2 entry in the region_map_update
function to ensure correct mapping for memory that was not previously
mapped based on xtensa_soc_mmu_ranges.
When a new memory domain is created, the arch_mem_domain_init function
allocates new ptables using the dup_table function. This function only
copies original L2 entries created from xtensa_soc_mmu_ranges with
OPTION_SAVE_ATTRS. Entries added later by arch_mem_map are replaced with
XTENSA_MMU_PTE_L2_ILLEGAL and have PPN (Physical Page Number) set to zero.
When adding a new partition to a memory domain, the update_region function
is used. It sets new access attributes in the L2 table without setting the
PPN value. As a result, it does not work correctly for memory that was not
previously mapped based on xtensa_soc_mmu_ranges.
Signed-off-by: Adrian Warecki <adrian.warecki@intel.com>
Add macro PTE_PPN_SET to simplify setting the physical page
number assigned in the page table entry.
Signed-off-by: Adrian Warecki <adrian.warecki@intel.com>
Add macro PTE_PPN_GET to simplify retrieval of the physical page
number assigned to a page table entry.
Signed-off-by: Adrian Warecki <adrian.warecki@intel.com>
xt-clang complains about l2_page_tables_counter_inc() being
unused but not GCC. So fix that by using it somewhere else.
Fixes#99753
Signed-off-by: Daniel Leung <daniel.leung@intel.com>
Adds function for app to obtain page table usage statistics,
allowing fine tuning of numbers of L1 and L2 page table
array.
Signed-off-by: Daniel Leung <daniel.leung@intel.com>
When adding new memory domains, we need to start from kernel
page tables by duplicating them. However, there is no need to
duplicate all in-use page tables as some memory regions remain
unchanged. Fror example, hardware register region where only
kernel has access does not need to be duplicated across all
memory domains. The same L2 table can be used among them.
This changes the L2 page table allocation to be copy-on-write
where we only need to duplicate a L2 table if changes need to
be made on it.
Signed-off-by: Daniel Leung <daniel.leung@intel.com>
If the page tables are not cached, there is no need to do any
cache ops to flush or invalidate the data in cache. So skip them
if the page tables are not cached. Saves a few CPU cycles.
Signed-off-by: Daniel Leung <daniel.leung@intel.com>
In l2_page_table_unmap(), invalidating TLBs should be limited to
the address and not the whole auto-refill TLBs. Also fix a bug
where the EXEC bit should be determined from the L2 PTE and not
the L1 PTE.
Signed-off-by: Daniel Leung <daniel.leung@intel.com>
Some macros for the page tables array are local to the file so
there is no need to prefix them with XTENSA_. Simplify by
removing the XTENSA_ prefix, and clairfy if needed.
Signed-off-by: Daniel Leung <daniel.leung@intel.com>
Almost all page table entry (PTE) related macros are only used
in one source file. These macros are considered the internal
working of MMU. There is no need to expose them in header file
for other to reference. So move them into the source file where
it is only place they are used. Bonus is that we can shorten
the macro names as they are now local to the file. Makes it
easier to read, and few keystrokes to input.
Signed-off-by: Daniel Leung <daniel.leung@intel.com>
This skips the redirection of backup attributes and ring via
some SW field macros, and use the PTE bits directly. Should
make it easier to decode what's going on.
Signed-off-by: Daniel Leung <daniel.leung@intel.com>
arch_mem_map() takes in some flags to describe the to-be mapped
memory regions' permissions and cache status. When the flags are
translated, they become attributes in PTEs. So for functions
being called by arch_mem_map() and beyond, rename flags to
attrs to better describe its purpose.
Signed-off-by: Daniel Leung <daniel.leung@intel.com>
In functions which manipulate both L1 and L2 tables, make
the variable names obvious by prefixing them with l1_ or l2_.
This is mainly done to avoid confusion when reading through
those functions.
Signed-off-by: Daniel Leung <daniel.leung@intel.com>
The assert error messages when l2_page_table_map() fails are not
correct. It returns false when it cannot allocate new L2 table,
and it is not able the address having already mapped. So correct
the error message.
Signed-off-by: Daniel Leung <daniel.leung@intel.com>
This adds a function to handle DTLB multihit exception when
userspace is enabled, as this exception may be raised due to
the same page being able to accessed by both kernel and user
threads. The auto-refill DTLBs may contain entries for same
page, one for kernel and one for user under some situations.
We need to invalidate those existing DTLB entries so that
hardware can reload from the page table.
This is an alternative to the kconfig invalidating DTLBs on
every swap: CONFIG_XTENSA_MMU_FLUSH_AUTOREFILL_DTLBS_ON_SWAP.
Although this does not have extra processing per context
switching, exception handling itself has a high cost. So
care needs to be taken on whether to enable that kconfig.
Signed-off-by: Daniel Leung <daniel.leung@intel.com>
When a page can be accessed by both kernel and user threads,
the autofill DTLB may contain an entry for kernel thread.
This will result in load/store ring exception when it is
accessed by user thread later. In this case, we need to
invalidate all associated TLBs related to kernel access so
hardware can reload the page table the correct permission
for user thread.
Signed-off-by: Daniel Leung <daniel.leung@intel.com>
The software bits inside PTE are used to store original PTE
attributes and ring value, and those bits are used to
restore PTE to previous state.
This modifies reset_region() to properly restore attributes
and ring value when resetting memory regions to the same as
when the system boots.
Signed-off-by: Daniel Leung <daniel.leung@intel.com>
Several static functions in ptables.c always return 0, make them void
to improve readability.
Signed-off-by: Guennadi Liakhovetski <guennadi.liakhovetski@linux.intel.com>
This adds a new kconfig and corresponding code to allow flushing
auto-refill data TLBs when page tables are swapped (e.g. during
context switching). This is mainly used to avoid multi-hit TLB
exception raised by certain memory access pattern. If memory is
only marked for user mode access but not inside a memory domain,
accessing that page in kernel mode would result in a TLB being
filled with kernel ASID. When going back into user mode, access
to the memory would result in another TLB being filled with
the user mode ASID. Now there are two entries on the same memory
page, and the multi-hit TLB exception will be raised if that
memory page is accessed. This type of access is better served
using memory partition and memory domain to share data. However,
this type of access is not prohibited but highly discouraged.
Wrapping the code in kconfig is simply because of the execution
penalty as there will be unnecessary TLB refilling being done.
So only enable this if necessary.
Fixes#88772
Signed-off-by: Daniel Leung <daniel.leung@intel.com>
Since the necessary register values are now pre-computed and
stored in the memory domain struct, we can use them directly
in various assembly locations, thus replacing the function
call to xtensa_swap_update_page_tables().
Signed-off-by: Daniel Leung <daniel.leung@intel.com>
Instead of computing all the needed register values when
swapping page tables, we can pre-compute those values when
the memory domain is first initialized. Should save some
time during context switching.
Signed-off-by: Daniel Leung <daniel.leung@intel.com>
Remove CONFIG_XTENSA_INVALIDATE_MEM_DOMAIN_TLB_ON_SWAP as it is
remnant from early MMU enabling work which is not needed as
the page table code is different from early version where
the PTEVADDR would be the same for all memory domains.
Signed-off-by: Daniel Leung <daniel.leung@intel.com>
thread_page_tables_get() is only used when userspace is
enabled. So move it with userspace #ifdef, or else
compiler would complain about it being unused.
Fixes#88421
Signed-off-by: Daniel Leung <daniel.leung@intel.com>
Inside map_memory() with double mapping enabled, we should not
be mapping the memory with the incoming attributes as-is since
the incoming address may be on un-cached region but with
caching attribute. So we need to sanitize the attributes
according to the incoming address.
Signed-off-by: Daniel Leung <daniel.leung@intel.com>
Mostly a revert of commit b1def7145f ("arch: deprecate `_current`").
This commit was part of PR #80716 whose initial purpose was about providing
an architecture specific optimization for _current. The actual deprecation
was sneaked in later on without proper discussion.
The Zephyr core always used _current before and that was fine. It is quite
prevalent as well and the alternative is proving rather verbose.
Furthermore, as a concept, the "current thread" is not something that is
necessarily architecture specific. Therefore the primary abstraction
should not carry the arch_ prefix.
Hence this revert.
Signed-off-by: Nicolas Pitre <npitre@baylibre.com>
