This supports de-initialization of memory domains to release
allocated page tables back to the pool.
Signed-off-by: Daniel Leung <daniel.leung@intel.com>
When a L2 table is no longer being used, we should set all PTEs
in the table to be illegal PTEs. This is simply a precautious
so that any stray references to the L2 table would not result
in incorrect permissions being applied.
Signed-off-by: Daniel Leung <daniel.leung@intel.com>
This reverts commit 8c02dde437.
For some unknown reasons, xt-clang emits two copies of
z_xt_init_pc if xtensa_mmu_init_paging() is in the same file
as xtensa_mmu_init() and xtensa_mmu_reinit(). So had to
revert the change.
Fixes#103055
Signed-off-by: Daniel Leung <daniel.leung@intel.com>
When using LLEXT, instruction TLB multi-hit becomes a reality
as the same memory space can be occupied by different modules
with different permissions. The ITLB cache may still contain
entries of the unloaded module. So we need to manually
invalidate any cached ITLB corresponding to the exception
address so the TLB associated with the newly loaded module
can be used.
Signed-off-by: Daniel Leung <daniel.leung@intel.com>
There is no need to invalidate the whole auto-refilled data TLB
cache when DTLB multi-hit exception is raised. Now it only
invalidates the TLB entries corresponding to the one causing
the DTLB multi-hit. This allows other non-related TLB entries
to remain in the cache so they don't need to be reloaded.
Signed-off-by: Daniel Leung <daniel.leung@intel.com>
Since we have split source files for page table related stuff
and MMU related stuff, move the MMU initialization functions
from the page table source file into MMU source file.
Signed-off-by: Daniel Leung <daniel.leung@intel.com>
This adds doxygen doc to the page table source file as
we are missing quite a bit of documentation there.
Signed-off-by: Daniel Leung <daniel.leung@intel.com>
If there are not enough free L2 tables to map all predefined
memory regions at boot, halt the system in case assertion is
not enabled. Without all the needed memory regions mapped,
it is very unlikely that anything will run properly.
Signed-off-by: Daniel Leung <daniel.leung@intel.com>
Adds some debug logs when we are allocating page tables.
This provides a more visible way of seeing whether
we need to have more free tables.
Signed-off-by: Daniel Leung <daniel.leung@intel.com>
Add an assertions to halt the system if L2 table allocation
fails when we need to duplicate an existing L2 table, as it is
a must-have and must-success operation.
Signed-off-by: Daniel Leung <daniel.leung@intel.com>
Use void * cleanly: avoid needless type-casts and use void * for
generic pointers instead of uint8_t *.
Signed-off-by: Guennadi Liakhovetski <guennadi.liakhovetski@linux.intel.com>
During L2 page table duplication, there are cases where we
should copy the table over instead of restoring the PTEs.
Add the ability to copy the table over.
Signed-off-by: Daniel Leung <daniel.leung@intel.com>
This adds a spinlock to guard page table counter updates.
For some reasons, without this, xt-clang generated code would
result in incorrect counter update possibly due to memory
access ordering (though GCC generated code works fine without
the spin lock).
Signed-off-by: Daniel Leung <daniel.leung@intel.com>
This removes the following kconfigs:
() CONFIG_XTENSA_RPO_CACHE
() CONFIG_XTENSA_CACHED_REGION
() CONFIG_XTENSA_UNCACHED_REGION
The associated feature is actually SoC specific and not general
architecture feature. Since the only user of these have its own
implementation, we can now safely remove these.
Signed-off-by: Daniel Leung <daniel.leung@intel.com>
This removes CONFIG_XTENSA_MMU_DOUBLE_MAP and its associated
code. The double mapping of a page into both cached and
uncached memory regions should be done in SoC and application,
as it gives finer control on what actually is being mapped.
For example, if a page only needs to mapped in cached region,
there is no need to map it in the uncached region. This may
save the need to allocate a L2 page table for it, and saves
some memory.
Signed-off-by: Daniel Leung <daniel.leung@intel.com>
Adds kconfig CONFIG_XTENSA_MMU_USE_DEFAULT_MAPPINGS for arch
code to initialize the page tables with a few default mappings
covering text, data, BSS and RODATA. Disabling this allows
the SoC to customize and fine-tune the mappings if needed.
Signed-off-by: Daniel Leung <daniel.leung@intel.com>
Set PPN (Physical Page Number) in L2 entry in the region_map_update
function to ensure correct mapping for memory that was not previously
mapped based on xtensa_soc_mmu_ranges.
When a new memory domain is created, the arch_mem_domain_init function
allocates new ptables using the dup_table function. This function only
copies original L2 entries created from xtensa_soc_mmu_ranges with
OPTION_SAVE_ATTRS. Entries added later by arch_mem_map are replaced with
XTENSA_MMU_PTE_L2_ILLEGAL and have PPN (Physical Page Number) set to zero.
When adding a new partition to a memory domain, the update_region function
is used. It sets new access attributes in the L2 table without setting the
PPN value. As a result, it does not work correctly for memory that was not
previously mapped based on xtensa_soc_mmu_ranges.
Signed-off-by: Adrian Warecki <adrian.warecki@intel.com>
Add macro PTE_PPN_SET to simplify setting the physical page
number assigned in the page table entry.
Signed-off-by: Adrian Warecki <adrian.warecki@intel.com>
Add macro PTE_PPN_GET to simplify retrieval of the physical page
number assigned to a page table entry.
Signed-off-by: Adrian Warecki <adrian.warecki@intel.com>
xt-clang complains about l2_page_tables_counter_inc() being
unused but not GCC. So fix that by using it somewhere else.
Fixes#99753
Signed-off-by: Daniel Leung <daniel.leung@intel.com>
Adds function for app to obtain page table usage statistics,
allowing fine tuning of numbers of L1 and L2 page table
array.
Signed-off-by: Daniel Leung <daniel.leung@intel.com>
When adding new memory domains, we need to start from kernel
page tables by duplicating them. However, there is no need to
duplicate all in-use page tables as some memory regions remain
unchanged. Fror example, hardware register region where only
kernel has access does not need to be duplicated across all
memory domains. The same L2 table can be used among them.
This changes the L2 page table allocation to be copy-on-write
where we only need to duplicate a L2 table if changes need to
be made on it.
Signed-off-by: Daniel Leung <daniel.leung@intel.com>
If the page tables are not cached, there is no need to do any
cache ops to flush or invalidate the data in cache. So skip them
if the page tables are not cached. Saves a few CPU cycles.
Signed-off-by: Daniel Leung <daniel.leung@intel.com>
In l2_page_table_unmap(), invalidating TLBs should be limited to
the address and not the whole auto-refill TLBs. Also fix a bug
where the EXEC bit should be determined from the L2 PTE and not
the L1 PTE.
Signed-off-by: Daniel Leung <daniel.leung@intel.com>
Some macros for the page tables array are local to the file so
there is no need to prefix them with XTENSA_. Simplify by
removing the XTENSA_ prefix, and clairfy if needed.
Signed-off-by: Daniel Leung <daniel.leung@intel.com>
Almost all page table entry (PTE) related macros are only used
in one source file. These macros are considered the internal
working of MMU. There is no need to expose them in header file
for other to reference. So move them into the source file where
it is only place they are used. Bonus is that we can shorten
the macro names as they are now local to the file. Makes it
easier to read, and few keystrokes to input.
Signed-off-by: Daniel Leung <daniel.leung@intel.com>
This skips the redirection of backup attributes and ring via
some SW field macros, and use the PTE bits directly. Should
make it easier to decode what's going on.
Signed-off-by: Daniel Leung <daniel.leung@intel.com>
arch_mem_map() takes in some flags to describe the to-be mapped
memory regions' permissions and cache status. When the flags are
translated, they become attributes in PTEs. So for functions
being called by arch_mem_map() and beyond, rename flags to
attrs to better describe its purpose.
Signed-off-by: Daniel Leung <daniel.leung@intel.com>
In functions which manipulate both L1 and L2 tables, make
the variable names obvious by prefixing them with l1_ or l2_.
This is mainly done to avoid confusion when reading through
those functions.
Signed-off-by: Daniel Leung <daniel.leung@intel.com>
The assert error messages when l2_page_table_map() fails are not
correct. It returns false when it cannot allocate new L2 table,
and it is not able the address having already mapped. So correct
the error message.
Signed-off-by: Daniel Leung <daniel.leung@intel.com>
This adds a function to handle DTLB multihit exception when
userspace is enabled, as this exception may be raised due to
the same page being able to accessed by both kernel and user
threads. The auto-refill DTLBs may contain entries for same
page, one for kernel and one for user under some situations.
We need to invalidate those existing DTLB entries so that
hardware can reload from the page table.
This is an alternative to the kconfig invalidating DTLBs on
every swap: CONFIG_XTENSA_MMU_FLUSH_AUTOREFILL_DTLBS_ON_SWAP.
Although this does not have extra processing per context
switching, exception handling itself has a high cost. So
care needs to be taken on whether to enable that kconfig.
Signed-off-by: Daniel Leung <daniel.leung@intel.com>
When a page can be accessed by both kernel and user threads,
the autofill DTLB may contain an entry for kernel thread.
This will result in load/store ring exception when it is
accessed by user thread later. In this case, we need to
invalidate all associated TLBs related to kernel access so
hardware can reload the page table the correct permission
for user thread.
Signed-off-by: Daniel Leung <daniel.leung@intel.com>
The software bits inside PTE are used to store original PTE
attributes and ring value, and those bits are used to
restore PTE to previous state.
This modifies reset_region() to properly restore attributes
and ring value when resetting memory regions to the same as
when the system boots.
Signed-off-by: Daniel Leung <daniel.leung@intel.com>
Several static functions in ptables.c always return 0, make them void
to improve readability.
Signed-off-by: Guennadi Liakhovetski <guennadi.liakhovetski@linux.intel.com>
This adds a new kconfig and corresponding code to allow flushing
auto-refill data TLBs when page tables are swapped (e.g. during
context switching). This is mainly used to avoid multi-hit TLB
exception raised by certain memory access pattern. If memory is
only marked for user mode access but not inside a memory domain,
accessing that page in kernel mode would result in a TLB being
filled with kernel ASID. When going back into user mode, access
to the memory would result in another TLB being filled with
the user mode ASID. Now there are two entries on the same memory
page, and the multi-hit TLB exception will be raised if that
memory page is accessed. This type of access is better served
using memory partition and memory domain to share data. However,
this type of access is not prohibited but highly discouraged.
Wrapping the code in kconfig is simply because of the execution
penalty as there will be unnecessary TLB refilling being done.
So only enable this if necessary.
Fixes#88772
Signed-off-by: Daniel Leung <daniel.leung@intel.com>
Since the necessary register values are now pre-computed and
stored in the memory domain struct, we can use them directly
in various assembly locations, thus replacing the function
call to xtensa_swap_update_page_tables().
Signed-off-by: Daniel Leung <daniel.leung@intel.com>
Instead of computing all the needed register values when
swapping page tables, we can pre-compute those values when
the memory domain is first initialized. Should save some
time during context switching.
Signed-off-by: Daniel Leung <daniel.leung@intel.com>
Remove CONFIG_XTENSA_INVALIDATE_MEM_DOMAIN_TLB_ON_SWAP as it is
remnant from early MMU enabling work which is not needed as
the page table code is different from early version where
the PTEVADDR would be the same for all memory domains.
Signed-off-by: Daniel Leung <daniel.leung@intel.com>
thread_page_tables_get() is only used when userspace is
enabled. So move it with userspace #ifdef, or else
compiler would complain about it being unused.
Fixes#88421
Signed-off-by: Daniel Leung <daniel.leung@intel.com>
Inside map_memory() with double mapping enabled, we should not
be mapping the memory with the incoming attributes as-is since
the incoming address may be on un-cached region but with
caching attribute. So we need to sanitize the attributes
according to the incoming address.
Signed-off-by: Daniel Leung <daniel.leung@intel.com>
Mostly a revert of commit b1def7145f ("arch: deprecate `_current`").
This commit was part of PR #80716 whose initial purpose was about providing
an architecture specific optimization for _current. The actual deprecation
was sneaked in later on without proper discussion.
The Zephyr core always used _current before and that was fine. It is quite
prevalent as well and the alternative is proving rather verbose.
Furthermore, as a concept, the "current thread" is not something that is
necessarily architecture specific. Therefore the primary abstraction
should not carry the arch_ prefix.
Hence this revert.
Signed-off-by: Nicolas Pitre <npitre@baylibre.com>
`_current` is now functionally equals to `arch_curr_thread()`, remove
its usage in-tree and deprecate it instead of removing it outright,
as it has been with us since forever.
Signed-off-by: Yong Cong Sin <ycsin@meta.com>
Signed-off-by: Yong Cong Sin <yongcong.sin@gmail.com>
xtensa_mmu_init() is called really early in the boot process
where the _kernel struct has not yet been initialized, and
thus we cannot use it to determine if the current CPU is
the boot CPU. In some cases, this may skip the call to
initialize the page tables which leaves us with incorrect
page table entries. Fix it by using a static variable to
determine whether the page tables have been initialized so
we only do it once per boot.
Fixes#76909
Signed-off-by: Daniel Leung <daniel.leung@intel.com>
ZSR_DEPC_SAVE is being used to determine whether we are faulting
inside double exception if this is not zero. It is possible that
the boot ROM or custom startup code leaves this non-zero, which
would result in a fake triple fault. So clear it at boot. Note
that the zeroing is done in MMU init code as these triple
faults are not actual hardware ones but only semantics, and will
occur once MMU is enabled.
Fixes#75194
Signed-off-by: Daniel Leung <daniel.leung@intel.com>
This adds a new function xtensa_mem_kernel_has_access() to
determine if a memory region can be accessed by kernel threads.
This allows checking for valid mapped memory before accessing
them to avoid relying on page faults to detect invalid access.
Also fixed an issue with arch_buffer_validate() on MPU where
it may return okay even if the incoming memory region has no
corresponding entry in the MPU table.
Signed-off-by: Daniel Leung <daniel.leung@intel.com>
arch_buffer_validate() is only to verify that user threads have
access to the memory region. It should not be used to verify
if kernel thread has access (which they should anyway). So
change the logic.
Signed-off-by: Daniel Leung <daniel.leung@intel.com>
Also any demand paging and page frame related bits are
renamed.
This is part of a series to move memory management related
stuff out of the Z_ namespace into its own namespace.
Signed-off-by: Daniel Leung <daniel.leung@intel.com>
With power managment is enabled, depending on the SoC power state
used when idle, the MMU may lose context and may need to be re-initialized.
When re-initializing the MMU, we must not re-create the page table
because it may overwrite changes done during the execution, but we still
need to set the asid and page table for the current context.
Signed-off-by: Flavio Ceolin <flavio.ceolin@intel.com>
The only page table duplicated is the kernel page table. This function
does not need a parameter.
Signed-off-by: Flavio Ceolin <flavio.ceolin@intel.com>
