EXCVADDR describes exceptions, not interrupts, and only the level 1
vector serves exceptions. The frames built by interrupt-only vectors
never have their excvaddr slot read (the C interrupt handlers do not
look at it, and nothing restores it on exit), so stop saving it
there. Fatal errors raised from ISRs still get a proper EXCVADDR:
they trap through the level 1 exception vector, which keeps the save.
Benchmarks on qemu_xtensa/dc233c (QEMU icount shift=6, 1 insn = 1
cycle) and ESP32-S3 (m5stack_stamps3 at 240 MHz), exercising a
level 3 software interrupt:
* thread_metric interrupt (higher is better):
- qemu_xtensa/dc233c: 759968 -> 762439 (+0.3%)
- ESP32-S3: 9281007 -> 9304992 (+0.3%)
* latency_measure: unchanged within +/-1 cycle alignment noise (its
ISR metrics start timing inside the ISR, after vector entry)
* code size: unchanged (vector slots are fixed-size)
Assisted-by: Claude:fable-5
Signed-off-by: Benjamin Cabé <benjamin@zephyrproject.org>
The Xtensa implementation validated the whole maxsize range with
arch_buffer_validate() before calling strnlen(), which breaks the
k_usermode_string_nlen() contract in two ways:
- With maxsize == 0 the string must not be touched at all, but the
validation probed the MPU/MMU region containing the pointer anyway
(alignment rounds a zero size up to a full region), wrongly
failing with -1 for inaccessible addresses.
- The validation required user mode accessibility, while the API
contract explicitly does not guarantee it: user access is checked
by the syscall marshalling layer via K_SYSCALL_MEMORY_READ() once
the length is known. Kernel mode callers measuring kernel strings
(e.g. a buffer on the kernel stack) were wrongly rejected.
Return early for maxsize == 0, and check kernel mode readability
with the new xtensa_buffer_is_kernel_readable() (MPU and MMU
variants) instead of arch_buffer_validate().
On the MMU side, extract a shared mem_buffer_validate() that takes a
ring parameter, so arch_buffer_validate() and the new
xtensa_buffer_is_kernel_readable() share the same logic without
duplication.
Fixeszephyrproject-rtos/zephyr#113722
Signed-off-by: Hongquan Li <hongquan.li@processmission.com>
This removes all the extra code path when pin only CPU mask
and kernel coherence are both enabled. The original idea was
that with CPU pinning only, we could skip all the stack
manipulation since threads are not going to migreate between
CPUs until they get pinned to another CPUs. However, this
creates an issue where a thread is created on one CPU and
its stack being populated for startup at this CPU. Then it
gets pinned to another CPU. The other CPU does not know that
it is first starting the thread and will need to invalidate
the cache to grab the new stack content. This problem also
applies to migration between CPUs as there is no code to
manipulate the cache on stack. To support the skipping of
cache manipulation with pin only CPU mask would require
some invasive changes to kernel. At this point, the only
Xtensa SoCs requiring kernel coherence are not using pin
only CPU mask. So we are safe to remove the special code
path, and also we unify stack coherence operations to
minimize differences.
Fixes#112850
Signed-off-by: Daniel Leung <daniel.leung@intel.com>
With the xtensa_mpu_mem_type_ranges[] array, the SoC or board
should be the one dictating memory types of various memory
regions. There is no need for this kconfig to be there and
we can use the defaults for the architecture.
Note that the previous P_NA_U_NA access rights are replaced
with the default one, as it should reflect the background map
where it is be used.
Signed-off-by: Daniel Leung <daniel.leung@intel.com>
Having different signature for xtensa_mpu_map_write() based on
kconfig is error-prone. So fix that.
Rename xtensa_mpu_map_write() to xtensa_map_thread_map_write()
to reflect this change and to signify that it takes thread
pointer as argument.
Signed-off-by: Daniel Leung <daniel.leung@intel.com>
We have two custom EXCCAUSE numbers to be used for raising
our own custom exceptions. Instead of having magic numbers
in the code, we give them names.
Signed-off-by: Daniel Leung <daniel.leung@intel.com>
When handling an ISR (which does not have a context from which to
restore its own value of LCOUNT), we must clear LCOUNT to prevents
incorrect zero-overhead execution if calling a function such as
memmove() which could be implemented using zero-overhead loop.
A function such as memmove() implemented using zero-overhead loop
assumes LCOUNT to have properly been setup before being called; but
an ISR calling memmove() in assembly, will likely not know that.
Signed-off-by: William Tambe <williamt@cadence.com>
This reverts commit 8c02dde437.
For some unknown reasons, xt-clang emits two copies of
z_xt_init_pc if xtensa_mmu_init_paging() is in the same file
as xtensa_mmu_init() and xtensa_mmu_reinit(). So had to
revert the change.
Fixes#103055
Signed-off-by: Daniel Leung <daniel.leung@intel.com>
When using LLEXT, instruction TLB multi-hit becomes a reality
as the same memory space can be occupied by different modules
with different permissions. The ITLB cache may still contain
entries of the unloaded module. So we need to manually
invalidate any cached ITLB corresponding to the exception
address so the TLB associated with the newly loaded module
can be used.
Signed-off-by: Daniel Leung <daniel.leung@intel.com>
There is no need to invalidate the whole auto-refilled data TLB
cache when DTLB multi-hit exception is raised. Now it only
invalidates the TLB entries corresponding to the one causing
the DTLB multi-hit. This allows other non-related TLB entries
to remain in the cache so they don't need to be reloaded.
Signed-off-by: Daniel Leung <daniel.leung@intel.com>
Since we have split source files for page table related stuff
and MMU related stuff, move the MMU initialization functions
from the page table source file into MMU source file.
Signed-off-by: Daniel Leung <daniel.leung@intel.com>
Preserve EXCCAUSE and EXCVADDR values on entry to _Level1Vector.
Restore EXCCAUSE when exiting TLB miss exception handling in the double
exception handler.
During first-level exception handling, a LoadStoreTLBMissException may
occur during the initial register dump to BSA. It modifies EXCCAUSE and
EXCVADDR registers before they are saved in BSA. Therefore, these values
must be captured as early as possible.
Signed-off-by: Adrian Warecki <adrian.warecki@intel.com>
Remove saving EXCCAUSE register in BSA through the _Level1Vector handler.
These value are later overwritten by the ODD_REG_SAVE macro called by
EXCINT_HANDLER, so saving it here is pointless.
Signed-off-by: Adrian Warecki <adrian.warecki@intel.com>
soc_per_core_init_hook() is usually called from arch_kernel_init() and
arch_secondary_cpu_init() which are C functions. As such, there is no need
to check for CONFIG_SOC_PER_CORE_INIT_HOOK since platform/hooks.h provides
a no-op function-like macro implementation if the Kconfig option is not
enabled.
Remove the Kconfig option check from all files.
Signed-off-by: Mathieu Choplain <mathieu.choplain-ext@st.com>
Almost all page table entry (PTE) related macros are only used
in one source file. These macros are considered the internal
working of MMU. There is no need to expose them in header file
for other to reference. So move them into the source file where
it is only place they are used. Bonus is that we can shorten
the macro names as they are now local to the file. Makes it
easier to read, and few keystrokes to input.
Signed-off-by: Daniel Leung <daniel.leung@intel.com>
This skips the redirection of backup attributes and ring via
some SW field macros, and use the PTE bits directly. Should
make it easier to decode what's going on.
Signed-off-by: Daniel Leung <daniel.leung@intel.com>
The fields in macro should be done with both shift and mask
values to make it clear. So amend some macros so that
the presentation are unified for them.
Signed-off-by: Daniel Leung <daniel.leung@intel.com>
This is simply done to conserve code space in the vector text
areas. These vector text areas are very small and we should
only put code that is absolutely necessary for interrupt and
exception entrance. The saving of PS into the thread struct
can be deferred a bit. So do that.
Signed-off-by: Daniel Leung <daniel.leung@intel.com>
xtensa_asm2_s.h is not exactly a C header. Rename it to
xtensa_asm2.inc.S to clearly state that it is has assembly
code in it as its main purpose is to declare assembly macros.
This is being done to keep checkpatch.pl from treating it as
C file and complaining about non-C syntax.
Signed-off-by: Daniel Leung <daniel.leung@intel.com>
This moves the block of FPU and HIFI registers in the Base Save
Area (BSA), used during interrupts and exceptions, to the end of
the block. This is done to minimize code usage in the vector
text section. During interrupt entrance, the code stores a small
set of registers first before jumping to the long handler. When
the offset to these registers from the beginning of BSA is small
enough, the compiler will emit S32I.N instead of S32I. This
saves us one byte per store (2 vs 3 bytes). This is mainly done
to avoid overflowing the vector text area.
Signed-off-by: Daniel Leung <daniel.leung@intel.com>
This adds a function to handle DTLB multihit exception when
userspace is enabled, as this exception may be raised due to
the same page being able to accessed by both kernel and user
threads. The auto-refill DTLBs may contain entries for same
page, one for kernel and one for user under some situations.
We need to invalidate those existing DTLB entries so that
hardware can reload from the page table.
This is an alternative to the kconfig invalidating DTLBs on
every swap: CONFIG_XTENSA_MMU_FLUSH_AUTOREFILL_DTLBS_ON_SWAP.
Although this does not have extra processing per context
switching, exception handling itself has a high cost. So
care needs to be taken on whether to enable that kconfig.
Signed-off-by: Daniel Leung <daniel.leung@intel.com>
When a page can be accessed by both kernel and user threads,
the autofill DTLB may contain an entry for kernel thread.
This will result in load/store ring exception when it is
accessed by user thread later. In this case, we need to
invalidate all associated TLBs related to kernel access so
hardware can reload the page table the correct permission
for user thread.
Signed-off-by: Daniel Leung <daniel.leung@intel.com>
This saves the EXCCAUSE and EXCVADDR into BSA during exception.
These will then be used during exception handling. The reason
for doing this instead of reading from both registers during
exception handing is that another exception will override
the value in these register (e.g. DTLB miss). When returning to
the previous exception handler, these register no longer
contains the original exception cause and address. We need to
save it so that we are actually handling the orignal exception.
Coredump also now uses the EXCCAUSE and EXCVADDR from BSA
instead of reading the registers.
Signed-off-by: Daniel Leung <daniel.leung@intel.com>
The software bits inside PTE are used to store original PTE
attributes and ring value, and those bits are used to
restore PTE to previous state.
This modifies reset_region() to properly restore attributes
and ring value when resetting memory regions to the same as
when the system boots.
Signed-off-by: Daniel Leung <daniel.leung@intel.com>
Fixes a flaw in the the xtensa implementation of arch_is_in_isr()
that could manifest on SMP systems. If the reading of the current
CPU's nested interrupt count is not fully atomic on an SMP system,
then an ill-timed context switch could occur leaving the caller
reading the nested interrupt of a different CPU.
Signed-off-by: Peter Mitsis <peter.mitsis@intel.com>
Splits HiFi sharing into two different models.
1. XTENSA_EAGER_HIFI_SHARING - unconditional save/restore
of the HiFi registers when context switching
2. XTENSA_LAZY_HIFI_SHARING - on demand save/restore of
of the HiFi registers. If a thread does not use the
HiFi registers, they are neither saved nor restored.
To maintain backwards compatibility, the eager model is the
default model when XTENSA_HIFI_SHARING is enabled.
Signed-off-by: Peter Mitsis <peter.mitsis@intel.com>
Add missing call to thread_switched_in for the purpose of tracing.
Fixeszephyrproject-rtos/zephyr#76057
Signed-off-by: Anas Nashif <anas.nashif@intel.com>
This adds a new kconfig and corresponding code to allow flushing
auto-refill data TLBs when page tables are swapped (e.g. during
context switching). This is mainly used to avoid multi-hit TLB
exception raised by certain memory access pattern. If memory is
only marked for user mode access but not inside a memory domain,
accessing that page in kernel mode would result in a TLB being
filled with kernel ASID. When going back into user mode, access
to the memory would result in another TLB being filled with
the user mode ASID. Now there are two entries on the same memory
page, and the multi-hit TLB exception will be raised if that
memory page is accessed. This type of access is better served
using memory partition and memory domain to share data. However,
this type of access is not prohibited but highly discouraged.
Wrapping the code in kconfig is simply because of the execution
penalty as there will be unnecessary TLB refilling being done.
So only enable this if necessary.
Fixes#88772
Signed-off-by: Daniel Leung <daniel.leung@intel.com>
Since the necessary register values are now pre-computed and
stored in the memory domain struct, we can use them directly
in various assembly locations, thus replacing the function
call to xtensa_swap_update_page_tables().
Signed-off-by: Daniel Leung <daniel.leung@intel.com>
When context switching and dealing with non-nested interrupts,
the context to be restored are saved in the thread stack.
When userspace is enabled, this means saving context into
the user stacks for user threads. This allows PS values to be
manipulated externally by setting PS.RING in the saved PS
value to 0, resulting in granting kernel access privilege when
the thread is restored. To prevent this, we store the PS value
into the thread struct instead, where user threads cannot
manipulate that. Note that nested interrupts and syscalls are
not using the user stack but the interrupt stack and thread
privileged stack respectively, where they are not accessible
under user mode.
Signed-off-by: Daniel Leung <daniel.leung@intel.com>
This extends arch_cohere_stacks() to handle privileged stacks of
user threads when userspace is enabled.
Signed-off-by: Daniel Leung <daniel.leung@intel.com>
Towards the end of interrupt handling, and before restoring
context, we would spill all register windows. This requires
A0 and A1 to be restored from the saved context so spilling
would work correct. However, when coherence is enabled,
window spilling has already been done earlier so there is
no need to spill the register windows again. So there is
no need to restore A0 and A1. They will be restored again
before returning from interrupt anyway.
Signed-off-by: Daniel Leung <daniel.leung@intel.com>
Instead of computing all the needed register values when
swapping page tables, we can pre-compute those values when
the memory domain is first initialized. Should save some
time during context switching.
Signed-off-by: Daniel Leung <daniel.leung@intel.com>
There is no need for ZSR_FLUSH when threads are pin only
(CONFIG_SCHED_CPU_MASK_PIN_ONLY=y), so there is no need to
reserve it.
Signed-off-by: Daniel Leung <daniel.leung@intel.com>
Xtensa ISA mentions that stack always needs to be aligned on
16 bytes. So we need to pad the stack frame to be also 16 bytes
aligned when dealing with interrupts. Or else the stack would
not be 16 bytes aligned when we add the stack frame to stack
during interrupt handling.
Signed-off-by: Daniel Leung <daniel.leung@intel.com>
When crossing stack during interrupt handling, we do two call4
to pivot to the interrupt stack, with arguments to these two
call4 stashed in A6, A10 and A11. However, A4-A11 may be marked
as invalid in the register file, and accessing them would
result in window overflowing. At that point, A0 and A1 are not
setup to handle window overflows, and will result in registers
being stashed in incorrect location, resulting in incorrect
value being restored during window underflowing. So move around
the code a bit to restore A0 and A1 properly before accessing
A4-A11.
Signed-off-by: Daniel Leung <daniel.leung@intel.com>
The logic to swap page tables or MPU entries is moved to the end
of cross stack call, since it is still running in the interrupt
stack instead of the thread stack. The old logic was calling
the swap functions in the outgoing thread stack, which is not
desirable.
Signed-off-by: Daniel Leung <daniel.leung@intel.com>
During arch_cohere_stacks(), the used portion of the outgoing
thread is cache flushed, and then the unused portion of cache
invalidated. However, this results in the cache line at
the stack pointer being flushed and then invalidated due to
how sys_cache_data_*() operates. If we are swapping back to
the same thread (e.g. after handling interrupt), this cache
line will need to be retrieved again from main memory since
it has already been invalidated. This creates unnecessary
data move between cache and main memory. So create our own
version of cache flushing and invalidation routines just for
arch_cohere_stacks(). Bouns is that these work directly with
bounding addresses and skips the size calculation which should
save a little bit amount of execution time.
Signed-off-by: Daniel Leung <daniel.leung@intel.com>
This removes _xstack_call1_* trampoline as we can simply use
callx4 to jump to the interrupt handler.
Signed-off-by: Daniel Leung <daniel.leung@intel.com>
Before cross stack call is setup correctly, we cannot allow
interrupts to be triggered or it may interfere with register
window spilling since we are clobbering registers needed for
that to work. However, there was a brief period where higher
level interrupts could fire due to code writing to PS with
lowered interrupt mask before raising it again. So rework
that part to avoid writing PS with intermediate value, and
now we mask interrupt until everything is setup correctly
before interrupt is enabled again.
Signed-off-by: Daniel Leung <daniel.leung@intel.com>
Syscall entrance makes use of ODD_REG_SAVE but it does not
really need to save FPU registers as it is technically
the same thread and same context. So extract call to
FPU_REG_SAVE to interrupt handling code.
Signed-off-by: Daniel Leung <daniel.leung@intel.com>
This adds ODD_REG_RESTORE as a counterpart to ODD_REG_SAVE.
Both the code in interrupt handling and syscall exit have
been refactored to use this new macro.
Signed-off-by: Daniel Leung <daniel.leung@intel.com>
This adds two parameters for ODD_REG_SAVE for scratch register
and BSA pointer, thus allowing a bit more flexibility on how
it can be called.
Signed-off-by: Daniel Leung <daniel.leung@intel.com>
Allow SoC to implement their custom per-core initialization function by
selecting `CONFIG_SOC_PER_CORE_INIT_HOOK` and implement
`soc_per_core_init_hook()`.
Signed-off-by: Maxim Adelman <imax@meta.com>
Signed-off-by: Yong Cong Sin <ycsin@meta.com>
Signed-off-by: Yong Cong Sin <yongcong.sin@gmail.com>
For code clarity, remove unnecessary `return` statements
in functions with a void return type they don't affect control flow.
Signed-off-by: Pisit Sawangvonganan <pisit@ndrsolution.com>
This adds a kconfig to enable making the interrupts
non-preemptible by other interrupts. Enabling this will set
the INTLEVEL to the max non-debug level before clearing
the EXCM bit.
Signed-off-by: Christopher J. Champagne <christopher.j.champagne@intel.com>
Signed-off-by: Daniel Leung <daniel.leung@intel.com>
