Add CONFIG_XTENSA_MMU_PTEVADDR_60000000 as a new PTEVADDR base address
choice in the Kconfig.
The PTEVADDR auto-refill mechanism places a self-referencing entry in
each memory domain's L1 page table at position:
(CONFIG_XTENSA_MMU_PTEVADDR >> 22) + ASID
With the default of 0x20000000 (L1 base position 128), ASIDs above 127
place the self-reference at L1 positions 256+, which on platforms like
intel_adsp collides with the uncached memory alias region at VA
0x40000000 (L1 position 256).
Add ASID_PTEVADDR_MAX = 256 - (PTEVADDR >> 22) - 1 and define
ASID_LAST_USER as MIN(XTENSA_MMU_SHARED_ASID - 1, ASID_PTEVADDR_MAX).
ASIDs beyond this limit are marked in-use during bitmap initialization.
Setting PTEVADDR to 0x60000000 (L1 base position 384) places all
possible self-reference entries at L1 positions 384-639, avoiding
collisions with both the uncached alias region (L1 256-383) and cached
SRAM (L1 640+).
For PTEVADDR 0x20000000: max usable ASID = 127.
For PTEVADDR 0x60000000: max usable ASID = 254 (no cap).
Signed-off-by: Jyri Sarha <jyri.sarha@linux.intel.com>
Previously, ASIDs were allocated monotonically and never reused. When a
memory domain was destroyed in arch_mem_domain_deinit(), its ASID was
lost. On platforms where short-lived memory domains are created
frequently (e.g. SOF DP tasks that each get their own domain), the
8-bit ASID space would eventually be exhausted.
Replace the monotonic counter with a bitmap-based allocator using
sys_bitarray covering the full 256-entry ASID space. Bit indices map
directly to ASID values with no translation: set bit (1) = in use or
reserved, clear bit (0) = free. Reserved ASIDs (0-3 and 255) are
marked in-use during initialization so they can never be allocated.
asid_alloc() uses sys_bitarray_alloc() to find a free ASID.
asid_free() releases the ASID and invalidates TLB autorefill entries
on all cores to prevent stale mappings from the previous holder.
Signed-off-by: Jyri Sarha <jyri.sarha@linux.intel.com>
When switching between threads in xtensa_switch, if incoming
and outgoing threads share the same memory domain, there is
no need to swap the page tables or MPU entries as they are
the same.
For MMU, CONFIG_XTENSA_MMU_FLUSH_AUTOREFILL_DTLBS_ON_SWAP must
be enabled to skip the swapping, as flushing of data TLBs is
quite expensive. Without the need to flush data TLBs, the page
table swapping is done via the macro SWAP_PAGE_TABLE which is
relatively lightweight. So there is no need to all the extra
checking code.
Signed-off-by: Daniel Leung <daniel.leung@intel.com>
The TLBI VAE1 address field is VA[55:12] regardless of the
translation granule (ARM ARM), but invalidate_tlb_page() shifted the
address by PAGE_SIZE_SHIFT. With 16 KB/64 KB pages the hardware then
invalidated a different entry and the intended page's translation
survived: paged-out frames stayed accessible through the stale TLB
entry, and the dirty-page write transition escalated into a fatal
fault loop.
Shift by 12 unconditionally, matching Linux's __TLBI_VADDR().
Fixes#115708
Signed-off-by: Hongquan Li <hongquan.li@processmission.com>
The fault handlers reach printk when LOG is disabled, and printk
has no level filtering, so the register dump and its format strings
are linked in unconditionally. Add EXCEPTION_DUMP, enabled by
default, so size constrained builds can compile the dump out.
The disabled variant keeps its arguments under an if (0) argument
checker, mirroring the logging subsystem, so format specifiers stay
type checked and argument side effects are preserved without
emitting unused variable warnings.
Assisted-by: Claude:opus-4-8
Signed-off-by: Sylvio Alves <sylvio.alves@espressif.com>
The round-robin ASID allocator in arch_mem_domain_init() could hand out
an ASID that was still in use by a live memory domain once the ASID
space wrapped. This broke TLB-based isolation because two live domains
would share the same ASID.
Fix by scanning domain_list before assigning an ASID and advancing the
round-robin counter until an unused ASID is found. If all ASIDs are in
use, return -ENOMEM.
Signed-off-by: Hongquan Li <hongquan.li@processmission.com>
arch_mem_domain_deinit() discards a domain's page tables but never
invalidated TLB entries tagged with that domain's ASID. ASIDs are
allocated round-robin (1..255), so after wrapping a new domain reuses
an old ASID. Stale TLB entries cached on any CPU that previously ran
threads in the old domain still match the reused ASID, allowing the
new domain's threads to access physical pages with the old domain's
permissions.
Add invalidate_tlb_all() after the page table teardown, matching the
pattern used by every other page-table mutation path (arch_mem_map,
arch_mem_unmap, private_map, etc.).
Fixes#113559
Signed-off-by: Hongquan Li <hongquan.li@processmission.com>
When a device using a second or third level interrupt makes a request to
set priority, the request should be routed via the multilevel interrupt
API. Otherwise, we will configure the interrupt priority for one of the
core interrupts on the ARC controller (which is usually not what the API
caller wants)
This occurs because the ARC interrupt priority register encodes
interrupts from 0-255, so a multilevel interrupt number would simply be
masked to the lowest 8 bits (which matches the interrupt number of that
device's parent controller)
Signed-off-by: Daniel DeGrasse <ddegrasse@tenstorrent.com>
Signed-off-by: Alex Apostolu <aapostolu@tenstorrent.com>
Update the arch_irq_enable/disable functions to support multilevel
interrupts on the ARC architecture.
Signed-off-by: Aaron Fong <afong@tenstorrent.com>
Signed-off-by: Alex Apostolu <apostolu240@gmail.com>
arch_buffer_validate() on ARC computes the end address of the
validated range using unprotected unsigned addition. When addr+size
wraps around the 32-bit address space the check can succeed for a
buffer that spans non-user-accessible memory, violating the
arch_buffer_validate() contract.
Add overflow detection using u32_add_overflow(), mirroring the fix
already applied to ARM32 (PR #23239) and Xtensa (PR #109000).
Handle size == 0 explicitly so that size - 1 does not underflow.
This affects ARC MPUv2/v3/v4/v6/v8 implementations.
Fixes#113832
Signed-off-by: Hongquan Li <hongquan.li@processmission.com>
We already determine whether the target is PMSAv8 or (PMSAv6/PMSAv7)
using the selected architecture profile (ARMv6-M/ARMv7-M or ARMv8-M
Baseline/Mainline) and define a symbol accordingly. Make use of that
symbol everywhere instead of repeating CPU checks all over arch code.
While at it, drop the allowlist which generated a build error when the
MPU was enabled on an "unknown" CPU. The header is only included from
SoC-specific code which knows whether an MPU exists or not, or generic
code gated behind CONFIG_ARM_MPU, which guarantees that the MPU exists
as CONFIG_ARM_MPU depends on CONFIG_CPU_HAS_ARM_MPU.
Note: this tangentially fixes a bug that misclassified ARMv8-R targets
as using the PMSAv6/PMSAv7 MPU. This bug did not seem to have any
adverse effect though...
Signed-off-by: Mathieu Choplain <mathieu.choplain-ext@st.com>
Xtensa architecture does not have native inter-processor
interrupts and must rely on mechanism on SoC or board to do
that. Because of this, the architecture should not select
CONFIG_ARCH_HAS_DIRECTED_IPIS and should let SoC or board
layer to do that, where CONFIG_SCHED_IPI_SUPPORTED is also
being selected.
Note that CONFIG_XTENSA_LAZY_HIFI_SHARING is now dependent
on CONFIG_ARCH_HAS_DIRECTED_IPIS as it uses directed IPI.
Due to that it keeps sending IPI under the other has saved
HIFI state, it is not really appopriate to broadcast IPIs.
That's why it can only be enabled when directed IPI is
supported.
Signed-off-by: Daniel Leung <daniel.leung@intel.com>
The Xtensa implementation validated the whole maxsize range with
arch_buffer_validate() before calling strnlen(), which breaks the
k_usermode_string_nlen() contract in two ways:
- With maxsize == 0 the string must not be touched at all, but the
validation probed the MPU/MMU region containing the pointer anyway
(alignment rounds a zero size up to a full region), wrongly
failing with -1 for inaccessible addresses.
- The validation required user mode accessibility, while the API
contract explicitly does not guarantee it: user access is checked
by the syscall marshalling layer via K_SYSCALL_MEMORY_READ() once
the length is known. Kernel mode callers measuring kernel strings
(e.g. a buffer on the kernel stack) were wrongly rejected.
Return early for maxsize == 0, and check kernel mode readability
with the new xtensa_buffer_is_kernel_readable() (MPU and MMU
variants) instead of arch_buffer_validate().
On the MMU side, extract a shared mem_buffer_validate() that takes a
ring parameter, so arch_buffer_validate() and the new
xtensa_buffer_is_kernel_readable() share the same logic without
duplication.
Fixeszephyrproject-rtos/zephyr#113722
Signed-off-by: Hongquan Li <hongquan.li@processmission.com>
The loop read s[counter] before checking whether maxsize bytes had
been examined (lp_count was set to maxsize + 1 and tested only after
the load), so it read s[maxsize] when the first maxsize bytes were
all non-NUL and dereferenced s[0] when maxsize == 0, violating the
strnlen() semantics required by arch_user_string_nlen(). The
over-read byte cannot change the returned length, but if it is not
accessible the exception fixup reports a spurious error and callers
reject valid input with EFAULT.
Set lp_count to maxsize and check it before the load instead.
Verified by build and disassembly for qemu_arc/qemu_arc_em; no ARC
simulator is available locally, runtime coverage is expected from
the syscalls suite on nsim platforms in CI.
Fixeszephyrproject-rtos/zephyr#113722
Signed-off-by: Hongquan Li <hongquan.li@processmission.com>
Both the ia32 and intel64 implementations loaded s[counter] before
checking counter == maxsize, so they read s[maxsize] when the first
maxsize bytes were all non-NUL and dereferenced s[0] when
maxsize == 0, violating the strnlen() semantics required by
arch_user_string_nlen(). The over-read byte cannot change the
returned length, but if it is not accessible the exception fixup
reports a spurious error and callers reject valid input with EFAULT.
Check the limit before the load instead.
Fixeszephyrproject-rtos/zephyr#113722
Signed-off-by: Hongquan Li <hongquan.li@processmission.com>
The loop loaded s[counter] before checking counter == maxsize, so it
read s[maxsize] when the first maxsize bytes were all non-NUL and
dereferenced s[0] when maxsize == 0, violating the strnlen()
semantics required by arch_user_string_nlen(). The over-read byte
cannot change the returned length, but if it is not accessible the
exception fixup reports a spurious error and callers reject valid
input with EFAULT.
Check the limit before the load instead.
Fixeszephyrproject-rtos/zephyr#113722
Signed-off-by: Hongquan Li <hongquan.li@processmission.com>
The loop loaded s[counter] before checking counter == maxsize, so it
read s[maxsize] when the first maxsize bytes were all non-NUL and
dereferenced s[0] when maxsize == 0, violating the strnlen()
semantics required by arch_user_string_nlen(). The over-read byte
cannot change the returned length, but if it is not accessible the
exception fixup reports a spurious error and callers reject valid
input with EFAULT.
Check the limit before the load instead.
Fixeszephyrproject-rtos/zephyr#113722
Signed-off-by: Hongquan Li <hongquan.li@processmission.com>
This reverts commit c18885e95f:
the issue that this commit worked around was fixed in the
previous commit by preserving the original stack alignment.
Signed-off-by: Laurie Fay <Laurie.Fay@arm.com>
Preserve the alignment indication and rebuild the hardware frame with its
padding word to ensure the stack is always 8-byte aligned when returning
from exception: this is required on Armv8-M.
When an exception is taken with a stack pointer that is not 8-byte
aligned, Armv8-M, and Armv7-M when configured, adds a 4-byte padding word
to align the exception frame. Bit 9 of xPSR records the presence of this
padding.
However, arm_m_switch_to_cpu() does not restore the padding word when
rebuilding the hardware frame. The existing workaround clears the xPSR
alignment bit, which is sufficient on Armv7-M but is invalid on Armv8-M.
Signed-off-by: Laurie Fay <Laurie.Fay@arm.com>
Declare dummy as a read-write output operand, to reflect how it
is used in the assembly block.
The inline assembly overwrites dummy before reading it, but declares the
operand as input-only. The compiler may therefore assume that the backing
register is unchanged and reuse it for another live value, even though the
assembly clobbers it.
Signed-off-by: Laurie Fay <Laurie.Fay@arm.com>
arch_timing_cycles_to_ns() multiplied before it divided, so the
intermediate cycles * NSEC_PER_SEC left the 64-bit range at 1.8e10
cycles. On a 3.2 GHz TSC that is under six seconds of uptime, after
which the returned time wraps back towards zero.
Anything sampling this clock for longer than that sees time jump
backwards. It shows up plainly in CTF tracing, where a capture longer
than a few seconds is rejected outright by babeltrace2: a stream whose
timestamps are not monotonic cannot be read, and per-CPU streams
cannot be merged.
Divide first and fold the remainder back in. The result is exact and
cannot overflow for any counter frequency below roughly 18 GHz, since
the remainder is by construction smaller than the frequency.
Assisted-by: Claude:claude-opus-5
Signed-off-by: Anas Nashif <anas.nashif@intel.com>
Add a Cortex-M perf backend that samples interrupted context from the
SysTick exception frame.
Install a SysTick wrapper when stack-sampling perf is enabled and
capture the interrupted stack pointer, EXC_RETURN, and r7 on exception
entry. Validate the complete hardware-stacked frame, including optional
alignment padding and extended FP state, against the Thread-mode or
Handler-mode stack before unwinding with arch_stack_walk().
The Arm stack walker uses EHABI unwind tables, so the backend does not
require CONFIG_FRAME_POINTER. Preserve r7 because EHABI unwind recipes
may use it to recover VSP even when frame pointers are not enabled
globally.
Return -EAGAIN when the interrupted context is invalid or unavailable
so the perf core discards that sample without reporting buffer
exhaustion.
Limit the backend to uniprocessor configurations because its captured
sample state is global and remote SMP sampling does not enter the
SysTick wrapper.
Disable the backend for Non-secure Trusted Execution images. A
Non-secure SysTick can be taken while the core executes Secure code,
but Non-secure firmware cannot access the Secure exception frame.
The stack pointer read by the wrapper instead refers to the suspended
Non-secure context, so it cannot be unwound as the interrupted frame.
Sampling across Security states is unsupported.
Signed-off-by: Sudan Landge <sudan.landge@arm.com>
Cortex-M exception frames can be stacked on MSP when execution is
interrupted in handler mode. The Arm EHABI stack walker currently seeds
VSP from the saved PSP unconditionally, which only works for frames
stacked on PSP.
Select the exception frame stack pointer from EXC_RETURN.SPSEL so
MSP-backed frames seed the unwinder from extra_info.msp, while preserving
the existing PSP path. This also allows perf sampling backends to unwind
handler-mode call chains captured on MSP.
Signed-off-by: Sudan Landge <sudan.landge@arm.com>
Otherwise, "warning: implicit declaration of function 'dump_fault'" is
generated, which may be promoted to an error when also using
CONFIG_COMPILER_WARNINGS_AS_ERRORS=y.
Some variables may be unused when the dump_fault isn't called, so mark them
as unused to avoid warnings.
Signed-off-by: Phil Hindman <phindman@xes-inc.com>
Commit 2222fa1426 removed all callers of
the function arm_core_mpu_mem_partition_config_update()... but not the
function itself, which became dead code lingering in tree since then...
Get rid of this function - it has served no purpose for long enough.
Signed-off-by: Mathieu Choplain <mathieu.choplain-ext@st.com>
Clear the fpu_recently_used flag when disabling FPU access for a
thread. This flag is used by the context switch path to decide whether
to re-enable FPU access; leaving it set caused k_float_disable() to be
silently reverted on the next context switch back to the thread.
Fixes#113645
Signed-off-by: Hongquan Li <hongquan.li@processmission.com>
Set K_FP_REGS in z_riscv_fpu_load() when a thread becomes the FPU
owner, and clear it in arch_float_disable(). This keeps user_options
consistent across lazy-FPU architectures rather than only clearing
the flag.
Fixes#113645
Signed-off-by: Hongquan Li <hongquan.li@processmission.com>
Introduce z_riscv_fpu_flush_thread() to perform the
architecture-specific FPU context flush without touching thread
options or fpu_recently_used, and make arch_float_disable() use it.
Reject NULL thread pointers with -EINVAL in arch_float_disable().
Unlike Cortex-M, RISC-V does not restrict the call to the current
thread or exclude ISR context.
Switch arch_spin_relax() to the new helper: it only needs to flush
the current CPU FPU context in response to a pending IPI, so it must
not go through arch_float_disable(), which now rejects NULL pointers
while _current_cpu->arch.fpu_owner may legitimately be NULL.
Fixes#113645
Signed-off-by: Hongquan Li <hongquan.li@processmission.com>
Rename pm_s2ram.S to pm_s2ram_asm.S and update the
Cortex-M source list to reference the new filename.
This avoids generating colliding object paths for the
C and assembly suspend-to-RAM sources when both are
built together.
Assisted-by: GitHub Copilot:GPT-5.4
Signed-off-by: Evan Chen <bugena123@gmail.com>
Code using this header is only built when CONFIG_ARM_MPU=y so it isn't
necessary to gate the header's declarations behind that option.
Signed-off-by: Mathieu Choplain <mathieu.choplain-ext@st.com>
`arm_core_mpu_dev.h` is the header describing the API that must be
implemented by drivers compatible with the core. Some functions were
were however accessed using an extern declaration inside the ARM MPU
Core module instead of a declaration in the API header.
Move the offending functions to the API header, making the extern
declaration unnecessary. While at it, also make `z_arm_mpu_init()`
part of the header because it is required, even if not consumed by
the ARM MPU Core module itself.
Signed-off-by: Mathieu Choplain <mathieu.choplain-ext@st.com>
XN on device-type regions only protects mapped regions; unmapped holes
in the MPU map remain reachable through the PRIVDEFENA background map,
both by Cortex-M7 speculative instruction fetches (Cortex-M7 TRM) and
by PLD linefills to faulting addresses (erratum 1013783, SDEN-1068427,
all M7 revisions).
Add opt-in CONFIG_ARM_MPU_CM7_UNMAPPED_REGION to program region 0 as a
4GB Strongly-ordered, no-access, Execute-Never catch-all, as
recommended by the erratum workaround; static regions start from
region 1 and take precedence. Opt-in because the catch-all defeats the
PRIVDEFENA background map: the static MPU region table must explicitly
cover all memory the firmware uses.
Signed-off-by: Lucien Zhao <lucien.zhao@nxp.com>
Header file include/zephyr/sys_clock.h is deprecated and will be removed
someday. Update the whole file tree to include zephyr/sys/clock.h
straight instead of zephyr/sys_clock.h.
This change was made running the sed shell command below:
$ sed -i 's/zephyr\/sys_clock\.h/zephyr\/sys\/clock\.h/' \
`grep -rsl "zephyr/sys_clock\.h" arch/`
Signed-off-by: Etienne Carriere <etienne.carriere@st.com>
Add support for stacktrace of threads. The shell command
"kernel unwind <thread_ptr>" now works.
Signed-off-by: Mike J. Chen <mjchen@google.com>
Co-authored-by: Filip Kokosinski <fkokosinski@antmicro.com>
Signed-off-by: Filip Kokosinski <fkokosinski@antmicro.com>
Mapping large memory regions at the soc level will consume too
many l2 entries, adding l1 section mapping functionality to
reduce excessive use of l2 entries.
Signed-off-by: CHEN Xing <xing.chen@microchip.com>
Select ARCH_SUPPORTS_COREDUMP_THREADS and ARCH_SUPPORTS_COREDUMP_STACK_PTR
for CPU_CORTEX_A in arch/arm64/core/Kconfig, enabling MEMORY_DUMP_THREADS
coredump mode on ARM64. Mirrors the existing Cortex-M declarations in
arch/Kconfig.
Implement arch_coredump_stack_ptr_get() using thread->callee_saved.sp_elx,
the EL1 stack pointer saved by the context-switch path for sleeping
threads. For the faulting thread the saved context is stale, so the
coredump falls back to dumping the full stack allocation,
which is always correct.
Signed-off-by: Appana Durga Kedareswara rao <appana.durga.kedareswara.rao@amd.com>
arm_cpu_boot_params stack pointer fields used pointer arithmetic on
two-dimensional K_KERNEL_STACK_ARRAY_DECLARE arrays before casting to
char *, producing invalid initialized pointer values.
Use K_KERNEL_STACK_BUFFER() on the CPU 0 stack elements instead,
matching the runtime initialization in arch_cpu_start().
Fixes#113125
Signed-off-by: Hongquan Li <hongquan.li@processmission.com>
In `movw_reloc_handler()`, the MOVZ/MOVN opcode selection for signed
AArch64 MOVW relocations was inverted: positive relocation values were
encoded with MOVN (negate) and negative values with MOVZ (zero), causing
all signed MOVW relocations to produce wrong immediate values.
Swap the two branches so that `x >= 0` selects MOVZ and `x < 0` selects
MOVN with immediate inversion, matching the AArch64 instruction semantics
where MOVZ zero-extends and MOVN negates the immediate.
Fixeszephyrproject-rtos/zephyr#113383.
Signed-off-by: Hongquan Li <hongquan.li@processmission.com>
When __arch_mem_map() fails to map a page, arch_mem_map() should
panic instead of continuing mapping the remaining pages.
Otherwise, there would be hole in the memory map which results
in confusing errors.
Signed-off-by: Daniel Leung <daniel.leung@intel.com>
Instead of every new xtensa core having to modify xtensa_backtrace.c
and provide a header file with custom functions needed by the common
xtensa backtrace functions to validate stack and code addresses, now
xtensa_backtrace.c uses default weak functions that the soc can replace
with their own implementations.
Modify the current in-tree instances that used to have custom helper
functions to provide the standardize weak replacements.
Signed-off-by: Mike J. Chen <mjchen@google.com>
Co-authored-by: Filip Kokosinski <fkokosinski@antmicro.com>
Signed-off-by: Filip Kokosinski <fkokosinski@antmicro.com>
This PR migrates the MIT-licensed Xen public headers into the
`include/zephyr/xen/public` directory to the zephyr-xenlib module.
Since the xen public headers require GCC extensions,
enable GNU_C_EXTENSIONS along with enabling CONFIG_XEN.
Signed-off-by: TOKITA Hiroshi <tokita.hiroshi@gmail.com>
Instead of defaulting to 0 in the absence of a devicetree value, set
cache line size to the arch bit width which is a good proxy for cache
line size, and it's safer to use in code.
This default will also eliminate the need for many redundant checks
across the tree of CACHE_LINE_SIZE's value.
Signed-off-by: Abderrahmane JARMOUNI <git@jarmouni.me>
z_main_thread is only needed if multithreading is enabled, so move its
definition inside a multithreading check.
Signed-off-by: Josh DeWitt <josh.dewitt@garmin.com>
Commit f5a1585426 removed the extra
pointers for the IAR toolchain tls pointer setup. Update the size
returned from arch_tls_stack_setup to keep in sync with the tls pointer.
Signed-off-by: Josh DeWitt <josh.dewitt@garmin.com>
Store SBI_ERR_NOT_SUPPORTED in the saved a0 slot for unsupported
SRST functions. The previous store used the saved t2 offset.
S-mode callers kept the old a0 value, and t2 was restored as -1.
Fixes#113149.
Signed-off-by: Hongquan Li <hongquan.li@processmission.com>
The RISC-V coredump hook arch_coredump_priv_stack_dump() derives the
privilege stack region from thread->arch.priv_stack_start. That field
is only initialized when a thread enters user mode via
arch_user_mode_enter(). For a K_USER thread that has been created but
has never run, priv_stack_start is still 0, causing the hook to dump a
low-address region near 0 and potentially trigger a nested memory
fault.
Add an early return when priv_stack_start is 0 so that unstarted user
threads are skipped.
Fixes#113875
Signed-off-by: Hongquan Li <hongquan.li@processmission.com>
arc_vpx_sem_init() initialized vpx_sem[0] once per CPU instead of
initializing each CPU's semaphore: the loop passed the array (decaying
to &vpx_sem[0]) to k_sem_init() rather than the i-th element, leaving
the semaphores of CPUs 1..N-1 uninitialized on multi-core
configurations.
Signed-off-by: Anas Nashif <anas.nashif@intel.com>
