orr fix is as reported in review:
```
The add causes a crash with IAR tools as the address loaded to r8
already has the lowest bit set, and the add causes it to be set to ARM
mode. The orr instruction works fine with both scenarios
```
`UDF 0` seems to break on IAR but `UDF #0` works for all.
Signed-off-by: Sudan Landge <sudan.landge@arm.com>
USE_SWITCH code unconditionally applied interrupt locking, which altered
BASEPRI handling and broke expected interrupt behavior on both
Baseline and Mainline CPUs when USE_SWITCH was disabled.
This commit restores the original behavior with USE_SWITCH disabled and
fixes tests/arch/arm/arm_interrupt failures.
Signed-off-by: Sudan Landge <sudan.landge@arm.com>
The ARM Ltd. FVP emulator (at least the variants run in Zephyr CI)
appears to have a bug with the stack alignment bit in xPSR. It's
common (it fails in the first 4-6 timer interrupts in
tests.syscalls.timeslicing) that we'll take an interrupt from a
seemingly aligned (!) stack with the bit set. If we then switch and
resume the thread from a different context later, popping the stack
goes wrong (more so than just a misalignment of four bytes: I usually
see it too low by 20 bytes) in a way that it doesn't if we return
synchronously. Presumably legacy PendSV didn't see this because it
used the unmodified exception frame.
Work around this by simply assuming all interrupted stacks were
aligned and clearing the bit. That is NOT correct in the general
case, but in practice it's enough to get tests to pass.
Signed-off-by: Andy Ross <andyross@google.com>
The exit from the SVC exception used for syscalls back into the
calling thread is done without locking. This means that the
intermediate states can be interrupted while the kernel-mode code is
still managing thread state like the mode bit, leading to mismatches.
This seems mostly robust when used with PendSV (though I'm a little
dubious), but the new arch_switch() code needs to be able to suspend
such an interrupted thread and restore it without going through a full
interrupt entry/exit again, so it needs locking for sure.
Take the lock unconditionally before exiting the call, and release it
in the thread once the magic is finished, just before calling the
handler. Then take it again before swapping stacks and dropping
privilege.
Even then there is a one-cycle race where the interrupted thread has
dropped the lock but still has privilege (the nPRIV bit is clear in
CONTROL). This thread will be resumed later WITHOUT privilege, which
means that trying to set CONTROL will fail. So there's detection of
this 1-instruction race that will skip over it.
Signed-off-by: Andy Ross <andyross@google.com>
Some toolchains don't support an __asm__(...) block at the top level
of a file and require that they live within function scope. That's
not a hardship as these two blocks were defining callable functions
anyway. Exploit the "naked" attribute to avoid wasted bytes in unused
entry/exit code.
Signed-off-by: Andy Ross <andyross@google.com>
Late-arriving clang-format-demanded changes that are too hard to split
and squash into the original patches. No behavior changes.
Signed-off-by: Andy Ross <andyross@google.com>
Some nitpicky hand-optimizations, no logic changes:
+ Shrink the assembly entry to put more of the logic into
compiler-optimizable C.
+ Split arm_m_must_switch() into two functions so that the first
doesn't look so big to the compiler. That allows it to spill (many)
fewer register on entry and speeds the (very) common early-exit case
where an interrupt returns without context switch.
Signed-off-by: Andy Ross <andyross@google.com>
When USE_SWITCH=y, the thread struct is now mostly degenerate. Only
the two words for ICI/IT state tracking are required. Eliminate all
the extra fields when not needed and save a bunch of SRAM.
Note a handful of spots in coredump/debug that need a location for the
new stack pointer (stored as the switch handle now) are also updated.
Signed-off-by: Andy Ross <andyross@google.com>
The new switch code no longer needs PendSV, but still calls the SVC
vector. Split them into separate files for hygiene and a few
microseconds of build time.
Signed-off-by: Andy Ross <andyross@google.com>
Micro-optimization: We don't need a full arch_irq_lock(), which is a
~6-instruction sequence on Cortex M. The lock will be dropped
unconditionally on interrupt exit, so take it unconditionally.
Signed-off-by: Andy Ross <andyross@google.com>
z_get_next_switch_handle() is a clean API, but implementing it as a
(comparatively large) callable function requires significant
entry/exit boilerplate and hides the very common "no switch needed"
early exit condition from the enclosing C code that calls it. (Most
architectures call this from assembly though and don't notice).
Provide an unwrapped version for the specific needs non-SMP builds.
It's compatible in all other ways.
Slightly ugly, but the gains are significant (like a dozen cycles or
so).
Signed-off-by: Andy Ross <andyross@google.com>
GCC/gas has a code generation bugglet on thumb. The R7 register is
the ABI-defined frame pointer, though it's usually unused in zephyr
due to -fomit-frame-pointer (and the fact the DWARF on ARM doesn't
really need it). But when it IS enabled, which sometimes seems to
happen due to toolchain internals, GCC is unable to allow its use in
the clobber list of an asm() block (I guess it can't generate
spill/fill code without using the frame?).
There is existing protection for this problem that sets
-fomit-frame-pointer unconditionally on the two files (sched.c and
init.c) that require it. But even with that, gcc sometimes gets
kicked back into "framed mode" due to internal state. Provide a
kconfig workaround that does an explicit spill/fill on the one
test/platform where we have trouble.
(I checked, btw: an ARM clang build appears not to have this
misfeature)
Signed-off-by: Andy Ross <andyross@google.com>
ARM Cortex M has what amounts to a design bug. The architecture
inherits several unpipelined/microcoded "ICI/IT" instruction forms
that take many cycles to complete (LDM/STM and the Thumb "IT"
conditional frame are the big ones). But out of a desire to minimize
interrupt latency, the CPU is allowed to halt and resume these
instructions mid-flight while they are partially completed. The
relevant bits of state are stored in the EPSR fields of the xPSR
register (see ARMv7-M manual B1.4.2). But (and this is the design
bug) those bits CANNOT BE WRITTEN BY SOFTWARE. They can only be
modified by exception return.
This means that if a Zephyr thread takes an interrupt
mid-ICI/IT-instruction, then switches to another thread on exit, and
then that thread is resumed by a cooperative switch and not an
interrupt, the instruction will lose the state and restart from
scratch. For LDM/STM that's generally idempotent for memory (but not
MMIO!), but for IT that means that the restart will re-execute
arbitrary instructions that may not be idempotent (e.g. "addeq r0, r0,
The fix is to check for this condition (which is very rare) on
interrupt exit when we are switching, and if we discover we've
interrupted such an instruction we swap the return address with a
trampoline that uses a UDF instruction to immediately trap to the
undefined instruction handler, which then recognizes the fixup address
as special and immediately returns back into the thread with the
correct EPSR value and resume PC (which have been stashed in the
thread struct). The overhead for the normal case is just a few cycles
for the test.
Signed-off-by: Andy Ross <andyross@google.com>
Integrate the new context layer, allowing it to be selected via the
pre-existing CONFIG_USE_SWITCH. Not a lot of changes, but notable
ones:
+ There was code in the MPU layer to adjust PSP on exception exit at a
stack overflow so that it remained inside the defined stack bounds.
With the new context layer though, exception exit will rewrite the
stack frame in a larger format, and needs PSP to be adjusted to make
room.
+ There was no such treatment in the PSPLIM case (the hardware prents
the SP from going that low), so I had to add similar code to
validate PSP at exit from fault handling.
+ The various return paths for fault/svc assembly handlers need to
call out to the switch code to do the needed scheduler work. Really
almost all of these can be replaced with C now, only userspace
syscall entry (which has to "return" into the privileged stack)
needs special treatment.
+ There is a gcc bug that prevents the arch_switch() inline assembly
from building when frame pointers are enabled (which they almost
never are on ARM): it disallows you from touching r7 (the thumb
frame pointer) entirely. But it's a context switch, we need to!
Worked around by enforcing -fomit-frame-pointer even in the two
scheduler files that can swap when NO_OPTIMIZATIONS=y.
Signed-off-by: Andy Ross <andyross@google.com>
Signed-off-by: Sudan Landge <sudan.landge@arm.com>
1. Mostly complete. Supports MPU, userspace, PSPLIM-based stack
guards, and FPU/DSP features. ARMv8-M secure mode "should" work but I
don't know how to test it.
2. Designed with an eye to uncompromising/best-in-industry cooperative
context switch performance. No PendSV exception nor hardware
stacking/unstacking, just a traditional "musical chairs" switch.
Context gets saved on process stacks only instead of split between
there and the thread struct. No branches in the core integer switch
code (and just one in the FPU bits that can't be avoided).
3. Minimal assembly use; arch_switch() itself is ALWAYS_INLINE, there
is an assembly stub for exception exit, and that's it beyond one/two
instruction inlines elsewhere.
4. Selectable at build time, interoperable with existing code. Just
use the pre-existing CONFIG_USE_SWITCH=y flag to enable it. Or turn
it off to evade regressions as this stabilizes.
5. Exception/interrupt returns in the common case need only a single C
function to be called at the tail, and then return naturally.
Effectively "all interrupts are direct now". This isn't a benefit
currently because the existing stubs haven't been removed (see #4),
but in the long term we can look at exploiting this. The boilerplate
previously required is now (mostly) empty.
6. No support for ARMv6 (Cortex M0 et. al.) thumb code. The expanded
instruction encodings in ARMv7 are a big (big) win, so the older cores
really need a separate port to avoid impacting newer hardware.
Thankfully there isn't that much code to port (see #3), so this should
be doable.
Signed-off-by: Andy Ross <andyross@google.com>
This was just a pedantic setting. I mean, of course it makes no sense
to have thread FPU management state features built when you aren't
including the scheduler in the build.
...unless you want to unit-test the context switch code without
tripping over itself on the way into the test code. In fact lots of
unit testing of low level primitives can be done with
MULTITHREADING=n.
Remove the dependency. It isn't actually doing anything useful.
Signed-off-by: Andy Ross <andyross@google.com>
ISR_TABLES_LOCAL_DECLARATION depends on GEN_IRQ_VECTOR_TABLE but
this is not enforced in Kconfig.
Building without the GEN_IRQ_VECTOR_TABLE and with LOCAL_DECLARATION
will produce the following misleading static assertion error:
"CONFIG_IRQ_VECTOR_TABLE_JUMP_BY_{ADDRESS,CODE} not set"
As the LOCAL_DECLARATION macros expect GEN_IRQ_VECTOR_TABLE to be
enabled. LOCAL_DECLARATION also depends on GEN_ISR_TABLES but that
is a dependency of GEN_IRQ_VECTOR_TABLE already.
Signed-off-by: Bjarki Arge Andreasen <bjarki.andreasen@nordicsemi.no>
The native_sim uses the pthread stack instead of the Zephyr allocated ones.
This adds CONFIG_ARCH_POSIX_UPDATE_STACK_INFO to the posix arch to make the
real stack bounds available in thread info. (CircuitPython uses this to do
it's own stack overflow checking and recovery.)
The original stack values are restored on abort for backwards
compatibility with CMSIS v1.
Signed-off-by: Scott Shawcroft <scott@adafruit.com>
Signed-off-by: Alberto Escolar Piedras <alberto.escolar.piedras@nordicsemi.no>
Using only the fence instruction to gate the management of data in cache
is insufficient to prevent unordered access after flushing in some cases.
Gate dcache instructions like icache instructions.
Signed-off-by: Camille BAUD <mail@massdriver.space>
Static MMU region entries populated via
MMU_REGION_DT_COMPAT_FOREACH_FLAT_ENTRY() pass raw DTS reg address and
size values to __add_map(), which asserts page-alignment. DTS nodes may
legitimately have non-page-aligned reg sizes reflecting actual hardware
register footprints, causing an assert crash during early boot when
CONFIG_ASSERT=y.
Align the base address down and size up to CONFIG_MMU_PAGE_SIZE in
add_arm_mmu_region(), mirroring the k_mem_region_align() logic already
used by the dynamic DEVICE_MMIO_MAP path in kernel/mmu.c. This ensures
all static platform MMU region entries are mapped with page-granular
parameters regardless of DTS reg values.
Signed-off-by: Appana Durga Kedareswara rao <appana.durga.kedareswara.rao@amd.com>
Setting SCR_ST_BIT actually traps CNTPS access to EL3, opposite
to what the comment says. Remove to allow secure EL1 access.
Also initialize CNTPS_CVAL_EL1 to prevent spurious interrupts.
Signed-off-by: Joakim Tjernlund <joakim.tjernlund@infinera.com>
Co-authored-by: Sudan Landge <sudan.landge@arm.com>
In addition to pool literal, we want to avoid jump tables generally
associated to Table Branch Byte (TBB) and Table Branch Halfword (TBH)
instructions.
Signed-off-by: Jérôme Pouiller <jerome.pouiller@silabs.com>
In addition to -mslow-flash-data, we must also ensure that the assembler
does not generate literal pools. They are automatically generated by the
LDR pseudo-instruction[1]:
- If the constant can be constructed with a MOV or MVN instruction, the
assembler emits the corresponding instruction.
- Otherwise (when the value does not fit on 16bits), the assembler places
the value in the next literal pool.
No options was found in GNU assembler to disable literal pool generation.
Therefore, this patch explicitly uses MOVT and MOVW when the assembler
would otherwise generate literal pool. Note, that LDR must be kept under
ifdef since Cortex-M0 does not support MOVT/ MOVW.
This patch only change four occurrences of LDR. The other occurrences do
not appear to generate literal pool (likely because the literal values are
< 0xFFFF). If a literal pool is generated in the future, it will introduce
a performance penalty. No other limitations are expected.
[1]: https://developer.arm.com/documentation/dui0204/f/ \
writing-arm-assembly-language/loading-constants-into-registers/ \
loading-with-ldr-rd---const?lang=en
Signed-off-by: Jérôme Pouiller <jerome.pouiller@silabs.com>
On some SoC, no data cache is associated with the main flash. Therefore,
all accesses to data stored in flash, especially literal pools[1] penalizes
performance. Fortunately, GCC and IAR provide options (-mslow-flash-data
and --no_literal_pool) to prevent the generation of literal pools.
Unfortunately, current GCC versions (14.x) do not support -mslow-flash-data
when Thread Local Storage (TLS) variables are used. A patch is currently
under review[2][3] to address this limitation. Without this gcc patch,
using -mslow-flash-data is not very user friendly. The user must rebuild
the libc (CONFIG_PICOLIBC_USE_MODULE=y) without TLS support
(CONFIG_THREAD_LOCAL_STORAGE=n), and must ensure that the application does
not rely on thread-safe "errno".
Because of these interactions with the compiler, this option can't be
automatically selected by the SoC. Thus, this patch leaves the option
hidden. The SoC may expose it if relevant.
[1]: https://en.wikipedia.org/wiki/Literal_pool
[2]: https://gcc.gnu.org/pipermail/gcc-patches/2026-February/707887.html
[3]: https://github.com/zephyrproject-rtos/gcc/pull/65
Signed-off-by: Jérôme Pouiller <jerome.pouiller@silabs.com>
Implement arch_mem_domain_deinit() for ARM64 to release page tables
back to the pool when a memory domain is de-initialized. This reuses
the existing discard_table() mechanism to recursively free all
sub-tables in the hierarchy.
Signed-off-by: Nicolas Pitre <npitre@baylibre.com>
gdb cannot unwind the stack from exceptions. This adds
CFI annotations to help gdb unwind.
Signed-off-by: Joakim Tjernlund <joakim.tjernlund@infinera.com>
Instead of using __ASSERT() with an empty string as message,
simply convert it to use __ASSERT_NO_MSG().
Signed-off-by: Daniel Leung <daniel.leung@intel.com>
The MAX32 RV32 core does not implement the fence instruction used by the
RISC-V synchronization intrinsic, so don't enable the builtin barriers for
that target.
Signed-off-by: Pete Johanson <pete.johanson@analog.com>
This supports de-initialization of memory domains to release
allocated page tables back to the pool.
Signed-off-by: Daniel Leung <daniel.leung@intel.com>
When a L2 table is no longer being used, we should set all PTEs
in the table to be illegal PTEs. This is simply a precautious
so that any stray references to the L2 table would not result
in incorrect permissions being applied.
Signed-off-by: Daniel Leung <daniel.leung@intel.com>
The ifdef guard in isr.S was written without the CONFIG_ prefix,
making the mtval fallback path dead code on all platforms including
QEMU (which previously worked via CONFIG_QEMU_TARGET).
Signed-off-by: William Markezana <william.markezana@gmail.com>
zeroing CNTHCTL_EL2 traps physical timer/counter access from EL1 to EL2,
but Zephyr has no hypervisor to handle those traps.
Enabling access is the standard EL2→EL1 drop behavior.
Signed-off-by: Joakim Tjernlund <joakim.tjernlund@infinera.com>
Select ARCH_SUPPORTS_COREDUMP_THREADS (if !SMP) and
ARCH_SUPPORTS_COREDUMP_STACK_PTR for RISC-V, and implement
arch_coredump_stack_ptr_get().
This enables CONFIG_DEBUG_COREDUMP_MEMORY_DUMP_THREADS and
CONFIG_DEBUG_COREDUMP_THREAD_STACK_TOP.
For non-current threads, return thread->callee_saved.sp.
For the faulting current thread in stack-top mode, return the
exception-time SP from z_riscv_get_sp_before_exc() (cached during
arch_coredump_info_dump()) instead of thread->callee_saved.sp,
which reflects switch-time state.
Signed-off-by: Mirai SHINJO <oss@mshinjo.com>
Expand the RISC-V coredump register block to all 33 GDB registers
(x0-x31, pc) in register-number order.
Previously only 18 registers were serialized. Populate zero, sp, gp,
tp, s0, and s1-s11 (when available).
Bump ARCH_HDR_VER from 1 to 3 (RISC-V 32-bit layout) and from 2 to 4
(RISC-V 64-bit layout) for the new wire format.
Keep the RISC-V 32-bit block fixed at 33 fields on the RISC-V RV32E
profile; registers not implemented by RV32E remain zero-filled so
version 3 always has a stable size.
Signed-off-by: Mirai SHINJO <oss@mshinjo.com>
When handling an ISR (which does not have a context from which to
restore its own value of LCOUNT), we must clear LCOUNT to prevents
incorrect zero-overhead execution if calling a function such as
memmove() which could be implemented using zero-overhead loop.
A function such as memmove() implemented using zero-overhead loop
assumes LCOUNT to have properly been setup before being called; but
an ISR calling memmove() in assembly, will likely not know that.
Signed-off-by: William Tambe <williamt@cadence.com>
Add CONFIG_PMP_UNLOCK_ROM_FOR_DEBUG option to conditionally disable
the lock bit (L=0) for the ROM region PMP entry. This allows debuggers
running in machine mode to access ROM for setting breakpoints and
reading instructions while preserving userspace protection.
When PMP lock bits are set, they restrict access even in machine mode,
causing "unable to halt hart" errors with hardware debuggers like
OpenOCD. This option provides a surgical fix that only affects the ROM
region - NULL pointer guards and stack guards remain locked to catch
critical bugs during development.
The option integrates with existing PMP_NO_LOCK_GLOBAL configuration
using nested COND_CODE_1 macros and defaults to disabled for production
builds.
Fixes: zephyrproject-rtos/zephyr#82729
Signed-off-by: Alex Lyrakis <alex_gfd@hotmail.com>
This reverts commit 8c02dde437.
For some unknown reasons, xt-clang emits two copies of
z_xt_init_pc if xtensa_mmu_init_paging() is in the same file
as xtensa_mmu_init() and xtensa_mmu_reinit(). So had to
revert the change.
Fixes#103055
Signed-off-by: Daniel Leung <daniel.leung@intel.com>
When using LLEXT, instruction TLB multi-hit becomes a reality
as the same memory space can be occupied by different modules
with different permissions. The ITLB cache may still contain
entries of the unloaded module. So we need to manually
invalidate any cached ITLB corresponding to the exception
address so the TLB associated with the newly loaded module
can be used.
Signed-off-by: Daniel Leung <daniel.leung@intel.com>
There is no need to invalidate the whole auto-refilled data TLB
cache when DTLB multi-hit exception is raised. Now it only
invalidates the TLB entries corresponding to the one causing
the DTLB multi-hit. This allows other non-related TLB entries
to remain in the cache so they don't need to be reloaded.
Signed-off-by: Daniel Leung <daniel.leung@intel.com>
For both data TLB multi-hit and load/store ring error, we should
return to the interrupted thread immediately so that it can get
past the exception generated code. It is because both of these
exceptions are the result of having cached TLB entries not
aligning to the correct access pattern. So once we have handled
the exception, go back to the interrupted thread to continue
to minimize the chance of having another incompatible TLB being
cached.
Signed-off-by: Daniel Leung <daniel.leung@intel.com>
There is no need for an extra switch block to manipulate
the value of is_fatal_error, which defaults to false, and is set
according to the actual exception above. So remove that.
Signed-off-by: Daniel Leung <daniel.leung@intel.com>
BTI requires that the C library be compiled with -mbranch-protection to
include BTI landing pads. Newlib from toolchains lacks this support, so
only minimal libc or picolibc built from source (PICOLIBC_USE_MODULE) can
be used with BTI.
Without this, the basic hello_world/ sample fails to execute.
Signed-off-by: Nicolas Pitre <npitre@baylibre.com>
Change the random number generator requirement for per-thread PAC keys
from a hard dependency to a more flexible approach:
- Use 'select CSPRNG_NEEDED' to automatically request cryptographic
RNG support rather than requiring specific RNG options to be
pre-enabled
- Use 'imply TEST_RANDOM_GENERATOR' as a fallback when no real CSPRNG
is available, enabling testing without hardware entropy
Signed-off-by: Nicolas Pitre <npitre@baylibre.com>
Enable the ARM_PACBTI Kconfig choice for ARM64 architectures (ARMV8_A
and ARMV9_A) in addition to the existing ARM32 ARMV8_1_M_MAINLINE
support. Add the corresponding -mbranch-protection compiler flags to
both GCC and Clang target files for ARM64.
Signed-off-by: Nicolas Pitre <npitre@baylibre.com>
Add support for ARMv8.5+ Branch Target Identification to protect against
Jump-Oriented Programming (JOP) attacks. This complements PAC to offer
complete protection against both ROP and JOP attacks, ensuring
comprehensive control flow integrity.
Signed-off-by: Nicolas Pitre <npitre@baylibre.com>
Add support for ARMv8.3+ Pointer Authentication to protect against
Return-Oriented Programming (ROP) attacks. This implementation provides
PAC functionality with per-thread key isolation, secure key management,
and integration with Zephyr's thread model.
Signed-off-by: Nicolas Pitre <npitre@baylibre.com>
The Xen events channel driver consume 72K of RAM, but may not be
required in all use cases.
Added a XEN_EVENTS Kconfig option so that Xen events can be gracefully
disabled if not required. Updated the relevant CMakeLists.txt and
Kconfig files to guard the inclusion of the Xen events driver and its
source files by this option.
Signed-off-by: Grygorii Strashko <grygorii_strashko@epam.com>
Signed-off-by: Svitlana Drozd <svitlana_drozd@epam.com>
Functions in assembler file pm_s2ram.S are declared with the usual:
SECTION_FUNC(TEXT, <function name>)
Note the first argument (section name) is `TEXT` in capital letters which
a define in `include/zephyr/linker/sections.h` should replace with `text`,
such that the functions are placed in section `.text.<function name>` which
matches the ".text.*" pattern in linker script. However, this file is not
included by pm_s2ram.S: as such, the substitution never happens and the
functions go in `.TEXT.<function name>` instead! This has not caused issues
thanks to a workaround in the Cortex-M linker script, which also has
".TEXT.*" as input section name pattern (unlike all other archs!), but is a
bug nonetheless.
Fix this issue by adding the missing include which ensures the functions
are placed in sections with the proper name.
Signed-off-by: Mathieu Choplain <mathieu.choplain-ext@st.com>
