The arm_core_mpu_enable() and arm_core_mpu_disable() functions were
defined in the ARM MPU driver but never declared in a header.
As a result, callers had to provide their own local `extern`
declarations to use them.
Add documented prototypes for both functions to
include/zephyr/arch/arm/mpu/arm_mpu.h and drop the redundant `extern`
declarations in arm_core_mpu.c and the userspace memory protection
test.
Signed-off-by: Andrej Butok <andrey.butok@nxp.com>
CONFIG_LINKER_USE_PINNED_SECTION is the second half of the selective
kernel-pinning model removed in issue #108773. With the kernel image
now always resident at boot (previous commit), the __pinned_*
attribute family is a no-op: every page they would have segregated is
already pinned by z_mem_manage_init()'s whole-image loop, so the
tagging contract neither adds safety nor remains maintainable.
Drop it.
Mechanical removals:
* All ~219 in-tree uses of __pinned_text, __pinned_rodata,
__pinned_data, __pinned_bss, __pinned_noinit, and __pinned_func
across arch/x86, drivers/interrupt_controller, drivers/timer,
arch/common, kernel, lib/libc, subsys/portability/posix, tests, and
the syscall code generator (scripts/build/gen_syscalls.py).
* The assembly aliases PINNED_TEXT/RODATA/DATA/BSS/NOINIT used in
arch/x86/core/ia32/*.S and drivers/interrupt_controller/
intc_loapic_spurious.S become plain TEXT/RODATA/DATA/BSS/NOINIT.
* K_KERNEL_PINNED_STACK_DEFINE, K_KERNEL_PINNED_STACK_ARRAY_DEFINE,
K_KERNEL_PINNED_STACK_ARRAY_DECLARE, K_THREAD_PINNED_STACK_DEFINE,
and K_THREAD_PINNED_STACK_ARRAY_DEFINE are removed. The few
in-tree callers (kernel/init.c, arch/arm/core/cortex_a_r/smp.c,
arch/arm64/core/fatal.c, arch/rx/core/prep_c.c,
arch/x86/core/prep_c.c, kernel/include/kernel_internal.h,
tests/bluetooth/hci_uart_async) move to the corresponding
non-pinned macros.
Machinery removals:
* Kconfig.zephyr drops CONFIG_LINKER_USE_PINNED_SECTION.
qemu_x86_tiny and qemu_x86_atom_virt drop their =y overrides.
* include/zephyr/linker/section_tags.h drops the __pinned_* macro
definitions (both arms). __isr collapses to an empty macro since
its only purpose was to alias __pinned_func.
* include/zephyr/linker/sections.h drops PINNED_TEXT_SECTION_NAME,
PINNED_BSS_SECTION_NAME, etc. and the bare PINNED_TEXT/RODATA/etc.
forwarders, plus the _APP_SMEM_PINNED_SECTION_NAME constant.
* include/zephyr/linker/linker-defs.h drops the lnkr_pinned_*
externs, the _app_smem_pinned_* externs, and the lnkr_is_pinned()
/ lnkr_is_region_pinned() inline helpers.
* include/zephyr/linker/utils.h drops the lnkr_pinned_rodata branch
in linker_is_in_rodata().
* include/zephyr/linker/app_smem_pinned{,_aligned,_unaligned}.ld
are deleted; cmake/linker/ld/target_configure.cmake stops
configuring them.
* boards/qemu/x86/qemu_x86_tiny.ld and
include/zephyr/arch/x86/ia32/linker.ld drop their pinned-section
blocks and the now-redundant #ifndef CONFIG_LINKER_USE_PINNED_SECTION
conditionals throughout the body. The
LIB_KERNEL_IN_SECT / LIB_ARCH_X86_IN_SECT / LIB_ZEPHYR_IN_SECT /
LIB_C_IN_SECT / LIB_DRIVERS_IN_SECT / LIB_SUBSYS_LOGGING_IN_SECT /
LIB_ZEPHYR_OBJECT_FILE_IN_SECT / ZEPHYR_KERNEL_FUNCS_IN_SECT macros
in qemu_x86_tiny.ld are deleted; they existed only to feed the
pinned text/rodata/data/bss/noinit sections.
* kernel/mmu.c drops the mark_linker_section_pinned(lnkr_pinned_start,
...) call. The mark_linker_section_pinned() helper survives but is
now gated only on CONFIG_LINKER_USE_BOOT_SECTION.
* arch/common/init.c and include/zephyr/arch/common/init.h drop
arch_bss_zero_pinned(); arch/x86/core/ia32/crt0.S drops the call
to it.
* arch/x86/core/userspace.c drops the eager k_mem_page_in() of the
thread's privileged stack on user-mode entry. With the kernel
image fully resident the stack is already mapped.
* arch/x86/gen_mmu.py drops map_region("lnkr_pinned") and the
set_region_perms() calls for lnkr_pinned_text / lnkr_pinned_rodata.
* CMakeLists.txt drops the LINKER_USE_PINNED_SECTION block that
generated APP_SMEM_PINNED_* variables and the
pinned_partitions target property feeding gen_app_partitions.py.
cmake/modules/extensions.cmake removes the PINNED_RODATA /
PINNED_RAM_SECTIONS / PINNED_DATA_SECTIONS zephyr_linker_sources()
location keywords and their snippet files.
scripts/build/gen_app_partitions.py drops --pinoutput /
--pinpartitions arguments and the pinned-output branch.
subsys/testsuite/coverage/CMakeLists.txt drops its
CONFIG_DEMAND_PAGING-conditional fork.
* scripts/build/gen_kobject_list.py drops the
app_smem_pinned_start / _end fallback for kobject placement
validation.
* tests/arch/x86/pagetables and tests/kernel/mem_protect/userspace
drop their lnkr_pinned_text / lnkr_pinned_rodata branches.
* include/zephyr/arch/x86/ia32/arch.h folds IRQSTUBS_TEXT_SECTION
to the unconditional ".text.irqstubs" form.
* tests/subsys/llext/src/syscalls_ext.c drops a stale comment about
syscalls landing in .pinned_text.
Targeted retentions:
* arch/x86/core/bootargs.c keeps multiboot_cmdline and efi_bootargs
in .noinit (was __pinned_noinit, which decayed to __noinit when
LINKER_USE_PINNED_SECTION was unset). The multiboot and zefi loader
paths write these buffers before Zephyr's BSS-zero step, so
zeroing them at boot loses the cmdline.
* arch/x86/core/ia32/fatal.c keeps _df_esf and _df_stack in .noinit.
They are scratch space written by the double-fault handler and have
no zero-init requirement; keeping them in .noinit also preserves
the historical post-noinit alignment that gen_mmu.py relies on
(z_mapped_size is computed before CMake-injected iterable sections
are appended to the linker script, so the post-noinit page padding
is what keeps those sections within the mapped region).
* include/zephyr/arch/x86/ia32/syscall.h and
include/zephyr/arch/x86/arch.h wrap the per-arch
arch_syscall_invoke* / arch_is_user_context / arch_k_cycle_get_*
implementations in @cond INTERNAL_HIDDEN. The public Doxygen
contract lives on the prototypes in
include/zephyr/arch/arch_interface.h; the per-arch implementations
are internal. Without this, removing the __pinned_func attribute
exposes the implementations to the doxygen-coverage delta check
as 10 newly-undocumented APIs.
Documentation updates are deferred to a separate commit.
Issue: #108773
Signed-off-by: Nicolas Pitre <npitre@baylibre.com>
TLS area must be defined also when multithreading is disabled. Not doing
so whould cause access to libc variables like 'errno' to fail with a
crash.
This commit adds the TLS area for the single thread case by reserving
some space at the top of the main stack area.
Signed-off-by: Valerio Setti <vsetti@baylibre.com>
Replace the opt-in locked marker with a legacy unlocked marker. With
this, SoC PM hooks use the locked IRQ contract by default, and only
legacy SoCs that still unmask interrupts from PM hooks select
PM_STATE_SET_IRQ_UNLOCKED.
Drop the old locked marker selections from already migrated NXP SoCs and
update the PM test expectation accordingly.
Signed-off-by: Holt Sun <holt.sun@nxp.com>
arm_m_switch_stack_buffer is only defined when CONFIG_USE_SWITCH is
enabled. With link-time optimization or certain no-optimization builds
using Clang, the symbol is referenced unconditionally and the linker
fails with:
ld.lld: error: undefined symbol: arm_m_switch_stack_buffer
Move the reference inside a preprocessor guard so that the symbol is
only accessed when the kernel is built with CONFIG_USE_SWITCH.
Assisted-by: GitHub Copilot:claude-sonnet-4.6
Signed-off-by: Anas Nashif <anas.nashif@intel.com>
Add support for generating llext veneers for Arm baseline cores.
The veneer is AAPCS-compliant: r0-r3, r5-r11 and LR are preserved,
r4 is saved and restored, and only r12 (ip) is clobbered as
permitted by AAPCS.
Tested on RP2040 (Cortex-M0+) loading an extension that calls
__gnu_thumb1_case_uqi.
Signed-off-by: Ibrahim Abdalkader <i.abdalkader@gmail.com>
Enabling the FPU forces the choice between CONFIG_FP_HARDABI
and CONFIG_FP_SOFTABI. Both of these options allow the compiler
to generate FP instructions. As a result, all threads must have
the K_FP_REGS options bit set because we can not predict where
the compiler will generate those instructions.
The forced enablement is keyed off those ABI Kconfig options
and not FPU sharing nor the FPU enablement. This leaves the
option open for a future (as yet not implemented) case where
someone might want FPU support but not have the compiler generate
any FPU instructions at all.
Fixes#108793
Signed-off-by: Peter Mitsis <peter.mitsis@intel.com>
Add support for the Cortex-A32 processor. The Cortex-A32 processor
is built on the ARMv8-A architecture and supports only the AArch32
execution state.
Signed-off-by: Silesh C V <silesh@alifsemi.com>
Add CONFIG_AARCH32_ARMV8_A_MONITOR_INIT to perform early Monitor mode
(EL3 AArch32) initialization at reset, modelled after z_arm64_el3_init()
in the arm64 port.
On ARMv8-A AArch32 platforms booting bare-metal without Trusted
Firmware-A, the core resets into Secure SVC mode. Certain system
registers (such as ICC_MSRE for GICv3) are only accessible from Monitor
mode. This option briefly switches to Monitor mode at reset to configure
such registers before returning to Secure SVC mode.
Currently used to configure ICC_MSRE, which enables GICv3 CPU
interface system register access at lower privilege levels (equivalent
to ICC_SRE_EL3 in the arm64 port).
ICC_MSRE_INIT reuses the existing ICC_SRE_ELx_* bit definitions from
cpu.h as the AArch32 ICC_MSRE shares the same bitfield layout as its
AArch64 counterpart ICC_SRE_EL3.
Signed-off-by: Silesh C V <silesh@alifsemi.com>
Add a new AARCH32_ARMV8_A Kconfig symbol for ARMv8-A processors
running in AArch32 execution state, modelled after the existing
ARMv8-R AArch32 support. ARMv8-A AArch32 execution state is used
by processors such as the Cortex-A32 that implement only the AArch32
state as well as by other ARMv8-A processors that support AArch32
at specific exception levels.
A dedicated Kconfig symbol is needed (compared to simply reusing
ARMV7_A)because ARMv8-A AArch32 shares some code paths with ARMv7-A
and others with ARMv8-R AArch32. This also means that rather than
introducing a new standalone architecture port, the implementation
selectively extends the code paths used by either of these
architectures based on the architectural requirements as detailed
below.
MMU/fault handling: The implementation uses the existing short
descriptor translation table format used by ARMv7-A, sharing
the MMU programming and FSR encodings.
exception modes stack initialization: ARMv8-A AArch32 implements the
same exception modes as ARMv7-A and so reuses the ARMv7-A exception
mode stack initialization framework directly.
VBAR programming: Unlike ARMv7-A, ARMv8-A AArch32 mandates the use of
VBAR to store the vector table base address. So extend the vector
table relocation support already used by ARMv8-R AArch32. Also map
the vectors region from arch-level code rather than requiring each
SoC to provide this.
Arm generic timer support: The Generic Timer is mandatory in ARMv8-A
AArch32 and is accessed via the system register interface rather than
the memory-mapped I/O interface used by some ARMv7-A implementations.
The system-register-based timer path is therefore shared with ARMv8-R
AArch32.
interrupt masking: Reuse the CPSR based interrupt masking used by
ARMv7-A and ARMv8-R AArch32.
SVC based fatal error path: Reuse the SVC based fatal error path
shared by ARMv7-A and ARMv8-R AArch32.
Signed-off-by: Silesh C V <silesh@alifsemi.com>
kernel.h implies kernel_structs.h via kernel_includes.h, making
explicit inclusion of kernel_structs.h unnecessary whenever kernel.h
is already included in the same translation unit.
Remove the redundant includes across arch, boards, drivers, kernel,
lib, samples, subsys, and tests trees.
in include/zephyr/kernel_structs.h:
* 2. kernel.h shall imply kernel_structs.h, such that it shall not be
* necessary to include kernel_structs.h explicitly when kernel.h is
* included.
Signed-off-by: Anas Nashif <anas.nashif@intel.com>
Restrict ARCH_HAS_LLEXT_VENEERS to ARMV7_M_ARMV8_M_MAINLINE instead
of all CPU_CORTEX_M targets, since the veneer implementation relies
on a Mainline-only Thumb-2 instruction sequence not supported on
Baseline cores.
Signed-off-by: Ibrahim Abdalkader <i.abdalkader@gmail.com>
Keep the original architecture IRQ key owned by idle across a
successful system PM transition.
Add architecture hooks and the PM_STATE_SET_IRQ_LOCKED migration
contract for SoCs that keep PM hooks from unmasking interrupts.
Signed-off-by: Holt Sun <holt.sun@nxp.com>
When an LLEXT is loaded in memory and calls kernel or libc symbols
located in flash, the target may fall outside the range of branches
on some architectures, such as Thumb-2 BL, causing relocations to
fail.
Add CONFIG_LLEXT_VENEERS which, when enabled, generates trampoline
stubs for such out-of-range relocations.
Stubs are allocated from the LLEXT heap into a new LLEXT_MEM_VENEER
region, instead of using new state variables to track the memory, to
leverage the existing llext machinery that flushes cache, frees memory
on unload, etc.
A test extension is added, compiled with -mno-long-calls to force
direct branches on Arm and exercise the veneer path against multiple
libc symbols.
Signed-off-by: Ibrahim Abdalkader <i.abdalkader@gmail.com>
Allow to use a switch-case instead of an array holding ISR entries.
When most of IRQs are not used, they share the same, default entry.
It results in most of the ISR array entries being identical duplicates.
This change allows to use dynamically generated function (after first
linker pass) that uses switch-case instead of a full array.
Default entries are handled only once, in a default section.
Used IRQs have their own case sections.
This can help reduce binary size.
Signed-off-by: Adam Szczygieł <adam.szczygiel@nordicsemi.no>
This relocation is used by the ARM TLS code to access thread local
variables. It is a simple absolute relocation that adds the symbol's
offset to the value at the location. This allows the code to access
thread local variables using a fixed offset from the thread pointer,
which is determined at runtime.
Signed-off-by: Luca Burelli <l.burelli@arduino.cc>
The generic ARM MPU nocache-memory cleanup path assumes Cortex-M
SCB dcache support whenever it needs to clean and invalidate
cache state before programming MPU regions.
That is correct for integrated ARCH_CACHE systems, but not for
cache backends such as NXP LMEM on RT11xx CM4 targets. Those
targets can select CPU_HAS_DCACHE and NOCACHE_MEMORY while using
a non-ARCH cache backend, which makes the direct SCB dcache
symbols unavailable and breaks builds in z_arm_mpu_init().
Keep the direct CMSIS SCB_CleanInvalidateDCache() call under the
ARCH_CACHE guard — since we already test SCB->CCR the integrated
cache controller is known to be present — and use the generic
cache API for other cache backends. This preserves the existing
integrated-cache behavior while allowing non-ARCH cache backends
to participate in the same MPU cleanup path.
Signed-off-by: Holt Sun <holt.sun@nxp.com>
ARMv8-R AArch32 cores determine the CPU start address on reset from
RVBAR (Reset Vector Base Address Register), which only stores bits
[31:5] — bits [4:0] are RES0. Any firmware or boot-loader that
programs RVBAR from the ELF entry point will silently truncate
a non-aligned address to a 32-byte boundary, causing the CPU to
begin executing at the wrong location.
Whether __start lands on a 32-byte boundary depends on the size of
code sections placed before it, which changes with Kconfig options.
This makes the failure non-deterministic: a build may work today and
break after enabling an unrelated feature like logging.
Force 32-byte alignment on z_arm_reset/__start for ARMv8-R so the
entry point survives RVBAR truncation on any SoC.
Signed-off-by: Appana Durga Kedareswara rao <appana.durga.kedareswara.rao@amd.com>
We will make use of the .exc_return member during walk_stackframe() to
know whether we have extended stack or standard stack.
Signed-off-by: Jisheng Zhang <jszhang@kernel.org>
Some ARMv6-M and ARMv8-M Baseline cores indeed support MPU
(CPU_HAS_ARM_MPU in soc Kconfig), so the exclusion should
not be based on ARMV6_M_ARMV8_M_BASELINE.
Signed-off-by: Andy Lin <andylinpersonal@gmail.com>
Extend the ARM Cortex-M coredump arch block to version 3 with metadata
that provides the offset to the callee_saved struct within k_thread.
This enables the coredump GDB stub to accurately retrieve callee-saved
registers (r4-r11) for non-faulting threads during multi-thread
debugging.
Signed-off-by: Mark Holden <mholden@meta.com>
The arch_dcache_enable() and arch_icache_enable() functions could
cause system crashes when called on caches that were already enabled.
This occurs because arch_dcache_invd_all() invalidates the entire
cache without first flushing dirty data, leading to memory corruption
when the cache was previously enabled.
This scenario happens in cache tests where test setup calls
sys_cache_data_enable(), but the SoC early init hook has already
enabled caches during boot.
Fix by checking the SCTLR register before performing cache operations:
- If D-cache is already enabled, perform clean+invalidate instead of
just invalidate to preserve dirty cache lines
- If I-cache is already enabled, perform invalidate only (no dirty
lines in I-cache)
- If cache is not enabled, proceed with normal enable sequence
This makes the enable functions safe to call multiple times without
risking data corruption or system crashes.
Signed-off-by: Appana Durga Kedareswara rao <appana.durga.kedareswara.rao@amd.com>
relocate_vector_table is called as part of z_arm_reset.
This is considered early-boot code before XIP.
At this stage, Program might not have access to optimized
compiler APIs that reside in FLASH.
Thus, its better for relocate_vector_table to use arch_early_memcpy.
Signed-off-by: Shreyas Shankar <s-shankar@ti.com>
Add calls to sys_trace_idle_exit before leaving idle state
to track CPU load.
Extend CPU_LOAD to CPU_AARCH32_CORTEX_R and CPU_AARCH32_CORTEX_A, thus
we can support CPU_LOAD for all CPU_CORTEX.
Signed-off-by: Jisheng Zhang <jszhang@kernel.org>
ASM is notoriously harder to maintain than C and requires core specific
adaptation which impairs even more the readability of the code.
As for performance concern, there's no difference of generated code
between ASM and C version.
ASM version:
<arch_cpu_idle>:
f57ff04f dsb sy
e320f003 wfi
f1080080 cpsie i
f57ff06f isb sy
e12fff1e bx lr
<arch_cpu_atomic_idle>:
f10c0080 cpsid i
f57ff04f dsb sy
e320f002 wfe
e3500000 cmp r0, #0
1a000000 bne 102ca8 <_irq_disabled>
f1080080 cpsie i
<_irq_disabled>:
e12fff1e bx lr
C version:
<arch_cpu_idle>:
f57ff04f dsb sy
e320f003 wfi
f1080080 cpsie i
f57ff06f isb sy
e12fff1e bx lr
<arch_cpu_atomic_idle>:
f10c0080 cpsid i
f57ff04f dsb sy
e320f002 wfe
e3500000 cmp r0, #0
112fff1e bxne lr
f1080080 cpsie i
e12fff1e bx lr
As can be seen, the C version use 'bxne lx' to return directly for irq
disabled case, cost one less instruction than asm version. So from this
PoV, C version not only improves the readability and maintainability
but also generates better code.
Signed-off-by: Jisheng Zhang <jszhang@kernel.org>
This allows to distinguish between f16 storage format support
(CONFIG_FP16) and actual f16 arithmetic capability.
CONFIG_FP16_ARITHMETIC requires either MVE float (ARMV8_1_M_MVEF) or a
Cortex-A core (CPU_CORTEX_A).
Signed-off-by: Martin Jäger <martin.jaeger@a-labs.io>
USE_SWITCH is a new feature and needs more testing before enabling it by
default. While all tests in upstream Zephyr CI passed, keeping this
config disabled helps in getting majority of the work in without causing
regression on upstream boards that are not tested in ci.
Signed-off-by: Sudan Landge <sudan.landge@arm.com>
Fix below issues when trying to build hello world with armclang:
```
Error: L6218E: Undefined symbol z_arm_exc_exit (referred from reset.o).
Error: L6218E: Undefined symbol z_arm_int_exit (referred from reset.o).
Error: L6218E: Undefined symbol z_arm_pendsv (referred from reset.o).
```
Signed-off-by: Sudan Landge <sudan.landge@arm.com>
orr fix is as reported in review:
```
The add causes a crash with IAR tools as the address loaded to r8
already has the lowest bit set, and the add causes it to be set to ARM
mode. The orr instruction works fine with both scenarios
```
`UDF 0` seems to break on IAR but `UDF #0` works for all.
Signed-off-by: Sudan Landge <sudan.landge@arm.com>
USE_SWITCH code unconditionally applied interrupt locking, which altered
BASEPRI handling and broke expected interrupt behavior on both
Baseline and Mainline CPUs when USE_SWITCH was disabled.
This commit restores the original behavior with USE_SWITCH disabled and
fixes tests/arch/arm/arm_interrupt failures.
Signed-off-by: Sudan Landge <sudan.landge@arm.com>
The ARM Ltd. FVP emulator (at least the variants run in Zephyr CI)
appears to have a bug with the stack alignment bit in xPSR. It's
common (it fails in the first 4-6 timer interrupts in
tests.syscalls.timeslicing) that we'll take an interrupt from a
seemingly aligned (!) stack with the bit set. If we then switch and
resume the thread from a different context later, popping the stack
goes wrong (more so than just a misalignment of four bytes: I usually
see it too low by 20 bytes) in a way that it doesn't if we return
synchronously. Presumably legacy PendSV didn't see this because it
used the unmodified exception frame.
Work around this by simply assuming all interrupted stacks were
aligned and clearing the bit. That is NOT correct in the general
case, but in practice it's enough to get tests to pass.
Signed-off-by: Andy Ross <andyross@google.com>
The exit from the SVC exception used for syscalls back into the
calling thread is done without locking. This means that the
intermediate states can be interrupted while the kernel-mode code is
still managing thread state like the mode bit, leading to mismatches.
This seems mostly robust when used with PendSV (though I'm a little
dubious), but the new arch_switch() code needs to be able to suspend
such an interrupted thread and restore it without going through a full
interrupt entry/exit again, so it needs locking for sure.
Take the lock unconditionally before exiting the call, and release it
in the thread once the magic is finished, just before calling the
handler. Then take it again before swapping stacks and dropping
privilege.
Even then there is a one-cycle race where the interrupted thread has
dropped the lock but still has privilege (the nPRIV bit is clear in
CONTROL). This thread will be resumed later WITHOUT privilege, which
means that trying to set CONTROL will fail. So there's detection of
this 1-instruction race that will skip over it.
Signed-off-by: Andy Ross <andyross@google.com>
Some toolchains don't support an __asm__(...) block at the top level
of a file and require that they live within function scope. That's
not a hardship as these two blocks were defining callable functions
anyway. Exploit the "naked" attribute to avoid wasted bytes in unused
entry/exit code.
Signed-off-by: Andy Ross <andyross@google.com>
Late-arriving clang-format-demanded changes that are too hard to split
and squash into the original patches. No behavior changes.
Signed-off-by: Andy Ross <andyross@google.com>
Some nitpicky hand-optimizations, no logic changes:
+ Shrink the assembly entry to put more of the logic into
compiler-optimizable C.
+ Split arm_m_must_switch() into two functions so that the first
doesn't look so big to the compiler. That allows it to spill (many)
fewer register on entry and speeds the (very) common early-exit case
where an interrupt returns without context switch.
Signed-off-by: Andy Ross <andyross@google.com>
When USE_SWITCH=y, the thread struct is now mostly degenerate. Only
the two words for ICI/IT state tracking are required. Eliminate all
the extra fields when not needed and save a bunch of SRAM.
Note a handful of spots in coredump/debug that need a location for the
new stack pointer (stored as the switch handle now) are also updated.
Signed-off-by: Andy Ross <andyross@google.com>
The new switch code no longer needs PendSV, but still calls the SVC
vector. Split them into separate files for hygiene and a few
microseconds of build time.
Signed-off-by: Andy Ross <andyross@google.com>
Micro-optimization: We don't need a full arch_irq_lock(), which is a
~6-instruction sequence on Cortex M. The lock will be dropped
unconditionally on interrupt exit, so take it unconditionally.
Signed-off-by: Andy Ross <andyross@google.com>
z_get_next_switch_handle() is a clean API, but implementing it as a
(comparatively large) callable function requires significant
entry/exit boilerplate and hides the very common "no switch needed"
early exit condition from the enclosing C code that calls it. (Most
architectures call this from assembly though and don't notice).
Provide an unwrapped version for the specific needs non-SMP builds.
It's compatible in all other ways.
Slightly ugly, but the gains are significant (like a dozen cycles or
so).
Signed-off-by: Andy Ross <andyross@google.com>
GCC/gas has a code generation bugglet on thumb. The R7 register is
the ABI-defined frame pointer, though it's usually unused in zephyr
due to -fomit-frame-pointer (and the fact the DWARF on ARM doesn't
really need it). But when it IS enabled, which sometimes seems to
happen due to toolchain internals, GCC is unable to allow its use in
the clobber list of an asm() block (I guess it can't generate
spill/fill code without using the frame?).
There is existing protection for this problem that sets
-fomit-frame-pointer unconditionally on the two files (sched.c and
init.c) that require it. But even with that, gcc sometimes gets
kicked back into "framed mode" due to internal state. Provide a
kconfig workaround that does an explicit spill/fill on the one
test/platform where we have trouble.
(I checked, btw: an ARM clang build appears not to have this
misfeature)
Signed-off-by: Andy Ross <andyross@google.com>
ARM Cortex M has what amounts to a design bug. The architecture
inherits several unpipelined/microcoded "ICI/IT" instruction forms
that take many cycles to complete (LDM/STM and the Thumb "IT"
conditional frame are the big ones). But out of a desire to minimize
interrupt latency, the CPU is allowed to halt and resume these
instructions mid-flight while they are partially completed. The
relevant bits of state are stored in the EPSR fields of the xPSR
register (see ARMv7-M manual B1.4.2). But (and this is the design
bug) those bits CANNOT BE WRITTEN BY SOFTWARE. They can only be
modified by exception return.
This means that if a Zephyr thread takes an interrupt
mid-ICI/IT-instruction, then switches to another thread on exit, and
then that thread is resumed by a cooperative switch and not an
interrupt, the instruction will lose the state and restart from
scratch. For LDM/STM that's generally idempotent for memory (but not
MMIO!), but for IT that means that the restart will re-execute
arbitrary instructions that may not be idempotent (e.g. "addeq r0, r0,
The fix is to check for this condition (which is very rare) on
interrupt exit when we are switching, and if we discover we've
interrupted such an instruction we swap the return address with a
trampoline that uses a UDF instruction to immediately trap to the
undefined instruction handler, which then recognizes the fixup address
as special and immediately returns back into the thread with the
correct EPSR value and resume PC (which have been stashed in the
thread struct). The overhead for the normal case is just a few cycles
for the test.
Signed-off-by: Andy Ross <andyross@google.com>
Integrate the new context layer, allowing it to be selected via the
pre-existing CONFIG_USE_SWITCH. Not a lot of changes, but notable
ones:
+ There was code in the MPU layer to adjust PSP on exception exit at a
stack overflow so that it remained inside the defined stack bounds.
With the new context layer though, exception exit will rewrite the
stack frame in a larger format, and needs PSP to be adjusted to make
room.
+ There was no such treatment in the PSPLIM case (the hardware prents
the SP from going that low), so I had to add similar code to
validate PSP at exit from fault handling.
+ The various return paths for fault/svc assembly handlers need to
call out to the switch code to do the needed scheduler work. Really
almost all of these can be replaced with C now, only userspace
syscall entry (which has to "return" into the privileged stack)
needs special treatment.
+ There is a gcc bug that prevents the arch_switch() inline assembly
from building when frame pointers are enabled (which they almost
never are on ARM): it disallows you from touching r7 (the thumb
frame pointer) entirely. But it's a context switch, we need to!
Worked around by enforcing -fomit-frame-pointer even in the two
scheduler files that can swap when NO_OPTIMIZATIONS=y.
Signed-off-by: Andy Ross <andyross@google.com>
Signed-off-by: Sudan Landge <sudan.landge@arm.com>
