Adds sys_ prefix to functions and typedef of the
word granular access library, as it is a sys API.
Signed-off-by: Lauren Murphy <lauren.murphy@intel.com>
This removes all the extra code path when pin only CPU mask
and kernel coherence are both enabled. The original idea was
that with CPU pinning only, we could skip all the stack
manipulation since threads are not going to migreate between
CPUs until they get pinned to another CPUs. However, this
creates an issue where a thread is created on one CPU and
its stack being populated for startup at this CPU. Then it
gets pinned to another CPU. The other CPU does not know that
it is first starting the thread and will need to invalidate
the cache to grab the new stack content. This problem also
applies to migration between CPUs as there is no code to
manipulate the cache on stack. To support the skipping of
cache manipulation with pin only CPU mask would require
some invasive changes to kernel. At this point, the only
Xtensa SoCs requiring kernel coherence are not using pin
only CPU mask. So we are safe to remove the special code
path, and also we unify stack coherence operations to
minimize differences.
Fixes#112850
Signed-off-by: Daniel Leung <daniel.leung@intel.com>
The Cortex-A/R interrupt wrapper enables IRQs to support nested interrupt
handling while running the registered ISR.
Move cpsie i after the spurious IRQ check so only valid ISR calls run
with IRQs enabled, and add cpsid i after the ISR returns. This prevents
EOI and z_arm_int_exit() from running with IRQs enabled.
Signed-off-by: Andrei-Edward Popa <andrei.popa105@yahoo.com>
ARCH_SUPPORTS_ROM_OFFSET is a hidden architecture-capability option, but
it was expressed as a reverse dependency on a hard-coded arch list
(ARM, X86, ARM64, RISCV, ARC) defaulting to y. This is inconsistent with
the surrounding ARCH_SUPPORTS_* symbols (e.g. ARCH_SUPPORTS_ROM_START),
which are selected by the architectures that provide the capability, and
the allowlist drifts as architectures are added.
Turn it into a plain selected capability symbol and have the supporting
architectures select it, matching the established convention. No
functional change.
Assisted-by: Claude:claude-opus-4-8
Signed-off-by: Anas Nashif <anas.nashif@intel.com>
ISR_TABLES_LOCAL_DECLARATION_SUPPORTED gated its availability on a
hard-coded "List of currently supported architectures" (ARM, ARM64,
RISCV). Whether an architecture supports local declaration of interrupt
tables placed by the linker is an arch property, so the arch should
declare it rather than have the option carry an allowlist that drifts.
Introduce a hidden ARCH_HAS_ISR_TABLES_LOCAL_DECLARATION capability
symbol, have the supporting architectures select it, and depend the
supported symbol on the capability instead of the arch list. The
toolchain and userspace dependencies are unchanged. No functional change.
Assisted-by: Claude:claude-opus-4-8
Signed-off-by: Anas Nashif <anas.nashif@intel.com>
CONFIG_SEMIHOST depended on a hard-coded list of architectures
(ARM, ARM64, RISCV, XTENSA). Each of these ships its own semihosting
backend under arch/*/core/semihost.c, so providing a semihosting
implementation is an architecture property that the arch should declare,
rather than having the common option maintain an allowlist that a new
arch adding a backend must remember to update.
Introduce a hidden ARCH_HAS_SEMIHOST capability symbol, have the
architectures that implement semihosting select it, and depend SEMIHOST
on the capability instead of the arch list. No functional change.
Assisted-by: Claude:claude-opus-4-8
Signed-off-by: Anas Nashif <anas.nashif@intel.com>
IRQ_OFFLOAD_NESTED defaulted to y based on a hard-coded list of
architectures (ARM64, X86, RISCV, XTENSA). Whether irq_offload() may
legally be called in interrupt context to cause a synchronous nested
interrupt is an architecture property, so the arch itself should declare
it rather than have the option maintain an allowlist that drifts.
Introduce a hidden ARCH_HAS_IRQ_OFFLOAD_NESTED capability symbol, have
the architectures whose irq_offload() supports nesting select it, and key
the default of IRQ_OFFLOAD_NESTED off the capability instead of the list.
No functional change.
Assisted-by: Claude:claude-opus-4-8
Signed-off-by: Anas Nashif <anas.nashif@intel.com>
arch/xtensa/core/irq_manage.c still defines the dynamic IRQ connect
routine under its pre-2019 z_arch_ name, which nothing references
since the z_arch_* to arch_* interface rename. The __weak
arch_irq_connect_dynamic() fallback in arch/common/dynamic_isr.c
silently took over, so the multi-level variant that routes to
z_soc_irq_connect_dynamic() never runs, and the only implementation
of that SoC hook (intel_adsp cavs) has had no caller ever since.
The common weak fallback is functionally equivalent on cavs: the
cavs intc driver registers IRQ_PARENT_ENTRY_DEFINE and
CAVS_ISR_TBL_OFFSET defaults to 2ND_LVL_ISR_TBL_OFFSET, so the
generic z_get_sw_isr_table_idx() computes the same table slot the
hand-rolled SoC hook did. Reviving the routing would also break
other multi-level Xtensa SoCs (ace, mt8xxx, imx8 DSPs) that never
provided the hook. Remove the dead functions, the hook declaration
and its cavs implementation instead.
Assisted-by: Claude:claude-fable-5
Signed-off-by: Anas Nashif <anas.nashif@intel.com>
The OpenRISC linker script already implements CODE_DATA_RELOCATION
support, but the Kconfig symbol was never selected, preventing the
feature from being enabled.
Signed-off-by: Joel Holdsworth <jholdsworth@nvidia.com>
When Secure firmware handles a fault with a recovered Non-Secure
exception frame, keep that context through fault classification.
Use it when reporting SecureFaults so the dump makes clear that the
fault originated in Non-Secure state and escalated into Secure state.
Also include guidance for the missing or disabled Non-Secure fault
handler case, where this escalation is commonly seen.
Handle the unexpected case where an exception frame cannot be recovered
without relying on __ASSERT(). Report the failure through the fault log
and terminate through z_arm_fatal_error(), so the path remains fatal
even when assertions are disabled.
Signed-off-by: Sudan Landge <sudan.landge@arm.com>
The fault assembly wrapper saves r4-r11 after exception entry. When
EXC_RETURN says callee stacking was skipped, the processor has already
saved the interrupted Secure r4-r11 in the Secure additional context
block, so the wrapper's live r4-r11 are not the interrupted values.
Keep a pointer to the skipped Secure additional context and use it to
populate extra_info.callee when available, so fault dumps and other ESF
consumers report the interrupted Secure context.
Signed-off-by: Sudan Landge <sudan.landge@arm.com>
Secure exception entry can place an integrity signature and additional
Secure context at the top of the stack before the basic exception frame.
Detect that signature when resolving the ESF and skip the additional
context before casting the stack pointer to struct arch_esf.
When EXC_RETURN says callee stacking was skipped, treat a missing
integrity signature as an invalid frame instead of silently using the
original stack pointer as the ESF. This avoids reporting misleading
fault state when the Secure stack shape is inconsistent or corrupt.
Signed-off-by: Sudan Landge <sudan.landge@arm.com>
Enables the __nocache attribute on ARCv2 with MPU v6.
Adds REGION_NOCACHE_ATTR and the DC bit to the public
MPU header so boards can declare nocache regions.
_region_init() preserves the DC bit through region programming.
Boards opt in by adding a nocache entry to mpu_regions[],
placed first for higher priority than SRAM.
nsim_hs_mpuv6 is the first consumer.
Adds a smoke test under tests/arch/arc/arc_nocache covering placement,
alignment, and .di-based cache-bypass validation on nsim_hs_mpuv6.
Signed-off-by: Rishav Chakraborty <reshav01@gmail.com>
Fix a compilation error when CONFIG_EXTRA_EXCEPTION_INFO is enabled
on platforms that do not define EXC_RETURN_STACK_FRAME_TYPE_Msk
(which is only defined for Cortex-M).
For Cortex-A/R devices we just default to the same code path as before
bce6f0de63
Signed-off-by: Alberto Escolar Piedras <alberto.escolar.piedras@nordicsemi.no>
PRIVILEGED_STACK_SIZE already has ARM64 defaults earlier in the
same Kconfig file.
The later unconditional default is never reached because the previous
unconditional default is the first matching default when FPU_SHARING is
disabled.
Remove the duplicate entry to avoid implying that the value is
overridden later.
Signed-off-by: Hongquan Li <hongquan.li@processmission.com>
Check the exc_return flags to determine if the FPU registers were actually
pushed onto the stack by the exception handler and only dump them if they
are used. Otherwise the stack locations for the FPU registers are invalid
and we're just dumping garbage.
Signed-off-by: Mike J. Chen <mjchen@google.com>
Add CONFIG_ARM_MPU_SKIP_ARCH_INIT for platforms where early boot code
(for example a TCM-resident stub) enables the MPU before z_arm_mpu_init()
runs. On Versal RPU split TCM/OCM images the kernel is linked in OCM while
reset code runs from TCM; z_arm_mpu_init() must not disable and reprogram
the MPU while the CPU is executing from an OCM-only region map.
When SCTLR.M is already set, z_arm_mpu_init() records the SoC static MPU
region count in static_regions_num and returns without touching the live
region table. That preserves the boot-time map (TCM, peripherals, OCM)
and keeps vector fetches at 0x0 executable after relocation.
SoCs that need this behavior select CONFIG_ARM_MPU_SKIP_ARCH_INIT from
their Kconfig; no Versal-specific logic is added under arch/.
Signed-off-by: Appana Durga Kedareswara rao <appana.durga.kedareswara.rao@amd.com>
Implement CONFIG_INIT_STACKS for the main stack when
CONFIG_MULTITHREADING is disabled. This keeps the config behavior
consistent between the main stack and the interrupt stack which is
memset in reset.S.
Signed-off-by: Josh DeWitt <josh.dewitt@garmin.com>
arch_user_mode_enter() pins the user entry point and its three
arguments into x0-x3 with local register variables, then executes an
eret into z_thread_entry() which expects them there. arch_irq_lock()
was called after that pinning. It is normally inlined, but when the
build disables inlining (code coverage adds -fno-inline) it becomes a
real function call placed between the register assignments and the
eret, clobbering the caller-saved x0-x3. The user thread then entered
with a garbage entry point and crashed.
Move arch_irq_lock() ahead of the register variable assignments so no
function call can sit between them and the eret. This is a latent
correctness fix independent of coverage.
Assisted-by: Claude:claude-opus-4-8
Signed-off-by: Anas Nashif <anas.nashif@intel.com>
Code built with COVERAGE_GCOV is instrumented to update per-arc
counters stored in the gcov BSS section. When userspace is enabled the
instrumented code also runs in user threads, so those counters must be
writable from EL0. The gcov BSS lives inside the zephyr_data region,
which is mapped kernel-only, and a user thread updating a counter took
a data abort.
Add a dedicated MMU flat range for the gcov area mapped P_RW_U_RW,
placed after zephyr_data so it overrides the kernel-only mapping for
that sub-range. This mirrors what x86 (gen_mmu.py) and the Cortex-M
MPU (arm_core_mpu.c) already do. The region is page aligned by the
linker, so no adjacent kernel data is exposed.
Assisted-by: Claude:claude-opus-4-8
Signed-off-by: Anas Nashif <anas.nashif@intel.com>
X86_VERY_EARLY_CONSOLE only works when
the compatible of the chosen zephyr,console is
ns16550. Add this add a dependency to the
Kconfig.
Signed-off-by: Fin Maaß <f.maass@vogl-electronic.com>
There can be platforms with a dcache or icache, but
without options for cache management. It that case
we should still be able to use the DCACHE_LINE_SIZE
without CACHE_MANAGEMENT if we want to use it to
align buffers for performance.
Signed-off-by: Fin Maaß <f.maass@vogl-electronic.com>
Adds LLEXT support for executing out of word granular
access instruction memory on Xtensa. This is achieved by
copying the text region onto the instruction heap
(assumed in instruction memory) but saving the
ELF address for use as the base address in
calculating relocation rel_addr (the location of the
relocation). Any rel_addr landing in the text region
is recalculated using the heap address instead.
Signed-off-by: Lauren Murphy <lauren.murphy@intel.com>
Signed-off-by: Anthony Giardina <anthony.giardina@intel.com>
Adds arch-level instruction memory API with functions
for memcpying / memsetting and determining whether a
region is in instruction memory. Implement API for
Xtensa and ESP32, with support for word granular access.
Adds definitions for instruction memory to
Xtensa ESP32 platforms.
Signed-off-by: Lauren Murphy <lauren.murphy@intel.com>
C++ extensions referencing a thread-local symbol such as errno emit
R_AARCH64_TLSLE_ADD_TPREL_HI12/_LO12_NC relocations and carry the
symbol in an SHF_TLS section. Loading such an extension currently
fails twice over: the .tbss section collides with .bss ("Multiple
SHT_NOBITS") and the relocation types are unknown to the arm64
relocator.
An extension has no TLS block of its own; it executes in one of the
loader's threads and shares that thread's TLS block. Skip SHF_TLS
sections in llext_map_sections() since they are never mapped as
runtime regions, and resolve the two local-exec relocations in the
arm64 relocator with tpoff = st_value + addend + TCB size, per the
AArch64 variant I TLS layout.
Add a C++ test extension (scenario llext.tls) that defines errno in
.tbss at the loader's offset and round-trips a value through the
loading thread's errno slot.
Signed-off-by: Jonathan E. Peace <jep@alphabetiq.com>
Since 5528dee556 (`device: Add asserts to DEVICE_API_GET`), the
following tests fail with a privileged-stack overflow during the
`log_panic()` syscall:
```
west twister -p hifive_unmatched/fu740/u74 -s logging.log_user
west twister -p hifive_unleashed/fu540/u54 -s logging.log_user
```
`qemu_riscv64` passes only because its defconfig sets
`CONFIG_PRIVILEGED_STACK_SIZE=2048`, lowering it to `1024` reproduces
the failure in a similar way.
The bad commit makes stack frames bigger along the `log_panic()` flush
chain. Peak privileged-stack usage goes from just-under to just-over
1024 bytes.
Signed-off-by: Adrian Śliwa <asliwa@internships.antmicro.com>
The ARM zImage header (header.S) ends with three words that follow
the Linux self-decompressor convention:
.long 0x016f2818 // Magic number
.long __rom_region_start // start address of zImage
.long __end // end address of zImage
A standard ARM zImage consumer (e.g. U-Boot bootz, or the Xen arm32
kernel loader) reads the third word and computes the on-disk file
size of the zImage as (__end - __rom_region_start). Linux establishes
the same invariant in arch/arm/boot/compressed/head.S, where the
analogous field is encoded as "_edata - start", i.e. the LMA of the
last byte of the image relative to its load address.
The current zimage_header.ld emits __end with:
KEEP(*(.image_header))
KEEP(*(.".image_header.*"))
__end = .;
zimage_header.ld is plugged into the linker script via the ROM_START
hook (see arch/arm/core/CMakeLists.txt) and runs immediately after
the 48-byte .image_header section is placed. At that point '.' is
still just past the header, so __end ends up only 0x30 bytes past
__rom_region_start regardless of how large the actual image is.
Every zImage built with CONFIG_ARM_ZIMAGE_HEADER=y therefore
advertises a size of 48 bytes in its header.
For Zephyr standalone this is invisible: the FVP / debugger loads
the whole file unconditionally and never consults the header. It
breaks any consumer that honours the header, though. On a Cortex-R52
FVP under Xen dom0less, the guest fails to boot:
(XEN) Loading zImage from 11000000 to 30000000-30000030
(XEN) CPU0: Unexpected Trap: Undefined Instruction
Xen copied only the 48 header bytes into the DomU and the guest
branched into uninitialised memory.
Fix it by computing __end the way Linux's head.S computes _edata:
take the LMA of the very last output section. Zephyr already exposes
that anchor as .last_section, and uses
LOADADDR(.last_section) + SIZEOF(.last_section)
elsewhere for the same purpose (e.g. _flash_used in
include/zephyr/arch/arm/cortex_a_r/scripts/linker.ld and the
equivalent cortex_m / arm64 / riscv linker scripts). The expression
is resolved lazily by the linker at final link time, so referring
to it from a ROM_START fragment that runs before .last_section is
emitted is safe.
After this change, a build of samples/hello_world for
fvp_baser_aemv8r/fvp_aemv8r_aarch32 with -DCONFIG_ARM_ZIMAGE_HEADER=y
produces a 27268-byte zephyr.bin whose header reads:
magic = 0x016f2818
start = 0x30000000
end = 0x30006a84 (end - start == file size)
`file(1)` now identifies it as "Linux kernel ARM boot executable
zImage", U-Boot bootz accepts it without complaint, and the same
binary boots cleanly as a Xen R52 dom0less DomU using Xen's standard
zImage loader path (no special payload-only handling required).
Signed-off-by: Ayan Kumar Halder <ayan.kumar.halder@amd.com>
Signed-off-by: Satya Sri <satyasri.katru@amd.com>
Fix the MCR instruction in switch.S to use the 'p15' coprocessor prefix
required by ARMv7-A toolchains.
Signed-off-by: Chidvilas Yerramsetti <cyerrams@qti.qualcomm.com>
z_riscv_fatal_error() may return: when a fatal error is handled (for
example an expected fault in ztest aborting the current thread), the
generic z_fatal_error() returns and the exception exit path in isr.S
takes care of rescheduling. isr.S explicitly sets the return address
to no_reschedule before tail-calling z_riscv_fault for this reason.
The CODE_UNREACHABLE hint made LLVM place a trapping instruction
(unimp) right after the call. When the handler returned, the CPU
executed the unimp and re-entered the fault path, so tests raising
expected faults hung in an endless fatal error loop when built with
clang. GCC builds only worked by chance, falling through into
whatever code the compiler laid out after the call.
Assisted-by: Claude:claude-opus-4-8
Signed-off-by: Anas Nashif <anas.nashif@intel.com>
Right now, the SBI hard-codes the usage of ld/sd instructions, which are
not available on 32-bit platforms. This commit changes this to sr/lr from
`asm_macros.inc`, which automatically resolves them to proper load/store
instructions on 32-bit and 64-bit configurations. It also ensures proper
timer handling through 32-bit registers.
Signed-off-by: Filip Kokosinski <fkokosinski@antmicro.com>
Co-authored-by: Jakub Klimczak <jklimczak@internships.antmicro.com>
Signed-off-by: Jakub Klimczak <jklimczak@internships.antmicro.com>
Add an AArch64 PMUv3 implementation behind CONFIG_ARM64_PMUV3 (pmuv3.c):
probe ID_AA64DFR0_EL1, calibrate CPU frequency (PMCCNTR_EL0 vs the
generic timer), and provide per-CPU counter configuration, enable/disable,
overflow handling, and cycle counter access. Initialization is explicit
via pmu_init() on each logical CPU that uses the PMU (no SYS_INIT).
Introduce include/zephyr/pmu.h for the portable pmu_*() API.
Architectural PMUv3 event codes (PMU_EVT_* in 0x00-0x1F) and
PMCR/PMUSERENR bit defines live in include/zephyr/arch/arm64/pmuv3.h for
AArch64 builds.
Add ARCH_HAS_PMU in arch/Kconfig (Cortex-A profiles select it); enable
CONFIG_ARM64_PMUV3 for the PMUv3 driver backend. Register access uses
explicit MRS/MSR inlines instead of read_sysreg()/write_sysreg()
statement expressions for static analysis.
Builds for versal_apu, versalnet_apu, and versal2_apu. On QEMU, PMU
access is often unavailable (-ENOTSUP). On Versal Net APU hardware with
PMU usable at the current EL, initialization succeeds.
Signed-off-by: Appana Durga Kedareswara rao <appana.durga.kedareswara.rao@amd.com>
The backtrace code was either calling the arch exception dump
hook or printk based on the value of its own config called
CONFIG_XTENSA_BACKTRACE_EXCEPTION_DUMP.
However, this made fault output inconsistent when CONFIG_LOG is enabled
because other output (like xtensa_dump_stack()) would use the macro
EXCEPTION_DUMP(), which would use LOG_ERR instead of printk. Fault output
would be a mix of log lines for register dumps and printk lines for
backtrace (no domains, timestamps, etc).
This commit changes backtrace to use the EXCEPTION_DUMP() macro so output
now is consistent. It also fixes an issue where if printk was going to
the logging subsystem (CONFIG_LOG_PRINTK=y), because the printk calls in
backtrace had no '\n' termination, the output would be lost in the log
format buffer.
Signed-off-by: Mike J. Chen <mjchen@google.com>
Use <> operator to include a Zephyr header file instead of "" that
is intended to local header files, not header files relative to
specifically defined search paths.
This change was made running the sed shell command below:
$ sed -i -E 's/#include "zephyr\/([^"]+)\.h"/#include <zephyr\/\1.h>/g' \
`grep -rsl "#include \"zephyr/" arch/`
Signed-off-by: Etienne Carriere <etienne.carriere@st.com>
Add RISCV_ISA_EXT_ZK (Scalar Cryptography) and RISCV_ISA_EXT_ZKS
(ShangMi Suite) Kconfig options, and append them to the GCC march
flag when enabled.
Signed-off-by: Liu Qian <liuqian.andy@picoheart.com>
CONFIG_XTENSA_SYSCALL_USE_HELPER currently only routes the 4/5/6-argument
syscall invocations through out-of-line helper functions. Toolchains that
reject inline-asm fixed-register operands in inlinable functions still
fail to build the 0-3 argument variants.
Add xtensa_syscall_helper_args_0..3 and route arch_syscall_invoke0..3
through them, gated behind a new opt-in
CONFIG_XTENSA_SYSCALL_USE_HELPER_ALL (default n, depends on
XTENSA_SYSCALL_USE_HELPER). This keeps the default behaviour unchanged -
the 0-3 argument syscalls stay on the inline fast path - and only routes
the low-argument variants through the helpers for toolchains that need
it, so there is no performance impact unless the option is explicitly
enabled. The inline-asm fast path is preserved under the #else branch.
Signed-off-by: Hiren Virapara <Hiren.Virapara@amd.com>
On ARM64, synchronous exceptions leave DAIF.I set independently of the
kernel IRQ lock. The generic arch_irq_unlock(key) fallback is therefore
insufficient to re-enable IRQ delivery during the coredump window.
Override arch_coredump_fatal_irq_unlock/lock to directly manipulate the
DAIF register: save current DAIF in the cookie, then clear DAIF.I so
that NIC TX-done interrupts can fire while the UDP coredump backend
drains its send queue. arch_coredump_fatal_irq_lock restores the saved
DAIF value and returns a fresh arch_irq_lock() key for the remainder of
z_fatal_error().
Signed-off-by: Appana Durga Kedareswara rao <appana.durga.kedareswara.rao@amd.com>
Setting symbol default value without 'configdefault', or without
explicit 'if' checks of dependencies, results in dependency weakening.
In this case, I/DCACHE_LINE_SIZE are redefined and set to the given
default value even if the dependency D/ICACHE is not enabled.
Signed-off-by: Abderrahmane JARMOUNI <git@jarmouni.me>
The arm_core_mpu_enable() and arm_core_mpu_disable() functions were
defined in the ARM MPU driver but never declared in a header.
As a result, callers had to provide their own local `extern`
declarations to use them.
Add documented prototypes for both functions to
include/zephyr/arch/arm/mpu/arm_mpu.h and drop the redundant `extern`
declarations in arm_core_mpu.c and the userspace memory protection
test.
Signed-off-by: Andrej Butok <andrey.butok@nxp.com>
Complier complains that idx_e may be used uninitialized.
So set the default as 0 so that the for loop where it can be
used uninitialized would be an no-op.
Signed-off-by: Daniel Leung <daniel.leung@intel.com>
Some issues found with previous region add mechanism:
() Previous mechanism cannot add new entry before the first
enabled entry.
() There are some cases where inserting a region would not
preserve access rights and memory type correctly.
This reworks the mechanism to fix these issue. Also remove
assertion within mpu_map_region_add() and let the caller
decide what to do.
Signed-off-by: Daniel Leung <daniel.leung@intel.com>
