PRIVILEGED_STACK_SIZE already has ARM64 defaults earlier in the
same Kconfig file.
The later unconditional default is never reached because the previous
unconditional default is the first matching default when FPU_SHARING is
disabled.
Remove the duplicate entry to avoid implying that the value is
overridden later.
Signed-off-by: Hongquan Li <hongquan.li@processmission.com>
arch_user_mode_enter() pins the user entry point and its three
arguments into x0-x3 with local register variables, then executes an
eret into z_thread_entry() which expects them there. arch_irq_lock()
was called after that pinning. It is normally inlined, but when the
build disables inlining (code coverage adds -fno-inline) it becomes a
real function call placed between the register assignments and the
eret, clobbering the caller-saved x0-x3. The user thread then entered
with a garbage entry point and crashed.
Move arch_irq_lock() ahead of the register variable assignments so no
function call can sit between them and the eret. This is a latent
correctness fix independent of coverage.
Assisted-by: Claude:claude-opus-4-8
Signed-off-by: Anas Nashif <anas.nashif@intel.com>
Code built with COVERAGE_GCOV is instrumented to update per-arc
counters stored in the gcov BSS section. When userspace is enabled the
instrumented code also runs in user threads, so those counters must be
writable from EL0. The gcov BSS lives inside the zephyr_data region,
which is mapped kernel-only, and a user thread updating a counter took
a data abort.
Add a dedicated MMU flat range for the gcov area mapped P_RW_U_RW,
placed after zephyr_data so it overrides the kernel-only mapping for
that sub-range. This mirrors what x86 (gen_mmu.py) and the Cortex-M
MPU (arm_core_mpu.c) already do. The region is page aligned by the
linker, so no adjacent kernel data is exposed.
Assisted-by: Claude:claude-opus-4-8
Signed-off-by: Anas Nashif <anas.nashif@intel.com>
C++ extensions referencing a thread-local symbol such as errno emit
R_AARCH64_TLSLE_ADD_TPREL_HI12/_LO12_NC relocations and carry the
symbol in an SHF_TLS section. Loading such an extension currently
fails twice over: the .tbss section collides with .bss ("Multiple
SHT_NOBITS") and the relocation types are unknown to the arm64
relocator.
An extension has no TLS block of its own; it executes in one of the
loader's threads and shares that thread's TLS block. Skip SHF_TLS
sections in llext_map_sections() since they are never mapped as
runtime regions, and resolve the two local-exec relocations in the
arm64 relocator with tpoff = st_value + addend + TCB size, per the
AArch64 variant I TLS layout.
Add a C++ test extension (scenario llext.tls) that defines errno in
.tbss at the loader's offset and round-trips a value through the
loading thread's errno slot.
Signed-off-by: Jonathan E. Peace <jep@alphabetiq.com>
Add an AArch64 PMUv3 implementation behind CONFIG_ARM64_PMUV3 (pmuv3.c):
probe ID_AA64DFR0_EL1, calibrate CPU frequency (PMCCNTR_EL0 vs the
generic timer), and provide per-CPU counter configuration, enable/disable,
overflow handling, and cycle counter access. Initialization is explicit
via pmu_init() on each logical CPU that uses the PMU (no SYS_INIT).
Introduce include/zephyr/pmu.h for the portable pmu_*() API.
Architectural PMUv3 event codes (PMU_EVT_* in 0x00-0x1F) and
PMCR/PMUSERENR bit defines live in include/zephyr/arch/arm64/pmuv3.h for
AArch64 builds.
Add ARCH_HAS_PMU in arch/Kconfig (Cortex-A profiles select it); enable
CONFIG_ARM64_PMUV3 for the PMUv3 driver backend. Register access uses
explicit MRS/MSR inlines instead of read_sysreg()/write_sysreg()
statement expressions for static analysis.
Builds for versal_apu, versalnet_apu, and versal2_apu. On QEMU, PMU
access is often unavailable (-ENOTSUP). On Versal Net APU hardware with
PMU usable at the current EL, initialization succeeds.
Signed-off-by: Appana Durga Kedareswara rao <appana.durga.kedareswara.rao@amd.com>
On ARM64, synchronous exceptions leave DAIF.I set independently of the
kernel IRQ lock. The generic arch_irq_unlock(key) fallback is therefore
insufficient to re-enable IRQ delivery during the coredump window.
Override arch_coredump_fatal_irq_unlock/lock to directly manipulate the
DAIF register: save current DAIF in the cookie, then clear DAIF.I so
that NIC TX-done interrupts can fire while the UDP coredump backend
drains its send queue. arch_coredump_fatal_irq_lock restores the saved
DAIF value and returns a fresh arch_irq_lock() key for the remainder of
z_fatal_error().
Signed-off-by: Appana Durga Kedareswara rao <appana.durga.kedareswara.rao@amd.com>
CONFIG_LINKER_USE_PINNED_SECTION is the second half of the selective
kernel-pinning model removed in issue #108773. With the kernel image
now always resident at boot (previous commit), the __pinned_*
attribute family is a no-op: every page they would have segregated is
already pinned by z_mem_manage_init()'s whole-image loop, so the
tagging contract neither adds safety nor remains maintainable.
Drop it.
Mechanical removals:
* All ~219 in-tree uses of __pinned_text, __pinned_rodata,
__pinned_data, __pinned_bss, __pinned_noinit, and __pinned_func
across arch/x86, drivers/interrupt_controller, drivers/timer,
arch/common, kernel, lib/libc, subsys/portability/posix, tests, and
the syscall code generator (scripts/build/gen_syscalls.py).
* The assembly aliases PINNED_TEXT/RODATA/DATA/BSS/NOINIT used in
arch/x86/core/ia32/*.S and drivers/interrupt_controller/
intc_loapic_spurious.S become plain TEXT/RODATA/DATA/BSS/NOINIT.
* K_KERNEL_PINNED_STACK_DEFINE, K_KERNEL_PINNED_STACK_ARRAY_DEFINE,
K_KERNEL_PINNED_STACK_ARRAY_DECLARE, K_THREAD_PINNED_STACK_DEFINE,
and K_THREAD_PINNED_STACK_ARRAY_DEFINE are removed. The few
in-tree callers (kernel/init.c, arch/arm/core/cortex_a_r/smp.c,
arch/arm64/core/fatal.c, arch/rx/core/prep_c.c,
arch/x86/core/prep_c.c, kernel/include/kernel_internal.h,
tests/bluetooth/hci_uart_async) move to the corresponding
non-pinned macros.
Machinery removals:
* Kconfig.zephyr drops CONFIG_LINKER_USE_PINNED_SECTION.
qemu_x86_tiny and qemu_x86_atom_virt drop their =y overrides.
* include/zephyr/linker/section_tags.h drops the __pinned_* macro
definitions (both arms). __isr collapses to an empty macro since
its only purpose was to alias __pinned_func.
* include/zephyr/linker/sections.h drops PINNED_TEXT_SECTION_NAME,
PINNED_BSS_SECTION_NAME, etc. and the bare PINNED_TEXT/RODATA/etc.
forwarders, plus the _APP_SMEM_PINNED_SECTION_NAME constant.
* include/zephyr/linker/linker-defs.h drops the lnkr_pinned_*
externs, the _app_smem_pinned_* externs, and the lnkr_is_pinned()
/ lnkr_is_region_pinned() inline helpers.
* include/zephyr/linker/utils.h drops the lnkr_pinned_rodata branch
in linker_is_in_rodata().
* include/zephyr/linker/app_smem_pinned{,_aligned,_unaligned}.ld
are deleted; cmake/linker/ld/target_configure.cmake stops
configuring them.
* boards/qemu/x86/qemu_x86_tiny.ld and
include/zephyr/arch/x86/ia32/linker.ld drop their pinned-section
blocks and the now-redundant #ifndef CONFIG_LINKER_USE_PINNED_SECTION
conditionals throughout the body. The
LIB_KERNEL_IN_SECT / LIB_ARCH_X86_IN_SECT / LIB_ZEPHYR_IN_SECT /
LIB_C_IN_SECT / LIB_DRIVERS_IN_SECT / LIB_SUBSYS_LOGGING_IN_SECT /
LIB_ZEPHYR_OBJECT_FILE_IN_SECT / ZEPHYR_KERNEL_FUNCS_IN_SECT macros
in qemu_x86_tiny.ld are deleted; they existed only to feed the
pinned text/rodata/data/bss/noinit sections.
* kernel/mmu.c drops the mark_linker_section_pinned(lnkr_pinned_start,
...) call. The mark_linker_section_pinned() helper survives but is
now gated only on CONFIG_LINKER_USE_BOOT_SECTION.
* arch/common/init.c and include/zephyr/arch/common/init.h drop
arch_bss_zero_pinned(); arch/x86/core/ia32/crt0.S drops the call
to it.
* arch/x86/core/userspace.c drops the eager k_mem_page_in() of the
thread's privileged stack on user-mode entry. With the kernel
image fully resident the stack is already mapped.
* arch/x86/gen_mmu.py drops map_region("lnkr_pinned") and the
set_region_perms() calls for lnkr_pinned_text / lnkr_pinned_rodata.
* CMakeLists.txt drops the LINKER_USE_PINNED_SECTION block that
generated APP_SMEM_PINNED_* variables and the
pinned_partitions target property feeding gen_app_partitions.py.
cmake/modules/extensions.cmake removes the PINNED_RODATA /
PINNED_RAM_SECTIONS / PINNED_DATA_SECTIONS zephyr_linker_sources()
location keywords and their snippet files.
scripts/build/gen_app_partitions.py drops --pinoutput /
--pinpartitions arguments and the pinned-output branch.
subsys/testsuite/coverage/CMakeLists.txt drops its
CONFIG_DEMAND_PAGING-conditional fork.
* scripts/build/gen_kobject_list.py drops the
app_smem_pinned_start / _end fallback for kobject placement
validation.
* tests/arch/x86/pagetables and tests/kernel/mem_protect/userspace
drop their lnkr_pinned_text / lnkr_pinned_rodata branches.
* include/zephyr/arch/x86/ia32/arch.h folds IRQSTUBS_TEXT_SECTION
to the unconditional ".text.irqstubs" form.
* tests/subsys/llext/src/syscalls_ext.c drops a stale comment about
syscalls landing in .pinned_text.
Targeted retentions:
* arch/x86/core/bootargs.c keeps multiboot_cmdline and efi_bootargs
in .noinit (was __pinned_noinit, which decayed to __noinit when
LINKER_USE_PINNED_SECTION was unset). The multiboot and zefi loader
paths write these buffers before Zephyr's BSS-zero step, so
zeroing them at boot loses the cmdline.
* arch/x86/core/ia32/fatal.c keeps _df_esf and _df_stack in .noinit.
They are scratch space written by the double-fault handler and have
no zero-init requirement; keeping them in .noinit also preserves
the historical post-noinit alignment that gen_mmu.py relies on
(z_mapped_size is computed before CMake-injected iterable sections
are appended to the linker script, so the post-noinit page padding
is what keeps those sections within the mapped region).
* include/zephyr/arch/x86/ia32/syscall.h and
include/zephyr/arch/x86/arch.h wrap the per-arch
arch_syscall_invoke* / arch_is_user_context / arch_k_cycle_get_*
implementations in @cond INTERNAL_HIDDEN. The public Doxygen
contract lives on the prototypes in
include/zephyr/arch/arch_interface.h; the per-arch implementations
are internal. Without this, removing the __pinned_func attribute
exposes the implementations to the doxygen-coverage delta check
as 10 newly-undocumented APIs.
Documentation updates are deferred to a separate commit.
Issue: #108773
Signed-off-by: Nicolas Pitre <npitre@baylibre.com>
kernel.h implies kernel_structs.h via kernel_includes.h, making
explicit inclusion of kernel_structs.h unnecessary whenever kernel.h
is already included in the same translation unit.
Remove the redundant includes across arch, boards, drivers, kernel,
lib, samples, subsys, and tests trees.
in include/zephyr/kernel_structs.h:
* 2. kernel.h shall imply kernel_structs.h, such that it shall not be
* necessary to include kernel_structs.h explicitly when kernel.h is
* included.
Signed-off-by: Anas Nashif <anas.nashif@intel.com>
The GIC register banks are accessed by the interrupt controller driver
through flat physical addresses before any driver can map them via the
device MMIO API, so they must be present in the page tables as soon as
the MMU is enabled. Until now every arm64 SoC had to repeat the same GIC
entries in its own mmu_regions.c to achieve this.
Map all the GIC reg banks in the arch core, generated from devicetree
and guarded by the presence of an arm,gic node, and provide a weak empty
mmu_config so a SoC that only needs the GIC no longer requires an
mmu_regions.c. Drop the now-redundant GIC entries from every arm64 SoC.
Signed-off-by: Carlo Caione <ccaione@baylibre.com>
Tuning CONFIG_MAX_XLAT_TABLES is currently trial-and-error: the only
feedback on overflow is a "too small" panic with no hint of how much
to bump.
Track the high-water mark of allocated translation tables and use it
to emit two signals from new_table(): a one-shot LOG_WRN when the
pool drops below 12.5 % free (always compiled in, advance notice
before the panic), and an opt-in LOG_INF on every new peak (under
CONFIG_ARM64_MMU_REPORT_XLAT_TABLES_USAGE) so the last logged "peak
N of M allocated" line gives a concrete lower bound on MAX for the
workload that ran.
Signed-off-by: Carlo Caione <ccaione@baylibre.com>
Add new implementations for entropy driver and random subsystem
based on ARM64 RNDRRS and RNDR instructions.
Signed-off-by: Christoph Busold <cbusold@qti.qualcomm.com>
Modern ARM64 SoCs with SMP and 36-bit virtual addressing require more
translation tables than the current default of 8. This is particularly
evident on TI K3 SoCs (AM62X, AM62LX) where there are more SoC
peripherals which are beyond the 2MB boundary.
Add a new default: 16 tables for SMP && (ARM64_VA_BITS >= 36)
Signed-off-by: Soumya Tripathy <s-tripathy@ti.com>
arch_mem_page_in() used to clear the access flag on paged-in pages so
the first access after page-in would fault and feed the tracking hook.
This predates the LRU eviction algorithm and is no longer carrying its
weight: the demand-paging core adds a freshly paged-in frame at the
tail of the LRU queue via k_mem_paging_eviction_add(), so the first-
access AF fault ends up moving the frame from tail to tail — wasted
work.
Measured on qemu_cortex_a53 with the mem_map demand_paging test:
total page faults, eviction counts, and new-head mark counts are
byte-identical before and after this change. Only the tracking-hit
count drops (308 -> 136), which exactly corresponds to the redundant
first-access minor faults we no longer take.
The new-head path in lru_pf_remove() still clears AF on the page at
the head of the queue to detect whether it is actually stale before
eviction — that mechanism is unchanged.
Signed-off-by: Nicolas Pitre <npitre@baylibre.com>
When a Data Abort or Instruction Abort exception includes a valid FAR,
walk the current TTBR0 page tables and display the PTE at each level.
This shows the actual page table content at the time of the fault,
which is invaluable for diagnosing permission faults where the PTE
may be correct (pointing to a TLB coherency issue) versus cases
where the PTE itself has wrong permissions.
Signed-off-by: Nicolas Pitre <npitre@baylibre.com>
Add compile-time macros to auto-generate ARM64 MMU region entries
from devicetree nodes with zephyr,memory-attr property:
- DT_MEM_ATTR_TO_MT(): converts DT memory attributes to ARM64 MT_*
flags at compile time using the composable-bitmask design — the
generic DT_MEM_CACHEABLE bit selects cacheable vs non-cacheable,
and the arch-specific ATTR_ARM64_CACHE_WB sub-bit selects
write-back vs write-through when cacheable
- MMU_REGION_DT_FLAT_ENTRY_FROM_DT(): generates a single MMU region
entry from a DT node, skipping nodes without zephyr,memory-attr
- MMU_REGION_DT_COMPAT_FOREACH_FLAT_ENTRY_FROM_DT(): iterates over
all status="okay" nodes of a given compatible
The convenience macros in memory-attr-arm64.h are defined as
composable combinations of generic and arch-specific attributes
rather than standalone one-hot enumerations:
DT_MEM_ARM64_MMU_NORMAL_NC = (0)
DT_MEM_ARM64_MMU_NORMAL_WT = (DT_MEM_CACHEABLE)
DT_MEM_ARM64_MMU_NORMAL = (DT_MEM_CACHEABLE | ATTR_ARM64_CACHE_WB)
Only Normal memory types are supported by the DT auto-mapping
mechanism. Device memory must be mapped through the MMIO device
API instead. Unsupported zephyr,memory-attr values are rejected
at compile time via BUILD_ASSERT.
The arch-level mmu.c automatically picks up all zephyr,memory-region
nodes with zephyr,memory-attr at boot, so individual SoC
mmu_regions.c files do not need any changes.
The DT region iteration is factored into max_region_bounds() and
map_mmu_regions() helpers to avoid code duplication in
setup_page_tables().
Introduces include/zephyr/dt-bindings/memory-attr/memory-attr-arm64.h
with ARM64-specific DT memory attribute sub-bits.
Add an ARM64 Developer Guide (doc/hardware/arch/arm64.rst)
documenting the DT-based MMU region mapping feature.
Signed-off-by: Appana Durga Kedareswara rao <appana.durga.kedareswara.rao@amd.com>
Since commit 0026a5610ac ("arm64: mm: use identity mapping for device
MMIO"), device_map() creates identity mappings (VA = PA) instead of
allocating virtual addresses from a contiguous pool. Each device at a
distinct 2MB-aligned physical address now requires its own L3 page
table, increasing the total number of translation tables needed.
Bump the USERSPACE && TEST default from 24 to 28 to accommodate the
additional tables required by identity-mapped device MMIO.
Signed-off-by: Carlo Caione <ccaione@baylibre.com>
The ARM64 coredump arch block did not include FP (x29) and SP,
making GDB unable to unwind the stack. The GDB stub also
misinterpreted SPSR as SP (tu[20] mapped to SP_EL0), producing
corrupted stack pointer values and broken backtraces.
Bump the arch block version to v2 (24 registers, 192 bytes)
adding FP and SP after the existing 22 registers. Update the
GDB stub to auto-detect v1 vs v2 blocks by payload size and
correctly map SPSR (skip), ELR (PC), FP (x29), and SP.
When CONFIG_ARM64_SAFE_EXCEPTION_STACK is enabled and the
exception originated from EL0, use the saved esf->sp (original
sp_el0 stored by the exception entry code) instead of computing
it from the ESF address, since the exception handler may be
running on a separate safe stack.
Fixes#99054
Signed-off-by: Anirudha Sarangi <anirudha.sarangi@amd.com>
Signed-off-by: Appana Durga Kedareswara rao <appana.durga.kedareswara.rao@amd.com>
On ARM64, Zephyr uses identity mappings (VA = PA) for kernel code, data and
boot-time device regions. The MMU fully supports address translation but
Zephyr uses it primarily for access permission enforcement.
There are currently two independent paths for mapping device MMIO regions:
1. SoC-level mmu_regions.c files use MMU_REGION_FLAT_ENTRY() to create
identity mappings (VA = PA) directly in the page tables at boot. This
bypasses the kernel's virtual memory tracking entirely. SoC maintainers
must manually list peripherals in mmu_regions.c for drivers that do not
use the device MMIO API (e.g. most existing drivers) or cannot use it
(e.g. the GIC, which is not a regular driver).
2. The device MMIO API (device_map()) goes through k_mem_map_phys_bare(),
which allocates a virtual address from the SRAM range and maps (VA !=
PA) device registers there. Mapping device MMIO into the SRAM virtual
address space is nonsensical: it conflates device registers with memory,
wastes virtual address pool space, and produces addresses that bear no
relation to the hardware.
The CONFIG_KERNEL_DIRECT_MAP mechanism already supports identity mapping
through k_mem_map_phys_bare() with the K_MEM_DIRECT_MAP flag, but it
requires each board defconfig to enable the Kconfig and each driver to
explicitly pass the flag.
Make identity-mapped device MMIO automatic on ARM64:
1. ARM64 CPU_CORTEX_A selects KERNEL_DIRECT_MAP when MMU is enabled. This
eliminates the need for per-board defconfig opt-in.
2. device_map() automatically injects K_MEM_DIRECT_MAP when
CONFIG_KERNEL_DIRECT_MAP is enabled. This is transparent to
drivers so no per-driver changes needed. The flag is gated on
CONFIG_KERNEL_DIRECT_MAP rather than CONFIG_ARM64, keeping it
architecture-agnostic.
Signed-off-by: Carlo Caione <ccaione@baylibre.com>
Implement thread-based unwinding to support the 'kernel thread unwind'
shell command on ARM64. This update ensures that 'thread' defaults to
'_current' when NULL, complying with the arch_stack_walk() API contract.
To enhance security, add stack bounds validation using stack_info,
and TLS pointers of the target thread. If these are not available, the
logic correctly falls back to is_address_mapped() to ensure robustness
during the unwinding process.
Signed-off-by: Archilis Wang <awm02289@gmail.com>
Add ARM64_PAGE_SIZE Kconfig choice allowing 4KB, 16KB and 64KB
page sizes. The MMU code already derived all constants from
PAGE_SIZE_SHIFT so most of the infrastructure was ready.
Changes:
- Add ARM64_PAGE_SIZE choice (4KB default, 16KB, 64KB) in Kconfig
- Derive PAGE_SIZE_SHIFT from CONFIG_MMU_PAGE_SIZE in mmu.h
- Select proper TCR granule bits (TG0/TG1) per page size in mmu.c
- Round ARCH_THREAD_STACK_RESERVED up to page alignment so that
the user-accessible stack buffer starts on a page boundary
- Fix MEM_REGION_ALLOC in mem_protect test to use CONFIG_MMU_PAGE_SIZE
Signed-off-by: Nicolas Pitre <npitre@baylibre.com>
arch_secondary_cpu_init() never returns (it ends with fn(arg) into the
scheduler) but its definition lacks FUNC_NORETURN. The compiler
generates a PACIASP/AUTIASP pair and turns the final fn(arg) into a
tail-call: AUTIASP followed by BR. The AUTIASP causes a PAC
authentication failure (FPAC exception) on secondary CPUs.
Fix by marking the definition FUNC_NORETURN with CODE_UNREACHABLE,
matching the extern declaration. The compiler then generates a plain
BLR without the AUTIASP epilogue.
Also fix the function signature to take no arguments, matching the
extern declaration and actual call sites, and move both
arch_secondary_cpu_init() and z_arm64_mm_init() declarations into
boot.h instead of scattering extern declarations across source files.
Also remove the dead arch_cache_init() call in z_arm64_secondary_prep_c()
that was placed after the noreturn call and could never execute. It is
absent from the primary CPU path in z_prep_c() and the implementation
is empty on arm64 anyway.
Signed-off-by: Nicolas Pitre <npitre@baylibre.com>
The ARM Architecture Reference Manual (DDI 0487) requires a context
synchronization event (ISB) between modifying SVE trap control registers
(CPTR_EL3.EZ, CPTR_EL2.TZ, CPACR_EL1.ZEN) and accessing the
corresponding ZCR_ELx registers: "The effect of the change is guaranteed
to be observable only after a Context synchronization event."
Without the ISB, the processor may still observe the old trap
configuration and generate an UNDEFINED exception on the ZCR write.
This also fixes the EL2 SVE initialization for non-VHE mode
(HCR_EL2.E2H=0): CPTR_EL2 bits [17:16] (ZEN) are RES0 in non-VHE
format and must not be set. SVE trapping at EL2 in non-VHE mode is
controlled by the TZ bit (bit 8) instead. The previous code wrote the
VHE-format ZEN bits which is architecturally UNPREDICTABLE in non-VHE
mode. Match the Linux kernel sequence (arch/arm64/include/asm/el2_setup.h).
Signed-off-by: Nicolas Pitre <npitre@baylibre.com>
ASM is notoriously harder to maintain than C and requires core specific
adaptation which impairs even more the readability of the code.
There's a bug in current arch_cpu_atomic_idle asm version:
tst x0, #(DAIF_IRQ_BIT) //here Z := (DAIF_IRQ_BIT == 0)
beq _irq_disabled //jump to _irq_disabled when Z is set
msr daifclr, #(DAIFCLR_IRQ_BIT)
_irq_disabled:
ret
As can be seen, the asm code jumps to _irq_disabled when Z is set, but
per aarch64 architecture reference, DAIF_IRQ == 0 means the IRQ is
unmasked, I.E enabled. So the asm logic here is wrong. I fixed this bug
in C version. This shows the benefit of ASM -> C
As for performance concern, except the bug fix above, there's no
difference of generated code between ASM and C version.
ASM version:
<arch_cpu_idle>:
d5033f9f dsb sy
d503207f wfi
d50342ff msr daifclr, #0x2
d65f03c0 ret
arch_cpu_atomic_idle>:
d50342df msr daifset, #0x2
d5033fdf isb
d503205f wfe
f279001f tst x0, #0x80
54000040 b.eq 1001d10 <_irq_disabled> // b.none
d50342ff msr daifclr, #0x2
_irq_disabled>:
d65f03c0 ret
C version:
<arch_cpu_idle>:
d5033f9f dsb sy
d503207f wfi
d50342ff msr daifclr, #0x2
d65f03c0 ret
<arch_cpu_atomic_idle>:
d50342df msr daifset, #0x2
d5033fdf isb
d503205f wfe
37380040 tbnz w0, #7, 1001d0c <arch_cpu_atomic_idle+0x14>
d50342ff msr daifclr, #0x2
d65f03c0 ret
And as can be seen, C version use the tbnz instruction to test bit and
branch. Unlike TST, TBNZ does not affect the Z, N, C, or V flags in the
processor state. So except the bug fix, C version looks a bit better
than asm version.
Other architectures such as x86, riscv, rx, xtensa, mips and even arm
cortex_m also use c version for cpu_idle, it's safe for ASM -> C.
Signed-off-by: Jisheng Zhang <jszhang@kernel.org>
Static MMU region entries populated via
MMU_REGION_DT_COMPAT_FOREACH_FLAT_ENTRY() pass raw DTS reg address and
size values to __add_map(), which asserts page-alignment. DTS nodes may
legitimately have non-page-aligned reg sizes reflecting actual hardware
register footprints, causing an assert crash during early boot when
CONFIG_ASSERT=y.
Align the base address down and size up to CONFIG_MMU_PAGE_SIZE in
add_arm_mmu_region(), mirroring the k_mem_region_align() logic already
used by the dynamic DEVICE_MMIO_MAP path in kernel/mmu.c. This ensures
all static platform MMU region entries are mapped with page-granular
parameters regardless of DTS reg values.
Signed-off-by: Appana Durga Kedareswara rao <appana.durga.kedareswara.rao@amd.com>
Setting SCR_ST_BIT actually traps CNTPS access to EL3, opposite
to what the comment says. Remove to allow secure EL1 access.
Also initialize CNTPS_CVAL_EL1 to prevent spurious interrupts.
Signed-off-by: Joakim Tjernlund <joakim.tjernlund@infinera.com>
Co-authored-by: Sudan Landge <sudan.landge@arm.com>
Implement arch_mem_domain_deinit() for ARM64 to release page tables
back to the pool when a memory domain is de-initialized. This reuses
the existing discard_table() mechanism to recursively free all
sub-tables in the hierarchy.
Signed-off-by: Nicolas Pitre <npitre@baylibre.com>
gdb cannot unwind the stack from exceptions. This adds
CFI annotations to help gdb unwind.
Signed-off-by: Joakim Tjernlund <joakim.tjernlund@infinera.com>
zeroing CNTHCTL_EL2 traps physical timer/counter access from EL1 to EL2,
but Zephyr has no hypervisor to handle those traps.
Enabling access is the standard EL2→EL1 drop behavior.
Signed-off-by: Joakim Tjernlund <joakim.tjernlund@infinera.com>
Add support for ARMv8.5+ Branch Target Identification to protect against
Jump-Oriented Programming (JOP) attacks. This complements PAC to offer
complete protection against both ROP and JOP attacks, ensuring
comprehensive control flow integrity.
Signed-off-by: Nicolas Pitre <npitre@baylibre.com>
Add support for ARMv8.3+ Pointer Authentication to protect against
Return-Oriented Programming (ROP) attacks. This implementation provides
PAC functionality with per-thread key isolation, secure key management,
and integration with Zephyr's thread model.
Signed-off-by: Nicolas Pitre <npitre@baylibre.com>
The Xen events channel driver consume 72K of RAM, but may not be
required in all use cases.
Added a XEN_EVENTS Kconfig option so that Xen events can be gracefully
disabled if not required. Updated the relevant CMakeLists.txt and
Kconfig files to guard the inclusion of the Xen events driver and its
source files by this option.
Signed-off-by: Grygorii Strashko <grygorii_strashko@epam.com>
Signed-off-by: Svitlana Drozd <svitlana_drozd@epam.com>
The arch_float_disable() function was not clearing the K_FP_REGS flag
from thread->base.user_options after disabling FPU access. This caused
the float_disable test to fail as it verifies the flag is properly
cleared after FPU disable.
Signed-off-by: Appana Durga Kedareswara rao <appana.durga.kedareswara.rao@amd.com>
After commit 02770ad963 ("debug: EXCEPTION_STACK_TRACE should depend
on arch Kconfigs"), the ARM64_EXCEPTION_STACK_TRACE isn't used any more,
remove it.
Signed-off-by: Jisheng Zhang <jszhang@kernel.org>
soc_per_core_init_hook() is usually called from arch_kernel_init() and
arch_secondary_cpu_init() which are C functions. As such, there is no need
to check for CONFIG_SOC_PER_CORE_INIT_HOOK since platform/hooks.h provides
a no-op function-like macro implementation if the Kconfig option is not
enabled.
Remove the Kconfig option check from all files.
Signed-off-by: Mathieu Choplain <mathieu.choplain-ext@st.com>
Move ARCH_HAS_STACKWALK under CPU_CORTEX_A section since only Cortex-A
implements arch_stack_walk(), while Cortex-R does not.
Signed-off-by: Sudan Landge <sudan.landge@arm.com>
When CONFIG_MAX_XLAT_TABLES is too small and new_table() cannot allocate
a translation table, the system must halt rather than continue with
undefined behavior.
This change ensures k_panic() is called after reporting the error,
preventing the system from proceeding when it runs out of translation
tables. Additionally, adds printk() fallback for configurations where
CONFIG_LOG is disabled to ensure the error is always visible.
Signed-off-by: Appana Durga Kedareswara rao <appana.durga.kedareswara.rao@amd.com>
Add a new Kconfig option CONFIG_XEN_SYSCTL_INTERFACE_VERSION that allows
to change the version of the Sysctl interface used by Zephyr to issue
sysctl hypercalls.
For now versions 0x15 is supported.
Signed-off-by: Mykyta Poturai <mykyta_poturai@epam.com>
Signed-off-by: Dmytro Firsov <dmytro_firsov@epam.com>
Add a new Kconfig option CONFIG_XEN_DOMCTL_INTERFACE_VERSION that allows
to change the version of the Domctl interface used by Zephyr to issue
domctl hypercalls. Add compile-time checks to enable or disable certain
Domctl operations based on the selected Domctl interface version.
For now versions 0x15, 0x16, and 0x17 are supported.
Also it required to correctly guard domctl call that were not supported
prior to specified version.
Signed-off-by: Mykyta Poturai <mykyta_poturai@epam.com>
Signed-off-by: Dmytro Firsov <dmytro_firsov@epam.com>
This hypercall can be used get some information about physical machine
and running guests:
- sysctl hypercall "xen_sysctl_getphysinfo" allows read information about
physical machine: number CPUs, memory sizes, hardware capabilities, etc.
- sysctl hypercall "xen_sysctl_getdomaininfolist" returns array of domain
info structures that provide information about particular domain(s).
Signed-off-by: Dmytro Semenets <dmytro_semenets@epam.com>
Signed-off-by: Mykyta Poturai <mykyta_poturai@epam.com>
Signed-off-by: Dmytro Firsov <dmytro_firsov@epam.com>
Xen API contains hypercall, which allows domains to identify Xen
version, that is currently used on the system. It can be used to check
if current version is supported by Zephyr or to change behavior of the
drivers or services.
Signed-off-by: Dmytro Semenets <dmytro_semenets@epam.com>
Signed-off-by: Dmytro Firsov <dmytro_firsov@epam.com>
Add wrappers for following XEN_DMOP_* hypercalls.
These enables Xen device model control path:
dm_op provides operations to create/manage the ioreq server
so guest MMIO accesses are trapped and handled by the hypervisor.
These are guarded by CONFIG_XEN_DMOP.
- dmop
- dmop_create_ioreq_server
XEN_DMOP_create_ioreq_server
- dmop_map_io_range_to_ioreq_server
XEN_DMOP_map_io_range_to_ioreq_server
- dmop_set_ioreq_server_state
XEN_DMOP_set_ioreq_server_state
- dmop_nr_vcpus
XEN_DMOP_nr_vcpus
- dmop_set_irq_level:
XEN_DMOP_set_irq_level
Signed-off-by: TOKITA Hiroshi <tokita.hiroshi@gmail.com>
Memory protection and userspace tests require more MMU translation
tables than the default. Without this increase, tests fail with:
E: CONFIG_MAX_XLAT_TABLES too small
ASSERTION FAIL [ret == 0] @ arch/arm64/core/mmu.c:1244
privatize_page_range() returned -12
Increase defaults when both USERSPACE and TEST are enabled:
- 32 tables for SMP configurations
- 24 tables for non-SMP configurations
This fixes:
- sample.kernel.memory_protection.shared_mem (all platforms)
- rtio.api.userspace (v8a, v9a)
- rtio.api.userspace.submit_sem (v8a, v9a)
- portability.posix.common.userspace
Consequently the demand paging test needed adjustment to its
qemu_cortex_a53 configs to keep working as this test is highly
sensitive to the amount of available free memory.
Signed-off-by: Nicolas Pitre <npitre@baylibre.com>
Increase ARM64 stack sizes to accommodate deeper call stacks in
userspace and SMP configurations when FPU_SHARING is enabled:
- PRIVILEGED_STACK_SIZE: 1024 → 4096 bytes (with FPU_SHARING)
- TEST_EXTRA_STACK_SIZE: 2048 → 4096 bytes (with FPU_SHARING)
The default 1KB privileged stack is insufficient for ARM64 userspace
syscalls when FPU context switching is enabled.
Symptom: Userspace tests crash with Data Abort (EC 0x24) near stack
boundaries during syscalls, particularly on SMP configurations where
multiple threads exercise FPU lazy switching.
Fixes previously failing CI test on fvp_base_revc_2xaem SMP variants:
- kernel.threads.dynamic
- Multiple userspace tests with FPU_SHARING enabled
Signed-off-by: Nicolas Pitre <npitre@baylibre.com>
Implement Scalable Vector Extension (SVE) context switching support,
enabling threads to use SVE and SVE2 instructions with lazy context
preservation across task switches.
The implementation is incremental: if only FPU instructions are used
then only the NEON access is granted and preserved to minimize context
switching overhead. If SVE is used then the NEON context is upgraded to
SVE and then full SVE access is granted and preserved from that point
onwards.
Signed-off-by: Nicolas Pitre <npitre@baylibre.com>
