arch_cpu_start() published the boot parameters through
sys_cache_data_flush_range(), which does nothing unless
CONFIG_CACHE_MANAGEMENT and CONFIG_DCACHE are both enabled: it returns
-ENOTSUP without touching the data cache, and the caller discarded that
value. A secondary core reads these parameters with the MMU and the data
cache disabled, so it could observe a stale mpid, sp, fn, arg or cpu_num.
Call arch_dcache_flush_range(), which is the operation the architecture
provides, and check the result. The flush completes with its own dsync
barrier, so nothing else is needed before the secondary core is
released. Include zephyr/arch/cache.h directly, because zephyr/cache.h
only pulls the arch cache interface in when a cache type is selected,
which is exactly the configuration that used to drop the clean.
The port turns the data cache on unconditionally as part of the MMU
setup, and the data cache maintenance operations are available
regardless of CONFIG_DCACHE.
Fixes#119663
Signed-off-by: Hongquan Li <hongquan.li@processmission.com>
z_arm64_el_highest_init() sets CNTFRQ_EL0 from
CONFIG_SYS_CLOCK_HW_CYCLES_PER_SEC. Since commit 864ea0ea3f ("kernel:
default SYS_CLOCK_HW_CYCLES_PER_SEC from timebase-frequency") that symbol
is only defined with SYS_CLOCK_EXISTS, so an arm64 build without a
system clock no longer compiles, for example samples/basic/minimal with
no-timers.conf on qemu_cortex_a53.
CNTFRQ_EL0 does not configure the counter: it only tells software the
counter frequency, and whatever runs at the highest EL is expected to
set it. The devicetree describes that frequency as timebase-frequency
whether or not the counter drives the system timer, so use it. Fall
back to the Kconfig value for boards that only provide that, and leave
the register alone when neither is known. That also avoids writing the
0 placeholder SYS_CLOCK_HW_CYCLES_PER_SEC defaults to with
TIMER_READS_ITS_FREQUENCY_AT_RUNTIME.
Boards that set the Kconfig value explicitly have no timebase-frequency,
and elsewhere the Kconfig default comes from timebase-frequency, so the
value written is unchanged wherever it was written before.
Fixes#121736
Signed-off-by: Nicolas Pitre <npitre@baylibre.com>
remap_mapping() splits a block when the range covers only part of it, and
ignored expand_to_table() returning NULL when the translation table pool
is empty. The attributes then landed on the whole block, granting the
domain access to everything the block maps rather than to the partition,
and the caller was told the partition had been added.
Return -ENOMEM instead and let it out through private_map(), which the
callers already propagate. privatize_page_range()'s error is passed on
the same way rather than merely asserted, now that there is somewhere for
it to go.
arch_mem_domain_thread_add() did not: it overwrote a map_thread_stack()
failure with the result of reset_map() and moved the thread to tables
where its stack is not mapped for user access. Return the error before
switching instead, leaving the thread on its old tables. Once the thread
runs on the new tables the move is done: if reset_map() fails then, the
old domain still maps the stack for user access, so revoke that in place,
which needs no new table since the private entries already cover the
stack.
Signed-off-by: Nicolas Pitre <npitre@baylibre.com>
globalize_table() asserts the source entry is a table whenever the range
stops in the middle of it. That holds when a range is globalized at the
granularity the kernel mapped it with, and not when a partition covers
only part of a range the kernel mapped as a block: reset_map() then walks
the partition's range against the kernel tables, finds a block spanning
more than the range, and the assertion fires on partition removal.
Split the source the way the destination is already split a few lines
down, which is also what set_mapping() does when a new mapping lands
inside an existing block. The mappings are unchanged, only the
granularity they are held at.
Signed-off-by: Nicolas Pitre <npitre@baylibre.com>
arch_mem_domain_partition_add() and map_thread_stack() both create the
domain private mapping with phys == virt, ignoring the entry the kernel
holds for that range. That assumes every partition is identity mapped
and always resident, and neither is guaranteed.
With demand paging, a range that is paged out when the partition is
added still gets a valid, access flagged, user permitted entry in the
domain, pointing at whatever frame sits at that physical address. The
kernel has already returned that frame to the free pool, so the domain
reads and writes memory that belongs to whoever gets it next, while the
kernel still considers the page evicted. Threads in the domain also miss
the page fault that would have brought the real contents back.
Apply the partition attributes to the entries the range already has
instead. Descriptor type, output address or paging location token, and
the access flag stay with the entry, so a paged out page stays paged
out and faults in normally, and a mapping whose virtual address is not
its physical address keeps pointing where it should. Ranges with
nothing mapped are left alone: access can only be granted to memory
that is actually mapped.
Signed-off-by: Nicolas Pitre <npitre@baylibre.com>
The descriptors are what tell a domain-private mapping apart from the
kernel entry it was built over: which bits came from the partition, which
were kept from the entry, and what the entry maps. Nothing in this file
exposes them, so a test can only observe the result indirectly, by
arranging an access and seeing whether it faults.
Add arm64_mmu_pte_get(), alongside the existing table accounting hooks,
walking a set of tables down to the leaf covering an address and
reporting it with its level.
Signed-off-by: Nicolas Pitre <npitre@baylibre.com>
Replace the open-coded tests of K_USER in a thread's user_options with
k_thread_is_user_thread(), and drop the private is_user() helper in
thread.c, which did the same thing. No functional change.
Signed-off-by: Anas Nashif <anas.nashif@intel.com>
The coredump freeze handshake in smp.c bounds its waits with
k_cycle_get_32() and k_ms_to_cyc_ceil32(), which need a system timer
and its frequency. Without SYS_CLOCK_EXISTS,
DEBUG_COREDUMP_SMP_FREEZE_CPUS still defaults to y on SMP and smp.c
fails to build.
Only select ARCH_SUPPORTS_COREDUMP_SMP_FREEZE when SYS_CLOCK_EXISTS is
set.
Should freezing ever be wanted without a system clock, the waits could
read the architectural counter directly instead: k_cycle_get_32() is a
CNTVCT_EL0 read on arm64, and that counter runs whether or not it
drives the system timer, with CNTFRQ_EL0 giving its rate. When
CNTFRQ_EL0 is unknown, the freeze wait would have to give up on the
other CPUs right away rather than wait forever on one with interrupts
masked.
Fixes#121737
Signed-off-by: Nicolas Pitre <npitre@baylibre.com>
Everything the primary core writes before the MMU is enabled bypasses
the data cache: the saved firmware arguments, the boot stack, .data,
.bss and then the translation tables built by z_arm64_mm_init(). Lines
a previous boot stage left over that RAM are untouched by those writes.
A clean one is read back in place of what was written once
enable_mmu_el1() turns on the MMU and data cache, and a dirty one can be
written back over it at any point before then. Either way the first
table walk or data access after the MMU enable may see garbage, which
shows up as a random crash right at that point.
Commit 6c6f1a5e99 ("arch: arm64: mmu: revert useless cache handling")
removed the invalidation setup_page_tables() used to do over these
ranges along with the per-mapping one, leaving nothing to cover this.
Invalidate the image RAM by VA in reset.S as soon as the primary core is
known, before the firmware arguments and the boot stack are written.
DC IVAC reaches caches beyond the cluster, which set/way operations do
not, and invalidating rather than cleaning avoids writing the stale
lines back. The boot vote before that point only writes
arm64_cpu_boot_params in .data, which the loader must already have left
clean, so lines over it can only be clean and are dropped as well.
Secondary cores must not do this: they start while the primary core
runs with its data cache on.
This does not depend on CONFIG_DCACHE: the data cache is enabled along
with the MMU either way.
Reproduced on fvp_corstone1000/a320 with cache state modelling enabled
and code standing in for a previous boot stage that leaves dirty lines
over .bss before jumping to Zephyr: the boot hangs without this change
and completes with it.
Signed-off-by: Nicolas Pitre <npitre@baylibre.com>
With the MMU off, a core's stack writes go straight to memory. Once
SCTLR_EL1.C is set, a stale cached line for the same address can be
read back instead, for example as a saved x19 or link register, and
the core faults during bring-up.
Invalidate the calling core's z_interrupt_stacks slot, found from sp,
right before the MMU is enabled. Other slots are left alone: they may
be live stacks of running cores, holding dirty lines. The arch cache
op is used, as sys_cache_data_invd_range() does nothing without
CONFIG_CACHE_MANAGEMENT.
Seen on RK3588 as a rare fault on a secondary core right after
z_arm64_mm_init().
Fixes#119630
Signed-off-by: Piter Zhang <zhangtdvip@163.com>
arch_mem_domain_partition_remove() ignored partition_id and rescanned
the whole domain, but the kernel zeroes partitions[p_idx].size only
after the arch hook returns, so the removed partition was programmed
right back into every started thread's MPU context: revocation of
dynamic memory domain partitions was silently ineffective.
Additionally, the partition loop in configure_dynamic_mpu_regions()
decremented the remaining-partition counter in the loop increment
expression, so it also ran for zero-sized holes in partitions[].
Each hole made the loop terminate one valid partition early, leaving
legitimately permitted memory unmapped and accesses to it faulting.
Exclude the removed partition from the rescan, and decrement the
remaining-partition counter only for non-empty partitions, matching
the arm32 implementation in arm_core_mpu.c.
Fixes: #115177
Signed-off-by: Hongquan Li <hongquan.li@processmission.com>
HCR_EL2 may be left configured by firmware/loaders before handing off
to Zephyr at EL2. z_arm64_el2_init() already clears TGE/AMO/IMO but not
FMO. With HCR_EL2.FMO=1, an EL1 write to ICC_SGI1R_EL1 is trapped to
EL2 before any SGI is generated. Zephyr runs at EL1 and installs no EL2
vectors, so the first IPI takes this trap with no handler in place and
crashes in firmware (observed on Phytium D3000M, where firmware leaves
FMO set on the primary CPU only).
Clear FMO alongside IMO/AMO so the ICC_SGI1R_EL1 access is not trapped
to EL2.
Signed-off-by: Chang Jianpeng <changjianpeng@kylinos.cn>
Use the existing custom-current architecture hooks to keep the running
thread pointer in TPIDR_EL1. A register read avoids the call and IRQ
masking needed to prevent migration between a CPU lookup and the
current-thread load.
Keep the per-CPU current pointer for cross-CPU readers. The scheduler
updates the register with preemption disabled, and reset initializes it
before the kernel installs the dummy thread on each CPU. TPIDR_EL0
remains available for TLS and TPIDRRO_EL0 for CPU and exception state.
Enable this for Cortex-A SMP without system power management.
Uniprocessor and PM builds keep the existing lookup; power-state
restoration is not extended to save the register.
A53 quad core microbenchmarks show ~1.2% higher throughput and up
to 4.7% lower latency.
Assisted-by: Codex:GPT-5
Signed-off-by: Aaron Wisner <aaronwisner@gmail.com>
Add a common hook declaration and Kconfig switch for firmware
arguments provided by architecture startup code.
The hook receives an architecture-provided argument array and count.
This keeps the common interface independent from register naming used
by a specific architecture.
On ARM64, preserve x0-x3 from the primary CPU reset path before
startup code reuses those registers. The saved values are stored only
after the primary CPU path is selected and are passed to the hook after
RAM has been initialized for C code. Secondary CPUs do not update the
saved storage.
Signed-off-by: Vladyslav Goncharuk <vladyslav_goncharuk@epam.com>
Assisted-by: Codex:gpt-5
Before z_arm64_mm_init() enables the MMU, all accesses are treated as
Device-nGnRnE, which the architecture requires to be naturally aligned.
This is independent of SCTLR_ELx.A: clearing the A bit relaxes alignment
checking for Normal memory only, it does not make unaligned Device
accesses legal.
Clang may lower struct assignments and small memory copies in this
library into 128-bit Advanced SIMD accesses (ldp/stp q, ldur/stur q).
When such an access is not 16-byte aligned and executes before the MMU
is enabled, it raises an alignment fault (data abort, ESR_ELx.DFSC
0b100001). As this happens before console initialisation, the failure
presents as a silent hang.
GCC already avoids this instruction class here via
-moverride=tune=no_ldp_stp_qregs, but there is no clang equivalent, and
clang rejects that argument. That option would also be insufficient on
its own, as it only suppresses ldp/stp pairs and not a single unaligned
128-bit access.
Apply -mstrict-align to this library when building with clang. The flag
is scoped with zephyr_library_compile_options() rather than
zephyr_cc_option() so that code running with the MMU enabled is not
restricted. Assembly is unaffected, so the FPU context save/restore in
fpu.S continues to use the Qn registers.
Signed-off-by: Shubhankar Kulkarni <shukul@qti.qualcomm.com>
ARM64_SAFE_EXCEPTION_STACK is user-selectable without
ARM64_STACK_PROTECTION, but thread->arch.stack_limit is only assigned
when the latter is on (arch_new_thread). With the safe exception stack
enabled standalone, current_stack_limit stays 0, so the overflow check
in z_arm64_quick_stack_check() can never trip: every EL1 exception pays
the entry overhead and the kernel stack overflow detection silently
does not exist.
Make ARM64_STACK_PROTECTION the only way in. No in-tree configuration
is affected: no AArch64 target selects ARM_MPU today, so
ARM64_SAFE_EXCEPTION_STACK was never enabled in any built config.
Fixes#118385
Signed-off-by: Hongquan Li <hongquan.li@processmission.com>
Assisted-by: Codex:GPT-5 gh
This changes the error handling of the call chain of both
arch_mem_map() and arch_mem_unmap() in ARM64 to return errors
if possible when actions fail, instead of causing kernel panic
or silently ignoring them. This allows the error code to be
propagated to the caller so it can be handled properly.
new_table() used to call k_panic() when running out of
translation tables, as its comment stated that many code paths
were not ready for failure. This is no longer the case since
expand_to_table(), dup_table() and set_mapping() all handle
the allocation failure already, so it simply returns NULL now.
Allocating the base translation table at boot keeps halting the
system as there is no way to recover from that.
del_mapping() needs to split a block mapping when only part of
it is being unmapped. It ignored the failure to allocate the
table for the split, and went on to clear the entry, which
unmapped the whole block instead of the requested range. It now
reports the failure and leaves the block alone, and this is
propagated through remove_map() and globalize_table().
sync_domains() only logged the failure to propagate mappings to
memory domains, so arch_mem_map() and arch_mem_unmap() could
report success while some domains were left inconsistent with
the kernel page tables. The error is now returned, though the
remaining domains are still synchronized first.
Domain synchronization and TLB invalidation are now performed
even if the operation has failed, since page tables may have
been modified before bailing out.
Assisted-by: Claude:claude-opus-5
Signed-off-by: Daniel Leung <daniel.leung@intel.com>
Mirror the arch_mem_map() change and give arch_mem_unmap() the same
treatment: change the prototype to return int instead of void, so
architecture code has a consistent way to report unmapping failures.
Assisted-by: Claude:claude-sonnet-5
Signed-off-by: Daniel Leung <daniel.leung@intel.com>
Previously arch_mem_map() was a void function, so architecture
code had no consistent way to report mapping failures to callers
other than panicking or asserting. Change arch_mem_map() to
return int.
Note that this mostly maintains the k_panic() calls unless
the error is easy to handle (like invalid arguments). Each
architecture will need to be amended in the future to properly
handle the error conditions. Also some of the users are
converted to do k_panic() after failure. This is to keep
the signature change commit easier to review.
Assisted-by: Claude:claude-sonnet-5
Signed-off-by: Daniel Leung <daniel.leung@intel.com>
arch_early_memset() only initialized the fill register x8 in the branch
taken when the destination is 8-byte aligned and the length is at least
8 bytes. Any call with a non-8-byte-aligned destination or a length < 8
reached the byte loop, which stores uninitialized register content
instead of the requested fill value. The C version this assembly
replaced in commit 696edbe841 ("arm64: move simple memcpy/memset
alternatives to assembly") handled all inputs correctly, so the
assembly port introduced the defect.
Rather than widening the fill value unconditionally, have the byte loop
store the low byte of the intact argument directly: x1 is preserved on
every path reaching the loop and strb already stores only the low byte,
so short and unaligned calls no longer pay for the 64-bit pattern
setup.
Fixes#118341
Suggested-by: Nicolas Pitre <npitre@baylibre.com>
Signed-off-by: Hongquan Li <hongquan.li@processmission.com>
Only modify CPACR_EL1 when it actually needs to be changed.
Read the current CPACR_EL1 value and compute the required FP/SVE access
permissions before writing it. Avoid the write and instruction barrier
when those permissions already match, both on exception entry and when
updating access before leaving an exception.
Preserve the existing access policy and execute the write and barrier
whenever permissions change. Keep the IRQ-disabled assertions and the
original lazy FPU ownership, context saving and migration behavior. This
reduces exception-path overhead even for integer-only execution.
Internal ARMv8 SMP microbenchmarks show 1.4-2.1% lower latency for this
change. Together with cached FPU-flush requests, the lazy-manager changes
measured 2.3% higher aggregate throughput and 3.8% lower aggregate latency
with integer hot paths held fixed. This patch's standalone throughput
effect was not isolated, and these are not fresh upstream-main results.
Assisted-by: Codex:GPT-5
Signed-off-by: Aaron Wisner <aaronwisner@gmail.com>
The alignment loop computes level_size with a 32-bit "1 <<" shift,
which is undefined behavior once the shift count reaches 32 (level 0
with 4KB granule, level 1 with 16KB/64KB granule). It also compares
int level against the unsigned BASE_XLAT_LEVEL, so with base level 0
the loop never terminates at level 0 and runs into negative levels,
leaving garbage in the returned alignment. Use 1ULL and cast the
bound to int.
The garbage alignment feeds virt_region_alloc(), which can
under-reserve the VA bitmap for large mappings.
Fixes#115139
Signed-off-by: Hongquan Li <hongquan.li@processmission.com>
A thread actively running on another CPU at panic time was previously
invisible to real debugging: coredump only captured the panicking CPU's
exception frame, so every other thread's registers came from its
k_thread.callee_saved struct -- correct for genuinely sleeping threads,
but stale (reflecting whenever it last voluntarily context switched) for
one that's actually running elsewhere right now.
Add CONFIG_DEBUG_COREDUMP_SMP_FREEZE_CPUS (default y where the arch sets
ARCH_SUPPORTS_COREDUMP_SMP_FREEZE, i.e. arm64 Cortex-A SMP). On panic,
coredump() freezes every other online CPU before dumping and thaws them
after the memory-region walk, and emits each frozen CPU's live registers
as a new coredump section (COREDUMP_CPU_SNAPSHOT_HDR_ID, tagged with CPU
index and thread pointer, reusing the arch-info block register layout).
The panicking CPU (never frozen, since freeze only targets other CPUs)
emits a zero-payload marker recording only which thread was panicking
there, which the host side uses to identify it unambiguously.
arm64 implementation (arch/arm64/core/smp.c, coredump.c): send a new SGI
to every other online CPU. Each handler captures its exact live state --
x0-x18/lr/spsr/elr via the arch_esf that _isr_wrapper() stashes at a
fixed, nesting-independent slot on its IRQ stack, plus x19-x29 via inline
asm (valid per AAPCS64) -- then spins holding whatever it held before
being frozen, untouched, until the dump finishes, and resumes via the
standard exception-return path exactly where it was interrupted. This
matters because a panic here can be recoverable (this tree's own
coredump_threads test doesn't halt/reboot after dumping).
The SGI uses a deliberately low priority (SGI_COREDUMP_FREEZE_PRIO), not
IRQ_DEFAULT_PRIORITY: a frozen CPU never returns from its handler so
_isr_wrapper() never EOIs, and per GIC priority rules only a strictly
higher-priority interrupt can preempt a still-active one. At the default
priority a frozen CPU could not service same-priority peripheral
interrupts (confirmed on hardware as Ethernet TX-done starvation during
the UDP backend); the lowest usable priority lets everything else keep
making progress while a CPU waits.
The freeze/thaw handshake uses two separate per-CPU signals: freeze_state
(IDLE/REQUESTED/CAPTURED) and a dedicated release_requested flag that
only thaw writes. Kept separate so a late-arriving CPU (whose SGI was
delayed past the freeze-side timeout) still sees a release signal instead
of stomping it with its own CAPTURED update and spinning forever; thaw
sets release_requested unconditionally for every other CPU. The handler
wait is also bounded by a 60-second wall-clock backstop (k_cycle_get_32(),
safe from any context) so no CPU is ever stuck indefinitely. A CPU that
never responds within the freeze-side timeout (never booted, or stuck
with IRQs masked) is skipped and never blocks the dump.
The host coredump log parser learns to parse the new snapshot section.
Signed-off-by: Appana Durga Kedareswara rao <appana.durga.kedareswara.rao@amd.com>
movw_reloc_handler() stored the relocation value in a 32-bit
variable, so the range check only saw the truncated value: 2^32 + 5
and the like aliased into range, and the instruction was silently
patched with a truncated immediate.
Widen the immediate to 64 bits so the single imm > UINT16_MAX
comparison covers both directions: the MOVN path stores the bitwise
inverse of the immediate, and ~ brings a negated value back under
UINT16_MAX exactly when the value is no smaller than
-(UINT16_MAX + 1).
Fixes#115707
Signed-off-by: Hongquan Li <hongquan.li@processmission.com>
imm_reloc_handler() tested its sign-extension check value (exactly 0
or -1 for values that fit the signed field) with (int64_t)x++ > 1,
which only rejects check values of 2 or more: a positive overflow by
one bit (check value 1) and every negative overflow pass, and the
instruction is then silently patched with a truncated immediate. The
post-increment is also dead: it compares the value before
incrementing and nothing reads x afterwards.
Reject everything but 0 and -1.
Fixes#115707
Signed-off-by: Hongquan Li <hongquan.li@processmission.com>
Remove the "zephyr,memory-region-mpu" property, deprecated in Zephyr 3.5
in favor of the "zephyr,memory-attr" bitmask.
The three BUILD_ASSERT tripwires in the ARM/ARM64 MPU drivers already
turned any remaining use of the property into a build failure, so they go
away with it, together with the now unused NODE_HAS_PROP_AND_OR helper.
The migration section in doc/services/mem_mgmt/index.rst documented a
property that no longer exists, so it is dropped and its value mapping
table moved to the 4.5 migration guide.
Signed-off-by: Benjamin Cabé <benjamin@zephyrproject.org>
Assisted-by: Claude:opus-5
ARMv9-A CPUs such as Cortex-A720 select ARMV9_A instead of ARMV8_A.
Allow Xen support to be enabled on these platforms.
Signed-off-by: Nam Nguyen <nam.nguyen.xn@renesas.com>
Drivers that need to touch an interrupt's latched pending state
currently reach into the interrupt controller themselves.
NVIC_ClearPendingIRQ() alone appears at 88 call sites across 50 files
under drivers/, NVIC_SetPendingIRQ() at 34 (26 of them in
drivers/counter/, which pends its own IRQ to fire an alarm whose
deadline has already passed) and NVIC_GetPendingIRQ() at 11. Each
site pulls cmsis_core.h and a CONFIG_CPU_CORTEX_M guard into
otherwise portable code, and several counter drivers had already
written private GIC-or-NVIC dispatch helpers for exactly these
operations.
Add k_irq_set_pending(), k_irq_clear_pending() and k_irq_is_pending()
as ALWAYS_INLINE wrappers over matching arch_irq_* functions, and
implement them for the ARM NVIC (AArch32 Cortex-M) and the GIC
(AArch32 Cortex-A/R and AArch64), where the GIC driver already
exported all three as arm_gic_irq_*.
The k_ prefix is deliberate: vendor HAL headers already declare
plain irq_set_pending()/irq_clear_pending() as their own functions
(the Realtek Ameba ROM ABI, pico-sdk's hardware/irq.h), and several
drivers carry private static helpers or API struct members with the
bare names. A namespaced API collides with none of them, needs no
macro tricks, and no renaming of existing code.
The operations are gated on a single capability symbol,
CONFIG_ARCH_HAS_IRQ_PENDING_OPS, so an unported target fails to
build instead of silently doing nothing. Both implemented backends
support all three operations; should an architecture with partial
support materialize (a RISC-V PLIC can report pending state but only
clears it by claiming, and cannot latch from software), the symbol
can be split then.
Assisted-by: Claude:claude-fable-5
Signed-off-by: Anas Nashif <anas.nashif@intel.com>
arch_mem_page_out()/arch_mem_page_in() and the AF/dirty sync paths call
sync_domains(), which globalizes the affected range into every memory
domain. globalize_table() replaced each destination leaf entry with the
global one, stripping the EL0 accessibility and nG attributes that
private_map() had set, so user threads lost access to their partitions
and stacks as soon as demand paging touched them.
Add a preserve_private mode to the globalization walk that keeps the
private permission attributes and only propagates the structural state
(descriptor type, physical address or location token, access flag). The
read-only bit is dirty-tracking state only where the mapping is
writable, so it is propagated only when the private entry marks the page
writable; a genuinely read-only private mapping keeps its RO bit even
after the kernel dirties the global entry. The guarded-page bit stays
on the private side as well, so BTI-guarded executable partitions
survive the sync.
The same paging updates also invalidate the TLB, but
invalidate_tlb_page() issues TLBI VAE1{IS} with the ASID field zero,
while domain page tables are allocated ASIDs from 1 up and their
entries are non-global: a stale translation survives under each
domain's ASID. For a page out that leaves a thread with a valid TLB
entry pointing at a frame the kernel may have handed to someone else;
for the dirty-tracking write the retried access faults again on a page
the kernel already made writable; and for the AF clear in
arch_page_info_get() the eviction algorithm stops seeing accesses and
NRU evicts hot pages. All callers are such paging paths, so switch the
implementation to TLBI VAAE1{IS}, which matches every ASID.
Signed-off-by: Hongquan Li <hongquan.li@processmission.com>
z_arm64_do_demand_paging() only accepted instruction/data aborts taken
from the current EL (EC 0x21/0x25). Aborts taken from EL0 (EC 0x20/0x24)
fell through to the fatal error path, so any user-mode access to a
paged-out page killed the thread instead of paging it back in.
Accept lower-EL aborts as well, and validate the access against the
faulting thread's own page tables: an abort from EL0 was raised against
the current domain's tables, so unless the leaf entry there grants EL0
the attempted access the fault is genuine and must not reach
k_mem_page_fault() or the AF/dirty handling on behalf of an
unprivileged thread. The validation walks the domain tables under
xlat_lock, as sync_domains() may free domain subtables on another CPU,
and requires more than AP_ELx: UXN must be clear for instruction
aborts, and PTE_SW_WRITABLE must be set for data writes, since the
dirty path judges writability from the kernel entry — without that
check a user write to a read-only partition page would still flip
ARCH_DATA_PAGE_DIRTY before the retry kills the thread.
Signed-off-by: Hongquan Li <hongquan.li@processmission.com>
The page table helpers run before the MMU is enabled, while the tables
are being built, and again at run time from arch_mem_map(). In the first
phase neither the logging subsystem nor printk()'s spinlock is usable,
because the atomics under them need the shareable memory attributes the
MMU provides, so a diagnostic there faults instead of printing. That is
where this started: a page table failure during early bring-up produced
silence.
Test the MMU and pick the path accordingly. mmu_is_enabled() reads
SCTLR_EL1 and tests M, which compiles to an mrs and a tbz. With the MMU
on the messages take the ordinary LOG_*() path, keeping level filtering,
timestamps, the backends and serialized output. With it off they degrade
to an unlocked printk() to the character hook.
Gating on the log core's own Z_LOG_CONST_LEVEL_CHECK() keeps the
build-time level filtering: a filtered-out level drops both branches and
its arguments. Verified on qemu_cortex_a53, where the INF format string
is in the binary with CONFIG_KERNEL_LOG_LEVEL_INF and gone with
CONFIG_KERNEL_LOG_LEVEL_ERR.
Errors here precede a k_panic() and must appear with logging compiled out
too, so they keep this file's existing printk() fallback.
Signed-off-by: Nicolas Pitre <npitre@baylibre.com>
z_arm64_sync_exc() demuxes SVC exceptions by the low 8 bits of the
SVC immediate (svc #1: IRQ offload, svc #2: runtime except, svc #3:
system call) without checking which exception level the SVC was
taken from. On ARM64, SVC from EL0 and EL1 both report ESR EC 0x15,
so hardware does not distinguish them.
With CONFIG_USERSPACE=y and CONFIG_IRQ_OFFLOAD=y, any user thread can
execute a raw svc #1 with x0/x1 pointing at an arbitrary kernel
function and argument; the handler then runs 'blr x1' at EL1, i.e.
privilege escalation / arbitrary code execution in kernel mode. The
RISC-V port avoids this class of bug by distinguishing ECALLU/ECALLM
via mcause.
Note that CONFIG_IRQ_OFFLOAD depends on TEST (arch/Kconfig), so this
is not reachable in a production build; only test builds that enable
irq_offload() are exposed. This distinction matters for backport
decisions.
Read the saved SPSR from the ESF and route any SVC taken from EL0 to
a user-only path that accepts just the system call and runtime-except
immediates. The runtime-except entry must stay reachable from EL0
because user threads raise stack-check failures and oopses through
it; z_arm64_do_kernel_oops() sanitizes the user-supplied reason, so
this is not a privilege boundary. All other EL0 SVCs raise a fatal
CPU exception. Kernel-mode SVCs keep their previous behavior, except
that svc #3 from EL1 is no longer dispatched to the syscall handler,
which kernel code never uses.
Fixes#116254
Signed-off-by: Hongquan Li <hongquan.li@processmission.com>
Expose the Xen xsm_op hypercall through the arm64 Xen hypercall
interface.
The arm64 Xen backend already keeps raw hypercall entry points in
hypercall.S and declares them in <zephyr/arch/arm64/hypercall.h>. Add
the missing xsm_op entry point there so callers that need Xen security
operations can issue the hypercall directly.
Signed-off-by: Vladyslav Goncharuk <vladyslav_goncharuk@epam.com>
Assisted-by: Codex:gpt-5.5
Expose the Xen vcpu_op hypercall through the arm64 Xen hypercall
interface.
The arm64 Xen backend already keeps raw hypercall entry points in
hypercall.S and declares them in <zephyr/arch/arm64/hypercall.h>. Add
the missing vcpu_op entry point there so callers that need Xen vCPU
operations can issue the hypercall directly.
Signed-off-by: Vladyslav Goncharuk <vladyslav_goncharuk@epam.com>
Assisted-by: Codex:gpt-5.5
The TLBI VAE1 address field is VA[55:12] regardless of the
translation granule (ARM ARM), but invalidate_tlb_page() shifted the
address by PAGE_SIZE_SHIFT. With 16 KB/64 KB pages the hardware then
invalidated a different entry and the intended page's translation
survived: paged-out frames stayed accessible through the stale TLB
entry, and the dirty-page write transition escalated into a fatal
fault loop.
Shift by 12 unconditionally, matching Linux's __TLBI_VADDR().
Fixes#115708
Signed-off-by: Hongquan Li <hongquan.li@processmission.com>
The round-robin ASID allocator in arch_mem_domain_init() could hand out
an ASID that was still in use by a live memory domain once the ASID
space wrapped. This broke TLB-based isolation because two live domains
would share the same ASID.
Fix by scanning domain_list before assigning an ASID and advancing the
round-robin counter until an unused ASID is found. If all ASIDs are in
use, return -ENOMEM.
Signed-off-by: Hongquan Li <hongquan.li@processmission.com>
arch_mem_domain_deinit() discards a domain's page tables but never
invalidated TLB entries tagged with that domain's ASID. ASIDs are
allocated round-robin (1..255), so after wrapping a new domain reuses
an old ASID. Stale TLB entries cached on any CPU that previously ran
threads in the old domain still match the reused ASID, allowing the
new domain's threads to access physical pages with the old domain's
permissions.
Add invalidate_tlb_all() after the page table teardown, matching the
pattern used by every other page-table mutation path (arch_mem_map,
arch_mem_unmap, private_map, etc.).
Fixes#113559
Signed-off-by: Hongquan Li <hongquan.li@processmission.com>
Select ARCH_SUPPORTS_COREDUMP_THREADS and ARCH_SUPPORTS_COREDUMP_STACK_PTR
for CPU_CORTEX_A in arch/arm64/core/Kconfig, enabling MEMORY_DUMP_THREADS
coredump mode on ARM64. Mirrors the existing Cortex-M declarations in
arch/Kconfig.
Implement arch_coredump_stack_ptr_get() using thread->callee_saved.sp_elx,
the EL1 stack pointer saved by the context-switch path for sleeping
threads. For the faulting thread the saved context is stale, so the
coredump falls back to dumping the full stack allocation,
which is always correct.
Signed-off-by: Appana Durga Kedareswara rao <appana.durga.kedareswara.rao@amd.com>
In `movw_reloc_handler()`, the MOVZ/MOVN opcode selection for signed
AArch64 MOVW relocations was inverted: positive relocation values were
encoded with MOVN (negate) and negative values with MOVZ (zero), causing
all signed MOVW relocations to produce wrong immediate values.
Swap the two branches so that `x >= 0` selects MOVZ and `x < 0` selects
MOVN with immediate inversion, matching the AArch64 instruction semantics
where MOVZ zero-extends and MOVN negates the immediate.
Fixeszephyrproject-rtos/zephyr#113383.
Signed-off-by: Hongquan Li <hongquan.li@processmission.com>
This PR migrates the MIT-licensed Xen public headers into the
`include/zephyr/xen/public` directory to the zephyr-xenlib module.
Since the xen public headers require GCC extensions,
enable GNU_C_EXTENSIONS along with enabling CONFIG_XEN.
Signed-off-by: TOKITA Hiroshi <tokita.hiroshi@gmail.com>
arch_buffer_validate() computes end = addr + size with a plain
64-bit add and never checks for unsigned wrap. When addr + size
wraps past 2^64, x1 < x0, so the blo abv_loop condition is false
after the first iteration — only the page containing addr is
validated, and the function returns 0 (success) for a range that
spans the entire address space including kernel memory.
Reject wrapping ranges immediately after computing the end
address, before entering the page loop. This mirrors how the
Xtensa MPU implementation (PR #109000) and ARM32 is_in_region()
(PR #23239) reject wrapping ranges.
Fixes#113800
Signed-off-by: Hongquan Li <hongquan.li@processmission.com>
The SMCCC macro in arch/arm64/core/smccc-call.S used x4 to load the
result pointer from the stack after the smc/hvc instruction. However,
per the SMCCC calling convention x4 holds the returned a4 value at that
point, so loading the pointer overwrote the real return value and
res->a4 ended up containing the result pointer itself.
Use x9 as the temporary register for the result pointer so that x4 is
preserved until it is stored into res->a4.
Fixes#113379
Signed-off-by: Hongquan Li <hongquan.li@processmission.com>
z_arm64_stack_corruption_check() computed the user-mode stack guard
page start by subtracting only Z_ARM64_STACK_GUARD_SIZE from
stack_info.start. However, the guard page is located below the
privileged stack area, so CONFIG_PRIVILEGED_STACK_SIZE must also be
subtracted. Without this, the computed guard_start falls inside the
privileged stack instead of the actual guard page, causing stack
overflow detection to miss faults in the guard region.
Fixes#113388
Signed-off-by: Hongquan Li <hongquan.li@processmission.com>
The architecture idle paths call sys_trace_idle() and
sys_trace_idle_exit() to notify subscribers when the CPU enters and
leaves the idle state. These calls were guarded by CONFIG_TRACING,
which tied idle-time accounting to the tracing subsystem even though
no tracing backend is required to service the hooks.
Introduce a hidden Kconfig symbol, SYS_IDLE_HOOKS, that any subsystem
needing these notifications can select, and have TRACING select it.
Switch the idle-path guards in every architecture and SoC that emits
the hooks from CONFIG_TRACING to CONFIG_SYS_IDLE_HOOKS. Because
TRACING selects the new symbol, existing tracing behaviour is
unchanged; the change only lets non-tracing consumers receive the
hooks.
This is a prerequisite for building the CPU load module without the
tracing subsystem.
Assisted-by: Claude:claude-opus-4-8
Signed-off-by: Anas Nashif <anas.nashif@intel.com>
MT_NORMAL_WT (Write-Through Normal memory) was defined in the MAIR
table but missing from the get_region_desc() switch. Mappings created
with K_MEM_CACHE_WT fell through to the implicit default, leaving
shareability unset (NON_SHAREABLE), PXN/UXN unset (executable), and
no GP bit for BTI.
Add MT_NORMAL_WT alongside MT_NORMAL so it inherits Inner Shareability
and the execute-never logic. Also fix the debug log to recognize
MT_NORMAL_WT as "MEM" instead of "DEV".
Fixes#113472
Signed-off-by: Hongquan Li <hongquan.li@processmission.com>
