Remove remaining uses of the internal __ASSERT_ON macro. Let __ASSERT()
handle disabled assertions, mark assert-only values as unused where needed,
and use CONFIG_ASSERT for assertion-only state.
Signed-off-by: Måns Ansgariusson <mansgariusson@gmail.com>
intc_system_apic.c is not an interrupt controller driver: it is the
x86 platform glue that implements arch_irq_enable(),
arch_irq_disable() and the vector programming hook by dispatching
between the IOAPIC and LOAPIC drivers, which expose their own APIs.
Interrupt controller drivers should only expose their own namespaced
API, so move the file to arch/x86/core where the glue belongs,
unchanged apart from a comment stating its role.
Signed-off-by: Anas Nashif <anas.nashif@intel.com>
With KPTI enabled, when a page fault happened, code would never load the
page when it should due an `ifdef` confusion.
Fixes: #116502
Signed-off-by: Ederson de Souza <ederson.desouza@intel.com>
Apply the ARG_UNUSED() macro to resolve unused parameter warnings
when building with "-Wextra".
Assisted-by: GitHub Copilot:GPT-5.3-Codex
Signed-off-by: Peter Mitsis <peter.mitsis@intel.com>
Apply the ARG_UNUSED() macro to resolve unused parameter warnings
when building with "-Wextra".
Assisted-by: GitHub Copilot:GPT-5.3-Codex
Signed-off-by: Peter Mitsis <peter.mitsis@intel.com>
Remove the deprecated CONFIG_SSE and CONFIG_SSE_FP_MATH aliases, which
were only selecting CONFIG_X86_SSE and CONFIG_X86_SSE_FP_MATH. They have
carried a "This option is deprecated" help text since well before Zephyr
4.2, and are due for removal in 4.5.
include/zephyr/arch/x86/ia32/arch.h was the last in-tree reader, and it
tested CONFIG_SSE while every other consumer (crt0.S, float.c, swap.S,
ia32/thread.h) already keys off CONFIG_X86_SSE. It now tests
CONFIG_X86_SSE too, so a configuration that enables X86_SSE directly
gets the required 16-byte dynamic thread object alignment for
fxsave/fxrstor instead of pointer alignment.
Signed-off-by: Benjamin Cabé <benjamin@zephyrproject.org>
Assisted-by: Claude:opus-5
Commit d78bf6c165 ("kconfig: Deprecate LEGACY_GENERATED_INCLUDE_PATH")
made it mandatory that generated includes need to include the
"zephyr" namespace, which was missed for a shadow stack related
include on `locore.S`.
Signed-off-by: Ederson de Souza <ederson.desouza@intel.com>
Both the ia32 and intel64 implementations loaded s[counter] before
checking counter == maxsize, so they read s[maxsize] when the first
maxsize bytes were all non-NUL and dereferenced s[0] when
maxsize == 0, violating the strnlen() semantics required by
arch_user_string_nlen(). The over-read byte cannot change the
returned length, but if it is not accessible the exception fixup
reports a spurious error and callers reject valid input with EFAULT.
Check the limit before the load instead.
Fixeszephyrproject-rtos/zephyr#113722
Signed-off-by: Hongquan Li <hongquan.li@processmission.com>
The CONFIG_X2APIC help text frames the option as safe to enable on any
APIC that supports x2APIC mode. This is not accurate: with
CONFIG_X2APIC=y, an AP reads its local APIC ID from the x2APIC ID MSR
unconditionally, even when the AP has not been switched into x2APIC
mode earlier in the bootflow. The MSR access then faults, and the
fault cannot be handled this early in SMP bring-up, so the AP hangs.
Signed-off-by: Patryk Koscik <pkoscik@antmicro.com>
Both architectures wait for the interrupt with interrupts enabled (x86
"sti; hlt", Xtensa "waiti 0"), so the wake-up ISR runs before the
idle-exit hook and can reschedule away from the idle thread. The CPU load
module therefore closes the idle window at ISR entry instead, which means
these architectures need to emit sys_trace_isr_enter().
- x86: the ia32 interrupt stub already emits the hook, but only under
CONFIG_TRACING_ISR. Emit it under CONFIG_SYS_IDLE_HOOKS as well. Also
restore the idle-exit hook after the halt as a fallback for a wake-up
that ran no ISR; closing the window is idempotent, so it is a no-op in
the common case. Only ia32 is enabled: intel64 has no ISR entry hook.
- Xtensa: the interrupt entry had no hook at all. Emit
sys_trace_isr_enter() from the common C interrupt handler, which also
gives Xtensa the ISR tracing it was missing.
Select ARCH_HAS_CPU_IDLE_HOOKS from both, and document in the capability's
help text the two ways an architecture can close the idle window.
Assisted-by: Claude:claude-opus-4-8
Signed-off-by: Anas Nashif <anas.nashif@intel.com>
The architecture idle paths call sys_trace_idle() and
sys_trace_idle_exit() to notify subscribers when the CPU enters and
leaves the idle state. These calls were guarded by CONFIG_TRACING,
which tied idle-time accounting to the tracing subsystem even though
no tracing backend is required to service the hooks.
Introduce a hidden Kconfig symbol, SYS_IDLE_HOOKS, that any subsystem
needing these notifications can select, and have TRACING select it.
Switch the idle-path guards in every architecture and SoC that emits
the hooks from CONFIG_TRACING to CONFIG_SYS_IDLE_HOOKS. Because
TRACING selects the new symbol, existing tracing behaviour is
unchanged; the change only lets non-tracing consumers receive the
hooks.
This is a prerequisite for building the CPU load module without the
tracing subsystem.
Assisted-by: Claude:claude-opus-4-8
Signed-off-by: Anas Nashif <anas.nashif@intel.com>
CONFIG_LINKER_USE_PINNED_SECTION is the second half of the selective
kernel-pinning model removed in issue #108773. With the kernel image
now always resident at boot (previous commit), the __pinned_*
attribute family is a no-op: every page they would have segregated is
already pinned by z_mem_manage_init()'s whole-image loop, so the
tagging contract neither adds safety nor remains maintainable.
Drop it.
Mechanical removals:
* All ~219 in-tree uses of __pinned_text, __pinned_rodata,
__pinned_data, __pinned_bss, __pinned_noinit, and __pinned_func
across arch/x86, drivers/interrupt_controller, drivers/timer,
arch/common, kernel, lib/libc, subsys/portability/posix, tests, and
the syscall code generator (scripts/build/gen_syscalls.py).
* The assembly aliases PINNED_TEXT/RODATA/DATA/BSS/NOINIT used in
arch/x86/core/ia32/*.S and drivers/interrupt_controller/
intc_loapic_spurious.S become plain TEXT/RODATA/DATA/BSS/NOINIT.
* K_KERNEL_PINNED_STACK_DEFINE, K_KERNEL_PINNED_STACK_ARRAY_DEFINE,
K_KERNEL_PINNED_STACK_ARRAY_DECLARE, K_THREAD_PINNED_STACK_DEFINE,
and K_THREAD_PINNED_STACK_ARRAY_DEFINE are removed. The few
in-tree callers (kernel/init.c, arch/arm/core/cortex_a_r/smp.c,
arch/arm64/core/fatal.c, arch/rx/core/prep_c.c,
arch/x86/core/prep_c.c, kernel/include/kernel_internal.h,
tests/bluetooth/hci_uart_async) move to the corresponding
non-pinned macros.
Machinery removals:
* Kconfig.zephyr drops CONFIG_LINKER_USE_PINNED_SECTION.
qemu_x86_tiny and qemu_x86_atom_virt drop their =y overrides.
* include/zephyr/linker/section_tags.h drops the __pinned_* macro
definitions (both arms). __isr collapses to an empty macro since
its only purpose was to alias __pinned_func.
* include/zephyr/linker/sections.h drops PINNED_TEXT_SECTION_NAME,
PINNED_BSS_SECTION_NAME, etc. and the bare PINNED_TEXT/RODATA/etc.
forwarders, plus the _APP_SMEM_PINNED_SECTION_NAME constant.
* include/zephyr/linker/linker-defs.h drops the lnkr_pinned_*
externs, the _app_smem_pinned_* externs, and the lnkr_is_pinned()
/ lnkr_is_region_pinned() inline helpers.
* include/zephyr/linker/utils.h drops the lnkr_pinned_rodata branch
in linker_is_in_rodata().
* include/zephyr/linker/app_smem_pinned{,_aligned,_unaligned}.ld
are deleted; cmake/linker/ld/target_configure.cmake stops
configuring them.
* boards/qemu/x86/qemu_x86_tiny.ld and
include/zephyr/arch/x86/ia32/linker.ld drop their pinned-section
blocks and the now-redundant #ifndef CONFIG_LINKER_USE_PINNED_SECTION
conditionals throughout the body. The
LIB_KERNEL_IN_SECT / LIB_ARCH_X86_IN_SECT / LIB_ZEPHYR_IN_SECT /
LIB_C_IN_SECT / LIB_DRIVERS_IN_SECT / LIB_SUBSYS_LOGGING_IN_SECT /
LIB_ZEPHYR_OBJECT_FILE_IN_SECT / ZEPHYR_KERNEL_FUNCS_IN_SECT macros
in qemu_x86_tiny.ld are deleted; they existed only to feed the
pinned text/rodata/data/bss/noinit sections.
* kernel/mmu.c drops the mark_linker_section_pinned(lnkr_pinned_start,
...) call. The mark_linker_section_pinned() helper survives but is
now gated only on CONFIG_LINKER_USE_BOOT_SECTION.
* arch/common/init.c and include/zephyr/arch/common/init.h drop
arch_bss_zero_pinned(); arch/x86/core/ia32/crt0.S drops the call
to it.
* arch/x86/core/userspace.c drops the eager k_mem_page_in() of the
thread's privileged stack on user-mode entry. With the kernel
image fully resident the stack is already mapped.
* arch/x86/gen_mmu.py drops map_region("lnkr_pinned") and the
set_region_perms() calls for lnkr_pinned_text / lnkr_pinned_rodata.
* CMakeLists.txt drops the LINKER_USE_PINNED_SECTION block that
generated APP_SMEM_PINNED_* variables and the
pinned_partitions target property feeding gen_app_partitions.py.
cmake/modules/extensions.cmake removes the PINNED_RODATA /
PINNED_RAM_SECTIONS / PINNED_DATA_SECTIONS zephyr_linker_sources()
location keywords and their snippet files.
scripts/build/gen_app_partitions.py drops --pinoutput /
--pinpartitions arguments and the pinned-output branch.
subsys/testsuite/coverage/CMakeLists.txt drops its
CONFIG_DEMAND_PAGING-conditional fork.
* scripts/build/gen_kobject_list.py drops the
app_smem_pinned_start / _end fallback for kobject placement
validation.
* tests/arch/x86/pagetables and tests/kernel/mem_protect/userspace
drop their lnkr_pinned_text / lnkr_pinned_rodata branches.
* include/zephyr/arch/x86/ia32/arch.h folds IRQSTUBS_TEXT_SECTION
to the unconditional ".text.irqstubs" form.
* tests/subsys/llext/src/syscalls_ext.c drops a stale comment about
syscalls landing in .pinned_text.
Targeted retentions:
* arch/x86/core/bootargs.c keeps multiboot_cmdline and efi_bootargs
in .noinit (was __pinned_noinit, which decayed to __noinit when
LINKER_USE_PINNED_SECTION was unset). The multiboot and zefi loader
paths write these buffers before Zephyr's BSS-zero step, so
zeroing them at boot loses the cmdline.
* arch/x86/core/ia32/fatal.c keeps _df_esf and _df_stack in .noinit.
They are scratch space written by the double-fault handler and have
no zero-init requirement; keeping them in .noinit also preserves
the historical post-noinit alignment that gen_mmu.py relies on
(z_mapped_size is computed before CMake-injected iterable sections
are appended to the linker script, so the post-noinit page padding
is what keeps those sections within the mapped region).
* include/zephyr/arch/x86/ia32/syscall.h and
include/zephyr/arch/x86/arch.h wrap the per-arch
arch_syscall_invoke* / arch_is_user_context / arch_k_cycle_get_*
implementations in @cond INTERNAL_HIDDEN. The public Doxygen
contract lives on the prototypes in
include/zephyr/arch/arch_interface.h; the per-arch implementations
are internal. Without this, removing the __pinned_func attribute
exposes the implementations to the doxygen-coverage delta check
as 10 newly-undocumented APIs.
Documentation updates are deferred to a separate commit.
Issue: #108773
Signed-off-by: Nicolas Pitre <npitre@baylibre.com>
With qemu_x86_tiny moved off the =n side of this knob in the previous
commit, no in-tree configuration sets
CONFIG_LINKER_GENERIC_SECTIONS_PRESENT_AT_BOOT=n anymore. The mode it
selected is also unsafe by construction (issue #108773), so there is
nothing to deprecate -- drop the Kconfig symbol and every #ifdef on
it.
Consequences:
* z_mem_manage_init() pins the whole Zephyr image unconditionally and
no longer calls arch_bss_zero() after demand-paging init. The
pinning loop's TODO about "we will need linker regions for a subset
of kernel code/data pages which are pinned in memory and may not be
evicted" is replaced with a brief note that the kernel image is
always resident now and that pageable kernel regions go through
__ondemand_*.
* arch/x86/core/ia32/crt0.S always calls arch_bss_zero() once
arch_bss_zero_boot and arch_bss_zero_pinned have run.
* kernel/kheap.c and kernel/userspace/userspace.c drop the
pre-kernel/post-kernel "skip non-pinned" dance and zero/init each
heap and app-shmem partition unconditionally at PRE_KERNEL_1.
* arch/x86/core/userspace.c drops the eager k_mem_page_in() of the
thread's privileged stack before dropping to user mode -- with the
full kernel image resident the stack is already mapped.
* arch/x86/gen_mmu.py always maps the Zephyr image with FLAG_P set
(and lnkr_boot_* / lnkr_pinned_* regions just inherit that).
* CMakeLists.txt no longer force-pins z_libc_partition into
pinned_partitions for app_smem.
* The EVICTION_LRU gate
"depends on LINKER_GENERIC_SECTIONS_PRESENT_AT_BOOT" goes away;
LRU is now safe by construction on any DEMAND_PAGING board and
becomes the default whenever ARCH_SUPPORTS_EVICTION_TRACKING.
* DEMAND_PAGING_PAGE_FRAMES_RESERVE loses its conditional default of
32 frames; the new model needs no reserve.
* boards/qemu/x86/qemu_x86_tiny.ld drops the FLASH MEMORY region and
the flash_load_offset block that arranged the demand-paged
generic-section layout. boards/qemu/x86/board.cmake drops the
8 MB-RAM bump and the --map flash gen_mmu argument that paired with
that mode.
* tests/arch/x86/pagetables and tests/kernel/fatal/exception drop the
code paths that were guarded on =n.
* tests/kernel/mem_protect/demand_paging/mem_map.lru replaces its
CONFIG_LINKER_GENERIC_SECTIONS_PRESENT_AT_BOOT=y override with an
explicit CONFIG_EVICTION_LRU=y, since the symbol it relied on is
gone.
The __pinned_* tagging convention still expands to its actual linker
sections under CONFIG_LINKER_USE_PINNED_SECTION; that symbol and the
~219 in-tree __pinned_* annotations are cleaned up in the following
commit.
Issue: #108773
Signed-off-by: Nicolas Pitre <npitre@baylibre.com>
This commit fixes a silent SMP hang on platforms where CPUs are
reported as MADT Type 9 (x2APIC) entries with non-sequential APIC
IDs. Zephyr only queried Type 0 (xAPIC) entries, leaving
x86_cpu_loapics[] as zero and sending the startup IPI to the wrong
CPU.
Add Type 9 (x2APIC) as the preferred MADT lookup in arch_cpu_start()
when CONFIG_X2APIC is enabled, with Type 0 (xAPIC) retained as
fallback for platforms that only publish xAPIC entries.
Signed-off-by: S Swetha <s.swetha@intel.com>
This commit introduces a fix for a silent SMP hang on systems that
report CPUs as MADT Type 9 (x2APIC) entries. On such systems, the
hardware operates in x2APIC mode where the xAPIC MMIO registers are
inaccessible. Secondary CPUs were reading their APIC ID from the
xAPIC MMIO register, causing an access fault and looping forever in
unknown_loapic_id, resulting in a silent SMP hang.
Read the 32-bit APIC ID from MSR 0x802 via rdmsr when CONFIG_X2APIC
is enabled, falling back to the xAPIC MMIO register when
CONFIG_X2APIC is disabled.
Signed-off-by: S Swetha <s.swetha@intel.com>
A component should not rely on the API it is implementing.
In this case, the arch layer is implementing the arch cache API
(include/zephyr/arch/cache.h), that is used by the public sys cache
API (include/zephyr/cache.h), so it can't call the latter.
Furthermore, it does not implement arch_cache_data_line_size_get(),
so sys_cache_data_line_size_get() will always give the value of
CONFIG_DCACHE_LINE_SIZE.
Signed-off-by: Abderrahmane JARMOUNI <git@jarmouni.me>
kernel.h implies kernel_structs.h via kernel_includes.h, making
explicit inclusion of kernel_structs.h unnecessary whenever kernel.h
is already included in the same translation unit.
Remove the redundant includes across arch, boards, drivers, kernel,
lib, samples, subsys, and tests trees.
in include/zephyr/kernel_structs.h:
* 2. kernel.h shall imply kernel_structs.h, such that it shall not be
* necessary to include kernel_structs.h explicitly when kernel.h is
* included.
Signed-off-by: Anas Nashif <anas.nashif@intel.com>
The LRU eviction algorithm needs to catch the first access to a loaded
page in order to call k_mem_paging_eviction_accessed() and move that
page to the tail of the queue. On ARM64 this is done with the MMU's
Access Flag: clearing AF causes a distinct fault on the next access.
On x86 there is no access-flag fault. The Accessed bit (PTE bit 5) is
set by hardware on access but never traps. The only way to force a
fault is to clear the Present bit, which already encodes the
"paged out" state — so a new state is needed:
PTE == 0 -> unmapped
P=0, A=1, upper=location -> paged out
P=0, G=1, upper=PFN -> LRU-tracked (new)
P=1 -> normally mapped
Bit G (Global, bit 8) is never set by Zephyr on x86 (CR4.PGE is not
used), so it is free to use as a private marker when P=0. No existing
PTE state needs to be displaced. This stays out of the way of the
KPTI path (which uses the PAT bit) and of the permission-backup bits
(IGNORED0..2) used for memory domain handling.
arch_page_info_get(addr, NULL, clear_accessed=true) is overloaded
under CONFIG_EVICTION_LRU to both query the prior flags and transition
the page to the LRU-tracked state via a new helper that updates all
domain ptables. arch_page_location_get() recognizes the tracked state
as paged-in so the core demand-paging code treats the page as resident.
The page fault handler intercepts LRU-tracking faults in-line before
k_mem_page_fault() dispatch: restore P, clear the tracking bit, and
call k_mem_paging_eviction_accessed() directly. This avoids the risk
of recursing through do_page_fault() with z_mm_lock held.
KPTI co-exists with demand paging but its PTE encoding is not yet
wired up to the LRU state, so tracking is gated on !X86_KPTI for now.
Fixes: #75132
Signed-off-by: Nicolas Pitre <npitre@baylibre.com>
For !TRACING, most arch_cpu_idle and arch_cpu_atomic_idle implementation
relies on the fact that there's weak stub implementations in
subsys/tracing/tracing_none.c, this works, but the arch_cpu_idle sits in
hot code path, so we'd better to make it as efficient as possible.
Take the riscv implementation for example,
Before the patch:
80000a66 <arch_cpu_idle>:
80000a66: 1141 addi sp,sp,-16
80000a68: c606 sw ra,12(sp)
80000a6a: 37c5 jal 80000a4a <sys_trace_idle>
80000a6c: 10500073 wfi
80000a70: 3ff1 jal 80000a4c <sys_trace_idle_exit>
80000a72: 47a1 li a5,8
80000a74: 3007a073 csrs mstatus,a5
80000a78: 40b2 lw ra,12(sp)
80000a7a: 0141 addi sp,sp,16
80000a7c: 8082 ret
NOTE: the sys_trace_idle and sys_trace_idle_exit are just stubs when
!TRACING
after the patch:
80000a62 <arch_cpu_idle>:
80000a62: 10500073 wfi
80000a66: 47a1 li a5,8
80000a68: 3007a073 csrs mstatus,a5
80000a6c: 8082 ret
Signed-off-by: Jisheng Zhang <jszhang@kernel.org>
Upgrades the thread user_options to 16 bits from an 8-bit value to
provide more space for future values.
Also, as the size of this field has changed, the values for the
existing architecture specific thread options have also shifted
from the upper end of the old 8-bit field, to the upper end of
the new 16-bit field.
Fixes#101034
Signed-off-by: Peter Mitsis <peter.mitsis@intel.com>
Remove duplicated #include directives within the same
preprocessor scope across the Zephyr tree.
Duplicates inside different #ifdef branches are preserved
as they may be intentional.
Signed-off-by: Sylvio Alves <sylvio.alves@espressif.com>
Due to slight differences in the way that LLVM and GNU linkers work,
the call to `z_stack_space_get()` is not dead-stripped when linking
with `lld` but it is dead-stripped when linking with GNU `ld`.
The `z_stack_space_get()` function is only available when
`CONFIG_INIT_STACKS` and `CONFIG_THREAD_STACK_INFO` are defined.
The issue is reproducible (although requires building LLVM and
setting up some environment variables) and goes away with the proposed
workaround.
Signed-off-by: Robin Kastberg <robin.kastberg@iar.com>
Signed-off-by: Chris Friedt <cfriedt@tenstorrent.com>
soc_prep_hook() is always called from z_prep_c() which is implemented
as a C function. As such, there is no need to check for the associated
CONFIG_SOC_PREP_HOOK since the platform/hooks.h header will define hooks
as no-op function-like macros if their associated Kconfig isn't enabled.
Remove the Kconfig check from all arch implementations of z_prep_c() and
call soc_prep_hook() directly instead, to avoid duplicating the Kconfig
check already performed in platform/hooks.h
Signed-off-by: Mathieu Choplain <mathieu.choplain-ext@st.com>
Do not directly include and use APIs from ksched.h outside of the
kernel. For now do this using more suitable (ipi.h and
kernel_internal.h) internal APIs until more cleanup is done.
Signed-off-by: Anas Nashif <anas.nashif@intel.com>
Do not use private API prefix and move to architecture interface as
those functions are primarily used across arches and can be defined by
the architecture.
Signed-off-by: Anas Nashif <anas.nashif@intel.com>
Do not use private API prefix and move to architecture interface as
those functions are primarily used across arches and can be defined by
the architecture.
Signed-off-by: Anas Nashif <anas.nashif@intel.com>
Cleanup init.c code and move early boot code into arch/ and make it
accessible outside of the boot process/kernel.
All of this code is not related to the 'kernel' and is mostly used
within the architecture boot / setup process.
The way it was done, some soc code was including kernel_internal.h
directly, which shouldn't be done.
Signed-off-by: Anas Nashif <anas.nashif@intel.com>
Not really a kernel feature, more for architecture, which is reflected
in how XIP is enabled and tested. Move it to architecture code to keep
which much of the 'implementation' and usage is.
Signed-off-by: Anas Nashif <anas.nashif@intel.com>
- No more need for special IRQ shadow stacks - just reuse the one
created for z_interrupt_stacks;
- Add the linker sections for the pairs of stack/shadow stack;
- Support shadow stack arrays.
Last item was a bit challenging: shadow stacks need to be initialised
before use, and this is done statically for normal shadow stacks. To
initialise the shadow stacks in the array, one needs how many entries it
has. While a simple approach would use `LISTIFY` to them do the
initialization on all entries, that is not possible as many stack arrays
are created using expressions instead of literals, such as
`CONFIG_MP_MAX_NUM_CPUS - 1`, which won't work with `LISTIFY`.
Instead, this patch uses a script, `gen_static_shstk_array.py` that
gathers all needed information and patches the ELF to initialize the
stack arrays. Note that this needs to be done before any other operation
on the ELF file that creates new representations, such as the .bin
output.
Signed-off-by: Ederson de Souza <ederson.desouza@intel.com>
It seems that, at least on tests, it's common to call k_thread_create()
on a thread multiple times. This trips a check for the CET shadow stack
- namely, set a shadow stack on a thread which already has a shadow
stack.
This patch adds a Kconfig option to allow that, iff the base address and
size of the new shadow stack are the same as before. This will trigger a
reset of the shadow stack, so it can be reused.
It may be the case that this behaviour (reusing threads) is more common
than only for tests, in which case it could make sense to change the
default - in this patch, is only true if ZTEST.
Even if being enabled by default becomes the reality, it would still
make sense to keep this option - more conscious apps could avoid the
need for the shadow stack reset code altogether.
Signed-off-by: Ederson de Souza <ederson.desouza@intel.com>
So that kernel created threads can use shadow stacks. Note that
CONFIG_X86_CET_SHADOW_STACK is abandoned in favour of
CONFIG_HW_SHADOW_STACK.
This means change some types, functions and macro throughout shadow
stack code.
Signed-off-by: Ederson de Souza <ederson.desouza@intel.com>
Some SoCs may need to do some preparatory work before changing the
current shadow stack pointer (and thus, currently used shadow stack).
This patch adds a way for that, shielded by a Kconfig
(CONFIG_X86_CET_SOC_PREPARE_SHADOW_STACK_SWITCH).
As currently only 32 bit SoC may use this, support is only added to the
32 bit code.
Signed-off-by: Ederson de Souza <ederson.desouza@intel.com>
Most notable difference on base support is the need to keep the shadow
stack tokens, which are 8 bytes, 8 bytes aligned. Some helper macros are
used for that.
Also, an `ssp` entry is added to the task state segment (TSS).
Signed-off-by: Ederson de Souza <ederson.desouza@intel.com>
Currently, it's permitted to have threads that don't have a shadow
stack. When those are run, shadow stack is disabled on the CPU. To
identify those, the thread `shstk_addr` member is checked.
This patch adds an optional check, behind
CONFIG_X86_CET_VERIFY_KERNEL_SHADOW_STACK, that checks if an outgoing
thread has this pointer NULL with shadow stack currently enabled on
the CPU, meaning a 1) bug or 2) some attempt to tamper with the pointer.
If the check fails, k_panic() is called. Note that this verification is
not enough to guarantee `shstk_addr` can't be tampered with. For
instance, it only works on a running thread. Ideally, all threads should
be shadow stack capable, so a missing `shstk_addr` would simply be a
hard fault, but that is still to come.
Signed-off-by: Ederson de Souza <ederson.desouza@intel.com>
Main peculiarity is that if an exception results in current thread being
aborted, we need to clear the busy bit on the shadow stack on the swap
to the new thread, otherwise future exceptions will fail when trying to
use a busy shadow stack.
Signed-off-by: Ederson de Souza <ederson.desouza@intel.com>
Nested interrupts are supported, on the normal stack, by creating a
stack whose size is a multiple of CONFIG_ISR_DEPTH, and updating the
pointer used by Interrupt Stack Table (IST) to point to a new base,
inside the "oversized" stack.
The same approach is used for the shadow stack: shadow stack size is
multiplied by CONFIG_ISR_DEPTH, and the pointer to the stack on the
shadow stack pointer table is update to point to the next base.
Signed-off-by: Ederson de Souza <ederson.desouza@intel.com>
For IRQs, shadow stack support a mechanism similar to the Interrupt
Stack Table (IST) for x86_64: a table, indexed by the IST index, pointing
to a 64 byte table in memory containing the address of seven shadow stacks
to be used by the interrupt service routines.
This patch adds support to this mechanism. It is worth noting that, as
Zephyr may exit from an interrupt by going to a different thread than
the one that was interrupted, some housekeeping is done to ensure that
the necessary shadow stack tokens are on the right shadow stack before
return from the interrupt.
Signed-off-by: Ederson de Souza <ederson.desouza@intel.com>
Shadow Stack is one of the capabilities provided by Intel Control-flow
Enforcement Technology (CET), aimed at defending against Return Oriented
Programming.
This patch enables it for x86_64 (32-bit support coming in future
patches):
- Add relevant Kconfigs;
- Shadow stacks should live in specially defined memory pages, so
gen_mmu.py was updated to allow that;
- A new macro, Z_X86_SHADOW_STACK_DEFINE, added to define the area
for a shadow stack;
- A new function, z_x86_thread_attach_shadow_stack(), added to
attach a shadow stack to a never started thread;
- locore.S changed to enable/disable shadow stack when a thread
using it comes in/out of execution.
As not all threads are currently shadow stack capable, threads that do
not use it will still run with shadow stack disabled. Ideally, at some
point in the future, all threads would use the shadow stack, so no need
to disable it at all.
Signed-off-by: Ederson de Souza <ederson.desouza@intel.com>
Add code to enable it and sprinkle `endbr64` on asm code, where needed.
Namely, IRQs and excepts entrypoints.
Finally, tests added to ensure IBT behaves sanely.
Signed-off-by: Ederson de Souza <ederson.desouza@intel.com>
Indirect Branch Tracking (IBT) is one of the capabilities provided by
Intel Control-flow Enforcement Technology (CET), aimed at defending
against Jump/Call Oriented Programming.
This patch enables it for x86 (32-bit, 64-bit support coming in future
patches):
- Add relevant Kconfigs (everything is behind X86_CET);
- Code to enable it;
- Enable compiler flags to enable it;
- Add `endbr32` instructions to asm code, where needed.
Points in the code where an indirect branch is expected to land need
special instructions that tell the CPU they are valid indirect branch
targets. Those are added by the compiler, so toolchain support is
necessary. Note that any code added to the final ELF also need those
markers, such as libc or libgcc.
Finally, tests added to ensure IBT behaves sanely.
Signed-off-by: Ederson de Souza <ederson.desouza@intel.com>
This adds exception handling of control protection exception
in fatal code.
Signed-off-by: Daniel Leung <daniel.leung@intel.com>
Signed-off-by: Ederson de Souza <ederson.desouza@intel.com>
Fix an issue where 1 vector is being requested when MSI-X is
enabled. The previous logic always assumed the PCIE device has
only fixed or single MSI when we are requesting 1 vector, which
is not entirely correct. So if there is no vector allocated
already, try to allocate one.
Fixes#93319
Signed-off-by: Daniel Leung <daniel.leung@intel.com>
arch_pcie_msi_vectors_allocate() has a return type of uint8_t.
One of the error path returns -1 which would result in 255
being returned. So fix that by returning 0 instead, as there is
no vector being allocated anyway.
Signed-off-by: Daniel Leung <daniel.leung@intel.com>
GCC 14.3 will happily delete any code that appears before
__builtin_unreachable that isn't separated with an obvious branch. That
includes __asm__ statements, even those which generate traps.
The failure case that I debugged was on x86 in
z_check_stack_sentinel. There is a store to restore the sentinel to the
correct value just before the ARCH_EXCEPT, and that macro emits 'int $32'
followed by CODE_UNREACHABLE. Because the compiler didn't understand that
ARCH_EXCEPT was changing execution flow, it decided that the sentinel
restoring store 'couldn't' be reached and elided it.
I added the "memory" clobber to the asm statement in ARCH_EXCEPT before
CODE_UNREACHABLE to enforce that all pending store operations be performed
before the asm statement occurs. This ensures that they are not deleted by
the compiler.
I think this might be a GCC bug. The GCC documentation explicitly documents
that asm statements which change the flow of control should be followed by
__builtin_unreachable.
Signed-off-by: Keith Packard <keithp@keithp.com>
