For !TRACING, most arch_cpu_idle and arch_cpu_atomic_idle implementation
relies on the fact that there's weak stub implementations in
subsys/tracing/tracing_none.c, this works, but the arch_cpu_idle sits in
hot code path, so we'd better to make it as efficient as possible.
Take the riscv implementation for example,
Before the patch:
80000a66 <arch_cpu_idle>:
80000a66: 1141 addi sp,sp,-16
80000a68: c606 sw ra,12(sp)
80000a6a: 37c5 jal 80000a4a <sys_trace_idle>
80000a6c: 10500073 wfi
80000a70: 3ff1 jal 80000a4c <sys_trace_idle_exit>
80000a72: 47a1 li a5,8
80000a74: 3007a073 csrs mstatus,a5
80000a78: 40b2 lw ra,12(sp)
80000a7a: 0141 addi sp,sp,16
80000a7c: 8082 ret
NOTE: the sys_trace_idle and sys_trace_idle_exit are just stubs when
!TRACING
after the patch:
80000a62 <arch_cpu_idle>:
80000a62: 10500073 wfi
80000a66: 47a1 li a5,8
80000a68: 3007a073 csrs mstatus,a5
80000a6c: 8082 ret
Signed-off-by: Jisheng Zhang <jszhang@kernel.org>
Upgrades the thread user_options to 16 bits from an 8-bit value to
provide more space for future values.
Also, as the size of this field has changed, the values for the
existing architecture specific thread options have also shifted
from the upper end of the old 8-bit field, to the upper end of
the new 16-bit field.
Fixes#101034
Signed-off-by: Peter Mitsis <peter.mitsis@intel.com>
soc_per_core_init_hook() is usually called from arch_kernel_init() and
arch_secondary_cpu_init() which are C functions. As such, there is no need
to check for CONFIG_SOC_PER_CORE_INIT_HOOK since platform/hooks.h provides
a no-op function-like macro implementation if the Kconfig option is not
enabled.
Remove the Kconfig option check from all files.
Signed-off-by: Mathieu Choplain <mathieu.choplain-ext@st.com>
Remove duplicated #include directives within the same
preprocessor scope across the Zephyr tree.
Duplicates inside different #ifdef branches are preserved
as they may be intentional.
Signed-off-by: Sylvio Alves <sylvio.alves@espressif.com>
Due to slight differences in the way that LLVM and GNU linkers work,
the call to `z_stack_space_get()` is not dead-stripped when linking
with `lld` but it is dead-stripped when linking with GNU `ld`.
The `z_stack_space_get()` function is only available when
`CONFIG_INIT_STACKS` and `CONFIG_THREAD_STACK_INFO` are defined.
The issue is reproducible (although requires building LLVM and
setting up some environment variables) and goes away with the proposed
workaround.
Signed-off-by: Robin Kastberg <robin.kastberg@iar.com>
Signed-off-by: Chris Friedt <cfriedt@tenstorrent.com>
soc_prep_hook() is always called from z_prep_c() which is implemented
as a C function. As such, there is no need to check for the associated
CONFIG_SOC_PREP_HOOK since the platform/hooks.h header will define hooks
as no-op function-like macros if their associated Kconfig isn't enabled.
Remove the Kconfig check from all arch implementations of z_prep_c() and
call soc_prep_hook() directly instead, to avoid duplicating the Kconfig
check already performed in platform/hooks.h
Signed-off-by: Mathieu Choplain <mathieu.choplain-ext@st.com>
Do not directly include and use APIs from ksched.h outside of the
kernel. For now do this using more suitable (ipi.h and
kernel_internal.h) internal APIs until more cleanup is done.
Signed-off-by: Anas Nashif <anas.nashif@intel.com>
Do not use private API prefix and move to architecture interface as
those functions are primarily used across arches and can be defined by
the architecture.
Signed-off-by: Anas Nashif <anas.nashif@intel.com>
Do not use private API prefix and move to architecture interface as
those functions are primarily used across arches and can be defined by
the architecture.
Signed-off-by: Anas Nashif <anas.nashif@intel.com>
Cleanup init.c code and move early boot code into arch/ and make it
accessible outside of the boot process/kernel.
All of this code is not related to the 'kernel' and is mostly used
within the architecture boot / setup process.
The way it was done, some soc code was including kernel_internal.h
directly, which shouldn't be done.
Signed-off-by: Anas Nashif <anas.nashif@intel.com>
Not really a kernel feature, more for architecture, which is reflected
in how XIP is enabled and tested. Move it to architecture code to keep
which much of the 'implementation' and usage is.
Signed-off-by: Anas Nashif <anas.nashif@intel.com>
- No more need for special IRQ shadow stacks - just reuse the one
created for z_interrupt_stacks;
- Add the linker sections for the pairs of stack/shadow stack;
- Support shadow stack arrays.
Last item was a bit challenging: shadow stacks need to be initialised
before use, and this is done statically for normal shadow stacks. To
initialise the shadow stacks in the array, one needs how many entries it
has. While a simple approach would use `LISTIFY` to them do the
initialization on all entries, that is not possible as many stack arrays
are created using expressions instead of literals, such as
`CONFIG_MP_MAX_NUM_CPUS - 1`, which won't work with `LISTIFY`.
Instead, this patch uses a script, `gen_static_shstk_array.py` that
gathers all needed information and patches the ELF to initialize the
stack arrays. Note that this needs to be done before any other operation
on the ELF file that creates new representations, such as the .bin
output.
Signed-off-by: Ederson de Souza <ederson.desouza@intel.com>
It seems that, at least on tests, it's common to call k_thread_create()
on a thread multiple times. This trips a check for the CET shadow stack
- namely, set a shadow stack on a thread which already has a shadow
stack.
This patch adds a Kconfig option to allow that, iff the base address and
size of the new shadow stack are the same as before. This will trigger a
reset of the shadow stack, so it can be reused.
It may be the case that this behaviour (reusing threads) is more common
than only for tests, in which case it could make sense to change the
default - in this patch, is only true if ZTEST.
Even if being enabled by default becomes the reality, it would still
make sense to keep this option - more conscious apps could avoid the
need for the shadow stack reset code altogether.
Signed-off-by: Ederson de Souza <ederson.desouza@intel.com>
So that kernel created threads can use shadow stacks. Note that
CONFIG_X86_CET_SHADOW_STACK is abandoned in favour of
CONFIG_HW_SHADOW_STACK.
This means change some types, functions and macro throughout shadow
stack code.
Signed-off-by: Ederson de Souza <ederson.desouza@intel.com>
Some SoCs may need to do some preparatory work before changing the
current shadow stack pointer (and thus, currently used shadow stack).
This patch adds a way for that, shielded by a Kconfig
(CONFIG_X86_CET_SOC_PREPARE_SHADOW_STACK_SWITCH).
As currently only 32 bit SoC may use this, support is only added to the
32 bit code.
Signed-off-by: Ederson de Souza <ederson.desouza@intel.com>
Most notable difference on base support is the need to keep the shadow
stack tokens, which are 8 bytes, 8 bytes aligned. Some helper macros are
used for that.
Also, an `ssp` entry is added to the task state segment (TSS).
Signed-off-by: Ederson de Souza <ederson.desouza@intel.com>
Currently, it's permitted to have threads that don't have a shadow
stack. When those are run, shadow stack is disabled on the CPU. To
identify those, the thread `shstk_addr` member is checked.
This patch adds an optional check, behind
CONFIG_X86_CET_VERIFY_KERNEL_SHADOW_STACK, that checks if an outgoing
thread has this pointer NULL with shadow stack currently enabled on
the CPU, meaning a 1) bug or 2) some attempt to tamper with the pointer.
If the check fails, k_panic() is called. Note that this verification is
not enough to guarantee `shstk_addr` can't be tampered with. For
instance, it only works on a running thread. Ideally, all threads should
be shadow stack capable, so a missing `shstk_addr` would simply be a
hard fault, but that is still to come.
Signed-off-by: Ederson de Souza <ederson.desouza@intel.com>
Main peculiarity is that if an exception results in current thread being
aborted, we need to clear the busy bit on the shadow stack on the swap
to the new thread, otherwise future exceptions will fail when trying to
use a busy shadow stack.
Signed-off-by: Ederson de Souza <ederson.desouza@intel.com>
Nested interrupts are supported, on the normal stack, by creating a
stack whose size is a multiple of CONFIG_ISR_DEPTH, and updating the
pointer used by Interrupt Stack Table (IST) to point to a new base,
inside the "oversized" stack.
The same approach is used for the shadow stack: shadow stack size is
multiplied by CONFIG_ISR_DEPTH, and the pointer to the stack on the
shadow stack pointer table is update to point to the next base.
Signed-off-by: Ederson de Souza <ederson.desouza@intel.com>
For IRQs, shadow stack support a mechanism similar to the Interrupt
Stack Table (IST) for x86_64: a table, indexed by the IST index, pointing
to a 64 byte table in memory containing the address of seven shadow stacks
to be used by the interrupt service routines.
This patch adds support to this mechanism. It is worth noting that, as
Zephyr may exit from an interrupt by going to a different thread than
the one that was interrupted, some housekeeping is done to ensure that
the necessary shadow stack tokens are on the right shadow stack before
return from the interrupt.
Signed-off-by: Ederson de Souza <ederson.desouza@intel.com>
Shadow Stack is one of the capabilities provided by Intel Control-flow
Enforcement Technology (CET), aimed at defending against Return Oriented
Programming.
This patch enables it for x86_64 (32-bit support coming in future
patches):
- Add relevant Kconfigs;
- Shadow stacks should live in specially defined memory pages, so
gen_mmu.py was updated to allow that;
- A new macro, Z_X86_SHADOW_STACK_DEFINE, added to define the area
for a shadow stack;
- A new function, z_x86_thread_attach_shadow_stack(), added to
attach a shadow stack to a never started thread;
- locore.S changed to enable/disable shadow stack when a thread
using it comes in/out of execution.
As not all threads are currently shadow stack capable, threads that do
not use it will still run with shadow stack disabled. Ideally, at some
point in the future, all threads would use the shadow stack, so no need
to disable it at all.
Signed-off-by: Ederson de Souza <ederson.desouza@intel.com>
Add code to enable it and sprinkle `endbr64` on asm code, where needed.
Namely, IRQs and excepts entrypoints.
Finally, tests added to ensure IBT behaves sanely.
Signed-off-by: Ederson de Souza <ederson.desouza@intel.com>
Indirect Branch Tracking (IBT) is one of the capabilities provided by
Intel Control-flow Enforcement Technology (CET), aimed at defending
against Jump/Call Oriented Programming.
This patch enables it for x86 (32-bit, 64-bit support coming in future
patches):
- Add relevant Kconfigs (everything is behind X86_CET);
- Code to enable it;
- Enable compiler flags to enable it;
- Add `endbr32` instructions to asm code, where needed.
Points in the code where an indirect branch is expected to land need
special instructions that tell the CPU they are valid indirect branch
targets. Those are added by the compiler, so toolchain support is
necessary. Note that any code added to the final ELF also need those
markers, such as libc or libgcc.
Finally, tests added to ensure IBT behaves sanely.
Signed-off-by: Ederson de Souza <ederson.desouza@intel.com>
This adds exception handling of control protection exception
in fatal code.
Signed-off-by: Daniel Leung <daniel.leung@intel.com>
Signed-off-by: Ederson de Souza <ederson.desouza@intel.com>
Fix an issue where 1 vector is being requested when MSI-X is
enabled. The previous logic always assumed the PCIE device has
only fixed or single MSI when we are requesting 1 vector, which
is not entirely correct. So if there is no vector allocated
already, try to allocate one.
Fixes#93319
Signed-off-by: Daniel Leung <daniel.leung@intel.com>
arch_pcie_msi_vectors_allocate() has a return type of uint8_t.
One of the error path returns -1 which would result in 255
being returned. So fix that by returning 0 instead, as there is
no vector being allocated anyway.
Signed-off-by: Daniel Leung <daniel.leung@intel.com>
GCC 14.3 will happily delete any code that appears before
__builtin_unreachable that isn't separated with an obvious branch. That
includes __asm__ statements, even those which generate traps.
The failure case that I debugged was on x86 in
z_check_stack_sentinel. There is a store to restore the sentinel to the
correct value just before the ARCH_EXCEPT, and that macro emits 'int $32'
followed by CODE_UNREACHABLE. Because the compiler didn't understand that
ARCH_EXCEPT was changing execution flow, it decided that the sentinel
restoring store 'couldn't' be reached and elided it.
I added the "memory" clobber to the asm statement in ARCH_EXCEPT before
CODE_UNREACHABLE to enforce that all pending store operations be performed
before the asm statement occurs. This ensures that they are not deleted by
the compiler.
I think this might be a GCC bug. The GCC documentation explicitly documents
that asm statements which change the flow of control should be followed by
__builtin_unreachable.
Signed-off-by: Keith Packard <keithp@keithp.com>
The intent of arch_coprocessors_disable() is to replace
arch_float_disable() in halt_thread() for the FPU will not
always be the only coprocessor that will need to be disabled.
Signed-off-by: Peter Mitsis <peter.mitsis@intel.com>
instead of depending on logging for exceptions and fatal messages, use
macro and dump exceptions if printk is enable with no logging.
Signed-off-by: Anas Nashif <anas.nashif@intel.com>
This is an X86 specific option and should not appear as generic debug
option.
Fixeszephyrproject-rtos/zephyr#52929
Signed-off-by: Anas Nashif <anas.nashif@intel.com>
Instead of both checking CONFIG_BOARD_QEMU_X86 and
CONFIG_BOARD_QEMU_X86_64, simply check for CONFIG_QEMU_TARGET.
This also helps qemu_x86_tiny, qemu_x86_lakemont and any
future/downstream QEMU targets to keep things simple.
Signed-off-by: Ederson de Souza <ederson.desouza@intel.com>
Stack walking on x86_64 can run with compiler optimization and
does not need CONFIG_NO_OPTIMIZATIONS as long as frame pointers
are not omitted. So remove the "depends on" for x86_64
from CONFIG_ARCH_HAS_STACKWALK.
Signed-off-by: Daniel Leung <daniel.leung@intel.com>
This ASSERT fails because the comparison is made between the physical
APIC ID and the virtual CPU's LAPIC ID. If CPU-1 is assigned to ACRN,
it considers that CPU as its CPU-0.
Signed-off-by: Anisetti Avinash Krishna <anisetti.avinash.krishna@intel.com>
There is no need for kernel/internal/smp.h as SOF does not call
z_sched_ipi(). Actually... git log over there has no mention of
z_sched_ipi() anywhere, just arch_sched_ipi().
And include <ksched.h> for source using z_sched_ipi() since
they are using scheduling functions, and would be the correct
file to include.
Signed-off-by: Daniel Leung <daniel.leung@intel.com>
It has been discovered that direct IPI support does not work
correctly when CONFIG_X2APIC is enabled. Until that can be
fixed, restrict this feature on x86 to platforms that do not
enable CONFIG_X2APIC.
Fixes#87046
Signed-off-by: Peter Mitsis <peter.mitsis@intel.com>
