Commit graph

208 commits

Author SHA1 Message Date
Anas Nashif
f5d7081710 kernel: do not include ksched.h in subsys/soc code
Do not directly include and use APIs from ksched.h outside of the
kernel. For now do this using more suitable (ipi.h and
kernel_internal.h) internal APIs until more cleanup is done.

Signed-off-by: Anas Nashif <anas.nashif@intel.com>
2025-09-09 11:45:06 +02:00
Anas Nashif
25938ec2bf arch: init: rename z_data_copy -> arch_data_copy
Do not use private API prefix and move to architecture interface as
those functions are primarily used across arches and can be defined by
the architecture.

Signed-off-by: Anas Nashif <anas.nashif@intel.com>
2025-09-08 15:51:07 -04:00
Anas Nashif
6b46c826aa arch: init: z_bss_zero -> arch_bss_zero
Do not use private API prefix and move to architecture interface as
those functions are primarily used across arches and can be defined by
the architecture.

Signed-off-by: Anas Nashif <anas.nashif@intel.com>
2025-09-08 15:51:07 -04:00
Anas Nashif
cf6db903e1 kernel: move xip into arch/common
Not really a kernel feature, more for architecture, which is reflected
in how XIP is enabled and tested. Move it to architecture code to keep
which much of the 'implementation' and usage is.

Signed-off-by: Anas Nashif <anas.nashif@intel.com>
2025-09-08 15:51:07 -04:00
Ederson de Souza
2d94c5d7aa arch/x86: Support for automatic shadow stacks
- No more need for special IRQ shadow stacks - just reuse the one
   created for z_interrupt_stacks;
   - Add the linker sections for the pairs of stack/shadow stack;
   - Support shadow stack arrays.

Last item was a bit challenging: shadow stacks need to be initialised
before use, and this is done statically for normal shadow stacks. To
initialise the shadow stacks in the array, one needs how many entries it
has. While a simple approach would use `LISTIFY` to them do the
initialization on all entries, that is not possible as many stack arrays
are created using expressions instead of literals, such as
`CONFIG_MP_MAX_NUM_CPUS - 1`, which won't work with `LISTIFY`.

Instead, this patch uses a script, `gen_static_shstk_array.py` that
gathers all needed information and patches the ELF to initialize the
stack arrays. Note that this needs to be done before any other operation
on the ELF file that creates new representations, such as the .bin
output.

Signed-off-by: Ederson de Souza <ederson.desouza@intel.com>
2025-09-02 07:56:48 +02:00
Ederson de Souza
10d6ef0ea5 kernel: Option to allow shadow stack to be reused
It seems that, at least on tests, it's common to call k_thread_create()
on a thread multiple times. This trips a check for the CET shadow stack
- namely, set a shadow stack on a thread which already has a shadow
stack.

This patch adds a Kconfig option to allow that, iff the base address and
size of the new shadow stack are the same as before. This will trigger a
reset of the shadow stack, so it can be reused.

It may be the case that this behaviour (reusing threads) is more common
than only for tests, in which case it could make sense to change the
default - in this patch, is only true if ZTEST.

Even if being enabled by default becomes the reality, it would still
make sense to keep this option - more conscious apps could avoid the
need for the shadow stack reset code altogether.

Signed-off-by: Ederson de Souza <ederson.desouza@intel.com>
2025-09-02 07:56:48 +02:00
Ederson de Souza
6a76424177 arch/x86: Use Zephyr HW shadow stack arch interface
So that kernel created threads can use shadow stacks. Note that
CONFIG_X86_CET_SHADOW_STACK is abandoned in favour of
CONFIG_HW_SHADOW_STACK.

This means change some types, functions and macro throughout shadow
stack code.

Signed-off-by: Ederson de Souza <ederson.desouza@intel.com>
2025-09-02 07:56:48 +02:00
Ederson de Souza
5df48fd5d9 arch/x86: Allow SoC to run preparatory steps on shadow stack
Some SoCs may need to do some preparatory work before changing the
current shadow stack pointer (and thus, currently used shadow stack).
This patch adds a way for that, shielded by a Kconfig
(CONFIG_X86_CET_SOC_PREPARE_SHADOW_STACK_SWITCH).

As currently only 32 bit SoC may use this, support is only added to the
32 bit code.

Signed-off-by: Ederson de Souza <ederson.desouza@intel.com>
2025-09-02 07:56:48 +02:00
Ederson de Souza
ad93c6a1ac arch/x86: Support shadow stack on IRQ on ia32
An IRQ shadow stack is created to be used by IRQ.

Signed-off-by: Ederson de Souza <ederson.desouza@intel.com>
2025-09-02 07:56:48 +02:00
Ederson de Souza
82de622d23 arch/x86: Support shadow stack on ia32
Most notable difference on base support is the need to keep the shadow
stack tokens, which are 8 bytes, 8 bytes aligned. Some helper macros are
used for that.

Also, an `ssp` entry is added to the task state segment (TSS).

Signed-off-by: Ederson de Souza <ederson.desouza@intel.com>
2025-09-02 07:56:48 +02:00
Ederson de Souza
1c7dc6930a arch/x86: Indirect Branch Tracking support
Indirect Branch Tracking (IBT) is one of the capabilities provided by
Intel Control-flow Enforcement Technology (CET), aimed at defending
against Jump/Call Oriented Programming.

This patch enables it for x86 (32-bit, 64-bit support coming in future
patches):

    - Add relevant Kconfigs (everything is behind X86_CET);
    - Code to enable it;
    - Enable compiler flags to enable it;
    - Add `endbr32` instructions to asm code, where needed.

Points in the code where an indirect branch is expected to land need
special instructions that tell the CPU they are valid indirect branch
targets. Those are added by the compiler, so toolchain support is
necessary. Note that any code added to the final ELF also need those
markers, such as libc or libgcc.

Finally, tests added to ensure IBT behaves sanely.

Signed-off-by: Ederson de Souza <ederson.desouza@intel.com>
2025-09-02 07:56:48 +02:00
Daniel Leung
d36813a27c x86: add exception handling for control protection exception
This adds exception handling of control protection exception
in fatal code.

Signed-off-by: Daniel Leung <daniel.leung@intel.com>
Signed-off-by: Ederson de Souza <ederson.desouza@intel.com>
2025-09-02 07:56:48 +02:00
Keith Packard
9051e2ea3e toolchain/gcc: Add "memory" clobber to asm traps before CODE_UNREACHABLE
GCC 14.3 will happily delete any code that appears before
__builtin_unreachable that isn't separated with an obvious branch. That
includes __asm__ statements, even those which generate traps.

The failure case that I debugged was on x86 in
z_check_stack_sentinel. There is a store to restore the sentinel to the
correct value just before the ARCH_EXCEPT, and that macro emits 'int $32'
followed by CODE_UNREACHABLE. Because the compiler didn't understand that
ARCH_EXCEPT was changing execution flow, it decided that the sentinel
restoring store 'couldn't' be reached and elided it.

I added the "memory" clobber to the asm statement in ARCH_EXCEPT before
CODE_UNREACHABLE to enforce that all pending store operations be performed
before the asm statement occurs. This ensures that they are not deleted by
the compiler.

I think this might be a GCC bug. The GCC documentation explicitly documents
that asm statements which change the flow of control should be followed by
__builtin_unreachable.

Signed-off-by: Keith Packard <keithp@keithp.com>
2025-08-18 22:01:08 +02:00
Peter Mitsis
d397a91c62 kernel: Add arch_coprocessors_disable()
The intent of arch_coprocessors_disable() is to replace
arch_float_disable() in halt_thread() for the FPU will not
always be the only coprocessor that will need to be disabled.

Signed-off-by: Peter Mitsis <peter.mitsis@intel.com>
2025-07-20 12:25:17 -04:00
Anas Nashif
fbb701371d x86: use EXCEPTION_DUMP macro for fatal messages
instead of depending on logging for exceptions and fatal messages, use
macro and dump exceptions if printk is enable with no logging.

Signed-off-by: Anas Nashif <anas.nashif@intel.com>
2025-07-19 15:49:28 -04:00
Peter Mitsis
c6bc09223e kernel: Move current_fp field out of z_kernel
The current_fp field in the z_kernel structure is only used
by 32-bit x86 (which does not support SMP). As such, it should
reside in the arch specific of section of _kernel.cpus[0].

This also changes the name of 'current_fp' to 'fpu_owner' to
be more consistent with other architectures.

Signed-off-by: Peter Mitsis <peter.mitsis@intel.com>
2025-03-14 05:47:10 +01:00
Nicolas Pitre
46aa6717ff Revert "arch: deprecate _current"
Mostly a revert of commit b1def7145f ("arch: deprecate `_current`").

This commit was part of PR #80716 whose initial purpose was about providing
an architecture specific optimization for _current. The actual deprecation
was sneaked in later on without proper discussion.

The Zephyr core always used _current before and that was fine. It is quite
prevalent as well and the alternative is proving rather verbose.
Furthermore, as a concept, the "current thread" is not something that is
necessarily architecture specific. Therefore the primary abstraction
should not carry the arch_ prefix.

Hence this revert.

Signed-off-by: Nicolas Pitre <npitre@baylibre.com>
2025-01-10 07:49:08 +01:00
Yong Cong Sin
b1def7145f arch: deprecate _current
`_current` is now functionally equals to `arch_curr_thread()`, remove
its usage in-tree and deprecate it instead of removing it outright,
as it has been with us since forever.

Signed-off-by: Yong Cong Sin <ycsin@meta.com>
Signed-off-by: Yong Cong Sin <yongcong.sin@gmail.com>
2024-11-23 20:12:24 -05:00
Daniel Leung
a3f4251ed5 x86: coredump: support dumping privilege stack
Adds the bits to support dumping privilege stack during
coredump.

Signed-off-by: Daniel Leung <daniel.leung@intel.com>
2024-09-21 11:29:39 +02:00
Anas Nashif
7e225efab7 arch: initialize irq_offload during boot, do not use SYS_INIT
Do not use SYS_INIT for initializing irq_offload when enabled, instead
using a new interface that is called during the boot process for all
architectures.

Signed-off-by: Anas Nashif <anas.nashif@intel.com>
2024-09-17 20:05:22 -04:00
Yong Cong Sin
42362c6fcc subsys/profiling: relocate stack unwind backends
Relocate stack unwind backends from `arch/` to perf's
`backends/` folder, just like logging/shell/..

Signed-off-by: Yong Cong Sin <ycsin@meta.com>
Signed-off-by: Yong Cong Sin <yongcong.sin@gmail.com>
2024-08-20 14:45:23 +02:00
Mikhail Kushnerov
140f9a57b7 arch: x86: ia32: Implement perf stack thrace func
Implement stack trace function for x86_32 arch, that get required
thread register values and unwind stack with it.

Signed-off-by: Mikhail Kushnerov <m.kushnerov@yadro.com>
2024-08-13 18:28:44 -04:00
Jordan Yates
07870934e3 everywhere: replace double words
Treewide search and replace on a range of double word combinations:
    * `the the`
    * `to to`
    * `if if`
    * `that that`
    * `on on`
    * `is is`
    * `from from`

Signed-off-by: Jordan Yates <jordan@embeint.com>
2024-06-22 05:40:22 -04:00
Daniel Leung
2d2bbc05d6 kernel: mm: rename Z_VM_KERNEL to K_MEM_IS_VM_KERNEL
This is part of a series to move memory management functions
away from the z_ namespace and into its own namespace.

Signed-off-by: Daniel Leung <daniel.leung@intel.com>
2024-06-12 21:13:26 -04:00
Daniel Leung
db9d3134c5 kernel: mm: rename Z_MEM_PHYS/VIRT_ADDR to K_MEM_*
This is part of a series to move memory management functions
away from the z_ namespace and into its own namespace. Also
make documentation available via doxygen.

Signed-off-by: Daniel Leung <daniel.leung@intel.com>
2024-06-12 21:13:26 -04:00
Flavio Ceolin
a3de27fce9 x86: pm: Don't use deprecated function
Use pm_system_resume instead of z_pm_save_idle_exit

Signed-off-by: Flavio Ceolin <flavio.ceolin@intel.com>
2024-06-05 17:36:22 -05:00
Yong Cong Sin
6a3cb93d88 arch: remove the use of z_arch_esf_t completely from internal
Created `GEN_OFFSET_STRUCT` & `GEN_NAMED_OFFSET_STRUCT` that
works for `struct`, and remove the use of `z_arch_esf_t`
completely.

Signed-off-by: Yong Cong Sin <ycsin@meta.com>
2024-06-04 14:02:51 -05:00
Yong Cong Sin
e54b27b967 arch: define struct arch_esf and deprecate z_arch_esf_t
Make `struct arch_esf` compulsory for all architectures by
declaring it in the `arch_interface.h` header.

After this commit, the named struct `z_arch_esf_t` is only used
internally to generate offsets, and is slated to be removed
from the `arch_interface.h` header in the future.

Signed-off-by: Yong Cong Sin <ycsin@meta.com>
2024-06-04 14:02:51 -05:00
Daniel Leung
027a1c30cd x86: add support for memory mapped stack for threads
This adds the necessary bits to enable memory mapping thread
stacks on both x86 and x86_64. Note that currently these do
not support multi level mappings (e.g. demand paging and
running in virtual address space: qemu_x86/atom/virt board)
as the mapped stacks require actual physical addresses.

Signed-off-by: Daniel Leung <daniel.leung@intel.com>
2024-04-10 07:44:27 -04:00
Daniel Leung
b69d2486fe kernel: rename Z_KERNEL_STACK_BUFFER to K_KERNEL_STACK_BUFFER
Simple rename to align the kernel naming scheme. This is being
used throughout the tree, especially in the architecture code.
As this is not a private API internal to kernel, prefix it
appropriately with K_.

Signed-off-by: Daniel Leung <daniel.leung@intel.com>
2024-03-27 19:27:10 -04:00
Daniel Leung
2ab367d149 x86: ia32/gdbstub: remove dead code
There is logically dead code which will never run. So remove.

Fixes #66848

Signed-off-by: Daniel Leung <daniel.leung@intel.com>
2024-01-08 20:54:16 -06:00
Anas Nashif
fb19d532ed arch: x86: z_x86_prep_c -> z_prep_c
Rename to use common naming for z_prep_c applied to all architectures.

Signed-off-by: Anas Nashif <anas.nashif@intel.com>
2023-12-11 18:23:52 -05:00
Dmitrii Golovanov
f308299ca2 debug: gdbstub: kconfig: Add GDBSTUB_TRACE config option
Add GDBSTUB_TRACE config option to extend GDB backend debug logging
for remote commands received and to debug the GDB stub itself.

Signed-off-by: Dmitrii Golovanov <dmitrii.golovanov@intel.com>
2023-12-06 17:52:18 +00:00
Daniel Leung
c972ef1a0f kernel: mm: move kernel mm functions under kernel includes
This moves the k_* memory management functions from sys/ into
kernel/ includes, as there are kernel public APIs. The z_*
functions are further separated into the kernel internal
header directory.

Also made a quick change to doxygen to group sys_mem_* into
the OS Memory Management group so they will appear in doc.

Signed-off-by: Daniel Leung <daniel.leung@intel.com>
2023-11-20 09:19:14 +01:00
Umar Nisar
31a6594212 drivers: loapic: add device tree support for loapic
As per #26393, Local APIC is using Kconfig based option for
the base address. This patch adds DTS binding support in the driver,
just like its conunter part I/O APIC.

Signed-off-by: Umar Nisar <umar.nisar@intel.com>
2023-09-01 16:36:18 +02:00
Anas Nashif
6baa622958 arch: move exc_handle.h under zephyr/arch/common
This header is private and included only in architecture code, no need for
it to be in the top of the public include directory.

Note: This might move to a more private location later. For now just
cleaning up the obvious issues.

Signed-off-by: Anas Nashif <anas.nashif@intel.com>
2023-08-31 09:19:19 -04:00
Daniel Leung
16bd0c861f x86: rename shadow variables
Rename shadow variables found by -Wshadow.

Signed-off-by: Daniel Leung <daniel.leung@intel.com>
2023-08-22 11:39:58 +02:00
Flavio Ceolin
596e77f562 x86: Early TLS initialization
Allow early boot code using thread local storage when
it is enabled.

Signed-off-by: Flavio Ceolin <flavio.ceolin@intel.com>
2023-08-08 19:08:04 -04:00
Qipeng Zha
1c095239b7 arch: x86: ia32: don't create FP context for NULL thread
Enhance for cases when call z_float_enable() with NULL thread.

Signed-off-by: Dong Wang <dong.d.wang@intel.com>
Signed-off-by: Qipeng Zha <qipeng.zha@intel.com>
2023-05-24 12:41:06 -04:00
Qipeng Zha
d963767369 arch: x86: add irq runtime statistics
Unlike tracing module mainly for debug usage, this is
to allow runtime profiling IRQ performance data, and
target to enable it in product release since platform
can choose to make it work with low weight protocol.

Enable this option and implement runtime_irq_stats()
in platform code, such as Intel ISH platform implement
with SHMI protocol to allow host profiling irq stats.

Signed-off-by: Qipeng Zha <qipeng.zha@intel.com>
2023-05-22 13:29:14 -04:00
Qipeng Zha
c626bac016 arch: x86: fix SSE init issue when enable paging
With paging config, need to use physical address as
paging is not enabled here.

From IA manual, LDMXCSR instruction description is,
Loads the source operand into the MXCSR control/status
register, the source operand is a 32-bit memory location.

Signed-off-by: Qipeng Zha <qipeng.zha@intel.com>
2023-05-08 16:55:27 -04:00
Flavio Ceolin
a6b4af4a93 ia32: irq: Remove unnecessary header
printk is not used anywhere in this file. Just remove it.

Signed-off-by: Flavio Ceolin <flavio.ceolin@intel.com>
2023-01-09 12:07:28 -05:00
Keith Packard
e802d53900 x86: buf return from gdb_reg_readone is not a string
The buffer contents returned from arch_gdb_reg_readone is a counted array
of bytes, not a C string. Use memcpy instead of strcpy for the failure
return path to avoid compiler warning about missing NUL termination.

Signed-off-by: Keith Packard <keithp@keithp.com>
2022-10-26 12:00:04 +02:00
Johann Fischer
3c971307dc arch/kernel/soc/samples: use unsigned int for irq_lock()
irq_lock() returns an unsigned integer key.
Generated by spatch using semantic patch
scripts/coccinelle/irq_lock.cocci

Signed-off-by: Johann Fischer <johann.fischer@nordicsemi.no>
2022-07-14 14:37:13 -05:00
Gerard Marull-Paretas
4b91c2d79f asm: update files with <zephyr/...> include prefix
Assembler files were not migrated with the new <zephyr/...> prefix.
Note that the conversion has been scripted, refer to #45388 for more
details.

Signed-off-by: Gerard Marull-Paretas <gerard.marull@nordicsemi.no>
2022-05-09 12:45:29 -04:00
Gerard Marull-Paretas
16811660ee arch: migrate includes to <zephyr/...>
In order to bring consistency in-tree, migrate all arch code to the new
prefix <zephyr/...>. Note that the conversion has been scripted, refer
to zephyrproject-rtos#45388 for more details.

Signed-off-by: Gerard Marull-Paretas <gerard.marull@nordicsemi.no>
2022-05-06 19:57:22 +02:00
Stephanos Ioannidis
f9a3f02b86 x86: Initialise FPU regs during thread creation for eager FPU sharing
When "eager FPU sharing" mode is enabled, FPU registers must be
initialised at the time of thread creation because the floating-point
context is always active and no further FPU initialisation is performed
later.

Note that, in case of the "lazy FPU sharing" mode, floating-point
context is inactive by default and the FPU is initialised when the
first floating-point instruction is executed.

Refer to the issue #44902 for more details.

Signed-off-by: Stephanos Ioannidis <root@stephanos.io>
2022-04-18 17:23:48 -07:00
Tomasz Bursztyka
f19f9db8df arch/x86: Expand cpu boot argument
In order to mitigate at runtime whether it booted on multiboot or EFI,
let's introduce a dedicated x86 cpu argument structure which holds the
type and the actual pointer delivered by the method (multiboot_info, or
efi_system_table)

Signed-off-by: Tomasz Bursztyka <tomasz.bursztyka@linux.intel.com>
2022-03-22 09:56:54 -04:00
Nazar Kazakov
9713f0d47c everywhere: fix typos
Fix a lot of typos

Signed-off-by: Nazar Kazakov <nazar.kazakov.work@gmail.com>
2022-03-14 20:22:24 -04:00
Daniel Leung
25f87aac87 x86: remove @return doc for void functions
For functions returning nothing, there is no need to document
with @return, as Doxgen complains about "documented empty
return type of ...".

Signed-off-by: Daniel Leung <daniel.leung@intel.com>
2022-01-12 16:02:16 -05:00