As part of an effort to abstract away the use of _sched_spinlock in
the kernel, this commit introduces z_reschedule_locked(). Callers
must already have _sched_spinlock held.
Signed-off-by: Peter Mitsis <peter.mitsis@intel.com>
As part of an effort to abstract away the use of _sched_spinlock in
the kernel, this commit introduces z_swap_locked(). Callers must
already have _sched_spinlock held.
Signed-off-by: Peter Mitsis <peter.mitsis@intel.com>
The scheduler's spinlock was originally confined to sched.c. However,
as the kernel has evolved and functionality has been moved around,
not only have its references proliferated, but the header files that
reference it have become somewhat brittle. This commits adds a new
private header that will abstract away most of the scheduler's
spinlock references to help keep things cleaner and applies them to
the kernel.
Signed-off-by: Peter Mitsis <peter.mitsis@intel.com>
The existing implementation unconditionally restores owner_orig_prio on
unlock, ignoring other held mutexes that still have high-priority waiters.
This causes incorrect priority restoration in nested mutex scenarios and
requires mutexes to be released in strict reverse acquisition order.
Additionally, priority boosts were not propagated through ownership chains
when the mutex owner was itself blocked on another mutex.
This change adds per-thread held_mutexes tracking and mutex_pended_on
pointer to enable:
- Correct priority recalculation on unlock by scanning all remaining held
mutexes
- Chained priority inheritance through the full ownership chain
- Deadlock detection: assert on K_FOREVER circular ownership where
every cycle member also waits forever; bounded chain walk prevents
livelock on cycles not involving the current thread
- Per-thread orig_prio field records true pre-inheritance priority,
fixing priority floor when mutexes are released in non-LIFO order
struct k_thread grows by three pointers plus a byte in default builds
(held_mutexes, mutex_pended_on, orig_prio).
Signed-off-by: Mayur Salve <msalve@qti.qualcomm.com>
Calling thread_schedule_new with a timeout not equal to K_NO_WAIT will
result in a z_add_timeout call, in the much common configuration
CONFIG_SYS_CLOCK_EXISTS=y, which returns early on K_FOREVER making the
compare redundant.
When !CONFIG_SYS_CLOCK_EXISTS the behavior is kept such that a timeout
value of K_FOREVER still results in a nop.
Signed-off-by: Emil Hammarström <emil.a.hammarstrom@gmail.com>
The non-SMP branch only asserted that cpu is 0 and silently
returned CPU 0 statistics for any out-of-range value with
CONFIG_ASSERT=n. The SMP branch had no validation at all and
passed the value through to z_sched_cpu_usage(), indexing the
per-CPU array out of bounds.
Validate cpu against arch_num_cpus() with CHECKIF() returning
-EINVAL, consistent with the existing NULL check for stats, and
unify the SMP and non-SMP branches since arch_num_cpus() is 1 in
the latter case.
Assisted-by: Claude:claude-fable-5
Signed-off-by: Anas Nashif <anas.nashif@intel.com>
With CONFIG_SPIN_VALIDATE, z_assert_can_swap() & halt_thread() check if
the value set in base.swap_data corresponds to &z_spinlock_abort_sentinel
(it will be set to this value in
subsys/testsuite/ztest/src/ztest_error_hook.c if there is a fatal error
or assert).
But nobody is initializing this (struct k_thread).base.swap_data to
anything.
So when there is no failures (or no ztest code), we are checking a random
value and potentially doing weird things.
Let's initialize it to NULL to avoid this.
This check was added in d844a4a861
Signed-off-by: Alberto Escolar Piedras <alberto.escolar.piedras@nordicsemi.no>
Header file include/zephyr/sys_clock.h is deprecated and will be removed
someday. Update the whole file tree to include zephyr/sys/clock.h
straight instead of zephyr/sys_clock.h.
This change was made running the sed shell command below:
$ sed -i 's/zephyr\/sys_clock\.h/zephyr\/sys\/clock\.h/' \
`grep -rsl "zephyr/sys_clock\.h" kernel/`
Signed-off-by: Etienne Carriere <etienne.carriere@st.com>
The per-CPU loops in k_thread_runtime_stats_all_get() and the runtime
stats enable/disable paths used a uint8_t index while the loop bound
arch_num_cpus() is unsigned int.
Widen the index to unsigned int to match the bound type.
Signed-off-by: Anas Nashif <anas.nashif@intel.com>
setup_thread_stack() interleaved stack-geometry math, optional virtual
memory mapping, and several independent #ifdef'd steps (init fill,
sentinel, TLS/local-data/random headroom, stack info) into one body,
juggling five locals across a dozen preprocessor branches.
Introduce a small struct stack_geometry and split the work into focused
helpers: compute_stack_geometry(), map_thread_stack(), fill_init_stack(),
set_stack_sentinel(), reserve_stack_headroom() and set_stack_info().
Each optional step keeps its #ifdef internally and collapses to an
ARG_UNUSED no-op when disabled, so the change is cosmetic with no
runtime cost. setup_thread_stack() is now a short linear sequence and
the headroom accounting (which must stay ordered: TLS, then local data,
then randomization, then alignment) is isolated in one place.
Assisted-by: Claude:claude-opus-4-8
Signed-off-by: Anas Nashif <anas.nashif@intel.com>
z_setup_new_thread() had grown into a ~140-line body dominated by
roughly fifteen #ifdef/#endif pairs interleaved with the actual setup
logic, making the initialization sequence hard to follow.
Push each optional-feature block down into its own static inline
helper (init_thread_obj_core, init_thread_userspace, init_thread_name,
add_thread_to_monitor, init_thread_usage, and so on). Each helper keeps
its #ifdef internally and collapses to an ARG_UNUSED no-op when its
feature is disabled, so the change is purely cosmetic with no runtime
cost. The function body now reads as a linear sequence of calls.
The only block that affects control flow, the
CONFIG_ARCH_HAS_CUSTOM_SWAP_TO_MAIN early return, is converted to an
IS_ENABLED() guard since _current and resource_pool always exist; it
dead-code-eliminates when the option is off. A no-op stub is added for
setup_shadow_stack so its call site needs no #ifdef either.
Assisted-by: Claude:claude-opus-4-8
Signed-off-by: Anas Nashif <anas.nashif@intel.com>
The strncpy + NUL-termination + arch_thread_name_set() hook sequence
behind CONFIG_THREAD_NAME was duplicated in z_impl_k_thread_name_set()
and z_setup_new_thread(). Extract it into a single set_thread_name()
helper and call it from both places, removing the duplicated logic and
the risk of the two copies drifting apart.
The helper treats a NULL name as "clear the name" (writes an empty
string), matching the previous z_setup_new_thread() behavior; the
k_thread_name_set() path always passed a non-NULL string, so its
behavior is unchanged.
Assisted-by: Claude:claude-opus-4-8
Signed-off-by: Anas Nashif <anas.nashif@intel.com>
Threads registered their object core type through a dedicated SYS_INIT.
Unlike the other object types they have no statically defined instances
to walk at boot: each thread links its own object core as it is created
in z_setup_new_thread(). The boot init therefore only registered the type
and its stats descriptor.
Add a K_OBJ_TYPE_DEFINE_TYPE_ONLY() variant that registers the type (and
optional stats descriptor) without walking a static object section, and
use it for threads. The type is still registered at PRE_KERNEL_1 via the
single object core init walk, before any thread is created, so ordering is
unchanged. Only the internal cpu/kernel system objects (which are not in
iterable sections) now remain outside the table.
Assisted-by: Claude:claude-opus-4-8
Signed-off-by: Anas Nashif <anas.nashif@intel.com>
k_thread_runtime_stack_safety_threshold_check() passed the thread's
unused stack threshold directly as the scan window size to
z_stack_space_get(). When CONFIG_STACK_SENTINEL is enabled,
z_stack_space_get() reserves the first 4 bytes of the stack buffer for
the sentinel and shrinks its scan window by that amount. As a result a
fully unused window reported "threshold - 4" unused bytes, which is
always less than the threshold, so the handler fired on every call
regardless of the actual unused stack space, making the abbreviated
check useless on sentinel-based platforms.
Grow the requested scan window by the sentinel reservation so that up to
"threshold" bytes of the usable stack are actually examined. The
firing condition is unchanged on platforms without the sentinel.
Assisted-by: Claude:claude-opus-4-8
Signed-off-by: Anas Nashif <anas.nashif@intel.com>
The runtime stack safety threshold syscalls were declared in kernel.h as
k_thread_runtime_stack_unused_threshold_{pct_set,set,get}, but their
implementations and verifiers in thread.c were named with an extra
"safety_" infix (k_thread_runtime_stack_safety_unused_threshold_*). Since
the __syscall declarations drive code generation, the generated _mrsh.c
files and the expected z_impl_/z_vrfy_ symbols never matched the
definitions, so the feature failed to build/link with
CONFIG_THREAD_RUNTIME_STACK_SAFETY and CONFIG_USERSPACE enabled.
Rename the implementations and verifiers to match the public API names
and fix the corresponding generated-header includes.
Also remove a duplicate, malformed z_vrfy for the threshold "get"
syscall that returned int instead of size_t and called the "set"
implementation with the wrong number of arguments. The correct verifier
is already defined alongside the other threshold syscalls.
Assisted-by: Claude:claude-opus-4-8
Signed-off-by: Anas Nashif <anas.nashif@intel.com>
thread_obj_validate()'s default case passed the (non-zero) error code
`ret` straight to K_SYSCALL_VERIFY_MSG(), which verifies its argument is
true. A non-zero `ret` therefore reads as "verified OK", so K_OOPS()
never raised and control fell through to CODE_UNREACHABLE. With GCC this
path isn't normally reached and __builtin_unreachable() is a no-op; with
Clang it traps (unimp), turning an access-denied into an illegal
instruction. Verify `ret == 0` so the oops is actually raised.
Signed-off-by: Anas Nashif <anas.nashif@intel.com>
Migrate scheduler-internal callers of z_abort_thread_timeout() to the
new z_try_abort_thread_timeout(). This covers the abort sites in
sched.c, thread.c, scheduler.c (z_sched_wake), events.c, and pipe.c.
The patterns used:
z_unpend_thread (sched.c) retries on -EAGAIN: if the timeout
handler is in flight on another CPU, drop _sched_spinlock so the
handler can run to completion and retry. This preserves 1b8c7a3's
unpend+abort atomicity from the caller's perspective.
halt_thread (sched.c) takes the caller's sched-lock key as a pointer
so its direct abort on the dying thread can retry on -EAGAIN.
Waiting for the handler is mandatory: a caller may free the thread's
storage as soon as halt_thread() returns, and without waiting, the
still-in-flight handler would later dereference freed memory.
_THREAD_DEAD is set before the abort, so the handler bails via the
killed check in z_sched_wake_thread_locked().
For next_up() (the scheduler-context caller), the key is not cleanly
available: K_SPINLOCK in z_get_next_switch_handle and do_swap's
(void)k_spin_lock both discard it. halt_thread is invoked on
_current with NULL key and no abort is performed -- _current is
running, so its base.timeout cannot be linked. The gap is closed at
the other end: z_thread_halt() spins on z_try_abort_thread_timeout()
outside any lock after the halt-queue wait completes, before
returning to the caller of k_thread_abort().
z_unpend_all_locked / unpend_all (sched.c) skip the local
ready_thread() on -EAGAIN and let the still-blocked handler ready
the thread when _sched_spinlock drops. The threads being woken are
not freed, so no UAF risk; end state is identical.
z_impl_k_wakeup (thread.c), z_sched_wake (scheduler.c) and
event_walk_op (events.c) perform the wake entirely under
_sched_spinlock, so a (void) abort is race-free -- a racing in-flight
handler is blocked on the same lock during the wake.
copy_to_pending_readers (pipe.c) is restructured to also wake the
reader under the scheduler lock instead of after it, so the
return-value set, unpend, abort, and ready all happen atomically.
The dticks-cancel check in z_thread_timeout() is preserved for now
because other callers (sem.c, mutex.c, ... via z_unpend_first_thread())
still use z_abort_thread_timeout() and rely on it for race protection.
A follow-up commit migrates those, and a final commit drops the
cancel check and removes z_abort_thread_timeout() itself.
Add z_try_abort_thread_timeout() as an inline wrapper.
Signed-off-by: Nicolas Pitre <npitre@baylibre.com>
Since commit 37717b229f ("sys: util: rename Z_MIN Z_MAX Z_CLAMP to min
max and clamp"), <zephyr/sys/util.h> unconditionally defines function-
like macros named `min`, `max`, and `clamp` in the global namespace (in
C mode). util.h gets pulled in transitively by very broad headers,
including the POSIX layer's <pthread.h>, so any third-party C code that
uses these names as ordinary identifiers (e.g. XNNPACK's static `clamp`
helper and its public `clamp` struct field) fails to build as soon as
<pthread.h> is included.
Following the approach used by Linux, move the lowercase `min`, `max`,
`min3`, `max3`, and `clamp` macros (and their helpers) into a new
<zephyr/sys/minmax.h> header that has to be included explicitly by
source files that want them. util.h keeps the uppercase MIN/MAX/CLAMP,
so most code is unaffected; only the (much smaller) set of files that
actually use the lowercase variants needs to pick up the new include.
Fixes#107853.
Signed-off-by: Nicolas Pitre <npitre@baylibre.com>
Inside k_thread_name_copy(), we call k_object_find() to find
the associated thread object of the incoming thread. However,
the finder can return NULL if incoming pointer address has
no kobj associated. So we need to check for NULL before
dereferencing k_object to look inside. Since k_object_find()
returns NULL if input object is NULL, there is no need to
specifically test thread pointer for NULL, and only need to
check for the return of k_object_find().
Signed-off-by: Daniel Leung <daniel.leung@intel.com>
Replace the lock/test/restore dance used to probe the current IRQ state
with a direct non-modifying read:
- z_spin_is_locked() (UP path) simply negates
arch_cpu_irqs_are_enabled().
- k_can_yield() and z_smp_cpu_mobile() likewise drop their lock/unlock
pair.
- arch_spin_relax() asserts IRQs are disabled without the sneaky
unpaired arch_irq_lock() it used to rely on.
- tests/arch/arm/arm_no_multithreading: same simplification on a
probe-only assertion.
Signed-off-by: Nicolas Pitre <npitre@baylibre.com>
Relocate k_thread_start(), k_thread_abort(), k_thread_suspend(), and
k_thread_resume() from sched.c to thread.c alongside related thread
lifecycle code.
Signed-off-by: Anas Nashif <anas.nashif@intel.com>
When CONFIG_WAITQ_SCALABLE=y, wake up all threads from a post-waitq-walk
callback which is invoked while the scheduler spinlock is still held. This
solves the race condition that was worked around via `no_wake_in_timeout`
flag in k_thread and `is_timeout` parameter of z_sched_wake_thread_locked()
which can now both be dropped.
Signed-off-by: Mathieu Choplain <mathieu.choplain-ext@st.com>
Upgrades the thread user_options to 16 bits from an 8-bit value to
provide more space for future values.
Also, as the size of this field has changed, the values for the
existing architecture specific thread options have also shifted
from the upper end of the old 8-bit field, to the upper end of
the new 16-bit field.
Fixes#101034
Signed-off-by: Peter Mitsis <peter.mitsis@intel.com>
Instead of performing a linear search to determine if a given
thread is running on another CPU, or if it is marked as being
preempted by a metaIRQ on any CPU do this in O(1) time.
On SMP systems, Zephyr already tracks the CPU on which a thread
executes (or lasted executed). This information is leveraged to
do the search in O(1) time.
Signed-off-by: Peter Mitsis <peter.mitsis@intel.com>
arch_mem_coherent() is cache related so it is better to move it
under cache subsys. It is renamed to sys_cache_is_mem_coherent()
to reflect this change.
The only user of arch_mem_coherent() is Xtensa. However, it is
not an architecture feature. That's why it is moved to the cache
subsys.
Signed-off-by: Daniel Leung <daniel.leung@intel.com>
Adds support for thread runtime stack safety. This kernel feature
allows a developer to run enhanced stack usage checks on threads
such that if the amount of unused stack space drops below a thread's
configured threshold, it will invoke a custom handler/callback.
This can be used by monitoring software to log warnings, suspend
or abort threads, or even reboot the system.
Signed-off-by: Peter Mitsis <peter.mitsis@intel.com>
The switch_handle for the outgoing thread is expected to be NULL
at the start of a context switch.
The previous code performed a redundant assignment to NULL.
This change replaces the assignment with an __ASSERT(). This makes the
code more robust by explicitly enforcing this precondition, helping to
catch potential scheduler bugs earlier.
Also, the switch_handle pointer is used to check a thread's state during a
context switch. For dummy threads, this pointer was left uninitialized,
potentially holding a unexpected value.
Set the handle to NULL during initialization to ensure these threads are
handled safely and predictably.
Signed-off-by: TaiJu Wu <tjwu1217@gmail.com>
Replace all in-function instances of MIN/MAX/CLAMP with the single
evaluation version min/max/clamp.
There's probably no race conditions in these files, but the single
evaluation ones save a couple of instructions each so they should save
few code bytes and potentially perform better, so they should be
preferred in general.
Signed-off-by: Fabio Baltieri <fabiobaltieri@google.com>
This patch modifies thread stack macros (such as K_KERNEL_STACK_DECLARE
or K_KERNEL_STACK_ARRAY_DECLARE) to also create a HW shadow stack (when
CONFIG_HW_SHADOW_STACK=y), as well as define a pairing between the
thread stack (or thread stack array) and the shadow stack (or shadow
stack array).
This pairing, which currently is simply an array of pairs (stack,
shadow_stack) is searched during thread setup to find the corresponding
shadow stack and attach it to the thread. If linear search on this array
proves to be a performance issue, the actual structure can be revisited.
To define the size of the shadow stack for a given stack, the stack size
is used. A new Kconfig, CONFIG_HW_SHADOW_STACK_PERCENTAGE_SIZE is used
to define how big the shadow stack is compared to the stack. Note that
this size is in *addition* to the stack size. To avoid some shadow
stacks becoming too small, CONFIG_HW_SHADOW_STACK_MIN_SIZE is used to
define a minimum size. Note that after this size is defined, platform
restrictions on the size of the shadow stack are still applied.
Signed-off-by: Ederson de Souza <ederson.desouza@intel.com>
K_KERNEL_STACK_RESERVED can be 0 which can give a warning with
-Wtype-limits. Only perform the check if ARCH_KERNEL_STACK_RESERVED
is set. Also remove the the unncessary sets in arch.h where it's
manually set to 0, it defaults to 0 anyways.
Signed-off-by: Ryan McClelland <ryanmcclelland@meta.com>
This function is getting quite involved and it also gained more callers
lately. This is not performance critical so Uninline it to save on
binary size.
Signed-off-by: Nicolas Pitre <npitre@baylibre.com>
Mostly a revert of commit b1def7145f ("arch: deprecate `_current`").
This commit was part of PR #80716 whose initial purpose was about providing
an architecture specific optimization for _current. The actual deprecation
was sneaked in later on without proper discussion.
The Zephyr core always used _current before and that was fine. It is quite
prevalent as well and the alternative is proving rather verbose.
Furthermore, as a concept, the "current thread" is not something that is
necessarily architecture specific. Therefore the primary abstraction
should not carry the arch_ prefix.
Hence this revert.
Signed-off-by: Nicolas Pitre <npitre@baylibre.com>
Sleeping and suspended are now orthogonal states. That is, a thread
may be both sleeping and suspended and the two do not interact. One
repercussion of this is that suspending a thread will no longer
abort its timeout.
Threads are now created in the 'sleeping' state instead of a
'suspended' state. This dovetails nicely with the start delay that
can be given to a newly created thread--it is as though the very
first operation that a thread with a start delay is a sleep.
Signed-off-by: Peter Mitsis <peter.mitsis@intel.com>
At the present time, Zephyr does has overlap between sleeping and
suspending. Not only should sleeping and suspended be orthogonal
states, but we should ensure users always employ the correct API.
For example, to wake a sleeping thread, k_wakeup() should be used,
and to resume a suspended thread, k_thread_resume() should be used.
However, at the present time k_thread_resume() can be used on a
thread that called k_sleep(K_FOREVER). Sleeping should have nothing
to do with suspension.
This commit introduces the new _THREAD_SLEEPING thread state along
with some prep-work to facilitate the decoupling of the sleeping and
suspended thread states.
Signed-off-by: Peter Mitsis <peter.mitsis@intel.com>
Traditionally threads have been initialized with a PRESTART flag set,
which gets cleared when the thread runs for the first time via either
its timeout or the k_thread_start() API.
But if you think about it, this is no different, semantically, than
SUSPENDED: the thread is prevented from running until the flag is
cleared.
So unify the two. Start threads in the SUSPENDED state, point
everyone looking at the PRESTART bit to the SUSPENDED flag, and make
k_thread_start() be a synonym for k_thread_resume().
There is some mild code size savings from the eliminated duplication,
but the real win here is that we make space in the thread flags byte,
which had run out.
Signed-off-by: Andy Ross <andyross@google.com>
This table just blindly assumed that the values of _THREAD_xxx state
bits wouldn't change. That's dangerous. (That's right, I tried to
change them and broke it.)
Signed-off-by: Andy Ross <andyross@google.com>
`_current` is now functionally equals to `arch_curr_thread()`, remove
its usage in-tree and deprecate it instead of removing it outright,
as it has been with us since forever.
Signed-off-by: Yong Cong Sin <ycsin@meta.com>
Signed-off-by: Yong Cong Sin <yongcong.sin@gmail.com>
Add the following arch-specific APIs:
- arch_curr_thread()
- arch_set_curr_thread()
which allow SMP architectures to implement a faster "get current
thread pointer" than the default provided by the kernel. The 'set'
function is required for the 'get' to work, more on that later.
When `CONFIG_ARCH_HAS_CUSTOM_CURRENT_IMPL` is selected, calls to
`_current` & `k_sched_current_thread_query()` will be redirected to
`arch_curr_thread()`, which ideally should translate into a single
instruction read, avoiding the current
"lock > read CPU > read current thread > unlock" path in SMP
architectures and thus greatly improves the read performance.
However, since the kernel relies on a copy of the "current thread"s on
every CPU for certain operations (i.e. to compare the priority of the
currently scheduled thread on another CPU to determine if IPI should be
sent), we can't eliminate the copy of "current thread" (`current`) from
the `struct _cpu` and therefore the kernel now has to invoke
`arch_set_curr_thread()` in addition to what it has been doing. This
means that it will take slightly longer (most likely one instruction
write) to change the current thread pointer on the current
CPU.
Signed-off-by: Yong Cong Sin <ycsin@meta.com>
Signed-off-by: Yong Cong Sin <yongcong.sin@gmail.com>
For code clarity, remove unnecessary `return` statements
in functions with a void return type they don't affect control flow.
Signed-off-by: Pisit Sawangvonganan <pisit@ndrsolution.com>
The intention of this API is to allow setting the posix thread
name equal to the zephyr thread name.
By defining it as an arch interface, the implementation becomes
generic.
Signed-off-by: Rubin Gerritsen <rubin.gerritsen@nordicsemi.no>
These functions were introduced alongside with the memory mapped
stack feature, and are currently only being used there only.
To avoid potential confusion with k_mem_un/map(), remove them
and use k_mem_map/unmap_phys_guard() directly instead.
Signed-off-by: Daniel Leung <daniel.leung@intel.com>
Namespaced the generated headers with `zephyr` to prevent
potential conflict with other headers.
Introduce a temporary Kconfig `LEGACY_GENERATED_INCLUDE_PATH`
that is enabled by default. This allows the developers to
continue the use of the old include paths for the time being
until it is deprecated and eventually removed. The Kconfig will
generate a build-time warning message, similar to the
`CONFIG_TIMER_RANDOM_GENERATOR`.
Updated the includes path of in-tree sources accordingly.
Most of the changes here are scripted, check the PR for more
info.
Signed-off-by: Yong Cong Sin <ycsin@meta.com>
- modified parameter types to receive a const pointer when a
non-const pointer is not needed
- avoided redundant casts
Signed-off-by: Hess Nathan <nhess@baumer.com>
This introduces support for memory mapped thread stacks,
where each thread stack is mapped into virtual memory
address space with two guard pages to catch
under-/over-flowing the stack. This is just on the kernel
side. Additional architecture code is required to fully
support this feature.
Signed-off-by: Daniel Leung <daniel.leung@intel.com>
