With CONFIG_SPIN_VALIDATE, z_assert_can_swap() & halt_thread() check if
the value set in base.swap_data corresponds to &z_spinlock_abort_sentinel
(it will be set to this value in
subsys/testsuite/ztest/src/ztest_error_hook.c if there is a fatal error
or assert).
But nobody is initializing this (struct k_thread).base.swap_data to
anything.
So when there is no failures (or no ztest code), we are checking a random
value and potentially doing weird things.
Let's initialize it to NULL to avoid this.
This check was added in d844a4a861
Signed-off-by: Alberto Escolar Piedras <alberto.escolar.piedras@nordicsemi.no>
Moves the assert checking that exactly one CPU in the mask is set
to be inside the locked section. This closes a hole where another
CPU could have called cpu_mask_mod() and performed an OR operation
setting more bits just before the current CPU read the cpu_mask
field leading to an incorrect assert evaulation.
Signed-off-by: Peter Mitsis <peter.mitsis@intel.com>
Header file include/zephyr/sys_clock.h is deprecated and will be removed
someday. Update the whole file tree to include zephyr/sys/clock.h
straight instead of zephyr/sys_clock.h.
This change was made running the sed shell command below:
$ sed -i 's/zephyr\/sys_clock\.h/zephyr\/sys\/clock\.h/' \
`grep -rsl "zephyr/sys_clock\.h" kernel/`
Signed-off-by: Etienne Carriere <etienne.carriere@st.com>
A blocking pend from ISR context is a programming error with no safe
recovery: it attempts to sleep whatever thread was interrupted and, on
CONFIG_SWAP_NONATOMIC, corrupts the scheduler (the qnode_dlist
double-link / NULL-deref traced in #111518).
Catch it at the funnel instead of per-API: every blocking primitive
(sem, mutex, msgq, queue, stack, pipe, poll, events, mailbox, mem_slab,
kheap, futex, ...) routes through z_pend_curr, and reaching it always
means a real block (callers handle K_NO_WAIT beforehand). So one check
there covers them all, with no per-API return-contract churn:
if (arch_is_in_isr()) {
__ASSERT(false, "blocking pend from ISR context");
k_panic();
}
__ASSERT(false, ...) gives the message and backtrace in debug builds;
k_panic() makes it fatal in every build, closing the production gap
(CONFIG_ASSERT defaults to n) that let the misuse ship and run for
hours. Located before _sched_spinlock is acquired, so no lock is held
on the panic path.
Also assert in add_to_waitq_locked() that a thread is not already on a
wait queue when added to a new one (pended_on == NULL) -- belt-and-
suspenders for a double-pend that reaches the scheduler despite the
above, e.g. a non-current thread re-pended without an intervening
unpend.
Suggested-by: Nicolas Pitre <npitre@baylibre.com>
Signed-off-by: Tibor Kiss <kiss.tibor@gmail.com>
z_add_timeout returns without any effect if timeout is equal to K_FOREVER,
and thus z_add_thread_timeout does this by extension and there is no need
to guard its timeout parameter from a K_FOREVER.
Signed-off-by: Emil Hammarström <emil.a.hammarstrom@gmail.com>
Move the declaration of arch_page_phys_get() from the internal
kernel_arch_interface.h to the public arch_interface.h, under the
existing arch-mmu group, so that drivers and subsystems can query the
physical address of an already-mapped virtual page without depending
on private kernel headers.
The declaration stays unconditional, as it was in the internal header:
callers such as munmap() in subsys/portability/posix/options/mmap.c
reference the function even when CONFIG_MMU is disabled (guarded only
by a runtime IS_ENABLED() check), so hiding the declaration behind
CONFIG_MMU breaks the build on MMU-less platforms.
Clean up the existing users that resorted to including the internal
header or to extern declarations:
- subsys/portability/posix/options/mmap.c
- subsys/portability/posix/options/shm.c
- subsys/portability/cmsis_rtos_v1/cmsis_thread.c
Fixes#113314
Signed-off-by: Hongquan Li <hongquan.li@processmission.com>
There is no need to grab CPU ID from _current_cpu->id down in
the ipi_work_process() call chain, as it can be passed as
a function argument and can be reused. Using _current_cpu can
be costly with assertion on as it calls z_smp_cpu_mobile for
each call.
Signed-off-by: Daniel Leung <daniel.leung@intel.com>
z_main_thread is only needed if multithreading is enabled, so move its
definition inside a multithreading check.
Signed-off-by: Josh DeWitt <josh.dewitt@garmin.com>
With low-power idle handoff now going through sys_clock_idle_enter(),
nothing passes idle == true to sys_clock_set_timeout() any more, so the
argument is dead. Drop it: sys_clock_set_timeout(uint32_t ticks) now
means exactly "program the next tick, N ticks out", nothing else.
This updates the prototype, the weak default, every in-tree timer driver
definition (including the out-of-tree-style board timer under boards/),
and the core call sites. For the five drivers with low-power behaviour
this only removes the now-unused idle argument that the previous change
left on set_timeout(); the handling itself stays in their
sys_clock_idle_enter().
Signed-off-by: Nicolas Pitre <npitre@baylibre.com>
The "no timeout pending, stop the clock" decision under
CONFIG_SYSTEM_CLOCK_SLOPPY_IDLE was expressed by next_timeout()
returning SYS_CLOCK_MAX_WAIT verbatim as a magic sentinel, which every
timer driver then had to recognise. Move the decision into the core and
give it an explicit, resumable interface.
Add a weak sys_clock_unused() hook (no-op default): the kernel calls it
when the timeout list is empty and sloppy idle allows uptime to drift,
in place of programming a wait. A driver may override it to actively
halt its counter; one that does not simply stops being reprogrammed and
quiesces on its own, so sloppy idle now works for every driver. Resume
is the next sys_clock_set_timeout(), exactly as before; the core keeps
no paused state.
next_timeout() no longer special-cases sloppy idle, so its empty-list
and far-timeout arms collapse to the same capped budget and
SYS_CLOCK_MAX_WAIT loses its sentinel meaning. The decision lives in
reprogram_next(), used only at the two sites where the list can drain
(abort, end of announce); the add path always has a pending timeout and
calls sys_clock_set_timeout() directly.
Signed-off-by: Nicolas Pitre <npitre@baylibre.com>
The per-CPU loops in k_thread_runtime_stats_all_get() and the runtime
stats enable/disable paths used a uint8_t index while the loop bound
arch_num_cpus() is unsigned int.
Widen the index to unsigned int to match the bound type.
Signed-off-by: Anas Nashif <anas.nashif@intel.com>
The guard page checks in k_mem_unmap_phys_guard() asserted
ret == 0 inside an if (ret == 0) branch, so the assertions
could never fire and the "cannot find guard page" diagnostic
was unreachable in any build.
Assert the intended invariant (ret != 0, i.e. the guard page
is unmapped) before the branch, matching the assert-then-
handle pattern used elsewhere in this function. Behavior with
CONFIG_ASSERT=n is unchanged.
Signed-off-by: Anas Nashif <anas.nashif@intel.com>
A CPU's usage counters only advance when it context switches, or when it
reports on itself through z_sched_cpu_usage(). Reading the runtime stats
of a *different* CPU therefore misses whatever time that CPU has spent in
its current thread since then.
The effect is worst for an idle CPU: a CPU that ran briefly and then went
idle has the active cycles of that run in its counters, but none of the
idle time that followed, because the idle thread has not been switched
out yet. Both k_thread_runtime_stats_cpu_get() and everything built on it
therefore see a window made up purely of execution cycles, i.e. a 100%
load for a CPU that is doing nothing.
Fold the in-progress cycles into the returned view, attributing them to
the idle thread or to the CPU depending on what it is currently running.
This is done for the returned stats only, without mutating the other
CPU's accounting, as that CPU will account for the same cycles itself
once it next updates. The read is consistent because usage_lock, which is
held here, also covers the usage0 timestamp and the counters.
Reading the current CPU is unchanged.
Assisted-by: Claude:claude-opus-4-8
Signed-off-by: Anas Nashif <anas.nashif@intel.com>
Several documentation defects in the two canonical arch interface
headers:
- The arch-timing group was defined twice, once in arch_interface.h
and once in kernel_arch_interface.h; turn the latter into an
addtogroup of the former.
- The arch-smp group was opened with addtogroup before its defgroup
appeared 300 lines later in the same file, leaving the SMP APIs
split across two blocks with the group defined mid-file; define
the group at first use and addtogroup at the second block.
- ARCH_STACK_PTR_ALIGN was the only stack macro documented with a
bare @def block and no __DOXYGEN__ stub definition, making it an
orphan @def; add the stub like its siblings.
- The ARCH_PCIE_IRQ_CONNECT doc block was wrapped in #ifdef
CONFIG_PCIE with no __DOXYGEN__ stub, so it never appeared in doc
builds; use the same __DOXYGEN__ pattern as the other macro slots.
- The arch-stackwalk group was closed with a malformed comment
block; use the standard group close.
- Fix a #return that should be @return in the
arch_thread_priv_stack_space_get() documentation.
In addition, give the ARCH_* macro slots real documentation: most of
them (ARCH_THREAD_STACK_RESERVED, ARCH_IRQ_CONNECT,
ARCH_PCIE_IRQ_CONNECT, ARCH_IRQ_DIRECT_CONNECT, the
ARCH_ISR_DIRECT_* family) carried nothing but a @see reference to
their public counterpart, leaving the porting-facing contract
undocumented. Describe what each hook must do and document its
parameters; add @brief lines to the stack alignment macros.
No functional change, comments and preprocessor-invisible
documentation stubs only.
Assisted-by: Claude:claude-fable-5
Signed-off-by: Anas Nashif <anas.nashif@intel.com>
signal_pending_ipi() used atomic_clear() on _kernel.pending_ipi, which
also drained the calling CPU's own pending bit. With directed IPIs,
arch_sched_directed_ipi() skips the calling CPU, so any IPI that had
been raised for self by another CPU was silently dropped. The lost
wakeup left the target CPU stuck until something else nudged the
scheduler.
This was latent for the existing call site in ipi_work_process(), but
became reachable on every no-swap path through do_swap() once
signal_pending_ipi() was added there, producing intermittent CPU1
starvation on SMP. tests/kernel/spinlock/spinlock_api test_trylock
(and test_spinlock_bounce on cortex_a53) reproduced it reliably on
qemu_riscv32/smp, qemu_riscv64/smp, and qemu_cortex_a53/smp: CPU0's
loop would acquire bounce_lock 10000 times in a row without CPU1
ever winning a single acquisition.
Switch to atomic_and() with the self bit as the mask, which preserves
our own pending bit (so the next interrupt entry on this CPU still
sees it and runs the scheduler) while atomically draining the bits
for the other CPUs we are about to dispatch IPIs to.
Assisted-by: Claude:claude-opus-4.7
Signed-off-by: Tom Burdick <thomas.burdick@infineon.com>
USE_SWITCH and USE_SWITCH_SUPPORTED describe a general context-switch
primitive (_arch_switch) that is usable on uniprocessor systems, not
just SMP configurations. Having them under the "SMP Options" menu in
kernel/smp/Kconfig was misleading.
Move both symbols to kernel/Kconfig proper, just before the SMP menu is
sourced. SMP still depends on USE_SWITCH; the guarding and arch selects
are unchanged.
Signed-off-by: Anas Nashif <anas.nashif@intel.com>
setup_thread_stack() interleaved stack-geometry math, optional virtual
memory mapping, and several independent #ifdef'd steps (init fill,
sentinel, TLS/local-data/random headroom, stack info) into one body,
juggling five locals across a dozen preprocessor branches.
Introduce a small struct stack_geometry and split the work into focused
helpers: compute_stack_geometry(), map_thread_stack(), fill_init_stack(),
set_stack_sentinel(), reserve_stack_headroom() and set_stack_info().
Each optional step keeps its #ifdef internally and collapses to an
ARG_UNUSED no-op when disabled, so the change is cosmetic with no
runtime cost. setup_thread_stack() is now a short linear sequence and
the headroom accounting (which must stay ordered: TLS, then local data,
then randomization, then alignment) is isolated in one place.
Assisted-by: Claude:claude-opus-4-8
Signed-off-by: Anas Nashif <anas.nashif@intel.com>
z_setup_new_thread() had grown into a ~140-line body dominated by
roughly fifteen #ifdef/#endif pairs interleaved with the actual setup
logic, making the initialization sequence hard to follow.
Push each optional-feature block down into its own static inline
helper (init_thread_obj_core, init_thread_userspace, init_thread_name,
add_thread_to_monitor, init_thread_usage, and so on). Each helper keeps
its #ifdef internally and collapses to an ARG_UNUSED no-op when its
feature is disabled, so the change is purely cosmetic with no runtime
cost. The function body now reads as a linear sequence of calls.
The only block that affects control flow, the
CONFIG_ARCH_HAS_CUSTOM_SWAP_TO_MAIN early return, is converted to an
IS_ENABLED() guard since _current and resource_pool always exist; it
dead-code-eliminates when the option is off. A no-op stub is added for
setup_shadow_stack so its call site needs no #ifdef either.
Assisted-by: Claude:claude-opus-4-8
Signed-off-by: Anas Nashif <anas.nashif@intel.com>
Add a single-level bucketed delta list as a fourth selectable timeout
backend: a simpler relative of the timer wheel that keeps the wheel's
O(1) near-future insertion without its second tier, sift/defer
machinery, or tickless-idle penalty.
Timeouts expiring within the next CONFIG_TIMEOUT_BUCKET_LISTS ticks go
into a per-tick bucket list (O(1) insert and remove), tracked by an
occupancy bitmap so "ticks until next expiry" is O(1). Everything
beyond goes into one overflow list sorted by absolute expiry (O(n)
insert, O(1) remove, no delta fix-up). As curr_tick crosses the bucket
window, overflow entries that have come within range migrate into
buckets.
Like the other backends it is a single implementation header
(kernel/timeout_bucket.h) included only by timeout.c. Two properties
distinguish it from the other non-default backends, both confirmed by
running the full kernel timer/context/common suites with the backend
forced on (qemu_x86, x86_64, cortex_a53 SMP, riscv64):
- It fits the generic next_gap/advance/pop_due announce loop -- no
backend-owned announce. Migration is on demand against the actual
next event, so an idle CPU with a distant timeout sleeps straight
to it: it passes tests/kernel/context cpu_idle / timer_interrupts,
which the wheel fails (the wheel wakes at least every 32 ticks).
- Same-tick firing order stays FIFO (bucket and overflow inserts
append; migration preserves order), so tests/kernel/common's
timeout_order passes, unlike the min-heap.
The per-node representation is the delta list's node + dticks with no
extra field: a bucket entry stores its bucket index (< BUCKET_LISTS) in
dticks, an overflow entry its absolute expiry (>= BUCKET_LISTS); the
ranges never overlap, so it shares the dlist per-node helpers. Absolute
expiry needs 64-bit ticks, so the backend depends on TIMEOUT_64BIT. It
inherits the shared z_add_timeout round-up and the inflight_timeout
synchronization like the other backends. EXPERIMENTAL.
Signed-off-by: Nicolas Pitre <npitre@baylibre.com>
Add a hierarchical timer wheel as a third selectable timeout backend.
Timeouts are bucketed by expiry distance: one list per tick for the
next 32 ticks ("soon"), one list per 32-tick band for the next ~1024
("later"), and a sorted overflow list ("distant"). Insertion and
removal are O(1) for the common near-future case; every 32 ticks the
announce path sifts the current "later" band into "soon" and refills
it from "distant". This scales well when many short-lived timeouts are
pending.
Unlike the dlist and min-heap backends, the wheel does not fit the
generic next_gap/advance/pop_due announce primitives: its per-tick
advance is a bitmap scan that jumps over empty ticks, and its sift is
a time-driven event tied to no single timeout. Rather than contort
that (already subtle) state machine, the wheel uses the backend-owned
announce escape hatch: it defines _TIMEOUT_BACKEND_OWNS_ANNOUNCE and
implements z_timeout_q_announce(), which sys_clock_announce_locked()
calls in place of the generic loop.
Like the other backends the wheel is a single implementation header
(kernel/timeout_wheel.h) included only by timeout.c, so its state, its
operations and that announce loop reach the shared state (curr_tick,
announce_remaining, inflight_timeout) directly; nothing extra needs
exposing. The SMP re-entry guard, announcing_cpu and the reprogram
remain in timeout.c. The wheel fires handlers through the same
inflight_timeout dance, so the post-#109977 abort/in-flight
synchronization works unchanged, and it carries no per-node
ANNOUNCING/ABORTED sentinels.
struct _timeout grows a wheel-only flags field (which wheel tier a
timeout occupies), gated by CONFIG_TIMEOUT_BACKEND_WHEEL so the dlist
and heap builds are unaffected.
The backend is EXPERIMENTAL. Two known limitations, inherent to the
wheel algorithm (not the abstraction):
- No same-tick firing-order guarantee (sifted timeouts are
prepended), so the timeout_order test does not apply.
- next_timeout() never exceeds 32 ticks because a sift is always
pending, so the wheel wakes a tickless-idle CPU at least every 32
ticks. This fails tests that assert zero spurious idle wakeups
(tests/kernel/context cpu_idle / timer_interrupts) and is a power
regression versus the dlist and heap backends.
Verified the timeout-functional suites (timer_api, timeout,
timepoints, sleep, sched/deadline) pass with the wheel on qemu_x86,
x86_64, cortex_a53 SMP and riscv64; dlist and heap remain unaffected.
The timer-wheel data structure, bucketing scheme and sift algorithm
are the work of Peter Mitsis (PR #108339), re-homed here behind the
timeout backend interface.
Co-authored-by: Peter Mitsis <peter.mitsis@intel.com>
Signed-off-by: Nicolas Pitre <npitre@baylibre.com>
Add a binary min-heap as a selectable timeout backend alongside the
sorted delta list, plugging into the backend abstraction rather than
forking timeout.c. Pending timeouts are kept in a min-heap keyed on
absolute expiry tick; insertion and arbitrary removal are O(log n),
which scales better than the delta list's O(n) insertion when many
timeouts are pending.
The backend is a new kernel/timeout_minheap.h: the heap instance (a
min_heap_ref from the previous commit), its comparator and the
z_timeout_q_*() operations, included only by timeout.c (after curr_tick)
so the whole backend stays private to that translation unit. struct
_timeout gains the abs_ticks + heap_handle representation, selected by
Kconfig (the delta list's node + dticks is the #else), with the common
fn pointer kept as a shared trailing member; the per-node helpers in
timeout_q.h grow a matching min-heap variant. The kernel shell thread
dump prints the backend's raw scheduling field (dticks or abs_ticks)
under #ifdef.
Because the shared z_add_timeout() already applies the post-#107452
conditional tick round-up, the heap inherits it: the backend's insert
simply stores abs_ticks = curr_tick + dticks. Likewise in-flight
handler synchronization (PR #109977) lives in timeout.c, so the heap
node carries no ANNOUNCING/ABORTED sentinels -- "not queued" is just
heap_handle.idx == 0.
The backend is EXPERIMENTAL, depends on TIMEOUT_64BIT (absolute ticks
need 64-bit precision), and uses a fixed-capacity heap
(CONFIG_TIMEOUT_HEAP_MAX_ENTRIES) whose overflow is a fatal error.
The min-heap algorithm, struct fields, Kconfig and capacity model are
derived from Sayooj K Karun's min-heap timeout subsystem (#106013),
reworked here to fit the pluggable backend and the current in-tree
in-flight-handler synchronization.
Co-authored-by: Sayooj K Karun <sayooj@aerlync.com>
Signed-off-by: Nicolas Pitre <npitre@baylibre.com>
The timeout queue data structure and the front-end logic in timeout.c
are entangled through first()/next()/remove_timeout()/next_timeout()
and the open-coded insertion and announce loops. This makes it hard to
offer alternative queue implementations (min-heap, timer wheel) without
either forking timeout.c or threading #ifdefs through its trickiest
code, the announce path in particular.
Introduce a backend interface and move the sorted delta list behind it.
The split mirrors how the queue is actually used:
- The per-node helpers z_init_timeout() / z_is_inactive_timeout()
operate on a single struct _timeout and are needed tree-wide
(timer.c, work.c, poll.c, ...), so they live in timeout_q.h and
depend only on struct _timeout's fields.
- The queue itself (its instance and the z_timeout_q_*() operations)
is used only by timeout.c, so it becomes a backend implementation
header (kernel/timeout_list.h) that timeout.c includes directly,
after its shared state. The header is private to that translation
unit; no other file sees the queue or its instance, so the instance
is plain static and needs no extern.
timeout.c keeps all shared state (curr_tick, announce_remaining,
announcing_cpu, inflight_timeout, timeout_lock), the add/abort/query API
translation, the in-flight handler synchronization, and a single
announce loop shared by every future backend.
The backend exposes:
- z_timeout_q_insert / _remove add and arbitrary remove
- z_timeout_q_remainder / _next_timeout queries
- z_timeout_q_next_gap / _advance / _pop_due announce-loop primitives
The announce loop is recast in terms of the last three primitives:
find the gap to the next event, advance the backend (and curr_tick) to
it, then drain whatever is due. For the delta list this is behaviorally
identical to the previous open-coded loop: the residual advance reuses
the same head->dticks fixup, same-tick ties drain via successive
pop_due() calls, and announce_remaining is still decremented per tick
group while announcing_cpu carries the announcing state.
So the front end no longer reaches into the queue representation, three
checks are made backend-neutral: z_add_timeout()'s assert and
z_timer_expiration_handler()'s "restarted?" test now use
z_is_inactive_timeout() instead of sys_dnode_is_linked(), and
Z_TIMER_INITIALIZER zero-inits the queue fields (.fn only) rather than
naming .node/.dticks. struct _timeout is otherwise untouched; no
sentinel or per-node state changes.
The Kconfig backend choice and the divergent node layouts are deferred
until a second backend lands; with only the delta list present they
would be churn with no benefit.
Verified no behavioral change: tests/kernel/timer/{timer_api,timeout},
tests/kernel/common, and tests/kernel/sleep pass on qemu_x86,
qemu_x86_64, qemu_cortex_a53 (SMP) and qemu_riscv64 (exercising the
SMP announcing_cpu path and 64-bit dticks).
Signed-off-by: Nicolas Pitre <npitre@baylibre.com>
Static heap init was converted from SYS_INIT(PRE_KERNEL_1, OBJECTS) to a
K_KERNEL_INIT_PRE() hook. Those hooks run before any PRE_KERNEL_1
SYS_INIT, which broke heap KASAN: K_HEAP_KASAN_ENABLE() registers a
heap's shadow at (PRE_KERNEL_1, 0), relying on running before static
heap init. With the hook, sys_heap_init() ran first, heap->kasan_ba
stayed NULL and tracking was silently disabled. On the system heap this
let the tests/lib/heap_kasan overflow cases corrupt real chunk metadata
and panic with "heap corruption (free chunk linkage)".
Revert only the kheap portion of that conversion and keep static heap
init as SYS_INIT(PRE_KERNEL_1, CONFIG_KERNEL_INIT_PRIORITY_OBJECTS),
restoring the ordering the KASAN shadow registration depends on. The
mem_slab and mailbox conversions in the same series are unaffected and
stay on K_KERNEL_INIT_PRE. A comment records why kheap must remain a
SYS_INIT so it is not reconverted.
Assisted-by: Claude:claude-opus-4-8
Signed-off-by: Anas Nashif <anas.nashif@intel.com>
Convert the two userspace PRE_KERNEL_1 SYS_INIT users to
K_KERNEL_INIT_PRE() hooks:
userspace.c app_shmem_bss_zero zero the BSS of app shared-memory
regions
mem_domain.c init_mem_domain_module set max partitions and build the
default memory domain
This removes the last SYS_INIT users in kernel/, so kernel-internal init
no longer uses SYS_INIT at all.
Assisted-by: Claude:claude-opus-4-8
Signed-off-by: Anas Nashif <anas.nashif@intel.com>
The kernel object is a singleton (_kernel) whose object core type was
registered and linked from init.c. It is fully self-contained -- its
k_obj_type is referenced only by that registration -- so move it next to
the object core framework in obj_core.c, alongside the registration walk.
obj_core.c already includes kernel_internal.h (for the kernel init hook
and the kernel stats helpers) and reaches _kernel via kernel_structs.h, so
no new dependencies are introduced. CONFIG_OBJ_CORE_SYSTEM depends on
CONFIG_OBJ_CORE, so obj_core.c is always compiled when this block is built.
Assisted-by: Claude:claude-opus-4-8
Signed-off-by: Anas Nashif <anas.nashif@intel.com>
Convert the object core registration walk from SYS_INIT (PRE_KERNEL_1) to
a K_KERNEL_INIT_PRE() hook. This was the last SYS_INIT in kernel/*.c, so
kernel-internal initialization no longer uses SYS_INIT at all.
obj_core.c is linked whenever the object core framework is used (its
helpers are referenced by every object's init), so the walk runs whenever
CONFIG_OBJ_CORE is enabled, exactly as before -- only the registration
mechanism changes.
Assisted-by: Claude:claude-opus-4-8
Signed-off-by: Anas Nashif <anas.nashif@intel.com>
Convert the statically defined kernel objects' boot-time init from
SYS_INIT (PRE_KERNEL_1) to K_KERNEL_INIT_PRE() hooks:
kheap build the free block list for each static heap
mem_slab build the free block list for each static slab
mailbox build the pool of asynchronous message descriptors
These initializers live in their respective translation units, which are
linked only when the application references the corresponding APIs, so
each init runs only when its subsystem is actually used (pay-per-use),
matching the previous SYS_INIT behavior without the level/priority.
Assisted-by: Claude:claude-opus-4-8
Signed-off-by: Anas Nashif <anas.nashif@intel.com>
Replace the system work queue's SYS_INIT (POST_KERNEL) with a
K_KERNEL_INIT_POST() hook. The init function and its hook entry live in
this translation unit, which is linked only when something references the
system work queue (e.g. k_sys_work_q), so an image that never uses it
neither links this code nor runs this init -- the same pay-per-use
behavior SYS_INIT gave, now without the init level/priority.
The hook runs before POST_KERNEL device init so that drivers initialized
there may submit work to the queue.
Assisted-by: Claude:claude-opus-4-8
Signed-off-by: Anas Nashif <anas.nashif@intel.com>
Add a lean, kernel-only initialization model to replace the kernel's
internal use of SYS_INIT. Kernel init order is fixed at compile time, so
these hooks drop the SYS_INIT level/priority machinery: a subsystem
registers a parameterless init function that the boot path runs at one of
two fixed phases.
K_KERNEL_INIT_PRE(fn) run in z_cstart() before PRE_KERNEL device init
K_KERNEL_INIT_POST(fn) run in bg_thread_main() before POST_KERNEL
device init
Each entry is a single function pointer placed, via an iterable section,
in the registering subsystem's own translation unit. The linker pulls
that unit (and the entry) into the image only when the subsystem is
otherwise referenced, so an init runs only when its subsystem is actually
linked. This preserves the pay-per-use linkage that SYS_INIT provided
while halving the per-entry size (one pointer versus init_entry's two) and
removing the level/priority sort. The boot-path walks reference only the
section bounds, so they never force a subsystem to be linked.
Assisted-by: Claude:claude-opus-4-8
Signed-off-by: Anas Nashif <anas.nashif@intel.com>
Extend the cpu_mask field in struct _thread_base to support up to
32 CPUs by selecting uint32_t when CONFIG_MP_MAX_NUM_CPUS > 16.
Fix undefined behavior in IPI_ALL_CPUS_MASK where shifting 1 by
CONFIG_MP_MAX_NUM_CPUS (which can be 32) overflows uint32_t.
Replace with UINT32_MAX >> (32 - N) to produce the correct mask.
Signed-off-by: Guang Li <guang.li@jaguarmicro.com>
z_try_abort_timeout() can only set ret to -EAGAIN inside a
CONFIG_SMP-gated branch, so the trailing arch_spin_relax() call is
dead code on non-SMP builds. Optimized builds drop it, but a
coverage build (-O0) keeps the call and the linker then fails with
an undefined reference to arch_spin_relax: its weak definition
lives in idle.c, which is only compiled when CONFIG_MULTITHREADING
is set, so a no-multithreading + coverage build has no definition.
Wrap the check with IS_ENABLED(CONFIG_SMP) so the front end folds
the branch (and the arch_spin_relax reference) away even at -O0.
This is behavior-preserving since ret is never -EAGAIN without SMP.
The issue also triggers when building with llvm.
Signed-off-by: Anas Nashif <anas.nashif@intel.com>
z_time_slice() computed z_time_slice_size(curr) to decide whether the
slice had expired, and then z_time_slice_reset() computed it a second
time to rearm the slice timeout. Cache the value instead: split the
rearm body into a slice_reset(slice_size) helper (z_time_slice_reset()
becomes a thin wrapper that still computes the size for its other
callers) and pass the already-known size from z_time_slice().
The size is still only computed when the slice has actually expired, so
the per-tick fast path is unchanged. In the CONFIG_TIMESLICE_PER_THREAD
case the expiry handler runs with the scheduler lock dropped and may
change the thread's slice configuration, so the cached value is
recomputed after the handler returns; behavior is therefore unchanged.
Assisted-by: Claude:claude-opus-4-8
Signed-off-by: Anas Nashif <anas.nashif@intel.com>
The strncpy + NUL-termination + arch_thread_name_set() hook sequence
behind CONFIG_THREAD_NAME was duplicated in z_impl_k_thread_name_set()
and z_setup_new_thread(). Extract it into a single set_thread_name()
helper and call it from both places, removing the duplicated logic and
the risk of the two copies drifting apart.
The helper treats a NULL name as "clear the name" (writes an empty
string), matching the previous z_setup_new_thread() behavior; the
k_thread_name_set() path always passed a non-NULL string, so its
behavior is unchanged.
Assisted-by: Claude:claude-opus-4-8
Signed-off-by: Anas Nashif <anas.nashif@intel.com>
z_impl_k_thread_stack_free() tested _THREAD_DUMMY and _THREAD_DEAD
with two separate z_is_thread_state_set() calls combined with a
logical OR. z_is_thread_state_set() returns (thread_state & state)
!= 0, so the two calls are equivalent to a single call with both
bits ORed into the mask. Collapse them into one call; this is
smaller and clearer with no change in behavior.
Assisted-by: Claude:claude-opus-4-8
Signed-off-by: Anas Nashif <anas.nashif@intel.com>
The CPU object core type was registered through a dedicated SYS_INIT. Like
threads, CPUs have no statically defined instances to walk at boot: each
CPU links its own object core in z_init_cpu(). The boot init only
registered the type and its stats descriptor.
Use K_OBJ_TYPE_DEFINE_TYPE_ONLY() for the CPU type, dropping
init_cpu_obj_core_list() and its SYS_INIT. The type is still registered at
PRE_KERNEL_1 by the single object core init walk, before z_init_cpu(0)
runs during prepare_multithreading(), so ordering is unchanged. The kernel
system object keeps its own init, as it links the singleton _kernel object.
Assisted-by: Claude:claude-opus-4-8
Signed-off-by: Anas Nashif <anas.nashif@intel.com>
Threads registered their object core type through a dedicated SYS_INIT.
Unlike the other object types they have no statically defined instances
to walk at boot: each thread links its own object core as it is created
in z_setup_new_thread(). The boot init therefore only registered the type
and its stats descriptor.
Add a K_OBJ_TYPE_DEFINE_TYPE_ONLY() variant that registers the type (and
optional stats descriptor) without walking a static object section, and
use it for threads. The type is still registered at PRE_KERNEL_1 via the
single object core init walk, before any thread is created, so ordering is
unchanged. Only the internal cpu/kernel system objects (which are not in
iterable sections) now remain outside the table.
Assisted-by: Claude:claude-opus-4-8
Signed-off-by: Anas Nashif <anas.nashif@intel.com>
mem_slab was the one statically defined object type left out of the
object core registration table, because its boot-time init bundled two
unrelated concerns: building each slab's free block list (create_free_list,
mandatory functional init) and its object core duties (type init, stats
descriptor init, linking and per-slab stats registration).
Separate the two. The object core duties now go through the registration
table like every other object type, and the per-slab statistics buffer is
registered generically by the table walk: the descriptor optionally carries
the offset and size of an embedded stats buffer (only present under
CONFIG_OBJ_CORE_STATS), set via the new K_OBJ_TYPE_DEFINE_STATS() macro.
K_OBJ_TYPE_DEFINE() forwards to it with no per-object stats.
create_free_list stays as its own SYS_INIT: it is required whether or not
the object core framework is enabled (CONFIG_OBJ_CORE is off by default),
so it cannot move into the object-core-gated table walk. mem_slab.c is
linked only when slab APIs are used, so this init carries no extra
footprint for builds that do not use slabs.
Assisted-by: Claude:claude-opus-4-8
Signed-off-by: Anas Nashif <anas.nashif@intel.com>
Each kernel object type that participates in the object core framework
previously supplied its own SYS_INIT routine to initialize its
k_obj_type and to walk its static-object linker section, linking each
object core. These routines were near-identical across 11 object types
and differed only in the type id, the object struct and the obj_core
offset.
Replace that per-type boilerplate with a declarative K_OBJ_TYPE_DEFINE()
macro that emits a const descriptor into a new iterable ROM section, and
walk those descriptors once from a single SYS_INIT in obj_core.c. The
descriptor captures the type storage, type id, obj_core offset, the
static object section bounds and the object stride, which is all the
shared loop needs to initialize and link every statically defined
object.
Converted: condvar, event, fifo, lifo, mailbox, msgq, mutex, pipe, sem,
stack and timer. The non-uniform initializers (thread, mem_slab and the
internal cpu/kernel objects) are left unchanged and will be dealt with
in followup commits.
Footprint on qemu_cortex_m3 (tests/kernel/obj_core/obj_core, all types
enabled): flash 32012 -> 30880 (-1132 B), RAM unchanged. With
CONFIG_OBJ_CORE disabled the image is byte-for-byte identical.
Assisted-by: Claude:claude-opus-4-8
Signed-off-by: Anas Nashif <anas.nashif@intel.com>
sys_clock_announce() reports ticks elapsed since the previous announce. The
quantity that must stay in range is therefore the distance of the next
timeout from that announce, not its distance from "now" as next_timeout()
previously bounded. When an early timeout is repeatedly replaced by a later
one, announces are deferred and that distance can exceed the range even
though each set_timeout() delay was in range, which every driver has had to
defend against on its own.
Cap it in the core instead: SYS_CLOCK_MAX_WAIT becomes UINT32_MAX/2 (the
clamp, with the upper half of the range left as slack for a late announce)
and next_timeout() clamps the inter-announce distance to it. Drivers no
longer need to clamp for the announce range and only have to honour their
own cycle-count limits.
The core no longer emits K_TICKS_FOREVER; the reported delay is always
finite, saturating at SYS_CLOCK_MAX_WAIT. Drivers that stop the timer
entirely when idle now key off ticks == SYS_CLOCK_MAX_WAIT under
CONFIG_SYSTEM_CLOCK_SLOPPY_IDLE.
z_get_next_timeout_expiry() still reports to the idle and PM paths as a
signed int32_t. It keeps its K_TICKS_FOREVER default and adopts the
next_timeout() value only when that fits, so the unsigned cap can never
surface there as a negative number.
Signed-off-by: Nicolas Pitre <npitre@baylibre.com>
The sys_clock_set_timeout(), sys_clock_announce() and
sys_clock_announce_locked() interfaces carry a number of ticks to be
scheduled or announced. Those ticks have no negative meaning, so a signed
argument is both wasteful and error prone: it invites incorrect handling in
drivers and it halves the range that sys_clock_announce() can represent.
Switch the tick argument of these interfaces, along with the internal
announce_remaining accounting, from int32_t to uint32_t. This is a
mechanical change: driver bodies perform plain arithmetic on the value and
are unaffected by the signedness, and the kernel never passes a negative
tick count. The freed sign bit doubles the representable announce range,
which a later change will use to move the announce range limit out of the
drivers and into the core.
One config is not strictly neutral: under CONFIG_TIMEOUT_64BIT,
K_TICKS_FOREVER is a 64-bit value that an unsigned 32-bit argument can no
longer compare equal to, so the K_TICKS_FOREVER tests in drivers stop
matching there. K_TICKS_FOREVER is a k_ticks_t concept and has no business
in this tick count interface; a follow-up removes it from the driver side
entirely, which closes this gap.
Signed-off-by: Nicolas Pitre <npitre@baylibre.com>
Bringing the kernel/ sources into the Doxygen requirements-traceability
INPUT (for @satisfies links) surfaces a handful of latent documentation
warnings that would break the -W docs build. Fix them.
With these, the kernel/ tree is clean under doxygen 1.17.
Assisted-by: Claude:claude-opus-4-8
Signed-off-by: Anas Nashif <anas.nashif@intel.com>
Lightweight write sanitizer for sys_heap. Uses
-fsanitize=kernel-address compiler instrumentation combined with a
per-heap shadow bitarray (one bit per granule) to detect buffer
overflows, underflows, and use-after-free on write accesses.
Ships its own lightweight sanitizer runtime (__asan_store* callbacks);
does not depend on an external ASAN library and supports debugging on
real embedded targets.
Instrumentation is opt-in per CMake target via
zephyr_target_enable_heap_kasan(), or per directory via
zephyr_heap_kasan_enable_directory(). Only writes are checked
(-asan-instrument-reads=0). Bulk-write library calls (memset,
memcpy, str*, printf family) are redirected to checked wrappers at
compile time via -Dfoo=__asan_foo, requiring no source changes in
application code.
Heap tracking is likewise opt-in: register each heap with
SYS_HEAP_KASAN_ENABLE() / K_HEAP_KASAN_ENABLE(), or enable
CONFIG_SYS_HEAP_KASAN_MALLOC / CONFIG_SYS_HEAP_KASAN_SYSTEM for the
common libc malloc and kernel system heaps.
Usage:
CONFIG_SYS_HEAP_KASAN=y
CONFIG_SYS_HEAP_KASAN_MALLOC=y # auto-track malloc/free
CONFIG_SYS_HEAP_KASAN_SYSTEM=y # auto-track k_malloc/k_free
# Instrument all sources of <target> (CMakeLists.txt)
zephyr_target_enable_heap_kasan(app)
# Or instrument sources under <dir>
zephyr_heap_kasan_enable_directory(src/mymodule)
/* Opt-in tracking for a custom heap */
K_HEAP_DEFINE(my_heap, 4096);
K_HEAP_KASAN_ENABLE(my_heap, 4096);
Signed-off-by: Jinming Zhao <jinmzhao@qti.qualcomm.com>
Some private symbol names were widely duplicated throughout the kernel
such as lock and handle_poll_event. This arguably made the kernel less
readable as the locality of the name lock is highly confusing.
Rename all compilation unit locks to match their usage (e.g.
mutex_lock). Improving readability.
Furthermore, by deduplicating these symbols we enable potential
amalgamation builds of the kernel where all C files are merged
into one large C file or compliation unit allowing for better
compiler visibility and optimization.
Signed-off-by: Tom Burdick <thomas.burdick@infineon.com>
The `thread_id` member should be used instead, because this also works for
work queues which run on a thread, that was not started by the work queue
implementation.
Signed-off-by: Michael Zimmermann <michael.zimmermann@sevenlab.de>
The `thread` field is not initialized, if it's being animated via
k_work_queue_run instead of k_work_queue_start.
Signed-off-by: Michael Zimmermann <michael.zimmermann@sevenlab.de>
