Splits z_waitq_head() into two versions: z_waitq_head_locked() and
z_waitq_head(). When the scheduler's spinlock is known to be already
held, z_waitq_head_locked() should be used--otherwise, z_waitq_head()
is to be used.
However, this approach uncovered a path where the scheduler spinlock
could be recursively taken when a thread is aborted. To work around the
recursion (see k_thread_perms_all_clear), knowledge of the scheduler's
spinlock state must be passed to the lower layers for use in the
cleanup routines for message queues, stacks and timers.
Fixes#115756
Signed-off-by: Peter Mitsis <peter.mitsis@intel.com>
As part of an effort to abstract away the use of _sched_spinlock in
the kernel, this commit introduces z_reschedule_locked(). Callers
must already have _sched_spinlock held.
Signed-off-by: Peter Mitsis <peter.mitsis@intel.com>
As part of an effort to abstract away the use of _sched_spinlock in
the kernel, this commit introduces z_swap_locked(). Callers must
already have _sched_spinlock held.
Signed-off-by: Peter Mitsis <peter.mitsis@intel.com>
The scheduler's spinlock was originally confined to sched.c. However,
as the kernel has evolved and functionality has been moved around,
not only have its references proliferated, but the header files that
reference it have become somewhat brittle. This commits adds a new
private header that will abstract away most of the scheduler's
spinlock references to help keep things cleaner and applies them to
the kernel.
Signed-off-by: Peter Mitsis <peter.mitsis@intel.com>
Updates the z_sched_waitq_walk() documentation to indicate that the
the walk_func callback may safefly remove the thread identified by
the callback's argument from the wait queue on the final iteration.
Signed-off-by: Peter Mitsis <peter.mitsis@intel.com>
Calling thread_schedule_new with a timeout not equal to K_NO_WAIT will
result in a z_add_timeout call, in the much common configuration
CONFIG_SYS_CLOCK_EXISTS=y, which returns early on K_FOREVER making the
compare redundant.
When !CONFIG_SYS_CLOCK_EXISTS the behavior is kept such that a timeout
value of K_FOREVER still results in a nop.
Signed-off-by: Emil Hammarström <emil.a.hammarstrom@gmail.com>
The idle path asks this for the time until the next wakeup, and gets a
tick count that never says "there is nothing to wake up for": an empty
timeout list is reported as the capped announce budget, exactly like a
deadline further out than can be programmed in one step. The power
management code cannot then tell the two apart, so it arms the timer in
both cases and a system with sloppy idle enabled keeps waking up for
nothing.
Make the same decision reprogram_next() makes. With
CONFIG_SYSTEM_CLOCK_SLOPPY_IDLE allowing the uptime to drift, an empty
list is reported as K_TICKS_FOREVER; anything else is a wait, whether a
real deadline or the synthetic one that keeps the announce range
covered. Without sloppy idle the empty case stays a wait, so the timer
remains armed and timekeeping is unaffected.
The return type becomes unsigned to match the tick type used throughout
the timer interface. The conversion at the only caller, in the idle
path, is value preserving in both directions: every wait is capped at
SYS_CLOCK_MAX_WAIT, which is INT32_MAX, and K_TICKS_FOREVER is the same
value read either way. Nothing outside the kernel sees the change; the
power management API keeps its signed tick counts.
Signed-off-by: Nicolas Pitre <npitre@baylibre.com>
kernel_offsets.h pulls in three headers before defining its own
include guard, so the guard is not the first thing in the file as
MISRA C:2012 Directive 4.10 requires.
Move the guard above the includes. The header is still only used by
the offsets generator, and the includes stay inside the guard.
Signed-off-by: Anas Nashif <anas.nashif@intel.com>
Move the declaration of arch_page_phys_get() from the internal
kernel_arch_interface.h to the public arch_interface.h, under the
existing arch-mmu group, so that drivers and subsystems can query the
physical address of an already-mapped virtual page without depending
on private kernel headers.
The declaration stays unconditional, as it was in the internal header:
callers such as munmap() in subsys/portability/posix/options/mmap.c
reference the function even when CONFIG_MMU is disabled (guarded only
by a runtime IS_ENABLED() check), so hiding the declaration behind
CONFIG_MMU breaks the build on MMU-less platforms.
Clean up the existing users that resorted to including the internal
header or to extern declarations:
- subsys/portability/posix/options/mmap.c
- subsys/portability/posix/options/shm.c
- subsys/portability/cmsis_rtos_v1/cmsis_thread.c
Fixes#113314
Signed-off-by: Hongquan Li <hongquan.li@processmission.com>
z_main_thread is only needed if multithreading is enabled, so move its
definition inside a multithreading check.
Signed-off-by: Josh DeWitt <josh.dewitt@garmin.com>
Several documentation defects in the two canonical arch interface
headers:
- The arch-timing group was defined twice, once in arch_interface.h
and once in kernel_arch_interface.h; turn the latter into an
addtogroup of the former.
- The arch-smp group was opened with addtogroup before its defgroup
appeared 300 lines later in the same file, leaving the SMP APIs
split across two blocks with the group defined mid-file; define
the group at first use and addtogroup at the second block.
- ARCH_STACK_PTR_ALIGN was the only stack macro documented with a
bare @def block and no __DOXYGEN__ stub definition, making it an
orphan @def; add the stub like its siblings.
- The ARCH_PCIE_IRQ_CONNECT doc block was wrapped in #ifdef
CONFIG_PCIE with no __DOXYGEN__ stub, so it never appeared in doc
builds; use the same __DOXYGEN__ pattern as the other macro slots.
- The arch-stackwalk group was closed with a malformed comment
block; use the standard group close.
- Fix a #return that should be @return in the
arch_thread_priv_stack_space_get() documentation.
In addition, give the ARCH_* macro slots real documentation: most of
them (ARCH_THREAD_STACK_RESERVED, ARCH_IRQ_CONNECT,
ARCH_PCIE_IRQ_CONNECT, ARCH_IRQ_DIRECT_CONNECT, the
ARCH_ISR_DIRECT_* family) carried nothing but a @see reference to
their public counterpart, leaving the porting-facing contract
undocumented. Describe what each hook must do and document its
parameters; add @brief lines to the stack alignment macros.
No functional change, comments and preprocessor-invisible
documentation stubs only.
Assisted-by: Claude:claude-fable-5
Signed-off-by: Anas Nashif <anas.nashif@intel.com>
Add a single-level bucketed delta list as a fourth selectable timeout
backend: a simpler relative of the timer wheel that keeps the wheel's
O(1) near-future insertion without its second tier, sift/defer
machinery, or tickless-idle penalty.
Timeouts expiring within the next CONFIG_TIMEOUT_BUCKET_LISTS ticks go
into a per-tick bucket list (O(1) insert and remove), tracked by an
occupancy bitmap so "ticks until next expiry" is O(1). Everything
beyond goes into one overflow list sorted by absolute expiry (O(n)
insert, O(1) remove, no delta fix-up). As curr_tick crosses the bucket
window, overflow entries that have come within range migrate into
buckets.
Like the other backends it is a single implementation header
(kernel/timeout_bucket.h) included only by timeout.c. Two properties
distinguish it from the other non-default backends, both confirmed by
running the full kernel timer/context/common suites with the backend
forced on (qemu_x86, x86_64, cortex_a53 SMP, riscv64):
- It fits the generic next_gap/advance/pop_due announce loop -- no
backend-owned announce. Migration is on demand against the actual
next event, so an idle CPU with a distant timeout sleeps straight
to it: it passes tests/kernel/context cpu_idle / timer_interrupts,
which the wheel fails (the wheel wakes at least every 32 ticks).
- Same-tick firing order stays FIFO (bucket and overflow inserts
append; migration preserves order), so tests/kernel/common's
timeout_order passes, unlike the min-heap.
The per-node representation is the delta list's node + dticks with no
extra field: a bucket entry stores its bucket index (< BUCKET_LISTS) in
dticks, an overflow entry its absolute expiry (>= BUCKET_LISTS); the
ranges never overlap, so it shares the dlist per-node helpers. Absolute
expiry needs 64-bit ticks, so the backend depends on TIMEOUT_64BIT. It
inherits the shared z_add_timeout round-up and the inflight_timeout
synchronization like the other backends. EXPERIMENTAL.
Signed-off-by: Nicolas Pitre <npitre@baylibre.com>
Add a hierarchical timer wheel as a third selectable timeout backend.
Timeouts are bucketed by expiry distance: one list per tick for the
next 32 ticks ("soon"), one list per 32-tick band for the next ~1024
("later"), and a sorted overflow list ("distant"). Insertion and
removal are O(1) for the common near-future case; every 32 ticks the
announce path sifts the current "later" band into "soon" and refills
it from "distant". This scales well when many short-lived timeouts are
pending.
Unlike the dlist and min-heap backends, the wheel does not fit the
generic next_gap/advance/pop_due announce primitives: its per-tick
advance is a bitmap scan that jumps over empty ticks, and its sift is
a time-driven event tied to no single timeout. Rather than contort
that (already subtle) state machine, the wheel uses the backend-owned
announce escape hatch: it defines _TIMEOUT_BACKEND_OWNS_ANNOUNCE and
implements z_timeout_q_announce(), which sys_clock_announce_locked()
calls in place of the generic loop.
Like the other backends the wheel is a single implementation header
(kernel/timeout_wheel.h) included only by timeout.c, so its state, its
operations and that announce loop reach the shared state (curr_tick,
announce_remaining, inflight_timeout) directly; nothing extra needs
exposing. The SMP re-entry guard, announcing_cpu and the reprogram
remain in timeout.c. The wheel fires handlers through the same
inflight_timeout dance, so the post-#109977 abort/in-flight
synchronization works unchanged, and it carries no per-node
ANNOUNCING/ABORTED sentinels.
struct _timeout grows a wheel-only flags field (which wheel tier a
timeout occupies), gated by CONFIG_TIMEOUT_BACKEND_WHEEL so the dlist
and heap builds are unaffected.
The backend is EXPERIMENTAL. Two known limitations, inherent to the
wheel algorithm (not the abstraction):
- No same-tick firing-order guarantee (sifted timeouts are
prepended), so the timeout_order test does not apply.
- next_timeout() never exceeds 32 ticks because a sift is always
pending, so the wheel wakes a tickless-idle CPU at least every 32
ticks. This fails tests that assert zero spurious idle wakeups
(tests/kernel/context cpu_idle / timer_interrupts) and is a power
regression versus the dlist and heap backends.
Verified the timeout-functional suites (timer_api, timeout,
timepoints, sleep, sched/deadline) pass with the wheel on qemu_x86,
x86_64, cortex_a53 SMP and riscv64; dlist and heap remain unaffected.
The timer-wheel data structure, bucketing scheme and sift algorithm
are the work of Peter Mitsis (PR #108339), re-homed here behind the
timeout backend interface.
Co-authored-by: Peter Mitsis <peter.mitsis@intel.com>
Signed-off-by: Nicolas Pitre <npitre@baylibre.com>
Add a binary min-heap as a selectable timeout backend alongside the
sorted delta list, plugging into the backend abstraction rather than
forking timeout.c. Pending timeouts are kept in a min-heap keyed on
absolute expiry tick; insertion and arbitrary removal are O(log n),
which scales better than the delta list's O(n) insertion when many
timeouts are pending.
The backend is a new kernel/timeout_minheap.h: the heap instance (a
min_heap_ref from the previous commit), its comparator and the
z_timeout_q_*() operations, included only by timeout.c (after curr_tick)
so the whole backend stays private to that translation unit. struct
_timeout gains the abs_ticks + heap_handle representation, selected by
Kconfig (the delta list's node + dticks is the #else), with the common
fn pointer kept as a shared trailing member; the per-node helpers in
timeout_q.h grow a matching min-heap variant. The kernel shell thread
dump prints the backend's raw scheduling field (dticks or abs_ticks)
under #ifdef.
Because the shared z_add_timeout() already applies the post-#107452
conditional tick round-up, the heap inherits it: the backend's insert
simply stores abs_ticks = curr_tick + dticks. Likewise in-flight
handler synchronization (PR #109977) lives in timeout.c, so the heap
node carries no ANNOUNCING/ABORTED sentinels -- "not queued" is just
heap_handle.idx == 0.
The backend is EXPERIMENTAL, depends on TIMEOUT_64BIT (absolute ticks
need 64-bit precision), and uses a fixed-capacity heap
(CONFIG_TIMEOUT_HEAP_MAX_ENTRIES) whose overflow is a fatal error.
The min-heap algorithm, struct fields, Kconfig and capacity model are
derived from Sayooj K Karun's min-heap timeout subsystem (#106013),
reworked here to fit the pluggable backend and the current in-tree
in-flight-handler synchronization.
Co-authored-by: Sayooj K Karun <sayooj@aerlync.com>
Signed-off-by: Nicolas Pitre <npitre@baylibre.com>
The timeout queue data structure and the front-end logic in timeout.c
are entangled through first()/next()/remove_timeout()/next_timeout()
and the open-coded insertion and announce loops. This makes it hard to
offer alternative queue implementations (min-heap, timer wheel) without
either forking timeout.c or threading #ifdefs through its trickiest
code, the announce path in particular.
Introduce a backend interface and move the sorted delta list behind it.
The split mirrors how the queue is actually used:
- The per-node helpers z_init_timeout() / z_is_inactive_timeout()
operate on a single struct _timeout and are needed tree-wide
(timer.c, work.c, poll.c, ...), so they live in timeout_q.h and
depend only on struct _timeout's fields.
- The queue itself (its instance and the z_timeout_q_*() operations)
is used only by timeout.c, so it becomes a backend implementation
header (kernel/timeout_list.h) that timeout.c includes directly,
after its shared state. The header is private to that translation
unit; no other file sees the queue or its instance, so the instance
is plain static and needs no extern.
timeout.c keeps all shared state (curr_tick, announce_remaining,
announcing_cpu, inflight_timeout, timeout_lock), the add/abort/query API
translation, the in-flight handler synchronization, and a single
announce loop shared by every future backend.
The backend exposes:
- z_timeout_q_insert / _remove add and arbitrary remove
- z_timeout_q_remainder / _next_timeout queries
- z_timeout_q_next_gap / _advance / _pop_due announce-loop primitives
The announce loop is recast in terms of the last three primitives:
find the gap to the next event, advance the backend (and curr_tick) to
it, then drain whatever is due. For the delta list this is behaviorally
identical to the previous open-coded loop: the residual advance reuses
the same head->dticks fixup, same-tick ties drain via successive
pop_due() calls, and announce_remaining is still decremented per tick
group while announcing_cpu carries the announcing state.
So the front end no longer reaches into the queue representation, three
checks are made backend-neutral: z_add_timeout()'s assert and
z_timer_expiration_handler()'s "restarted?" test now use
z_is_inactive_timeout() instead of sys_dnode_is_linked(), and
Z_TIMER_INITIALIZER zero-inits the queue fields (.fn only) rather than
naming .node/.dticks. struct _timeout is otherwise untouched; no
sentinel or per-node state changes.
The Kconfig backend choice and the divergent node layouts are deferred
until a second backend lands; with only the delta list present they
would be churn with no benefit.
Verified no behavioral change: tests/kernel/timer/{timer_api,timeout},
tests/kernel/common, and tests/kernel/sleep pass on qemu_x86,
qemu_x86_64, qemu_cortex_a53 (SMP) and qemu_riscv64 (exercising the
SMP announcing_cpu path and 64-bit dticks).
Signed-off-by: Nicolas Pitre <npitre@baylibre.com>
Add a lean, kernel-only initialization model to replace the kernel's
internal use of SYS_INIT. Kernel init order is fixed at compile time, so
these hooks drop the SYS_INIT level/priority machinery: a subsystem
registers a parameterless init function that the boot path runs at one of
two fixed phases.
K_KERNEL_INIT_PRE(fn) run in z_cstart() before PRE_KERNEL device init
K_KERNEL_INIT_POST(fn) run in bg_thread_main() before POST_KERNEL
device init
Each entry is a single function pointer placed, via an iterable section,
in the registering subsystem's own translation unit. The linker pulls
that unit (and the entry) into the image only when the subsystem is
otherwise referenced, so an init runs only when its subsystem is actually
linked. This preserves the pay-per-use linkage that SYS_INIT provided
while halving the per-entry size (one pointer versus init_entry's two) and
removing the level/priority sort. The boot-path walks reference only the
section bounds, so they never force a subsystem to be linked.
Assisted-by: Claude:claude-opus-4-8
Signed-off-by: Anas Nashif <anas.nashif@intel.com>
Extend the cpu_mask field in struct _thread_base to support up to
32 CPUs by selecting uint32_t when CONFIG_MP_MAX_NUM_CPUS > 16.
Fix undefined behavior in IPI_ALL_CPUS_MASK where shifting 1 by
CONFIG_MP_MAX_NUM_CPUS (which can be 32) overflows uint32_t.
Replace with UINT32_MAX >> (32 - N) to produce the correct mask.
Signed-off-by: Guang Li <guang.li@jaguarmicro.com>
z_fatal_error() locks interrupts before calling coredump(). Some
backends (e.g. a UDP socket backend) need IRQ delivery to complete
their TX path. Introduce two weak arch hooks that bracket the
coredump() call:
arch_coredump_fatal_irq_unlock(key, cookie) -- re-enables IRQs
arch_coredump_fatal_irq_lock(cookie) -- restores the mask
Both are declared in kernel_arch_interface.h under
CONFIG_DEBUG_COREDUMP_FATAL_UNLOCK_IRQS. The default weak
implementations in subsys/debug/coredump/ delegate to the standard
arch_irq_unlock()/arch_irq_lock() pair. Architectures where exception
entry independently masks IRQs (e.g. ARM64 DAIF.I) must override them.
Signed-off-by: Appana Durga Kedareswara rao <appana.durga.kedareswara.rao@amd.com>
Inlining z_sched_wake() gives the thread_metric synchronization
benchmark an almost 40% boost. This restores it to its previous
levels when k_sem_give() was using z_unpend_first_thread().
Signed-off-by: Peter Mitsis <peter.mitsis@intel.com>
CONFIG_LINKER_USE_PINNED_SECTION is the second half of the selective
kernel-pinning model removed in issue #108773. With the kernel image
now always resident at boot (previous commit), the __pinned_*
attribute family is a no-op: every page they would have segregated is
already pinned by z_mem_manage_init()'s whole-image loop, so the
tagging contract neither adds safety nor remains maintainable.
Drop it.
Mechanical removals:
* All ~219 in-tree uses of __pinned_text, __pinned_rodata,
__pinned_data, __pinned_bss, __pinned_noinit, and __pinned_func
across arch/x86, drivers/interrupt_controller, drivers/timer,
arch/common, kernel, lib/libc, subsys/portability/posix, tests, and
the syscall code generator (scripts/build/gen_syscalls.py).
* The assembly aliases PINNED_TEXT/RODATA/DATA/BSS/NOINIT used in
arch/x86/core/ia32/*.S and drivers/interrupt_controller/
intc_loapic_spurious.S become plain TEXT/RODATA/DATA/BSS/NOINIT.
* K_KERNEL_PINNED_STACK_DEFINE, K_KERNEL_PINNED_STACK_ARRAY_DEFINE,
K_KERNEL_PINNED_STACK_ARRAY_DECLARE, K_THREAD_PINNED_STACK_DEFINE,
and K_THREAD_PINNED_STACK_ARRAY_DEFINE are removed. The few
in-tree callers (kernel/init.c, arch/arm/core/cortex_a_r/smp.c,
arch/arm64/core/fatal.c, arch/rx/core/prep_c.c,
arch/x86/core/prep_c.c, kernel/include/kernel_internal.h,
tests/bluetooth/hci_uart_async) move to the corresponding
non-pinned macros.
Machinery removals:
* Kconfig.zephyr drops CONFIG_LINKER_USE_PINNED_SECTION.
qemu_x86_tiny and qemu_x86_atom_virt drop their =y overrides.
* include/zephyr/linker/section_tags.h drops the __pinned_* macro
definitions (both arms). __isr collapses to an empty macro since
its only purpose was to alias __pinned_func.
* include/zephyr/linker/sections.h drops PINNED_TEXT_SECTION_NAME,
PINNED_BSS_SECTION_NAME, etc. and the bare PINNED_TEXT/RODATA/etc.
forwarders, plus the _APP_SMEM_PINNED_SECTION_NAME constant.
* include/zephyr/linker/linker-defs.h drops the lnkr_pinned_*
externs, the _app_smem_pinned_* externs, and the lnkr_is_pinned()
/ lnkr_is_region_pinned() inline helpers.
* include/zephyr/linker/utils.h drops the lnkr_pinned_rodata branch
in linker_is_in_rodata().
* include/zephyr/linker/app_smem_pinned{,_aligned,_unaligned}.ld
are deleted; cmake/linker/ld/target_configure.cmake stops
configuring them.
* boards/qemu/x86/qemu_x86_tiny.ld and
include/zephyr/arch/x86/ia32/linker.ld drop their pinned-section
blocks and the now-redundant #ifndef CONFIG_LINKER_USE_PINNED_SECTION
conditionals throughout the body. The
LIB_KERNEL_IN_SECT / LIB_ARCH_X86_IN_SECT / LIB_ZEPHYR_IN_SECT /
LIB_C_IN_SECT / LIB_DRIVERS_IN_SECT / LIB_SUBSYS_LOGGING_IN_SECT /
LIB_ZEPHYR_OBJECT_FILE_IN_SECT / ZEPHYR_KERNEL_FUNCS_IN_SECT macros
in qemu_x86_tiny.ld are deleted; they existed only to feed the
pinned text/rodata/data/bss/noinit sections.
* kernel/mmu.c drops the mark_linker_section_pinned(lnkr_pinned_start,
...) call. The mark_linker_section_pinned() helper survives but is
now gated only on CONFIG_LINKER_USE_BOOT_SECTION.
* arch/common/init.c and include/zephyr/arch/common/init.h drop
arch_bss_zero_pinned(); arch/x86/core/ia32/crt0.S drops the call
to it.
* arch/x86/core/userspace.c drops the eager k_mem_page_in() of the
thread's privileged stack on user-mode entry. With the kernel
image fully resident the stack is already mapped.
* arch/x86/gen_mmu.py drops map_region("lnkr_pinned") and the
set_region_perms() calls for lnkr_pinned_text / lnkr_pinned_rodata.
* CMakeLists.txt drops the LINKER_USE_PINNED_SECTION block that
generated APP_SMEM_PINNED_* variables and the
pinned_partitions target property feeding gen_app_partitions.py.
cmake/modules/extensions.cmake removes the PINNED_RODATA /
PINNED_RAM_SECTIONS / PINNED_DATA_SECTIONS zephyr_linker_sources()
location keywords and their snippet files.
scripts/build/gen_app_partitions.py drops --pinoutput /
--pinpartitions arguments and the pinned-output branch.
subsys/testsuite/coverage/CMakeLists.txt drops its
CONFIG_DEMAND_PAGING-conditional fork.
* scripts/build/gen_kobject_list.py drops the
app_smem_pinned_start / _end fallback for kobject placement
validation.
* tests/arch/x86/pagetables and tests/kernel/mem_protect/userspace
drop their lnkr_pinned_text / lnkr_pinned_rodata branches.
* include/zephyr/arch/x86/ia32/arch.h folds IRQSTUBS_TEXT_SECTION
to the unconditional ".text.irqstubs" form.
* tests/subsys/llext/src/syscalls_ext.c drops a stale comment about
syscalls landing in .pinned_text.
Targeted retentions:
* arch/x86/core/bootargs.c keeps multiboot_cmdline and efi_bootargs
in .noinit (was __pinned_noinit, which decayed to __noinit when
LINKER_USE_PINNED_SECTION was unset). The multiboot and zefi loader
paths write these buffers before Zephyr's BSS-zero step, so
zeroing them at boot loses the cmdline.
* arch/x86/core/ia32/fatal.c keeps _df_esf and _df_stack in .noinit.
They are scratch space written by the double-fault handler and have
no zero-init requirement; keeping them in .noinit also preserves
the historical post-noinit alignment that gen_mmu.py relies on
(z_mapped_size is computed before CMake-injected iterable sections
are appended to the linker script, so the post-noinit page padding
is what keeps those sections within the mapped region).
* include/zephyr/arch/x86/ia32/syscall.h and
include/zephyr/arch/x86/arch.h wrap the per-arch
arch_syscall_invoke* / arch_is_user_context / arch_k_cycle_get_*
implementations in @cond INTERNAL_HIDDEN. The public Doxygen
contract lives on the prototypes in
include/zephyr/arch/arch_interface.h; the per-arch implementations
are internal. Without this, removing the __pinned_func attribute
exposes the implementations to the doxygen-coverage delta check
as 10 newly-undocumented APIs.
Documentation updates are deferred to a separate commit.
Issue: #108773
Signed-off-by: Nicolas Pitre <npitre@baylibre.com>
The only reader was z_is_aborted_thread_timeout() in kernel/sleep.c,
used to distinguish a normal timeout-driven wakeup from one caused by
k_wakeup(). That distinction can be made entirely from the time
remaining: a normal wakeup leaves left_ticks <= 0, while an early
k_wakeup() leaves a positive remainder. The signed_left > 0 check
that already follows handles both cases correctly. The
z_is_aborted_thread_timeout() check was an early-return optimization
that the time computation makes redundant.
Drop the early-return in z_tick_sleep(), remove z_is_aborted_timeout(),
z_is_aborted_thread_timeout(), and the TIMEOUT_DTICKS_ABORTED macro.
Stop writing dticks = ABORTED in z_try_abort_timeout()'s linked-removal
path; the same-CPU IRQ branch is simplified to a flat -EINVAL fall-
through (the comment is moved to the cross-CPU/-EAGAIN side which is
the only case the caller needs to distinguish).
After this commit, dticks is purely a queue-management delta; there
are no more in-band sentinel values.
Signed-off-by: Nicolas Pitre <npitre@baylibre.com>
With every caller now using z_try_abort_timeout() and every handler
relying on subsystem-local wake-ownership flags (queue->finished,
K_WORK_DELAYED_BIT, poller.is_polling, killed/pended_on for threads)
rather than the dticks-cancel check, the in-band cancel mechanism is
dead code. Remove it:
- z_abort_timeout(): no callers; remove the function definition
in kernel/timeout.c and the declaration in kernel/include/timeout_q.h.
- z_is_timeout_handler_canceled(): no callers (handlers no longer
bail on dticks); remove.
- TIMEOUT_DTICKS_ANNOUNCING: nothing reads it; remove the macro and
the corresponding t->dticks = ANNOUNCING write in
sys_clock_announce_locked()'s dispatch loop.
TIMEOUT_DTICKS_ABORTED stays. z_is_aborted_timeout() (used by
kernel/sleep.c via z_is_aborted_thread_timeout to distinguish a
k_wakeup'd thread from one that timed out) reads it, and
z_try_abort_timeout() still writes it on the linked-removal and
the same-CPU IRQ best-effort paths.
Signed-off-by: Nicolas Pitre <npitre@baylibre.com>
With every caller of z_abort_thread_timeout() now migrated to
z_try_abort_thread_timeout() (sched.c, thread.c, scheduler.c,
events.c, pipe.c) or to z_unpend_first_thread_locked() (sem.c,
mutex.c, mem_slab.c, stack.c, condvar.c, msg_q.c, queue.c, futex.c),
remove the inline wrapper (and !CONFIG_SYS_CLOCK_EXISTS stub) for
z_abort_thread_timeout().
z_thread_timeout() still needs a cancellation check, but it no longer
relies on the dticks=ANNOUNCING sentinel: switch it to
z_timeout_inflight_superseded(). The check is still required, and for
the same reason 1b8c7a3 added it. A concurrent waker on another CPU
(e.g. a sem give via z_unpend_first_thread_locked()) can unpend and
ready the thread while this timeout's handler is blocked on
_sched_spinlock; the thread may then run and re-pend on a different
object -- possibly with no timeout (K_FOREVER). The waker aborts this
timeout, which flags it superseded, and z_thread_timeout() bails on
that flag so it does not wake the thread from its new wait. The
atomic wake-under-_sched_spinlock closes the swap_retval window; the
superseded check closes this re-pend window.
The remaining TIMEOUT_DTICKS_ANNOUNCING sentinel and
z_is_timeout_handler_canceled() helper still have other users
(kernel/timer.c, kernel/poll.c, kernel/work.c) and are removed in the
later cleanup commit once those subsystems have been migrated as well.
Signed-off-by: Nicolas Pitre <npitre@baylibre.com>
Replace z_unpend_first_thread() with z_unpend_first_thread_locked() and
migrate every caller across the kernel. The old function dropped the
scheduler spinlock before returning, exposing a race window between
its caller's "arch_thread_return_value_set + z_ready_thread" pair and
a still-in-flight timeout handler that could ready the thread first --
the woken thread might then run on another CPU and see an uninitialized
swap_retval. Pre-1b8c7a3 the dticks-cancel check made the handler bail;
here we fix it cleanly by requiring the caller to hold _sched_spinlock
across the entire wake, so the handler is blocked for the duration and
runs as a no-op afterwards.
z_unpend_first_thread_locked() requires the caller to be inside a
locked region and must be paired with z_sched_ready_locked() (and
whatever return-value setup is needed) under the same lock acquisition.
Sites migrated:
Simple "set retval [+ swap_data] and ready" callers use the existing
z_sched_wake() convenience wrapper, refactored to use the new
helper internally:
sem (give, reset), mem_slab (free), stack (push),
condvar (signal, broadcast), msgq (purge),
queue (cancel_wait, queue_insert, append_list),
futex (wake).
Sites that need additional setup on the woken thread use
LOCK_SCHED_SPINLOCK + z_unpend_first_thread_locked() + custom wake:
mutex (unlock -- needs the thread reference to track new owner),
msgq put / get (needs memcpy into the receiver's swap_data
buffer before the return value is set).
The dticks-cancel check in z_thread_timeout() is left in place; it is
no longer load-bearing once z_abort_thread_timeout() has no callers,
and is removed in the next commit.
Signed-off-by: Nicolas Pitre <npitre@baylibre.com>
Migrate scheduler-internal callers of z_abort_thread_timeout() to the
new z_try_abort_thread_timeout(). This covers the abort sites in
sched.c, thread.c, scheduler.c (z_sched_wake), events.c, and pipe.c.
The patterns used:
z_unpend_thread (sched.c) retries on -EAGAIN: if the timeout
handler is in flight on another CPU, drop _sched_spinlock so the
handler can run to completion and retry. This preserves 1b8c7a3's
unpend+abort atomicity from the caller's perspective.
halt_thread (sched.c) takes the caller's sched-lock key as a pointer
so its direct abort on the dying thread can retry on -EAGAIN.
Waiting for the handler is mandatory: a caller may free the thread's
storage as soon as halt_thread() returns, and without waiting, the
still-in-flight handler would later dereference freed memory.
_THREAD_DEAD is set before the abort, so the handler bails via the
killed check in z_sched_wake_thread_locked().
For next_up() (the scheduler-context caller), the key is not cleanly
available: K_SPINLOCK in z_get_next_switch_handle and do_swap's
(void)k_spin_lock both discard it. halt_thread is invoked on
_current with NULL key and no abort is performed -- _current is
running, so its base.timeout cannot be linked. The gap is closed at
the other end: z_thread_halt() spins on z_try_abort_thread_timeout()
outside any lock after the halt-queue wait completes, before
returning to the caller of k_thread_abort().
z_unpend_all_locked / unpend_all (sched.c) skip the local
ready_thread() on -EAGAIN and let the still-blocked handler ready
the thread when _sched_spinlock drops. The threads being woken are
not freed, so no UAF risk; end state is identical.
z_impl_k_wakeup (thread.c), z_sched_wake (scheduler.c) and
event_walk_op (events.c) perform the wake entirely under
_sched_spinlock, so a (void) abort is race-free -- a racing in-flight
handler is blocked on the same lock during the wake.
copy_to_pending_readers (pipe.c) is restructured to also wake the
reader under the scheduler lock instead of after it, so the
return-value set, unpend, abort, and ready all happen atomically.
The dticks-cancel check in z_thread_timeout() is preserved for now
because other callers (sem.c, mutex.c, ... via z_unpend_first_thread())
still use z_abort_thread_timeout() and rely on it for race protection.
A follow-up commit migrates those, and a final commit drops the
cancel check and removes z_abort_thread_timeout() itself.
Add z_try_abort_thread_timeout() as an inline wrapper.
Signed-off-by: Nicolas Pitre <npitre@baylibre.com>
When z_try_abort_timeout() finds the target timeout already popped
from the queue and in flight (its handler dispatching), the abort
cannot remove anything from the queue. The handler is about to run
(or is running) the timeout's callback; the aborter needs a way to
tell it "you were aborted, skip your side effects". main carried that
signal in the per-timeout dticks=ABORTED sentinel, which any CPU
could write under timeout_lock and the handler checked at entry via
z_is_timeout_handler_canceled(). Later commits in this series remove
the dticks-cancel mechanism, so the signal needs a new home.
Encode it in the low bit of the file-local inflight_timeout pointer
(struct _timeout is pointer-aligned, so bit 0 is free):
inflight_timeout == NULL no handler in flight
inflight_timeout == t handler in flight, not superseded
inflight_timeout == t | 1 handler in flight, superseded
z_try_abort_timeout() sets the bit whenever the target is the
in-flight timeout -- on the announcing CPU (same-CPU IRQ that
preempted the dispatch, or a stop after a re-arm) and on another CPU
racing the handler.
A handler with non-idempotent side effects (currently only k_timer's
expiry_fn) checks z_timeout_inflight_superseded() at entry and bails
if set. Idempotent handlers (z_thread_timeout, work, poll, ...)
tolerate the race and don't need the check.
The bit is a best-effort signal, not a barrier: an aborter that sets
it after the handler has passed its check has no effect (the handler
already committed), exactly as the dticks sentinel behaved on main.
The -EAGAIN return for the cross-CPU case is a separate mechanism,
used only by callers that must wait for the handler to fully complete
(e.g. before freeing the timeout's storage); they spin, while
best-effort callers ignore -EAGAIN and rely on the bit.
Signed-off-by: Nicolas Pitre <npitre@baylibre.com>
Add an `inflight_timeout` pointer in the timeout subsystem, set by
sys_clock_announce_locked() under timeout_lock before a handler is
dispatched and cleared after the handler returns. The single-announcer
SMP invariant (only one CPU is in the dispatch loop at a time) means a
single global pointer suffices, mirroring announcing_cpu.
Add a new abort function z_try_abort_timeout() that uses this pointer
to detect "popped from the queue but handler not yet finished" without
relying on the in-band dticks == ANNOUNCING sentinel that survives
across the timeout's storage being freed only by coincidence.
The new function returns -EAGAIN when a handler is in flight on
another CPU, signalling that the caller must drop any outer lock the
handler may need and retry. A short arch_spin_relax() is performed
before -EAGAIN is returned so callers don't need to add their own
back-off in the retry path. inflight_timeout itself is kept private
to the timeout subsystem; callers only ever see the return value.
This commit introduces the infrastructure but migrates no callers.
The existing z_abort_timeout() and TIMEOUT_DTICKS_ANNOUNCING-based
cancellation in handlers continue to work; the dispatch loop sets
both inflight_timeout and ANNOUNCING so the two paths coexist while
callers are migrated one module at a time.
Signed-off-by: Nicolas Pitre <npitre@baylibre.com>
Replace the single in-place assertion in do_swap/z_swap_irqlock with a
z_assert_can_swap() helper performing three checks: held spinlock
pointer, hold count, and IRQ state.
To support the hold-count check, add per-CPU tracking arrays
(z_held_spinlock[] and z_held_spinlock_count[]).
Add z_spin_lock_transfer_owner() to update lock ownership after a
context switch, z_spinlock_abort_sentinel to exempt threads aborted by
a ztest expected-fault scenario, and z_spin_validate_reset() to reset
stale per-CPU lock tracking left by such an abort so subsequent tests
can proceed cleanly.
Signed-off-by: Jinming Zhao <jinmzhao@qti.qualcomm.com>
In CONFIG_SCHED_CPU_MASK_PIN_ONLY a thread must always be pinned to
exactly one CPU. Two related gaps let a zero-mask thread slip
through silently:
1. thread_runq() (run_q.h) had an explicit if/else that silently
routed a zero-masked thread to CPU 0 instead of catching the
violation. Replace it with an __ASSERT that fires at the point
where the bad state is used.
2. cpu_mask_mod() (cpu_mask.c) checked '(m == 0) || power-of-two',
which accepted a cleared mask as valid. Tighten the check to
require strictly a single set bit ('m != 0 && power-of-two') so
any API call that would leave the thread with an empty mask (e.g.
k_thread_cpu_mask_clear()) traps at the API boundary rather than
later at queue time.
The two enforcement points now agree: every thread in PIN_ONLY mode
must carry exactly one CPU bit from the moment the mask is written
until the thread is queued.
Assisted-by: GitHub Copilot:claude-sonnet-4-5
Signed-off-by: Anas Nashif <anas.nashif@intel.com>
Now SCHED_CPU_MASK is no longer restricted to SCHED_SIMPLE. Dropped
dependency on SCHED_SIMPLE and made SCHED_CPU_MASK work on all scheduler
types by implementing z_priq_rb_mask_best and z_priq_mq_mask_best for
both scalable and multiq schedulers.
Update the help text and remove the stale claim that the feature only
works with the simple scheduler and document the mask-aware best-thread
search algorithm and its performance characteristics for each of the
three supported backends:
SCHED_SIMPLE: O(N) sorted-list scan; terminates at the first
priority-eligible thread, so cost is proportional to
the number of higher-priority threads pinned away from
the current CPU.
SCHED_SCALABLE: O(N) in-order rbtree walk; benefits from priority
ordering so the walk usually terminates early.
SCHED_MULTIQ: O(P) bitmap iteration over non-empty priority levels
plus an inner O(N) per-level list scan; fast when
affinity-constrained threads are sparse, degrades when
many same-priority threads are pinned away.
Assisted-by: GitHub Copilot:claude-sonnet-4.6
Signed-off-by: Anas Nashif <anas.nashif@intel.com>
Add __DOXYGEN__ placeholder definitions for ARCH_DATA_PAGE_* macros so
that Doxygen, which does not know about how these macros _might_ be
defined by architectures, can still bind documentation to them.
Signed-off-by: Benjamin Cabé <benjamin@zephyrproject.org>
Keep the original architecture IRQ key owned by idle across a
successful system PM transition.
Add architecture hooks and the PM_STATE_SET_IRQ_LOCKED migration
contract for SoCs that keep PM hooks from unmasking interrupts.
Signed-off-by: Holt Sun <holt.sun@nxp.com>
Unlike Z_TIMER_INITIALIZER, k_timer_init() does not fully initialize the
provided timer.
This results on valgrind warning about a Conditional jump on uninitialised
value when calling k_timer_start() on that object later when dticks is
checked.
Let's initialize it to avoid this warning.
Signed-off-by: Alberto Escolar Piedras <alberto.escolar.piedras@nordicsemi.no>
Fix the early TLS initialization sequence on RISC-V to correctly set up
the thread pointer (tp) before any C code runs, and save/restore
callee-saved registers as required by the ABI.
Signed-off-by: Mayur Salve <msalve@qti.qualcomm.com>
This change enables per thread stack canary for RISC-V.
RISC-V GCC accesses the stack canary via a fixed offset from the
thread pointer (tp) when -mstack-protector-guard=tls is used. The
compiler emits code equivalent to:
lw t0, 0(tp) # load canary from tp+0
Additionally, tp is zeroed in arch_kernel_init() when TLS is enabled,
which means any C function called before thread setup completes (such
as z_early_rand_get or data_copy_xip_relocation) would fault trying
to access the canary.
Introduce STACK_CANARIES_TLS_PREPEND, which places the
.stack_chk.guard section at offset 0 of the TLS block, before .tdata
and .tbss. The compiler flags -mstack-protector-guard-reg=tp and
-mstack-protector-guard-offset=0 are passed so GCC generates the
correct canary access.
With STACK_CANARIES_TLS_PREPEND the per-thread TLS block layout is:
tp --> +------------------+ offset 0
| .stack_chk.guard | (__stack_chk_guard)
+------------------+
| .tdata | (initialized TLS data)
+------------------+
| .tbss | (zero-initialized TLS data)
+------------------+
The RISC-V reset path is extended to initialize tp before any C code
runs by allocating a TLS area on the boot stack and calling
arch_riscv_early_tls_stack_update(). Early boot functions that run
before tp is set up (z_early_rand_get, data_copy_xip_relocation) are
marked FUNC_NO_STACK_PROTECTOR to avoid canary access before tp is
valid.
Signed-off-by: Mayur Salve <msalve@qti.qualcomm.com>
This fixes a subtle race condition in the thread timeout expiration
handler z_thread_timeout(). There was a small window of opportunity
between when sys_clock_announce() unlocked interrupts and that the
handler re-locked them that one or more higher priority interrupts
(or threads running on another CPU if in an SMP environment) could
abort the thread's timeout.
The fix has two parts. Part one ensures that _sched_spinlock is held
in every location before a thread's time can be canceled. Of the
various locations, only z_unpend_thread() was found to need updating.
Part two updates the timeout handler z_thread_timeout() to bail early
if the thread's timeout has been found to be canceled (or re-used)
during that aforementioned window.
Fixes#106653
Signed-off-by: Peter Mitsis <peter.mitsis@intel.com>
This partially reverts commit 9cef0da05c ("kernel: avoid recursive
scheduler lock in k_heap_free path"), keeping only the sched.c changes
(z_unpend_all_locked / z_unpend_all refactoring).
The _sched_locked variants of k_free, k_heap_free, k_msgq_cleanup,
k_stack_cleanup and the sched_locked parameter plumbing through
unref_check were an overcomplicated approach. A simpler fix follows.
Signed-off-by: Nicolas Pitre <npitre@baylibre.com>
IAR emits diagnostic Go004 ("function cannot be inlined") for
every ALWAYS_INLINE function when optimisation is disabled, e.g.
in debug builds. The previous workaround wrapped each affected
function in per-function preprocessor guard pairs:
#ifdef IAR_SUPPRESS_ALWAYS_INLINE_WARNING_FLAG
TOOLCHAIN_DISABLE_WARNING(TOOLCHAIN_WARNING_ALWAYS_INLINE)
#endif
static ALWAYS_INLINE void foo(...) { ... }
#ifdef IAR_SUPPRESS_ALWAYS_INLINE_WARNING_FLAG
TOOLCHAIN_ENABLE_WARNING(TOOLCHAIN_WARNING_ALWAYS_INLINE)
#endif
This pattern is highly intrusive, scatters toolchain-specific
knowledge across generic source files, and requires a guard pair
every time a new ALWAYS_INLINE function is added for IAR.
Replace it with a single override of ALWAYS_INLINE inside
iccarm.h, using the C99 _Pragma operator to embed the diagnostic
suppression in the macro itself.
Assisted-by: GitHub Copilot:claude-sonnet-4.6
Signed-off-by: Anas Nashif <anas.nashif@intel.com>
gen_offset.h is an architecture-specific header, not a kernel one.
Move it under the arch tree where it belongs.
Signed-off-by: Anas Nashif <anas.nashif@intel.com>
Migrate scheduler API implementations (k_sched_lock/unlock,
z_reschedule, z_yield_current, etc.) and their private declarations
from ksched.h/sched.c into scheduler.c and scheduler.h.
Signed-off-by: Anas Nashif <anas.nashif@intel.com>
Move time-slice related declarations from ksched.h into the
dedicated kernel/include/timeslicing.h header.
Signed-off-by: Anas Nashif <anas.nashif@intel.com>
Migrate z_add_thread_to_ready_q(), z_remove_thread_from_ready_q(),
and related helpers from sched.c to scheduler.c.
Signed-off-by: Anas Nashif <anas.nashif@intel.com>
Move z_sched_init to scheduler.c and somplify implementation getting rid
of single use init_ready_q.
Signed-off-by: Anas Nashif <anas.nashif@intel.com>
Move run-queue management functions (add/remove/peek thread,
choose_next_thread) from sched.c into the new
kernel/include/run_q.h header.
Signed-off-by: Anas Nashif <anas.nashif@intel.com>
