Commit graph zephyr/kernel
Author SHA1 Message Date
Yiren Guo
fc7a8936a4 kernel: usage: fix double-count in k_thread_runtime_stats_disable()
Disabling stats on the current thread accounts the in-progress window
into the CPU stats but forgets to update cpu->usage0. The next
z_sched_usage_stop() recomputes the same window and adds it again.

Signed-off-by: Yiren Guo <guoyr_2013@hotmail.com>
2026-08-15 08:11:37 -04:00
Anas Nashif
7170b2dba2 kernel: fatal: use EXCEPTION_DUMP in fatal error handler
If logging is disabled, we want to get the same exception and
information, right now half of the information is being dropped as they
go through LOG_ERR only while the rest go via EXCEPTION_DUMP/printk.

Signed-off-by: Anas Nashif <anas.nashif@intel.com>
2026-08-14 15:13:19 +02:00
Anas Nashif
32c0ba4c7e Revert "kernel: move the sleep API declarations to their own header"
This reverts commit 274d67b107.

Signed-off-by: Anas Nashif <anas.nashif@intel.com>
2026-08-13 17:48:21 -04:00
Anas Nashif
2dd9f5f414 Revert "kernel: inline k_sleep() and k_usleep() over a tick primitive"
This reverts commit d38477cdf6.

Signed-off-by: Anas Nashif <anas.nashif@intel.com>
2026-08-13 17:48:21 -04:00
Nicolas Pitre
d38477cdf6 kernel: inline k_sleep() and k_usleep() over a tick primitive
k_sleep() and k_usleep() resolve to an out of line implementation living in
another translation unit, a system call under CONFIG_USERSPACE and a plain
call otherwise.  Either way the compiler sees neither whether the requested
duration is a constant nor whether the caller cares about the returned
value, so it always emits the tick to millisecond and microsecond
conversions.  Those conversions are the only thing that drags the 64 bit
division helper into many small builds: CONFIG_SYS_CLOCK_TICKS_PER_SEC
defaults to 10000 on tickless platforms, which lands in the integer
division path and divides a 64 bit value by ten, something GCC will not
strength reduce on a 32 bit target.

Promote the former z_tick_sleep() helper to a k_sleep_ticks() system call
that reports the remaining time in ticks, and rebuild k_sleep() and
k_usleep() as inlines on top of it.  The conversions now live at the call
site where the compiler can fold or discard them.  Nearly every caller
discards them: of the 4036 sleep call sites in the tree, exactly six use
the returned value, and only three of those are outside of tests.

What this removes is best seen in the two out of line entry points that
disappear, which on a nucleo_f030r8 (Cortex-M0, 10000 ticks/s) were the
only two callers of the 64 bit division helper in the whole image:

  08002594 <z_impl_k_sleep>:
        bl      8001... <z_tick_sleep>
        ...
        movs    r0, #9            /* the ceil() bias, 10 - 1 */
        adds    r0, r0, r2
        adcs    r1, r3
        movs    r2, #10           /* ticks / 10 -> milliseconds */
        bl      8000180 <__aeabi_uldivmod>

  080025c0 <z_impl_k_usleep>:
        movs    r0, #99           /* the ceil() bias, 100 - 1 */
        ...
        movs    r2, #100          /* microseconds / 100 -> ticks */
        bl      8000180 <__aeabi_uldivmod>
        bl      8001... <z_tick_sleep>

Both are gone, and so are __aeabi_uldivmod and __udivmoddi4, which are no
longer referenced anywhere.  A k_usleep(250) call site now passes the
literal 3 ticks the compiler worked out on its own, where it used to pass
250 microseconds to a helper that divided by 100 at runtime.  For a loop
calling k_msleep(100), k_sleep(K_SECONDS(1)) and k_usleep(250) without
using the results, FLASH drops from 11028 to 10600 bytes.

Tracing stays in the out of line implementation.  It cannot move into the
inlines because every tracing backend header includes kernel.h, so a
translation unit that reaches a backend header first would expand
SYS_PORT_TRACING_* before those macros exist.  A consequence is that the
sleep_exit hook now reports ticks rather than milliseconds and the usleep
hook is no longer emitted; both are covered in a following commit.

The microsecond duration is clamped to zero the way Z_TIMEOUT_US() already
does.  k_usleep() never did that, but it used to truncate the 64 bit
conversion into an int32_t before handing it over, which hid the worst of
it.
Passing the conversion through whole would turn a negative duration into a
sleep of roughly 1.8e17 ticks rather than a bogus absolute deadline.

This also removes a potential link error.  k_usleep() had no
implementation at all with CONFIG_MULTITHREADING=n, since nothread.c only
ever provided z_impl_k_sleep(), so any caller failed to link.  Nothing in
the tree happens to call it in that configuration, which is presumably why
it went unnoticed; a single z_impl_k_sleep_ticks() now serves both.

Signed-off-by: Nicolas Pitre <npitre@baylibre.com>
2026-08-13 16:37:05 -04:00
Nicolas Pitre
274d67b107 kernel: move the sleep API declarations to their own header
k_sleep(), k_msleep() and k_usleep() are declared in kernel.h today.
Later commits grow that API with a tick based primitive plus inline unit
conversions, which is more than kernel.h ought to carry for a single
service.

Move the three declarations verbatim into a new include/zephyr/sleep.h
and have kernel.h include it, so existing users need no change. The new
header has to be registered with zephyr_syscall_header(), otherwise the
marshalling stubs for k_sleep() and k_usleep() stop being generated and
CONFIG_USERSPACE builds fail to find k_sleep_mrsh.c.

No functional change. A test application built for nucleo_f030r8 comes
out byte identical, at 11028 bytes of FLASH before and after.

Signed-off-by: Nicolas Pitre <npitre@baylibre.com>
2026-08-13 16:37:05 -04:00
Nicolas Pitre
51b33f884a kernel: timeout: report the no-deadline case from the idle path
The idle path asks this for the time until the next wakeup, and gets a
tick count that never says "there is nothing to wake up for": an empty
timeout list is reported as the capped announce budget, exactly like a
deadline further out than can be programmed in one step. The power
management code cannot then tell the two apart, so it arms the timer in
both cases and a system with sloppy idle enabled keeps waking up for
nothing.

Make the same decision reprogram_next() makes. With
CONFIG_SYSTEM_CLOCK_SLOPPY_IDLE allowing the uptime to drift, an empty
list is reported as K_TICKS_FOREVER; anything else is a wait, whether a
real deadline or the synthetic one that keeps the announce range
covered. Without sloppy idle the empty case stays a wait, so the timer
remains armed and timekeeping is unaffected.

The return type becomes unsigned to match the tick type used throughout
the timer interface. The conversion at the only caller, in the idle
path, is value preserving in both directions: every wait is capped at
SYS_CLOCK_MAX_WAIT, which is INT32_MAX, and K_TICKS_FOREVER is the same
value read either way. Nothing outside the kernel sees the change; the
power management API keeps its signed tick counts.

Signed-off-by: Nicolas Pitre <npitre@baylibre.com>
2026-08-12 19:27:28 -04:00
Nicolas Pitre
360f4fb1fd kernel: timeout: add sys_clock_no_timeout() for the empty timeout list
An empty timeout list normally still gets the driver a wakeup request.
That is a synthetic timeout: nothing is waiting for it, it is there to
keep the announce baseline moving so uptime stays correct.
CONFIG_SYSTEM_CLOCK_SLOPPY_IDLE is the permission to skip it, which is
why an empty list only becomes visible to a driver when that option is
set, and today it becomes visible as a tick value the driver has to
recognise rather than as a condition of its own.

Add a weak sys_clock_no_timeout() hook, called in place of
sys_clock_set_timeout() when the list is empty and sloppy idle allows the
uptime to drift. Its default asks sys_clock_set_timeout() for UINT32_MAX
ticks, which is what K_TICKS_FOREVER is in the unsigned tick domain this
interface now uses, so an out-of-tree driver that has not migrated still
sees the "no deadline" value it always did. What gets deprecated is the
special meaning, not the call.

next_timeout() no longer decides sloppy idle, so its empty-list and
far-timeout arms collapse into one capped budget. The far-timeout arm no
longer stops a tick short of the cap either: it only did so to stay
distinguishable from the empty case, which now travels out of band.

Signed-off-by: Nicolas Pitre <npitre@baylibre.com>
2026-08-12 19:27:28 -04:00
Gustav Holmberg
8483d28b42 toolchain: fix deprecation warning for iar
In commit 4f9df472fc1d06a6deeb7dddf6ab167c7cba984f
TOOLCHAIN_DISABLE_WARNING with a hard coded argument in work.c. This
created warnings when using the iar toolchain. Fix it by using a
preprocessor macro.

Signed-off-by: Gustav Holmberg <gustav.holmberg@qt.io>
2026-08-12 13:11:27 -04:00
Anas Nashif
5c12262de7 kernel: kernel_offsets: put the include guard before the includes
kernel_offsets.h pulls in three headers before defining its own
include guard, so the guard is not the first thing in the file as
MISRA C:2012 Directive 4.10 requires.

Move the guard above the includes.  The header is still only used by
the offsets generator, and the includes stay inside the guard.

Signed-off-by: Anas Nashif <anas.nashif@intel.com>
2026-08-11 17:09:37 -04:00
Anas Nashif
e84e2f7955 kernel: rename the reserved 'exit' labels to 'out'
'exit' is reserved by the C standard for any use, so using it as a
label violates MISRA C:2012 Rule 21.2.  Rename the labels in the
message queue and pipe implementations to 'out', which is the spelling
already used elsewhere in the kernel.

Pure rename, no functional change.

Signed-off-by: Anas Nashif <anas.nashif@intel.com>
2026-08-11 17:08:24 -04:00
Anas Nashif
7a72151634 kernel: smp: replace SMP_BOOT_DELAY with per-CPU devicetree deferral
CONFIG_SMP_BOOT_DELAY was a global, application-level switch that made
the kernel skip starting every secondary CPU at boot, used in-tree only
by two tests and in practice by one platform class (intel_adsp, where
the host or PM policy brings DSP cores up on demand). Boot topology is
a hardware/platform property, and all-or-nothing is needlessly coarse.

Replace it with a per-CPU devicetree flag, zephyr,deferred-start, on
the /cpus children (mirroring zephyr,deferred-init for devices):
z_smp_init() now always runs and simply skips flagged CPUs, which are
brought up at run time with the existing k_smp_cpu_start() (or
k_smp_cpu_resume()). Deferral is per CPU, so asymmetric bring-up
(start some cores at boot, defer others) is now expressible, and the
special-case branch disappears from the boot path.

The flag is declared in the common cpu.yaml binding. The lookup uses
DT_PROP_OR() so cpu nodes whose binding does not cover the property
simply cannot be deferred rather than breaking the build; a binding
for the intel,x86_64 compatible used by qemu_x86_64's cpu nodes was
missing entirely and is added.

The two users are converted: tests/kernel/multiprocessing/
smp_boot_delay marks the secondary CPUs in per-board overlays (both
tests pass on qemu_x86_64) and tests/boards/intel_adsp/smoke gains
overlays for its four platforms (all build). Normal SMP boot is
unaffected (verified on qemu_x86_64, all CPUs online).

Out-of-tree users migrate by dropping CONFIG_SMP_BOOT_DELAY=y and
adding zephyr,deferred-start to the deferred cpu nodes in their board
overlay.

Assisted-by: Claude:claude-fable-5
Signed-off-by: Anas Nashif <anas.nashif@intel.com>
2026-08-11 17:01:49 -04:00
Tim Pambor
e1b4cf8615 kernel: Update CONFIG_SWITCH workaround for clang
Clang (pre-v20) enables frame pointers at all optimization levels,
unlike gcc which only enables them only at -O0. This causes the
same r7 clobber issue in arch_switch inline assembly as for GCC,
if CONFIG_SWITCH=y.

Extend the -fomit-frame-pointer workaround to apply unconditionally
for clang and also add sleep.c and thread.c which also started to
need the flag since the workaround was added.

Signed-off-by: Tim Pambor <tim.pambor@codewrights.de>
2026-08-11 13:45:59 -04:00
Tim Pambor
6527ead56b kernel: format CMakeLists.txt
Format CMakeLists.txt according to CMake Style Guidelines.

Signed-off-by: Tim Pambor <tim.pambor@codewrights.de>
2026-08-11 13:45:59 -04:00
Anas Nashif
1036694889 kernel: device: restrict the device_deinit syscall to driver objects
z_vrfy_device_deinit() validated its argument with K_OBJ_ANY, so any kernel
object the calling thread has access to could be passed as a struct device.
z_impl_device_deinit() then calls dev->ops.deinit(dev), an indirect call
through a function pointer read out of the confused object, giving user
mode an arbitrary call primitive when CONFIG_DEVICE_DEINIT_SUPPORT=y.

Use K_OBJ_DRIVER_ANY, matching z_vrfy_device_init() and
z_vrfy_device_is_ready().

Signed-off-by: Anas Nashif <anas.nashif@intel.com>
2026-08-10 09:14:56 -07:00
Anas Nashif
85c1c4c219 kernel: userspace: reject size overflow in dynamic object allocation
dynamic_object_create() computed the allocation size as
obj_size_get(otype) + size, where size comes straight from user mode via
the k_object_alloc_size() syscall - the verifier is a bare pass-through and
z_object_alloc() only range-checks otype. A size close to SIZE_MAX wraps,
so the allocation is small while the object is tagged with the requested
type; the matching init syscall then writes a full object over it. The
K_OBJ_THREAD_STACK_ELEMENT branch has the same problem through
STACK_ELEMENT_DATA_SIZE(), which rounds up and adds overhead.

Reject both overflows and free the descriptor.

Signed-off-by: Anas Nashif <anas.nashif@intel.com>
2026-08-10 09:14:40 -07:00
Peter Mitsis
7d4140aee7 kernel: Streamline k_work_submit() tracing
Updates k_work_submit() to use z_work_submit_to_queue() instead of
k_work_submit_to_queue() to remove superfluous tracing calls.

Signed-off-by: Peter Mitsis <peter.mitsis@intel.com>
2026-08-04 06:53:32 -04:00
Peter Mitsis
cc80daf522 kernel: z_work_submit_to_queue() may reschedule
Migrates the schedule point from k_work_submit_to_queue() to the
internal routine z_work_submit_to_queue().

Though the function description previously indicated that
z_work_submit_to_queue() was not to yield the thread, investigation
showed that its only use outside the work.c file was poll.c and that
interrupts are locked at each of those call points. As interrupts
were already locked prior to calling this routine from poll.c,
z_reschedule() will not swap out the current thread and the behavior
remains unchanged.

Signed-off-by: Peter Mitsis <peter.mitsis@intel.com>
2026-08-04 06:53:32 -04:00
Emil Hammarström
76c0a34e3d kernel: poll: Remove redundant timeout check
z_add_timeout returns without any effect if timeout is equal to K_FOREVER
so there is no need to guard its timeout parameter from a K_FOREVER.

Signed-off-by: Emil Hammarström <emil.a.hammarstrom@gmail.com>
2026-08-02 11:43:59 -04:00
Daniel Leung
03e4556134 kernel: sched: use function aliases for simple wrappers
There are a few functions inside kernel/sched.c that are simply
one line wrappers to functions having the exact same signatures.
Unifying the naming would be ideal but the underlying functions
are being called by others under different kernel namespace.
So for now, the simplest is to use function aliasing and let
linker do its magic to avoid unnecessary trampoline. Saves
a few CPU cycles too.

Signed-off-by: Daniel Leung <daniel.leung@intel.com>
2026-08-02 11:43:47 -04:00
Alberto Escolar Piedras
2f7f6af693 kernel: initialize swap_data if CONFIG_SPIN_VALIDATE
With CONFIG_SPIN_VALIDATE, z_assert_can_swap() & halt_thread() check if
the value set in base.swap_data corresponds to &z_spinlock_abort_sentinel
(it will be set to this value in
subsys/testsuite/ztest/src/ztest_error_hook.c if there is a fatal error
or assert).
But nobody is initializing this (struct k_thread).base.swap_data to
anything.
So when there is no failures (or no ztest code), we are checking a random
value and potentially doing weird things.

Let's initialize it to NULL to avoid this.

This check was added in d844a4a861

Signed-off-by: Alberto Escolar Piedras <alberto.escolar.piedras@nordicsemi.no>
2026-07-31 14:57:37 -04:00
Peter Mitsis
eff636cd88 kernel: fix race in cpu_mask_mod()
Moves the assert checking that exactly one CPU in the mask is set
to be inside the locked section. This closes a hole where another
CPU could have called cpu_mask_mod() and performed an OR operation
setting more bits just before the current CPU read the cpu_mask
field leading to an incorrect assert evaulation.

Signed-off-by: Peter Mitsis <peter.mitsis@intel.com>
2026-07-30 17:39:47 -04:00
Etienne Carriere
f6c0abb54c kernel: timer: Include include/zephyr/sys/clock.h
Header file include/zephyr/sys_clock.h is deprecated and will be removed
someday. Update the whole file tree to include zephyr/sys/clock.h
straight instead of zephyr/sys_clock.h.

This change was made running the sed shell command below:
$ sed -i 's/zephyr\/sys_clock\.h/zephyr\/sys\/clock\.h/' \
      `grep -rsl "zephyr/sys_clock\.h" kernel/`

Signed-off-by: Etienne Carriere <etienne.carriere@st.com>
2026-07-30 07:45:05 -05:00
Tibor Kiss
4e5a1c1f99 kernel: sched: panic on blocking pend from ISR context
A blocking pend from ISR context is a programming error with no safe
recovery: it attempts to sleep whatever thread was interrupted and, on
CONFIG_SWAP_NONATOMIC, corrupts the scheduler (the qnode_dlist
double-link / NULL-deref traced in #111518).

Catch it at the funnel instead of per-API: every blocking primitive
(sem, mutex, msgq, queue, stack, pipe, poll, events, mailbox, mem_slab,
kheap, futex, ...) routes through z_pend_curr, and reaching it always
means a real block (callers handle K_NO_WAIT beforehand). So one check
there covers them all, with no per-API return-contract churn:

  if (arch_is_in_isr()) {
      __ASSERT(false, "blocking pend from ISR context");
      k_panic();
  }

__ASSERT(false, ...) gives the message and backtrace in debug builds;
k_panic() makes it fatal in every build, closing the production gap
(CONFIG_ASSERT defaults to n) that let the misuse ship and run for
hours. Located before _sched_spinlock is acquired, so no lock is held
on the panic path.

Also assert in add_to_waitq_locked() that a thread is not already on a
wait queue when added to a new one (pended_on == NULL) -- belt-and-
suspenders for a double-pend that reaches the scheduler despite the
above, e.g. a non-current thread re-pended without an intervening
unpend.

Suggested-by: Nicolas Pitre <npitre@baylibre.com>
Signed-off-by: Tibor Kiss <kiss.tibor@gmail.com>
2026-07-30 07:44:12 -05:00
Emil Hammarström
f6e012d1bc kernel: sched: Remove redundant timeout check
z_add_timeout returns without any effect if timeout is equal to K_FOREVER,
and thus z_add_thread_timeout does this by extension and there is no need
to guard its timeout parameter from a K_FOREVER.

Signed-off-by: Emil Hammarström <emil.a.hammarstrom@gmail.com>
2026-07-28 11:34:34 +01:00
Hongquan Li
4f3afd3199 arch: expose arch_page_phys_get() in public arch_interface.h
Move the declaration of arch_page_phys_get() from the internal
kernel_arch_interface.h to the public arch_interface.h, under the
existing arch-mmu group, so that drivers and subsystems can query the
physical address of an already-mapped virtual page without depending
on private kernel headers.

The declaration stays unconditional, as it was in the internal header:
callers such as munmap() in subsys/portability/posix/options/mmap.c
reference the function even when CONFIG_MMU is disabled (guarded only
by a runtime IS_ENABLED() check), so hiding the declaration behind
CONFIG_MMU breaks the build on MMU-less platforms.

Clean up the existing users that resorted to including the internal
header or to extern declarations:

- subsys/portability/posix/options/mmap.c
- subsys/portability/posix/options/shm.c
- subsys/portability/cmsis_rtos_v1/cmsis_thread.c

Fixes #113314

Signed-off-by: Hongquan Li <hongquan.li@processmission.com>
2026-07-28 11:33:29 +01:00
Daniel Leung
c25fa5fcc4 kernel: smp/ipi: pass cpu_id as function argument
There is no need to grab CPU ID from _current_cpu->id down in
the ipi_work_process() call chain, as it can be passed as
a function argument and can be reused. Using _current_cpu can
be costly with assertion on as it calls z_smp_cpu_mobile for
each call.

Signed-off-by: Daniel Leung <daniel.leung@intel.com>
2026-07-28 11:32:25 +01:00
Fin Maaß
f6dfffb7ea kernel: remove duplicate config OBJ_CORE_SEM
remove duplicate config OBJ_CORE_SEM

Signed-off-by: Fin Maaß <f.maass@vogl-electronic.com>
2026-07-27 11:45:01 -07:00
Josh DeWitt
d2fbd32a63 kernel: Only define z_main_thread if multithreading is enabled
z_main_thread is only needed if multithreading is enabled, so move its
definition inside a multithreading check.

Signed-off-by: Josh DeWitt <josh.dewitt@garmin.com>
2026-07-24 15:34:40 -04:00
Anas Nashif
12c33a1e5a Revert "kernel: timeout: add sys_clock_unused() and decide sloppy idle centrally"
This reverts commit 926f7c4c0f.

Signed-off-by: Anas Nashif <anas.nashif@intel.com>
2026-07-22 10:27:24 -04:00
Anas Nashif
3d7be78899 Revert "kernel: timeout: drop the idle argument of sys_clock_set_timeout()"
This reverts commit fa9b836416.

Signed-off-by: Anas Nashif <anas.nashif@intel.com>
2026-07-22 10:27:24 -04:00
Nicolas Pitre
fa9b836416 kernel: timeout: drop the idle argument of sys_clock_set_timeout()
With low-power idle handoff now going through sys_clock_idle_enter(),
nothing passes idle == true to sys_clock_set_timeout() any more, so the
argument is dead. Drop it: sys_clock_set_timeout(uint32_t ticks) now
means exactly "program the next tick, N ticks out", nothing else.

This updates the prototype, the weak default, every in-tree timer driver
definition (including the out-of-tree-style board timer under boards/),
and the core call sites. For the five drivers with low-power behaviour
this only removes the now-unused idle argument that the previous change
left on set_timeout(); the handling itself stays in their
sys_clock_idle_enter().

Signed-off-by: Nicolas Pitre <npitre@baylibre.com>
2026-07-21 18:02:43 -04:00
Nicolas Pitre
926f7c4c0f kernel: timeout: add sys_clock_unused() and decide sloppy idle centrally
The "no timeout pending, stop the clock" decision under
CONFIG_SYSTEM_CLOCK_SLOPPY_IDLE was expressed by next_timeout()
returning SYS_CLOCK_MAX_WAIT verbatim as a magic sentinel, which every
timer driver then had to recognise. Move the decision into the core and
give it an explicit, resumable interface.

Add a weak sys_clock_unused() hook (no-op default): the kernel calls it
when the timeout list is empty and sloppy idle allows uptime to drift,
in place of programming a wait. A driver may override it to actively
halt its counter; one that does not simply stops being reprogrammed and
quiesces on its own, so sloppy idle now works for every driver. Resume
is the next sys_clock_set_timeout(), exactly as before; the core keeps
no paused state.

next_timeout() no longer special-cases sloppy idle, so its empty-list
and far-timeout arms collapse to the same capped budget and
SYS_CLOCK_MAX_WAIT loses its sentinel meaning. The decision lives in
reprogram_next(), used only at the two sites where the list can drain
(abort, end of announce); the add path always has a pending timeout and
calls sys_clock_set_timeout() directly.

Signed-off-by: Nicolas Pitre <npitre@baylibre.com>
2026-07-21 18:02:43 -04:00
Anas Nashif
cdb04e25b9 kernel: widen CPU-iteration loop indices to match bound
The per-CPU loops in k_thread_runtime_stats_all_get() and the runtime
stats enable/disable paths used a uint8_t index while the loop bound
arch_num_cpus() is unsigned int.
Widen the index to unsigned int to match the bound type.

Signed-off-by: Anas Nashif <anas.nashif@intel.com>
2026-07-21 14:09:50 -04:00
Anas Nashif
85c4346a3c kernel: mmu: fix redundantl guard page assertions
The guard page checks in k_mem_unmap_phys_guard() asserted
ret == 0 inside an if (ret == 0) branch, so the assertions
could never fire and the "cannot find guard page" diagnostic
was unreachable in any build.

Assert the intended invariant (ret != 0, i.e. the guard page
is unmapped) before the branch, matching the assert-then-
handle pattern used elsewhere in this function. Behavior with
CONFIG_ASSERT=n is unchanged.

Signed-off-by: Anas Nashif <anas.nashif@intel.com>
2026-07-21 10:16:48 -04:00
Anas Nashif
11426c0d7b kernel: usage: account in-progress cycles of another CPU
A CPU's usage counters only advance when it context switches, or when it
reports on itself through z_sched_cpu_usage(). Reading the runtime stats
of a *different* CPU therefore misses whatever time that CPU has spent in
its current thread since then.

The effect is worst for an idle CPU: a CPU that ran briefly and then went
idle has the active cycles of that run in its counters, but none of the
idle time that followed, because the idle thread has not been switched
out yet. Both k_thread_runtime_stats_cpu_get() and everything built on it
therefore see a window made up purely of execution cycles, i.e. a 100%
load for a CPU that is doing nothing.

Fold the in-progress cycles into the returned view, attributing them to
the idle thread or to the CPU depending on what it is currently running.
This is done for the returned stats only, without mutating the other
CPU's accounting, as that CPU will account for the same cycles itself
once it next updates. The read is consistent because usage_lock, which is
held here, also covers the usage0 timestamp and the counters.

Reading the current CPU is unchanged.

Assisted-by: Claude:claude-opus-4-8
Signed-off-by: Anas Nashif <anas.nashif@intel.com>
2026-07-17 11:55:28 -04:00
Anas Nashif
c5fd691a58 arch: fix Doxygen structure and document macro slots in arch interface
Several documentation defects in the two canonical arch interface
headers:

- The arch-timing group was defined twice, once in arch_interface.h
  and once in kernel_arch_interface.h; turn the latter into an
  addtogroup of the former.
- The arch-smp group was opened with addtogroup before its defgroup
  appeared 300 lines later in the same file, leaving the SMP APIs
  split across two blocks with the group defined mid-file; define
  the group at first use and addtogroup at the second block.
- ARCH_STACK_PTR_ALIGN was the only stack macro documented with a
  bare @def block and no __DOXYGEN__ stub definition, making it an
  orphan @def; add the stub like its siblings.
- The ARCH_PCIE_IRQ_CONNECT doc block was wrapped in #ifdef
  CONFIG_PCIE with no __DOXYGEN__ stub, so it never appeared in doc
  builds; use the same __DOXYGEN__ pattern as the other macro slots.
- The arch-stackwalk group was closed with a malformed comment
  block; use the standard group close.
- Fix a #return that should be @return in the
  arch_thread_priv_stack_space_get() documentation.

In addition, give the ARCH_* macro slots real documentation: most of
them (ARCH_THREAD_STACK_RESERVED, ARCH_IRQ_CONNECT,
ARCH_PCIE_IRQ_CONNECT, ARCH_IRQ_DIRECT_CONNECT, the
ARCH_ISR_DIRECT_* family) carried nothing but a @see reference to
their public counterpart, leaving the porting-facing contract
undocumented. Describe what each hook must do and document its
parameters; add @brief lines to the stack alignment macros.

No functional change, comments and preprocessor-invisible
documentation stubs only.

Assisted-by: Claude:claude-fable-5
Signed-off-by: Anas Nashif <anas.nashif@intel.com>
2026-07-14 20:08:55 -04:00
Tom Burdick
093325206d kernel: smp: ipi: do not drain self bit in signal_pending_ipi()
signal_pending_ipi() used atomic_clear() on _kernel.pending_ipi, which
also drained the calling CPU's own pending bit. With directed IPIs,
arch_sched_directed_ipi() skips the calling CPU, so any IPI that had
been raised for self by another CPU was silently dropped. The lost
wakeup left the target CPU stuck until something else nudged the
scheduler.

This was latent for the existing call site in ipi_work_process(), but
became reachable on every no-swap path through do_swap() once
signal_pending_ipi() was added there, producing intermittent CPU1
starvation on SMP. tests/kernel/spinlock/spinlock_api test_trylock
(and test_spinlock_bounce on cortex_a53) reproduced it reliably on
qemu_riscv32/smp, qemu_riscv64/smp, and qemu_cortex_a53/smp: CPU0's
loop would acquire bounce_lock 10000 times in a row without CPU1
ever winning a single acquisition.

Switch to atomic_and() with the self bit as the mask, which preserves
our own pending bit (so the next interrupt entry on this CPU still
sees it and runs the scheduler) while atomically draining the bits
for the other CPUs we are about to dispatch IPIs to.

Assisted-by: Claude:claude-opus-4.7
Signed-off-by: Tom Burdick <thomas.burdick@infineon.com>
2026-07-14 09:45:40 +02:00
Anas Nashif
f1521edd22 kernel: move USE_SWITCH out of the SMP options menu
USE_SWITCH and USE_SWITCH_SUPPORTED describe a general context-switch
primitive (_arch_switch) that is usable on uniprocessor systems, not
just SMP configurations. Having them under the "SMP Options" menu in
kernel/smp/Kconfig was misleading.

Move both symbols to kernel/Kconfig proper, just before the SMP menu is
sourced. SMP still depends on USE_SWITCH; the guarding and arch selects
are unchanged.

Signed-off-by: Anas Nashif <anas.nashif@intel.com>
2026-07-12 20:02:52 -04:00
Anas Nashif
cde0a961d8 kernel: thread: factor setup_thread_stack into helpers
setup_thread_stack() interleaved stack-geometry math, optional virtual
memory mapping, and several independent #ifdef'd steps (init fill,
sentinel, TLS/local-data/random headroom, stack info) into one body,
juggling five locals across a dozen preprocessor branches.

Introduce a small struct stack_geometry and split the work into focused
helpers: compute_stack_geometry(), map_thread_stack(), fill_init_stack(),
set_stack_sentinel(), reserve_stack_headroom() and set_stack_info().
Each optional step keeps its #ifdef internally and collapses to an
ARG_UNUSED no-op when disabled, so the change is cosmetic with no
runtime cost. setup_thread_stack() is now a short linear sequence and
the headroom accounting (which must stay ordered: TLS, then local data,
then randomization, then alignment) is isolated in one place.

Assisted-by: Claude:claude-opus-4-8
Signed-off-by: Anas Nashif <anas.nashif@intel.com>
2026-07-09 18:11:09 -04:00
Anas Nashif
1c47cb6eb4 kernel: thread: factor z_setup_new_thread config blocks into helpers
z_setup_new_thread() had grown into a ~140-line body dominated by
roughly fifteen #ifdef/#endif pairs interleaved with the actual setup
logic, making the initialization sequence hard to follow.

Push each optional-feature block down into its own static inline
helper (init_thread_obj_core, init_thread_userspace, init_thread_name,
add_thread_to_monitor, init_thread_usage, and so on). Each helper keeps
its #ifdef internally and collapses to an ARG_UNUSED no-op when its
feature is disabled, so the change is purely cosmetic with no runtime
cost. The function body now reads as a linear sequence of calls.

The only block that affects control flow, the
CONFIG_ARCH_HAS_CUSTOM_SWAP_TO_MAIN early return, is converted to an
IS_ENABLED() guard since _current and resource_pool always exist; it
dead-code-eliminates when the option is off. A no-op stub is added for
setup_shadow_stack so its call site needs no #ifdef either.

Assisted-by: Claude:claude-opus-4-8
Signed-off-by: Anas Nashif <anas.nashif@intel.com>
2026-07-09 18:11:09 -04:00
Nicolas Pitre
d33280510a kernel: timeout: add bucketed delta-list backend
Add a single-level bucketed delta list as a fourth selectable timeout
backend: a simpler relative of the timer wheel that keeps the wheel's
O(1) near-future insertion without its second tier, sift/defer
machinery, or tickless-idle penalty.

Timeouts expiring within the next CONFIG_TIMEOUT_BUCKET_LISTS ticks go
into a per-tick bucket list (O(1) insert and remove), tracked by an
occupancy bitmap so "ticks until next expiry" is O(1). Everything
beyond goes into one overflow list sorted by absolute expiry (O(n)
insert, O(1) remove, no delta fix-up). As curr_tick crosses the bucket
window, overflow entries that have come within range migrate into
buckets.

Like the other backends it is a single implementation header
(kernel/timeout_bucket.h) included only by timeout.c. Two properties
distinguish it from the other non-default backends, both confirmed by
running the full kernel timer/context/common suites with the backend
forced on (qemu_x86, x86_64, cortex_a53 SMP, riscv64):

  - It fits the generic next_gap/advance/pop_due announce loop -- no
    backend-owned announce. Migration is on demand against the actual
    next event, so an idle CPU with a distant timeout sleeps straight
    to it: it passes tests/kernel/context cpu_idle / timer_interrupts,
    which the wheel fails (the wheel wakes at least every 32 ticks).
  - Same-tick firing order stays FIFO (bucket and overflow inserts
    append; migration preserves order), so tests/kernel/common's
    timeout_order passes, unlike the min-heap.

The per-node representation is the delta list's node + dticks with no
extra field: a bucket entry stores its bucket index (< BUCKET_LISTS) in
dticks, an overflow entry its absolute expiry (>= BUCKET_LISTS); the
ranges never overlap, so it shares the dlist per-node helpers. Absolute
expiry needs 64-bit ticks, so the backend depends on TIMEOUT_64BIT. It
inherits the shared z_add_timeout round-up and the inflight_timeout
synchronization like the other backends. EXPERIMENTAL.

Signed-off-by: Nicolas Pitre <npitre@baylibre.com>
2026-07-09 18:02:12 -04:00
Nicolas Pitre
e0244d279b kernel: timeout: add timer-wheel backend
Add a hierarchical timer wheel as a third selectable timeout backend.
Timeouts are bucketed by expiry distance: one list per tick for the
next 32 ticks ("soon"), one list per 32-tick band for the next ~1024
("later"), and a sorted overflow list ("distant"). Insertion and
removal are O(1) for the common near-future case; every 32 ticks the
announce path sifts the current "later" band into "soon" and refills
it from "distant". This scales well when many short-lived timeouts are
pending.

Unlike the dlist and min-heap backends, the wheel does not fit the
generic next_gap/advance/pop_due announce primitives: its per-tick
advance is a bitmap scan that jumps over empty ticks, and its sift is
a time-driven event tied to no single timeout. Rather than contort
that (already subtle) state machine, the wheel uses the backend-owned
announce escape hatch: it defines _TIMEOUT_BACKEND_OWNS_ANNOUNCE and
implements z_timeout_q_announce(), which sys_clock_announce_locked()
calls in place of the generic loop.

Like the other backends the wheel is a single implementation header
(kernel/timeout_wheel.h) included only by timeout.c, so its state, its
operations and that announce loop reach the shared state (curr_tick,
announce_remaining, inflight_timeout) directly; nothing extra needs
exposing. The SMP re-entry guard, announcing_cpu and the reprogram
remain in timeout.c. The wheel fires handlers through the same
inflight_timeout dance, so the post-#109977 abort/in-flight
synchronization works unchanged, and it carries no per-node
ANNOUNCING/ABORTED sentinels.

struct _timeout grows a wheel-only flags field (which wheel tier a
timeout occupies), gated by CONFIG_TIMEOUT_BACKEND_WHEEL so the dlist
and heap builds are unaffected.

The backend is EXPERIMENTAL. Two known limitations, inherent to the
wheel algorithm (not the abstraction):
  - No same-tick firing-order guarantee (sifted timeouts are
    prepended), so the timeout_order test does not apply.
  - next_timeout() never exceeds 32 ticks because a sift is always
    pending, so the wheel wakes a tickless-idle CPU at least every 32
    ticks. This fails tests that assert zero spurious idle wakeups
    (tests/kernel/context cpu_idle / timer_interrupts) and is a power
    regression versus the dlist and heap backends.
Verified the timeout-functional suites (timer_api, timeout,
timepoints, sleep, sched/deadline) pass with the wheel on qemu_x86,
x86_64, cortex_a53 SMP and riscv64; dlist and heap remain unaffected.

The timer-wheel data structure, bucketing scheme and sift algorithm
are the work of Peter Mitsis (PR #108339), re-homed here behind the
timeout backend interface.

Co-authored-by: Peter Mitsis <peter.mitsis@intel.com>
Signed-off-by: Nicolas Pitre <npitre@baylibre.com>
2026-07-09 18:02:12 -04:00
Nicolas Pitre
5188d5009b kernel: timeout: add min-heap backend
Add a binary min-heap as a selectable timeout backend alongside the
sorted delta list, plugging into the backend abstraction rather than
forking timeout.c. Pending timeouts are kept in a min-heap keyed on
absolute expiry tick; insertion and arbitrary removal are O(log n),
which scales better than the delta list's O(n) insertion when many
timeouts are pending.

The backend is a new kernel/timeout_minheap.h: the heap instance (a
min_heap_ref from the previous commit), its comparator and the
z_timeout_q_*() operations, included only by timeout.c (after curr_tick)
so the whole backend stays private to that translation unit. struct
_timeout gains the abs_ticks + heap_handle representation, selected by
Kconfig (the delta list's node + dticks is the #else), with the common
fn pointer kept as a shared trailing member; the per-node helpers in
timeout_q.h grow a matching min-heap variant. The kernel shell thread
dump prints the backend's raw scheduling field (dticks or abs_ticks)
under #ifdef.

Because the shared z_add_timeout() already applies the post-#107452
conditional tick round-up, the heap inherits it: the backend's insert
simply stores abs_ticks = curr_tick + dticks. Likewise in-flight
handler synchronization (PR #109977) lives in timeout.c, so the heap
node carries no ANNOUNCING/ABORTED sentinels -- "not queued" is just
heap_handle.idx == 0.

The backend is EXPERIMENTAL, depends on TIMEOUT_64BIT (absolute ticks
need 64-bit precision), and uses a fixed-capacity heap
(CONFIG_TIMEOUT_HEAP_MAX_ENTRIES) whose overflow is a fatal error.

The min-heap algorithm, struct fields, Kconfig and capacity model are
derived from Sayooj K Karun's min-heap timeout subsystem (#106013),
reworked here to fit the pluggable backend and the current in-tree
in-flight-handler synchronization.

Co-authored-by: Sayooj K Karun <sayooj@aerlync.com>
Signed-off-by: Nicolas Pitre <npitre@baylibre.com>
2026-07-09 18:02:12 -04:00
Nicolas Pitre
b381345043 kernel: timeout: factor delta-list queue into a pluggable backend
The timeout queue data structure and the front-end logic in timeout.c
are entangled through first()/next()/remove_timeout()/next_timeout()
and the open-coded insertion and announce loops. This makes it hard to
offer alternative queue implementations (min-heap, timer wheel) without
either forking timeout.c or threading #ifdefs through its trickiest
code, the announce path in particular.

Introduce a backend interface and move the sorted delta list behind it.
The split mirrors how the queue is actually used:

  - The per-node helpers z_init_timeout() / z_is_inactive_timeout()
    operate on a single struct _timeout and are needed tree-wide
    (timer.c, work.c, poll.c, ...), so they live in timeout_q.h and
    depend only on struct _timeout's fields.

  - The queue itself (its instance and the z_timeout_q_*() operations)
    is used only by timeout.c, so it becomes a backend implementation
    header (kernel/timeout_list.h) that timeout.c includes directly,
    after its shared state. The header is private to that translation
    unit; no other file sees the queue or its instance, so the instance
    is plain static and needs no extern.

timeout.c keeps all shared state (curr_tick, announce_remaining,
announcing_cpu, inflight_timeout, timeout_lock), the add/abort/query API
translation, the in-flight handler synchronization, and a single
announce loop shared by every future backend.

The backend exposes:
  - z_timeout_q_insert / _remove             add and arbitrary remove
  - z_timeout_q_remainder / _next_timeout    queries
  - z_timeout_q_next_gap / _advance / _pop_due   announce-loop primitives

The announce loop is recast in terms of the last three primitives:
find the gap to the next event, advance the backend (and curr_tick) to
it, then drain whatever is due. For the delta list this is behaviorally
identical to the previous open-coded loop: the residual advance reuses
the same head->dticks fixup, same-tick ties drain via successive
pop_due() calls, and announce_remaining is still decremented per tick
group while announcing_cpu carries the announcing state.

So the front end no longer reaches into the queue representation, three
checks are made backend-neutral: z_add_timeout()'s assert and
z_timer_expiration_handler()'s "restarted?" test now use
z_is_inactive_timeout() instead of sys_dnode_is_linked(), and
Z_TIMER_INITIALIZER zero-inits the queue fields (.fn only) rather than
naming .node/.dticks. struct _timeout is otherwise untouched; no
sentinel or per-node state changes.

The Kconfig backend choice and the divergent node layouts are deferred
until a second backend lands; with only the delta list present they
would be churn with no benefit.

Verified no behavioral change: tests/kernel/timer/{timer_api,timeout},
tests/kernel/common, and tests/kernel/sleep pass on qemu_x86,
qemu_x86_64, qemu_cortex_a53 (SMP) and qemu_riscv64 (exercising the
SMP announcing_cpu path and 64-bit dticks).

Signed-off-by: Nicolas Pitre <npitre@baylibre.com>
2026-07-09 18:02:12 -04:00
Anas Nashif
1e265cf776 kernel: kheap: keep static heap init as a PRE_KERNEL_1 SYS_INIT
Static heap init was converted from SYS_INIT(PRE_KERNEL_1, OBJECTS) to a
K_KERNEL_INIT_PRE() hook. Those hooks run before any PRE_KERNEL_1
SYS_INIT, which broke heap KASAN: K_HEAP_KASAN_ENABLE() registers a
heap's shadow at (PRE_KERNEL_1, 0), relying on running before static
heap init. With the hook, sys_heap_init() ran first, heap->kasan_ba
stayed NULL and tracking was silently disabled. On the system heap this
let the tests/lib/heap_kasan overflow cases corrupt real chunk metadata
and panic with "heap corruption (free chunk linkage)".

Revert only the kheap portion of that conversion and keep static heap
init as SYS_INIT(PRE_KERNEL_1, CONFIG_KERNEL_INIT_PRIORITY_OBJECTS),
restoring the ordering the KASAN shadow registration depends on. The
mem_slab and mailbox conversions in the same series are unaffected and
stay on K_KERNEL_INIT_PRE. A comment records why kheap must remain a
SYS_INIT so it is not reconverted.

Assisted-by: Claude:claude-opus-4-8
Signed-off-by: Anas Nashif <anas.nashif@intel.com>
2026-07-06 19:42:44 -04:00
Anas Nashif
9a27e2e135 kernel: userspace: use kernel init hooks for mem domain and shmem
Convert the two userspace PRE_KERNEL_1 SYS_INIT users to
K_KERNEL_INIT_PRE() hooks:

  userspace.c   app_shmem_bss_zero  zero the BSS of app shared-memory
                                    regions
  mem_domain.c  init_mem_domain_module  set max partitions and build the
                                        default memory domain

This removes the last SYS_INIT users in kernel/, so kernel-internal init
no longer uses SYS_INIT at all.

Assisted-by: Claude:claude-opus-4-8
Signed-off-by: Anas Nashif <anas.nashif@intel.com>
2026-07-06 15:10:55 -04:00
Anas Nashif
ff6841547c kernel: obj_core: move kernel system object registration
The kernel object is a singleton (_kernel) whose object core type was
registered and linked from init.c. It is fully self-contained -- its
k_obj_type is referenced only by that registration -- so move it next to
the object core framework in obj_core.c, alongside the registration walk.

obj_core.c already includes kernel_internal.h (for the kernel init hook
and the kernel stats helpers) and reaches _kernel via kernel_structs.h, so
no new dependencies are introduced. CONFIG_OBJ_CORE_SYSTEM depends on
CONFIG_OBJ_CORE, so obj_core.c is always compiled when this block is built.

Assisted-by: Claude:claude-opus-4-8
Signed-off-by: Anas Nashif <anas.nashif@intel.com>
2026-07-06 15:10:55 -04:00
Anas Nashif
d78ced0f09 kernel: obj_core: register object-type init via kernel init hook
Convert the object core registration walk from SYS_INIT (PRE_KERNEL_1) to
a K_KERNEL_INIT_PRE() hook. This was the last SYS_INIT in kernel/*.c, so
kernel-internal initialization no longer uses SYS_INIT at all.

obj_core.c is linked whenever the object core framework is used (its
helpers are referenced by every object's init), so the walk runs whenever
CONFIG_OBJ_CORE is enabled, exactly as before -- only the registration
mechanism changes.

Assisted-by: Claude:claude-opus-4-8
Signed-off-by: Anas Nashif <anas.nashif@intel.com>
2026-07-06 15:10:55 -04:00
Anas Nashif
3a1d70f450 kernel: kheap, mem_slab, mailbox: use kernel init hooks
Convert the statically defined kernel objects' boot-time init from
SYS_INIT (PRE_KERNEL_1) to K_KERNEL_INIT_PRE() hooks:

  kheap     build the free block list for each static heap
  mem_slab  build the free block list for each static slab
  mailbox   build the pool of asynchronous message descriptors

These initializers live in their respective translation units, which are
linked only when the application references the corresponding APIs, so
each init runs only when its subsystem is actually used (pay-per-use),
matching the previous SYS_INIT behavior without the level/priority.

Assisted-by: Claude:claude-opus-4-8
Signed-off-by: Anas Nashif <anas.nashif@intel.com>
2026-07-06 15:10:55 -04:00