On uniprocessor builds, update_cache() starts the time slice of the
thread it picks to preempt _current, and do_swap() starts it again when
it switches to that thread. Each start aborts and adds the slice
timeout, which reprograms the system timer.
do_swap() starts the slice for SMP, where the scheduler state can
change on another CPU and a switch is only known at the last moment.
Only do it there: on uniprocessor builds every switch goes to the
thread update_cache() picked, the only place that sets
_kernel.ready_q.cache.
Assisted-by: Claude:claude-opus-5-5
Signed-off-by: TaiJu Wu <tjwu1217@gmail.com>
add_thread_locked() links the thread into the new domain's list and
points mem_domain_info.mem_domain at it before calling
arch_mem_domain_thread_add(), which may fail for lack of translation
tables. k_mem_domain_add_thread() left both in place when it did, so the
thread was recorded in a domain whose memory configuration it never got,
and a retry was a no-op since the thread already looked like a member.
Put the thread back in its old domain when the move fails, and state in
the arch_mem_domain_thread_add() documentation that the architecture
leaves the thread configured for that domain in that case.
Signed-off-by: Nicolas Pitre <npitre@baylibre.com>
Move the implementation of device_is_ready() into device.h as a static
inline so the readiness check is expanded at the call site. Most callers
pass a link-time constant from DEVICE_DT_GET(), which lets the compiler
drop the NULL check and reduce the call to two loads and a branch
instead of a function call with stack frame setup.
On qemu_riscv32 this saves 2 bytes of text for hello_world and 26 bytes
for tests/kernel/device with userspace enabled; data and bss are
unchanged.
Assisted-by: Claude:claude-fable-5.1
Signed-off-by: Fin Maaß <f.maass@vogl-electronic.com>
k_mem_update_flags() looks up the physical address of the first page
only, then unmaps the whole region and maps it again as one physically
contiguous range starting at that address. Regions from k_mem_map() are
built from arbitrary free page frames and need not be physically
contiguous, so every page after the first may end up mapped to a frame
it did not own, exposing or corrupting unrelated memory while the
original frames stay allocated.
Look up the physical address of each page and unmap and map the region
one page at a time, stopping at the first error.
Assisted-by: Claude:claude-opus-5.5
Signed-off-by: Daniel Leung <daniel.leung@intel.com>
The SMP and uniprocessor schedulers were interleaved in sched.c, kswap.h,
init.c and thread.c through CONFIG_SMP conditionals, so a uniprocessor
build still had to compile around the SMP code.
Add kernel/include/ksmp.h, a set of hooks through which the common code
reaches the SMP support: CPU and thread field setup, swap_ok tracking,
halting a thread that runs on another CPU, binding and requeueing threads
at a context switch, and starting the secondary CPUs. Every hook has a
uniprocessor variant that compiles to nothing. The SMP next-thread
selection, z_get_next_switch_handle(), z_swap_next_thread(), the
halt-spin and the remote-halt path move to kernel/smp/sched_smp.c, which
is only compiled with CONFIG_SMP. The uniprocessor ready queue cache stays
in sched.c; the remaining CONFIG_SMP conditionals there, in kswap.h and
in ksched.h all select between the cache and the SMP selection.
The halt helper is exported as z_sched_halt_thread_locked() for the SMP
self-halt path and is no longer forced inline: it was a hundred lines
expanded at three call sites. unpend_all() is exported as
z_sched_unpend_all_locked() for the halt queue. set_current() moves to
ksched.h as z_sched_set_current().
No functional change intended.
Assisted-by: Claude:claude-fable-5-1
Signed-off-by: Anas Nashif <anas.nashif@intel.com>
z_add_timeout() returns the absolute expiry tick of the scheduled
timeout, which z_impl_k_sleep_ticks() records to compute the time
remaining when a sleep is cut short by k_wakeup(). For absolute
timeouts the raw encoded value (Z_TICK_ABS, a negative number) was
returned instead of the decoded expiry, so k_sleep() with an absolute
timeout reported a garbage remainder - typically 0, indistinguishable
from normal expiry - when woken early.
Return the decoded absolute expiry, matching what the relative branch
reports, and add a regression case to the kernel sleep testsuite
covering the woken-early remainder and re-arming against the same
absolute deadline.
Co-authored-by: Chris Friedt <chris@fr4.co>
Assisted-by: Claude:claude-opus-5.5
Signed-off-by: Benjamin Cabé <benjamin@zephyrproject.org>
ring_buf_init() only asserts that the size fits the ring buffer index
width. Reject a user-supplied buffer_size above RING_BUFFER_MAX_SIZE in
z_vrfy_k_pipe_init() so the ring buffer index arithmetic cannot wrap.
Signed-off-by: Flavio Ceolin <flavio@hubble.com>
z_try_abort_timeout() promises that, unless it returns -EAGAIN, the
timeout's handler is not running on another CPU. k_timer_cleanup()
relies on that before letting its caller free the timer.
The promise does not hold for a periodic k_timer, whose handler
re-queues the timeout for the next period before calling expiry_fn:
CPU0 CPU1
handler: re-queue timeout
handler: call expiry_fn ... k_timer_cleanup()
abort: timeout is queued,
remove it, return 0
caller frees the timer
... expiry_fn uses the timer (use after free)
The abort checks "is it queued?" and "is its handler running?" as
either/or, so a timeout that is both is reported as simply removed.
Check for a running handler after removing it too, and return -EAGAIN
in that case. k_timer_cleanup() and the other callers that wait on
-EAGAIN then wait for the handler to finish, without any change. The
rtio timeout iodev and k_timer_stop(), which also looked at the result,
no longer do since the previous commits.
Signed-off-by: Nicolas Pitre <npitre@baylibre.com>
k_timer_stop() must deliver the stop (stop_fn, wake a
k_timer_status_sync() waiter) unless the expiry was delivered. It
returns early when z_try_abort_timeout() doesn't dequeue the timeout,
but the abort also marks an in-flight handler superseded, so a handler
not yet past its check bails too. Neither stop_fn nor expiry_fn runs
and a k_timer_status_sync() waiter stays blocked:
CPU0 CPU1
announce: pop timer timeout
k_timer_stop(): lock timer_lock
abort: not queued, return
handler: lock timer_lock
superseded: bail
k_timer_status_sync() decides whether to wait from the timeout's queue
linkage, which the announcer changes under timeout_lock, not
timer_lock: a timeout just popped looks stopped, and the wheel backend
briefly unlinks pending timeouts it moves between lists.
Keep the armed state in the top bit of timer->status, changed only
under timer_lock: set by k_timer_start(), cleared by k_timer_stop(),
k_timer_cleanup() and by the handler on a one-shot expiry. The handler
bails if it is clear, so whichever of the stop and the expiry takes
timer_lock first is the one delivered. The other bits still count
expiries, now saturating at 2^31 - 1.
Signed-off-by: Nicolas Pitre <npitre@baylibre.com>
k_heap_alloc() and k_heap_realloc() checked the caller's original
timeout, then refreshed it from the deadline only after deciding to
pend. A waiter resumed on the tick its deadline expired pended again
with that stale timeout, so the call blocked for one more tick.
k_heap_realloc() of size 0 frees the block and returns NULL. The same
loop treated that NULL as another attempt, freeing the block twice and
leaving waiters asleep. Stop after a zero-size realloc and wake the
heap wait queue, matching k_heap_free().
Assisted-by: Cursor:grok-4.7
Signed-off-by: Ajay Rajan <ajaycrajan@gmail.com>
Every k_object carries a perms[] bitmap with one bit per thread index,
but the code that sets, clears, tests and copies those bits took a
struct k_thread * and looked the index up each time, in five places,
with the "no index" value spelled -1 against an unsigned return type
and cast back and forth between int and uintptr_t.
Split the two concerns. thread_index_get() is now the single place
that maps a thread to its permission index, returning PERM_INDEX_NONE
for threads that are not kernel objects. The bitmap work moves into
perms_set_index(), perms_clear_index(), perms_all_clear_index(),
perms_test_index() and perms_inherit_index(), which take an index and
know nothing about threads. The k_thread_perms_*() functions and
thread_perms_test() become one-line wrappers, so no caller changes.
The behaviour of every path is unchanged, including the lists_lock
handling around unref_check() and the parent-thread exclusion when
permissions are inherited. The split lets a principal other than a
thread hold a permission index later without touching the bitmap code
again.
Assisted-by: Claude:claude-fable-5-1
Signed-off-by: Anas Nashif <anas.nashif@intel.com>
signal_pending_ipi() told every reschedule caller to hold
_sched_spinlock. The no-swap paths do not, and that is the normal
return from z_sched_wake() and z_ready_thread(). Draining without
the scheduler lock does not drop the wakeup: the target still takes
the IPI and runs the scheduler after the flagging CPU releases the
lock. Taking the lock there adds a round trip on every non-swapping
z_reschedule() and gains nothing.
Require the scheduler lock only when the drain is atomic with
picking the next thread. Drop the TODO in z_reschedule_irqlock().
Tested on qemu_cortex_m3: tests/kernel/sched/schedule_api passed
21, skipped 8. Skips are ZTEST_USER; this board has USERSPACE=n.
Assisted-by: Cursor:Grok-4.7
Signed-off-by: Ajay Rajan <ajaycrajan@gmail.com>
This allows locking and unlocking uncontended mutexes without
syscalls. The futex stores the TID of the owner. The LSB should
be free, since TIDs are 4-byte aligned, and is used to indicate
if a thread is waiting.
This does not implement priority inheritance (PI), which requires
a different kind of futex.
Update futex test to use k_mutex as non-futex, since sys_mutex
now contains a futex.
Add separate test for futex-based sys_mutex, which does not cover
PI and instead tests the contested path.
Signed-off-by: Christoph Busold <cbusold@qti.qualcomm.com>
This is required to extend the value on 64-bit targets so that
pointers can be used as futex values.
Signed-off-by: Christoph Busold <cbusold@qti.qualcomm.com>
z_sched_wake() is ALWAYS_INLINE but the unpend_thread_no_timeout() it
calls is only static inline, and with -Os GCC often emits it as a call
instead, depending on the target and on how many wake sites a file has.
That puts a call and return on the wake path of the kernel objects that
wake a waiter.
Mark it ALWAYS_INLINE like its caller.
The outlined copy costs a call and a return, plus a stack frame and
callee-saved register spills when the copy needs them. That is about 11
instructions on x86 and 3 to 9 on cortex_a53. Where GCC already inlined
it, nothing changes.
Time saved by tests/benchmarks/latency_measure, default -Os, on wake
operations that switch to the woken thread (kernel threads; QEMU with
icount):
x86 cortex_a53 riscv64
k_sem_give() 2.6% 0 0
k_fifo/k_lifo put 2.3% 1.4% 1.5%
k_event set/post 0.9% 3.2% 3.8%
k_condvar_signal() 2.0% 0 0
With CONFIG_USERSPACE or CONFIG_OBJ_CORE the same operations take
longer, so the saving ranges from 0.3% to 3.6%. Other changes stay under
0.3% and come from riscv64 timer granularity.
The regressions: on x86, thread abort and condvar wait get 0.1% to 0.3%
slower, and text grows by 0 to 28 bytes over samples/synchronization,
tests/benchmarks/footprints and tests/benchmarks/latency_measure on
cortex_a53, cortex_m3, riscv32, riscv64 and x86. That is a good
compromise given the gains above.
Signed-off-by: Nicolas Pitre <npitre@baylibre.com>
device_is_ready() documents that a NULL pointer yields false, and the
implementation checks for it. The user mode verifier however validates
the pointer as a kernel object first, and NULL is not one, so a call
from a user thread ends in a kernel oops before the check is reached.
Check for NULL in the verifier before the kernel object validation, and
add a user mode test for it.
Fixes#121103
Assisted-by: Claude:claude-fable-5-1
Signed-off-by: Anas Nashif <anas.nashif@intel.com>
do_device_init() set dev->state->initialized even when the init function
returned an error, and device_deinit() only checks that flag. The deinit
function could therefore run on a device that never completed its
initialization and release resources it never acquired. The flag also
made device_init() return -EALREADY for such a device, so a failed
initialization could never be retried without poking the state directly.
Set the flag only when the init function succeeds. A device whose
initialization failed now stays uninitialized: device_deinit() refuses it
with -EPERM and device_init() runs the init function again, which is what
the deferred initialization documentation already promises for devices
that are not initialized.
Extend the deferred initialization failure test accordingly and add a
test that takes a device through failed init, retry, deinit and re-init.
The net iface test, which forced a device ready by clearing init_res
directly, now calls device_init() instead.
Assisted-by: Claude:claude-fable-5-1
Signed-off-by: Anas Nashif <anas.nashif@intel.com>
do_device_init() only wrote dev->state->init_res when the init function
failed. A device whose initialization failed once, and which is later
initialized again successfully through device_init(), kept the stale
error code, so device_is_ready() reported it as not ready although its
init function had succeeded.
Record the result of every initialization, so that a successful run
clears a previous failure.
Assisted-by: Claude:claude-fable-5-1
Signed-off-by: Anas Nashif <anas.nashif@intel.com>
finalize_cancel_locked() walks pending_cancels to release threads
blocked in k_work_cancel_sync(). Commit 39f710136e ("sched:
finalize_cancel_locked can early return") stopped after the first
match, but the same work item can appear more than once when several
threads cancel it. Remaining waiters then block forever on their
canceller semaphores after the item is already idle.
Resume waking every matching waiter, and keep the slist predecessor
pointer on the node before a removal so later matches unlink correctly.
The new test sleeps after submit and after starting the helpers. The
ztest thread is cooperative, so those helpers do not preempt it.
Fixes#119700
Assisted-by: Codex:GPT-6
Signed-off-by: Ajay Rajan <39146318+Ajay-Embed@users.noreply.github.com>
Signed-off-by: Ajay Rajan <ajaycrajan@gmail.com>
Return any error from z_pend_curr() before reading swap_data, and
report the same result through the tracing hook. Only a successful
wait supplies a value to the caller.
Assisted-by: Codex:GPT-6
Signed-off-by: Ajay Rajan <ajaycrajan@gmail.com>
CHECKIF() removes the full-stack and pending-waiter checks with
CONFIG_NO_RUNTIME_CHECKS and asserts with CONFIG_ASSERT_ON_ERRORS.
Use plain conditions so push returns -ENOMEM and cleanup returns
-EAGAIN without overflowing the buffer or freeing it under a waiter.
Assisted-by: Codex:GPT-6
Signed-off-by: Ajay Rajan <ajaycrajan@gmail.com>
k_stack_init() can retain K_STACK_FLAG_ALLOC from an earlier allocation.
Clear flags when installing a caller-owned buffer so cleanup does not
free that buffer. Check the flag before cleanup in the regression test.
Assisted-by: Codex:GPT-6
Signed-off-by: Ajay Rajan <ajaycrajan@gmail.com>
k_pipe_read() on a closed empty pipe returns -EPIPE immediately, but
k_poll() only checked the ring buffer. A closed empty pipe therefore
never became ready, so a poller could wait forever, and k_pipe_close()
woke wait-queue threads but not registered pollers.
Treat a closed pipe as K_POLL_STATE_PIPE_DATA_AVAILABLE in
is_condition_met(), and wake all pipe pollers from k_pipe_close()
(with a reschedule, matching the read/write paths).
Assisted-by: Codex:GPT-6
Signed-off-by: Ajay Rajan <39146318+Ajay-Embed@users.noreply.github.com>
Signed-off-by: Ajay Rajan <ajaycrajan@gmail.com>
Re-initializing an owned mutex leaves its held_node linked in the
owner's held_mutexes list and resets the wait queue under pending
waiters. Reject it from user mode, and reset the fields under
mutex_lock so concurrent lockers never see a half-initialized object.
Assisted-by: Claude:claude-opus-5.5
Signed-off-by: Flavio Ceolin <flavio@hubble.com>
Group the header-only timeout queue backends under a dedicated
subdirectory so the algorithm implementations sit together, and
update timeout.c includes accordingly.
Signed-off-by: Dhruv Menon <dhruvmenon1104@gmail.com>
Signed-off-by: Dhruv Menon <dhrumeno@qti.qualcomm.com>
Export the SMP IRQ lock helpers used by irq_lock()/irq_unlock() and
the 32-bit counter used by k_cycle_get_32() and spinlock time limits.
These existing inline APIs leave imports in loadable extensions that
cannot resolve them without entries in the built-in symbol table.
Signed-off-by: Jinming Zhao <jinmzhao@qti.qualcomm.com>
z_timeout_remaining() subtracted elapsed() from the queued remainder
with no lower bound. A late sys_clock_announce() can make elapsed()
exceed remainder, wrapping unsigned k_ticks_t or going negative with
CONFIG_TIMEOUT_64BIT. Saturate at 0, matching next_timeout().
Compare the observed time against the timer deadline so the regression
test requires an overdue timer, including the relative-timeout round-up.
Assisted-by: Codex:GPT-6
Signed-off-by: Ajay Rajan <39146318+Ajay-Embed@users.noreply.github.com>
Signed-off-by: Ajay Rajan <ajaycrajan@gmail.com>
Queues were the one object type that object tracking listed but object
cores did not know. Add OBJ_CORE_QUEUE, the queue object type and its
registration in k_queue_init(), so that object cores cover every object
type that object tracking does.
Assisted-by: Claude:claude-fable-5-1
Signed-off-by: Anas Nashif <anas.nashif@intel.com>
An object type enumerates its permanent objects from one range. The
objects of a type can also be embedded in another type's statically
defined instances, as the queue inside every K_FIFO_DEFINE()d FIFO is,
and those instances live in a section of their own.
Let a type carry up to K_OBJ_TYPE_MAX_RANGES ranges, defined with
K_OBJ_TYPE_DEFINE_RANGES() and K_OBJ_RANGE_SECTION(), so that such
objects are walked in place too instead of each taking a registry entry.
Assisted-by: Claude:claude-fable-5-1
Signed-off-by: Anas Nashif <anas.nashif@intel.com>
A kernel object embedded in a k_malloc()ed or slab-allocated structure
ends its life with the block that holds it, and nothing calls the object
core API when that block is released. Its registry entry then stays
until the storage is reused by an object of another type.
Add a hidden SYS_HEAP_RELEASE_HOOK that the heap calls with the released
memory while it is still valid, from sys_heap_free() and from the tail an
in-place shrink gives up, and have k_mem_slab_free() report the released
block the same way. Object cores provide the hook under
OBJ_CORE_EVICT_ON_FREE, default on, and drop every entry located in the
released range. User mode frees are ignored: the registry is kernel
data and user mode never holds registered objects.
Assisted-by: Claude:claude-fable-5-1
Signed-off-by: Anas Nashif <anas.nashif@intel.com>
The kernel knows when it ends an object's life itself: k_object_free()
releases a dynamically allocated kernel object, k_msgq_cleanup() and
k_stack_cleanup() release a buffer the object owned, and
k_timer_cleanup() is the last call before a timer's storage is freed.
Remove the object from the registry at those points so it stops being
reported at once instead of when its storage is reused. Cleanup of a
caller-provided buffer leaves the registration in place, as the object
stays usable.
Assisted-by: Claude:claude-fable-5-1
Signed-off-by: Anas Nashif <anas.nashif@intel.com>
A kernel object declared in a function body, such as the semaphore in a
k_work_sync or the one a blocking can_send() waits on, ends with the
frame that holds it. Registering it makes the registry report an object
that does not exist and spends an entry on it for every call.
Refuse to register an object located in the current thread's stack or in
the interrupt stack and count it in the object type instead. This is the
policy the user mode object table already follows for objects declared
on stacks. OBJ_CORE selects THREAD_STACK_INFO for the stack bounds.
Assisted-by: Claude:claude-fable-5-1
Signed-off-by: Anas Nashif <anas.nashif@intel.com>
K_OBJ_TYPE_DEFINE() emitted a descriptor that a kernel init pass used to
fill in a separate, zero-initialized struct k_obj_type. An object
initialized before that pass, for instance from an EARLY init level,
registered with a type that had no ID and no range yet.
Make the macros define the struct k_obj_type itself, fully initialized,
in an iterable section. k_obj_type_find() searches that section before
the run-time type list, so the types are usable from the first
instruction. The init pass now only adds them to the type list and
initializes the object cores of their permanent objects. The macros give
the type variable internal linkage; the FIFO and LIFO types, which
k_fifo_init() and k_lifo_init() reference from kernel.h, use the
initializer macro instead.
Assisted-by: Claude:claude-fable-5-1
Signed-off-by: Anas Nashif <anas.nashif@intel.com>
Object cores linked every object into a per-type slist through a node
embedded in the object, so an object discarded without an unlink left
the list tail pointing into dead memory and the next k_sem_init() wrote
through it. Re-initializing a linked object truncated the list, and
freed dynamic objects left dangling entries. Kernel objects have no
mandated end of life: semaphores and mutexes are routinely stack locals,
members of freed structs or re-initialized in place, so requiring an
unlink from every such user is not an option.
Keep no registry state inside the objects. Statically defined objects
are enumerated from their iterable sections through a range recorded in
the object type. Objects initialized at run time are referenced from a
bounded table keyed by address: re-registration refreshes the entry, an
object whose storage was reused is detected by its type tag and dropped,
and a full table refuses the registration and counts it. Walks visit the
static range and then the table, so a discarded object can only be
stale, never corrupting.
Fixes#114856
Assisted-by: Claude:claude-fable-5-1
Signed-off-by: Anas Nashif <anas.nashif@intel.com>
Replace scripts/kconfig/hardened.csv with a distributed YAML database.
The CSV format could not carry any explanation for its recommendations,
had no validation (a malformed line was silently dropped), no notion of
profiles, no ownership story, and no way to extend it out of tree.
The database now lives where the code lives: the central
scripts/kconfig/hardening.yaml defines the hardening profiles (plus
rules for top-level Kconfig symbols), and per-subsystem hardening.yaml
fragments next to the relevant Kconfig files carry the rules, so each
rule is reviewed by the maintainers of the code it hardens. Every rule
has a required rationale and optional references, supports exact-value
and integer min/max recommendations (STACK_POINTER_RANDOM is now
'>=100' instead of '==100'), and belongs to a profile: 'base' (memory
protection and inherently insecure features) or 'strict' (extends base
with debug/observability feature removal). The default profile is
'strict', matching the rule set the CSV checked.
hardenconfig is restructured around a serializable HardeningReport
model so results can be consumed programmatically, and gains:
- a Rationale column in the report
- profile selection:
west build -t hardenconfig -- -DHARDENCONFIG_PROFILE=base
- HARDENCONFIG_SHOW_ALL to also list PASS/NA results
- HARDENCONFIG_STRICT to exit non-zero on failures, for CI use
- HARDENCONFIG_JSON to write machine-readable results, including which
database file each rule came from
- HARDENCONFIG_EXTRA_SOURCES to layer out-of-tree database files that
may add profiles/rules and override in-tree ones
The EXPERIMENTAL/DEPRECATED/NOT_SECURE marker symbol detection is
preserved unchanged.
Three CSV entries are intentionally not carried over because they were
redundant: BT_CONN_DISABLE_SECURITY and BT_OOB_DATA_FIXED can only be
enabled together with BT_TESTING, and TIMER_RANDOM_GENERATOR together
with TEST_RANDOM_GENERATOR, all of which the database already
recommends disabling for the same profiles.
Behavioral fix: the CSV's DEBUG_COREDUMP line used '=' instead of ','
and was silently ignored, so CONFIG_DEBUG_COREDUMP was never actually
checked; it now is.
Also add tabulate to requirements-base.txt: hardenconfig has always
imported it, but it was only pulled in via other requirements files.
Assisted-by: Claude:fable-5
Signed-off-by: Benjamin Cabé <benjamin@zephyrproject.org>
Default SYS_CLOCK_HW_CYCLES_PER_SEC to the timebase-frequency property
of the first CPU node when it is set, like DCACHE_LINE_SIZE does with
d-cache-line-size. The property describes exactly this value: the
frequency at which the CPU timebase (on RISC-V the mtime counter)
advances. edtlib falls back to /cpus for CPU properties, so this
covers the property placed on either /cpus or /cpus/cpu@0.
SoC and board Kconfig.defconfig defaults still take precedence.
Assisted-by: Claude:claude-opus-5.5
Signed-off-by: Fin Maaß <f.maass@vogl-electronic.com>
z_vrfy_k_thread_stack_space_get() consumed its K_SYSCALL_OBJ() result
with CHECKIF(), which is compiled out under CONFIG_NO_RUNTIME_CHECKS
and CONFIG_ASSERT_ON_ERRORS. Wrap it in K_OOPS() instead, leaving
CHECKIF() on the remaining error propagation.
dynamic_disabled.c had no verifiers for k_thread_stack_alloc() and
k_thread_stack_free(). Add pass-throughs; neither argument is
dereferenced there, and checking the stack object would replace the
documented -ENOSYS return.
Signed-off-by: Benjamin Cabé <benjamin@zephyrproject.org>
Assisted-by: Claude:opus-5
The compiler cannot prove the payload copy leaves the queue fields
alone, so it reloaded the ring pointers after it. Pick the slot and
advance the pointer first.
thread_metric message processing, base / reordered / delta:
x86 133267 / 136881 / +2.71%
Xtensa DC233C 84595 / 87528 / +3.47%
RISC-V 32 80998 / 84130 / +3.87%
ARM Cortex-A53 497986 / 523815 / +5.19%
ARM Cortex-M3 96465 / 103732 / +7.53%
ARM Cortex-M4 410042 / 442637 / +7.95%
The first five are QEMU under icount; the Cortex-M4 figure is an MXChip
AZ3166 at 96 MHz. app_kernel improves 4.5% to 5.5% on one and four byte
messages and 0.5% at 192 bytes, where the copy dominates. Text shrinks
by 20 bytes and there is no impact on RAM.
Assisted-by: Claude:claude-opus-5
Signed-off-by: Benjamin Cabé <benjamin@zephyrproject.org>
An architecture may cache the current thread in a register rather than
read it from the per-CPU structure, which is what
CONFIG_ARCH_HAS_CUSTOM_CURRENT_IMPL is for. That is worth doing on the
hottest path in the kernel, but it only pays if the compiler is allowed
to reuse a value it has already read, which in turn requires that no
code depend on _current changing underneath it.
The uniprocessor half of z_get_next_switch_handle() stores the outgoing
thread's switch handle through _current, sets the new current thread,
then reads _current again expecting the incoming one. Give it the same
treatment as its SMP counterpart and name the target once. It also stops
reading _kernel.ready_q.cache three times.
With this and the two preceding commits, nothing in the core reads the
current thread back after setting it.
Signed-off-by: Nicolas Pitre <npitre@baylibre.com>
z_spin_lock_set_owner() and z_spin_unlock_valid() need the thread that
is current at the moment they run. The unlock case in particular has to
see the dummy thread that z_thread_halt() installs when an ISR aborts
the thread it interrupted, which is how it recognizes that the abandoned
lock has no owner left to match.
They read _current, which an architecture may cache in a register
(CONFIG_ARCH_HAS_CUSTOM_CURRENT_IMPL), letting the compiler carry a
value across the store that made the new thread current. Use
_raw_current, which cannot be cached. Both are called from k_spin_lock()
and k_spin_unlock() with the lock held, so interrupts are locked as that
requires.
No functional change where the architecture does not cache the current
thread, _current being _raw_current there already.
z_assert_can_swap() keeps using _current: it runs before the swap rather
than after it, and it is reachable with interrupts unlocked, where
_raw_current is not allowed.
Signed-off-by: Nicolas Pitre <npitre@baylibre.com>
z_spin_lock_transfer_owner() is called from do_swap() and from
z_get_next_switch_handle() right after z_current_thread_set(), and it read
the incoming thread back from _current. An architecture that caches the
current thread in a register (CONFIG_ARCH_HAS_CUSTOM_CURRENT_IMPL) lets the
compiler reuse a value read before the store, so the lock is recorded as
still owned by the outgoing thread and the next release fails
z_spin_unlock_valid(). Reproducible under LTO.
Both call sites have the incoming thread in a local already. Pass it.
Signed-off-by: Nicolas Pitre <npitre@baylibre.com>
_current is one of three things: the field the kernel records in the
per-CPU structure, a call that reads that field under an interrupt lock,
or a register an architecture keeps it in
(CONFIG_ARCH_HAS_CUSTOM_CURRENT_IMPL). Only the first is guaranteed to
report the incoming thread between z_current_thread_set() and the switch
that follows it, and code needing that has no accessor for it, so it
open-codes _current_cpu->current.
Name the field _raw_current, document what using it requires, and build
the rest on top: the uniprocessor _current is now that accessor,
z_smp_current_get() reads it under the lock it already takes, and
z_current_thread_set() writes through it.
No functional change.
Signed-off-by: Nicolas Pitre <npitre@baylibre.com>
The disco_l475_iot1 is the board on which the thread_metric benchmarks
have been primarily executed.
Applying the likely() attribute to the non-SMP case where the thread
being suspended is the currently executing thread has been osbserved
to improve the thread_metric benchmark performance by about 1% on the
disco_l475_iot1 board.
This attribute resulted in the following observed changes in the
generated assembly.
1. The branch to the preferred code path is no longer a forward
branch but a fall through. CPUs typically favor fall throughs
and reverse branches over forward branches, so this helps
leverage that bias for performance.
2. The fall through to the preferred z_thread_suspend_current()
path has been observed to be in the same cache line as the
preceding instructions, whereas before this change, it was in
a different cache line. Long term, this does not guarantee that
it will always be in the same cache line as that will ultimately
depend upon the linker, but it does increase the odds.
Signed-off-by: Peter Mitsis <peter.mitsis@intel.com>
Return before taking usage_lock when the current CPU's accounting window
is already stopped. This avoids a redundant global-lock acquisition when
architecture IRQ-entry accounting precedes scheduler IRQ-exit accounting.
The hook already requires local interrupts to be masked, and only the
local CPU writes usage0. Keep the existing lock for every shared-counter
update and preserve runtime-statistics behavior.
Microbenchmarks on a quad-core ARMv8 SMP platform with runtime statistics
enabled showed 2.5-4.4% lower latency and up to 12.8% higher throughput.
Assisted-by: Codex:GPT-5
Signed-off-by: Aaron Wisner <aaronwisner@gmail.com>
k_pipe_write() reported K_POLL_STATE_PIPE_DATA_AVAILABLE before it put
anything into the ring buffer, and did so whether or not the write
stored any data. A thread blocked in k_poll() on
K_POLL_TYPE_PIPE_DATA_AVAILABLE was therefore woken by writes that left
the pipe with nothing to read.
The case that always misreports is an unbuffered pipe, which
k_pipe_init() accepts with a NULL buffer and a size of zero. Such a pipe
has no storage, so data only moves by direct copy to a pending reader. A
write that finds no reader transfers nothing and returns -EAGAIN, yet it
still told pollers that data was available. A write consumed entirely by
pending readers, and a write that stores nothing because the pipe is
full, misreport the same way.
Store the data first and report availability only when ring_buf_put()
accepted at least one byte. Data handed straight to a pending reader
never reaches the pipe, so it is not reported.
Enable CONFIG_POLL in the pipe test scenario. Verify that data
transferred directly from an unbuffered pipe writer to a pending reader
does not wake a poller, while data stored in a buffered pipe does.
Ran kernel.pipe.api.poll on native_sim: 21 of 21 test cases passed.
Fixes: a9ab8cb779 ("feat: enable polling support for k_pipe interface")
Assisted-by: Codex:gpt-5
Signed-off-by: Jaeyeong Lee <lee@jaeyeong.cc>
Add a common hook declaration and Kconfig switch for firmware
arguments provided by architecture startup code.
The hook receives an architecture-provided argument array and count.
This keeps the common interface independent from register naming used
by a specific architecture.
On ARM64, preserve x0-x3 from the primary CPU reset path before
startup code reuses those registers. The saved values are stored only
after the primary CPU path is selected and are passed to the hook after
RAM has been initialized for C code. Secondary CPUs do not update the
saved storage.
Signed-off-by: Vladyslav Goncharuk <vladyslav_goncharuk@epam.com>
Assisted-by: Codex:gpt-5
Add a check in thread monitor to assert if a TCB is reused while it is
still present in the kernel thread list.
This prevents undefined behavior due to TID reuse before proper cleanup.
Signed-off-by: Chidvilas Yerramsetti <cyerrams@qti.qualcomm.com>
The slim ring buffer API is now header-only and always available and the
tree has already migrated from the _claim/_finish API, so the RING_BUFFER
symbol no longer gates any code.
Remove the now-redundant "select RING_BUFFER" and CONFIG_RING_BUFFER=y
entries across drivers, subsystems, samples and tests.
Signed-off-by: Måns Ansgariusson <mansgariusson@gmail.com>
k_msgq_get() falls straight through to the empty case and leaves the
transfer behind a taken branch, though the transfer is both the common
case and the only one that copies anything.
A transfer costs five cycles less and a get on an empty queue two more,
on a path that returns without copying and that a caller hitting it
routinely would normally be blocking on rather than polling anyway.
thread_metric message processing, base / likely() / delta:
x86 133267 / 133267 / +0.00%
Xtensa DC233C 84595 / 84595 / +0.00%
ARM Cortex-M3 96465 / 97083 / +0.64%
ARM Cortex-A53 497986 / 501946 / +0.80%
RISC-V 32 80998 / 81839 / +1.04%
ARM Cortex-M4 410042 / 418985 / +2.18%
The first five are QEMU under icount; the Cortex-M4 figure is an MXChip
AZ3166 at 96 MHz, where flash wait states make a taken branch dearer.
Text shrinks by 4 bytes and there is no impact on RAM.
Assisted-by: Claude:claude-opus-5
Signed-off-by: Benjamin Cabé <benjamin@zephyrproject.org>