sys_sflist stores flags in unused low bits of node links. The number of
available bits follows the ABI's natural pointer alignment, but the generic
header currently forces pointer-size alignment and globally requires two
bits.
That guarantee does not hold for borrowed nodes such as intrusive k_queue
items, which are caller-owned objects cast to sys_sfnode_t. On ABIs with
four-byte pointers and two-byte pointer alignment, only bit 0 is reliably
free.
Remove the forced alignment, expose SYS_SFLIST_FLAG_BITS, and let consumers
assert their own requirements. k_queue and the MMU free-page list each
require one bit. Make the unit test honor the capacity reported by the ABI
and clarify the intrusive queue alignment documentation.
This preserves generic flagged lists and safely supports one-bit ABIs.
Assisted-by: ChatGPT:gpt-5.6
Signed-off-by: Dimitri Varpusvuori <dimitri.varpusvuori@gmail.com>
An empty k_queue_get() emits the blocking trace before checking
K_NO_WAIT. This records a block that never occurs and invokes the
tracing backend on a non-blocking fast path.
Move the trace below the no-wait return so only calls that can pend
emit it. Add CTF coverage for an empty no-wait get.
Signed-off-by: Magnus Strømme <magnus.henrik@hotmail.com>
k_queue_alloc_append() and k_queue_alloc_prepend() wrap the caller's data
in an internal alloc_node before inserting it into the queue. The old
k_queue_remove() implementation searched for the caller data pointer as
if it were the linked-list node, so removal returned false and left the
allocated queue item in place.
Walk the queue and compare the unwrapped payload pointer instead. Free
the alloc_node wrapper after removing a matching item, while preserving
the existing behavior for directly inserted queue nodes.
Add coverage for removing items inserted through both alloc APIs.
Tested:
- ZEPHYR_TOOLCHAIN_VARIANT=gnuarmemb GNUARMEMB_TOOLCHAIN_PATH=/usr
west build -d /tmp/zephyr-queue-gnuarmemb
- qemu_cortex_m3: queue_api 15/15 and queue_api_1cpu 7/7
Signed-off-by: Hui Su <3164683437@qq.com>
Each kernel object type that participates in the object core framework
previously supplied its own SYS_INIT routine to initialize its
k_obj_type and to walk its static-object linker section, linking each
object core. These routines were near-identical across 11 object types
and differed only in the type id, the object struct and the obj_core
offset.
Replace that per-type boilerplate with a declarative K_OBJ_TYPE_DEFINE()
macro that emits a const descriptor into a new iterable ROM section, and
walk those descriptors once from a single SYS_INIT in obj_core.c. The
descriptor captures the type storage, type id, obj_core offset, the
static object section bounds and the object stride, which is all the
shared loop needs to initialize and link every statically defined
object.
Converted: condvar, event, fifo, lifo, mailbox, msgq, mutex, pipe, sem,
stack and timer. The non-uniform initializers (thread, mem_slab and the
internal cpu/kernel objects) are left unchanged and will be dealt with
in followup commits.
Footprint on qemu_cortex_m3 (tests/kernel/obj_core/obj_core, all types
enabled): flash 32012 -> 30880 (-1132 B), RAM unchanged. With
CONFIG_OBJ_CORE disabled the image is byte-for-byte identical.
Assisted-by: Claude:claude-opus-4-8
Signed-off-by: Anas Nashif <anas.nashif@intel.com>
Some private symbol names were widely duplicated throughout the kernel
such as lock and handle_poll_event. This arguably made the kernel less
readable as the locality of the name lock is highly confusing.
Rename all compilation unit locks to match their usage (e.g.
mutex_lock). Improving readability.
Furthermore, by deduplicating these symbols we enable potential
amalgamation builds of the kernel where all C files are merged
into one large C file or compliation unit allowing for better
compiler visibility and optimization.
Signed-off-by: Tom Burdick <thomas.burdick@infineon.com>
Replace z_unpend_first_thread() with z_unpend_first_thread_locked() and
migrate every caller across the kernel. The old function dropped the
scheduler spinlock before returning, exposing a race window between
its caller's "arch_thread_return_value_set + z_ready_thread" pair and
a still-in-flight timeout handler that could ready the thread first --
the woken thread might then run on another CPU and see an uninitialized
swap_retval. Pre-1b8c7a3 the dticks-cancel check made the handler bail;
here we fix it cleanly by requiring the caller to hold _sched_spinlock
across the entire wake, so the handler is blocked for the duration and
runs as a no-op afterwards.
z_unpend_first_thread_locked() requires the caller to be inside a
locked region and must be paired with z_sched_ready_locked() (and
whatever return-value setup is needed) under the same lock acquisition.
Sites migrated:
Simple "set retval [+ swap_data] and ready" callers use the existing
z_sched_wake() convenience wrapper, refactored to use the new
helper internally:
sem (give, reset), mem_slab (free), stack (push),
condvar (signal, broadcast), msgq (purge),
queue (cancel_wait, queue_insert, append_list),
futex (wake).
Sites that need additional setup on the woken thread use
LOCK_SCHED_SPINLOCK + z_unpend_first_thread_locked() + custom wake:
mutex (unlock -- needs the thread reference to track new owner),
msgq put / get (needs memcpy into the receiver's swap_data
buffer before the return value is set).
The dticks-cancel check in z_thread_timeout() is left in place; it is
no longer load-bearing once z_abort_thread_timeout() has no callers,
and is removed in the next commit.
Signed-off-by: Nicolas Pitre <npitre@baylibre.com>
Fixes a TOCTOU race in k_queue_unique_append() by locking the
queue's spinlock around both the search and insert operations.
Signed-off-by: Peter Mitsis <peter.mitsis@intel.com>
The k_queue helper routine queue_insert() now requires that
the queue's spinlock be obtained prior to calling it. Its
key is now passed as a parameter into it so that the lock
will be released before it returns.
Signed-off-by: Peter Mitsis <peter.mitsis@intel.com>
To avoid a TOCTOU type error in k_queue_remove(), it must lock
the queue's spinlock for the duration of the operation.
Signed-off-by: Peter Mitsis <peter.mitsis@intel.com>
When peeking at an allocated node, the allocated node must be
dereferenced. Unless the queue's spinlock is held that allocated
node could be freed and that memory re-used for something else
entirely leading to the kernel dereferencing an invalid pointer.
Signed-off-by: Peter Mitsis <peter.mitsis@intel.com>
kernel.h implies kernel_structs.h via kernel_includes.h, making
explicit inclusion of kernel_structs.h unnecessary whenever kernel.h
is already included in the same translation unit.
Remove the redundant includes across arch, boards, drivers, kernel,
lib, samples, subsys, and tests trees.
in include/zephyr/kernel_structs.h:
* 2. kernel.h shall imply kernel_structs.h, such that it shall not be
* necessary to include kernel_structs.h explicitly when kernel.h is
* included.
Signed-off-by: Anas Nashif <anas.nashif@intel.com>
queue_insert emitted SYS_PORT_TRACING_OBJ_FUNC_BLOCKING twice:
1. When a pending thread is found and woken (correct).
2. Unconditionally before sys_sflist_insert when no thread is
pending and the item is placed directly on the list (incorrect).
The second emission is wrong: no blocking occurs in that path — the
caller's data is simply enqueued and the function returns. Emitting
a BLOCKING event there misrepresents the operation to tracing
consumers and is likely a copy-paste error from the first branch.
Remove the second SYS_PORT_TRACING_OBJ_FUNC_BLOCKING call.
Assisted-by: GitHub Copilot:claude-opus-4.6
Signed-off-by: Anas Nashif <anas.nashif@intel.com>
Updates the queue code to both allow for a fast return path in a
few routines when the operation did not wake or signal another
thread.
Signed-off-by: Peter Mitsis <peter.mitsis@intel.com>
Propagates the return value from z_handle_obj_poll_events()
within the message queue, pipes, queue and semaphore objects.
This allows the kernel object code to determine whether it
needs to perform a full reschedule, or if it can perform a
more optimized exit strategy.
Signed-off-by: Peter Mitsis <peter.mitsis@intel.com>
Mostly a revert of commit b1def7145f ("arch: deprecate `_current`").
This commit was part of PR #80716 whose initial purpose was about providing
an architecture specific optimization for _current. The actual deprecation
was sneaked in later on without proper discussion.
The Zephyr core always used _current before and that was fine. It is quite
prevalent as well and the alternative is proving rather verbose.
Furthermore, as a concept, the "current thread" is not something that is
necessarily architecture specific. Therefore the primary abstraction
should not carry the arch_ prefix.
Hence this revert.
Signed-off-by: Nicolas Pitre <npitre@baylibre.com>
`_current` is now functionally equals to `arch_curr_thread()`, remove
its usage in-tree and deprecate it instead of removing it outright,
as it has been with us since forever.
Signed-off-by: Yong Cong Sin <ycsin@meta.com>
Signed-off-by: Yong Cong Sin <yongcong.sin@gmail.com>
Applies the 'unlikely' attribute to various kernel objects that
use z_unpend_first_thread() to optimize for the non-blocking path.
This boosts the thread_metric synchronization benchmark numbers
on the frdm_k64f board by about 10%.
Signed-off-by: Peter Mitsis <peter.mitsis@intel.com>
Namespaced the generated headers with `zephyr` to prevent
potential conflict with other headers.
Introduce a temporary Kconfig `LEGACY_GENERATED_INCLUDE_PATH`
that is enabled by default. This allows the developers to
continue the use of the old include paths for the time being
until it is deprecated and eventually removed. The Kconfig will
generate a build-time warning message, similar to the
`CONFIG_TIMER_RANDOM_GENERATOR`.
Updated the includes path of in-tree sources accordingly.
Most of the changes here are scripted, check the PR for more
info.
Signed-off-by: Yong Cong Sin <ycsin@meta.com>
Add a closing comment to the endif with the configuration
information to which the endif belongs too.
To make the code more clearer if the configs need adaptions.
Signed-off-by: Simon Hein <Shein@baumer.com>
Move the syscall_handler.h header, used internally only to a dedicated
internal folder that should not be used outside of Zephyr.
Signed-off-by: Anas Nashif <anas.nashif@intel.com>
This header does not expose any public APIs, so move it under
kernel/include and change files including it.
Signed-off-by: Anas Nashif <anas.nashif@intel.com>
In order to bring consistency in-tree, migrate all kernel code to the
new prefix <zephyr/...>. Note that the conversion has been scripted,
refer to zephyrproject-rtos#45388 for more details.
Signed-off-by: Gerard Marull-Paretas <gerard.marull@nordicsemi.no>
Remove this intrusive tracing feature in favor of the new object tracing
using the main tracing feature in zephyr. See #33603 for the new tracing
coverage for all objects.
This will allow for support in more tools and less reliance on GDB for
tracing objects.
Signed-off-by: Anas Nashif <anas.nashif@intel.com>
k_queue_append and k_queue_alloc_append are in a race
condition.
Scenario:
Using either of the above mentioned functions from a
preemptive context, can lead to the element being
appended to the queue to be lost. The element loss happens
when the k_queue_append is interrupted at a critical point,
when the list only contains one element and k_queue_get is
called, before the k_queue_append is allowed to complete.
Fix:
Move sys_sflist_peek_tail inside queue_insert(). Add additional
bool paramenter to queue_insert(), to indicate append/prepend
operation.
Fixes#29257
Signed-off-by: iva kik <megatheriumiva@gmail.com>
Now that device_api attribute is unmodified at runtime, as well as all
the other attributes, it is possible to switch all device driver
instance to be constant.
A coccinelle rule is used for this:
@r_const_dev_1
disable optional_qualifier
@
@@
-struct device *
+const struct device *
@r_const_dev_2
disable optional_qualifier
@
@@
-struct device * const
+const struct device *
Fixes#27399
Signed-off-by: Tomasz Bursztyka <tomasz.bursztyka@linux.intel.com>
The k_queue data structure, when CONFIG_POLL was enabled, would
inexplicably use k_poll() as its blocking mechanism instead of the
original wait_q/pend() code. This was actually racy, see commit
b173e4353f. The code was structured as a condition variable: using
a spinlock around the queue data before deciding to block. But unlike
pend_current_thread(), k_poll() cannot atomically release a lock.
A workaround had been in place for this, and then accidentally
reverted (both by me!) because the code looked "wrong".
This is just fragile, there's no reason to have two implementations of
k_queue_get(). Remove.
Note that this also removes a test case in the work_queue test where
(when CONFIG_POLL was enabled, but not otherwise) it was checking for
the ability to immediately cancel a delayed work item that was
submitted with a timeout of K_NO_WAIT (i.e. "queue it immediately").
This DOES NOT work with the origina/non-poll queue backend, and has
never been a documented behavior of k_delayed_work_submit_to_queue()
under any circumstances. I don't know why we were testing this.
Fixes#25904
Signed-off-by: Andy Ross <andrew.j.ross@intel.com>
Add a k_timeout_t type, and use it everywhere that kernel API
functions were accepting a millisecond timeout argument. Instead of
forcing milliseconds everywhere (which are often not integrally
representable as system ticks), do the conversion to ticks at the
point where the timeout is created. This avoids an extra unit
conversion in some application code, and allows us to express the
timeout in units other than milliseconds to achieve greater precision.
The existing K_MSEC() et. al. macros now return initializers for a
k_timeout_t.
The K_NO_WAIT and K_FOREVER constants have now become k_timeout_t
values, which means they cannot be operated on as integers.
Applications which have their own APIs that need to inspect these
vs. user-provided timeouts can now use a K_TIMEOUT_EQ() predicate to
test for equality.
Timer drivers, which receive an integer tick count in ther
z_clock_set_timeout() functions, now use the integer-valued
K_TICKS_FOREVER constant instead of K_FOREVER.
For the initial release, to preserve source compatibility, a
CONFIG_LEGACY_TIMEOUT_API kconfig is provided. When true, the
k_timeout_t will remain a compatible 32 bit value that will work with
any legacy Zephyr application.
Some subsystems present timeout (or timeout-like) values to their own
users as APIs that would re-use the kernel's own constants and
conventions. These will require some minor design work to adapt to
the new scheme (in most cases just using k_timeout_t directly in their
own API), and they have not been changed in this patch, instead
selecting CONFIG_LEGACY_TIMEOUT_API via kconfig. These subsystems
include: CAN Bus, the Microbit display driver, I2S, LoRa modem
drivers, the UART Async API, Video hardware drivers, the console
subsystem, and the network buffer abstraction.
k_sleep() now takes a k_timeout_t argument, with a k_msleep() variant
provided that works identically to the original API.
Most of the changes here are just type/configuration management and
documentation, but there are logic changes in mempool, where a loop
that used a timeout numerically has been reworked using a new
z_timeout_end_calc() predicate. Also in queue.c, a (when POLL was
enabled) a similar loop was needlessly used to try to retry the
k_poll() call after a spurious failure. But k_poll() does not fail
spuriously, so the loop was removed.
Signed-off-by: Andy Ross <andrew.j.ross@intel.com>
Calling z_ready_thread() means the thread is now ready and can wake up
at any moment on another CPU. But we weren't finished setting the
return value! So the other side could wake up with a spurious "error"
condition if it ran too soon. Note that on systems with a working
IPI, that wakeup can happen much faster than you might think.
Signed-off-by: Andy Ross <andrew.j.ross@intel.com>
z_set_thread_return_value is part of the core kernel -> arch
interface and has been renamed to z_arch_thread_return_value_set.
z_set_thread_return_value_with_data renamed to
z_thread_return_value_set_with_data for consistency.
Signed-off-by: Andrew Boie <andrew.p.boie@intel.com>
The semi-automated API changes weren't checkpatch aware. Fix up
whitespace warnings that snuck into the previous patches. Really this
should be squashed, but that's somewhat difficult given the structure
of the series.
Signed-off-by: Andy Ross <andrew.j.ross@intel.com>
System call arguments, at the arch layer, are single words. So
passing wider values requires splitting them into two registers at
call time. This gets even more complicated for values (e.g
k_timeout_t) that may have different sizes depending on configuration.
This patch adds a feature to gen_syscalls.py to detect functions with
wide arguments and automatically generates code to split/unsplit them.
Unfortunately the current scheme of Z_SYSCALL_DECLARE_* macros won't
work with functions like this, because for N arguments (our current
maximum N is 10) there are 2^N possible configurations of argument
widths. So this generates the complete functions for each handler and
wrapper, effectively doing in python what was originally done in the
preprocessor.
Another complexity is that traditional the z_hdlr_*() function for a
system call has taken the raw list of word arguments, which does not
work when some of those arguments must be 64 bit types. So instead of
using a single Z_SYSCALL_HANDLER macro, this splits the job of
z_hdlr_*() into two steps: An automatically-generated unmarshalling
function, z_mrsh_*(), which then calls a user-supplied verification
function z_vrfy_*(). The verification function is typesafe, and is a
simple C function with exactly the same argument and return signature
as the syscall impl function. It is also not responsible for
validating the pointers to the extra parameter array or a wide return
value, that code gets automatically generated.
This commit includes new vrfy/msrh handling for all syscalls invoked
during CI runs. Future commits will port the less testable code.
Signed-off-by: Andy Ross <andrew.j.ross@intel.com>
move misc/sflist.h to sys/sflist.h and
create a shim for backward-compatibility.
No functional changes to the headers.
A warning in the shim can be controlled with CONFIG_COMPAT_INCLUDES.
Related to #16539
Signed-off-by: Anas Nashif <anas.nashif@intel.com>
The first word is used as a pointer, meaning it is 64 bits on 64-bit
systems. To reserve it, it has to be either a pointer, a long, or an
intptr_t. Not an int nor an u32_t.
Signed-off-by: Nicolas Pitre <npitre@baylibre.com>
Given that the section name and boundary simbols can be inferred from
the struct object name, it makes sense to create an iterator that
abstracts away the access details and reduce the possibility for
mistakes.
Signed-off-by: Nicolas Pitre <npitre@baylibre.com>
Update reserved function names starting with one underscore, replacing
them as follows:
'_k_' with 'z_'
'_K_' with 'Z_'
'_handler_' with 'z_handl_'
'_Cstart' with 'z_cstart'
'_Swap' with 'z_swap'
This renaming is done on both global and those static function names
in kernel/include and include/. Other static function names in kernel/
are renamed by removing the leading underscore. Other function names
not starting with any prefix listed above are renamed starting with
a 'z_' or 'Z_' prefix.
Function names starting with two or three leading underscores are not
automatcally renamed since these names will collide with the variants
with two or three leading underscores.
Various generator scripts have also been updated as well as perf,
linker and usb files. These are
drivers/serial/uart_handlers.c
include/linker/kobject-text.ld
kernel/include/syscall_handler.h
scripts/gen_kobject_list.py
scripts/gen_syscall_header.py
Signed-off-by: Patrik Flykt <patrik.flykt@intel.com>
Straightforward port. Each struct k_queue object gets a spinlock to
control obvious data ownership.
Note that this port actually discovered a preexisting bug: the -ENOMEM
case in queue_insert() was failing to release the lock. But because
the tests that hit that path didn't rely on other threads being
scheduled, they ran to successful completion even with interrupts
disabled. The spinlock API detects that as a recursive lock when
asserts are enabled.
Signed-off-by: Andy Ross <andrew.j.ross@intel.com>
Just like with _Swap(), we need two variants of these utilities which
can atomically release a lock and context switch. The naming shifts
(for byte count reasons) to _reschedule/_pend_curr, and both have an
_irqlock variant which takes the traditional locking.
Just refactoring. No logic changes.
Signed-off-by: Andy Ross <andrew.j.ross@intel.com>
These files were using z_thread_malloc() without including
kernel_internal.h. On existing architectures that works due to
transitive includes, but x86_64 has a thinner include layer and
doesn't do it for us. Include the files required for the APIs we use.
Signed-off-by: Andy Ross <andrew.j.ross@intel.com>
