The compiler cannot prove the payload copy leaves the queue fields
alone, so it reloaded the ring pointers after it. Pick the slot and
advance the pointer first.
thread_metric message processing, base / reordered / delta:
x86 133267 / 136881 / +2.71%
Xtensa DC233C 84595 / 87528 / +3.47%
RISC-V 32 80998 / 84130 / +3.87%
ARM Cortex-A53 497986 / 523815 / +5.19%
ARM Cortex-M3 96465 / 103732 / +7.53%
ARM Cortex-M4 410042 / 442637 / +7.95%
The first five are QEMU under icount; the Cortex-M4 figure is an MXChip
AZ3166 at 96 MHz. app_kernel improves 4.5% to 5.5% on one and four byte
messages and 0.5% at 192 bytes, where the copy dominates. Text shrinks
by 20 bytes and there is no impact on RAM.
Assisted-by: Claude:claude-opus-5
Signed-off-by: Benjamin Cabé <benjamin@zephyrproject.org>
k_msgq_get() falls straight through to the empty case and leaves the
transfer behind a taken branch, though the transfer is both the common
case and the only one that copies anything.
A transfer costs five cycles less and a get on an empty queue two more,
on a path that returns without copying and that a caller hitting it
routinely would normally be blocking on rather than polling anyway.
thread_metric message processing, base / likely() / delta:
x86 133267 / 133267 / +0.00%
Xtensa DC233C 84595 / 84595 / +0.00%
ARM Cortex-M3 96465 / 97083 / +0.64%
ARM Cortex-A53 497986 / 501946 / +0.80%
RISC-V 32 80998 / 81839 / +1.04%
ARM Cortex-M4 410042 / 418985 / +2.18%
The first five are QEMU under icount; the Cortex-M4 figure is an MXChip
AZ3166 at 96 MHz, where flash wait states make a taken branch dearer.
Text shrinks by 4 bytes and there is no impact on RAM.
Assisted-by: Claude:claude-opus-5
Signed-off-by: Benjamin Cabé <benjamin@zephyrproject.org>
Splits z_waitq_head() into two versions: z_waitq_head_locked() and
z_waitq_head(). When the scheduler's spinlock is known to be already
held, z_waitq_head_locked() should be used--otherwise, z_waitq_head()
is to be used.
However, this approach uncovered a path where the scheduler spinlock
could be recursively taken when a thread is aborted. To work around the
recursion (see k_thread_perms_all_clear), knowledge of the scheduler's
spinlock state must be passed to the lower layers for use in the
cleanup routines for message queues, stacks and timers.
Fixes#115756
Signed-off-by: Peter Mitsis <peter.mitsis@intel.com>
Updates the k_stack_cleanup() and k_msgq_cleanup() routines
to lock the appropriate lock to prevent two execution contexts
from trying to clean up the same object at the same time.
Signed-off-by: Peter Mitsis <peter.mitsis@intel.com>
'exit' is reserved by the C standard for any use, so using it as a
label violates MISRA C:2012 Rule 21.2. Rename the labels in the
message queue and pipe implementations to 'out', which is the spelling
already used elsewhere in the kernel.
Pure rename, no functional change.
Signed-off-by: Anas Nashif <anas.nashif@intel.com>
Each kernel object type that participates in the object core framework
previously supplied its own SYS_INIT routine to initialize its
k_obj_type and to walk its static-object linker section, linking each
object core. These routines were near-identical across 11 object types
and differed only in the type id, the object struct and the obj_core
offset.
Replace that per-type boilerplate with a declarative K_OBJ_TYPE_DEFINE()
macro that emits a const descriptor into a new iterable ROM section, and
walk those descriptors once from a single SYS_INIT in obj_core.c. The
descriptor captures the type storage, type id, obj_core offset, the
static object section bounds and the object stride, which is all the
shared loop needs to initialize and link every statically defined
object.
Converted: condvar, event, fifo, lifo, mailbox, msgq, mutex, pipe, sem,
stack and timer. The non-uniform initializers (thread, mem_slab and the
internal cpu/kernel objects) are left unchanged and will be dealt with
in followup commits.
Footprint on qemu_cortex_m3 (tests/kernel/obj_core/obj_core, all types
enabled): flash 32012 -> 30880 (-1132 B), RAM unchanged. With
CONFIG_OBJ_CORE disabled the image is byte-for-byte identical.
Assisted-by: Claude:claude-opus-4-8
Signed-off-by: Anas Nashif <anas.nashif@intel.com>
Some private symbol names were widely duplicated throughout the kernel
such as lock and handle_poll_event. This arguably made the kernel less
readable as the locality of the name lock is highly confusing.
Rename all compilation unit locks to match their usage (e.g.
mutex_lock). Improving readability.
Furthermore, by deduplicating these symbols we enable potential
amalgamation builds of the kernel where all C files are merged
into one large C file or compliation unit allowing for better
compiler visibility and optimization.
Signed-off-by: Tom Burdick <thomas.burdick@infineon.com>
Replace z_unpend_first_thread() with z_unpend_first_thread_locked() and
migrate every caller across the kernel. The old function dropped the
scheduler spinlock before returning, exposing a race window between
its caller's "arch_thread_return_value_set + z_ready_thread" pair and
a still-in-flight timeout handler that could ready the thread first --
the woken thread might then run on another CPU and see an uninitialized
swap_retval. Pre-1b8c7a3 the dticks-cancel check made the handler bail;
here we fix it cleanly by requiring the caller to hold _sched_spinlock
across the entire wake, so the handler is blocked for the duration and
runs as a no-op afterwards.
z_unpend_first_thread_locked() requires the caller to be inside a
locked region and must be paired with z_sched_ready_locked() (and
whatever return-value setup is needed) under the same lock acquisition.
Sites migrated:
Simple "set retval [+ swap_data] and ready" callers use the existing
z_sched_wake() convenience wrapper, refactored to use the new
helper internally:
sem (give, reset), mem_slab (free), stack (push),
condvar (signal, broadcast), msgq (purge),
queue (cancel_wait, queue_insert, append_list),
futex (wake).
Sites that need additional setup on the woken thread use
LOCK_SCHED_SPINLOCK + z_unpend_first_thread_locked() + custom wake:
mutex (unlock -- needs the thread reference to track new owner),
msgq put / get (needs memcpy into the receiver's swap_data
buffer before the return value is set).
The dticks-cancel check in z_thread_timeout() is left in place; it is
no longer load-bearing once z_abort_thread_timeout() has no callers,
and is removed in the next commit.
Signed-off-by: Nicolas Pitre <npitre@baylibre.com>
kernel.h implies kernel_structs.h via kernel_includes.h, making
explicit inclusion of kernel_structs.h unnecessary whenever kernel.h
is already included in the same translation unit.
Remove the redundant includes across arch, boards, drivers, kernel,
lib, samples, subsys, and tests trees.
in include/zephyr/kernel_structs.h:
* 2. kernel.h shall imply kernel_structs.h, such that it shall not be
* necessary to include kernel_structs.h explicitly when kernel.h is
* included.
Signed-off-by: Anas Nashif <anas.nashif@intel.com>
This partially reverts commit 9cef0da05c ("kernel: avoid recursive
scheduler lock in k_heap_free path"), keeping only the sched.c changes
(z_unpend_all_locked / z_unpend_all refactoring).
The _sched_locked variants of k_free, k_heap_free, k_msgq_cleanup,
k_stack_cleanup and the sched_locked parameter plumbing through
unref_check were an overcomplicated approach. A simpler fix follows.
Signed-off-by: Nicolas Pitre <npitre@baylibre.com>
When halt_thread() calls k_thread_perms_all_clear() under
_sched_spinlock, the permission cleanup can trigger k_free() on
dynamic objects. k_heap_free() then calls z_unpend_all() which
attempts to take _sched_spinlock again, causing a recursive lock.
Fix this by introducing k_heap_free_sched_locked() and
k_free_sched_locked() variants that use z_unpend_all_locked()
to operate on the wait queue without re-acquiring the scheduler
lock. The existing z_unpend_all() becomes a wrapper that takes
the lock and delegates to z_unpend_all_locked().
unref_check() gains a sched_locked parameter: the abort path
(clear_perms_cb) passes true to use the locked free variant,
while k_thread_perms_clear() passes false for the normal path.
Fixes#106659
Signed-off-by: Nicolas Pitre <npitre@baylibre.com>
Not only checks if writer_ptr is smaller than buffer_end but also
checks that write_ptr + msg_size is smaller than buffer_end to
avoid overflow when copying data.
Signed-off-by: Flavio Ceolin <flavio@hubblenetwork.com>
The message queue 'buffer_end' field points to the next address AFTER
the end of the buffer. When the buffer goes to the last addressable
byte, the next byte is 0x0. To ensure proper evaluation of the bounds
the __ASSERT_NO_MSG() checks must not use "< buffer_end", but
"<= buffer_end - 1".
Signed-off-by: Peter Mitsis <peter.mitsis@intel.com>
When the buffer is full, Thread A gets pended (blocked).
If Thread B later calls the get function, it will unpend Thread A,
allowing it to resume and put the message into the queue.
In this situation, we need to know whether Thread A should
continue with put to front or put to end.
In order to resolve this issue, we don't allow set timeout
parameter for `k_msgq_put_front` and this parameter is always
`K_NO_WAIT`.
Signed-off-by: TaiJu Wu <tjwu1217@gmail.com>
This commit adds the tracing macros and functions related
specifically to the k_msgq_put_front API.
Signed-off-by: Alexander Paschoaletto <axelpinheiro@gmail.com>
This commit introduces the k_msgq_put_front API for sending
messages to a queue in a LIFO scheme.
Signed-off-by: Alexander Paschoaletto <axelpinheiro@gmail.com>
1. Fixes a performance issue in k_msgq_put() to allow for a fast return
path when handling a poll event does nothing.
2. Allows for a fast return path in k_msgq_purge() when no threads were
awakened.
Signed-off-by: Peter Mitsis <peter.mitsis@intel.com>
Propagates the return value from z_handle_obj_poll_events()
within the message queue, pipes, queue and semaphore objects.
This allows the kernel object code to determine whether it
needs to perform a full reschedule, or if it can perform a
more optimized exit strategy.
Signed-off-by: Peter Mitsis <peter.mitsis@intel.com>
Mostly a revert of commit b1def7145f ("arch: deprecate `_current`").
This commit was part of PR #80716 whose initial purpose was about providing
an architecture specific optimization for _current. The actual deprecation
was sneaked in later on without proper discussion.
The Zephyr core always used _current before and that was fine. It is quite
prevalent as well and the alternative is proving rather verbose.
Furthermore, as a concept, the "current thread" is not something that is
necessarily architecture specific. Therefore the primary abstraction
should not carry the arch_ prefix.
Hence this revert.
Signed-off-by: Nicolas Pitre <npitre@baylibre.com>
`_current` is now functionally equals to `arch_curr_thread()`, remove
its usage in-tree and deprecate it instead of removing it outright,
as it has been with us since forever.
Signed-off-by: Yong Cong Sin <ycsin@meta.com>
Signed-off-by: Yong Cong Sin <yongcong.sin@gmail.com>
Applies the 'unlikely' attribute to various kernel objects that
use z_unpend_first_thread() to optimize for the non-blocking path.
This boosts the thread_metric synchronization benchmark numbers
on the frdm_k64f board by about 10%.
Signed-off-by: Peter Mitsis <peter.mitsis@intel.com>
Namespaced the generated headers with `zephyr` to prevent
potential conflict with other headers.
Introduce a temporary Kconfig `LEGACY_GENERATED_INCLUDE_PATH`
that is enabled by default. This allows the developers to
continue the use of the old include paths for the time being
until it is deprecated and eventually removed. The Kconfig will
generate a build-time warning message, similar to the
`CONFIG_TIMER_RANDOM_GENERATOR`.
Updated the includes path of in-tree sources accordingly.
Most of the changes here are scripted, check the PR for more
info.
Signed-off-by: Yong Cong Sin <ycsin@meta.com>
Add a closing comment to the endif with the configuration
information to which the endif belongs too.
To make the code more clearer if the configs need adaptions.
Signed-off-by: Simon Hein <Shein@baumer.com>
Move the syscall_handler.h header, used internally only to a dedicated
internal folder that should not be used outside of Zephyr.
Signed-off-by: Anas Nashif <anas.nashif@intel.com>
This header does not expose any public APIs, so move it under
kernel/include and change files including it.
Signed-off-by: Anas Nashif <anas.nashif@intel.com>
Without these parentheses, specifying a q_max_msgs of e.g.
`MY_DEFAULT_QUEUESIZE+1` would result in a buffer of size
(1 element + MY_DEFAULT_QUEUESIZE bytes).
This would then lead to an unbounded buffer overflow because the queue
never reaches the exact (offset by MY_DEFAULT_QUEUESIZE bytes)
`buffer_end` and just keeps writing.
Additionally, add asserts to make sure this can't happen again.
Signed-off-by: Armin Brauns <armin.brauns@embedded-solutions.at>
In order to bring consistency in-tree, migrate all kernel code to the
new prefix <zephyr/...>. Note that the conversion has been scripted,
refer to zephyrproject-rtos#45388 for more details.
Signed-off-by: Gerard Marull-Paretas <gerard.marull@nordicsemi.no>
Remove this intrusive tracing feature in favor of the new object tracing
using the main tracing feature in zephyr. See #33603 for the new tracing
coverage for all objects.
This will allow for support in more tools and less reliance on GDB for
tracing objects.
Signed-off-by: Anas Nashif <anas.nashif@intel.com>
This commit adds the ability to use a message queue as a
k_poll object. It follows the same pattern as polling on
FIFOs.
This change has been proven in practice at Samsara.
Fixes: #26728
Signed-off-by: Nick Graves <nicholas.graves@samsara.com>
Add a 'U' suffix to values when computing and comparing against
unsigned variables and other related fixes of the same MISRA rule (10.4)
Signed-off-by: Anas Nashif <anas.nashif@intel.com>
Amend Doxygen documentation for k_msgq_put to note that the message
content will not be modified as a result of the function call and
constify data parameter in function prototype.
Fixes#22301
Signed-off-by: Lauren Murphy <lauren.murphy@intel.com>
Now that device_api attribute is unmodified at runtime, as well as all
the other attributes, it is possible to switch all device driver
instance to be constant.
A coccinelle rule is used for this:
@r_const_dev_1
disable optional_qualifier
@
@@
-struct device *
+const struct device *
@r_const_dev_2
disable optional_qualifier
@
@@
-struct device * const
+const struct device *
Fixes#27399
Signed-off-by: Tomasz Bursztyka <tomasz.bursztyka@linux.intel.com>
Add a k_timeout_t type, and use it everywhere that kernel API
functions were accepting a millisecond timeout argument. Instead of
forcing milliseconds everywhere (which are often not integrally
representable as system ticks), do the conversion to ticks at the
point where the timeout is created. This avoids an extra unit
conversion in some application code, and allows us to express the
timeout in units other than milliseconds to achieve greater precision.
The existing K_MSEC() et. al. macros now return initializers for a
k_timeout_t.
The K_NO_WAIT and K_FOREVER constants have now become k_timeout_t
values, which means they cannot be operated on as integers.
Applications which have their own APIs that need to inspect these
vs. user-provided timeouts can now use a K_TIMEOUT_EQ() predicate to
test for equality.
Timer drivers, which receive an integer tick count in ther
z_clock_set_timeout() functions, now use the integer-valued
K_TICKS_FOREVER constant instead of K_FOREVER.
For the initial release, to preserve source compatibility, a
CONFIG_LEGACY_TIMEOUT_API kconfig is provided. When true, the
k_timeout_t will remain a compatible 32 bit value that will work with
any legacy Zephyr application.
Some subsystems present timeout (or timeout-like) values to their own
users as APIs that would re-use the kernel's own constants and
conventions. These will require some minor design work to adapt to
the new scheme (in most cases just using k_timeout_t directly in their
own API), and they have not been changed in this patch, instead
selecting CONFIG_LEGACY_TIMEOUT_API via kconfig. These subsystems
include: CAN Bus, the Microbit display driver, I2S, LoRa modem
drivers, the UART Async API, Video hardware drivers, the console
subsystem, and the network buffer abstraction.
k_sleep() now takes a k_timeout_t argument, with a k_msleep() variant
provided that works identically to the original API.
Most of the changes here are just type/configuration management and
documentation, but there are logic changes in mempool, where a loop
that used a timeout numerically has been reworked using a new
z_timeout_end_calc() predicate. Also in queue.c, a (when POLL was
enabled) a similar loop was needlessly used to try to retry the
k_poll() call after a spurious failure. But k_poll() does not fail
spuriously, so the loop was removed.
Signed-off-by: Andy Ross <andrew.j.ross@intel.com>
Add runtime error handling for k_msgq_cleanup. We return 0 on success
now and -EAGAIN when cleanup is not possible.
Signed-off-by: Anas Nashif <anas.nashif@intel.com>
Promote the private z_arch_* namespace, which specifies
the interface between the core kernel and the
architecture code, to a new top-level namespace named
arch_*.
This allows our documentation generation to create
online documentation for this set of interfaces,
and this set of interfaces is worth treating in a
more formal way anyway.
Signed-off-by: Andrew Boie <andrew.p.boie@intel.com>
This commit refactors kernel and arch headers to establish a boundary
between private and public interface headers.
The refactoring strategy used in this commit is detailed in the issue
This commit introduces the following major changes:
1. Establish a clear boundary between private and public headers by
removing "kernel/include" and "arch/*/include" from the global
include paths. Ideally, only kernel/ and arch/*/ source files should
reference the headers in these directories. If these headers must be
used by a component, these include paths shall be manually added to
the CMakeLists.txt file of the component. This is intended to
discourage applications from including private kernel and arch
headers either knowingly and unknowingly.
- kernel/include/ (PRIVATE)
This directory contains the private headers that provide private
kernel definitions which should not be visible outside the kernel
and arch source code. All public kernel definitions must be added
to an appropriate header located under include/.
- arch/*/include/ (PRIVATE)
This directory contains the private headers that provide private
architecture-specific definitions which should not be visible
outside the arch and kernel source code. All public architecture-
specific definitions must be added to an appropriate header located
under include/arch/*/.
- include/ AND include/sys/ (PUBLIC)
This directory contains the public headers that provide public
kernel definitions which can be referenced by both kernel and
application code.
- include/arch/*/ (PUBLIC)
This directory contains the public headers that provide public
architecture-specific definitions which can be referenced by both
kernel and application code.
2. Split arch_interface.h into "kernel-to-arch interface" and "public
arch interface" divisions.
- kernel/include/kernel_arch_interface.h
* provides private "kernel-to-arch interface" definition.
* includes arch/*/include/kernel_arch_func.h to ensure that the
interface function implementations are always available.
* includes sys/arch_interface.h so that public arch interface
definitions are automatically included when including this file.
- arch/*/include/kernel_arch_func.h
* provides architecture-specific "kernel-to-arch interface"
implementation.
* only the functions that will be used in kernel and arch source
files are defined here.
- include/sys/arch_interface.h
* provides "public arch interface" definition.
* includes include/arch/arch_inlines.h to ensure that the
architecture-specific public inline interface function
implementations are always available.
- include/arch/arch_inlines.h
* includes architecture-specific arch_inlines.h in
include/arch/*/arch_inline.h.
- include/arch/*/arch_inline.h
* provides architecture-specific "public arch interface" inline
function implementation.
* supersedes include/sys/arch_inline.h.
3. Refactor kernel and the existing architecture implementations.
- Remove circular dependency of kernel and arch headers. The
following general rules should be observed:
* Never include any private headers from public headers
* Never include kernel_internal.h in kernel_arch_data.h
* Always include kernel_arch_data.h from kernel_arch_func.h
* Never include kernel.h from kernel_struct.h either directly or
indirectly. Only add the kernel structures that must be referenced
from public arch headers in this file.
- Relocate syscall_handler.h to include/ so it can be used in the
public code. This is necessary because many user-mode public codes
reference the functions defined in this header.
- Relocate kernel_arch_thread.h to include/arch/*/thread.h. This is
necessary to provide architecture-specific thread definition for
'struct k_thread' in kernel.h.
- Remove any private header dependencies from public headers using
the following methods:
* If dependency is not required, simply omit
* If dependency is required,
- Relocate a portion of the required dependencies from the
private header to an appropriate public header OR
- Relocate the required private header to make it public.
This commit supersedes #20047, addresses #19666, and fixes#3056.
Signed-off-by: Stephanos Ioannidis <root@stephanos.io>
