Introduces K_MEM_SLAB_DEFINE_TYPE() and K_MEM_SLAB_DEFINE_STATIC_TYPE()
helpers to allow the user to declare slabs for types without having to
manually ensure the alignment is correct.
Manual slab alignment was very error-prone and this change fixes several
instances of misalignment that would be trapped by the undefined
behavior sanitizer when running on 64-bit targets.
Signed-off-by: Egill Sigurdur <egill@egill.xyz>
Add Kconfig options to control where RTIO SQEs and the RTIO block pool
are allocated: default RAM, DTCM, or a Zephyr nocache region.
This improves robustness and performance for DMA-heavy RTIO users:
- Avoids D-cache coherency pitfalls (especially with tiny DMA buffers).
- Enables using DTCM for lower-latency RTIO queue and pool access.
Signed-off-by: Peter van der Perk <peter.vanderperk@nxp.com>
Set unique names for the RTIO workqueue threads. If
`CONFIG_RTIO_WORKQ_THREADS_POOL` goes over 9, the numbers repeat. This
is not expected to occur in practice since the two defaults for this
config are `1` and `2`, and fully handling it is not worth the code
complexity.
Signed-off-by: Jordan Yates <jordan@embeint.com>
This check was asserting on the wrong struct size and was not taking in
the architecture configured cache line size. The build assert now checks
the right struct size and takes into account the configured cache line
size or sets a default.
Additional makes the check entire opt in.
Signed-off-by: Tom Burdick <thomas.burdick@infineon.com>
OP_DELAY requires embedding in the submission queue entry a _timeout
struct which can push the struct size over the specified 64 byte (cache
line) limit.
Signed-off-by: Tom Burdick <thomas.burdick@infineon.com>
`rtio_executor_handle_multishot` releases the RX buffer in case of
cancellation or error. However, in case of an sqe that uses a mempool,
it reset the buffer before that, making the release a NOOP. Fix that by
moving the reset of the buffer to right before pushing the sqe back onto
the queue.
Signed-off-by: Wouter Horré <wouter@versasense.com>
Callbacks now take a result parameter which may, if the callback was
linkd to by a previous submissions, have the result code from the last
submission.
Additionally the userdata is now const by default making it easier to
use the device pointer as the userdata which is very common.
Signed-off-by: Tom Burdick <thomas.burdick@intel.com>
Allow the rtio executor to handle the RTIO_OP_AWAIT for cases where no
rtio_iodev needs to be blocked while waiting for the signal.
When the RTIO_OP_AWAIT was introduced, the highlighted usecase was
performing transactions within an exact time window. This can be
achieved by blocking the rtio_iodev, effectively locking the bus, while
awaiting.
However, a relaxed implementation which doesn't block any rtio_iodev is
really useful for device drivers that use some kind of ready-event
during transactions. For example in order to read configuration out of
a sensor, the following flow may be required:
1. Write some cmd like "read accelerometer scale"
2. Await data-ready GPIO rising edge (might take "a long time")
3. Read the requested data payload from the sensor
Using a relaxed variant of the RTIO_OP_AWAIT this can be elegantly put
together in a single chained rtio sequence. And by not blocking the
rtio_iodev, other devices can be operated during the await period.
Of course, not blocking the rtio_iodev also means that the bus may be
busy when the RTIO_OP_AWAIT completes and thus, step 3 may be blocked
for a some time and the guarantee of an exact time window cannot be
achieved with the relaxed variant.
To make the API clearer, separate helpers are added for the strict
(blocking iodev) and relaxed (blocking only the sqe chain) variant
respectively.
Signed-off-by: Emil Dahl Juhl <emil@s16s.ai>
Adds a context pool that can be used when a number of threads may be
dynamically created to use with RTIO. While the pool has a small cost to
it, the cost of verifying a statically declared kobject likely makes it
worth avoiding dynamically allocating these with the typical
kobject_alloc method. Instead this arena style pool of objects can be
used where the kobject validation uses the gperf hash created at build
time.
Signed-off-by: Tom Burdick <thomas.burdick@intel.com>
* Remove `RTIO_WORKQ_STACK_SIZE` as it is no longer used.
* Fix description of `RTIO_WORKQ_THREADS_POOL_STACK_SIZE`.
Signed-off-by: Jonas Spinner <jonas.spinner@burkert.com>
As an error-handling strategy. The client then can recover from this
depending on the error and the number of occurrences.
Signed-off-by: Luis Ubieda <luisf@croxel.com>
As their handling is not that similar and mixing them both affects
code readability. This patch does not affect functionality and does
not break any testsuite under rtio API.
Signed-off-by: Luis Ubieda <luisf@croxel.com>
There's no use-case covering this case and enforcing it will allow
simplify the logic for processing items with each of these items when
mutually exclusive.
Signed-off-by: Luis Ubieda <luisf@croxel.com>
MISRA-C says the right hand of a logical and should not contain side
effect operations. Instead nest the branches to get the same logic
without the MISRA-C rule breakage.
Signed-off-by: Tom Burdick <thomas.burdick@intel.com>
Based on a discussion around P4WQ limitations for our application, it
was determined that the RTIO workqueue required the ability to use
additional threads from the pool in spite of work items blocked.
Since this is not covered, nor desired for the P4WQ, then remove this
dependency and re-implement it in a way that covers also this use-case.
Signed-off-by: Luis Ubieda <luisf@croxel.com>
SQE items with this OP will take the specified amount of time
(asynchronously) before completing. This allows to serve as an
asynchronous delay in between SQE items (e.g: A sensor measurement
requested, which requires 50-ms before having the result available).
Signed-off-by: Luis Ubieda <luisf@croxel.com>
Add an assert ensuring programming errors where the executor code
assumes there is a next node in the list for transaction/chained
submissions.
Signed-off-by: Tom Burdick <thomas.burdick@intel.com>
As to facilitate work items preemption, common for sensor read ops.
e.g:
- Sensor read (submitted through workqueue).
- SPI read (submitted through workqueue using default handler).
This requires at least 2 thread pools to allow SPI read to complete,
to then come back and complete sensor reading.
Signed-off-by: Luis Ubieda <luisf@croxel.com>
Otherwise the RTIO Workqueue is not functional for devices during init.
An example of this issue is devices using SPI transfers with default
spi_rtio.
Signed-off-by: Luis Ubieda <luisf@croxel.com>
This commit fixes the bug that the memory of the work request is
freed up in the work handler, before it is not needed anymore by the p4wq.
This is fixed now, by using the new done_handler in the p4wq
for freeing up that memory.
Signed-off-by: Florian Weber <Florian.Weber@live.de>
Transmit/write buffers are expected to be constant values given to the
operation to transmit but not mutate. Seperate the OP_TX and OP_RX
operation description unions.
Signed-off-by: Tom Burdick <thomas.burdick@intel.com>
The default behavior for thread pending of completions should use
semaphores rather than yield/wait looping when multithreading is
available.
Disable by default only when multithreading isn't available.
Signed-off-by: Tom Burdick <thomas.burdick@intel.com>
As the items come from the memory slab, their initialization state
can't be guaranteed. This is causing some devices triggering an assert
in p4wq where the item's thread is not null (not zero).
Signed-off-by: Luis Ubieda <luisf@croxel.com>
Adds ability to process synchronous requests in an asynchronous
fashion, relying on dedicated P4WQ's and pre-allocated work items.
Signed-off-by: Luis Ubieda <luisf@croxel.com>
Moves the rtio_ prefixed lockfree queues to sys alongside existing
mpsc/spsc pbuf, ringbuf, and similar queue-like data structures.
Signed-off-by: Tom Burdick <thomas.burdick@intel.com>
Callbacks were a bit neglected in terms of test coverage, especially
when used in chains. It was clear from the code that chained callbacks
may not actually work, and callback ordering then was hard to verify.
Test callbacks chained to transactions work as expected.
The test iodev had built up some cruft over time and in the process
showed a few bugs once callback chaining was fixed so the test iodev now
better matches typical iodev implementations at this point.
Cancellation testing now includes an added case for cancelling a the
second submission in the chain prior to calling submit noting that no
completions notifications should be given back for those.
Signed-off-by: Tom Burdick <thomas.burdick@intel.com>
Namespaced the generated headers with `zephyr` to prevent
potential conflict with other headers.
Introduce a temporary Kconfig `LEGACY_GENERATED_INCLUDE_PATH`
that is enabled by default. This allows the developers to
continue the use of the old include paths for the time being
until it is deprecated and eventually removed. The Kconfig will
generate a build-time warning message, similar to the
`CONFIG_TIMER_RANDOM_GENERATOR`.
Updated the includes path of in-tree sources accordingly.
Most of the changes here are scripted, check the PR for more
info.
Signed-off-by: Yong Cong Sin <ycsin@meta.com>
Move the syscall_handler.h header, used internally only to a dedicated
internal folder that should not be used outside of Zephyr.
Signed-off-by: Anas Nashif <anas.nashif@intel.com>
This adds a few line use zephyr_syscall_header() to include
headers containing syscall function prototypes.
Signed-off-by: Daniel Leung <daniel.leung@intel.com>
Until now iterable sections APIs have been part of the toolchain
(common) headers. They are not strictly related to a toolchain, they
just rely on linker providing support for sections. Most files relied on
indirect includes to access the API, now, it is included as needed.
Signed-off-by: Gerard Marull-Paretas <gerard.marull@nordicsemi.no>
- Add a new API `rtio_sqe_cancel` to attempt canceling a queued SQE
- Add a new syscall `rtio_sqe_copy_in_get_handles` which allows getting
back the SQE handles generated by the copy_in operation so that they
can be canceled.
Signed-off-by: Yuval Peress <peress@google.com>
Reworks the zephyr macros and pools to be objects in their own right. Each
pool can be statically defined with a Z_ private macro. The objects can
then be initialized with an rtio instance statically.
This cleans up a lot of code that was otherwise doing little bits of
management around allocation/freeing and reduces the scope those functions
has to the data it needs.
This should enable sharing the pools of sqe, cqe, and mem blocks among rtio
instances in a future improvement easily.
Signed-off-by: Tom Burdick <thomas.burdick@intel.com>
Rather than the rings, which weren't shared between userspace and kernel
space in Zephyr like they are in Linux with io_uring, use atomic mpsc
queues for submission and completion queues.
Most importantly this removes a potential head of line blocker in the
submission queue as the sqe would be held until a task is completed.
As additional bonuses this avoids some additional locks and restrictions
about what can be submitted and where. It also removes the need for
two executors as all chains/transactions are done concurrently.
Lastly this opens up the possibility for a common pool of sqe's to
allocate from potentially saving lots of memory.
Signed-off-by: Tom Burdick <thomas.burdick@intel.com>
It was previously assumed that the 'sys_mem_blocks' struct would maintain
information about contiguous blocks allocated so the release API only
took the starting address. This led to an issue where allocating 2+
blocks would end up with a memory leak because any block not being the
first would never be released.
Add the buffer length as an argument so the correct number of blocks can
be released. Also, ammend the tests to match and verify.
Signed-off-by: Yuval Peress <peress@google.com>
- Introduce a new Kconfig to enable mempool in RTIO
- Introduce a new RTIO_DEFINE_WITH_MEMPOOL to allocate an RTIO context
with an associated memory pool.
- Add a new sqe read function rtio_sqe_read_with_pool() for memory pool
enabled RTIO contexts
- Allow IODevs to allocate only the memory they need via rtio_sqe_rx_buf()
- Allow the consumer to get the allocated buffer via
rtio_cqe_get_mempool_buffer()
- Consumers need to release the buffer via rtio_release_buffer() when
processing is complete.
Signed-off-by: Yuval Peress <peress@google.com>
Update the policy such that every completed sqe has a parallel cqe.
This has the primary purpose of making any reads in the sqe visible
to the consumer (since they might have different buffers).
Signed-off-by: Yuval Peress <peress@google.com>
Adds the transceive op which is needed for full SPI support. Notably
in RTIO transceive is expected to be a balanced buffer pair of the same
length and have non-null tx/rx bufs.
Signed-off-by: Tom Burdick <thomas.burdick@intel.com>
Adds a callback op to RTIO enabling C logic to be interspersed with
I/O operations. This is not safe from userspace but allowable in kernel
space.
Signed-off-by: Tom Burdick <thomas.burdick@intel.com>
When sending small buffers out it makes sense to copy rather than
reference to avoid having to keep the small buffer around for the
lifetime of the write request.
Adjusts the op numbers to always be +1 from the previously defined op
id making it easier to re-arrange if needed in the future.
Signed-off-by: Tom Burdick <thomas.burdick@intel.com>
The pending_sqe logic to track where in the ring queue the concurrent
executor had left off was slightly flawed. It didn't account for starting
all sqes in the queue and ending back up at the beginning.
Instead track the last SQE in the queue, from which the next one in the
queue will the one to start next.
If we happen to sweep the last known SQE in the queue, reset it to NULL
so the next time prepare is called we start at the beginning of the queue
again.
Signed-off-by: Tom Burdick <thomas.burdick@intel.com>
Transactional submissions treat a sequence of sqes as a single atomic
submission given to a single iodev and expect as a reply a single
completion.
This is useful for scatter gather like APIs that exist in Zephyr already
for I2C and SPI.
Signed-off-by: Tom Burdick <thomas.burdick@intel.com>
