Namespaced the generated headers with `zephyr` to prevent
potential conflict with other headers.
Introduce a temporary Kconfig `LEGACY_GENERATED_INCLUDE_PATH`
that is enabled by default. This allows the developers to
continue the use of the old include paths for the time being
until it is deprecated and eventually removed. The Kconfig will
generate a build-time warning message, similar to the
`CONFIG_TIMER_RANDOM_GENERATOR`.
Updated the includes path of in-tree sources accordingly.
Most of the changes here are scripted, check the PR for more
info.
Signed-off-by: Yong Cong Sin <ycsin@meta.com>
Found issues when using ## to concatenate tokens in macros where the
resulting symbolic names were including the ## tokens themselves. Using
CONCAT everywhere fixes the issue.
Signed-off-by: Tom Burdick <thomas.burdick@intel.com>
Based on information from doc/develop/api/overview.rst, add current
version for some groups representing APIs, following the following
table:
- Experimental: 0.1.0
- Unstable: 0.8.0
- Stable: 1.0.0
Also based on doc/develop/api/overview.rst, add 'since' tag to the
groups.
Signed-off-by: Ederson de Souza <ederson.desouza@intel.com>
Signed-off-by: Anas Nashif <anas.nashif@intel.com>
In order to serialize all of the i2c functionality an added op code for
configuring the i2c controller was needed. Thankfully very simple.
Signed-off-by: Tom Burdick <thomas.burdick@intel.com>
When compiling with CONFIG_RTIO_SYS_MEM_BLOCKS=n, rtio.h causes unused
argument warnings. Align the implementation of rtio_block_pool_free with
rtio_block_pool_alloc and add ARG_UNUSED() statements for the arguments.
Signed-off-by: Marco Widmer <marco.widmer@bytesatwork.ch>
Coverity reported (rightfully) a possible divide by zero when trying to
allocate from a pool. Return -ENOMEM in such scenarios as it means there
is no memory to allocate from.
Signed-off-by: Tom Burdick <thomas.burdick@intel.com>
The current blocking model will try calling `rtio_mpsc_pop` twice in a
row without any delay. Probably not the actual intent.
Signed-off-by: Yuval Peress <peress@google.com>
Rearranges the sys_mem_blocks fields so that information that describes
how much of the memory block is used is co-located. This will allow
easier of its statistics into the object core statistics reporting
framework.
Signed-off-by: Peter Mitsis <peter.mitsis@intel.com>
Rename shadow variables found by -Wshadow, as the ones in
the macros have a tendency to conflict with outer scope ones
when used. So they need to be more descriptive.
Signed-off-by: Daniel Leung <daniel.leung@intel.com>
atomic_t is a machine word signed integer but cq_count is meant to be an
unsigned positive only count of total completions. Cast when needed to
ensure the correct math and comparisons are being done.
Signed-off-by: Tom Burdick <thomas.burdick@intel.com>
The RTIO API docs weren't being shown. I made a poor assumption thinking
I could simply include the top level doxygen group in index.rst which isn't
the case. Fix the groups, and remove a note about a poll function
that doesn't exist yet.
Signed-off-by: Tom Burdick <thomas.burdick@intel.com>
Rather than looking at the pool of completions for spinning use an
atomic counter of total completions ever done. The relative number of
completions being waited on by rtio_submit may then always be correctly
done.
Prior to this a race was possible, and understood, as
rtio_cqe_consumable was a likely but not guaranteed count of completions.
Sure enough on an SMP system the likely count was ahead of the actual
available completions and a race was caught by the simple test case.
Signed-off-by: Tom Burdick <thomas.burdick@intel.com>
Until now iterable sections APIs have been part of the toolchain
(common) headers. They are not strictly related to a toolchain, they
just rely on linker providing support for sections. Most files relied on
indirect includes to access the API, now, it is included as needed.
Signed-off-by: Gerard Marull-Paretas <gerard.marull@nordicsemi.no>
Flags are needed for i2c support to enable arbitrary stop and restart
setups where in a single i2c transaction the caller would like to direct
the hardware to either send a stop or not after any particular read/write.
These can be reused more generally to direct devices to use different
methods of performing the operations on an operation by operation basis
such as perhaps a hint to use a polling, interrupt, or dma read/write.
Signed-off-by: Tom Burdick <thomas.burdick@intel.com>
- Add a new API `rtio_sqe_cancel` to attempt canceling a queued SQE
- Add a new syscall `rtio_sqe_copy_in_get_handles` which allows getting
back the SQE handles generated by the copy_in operation so that they
can be canceled.
Signed-off-by: Yuval Peress <peress@google.com>
Reworks the zephyr macros and pools to be objects in their own right. Each
pool can be statically defined with a Z_ private macro. The objects can
then be initialized with an rtio instance statically.
This cleans up a lot of code that was otherwise doing little bits of
management around allocation/freeing and reduces the scope those functions
has to the data it needs.
This should enable sharing the pools of sqe, cqe, and mem blocks among rtio
instances in a future improvement easily.
Signed-off-by: Tom Burdick <thomas.burdick@intel.com>
On non-smp systems where multiple cores aren't in play atomics aren't
really necessary and volatile can be used in stead.
Additionally marks the push function as ALWAYS_INLINE as I saw at times
it was not being inlined.
MPSC operation speed is crucial to the performance of rtio, these changes
provided a 30% throughput improvmement in the throughput test.
Signed-off-by: Tom Burdick <thomas.burdick@intel.com>
Rather than the rings, which weren't shared between userspace and kernel
space in Zephyr like they are in Linux with io_uring, use atomic mpsc
queues for submission and completion queues.
Most importantly this removes a potential head of line blocker in the
submission queue as the sqe would be held until a task is completed.
As additional bonuses this avoids some additional locks and restrictions
about what can be submitted and where. It also removes the need for
two executors as all chains/transactions are done concurrently.
Lastly this opens up the possibility for a common pool of sqe's to
allocate from potentially saving lots of memory.
Signed-off-by: Tom Burdick <thomas.burdick@intel.com>
Add the missing app_memdomain.h header needed if userspace is enabled.
This snuck in because the file that was using it already had included
the same header.
Signed-off-by: Yuval Peress <peress@google.com>
It was previously assumed that the 'sys_mem_blocks' struct would maintain
information about contiguous blocks allocated so the release API only
took the starting address. This led to an issue where allocating 2+
blocks would end up with a memory leak because any block not being the
first would never be released.
Add the buffer length as an argument so the correct number of blocks can
be released. Also, ammend the tests to match and verify.
Signed-off-by: Yuval Peress <peress@google.com>
The previous RTIO mempool allocation was found to be faulty. Since the
map entries were indexed using the sqe, as soon as the sqe and cqe were
no longer in sync, there was a contention. Effectively a new sqe needing
memory would assume the entry was already consumed by themselves. This
was caused by 'rtio_sqe_rx_buf()' being effectively reentrant (i.e. if
called more than once, it'll return the same buffer). So when the map
entry was marked as allocated it was assumed that it belonged to the sqe
in question and used. This led to the producer writing to memory that
was actively being read by the consumer.
Instead, the sqe's own buf/buf_len fields are used to store the memory.
Once allocated, the memory is stored there and no mapping is needed.
Signed-off-by: Yuval Peress <peress@google.com>
- Fix initialization order (C++ requires struct fields to be
initialized in the same order they were declared).
- Fix casting issue in 'for' loop conditions
- Fix casting issues when getting atomic values
Signed-off-by: Yuval Peress <peress@google.com>
Commit 53da110dbf deprecated
ceiling_fraction in favor of DIV_ROUND_UP. Apply this to the rtio
header as well.
Signed-off-by: Carles Cufi <carles.cufi@nordicsemi.no>
- Introduce a new Kconfig to enable mempool in RTIO
- Introduce a new RTIO_DEFINE_WITH_MEMPOOL to allocate an RTIO context
with an associated memory pool.
- Add a new sqe read function rtio_sqe_read_with_pool() for memory pool
enabled RTIO contexts
- Allow IODevs to allocate only the memory they need via rtio_sqe_rx_buf()
- Allow the consumer to get the allocated buffer via
rtio_cqe_get_mempool_buffer()
- Consumers need to release the buffer via rtio_release_buffer() when
processing is complete.
Signed-off-by: Yuval Peress <peress@google.com>
Adds the transceive op which is needed for full SPI support. Notably
in RTIO transceive is expected to be a balanced buffer pair of the same
length and have non-null tx/rx bufs.
Signed-off-by: Tom Burdick <thomas.burdick@intel.com>
Adds a callback op to RTIO enabling C logic to be interspersed with
I/O operations. This is not safe from userspace but allowable in kernel
space.
Signed-off-by: Tom Burdick <thomas.burdick@intel.com>
When sending small buffers out it makes sense to copy rather than
reference to avoid having to keep the small buffer around for the
lifetime of the write request.
Adjusts the op numbers to always be +1 from the previously defined op
id making it easier to re-arrange if needed in the future.
Signed-off-by: Tom Burdick <thomas.burdick@intel.com>
When the consume semaphore is enabled always give and take from the
semaphore. It's expected that rtio_cqe_consume and rtio_cqe_consume_block
will be used exclusively rather than directly poking at the SPSC queues.
Signed-off-by: Tom Burdick <thomas.burdick@intel.com>
C++ requires casting void * as the implicit cast isn't enough and
the C++ sample app fails to build without duplicating type information
here. Do that, and wrap it in extern C allowing the C++ to include
rtio_mpsc.h directly or indirectly.
Signed-off-by: Tom Burdick <thomas.burdick@intel.com>
The pending_sqe logic to track where in the ring queue the concurrent
executor had left off was slightly flawed. It didn't account for starting
all sqes in the queue and ending back up at the beginning.
Instead track the last SQE in the queue, from which the next one in the
queue will the one to start next.
If we happen to sweep the last known SQE in the queue, reset it to NULL
so the next time prepare is called we start at the beginning of the queue
again.
Signed-off-by: Tom Burdick <thomas.burdick@intel.com>
Transactional submissions treat a sequence of sqes as a single atomic
submission given to a single iodev and expect as a reply a single
completion.
This is useful for scatter gather like APIs that exist in Zephyr already
for I2C and SPI.
Signed-off-by: Tom Burdick <thomas.burdick@intel.com>
Tests were failing when the sqe was allocated on the stack for some
architectures. Initialization of variables on the stack is not guaranteed
to be zeroed like static data. This caused undefined behavior.
Secondly if the sqe is recycled and had a flag set, there would
unexpected behavior as well.
Signed-off-by: Tom Burdick <thomas.burdick@intel.com>
A new Z_SPIN_DELAY() macro has been added which
can be used to reduce a bit the amount of noise
due to the POSIX arch need to break busy loops with
k_busy_wait().
Use it.
Signed-off-by: Alberto Escolar Piedras <alberto.escolar.piedras@nordicsemi.no>
The same k_busy_wait needed for native_posix is needed for
all POSIX arch platforms.
Also increase that busy wait a bit. There is no point of
waiting for just 1 microsecond at a time, and waiting a bit
longer speeds up the test.
Signed-off-by: Alberto Escolar Piedras <alberto.escolar.piedras@nordicsemi.no>
By using an mpsc queue for each iodev, the iodev itself is shareable across
contexts. Since its lock free, submits may occur even from an ISR context.
Rather than a fixed size queue, and with it the possibility of running
out of pre-allocated spots, each iodev now holds a wait-free mpsc
queue head.
This changes the parameter of iodev submit to be a struct containing 4
pointers for the rtio context, the submission queue entry, and the mpsc
node for the iodevs submission queue.
This solves the problem involving busy iodevs working with real
devices. For example a busy SPI bus driver could enqueue, without locking,
a request to start once the current request is done.
The queue entries are expected to be owned and allocated by the
executor rather than the iodev. This helps simplify potential
tuning knobs to one place, the RTIO context and its executor an
application directly uses.
As the test case shows iodevs can operate effectively lock free
with the mpsc queue and a single atomic denoting the current task.
Signed-off-by: Tom Burdick <thomas.burdick@intel.com>
