When USE_SWITCH=y, the thread struct is now mostly degenerate. Only
the two words for ICI/IT state tracking are required. Eliminate all
the extra fields when not needed and save a bunch of SRAM.
Note a handful of spots in coredump/debug that need a location for the
new stack pointer (stored as the switch handle now) are also updated.
Signed-off-by: Andy Ross <andyross@google.com>
The new switch code no longer needs PendSV, but still calls the SVC
vector. Split them into separate files for hygiene and a few
microseconds of build time.
Signed-off-by: Andy Ross <andyross@google.com>
Micro-optimization: We don't need a full arch_irq_lock(), which is a
~6-instruction sequence on Cortex M. The lock will be dropped
unconditionally on interrupt exit, so take it unconditionally.
Signed-off-by: Andy Ross <andyross@google.com>
z_get_next_switch_handle() is a clean API, but implementing it as a
(comparatively large) callable function requires significant
entry/exit boilerplate and hides the very common "no switch needed"
early exit condition from the enclosing C code that calls it. (Most
architectures call this from assembly though and don't notice).
Provide an unwrapped version for the specific needs non-SMP builds.
It's compatible in all other ways.
Slightly ugly, but the gains are significant (like a dozen cycles or
so).
Signed-off-by: Andy Ross <andyross@google.com>
This is a mistake. IRQ_OFFLOAD_NESTED is not intended to be an app
tunable. It reflects whether or not platform layer irq_offload()
implementation supports its use from within an ISR (i.e. to invoke a
nested interrupt for testing purposes). Not all architectures can do
that (and I know of at least one Xtensa soc that didn't give a
software interrupt the right priority and can't either). Basically
this should have been a "ARCH_HAS_" kind of kconfig, it's just named
funny.
And in any case the thread_metric tests don't do nested interrupts,
which as I understand it aren't even possible in the FreeRTOS
environment it was written on.
Signed-off-by: Andy Ross <andyross@google.com>
This microbenchmark has been super-useful. Add similar cases looking
at targetting interupt latency:
+ "IRQ" tests the cycles taken from the instant of an interrupt
trigger (as implemented, it's an arch_irq_lock() being released with
a timer interrupt pending) through a noop ISR (a k_timer callback),
and back to the interrupting thread.
+ "IRQ_P" is a similar preemption test: it measures the time taken
from the same trigger, into a higher-priority thread that
timestamps the instant of its wakeup (in this case as a return from
k_sem_take()).
Signed-off-by: Andy Ross <andyross@google.com>
This gets turned on at the platform layer by a handful of boards,
despite note being used for anything in the test. It has performance
impact, so really shouldn't be here.
Note that ARM_MPU also needs an =n, as that will select MPU, seemingly
incorrectly.
Signed-off-by: Andy Ross <andyross@google.com>
Running this with USE_SWITCH=y takes more space as expected, but it
oddly seems variable. Even at 2kb, I still see fairly regular
failures. It wants 3kb to be stable. Maybe the non-determinism of
the two-qemu setup is involved? Requests batch up and are handled
recursively? Too much code in the third party stack to make it easy
to tell.
Signed-off-by: Andy Ross <andyross@google.com>
GCC/gas has a code generation bugglet on thumb. The R7 register is
the ABI-defined frame pointer, though it's usually unused in zephyr
due to -fomit-frame-pointer (and the fact the DWARF on ARM doesn't
really need it). But when it IS enabled, which sometimes seems to
happen due to toolchain internals, GCC is unable to allow its use in
the clobber list of an asm() block (I guess it can't generate
spill/fill code without using the frame?).
There is existing protection for this problem that sets
-fomit-frame-pointer unconditionally on the two files (sched.c and
init.c) that require it. But even with that, gcc sometimes gets
kicked back into "framed mode" due to internal state. Provide a
kconfig workaround that does an explicit spill/fill on the one
test/platform where we have trouble.
(I checked, btw: an ARM clang build appears not to have this
misfeature)
Signed-off-by: Andy Ross <andyross@google.com>
Benchmarking context switch in the absence of other polluting factors
is actually really hard. I wanted a microbenchmark of just z_swap()
for tuning the ARM arch_switch() code, and nothing we have is very
useful. This one works well, measuring ONLY the time taken for a
thread to enter z_swap(), switch to another thread already suspended
in z_swap(), and return to a timestamp on the other side. It runs a
few thousand iterations and reporting average and standard deviation.
Signed-off-by: Andy Ross <andyross@google.com>
Pick some low hanging fruit on non-SMP code paths:
+ The scheduler spinlock is always taken, but as we're already in an
irqlocked state that's a noop. But the optmizer can't tell, because
arch_irq_lock() involves an asm block it can't see inside. Elide
the call when possible.
+ The z_swap_next_thread() function evaluates to just a single load of
_kernel.ready_q.cache when !SMP, but wasn't being inlined because of
function location. Move that test up into do_swap() so it's always
done correctly.
Signed-off-by: Andy Ross <andyross@google.com>
ARM Cortex M has what amounts to a design bug. The architecture
inherits several unpipelined/microcoded "ICI/IT" instruction forms
that take many cycles to complete (LDM/STM and the Thumb "IT"
conditional frame are the big ones). But out of a desire to minimize
interrupt latency, the CPU is allowed to halt and resume these
instructions mid-flight while they are partially completed. The
relevant bits of state are stored in the EPSR fields of the xPSR
register (see ARMv7-M manual B1.4.2). But (and this is the design
bug) those bits CANNOT BE WRITTEN BY SOFTWARE. They can only be
modified by exception return.
This means that if a Zephyr thread takes an interrupt
mid-ICI/IT-instruction, then switches to another thread on exit, and
then that thread is resumed by a cooperative switch and not an
interrupt, the instruction will lose the state and restart from
scratch. For LDM/STM that's generally idempotent for memory (but not
MMIO!), but for IT that means that the restart will re-execute
arbitrary instructions that may not be idempotent (e.g. "addeq r0, r0,
The fix is to check for this condition (which is very rare) on
interrupt exit when we are switching, and if we discover we've
interrupted such an instruction we swap the return address with a
trampoline that uses a UDF instruction to immediately trap to the
undefined instruction handler, which then recognizes the fixup address
as special and immediately returns back into the thread with the
correct EPSR value and resume PC (which have been stashed in the
thread struct). The overhead for the normal case is just a few cycles
for the test.
Signed-off-by: Andy Ross <andyross@google.com>
Integrate the new context layer, allowing it to be selected via the
pre-existing CONFIG_USE_SWITCH. Not a lot of changes, but notable
ones:
+ There was code in the MPU layer to adjust PSP on exception exit at a
stack overflow so that it remained inside the defined stack bounds.
With the new context layer though, exception exit will rewrite the
stack frame in a larger format, and needs PSP to be adjusted to make
room.
+ There was no such treatment in the PSPLIM case (the hardware prents
the SP from going that low), so I had to add similar code to
validate PSP at exit from fault handling.
+ The various return paths for fault/svc assembly handlers need to
call out to the switch code to do the needed scheduler work. Really
almost all of these can be replaced with C now, only userspace
syscall entry (which has to "return" into the privileged stack)
needs special treatment.
+ There is a gcc bug that prevents the arch_switch() inline assembly
from building when frame pointers are enabled (which they almost
never are on ARM): it disallows you from touching r7 (the thumb
frame pointer) entirely. But it's a context switch, we need to!
Worked around by enforcing -fomit-frame-pointer even in the two
scheduler files that can swap when NO_OPTIMIZATIONS=y.
Signed-off-by: Andy Ross <andyross@google.com>
Signed-off-by: Sudan Landge <sudan.landge@arm.com>
This is a standalone unit test of the new context code, which builds
in isolation as a MULTITHREADING=n binary. No integration with the
core arch layer is required.
Signed-off-by: Andy Ross <andyross@google.com>
1. Mostly complete. Supports MPU, userspace, PSPLIM-based stack
guards, and FPU/DSP features. ARMv8-M secure mode "should" work but I
don't know how to test it.
2. Designed with an eye to uncompromising/best-in-industry cooperative
context switch performance. No PendSV exception nor hardware
stacking/unstacking, just a traditional "musical chairs" switch.
Context gets saved on process stacks only instead of split between
there and the thread struct. No branches in the core integer switch
code (and just one in the FPU bits that can't be avoided).
3. Minimal assembly use; arch_switch() itself is ALWAYS_INLINE, there
is an assembly stub for exception exit, and that's it beyond one/two
instruction inlines elsewhere.
4. Selectable at build time, interoperable with existing code. Just
use the pre-existing CONFIG_USE_SWITCH=y flag to enable it. Or turn
it off to evade regressions as this stabilizes.
5. Exception/interrupt returns in the common case need only a single C
function to be called at the tail, and then return naturally.
Effectively "all interrupts are direct now". This isn't a benefit
currently because the existing stubs haven't been removed (see #4),
but in the long term we can look at exploiting this. The boilerplate
previously required is now (mostly) empty.
6. No support for ARMv6 (Cortex M0 et. al.) thumb code. The expanded
instruction encodings in ARMv7 are a big (big) win, so the older cores
really need a separate port to avoid impacting newer hardware.
Thankfully there isn't that much code to port (see #3), so this should
be doable.
Signed-off-by: Andy Ross <andyross@google.com>
This is a low-level/whitebox test of the older context layer and can't
pass with the new code (which has its own unit test).
Signed-off-by: Andy Ross <andyross@google.com>
This test seemed confused. Per the description, it's trying to create
a priority boost situation where a low priority thread (the main ztest
thread running test_mutex_timeout_race_during_priority_inversion())
holds a mutex and is boosted by a high priority thread (running
tThread_mutex_lock_should_fail()) trying to lock the mutex.
But that's not what the code was doing. It was creating the thread at
a LOWER priority, so no priority inheritance was happening. The test
would pass simply because the thread ordering via the natural
scheduler caused the spawned thread to run first. Fix the priorities
to be explicit. And add some comments explaining what's happening.
Also clean up the timeout whiteboxing, which apparently an attempt to
evade userspace protections on the stack. Just use a shared pointer
to ZTEST_DMEM. And remove a voodoo "align to tick boundary" sleep;
that's needless here as the timeout is explicitly constructed to be
simultaneous, we don't care when we go to sleep, only that we wake up
out of the same ISR.
Signed-off-by: Andy Ross <andyross@google.com>
This was just a pedantic setting. I mean, of course it makes no sense
to have thread FPU management state features built when you aren't
including the scheduler in the build.
...unless you want to unit-test the context switch code without
tripping over itself on the way into the test code. In fact lots of
unit testing of low level primitives can be done with
MULTITHREADING=n.
Remove the dependency. It isn't actually doing anything useful.
Signed-off-by: Andy Ross <andyross@google.com>
ISR_TABLES_LOCAL_DECLARATION depends on GEN_IRQ_VECTOR_TABLE but
this is not enforced in Kconfig.
Building without the GEN_IRQ_VECTOR_TABLE and with LOCAL_DECLARATION
will produce the following misleading static assertion error:
"CONFIG_IRQ_VECTOR_TABLE_JUMP_BY_{ADDRESS,CODE} not set"
As the LOCAL_DECLARATION macros expect GEN_IRQ_VECTOR_TABLE to be
enabled. LOCAL_DECLARATION also depends on GEN_ISR_TABLES but that
is a dependency of GEN_IRQ_VECTOR_TABLE already.
Signed-off-by: Bjarki Arge Andreasen <bjarki.andreasen@nordicsemi.no>
Add missing Kconfig selection to enable the Data Watch and Trace unit for
the PSOC6 family of devices using a Cortex-M4 core.
Signed-off-by: Alexander Roodt <alexander.roodt@deepgate.ai>
Add software state check (!data->wdt_started) to wdt_feed() validation
to ensure feed operations are only allowed after wdt_setup() has been
called.
Signed-off-by: Harini T <harini.t@amd.com>
The test's mock SMC callbacks were requesting shared memory allocations
of only 1 byte, but subsequently writing full optee_msg_arg structures
(~64 bytes) to these buffers, causing heap buffer overflows.
This heap corruption was latent and only manifested as a crash when
unrelated changes (such as increasing BSS size) altered the heap layout
enough for the overflow to corrupt critical data structures like the
optee_notify linked list nodes.
Fix by:
- Requesting the correct allocation size using OPTEE_MSG_GET_ARG_SIZE()
to match the number of parameters being written
- Adding assertions to verify the shared memory buffer is large enough
before writing to it, to catch similar issues in the future
Signed-off-by: Joakim Andersson <joerchan@gmail.com>
Add systimer compensation for time spent in light sleep while stalled.
Add sys_clock_idle_exit() for dticks announcement when leaving
light sleep.
Signed-off-by: Raffael Rostagno <raffael.rostagno@espressif.com>
Fix implementation so that sleep start routine is called at
pm_state_set() (entry) instead of pm_state_exit_post_ops().
Add RTC timer wakeup management, as well as code for deep
sleep.
Signed-off-by: Raffael Rostagno <raffael.rostagno@espressif.com>
Adjust PM residency times to more realistic values. Add power
states to ESP32-C2. Mark soft-off state as disabled, as it must
be called explicitly by application.
Signed-off-by: Raffael Rostagno <raffael.rostagno@espressif.com>
Add devicetree binding for the Trinamic TMCM-3216 3-axis stepper
motor controller. The TMCM-3216 communicates over RS485 using the
TMCL protocol and supports up to 3 stepper motors.
Signed-off-by: Alexis Czezar Torreno <alexisczezar.torreno@analog.com>
The SWO backend's panic() handler was a no-op. When LOG_PANIC() is
called (e.g., before a bootloader jumps to an app, or during a fatal
error), the logging subsystem flushes software buffers to the ITM
FIFO, but the TPIU hardware pipeline still has bytes in-flight on
the SWO pin. The debug probe loses these final messages.
Spin on ITM_TCR.BUSY until the formatter is idle, then busy-wait
for the TPIU output stage to clock remaining bytes out at the
configured SWO baud rate.
Signed-off-by: Jay Beavers <jay@tolttechnologies.com>
Add a shield for the Raspberry Pi Camera Module 2, featuring the IMX219
sensor. This module is suitable for bot the "normal" and "No IR" (no
infra-red) variants and uses the rasapberry pi connector.
Signed-off-by: Josuah Demangeon <me@josuah.net>
Co-authored-by: Alain Volmat <alain.volmat@foss.st.com>
Introduce a node alias called csi_capture_port in order to allow
having non ST shield using the chosen csi_capture_port to indicate
the camera interface.
Signed-off-by: Alain Volmat <alain.volmat@foss.st.com>
Build the IMX219 image sensor driver on the build-all tests.
No interconnection with any other device is done, which is not mandatory
at the time as no link-related parameter is used.
Signed-off-by: Josuah Demangeon <me@josuah.net>
Add support for the Sony IMX219 CSI sensor.
This sensor supports resolution of 3280x2464 in RGGB bayer format
either 8 or 10 bits and using 2 or 4 CSI lanes.
Only 10 bits on 2 CSI lanes is currently supported, and only in
1920x1080 pixel resolution using cropping using the video_set_format(),
and will need to be converted to video_set_selection() instead.
Signed-off-by: Josuah Demangeon <me@josuah.net>
Co-authored-by: Alan Shaju <alanshaju@rideltech.com>
Co-authored-by: Alain Volmat <alain.volmat@foss.st.com>
Switch to zephyr-keep-sorted-start/stop to keep checking the new
driver imports are always sorted, which reduces chances of git
conflict while adding new drivers from separate branches.
Signed-off-by: Josuah Demangeon <me@josuah.net>
Only VIDEO_CID_ANALOGUE_GAIN and VIDEO_CID_GAIN were defined. Also add
the complementary VIDEO_CID_DIGITAL_GAIN.
Signed-off-by: Josuah Demangeon <me@josuah.net>
When routing packets, check if the destination address matches an
on-link prefix on any interface before using the default router.
This ensures packets destined for on-link prefixes are not
incorrectly sent to the default router.
Signed-off-by: Jani Hirsimäki <jani.hirsimaki@nordicsemi.no>
