This trick turns out also to be needed by the abort/join code.
Promote it to a more formal-looking internal API and clean up the
documentation to (hopefully) clarify the exact behavior and better
explain the need.
This is one of the more... enchanted bits of the scheduler, and while
the trick is IMHO pretty clean, it remains a big SMP footgun.
Signed-off-by: Andy Ross <andyross@google.com>
Update current stack limit on every context switch, including switching
to irq stack and switching back to thread stack.
Signed-off-by: Jaxson Han <jaxson.han@arm.com>
This commit mainly enable the safe exception stack including the stack
switch. Init the safe exception stack by calling
z_arm64_safe_exception_stack during the boot stage on every core. Also,
tweaks several files to properly switch the mode with different cases.
1) The same as before, when executing in userspace, SP_EL0 holds the
user stack and SP_EL1 holds the privileged stack, using EL1h mode.
2) When entering exception from EL0 then SP_EL0 will be saved in the
_esf_t structure. SP_EL1 will be the current SP, then retrieves the safe
exception stack to SP_EL0, making sure the always pointing to safe
exception stack as long as the system running in kernel space.
3) When exiting exception from EL1 to EL0 then SP_EL0 will be restored
from the stack value previously saved in the _esf_t structure. Still at
EL1h mode.
4) Either entering or exiting exception from EL1 to EL1, SP_EL0 will
keep holding the safe exception stack unchanged as memtioned above.
5) Do a quick stack check every time entering the exception from EL1 to
EL1. If check fail, set SP_EL1 to safe exception stack, and then handle
the fatal error.
Overall, the exception from user mode will be handled with kernel stack
at the assumption that it is impossible the stackoverflow happens at the
entry of exception from EL0 to EL1. However the exception from kernel
mode will be firstly checked with the safe exception stack to see if the
kernel stack overflows, because the exception might be triggered by
stack invalid accessing.
Signed-off-by: Jaxson Han <jaxson.han@arm.com>
Assembler files were not migrated with the new <zephyr/...> prefix.
Note that the conversion has been scripted, refer to #45388 for more
details.
Signed-off-by: Gerard Marull-Paretas <gerard.marull@nordicsemi.no>
In ARM parlance, the subroutine call return address is stored in the
"link register" or simply lr. Refer to it as lr which is clearer than
the anonymous x30 designation.
Signed-off-by: Nicolas Pitre <npitre@baylibre.com>
Make it optimal without the need for an SVC/exception roundtrip on
every context switch. Performance numbers from tests/benchmarks/sched:
Before:
unpend 85 ready 58 switch 258 pend 231 tot 632 (avg 699)
After:
unpend 85 ready 59 switch 115 pend 138 tot 397 (avg 478)
Signed-off-by: Nicolas Pitre <npitre@baylibre.com>
Get rid of all those global variables and scheduler locking.
Use the reguler IRQ exit path to let tests properly validate preemption.
Signed-off-by: Nicolas Pitre <npitre@baylibre.com>
Avoid executing ISRs using the thread stack as it might not be sized
for that. Plus, we do have IRQ stacks already set up for us.
The non-nested IRQ context is still (and has to be) saved on the thread
stack as the thread could be preempted.
The irq_offload case is never nested and always invoked with the
sched_lock held so it can be simplified a bit.
Signed-off-by: Nicolas Pitre <npitre@baylibre.com>
This is an uint32_t so the proper register width must be used, otherwise
the adjacent structure member will be overwritten (didn't happen in
practice because of struct member alignment but still). This makes the
inc_nest_counter and dec_nest_counter macros rather unwieldy, especially
with upcoming changes, so let's just remove them.
Signed-off-by: Nicolas Pitre <npitre@baylibre.com>
Include the new introduced include/arch/arm64/mm.h instead of the
arm_mmu.h or arm_mpu.h.
Unify function names z_arm64_thread_pt_init/z_arm64_swap_ptables with
z_arm64_thread_mem_domains_init/z_arm64_swap_mem_domains for mmu and
mpu, because:
1. mmu and mpu have almost the same logic.
2. mpu doesn't have ptables.
3. using the function names help reducing "#if define" macros.
Similarly, change z_arm64_ptable_ipi to z_arm64_domain_sync_ipi
And fix a log bug in arm_mmu.c.
Signed-off-by: Jaxson Han <jaxson.han@arm.com>
We can use build-time offsets from a struct k_thread pointer directly
to struct _callee_saved members. No need to compute that at run time.
Signed-off-by: Nicolas Pitre <npitre@baylibre.com>
This adds FPU sharing support with a lazy context switching algorithm.
Every thread is allowed to use FPU/SIMD registers. In fact, the compiler
may insert FPU reg accesses in anycontext to optimize even non-FP code
unless the -mgeneral-regs-only compiler flag is used, but Zephyr
currently doesn't support such a build.
It is therefore possible to do FP access in IRS as well with this patch
although IRQs are then disabled to prevent nested IRQs in such cases.
Because the thread object grows in size, some tests have to be adjusted.
Signed-off-by: Nicolas Pitre <npitre@baylibre.com>
Add the exception depth count to tpidrro_el0 and make it available
through the arch_exception_depth() accessor.
The IN_EL0 flag is now updated unconditionally even if userspace is
not configured. Doing otherwise made the code rather hairy and
I doubt the overhead is measurable.
Signed-off-by: Nicolas Pitre <npitre@baylibre.com>
The ARM64 port is currently using SP_EL0 for everything: kernel threads,
user threads and exceptions. In addition when taking an exception the
exception code is still using the thread SP without relying on any
interrupt stack.
If from one hand this makes the context switch really quick because the
thread context is already on the thread stack so we have only to save
one register (SP) for the whole context, on the other hand the major
limitation introduced by this choice is that if for some reason the
thread SP is corrupted or pointing to some unaccessible location (for
example in case of stack overflow), the exception code is unable to
recover or even deal with it.
The usual way of dealing with this kind of problems is to use a
dedicated interrupt stack on SP_EL1 when servicing the exceptions. The
real drawback of this is that, in case of context switch, all the
context must be copied from the shared interrupt stack into a
thread-specific stack or structure, so it is really slow.
We use here an hybrid approach, sacrificing a bit of stack space for a
quicker context switch. While nothing really changes for kernel threads,
for user threads we now use the privileged stack (already present to
service syscalls) as interrupt stack.
When an exception arrives the code now switches to use SP_EL1 that for
user threads is always pointing inside the privileged portion of the
stack of the current running thread. This achieves two things: (1)
isolate exceptions and syscall code to use a stack that is isolated,
privileged and not accessible to user threads and (2) the thread SP is
not touched at all during exceptions, so it can be invalid or corrupted
without any direct consequence.
Signed-off-by: Carlo Caione <ccaione@baylibre.com>
Split ARM and ARM64 architectures.
Details:
- CONFIG_ARM64 is decoupled from CONFIG_ARM (not a subset anymore)
- Arch and include AArch64 files are in a dedicated directory
(arch/arm64 and include/arch/arm64)
- AArch64 boards and SoC are moved to soc/arm64 and boards/arm64
- AArch64-specific DTS files are moved to dts/arm64
- The A72 support for the bcm_vk/viper board is moved in the
boards/bcm_vk/viper directory
Signed-off-by: Carlo Caione <ccaione@baylibre.com>
2021-03-31 10:34:33 -05:00
Renamed from arch/arm/core/aarch64/switch.S (Browse further)
