michaelh/zephyr
1
0
Fork 0
Code Releases Activity
zephyr/arch/arm64/core/vector_table.S

358 lines
9.5 KiB
ArmAsm
Raw Permalink Normal View History

arch: arm64: Introduce ARM64 (AArch64) architecture Introduce the basic ARM64 architecture support. A new CONFIG_ARM64 symbol is introduced for the new architecture and new cmake / Kconfig files are added to switch between ARM and ARM64. Signed-off-by: Carlo Caione <ccaione@baylibre.com>
2019-11-10 17:17:19 +01:00
/*
* Copyright (c) 2019 Carlo Caione <ccaione@baylibre.com>
*
* SPDX-License-Identifier: Apache-2.0
*/
arch: arm64: Reword comments Fix, reword and rework comments. Signed-off-by: Carlo Caione <ccaione@baylibre.com>
2020-09-18 09:04:43 +02:00
/*
* Populated vector table
arch: arm64: Introduce ARM64 (AArch64) architecture Introduce the basic ARM64 architecture support. A new CONFIG_ARM64 symbol is introduced for the new architecture and new cmake / Kconfig files are added to switch between ARM and ARM64. Signed-off-by: Carlo Caione <ccaione@baylibre.com>
2019-11-10 17:17:19 +01:00
*/
asm: update files with <zephyr/...> include prefix Assembler files were not migrated with the new <zephyr/...> prefix. Note that the conversion has been scripted, refer to #45388 for more details. Signed-off-by: Gerard Marull-Paretas <gerard.marull@nordicsemi.no>
2022-05-09 13:56:13 +02:00
#include <zephyr/toolchain.h>
#include <zephyr/linker/sections.h>
build: namespace the generated headers with `zephyr/` Namespaced the generated headers with `zephyr` to prevent potential conflict with other headers. Introduce a temporary Kconfig `LEGACY_GENERATED_INCLUDE_PATH` that is enabled by default. This allows the developers to continue the use of the old include paths for the time being until it is deprecated and eventually removed. The Kconfig will generate a build-time warning message, similar to the `CONFIG_TIMER_RANDOM_GENERATOR`. Updated the includes path of in-tree sources accordingly. Most of the changes here are scripted, check the PR for more info. Signed-off-by: Yong Cong Sin <ycsin@meta.com>
2024-01-24 10:35:04 +01:00
#include <zephyr/offsets.h>
asm: update files with <zephyr/...> include prefix Assembler files were not migrated with the new <zephyr/...> prefix. Note that the conversion has been scripted, refer to #45388 for more details. Signed-off-by: Gerard Marull-Paretas <gerard.marull@nordicsemi.no>
2022-05-09 13:56:13 +02:00
#include <zephyr/arch/cpu.h>
#include <zephyr/arch/arm64/tpidrro_el0.h>
arch: arm64: Enable safe exception stack This commit mainly enable the safe exception stack including the stack switch. Init the safe exception stack by calling z_arm64_safe_exception_stack during the boot stage on every core. Also, tweaks several files to properly switch the mode with different cases. 1) The same as before, when executing in userspace, SP_EL0 holds the user stack and SP_EL1 holds the privileged stack, using EL1h mode. 2) When entering exception from EL0 then SP_EL0 will be saved in the _esf_t structure. SP_EL1 will be the current SP, then retrieves the safe exception stack to SP_EL0, making sure the always pointing to safe exception stack as long as the system running in kernel space. 3) When exiting exception from EL1 to EL0 then SP_EL0 will be restored from the stack value previously saved in the _esf_t structure. Still at EL1h mode. 4) Either entering or exiting exception from EL1 to EL1, SP_EL0 will keep holding the safe exception stack unchanged as memtioned above. 5) Do a quick stack check every time entering the exception from EL1 to EL1. If check fail, set SP_EL1 to safe exception stack, and then handle the fatal error. Overall, the exception from user mode will be handled with kernel stack at the assumption that it is impossible the stackoverflow happens at the entry of exception from EL0 to EL1. However the exception from kernel mode will be firstly checked with the safe exception stack to see if the kernel stack overflows, because the exception might be triggered by stack invalid accessing. Signed-off-by: Jaxson Han <jaxson.han@arm.com>
2022-11-01 09:11:45 +01:00
#include <offsets_short.h>
#include "macro_priv.inc"
arch: arm64: Introduce ARM64 (AArch64) architecture Introduce the basic ARM64 architecture support. A new CONFIG_ARM64 symbol is introduced for the new architecture and new cmake / Kconfig files are added to switch between ARM and ARM64. Signed-off-by: Carlo Caione <ccaione@baylibre.com>
2019-11-10 17:17:19 +01:00
arm: aarch64: Reintroduce _ASM_FILE_PROLOGUE This is currently missing from the AArch64 assembly files. Signed-off-by: Carlo Caione <ccaione@baylibre.com>
2020-03-10 18:26:58 +01:00
_ASM_FILE_PROLOGUE
aarch64: rationalize exception entry/exit code Each vector slot has room for 32 instructions. The exception context saving needs 15 instructions already. Rather than duplicating those instructions in each out-of-line exception routines, let's store them directly in the vector table. That vector space is otherwise wasted anyway. Move the z_arm64_enter_exc macro into vector_table.S as this is the only place where it should be used. To further reduce code size, let's make z_arm64_exit_exc into a function of its own to avoid code duplication again. It is put in vector_table.S as this is the most logical location to go with its z_arm64_enter_exc counterpart. Signed-off-by: Nicolas Pitre <npitre@baylibre.com>
2021-02-23 20:48:26 +01:00
/*
arm64: refer to the link register as "lr" rather than "x30" In ARM parlance, the subroutine call return address is stored in the "link register" or simply lr. Refer to it as lr which is clearer than the anonymous x30 designation. Signed-off-by: Nicolas Pitre <npitre@baylibre.com>
2022-03-16 18:12:39 +01:00
* Save volatile registers, LR, SPSR_EL1 and ELR_EL1
aarch64: rationalize exception entry/exit code Each vector slot has room for 32 instructions. The exception context saving needs 15 instructions already. Rather than duplicating those instructions in each out-of-line exception routines, let's store them directly in the vector table. That vector space is otherwise wasted anyway. Move the z_arm64_enter_exc macro into vector_table.S as this is the only place where it should be used. To further reduce code size, let's make z_arm64_exit_exc into a function of its own to avoid code duplication again. It is put in vector_table.S as this is the most logical location to go with its z_arm64_enter_exc counterpart. Signed-off-by: Nicolas Pitre <npitre@baylibre.com>
2021-02-23 20:48:26 +01:00
*
arm64: refer to the link register as "lr" rather than "x30" In ARM parlance, the subroutine call return address is stored in the "link register" or simply lr. Refer to it as lr which is clearer than the anonymous x30 designation. Signed-off-by: Nicolas Pitre <npitre@baylibre.com>
2022-03-16 18:12:39 +01:00
* Save the volatile registers and LR on the process stack. This is
aarch64: rationalize exception entry/exit code Each vector slot has room for 32 instructions. The exception context saving needs 15 instructions already. Rather than duplicating those instructions in each out-of-line exception routines, let's store them directly in the vector table. That vector space is otherwise wasted anyway. Move the z_arm64_enter_exc macro into vector_table.S as this is the only place where it should be used. To further reduce code size, let's make z_arm64_exit_exc into a function of its own to avoid code duplication again. It is put in vector_table.S as this is the most logical location to go with its z_arm64_enter_exc counterpart. Signed-off-by: Nicolas Pitre <npitre@baylibre.com>
2021-02-23 20:48:26 +01:00
* needed if the thread is switched out because they can be clobbered by the
* ISR and/or context switch.
*/
arch: arm64: Add el label for vector entry macro Add a new label el for z_arm64_enter_exc to indicate which el the exception comes from. Signed-off-by: Jaxson Han <jaxson.han@arm.com>
2022-10-25 08:55:47 +02:00
.macro z_arm64_enter_exc xreg0, xreg1, el
aarch64: rationalize exception entry/exit code Each vector slot has room for 32 instructions. The exception context saving needs 15 instructions already. Rather than duplicating those instructions in each out-of-line exception routines, let's store them directly in the vector table. That vector space is otherwise wasted anyway. Move the z_arm64_enter_exc macro into vector_table.S as this is the only place where it should be used. To further reduce code size, let's make z_arm64_exit_exc into a function of its own to avoid code duplication again. It is put in vector_table.S as this is the most logical location to go with its z_arm64_enter_exc counterpart. Signed-off-by: Nicolas Pitre <npitre@baylibre.com>
2021-02-23 20:48:26 +01:00
/*
* Two things can happen to the remaining registers:
*
* - No context-switch: in this case x19-x28 are callee-saved register
* so we can be sure they are not going to be clobbered by ISR.
* - Context-switch: the callee-saved registers are saved by
* z_arm64_context_switch() in the kernel structure.
*/
sub sp, sp, ___esf_t_SIZEOF
arch: arm64: Enable safe exception stack This commit mainly enable the safe exception stack including the stack switch. Init the safe exception stack by calling z_arm64_safe_exception_stack during the boot stage on every core. Also, tweaks several files to properly switch the mode with different cases. 1) The same as before, when executing in userspace, SP_EL0 holds the user stack and SP_EL1 holds the privileged stack, using EL1h mode. 2) When entering exception from EL0 then SP_EL0 will be saved in the _esf_t structure. SP_EL1 will be the current SP, then retrieves the safe exception stack to SP_EL0, making sure the always pointing to safe exception stack as long as the system running in kernel space. 3) When exiting exception from EL1 to EL0 then SP_EL0 will be restored from the stack value previously saved in the _esf_t structure. Still at EL1h mode. 4) Either entering or exiting exception from EL1 to EL1, SP_EL0 will keep holding the safe exception stack unchanged as memtioned above. 5) Do a quick stack check every time entering the exception from EL1 to EL1. If check fail, set SP_EL1 to safe exception stack, and then handle the fatal error. Overall, the exception from user mode will be handled with kernel stack at the assumption that it is impossible the stackoverflow happens at the entry of exception from EL0 to EL1. However the exception from kernel mode will be firstly checked with the safe exception stack to see if the kernel stack overflows, because the exception might be triggered by stack invalid accessing. Signed-off-by: Jaxson Han <jaxson.han@arm.com>
2022-11-01 09:11:45 +01:00
#ifdef CONFIG_ARM64_SAFE_EXCEPTION_STACK
.if \el == el1
/*
* EL1t mode cannot access sp_el1, so set x0 to sp_el1 without corrupt
* other registers
*/
add sp, sp, x0 // sp' = sp + x0
sub x0, sp, x0 // x0' = sp' - x0 = sp
msr SPSel, #0
stp x16, x17, [sp, -(___esf_t_SIZEOF - ___esf_t_x16_x17_OFFSET)]
stp x18, lr, [sp, -(___esf_t_SIZEOF - ___esf_t_x18_lr_OFFSET)]
bl z_arm64_quick_stack_check
.endif
#endif
aarch64: rationalize exception entry/exit code Each vector slot has room for 32 instructions. The exception context saving needs 15 instructions already. Rather than duplicating those instructions in each out-of-line exception routines, let's store them directly in the vector table. That vector space is otherwise wasted anyway. Move the z_arm64_enter_exc macro into vector_table.S as this is the only place where it should be used. To further reduce code size, let's make z_arm64_exit_exc into a function of its own to avoid code duplication again. It is put in vector_table.S as this is the most logical location to go with its z_arm64_enter_exc counterpart. Signed-off-by: Nicolas Pitre <npitre@baylibre.com>
2021-02-23 20:48:26 +01:00
stp x0, x1, [sp, ___esf_t_x0_x1_OFFSET]
stp x2, x3, [sp, ___esf_t_x2_x3_OFFSET]
stp x4, x5, [sp, ___esf_t_x4_x5_OFFSET]
stp x6, x7, [sp, ___esf_t_x6_x7_OFFSET]
stp x8, x9, [sp, ___esf_t_x8_x9_OFFSET]
stp x10, x11, [sp, ___esf_t_x10_x11_OFFSET]
stp x12, x13, [sp, ___esf_t_x12_x13_OFFSET]
stp x14, x15, [sp, ___esf_t_x14_x15_OFFSET]
arch: arm64: Enable safe exception stack This commit mainly enable the safe exception stack including the stack switch. Init the safe exception stack by calling z_arm64_safe_exception_stack during the boot stage on every core. Also, tweaks several files to properly switch the mode with different cases. 1) The same as before, when executing in userspace, SP_EL0 holds the user stack and SP_EL1 holds the privileged stack, using EL1h mode. 2) When entering exception from EL0 then SP_EL0 will be saved in the _esf_t structure. SP_EL1 will be the current SP, then retrieves the safe exception stack to SP_EL0, making sure the always pointing to safe exception stack as long as the system running in kernel space. 3) When exiting exception from EL1 to EL0 then SP_EL0 will be restored from the stack value previously saved in the _esf_t structure. Still at EL1h mode. 4) Either entering or exiting exception from EL1 to EL1, SP_EL0 will keep holding the safe exception stack unchanged as memtioned above. 5) Do a quick stack check every time entering the exception from EL1 to EL1. If check fail, set SP_EL1 to safe exception stack, and then handle the fatal error. Overall, the exception from user mode will be handled with kernel stack at the assumption that it is impossible the stackoverflow happens at the entry of exception from EL0 to EL1. However the exception from kernel mode will be firstly checked with the safe exception stack to see if the kernel stack overflows, because the exception might be triggered by stack invalid accessing. Signed-off-by: Jaxson Han <jaxson.han@arm.com>
2022-11-01 09:11:45 +01:00
#ifdef CONFIG_ARM64_SAFE_EXCEPTION_STACK
/* The expection from el1 does not need to save x16, x17, x18 and lr */
.if \el == el0
#endif
aarch64: rationalize exception entry/exit code Each vector slot has room for 32 instructions. The exception context saving needs 15 instructions already. Rather than duplicating those instructions in each out-of-line exception routines, let's store them directly in the vector table. That vector space is otherwise wasted anyway. Move the z_arm64_enter_exc macro into vector_table.S as this is the only place where it should be used. To further reduce code size, let's make z_arm64_exit_exc into a function of its own to avoid code duplication again. It is put in vector_table.S as this is the most logical location to go with its z_arm64_enter_exc counterpart. Signed-off-by: Nicolas Pitre <npitre@baylibre.com>
2021-02-23 20:48:26 +01:00
stp x16, x17, [sp, ___esf_t_x16_x17_OFFSET]
arm64: refer to the link register as "lr" rather than "x30" In ARM parlance, the subroutine call return address is stored in the "link register" or simply lr. Refer to it as lr which is clearer than the anonymous x30 designation. Signed-off-by: Nicolas Pitre <npitre@baylibre.com>
2022-03-16 18:12:39 +01:00
stp x18, lr, [sp, ___esf_t_x18_lr_OFFSET]
arch: arm64: Enable safe exception stack This commit mainly enable the safe exception stack including the stack switch. Init the safe exception stack by calling z_arm64_safe_exception_stack during the boot stage on every core. Also, tweaks several files to properly switch the mode with different cases. 1) The same as before, when executing in userspace, SP_EL0 holds the user stack and SP_EL1 holds the privileged stack, using EL1h mode. 2) When entering exception from EL0 then SP_EL0 will be saved in the _esf_t structure. SP_EL1 will be the current SP, then retrieves the safe exception stack to SP_EL0, making sure the always pointing to safe exception stack as long as the system running in kernel space. 3) When exiting exception from EL1 to EL0 then SP_EL0 will be restored from the stack value previously saved in the _esf_t structure. Still at EL1h mode. 4) Either entering or exiting exception from EL1 to EL1, SP_EL0 will keep holding the safe exception stack unchanged as memtioned above. 5) Do a quick stack check every time entering the exception from EL1 to EL1. If check fail, set SP_EL1 to safe exception stack, and then handle the fatal error. Overall, the exception from user mode will be handled with kernel stack at the assumption that it is impossible the stackoverflow happens at the entry of exception from EL0 to EL1. However the exception from kernel mode will be firstly checked with the safe exception stack to see if the kernel stack overflows, because the exception might be triggered by stack invalid accessing. Signed-off-by: Jaxson Han <jaxson.han@arm.com>
2022-11-01 09:11:45 +01:00
#ifdef CONFIG_ARM64_SAFE_EXCEPTION_STACK
.endif
#endif
aarch64: rationalize exception entry/exit code Each vector slot has room for 32 instructions. The exception context saving needs 15 instructions already. Rather than duplicating those instructions in each out-of-line exception routines, let's store them directly in the vector table. That vector space is otherwise wasted anyway. Move the z_arm64_enter_exc macro into vector_table.S as this is the only place where it should be used. To further reduce code size, let's make z_arm64_exit_exc into a function of its own to avoid code duplication again. It is put in vector_table.S as this is the most logical location to go with its z_arm64_enter_exc counterpart. Signed-off-by: Nicolas Pitre <npitre@baylibre.com>
2021-02-23 20:48:26 +01:00
arm64: Add frame-pointer based stack unwinding Add the frame-pointer based stack unwinding on ARM64. This is a typical output with the feature enabled: *** Booting Zephyr OS build zephyr-v3.4.0-1029-gae22ff648c16 *** E: ELR_ELn: 0x00000000400011c8 E: ESR_ELn: 0x0000000096000046 E: EC: 0x25 (Data Abort taken without a change in Exception level) E: IL: 0x1 E: ISS: 0x46 E: FAR_ELn: 0x0000000000000000 E: TPIDRRO: 0x0100000040011650 E: x0: 0x0000000000000000 x1: 0x0000000000000003 E: x2: 0x0000000000000003 x3: 0x000000004005c6a0 E: x4: 0x0000000000000000 x5: 0x000000004005c7f0 E: x6: 0x0000000048000000 x7: 0x0000000048000000 E: x8: 0x0000000000000005 x9: 0x0000000000000000 E: x10: 0x0000000000000000 x11: 0x0000000000000000 E: x12: 0x0000000000000000 x13: 0x0000000000000000 E: x14: 0x0000000000000000 x15: 0x0000000000000000 E: x16: 0x0000000040004290 x17: 0x0000000000000000 E: x18: 0x0000000000000000 lr: 0x0000000040001208 E: E: backtrace 0: fp: 0x000000004005c690 lr: 0x0000000040001270 E: backtrace 1: fp: 0x000000004005c6b0 lr: 0x0000000040001290 E: backtrace 2: fp: 0x000000004005c7d0 lr: 0x0000000040004ac0 E: backtrace 3: fp: 0x000000004005c7f0 lr: 0x00000000400013a4 E: backtrace 4: fp: 0x000000004005c800 lr: 0x0000000000000000 E: E: >>> ZEPHYR FATAL ERROR 0: CPU exception on CPU 0 E: Current thread: 0x40011310 (unknown) E: Halting system Signed-off-by: Carlo Caione <ccaione@baylibre.com>
2023-07-13 09:03:26 +02:00
#ifdef CONFIG_ARM64_ENABLE_FRAME_POINTER
str x29, [sp, ___esf_t_fp_OFFSET]
#endif
aarch64: rationalize exception entry/exit code Each vector slot has room for 32 instructions. The exception context saving needs 15 instructions already. Rather than duplicating those instructions in each out-of-line exception routines, let's store them directly in the vector table. That vector space is otherwise wasted anyway. Move the z_arm64_enter_exc macro into vector_table.S as this is the only place where it should be used. To further reduce code size, let's make z_arm64_exit_exc into a function of its own to avoid code duplication again. It is put in vector_table.S as this is the most logical location to go with its z_arm64_enter_exc counterpart. Signed-off-by: Nicolas Pitre <npitre@baylibre.com>
2021-02-23 20:48:26 +01:00
mrs \xreg0, spsr_el1
mrs \xreg1, elr_el1
stp \xreg0, \xreg1, [sp, ___esf_t_spsr_elr_OFFSET]
arch: arm64: Enable safe exception stack This commit mainly enable the safe exception stack including the stack switch. Init the safe exception stack by calling z_arm64_safe_exception_stack during the boot stage on every core. Also, tweaks several files to properly switch the mode with different cases. 1) The same as before, when executing in userspace, SP_EL0 holds the user stack and SP_EL1 holds the privileged stack, using EL1h mode. 2) When entering exception from EL0 then SP_EL0 will be saved in the _esf_t structure. SP_EL1 will be the current SP, then retrieves the safe exception stack to SP_EL0, making sure the always pointing to safe exception stack as long as the system running in kernel space. 3) When exiting exception from EL1 to EL0 then SP_EL0 will be restored from the stack value previously saved in the _esf_t structure. Still at EL1h mode. 4) Either entering or exiting exception from EL1 to EL1, SP_EL0 will keep holding the safe exception stack unchanged as memtioned above. 5) Do a quick stack check every time entering the exception from EL1 to EL1. If check fail, set SP_EL1 to safe exception stack, and then handle the fatal error. Overall, the exception from user mode will be handled with kernel stack at the assumption that it is impossible the stackoverflow happens at the entry of exception from EL0 to EL1. However the exception from kernel mode will be firstly checked with the safe exception stack to see if the kernel stack overflows, because the exception might be triggered by stack invalid accessing. Signed-off-by: Jaxson Han <jaxson.han@arm.com>
2022-11-01 09:11:45 +01:00
#ifdef CONFIG_ARM64_SAFE_EXCEPTION_STACK
.if \el == el0
mrs x0, sp_el0
str x0, [sp, ___esf_t_sp_el0_OFFSET]
/* Retrieving safe exception stack */
get_cpu x0
ldr x1, [x0, #_cpu_offset_to_safe_exception_stack]
msr sp_el0, x1
.endif
#endif
arm64: implement exception depth count Add the exception depth count to tpidrro_el0 and make it available through the arch_exception_depth() accessor. The IN_EL0 flag is now updated unconditionally even if userspace is not configured. Doing otherwise made the code rather hairy and I doubt the overhead is measurable. Signed-off-by: Nicolas Pitre <npitre@baylibre.com>
2021-04-13 06:42:00 +02:00
/* Clear usermode flag and increment exception depth */
aarch64: userspace: Introduce arch_is_user_context The arch_is_user_context() function is relying on the content of the tpidrro_el0 register to determine whether we are in user context or not. This register is set to '1' when in EL1 and set back to '0' when user threads are running in userspace. Signed-off-by: Carlo Caione <ccaione@baylibre.com>
2020-11-25 15:54:12 +01:00
mrs \xreg0, tpidrro_el0
arm64: implement exception depth count Add the exception depth count to tpidrro_el0 and make it available through the arch_exception_depth() accessor. The IN_EL0 flag is now updated unconditionally even if userspace is not configured. Doing otherwise made the code rather hairy and I doubt the overhead is measurable. Signed-off-by: Nicolas Pitre <npitre@baylibre.com>
2021-04-13 06:42:00 +02:00
mov \xreg1, #TPIDRROEL0_EXC_UNIT
arm64: hold curr_cpu instance in tpidrro_el0 Let's fully exploit tpidrro_el0 by storing in it the current CPU's struct _cpu instance alongside the userspace mode flag bit. This greatly simplifies the code needed to get at the cpu structure, and this paves the way to much simpler multi cluster support, as there is no longer the need to decode MPIDR all the time. The same code is used in the !SMP case as there are benefits there too such as avoiding the literal pool, and it looks cleaner. The tpidrro_el0 value is no longer stored in the exception stack frame. Instead, we simply restore the user mode flag based on the SPSR value. This way, more flag bits could be used independently in the future. Signed-off-by: Nicolas Pitre <npitre@baylibre.com>
2021-04-10 05:01:18 +02:00
bic \xreg0, \xreg0, #TPIDRROEL0_IN_EL0
arm64: implement exception depth count Add the exception depth count to tpidrro_el0 and make it available through the arch_exception_depth() accessor. The IN_EL0 flag is now updated unconditionally even if userspace is not configured. Doing otherwise made the code rather hairy and I doubt the overhead is measurable. Signed-off-by: Nicolas Pitre <npitre@baylibre.com>
2021-04-13 06:42:00 +02:00
add \xreg0, \xreg0, \xreg1
arm64: hold curr_cpu instance in tpidrro_el0 Let's fully exploit tpidrro_el0 by storing in it the current CPU's struct _cpu instance alongside the userspace mode flag bit. This greatly simplifies the code needed to get at the cpu structure, and this paves the way to much simpler multi cluster support, as there is no longer the need to decode MPIDR all the time. The same code is used in the !SMP case as there are benefits there too such as avoiding the literal pool, and it looks cleaner. The tpidrro_el0 value is no longer stored in the exception stack frame. Instead, we simply restore the user mode flag based on the SPSR value. This way, more flag bits could be used independently in the future. Signed-off-by: Nicolas Pitre <npitre@baylibre.com>
2021-04-10 05:01:18 +02:00
msr tpidrro_el0, \xreg0
aarch64: userspace: Introduce arch_is_user_context The arch_is_user_context() function is relying on the content of the tpidrro_el0 register to determine whether we are in user context or not. This register is set to '1' when in EL1 and set back to '0' when user threads are running in userspace. Signed-off-by: Carlo Caione <ccaione@baylibre.com>
2020-11-25 15:54:12 +01:00
arm64: FPU context switching support This adds FPU sharing support with a lazy context switching algorithm. Every thread is allowed to use FPU/SIMD registers. In fact, the compiler may insert FPU reg accesses in anycontext to optimize even non-FP code unless the -mgeneral-regs-only compiler flag is used, but Zephyr currently doesn't support such a build. It is therefore possible to do FP access in IRS as well with this patch although IRQs are then disabled to prevent nested IRQs in such cases. Because the thread object grows in size, some tests have to be adjusted. Signed-off-by: Nicolas Pitre <npitre@baylibre.com>
2021-04-08 05:31:44 +02:00
#ifdef CONFIG_FPU_SHARING
bl z_arm64_fpu_enter_exc
#endif
aarch64: rationalize exception entry/exit code Each vector slot has room for 32 instructions. The exception context saving needs 15 instructions already. Rather than duplicating those instructions in each out-of-line exception routines, let's store them directly in the vector table. That vector space is otherwise wasted anyway. Move the z_arm64_enter_exc macro into vector_table.S as this is the only place where it should be used. To further reduce code size, let's make z_arm64_exit_exc into a function of its own to avoid code duplication again. It is put in vector_table.S as this is the most logical location to go with its z_arm64_enter_exc counterpart. Signed-off-by: Nicolas Pitre <npitre@baylibre.com>
2021-02-23 20:48:26 +01:00
.endm
arch: arm64: Introduce ARM64 (AArch64) architecture Introduce the basic ARM64 architecture support. A new CONFIG_ARM64 symbol is introduced for the new architecture and new cmake / Kconfig files are added to switch between ARM and ARM64. Signed-off-by: Carlo Caione <ccaione@baylibre.com>
2019-11-10 17:17:19 +01:00
/*
* Four types of exceptions:
* - synchronous: aborts from MMU, SP/CP alignment checking, unallocated
* instructions, SVCs/SMCs/HVCs, ...)
* - IRQ: group 1 (normal) interrupts
* - FIQ: group 0 or secure interrupts
* - SError: fatal system errors
*
* Four different contexts:
* - from same exception level, when using the SP_EL0 stack pointer
* - from same exception level, when using the SP_ELx stack pointer
* - from lower exception level, when this is AArch64
* - from lower exception level, when this is AArch32
*
* +------------------+------------------+-------------------------+
* | Address | Exception type | Description |
* +------------------+------------------+-------------------------+
* | VBAR_ELn + 0x000 | Synchronous | Current EL with SP0 |
* | + 0x080 | IRQ / vIRQ | |
* | + 0x100 | FIQ / vFIQ | |
* | + 0x180 | SError / vSError | |
* +------------------+------------------+-------------------------+
* | + 0x200 | Synchronous | Current EL with SPx |
* | + 0x280 | IRQ / vIRQ | |
* | + 0x300 | FIQ / vFIQ | |
* | + 0x380 | SError / vSError | |
* +------------------+------------------+-------------------------+
arch: arm64: Add missing vector table entries The current vector table is missing some (not used) entries. Fill these in for the sake of completeness. Signed-off-by: Carlo Caione <ccaione@baylibre.com>
2020-09-07 15:33:37 +02:00
* | + 0x400 | Synchronous | Lower EL using AArch64 |
arch: arm64: Introduce ARM64 (AArch64) architecture Introduce the basic ARM64 architecture support. A new CONFIG_ARM64 symbol is introduced for the new architecture and new cmake / Kconfig files are added to switch between ARM and ARM64. Signed-off-by: Carlo Caione <ccaione@baylibre.com>
2019-11-10 17:17:19 +01:00
* | + 0x480 | IRQ / vIRQ | |
* | + 0x500 | FIQ / vFIQ | |
* | + 0x580 | SError / vSError | |
* +------------------+------------------+-------------------------+
arch: arm64: Add missing vector table entries The current vector table is missing some (not used) entries. Fill these in for the sake of completeness. Signed-off-by: Carlo Caione <ccaione@baylibre.com>
2020-09-07 15:33:37 +02:00
* | + 0x600 | Synchronous | Lower EL using AArch32 |
arch: arm64: Introduce ARM64 (AArch64) architecture Introduce the basic ARM64 architecture support. A new CONFIG_ARM64 symbol is introduced for the new architecture and new cmake / Kconfig files are added to switch between ARM and ARM64. Signed-off-by: Carlo Caione <ccaione@baylibre.com>
2019-11-10 17:17:19 +01:00
* | + 0x680 | IRQ / vIRQ | |
* | + 0x700 | FIQ / vFIQ | |
* | + 0x780 | SError / vSError | |
* +------------------+------------------+-------------------------+
*/
arm64: decrustify and extend SMP boot code The SMP boot code depends on physical CPU #0 to be first to boot and subsequent CPUs to follow suit in a linear fashion. Let's decouple physical and logical numbering so that any physical CPU can be the boot CPU. This is based on a prior code proposal from Jiafei Pan <Jiafei.Pan@nxp.com>. This, however, was about to turn the boot code into some hairy mess. So let's clean things up and simplify the code as well while at it. Both the extension and the clean up aren't separate commits because they actually depend on each other. The BOOT_PARAM_*_OFFSET defines are locally hardcoded as there is no point exposing the related structure widely. Build time assertions ensure they don't go out of sync with the struct definition. And vector_table.h is repurposed into boot.h to gather boot related definitions. Signed-off-by: Nicolas Pitre <npitre@baylibre.com> Signed-off-by: Jiafei Pan <Jiafei.Pan@nxp.com>
2021-04-16 23:45:00 +02:00
GDATA(_vector_table)
arch: arm64: Introduce ARM64 (AArch64) architecture Introduce the basic ARM64 architecture support. A new CONFIG_ARM64 symbol is introduced for the new architecture and new cmake / Kconfig files are added to switch between ARM and ARM64. Signed-off-by: Carlo Caione <ccaione@baylibre.com>
2019-11-10 17:17:19 +01:00
SECTION_SUBSEC_FUNC(exc_vector_table,_vector_table_section,_vector_table)
arm64: decrustify and extend SMP boot code The SMP boot code depends on physical CPU #0 to be first to boot and subsequent CPUs to follow suit in a linear fashion. Let's decouple physical and logical numbering so that any physical CPU can be the boot CPU. This is based on a prior code proposal from Jiafei Pan <Jiafei.Pan@nxp.com>. This, however, was about to turn the boot code into some hairy mess. So let's clean things up and simplify the code as well while at it. Both the extension and the clean up aren't separate commits because they actually depend on each other. The BOOT_PARAM_*_OFFSET defines are locally hardcoded as there is no point exposing the related structure widely. Build time assertions ensure they don't go out of sync with the struct definition. And vector_table.h is repurposed into boot.h to gather boot related definitions. Signed-off-by: Nicolas Pitre <npitre@baylibre.com> Signed-off-by: Jiafei Pan <Jiafei.Pan@nxp.com>
2021-04-16 23:45:00 +02:00
/* The whole table must be 2K aligned */
arch: arm64: correct vector_table alignment The 2K alignment assembler directives should be under 'SECTION_SUBSEC_FUNC(exc_vector_table,_vector_table_section,_vector_table)' Otherwise the _vector_table is actually 0x80 bytes aligned. Signed-off-by: Peng Fan <peng.fan@nxp.com>
2020-11-10 04:10:48 +01:00
.align 11
arch: arm64: Introduce ARM64 (AArch64) architecture Introduce the basic ARM64 architecture support. A new CONFIG_ARM64 symbol is introduced for the new architecture and new cmake / Kconfig files are added to switch between ARM and ARM64. Signed-off-by: Carlo Caione <ccaione@baylibre.com>
2019-11-10 17:17:19 +01:00
/* Current EL with SP0 / Synchronous */
.align 7
arch: arm64: Add el label for vector entry macro Add a new label el for z_arm64_enter_exc to indicate which el the exception comes from. Signed-off-by: Jaxson Han <jaxson.han@arm.com>
2022-10-25 08:55:47 +02:00
z_arm64_enter_exc x0, x1, el1
arch: aarch64: Use SP_EL0 instead of SP_ELx ARM64 is currently using SP_ELx as stack pointer for kernel and threads because everything is running in EL1. If support for EL0 is required, it is necessary to switch to use SP_EL0 instead, that is the only stack pointer that can be accessed at all exception levels by threads. While it is not required to keep using SP_EL0 also during the exceptions, the current code implementation makes it easier to use the same stack pointer as the one used by threads also during the exceptions. This patch moves the code from using SP_ELx to SP_EL0 and fill in the missing entries in the vector table. Signed-off-by: Carlo Caione <ccaione@baylibre.com>
2020-11-06 12:21:52 +01:00
b z_arm64_sync_exc
arch: arm64: Introduce ARM64 (AArch64) architecture Introduce the basic ARM64 architecture support. A new CONFIG_ARM64 symbol is introduced for the new architecture and new cmake / Kconfig files are added to switch between ARM and ARM64. Signed-off-by: Carlo Caione <ccaione@baylibre.com>
2019-11-10 17:17:19 +01:00
/* Current EL with SP0 / IRQ */
.align 7
arch: arm64: Add el label for vector entry macro Add a new label el for z_arm64_enter_exc to indicate which el the exception comes from. Signed-off-by: Jaxson Han <jaxson.han@arm.com>
2022-10-25 08:55:47 +02:00
z_arm64_enter_exc x0, x1, el1
arch: aarch64: Use SP_EL0 instead of SP_ELx ARM64 is currently using SP_ELx as stack pointer for kernel and threads because everything is running in EL1. If support for EL0 is required, it is necessary to switch to use SP_EL0 instead, that is the only stack pointer that can be accessed at all exception levels by threads. While it is not required to keep using SP_EL0 also during the exceptions, the current code implementation makes it easier to use the same stack pointer as the one used by threads also during the exceptions. This patch moves the code from using SP_ELx to SP_EL0 and fill in the missing entries in the vector table. Signed-off-by: Carlo Caione <ccaione@baylibre.com>
2020-11-06 12:21:52 +01:00
#ifdef CONFIG_GEN_SW_ISR_TABLE
b _isr_wrapper
#else
b z_irq_spurious
#endif
arch: arm64: Introduce ARM64 (AArch64) architecture Introduce the basic ARM64 architecture support. A new CONFIG_ARM64 symbol is introduced for the new architecture and new cmake / Kconfig files are added to switch between ARM and ARM64. Signed-off-by: Carlo Caione <ccaione@baylibre.com>
2019-11-10 17:17:19 +01:00
/* Current EL with SP0 / FIQ */
.align 7
b .
/* Current EL with SP0 / SError */
.align 7
arch: arm64: Add el label for vector entry macro Add a new label el for z_arm64_enter_exc to indicate which el the exception comes from. Signed-off-by: Jaxson Han <jaxson.han@arm.com>
2022-10-25 08:55:47 +02:00
z_arm64_enter_exc x0, x1, el1
arch: aarch64: Use SP_EL0 instead of SP_ELx ARM64 is currently using SP_ELx as stack pointer for kernel and threads because everything is running in EL1. If support for EL0 is required, it is necessary to switch to use SP_EL0 instead, that is the only stack pointer that can be accessed at all exception levels by threads. While it is not required to keep using SP_EL0 also during the exceptions, the current code implementation makes it easier to use the same stack pointer as the one used by threads also during the exceptions. This patch moves the code from using SP_ELx to SP_EL0 and fill in the missing entries in the vector table. Signed-off-by: Carlo Caione <ccaione@baylibre.com>
2020-11-06 12:21:52 +01:00
b z_arm64_serror
arch: arm64: Introduce ARM64 (AArch64) architecture Introduce the basic ARM64 architecture support. A new CONFIG_ARM64 symbol is introduced for the new architecture and new cmake / Kconfig files are added to switch between ARM and ARM64. Signed-off-by: Carlo Caione <ccaione@baylibre.com>
2019-11-10 17:17:19 +01:00
/* Current EL with SPx / Synchronous */
.align 7
arch: arm64: Add el label for vector entry macro Add a new label el for z_arm64_enter_exc to indicate which el the exception comes from. Signed-off-by: Jaxson Han <jaxson.han@arm.com>
2022-10-25 08:55:47 +02:00
z_arm64_enter_exc x0, x1, el1
arch: arm64: Rename z_arm64_svc to z_arm64_sync_exc The SVC handler is not only used for the SVC call but in general for all the synchronous exceptions. Reflect this in the handler name. Signed-off-by: Carlo Caione <ccaione@baylibre.com>
2020-09-18 09:19:01 +02:00
b z_arm64_sync_exc
arch: arm64: Introduce ARM64 (AArch64) architecture Introduce the basic ARM64 architecture support. A new CONFIG_ARM64 symbol is introduced for the new architecture and new cmake / Kconfig files are added to switch between ARM and ARM64. Signed-off-by: Carlo Caione <ccaione@baylibre.com>
2019-11-10 17:17:19 +01:00
/* Current EL with SPx / IRQ */
.align 7
arch: arm64: Add el label for vector entry macro Add a new label el for z_arm64_enter_exc to indicate which el the exception comes from. Signed-off-by: Jaxson Han <jaxson.han@arm.com>
2022-10-25 08:55:47 +02:00
z_arm64_enter_exc x0, x1, el1
arch: arm: Fix undefined symbol reference _isr_wrapper is not defined when building with CONFIG_GEN_SW_ISR_TABLE = n. Signed-off-by: Flavio Ceolin <flavio.ceolin@intel.com>
2020-09-22 07:06:20 +02:00
#ifdef CONFIG_GEN_SW_ISR_TABLE
b _isr_wrapper
#else
b z_irq_spurious
#endif
arch: arm64: Introduce ARM64 (AArch64) architecture Introduce the basic ARM64 architecture support. A new CONFIG_ARM64 symbol is introduced for the new architecture and new cmake / Kconfig files are added to switch between ARM and ARM64. Signed-off-by: Carlo Caione <ccaione@baylibre.com>
2019-11-10 17:17:19 +01:00
/* Current EL with SPx / FIQ */
.align 7
b .
/* Current EL with SPx / SError */
.align 7
arch: arm64: Add el label for vector entry macro Add a new label el for z_arm64_enter_exc to indicate which el the exception comes from. Signed-off-by: Jaxson Han <jaxson.han@arm.com>
2022-10-25 08:55:47 +02:00
z_arm64_enter_exc x0, x1, el1
arch: aarch64: Branch SError vector table entry Each vector table entry has 128-bytes to host the vector code. This is not always enough and in general it's better to branch to the actual exception handler elsewhere in memory. Move the SError entry to a branched code. Signed-off-by: Carlo Caione <ccaione@baylibre.com>
2020-11-09 14:17:28 +01:00
b z_arm64_serror
arch: arm64: Introduce ARM64 (AArch64) architecture Introduce the basic ARM64 architecture support. A new CONFIG_ARM64 symbol is introduced for the new architecture and new cmake / Kconfig files are added to switch between ARM and ARM64. Signed-off-by: Carlo Caione <ccaione@baylibre.com>
2019-11-10 17:17:19 +01:00
arch: arm64: Add missing vector table entries The current vector table is missing some (not used) entries. Fill these in for the sake of completeness. Signed-off-by: Carlo Caione <ccaione@baylibre.com>
2020-09-07 15:33:37 +02:00
/* Lower EL using AArch64 / Synchronous */
.align 7
arch: arm64: Add el label for vector entry macro Add a new label el for z_arm64_enter_exc to indicate which el the exception comes from. Signed-off-by: Jaxson Han <jaxson.han@arm.com>
2022-10-25 08:55:47 +02:00
z_arm64_enter_exc x0, x1, el0
aarch64: userspace: Add missing entries in vector table To support exceptions taken in EL0. Signed-off-by: Carlo Caione <ccaione@baylibre.com>
2020-11-26 16:08:22 +01:00
b z_arm64_sync_exc
arch: arm64: Add missing vector table entries The current vector table is missing some (not used) entries. Fill these in for the sake of completeness. Signed-off-by: Carlo Caione <ccaione@baylibre.com>
2020-09-07 15:33:37 +02:00
/* Lower EL using AArch64 / IRQ */
.align 7
arch: arm64: Add el label for vector entry macro Add a new label el for z_arm64_enter_exc to indicate which el the exception comes from. Signed-off-by: Jaxson Han <jaxson.han@arm.com>
2022-10-25 08:55:47 +02:00
z_arm64_enter_exc x0, x1, el0
aarch64: userspace: Add missing entries in vector table To support exceptions taken in EL0. Signed-off-by: Carlo Caione <ccaione@baylibre.com>
2020-11-26 16:08:22 +01:00
#ifdef CONFIG_GEN_SW_ISR_TABLE
b _isr_wrapper
#else
b z_irq_spurious
#endif
arch: arm64: Add missing vector table entries The current vector table is missing some (not used) entries. Fill these in for the sake of completeness. Signed-off-by: Carlo Caione <ccaione@baylibre.com>
2020-09-07 15:33:37 +02:00
/* Lower EL using AArch64 / FIQ */
.align 7
b .
/* Lower EL using AArch64 / SError */
.align 7
arch: arm64: Add el label for vector entry macro Add a new label el for z_arm64_enter_exc to indicate which el the exception comes from. Signed-off-by: Jaxson Han <jaxson.han@arm.com>
2022-10-25 08:55:47 +02:00
z_arm64_enter_exc x0, x1, el0
aarch64: userspace: Add missing entries in vector table To support exceptions taken in EL0. Signed-off-by: Carlo Caione <ccaione@baylibre.com>
2020-11-26 16:08:22 +01:00
b z_arm64_serror
arch: arm64: Add missing vector table entries The current vector table is missing some (not used) entries. Fill these in for the sake of completeness. Signed-off-by: Carlo Caione <ccaione@baylibre.com>
2020-09-07 15:33:37 +02:00
/* Lower EL using AArch32 / Synchronous */
.align 7
b .
/* Lower EL using AArch32 / IRQ */
.align 7
b .
/* Lower EL using AArch32 / FIQ */
.align 7
b .
/* Lower EL using AArch32 / SError */
.align 7
b .
arch: aarch64: Branch SError vector table entry Each vector table entry has 128-bytes to host the vector code. This is not always enough and in general it's better to branch to the actual exception handler elsewhere in memory. Move the SError entry to a branched code. Signed-off-by: Carlo Caione <ccaione@baylibre.com>
2020-11-09 14:17:28 +01:00
GTEXT(z_arm64_serror)
SECTION_FUNC(TEXT, z_arm64_serror)
mov x1, sp
mov x0, #0 /* K_ERR_CPU_EXCEPTION */
aarch64: error: Enable recoverable errors For some kind of faults we want to be able to put in action some corrective actions and keep executing the code. Signed-off-by: Carlo Caione <ccaione@baylibre.com>
2020-11-09 16:03:35 +01:00
bl z_arm64_fatal_error
/* Return here only in case of recoverable error */
aarch64: rationalize exception entry/exit code Each vector slot has room for 32 instructions. The exception context saving needs 15 instructions already. Rather than duplicating those instructions in each out-of-line exception routines, let's store them directly in the vector table. That vector space is otherwise wasted anyway. Move the z_arm64_enter_exc macro into vector_table.S as this is the only place where it should be used. To further reduce code size, let's make z_arm64_exit_exc into a function of its own to avoid code duplication again. It is put in vector_table.S as this is the most logical location to go with its z_arm64_enter_exc counterpart. Signed-off-by: Nicolas Pitre <npitre@baylibre.com>
2021-02-23 20:48:26 +01:00
b z_arm64_exit_exc
arch: arm64: Enable safe exception stack This commit mainly enable the safe exception stack including the stack switch. Init the safe exception stack by calling z_arm64_safe_exception_stack during the boot stage on every core. Also, tweaks several files to properly switch the mode with different cases. 1) The same as before, when executing in userspace, SP_EL0 holds the user stack and SP_EL1 holds the privileged stack, using EL1h mode. 2) When entering exception from EL0 then SP_EL0 will be saved in the _esf_t structure. SP_EL1 will be the current SP, then retrieves the safe exception stack to SP_EL0, making sure the always pointing to safe exception stack as long as the system running in kernel space. 3) When exiting exception from EL1 to EL0 then SP_EL0 will be restored from the stack value previously saved in the _esf_t structure. Still at EL1h mode. 4) Either entering or exiting exception from EL1 to EL1, SP_EL0 will keep holding the safe exception stack unchanged as memtioned above. 5) Do a quick stack check every time entering the exception from EL1 to EL1. If check fail, set SP_EL1 to safe exception stack, and then handle the fatal error. Overall, the exception from user mode will be handled with kernel stack at the assumption that it is impossible the stackoverflow happens at the entry of exception from EL0 to EL1. However the exception from kernel mode will be firstly checked with the safe exception stack to see if the kernel stack overflows, because the exception might be triggered by stack invalid accessing. Signed-off-by: Jaxson Han <jaxson.han@arm.com>
2022-11-01 09:11:45 +01:00
#ifdef CONFIG_ARM64_SAFE_EXCEPTION_STACK
GTEXT(z_arm64_quick_stack_check)
SECTION_FUNC(TEXT, z_arm64_quick_stack_check)
/*
* x0 is SP_EL1
* Retrieve the current stack limit
*/
get_cpu x16
ldr x17, [x16, #_cpu_offset_to_current_stack_limit]
/*
* If priv sp <= the stack limit, then keep the safe exception stack
* go to the stack overflow process.
*/
cmp x0, x17
/* Restore the sp_el1 */
msr SPSel, #1 // switch sp to sp_el1
sub x0, sp, x0 // x0'' = sp' - x0' = x0
sub sp, sp, x0 // sp'' = sp' - x0 = sp
ble 1f
/*
* If the stack does not overflow, keep using sp_el1, copy the original
* x16, x17, x18, lr from sp_el0 (safe_exception_stack) to sp_el1. So
* the four registers can be restroed directly from sp_el1 without a
* stack mode switch.
*/
mrs x18, sp_el0
ldp x16, x17, [x18, -(___esf_t_SIZEOF - ___esf_t_x16_x17_OFFSET)]
stp x16, x17, [sp, ___esf_t_x16_x17_OFFSET]
ldp x16, x17, [x18, -(___esf_t_SIZEOF - ___esf_t_x18_lr_OFFSET)]
stp x16, x17, [sp, ___esf_t_x18_lr_OFFSET]
ret
1: /*
* If stack overflow, save the current sp and then switch sp to safe
* exception stack
* x16 is still the current _cpu
*/
mrs x18, sp_el0
mov x17, sp
str x17, [x16, #_cpu_offset_to_corrupted_sp]
/*
* switch sp to safe exception stack, which means we handle the fatal
* error with safe exception stack.
*/
sub sp, x18, ___esf_t_SIZEOF
ret
#endif
aarch64: rationalize exception entry/exit code Each vector slot has room for 32 instructions. The exception context saving needs 15 instructions already. Rather than duplicating those instructions in each out-of-line exception routines, let's store them directly in the vector table. That vector space is otherwise wasted anyway. Move the z_arm64_enter_exc macro into vector_table.S as this is the only place where it should be used. To further reduce code size, let's make z_arm64_exit_exc into a function of its own to avoid code duplication again. It is put in vector_table.S as this is the most logical location to go with its z_arm64_enter_exc counterpart. Signed-off-by: Nicolas Pitre <npitre@baylibre.com>
2021-02-23 20:48:26 +01:00
/*
arm64: refer to the link register as "lr" rather than "x30" In ARM parlance, the subroutine call return address is stored in the "link register" or simply lr. Refer to it as lr which is clearer than the anonymous x30 designation. Signed-off-by: Nicolas Pitre <npitre@baylibre.com>
2022-03-16 18:12:39 +01:00
* Restore volatile registers, LR, SPSR_EL1 and ELR_EL1
aarch64: rationalize exception entry/exit code Each vector slot has room for 32 instructions. The exception context saving needs 15 instructions already. Rather than duplicating those instructions in each out-of-line exception routines, let's store them directly in the vector table. That vector space is otherwise wasted anyway. Move the z_arm64_enter_exc macro into vector_table.S as this is the only place where it should be used. To further reduce code size, let's make z_arm64_exit_exc into a function of its own to avoid code duplication again. It is put in vector_table.S as this is the most logical location to go with its z_arm64_enter_exc counterpart. Signed-off-by: Nicolas Pitre <npitre@baylibre.com>
2021-02-23 20:48:26 +01:00
*
* This is the common exit point for z_arm64_sync_exc() and _isr_wrapper().
*/
GTEXT(z_arm64_exit_exc)
SECTION_FUNC(TEXT, z_arm64_exit_exc)
arm64: FPU context switching support This adds FPU sharing support with a lazy context switching algorithm. Every thread is allowed to use FPU/SIMD registers. In fact, the compiler may insert FPU reg accesses in anycontext to optimize even non-FP code unless the -mgeneral-regs-only compiler flag is used, but Zephyr currently doesn't support such a build. It is therefore possible to do FP access in IRS as well with this patch although IRQs are then disabled to prevent nested IRQs in such cases. Because the thread object grows in size, some tests have to be adjusted. Signed-off-by: Nicolas Pitre <npitre@baylibre.com>
2021-04-08 05:31:44 +02:00
#ifdef CONFIG_FPU_SHARING
bl z_arm64_fpu_exit_exc
GTEXT(z_arm64_exit_exc_fpu_done)
arm64: don't create a section for z_arm64_exit_exc_fpu_done Both z_arm64_exit_exc and z_arm64_exit_exc_fpu_done must be within the same section as execution falls through here. If z_arm64_exit_exc_fpu_done creates a section of its own then the linker is free to disjoint the code and we absolutely don't want that. Signed-off-by: Nicolas Pitre <npitre@baylibre.com>
2021-05-03 18:39:11 +02:00
z_arm64_exit_exc_fpu_done:
arm64: FPU context switching support This adds FPU sharing support with a lazy context switching algorithm. Every thread is allowed to use FPU/SIMD registers. In fact, the compiler may insert FPU reg accesses in anycontext to optimize even non-FP code unless the -mgeneral-regs-only compiler flag is used, but Zephyr currently doesn't support such a build. It is therefore possible to do FP access in IRS as well with this patch although IRQs are then disabled to prevent nested IRQs in such cases. Because the thread object grows in size, some tests have to be adjusted. Signed-off-by: Nicolas Pitre <npitre@baylibre.com>
2021-04-08 05:31:44 +02:00
#endif
aarch64: rationalize exception entry/exit code Each vector slot has room for 32 instructions. The exception context saving needs 15 instructions already. Rather than duplicating those instructions in each out-of-line exception routines, let's store them directly in the vector table. That vector space is otherwise wasted anyway. Move the z_arm64_enter_exc macro into vector_table.S as this is the only place where it should be used. To further reduce code size, let's make z_arm64_exit_exc into a function of its own to avoid code duplication again. It is put in vector_table.S as this is the most logical location to go with its z_arm64_enter_exc counterpart. Signed-off-by: Nicolas Pitre <npitre@baylibre.com>
2021-02-23 20:48:26 +01:00
ldp x0, x1, [sp, ___esf_t_spsr_elr_OFFSET]
msr spsr_el1, x0
msr elr_el1, x1
arm64: implement exception depth count Add the exception depth count to tpidrro_el0 and make it available through the arch_exception_depth() accessor. The IN_EL0 flag is now updated unconditionally even if userspace is not configured. Doing otherwise made the code rather hairy and I doubt the overhead is measurable. Signed-off-by: Nicolas Pitre <npitre@baylibre.com>
2021-04-13 06:42:00 +02:00
/* Restore the kernel/user mode flag and decrement exception depth */
arm64: hold curr_cpu instance in tpidrro_el0 Let's fully exploit tpidrro_el0 by storing in it the current CPU's struct _cpu instance alongside the userspace mode flag bit. This greatly simplifies the code needed to get at the cpu structure, and this paves the way to much simpler multi cluster support, as there is no longer the need to decode MPIDR all the time. The same code is used in the !SMP case as there are benefits there too such as avoiding the literal pool, and it looks cleaner. The tpidrro_el0 value is no longer stored in the exception stack frame. Instead, we simply restore the user mode flag based on the SPSR value. This way, more flag bits could be used independently in the future. Signed-off-by: Nicolas Pitre <npitre@baylibre.com>
2021-04-10 05:01:18 +02:00
tst x0, #SPSR_MODE_MASK /* EL0 == 0 */
mrs x0, tpidrro_el0
arm64: implement exception depth count Add the exception depth count to tpidrro_el0 and make it available through the arch_exception_depth() accessor. The IN_EL0 flag is now updated unconditionally even if userspace is not configured. Doing otherwise made the code rather hairy and I doubt the overhead is measurable. Signed-off-by: Nicolas Pitre <npitre@baylibre.com>
2021-04-13 06:42:00 +02:00
mov x1, #TPIDRROEL0_EXC_UNIT
orr x2, x0, #TPIDRROEL0_IN_EL0
csel x0, x2, x0, eq
sub x0, x0, x1
aarch64: userspace: Introduce arch_is_user_context The arch_is_user_context() function is relying on the content of the tpidrro_el0 register to determine whether we are in user context or not. This register is set to '1' when in EL1 and set back to '0' when user threads are running in userspace. Signed-off-by: Carlo Caione <ccaione@baylibre.com>
2020-11-25 15:54:12 +01:00
msr tpidrro_el0, x0
arch: arm64: Enable safe exception stack This commit mainly enable the safe exception stack including the stack switch. Init the safe exception stack by calling z_arm64_safe_exception_stack during the boot stage on every core. Also, tweaks several files to properly switch the mode with different cases. 1) The same as before, when executing in userspace, SP_EL0 holds the user stack and SP_EL1 holds the privileged stack, using EL1h mode. 2) When entering exception from EL0 then SP_EL0 will be saved in the _esf_t structure. SP_EL1 will be the current SP, then retrieves the safe exception stack to SP_EL0, making sure the always pointing to safe exception stack as long as the system running in kernel space. 3) When exiting exception from EL1 to EL0 then SP_EL0 will be restored from the stack value previously saved in the _esf_t structure. Still at EL1h mode. 4) Either entering or exiting exception from EL1 to EL1, SP_EL0 will keep holding the safe exception stack unchanged as memtioned above. 5) Do a quick stack check every time entering the exception from EL1 to EL1. If check fail, set SP_EL1 to safe exception stack, and then handle the fatal error. Overall, the exception from user mode will be handled with kernel stack at the assumption that it is impossible the stackoverflow happens at the entry of exception from EL0 to EL1. However the exception from kernel mode will be firstly checked with the safe exception stack to see if the kernel stack overflows, because the exception might be triggered by stack invalid accessing. Signed-off-by: Jaxson Han <jaxson.han@arm.com>
2022-11-01 09:11:45 +01:00
#ifdef CONFIG_ARM64_SAFE_EXCEPTION_STACK
bne 1f
ldr x0, [sp, ___esf_t_sp_el0_OFFSET]
msr sp_el0, x0
1:
#endif
aarch64: rationalize exception entry/exit code Each vector slot has room for 32 instructions. The exception context saving needs 15 instructions already. Rather than duplicating those instructions in each out-of-line exception routines, let's store them directly in the vector table. That vector space is otherwise wasted anyway. Move the z_arm64_enter_exc macro into vector_table.S as this is the only place where it should be used. To further reduce code size, let's make z_arm64_exit_exc into a function of its own to avoid code duplication again. It is put in vector_table.S as this is the most logical location to go with its z_arm64_enter_exc counterpart. Signed-off-by: Nicolas Pitre <npitre@baylibre.com>
2021-02-23 20:48:26 +01:00
ldp x0, x1, [sp, ___esf_t_x0_x1_OFFSET]
ldp x2, x3, [sp, ___esf_t_x2_x3_OFFSET]
ldp x4, x5, [sp, ___esf_t_x4_x5_OFFSET]
ldp x6, x7, [sp, ___esf_t_x6_x7_OFFSET]
ldp x8, x9, [sp, ___esf_t_x8_x9_OFFSET]
ldp x10, x11, [sp, ___esf_t_x10_x11_OFFSET]
ldp x12, x13, [sp, ___esf_t_x12_x13_OFFSET]
ldp x14, x15, [sp, ___esf_t_x14_x15_OFFSET]
ldp x16, x17, [sp, ___esf_t_x16_x17_OFFSET]
arm64: refer to the link register as "lr" rather than "x30" In ARM parlance, the subroutine call return address is stored in the "link register" or simply lr. Refer to it as lr which is clearer than the anonymous x30 designation. Signed-off-by: Nicolas Pitre <npitre@baylibre.com>
2022-03-16 18:12:39 +01:00
ldp x18, lr, [sp, ___esf_t_x18_lr_OFFSET]
aarch64: rationalize exception entry/exit code Each vector slot has room for 32 instructions. The exception context saving needs 15 instructions already. Rather than duplicating those instructions in each out-of-line exception routines, let's store them directly in the vector table. That vector space is otherwise wasted anyway. Move the z_arm64_enter_exc macro into vector_table.S as this is the only place where it should be used. To further reduce code size, let's make z_arm64_exit_exc into a function of its own to avoid code duplication again. It is put in vector_table.S as this is the most logical location to go with its z_arm64_enter_exc counterpart. Signed-off-by: Nicolas Pitre <npitre@baylibre.com>
2021-02-23 20:48:26 +01:00
arm64: Add frame-pointer based stack unwinding Add the frame-pointer based stack unwinding on ARM64. This is a typical output with the feature enabled: *** Booting Zephyr OS build zephyr-v3.4.0-1029-gae22ff648c16 *** E: ELR_ELn: 0x00000000400011c8 E: ESR_ELn: 0x0000000096000046 E: EC: 0x25 (Data Abort taken without a change in Exception level) E: IL: 0x1 E: ISS: 0x46 E: FAR_ELn: 0x0000000000000000 E: TPIDRRO: 0x0100000040011650 E: x0: 0x0000000000000000 x1: 0x0000000000000003 E: x2: 0x0000000000000003 x3: 0x000000004005c6a0 E: x4: 0x0000000000000000 x5: 0x000000004005c7f0 E: x6: 0x0000000048000000 x7: 0x0000000048000000 E: x8: 0x0000000000000005 x9: 0x0000000000000000 E: x10: 0x0000000000000000 x11: 0x0000000000000000 E: x12: 0x0000000000000000 x13: 0x0000000000000000 E: x14: 0x0000000000000000 x15: 0x0000000000000000 E: x16: 0x0000000040004290 x17: 0x0000000000000000 E: x18: 0x0000000000000000 lr: 0x0000000040001208 E: E: backtrace 0: fp: 0x000000004005c690 lr: 0x0000000040001270 E: backtrace 1: fp: 0x000000004005c6b0 lr: 0x0000000040001290 E: backtrace 2: fp: 0x000000004005c7d0 lr: 0x0000000040004ac0 E: backtrace 3: fp: 0x000000004005c7f0 lr: 0x00000000400013a4 E: backtrace 4: fp: 0x000000004005c800 lr: 0x0000000000000000 E: E: >>> ZEPHYR FATAL ERROR 0: CPU exception on CPU 0 E: Current thread: 0x40011310 (unknown) E: Halting system Signed-off-by: Carlo Caione <ccaione@baylibre.com>
2023-07-13 09:03:26 +02:00
#ifdef CONFIG_ARM64_ENABLE_FRAME_POINTER
ldr x29, [sp, ___esf_t_fp_OFFSET]
#endif
aarch64: rationalize exception entry/exit code Each vector slot has room for 32 instructions. The exception context saving needs 15 instructions already. Rather than duplicating those instructions in each out-of-line exception routines, let's store them directly in the vector table. That vector space is otherwise wasted anyway. Move the z_arm64_enter_exc macro into vector_table.S as this is the only place where it should be used. To further reduce code size, let's make z_arm64_exit_exc into a function of its own to avoid code duplication again. It is put in vector_table.S as this is the most logical location to go with its z_arm64_enter_exc counterpart. Signed-off-by: Nicolas Pitre <npitre@baylibre.com>
2021-02-23 20:48:26 +01:00
add sp, sp, ___esf_t_SIZEOF
/*
* In general in the ELR_EL1 register we can find:
*
* - The address of ret in z_arm64_call_svc()
* - The address of the next instruction at the time of the IRQ when the
* thread was switched out.
* - The address of z_thread_entry() for new threads (see thread.c).
*/
eret
Copy permalink