michaelh/zephyr
1
0
Fork 0
Code Releases Activity
zephyr/arch/riscv/core/isr.S

647 lines
17 KiB
ArmAsm
Raw Normal View History

arch: added support for the riscv32 architecture RISC-V is an open-source instruction set architecture. Added support for the 32bit version of RISC-V to Zephyr. 1) exceptions/interrupts/faults are handled at the architecture level via the __irq_wrapper handler. Context saving/restoring of registers can be handled at both architecture and SOC levels. If SOC-specific registers need to be saved, SOC level needs to provide __soc_save_context and __soc_restore_context functions that shall be accounted by the architecture level, when corresponding config variable RISCV_SOC_CONTEXT_SAVE is set. 2) As RISC-V architecture does not provide a clear ISA specification about interrupt handling, each RISC-V SOC handles it in its own way. Hence, at the architecture level, the __irq_wrapper handler expects the following functions to be provided by the SOC level: __soc_is_irq: to check if the exception is the result of an interrupt or not. __soc_handle_irq: handle pending IRQ at SOC level (ex: clear pending IRQ in SOC-specific IRQ register) 3) Thread/task scheduling, as well as IRQ offloading are handled via the RISC-V system call ("ecall"), which is also handled via the __irq_wrapper handler. The _Swap asm function just calls "ecall" to generate an exception. 4) As there is no conventional way of handling CPU power save in RISC-V, the default nano_cpu_idle and nano_cpu_atomic_idle functions just unlock interrupts and return to the caller, without issuing any CPU power saving instruction. Nonetheless, to allow SOC-level to implement proper CPU power save, nano_cpu_idle and nano_cpu_atomic_idle functions are defined as __weak at the architecture level. Change-Id: I980a161d0009f3f404ad22b226a6229fbb492389 Signed-off-by: Jean-Paul Etienne <fractalclone@gmail.com>
2017-01-11 00:24:30 +01:00
/*
* Copyright (c) 2016 Jean-Paul Etienne <fractalclone@gmail.com>
arch: riscv32: provide a general mechanism for saving SoC context RISC-V permits myriad extensions to the ISA, any of which may imply additional context that must be saved and restored on ISR entry and exit. The current in-tree example is the Pulpino core, which has extra registers used by ISA extensions for running loops that shouldn't get clobbered by an ISR. This is currently supported by including pulpino-specific definitions in the generic architecture code. This works, but it's a bit inelegant and is something of a layering violation. A more generic mechanism is required to support other RISC-V SoCs with similar requirements without cluttering the arch code too much. Provide that by extending the semantics of the existing CONFIG_RISCV_SOC_CONTEXT_SAVE option to allow other SoCs to allocate space for saving and restoring their own state, promoting the currently pulpino-specific __soc_save_context / __soc_restore_context routines to a RISC-V arch API. The cost of making this generic is two more instructions in each ISR to pass the SoC specific context to these routines in a0 rather than just assuming the stack points to the right place. This is minimal, and should have been done anyway to keep with the ABI. As a first (and currently only in-tree) customer, convert the Pulpino SoC code to this new mechanism. Signed-off-by: Marti Bolivar <marti@foundries.io>
2018-11-01 22:25:13 -06:00
* Copyright (c) 2018 Foundries.io Ltd
arch: riscv: add memory protection support The IRQ handler has had a major changes to manage syscall, reschedule and interrupt from user thread and stack guard. Add userspace support: - Use a global variable to know if the current execution is user or machine. The location of this variable is read only for all user thread and read/write for kernel thread. - Memory shared is supported. - Use dynamic allocation to optimize PMP slot usage. If the area size is a power of 2, only one PMP slot is used, else 2 are used. Add stack guard support: - Use MPRV bit to force PMP rules to machine mode execution. - IRQ stack have a locked stack guard to avoid re-write PMP configuration registers for each interruption and then win some cycle. - The IRQ stack is used as "temporary" stack at the beginning of IRQ handler to save current ESF. That avoid to trigger write fault on thread stack during store ESF which that call IRQ handler to infinity. - A stack guard is also setup for privileged stack of a user thread. Thread: - A PMP setup is specific to each thread. PMP setup are saved in each thread structure to improve reschedule performance. Signed-off-by: Alexandre Mergnat <amergnat@baylibre.com> Reviewed-by: Nicolas Royer <nroyer@baylibre.com>
2020-07-21 16:00:39 +02:00
* Copyright (c) 2020 BayLibre, SAS
arch: added support for the riscv32 architecture RISC-V is an open-source instruction set architecture. Added support for the 32bit version of RISC-V to Zephyr. 1) exceptions/interrupts/faults are handled at the architecture level via the __irq_wrapper handler. Context saving/restoring of registers can be handled at both architecture and SOC levels. If SOC-specific registers need to be saved, SOC level needs to provide __soc_save_context and __soc_restore_context functions that shall be accounted by the architecture level, when corresponding config variable RISCV_SOC_CONTEXT_SAVE is set. 2) As RISC-V architecture does not provide a clear ISA specification about interrupt handling, each RISC-V SOC handles it in its own way. Hence, at the architecture level, the __irq_wrapper handler expects the following functions to be provided by the SOC level: __soc_is_irq: to check if the exception is the result of an interrupt or not. __soc_handle_irq: handle pending IRQ at SOC level (ex: clear pending IRQ in SOC-specific IRQ register) 3) Thread/task scheduling, as well as IRQ offloading are handled via the RISC-V system call ("ecall"), which is also handled via the __irq_wrapper handler. The _Swap asm function just calls "ecall" to generate an exception. 4) As there is no conventional way of handling CPU power save in RISC-V, the default nano_cpu_idle and nano_cpu_atomic_idle functions just unlock interrupts and return to the caller, without issuing any CPU power saving instruction. Nonetheless, to allow SOC-level to implement proper CPU power save, nano_cpu_idle and nano_cpu_atomic_idle functions are defined as __weak at the architecture level. Change-Id: I980a161d0009f3f404ad22b226a6229fbb492389 Signed-off-by: Jean-Paul Etienne <fractalclone@gmail.com>
2017-01-11 00:24:30 +01:00
*
license: Replace Apache boilerplate with SPDX tag Replace the existing Apache 2.0 boilerplate header with an SPDX tag throughout the zephyr code tree. This patch was generated via a script run over the master branch. Also updated doc/porting/application.rst that had a dependency on line numbers in a literal include. Manually updated subsys/logging/sys_log.c that had a malformed header in the original file. Also cleanup several cases that already had a SPDX tag and we either got a duplicate or missed updating. Jira: ZEP-1457 Change-Id: I6131a1d4ee0e58f5b938300c2d2fc77d2e69572c Signed-off-by: David B. Kinder <david.b.kinder@intel.com> Signed-off-by: Kumar Gala <kumar.gala@linaro.org>
2017-01-18 17:01:01 -08:00
* SPDX-License-Identifier: Apache-2.0
arch: added support for the riscv32 architecture RISC-V is an open-source instruction set architecture. Added support for the 32bit version of RISC-V to Zephyr. 1) exceptions/interrupts/faults are handled at the architecture level via the __irq_wrapper handler. Context saving/restoring of registers can be handled at both architecture and SOC levels. If SOC-specific registers need to be saved, SOC level needs to provide __soc_save_context and __soc_restore_context functions that shall be accounted by the architecture level, when corresponding config variable RISCV_SOC_CONTEXT_SAVE is set. 2) As RISC-V architecture does not provide a clear ISA specification about interrupt handling, each RISC-V SOC handles it in its own way. Hence, at the architecture level, the __irq_wrapper handler expects the following functions to be provided by the SOC level: __soc_is_irq: to check if the exception is the result of an interrupt or not. __soc_handle_irq: handle pending IRQ at SOC level (ex: clear pending IRQ in SOC-specific IRQ register) 3) Thread/task scheduling, as well as IRQ offloading are handled via the RISC-V system call ("ecall"), which is also handled via the __irq_wrapper handler. The _Swap asm function just calls "ecall" to generate an exception. 4) As there is no conventional way of handling CPU power save in RISC-V, the default nano_cpu_idle and nano_cpu_atomic_idle functions just unlock interrupts and return to the caller, without issuing any CPU power saving instruction. Nonetheless, to allow SOC-level to implement proper CPU power save, nano_cpu_idle and nano_cpu_atomic_idle functions are defined as __weak at the architecture level. Change-Id: I980a161d0009f3f404ad22b226a6229fbb492389 Signed-off-by: Jean-Paul Etienne <fractalclone@gmail.com>
2017-01-11 00:24:30 +01:00
*/
asm: update files with <zephyr/...> include prefix Assembler files were not migrated with the new <zephyr/...> prefix. Note that the conversion has been scripted, refer to #45388 for more details. Signed-off-by: Gerard Marull-Paretas <gerard.marull@nordicsemi.no>
2022-05-09 13:56:13 +02:00
#include <zephyr/toolchain.h>
#include <zephyr/linker/sections.h>
arch: added support for the riscv32 architecture RISC-V is an open-source instruction set architecture. Added support for the 32bit version of RISC-V to Zephyr. 1) exceptions/interrupts/faults are handled at the architecture level via the __irq_wrapper handler. Context saving/restoring of registers can be handled at both architecture and SOC levels. If SOC-specific registers need to be saved, SOC level needs to provide __soc_save_context and __soc_restore_context functions that shall be accounted by the architecture level, when corresponding config variable RISCV_SOC_CONTEXT_SAVE is set. 2) As RISC-V architecture does not provide a clear ISA specification about interrupt handling, each RISC-V SOC handles it in its own way. Hence, at the architecture level, the __irq_wrapper handler expects the following functions to be provided by the SOC level: __soc_is_irq: to check if the exception is the result of an interrupt or not. __soc_handle_irq: handle pending IRQ at SOC level (ex: clear pending IRQ in SOC-specific IRQ register) 3) Thread/task scheduling, as well as IRQ offloading are handled via the RISC-V system call ("ecall"), which is also handled via the __irq_wrapper handler. The _Swap asm function just calls "ecall" to generate an exception. 4) As there is no conventional way of handling CPU power save in RISC-V, the default nano_cpu_idle and nano_cpu_atomic_idle functions just unlock interrupts and return to the caller, without issuing any CPU power saving instruction. Nonetheless, to allow SOC-level to implement proper CPU power save, nano_cpu_idle and nano_cpu_atomic_idle functions are defined as __weak at the architecture level. Change-Id: I980a161d0009f3f404ad22b226a6229fbb492389 Signed-off-by: Jean-Paul Etienne <fractalclone@gmail.com>
2017-01-11 00:24:30 +01:00
#include <offsets_short.h>
asm: update files with <zephyr/...> include prefix Assembler files were not migrated with the new <zephyr/...> prefix. Note that the conversion has been scripted, refer to #45388 for more details. Signed-off-by: Gerard Marull-Paretas <gerard.marull@nordicsemi.no>
2022-05-09 13:56:13 +02:00
#include <zephyr/arch/cpu.h>
#include <zephyr/sys/util.h>
#include <zephyr/kernel.h>
#include <zephyr/syscall.h>
#include <zephyr/arch/riscv/csr.h>
#include <zephyr/arch/riscv/syscall.h>
riscv: better abstraction for register-wide load/store opcodes Those are prominent enough that having RV_OP_LOADREG and RV_OP_STOREREG shouting at you all over the place is rather unpleasant and bad taste. Let's create pseudo-instructions of our own with assembler macros rather than preprocessor defines and only in assembly scope. This makes the asm code way more uniform and readable. Signed-off-by: Nicolas Pitre <npitre@baylibre.com>
2022-02-22 14:09:55 -05:00
#include "asm_macros.inc"
riscv: Add support for floating point This change adds full shared floating point support for the RISCV architecture with minimal impact on threads with floating point support not enabled. Signed-off-by: Corey Wharton <coreyw7@fb.com>
2020-03-11 18:15:29 -07:00
riscv: Add support for hardware stacking / unstacking Some RISC-V SoCs implement a mechanism for hardware supported stacking / unstacking of registers during ISR / exceptions. What happens is that on ISR / exception entry part of the context is automatically saved by the hardware on the stack without software intervention, and the same part of the context is restored by the hardware usually on mret. This is currently not yet supported by Zephyr, where the full context must be saved by software in the full fledged ESF. This patcheset is trying to address exactly this case. At least three things are needed to support in a general fashion this problem: (1) a way to store in software only the part of the ESF not already stacked by hardware, (2) a way to restore in software only the part of the context that is not going to be restored by hardware and (3) a way to define a custom ESF. Point (3) is important because the full ESF frame is now composed by a custom part depending on the hardware (that can choose which register to stack / unstack and the order they are saved onto the stack) and a part defined in software for the remaining part of the context. In this patch a new CONFIG_RISCV_SOC_HAS_ISR_STACKING is introduced that enables the code path supporting the three points by the mean of three macros that must be implemented by the user in a soc_stacking.h file: SOC_ISR_SW_STACKING, SOC_ISR_SW_UNSTACKING and SOC_ISR_STACKING_ESF (refer to the symbol help for more details). This is an example of soc_isr_stacking.h for an hardware that doesn't do any hardware stacking / unstacking but everything is managed in software: #ifndef __SOC_ISR_STACKING #define __SOC_ISR_STACKING #if !defined(_ASMLANGUAGE) #define SOC_ISR_STACKING_ESF_DECLARE \ struct __esf { \ unsigned long ra; \ unsigned long t0; \ unsigned long t1; \ unsigned long t2; \ unsigned long t3; \ unsigned long t4; \ unsigned long t5; \ unsigned long t6; \ unsigned long a0; \ unsigned long a1; \ unsigned long a2; \ unsigned long a3; \ unsigned long a4; \ unsigned long a5; \ unsigned long a6; \ unsigned long a7; \ unsigned long mepc; \ unsigned long mstatus; \ unsigned long s0; \ } __aligned(16) #else #define SOC_ISR_SW_STACKING \ addi sp, sp, -__z_arch_esf_t_SIZEOF; \ DO_CALLER_SAVED(sr); #define SOC_ISR_SW_UNSTACKING \ DO_CALLER_SAVED(lr); #endif /* _ASMLANGUAGE */ #endif /* __SOC_ISR_STACKING */ Signed-off-by: Carlo Caione <ccaione@baylibre.com>
2022-12-27 20:14:51 +01:00
#ifdef CONFIG_RISCV_SOC_HAS_ISR_STACKING
#include <soc_isr_stacking.h>
#endif
riscv: integrate the new FPU context switching support FPU context switching is always performed on demand through the FPU access exception handler. Actual task switching only grants or denies FPU access depending on the current FPU owner. Because RISC-V doesn't have a dedicated FPU access exception, we must catch the Illegal Instruction exception and look for actual FP opcodes. There is no longer a need to allocate FPU storage on the stack for every exception making esf smaller and stack overflows less likely. Signed-off-by: Nicolas Pitre <npitre@baylibre.com>
2023-01-17 23:32:21 -05:00
/* Convenience macro for loading/storing register states. */
riscv: new TLS-based arch_is_user_context() implementation This reverts the bulk of commit c8bfc2afda2c ("riscv: make arch_is_user_context() SMP compatible") and replaces it with a flag stored in the thread local storage (TLS) area, therefore making TLS mandatory for userspace support on RISC-V. This has many advantages: - The tp (x4) register is already dedicated by the standard for this purpose, making TLS support almost free. - This is very efficient, requiring only a single instruction to clear and 2 instructions to set. - This makes the SMP case much more efficient. No need for funky exception code any longer. - SMP and non-SMP now use the same implementation making maintenance easier. - The is_user_mode variable no longer requires a dedicated PMP mapping and therefore freeing one PMP slot for other purposes. Signed-off-by: Nicolas Pitre <npitre@baylibre.com> 5f65dbcc9dab3d39473b05397e05.
2022-06-07 09:37:59 -04:00
#define DO_CALLER_SAVED(op) \
riscv: abstract RV32E register access ... and avoid macro duplication. Signed-off-by: Nicolas Pitre <npitre@baylibre.com>
2022-06-15 18:41:36 -04:00
RV_E( op t0, __z_arch_esf_t_t0_OFFSET(sp) );\
riscv: new TLS-based arch_is_user_context() implementation This reverts the bulk of commit c8bfc2afda2c ("riscv: make arch_is_user_context() SMP compatible") and replaces it with a flag stored in the thread local storage (TLS) area, therefore making TLS mandatory for userspace support on RISC-V. This has many advantages: - The tp (x4) register is already dedicated by the standard for this purpose, making TLS support almost free. - This is very efficient, requiring only a single instruction to clear and 2 instructions to set. - This makes the SMP case much more efficient. No need for funky exception code any longer. - SMP and non-SMP now use the same implementation making maintenance easier. - The is_user_mode variable no longer requires a dedicated PMP mapping and therefore freeing one PMP slot for other purposes. Signed-off-by: Nicolas Pitre <npitre@baylibre.com> 5f65dbcc9dab3d39473b05397e05.
2022-06-07 09:37:59 -04:00
RV_E( op t1, __z_arch_esf_t_t1_OFFSET(sp) );\
riscv: abstract RV32E register access ... and avoid macro duplication. Signed-off-by: Nicolas Pitre <npitre@baylibre.com>
2022-06-15 18:41:36 -04:00
RV_E( op t2, __z_arch_esf_t_t2_OFFSET(sp) );\
RV_I( op t3, __z_arch_esf_t_t3_OFFSET(sp) );\
RV_I( op t4, __z_arch_esf_t_t4_OFFSET(sp) );\
RV_I( op t5, __z_arch_esf_t_t5_OFFSET(sp) );\
RV_I( op t6, __z_arch_esf_t_t6_OFFSET(sp) );\
RV_E( op a0, __z_arch_esf_t_a0_OFFSET(sp) );\
RV_E( op a1, __z_arch_esf_t_a1_OFFSET(sp) );\
RV_E( op a2, __z_arch_esf_t_a2_OFFSET(sp) );\
RV_E( op a3, __z_arch_esf_t_a3_OFFSET(sp) );\
RV_E( op a4, __z_arch_esf_t_a4_OFFSET(sp) );\
RV_E( op a5, __z_arch_esf_t_a5_OFFSET(sp) );\
RV_I( op a6, __z_arch_esf_t_a6_OFFSET(sp) );\
RV_I( op a7, __z_arch_esf_t_a7_OFFSET(sp) );\
RV_E( op ra, __z_arch_esf_t_ra_OFFSET(sp) )
arch: riscv: add memory protection support The IRQ handler has had a major changes to manage syscall, reschedule and interrupt from user thread and stack guard. Add userspace support: - Use a global variable to know if the current execution is user or machine. The location of this variable is read only for all user thread and read/write for kernel thread. - Memory shared is supported. - Use dynamic allocation to optimize PMP slot usage. If the area size is a power of 2, only one PMP slot is used, else 2 are used. Add stack guard support: - Use MPRV bit to force PMP rules to machine mode execution. - IRQ stack have a locked stack guard to avoid re-write PMP configuration registers for each interruption and then win some cycle. - The IRQ stack is used as "temporary" stack at the beginning of IRQ handler to save current ESF. That avoid to trigger write fault on thread stack during store ESF which that call IRQ handler to infinity. - A stack guard is also setup for privileged stack of a user thread. Thread: - A PMP setup is specific to each thread. PMP setup are saved in each thread structure to improve reschedule performance. Signed-off-by: Alexandre Mergnat <amergnat@baylibre.com> Reviewed-by: Nicolas Royer <nroyer@baylibre.com>
2020-07-21 16:00:39 +02:00
riscv: decouple the Zephyr CPU number from the hart ID Currently it is assumed that Zephyr CPU numbers match their hartid value one for one. This assumption was relied upon to efficiently retrieve the current CPU's `struct _cpu` pointer. People are starting to have systems with a mix of different usage for each CPU and such assumption may no longer be true. Let's completely decouple the hartid from the Zephyr CPU number by stuffing each CPU's `struct _cpu` pointer in their respective scratch register instead. `arch_curr_cpu()` becomes more efficient as well. Since the scratch register was previously used to store userspace's exception stack pointer, that is now moved into `struct _cpu_arch` which implied minor user space entry code cleanup and rationalization. Signed-off-by: Nicolas Pitre <npitre@baylibre.com>
2023-01-06 17:46:21 -05:00
.macro get_current_cpu dst
#if defined(CONFIG_SMP) || defined(CONFIG_USERSPACE)
csrr \dst, mscratch
riscv: isr.S: compute _current_cpu using CPU number on SMP To do so efficiently on systems without the mul instruction, we use shifts and adds which is faster and sometimes smaller than a plain loop. Signed-off-by: Nicolas Pitre <npitre@baylibre.com>
2022-03-09 23:47:38 -05:00
#else
riscv: decouple the Zephyr CPU number from the hart ID Currently it is assumed that Zephyr CPU numbers match their hartid value one for one. This assumption was relied upon to efficiently retrieve the current CPU's `struct _cpu` pointer. People are starting to have systems with a mix of different usage for each CPU and such assumption may no longer be true. Let's completely decouple the hartid from the Zephyr CPU number by stuffing each CPU's `struct _cpu` pointer in their respective scratch register instead. `arch_curr_cpu()` becomes more efficient as well. Since the scratch register was previously used to store userspace's exception stack pointer, that is now moved into `struct _cpu_arch` which implied minor user space entry code cleanup and rationalization. Signed-off-by: Nicolas Pitre <npitre@baylibre.com>
2023-01-06 17:46:21 -05:00
la \dst, _kernel + ___kernel_t_cpus_OFFSET
riscv: isr.S: compute _current_cpu using CPU number on SMP To do so efficiently on systems without the mul instruction, we use shifts and adds which is faster and sometimes smaller than a plain loop. Signed-off-by: Nicolas Pitre <npitre@baylibre.com>
2022-03-09 23:47:38 -05:00
#endif
riscv: decouple the Zephyr CPU number from the hart ID Currently it is assumed that Zephyr CPU numbers match their hartid value one for one. This assumption was relied upon to efficiently retrieve the current CPU's `struct _cpu` pointer. People are starting to have systems with a mix of different usage for each CPU and such assumption may no longer be true. Let's completely decouple the hartid from the Zephyr CPU number by stuffing each CPU's `struct _cpu` pointer in their respective scratch register instead. `arch_curr_cpu()` becomes more efficient as well. Since the scratch register was previously used to store userspace's exception stack pointer, that is now moved into `struct _cpu_arch` which implied minor user space entry code cleanup and rationalization. Signed-off-by: Nicolas Pitre <npitre@baylibre.com>
2023-01-06 17:46:21 -05:00
.endm
riscv: isr.S: compute _current_cpu using CPU number on SMP To do so efficiently on systems without the mul instruction, we use shifts and adds which is faster and sometimes smaller than a plain loop. Signed-off-by: Nicolas Pitre <npitre@baylibre.com>
2022-03-09 23:47:38 -05:00
arch: added support for the riscv32 architecture RISC-V is an open-source instruction set architecture. Added support for the 32bit version of RISC-V to Zephyr. 1) exceptions/interrupts/faults are handled at the architecture level via the __irq_wrapper handler. Context saving/restoring of registers can be handled at both architecture and SOC levels. If SOC-specific registers need to be saved, SOC level needs to provide __soc_save_context and __soc_restore_context functions that shall be accounted by the architecture level, when corresponding config variable RISCV_SOC_CONTEXT_SAVE is set. 2) As RISC-V architecture does not provide a clear ISA specification about interrupt handling, each RISC-V SOC handles it in its own way. Hence, at the architecture level, the __irq_wrapper handler expects the following functions to be provided by the SOC level: __soc_is_irq: to check if the exception is the result of an interrupt or not. __soc_handle_irq: handle pending IRQ at SOC level (ex: clear pending IRQ in SOC-specific IRQ register) 3) Thread/task scheduling, as well as IRQ offloading are handled via the RISC-V system call ("ecall"), which is also handled via the __irq_wrapper handler. The _Swap asm function just calls "ecall" to generate an exception. 4) As there is no conventional way of handling CPU power save in RISC-V, the default nano_cpu_idle and nano_cpu_atomic_idle functions just unlock interrupts and return to the caller, without issuing any CPU power saving instruction. Nonetheless, to allow SOC-level to implement proper CPU power save, nano_cpu_idle and nano_cpu_atomic_idle functions are defined as __weak at the architecture level. Change-Id: I980a161d0009f3f404ad22b226a6229fbb492389 Signed-off-by: Jean-Paul Etienne <fractalclone@gmail.com>
2017-01-11 00:24:30 +01:00
/* imports */
GDATA(_sw_isr_table)
GTEXT(__soc_is_irq)
GTEXT(__soc_handle_irq)
GTEXT(_Fault)
arch: riscv32: provide a general mechanism for saving SoC context RISC-V permits myriad extensions to the ISA, any of which may imply additional context that must be saved and restored on ISR entry and exit. The current in-tree example is the Pulpino core, which has extra registers used by ISA extensions for running loops that shouldn't get clobbered by an ISR. This is currently supported by including pulpino-specific definitions in the generic architecture code. This works, but it's a bit inelegant and is something of a layering violation. A more generic mechanism is required to support other RISC-V SoCs with similar requirements without cluttering the arch code too much. Provide that by extending the semantics of the existing CONFIG_RISCV_SOC_CONTEXT_SAVE option to allow other SoCs to allocate space for saving and restoring their own state, promoting the currently pulpino-specific __soc_save_context / __soc_restore_context routines to a RISC-V arch API. The cost of making this generic is two more instructions in each ISR to pass the SoC specific context to these routines in a0 rather than just assuming the stack points to the right place. This is minimal, and should have been done anyway to keep with the ABI. As a first (and currently only in-tree) customer, convert the Pulpino SoC code to this new mechanism. Signed-off-by: Marti Bolivar <marti@foundries.io>
2018-11-01 22:25:13 -06:00
#ifdef CONFIG_RISCV_SOC_CONTEXT_SAVE
GTEXT(__soc_save_context)
GTEXT(__soc_restore_context)
#endif /* CONFIG_RISCV_SOC_CONTEXT_SAVE */
arch: added support for the riscv32 architecture RISC-V is an open-source instruction set architecture. Added support for the 32bit version of RISC-V to Zephyr. 1) exceptions/interrupts/faults are handled at the architecture level via the __irq_wrapper handler. Context saving/restoring of registers can be handled at both architecture and SOC levels. If SOC-specific registers need to be saved, SOC level needs to provide __soc_save_context and __soc_restore_context functions that shall be accounted by the architecture level, when corresponding config variable RISCV_SOC_CONTEXT_SAVE is set. 2) As RISC-V architecture does not provide a clear ISA specification about interrupt handling, each RISC-V SOC handles it in its own way. Hence, at the architecture level, the __irq_wrapper handler expects the following functions to be provided by the SOC level: __soc_is_irq: to check if the exception is the result of an interrupt or not. __soc_handle_irq: handle pending IRQ at SOC level (ex: clear pending IRQ in SOC-specific IRQ register) 3) Thread/task scheduling, as well as IRQ offloading are handled via the RISC-V system call ("ecall"), which is also handled via the __irq_wrapper handler. The _Swap asm function just calls "ecall" to generate an exception. 4) As there is no conventional way of handling CPU power save in RISC-V, the default nano_cpu_idle and nano_cpu_atomic_idle functions just unlock interrupts and return to the caller, without issuing any CPU power saving instruction. Nonetheless, to allow SOC-level to implement proper CPU power save, nano_cpu_idle and nano_cpu_atomic_idle functions are defined as __weak at the architecture level. Change-Id: I980a161d0009f3f404ad22b226a6229fbb492389 Signed-off-by: Jean-Paul Etienne <fractalclone@gmail.com>
2017-01-11 00:24:30 +01:00
riscv: exception code mega simplification and optimization Complete revamp of the exception entry code, including syscall handling. Proper syscall frame exception trigger. Many correctness fixes, hacks removal, etc. etc. I tried to make this into several commits, but this stuff is all inter-related and a pain to split. The diffstat summary: 14 files changed, 250 insertions(+), 802 deletions(-) Binary size (before): text data bss dec hex filename 1104 0 0 1104 450 isr.S.obj 64 0 0 64 40 userspace.S.obj Binary size (after): text data bss dec hex filename 600 0 0 600 258 isr.S.obj 36 0 0 36 24 userspace.S.obj Run of samples/userspace/syscall_perf (before): *** Booting Zephyr OS build zephyr-v3.0.0-325-g3748accae018 *** Main Thread started; qemu_riscv32 Supervisor thread started User thread started Supervisor thread(0x80010048): 384 cycles 509 instructions User thread(0x80010140): 77312 cycles 77437 instructions Run of samples/userspace/syscall_perf (after): *** Booting Zephyr OS build zephyr-v3.0.0-326-g4c877a2753b3 *** Main Thread started; qemu_riscv32 Supervisor thread started User thread started Supervisor thread(0x80010048): 384 cycles 509 instructions User thread(0x80010138): 7040 cycles 7165 instructions Yes, that's more than a 10x speed-up! Signed-off-by: Nicolas Pitre <npitre@baylibre.com>
2022-02-24 22:30:03 -05:00
GTEXT(z_riscv_fatal_error)
riscv: implement arch_switch() The move to arch_switch() is a prerequisite for SMP support. Make it optimal without the need for an ECALL roundtrip on every context switch. Performance numbers from tests/benchmarks/sched: Before: unpend 107 ready 102 switch 188 pend 218 tot 615 (avg 615) After: unpend 107 ready 102 switch 170 pend 217 tot 596 (avg 595) Signed-off-by: Nicolas Pitre <npitre@baylibre.com>
2022-03-07 17:01:36 -05:00
GTEXT(z_get_next_switch_handle)
GTEXT(z_riscv_switch)
GTEXT(z_riscv_thread_start)
arch: added support for the riscv32 architecture RISC-V is an open-source instruction set architecture. Added support for the 32bit version of RISC-V to Zephyr. 1) exceptions/interrupts/faults are handled at the architecture level via the __irq_wrapper handler. Context saving/restoring of registers can be handled at both architecture and SOC levels. If SOC-specific registers need to be saved, SOC level needs to provide __soc_save_context and __soc_restore_context functions that shall be accounted by the architecture level, when corresponding config variable RISCV_SOC_CONTEXT_SAVE is set. 2) As RISC-V architecture does not provide a clear ISA specification about interrupt handling, each RISC-V SOC handles it in its own way. Hence, at the architecture level, the __irq_wrapper handler expects the following functions to be provided by the SOC level: __soc_is_irq: to check if the exception is the result of an interrupt or not. __soc_handle_irq: handle pending IRQ at SOC level (ex: clear pending IRQ in SOC-specific IRQ register) 3) Thread/task scheduling, as well as IRQ offloading are handled via the RISC-V system call ("ecall"), which is also handled via the __irq_wrapper handler. The _Swap asm function just calls "ecall" to generate an exception. 4) As there is no conventional way of handling CPU power save in RISC-V, the default nano_cpu_idle and nano_cpu_atomic_idle functions just unlock interrupts and return to the caller, without issuing any CPU power saving instruction. Nonetheless, to allow SOC-level to implement proper CPU power save, nano_cpu_idle and nano_cpu_atomic_idle functions are defined as __weak at the architecture level. Change-Id: I980a161d0009f3f404ad22b226a6229fbb492389 Signed-off-by: Jean-Paul Etienne <fractalclone@gmail.com>
2017-01-11 00:24:30 +01:00
kernel: event_logger: remove kernel_event_logger Move to more generic tracing hooks that can be implemented in different ways and do not interfere with the kernel. Signed-off-by: Anas Nashif <anas.nashif@intel.com>
2018-04-06 07:48:53 -04:00
#ifdef CONFIG_TRACING
debug: tracing: Remove unneeded abstraction Various C and Assembly modules make function calls to z_sys_trace_*. These merely call corresponding functions sys_trace_*. This commit is to simplify these by making direct function calls to the sys_trace_* functions from these modules. Subsequently, the z_sys_trace_* functions are removed. Signed-off-by: Mrinal Sen <msen@oticon.com>
2019-09-19 09:25:19 +02:00
GTEXT(sys_trace_isr_enter)
tracing: riscv: Add missing invoke of sys_trace_isr_exit() Change suggested by @WealianLiao in #41995. Signed-off-by: Robert Szczepanski <rszczepanski@antmicro.com>
2022-05-11 12:29:46 +02:00
GTEXT(sys_trace_isr_exit)
arch: added support for the riscv32 architecture RISC-V is an open-source instruction set architecture. Added support for the 32bit version of RISC-V to Zephyr. 1) exceptions/interrupts/faults are handled at the architecture level via the __irq_wrapper handler. Context saving/restoring of registers can be handled at both architecture and SOC levels. If SOC-specific registers need to be saved, SOC level needs to provide __soc_save_context and __soc_restore_context functions that shall be accounted by the architecture level, when corresponding config variable RISCV_SOC_CONTEXT_SAVE is set. 2) As RISC-V architecture does not provide a clear ISA specification about interrupt handling, each RISC-V SOC handles it in its own way. Hence, at the architecture level, the __irq_wrapper handler expects the following functions to be provided by the SOC level: __soc_is_irq: to check if the exception is the result of an interrupt or not. __soc_handle_irq: handle pending IRQ at SOC level (ex: clear pending IRQ in SOC-specific IRQ register) 3) Thread/task scheduling, as well as IRQ offloading are handled via the RISC-V system call ("ecall"), which is also handled via the __irq_wrapper handler. The _Swap asm function just calls "ecall" to generate an exception. 4) As there is no conventional way of handling CPU power save in RISC-V, the default nano_cpu_idle and nano_cpu_atomic_idle functions just unlock interrupts and return to the caller, without issuing any CPU power saving instruction. Nonetheless, to allow SOC-level to implement proper CPU power save, nano_cpu_idle and nano_cpu_atomic_idle functions are defined as __weak at the architecture level. Change-Id: I980a161d0009f3f404ad22b226a6229fbb492389 Signed-off-by: Jean-Paul Etienne <fractalclone@gmail.com>
2017-01-11 00:24:30 +01:00
#endif
arch: riscv: add memory protection support The IRQ handler has had a major changes to manage syscall, reschedule and interrupt from user thread and stack guard. Add userspace support: - Use a global variable to know if the current execution is user or machine. The location of this variable is read only for all user thread and read/write for kernel thread. - Memory shared is supported. - Use dynamic allocation to optimize PMP slot usage. If the area size is a power of 2, only one PMP slot is used, else 2 are used. Add stack guard support: - Use MPRV bit to force PMP rules to machine mode execution. - IRQ stack have a locked stack guard to avoid re-write PMP configuration registers for each interruption and then win some cycle. - The IRQ stack is used as "temporary" stack at the beginning of IRQ handler to save current ESF. That avoid to trigger write fault on thread stack during store ESF which that call IRQ handler to infinity. - A stack guard is also setup for privileged stack of a user thread. Thread: - A PMP setup is specific to each thread. PMP setup are saved in each thread structure to improve reschedule performance. Signed-off-by: Alexandre Mergnat <amergnat@baylibre.com> Reviewed-by: Nicolas Royer <nroyer@baylibre.com>
2020-07-21 16:00:39 +02:00
#ifdef CONFIG_USERSPACE
riscv: exception code mega simplification and optimization Complete revamp of the exception entry code, including syscall handling. Proper syscall frame exception trigger. Many correctness fixes, hacks removal, etc. etc. I tried to make this into several commits, but this stuff is all inter-related and a pain to split. The diffstat summary: 14 files changed, 250 insertions(+), 802 deletions(-) Binary size (before): text data bss dec hex filename 1104 0 0 1104 450 isr.S.obj 64 0 0 64 40 userspace.S.obj Binary size (after): text data bss dec hex filename 600 0 0 600 258 isr.S.obj 36 0 0 36 24 userspace.S.obj Run of samples/userspace/syscall_perf (before): *** Booting Zephyr OS build zephyr-v3.0.0-325-g3748accae018 *** Main Thread started; qemu_riscv32 Supervisor thread started User thread started Supervisor thread(0x80010048): 384 cycles 509 instructions User thread(0x80010140): 77312 cycles 77437 instructions Run of samples/userspace/syscall_perf (after): *** Booting Zephyr OS build zephyr-v3.0.0-326-g4c877a2753b3 *** Main Thread started; qemu_riscv32 Supervisor thread started User thread started Supervisor thread(0x80010048): 384 cycles 509 instructions User thread(0x80010138): 7040 cycles 7165 instructions Yes, that's more than a 10x speed-up! Signed-off-by: Nicolas Pitre <npitre@baylibre.com>
2022-02-24 22:30:03 -05:00
GDATA(_k_syscall_table)
arch: riscv: add memory protection support The IRQ handler has had a major changes to manage syscall, reschedule and interrupt from user thread and stack guard. Add userspace support: - Use a global variable to know if the current execution is user or machine. The location of this variable is read only for all user thread and read/write for kernel thread. - Memory shared is supported. - Use dynamic allocation to optimize PMP slot usage. If the area size is a power of 2, only one PMP slot is used, else 2 are used. Add stack guard support: - Use MPRV bit to force PMP rules to machine mode execution. - IRQ stack have a locked stack guard to avoid re-write PMP configuration registers for each interruption and then win some cycle. - The IRQ stack is used as "temporary" stack at the beginning of IRQ handler to save current ESF. That avoid to trigger write fault on thread stack during store ESF which that call IRQ handler to infinity. - A stack guard is also setup for privileged stack of a user thread. Thread: - A PMP setup is specific to each thread. PMP setup are saved in each thread structure to improve reschedule performance. Signed-off-by: Alexandre Mergnat <amergnat@baylibre.com> Reviewed-by: Nicolas Royer <nroyer@baylibre.com>
2020-07-21 16:00:39 +02:00
#endif
arch: added support for the riscv32 architecture RISC-V is an open-source instruction set architecture. Added support for the 32bit version of RISC-V to Zephyr. 1) exceptions/interrupts/faults are handled at the architecture level via the __irq_wrapper handler. Context saving/restoring of registers can be handled at both architecture and SOC levels. If SOC-specific registers need to be saved, SOC level needs to provide __soc_save_context and __soc_restore_context functions that shall be accounted by the architecture level, when corresponding config variable RISCV_SOC_CONTEXT_SAVE is set. 2) As RISC-V architecture does not provide a clear ISA specification about interrupt handling, each RISC-V SOC handles it in its own way. Hence, at the architecture level, the __irq_wrapper handler expects the following functions to be provided by the SOC level: __soc_is_irq: to check if the exception is the result of an interrupt or not. __soc_handle_irq: handle pending IRQ at SOC level (ex: clear pending IRQ in SOC-specific IRQ register) 3) Thread/task scheduling, as well as IRQ offloading are handled via the RISC-V system call ("ecall"), which is also handled via the __irq_wrapper handler. The _Swap asm function just calls "ecall" to generate an exception. 4) As there is no conventional way of handling CPU power save in RISC-V, the default nano_cpu_idle and nano_cpu_atomic_idle functions just unlock interrupts and return to the caller, without issuing any CPU power saving instruction. Nonetheless, to allow SOC-level to implement proper CPU power save, nano_cpu_idle and nano_cpu_atomic_idle functions are defined as __weak at the architecture level. Change-Id: I980a161d0009f3f404ad22b226a6229fbb492389 Signed-off-by: Jean-Paul Etienne <fractalclone@gmail.com>
2017-01-11 00:24:30 +01:00
/* exports */
riscv: Rename __irq_wrapper to _isr_wrapper For some reasons RISCV is the only arch where the vector table entry is called __irq_wrapper instead of _isr_wrapper. This is not only a cosmetic change but Zephyr expects the common ISR handler to be called _isr_wrapper (for example when generating the IRQ vector table). Change it. find ./ -type f -exec sed -i 's/__irq_wrapper/_isr_wrapper/g' {} \; Signed-off-by: Carlo Caione <ccaione@baylibre.com>
2022-06-21 10:43:40 +02:00
GTEXT(_isr_wrapper)
arch: added support for the riscv32 architecture RISC-V is an open-source instruction set architecture. Added support for the 32bit version of RISC-V to Zephyr. 1) exceptions/interrupts/faults are handled at the architecture level via the __irq_wrapper handler. Context saving/restoring of registers can be handled at both architecture and SOC levels. If SOC-specific registers need to be saved, SOC level needs to provide __soc_save_context and __soc_restore_context functions that shall be accounted by the architecture level, when corresponding config variable RISCV_SOC_CONTEXT_SAVE is set. 2) As RISC-V architecture does not provide a clear ISA specification about interrupt handling, each RISC-V SOC handles it in its own way. Hence, at the architecture level, the __irq_wrapper handler expects the following functions to be provided by the SOC level: __soc_is_irq: to check if the exception is the result of an interrupt or not. __soc_handle_irq: handle pending IRQ at SOC level (ex: clear pending IRQ in SOC-specific IRQ register) 3) Thread/task scheduling, as well as IRQ offloading are handled via the RISC-V system call ("ecall"), which is also handled via the __irq_wrapper handler. The _Swap asm function just calls "ecall" to generate an exception. 4) As there is no conventional way of handling CPU power save in RISC-V, the default nano_cpu_idle and nano_cpu_atomic_idle functions just unlock interrupts and return to the caller, without issuing any CPU power saving instruction. Nonetheless, to allow SOC-level to implement proper CPU power save, nano_cpu_idle and nano_cpu_atomic_idle functions are defined as __weak at the architecture level. Change-Id: I980a161d0009f3f404ad22b226a6229fbb492389 Signed-off-by: Jean-Paul Etienne <fractalclone@gmail.com>
2017-01-11 00:24:30 +01:00
/* use ABI name of registers for the sake of simplicity */
/*
arch: riscv32: provide a general mechanism for saving SoC context RISC-V permits myriad extensions to the ISA, any of which may imply additional context that must be saved and restored on ISR entry and exit. The current in-tree example is the Pulpino core, which has extra registers used by ISA extensions for running loops that shouldn't get clobbered by an ISR. This is currently supported by including pulpino-specific definitions in the generic architecture code. This works, but it's a bit inelegant and is something of a layering violation. A more generic mechanism is required to support other RISC-V SoCs with similar requirements without cluttering the arch code too much. Provide that by extending the semantics of the existing CONFIG_RISCV_SOC_CONTEXT_SAVE option to allow other SoCs to allocate space for saving and restoring their own state, promoting the currently pulpino-specific __soc_save_context / __soc_restore_context routines to a RISC-V arch API. The cost of making this generic is two more instructions in each ISR to pass the SoC specific context to these routines in a0 rather than just assuming the stack points to the right place. This is minimal, and should have been done anyway to keep with the ABI. As a first (and currently only in-tree) customer, convert the Pulpino SoC code to this new mechanism. Signed-off-by: Marti Bolivar <marti@foundries.io>
2018-11-01 22:25:13 -06:00
* Generic architecture-level IRQ handling, along with callouts to
* SoC-specific routines.
*
* Architecture level IRQ handling includes basic context save/restore
* of standard registers and calling ISRs registered at Zephyr's driver
* level.
*
* Since RISC-V does not completely prescribe IRQ handling behavior,
* implementations vary (some implementations also deviate from
* what standard behavior is defined). Hence, the arch level code expects
* the following functions to be provided at the SOC level:
*
* - __soc_is_irq: decide if we're handling an interrupt or an exception
* - __soc_handle_irq: handle SoC-specific details for a pending IRQ
* (e.g. clear a pending bit in a SoC-specific register)
arch: added support for the riscv32 architecture RISC-V is an open-source instruction set architecture. Added support for the 32bit version of RISC-V to Zephyr. 1) exceptions/interrupts/faults are handled at the architecture level via the __irq_wrapper handler. Context saving/restoring of registers can be handled at both architecture and SOC levels. If SOC-specific registers need to be saved, SOC level needs to provide __soc_save_context and __soc_restore_context functions that shall be accounted by the architecture level, when corresponding config variable RISCV_SOC_CONTEXT_SAVE is set. 2) As RISC-V architecture does not provide a clear ISA specification about interrupt handling, each RISC-V SOC handles it in its own way. Hence, at the architecture level, the __irq_wrapper handler expects the following functions to be provided by the SOC level: __soc_is_irq: to check if the exception is the result of an interrupt or not. __soc_handle_irq: handle pending IRQ at SOC level (ex: clear pending IRQ in SOC-specific IRQ register) 3) Thread/task scheduling, as well as IRQ offloading are handled via the RISC-V system call ("ecall"), which is also handled via the __irq_wrapper handler. The _Swap asm function just calls "ecall" to generate an exception. 4) As there is no conventional way of handling CPU power save in RISC-V, the default nano_cpu_idle and nano_cpu_atomic_idle functions just unlock interrupts and return to the caller, without issuing any CPU power saving instruction. Nonetheless, to allow SOC-level to implement proper CPU power save, nano_cpu_idle and nano_cpu_atomic_idle functions are defined as __weak at the architecture level. Change-Id: I980a161d0009f3f404ad22b226a6229fbb492389 Signed-off-by: Jean-Paul Etienne <fractalclone@gmail.com>
2017-01-11 00:24:30 +01:00
*
arch: riscv32: Fix trivial comment The Kconfig sybmol referenced in the comment is called CONFIG_RISCV_SOC_CONTEXT_SAVE not CONFIG_RISCV_SOC_CONTEXT. Signed-off-by: Kumar Gala <kumar.gala@linaro.org>
2019-01-31 04:40:07 -06:00
* If CONFIG_RISCV_SOC_CONTEXT_SAVE=y, calls to SoC-level context save/restore
arch: riscv32: provide a general mechanism for saving SoC context RISC-V permits myriad extensions to the ISA, any of which may imply additional context that must be saved and restored on ISR entry and exit. The current in-tree example is the Pulpino core, which has extra registers used by ISA extensions for running loops that shouldn't get clobbered by an ISR. This is currently supported by including pulpino-specific definitions in the generic architecture code. This works, but it's a bit inelegant and is something of a layering violation. A more generic mechanism is required to support other RISC-V SoCs with similar requirements without cluttering the arch code too much. Provide that by extending the semantics of the existing CONFIG_RISCV_SOC_CONTEXT_SAVE option to allow other SoCs to allocate space for saving and restoring their own state, promoting the currently pulpino-specific __soc_save_context / __soc_restore_context routines to a RISC-V arch API. The cost of making this generic is two more instructions in each ISR to pass the SoC specific context to these routines in a0 rather than just assuming the stack points to the right place. This is minimal, and should have been done anyway to keep with the ABI. As a first (and currently only in-tree) customer, convert the Pulpino SoC code to this new mechanism. Signed-off-by: Marti Bolivar <marti@foundries.io>
2018-11-01 22:25:13 -06:00
* routines are also made here. For details, see the Kconfig help text.
arch: added support for the riscv32 architecture RISC-V is an open-source instruction set architecture. Added support for the 32bit version of RISC-V to Zephyr. 1) exceptions/interrupts/faults are handled at the architecture level via the __irq_wrapper handler. Context saving/restoring of registers can be handled at both architecture and SOC levels. If SOC-specific registers need to be saved, SOC level needs to provide __soc_save_context and __soc_restore_context functions that shall be accounted by the architecture level, when corresponding config variable RISCV_SOC_CONTEXT_SAVE is set. 2) As RISC-V architecture does not provide a clear ISA specification about interrupt handling, each RISC-V SOC handles it in its own way. Hence, at the architecture level, the __irq_wrapper handler expects the following functions to be provided by the SOC level: __soc_is_irq: to check if the exception is the result of an interrupt or not. __soc_handle_irq: handle pending IRQ at SOC level (ex: clear pending IRQ in SOC-specific IRQ register) 3) Thread/task scheduling, as well as IRQ offloading are handled via the RISC-V system call ("ecall"), which is also handled via the __irq_wrapper handler. The _Swap asm function just calls "ecall" to generate an exception. 4) As there is no conventional way of handling CPU power save in RISC-V, the default nano_cpu_idle and nano_cpu_atomic_idle functions just unlock interrupts and return to the caller, without issuing any CPU power saving instruction. Nonetheless, to allow SOC-level to implement proper CPU power save, nano_cpu_idle and nano_cpu_atomic_idle functions are defined as __weak at the architecture level. Change-Id: I980a161d0009f3f404ad22b226a6229fbb492389 Signed-off-by: Jean-Paul Etienne <fractalclone@gmail.com>
2017-01-11 00:24:30 +01:00
*/
/*
* Handler called upon each exception/interrupt/fault
*/
riscv: Rename __irq_wrapper to _isr_wrapper For some reasons RISCV is the only arch where the vector table entry is called __irq_wrapper instead of _isr_wrapper. This is not only a cosmetic change but Zephyr expects the common ISR handler to be called _isr_wrapper (for example when generating the IRQ vector table). Change it. find ./ -type f -exec sed -i 's/__irq_wrapper/_isr_wrapper/g' {} \; Signed-off-by: Carlo Caione <ccaione@baylibre.com>
2022-06-21 10:43:40 +02:00
SECTION_FUNC(exception.entry, _isr_wrapper)
arch: added support for the riscv32 architecture RISC-V is an open-source instruction set architecture. Added support for the 32bit version of RISC-V to Zephyr. 1) exceptions/interrupts/faults are handled at the architecture level via the __irq_wrapper handler. Context saving/restoring of registers can be handled at both architecture and SOC levels. If SOC-specific registers need to be saved, SOC level needs to provide __soc_save_context and __soc_restore_context functions that shall be accounted by the architecture level, when corresponding config variable RISCV_SOC_CONTEXT_SAVE is set. 2) As RISC-V architecture does not provide a clear ISA specification about interrupt handling, each RISC-V SOC handles it in its own way. Hence, at the architecture level, the __irq_wrapper handler expects the following functions to be provided by the SOC level: __soc_is_irq: to check if the exception is the result of an interrupt or not. __soc_handle_irq: handle pending IRQ at SOC level (ex: clear pending IRQ in SOC-specific IRQ register) 3) Thread/task scheduling, as well as IRQ offloading are handled via the RISC-V system call ("ecall"), which is also handled via the __irq_wrapper handler. The _Swap asm function just calls "ecall" to generate an exception. 4) As there is no conventional way of handling CPU power save in RISC-V, the default nano_cpu_idle and nano_cpu_atomic_idle functions just unlock interrupts and return to the caller, without issuing any CPU power saving instruction. Nonetheless, to allow SOC-level to implement proper CPU power save, nano_cpu_idle and nano_cpu_atomic_idle functions are defined as __weak at the architecture level. Change-Id: I980a161d0009f3f404ad22b226a6229fbb492389 Signed-off-by: Jean-Paul Etienne <fractalclone@gmail.com>
2017-01-11 00:24:30 +01:00
riscv: exception code mega simplification and optimization Complete revamp of the exception entry code, including syscall handling. Proper syscall frame exception trigger. Many correctness fixes, hacks removal, etc. etc. I tried to make this into several commits, but this stuff is all inter-related and a pain to split. The diffstat summary: 14 files changed, 250 insertions(+), 802 deletions(-) Binary size (before): text data bss dec hex filename 1104 0 0 1104 450 isr.S.obj 64 0 0 64 40 userspace.S.obj Binary size (after): text data bss dec hex filename 600 0 0 600 258 isr.S.obj 36 0 0 36 24 userspace.S.obj Run of samples/userspace/syscall_perf (before): *** Booting Zephyr OS build zephyr-v3.0.0-325-g3748accae018 *** Main Thread started; qemu_riscv32 Supervisor thread started User thread started Supervisor thread(0x80010048): 384 cycles 509 instructions User thread(0x80010140): 77312 cycles 77437 instructions Run of samples/userspace/syscall_perf (after): *** Booting Zephyr OS build zephyr-v3.0.0-326-g4c877a2753b3 *** Main Thread started; qemu_riscv32 Supervisor thread started User thread started Supervisor thread(0x80010048): 384 cycles 509 instructions User thread(0x80010138): 7040 cycles 7165 instructions Yes, that's more than a 10x speed-up! Signed-off-by: Nicolas Pitre <npitre@baylibre.com>
2022-02-24 22:30:03 -05:00
#ifdef CONFIG_USERSPACE
riscv: decouple the Zephyr CPU number from the hart ID Currently it is assumed that Zephyr CPU numbers match their hartid value one for one. This assumption was relied upon to efficiently retrieve the current CPU's `struct _cpu` pointer. People are starting to have systems with a mix of different usage for each CPU and such assumption may no longer be true. Let's completely decouple the hartid from the Zephyr CPU number by stuffing each CPU's `struct _cpu` pointer in their respective scratch register instead. `arch_curr_cpu()` becomes more efficient as well. Since the scratch register was previously used to store userspace's exception stack pointer, that is now moved into `struct _cpu_arch` which implied minor user space entry code cleanup and rationalization. Signed-off-by: Nicolas Pitre <npitre@baylibre.com>
2023-01-06 17:46:21 -05:00
/* retrieve address of _current_cpu preserving s0 */
csrrw s0, mscratch, s0
arch: added support for the riscv32 architecture RISC-V is an open-source instruction set architecture. Added support for the 32bit version of RISC-V to Zephyr. 1) exceptions/interrupts/faults are handled at the architecture level via the __irq_wrapper handler. Context saving/restoring of registers can be handled at both architecture and SOC levels. If SOC-specific registers need to be saved, SOC level needs to provide __soc_save_context and __soc_restore_context functions that shall be accounted by the architecture level, when corresponding config variable RISCV_SOC_CONTEXT_SAVE is set. 2) As RISC-V architecture does not provide a clear ISA specification about interrupt handling, each RISC-V SOC handles it in its own way. Hence, at the architecture level, the __irq_wrapper handler expects the following functions to be provided by the SOC level: __soc_is_irq: to check if the exception is the result of an interrupt or not. __soc_handle_irq: handle pending IRQ at SOC level (ex: clear pending IRQ in SOC-specific IRQ register) 3) Thread/task scheduling, as well as IRQ offloading are handled via the RISC-V system call ("ecall"), which is also handled via the __irq_wrapper handler. The _Swap asm function just calls "ecall" to generate an exception. 4) As there is no conventional way of handling CPU power save in RISC-V, the default nano_cpu_idle and nano_cpu_atomic_idle functions just unlock interrupts and return to the caller, without issuing any CPU power saving instruction. Nonetheless, to allow SOC-level to implement proper CPU power save, nano_cpu_idle and nano_cpu_atomic_idle functions are defined as __weak at the architecture level. Change-Id: I980a161d0009f3f404ad22b226a6229fbb492389 Signed-off-by: Jean-Paul Etienne <fractalclone@gmail.com>
2017-01-11 00:24:30 +01:00
riscv: decouple the Zephyr CPU number from the hart ID Currently it is assumed that Zephyr CPU numbers match their hartid value one for one. This assumption was relied upon to efficiently retrieve the current CPU's `struct _cpu` pointer. People are starting to have systems with a mix of different usage for each CPU and such assumption may no longer be true. Let's completely decouple the hartid from the Zephyr CPU number by stuffing each CPU's `struct _cpu` pointer in their respective scratch register instead. `arch_curr_cpu()` becomes more efficient as well. Since the scratch register was previously used to store userspace's exception stack pointer, that is now moved into `struct _cpu_arch` which implied minor user space entry code cleanup and rationalization. Signed-off-by: Nicolas Pitre <npitre@baylibre.com>
2023-01-06 17:46:21 -05:00
/* preserve t0 and t1 temporarily */
sr t0, _curr_cpu_arch_user_exc_tmp0(s0)
sr t1, _curr_cpu_arch_user_exc_tmp1(s0)
Revert "arch/riscv: Use arch_switch() for context swap" This reverts commit be28de692c5e2c5546841c49d6a85d5f046474a4. The purpose of this commit will be reintroduced later on top of a cleaner codebase. Signed-off-by: Nicolas Pitre <npitre@baylibre.com>
2022-03-09 17:46:30 -05:00
riscv: decouple the Zephyr CPU number from the hart ID Currently it is assumed that Zephyr CPU numbers match their hartid value one for one. This assumption was relied upon to efficiently retrieve the current CPU's `struct _cpu` pointer. People are starting to have systems with a mix of different usage for each CPU and such assumption may no longer be true. Let's completely decouple the hartid from the Zephyr CPU number by stuffing each CPU's `struct _cpu` pointer in their respective scratch register instead. `arch_curr_cpu()` becomes more efficient as well. Since the scratch register was previously used to store userspace's exception stack pointer, that is now moved into `struct _cpu_arch` which implied minor user space entry code cleanup and rationalization. Signed-off-by: Nicolas Pitre <npitre@baylibre.com>
2023-01-06 17:46:21 -05:00
/* determine if we come from user space */
csrr t0, mstatus
li t1, MSTATUS_MPP
and t0, t0, t1
riscv: exception code mega simplification and optimization Complete revamp of the exception entry code, including syscall handling. Proper syscall frame exception trigger. Many correctness fixes, hacks removal, etc. etc. I tried to make this into several commits, but this stuff is all inter-related and a pain to split. The diffstat summary: 14 files changed, 250 insertions(+), 802 deletions(-) Binary size (before): text data bss dec hex filename 1104 0 0 1104 450 isr.S.obj 64 0 0 64 40 userspace.S.obj Binary size (after): text data bss dec hex filename 600 0 0 600 258 isr.S.obj 36 0 0 36 24 userspace.S.obj Run of samples/userspace/syscall_perf (before): *** Booting Zephyr OS build zephyr-v3.0.0-325-g3748accae018 *** Main Thread started; qemu_riscv32 Supervisor thread started User thread started Supervisor thread(0x80010048): 384 cycles 509 instructions User thread(0x80010140): 77312 cycles 77437 instructions Run of samples/userspace/syscall_perf (after): *** Booting Zephyr OS build zephyr-v3.0.0-326-g4c877a2753b3 *** Main Thread started; qemu_riscv32 Supervisor thread started User thread started Supervisor thread(0x80010048): 384 cycles 509 instructions User thread(0x80010138): 7040 cycles 7165 instructions Yes, that's more than a 10x speed-up! Signed-off-by: Nicolas Pitre <npitre@baylibre.com>
2022-02-24 22:30:03 -05:00
bnez t0, 1f
arch: riscv: add memory protection support The IRQ handler has had a major changes to manage syscall, reschedule and interrupt from user thread and stack guard. Add userspace support: - Use a global variable to know if the current execution is user or machine. The location of this variable is read only for all user thread and read/write for kernel thread. - Memory shared is supported. - Use dynamic allocation to optimize PMP slot usage. If the area size is a power of 2, only one PMP slot is used, else 2 are used. Add stack guard support: - Use MPRV bit to force PMP rules to machine mode execution. - IRQ stack have a locked stack guard to avoid re-write PMP configuration registers for each interruption and then win some cycle. - The IRQ stack is used as "temporary" stack at the beginning of IRQ handler to save current ESF. That avoid to trigger write fault on thread stack during store ESF which that call IRQ handler to infinity. - A stack guard is also setup for privileged stack of a user thread. Thread: - A PMP setup is specific to each thread. PMP setup are saved in each thread structure to improve reschedule performance. Signed-off-by: Alexandre Mergnat <amergnat@baylibre.com> Reviewed-by: Nicolas Royer <nroyer@baylibre.com>
2020-07-21 16:00:39 +02:00
riscv: decouple the Zephyr CPU number from the hart ID Currently it is assumed that Zephyr CPU numbers match their hartid value one for one. This assumption was relied upon to efficiently retrieve the current CPU's `struct _cpu` pointer. People are starting to have systems with a mix of different usage for each CPU and such assumption may no longer be true. Let's completely decouple the hartid from the Zephyr CPU number by stuffing each CPU's `struct _cpu` pointer in their respective scratch register instead. `arch_curr_cpu()` becomes more efficient as well. Since the scratch register was previously used to store userspace's exception stack pointer, that is now moved into `struct _cpu_arch` which implied minor user space entry code cleanup and rationalization. Signed-off-by: Nicolas Pitre <npitre@baylibre.com>
2023-01-06 17:46:21 -05:00
/* in user space we were: switch to our privileged stack */
mv t0, sp
lr sp, _curr_cpu_arch_user_exc_sp(s0)
/* Save user stack value. Coming from user space, we know this
* can't overflow the privileged stack. The esf will be allocated
* later but it is safe to store our saved user sp here. */
sr t0, (-__z_arch_esf_t_SIZEOF + __z_arch_esf_t_sp_OFFSET)(sp)
arch: riscv: add memory protection support The IRQ handler has had a major changes to manage syscall, reschedule and interrupt from user thread and stack guard. Add userspace support: - Use a global variable to know if the current execution is user or machine. The location of this variable is read only for all user thread and read/write for kernel thread. - Memory shared is supported. - Use dynamic allocation to optimize PMP slot usage. If the area size is a power of 2, only one PMP slot is used, else 2 are used. Add stack guard support: - Use MPRV bit to force PMP rules to machine mode execution. - IRQ stack have a locked stack guard to avoid re-write PMP configuration registers for each interruption and then win some cycle. - The IRQ stack is used as "temporary" stack at the beginning of IRQ handler to save current ESF. That avoid to trigger write fault on thread stack during store ESF which that call IRQ handler to infinity. - A stack guard is also setup for privileged stack of a user thread. Thread: - A PMP setup is specific to each thread. PMP setup are saved in each thread structure to improve reschedule performance. Signed-off-by: Alexandre Mergnat <amergnat@baylibre.com> Reviewed-by: Nicolas Royer <nroyer@baylibre.com>
2020-07-21 16:00:39 +02:00
riscv: stop preserving the tp register needlessly The tp (x4) register is neither caller nor callee saved according to the RISC-V standard calling convention. It only has to be set on thread context switching and is otherwise read-only. To protect the kernel against a possible rogue user thread, the tp is also re-set on exception entry from u-mode. Signed-off-by: Nicolas Pitre <npitre@baylibre.com>
2022-06-07 08:04:55 -04:00
/* Make sure tls pointer is sane */
lr t0, ___cpu_t_current_OFFSET(s0)
lr tp, _thread_offset_to_tls(t0)
riscv: new TLS-based arch_is_user_context() implementation This reverts the bulk of commit c8bfc2afda2c ("riscv: make arch_is_user_context() SMP compatible") and replaces it with a flag stored in the thread local storage (TLS) area, therefore making TLS mandatory for userspace support on RISC-V. This has many advantages: - The tp (x4) register is already dedicated by the standard for this purpose, making TLS support almost free. - This is very efficient, requiring only a single instruction to clear and 2 instructions to set. - This makes the SMP case much more efficient. No need for funky exception code any longer. - SMP and non-SMP now use the same implementation making maintenance easier. - The is_user_mode variable no longer requires a dedicated PMP mapping and therefore freeing one PMP slot for other purposes. Signed-off-by: Nicolas Pitre <npitre@baylibre.com> 5f65dbcc9dab3d39473b05397e05.
2022-06-07 09:37:59 -04:00
/* Clear our per-thread usermode flag */
lui t0, %tprel_hi(is_user_mode)
add t0, t0, tp, %tprel_add(is_user_mode)
sb zero, %tprel_lo(is_user_mode)(t0)
riscv: decouple the Zephyr CPU number from the hart ID Currently it is assumed that Zephyr CPU numbers match their hartid value one for one. This assumption was relied upon to efficiently retrieve the current CPU's `struct _cpu` pointer. People are starting to have systems with a mix of different usage for each CPU and such assumption may no longer be true. Let's completely decouple the hartid from the Zephyr CPU number by stuffing each CPU's `struct _cpu` pointer in their respective scratch register instead. `arch_curr_cpu()` becomes more efficient as well. Since the scratch register was previously used to store userspace's exception stack pointer, that is now moved into `struct _cpu_arch` which implied minor user space entry code cleanup and rationalization. Signed-off-by: Nicolas Pitre <npitre@baylibre.com>
2023-01-06 17:46:21 -05:00
1:
/* retrieve original t0/t1 values */
lr t0, _curr_cpu_arch_user_exc_tmp0(s0)
lr t1, _curr_cpu_arch_user_exc_tmp1(s0)
/* retrieve original s0 and restore _current_cpu in mscratch */
csrrw s0, mscratch, s0
riscv: exception code mega simplification and optimization Complete revamp of the exception entry code, including syscall handling. Proper syscall frame exception trigger. Many correctness fixes, hacks removal, etc. etc. I tried to make this into several commits, but this stuff is all inter-related and a pain to split. The diffstat summary: 14 files changed, 250 insertions(+), 802 deletions(-) Binary size (before): text data bss dec hex filename 1104 0 0 1104 450 isr.S.obj 64 0 0 64 40 userspace.S.obj Binary size (after): text data bss dec hex filename 600 0 0 600 258 isr.S.obj 36 0 0 36 24 userspace.S.obj Run of samples/userspace/syscall_perf (before): *** Booting Zephyr OS build zephyr-v3.0.0-325-g3748accae018 *** Main Thread started; qemu_riscv32 Supervisor thread started User thread started Supervisor thread(0x80010048): 384 cycles 509 instructions User thread(0x80010140): 77312 cycles 77437 instructions Run of samples/userspace/syscall_perf (after): *** Booting Zephyr OS build zephyr-v3.0.0-326-g4c877a2753b3 *** Main Thread started; qemu_riscv32 Supervisor thread started User thread started Supervisor thread(0x80010048): 384 cycles 509 instructions User thread(0x80010138): 7040 cycles 7165 instructions Yes, that's more than a 10x speed-up! Signed-off-by: Nicolas Pitre <npitre@baylibre.com>
2022-02-24 22:30:03 -05:00
#endif
riscv: decouple the Zephyr CPU number from the hart ID Currently it is assumed that Zephyr CPU numbers match their hartid value one for one. This assumption was relied upon to efficiently retrieve the current CPU's `struct _cpu` pointer. People are starting to have systems with a mix of different usage for each CPU and such assumption may no longer be true. Let's completely decouple the hartid from the Zephyr CPU number by stuffing each CPU's `struct _cpu` pointer in their respective scratch register instead. `arch_curr_cpu()` becomes more efficient as well. Since the scratch register was previously used to store userspace's exception stack pointer, that is now moved into `struct _cpu_arch` which implied minor user space entry code cleanup and rationalization. Signed-off-by: Nicolas Pitre <npitre@baylibre.com>
2023-01-06 17:46:21 -05:00
#ifdef CONFIG_RISCV_SOC_HAS_ISR_STACKING
SOC_ISR_SW_STACKING
#else
/* Save caller-saved registers on current thread stack. */
addi sp, sp, -__z_arch_esf_t_SIZEOF
DO_CALLER_SAVED(sr) ;
#endif /* CONFIG_RISCV_SOC_HAS_ISR_STACKING */
/* Save s0 in the esf and load it with &_current_cpu. */
sr s0, __z_arch_esf_t_s0_OFFSET(sp)
get_current_cpu s0
riscv: isr.S: optimize FP regs save/restore decision Rely on mstatus rather than thread->base.user_options since it is always up to date (updated by z_riscv_switch) to simplify the code and be SMP proof. Also carry over SF_INIT to the mstatus being restored in case it was changed in the mean time. Signed-off-by: Nicolas Pitre <npitre@baylibre.com>
2022-03-07 17:31:13 -05:00
/* Save MEPC register */
csrr t0, mepc
sr t0, __z_arch_esf_t_mepc_OFFSET(sp)
/* Save MSTATUS register */
riscv: Optimize t* registers usage In preparation for the support of RV32E optimize a bit the t* registers usage limiting that to t{0-2}. Signed-off-by: Carlo Caione <ccaione@baylibre.com>
2022-05-18 16:48:17 +02:00
csrr t2, mstatus
sr t2, __z_arch_esf_t_mstatus_OFFSET(sp)
riscv: isr.S: optimize FP regs save/restore decision Rely on mstatus rather than thread->base.user_options since it is always up to date (updated by z_riscv_switch) to simplify the code and be SMP proof. Also carry over SF_INIT to the mstatus being restored in case it was changed in the mean time. Signed-off-by: Nicolas Pitre <npitre@baylibre.com>
2022-03-07 17:31:13 -05:00
riscv: integrate the new FPU context switching support FPU context switching is always performed on demand through the FPU access exception handler. Actual task switching only grants or denies FPU access depending on the current FPU owner. Because RISC-V doesn't have a dedicated FPU access exception, we must catch the Illegal Instruction exception and look for actual FP opcodes. There is no longer a need to allocate FPU storage on the stack for every exception making esf smaller and stack overflows less likely. Signed-off-by: Nicolas Pitre <npitre@baylibre.com>
2023-01-17 23:32:21 -05:00
#if defined(CONFIG_FPU_SHARING)
/* determine if this is an Illegal Instruction exception */
csrr t0, mcause
li t1, 2 /* 2 = illegal instruction */
bne t0, t1, no_fp
/* determine if FPU access was disabled */
csrr t0, mstatus
li t1, MSTATUS_FS
and t0, t0, t1
bnez t0, no_fp
/* determine if we trapped on an FP instruction. */
csrr t2, mtval /* get faulting instruction */
andi t0, t2, 0x7f /* keep only the opcode bits */
xori t1, t0, 0b1010011 /* OP-FP */
beqz t1, is_fp
ori t0, t0, 0b0100000
xori t1, t0, 0b0100111 /* LOAD-FP / STORE-FP */
#if !defined(CONFIG_RISCV_ISA_EXT_C)
bnez t1, no_fp
#else
beqz t1, is_fp
/* remaining non RVC (0b11) and RVC with 0b01 are not FP instructions */
andi t1, t0, 1
bnez t1, no_fp
/*
* 001...........00 = C.FLD RV32/64 (RV128 = C.LQ)
* 001...........10 = C.FLDSP RV32/64 (RV128 = C.LQSP)
* 011...........00 = C.FLW RV32 (RV64/128 = C.LD)
* 011...........10 = C.FLWSPP RV32 (RV64/128 = C.LDSP)
* 101...........00 = C.FSD RV32/64 (RV128 = C.SQ)
* 101...........10 = C.FSDSP RV32/64 (RV128 = C.SQSP)
* 111...........00 = C.FSW RV32 (RV64/128 = C.SD)
* 111...........10 = C.FSWSP RV32 (RV64/128 = C.SDSP)
*
* so must be .01............. on RV64 and ..1............. on RV32.
*/
srli t0, t2, 8
#if defined(CONFIG_64BIT)
andi t1, t0, 0b01100000
xori t1, t1, 0b00100000
bnez t1, no_fp
#else
andi t1, t0, 0b00100000
beqz t1, no_fp
#endif
#endif /* CONFIG_RISCV_ISA_EXT_C */
is_fp: /* Process the FP trap and quickly return from exception */
la ra, fp_trap_exit
mv a0, sp
tail z_riscv_fpu_trap
no_fp: /* increment _current->arch.exception_depth */
lr t0, ___cpu_t_current_OFFSET(s0)
lb t1, _thread_offset_to_exception_depth(t0)
add t1, t1, 1
sb t1, _thread_offset_to_exception_depth(t0)
/* configure the FPU for exception mode */
call z_riscv_fpu_enter_exc
#endif
riscv: exception code mega simplification and optimization Complete revamp of the exception entry code, including syscall handling. Proper syscall frame exception trigger. Many correctness fixes, hacks removal, etc. etc. I tried to make this into several commits, but this stuff is all inter-related and a pain to split. The diffstat summary: 14 files changed, 250 insertions(+), 802 deletions(-) Binary size (before): text data bss dec hex filename 1104 0 0 1104 450 isr.S.obj 64 0 0 64 40 userspace.S.obj Binary size (after): text data bss dec hex filename 600 0 0 600 258 isr.S.obj 36 0 0 36 24 userspace.S.obj Run of samples/userspace/syscall_perf (before): *** Booting Zephyr OS build zephyr-v3.0.0-325-g3748accae018 *** Main Thread started; qemu_riscv32 Supervisor thread started User thread started Supervisor thread(0x80010048): 384 cycles 509 instructions User thread(0x80010140): 77312 cycles 77437 instructions Run of samples/userspace/syscall_perf (after): *** Booting Zephyr OS build zephyr-v3.0.0-326-g4c877a2753b3 *** Main Thread started; qemu_riscv32 Supervisor thread started User thread started Supervisor thread(0x80010048): 384 cycles 509 instructions User thread(0x80010138): 7040 cycles 7165 instructions Yes, that's more than a 10x speed-up! Signed-off-by: Nicolas Pitre <npitre@baylibre.com>
2022-02-24 22:30:03 -05:00
#ifdef CONFIG_RISCV_SOC_CONTEXT_SAVE
/* Handle context saving at SOC level. */
addi a0, sp, __z_arch_esf_t_soc_context_OFFSET
jal ra, __soc_save_context
#endif /* CONFIG_RISCV_SOC_CONTEXT_SAVE */
arch: riscv: add memory protection support The IRQ handler has had a major changes to manage syscall, reschedule and interrupt from user thread and stack guard. Add userspace support: - Use a global variable to know if the current execution is user or machine. The location of this variable is read only for all user thread and read/write for kernel thread. - Memory shared is supported. - Use dynamic allocation to optimize PMP slot usage. If the area size is a power of 2, only one PMP slot is used, else 2 are used. Add stack guard support: - Use MPRV bit to force PMP rules to machine mode execution. - IRQ stack have a locked stack guard to avoid re-write PMP configuration registers for each interruption and then win some cycle. - The IRQ stack is used as "temporary" stack at the beginning of IRQ handler to save current ESF. That avoid to trigger write fault on thread stack during store ESF which that call IRQ handler to infinity. - A stack guard is also setup for privileged stack of a user thread. Thread: - A PMP setup is specific to each thread. PMP setup are saved in each thread structure to improve reschedule performance. Signed-off-by: Alexandre Mergnat <amergnat@baylibre.com> Reviewed-by: Nicolas Royer <nroyer@baylibre.com>
2020-07-21 16:00:39 +02:00
Revert "arch/riscv: Use arch_switch() for context swap" This reverts commit be28de692c5e2c5546841c49d6a85d5f046474a4. The purpose of this commit will be reintroduced later on top of a cleaner codebase. Signed-off-by: Nicolas Pitre <npitre@baylibre.com>
2022-03-09 17:46:30 -05:00
/*
* Check if exception is the result of an interrupt or not.
* (SOC dependent). Following the RISC-V architecture spec, the MSB
* of the mcause register is used to indicate whether an exception
* is the result of an interrupt or an exception/fault. But for some
* SOCs (like pulpino or riscv-qemu), the MSB is never set to indicate
* interrupt. Hence, check for interrupt/exception via the __soc_is_irq
* function (that needs to be implemented by each SOC). The result is
* returned via register a0 (1: interrupt, 0 exception)
*/
jal ra, __soc_is_irq
/* If a0 != 0, jump to is_interrupt */
bnez a0, is_interrupt
arch: added support for the riscv32 architecture RISC-V is an open-source instruction set architecture. Added support for the 32bit version of RISC-V to Zephyr. 1) exceptions/interrupts/faults are handled at the architecture level via the __irq_wrapper handler. Context saving/restoring of registers can be handled at both architecture and SOC levels. If SOC-specific registers need to be saved, SOC level needs to provide __soc_save_context and __soc_restore_context functions that shall be accounted by the architecture level, when corresponding config variable RISCV_SOC_CONTEXT_SAVE is set. 2) As RISC-V architecture does not provide a clear ISA specification about interrupt handling, each RISC-V SOC handles it in its own way. Hence, at the architecture level, the __irq_wrapper handler expects the following functions to be provided by the SOC level: __soc_is_irq: to check if the exception is the result of an interrupt or not. __soc_handle_irq: handle pending IRQ at SOC level (ex: clear pending IRQ in SOC-specific IRQ register) 3) Thread/task scheduling, as well as IRQ offloading are handled via the RISC-V system call ("ecall"), which is also handled via the __irq_wrapper handler. The _Swap asm function just calls "ecall" to generate an exception. 4) As there is no conventional way of handling CPU power save in RISC-V, the default nano_cpu_idle and nano_cpu_atomic_idle functions just unlock interrupts and return to the caller, without issuing any CPU power saving instruction. Nonetheless, to allow SOC-level to implement proper CPU power save, nano_cpu_idle and nano_cpu_atomic_idle functions are defined as __weak at the architecture level. Change-Id: I980a161d0009f3f404ad22b226a6229fbb492389 Signed-off-by: Jean-Paul Etienne <fractalclone@gmail.com>
2017-01-11 00:24:30 +01:00
Revert "arch/riscv: Use arch_switch() for context swap" This reverts commit be28de692c5e2c5546841c49d6a85d5f046474a4. The purpose of this commit will be reintroduced later on top of a cleaner codebase. Signed-off-by: Nicolas Pitre <npitre@baylibre.com>
2022-03-09 17:46:30 -05:00
/*
* If the exception is the result of an ECALL, check whether to
* perform a context-switch or an IRQ offload. Otherwise call _Fault
* to report the exception.
*/
csrr t0, mcause
li t2, SOC_MCAUSE_EXP_MASK
and t0, t0, t2
arch/riscv: Use arch_switch() for context swap Enable `arch_switch()` as preparation for SMP support. This patch doesn't try to keep support for old style context swap - only switch based swap is supported, to keep things simple. A fair amount of refactoring was done in this patch, specially regarding the code that decides what to do about the ISR. In RISC-V, ECALL instructions are used to signalize several events, such as user space system calls, forced syscall, IRQ offload, return from syscall and context switch. All those handled by the ISR - which also handles interrupts. After refactor, this "dispatching" step is done at the beginning of ISR (just after saving generic registers). As with other platforms, the thread object itself is used as the thread "switch handle" for the context swap. Signed-off-by: Ederson de Souza <ederson.desouza@intel.com>
2021-12-21 16:44:57 -08:00
Revert "arch/riscv: Use arch_switch() for context swap" This reverts commit be28de692c5e2c5546841c49d6a85d5f046474a4. The purpose of this commit will be reintroduced later on top of a cleaner codebase. Signed-off-by: Nicolas Pitre <npitre@baylibre.com>
2022-03-09 17:46:30 -05:00
/*
* If mcause == SOC_MCAUSE_ECALL_EXP, handle system call from
* kernel thread.
arch: riscv: add memory protection support The IRQ handler has had a major changes to manage syscall, reschedule and interrupt from user thread and stack guard. Add userspace support: - Use a global variable to know if the current execution is user or machine. The location of this variable is read only for all user thread and read/write for kernel thread. - Memory shared is supported. - Use dynamic allocation to optimize PMP slot usage. If the area size is a power of 2, only one PMP slot is used, else 2 are used. Add stack guard support: - Use MPRV bit to force PMP rules to machine mode execution. - IRQ stack have a locked stack guard to avoid re-write PMP configuration registers for each interruption and then win some cycle. - The IRQ stack is used as "temporary" stack at the beginning of IRQ handler to save current ESF. That avoid to trigger write fault on thread stack during store ESF which that call IRQ handler to infinity. - A stack guard is also setup for privileged stack of a user thread. Thread: - A PMP setup is specific to each thread. PMP setup are saved in each thread structure to improve reschedule performance. Signed-off-by: Alexandre Mergnat <amergnat@baylibre.com> Reviewed-by: Nicolas Royer <nroyer@baylibre.com>
2020-07-21 16:00:39 +02:00
*/
riscv: exception code mega simplification and optimization Complete revamp of the exception entry code, including syscall handling. Proper syscall frame exception trigger. Many correctness fixes, hacks removal, etc. etc. I tried to make this into several commits, but this stuff is all inter-related and a pain to split. The diffstat summary: 14 files changed, 250 insertions(+), 802 deletions(-) Binary size (before): text data bss dec hex filename 1104 0 0 1104 450 isr.S.obj 64 0 0 64 40 userspace.S.obj Binary size (after): text data bss dec hex filename 600 0 0 600 258 isr.S.obj 36 0 0 36 24 userspace.S.obj Run of samples/userspace/syscall_perf (before): *** Booting Zephyr OS build zephyr-v3.0.0-325-g3748accae018 *** Main Thread started; qemu_riscv32 Supervisor thread started User thread started Supervisor thread(0x80010048): 384 cycles 509 instructions User thread(0x80010140): 77312 cycles 77437 instructions Run of samples/userspace/syscall_perf (after): *** Booting Zephyr OS build zephyr-v3.0.0-326-g4c877a2753b3 *** Main Thread started; qemu_riscv32 Supervisor thread started User thread started Supervisor thread(0x80010048): 384 cycles 509 instructions User thread(0x80010138): 7040 cycles 7165 instructions Yes, that's more than a 10x speed-up! Signed-off-by: Nicolas Pitre <npitre@baylibre.com>
2022-02-24 22:30:03 -05:00
li t1, SOC_MCAUSE_ECALL_EXP
Revert "arch/riscv: Use arch_switch() for context swap" This reverts commit be28de692c5e2c5546841c49d6a85d5f046474a4. The purpose of this commit will be reintroduced later on top of a cleaner codebase. Signed-off-by: Nicolas Pitre <npitre@baylibre.com>
2022-03-09 17:46:30 -05:00
beq t0, t1, is_kernel_syscall
arch: riscv: add memory protection support The IRQ handler has had a major changes to manage syscall, reschedule and interrupt from user thread and stack guard. Add userspace support: - Use a global variable to know if the current execution is user or machine. The location of this variable is read only for all user thread and read/write for kernel thread. - Memory shared is supported. - Use dynamic allocation to optimize PMP slot usage. If the area size is a power of 2, only one PMP slot is used, else 2 are used. Add stack guard support: - Use MPRV bit to force PMP rules to machine mode execution. - IRQ stack have a locked stack guard to avoid re-write PMP configuration registers for each interruption and then win some cycle. - The IRQ stack is used as "temporary" stack at the beginning of IRQ handler to save current ESF. That avoid to trigger write fault on thread stack during store ESF which that call IRQ handler to infinity. - A stack guard is also setup for privileged stack of a user thread. Thread: - A PMP setup is specific to each thread. PMP setup are saved in each thread structure to improve reschedule performance. Signed-off-by: Alexandre Mergnat <amergnat@baylibre.com> Reviewed-by: Nicolas Royer <nroyer@baylibre.com>
2020-07-21 16:00:39 +02:00
#ifdef CONFIG_USERSPACE
Revert "arch/riscv: Use arch_switch() for context swap" This reverts commit be28de692c5e2c5546841c49d6a85d5f046474a4. The purpose of this commit will be reintroduced later on top of a cleaner codebase. Signed-off-by: Nicolas Pitre <npitre@baylibre.com>
2022-03-09 17:46:30 -05:00
/*
riscv: exception code mega simplification and optimization Complete revamp of the exception entry code, including syscall handling. Proper syscall frame exception trigger. Many correctness fixes, hacks removal, etc. etc. I tried to make this into several commits, but this stuff is all inter-related and a pain to split. The diffstat summary: 14 files changed, 250 insertions(+), 802 deletions(-) Binary size (before): text data bss dec hex filename 1104 0 0 1104 450 isr.S.obj 64 0 0 64 40 userspace.S.obj Binary size (after): text data bss dec hex filename 600 0 0 600 258 isr.S.obj 36 0 0 36 24 userspace.S.obj Run of samples/userspace/syscall_perf (before): *** Booting Zephyr OS build zephyr-v3.0.0-325-g3748accae018 *** Main Thread started; qemu_riscv32 Supervisor thread started User thread started Supervisor thread(0x80010048): 384 cycles 509 instructions User thread(0x80010140): 77312 cycles 77437 instructions Run of samples/userspace/syscall_perf (after): *** Booting Zephyr OS build zephyr-v3.0.0-326-g4c877a2753b3 *** Main Thread started; qemu_riscv32 Supervisor thread started User thread started Supervisor thread(0x80010048): 384 cycles 509 instructions User thread(0x80010138): 7040 cycles 7165 instructions Yes, that's more than a 10x speed-up! Signed-off-by: Nicolas Pitre <npitre@baylibre.com>
2022-02-24 22:30:03 -05:00
* If mcause == SOC_MCAUSE_USER_ECALL_EXP, handle system call
* for user mode thread.
arch: added support for the riscv32 architecture RISC-V is an open-source instruction set architecture. Added support for the 32bit version of RISC-V to Zephyr. 1) exceptions/interrupts/faults are handled at the architecture level via the __irq_wrapper handler. Context saving/restoring of registers can be handled at both architecture and SOC levels. If SOC-specific registers need to be saved, SOC level needs to provide __soc_save_context and __soc_restore_context functions that shall be accounted by the architecture level, when corresponding config variable RISCV_SOC_CONTEXT_SAVE is set. 2) As RISC-V architecture does not provide a clear ISA specification about interrupt handling, each RISC-V SOC handles it in its own way. Hence, at the architecture level, the __irq_wrapper handler expects the following functions to be provided by the SOC level: __soc_is_irq: to check if the exception is the result of an interrupt or not. __soc_handle_irq: handle pending IRQ at SOC level (ex: clear pending IRQ in SOC-specific IRQ register) 3) Thread/task scheduling, as well as IRQ offloading are handled via the RISC-V system call ("ecall"), which is also handled via the __irq_wrapper handler. The _Swap asm function just calls "ecall" to generate an exception. 4) As there is no conventional way of handling CPU power save in RISC-V, the default nano_cpu_idle and nano_cpu_atomic_idle functions just unlock interrupts and return to the caller, without issuing any CPU power saving instruction. Nonetheless, to allow SOC-level to implement proper CPU power save, nano_cpu_idle and nano_cpu_atomic_idle functions are defined as __weak at the architecture level. Change-Id: I980a161d0009f3f404ad22b226a6229fbb492389 Signed-off-by: Jean-Paul Etienne <fractalclone@gmail.com>
2017-01-11 00:24:30 +01:00
*/
riscv: exception code mega simplification and optimization Complete revamp of the exception entry code, including syscall handling. Proper syscall frame exception trigger. Many correctness fixes, hacks removal, etc. etc. I tried to make this into several commits, but this stuff is all inter-related and a pain to split. The diffstat summary: 14 files changed, 250 insertions(+), 802 deletions(-) Binary size (before): text data bss dec hex filename 1104 0 0 1104 450 isr.S.obj 64 0 0 64 40 userspace.S.obj Binary size (after): text data bss dec hex filename 600 0 0 600 258 isr.S.obj 36 0 0 36 24 userspace.S.obj Run of samples/userspace/syscall_perf (before): *** Booting Zephyr OS build zephyr-v3.0.0-325-g3748accae018 *** Main Thread started; qemu_riscv32 Supervisor thread started User thread started Supervisor thread(0x80010048): 384 cycles 509 instructions User thread(0x80010140): 77312 cycles 77437 instructions Run of samples/userspace/syscall_perf (after): *** Booting Zephyr OS build zephyr-v3.0.0-326-g4c877a2753b3 *** Main Thread started; qemu_riscv32 Supervisor thread started User thread started Supervisor thread(0x80010048): 384 cycles 509 instructions User thread(0x80010138): 7040 cycles 7165 instructions Yes, that's more than a 10x speed-up! Signed-off-by: Nicolas Pitre <npitre@baylibre.com>
2022-02-24 22:30:03 -05:00
li t1, SOC_MCAUSE_USER_ECALL_EXP
Revert "arch/riscv: Use arch_switch() for context swap" This reverts commit be28de692c5e2c5546841c49d6a85d5f046474a4. The purpose of this commit will be reintroduced later on top of a cleaner codebase. Signed-off-by: Nicolas Pitre <npitre@baylibre.com>
2022-03-09 17:46:30 -05:00
beq t0, t1, is_user_syscall
#endif /* CONFIG_USERSPACE */
arch: riscv32: Move MEPC increment in irq_wrapper Move the MEPC increment logic in __irq_wrapper such that the saved MEPC is only incremented on a syscall (ecall instruction). Signed-off-by: Nathaniel Graff <nathaniel.graff@sifive.com>
2018-08-02 10:38:57 -07:00
/*
* Call _Fault to handle exception.
kernel: rename NANO_ESF This is now called z_arch_esf_t, conforming to our naming convention. This needs to remain a typedef due to how our offset generation header mechanism works. Signed-off-by: Andrew Boie <andrew.p.boie@intel.com>
2019-07-16 15:21:19 -07:00
* Stack pointer is pointing to a z_arch_esf_t structure, pass it
arch: riscv32: Move MEPC increment in irq_wrapper Move the MEPC increment logic in __irq_wrapper such that the saved MEPC is only incremented on a syscall (ecall instruction). Signed-off-by: Nathaniel Graff <nathaniel.graff@sifive.com>
2018-08-02 10:38:57 -07:00
* to _Fault (via register a0).
arch: riscv: add memory protection support The IRQ handler has had a major changes to manage syscall, reschedule and interrupt from user thread and stack guard. Add userspace support: - Use a global variable to know if the current execution is user or machine. The location of this variable is read only for all user thread and read/write for kernel thread. - Memory shared is supported. - Use dynamic allocation to optimize PMP slot usage. If the area size is a power of 2, only one PMP slot is used, else 2 are used. Add stack guard support: - Use MPRV bit to force PMP rules to machine mode execution. - IRQ stack have a locked stack guard to avoid re-write PMP configuration registers for each interruption and then win some cycle. - The IRQ stack is used as "temporary" stack at the beginning of IRQ handler to save current ESF. That avoid to trigger write fault on thread stack during store ESF which that call IRQ handler to infinity. - A stack guard is also setup for privileged stack of a user thread. Thread: - A PMP setup is specific to each thread. PMP setup are saved in each thread structure to improve reschedule performance. Signed-off-by: Alexandre Mergnat <amergnat@baylibre.com> Reviewed-by: Nicolas Royer <nroyer@baylibre.com>
2020-07-21 16:00:39 +02:00
* If _Fault shall return, set return address to
* no_reschedule to restore stack.
arch: riscv32: Move MEPC increment in irq_wrapper Move the MEPC increment logic in __irq_wrapper such that the saved MEPC is only incremented on a syscall (ecall instruction). Signed-off-by: Nathaniel Graff <nathaniel.graff@sifive.com>
2018-08-02 10:38:57 -07:00
*/
riscv: use simplest asm expression when possible Let's take advantage of assembler pseudoinstructions: - convert `addi rd, rs, 0` to `mv rd, rs` - convert `jal x0, somewhere` to `j somewhere` - convert `csrrs x0, csrreg, rs` to `csrs csrreg, rs` - convert `fscsr x0, rs` to `fscsr rs` And simplify zero offsets to simply 0. Signed-off-by: Nicolas Pitre <npitre@baylibre.com>
2022-02-21 21:36:55 -05:00
mv a0, sp
arch: riscv: fix sp of supervisor thread in _Fault function. Although CONFIG_USERSPACE is enabled, there are supervisor threads who don't have privileged stack using exception handler. Only let user threads to switch to privileged stack in exception handler. Signed-off-by: Jim Shu <cwshu@andestech.com>
2021-07-18 02:07:48 +08:00
la ra, no_reschedule
arch: riscv32: Move MEPC increment in irq_wrapper Move the MEPC increment logic in __irq_wrapper such that the saved MEPC is only incremented on a syscall (ecall instruction). Signed-off-by: Nathaniel Graff <nathaniel.graff@sifive.com>
2018-08-02 10:38:57 -07:00
tail _Fault
arch: added support for the riscv32 architecture RISC-V is an open-source instruction set architecture. Added support for the 32bit version of RISC-V to Zephyr. 1) exceptions/interrupts/faults are handled at the architecture level via the __irq_wrapper handler. Context saving/restoring of registers can be handled at both architecture and SOC levels. If SOC-specific registers need to be saved, SOC level needs to provide __soc_save_context and __soc_restore_context functions that shall be accounted by the architecture level, when corresponding config variable RISCV_SOC_CONTEXT_SAVE is set. 2) As RISC-V architecture does not provide a clear ISA specification about interrupt handling, each RISC-V SOC handles it in its own way. Hence, at the architecture level, the __irq_wrapper handler expects the following functions to be provided by the SOC level: __soc_is_irq: to check if the exception is the result of an interrupt or not. __soc_handle_irq: handle pending IRQ at SOC level (ex: clear pending IRQ in SOC-specific IRQ register) 3) Thread/task scheduling, as well as IRQ offloading are handled via the RISC-V system call ("ecall"), which is also handled via the __irq_wrapper handler. The _Swap asm function just calls "ecall" to generate an exception. 4) As there is no conventional way of handling CPU power save in RISC-V, the default nano_cpu_idle and nano_cpu_atomic_idle functions just unlock interrupts and return to the caller, without issuing any CPU power saving instruction. Nonetheless, to allow SOC-level to implement proper CPU power save, nano_cpu_idle and nano_cpu_atomic_idle functions are defined as __weak at the architecture level. Change-Id: I980a161d0009f3f404ad22b226a6229fbb492389 Signed-off-by: Jean-Paul Etienne <fractalclone@gmail.com>
2017-01-11 00:24:30 +01:00
Revert "arch/riscv: Use arch_switch() for context swap" This reverts commit be28de692c5e2c5546841c49d6a85d5f046474a4. The purpose of this commit will be reintroduced later on top of a cleaner codebase. Signed-off-by: Nicolas Pitre <npitre@baylibre.com>
2022-03-09 17:46:30 -05:00
is_kernel_syscall:
/*
* A syscall is the result of an ecall instruction, in which case the
* MEPC will contain the address of the ecall instruction.
* Increment saved MEPC by 4 to prevent triggering the same ecall
* again upon exiting the ISR.
*
* It's safe to always increment by 4, even with compressed
* instructions, because the ecall instruction is always 4 bytes.
*/
riscv: better abstraction for register-wide load/store opcodes Those are prominent enough that having RV_OP_LOADREG and RV_OP_STOREREG shouting at you all over the place is rather unpleasant and bad taste. Let's create pseudo-instructions of our own with assembler macros rather than preprocessor defines and only in assembly scope. This makes the asm code way more uniform and readable. Signed-off-by: Nicolas Pitre <npitre@baylibre.com>
2022-02-22 14:09:55 -05:00
lr t0, __z_arch_esf_t_mepc_OFFSET(sp)
Revert "arch/riscv: Use arch_switch() for context swap" This reverts commit be28de692c5e2c5546841c49d6a85d5f046474a4. The purpose of this commit will be reintroduced later on top of a cleaner codebase. Signed-off-by: Nicolas Pitre <npitre@baylibre.com>
2022-03-09 17:46:30 -05:00
addi t0, t0, 4
riscv: better abstraction for register-wide load/store opcodes Those are prominent enough that having RV_OP_LOADREG and RV_OP_STOREREG shouting at you all over the place is rather unpleasant and bad taste. Let's create pseudo-instructions of our own with assembler macros rather than preprocessor defines and only in assembly scope. This makes the asm code way more uniform and readable. Signed-off-by: Nicolas Pitre <npitre@baylibre.com>
2022-02-22 14:09:55 -05:00
sr t0, __z_arch_esf_t_mepc_OFFSET(sp)
Revert "arch/riscv: Use arch_switch() for context swap" This reverts commit be28de692c5e2c5546841c49d6a85d5f046474a4. The purpose of this commit will be reintroduced later on top of a cleaner codebase. Signed-off-by: Nicolas Pitre <npitre@baylibre.com>
2022-03-09 17:46:30 -05:00
riscv: pmp: switch over to the new implementation Add the appropriate hooks effectively replacing the old implementation with the new one. Also the stackguard wasn't properly enforced especially with the usermode combination. This is now fixed. Signed-off-by: Nicolas Pitre <npitre@baylibre.com>
2022-04-06 22:03:54 -04:00
#ifdef CONFIG_PMP_STACK_GUARD
/* Re-activate PMP for m-mode */
li t1, MSTATUS_MPP
csrc mstatus, t1
li t1, MSTATUS_MPRV
csrs mstatus, t1
#endif
riscv: Move syscall parameter from a7 to t0 To prepare for RV32E support. Signed-off-by: Carlo Caione <ccaione@baylibre.com>
2022-06-06 17:04:25 +02:00
/* Determine what to do. Operation code is in t0. */
lr t0, __z_arch_esf_t_t0_OFFSET(sp)
Revert "arch/riscv: Use arch_switch() for context swap" This reverts commit be28de692c5e2c5546841c49d6a85d5f046474a4. The purpose of this commit will be reintroduced later on top of a cleaner codebase. Signed-off-by: Nicolas Pitre <npitre@baylibre.com>
2022-03-09 17:46:30 -05:00
riscv: _isr_wrapper: get rid of the ASSUME_EQUAL() macro This is really useful only for one case i.e. when testing against zero. Do that test inline where it is needed and make the rest of the code independent from the actual numerical value being tested to make code maintenance easier if/when new cases are added. Signed-off-by: Nicolas Pitre <npitre@baylibre.com>
2022-06-30 13:02:39 -04:00
.if RV_ECALL_RUNTIME_EXCEPT != 0; .err; .endif
riscv: Move syscall parameter from a7 to t0 To prepare for RV32E support. Signed-off-by: Carlo Caione <ccaione@baylibre.com>
2022-06-06 17:04:25 +02:00
beqz t0, do_fault
arch: riscv: add memory protection support The IRQ handler has had a major changes to manage syscall, reschedule and interrupt from user thread and stack guard. Add userspace support: - Use a global variable to know if the current execution is user or machine. The location of this variable is read only for all user thread and read/write for kernel thread. - Memory shared is supported. - Use dynamic allocation to optimize PMP slot usage. If the area size is a power of 2, only one PMP slot is used, else 2 are used. Add stack guard support: - Use MPRV bit to force PMP rules to machine mode execution. - IRQ stack have a locked stack guard to avoid re-write PMP configuration registers for each interruption and then win some cycle. - The IRQ stack is used as "temporary" stack at the beginning of IRQ handler to save current ESF. That avoid to trigger write fault on thread stack during store ESF which that call IRQ handler to infinity. - A stack guard is also setup for privileged stack of a user thread. Thread: - A PMP setup is specific to each thread. PMP setup are saved in each thread structure to improve reschedule performance. Signed-off-by: Alexandre Mergnat <amergnat@baylibre.com> Reviewed-by: Nicolas Royer <nroyer@baylibre.com>
2020-07-21 16:00:39 +02:00
riscv: exception code mega simplification and optimization Complete revamp of the exception entry code, including syscall handling. Proper syscall frame exception trigger. Many correctness fixes, hacks removal, etc. etc. I tried to make this into several commits, but this stuff is all inter-related and a pain to split. The diffstat summary: 14 files changed, 250 insertions(+), 802 deletions(-) Binary size (before): text data bss dec hex filename 1104 0 0 1104 450 isr.S.obj 64 0 0 64 40 userspace.S.obj Binary size (after): text data bss dec hex filename 600 0 0 600 258 isr.S.obj 36 0 0 36 24 userspace.S.obj Run of samples/userspace/syscall_perf (before): *** Booting Zephyr OS build zephyr-v3.0.0-325-g3748accae018 *** Main Thread started; qemu_riscv32 Supervisor thread started User thread started Supervisor thread(0x80010048): 384 cycles 509 instructions User thread(0x80010140): 77312 cycles 77437 instructions Run of samples/userspace/syscall_perf (after): *** Booting Zephyr OS build zephyr-v3.0.0-326-g4c877a2753b3 *** Main Thread started; qemu_riscv32 Supervisor thread started User thread started Supervisor thread(0x80010048): 384 cycles 509 instructions User thread(0x80010138): 7040 cycles 7165 instructions Yes, that's more than a 10x speed-up! Signed-off-by: Nicolas Pitre <npitre@baylibre.com>
2022-02-24 22:30:03 -05:00
#if defined(CONFIG_IRQ_OFFLOAD)
riscv: _isr_wrapper: get rid of the ASSUME_EQUAL() macro This is really useful only for one case i.e. when testing against zero. Do that test inline where it is needed and make the rest of the code independent from the actual numerical value being tested to make code maintenance easier if/when new cases are added. Signed-off-by: Nicolas Pitre <npitre@baylibre.com>
2022-06-30 13:02:39 -04:00
li t1, RV_ECALL_IRQ_OFFLOAD
beq t0, t1, do_irq_offload
riscv: exception code mega simplification and optimization Complete revamp of the exception entry code, including syscall handling. Proper syscall frame exception trigger. Many correctness fixes, hacks removal, etc. etc. I tried to make this into several commits, but this stuff is all inter-related and a pain to split. The diffstat summary: 14 files changed, 250 insertions(+), 802 deletions(-) Binary size (before): text data bss dec hex filename 1104 0 0 1104 450 isr.S.obj 64 0 0 64 40 userspace.S.obj Binary size (after): text data bss dec hex filename 600 0 0 600 258 isr.S.obj 36 0 0 36 24 userspace.S.obj Run of samples/userspace/syscall_perf (before): *** Booting Zephyr OS build zephyr-v3.0.0-325-g3748accae018 *** Main Thread started; qemu_riscv32 Supervisor thread started User thread started Supervisor thread(0x80010048): 384 cycles 509 instructions User thread(0x80010140): 77312 cycles 77437 instructions Run of samples/userspace/syscall_perf (after): *** Booting Zephyr OS build zephyr-v3.0.0-326-g4c877a2753b3 *** Main Thread started; qemu_riscv32 Supervisor thread started User thread started Supervisor thread(0x80010048): 384 cycles 509 instructions User thread(0x80010138): 7040 cycles 7165 instructions Yes, that's more than a 10x speed-up! Signed-off-by: Nicolas Pitre <npitre@baylibre.com>
2022-02-24 22:30:03 -05:00
#endif
arch: riscv: add memory protection support The IRQ handler has had a major changes to manage syscall, reschedule and interrupt from user thread and stack guard. Add userspace support: - Use a global variable to know if the current execution is user or machine. The location of this variable is read only for all user thread and read/write for kernel thread. - Memory shared is supported. - Use dynamic allocation to optimize PMP slot usage. If the area size is a power of 2, only one PMP slot is used, else 2 are used. Add stack guard support: - Use MPRV bit to force PMP rules to machine mode execution. - IRQ stack have a locked stack guard to avoid re-write PMP configuration registers for each interruption and then win some cycle. - The IRQ stack is used as "temporary" stack at the beginning of IRQ handler to save current ESF. That avoid to trigger write fault on thread stack during store ESF which that call IRQ handler to infinity. - A stack guard is also setup for privileged stack of a user thread. Thread: - A PMP setup is specific to each thread. PMP setup are saved in each thread structure to improve reschedule performance. Signed-off-by: Alexandre Mergnat <amergnat@baylibre.com> Reviewed-by: Nicolas Royer <nroyer@baylibre.com>
2020-07-21 16:00:39 +02:00
riscv: Introduce RISCV_ALWAYS_SWITCH_THROUGH_ECALL Some early RISC-V SoCs have a problem when an `mret` instruction is used outside a trap handler. After the latest Zephyr RISC-V huge rework, the arch_switch code is indeed calling `mret` when not in handler mode, breaking some early RISC-V platforms. Optionally restore the old behavior by adding a new CONFIG_RISCV_ALWAYS_SWITCH_THROUGH_ECALL symbol. Signed-off-by: Nicolas Pitre <npitre@baylibre.com> Signed-off-by: Carlo Caione <ccaione@baylibre.com>
2022-06-30 14:40:52 +02:00
#ifdef CONFIG_RISCV_ALWAYS_SWITCH_THROUGH_ECALL
li t1, RV_ECALL_SCHEDULE
bne t0, t1, skip_schedule
lr a0, __z_arch_esf_t_a0_OFFSET(sp)
lr a1, __z_arch_esf_t_a1_OFFSET(sp)
j reschedule
skip_schedule:
#endif
riscv: exception code mega simplification and optimization Complete revamp of the exception entry code, including syscall handling. Proper syscall frame exception trigger. Many correctness fixes, hacks removal, etc. etc. I tried to make this into several commits, but this stuff is all inter-related and a pain to split. The diffstat summary: 14 files changed, 250 insertions(+), 802 deletions(-) Binary size (before): text data bss dec hex filename 1104 0 0 1104 450 isr.S.obj 64 0 0 64 40 userspace.S.obj Binary size (after): text data bss dec hex filename 600 0 0 600 258 isr.S.obj 36 0 0 36 24 userspace.S.obj Run of samples/userspace/syscall_perf (before): *** Booting Zephyr OS build zephyr-v3.0.0-325-g3748accae018 *** Main Thread started; qemu_riscv32 Supervisor thread started User thread started Supervisor thread(0x80010048): 384 cycles 509 instructions User thread(0x80010140): 77312 cycles 77437 instructions Run of samples/userspace/syscall_perf (after): *** Booting Zephyr OS build zephyr-v3.0.0-326-g4c877a2753b3 *** Main Thread started; qemu_riscv32 Supervisor thread started User thread started Supervisor thread(0x80010048): 384 cycles 509 instructions User thread(0x80010138): 7040 cycles 7165 instructions Yes, that's more than a 10x speed-up! Signed-off-by: Nicolas Pitre <npitre@baylibre.com>
2022-02-24 22:30:03 -05:00
/* default fault code is K_ERR_KERNEL_OOPS */
li a0, 3
j 1f
arch: riscv: add memory protection support The IRQ handler has had a major changes to manage syscall, reschedule and interrupt from user thread and stack guard. Add userspace support: - Use a global variable to know if the current execution is user or machine. The location of this variable is read only for all user thread and read/write for kernel thread. - Memory shared is supported. - Use dynamic allocation to optimize PMP slot usage. If the area size is a power of 2, only one PMP slot is used, else 2 are used. Add stack guard support: - Use MPRV bit to force PMP rules to machine mode execution. - IRQ stack have a locked stack guard to avoid re-write PMP configuration registers for each interruption and then win some cycle. - The IRQ stack is used as "temporary" stack at the beginning of IRQ handler to save current ESF. That avoid to trigger write fault on thread stack during store ESF which that call IRQ handler to infinity. - A stack guard is also setup for privileged stack of a user thread. Thread: - A PMP setup is specific to each thread. PMP setup are saved in each thread structure to improve reschedule performance. Signed-off-by: Alexandre Mergnat <amergnat@baylibre.com> Reviewed-by: Nicolas Royer <nroyer@baylibre.com>
2020-07-21 16:00:39 +02:00
riscv: exception code mega simplification and optimization Complete revamp of the exception entry code, including syscall handling. Proper syscall frame exception trigger. Many correctness fixes, hacks removal, etc. etc. I tried to make this into several commits, but this stuff is all inter-related and a pain to split. The diffstat summary: 14 files changed, 250 insertions(+), 802 deletions(-) Binary size (before): text data bss dec hex filename 1104 0 0 1104 450 isr.S.obj 64 0 0 64 40 userspace.S.obj Binary size (after): text data bss dec hex filename 600 0 0 600 258 isr.S.obj 36 0 0 36 24 userspace.S.obj Run of samples/userspace/syscall_perf (before): *** Booting Zephyr OS build zephyr-v3.0.0-325-g3748accae018 *** Main Thread started; qemu_riscv32 Supervisor thread started User thread started Supervisor thread(0x80010048): 384 cycles 509 instructions User thread(0x80010140): 77312 cycles 77437 instructions Run of samples/userspace/syscall_perf (after): *** Booting Zephyr OS build zephyr-v3.0.0-326-g4c877a2753b3 *** Main Thread started; qemu_riscv32 Supervisor thread started User thread started Supervisor thread(0x80010048): 384 cycles 509 instructions User thread(0x80010138): 7040 cycles 7165 instructions Yes, that's more than a 10x speed-up! Signed-off-by: Nicolas Pitre <npitre@baylibre.com>
2022-02-24 22:30:03 -05:00
do_fault:
/* Handle RV_ECALL_RUNTIME_EXCEPT. Retrieve reason in a0, esf in A1. */
riscv: better abstraction for register-wide load/store opcodes Those are prominent enough that having RV_OP_LOADREG and RV_OP_STOREREG shouting at you all over the place is rather unpleasant and bad taste. Let's create pseudo-instructions of our own with assembler macros rather than preprocessor defines and only in assembly scope. This makes the asm code way more uniform and readable. Signed-off-by: Nicolas Pitre <npitre@baylibre.com>
2022-02-22 14:09:55 -05:00
lr a0, __z_arch_esf_t_a0_OFFSET(sp)
riscv: exception code mega simplification and optimization Complete revamp of the exception entry code, including syscall handling. Proper syscall frame exception trigger. Many correctness fixes, hacks removal, etc. etc. I tried to make this into several commits, but this stuff is all inter-related and a pain to split. The diffstat summary: 14 files changed, 250 insertions(+), 802 deletions(-) Binary size (before): text data bss dec hex filename 1104 0 0 1104 450 isr.S.obj 64 0 0 64 40 userspace.S.obj Binary size (after): text data bss dec hex filename 600 0 0 600 258 isr.S.obj 36 0 0 36 24 userspace.S.obj Run of samples/userspace/syscall_perf (before): *** Booting Zephyr OS build zephyr-v3.0.0-325-g3748accae018 *** Main Thread started; qemu_riscv32 Supervisor thread started User thread started Supervisor thread(0x80010048): 384 cycles 509 instructions User thread(0x80010140): 77312 cycles 77437 instructions Run of samples/userspace/syscall_perf (after): *** Booting Zephyr OS build zephyr-v3.0.0-326-g4c877a2753b3 *** Main Thread started; qemu_riscv32 Supervisor thread started User thread started Supervisor thread(0x80010048): 384 cycles 509 instructions User thread(0x80010138): 7040 cycles 7165 instructions Yes, that's more than a 10x speed-up! Signed-off-by: Nicolas Pitre <npitre@baylibre.com>
2022-02-24 22:30:03 -05:00
1: mv a1, sp
tail z_riscv_fatal_error
arch: riscv: add memory protection support The IRQ handler has had a major changes to manage syscall, reschedule and interrupt from user thread and stack guard. Add userspace support: - Use a global variable to know if the current execution is user or machine. The location of this variable is read only for all user thread and read/write for kernel thread. - Memory shared is supported. - Use dynamic allocation to optimize PMP slot usage. If the area size is a power of 2, only one PMP slot is used, else 2 are used. Add stack guard support: - Use MPRV bit to force PMP rules to machine mode execution. - IRQ stack have a locked stack guard to avoid re-write PMP configuration registers for each interruption and then win some cycle. - The IRQ stack is used as "temporary" stack at the beginning of IRQ handler to save current ESF. That avoid to trigger write fault on thread stack during store ESF which that call IRQ handler to infinity. - A stack guard is also setup for privileged stack of a user thread. Thread: - A PMP setup is specific to each thread. PMP setup are saved in each thread structure to improve reschedule performance. Signed-off-by: Alexandre Mergnat <amergnat@baylibre.com> Reviewed-by: Nicolas Royer <nroyer@baylibre.com>
2020-07-21 16:00:39 +02:00
riscv: exception code mega simplification and optimization Complete revamp of the exception entry code, including syscall handling. Proper syscall frame exception trigger. Many correctness fixes, hacks removal, etc. etc. I tried to make this into several commits, but this stuff is all inter-related and a pain to split. The diffstat summary: 14 files changed, 250 insertions(+), 802 deletions(-) Binary size (before): text data bss dec hex filename 1104 0 0 1104 450 isr.S.obj 64 0 0 64 40 userspace.S.obj Binary size (after): text data bss dec hex filename 600 0 0 600 258 isr.S.obj 36 0 0 36 24 userspace.S.obj Run of samples/userspace/syscall_perf (before): *** Booting Zephyr OS build zephyr-v3.0.0-325-g3748accae018 *** Main Thread started; qemu_riscv32 Supervisor thread started User thread started Supervisor thread(0x80010048): 384 cycles 509 instructions User thread(0x80010140): 77312 cycles 77437 instructions Run of samples/userspace/syscall_perf (after): *** Booting Zephyr OS build zephyr-v3.0.0-326-g4c877a2753b3 *** Main Thread started; qemu_riscv32 Supervisor thread started User thread started Supervisor thread(0x80010048): 384 cycles 509 instructions User thread(0x80010138): 7040 cycles 7165 instructions Yes, that's more than a 10x speed-up! Signed-off-by: Nicolas Pitre <npitre@baylibre.com>
2022-02-24 22:30:03 -05:00
#if defined(CONFIG_IRQ_OFFLOAD)
do_irq_offload:
riscv: irq_offload: simpler implementation Get rid of all those global variables and IRQ locking. Use the regular IRQ exit path to let tests validate preemption properly. Signed-off-by: Nicolas Pitre <npitre@baylibre.com>
2022-03-07 13:24:38 -05:00
/*
* Retrieve provided routine and argument from the stack.
* Routine pointer is in saved a0, argument in saved a1
* so we load them with a1/a0 (reversed).
*/
lr a1, __z_arch_esf_t_a0_OFFSET(sp)
lr a0, __z_arch_esf_t_a1_OFFSET(sp)
riscv: isr.S: dedicate a register to &current_cpu Stop using &_kernel as this is not SMP friendly. Let's use s0 (after preserving its content) to hold &current_cpu instead so it won't have to be reloaded each time it is needed. This will be even more relevant when SMP support is added. Signed-off-by: Nicolas Pitre <npitre@baylibre.com>
2022-03-09 17:21:27 -05:00
/* Increment _current_cpu->nested */
riscv: Optimize t* registers usage In preparation for the support of RV32E optimize a bit the t* registers usage limiting that to t{0-2}. Signed-off-by: Carlo Caione <ccaione@baylibre.com>
2022-05-18 16:48:17 +02:00
lw t1, ___cpu_t_nested_OFFSET(s0)
addi t2, t1, 1
sw t2, ___cpu_t_nested_OFFSET(s0)
bnez t1, 1f
arch: added support for the riscv32 architecture RISC-V is an open-source instruction set architecture. Added support for the 32bit version of RISC-V to Zephyr. 1) exceptions/interrupts/faults are handled at the architecture level via the __irq_wrapper handler. Context saving/restoring of registers can be handled at both architecture and SOC levels. If SOC-specific registers need to be saved, SOC level needs to provide __soc_save_context and __soc_restore_context functions that shall be accounted by the architecture level, when corresponding config variable RISCV_SOC_CONTEXT_SAVE is set. 2) As RISC-V architecture does not provide a clear ISA specification about interrupt handling, each RISC-V SOC handles it in its own way. Hence, at the architecture level, the __irq_wrapper handler expects the following functions to be provided by the SOC level: __soc_is_irq: to check if the exception is the result of an interrupt or not. __soc_handle_irq: handle pending IRQ at SOC level (ex: clear pending IRQ in SOC-specific IRQ register) 3) Thread/task scheduling, as well as IRQ offloading are handled via the RISC-V system call ("ecall"), which is also handled via the __irq_wrapper handler. The _Swap asm function just calls "ecall" to generate an exception. 4) As there is no conventional way of handling CPU power save in RISC-V, the default nano_cpu_idle and nano_cpu_atomic_idle functions just unlock interrupts and return to the caller, without issuing any CPU power saving instruction. Nonetheless, to allow SOC-level to implement proper CPU power save, nano_cpu_idle and nano_cpu_atomic_idle functions are defined as __weak at the architecture level. Change-Id: I980a161d0009f3f404ad22b226a6229fbb492389 Signed-off-by: Jean-Paul Etienne <fractalclone@gmail.com>
2017-01-11 00:24:30 +01:00
riscv: exception code mega simplification and optimization Complete revamp of the exception entry code, including syscall handling. Proper syscall frame exception trigger. Many correctness fixes, hacks removal, etc. etc. I tried to make this into several commits, but this stuff is all inter-related and a pain to split. The diffstat summary: 14 files changed, 250 insertions(+), 802 deletions(-) Binary size (before): text data bss dec hex filename 1104 0 0 1104 450 isr.S.obj 64 0 0 64 40 userspace.S.obj Binary size (after): text data bss dec hex filename 600 0 0 600 258 isr.S.obj 36 0 0 36 24 userspace.S.obj Run of samples/userspace/syscall_perf (before): *** Booting Zephyr OS build zephyr-v3.0.0-325-g3748accae018 *** Main Thread started; qemu_riscv32 Supervisor thread started User thread started Supervisor thread(0x80010048): 384 cycles 509 instructions User thread(0x80010140): 77312 cycles 77437 instructions Run of samples/userspace/syscall_perf (after): *** Booting Zephyr OS build zephyr-v3.0.0-326-g4c877a2753b3 *** Main Thread started; qemu_riscv32 Supervisor thread started User thread started Supervisor thread(0x80010048): 384 cycles 509 instructions User thread(0x80010138): 7040 cycles 7165 instructions Yes, that's more than a 10x speed-up! Signed-off-by: Nicolas Pitre <npitre@baylibre.com>
2022-02-24 22:30:03 -05:00
/* Switch to interrupt stack */
mv t0, sp
riscv: isr.S: dedicate a register to &current_cpu Stop using &_kernel as this is not SMP friendly. Let's use s0 (after preserving its content) to hold &current_cpu instead so it won't have to be reloaded each time it is needed. This will be even more relevant when SMP support is added. Signed-off-by: Nicolas Pitre <npitre@baylibre.com>
2022-03-09 17:21:27 -05:00
lr sp, ___cpu_t_irq_stack_OFFSET(s0)
arch: riscv: add memory protection support The IRQ handler has had a major changes to manage syscall, reschedule and interrupt from user thread and stack guard. Add userspace support: - Use a global variable to know if the current execution is user or machine. The location of this variable is read only for all user thread and read/write for kernel thread. - Memory shared is supported. - Use dynamic allocation to optimize PMP slot usage. If the area size is a power of 2, only one PMP slot is used, else 2 are used. Add stack guard support: - Use MPRV bit to force PMP rules to machine mode execution. - IRQ stack have a locked stack guard to avoid re-write PMP configuration registers for each interruption and then win some cycle. - The IRQ stack is used as "temporary" stack at the beginning of IRQ handler to save current ESF. That avoid to trigger write fault on thread stack during store ESF which that call IRQ handler to infinity. - A stack guard is also setup for privileged stack of a user thread. Thread: - A PMP setup is specific to each thread. PMP setup are saved in each thread structure to improve reschedule performance. Signed-off-by: Alexandre Mergnat <amergnat@baylibre.com> Reviewed-by: Nicolas Royer <nroyer@baylibre.com>
2020-07-21 16:00:39 +02:00
riscv: implement CONFIG_IRQ_OFFLOAD_NESTED It can easily be done now, so why not. Suffice to increment the nested count like with actual IRQs. Signed-off-by: Nicolas Pitre <npitre@baylibre.com>
2022-03-14 14:51:04 -04:00
/* Save thread stack pointer on interrupt stack */
riscv: exception code mega simplification and optimization Complete revamp of the exception entry code, including syscall handling. Proper syscall frame exception trigger. Many correctness fixes, hacks removal, etc. etc. I tried to make this into several commits, but this stuff is all inter-related and a pain to split. The diffstat summary: 14 files changed, 250 insertions(+), 802 deletions(-) Binary size (before): text data bss dec hex filename 1104 0 0 1104 450 isr.S.obj 64 0 0 64 40 userspace.S.obj Binary size (after): text data bss dec hex filename 600 0 0 600 258 isr.S.obj 36 0 0 36 24 userspace.S.obj Run of samples/userspace/syscall_perf (before): *** Booting Zephyr OS build zephyr-v3.0.0-325-g3748accae018 *** Main Thread started; qemu_riscv32 Supervisor thread started User thread started Supervisor thread(0x80010048): 384 cycles 509 instructions User thread(0x80010140): 77312 cycles 77437 instructions Run of samples/userspace/syscall_perf (after): *** Booting Zephyr OS build zephyr-v3.0.0-326-g4c877a2753b3 *** Main Thread started; qemu_riscv32 Supervisor thread started User thread started Supervisor thread(0x80010048): 384 cycles 509 instructions User thread(0x80010138): 7040 cycles 7165 instructions Yes, that's more than a 10x speed-up! Signed-off-by: Nicolas Pitre <npitre@baylibre.com>
2022-02-24 22:30:03 -05:00
addi sp, sp, -16
sr t0, 0(sp)
riscv: implement CONFIG_IRQ_OFFLOAD_NESTED It can easily be done now, so why not. Suffice to increment the nested count like with actual IRQs. Signed-off-by: Nicolas Pitre <npitre@baylibre.com>
2022-03-14 14:51:04 -04:00
1:
riscv: irq_offload: simpler implementation Get rid of all those global variables and IRQ locking. Use the regular IRQ exit path to let tests validate preemption properly. Signed-off-by: Nicolas Pitre <npitre@baylibre.com>
2022-03-07 13:24:38 -05:00
/* Execute provided routine (argument is in a0 already). */
jalr ra, a1, 0
riscv: exception code mega simplification and optimization Complete revamp of the exception entry code, including syscall handling. Proper syscall frame exception trigger. Many correctness fixes, hacks removal, etc. etc. I tried to make this into several commits, but this stuff is all inter-related and a pain to split. The diffstat summary: 14 files changed, 250 insertions(+), 802 deletions(-) Binary size (before): text data bss dec hex filename 1104 0 0 1104 450 isr.S.obj 64 0 0 64 40 userspace.S.obj Binary size (after): text data bss dec hex filename 600 0 0 600 258 isr.S.obj 36 0 0 36 24 userspace.S.obj Run of samples/userspace/syscall_perf (before): *** Booting Zephyr OS build zephyr-v3.0.0-325-g3748accae018 *** Main Thread started; qemu_riscv32 Supervisor thread started User thread started Supervisor thread(0x80010048): 384 cycles 509 instructions User thread(0x80010140): 77312 cycles 77437 instructions Run of samples/userspace/syscall_perf (after): *** Booting Zephyr OS build zephyr-v3.0.0-326-g4c877a2753b3 *** Main Thread started; qemu_riscv32 Supervisor thread started User thread started Supervisor thread(0x80010048): 384 cycles 509 instructions User thread(0x80010138): 7040 cycles 7165 instructions Yes, that's more than a 10x speed-up! Signed-off-by: Nicolas Pitre <npitre@baylibre.com>
2022-02-24 22:30:03 -05:00
riscv: irq_offload: simpler implementation Get rid of all those global variables and IRQ locking. Use the regular IRQ exit path to let tests validate preemption properly. Signed-off-by: Nicolas Pitre <npitre@baylibre.com>
2022-03-07 13:24:38 -05:00
/* Leave through the regular IRQ exit path */
j irq_done
riscv: exception code mega simplification and optimization Complete revamp of the exception entry code, including syscall handling. Proper syscall frame exception trigger. Many correctness fixes, hacks removal, etc. etc. I tried to make this into several commits, but this stuff is all inter-related and a pain to split. The diffstat summary: 14 files changed, 250 insertions(+), 802 deletions(-) Binary size (before): text data bss dec hex filename 1104 0 0 1104 450 isr.S.obj 64 0 0 64 40 userspace.S.obj Binary size (after): text data bss dec hex filename 600 0 0 600 258 isr.S.obj 36 0 0 36 24 userspace.S.obj Run of samples/userspace/syscall_perf (before): *** Booting Zephyr OS build zephyr-v3.0.0-325-g3748accae018 *** Main Thread started; qemu_riscv32 Supervisor thread started User thread started Supervisor thread(0x80010048): 384 cycles 509 instructions User thread(0x80010140): 77312 cycles 77437 instructions Run of samples/userspace/syscall_perf (after): *** Booting Zephyr OS build zephyr-v3.0.0-326-g4c877a2753b3 *** Main Thread started; qemu_riscv32 Supervisor thread started User thread started Supervisor thread(0x80010048): 384 cycles 509 instructions User thread(0x80010138): 7040 cycles 7165 instructions Yes, that's more than a 10x speed-up! Signed-off-by: Nicolas Pitre <npitre@baylibre.com>
2022-02-24 22:30:03 -05:00
#endif /* CONFIG_IRQ_OFFLOAD */
arch: added support for the riscv32 architecture RISC-V is an open-source instruction set architecture. Added support for the 32bit version of RISC-V to Zephyr. 1) exceptions/interrupts/faults are handled at the architecture level via the __irq_wrapper handler. Context saving/restoring of registers can be handled at both architecture and SOC levels. If SOC-specific registers need to be saved, SOC level needs to provide __soc_save_context and __soc_restore_context functions that shall be accounted by the architecture level, when corresponding config variable RISCV_SOC_CONTEXT_SAVE is set. 2) As RISC-V architecture does not provide a clear ISA specification about interrupt handling, each RISC-V SOC handles it in its own way. Hence, at the architecture level, the __irq_wrapper handler expects the following functions to be provided by the SOC level: __soc_is_irq: to check if the exception is the result of an interrupt or not. __soc_handle_irq: handle pending IRQ at SOC level (ex: clear pending IRQ in SOC-specific IRQ register) 3) Thread/task scheduling, as well as IRQ offloading are handled via the RISC-V system call ("ecall"), which is also handled via the __irq_wrapper handler. The _Swap asm function just calls "ecall" to generate an exception. 4) As there is no conventional way of handling CPU power save in RISC-V, the default nano_cpu_idle and nano_cpu_atomic_idle functions just unlock interrupts and return to the caller, without issuing any CPU power saving instruction. Nonetheless, to allow SOC-level to implement proper CPU power save, nano_cpu_idle and nano_cpu_atomic_idle functions are defined as __weak at the architecture level. Change-Id: I980a161d0009f3f404ad22b226a6229fbb492389 Signed-off-by: Jean-Paul Etienne <fractalclone@gmail.com>
2017-01-11 00:24:30 +01:00
arch: riscv: add memory protection support The IRQ handler has had a major changes to manage syscall, reschedule and interrupt from user thread and stack guard. Add userspace support: - Use a global variable to know if the current execution is user or machine. The location of this variable is read only for all user thread and read/write for kernel thread. - Memory shared is supported. - Use dynamic allocation to optimize PMP slot usage. If the area size is a power of 2, only one PMP slot is used, else 2 are used. Add stack guard support: - Use MPRV bit to force PMP rules to machine mode execution. - IRQ stack have a locked stack guard to avoid re-write PMP configuration registers for each interruption and then win some cycle. - The IRQ stack is used as "temporary" stack at the beginning of IRQ handler to save current ESF. That avoid to trigger write fault on thread stack during store ESF which that call IRQ handler to infinity. - A stack guard is also setup for privileged stack of a user thread. Thread: - A PMP setup is specific to each thread. PMP setup are saved in each thread structure to improve reschedule performance. Signed-off-by: Alexandre Mergnat <amergnat@baylibre.com> Reviewed-by: Nicolas Royer <nroyer@baylibre.com>
2020-07-21 16:00:39 +02:00
#ifdef CONFIG_USERSPACE
is_user_syscall:
riscv: pmp: switch over to the new implementation Add the appropriate hooks effectively replacing the old implementation with the new one. Also the stackguard wasn't properly enforced especially with the usermode combination. This is now fixed. Signed-off-by: Nicolas Pitre <npitre@baylibre.com>
2022-04-06 22:03:54 -04:00
#ifdef CONFIG_PMP_STACK_GUARD
/*
* We came from userspace and need to reconfigure the
* PMP for kernel mode stack guard.
*/
lr a0, ___cpu_t_current_OFFSET(s0)
call z_riscv_pmp_stackguard_enable
#endif
riscv: exception code mega simplification and optimization Complete revamp of the exception entry code, including syscall handling. Proper syscall frame exception trigger. Many correctness fixes, hacks removal, etc. etc. I tried to make this into several commits, but this stuff is all inter-related and a pain to split. The diffstat summary: 14 files changed, 250 insertions(+), 802 deletions(-) Binary size (before): text data bss dec hex filename 1104 0 0 1104 450 isr.S.obj 64 0 0 64 40 userspace.S.obj Binary size (after): text data bss dec hex filename 600 0 0 600 258 isr.S.obj 36 0 0 36 24 userspace.S.obj Run of samples/userspace/syscall_perf (before): *** Booting Zephyr OS build zephyr-v3.0.0-325-g3748accae018 *** Main Thread started; qemu_riscv32 Supervisor thread started User thread started Supervisor thread(0x80010048): 384 cycles 509 instructions User thread(0x80010140): 77312 cycles 77437 instructions Run of samples/userspace/syscall_perf (after): *** Booting Zephyr OS build zephyr-v3.0.0-326-g4c877a2753b3 *** Main Thread started; qemu_riscv32 Supervisor thread started User thread started Supervisor thread(0x80010048): 384 cycles 509 instructions User thread(0x80010138): 7040 cycles 7165 instructions Yes, that's more than a 10x speed-up! Signed-off-by: Nicolas Pitre <npitre@baylibre.com>
2022-02-24 22:30:03 -05:00
/* It is safe to re-enable IRQs now */
csrs mstatus, MSTATUS_IEN
arch: riscv: add memory protection support The IRQ handler has had a major changes to manage syscall, reschedule and interrupt from user thread and stack guard. Add userspace support: - Use a global variable to know if the current execution is user or machine. The location of this variable is read only for all user thread and read/write for kernel thread. - Memory shared is supported. - Use dynamic allocation to optimize PMP slot usage. If the area size is a power of 2, only one PMP slot is used, else 2 are used. Add stack guard support: - Use MPRV bit to force PMP rules to machine mode execution. - IRQ stack have a locked stack guard to avoid re-write PMP configuration registers for each interruption and then win some cycle. - The IRQ stack is used as "temporary" stack at the beginning of IRQ handler to save current ESF. That avoid to trigger write fault on thread stack during store ESF which that call IRQ handler to infinity. - A stack guard is also setup for privileged stack of a user thread. Thread: - A PMP setup is specific to each thread. PMP setup are saved in each thread structure to improve reschedule performance. Signed-off-by: Alexandre Mergnat <amergnat@baylibre.com> Reviewed-by: Nicolas Royer <nroyer@baylibre.com>
2020-07-21 16:00:39 +02:00
Revert "arch/riscv: Use arch_switch() for context swap" This reverts commit be28de692c5e2c5546841c49d6a85d5f046474a4. The purpose of this commit will be reintroduced later on top of a cleaner codebase. Signed-off-by: Nicolas Pitre <npitre@baylibre.com>
2022-03-09 17:46:30 -05:00
/*
riscv: exception code mega simplification and optimization Complete revamp of the exception entry code, including syscall handling. Proper syscall frame exception trigger. Many correctness fixes, hacks removal, etc. etc. I tried to make this into several commits, but this stuff is all inter-related and a pain to split. The diffstat summary: 14 files changed, 250 insertions(+), 802 deletions(-) Binary size (before): text data bss dec hex filename 1104 0 0 1104 450 isr.S.obj 64 0 0 64 40 userspace.S.obj Binary size (after): text data bss dec hex filename 600 0 0 600 258 isr.S.obj 36 0 0 36 24 userspace.S.obj Run of samples/userspace/syscall_perf (before): *** Booting Zephyr OS build zephyr-v3.0.0-325-g3748accae018 *** Main Thread started; qemu_riscv32 Supervisor thread started User thread started Supervisor thread(0x80010048): 384 cycles 509 instructions User thread(0x80010140): 77312 cycles 77437 instructions Run of samples/userspace/syscall_perf (after): *** Booting Zephyr OS build zephyr-v3.0.0-326-g4c877a2753b3 *** Main Thread started; qemu_riscv32 Supervisor thread started User thread started Supervisor thread(0x80010048): 384 cycles 509 instructions User thread(0x80010138): 7040 cycles 7165 instructions Yes, that's more than a 10x speed-up! Signed-off-by: Nicolas Pitre <npitre@baylibre.com>
2022-02-24 22:30:03 -05:00
* Same as for is_kernel_syscall: increment saved MEPC by 4 to
* prevent triggering the same ecall again upon exiting the ISR.
Revert "arch/riscv: Use arch_switch() for context swap" This reverts commit be28de692c5e2c5546841c49d6a85d5f046474a4. The purpose of this commit will be reintroduced later on top of a cleaner codebase. Signed-off-by: Nicolas Pitre <npitre@baylibre.com>
2022-03-09 17:46:30 -05:00
*/
riscv: better abstraction for register-wide load/store opcodes Those are prominent enough that having RV_OP_LOADREG and RV_OP_STOREREG shouting at you all over the place is rather unpleasant and bad taste. Let's create pseudo-instructions of our own with assembler macros rather than preprocessor defines and only in assembly scope. This makes the asm code way more uniform and readable. Signed-off-by: Nicolas Pitre <npitre@baylibre.com>
2022-02-22 14:09:55 -05:00
lr t1, __z_arch_esf_t_mepc_OFFSET(sp)
Revert "arch/riscv: Use arch_switch() for context swap" This reverts commit be28de692c5e2c5546841c49d6a85d5f046474a4. The purpose of this commit will be reintroduced later on top of a cleaner codebase. Signed-off-by: Nicolas Pitre <npitre@baylibre.com>
2022-03-09 17:46:30 -05:00
addi t1, t1, 4
riscv: better abstraction for register-wide load/store opcodes Those are prominent enough that having RV_OP_LOADREG and RV_OP_STOREREG shouting at you all over the place is rather unpleasant and bad taste. Let's create pseudo-instructions of our own with assembler macros rather than preprocessor defines and only in assembly scope. This makes the asm code way more uniform and readable. Signed-off-by: Nicolas Pitre <npitre@baylibre.com>
2022-02-22 14:09:55 -05:00
sr t1, __z_arch_esf_t_mepc_OFFSET(sp)
riscv: exception code mega simplification and optimization Complete revamp of the exception entry code, including syscall handling. Proper syscall frame exception trigger. Many correctness fixes, hacks removal, etc. etc. I tried to make this into several commits, but this stuff is all inter-related and a pain to split. The diffstat summary: 14 files changed, 250 insertions(+), 802 deletions(-) Binary size (before): text data bss dec hex filename 1104 0 0 1104 450 isr.S.obj 64 0 0 64 40 userspace.S.obj Binary size (after): text data bss dec hex filename 600 0 0 600 258 isr.S.obj 36 0 0 36 24 userspace.S.obj Run of samples/userspace/syscall_perf (before): *** Booting Zephyr OS build zephyr-v3.0.0-325-g3748accae018 *** Main Thread started; qemu_riscv32 Supervisor thread started User thread started Supervisor thread(0x80010048): 384 cycles 509 instructions User thread(0x80010140): 77312 cycles 77437 instructions Run of samples/userspace/syscall_perf (after): *** Booting Zephyr OS build zephyr-v3.0.0-326-g4c877a2753b3 *** Main Thread started; qemu_riscv32 Supervisor thread started User thread started Supervisor thread(0x80010048): 384 cycles 509 instructions User thread(0x80010138): 7040 cycles 7165 instructions Yes, that's more than a 10x speed-up! Signed-off-by: Nicolas Pitre <npitre@baylibre.com>
2022-02-24 22:30:03 -05:00
arch: riscv: add memory protection support The IRQ handler has had a major changes to manage syscall, reschedule and interrupt from user thread and stack guard. Add userspace support: - Use a global variable to know if the current execution is user or machine. The location of this variable is read only for all user thread and read/write for kernel thread. - Memory shared is supported. - Use dynamic allocation to optimize PMP slot usage. If the area size is a power of 2, only one PMP slot is used, else 2 are used. Add stack guard support: - Use MPRV bit to force PMP rules to machine mode execution. - IRQ stack have a locked stack guard to avoid re-write PMP configuration registers for each interruption and then win some cycle. - The IRQ stack is used as "temporary" stack at the beginning of IRQ handler to save current ESF. That avoid to trigger write fault on thread stack during store ESF which that call IRQ handler to infinity. - A stack guard is also setup for privileged stack of a user thread. Thread: - A PMP setup is specific to each thread. PMP setup are saved in each thread structure to improve reschedule performance. Signed-off-by: Alexandre Mergnat <amergnat@baylibre.com> Reviewed-by: Nicolas Royer <nroyer@baylibre.com>
2020-07-21 16:00:39 +02:00
/* Restore argument registers from user stack */
riscv: better abstraction for register-wide load/store opcodes Those are prominent enough that having RV_OP_LOADREG and RV_OP_STOREREG shouting at you all over the place is rather unpleasant and bad taste. Let's create pseudo-instructions of our own with assembler macros rather than preprocessor defines and only in assembly scope. This makes the asm code way more uniform and readable. Signed-off-by: Nicolas Pitre <npitre@baylibre.com>
2022-02-22 14:09:55 -05:00
lr a0, __z_arch_esf_t_a0_OFFSET(sp)
lr a1, __z_arch_esf_t_a1_OFFSET(sp)
lr a2, __z_arch_esf_t_a2_OFFSET(sp)
lr a3, __z_arch_esf_t_a3_OFFSET(sp)
lr a4, __z_arch_esf_t_a4_OFFSET(sp)
lr a5, __z_arch_esf_t_a5_OFFSET(sp)
riscv: Move syscall parameter from a7 to t0 To prepare for RV32E support. Signed-off-by: Carlo Caione <ccaione@baylibre.com>
2022-06-06 17:04:25 +02:00
lr t0, __z_arch_esf_t_t0_OFFSET(sp)
riscv: Introduce support for RV32E Introduce support for RV32E. Signed-off-by: Carlo Caione <ccaione@baylibre.com>
2022-06-07 14:18:11 +02:00
#if defined(CONFIG_RISCV_ISA_RV32E)
/* Stack alignment for RV32E is 4 bytes */
addi sp, sp, -4
mv t1, sp
sw t1, 0(sp)
#else
arch: riscv: add memory protection support The IRQ handler has had a major changes to manage syscall, reschedule and interrupt from user thread and stack guard. Add userspace support: - Use a global variable to know if the current execution is user or machine. The location of this variable is read only for all user thread and read/write for kernel thread. - Memory shared is supported. - Use dynamic allocation to optimize PMP slot usage. If the area size is a power of 2, only one PMP slot is used, else 2 are used. Add stack guard support: - Use MPRV bit to force PMP rules to machine mode execution. - IRQ stack have a locked stack guard to avoid re-write PMP configuration registers for each interruption and then win some cycle. - The IRQ stack is used as "temporary" stack at the beginning of IRQ handler to save current ESF. That avoid to trigger write fault on thread stack during store ESF which that call IRQ handler to infinity. - A stack guard is also setup for privileged stack of a user thread. Thread: - A PMP setup is specific to each thread. PMP setup are saved in each thread structure to improve reschedule performance. Signed-off-by: Alexandre Mergnat <amergnat@baylibre.com> Reviewed-by: Nicolas Royer <nroyer@baylibre.com>
2020-07-21 16:00:39 +02:00
mv a6, sp
riscv: Introduce support for RV32E Introduce support for RV32E. Signed-off-by: Carlo Caione <ccaione@baylibre.com>
2022-06-07 14:18:11 +02:00
#endif /* CONFIG_RISCV_ISA_RV32E */
arch: riscv: add memory protection support The IRQ handler has had a major changes to manage syscall, reschedule and interrupt from user thread and stack guard. Add userspace support: - Use a global variable to know if the current execution is user or machine. The location of this variable is read only for all user thread and read/write for kernel thread. - Memory shared is supported. - Use dynamic allocation to optimize PMP slot usage. If the area size is a power of 2, only one PMP slot is used, else 2 are used. Add stack guard support: - Use MPRV bit to force PMP rules to machine mode execution. - IRQ stack have a locked stack guard to avoid re-write PMP configuration registers for each interruption and then win some cycle. - The IRQ stack is used as "temporary" stack at the beginning of IRQ handler to save current ESF. That avoid to trigger write fault on thread stack during store ESF which that call IRQ handler to infinity. - A stack guard is also setup for privileged stack of a user thread. Thread: - A PMP setup is specific to each thread. PMP setup are saved in each thread structure to improve reschedule performance. Signed-off-by: Alexandre Mergnat <amergnat@baylibre.com> Reviewed-by: Nicolas Royer <nroyer@baylibre.com>
2020-07-21 16:00:39 +02:00
/* validate syscall limit */
riscv: Move syscall parameter from a7 to t0 To prepare for RV32E support. Signed-off-by: Carlo Caione <ccaione@baylibre.com>
2022-06-06 17:04:25 +02:00
li t1, K_SYSCALL_LIMIT
bltu t0, t1, valid_syscall_id
arch: riscv: add memory protection support The IRQ handler has had a major changes to manage syscall, reschedule and interrupt from user thread and stack guard. Add userspace support: - Use a global variable to know if the current execution is user or machine. The location of this variable is read only for all user thread and read/write for kernel thread. - Memory shared is supported. - Use dynamic allocation to optimize PMP slot usage. If the area size is a power of 2, only one PMP slot is used, else 2 are used. Add stack guard support: - Use MPRV bit to force PMP rules to machine mode execution. - IRQ stack have a locked stack guard to avoid re-write PMP configuration registers for each interruption and then win some cycle. - The IRQ stack is used as "temporary" stack at the beginning of IRQ handler to save current ESF. That avoid to trigger write fault on thread stack during store ESF which that call IRQ handler to infinity. - A stack guard is also setup for privileged stack of a user thread. Thread: - A PMP setup is specific to each thread. PMP setup are saved in each thread structure to improve reschedule performance. Signed-off-by: Alexandre Mergnat <amergnat@baylibre.com> Reviewed-by: Nicolas Royer <nroyer@baylibre.com>
2020-07-21 16:00:39 +02:00
/* bad syscall id. Set arg1 to bad id and set call_id to SYSCALL_BAD */
riscv: Move syscall parameter from a7 to t0 To prepare for RV32E support. Signed-off-by: Carlo Caione <ccaione@baylibre.com>
2022-06-06 17:04:25 +02:00
mv a0, t0
li t0, K_SYSCALL_BAD
arch: riscv: add memory protection support The IRQ handler has had a major changes to manage syscall, reschedule and interrupt from user thread and stack guard. Add userspace support: - Use a global variable to know if the current execution is user or machine. The location of this variable is read only for all user thread and read/write for kernel thread. - Memory shared is supported. - Use dynamic allocation to optimize PMP slot usage. If the area size is a power of 2, only one PMP slot is used, else 2 are used. Add stack guard support: - Use MPRV bit to force PMP rules to machine mode execution. - IRQ stack have a locked stack guard to avoid re-write PMP configuration registers for each interruption and then win some cycle. - The IRQ stack is used as "temporary" stack at the beginning of IRQ handler to save current ESF. That avoid to trigger write fault on thread stack during store ESF which that call IRQ handler to infinity. - A stack guard is also setup for privileged stack of a user thread. Thread: - A PMP setup is specific to each thread. PMP setup are saved in each thread structure to improve reschedule performance. Signed-off-by: Alexandre Mergnat <amergnat@baylibre.com> Reviewed-by: Nicolas Royer <nroyer@baylibre.com>
2020-07-21 16:00:39 +02:00
valid_syscall_id:
riscv: Move syscall parameter from a7 to t0 To prepare for RV32E support. Signed-off-by: Carlo Caione <ccaione@baylibre.com>
2022-06-06 17:04:25 +02:00
la t2, _k_syscall_table
arch: riscv: add memory protection support The IRQ handler has had a major changes to manage syscall, reschedule and interrupt from user thread and stack guard. Add userspace support: - Use a global variable to know if the current execution is user or machine. The location of this variable is read only for all user thread and read/write for kernel thread. - Memory shared is supported. - Use dynamic allocation to optimize PMP slot usage. If the area size is a power of 2, only one PMP slot is used, else 2 are used. Add stack guard support: - Use MPRV bit to force PMP rules to machine mode execution. - IRQ stack have a locked stack guard to avoid re-write PMP configuration registers for each interruption and then win some cycle. - The IRQ stack is used as "temporary" stack at the beginning of IRQ handler to save current ESF. That avoid to trigger write fault on thread stack during store ESF which that call IRQ handler to infinity. - A stack guard is also setup for privileged stack of a user thread. Thread: - A PMP setup is specific to each thread. PMP setup are saved in each thread structure to improve reschedule performance. Signed-off-by: Alexandre Mergnat <amergnat@baylibre.com> Reviewed-by: Nicolas Royer <nroyer@baylibre.com>
2020-07-21 16:00:39 +02:00
riscv: Move syscall parameter from a7 to t0 To prepare for RV32E support. Signed-off-by: Carlo Caione <ccaione@baylibre.com>
2022-06-06 17:04:25 +02:00
slli t1, t0, RV_REGSHIFT # Determine offset from indice value
add t2, t2, t1 # Table addr + offset = function addr
lr t2, 0(t2) # Load function address
arch: riscv: add memory protection support The IRQ handler has had a major changes to manage syscall, reschedule and interrupt from user thread and stack guard. Add userspace support: - Use a global variable to know if the current execution is user or machine. The location of this variable is read only for all user thread and read/write for kernel thread. - Memory shared is supported. - Use dynamic allocation to optimize PMP slot usage. If the area size is a power of 2, only one PMP slot is used, else 2 are used. Add stack guard support: - Use MPRV bit to force PMP rules to machine mode execution. - IRQ stack have a locked stack guard to avoid re-write PMP configuration registers for each interruption and then win some cycle. - The IRQ stack is used as "temporary" stack at the beginning of IRQ handler to save current ESF. That avoid to trigger write fault on thread stack during store ESF which that call IRQ handler to infinity. - A stack guard is also setup for privileged stack of a user thread. Thread: - A PMP setup is specific to each thread. PMP setup are saved in each thread structure to improve reschedule performance. Signed-off-by: Alexandre Mergnat <amergnat@baylibre.com> Reviewed-by: Nicolas Royer <nroyer@baylibre.com>
2020-07-21 16:00:39 +02:00
riscv: exception code mega simplification and optimization Complete revamp of the exception entry code, including syscall handling. Proper syscall frame exception trigger. Many correctness fixes, hacks removal, etc. etc. I tried to make this into several commits, but this stuff is all inter-related and a pain to split. The diffstat summary: 14 files changed, 250 insertions(+), 802 deletions(-) Binary size (before): text data bss dec hex filename 1104 0 0 1104 450 isr.S.obj 64 0 0 64 40 userspace.S.obj Binary size (after): text data bss dec hex filename 600 0 0 600 258 isr.S.obj 36 0 0 36 24 userspace.S.obj Run of samples/userspace/syscall_perf (before): *** Booting Zephyr OS build zephyr-v3.0.0-325-g3748accae018 *** Main Thread started; qemu_riscv32 Supervisor thread started User thread started Supervisor thread(0x80010048): 384 cycles 509 instructions User thread(0x80010140): 77312 cycles 77437 instructions Run of samples/userspace/syscall_perf (after): *** Booting Zephyr OS build zephyr-v3.0.0-326-g4c877a2753b3 *** Main Thread started; qemu_riscv32 Supervisor thread started User thread started Supervisor thread(0x80010048): 384 cycles 509 instructions User thread(0x80010138): 7040 cycles 7165 instructions Yes, that's more than a 10x speed-up! Signed-off-by: Nicolas Pitre <npitre@baylibre.com>
2022-02-24 22:30:03 -05:00
/* Execute syscall function */
riscv: Optimize t* registers usage In preparation for the support of RV32E optimize a bit the t* registers usage limiting that to t{0-2}. Signed-off-by: Carlo Caione <ccaione@baylibre.com>
2022-05-18 16:48:17 +02:00
jalr ra, t2, 0
arch: riscv: add memory protection support The IRQ handler has had a major changes to manage syscall, reschedule and interrupt from user thread and stack guard. Add userspace support: - Use a global variable to know if the current execution is user or machine. The location of this variable is read only for all user thread and read/write for kernel thread. - Memory shared is supported. - Use dynamic allocation to optimize PMP slot usage. If the area size is a power of 2, only one PMP slot is used, else 2 are used. Add stack guard support: - Use MPRV bit to force PMP rules to machine mode execution. - IRQ stack have a locked stack guard to avoid re-write PMP configuration registers for each interruption and then win some cycle. - The IRQ stack is used as "temporary" stack at the beginning of IRQ handler to save current ESF. That avoid to trigger write fault on thread stack during store ESF which that call IRQ handler to infinity. - A stack guard is also setup for privileged stack of a user thread. Thread: - A PMP setup is specific to each thread. PMP setup are saved in each thread structure to improve reschedule performance. Signed-off-by: Alexandre Mergnat <amergnat@baylibre.com> Reviewed-by: Nicolas Royer <nroyer@baylibre.com>
2020-07-21 16:00:39 +02:00
riscv: Introduce support for RV32E Introduce support for RV32E. Signed-off-by: Carlo Caione <ccaione@baylibre.com>
2022-06-07 14:18:11 +02:00
#if defined(CONFIG_RISCV_ISA_RV32E)
addi sp, sp, 4
#endif /* CONFIG_RISCV_ISA_RV32E */
riscv: exception code mega simplification and optimization Complete revamp of the exception entry code, including syscall handling. Proper syscall frame exception trigger. Many correctness fixes, hacks removal, etc. etc. I tried to make this into several commits, but this stuff is all inter-related and a pain to split. The diffstat summary: 14 files changed, 250 insertions(+), 802 deletions(-) Binary size (before): text data bss dec hex filename 1104 0 0 1104 450 isr.S.obj 64 0 0 64 40 userspace.S.obj Binary size (after): text data bss dec hex filename 600 0 0 600 258 isr.S.obj 36 0 0 36 24 userspace.S.obj Run of samples/userspace/syscall_perf (before): *** Booting Zephyr OS build zephyr-v3.0.0-325-g3748accae018 *** Main Thread started; qemu_riscv32 Supervisor thread started User thread started Supervisor thread(0x80010048): 384 cycles 509 instructions User thread(0x80010140): 77312 cycles 77437 instructions Run of samples/userspace/syscall_perf (after): *** Booting Zephyr OS build zephyr-v3.0.0-326-g4c877a2753b3 *** Main Thread started; qemu_riscv32 Supervisor thread started User thread started Supervisor thread(0x80010048): 384 cycles 509 instructions User thread(0x80010138): 7040 cycles 7165 instructions Yes, that's more than a 10x speed-up! Signed-off-by: Nicolas Pitre <npitre@baylibre.com>
2022-02-24 22:30:03 -05:00
/* Update a0 (return value) on the stack */
riscv: better abstraction for register-wide load/store opcodes Those are prominent enough that having RV_OP_LOADREG and RV_OP_STOREREG shouting at you all over the place is rather unpleasant and bad taste. Let's create pseudo-instructions of our own with assembler macros rather than preprocessor defines and only in assembly scope. This makes the asm code way more uniform and readable. Signed-off-by: Nicolas Pitre <npitre@baylibre.com>
2022-02-22 14:09:55 -05:00
sr a0, __z_arch_esf_t_a0_OFFSET(sp)
arch: riscv: add memory protection support The IRQ handler has had a major changes to manage syscall, reschedule and interrupt from user thread and stack guard. Add userspace support: - Use a global variable to know if the current execution is user or machine. The location of this variable is read only for all user thread and read/write for kernel thread. - Memory shared is supported. - Use dynamic allocation to optimize PMP slot usage. If the area size is a power of 2, only one PMP slot is used, else 2 are used. Add stack guard support: - Use MPRV bit to force PMP rules to machine mode execution. - IRQ stack have a locked stack guard to avoid re-write PMP configuration registers for each interruption and then win some cycle. - The IRQ stack is used as "temporary" stack at the beginning of IRQ handler to save current ESF. That avoid to trigger write fault on thread stack during store ESF which that call IRQ handler to infinity. - A stack guard is also setup for privileged stack of a user thread. Thread: - A PMP setup is specific to each thread. PMP setup are saved in each thread structure to improve reschedule performance. Signed-off-by: Alexandre Mergnat <amergnat@baylibre.com> Reviewed-by: Nicolas Royer <nroyer@baylibre.com>
2020-07-21 16:00:39 +02:00
riscv: exception code mega simplification and optimization Complete revamp of the exception entry code, including syscall handling. Proper syscall frame exception trigger. Many correctness fixes, hacks removal, etc. etc. I tried to make this into several commits, but this stuff is all inter-related and a pain to split. The diffstat summary: 14 files changed, 250 insertions(+), 802 deletions(-) Binary size (before): text data bss dec hex filename 1104 0 0 1104 450 isr.S.obj 64 0 0 64 40 userspace.S.obj Binary size (after): text data bss dec hex filename 600 0 0 600 258 isr.S.obj 36 0 0 36 24 userspace.S.obj Run of samples/userspace/syscall_perf (before): *** Booting Zephyr OS build zephyr-v3.0.0-325-g3748accae018 *** Main Thread started; qemu_riscv32 Supervisor thread started User thread started Supervisor thread(0x80010048): 384 cycles 509 instructions User thread(0x80010140): 77312 cycles 77437 instructions Run of samples/userspace/syscall_perf (after): *** Booting Zephyr OS build zephyr-v3.0.0-326-g4c877a2753b3 *** Main Thread started; qemu_riscv32 Supervisor thread started User thread started Supervisor thread(0x80010048): 384 cycles 509 instructions User thread(0x80010138): 7040 cycles 7165 instructions Yes, that's more than a 10x speed-up! Signed-off-by: Nicolas Pitre <npitre@baylibre.com>
2022-02-24 22:30:03 -05:00
/* Disable IRQs again before leaving */
csrc mstatus, MSTATUS_IEN
riscv: isr.S: dedicate a register to &current_cpu Stop using &_kernel as this is not SMP friendly. Let's use s0 (after preserving its content) to hold &current_cpu instead so it won't have to be reloaded each time it is needed. This will be even more relevant when SMP support is added. Signed-off-by: Nicolas Pitre <npitre@baylibre.com>
2022-03-09 17:21:27 -05:00
j might_have_rescheduled
arch: riscv: add memory protection support The IRQ handler has had a major changes to manage syscall, reschedule and interrupt from user thread and stack guard. Add userspace support: - Use a global variable to know if the current execution is user or machine. The location of this variable is read only for all user thread and read/write for kernel thread. - Memory shared is supported. - Use dynamic allocation to optimize PMP slot usage. If the area size is a power of 2, only one PMP slot is used, else 2 are used. Add stack guard support: - Use MPRV bit to force PMP rules to machine mode execution. - IRQ stack have a locked stack guard to avoid re-write PMP configuration registers for each interruption and then win some cycle. - The IRQ stack is used as "temporary" stack at the beginning of IRQ handler to save current ESF. That avoid to trigger write fault on thread stack during store ESF which that call IRQ handler to infinity. - A stack guard is also setup for privileged stack of a user thread. Thread: - A PMP setup is specific to each thread. PMP setup are saved in each thread structure to improve reschedule performance. Signed-off-by: Alexandre Mergnat <amergnat@baylibre.com> Reviewed-by: Nicolas Royer <nroyer@baylibre.com>
2020-07-21 16:00:39 +02:00
#endif /* CONFIG_USERSPACE */
arch: added support for the riscv32 architecture RISC-V is an open-source instruction set architecture. Added support for the 32bit version of RISC-V to Zephyr. 1) exceptions/interrupts/faults are handled at the architecture level via the __irq_wrapper handler. Context saving/restoring of registers can be handled at both architecture and SOC levels. If SOC-specific registers need to be saved, SOC level needs to provide __soc_save_context and __soc_restore_context functions that shall be accounted by the architecture level, when corresponding config variable RISCV_SOC_CONTEXT_SAVE is set. 2) As RISC-V architecture does not provide a clear ISA specification about interrupt handling, each RISC-V SOC handles it in its own way. Hence, at the architecture level, the __irq_wrapper handler expects the following functions to be provided by the SOC level: __soc_is_irq: to check if the exception is the result of an interrupt or not. __soc_handle_irq: handle pending IRQ at SOC level (ex: clear pending IRQ in SOC-specific IRQ register) 3) Thread/task scheduling, as well as IRQ offloading are handled via the RISC-V system call ("ecall"), which is also handled via the __irq_wrapper handler. The _Swap asm function just calls "ecall" to generate an exception. 4) As there is no conventional way of handling CPU power save in RISC-V, the default nano_cpu_idle and nano_cpu_atomic_idle functions just unlock interrupts and return to the caller, without issuing any CPU power saving instruction. Nonetheless, to allow SOC-level to implement proper CPU power save, nano_cpu_idle and nano_cpu_atomic_idle functions are defined as __weak at the architecture level. Change-Id: I980a161d0009f3f404ad22b226a6229fbb492389 Signed-off-by: Jean-Paul Etienne <fractalclone@gmail.com>
2017-01-11 00:24:30 +01:00
is_interrupt:
riscv: pmp: switch over to the new implementation Add the appropriate hooks effectively replacing the old implementation with the new one. Also the stackguard wasn't properly enforced especially with the usermode combination. This is now fixed. Signed-off-by: Nicolas Pitre <npitre@baylibre.com>
2022-04-06 22:03:54 -04:00
#ifdef CONFIG_PMP_STACK_GUARD
#ifdef CONFIG_USERSPACE
/*
* If we came from userspace then we need to reconfigure the
* PMP for kernel mode stack guard.
*/
lr t0, __z_arch_esf_t_mstatus_OFFSET(sp)
li t1, MSTATUS_MPP
and t0, t0, t1
bnez t0, 1f
lr a0, ___cpu_t_current_OFFSET(s0)
call z_riscv_pmp_stackguard_enable
j 2f
#endif /* CONFIG_USERSPACE */
1: /* Re-activate PMP for m-mode */
li t1, MSTATUS_MPP
csrc mstatus, t1
li t1, MSTATUS_MPRV
csrs mstatus, t1
2:
#endif
riscv: isr.S: dedicate a register to &current_cpu Stop using &_kernel as this is not SMP friendly. Let's use s0 (after preserving its content) to hold &current_cpu instead so it won't have to be reloaded each time it is needed. This will be even more relevant when SMP support is added. Signed-off-by: Nicolas Pitre <npitre@baylibre.com>
2022-03-09 17:21:27 -05:00
/* Increment _current_cpu->nested */
riscv: Optimize t* registers usage In preparation for the support of RV32E optimize a bit the t* registers usage limiting that to t{0-2}. Signed-off-by: Carlo Caione <ccaione@baylibre.com>
2022-05-18 16:48:17 +02:00
lw t1, ___cpu_t_nested_OFFSET(s0)
addi t2, t1, 1
sw t2, ___cpu_t_nested_OFFSET(s0)
bnez t1, on_irq_stack
arch: added support for the riscv32 architecture RISC-V is an open-source instruction set architecture. Added support for the 32bit version of RISC-V to Zephyr. 1) exceptions/interrupts/faults are handled at the architecture level via the __irq_wrapper handler. Context saving/restoring of registers can be handled at both architecture and SOC levels. If SOC-specific registers need to be saved, SOC level needs to provide __soc_save_context and __soc_restore_context functions that shall be accounted by the architecture level, when corresponding config variable RISCV_SOC_CONTEXT_SAVE is set. 2) As RISC-V architecture does not provide a clear ISA specification about interrupt handling, each RISC-V SOC handles it in its own way. Hence, at the architecture level, the __irq_wrapper handler expects the following functions to be provided by the SOC level: __soc_is_irq: to check if the exception is the result of an interrupt or not. __soc_handle_irq: handle pending IRQ at SOC level (ex: clear pending IRQ in SOC-specific IRQ register) 3) Thread/task scheduling, as well as IRQ offloading are handled via the RISC-V system call ("ecall"), which is also handled via the __irq_wrapper handler. The _Swap asm function just calls "ecall" to generate an exception. 4) As there is no conventional way of handling CPU power save in RISC-V, the default nano_cpu_idle and nano_cpu_atomic_idle functions just unlock interrupts and return to the caller, without issuing any CPU power saving instruction. Nonetheless, to allow SOC-level to implement proper CPU power save, nano_cpu_idle and nano_cpu_atomic_idle functions are defined as __weak at the architecture level. Change-Id: I980a161d0009f3f404ad22b226a6229fbb492389 Signed-off-by: Jean-Paul Etienne <fractalclone@gmail.com>
2017-01-11 00:24:30 +01:00
/* Switch to interrupt stack */
riscv: exception code mega simplification and optimization Complete revamp of the exception entry code, including syscall handling. Proper syscall frame exception trigger. Many correctness fixes, hacks removal, etc. etc. I tried to make this into several commits, but this stuff is all inter-related and a pain to split. The diffstat summary: 14 files changed, 250 insertions(+), 802 deletions(-) Binary size (before): text data bss dec hex filename 1104 0 0 1104 450 isr.S.obj 64 0 0 64 40 userspace.S.obj Binary size (after): text data bss dec hex filename 600 0 0 600 258 isr.S.obj 36 0 0 36 24 userspace.S.obj Run of samples/userspace/syscall_perf (before): *** Booting Zephyr OS build zephyr-v3.0.0-325-g3748accae018 *** Main Thread started; qemu_riscv32 Supervisor thread started User thread started Supervisor thread(0x80010048): 384 cycles 509 instructions User thread(0x80010140): 77312 cycles 77437 instructions Run of samples/userspace/syscall_perf (after): *** Booting Zephyr OS build zephyr-v3.0.0-326-g4c877a2753b3 *** Main Thread started; qemu_riscv32 Supervisor thread started User thread started Supervisor thread(0x80010048): 384 cycles 509 instructions User thread(0x80010138): 7040 cycles 7165 instructions Yes, that's more than a 10x speed-up! Signed-off-by: Nicolas Pitre <npitre@baylibre.com>
2022-02-24 22:30:03 -05:00
mv t0, sp
riscv: isr.S: dedicate a register to &current_cpu Stop using &_kernel as this is not SMP friendly. Let's use s0 (after preserving its content) to hold &current_cpu instead so it won't have to be reloaded each time it is needed. This will be even more relevant when SMP support is added. Signed-off-by: Nicolas Pitre <npitre@baylibre.com>
2022-03-09 17:21:27 -05:00
lr sp, ___cpu_t_irq_stack_OFFSET(s0)
arch: added support for the riscv32 architecture RISC-V is an open-source instruction set architecture. Added support for the 32bit version of RISC-V to Zephyr. 1) exceptions/interrupts/faults are handled at the architecture level via the __irq_wrapper handler. Context saving/restoring of registers can be handled at both architecture and SOC levels. If SOC-specific registers need to be saved, SOC level needs to provide __soc_save_context and __soc_restore_context functions that shall be accounted by the architecture level, when corresponding config variable RISCV_SOC_CONTEXT_SAVE is set. 2) As RISC-V architecture does not provide a clear ISA specification about interrupt handling, each RISC-V SOC handles it in its own way. Hence, at the architecture level, the __irq_wrapper handler expects the following functions to be provided by the SOC level: __soc_is_irq: to check if the exception is the result of an interrupt or not. __soc_handle_irq: handle pending IRQ at SOC level (ex: clear pending IRQ in SOC-specific IRQ register) 3) Thread/task scheduling, as well as IRQ offloading are handled via the RISC-V system call ("ecall"), which is also handled via the __irq_wrapper handler. The _Swap asm function just calls "ecall" to generate an exception. 4) As there is no conventional way of handling CPU power save in RISC-V, the default nano_cpu_idle and nano_cpu_atomic_idle functions just unlock interrupts and return to the caller, without issuing any CPU power saving instruction. Nonetheless, to allow SOC-level to implement proper CPU power save, nano_cpu_idle and nano_cpu_atomic_idle functions are defined as __weak at the architecture level. Change-Id: I980a161d0009f3f404ad22b226a6229fbb492389 Signed-off-by: Jean-Paul Etienne <fractalclone@gmail.com>
2017-01-11 00:24:30 +01:00
/*
* Save thread stack pointer on interrupt stack
* In RISC-V, stack pointer needs to be 16-byte aligned
*/
addi sp, sp, -16
riscv: better abstraction for register-wide load/store opcodes Those are prominent enough that having RV_OP_LOADREG and RV_OP_STOREREG shouting at you all over the place is rather unpleasant and bad taste. Let's create pseudo-instructions of our own with assembler macros rather than preprocessor defines and only in assembly scope. This makes the asm code way more uniform and readable. Signed-off-by: Nicolas Pitre <npitre@baylibre.com>
2022-02-22 14:09:55 -05:00
sr t0, 0(sp)
arch: added support for the riscv32 architecture RISC-V is an open-source instruction set architecture. Added support for the 32bit version of RISC-V to Zephyr. 1) exceptions/interrupts/faults are handled at the architecture level via the __irq_wrapper handler. Context saving/restoring of registers can be handled at both architecture and SOC levels. If SOC-specific registers need to be saved, SOC level needs to provide __soc_save_context and __soc_restore_context functions that shall be accounted by the architecture level, when corresponding config variable RISCV_SOC_CONTEXT_SAVE is set. 2) As RISC-V architecture does not provide a clear ISA specification about interrupt handling, each RISC-V SOC handles it in its own way. Hence, at the architecture level, the __irq_wrapper handler expects the following functions to be provided by the SOC level: __soc_is_irq: to check if the exception is the result of an interrupt or not. __soc_handle_irq: handle pending IRQ at SOC level (ex: clear pending IRQ in SOC-specific IRQ register) 3) Thread/task scheduling, as well as IRQ offloading are handled via the RISC-V system call ("ecall"), which is also handled via the __irq_wrapper handler. The _Swap asm function just calls "ecall" to generate an exception. 4) As there is no conventional way of handling CPU power save in RISC-V, the default nano_cpu_idle and nano_cpu_atomic_idle functions just unlock interrupts and return to the caller, without issuing any CPU power saving instruction. Nonetheless, to allow SOC-level to implement proper CPU power save, nano_cpu_idle and nano_cpu_atomic_idle functions are defined as __weak at the architecture level. Change-Id: I980a161d0009f3f404ad22b226a6229fbb492389 Signed-off-by: Jean-Paul Etienne <fractalclone@gmail.com>
2017-01-11 00:24:30 +01:00
on_irq_stack:
tracing: add TRACING_ISR Kconfig Add TRACING_ISR Kconfig to help high latency backend working well. Currently the ISR tracing hook function is put at the begining and ending of ISR wrapper, when there is ISR needed in the tracing path (especially tracing backend), it will cause tracing buffer easily be exhausted if async tracing method enabled. Also it will increase system latency if all the ISRs are traced. So add TRACING_ISR to enable/disable ISR tracing here. Later a filter out mechanism based on irq number will be added. Signed-off-by: Wentong Wu <wentong.wu@intel.com>
2020-01-28 16:44:59 +08:00
#ifdef CONFIG_TRACING_ISR
debug: tracing: Remove unneeded abstraction Various C and Assembly modules make function calls to z_sys_trace_*. These merely call corresponding functions sys_trace_*. This commit is to simplify these by making direct function calls to the sys_trace_* functions from these modules. Subsequently, the z_sys_trace_* functions are removed. Signed-off-by: Mrinal Sen <msen@oticon.com>
2019-09-19 09:25:19 +02:00
call sys_trace_isr_enter
arch: added support for the riscv32 architecture RISC-V is an open-source instruction set architecture. Added support for the 32bit version of RISC-V to Zephyr. 1) exceptions/interrupts/faults are handled at the architecture level via the __irq_wrapper handler. Context saving/restoring of registers can be handled at both architecture and SOC levels. If SOC-specific registers need to be saved, SOC level needs to provide __soc_save_context and __soc_restore_context functions that shall be accounted by the architecture level, when corresponding config variable RISCV_SOC_CONTEXT_SAVE is set. 2) As RISC-V architecture does not provide a clear ISA specification about interrupt handling, each RISC-V SOC handles it in its own way. Hence, at the architecture level, the __irq_wrapper handler expects the following functions to be provided by the SOC level: __soc_is_irq: to check if the exception is the result of an interrupt or not. __soc_handle_irq: handle pending IRQ at SOC level (ex: clear pending IRQ in SOC-specific IRQ register) 3) Thread/task scheduling, as well as IRQ offloading are handled via the RISC-V system call ("ecall"), which is also handled via the __irq_wrapper handler. The _Swap asm function just calls "ecall" to generate an exception. 4) As there is no conventional way of handling CPU power save in RISC-V, the default nano_cpu_idle and nano_cpu_atomic_idle functions just unlock interrupts and return to the caller, without issuing any CPU power saving instruction. Nonetheless, to allow SOC-level to implement proper CPU power save, nano_cpu_idle and nano_cpu_atomic_idle functions are defined as __weak at the architecture level. Change-Id: I980a161d0009f3f404ad22b226a6229fbb492389 Signed-off-by: Jean-Paul Etienne <fractalclone@gmail.com>
2017-01-11 00:24:30 +01:00
#endif
/* Get IRQ causing interrupt */
csrr a0, mcause
riscv32: move riscv privileged architecture specifics within a common header file Added a riscv-privilege.h header file that contains common definitions for all riscv SOCs supporting the riscv privileged architecture specification. This shall ease addition of future riscv SOCs supporting the riscv privileged architecture spec. Change-Id: I5714bf70eeda738a25967ed26d3d0d2aaa0c9989 Signed-off-by: Jean-Paul Etienne <fractalclone@gmail.com>
2017-02-13 01:17:01 +01:00
li t0, SOC_MCAUSE_EXP_MASK
arch: added support for the riscv32 architecture RISC-V is an open-source instruction set architecture. Added support for the 32bit version of RISC-V to Zephyr. 1) exceptions/interrupts/faults are handled at the architecture level via the __irq_wrapper handler. Context saving/restoring of registers can be handled at both architecture and SOC levels. If SOC-specific registers need to be saved, SOC level needs to provide __soc_save_context and __soc_restore_context functions that shall be accounted by the architecture level, when corresponding config variable RISCV_SOC_CONTEXT_SAVE is set. 2) As RISC-V architecture does not provide a clear ISA specification about interrupt handling, each RISC-V SOC handles it in its own way. Hence, at the architecture level, the __irq_wrapper handler expects the following functions to be provided by the SOC level: __soc_is_irq: to check if the exception is the result of an interrupt or not. __soc_handle_irq: handle pending IRQ at SOC level (ex: clear pending IRQ in SOC-specific IRQ register) 3) Thread/task scheduling, as well as IRQ offloading are handled via the RISC-V system call ("ecall"), which is also handled via the __irq_wrapper handler. The _Swap asm function just calls "ecall" to generate an exception. 4) As there is no conventional way of handling CPU power save in RISC-V, the default nano_cpu_idle and nano_cpu_atomic_idle functions just unlock interrupts and return to the caller, without issuing any CPU power saving instruction. Nonetheless, to allow SOC-level to implement proper CPU power save, nano_cpu_idle and nano_cpu_atomic_idle functions are defined as __weak at the architecture level. Change-Id: I980a161d0009f3f404ad22b226a6229fbb492389 Signed-off-by: Jean-Paul Etienne <fractalclone@gmail.com>
2017-01-11 00:24:30 +01:00
and a0, a0, t0
/*
* Clear pending IRQ generating the interrupt at SOC level
* Pass IRQ number to __soc_handle_irq via register a0
*/
jal ra, __soc_handle_irq
/*
* Call corresponding registered function in _sw_isr_table.
riscv: make core code 64-bit compatible There are two aspects to this: CPU registers are twice as big, and the load and store instructions must use the 'd' suffix instead of the 'w' one. To abstract register differences, we simply use a ulong_t instead of u32_t given that RISC-V is either ILP32 or LP64. And the relevant lw/sw instructions are replaced by LR/SR (load/store register) that get defined as either lw/sw or ld/sd. Finally a few constants to deal with register offsets are also provided. Signed-off-by: Nicolas Pitre <npitre@baylibre.com>
2019-07-24 16:21:58 -04:00
* (table is 2-word wide, we should shift index accordingly)
arch: added support for the riscv32 architecture RISC-V is an open-source instruction set architecture. Added support for the 32bit version of RISC-V to Zephyr. 1) exceptions/interrupts/faults are handled at the architecture level via the __irq_wrapper handler. Context saving/restoring of registers can be handled at both architecture and SOC levels. If SOC-specific registers need to be saved, SOC level needs to provide __soc_save_context and __soc_restore_context functions that shall be accounted by the architecture level, when corresponding config variable RISCV_SOC_CONTEXT_SAVE is set. 2) As RISC-V architecture does not provide a clear ISA specification about interrupt handling, each RISC-V SOC handles it in its own way. Hence, at the architecture level, the __irq_wrapper handler expects the following functions to be provided by the SOC level: __soc_is_irq: to check if the exception is the result of an interrupt or not. __soc_handle_irq: handle pending IRQ at SOC level (ex: clear pending IRQ in SOC-specific IRQ register) 3) Thread/task scheduling, as well as IRQ offloading are handled via the RISC-V system call ("ecall"), which is also handled via the __irq_wrapper handler. The _Swap asm function just calls "ecall" to generate an exception. 4) As there is no conventional way of handling CPU power save in RISC-V, the default nano_cpu_idle and nano_cpu_atomic_idle functions just unlock interrupts and return to the caller, without issuing any CPU power saving instruction. Nonetheless, to allow SOC-level to implement proper CPU power save, nano_cpu_idle and nano_cpu_atomic_idle functions are defined as __weak at the architecture level. Change-Id: I980a161d0009f3f404ad22b226a6229fbb492389 Signed-off-by: Jean-Paul Etienne <fractalclone@gmail.com>
2017-01-11 00:24:30 +01:00
*/
la t0, _sw_isr_table
riscv: make core code 64-bit compatible There are two aspects to this: CPU registers are twice as big, and the load and store instructions must use the 'd' suffix instead of the 'w' one. To abstract register differences, we simply use a ulong_t instead of u32_t given that RISC-V is either ILP32 or LP64. And the relevant lw/sw instructions are replaced by LR/SR (load/store register) that get defined as either lw/sw or ld/sd. Finally a few constants to deal with register offsets are also provided. Signed-off-by: Nicolas Pitre <npitre@baylibre.com>
2019-07-24 16:21:58 -04:00
slli a0, a0, (RV_REGSHIFT + 1)
arch: added support for the riscv32 architecture RISC-V is an open-source instruction set architecture. Added support for the 32bit version of RISC-V to Zephyr. 1) exceptions/interrupts/faults are handled at the architecture level via the __irq_wrapper handler. Context saving/restoring of registers can be handled at both architecture and SOC levels. If SOC-specific registers need to be saved, SOC level needs to provide __soc_save_context and __soc_restore_context functions that shall be accounted by the architecture level, when corresponding config variable RISCV_SOC_CONTEXT_SAVE is set. 2) As RISC-V architecture does not provide a clear ISA specification about interrupt handling, each RISC-V SOC handles it in its own way. Hence, at the architecture level, the __irq_wrapper handler expects the following functions to be provided by the SOC level: __soc_is_irq: to check if the exception is the result of an interrupt or not. __soc_handle_irq: handle pending IRQ at SOC level (ex: clear pending IRQ in SOC-specific IRQ register) 3) Thread/task scheduling, as well as IRQ offloading are handled via the RISC-V system call ("ecall"), which is also handled via the __irq_wrapper handler. The _Swap asm function just calls "ecall" to generate an exception. 4) As there is no conventional way of handling CPU power save in RISC-V, the default nano_cpu_idle and nano_cpu_atomic_idle functions just unlock interrupts and return to the caller, without issuing any CPU power saving instruction. Nonetheless, to allow SOC-level to implement proper CPU power save, nano_cpu_idle and nano_cpu_atomic_idle functions are defined as __weak at the architecture level. Change-Id: I980a161d0009f3f404ad22b226a6229fbb492389 Signed-off-by: Jean-Paul Etienne <fractalclone@gmail.com>
2017-01-11 00:24:30 +01:00
add t0, t0, a0
/* Load argument in a0 register */
riscv: better abstraction for register-wide load/store opcodes Those are prominent enough that having RV_OP_LOADREG and RV_OP_STOREREG shouting at you all over the place is rather unpleasant and bad taste. Let's create pseudo-instructions of our own with assembler macros rather than preprocessor defines and only in assembly scope. This makes the asm code way more uniform and readable. Signed-off-by: Nicolas Pitre <npitre@baylibre.com>
2022-02-22 14:09:55 -05:00
lr a0, 0(t0)
arch: added support for the riscv32 architecture RISC-V is an open-source instruction set architecture. Added support for the 32bit version of RISC-V to Zephyr. 1) exceptions/interrupts/faults are handled at the architecture level via the __irq_wrapper handler. Context saving/restoring of registers can be handled at both architecture and SOC levels. If SOC-specific registers need to be saved, SOC level needs to provide __soc_save_context and __soc_restore_context functions that shall be accounted by the architecture level, when corresponding config variable RISCV_SOC_CONTEXT_SAVE is set. 2) As RISC-V architecture does not provide a clear ISA specification about interrupt handling, each RISC-V SOC handles it in its own way. Hence, at the architecture level, the __irq_wrapper handler expects the following functions to be provided by the SOC level: __soc_is_irq: to check if the exception is the result of an interrupt or not. __soc_handle_irq: handle pending IRQ at SOC level (ex: clear pending IRQ in SOC-specific IRQ register) 3) Thread/task scheduling, as well as IRQ offloading are handled via the RISC-V system call ("ecall"), which is also handled via the __irq_wrapper handler. The _Swap asm function just calls "ecall" to generate an exception. 4) As there is no conventional way of handling CPU power save in RISC-V, the default nano_cpu_idle and nano_cpu_atomic_idle functions just unlock interrupts and return to the caller, without issuing any CPU power saving instruction. Nonetheless, to allow SOC-level to implement proper CPU power save, nano_cpu_idle and nano_cpu_atomic_idle functions are defined as __weak at the architecture level. Change-Id: I980a161d0009f3f404ad22b226a6229fbb492389 Signed-off-by: Jean-Paul Etienne <fractalclone@gmail.com>
2017-01-11 00:24:30 +01:00
/* Load ISR function address in register t1 */
riscv: better abstraction for register-wide load/store opcodes Those are prominent enough that having RV_OP_LOADREG and RV_OP_STOREREG shouting at you all over the place is rather unpleasant and bad taste. Let's create pseudo-instructions of our own with assembler macros rather than preprocessor defines and only in assembly scope. This makes the asm code way more uniform and readable. Signed-off-by: Nicolas Pitre <npitre@baylibre.com>
2022-02-22 14:09:55 -05:00
lr t1, RV_REGSIZE(t0)
arch: added support for the riscv32 architecture RISC-V is an open-source instruction set architecture. Added support for the 32bit version of RISC-V to Zephyr. 1) exceptions/interrupts/faults are handled at the architecture level via the __irq_wrapper handler. Context saving/restoring of registers can be handled at both architecture and SOC levels. If SOC-specific registers need to be saved, SOC level needs to provide __soc_save_context and __soc_restore_context functions that shall be accounted by the architecture level, when corresponding config variable RISCV_SOC_CONTEXT_SAVE is set. 2) As RISC-V architecture does not provide a clear ISA specification about interrupt handling, each RISC-V SOC handles it in its own way. Hence, at the architecture level, the __irq_wrapper handler expects the following functions to be provided by the SOC level: __soc_is_irq: to check if the exception is the result of an interrupt or not. __soc_handle_irq: handle pending IRQ at SOC level (ex: clear pending IRQ in SOC-specific IRQ register) 3) Thread/task scheduling, as well as IRQ offloading are handled via the RISC-V system call ("ecall"), which is also handled via the __irq_wrapper handler. The _Swap asm function just calls "ecall" to generate an exception. 4) As there is no conventional way of handling CPU power save in RISC-V, the default nano_cpu_idle and nano_cpu_atomic_idle functions just unlock interrupts and return to the caller, without issuing any CPU power saving instruction. Nonetheless, to allow SOC-level to implement proper CPU power save, nano_cpu_idle and nano_cpu_atomic_idle functions are defined as __weak at the architecture level. Change-Id: I980a161d0009f3f404ad22b226a6229fbb492389 Signed-off-by: Jean-Paul Etienne <fractalclone@gmail.com>
2017-01-11 00:24:30 +01:00
/* Call ISR function */
riscv: fix non-standard assembly of RISC-V Non-standard `jalr rd, rs` pseudo-instructions are used. This commit changes them to `ret` for standard return pseudo-instruction or `jalr rd, rs, 0` for no offset jump register and link. Fixes #41100. Signed-off-by: Henry Hsieh <r901042004@yahoo.com.tw>
2021-12-14 22:46:42 +08:00
jalr ra, t1, 0
arch: added support for the riscv32 architecture RISC-V is an open-source instruction set architecture. Added support for the 32bit version of RISC-V to Zephyr. 1) exceptions/interrupts/faults are handled at the architecture level via the __irq_wrapper handler. Context saving/restoring of registers can be handled at both architecture and SOC levels. If SOC-specific registers need to be saved, SOC level needs to provide __soc_save_context and __soc_restore_context functions that shall be accounted by the architecture level, when corresponding config variable RISCV_SOC_CONTEXT_SAVE is set. 2) As RISC-V architecture does not provide a clear ISA specification about interrupt handling, each RISC-V SOC handles it in its own way. Hence, at the architecture level, the __irq_wrapper handler expects the following functions to be provided by the SOC level: __soc_is_irq: to check if the exception is the result of an interrupt or not. __soc_handle_irq: handle pending IRQ at SOC level (ex: clear pending IRQ in SOC-specific IRQ register) 3) Thread/task scheduling, as well as IRQ offloading are handled via the RISC-V system call ("ecall"), which is also handled via the __irq_wrapper handler. The _Swap asm function just calls "ecall" to generate an exception. 4) As there is no conventional way of handling CPU power save in RISC-V, the default nano_cpu_idle and nano_cpu_atomic_idle functions just unlock interrupts and return to the caller, without issuing any CPU power saving instruction. Nonetheless, to allow SOC-level to implement proper CPU power save, nano_cpu_idle and nano_cpu_atomic_idle functions are defined as __weak at the architecture level. Change-Id: I980a161d0009f3f404ad22b226a6229fbb492389 Signed-off-by: Jean-Paul Etienne <fractalclone@gmail.com>
2017-01-11 00:24:30 +01:00
tracing: riscv: Add missing invoke of sys_trace_isr_exit() Change suggested by @WealianLiao in #41995. Signed-off-by: Robert Szczepanski <rszczepanski@antmicro.com>
2022-05-11 12:29:46 +02:00
#ifdef CONFIG_TRACING_ISR
call sys_trace_isr_exit
#endif
riscv: irq_offload: simpler implementation Get rid of all those global variables and IRQ locking. Use the regular IRQ exit path to let tests validate preemption properly. Signed-off-by: Nicolas Pitre <npitre@baylibre.com>
2022-03-07 13:24:38 -05:00
irq_done:
riscv: isr.S: dedicate a register to &current_cpu Stop using &_kernel as this is not SMP friendly. Let's use s0 (after preserving its content) to hold &current_cpu instead so it won't have to be reloaded each time it is needed. This will be even more relevant when SMP support is added. Signed-off-by: Nicolas Pitre <npitre@baylibre.com>
2022-03-09 17:21:27 -05:00
/* Decrement _current_cpu->nested */
lw t2, ___cpu_t_nested_OFFSET(s0)
arch: added support for the riscv32 architecture RISC-V is an open-source instruction set architecture. Added support for the 32bit version of RISC-V to Zephyr. 1) exceptions/interrupts/faults are handled at the architecture level via the __irq_wrapper handler. Context saving/restoring of registers can be handled at both architecture and SOC levels. If SOC-specific registers need to be saved, SOC level needs to provide __soc_save_context and __soc_restore_context functions that shall be accounted by the architecture level, when corresponding config variable RISCV_SOC_CONTEXT_SAVE is set. 2) As RISC-V architecture does not provide a clear ISA specification about interrupt handling, each RISC-V SOC handles it in its own way. Hence, at the architecture level, the __irq_wrapper handler expects the following functions to be provided by the SOC level: __soc_is_irq: to check if the exception is the result of an interrupt or not. __soc_handle_irq: handle pending IRQ at SOC level (ex: clear pending IRQ in SOC-specific IRQ register) 3) Thread/task scheduling, as well as IRQ offloading are handled via the RISC-V system call ("ecall"), which is also handled via the __irq_wrapper handler. The _Swap asm function just calls "ecall" to generate an exception. 4) As there is no conventional way of handling CPU power save in RISC-V, the default nano_cpu_idle and nano_cpu_atomic_idle functions just unlock interrupts and return to the caller, without issuing any CPU power saving instruction. Nonetheless, to allow SOC-level to implement proper CPU power save, nano_cpu_idle and nano_cpu_atomic_idle functions are defined as __weak at the architecture level. Change-Id: I980a161d0009f3f404ad22b226a6229fbb492389 Signed-off-by: Jean-Paul Etienne <fractalclone@gmail.com>
2017-01-11 00:24:30 +01:00
addi t2, t2, -1
riscv: isr.S: dedicate a register to &current_cpu Stop using &_kernel as this is not SMP friendly. Let's use s0 (after preserving its content) to hold &current_cpu instead so it won't have to be reloaded each time it is needed. This will be even more relevant when SMP support is added. Signed-off-by: Nicolas Pitre <npitre@baylibre.com>
2022-03-09 17:21:27 -05:00
sw t2, ___cpu_t_nested_OFFSET(s0)
riscv: exception code mega simplification and optimization Complete revamp of the exception entry code, including syscall handling. Proper syscall frame exception trigger. Many correctness fixes, hacks removal, etc. etc. I tried to make this into several commits, but this stuff is all inter-related and a pain to split. The diffstat summary: 14 files changed, 250 insertions(+), 802 deletions(-) Binary size (before): text data bss dec hex filename 1104 0 0 1104 450 isr.S.obj 64 0 0 64 40 userspace.S.obj Binary size (after): text data bss dec hex filename 600 0 0 600 258 isr.S.obj 36 0 0 36 24 userspace.S.obj Run of samples/userspace/syscall_perf (before): *** Booting Zephyr OS build zephyr-v3.0.0-325-g3748accae018 *** Main Thread started; qemu_riscv32 Supervisor thread started User thread started Supervisor thread(0x80010048): 384 cycles 509 instructions User thread(0x80010140): 77312 cycles 77437 instructions Run of samples/userspace/syscall_perf (after): *** Booting Zephyr OS build zephyr-v3.0.0-326-g4c877a2753b3 *** Main Thread started; qemu_riscv32 Supervisor thread started User thread started Supervisor thread(0x80010048): 384 cycles 509 instructions User thread(0x80010138): 7040 cycles 7165 instructions Yes, that's more than a 10x speed-up! Signed-off-by: Nicolas Pitre <npitre@baylibre.com>
2022-02-24 22:30:03 -05:00
bnez t2, no_reschedule
arch: added support for the riscv32 architecture RISC-V is an open-source instruction set architecture. Added support for the 32bit version of RISC-V to Zephyr. 1) exceptions/interrupts/faults are handled at the architecture level via the __irq_wrapper handler. Context saving/restoring of registers can be handled at both architecture and SOC levels. If SOC-specific registers need to be saved, SOC level needs to provide __soc_save_context and __soc_restore_context functions that shall be accounted by the architecture level, when corresponding config variable RISCV_SOC_CONTEXT_SAVE is set. 2) As RISC-V architecture does not provide a clear ISA specification about interrupt handling, each RISC-V SOC handles it in its own way. Hence, at the architecture level, the __irq_wrapper handler expects the following functions to be provided by the SOC level: __soc_is_irq: to check if the exception is the result of an interrupt or not. __soc_handle_irq: handle pending IRQ at SOC level (ex: clear pending IRQ in SOC-specific IRQ register) 3) Thread/task scheduling, as well as IRQ offloading are handled via the RISC-V system call ("ecall"), which is also handled via the __irq_wrapper handler. The _Swap asm function just calls "ecall" to generate an exception. 4) As there is no conventional way of handling CPU power save in RISC-V, the default nano_cpu_idle and nano_cpu_atomic_idle functions just unlock interrupts and return to the caller, without issuing any CPU power saving instruction. Nonetheless, to allow SOC-level to implement proper CPU power save, nano_cpu_idle and nano_cpu_atomic_idle functions are defined as __weak at the architecture level. Change-Id: I980a161d0009f3f404ad22b226a6229fbb492389 Signed-off-by: Jean-Paul Etienne <fractalclone@gmail.com>
2017-01-11 00:24:30 +01:00
riscv: exception code mega simplification and optimization Complete revamp of the exception entry code, including syscall handling. Proper syscall frame exception trigger. Many correctness fixes, hacks removal, etc. etc. I tried to make this into several commits, but this stuff is all inter-related and a pain to split. The diffstat summary: 14 files changed, 250 insertions(+), 802 deletions(-) Binary size (before): text data bss dec hex filename 1104 0 0 1104 450 isr.S.obj 64 0 0 64 40 userspace.S.obj Binary size (after): text data bss dec hex filename 600 0 0 600 258 isr.S.obj 36 0 0 36 24 userspace.S.obj Run of samples/userspace/syscall_perf (before): *** Booting Zephyr OS build zephyr-v3.0.0-325-g3748accae018 *** Main Thread started; qemu_riscv32 Supervisor thread started User thread started Supervisor thread(0x80010048): 384 cycles 509 instructions User thread(0x80010140): 77312 cycles 77437 instructions Run of samples/userspace/syscall_perf (after): *** Booting Zephyr OS build zephyr-v3.0.0-326-g4c877a2753b3 *** Main Thread started; qemu_riscv32 Supervisor thread started User thread started Supervisor thread(0x80010048): 384 cycles 509 instructions User thread(0x80010138): 7040 cycles 7165 instructions Yes, that's more than a 10x speed-up! Signed-off-by: Nicolas Pitre <npitre@baylibre.com>
2022-02-24 22:30:03 -05:00
/* nested count is back to 0: Return to thread stack */
lr sp, 0(sp)
stack_sentinel: change cooperative check One of the stack sentinel policies was to check the sentinel any time a cooperative context switch is done (i.e, _Swap is called). This was done by adding a hook to _check_stack_sentinel in every arch's __swap function. This way is cleaner as we just have the hook in one inline function rather than implemented in several different assembly dialects. The check upon interrupt is now made unconditionally rather than checking if we are calling __swap, since the check now is only called on cooperative _Swap(). The interrupt is always serviced first. Issue: ZEP-2244 Signed-off-by: Andrew Boie <andrew.p.boie@intel.com>
2017-06-07 09:33:16 -07:00
debug: add stack sentinel feature This places a sentinel value at the lowest 4 bytes of a stack memory region and checks it at various intervals, including when servicing interrupts or context switching. This is implemented on all arches except ARC, which supports stack bounds checking directly in hardware. Signed-off-by: Andrew Boie <andrew.p.boie@intel.com>
2017-05-11 13:29:15 -07:00
#ifdef CONFIG_STACK_SENTINEL
all: Update reserved function names Update reserved function names starting with one underscore, replacing them as follows: '_k_' with 'z_' '_K_' with 'Z_' '_handler_' with 'z_handl_' '_Cstart' with 'z_cstart' '_Swap' with 'z_swap' This renaming is done on both global and those static function names in kernel/include and include/. Other static function names in kernel/ are renamed by removing the leading underscore. Other function names not starting with any prefix listed above are renamed starting with a 'z_' or 'Z_' prefix. Function names starting with two or three leading underscores are not automatcally renamed since these names will collide with the variants with two or three leading underscores. Various generator scripts have also been updated as well as perf, linker and usb files. These are drivers/serial/uart_handlers.c include/linker/kobject-text.ld kernel/include/syscall_handler.h scripts/gen_kobject_list.py scripts/gen_syscall_header.py Signed-off-by: Patrik Flykt <patrik.flykt@intel.com>
2019-03-08 14:19:05 -07:00
call z_check_stack_sentinel
debug: add stack sentinel feature This places a sentinel value at the lowest 4 bytes of a stack memory region and checks it at various intervals, including when servicing interrupts or context switching. This is implemented on all arches except ARC, which supports stack bounds checking directly in hardware. Signed-off-by: Andrew Boie <andrew.p.boie@intel.com>
2017-05-11 13:29:15 -07:00
#endif
riscv: Introduce RISCV_ALWAYS_SWITCH_THROUGH_ECALL Some early RISC-V SoCs have a problem when an `mret` instruction is used outside a trap handler. After the latest Zephyr RISC-V huge rework, the arch_switch code is indeed calling `mret` when not in handler mode, breaking some early RISC-V platforms. Optionally restore the old behavior by adding a new CONFIG_RISCV_ALWAYS_SWITCH_THROUGH_ECALL symbol. Signed-off-by: Nicolas Pitre <npitre@baylibre.com> Signed-off-by: Carlo Caione <ccaione@baylibre.com>
2022-06-30 14:40:52 +02:00
check_reschedule:
Revert "arch/riscv: Use arch_switch() for context swap" This reverts commit be28de692c5e2c5546841c49d6a85d5f046474a4. The purpose of this commit will be reintroduced later on top of a cleaner codebase. Signed-off-by: Nicolas Pitre <npitre@baylibre.com>
2022-03-09 17:46:30 -05:00
riscv: isr.S: dedicate a register to &current_cpu Stop using &_kernel as this is not SMP friendly. Let's use s0 (after preserving its content) to hold &current_cpu instead so it won't have to be reloaded each time it is needed. This will be even more relevant when SMP support is added. Signed-off-by: Nicolas Pitre <npitre@baylibre.com>
2022-03-09 17:21:27 -05:00
/* Get pointer to current thread on this CPU */
lr a1, ___cpu_t_current_OFFSET(s0)
arch: riscv: add memory protection support The IRQ handler has had a major changes to manage syscall, reschedule and interrupt from user thread and stack guard. Add userspace support: - Use a global variable to know if the current execution is user or machine. The location of this variable is read only for all user thread and read/write for kernel thread. - Memory shared is supported. - Use dynamic allocation to optimize PMP slot usage. If the area size is a power of 2, only one PMP slot is used, else 2 are used. Add stack guard support: - Use MPRV bit to force PMP rules to machine mode execution. - IRQ stack have a locked stack guard to avoid re-write PMP configuration registers for each interruption and then win some cycle. - The IRQ stack is used as "temporary" stack at the beginning of IRQ handler to save current ESF. That avoid to trigger write fault on thread stack during store ESF which that call IRQ handler to infinity. - A stack guard is also setup for privileged stack of a user thread. Thread: - A PMP setup is specific to each thread. PMP setup are saved in each thread structure to improve reschedule performance. Signed-off-by: Alexandre Mergnat <amergnat@baylibre.com> Reviewed-by: Nicolas Royer <nroyer@baylibre.com>
2020-07-21 16:00:39 +02:00
arch: added support for the riscv32 architecture RISC-V is an open-source instruction set architecture. Added support for the 32bit version of RISC-V to Zephyr. 1) exceptions/interrupts/faults are handled at the architecture level via the __irq_wrapper handler. Context saving/restoring of registers can be handled at both architecture and SOC levels. If SOC-specific registers need to be saved, SOC level needs to provide __soc_save_context and __soc_restore_context functions that shall be accounted by the architecture level, when corresponding config variable RISCV_SOC_CONTEXT_SAVE is set. 2) As RISC-V architecture does not provide a clear ISA specification about interrupt handling, each RISC-V SOC handles it in its own way. Hence, at the architecture level, the __irq_wrapper handler expects the following functions to be provided by the SOC level: __soc_is_irq: to check if the exception is the result of an interrupt or not. __soc_handle_irq: handle pending IRQ at SOC level (ex: clear pending IRQ in SOC-specific IRQ register) 3) Thread/task scheduling, as well as IRQ offloading are handled via the RISC-V system call ("ecall"), which is also handled via the __irq_wrapper handler. The _Swap asm function just calls "ecall" to generate an exception. 4) As there is no conventional way of handling CPU power save in RISC-V, the default nano_cpu_idle and nano_cpu_atomic_idle functions just unlock interrupts and return to the caller, without issuing any CPU power saving instruction. Nonetheless, to allow SOC-level to implement proper CPU power save, nano_cpu_idle and nano_cpu_atomic_idle functions are defined as __weak at the architecture level. Change-Id: I980a161d0009f3f404ad22b226a6229fbb492389 Signed-off-by: Jean-Paul Etienne <fractalclone@gmail.com>
2017-01-11 00:24:30 +01:00
/*
riscv: implement arch_switch() The move to arch_switch() is a prerequisite for SMP support. Make it optimal without the need for an ECALL roundtrip on every context switch. Performance numbers from tests/benchmarks/sched: Before: unpend 107 ready 102 switch 188 pend 218 tot 615 (avg 615) After: unpend 107 ready 102 switch 170 pend 217 tot 596 (avg 595) Signed-off-by: Nicolas Pitre <npitre@baylibre.com>
2022-03-07 17:01:36 -05:00
* Get next thread to schedule with z_get_next_switch_handle().
* We pass it a NULL as we didn't save the whole thread context yet.
* If no scheduling is necessary then NULL will be returned.
arch: added support for the riscv32 architecture RISC-V is an open-source instruction set architecture. Added support for the 32bit version of RISC-V to Zephyr. 1) exceptions/interrupts/faults are handled at the architecture level via the __irq_wrapper handler. Context saving/restoring of registers can be handled at both architecture and SOC levels. If SOC-specific registers need to be saved, SOC level needs to provide __soc_save_context and __soc_restore_context functions that shall be accounted by the architecture level, when corresponding config variable RISCV_SOC_CONTEXT_SAVE is set. 2) As RISC-V architecture does not provide a clear ISA specification about interrupt handling, each RISC-V SOC handles it in its own way. Hence, at the architecture level, the __irq_wrapper handler expects the following functions to be provided by the SOC level: __soc_is_irq: to check if the exception is the result of an interrupt or not. __soc_handle_irq: handle pending IRQ at SOC level (ex: clear pending IRQ in SOC-specific IRQ register) 3) Thread/task scheduling, as well as IRQ offloading are handled via the RISC-V system call ("ecall"), which is also handled via the __irq_wrapper handler. The _Swap asm function just calls "ecall" to generate an exception. 4) As there is no conventional way of handling CPU power save in RISC-V, the default nano_cpu_idle and nano_cpu_atomic_idle functions just unlock interrupts and return to the caller, without issuing any CPU power saving instruction. Nonetheless, to allow SOC-level to implement proper CPU power save, nano_cpu_idle and nano_cpu_atomic_idle functions are defined as __weak at the architecture level. Change-Id: I980a161d0009f3f404ad22b226a6229fbb492389 Signed-off-by: Jean-Paul Etienne <fractalclone@gmail.com>
2017-01-11 00:24:30 +01:00
*/
riscv: implement arch_switch() The move to arch_switch() is a prerequisite for SMP support. Make it optimal without the need for an ECALL roundtrip on every context switch. Performance numbers from tests/benchmarks/sched: Before: unpend 107 ready 102 switch 188 pend 218 tot 615 (avg 615) After: unpend 107 ready 102 switch 170 pend 217 tot 596 (avg 595) Signed-off-by: Nicolas Pitre <npitre@baylibre.com>
2022-03-07 17:01:36 -05:00
addi sp, sp, -16
sr a1, 0(sp)
mv a0, zero
call z_get_next_switch_handle
lr a1, 0(sp)
addi sp, sp, 16
beqz a0, no_reschedule
riscv: Add support for floating point This change adds full shared floating point support for the RISCV architecture with minimal impact on threads with floating point support not enabled. Signed-off-by: Corey Wharton <coreyw7@fb.com>
2020-03-11 18:15:29 -07:00
riscv: Introduce RISCV_ALWAYS_SWITCH_THROUGH_ECALL Some early RISC-V SoCs have a problem when an `mret` instruction is used outside a trap handler. After the latest Zephyr RISC-V huge rework, the arch_switch code is indeed calling `mret` when not in handler mode, breaking some early RISC-V platforms. Optionally restore the old behavior by adding a new CONFIG_RISCV_ALWAYS_SWITCH_THROUGH_ECALL symbol. Signed-off-by: Nicolas Pitre <npitre@baylibre.com> Signed-off-by: Carlo Caione <ccaione@baylibre.com>
2022-06-30 14:40:52 +02:00
reschedule:
riscv: Add support for floating point This change adds full shared floating point support for the RISCV architecture with minimal impact on threads with floating point support not enabled. Signed-off-by: Corey Wharton <coreyw7@fb.com>
2020-03-11 18:15:29 -07:00
/*
riscv: implement arch_switch() The move to arch_switch() is a prerequisite for SMP support. Make it optimal without the need for an ECALL roundtrip on every context switch. Performance numbers from tests/benchmarks/sched: Before: unpend 107 ready 102 switch 188 pend 218 tot 615 (avg 615) After: unpend 107 ready 102 switch 170 pend 217 tot 596 (avg 595) Signed-off-by: Nicolas Pitre <npitre@baylibre.com>
2022-03-07 17:01:36 -05:00
* Perform context switch:
* a0 = new thread
* a1 = old thread
riscv: Add support for floating point This change adds full shared floating point support for the RISCV architecture with minimal impact on threads with floating point support not enabled. Signed-off-by: Corey Wharton <coreyw7@fb.com>
2020-03-11 18:15:29 -07:00
*/
riscv: implement arch_switch() The move to arch_switch() is a prerequisite for SMP support. Make it optimal without the need for an ECALL roundtrip on every context switch. Performance numbers from tests/benchmarks/sched: Before: unpend 107 ready 102 switch 188 pend 218 tot 615 (avg 615) After: unpend 107 ready 102 switch 170 pend 217 tot 596 (avg 595) Signed-off-by: Nicolas Pitre <npitre@baylibre.com>
2022-03-07 17:01:36 -05:00
call z_riscv_switch
tests: benchmarks: timing_info: Enable benchmarks for riscv32. This patch provides support needed to get timing related information from riscv32 based SOC. Signed-off-by: Adithya Baglody <adithya.nagaraj.baglody@intel.com>
2018-07-23 15:54:22 +05:30
riscv: implement arch_switch() The move to arch_switch() is a prerequisite for SMP support. Make it optimal without the need for an ECALL roundtrip on every context switch. Performance numbers from tests/benchmarks/sched: Before: unpend 107 ready 102 switch 188 pend 218 tot 615 (avg 615) After: unpend 107 ready 102 switch 170 pend 217 tot 596 (avg 595) Signed-off-by: Nicolas Pitre <npitre@baylibre.com>
2022-03-07 17:01:36 -05:00
z_riscv_thread_start:
riscv: isr.S: dedicate a register to &current_cpu Stop using &_kernel as this is not SMP friendly. Let's use s0 (after preserving its content) to hold &current_cpu instead so it won't have to be reloaded each time it is needed. This will be even more relevant when SMP support is added. Signed-off-by: Nicolas Pitre <npitre@baylibre.com>
2022-03-09 17:21:27 -05:00
might_have_rescheduled:
riscv: integrate the new FPU context switching support FPU context switching is always performed on demand through the FPU access exception handler. Actual task switching only grants or denies FPU access depending on the current FPU owner. Because RISC-V doesn't have a dedicated FPU access exception, we must catch the Illegal Instruction exception and look for actual FP opcodes. There is no longer a need to allocate FPU storage on the stack for every exception making esf smaller and stack overflows less likely. Signed-off-by: Nicolas Pitre <npitre@baylibre.com>
2023-01-17 23:32:21 -05:00
/* reload s0 with &_current_cpu as it might have changed or be unset */
riscv: decouple the Zephyr CPU number from the hart ID Currently it is assumed that Zephyr CPU numbers match their hartid value one for one. This assumption was relied upon to efficiently retrieve the current CPU's `struct _cpu` pointer. People are starting to have systems with a mix of different usage for each CPU and such assumption may no longer be true. Let's completely decouple the hartid from the Zephyr CPU number by stuffing each CPU's `struct _cpu` pointer in their respective scratch register instead. `arch_curr_cpu()` becomes more efficient as well. Since the scratch register was previously used to store userspace's exception stack pointer, that is now moved into `struct _cpu_arch` which implied minor user space entry code cleanup and rationalization. Signed-off-by: Nicolas Pitre <npitre@baylibre.com>
2023-01-06 17:46:21 -05:00
get_current_cpu s0
riscv: isr.S: dedicate a register to &current_cpu Stop using &_kernel as this is not SMP friendly. Let's use s0 (after preserving its content) to hold &current_cpu instead so it won't have to be reloaded each time it is needed. This will be even more relevant when SMP support is added. Signed-off-by: Nicolas Pitre <npitre@baylibre.com>
2022-03-09 17:21:27 -05:00
riscv: exception code mega simplification and optimization Complete revamp of the exception entry code, including syscall handling. Proper syscall frame exception trigger. Many correctness fixes, hacks removal, etc. etc. I tried to make this into several commits, but this stuff is all inter-related and a pain to split. The diffstat summary: 14 files changed, 250 insertions(+), 802 deletions(-) Binary size (before): text data bss dec hex filename 1104 0 0 1104 450 isr.S.obj 64 0 0 64 40 userspace.S.obj Binary size (after): text data bss dec hex filename 600 0 0 600 258 isr.S.obj 36 0 0 36 24 userspace.S.obj Run of samples/userspace/syscall_perf (before): *** Booting Zephyr OS build zephyr-v3.0.0-325-g3748accae018 *** Main Thread started; qemu_riscv32 Supervisor thread started User thread started Supervisor thread(0x80010048): 384 cycles 509 instructions User thread(0x80010140): 77312 cycles 77437 instructions Run of samples/userspace/syscall_perf (after): *** Booting Zephyr OS build zephyr-v3.0.0-326-g4c877a2753b3 *** Main Thread started; qemu_riscv32 Supervisor thread started User thread started Supervisor thread(0x80010048): 384 cycles 509 instructions User thread(0x80010138): 7040 cycles 7165 instructions Yes, that's more than a 10x speed-up! Signed-off-by: Nicolas Pitre <npitre@baylibre.com>
2022-02-24 22:30:03 -05:00
no_reschedule:
arch: riscv: add memory protection support The IRQ handler has had a major changes to manage syscall, reschedule and interrupt from user thread and stack guard. Add userspace support: - Use a global variable to know if the current execution is user or machine. The location of this variable is read only for all user thread and read/write for kernel thread. - Memory shared is supported. - Use dynamic allocation to optimize PMP slot usage. If the area size is a power of 2, only one PMP slot is used, else 2 are used. Add stack guard support: - Use MPRV bit to force PMP rules to machine mode execution. - IRQ stack have a locked stack guard to avoid re-write PMP configuration registers for each interruption and then win some cycle. - The IRQ stack is used as "temporary" stack at the beginning of IRQ handler to save current ESF. That avoid to trigger write fault on thread stack during store ESF which that call IRQ handler to infinity. - A stack guard is also setup for privileged stack of a user thread. Thread: - A PMP setup is specific to each thread. PMP setup are saved in each thread structure to improve reschedule performance. Signed-off-by: Alexandre Mergnat <amergnat@baylibre.com> Reviewed-by: Nicolas Royer <nroyer@baylibre.com>
2020-07-21 16:00:39 +02:00
#ifdef CONFIG_RISCV_SOC_CONTEXT_SAVE
/* Restore context at SOC level */
addi a0, sp, __z_arch_esf_t_soc_context_OFFSET
jal ra, __soc_restore_context
#endif /* CONFIG_RISCV_SOC_CONTEXT_SAVE */
riscv: integrate the new FPU context switching support FPU context switching is always performed on demand through the FPU access exception handler. Actual task switching only grants or denies FPU access depending on the current FPU owner. Because RISC-V doesn't have a dedicated FPU access exception, we must catch the Illegal Instruction exception and look for actual FP opcodes. There is no longer a need to allocate FPU storage on the stack for every exception making esf smaller and stack overflows less likely. Signed-off-by: Nicolas Pitre <npitre@baylibre.com>
2023-01-17 23:32:21 -05:00
#if defined(CONFIG_FPU_SHARING)
/* FPU handling upon exception mode exit */
mv a0, sp
call z_riscv_fpu_exit_exc
riscv: isr.S: optimize FP regs save/restore decision Rely on mstatus rather than thread->base.user_options since it is always up to date (updated by z_riscv_switch) to simplify the code and be SMP proof. Also carry over SF_INIT to the mstatus being restored in case it was changed in the mean time. Signed-off-by: Nicolas Pitre <npitre@baylibre.com>
2022-03-07 17:31:13 -05:00
riscv: integrate the new FPU context switching support FPU context switching is always performed on demand through the FPU access exception handler. Actual task switching only grants or denies FPU access depending on the current FPU owner. Because RISC-V doesn't have a dedicated FPU access exception, we must catch the Illegal Instruction exception and look for actual FP opcodes. There is no longer a need to allocate FPU storage on the stack for every exception making esf smaller and stack overflows less likely. Signed-off-by: Nicolas Pitre <npitre@baylibre.com>
2023-01-17 23:32:21 -05:00
/* decrement _current->arch.exception_depth */
lr t0, ___cpu_t_current_OFFSET(s0)
lb t1, _thread_offset_to_exception_depth(t0)
add t1, t1, -1
sb t1, _thread_offset_to_exception_depth(t0)
fp_trap_exit:
#endif
riscv: isr.S: optimize FP regs save/restore decision Rely on mstatus rather than thread->base.user_options since it is always up to date (updated by z_riscv_switch) to simplify the code and be SMP proof. Also carry over SF_INIT to the mstatus being restored in case it was changed in the mean time. Signed-off-by: Nicolas Pitre <npitre@baylibre.com>
2022-03-07 17:31:13 -05:00
riscv: integrate the new FPU context switching support FPU context switching is always performed on demand through the FPU access exception handler. Actual task switching only grants or denies FPU access depending on the current FPU owner. Because RISC-V doesn't have a dedicated FPU access exception, we must catch the Illegal Instruction exception and look for actual FP opcodes. There is no longer a need to allocate FPU storage on the stack for every exception making esf smaller and stack overflows less likely. Signed-off-by: Nicolas Pitre <npitre@baylibre.com>
2023-01-17 23:32:21 -05:00
/* Restore MEPC and MSTATUS registers */
lr t0, __z_arch_esf_t_mepc_OFFSET(sp)
lr t2, __z_arch_esf_t_mstatus_OFFSET(sp)
csrw mepc, t0
csrw mstatus, t2
riscv32: fixed context restore upon exiting ISR By now, t0 register restored value is overwritten by mepc and mstatus values prior to returning from ISR. Fixed by restoring mstatus and mepc registers before restoring the caller-saved registers. As t0 is a temporary register within the riscv ABI, this issue was unnoticed for most applications, except for computation intensive apps, like crypto tests. Signed-off-by: Jean-Paul Etienne <fractalclone@gmail.com>
2017-06-29 22:45:55 +02:00
riscv: exception code mega simplification and optimization Complete revamp of the exception entry code, including syscall handling. Proper syscall frame exception trigger. Many correctness fixes, hacks removal, etc. etc. I tried to make this into several commits, but this stuff is all inter-related and a pain to split. The diffstat summary: 14 files changed, 250 insertions(+), 802 deletions(-) Binary size (before): text data bss dec hex filename 1104 0 0 1104 450 isr.S.obj 64 0 0 64 40 userspace.S.obj Binary size (after): text data bss dec hex filename 600 0 0 600 258 isr.S.obj 36 0 0 36 24 userspace.S.obj Run of samples/userspace/syscall_perf (before): *** Booting Zephyr OS build zephyr-v3.0.0-325-g3748accae018 *** Main Thread started; qemu_riscv32 Supervisor thread started User thread started Supervisor thread(0x80010048): 384 cycles 509 instructions User thread(0x80010140): 77312 cycles 77437 instructions Run of samples/userspace/syscall_perf (after): *** Booting Zephyr OS build zephyr-v3.0.0-326-g4c877a2753b3 *** Main Thread started; qemu_riscv32 Supervisor thread started User thread started Supervisor thread(0x80010048): 384 cycles 509 instructions User thread(0x80010138): 7040 cycles 7165 instructions Yes, that's more than a 10x speed-up! Signed-off-by: Nicolas Pitre <npitre@baylibre.com>
2022-02-24 22:30:03 -05:00
#ifdef CONFIG_USERSPACE
riscv: Restore floating-point caller saved registers before integer This fixes an issue where the t0 register is overwritten after it has been restored. Signed-off-by: Corey Wharton <coreyw7@fb.com>
2020-04-14 14:56:12 -07:00
/*
riscv: exception code mega simplification and optimization Complete revamp of the exception entry code, including syscall handling. Proper syscall frame exception trigger. Many correctness fixes, hacks removal, etc. etc. I tried to make this into several commits, but this stuff is all inter-related and a pain to split. The diffstat summary: 14 files changed, 250 insertions(+), 802 deletions(-) Binary size (before): text data bss dec hex filename 1104 0 0 1104 450 isr.S.obj 64 0 0 64 40 userspace.S.obj Binary size (after): text data bss dec hex filename 600 0 0 600 258 isr.S.obj 36 0 0 36 24 userspace.S.obj Run of samples/userspace/syscall_perf (before): *** Booting Zephyr OS build zephyr-v3.0.0-325-g3748accae018 *** Main Thread started; qemu_riscv32 Supervisor thread started User thread started Supervisor thread(0x80010048): 384 cycles 509 instructions User thread(0x80010140): 77312 cycles 77437 instructions Run of samples/userspace/syscall_perf (after): *** Booting Zephyr OS build zephyr-v3.0.0-326-g4c877a2753b3 *** Main Thread started; qemu_riscv32 Supervisor thread started User thread started Supervisor thread(0x80010048): 384 cycles 509 instructions User thread(0x80010138): 7040 cycles 7165 instructions Yes, that's more than a 10x speed-up! Signed-off-by: Nicolas Pitre <npitre@baylibre.com>
2022-02-24 22:30:03 -05:00
* Check if we are returning to user mode. If so then we must
riscv: decouple the Zephyr CPU number from the hart ID Currently it is assumed that Zephyr CPU numbers match their hartid value one for one. This assumption was relied upon to efficiently retrieve the current CPU's `struct _cpu` pointer. People are starting to have systems with a mix of different usage for each CPU and such assumption may no longer be true. Let's completely decouple the hartid from the Zephyr CPU number by stuffing each CPU's `struct _cpu` pointer in their respective scratch register instead. `arch_curr_cpu()` becomes more efficient as well. Since the scratch register was previously used to store userspace's exception stack pointer, that is now moved into `struct _cpu_arch` which implied minor user space entry code cleanup and rationalization. Signed-off-by: Nicolas Pitre <npitre@baylibre.com>
2023-01-06 17:46:21 -05:00
* set is_user_mode to true and preserve our kernel mode stack for
* the next exception to come.
riscv: Restore floating-point caller saved registers before integer This fixes an issue where the t0 register is overwritten after it has been restored. Signed-off-by: Corey Wharton <coreyw7@fb.com>
2020-04-14 14:56:12 -07:00
*/
riscv: exception code mega simplification and optimization Complete revamp of the exception entry code, including syscall handling. Proper syscall frame exception trigger. Many correctness fixes, hacks removal, etc. etc. I tried to make this into several commits, but this stuff is all inter-related and a pain to split. The diffstat summary: 14 files changed, 250 insertions(+), 802 deletions(-) Binary size (before): text data bss dec hex filename 1104 0 0 1104 450 isr.S.obj 64 0 0 64 40 userspace.S.obj Binary size (after): text data bss dec hex filename 600 0 0 600 258 isr.S.obj 36 0 0 36 24 userspace.S.obj Run of samples/userspace/syscall_perf (before): *** Booting Zephyr OS build zephyr-v3.0.0-325-g3748accae018 *** Main Thread started; qemu_riscv32 Supervisor thread started User thread started Supervisor thread(0x80010048): 384 cycles 509 instructions User thread(0x80010140): 77312 cycles 77437 instructions Run of samples/userspace/syscall_perf (after): *** Booting Zephyr OS build zephyr-v3.0.0-326-g4c877a2753b3 *** Main Thread started; qemu_riscv32 Supervisor thread started User thread started Supervisor thread(0x80010048): 384 cycles 509 instructions User thread(0x80010138): 7040 cycles 7165 instructions Yes, that's more than a 10x speed-up! Signed-off-by: Nicolas Pitre <npitre@baylibre.com>
2022-02-24 22:30:03 -05:00
li t1, MSTATUS_MPP
riscv: Optimize t* registers usage In preparation for the support of RV32E optimize a bit the t* registers usage limiting that to t{0-2}. Signed-off-by: Carlo Caione <ccaione@baylibre.com>
2022-05-18 16:48:17 +02:00
and t0, t2, t1
riscv: exception code mega simplification and optimization Complete revamp of the exception entry code, including syscall handling. Proper syscall frame exception trigger. Many correctness fixes, hacks removal, etc. etc. I tried to make this into several commits, but this stuff is all inter-related and a pain to split. The diffstat summary: 14 files changed, 250 insertions(+), 802 deletions(-) Binary size (before): text data bss dec hex filename 1104 0 0 1104 450 isr.S.obj 64 0 0 64 40 userspace.S.obj Binary size (after): text data bss dec hex filename 600 0 0 600 258 isr.S.obj 36 0 0 36 24 userspace.S.obj Run of samples/userspace/syscall_perf (before): *** Booting Zephyr OS build zephyr-v3.0.0-325-g3748accae018 *** Main Thread started; qemu_riscv32 Supervisor thread started User thread started Supervisor thread(0x80010048): 384 cycles 509 instructions User thread(0x80010140): 77312 cycles 77437 instructions Run of samples/userspace/syscall_perf (after): *** Booting Zephyr OS build zephyr-v3.0.0-326-g4c877a2753b3 *** Main Thread started; qemu_riscv32 Supervisor thread started User thread started Supervisor thread(0x80010048): 384 cycles 509 instructions User thread(0x80010138): 7040 cycles 7165 instructions Yes, that's more than a 10x speed-up! Signed-off-by: Nicolas Pitre <npitre@baylibre.com>
2022-02-24 22:30:03 -05:00
bnez t0, 1f
riscv: Restore floating-point caller saved registers before integer This fixes an issue where the t0 register is overwritten after it has been restored. Signed-off-by: Corey Wharton <coreyw7@fb.com>
2020-04-14 14:56:12 -07:00
riscv: pmp: switch over to the new implementation Add the appropriate hooks effectively replacing the old implementation with the new one. Also the stackguard wasn't properly enforced especially with the usermode combination. This is now fixed. Signed-off-by: Nicolas Pitre <npitre@baylibre.com>
2022-04-06 22:03:54 -04:00
#ifdef CONFIG_PMP_STACK_GUARD
/* Remove kernel stack guard and Reconfigure PMP for user mode */
lr a0, ___cpu_t_current_OFFSET(s0)
call z_riscv_pmp_usermode_enable
#endif
riscv: new TLS-based arch_is_user_context() implementation This reverts the bulk of commit c8bfc2afda2c ("riscv: make arch_is_user_context() SMP compatible") and replaces it with a flag stored in the thread local storage (TLS) area, therefore making TLS mandatory for userspace support on RISC-V. This has many advantages: - The tp (x4) register is already dedicated by the standard for this purpose, making TLS support almost free. - This is very efficient, requiring only a single instruction to clear and 2 instructions to set. - This makes the SMP case much more efficient. No need for funky exception code any longer. - SMP and non-SMP now use the same implementation making maintenance easier. - The is_user_mode variable no longer requires a dedicated PMP mapping and therefore freeing one PMP slot for other purposes. Signed-off-by: Nicolas Pitre <npitre@baylibre.com> 5f65dbcc9dab3d39473b05397e05.
2022-06-07 09:37:59 -04:00
/* Set our per-thread usermode flag */
li t1, 1
lui t0, %tprel_hi(is_user_mode)
add t0, t0, tp, %tprel_add(is_user_mode)
sb t1, %tprel_lo(is_user_mode)(t0)
riscv: exception code mega simplification and optimization Complete revamp of the exception entry code, including syscall handling. Proper syscall frame exception trigger. Many correctness fixes, hacks removal, etc. etc. I tried to make this into several commits, but this stuff is all inter-related and a pain to split. The diffstat summary: 14 files changed, 250 insertions(+), 802 deletions(-) Binary size (before): text data bss dec hex filename 1104 0 0 1104 450 isr.S.obj 64 0 0 64 40 userspace.S.obj Binary size (after): text data bss dec hex filename 600 0 0 600 258 isr.S.obj 36 0 0 36 24 userspace.S.obj Run of samples/userspace/syscall_perf (before): *** Booting Zephyr OS build zephyr-v3.0.0-325-g3748accae018 *** Main Thread started; qemu_riscv32 Supervisor thread started User thread started Supervisor thread(0x80010048): 384 cycles 509 instructions User thread(0x80010140): 77312 cycles 77437 instructions Run of samples/userspace/syscall_perf (after): *** Booting Zephyr OS build zephyr-v3.0.0-326-g4c877a2753b3 *** Main Thread started; qemu_riscv32 Supervisor thread started User thread started Supervisor thread(0x80010048): 384 cycles 509 instructions User thread(0x80010138): 7040 cycles 7165 instructions Yes, that's more than a 10x speed-up! Signed-off-by: Nicolas Pitre <npitre@baylibre.com>
2022-02-24 22:30:03 -05:00
riscv: decouple the Zephyr CPU number from the hart ID Currently it is assumed that Zephyr CPU numbers match their hartid value one for one. This assumption was relied upon to efficiently retrieve the current CPU's `struct _cpu` pointer. People are starting to have systems with a mix of different usage for each CPU and such assumption may no longer be true. Let's completely decouple the hartid from the Zephyr CPU number by stuffing each CPU's `struct _cpu` pointer in their respective scratch register instead. `arch_curr_cpu()` becomes more efficient as well. Since the scratch register was previously used to store userspace's exception stack pointer, that is now moved into `struct _cpu_arch` which implied minor user space entry code cleanup and rationalization. Signed-off-by: Nicolas Pitre <npitre@baylibre.com>
2023-01-06 17:46:21 -05:00
/* preserve stack pointer for next exception entry */
riscv: exception code mega simplification and optimization Complete revamp of the exception entry code, including syscall handling. Proper syscall frame exception trigger. Many correctness fixes, hacks removal, etc. etc. I tried to make this into several commits, but this stuff is all inter-related and a pain to split. The diffstat summary: 14 files changed, 250 insertions(+), 802 deletions(-) Binary size (before): text data bss dec hex filename 1104 0 0 1104 450 isr.S.obj 64 0 0 64 40 userspace.S.obj Binary size (after): text data bss dec hex filename 600 0 0 600 258 isr.S.obj 36 0 0 36 24 userspace.S.obj Run of samples/userspace/syscall_perf (before): *** Booting Zephyr OS build zephyr-v3.0.0-325-g3748accae018 *** Main Thread started; qemu_riscv32 Supervisor thread started User thread started Supervisor thread(0x80010048): 384 cycles 509 instructions User thread(0x80010140): 77312 cycles 77437 instructions Run of samples/userspace/syscall_perf (after): *** Booting Zephyr OS build zephyr-v3.0.0-326-g4c877a2753b3 *** Main Thread started; qemu_riscv32 Supervisor thread started User thread started Supervisor thread(0x80010048): 384 cycles 509 instructions User thread(0x80010138): 7040 cycles 7165 instructions Yes, that's more than a 10x speed-up! Signed-off-by: Nicolas Pitre <npitre@baylibre.com>
2022-02-24 22:30:03 -05:00
add t0, sp, __z_arch_esf_t_SIZEOF
riscv: decouple the Zephyr CPU number from the hart ID Currently it is assumed that Zephyr CPU numbers match their hartid value one for one. This assumption was relied upon to efficiently retrieve the current CPU's `struct _cpu` pointer. People are starting to have systems with a mix of different usage for each CPU and such assumption may no longer be true. Let's completely decouple the hartid from the Zephyr CPU number by stuffing each CPU's `struct _cpu` pointer in their respective scratch register instead. `arch_curr_cpu()` becomes more efficient as well. Since the scratch register was previously used to store userspace's exception stack pointer, that is now moved into `struct _cpu_arch` which implied minor user space entry code cleanup and rationalization. Signed-off-by: Nicolas Pitre <npitre@baylibre.com>
2023-01-06 17:46:21 -05:00
sr t0, _curr_cpu_arch_user_exc_sp(s0)
j 2f
riscv: exception code mega simplification and optimization Complete revamp of the exception entry code, including syscall handling. Proper syscall frame exception trigger. Many correctness fixes, hacks removal, etc. etc. I tried to make this into several commits, but this stuff is all inter-related and a pain to split. The diffstat summary: 14 files changed, 250 insertions(+), 802 deletions(-) Binary size (before): text data bss dec hex filename 1104 0 0 1104 450 isr.S.obj 64 0 0 64 40 userspace.S.obj Binary size (after): text data bss dec hex filename 600 0 0 600 258 isr.S.obj 36 0 0 36 24 userspace.S.obj Run of samples/userspace/syscall_perf (before): *** Booting Zephyr OS build zephyr-v3.0.0-325-g3748accae018 *** Main Thread started; qemu_riscv32 Supervisor thread started User thread started Supervisor thread(0x80010048): 384 cycles 509 instructions User thread(0x80010140): 77312 cycles 77437 instructions Run of samples/userspace/syscall_perf (after): *** Booting Zephyr OS build zephyr-v3.0.0-326-g4c877a2753b3 *** Main Thread started; qemu_riscv32 Supervisor thread started User thread started Supervisor thread(0x80010048): 384 cycles 509 instructions User thread(0x80010138): 7040 cycles 7165 instructions Yes, that's more than a 10x speed-up! Signed-off-by: Nicolas Pitre <npitre@baylibre.com>
2022-02-24 22:30:03 -05:00
1:
riscv: decouple the Zephyr CPU number from the hart ID Currently it is assumed that Zephyr CPU numbers match their hartid value one for one. This assumption was relied upon to efficiently retrieve the current CPU's `struct _cpu` pointer. People are starting to have systems with a mix of different usage for each CPU and such assumption may no longer be true. Let's completely decouple the hartid from the Zephyr CPU number by stuffing each CPU's `struct _cpu` pointer in their respective scratch register instead. `arch_curr_cpu()` becomes more efficient as well. Since the scratch register was previously used to store userspace's exception stack pointer, that is now moved into `struct _cpu_arch` which implied minor user space entry code cleanup and rationalization. Signed-off-by: Nicolas Pitre <npitre@baylibre.com>
2023-01-06 17:46:21 -05:00
/*
* We are returning to kernel mode. Store the stack pointer to
* be re-loaded further down.
*/
addi t0, sp, __z_arch_esf_t_SIZEOF
sr t0, __z_arch_esf_t_sp_OFFSET(sp)
2:
riscv: exception code mega simplification and optimization Complete revamp of the exception entry code, including syscall handling. Proper syscall frame exception trigger. Many correctness fixes, hacks removal, etc. etc. I tried to make this into several commits, but this stuff is all inter-related and a pain to split. The diffstat summary: 14 files changed, 250 insertions(+), 802 deletions(-) Binary size (before): text data bss dec hex filename 1104 0 0 1104 450 isr.S.obj 64 0 0 64 40 userspace.S.obj Binary size (after): text data bss dec hex filename 600 0 0 600 258 isr.S.obj 36 0 0 36 24 userspace.S.obj Run of samples/userspace/syscall_perf (before): *** Booting Zephyr OS build zephyr-v3.0.0-325-g3748accae018 *** Main Thread started; qemu_riscv32 Supervisor thread started User thread started Supervisor thread(0x80010048): 384 cycles 509 instructions User thread(0x80010140): 77312 cycles 77437 instructions Run of samples/userspace/syscall_perf (after): *** Booting Zephyr OS build zephyr-v3.0.0-326-g4c877a2753b3 *** Main Thread started; qemu_riscv32 Supervisor thread started User thread started Supervisor thread(0x80010048): 384 cycles 509 instructions User thread(0x80010138): 7040 cycles 7165 instructions Yes, that's more than a 10x speed-up! Signed-off-by: Nicolas Pitre <npitre@baylibre.com>
2022-02-24 22:30:03 -05:00
#endif
riscv: Restore floating-point caller saved registers before integer This fixes an issue where the t0 register is overwritten after it has been restored. Signed-off-by: Corey Wharton <coreyw7@fb.com>
2020-04-14 14:56:12 -07:00
riscv: isr.S: dedicate a register to &current_cpu Stop using &_kernel as this is not SMP friendly. Let's use s0 (after preserving its content) to hold &current_cpu instead so it won't have to be reloaded each time it is needed. This will be even more relevant when SMP support is added. Signed-off-by: Nicolas Pitre <npitre@baylibre.com>
2022-03-09 17:21:27 -05:00
/* Restore s0 (it is no longer ours) */
lr s0, __z_arch_esf_t_s0_OFFSET(sp)
riscv: Add support for hardware stacking / unstacking Some RISC-V SoCs implement a mechanism for hardware supported stacking / unstacking of registers during ISR / exceptions. What happens is that on ISR / exception entry part of the context is automatically saved by the hardware on the stack without software intervention, and the same part of the context is restored by the hardware usually on mret. This is currently not yet supported by Zephyr, where the full context must be saved by software in the full fledged ESF. This patcheset is trying to address exactly this case. At least three things are needed to support in a general fashion this problem: (1) a way to store in software only the part of the ESF not already stacked by hardware, (2) a way to restore in software only the part of the context that is not going to be restored by hardware and (3) a way to define a custom ESF. Point (3) is important because the full ESF frame is now composed by a custom part depending on the hardware (that can choose which register to stack / unstack and the order they are saved onto the stack) and a part defined in software for the remaining part of the context. In this patch a new CONFIG_RISCV_SOC_HAS_ISR_STACKING is introduced that enables the code path supporting the three points by the mean of three macros that must be implemented by the user in a soc_stacking.h file: SOC_ISR_SW_STACKING, SOC_ISR_SW_UNSTACKING and SOC_ISR_STACKING_ESF (refer to the symbol help for more details). This is an example of soc_isr_stacking.h for an hardware that doesn't do any hardware stacking / unstacking but everything is managed in software: #ifndef __SOC_ISR_STACKING #define __SOC_ISR_STACKING #if !defined(_ASMLANGUAGE) #define SOC_ISR_STACKING_ESF_DECLARE \ struct __esf { \ unsigned long ra; \ unsigned long t0; \ unsigned long t1; \ unsigned long t2; \ unsigned long t3; \ unsigned long t4; \ unsigned long t5; \ unsigned long t6; \ unsigned long a0; \ unsigned long a1; \ unsigned long a2; \ unsigned long a3; \ unsigned long a4; \ unsigned long a5; \ unsigned long a6; \ unsigned long a7; \ unsigned long mepc; \ unsigned long mstatus; \ unsigned long s0; \ } __aligned(16) #else #define SOC_ISR_SW_STACKING \ addi sp, sp, -__z_arch_esf_t_SIZEOF; \ DO_CALLER_SAVED(sr); #define SOC_ISR_SW_UNSTACKING \ DO_CALLER_SAVED(lr); #endif /* _ASMLANGUAGE */ #endif /* __SOC_ISR_STACKING */ Signed-off-by: Carlo Caione <ccaione@baylibre.com>
2022-12-27 20:14:51 +01:00
#ifdef CONFIG_RISCV_SOC_HAS_ISR_STACKING
SOC_ISR_SW_UNSTACKING
#else
arch: added support for the riscv32 architecture RISC-V is an open-source instruction set architecture. Added support for the 32bit version of RISC-V to Zephyr. 1) exceptions/interrupts/faults are handled at the architecture level via the __irq_wrapper handler. Context saving/restoring of registers can be handled at both architecture and SOC levels. If SOC-specific registers need to be saved, SOC level needs to provide __soc_save_context and __soc_restore_context functions that shall be accounted by the architecture level, when corresponding config variable RISCV_SOC_CONTEXT_SAVE is set. 2) As RISC-V architecture does not provide a clear ISA specification about interrupt handling, each RISC-V SOC handles it in its own way. Hence, at the architecture level, the __irq_wrapper handler expects the following functions to be provided by the SOC level: __soc_is_irq: to check if the exception is the result of an interrupt or not. __soc_handle_irq: handle pending IRQ at SOC level (ex: clear pending IRQ in SOC-specific IRQ register) 3) Thread/task scheduling, as well as IRQ offloading are handled via the RISC-V system call ("ecall"), which is also handled via the __irq_wrapper handler. The _Swap asm function just calls "ecall" to generate an exception. 4) As there is no conventional way of handling CPU power save in RISC-V, the default nano_cpu_idle and nano_cpu_atomic_idle functions just unlock interrupts and return to the caller, without issuing any CPU power saving instruction. Nonetheless, to allow SOC-level to implement proper CPU power save, nano_cpu_idle and nano_cpu_atomic_idle functions are defined as __weak at the architecture level. Change-Id: I980a161d0009f3f404ad22b226a6229fbb492389 Signed-off-by: Jean-Paul Etienne <fractalclone@gmail.com>
2017-01-11 00:24:30 +01:00
/* Restore caller-saved registers from thread stack */
riscv: new TLS-based arch_is_user_context() implementation This reverts the bulk of commit c8bfc2afda2c ("riscv: make arch_is_user_context() SMP compatible") and replaces it with a flag stored in the thread local storage (TLS) area, therefore making TLS mandatory for userspace support on RISC-V. This has many advantages: - The tp (x4) register is already dedicated by the standard for this purpose, making TLS support almost free. - This is very efficient, requiring only a single instruction to clear and 2 instructions to set. - This makes the SMP case much more efficient. No need for funky exception code any longer. - SMP and non-SMP now use the same implementation making maintenance easier. - The is_user_mode variable no longer requires a dedicated PMP mapping and therefore freeing one PMP slot for other purposes. Signed-off-by: Nicolas Pitre <npitre@baylibre.com> 5f65dbcc9dab3d39473b05397e05.
2022-06-07 09:37:59 -04:00
DO_CALLER_SAVED(lr)
riscv: Add support for hardware stacking / unstacking Some RISC-V SoCs implement a mechanism for hardware supported stacking / unstacking of registers during ISR / exceptions. What happens is that on ISR / exception entry part of the context is automatically saved by the hardware on the stack without software intervention, and the same part of the context is restored by the hardware usually on mret. This is currently not yet supported by Zephyr, where the full context must be saved by software in the full fledged ESF. This patcheset is trying to address exactly this case. At least three things are needed to support in a general fashion this problem: (1) a way to store in software only the part of the ESF not already stacked by hardware, (2) a way to restore in software only the part of the context that is not going to be restored by hardware and (3) a way to define a custom ESF. Point (3) is important because the full ESF frame is now composed by a custom part depending on the hardware (that can choose which register to stack / unstack and the order they are saved onto the stack) and a part defined in software for the remaining part of the context. In this patch a new CONFIG_RISCV_SOC_HAS_ISR_STACKING is introduced that enables the code path supporting the three points by the mean of three macros that must be implemented by the user in a soc_stacking.h file: SOC_ISR_SW_STACKING, SOC_ISR_SW_UNSTACKING and SOC_ISR_STACKING_ESF (refer to the symbol help for more details). This is an example of soc_isr_stacking.h for an hardware that doesn't do any hardware stacking / unstacking but everything is managed in software: #ifndef __SOC_ISR_STACKING #define __SOC_ISR_STACKING #if !defined(_ASMLANGUAGE) #define SOC_ISR_STACKING_ESF_DECLARE \ struct __esf { \ unsigned long ra; \ unsigned long t0; \ unsigned long t1; \ unsigned long t2; \ unsigned long t3; \ unsigned long t4; \ unsigned long t5; \ unsigned long t6; \ unsigned long a0; \ unsigned long a1; \ unsigned long a2; \ unsigned long a3; \ unsigned long a4; \ unsigned long a5; \ unsigned long a6; \ unsigned long a7; \ unsigned long mepc; \ unsigned long mstatus; \ unsigned long s0; \ } __aligned(16) #else #define SOC_ISR_SW_STACKING \ addi sp, sp, -__z_arch_esf_t_SIZEOF; \ DO_CALLER_SAVED(sr); #define SOC_ISR_SW_UNSTACKING \ DO_CALLER_SAVED(lr); #endif /* _ASMLANGUAGE */ #endif /* __SOC_ISR_STACKING */ Signed-off-by: Carlo Caione <ccaione@baylibre.com>
2022-12-27 20:14:51 +01:00
#endif /* CONFIG_RISCV_SOC_HAS_ISR_STACKING */
riscv: exception code mega simplification and optimization Complete revamp of the exception entry code, including syscall handling. Proper syscall frame exception trigger. Many correctness fixes, hacks removal, etc. etc. I tried to make this into several commits, but this stuff is all inter-related and a pain to split. The diffstat summary: 14 files changed, 250 insertions(+), 802 deletions(-) Binary size (before): text data bss dec hex filename 1104 0 0 1104 450 isr.S.obj 64 0 0 64 40 userspace.S.obj Binary size (after): text data bss dec hex filename 600 0 0 600 258 isr.S.obj 36 0 0 36 24 userspace.S.obj Run of samples/userspace/syscall_perf (before): *** Booting Zephyr OS build zephyr-v3.0.0-325-g3748accae018 *** Main Thread started; qemu_riscv32 Supervisor thread started User thread started Supervisor thread(0x80010048): 384 cycles 509 instructions User thread(0x80010140): 77312 cycles 77437 instructions Run of samples/userspace/syscall_perf (after): *** Booting Zephyr OS build zephyr-v3.0.0-326-g4c877a2753b3 *** Main Thread started; qemu_riscv32 Supervisor thread started User thread started Supervisor thread(0x80010048): 384 cycles 509 instructions User thread(0x80010138): 7040 cycles 7165 instructions Yes, that's more than a 10x speed-up! Signed-off-by: Nicolas Pitre <npitre@baylibre.com>
2022-02-24 22:30:03 -05:00
#ifdef CONFIG_USERSPACE
/* retrieve saved stack pointer */
lr sp, __z_arch_esf_t_sp_OFFSET(sp)
#else
/* remove esf from the stack */
addi sp, sp, __z_arch_esf_t_SIZEOF
#endif
arch: added support for the riscv32 architecture RISC-V is an open-source instruction set architecture. Added support for the 32bit version of RISC-V to Zephyr. 1) exceptions/interrupts/faults are handled at the architecture level via the __irq_wrapper handler. Context saving/restoring of registers can be handled at both architecture and SOC levels. If SOC-specific registers need to be saved, SOC level needs to provide __soc_save_context and __soc_restore_context functions that shall be accounted by the architecture level, when corresponding config variable RISCV_SOC_CONTEXT_SAVE is set. 2) As RISC-V architecture does not provide a clear ISA specification about interrupt handling, each RISC-V SOC handles it in its own way. Hence, at the architecture level, the __irq_wrapper handler expects the following functions to be provided by the SOC level: __soc_is_irq: to check if the exception is the result of an interrupt or not. __soc_handle_irq: handle pending IRQ at SOC level (ex: clear pending IRQ in SOC-specific IRQ register) 3) Thread/task scheduling, as well as IRQ offloading are handled via the RISC-V system call ("ecall"), which is also handled via the __irq_wrapper handler. The _Swap asm function just calls "ecall" to generate an exception. 4) As there is no conventional way of handling CPU power save in RISC-V, the default nano_cpu_idle and nano_cpu_atomic_idle functions just unlock interrupts and return to the caller, without issuing any CPU power saving instruction. Nonetheless, to allow SOC-level to implement proper CPU power save, nano_cpu_idle and nano_cpu_atomic_idle functions are defined as __weak at the architecture level. Change-Id: I980a161d0009f3f404ad22b226a6229fbb492389 Signed-off-by: Jean-Paul Etienne <fractalclone@gmail.com>
2017-01-11 00:24:30 +01:00
soc: riscv: remove usage of SOC_ERET All SOC_ERET definitions expand to the mret instruction (used to return from a trap: exception or interruption). The 'eret' instruction existed in previous RISC-V privileged specs, but it doesn't seem to be used in Zephyr (ref. RISC-V Privileged Architectures 3.2.2). Signed-off-by: Gerard Marull-Paretas <gerard.marull@nordicsemi.no>
2022-08-03 12:51:34 +02:00
mret
Copy permalink