CLIC supports mintstatus.MIL (RO) and mcause.MPIL (RW) for the current
interrupt level and the previous interrut level before a trap. Each ISR
must execute MRET to set mcause.MPIL back to mintstatus.MIL.
This commit introduces CONFIG_CLIC_SUPPORT_INTERRUPT_LEVEL to handle
mcause.MPIL for interrupt preemption in nested ISR, and uses
CONFIG_RISCV_ALWAYS_SWITCH_THROUGH_ECALL to ensure ISR always switch out
with MRET.
e.g.
With CONFIG_RISCV_ALWAYS_SWITCH_THROUGH_ECALL=n, a context-switch in
ISR may skip MRET in this flow:
IRQ -> _isr_wrapper -> z_riscv_switch() -> retrun to arch_switch()
Signed-off-by: Jimmy Zheng <jimmyzhe@andestech.com>
The exception fault handler function is named _Fault which is inconsistent
with all other arch-specific functions present in Zephyr. This patch
replaces the name with z_riscv_fault.
Signed-off-by: Joel Holdsworth <jholdsworth@nvidia.com>
When RISCV_SOC_HAS_ISR_STACKING is used, it may
be needed to initialize custom hw stacked esf members.
Some initial values may need to be aligned with
hw stacking mechanism to avoid any side effects.
Signed-off-by: Łukasz Stępnicki <lukasz.stepnicki@nordicsemi.no>
The RISC-V architecture-specific relocations need to check whether
each required relocation can fit into the modified instruction's
immediate. All immediates in RISC-V are encoded as two's complement.
The current truncation check has an off-by-one error for checking
the maximum negative distance, as two's complement encoding can
represent a negative value that is the maximum positive value plus
one, causing LLEXT to refuse loading valid code.
This commit adds an additional condition to the check that fixes
the aforementioned issue.
Signed-off-by: Eric Ackermann <eric.ackermann@cispa.de>
Mostly a revert of commit b1def7145f ("arch: deprecate `_current`").
This commit was part of PR #80716 whose initial purpose was about providing
an architecture specific optimization for _current. The actual deprecation
was sneaked in later on without proper discussion.
The Zephyr core always used _current before and that was fine. It is quite
prevalent as well and the alternative is proving rather verbose.
Furthermore, as a concept, the "current thread" is not something that is
necessarily architecture specific. Therefore the primary abstraction
should not carry the arch_ prefix.
Hence this revert.
Signed-off-by: Nicolas Pitre <npitre@baylibre.com>
Introduce `CONFIG_RISCV_GP_PURPOSE` choice to make sure that only
one of `CONFIG_RISCV_GP` or `CONFIG_RISCV_CURRENT_VIA_GP` can be
enabled, instead of relying of dependencies.
To do that, introduce a new
`CONFIG_RISCV_SOC_HAS_GP_RELATIVE_ADDRESSING` that can be selected
by SoC when it implemented global pointer (GP) initialization for
relative addressing in its linker.
`CONFIG_RISCV_GP` will be the default choice when
`CONFIG_RISCV_SOC_HAS_GP_RELATIVE_ADDRESSING=y`
Signed-off-by: Yong Cong Sin <ycsin@meta.com>
Signed-off-by: Yong Cong Sin <yongcong.sin@gmail.com>
Reset the the `gp` register to `_kernel->cpus[i].current` when
`CONFIG_USERSPACE` is enabled on exception to keep it sane.
Updated the testcase to test both `CONFIG_RISCV_GP` and
`CONFIG_RISCV_CURRENT_VIA_GP`.
Signed-off-by: Yong Cong Sin <ycsin@meta.com>
Signed-off-by: Yong Cong Sin <yongcong.sin@gmail.com>
In most implements, the msip base address is 0x2000000. But the address
is not fixed in all boards.
Signed-off-by: Yang Jialong <yangjialong@vcore.com>
`_current` is now functionally equals to `arch_curr_thread()`, remove
its usage in-tree and deprecate it instead of removing it outright,
as it has been with us since forever.
Signed-off-by: Yong Cong Sin <ycsin@meta.com>
Signed-off-by: Yong Cong Sin <yongcong.sin@gmail.com>
Implement `arch_curr_thread()` & `arch_set_curr_thread()`
with the global pointer (GP) register.
Signed-off-by: Yong Cong Sin <ycsin@meta.com>
Signed-off-by: Yong Cong Sin <yongcong.sin@gmail.com>
Previously PMP was avaible only with Multithreading, since it's now
available without MT - add extra checks to prevent user/kernel mode
reconfiguration.
Signed-off-by: Volodymyr Fialko <vfialko@marvell.com>
Signed-off-by: Anas Nashif <anas.nashif@intel.com>
Without multithreading only two stacks present: ISR and main.
As any stack they also could overflow, so it make sense to add stack
guard for them also.
Remove stack guard dependency on multithreading and mark
`Z_RISCV_STACK_GUARD_SIZE` bytes at the beginning of stack as read-only
region with PMP entry.
Signed-off-by: Volodymyr Fialko <vfialko@marvell.com>
According to the RISC-V Instruction Set Manual: Volume II, Version 20240411
(Section 3.1.6.6), some implementations may choose to track the dirtiness
of the floating-point register state imprecisely by reporting the state to
be dirty even when it has not been modified. This option reflects that.
Also add a filter in `tests/arch/riscv/fpu_sharing/` based on imprecise
FPU state tracking
Signed-off-by: Jakub Wasilewski <jwasilewski@internships.antmicro.com>
Signed-off-by: Filip Kokosinski <fkokosinski@antmicro.com>
Allow SoC to implement their custom per-core initialization function by
selecting `CONFIG_SOC_PER_CORE_INIT_HOOK` and implement
`soc_per_core_init_hook()`.
Signed-off-by: Maxim Adelman <imax@meta.com>
Signed-off-by: Yong Cong Sin <ycsin@meta.com>
Signed-off-by: Yong Cong Sin <yongcong.sin@gmail.com>
The currently IPI implementation assumes that CLINT exists in the
system, however, that might not be the case as IPI can be implemented
with PLIC that supports software-triggering as well, such as the Andes
NCEPLIC100.
Refactor the CLINT-based IPI implementations into `ipi_clint.c`, and
create Kconfig that selects the CLINT implementation when
`sifive-clint0` exists and enabled, otherwise default to
`RISCV_SMP_IPI_CUSTOM` which allows OOT implementation. This also
makes way for the upstreaming of non-clint IPI implementation later.
Signed-off-by: Yong Cong Sin <ycsin@meta.com>
Reset the gp on exception entry from u-mode to protect the kernel
against a possible rogue user thread.
Signed-off-by: Yong Cong Sin <yongcong.sin@gmail.com>
Signed-off-by: Yong Cong Sin <ycsin@meta.com>
The `CONFIG_PLIC_IRQ_AFFINITY` depends on `CONFIG_SMP` in the
Kconfig layer, nested #ifdef can be avoided for readability.
Signed-off-by: Yong Cong Sin <ycsin@meta.com>
Signed-off-by: Yong Cong Sin <yongcong.sin@gmail.com>
This commit introduces architecture-specific ELF relocations for RISC-V,
in accordance with the RISC-V PSABI specification:
https://github.com/riscv-non-isa/riscv-elf-psabi-doc/blob/master/riscv-elf.adoc
Also, the necessary compiler configurations for compiling LLEXT
extensions on RISC-V are added, and the llext tests are executed on
RISC-V targets.
Calling llext extensions from user threads in RISC-V is still
unsupported as of this commit.
Signed-off-by: Eric Ackermann <eric.ackermann@cispa.de>
- Implement irq-set-affinity in RISCV PLIC.
- Added new affinity shell command to get/set the irq(s)
affinity in runtime, when `0` is sent as the `local_irq`, it
means set/get all IRQs affinity.
- Some minor optimizations
Updated the build_all test to build this new configuration.
Signed-off-by: Yong Cong Sin <ycsin@meta.com>
Signed-off-by: Yong Cong Sin <yongcong.sin@gmail.com>
Restore the s0 we saved early in ISR entry so it shows up
properly in the CSF.
Signed-off-by: David Reiss <dreiss@meta.com>
Signed-off-by: Yong Cong Sin <ycsin@meta.com>
Up until now, the `__thread` keyword has been used for declaring
variables as Thread local storage. However, `__thread` is a GNU
specific keyword which thus limits compatibility with other
toolchains (for instance IAR).
This PR intoduces a new macro `Z_THREAD_LOCAL` which expands to the
corresponding C11, C23 or C++11 standard keyword based on the standard
that is specified during compilation, else it uses the old `__thread`
keyword.
Signed-off-by: Daniel Flodin <daniel.flodin@iar.com>
Do not use SYS_INIT for initializing irq_offload when enabled, instead
using a new interface that is called during the boot process for all
architectures.
Signed-off-by: Anas Nashif <anas.nashif@intel.com>
Add a new call for initializing cache on architectures that need that.
Avoid using SYS_INIT for this and instead call the hook in a fixed place
and run if implemented.
Signed-off-by: Anas Nashif <anas.nashif@intel.com>
Fill the memory of all CPU's IRQ stack with 0xAA on init, so
that `z_stack_space_get` can calculate the remaining space
correctly.
Signed-off-by: Nicolas Pitre <npitre@baylibre.com>
Signed-off-by: Yong Cong Sin <ycsin@meta.com>
Signed-off-by: Yong Cong Sin <yongcong.sin@gmail.com>
The callee-saved-registers can be helpful to debug the state of
a core upon an exception, however, currently there's no way to
access that information in user-implemented
`k_sys_fatal_error_handler()`, even though the csf is already stored
in the stack.
This patch conditionally add a `csf` member in the `arch_esf` when
`CONFIG_EXTRA_EXCEPTION_INFO=y`*, which the `_isr_wrapper` would update
when a fatal error occurs before invoking `z_riscv_fatal_error_csf()`.
Functions such as `k_sys_fatal_error_handler()` would then be able
to access the callee-saved-registers at the time of exception via
`esf->csf`.
* For SoCs that select `RISCV_SOC_HAS_ISR_STACKING`, the
`SOC_ISR_STACKING_ESF_DECLARE` has to include the `csf` member,
otherwise the build would fail.
Signed-off-by: Yong Cong Sin <ycsin@meta.com>
Signed-off-by: Yong Cong Sin <yongcong.sin@gmail.com>
Get rid of the switch statement and use an string array
for the cause instead. This saves about ~600 bytes.
Signed-off-by: Yong Cong Sin <ycsin@meta.com>
Signed-off-by: Yong Cong Sin <yongcong.sin@gmail.com>
Rename `cause_str` to `z_riscv_mcause_str` and make it non-static,
so that it can be used in user-implemented `k_sys_fatal_error_handler`.
Also, this function should return a constant string, so add `const`
to it.
Signed-off-by: Yong Cong Sin <ycsin@meta.com>
Signed-off-by: Yong Cong Sin <yongcong.sin@gmail.com>
Relocate the logging of mcause & mtval from `_Fault` to
`z_riscv_fatal_error_csf` so that they are always printed
upon exception.
Signed-off-by: Yong Cong Sin <ycsin@meta.com>
Signed-off-by: Yong Cong Sin <yongcong.sin@gmail.com>
Introduce soc and board hooks to replace arch specific code
and replace usages of SYS_INIT for platform initialization.
include/zephyr/platform/hooks.h introduces the hooks to be implemented
by boards and SoCs.
Signed-off-by: Anas Nashif <anas.nashif@intel.com>
Add the possibility to disable fault handling in spurious
interrupt handler on RISCs and replacce it with an infinite loop.
Signed-off-by: Magdalena Pastula <magdalena.pastula@nordicsemi.no>
The change of alignment check in #76045 could be wrong and
isn't unnecessary to fix the stack traces output, undo it for
now.
Signed-off-by: Yong Cong Sin <ycsin@meta.com>
Signed-off-by: Yong Cong Sin <yongcong.sin@gmail.com>
This implements arch_thread_priv_stack_space_get() so this can
be used to figure out how much privileged stack space is used.
Signed-off-by: Daniel Leung <daniel.leung@intel.com>
This adds the bits to initialize the privileged stack when
a thread is transitioning to user mode. This prevents
information leaking if the stack is reused, and also aids
in calculating stack space usage during system calls.
Signed-off-by: Daniel Leung <daniel.leung@intel.com>
Having `CONFIG_EXCEPTION_STACK_TRACE_SYMTAB` to select the
`CONFIG_SYMTAB` or to explicitly not print the symbol name
during exception stack unwind seems unnecessary, as the extra
code to print the symbol name is negligible when compared with
the symbol table, so just remove it.
Signed-off-by: Yong Cong Sin <ycsin@meta.com>
Signed-off-by: Yong Cong Sin <yongcong.sin@gmail.com>
This commit introduces a new ARCH_STACKWALK Kconfig which
determines if the `arch_stack_walk()` is available should the
arch supports it.
Starting from RISCV, this will be able to converge the exception
stack trace implementation & stack walking features. Existing
exception stack trace implementation will be updated later.
Eventually we will end up with the following:
1. If an arch implements `arch_stack_walk()`
`ARCH_HAS_STACKWALK` should be selected.
2. If the above is enabled, `ARCH_SUPPORTS_STACKWALK` indicates
if the dependencies are met for arch to enable stack walking.
This Kconfig replaces `<arch>_EXCEPTION_STACK_TRACE`
2. If the above is enabled, then, `ARCH_STACKWALK` determines
if `arch_stack_walk()` should be compiled.
3. `EXCEPTION_STACK_TRACE` should build on top of the
`ARCH_STACKWALK`, stack traces will be printed when it
is enabled.
4. `ARCH_STACKWALK_MAX_FRAMES` will be removed as it is
replaced by `ARCH_STACKWALK_MAX_FRAMES`
Signed-off-by: Yong Cong Sin <ycsin@meta.com>
Signed-off-by: Yong Cong Sin <yongcong.sin@gmail.com>
Relocate stack unwind backends from `arch/` to perf's
`backends/` folder, just like logging/shell/..
Signed-off-by: Yong Cong Sin <ycsin@meta.com>
Signed-off-by: Yong Cong Sin <yongcong.sin@gmail.com>
Implement stack trace function for riscv arch, that get required
thread register values and unwind stack with it.
Signed-off-by: Mikhail Kushnerov <m.kushnerov@yadro.com>
custom arch_cpu_idle and arch_cpu_atomic_idle implementation was done
differently on different architectures. riscv implemented those as weak
symbols, xtensa used a kconfig and all other architectures did not
really care, but this was a global kconfig that should apply to all
architectures.
Signed-off-by: Anas Nashif <anas.nashif@intel.com>
Account for the scenario when we are doing `esf`-based
unwinding from a function which doesn't have any callee.
In this case the `ra` is not saved on the stack and the
second function from the top of the frame could be missing.
Signed-off-by: Yong Cong Sin <ycsin@meta.com>
Signed-off-by: Yong Cong Sin <yongcong.sin@gmail.com>
According to the riscv's `arch.h`:
+------------+ <- thread.arch.priv_stack_start
| Guard | } Z_RISCV_STACK_GUARD_SIZE
+------------+
| Priv Stack | } CONFIG_PRIVILEGED_STACK_SIZE
+------------+ <- thread.arch.priv_stack_start +
CONFIG_PRIVILEGED_STACK_SIZE +
Z_RISCV_STACK_GUARD_SIZE
The start of the privilege stack should be:
`thread.arch.priv_stack_start + Z_RISCV_STACK_GUARD_SIZE`
Instead of
`thread.arch.priv_stack_start - CONFIG_PRIVILEGED_STACK_SIZE`
For the `end`, use the same equation of `top_of_priv_stack` in
the `arch_user_mode_enter()`
Signed-off-by: Yong Cong Sin <ycsin@meta.com>
When RISCV_ALWAYS_SWITCH_THROUGH_ECALL is enabled, do_swap() enables PMP
checking in is_kernel_syscall.
If the PMP stack guard is triggered and do_swap() is called from the
fault handler, a PMP error occurs because the stack usage violates the
previous PMP setting.
Remove the stack guard setting during a stack overflow handler to allow
enabling PMP checking safely in fault handler.
Signed-off-by: Jimmy Zheng <jimmyzhe@andestech.com>
When RISCV_ALWAYS_SWITCH_THROUGH_ECALL is enabled, do_swap() enables PMP
checking in is_kernel_syscall.
If a user thread violates memory protection and do_swap() is called from
the fault handler, a PMP error occurs because the thread is in privileged
mode but still using the old user mode PMP setting.
Update the PMP setting to privileged mode for fault handler.
This also enables the stack guard for user thread's privileged stack in
fault handler.
Signed-off-by: Jimmy Zheng <jimmyzhe@andestech.com>
When 'arch_switch()' switches though Ecall, 'exception_depth' is
incorrectly added to the next thread because the current thread is updated
before arch_switch().
Add 'exception_depth' back to the previous thread when Ecall is called from
'arch_switch()'.
Signed-off-by: Jimmy Zheng <jimmyzhe@andestech.com>
Before this, stack protection would be effective only after switching to
the first thread.
Even before the first thread is created, the kernel init code uses the
IRQ stack to set things up. Let's make sure this is safeguarded as well.
This also fixes the incompatibility between CONFIG_RISCV_PMP and
CONFIG_RISCV_ALWAYS_SWITCH_THROUGH_ECALL, the later needing an exception
call to switch to the first thread and exception code assuming stack
guard is already set up in the PMP.
Signed-off-by: Nicolas Pitre <npitre@baylibre.com>
Update the description of the `INCLUDE_RESET_VECTOR` Kconfig so
that it is more clear to the user what it does.
Signed-off-by: Yong Cong Sin <ycsin@meta.com>
Zephyr's thread stack size is not fixed, in most cases we would
need the `thread` argument to obtain the `stack_info`, unless
we are unwinding the irq stack, since that is fixed.
Otherwise we can only safely print the current `mepc` register,
unwinding the esf without the stack info of a thread can
result in undefined behavior.
Signed-off-by: Yong Cong Sin <ycsin@meta.com>
