Introduce `CONFIG_PMP_KERNEL_MODE_DYNAMIC` to enable dynamic
configuration and activation of Machine mode PMP entries. This allows
PMP settings to be managed efficiently during transitions between
kernel and thread contexts.
Signed-off-by: Firas Sammoura <fsammoura@google.com>
Rename the `z_riscv_pmp_stackguard_*` functions to
`z_riscv_pmp_kernelmode_*`. This change better reflects that
these functions are used for general kernel mode PMP configuration,
not strictly limited to stack guard purposes.
Call sites in fatal.c, isr.S, and switch.S have been updated accordingly.
Signed-off-by: Firas Sammoura <fsammoura@google.com>
Add option to enable NMI delivery on boot for SMRNMI hardware.
Changes:
- Add CONFIG_RISCV_SMRNMI_ENABLE_NMI_DELIVERY Kconfig option
- Define SMRNMI CSRs in arch/riscv/include/csr.h
- Set NMIE bit during boot to enable NMI delivery
SMRNMI hardware generates but doesn't deliver NMIs when NMIE=0 (default).
This causes twister test failures and prevents handling of critical
hardware events like watchdog NMIs and ECC errors.
Setting NMIE=1 enables NMI delivery, but note that this implementation
only sets the enable bit - it does not provide full SMRNMI support
(no mnret instruction handling, no RNMI handlers). Users must implement
proper RNMI handlers in SoC-specific code to avoid undefined behavior.
Signed-off-by: Afonso Oliveira <afonsoo@synopsys.com>
Adds function for app to obtain page table usage statistics,
allowing fine tuning of numbers of L1 and L2 page table
array.
Signed-off-by: Daniel Leung <daniel.leung@intel.com>
When adding new memory domains, we need to start from kernel
page tables by duplicating them. However, there is no need to
duplicate all in-use page tables as some memory regions remain
unchanged. Fror example, hardware register region where only
kernel has access does not need to be duplicated across all
memory domains. The same L2 table can be used among them.
This changes the L2 page table allocation to be copy-on-write
where we only need to duplicate a L2 table if changes need to
be made on it.
Signed-off-by: Daniel Leung <daniel.leung@intel.com>
If the page tables are not cached, there is no need to do any
cache ops to flush or invalidate the data in cache. So skip them
if the page tables are not cached. Saves a few CPU cycles.
Signed-off-by: Daniel Leung <daniel.leung@intel.com>
In l2_page_table_unmap(), invalidating TLBs should be limited to
the address and not the whole auto-refill TLBs. Also fix a bug
where the EXEC bit should be determined from the L2 PTE and not
the L1 PTE.
Signed-off-by: Daniel Leung <daniel.leung@intel.com>
Some macros for the page tables array are local to the file so
there is no need to prefix them with XTENSA_. Simplify by
removing the XTENSA_ prefix, and clairfy if needed.
Signed-off-by: Daniel Leung <daniel.leung@intel.com>
Almost all page table entry (PTE) related macros are only used
in one source file. These macros are considered the internal
working of MMU. There is no need to expose them in header file
for other to reference. So move them into the source file where
it is only place they are used. Bonus is that we can shorten
the macro names as they are now local to the file. Makes it
easier to read, and few keystrokes to input.
Signed-off-by: Daniel Leung <daniel.leung@intel.com>
This skips the redirection of backup attributes and ring via
some SW field macros, and use the PTE bits directly. Should
make it easier to decode what's going on.
Signed-off-by: Daniel Leung <daniel.leung@intel.com>
The fields in macro should be done with both shift and mask
values to make it clear. So amend some macros so that
the presentation are unified for them.
Signed-off-by: Daniel Leung <daniel.leung@intel.com>
Add support for the RISC-V Smcsrind extension, which provides
indirect access to CSRs through the MISELECT and MIREG registers.
Changes:
- Added CONFIG_RISCV_ISA_EXT_SMCSRIND Kconfig option
- Implemented 4 helper functions for indirect CSR access:
* icsr_read/write - basic access
* icsr_read_set/clear - bit manipulation
- Defined 7 CSR registers (MISELECT, MIREG, MIREG2-6)
This is a CSR-only extension that does not require any compiler
support or march flags. The helper functions compile to standard
CSR instructions and work with any toolchain that supports Zicsr.
Primary use case: RISC-V AIA (Advanced Interrupt Architecture)
uses indirect CSRs to access IMSIC (Incoming MSI Controller)
registers.
Signed-off-by: Afonso Oliveira <afonsoo@synopsys.com>
Adds a new test suite to verify the behavior of `riscv_pmp_clear_all()`.
These tests ensure that the function correctly clears all unlocked PMP
entries while preserving any entries that are locked.
Signed-off-by: Firas Sammoura <fsammoura@google.com>
Introduce the new function `riscv_pmp_clear_all()` to reset the Physical
Memory Protection (PMP) configuration.
This function iterates through all configured PMP slots. For each entry,
it writes 0x0 to the entry's 8-bit configuration register. This action
attempts to clear all fields, including the Address Matching Mode (A) bits
(setting the region type to OFF), the permission bits (R, W, X), and
the Lock (L) bit.
According to the RISC-V specification, any writes to the configuration
or address registers of a locked PMP entry are ignored. Thus, locked
entries will remain unchanged, while all non-locked entries will be
effectively disabled and their permissions cleared.
The function ensures it operates in Machine mode with MSTATUS.MPRV = 0
and MSTATUS.MPP = M-mode before modifying any PMP Control and Status
Registers (CSRs).
This provides a mechanism to clear all non-locked PMP regions,
returning them to a default disabled state. The function declaration is
exposed in the `include/zephyr/arch/riscv/pmp.h` header file, making it
available for inclusion and use by external modules.
It is recommended for firmware to call this function before transitioning
from a Read-Only (RO) stage to a Read-Write (RW) stage. This ensures
that any PMP settings established during the RO phase, which might no
longer be appropriate, are cleared, providing a clean and secure base
PMP configuration for the RW firmware.
Signed-off-by: Firas Sammoura <fsammoura@google.com>
The ATOMIC_OPERATIONS_* Kconfig option is not a choice, so it does not
have a default. However, the file that determines which actual atomic
operations backend will be used does default to
ATOMIC_OPERATIONS_BUILTIN:
3e537db71e/include/zephyr/sys/atomic.h (L26-L41)
Since we want to ensure that all SoCs intentionally select the atomic
operations backend they want to use, select it at the SoC level for all
SoCs, as well as for the Cortex-M arch when the Armv8-M baseline profile
is selected.
Signed-off-by: Carles Cufi <carles.cufi@nordicsemi.no>
This reverts the following commits:
commit c9b534c4eb
("arch: arm64: mmu: avoid using of set/way cache instructions")
commit c4ffadb0b6
("arch: arm64: avoid invalidating of RO mem after mem map")
The reason for the former is about Xen not virtualizing set/way cache
operations used by sys_cache_data_invd_all() originally used prior to
enabling the MMU and data cache. But the cure is worse than the Xen
issue as:
- Cache invalidation is performed on _every_ mapping change.
- Those invalidations are completely unnecessary with a PIPT data cache.
ARM64 implementations use Physically Indexed, Physically Tagged (PIPT)
data caches where cache maintenance is not needed during MMU operations.
- arch_mem_map() invoked with K_MEM_MAP_UNPAGED triggers page faults
when accessing the unmapped region for cache operations. The page
fault handler in do_page_fault() tries to reacquire z_mm_lock which
is already held by the caller of arch_mem_map(). This results in a
deadlock.
And the latter commit disables cache operations for read-only mappings,
effectively rendering the workaround described in the first commit
inoperative on half the mappings, making the performance cost of the
first commit's approach unjustifiable since it doesn't actually solve
the problem it set out to fix.
Given the above, the actual "fix" should simply have been the removal of
the sys_cache_data_invd_all() as, in theory, it isn't strictly needed
and its replacement is already ineffective on read-only areas as mentioned.
So let's revert them, which fixes the deadlock-induced CI test failures
on ARM FVP SMP configurations that were triggered when demand paging or
memory mapping operations were involved.
Signed-off-by: Nicolas Pitre <npitre@baylibre.com>
Use Inner Shareable (IS) TLB invalidation instructions in SMP
configurations to broadcast TLB invalidations to all CPUs.
Use TLBI VMALLE1IS instead of VMALLE1 in invalidate_tlb_all().
While at it, implement proper page-specific invalidation using TLBI VAE1IS
in invalidate_tlb_page() instead of falling back to full invalidation.
This fixes many SMP test failures with userspace enabled onArm's FVP.
Signed-off-by: Nicolas Pitre <npitre@baylibre.com>
Disabling multithreading is not possible when enabling SMP (logically)
so depend on SMP being disabled to enable
ARCH_HAS_SINGLE_THREAD_SUPPORT.
Signed-off-by: Carles Cufi <carles.cufi@nordicsemi.no>
This change suppress the warning:
variable 'ie' is used uninitialized whenever switch default is
taken [-Wsometimes-uninitialized]
Signed-off-by: William Tambe <williamt@cadence.com>
For RISCV vector table needs to be aligned depending on
CONFIG_ARCH_IRQ_VECTOR_TABLE_ALIGN. This was missing
when using LTO making issues when direct ISR were in use.
Signed-off-by: Łukasz Stępnicki <lukasz.stepnicki@nordicsemi.no>
use CONFIG_RISCV_ISA_EXT_F to set CONFIG_CPU_HAS_FPU.
Same for CONFIG_RISCV_ISA_EXT_D and
CONFIG_CPU_HAS_FPU_DOUBLE_PRECISION.
Signed-off-by: Fin Maaß <f.maass@vogl-electronic.com>
`CONFIG_EXTRA_EXCEPTION_INFO` that was added in #78065 doesn't
seem necessary, as we were already storing and printing the
callee-saved-registers before that. All `CONFIG_EXTRA_EXCEPTION_INFO`
does in RISCV is to add an additional `_callee_saved_t *csf` in the
`struct arch_esf`, which overhead is negligible to what's being enabled
by `CONFIG_EXCEPTION_DEBUG`.
Let's remove `CONFIG_EXTRA_EXCEPTION_INFO`, and have that extra
`_callee_saved_t *csf` in the `struct arch_esf` as long as
`CONFIG_EXCEPTION_DEBUG` is enabled.
TL;DR: it doesn't make sense to not enable `CONFIG_EXTRA_EXCEPTION_INFO`
when `CONFIG_EXCEPTION_DEBUG` is enabled, so let's merge them.
Then, since `*csf` is always available in the `struct arch_esf` when
`CONFIG_EXCEPTION_DEBUG=y`, we can simply rely on that pointer in
`z_riscv_fatal_error()` instead of an additional argument in
`z_riscv_fatal_error_csf()`, rendering the latter redundant and thus
can be removed.
Additionally, save the callee-saved registers before jumping to
to `z_riscv_fault()`, so that callee-saved-registers are printed on
generic CPU exception as well.
Signed-off-by: Yong Cong Sin <yongcong.sin@gmail.com>
Signed-off-by: Yong Cong Sin <ycsin@meta.com>
Fixes local ISR generation so that devices actually boot, also
allows enabling LTO for these builds (tested working on nrf54l15
flpr device)
Signed-off-by: Jamie McCrae <spam@helper3000.net>
Let's move the null pointer trap first so to give set_pmp_entry() the
opportunity to use a single-slot TOR entry.
Signed-off-by: Nicolas Pitre <npitre@baylibre.com>
Update include of header file arm_mpu_mem_cfg.h which has been moved
to a Cortex-M/-R-agnostic include directory.
Signed-off-by: Immo Birnbaum <mail@birnbaum.immo>
Adds a new kconfig CONFIG_MEM_DOMAIN_HAS_THREAD_LIST so that
only the architectures requiring to keep track of threads in
memory domains will have the necessary list struct inside
the memory domain structs. Saves a few bytes for those arch
not needing this.
Also rename the struct fields to be most descriptive of what
they are.
Signed-off-by: Daniel Leung <daniel.leung@intel.com>
Refactor the code to read all PMP address CSRs (pmpaddr0 through
pmpaddrN) into a new helper function, `z_riscv_pmp_read_addr`.
This change encapsulates the register reading loop, improving code
organization and potential reusability. The new function includes size
assertions
Signed-off-by: Firas Sammoura <fsammoura@google.com>
Allow to configure the MMU for non-cacheable normal memories.
This mode is needed for instance by net samples to access to
to non word-aligned memory.
Signed-off-by: Arnaud Pouliquen <arnaud.pouliquen@foss.st.com>
Replace `sizeof(unsigned long)` with `PMPCFG_STRIDE` (defined as
`__riscv_xlen / 8`) for dimensioning PMP configuration register arrays
(`pmpcfg_regs`).
The size of PMP configuration registers should be derived directly from
the target architecture's XLEN. Using `sizeof(unsigned long)` can cause
size mismatches, particularly with static analysis tools like SonarQube.
These tools might assume a host-specific size for `unsigned long`,
leading to spurious out-of-bounds access warnings when analyzing code
for different RISCV base architectures (e.g., RV32 vs. RV64).
This change ensures the array sizing is correctly and consistently tied
to the target's register width (XLEN).
Signed-off-by: Firas Sammoura <fsammoura@google.com>
soc_prep_hook() is always called from z_prep_c() which is implemented
as a C function. As such, there is no need to check for the associated
CONFIG_SOC_PREP_HOOK since the platform/hooks.h header will define hooks
as no-op function-like macros if their associated Kconfig isn't enabled.
Remove the Kconfig check from all arch implementations of z_prep_c() and
call soc_prep_hook() directly instead, to avoid duplicating the Kconfig
check already performed in platform/hooks.h
Signed-off-by: Mathieu Choplain <mathieu.choplain-ext@st.com>
Introduce `z_riscv_pmp_write_config` to abstract writing to pmpcfg
registers. This function handles the differing register layouts and slot
counts between RV32 and RV64 architectures, writing to the appropriate
pmpcfg0, pmpcfg1, pmpcfg2, or pmpcfg3 CSRs as needed based on
CONFIG_PMP_SLOTS.
Signed-off-by: Firas Sammoura <fsammoura@google.com>
arch_mem_map() takes in some flags to describe the to-be mapped
memory regions' permissions and cache status. When the flags are
translated, they become attributes in PTEs. So for functions
being called by arch_mem_map() and beyond, rename flags to
attrs to better describe its purpose.
Signed-off-by: Daniel Leung <daniel.leung@intel.com>
In functions which manipulate both L1 and L2 tables, make
the variable names obvious by prefixing them with l1_ or l2_.
This is mainly done to avoid confusion when reading through
those functions.
Signed-off-by: Daniel Leung <daniel.leung@intel.com>
The assert error messages when l2_page_table_map() fails are not
correct. It returns false when it cannot allocate new L2 table,
and it is not able the address having already mapped. So correct
the error message.
Signed-off-by: Daniel Leung <daniel.leung@intel.com>
This is simply done to conserve code space in the vector text
areas. These vector text areas are very small and we should
only put code that is absolutely necessary for interrupt and
exception entrance. The saving of PS into the thread struct
can be deferred a bit. So do that.
Signed-off-by: Daniel Leung <daniel.leung@intel.com>
xtensa_asm2_s.h is not exactly a C header. Rename it to
xtensa_asm2.inc.S to clearly state that it is has assembly
code in it as its main purpose is to declare assembly macros.
This is being done to keep checkpatch.pl from treating it as
C file and complaining about non-C syntax.
Signed-off-by: Daniel Leung <daniel.leung@intel.com>
This moves the block of FPU and HIFI registers in the Base Save
Area (BSA), used during interrupts and exceptions, to the end of
the block. This is done to minimize code usage in the vector
text section. During interrupt entrance, the code stores a small
set of registers first before jumping to the long handler. When
the offset to these registers from the beginning of BSA is small
enough, the compiler will emit S32I.N instead of S32I. This
saves us one byte per store (2 vs 3 bytes). This is mainly done
to avoid overflowing the vector text area.
Signed-off-by: Daniel Leung <daniel.leung@intel.com>
This adds a function to handle DTLB multihit exception when
userspace is enabled, as this exception may be raised due to
the same page being able to accessed by both kernel and user
threads. The auto-refill DTLBs may contain entries for same
page, one for kernel and one for user under some situations.
We need to invalidate those existing DTLB entries so that
hardware can reload from the page table.
This is an alternative to the kconfig invalidating DTLBs on
every swap: CONFIG_XTENSA_MMU_FLUSH_AUTOREFILL_DTLBS_ON_SWAP.
Although this does not have extra processing per context
switching, exception handling itself has a high cost. So
care needs to be taken on whether to enable that kconfig.
Signed-off-by: Daniel Leung <daniel.leung@intel.com>
When a page can be accessed by both kernel and user threads,
the autofill DTLB may contain an entry for kernel thread.
This will result in load/store ring exception when it is
accessed by user thread later. In this case, we need to
invalidate all associated TLBs related to kernel access so
hardware can reload the page table the correct permission
for user thread.
Signed-off-by: Daniel Leung <daniel.leung@intel.com>
This saves the EXCCAUSE and EXCVADDR into BSA during exception.
These will then be used during exception handling. The reason
for doing this instead of reading from both registers during
exception handing is that another exception will override
the value in these register (e.g. DTLB miss). When returning to
the previous exception handler, these register no longer
contains the original exception cause and address. We need to
save it so that we are actually handling the orignal exception.
Coredump also now uses the EXCCAUSE and EXCVADDR from BSA
instead of reading the registers.
Signed-off-by: Daniel Leung <daniel.leung@intel.com>
The software bits inside PTE are used to store original PTE
attributes and ring value, and those bits are used to
restore PTE to previous state.
This modifies reset_region() to properly restore attributes
and ring value when resetting memory regions to the same as
when the system boots.
Signed-off-by: Daniel Leung <daniel.leung@intel.com>
