Commit graph

263 commits

Author SHA1 Message Date
Nicolas Pitre
6255bf1ef4 arch/arm64/mmu: allow partial unmap of block mappings again
Before commit baa70d8d36 ("arch/arm64/mmu: fix page table reference
counting part 2")  it was possible to perform a partial unmap of a block
mapping. Restore that ability and provide a test case to validate it.

Signed-off-by: Nicolas Pitre <npitre@baylibre.com>
2024-08-27 15:14:43 -04:00
Yong Cong Sin
ab676fdb86 arch: arm64: implement arch_stack_walk()
Currently it supports `esf` based unwinding only.

Then, update the exception stack unwinding to use
`arch_stack_walk()`, and update the Kconfigs & testcase
accordingly.

Also, `EXCEPTION_STACK_TRACE_MAX_FRAMES` is unused and
made redundant after this change, so remove it.

Signed-off-by: Yong Cong Sin <ycsin@meta.com>
Signed-off-by: Yong Cong Sin <yongcong.sin@gmail.com>
2024-08-26 14:44:53 -04:00
Anas Nashif
a91c6e56c8 arch: use same syntax for custom arch calls
Use same Kconfig syntax for those  custom arch call.

Signed-off-by: Anas Nashif <anas.nashif@intel.com>
2024-08-12 12:43:36 +02:00
Anas Nashif
7f52fc4188 arch: custom cpu_idle and cpu_atomic harmonization
custom arch_cpu_idle and arch_cpu_atomic_idle implementation was done
differently on different architectures. riscv implemented those as weak
symbols, xtensa used a kconfig and all other architectures did not
really care, but this was a global kconfig that should apply to all
architectures.

Signed-off-by: Anas Nashif <anas.nashif@intel.com>
2024-08-12 12:43:36 +02:00
Nicolas Pitre
b5be646e20 arch/arm64/mmu: improve debugging output
Rationalize the debug log to make it easier to use.

Signed-off-by: Nicolas Pitre <npitre@baylibre.com>
2024-06-26 13:06:42 -04:00
Nicolas Pitre
d7e9ea0b71 arch/arm64/mmu: fix page table reference counting part 3
Commit f7e11649fd ("arch/arm64/mmu: fix page table reference
counting") missed another case where the freeing of a whole table
"branch" didn't take into account the fact that some sub-tables might
be shared and therefore must be cleared only if the reference count is
down to 1.

Signed-off-by: Nicolas Pitre <npitre@baylibre.com>
2024-06-26 13:06:42 -04:00
Nicolas Pitre
a4ebd5e8db arch/arm64/mmu: move hardware specifics to private header
None of the moved definitions are meant to be used by any code outside
of arch/arm64/core/mmu.c. Move them away from global scope to the
private header where more such definitions already live.

This is especially relevant as the previous commit fixed some of those
definitions which then caused conflicts with some external SDK that
carries a copy of those original buggy definitions.

Signed-off-by: Nicolas Pitre <npitre@baylibre.com>
2024-06-24 22:24:50 -04:00
Nicolas Pitre
1e27100442 arch/arm64/mmu: use 64-bit mask values with page table descriptors
Inverting a mask whose type has only 32 bits  doesn't produce the
expected result. Fix those to be 64-bit values.

Signed-off-by: Nicolas Pitre <npitre@baylibre.com>
2024-06-24 22:24:50 -04:00
Nicolas Pitre
696edbe841 arm64: move simple memcpy/memset alternatives to assembly
Assembly implementation for z_early_memset() and z_early_memcpy().
Otherwise the compiler will happily replace our C code with a direct
call to memset/memcpy which kind of defeats the purpose.

Signed-off-by: Nicolas Pitre <npitre@baylibre.com>
2024-06-14 17:11:40 -04:00
Nicolas Pitre
64855973c0 arm64: speed up simple memcpy/memset alternatives
We need those simple alternatives to be used during early boot when the
MMU is not yet enabled. However they don't have to be the slowest they
can be. Those functions are mainly used to clear .bss sections and copy
.data to final destination when doing XIP, etc. Therefore it is very
likely for provided pointers to be 64-bit aligned. Let's optimize for
that case.

Signed-off-by: Nicolas Pitre <npitre@baylibre.com>
2024-06-14 17:11:40 -04:00
Nicolas Pitre
baa70d8d36 arch/arm64/mmu: fix page table reference counting part 2
Commit f7e11649fd ("arch/arm64/mmu: fix page table reference
counting") missed a case where the freeing of a table wasn't propagated
properly to all domains. To fix this, the page freeing logic was removed
from set_mapping() and a del_mapping() was created instead, to be usedby
both by remove_map() and globalize_table().

A test covering this case should definitely be created but that'll come
later. Proper operation was verified through manual debug log
inspection for now.

Signed-off-by: Nicolas Pitre <npitre@baylibre.com>
2024-06-13 17:03:33 -04:00
Nicolas Pitre
6cc9d7ec20 tests: arm64: exercize MMU page table allocation and recycling
This code was never formally tested before... and without the preceding
commit it obviously didn't work either.

Signed-off-by: Nicolas Pitre <npitre@baylibre.com>
2024-06-13 05:47:24 -04:00
Nicolas Pitre
f7e11649fd arch/arm64/mmu: fix page table reference counting
Existing code confused table usage and table reference counts together.
This obviously doesn't work. A table with one reference to it and one
populated PTE is not the same as a table with 2 references to it and
no PTe in use.

So split the two concepts and adjust the code accordingly. A page needs
to have its PTE usage count drop to zero before the last reference is
released. When both counts are 0 then the page is free.

Signed-off-by: Nicolas Pitre <npitre@baylibre.com>
2024-06-13 05:47:24 -04:00
Anas Nashif
c20e798646 arch: call arch_smp_init() directly, do not use SYS_INIT
Move this to a call in the init process. arch_* calls are no services
and should be called consistently during initialization.

Place it between PRE_KERNEL_1 and PRE_KERNEL_2 as some drivers
initialized in PRE_KERNEL_2 might depend on SMP being setup.

Signed-off-by: Anas Nashif <anas.nashif@intel.com>
2024-06-12 18:23:54 -04:00
Peter Mitsis
0bcdae2c62 kernel: Add CONFIG_ARCH_HAS_DIRECTED_IPIS
Platforms that support IPIs allow them to be broadcast via the
new arch_sched_broadcast_ipi() routine (replacing arch_sched_ipi()).
Those that also allow IPIs to be directed to specific CPUs may
use arch_sched_directed_ipi() to do so.

As the kernel has the capability to track which CPUs may need an IPI
(see CONFIG_IPI_OPTIMIZE), this commit updates the signalling of
tracked IPIs to use the directed version if supported; otherwise
they continue to use the broadcast version.

Platforms that allow directed IPIs may see a significant reduction
in the number of IPI related ISRs when CONFIG_IPI_OPTIMIZE is
enabled and the number of CPUs increases.  These platforms can be
identified by the Kconfig option CONFIG_ARCH_HAS_DIRECTED_IPIS.

Signed-off-by: Peter Mitsis <peter.mitsis@intel.com>
2024-06-04 22:35:54 -04:00
Yong Cong Sin
e54b27b967 arch: define struct arch_esf and deprecate z_arch_esf_t
Make `struct arch_esf` compulsory for all architectures by
declaring it in the `arch_interface.h` header.

After this commit, the named struct `z_arch_esf_t` is only used
internally to generate offsets, and is slated to be removed
from the `arch_interface.h` header in the future.

Signed-off-by: Yong Cong Sin <ycsin@meta.com>
2024-06-04 14:02:51 -05:00
Yong Cong Sin
3998e18ec4 arch: rename all esf struct to struct arch_esf
Rename every architecture's esf struct to `struct esf`.

Signed-off-by: Yong Cong Sin <ycsin@meta.com>
2024-06-04 14:02:51 -05:00
Yong Cong Sin
02770ad963 debug: EXCEPTION_STACK_TRACE should depend on arch Kconfigs
Fix the dependencies of `CONFIG_EXCEPTION_STACK_TRACE`:
- Architecture-specific Kconfig, i.e.
  `X86_EXCEPTION_STACK_TRACE`, will be enabled automatically
  when all the dependencies are met.
- `EXCEPTION_STACK_TRACE` depends on architecture-specific
  Kconfig to be enabled.
- The stack trace implementations should be compiled only if
  user enables `CONFIG_EXCEPTION_STACK_TRACE`.

Signed-off-by: Yong Cong Sin <ycsin@meta.com>
2024-06-03 03:02:04 -07:00
Yong Cong Sin
190777dccf arch: arm64: create ARM64_EXCEPTION_STACK_TRACE
Currently, the stack trace in ARM64 implementation depends on
frame pointer Kconfigs combo to be enabled. Create a dedicated
Kconfig for that instead, so that it is consistent with x86 and
riscv, and update the source accordingly.

Signed-off-by: Yong Cong Sin <ycsin@meta.com>
2024-06-03 03:02:04 -07:00
Yong Cong Sin
94346d2441 arch: arm64: fatal: limit max number of stack traces
In some cases, the `fp` will never be `NULL` and the stack
unwinding can go on and on forever, limit the max depth so that
this will not happen.

Signed-off-by: Yong Cong Sin <ycsin@meta.com>
2024-05-30 03:00:50 -07:00
Yong Cong Sin
c87dc641bc arch: generalize frame pointer via CONFIG_FRAME_POINTER introduction
Enabling `CONFIG_FRAME_POINTER` allows the users to build the
kernel with frame-pointer.

Signed-off-by: Yong Cong Sin <ycsin@meta.com>
2024-05-30 03:00:40 -07:00
Yong Cong Sin
bbe5e1e6eb build: namespace the generated headers with zephyr/
Namespaced the generated headers with `zephyr` to prevent
potential conflict with other headers.

Introduce a temporary Kconfig `LEGACY_GENERATED_INCLUDE_PATH`
that is enabled by default. This allows the developers to
continue the use of the old include paths for the time being
until it is deprecated and eventually removed. The Kconfig will
generate a build-time warning message, similar to the
`CONFIG_TIMER_RANDOM_GENERATOR`.

Updated the includes path of in-tree sources accordingly.

Most of the changes here are scripted, check the PR for more
info.

Signed-off-by: Yong Cong Sin <ycsin@meta.com>
2024-05-28 22:03:55 +02:00
Yong Cong Sin
2a3d9d0d90 arch: arm64: use symtab to print function name in stack trace
Selecting `CONFIG_SYMTAB` will
enable the symtab generation which will be used in the
stack trace to print the function name of the return
address.

Added `arm64` to the `arch.common.stack_unwind.symtab` test.

Signed-off-by: Yong Cong Sin <ycsin@meta.com>
2024-05-23 11:52:08 -04:00
Hess Nathan
ed12a0cc35 coding guidelines: comply with MISRA Rule 11.8
modified parameter types to receive a const pointer when a
non-const pointer is not needed

Signed-off-by: Hess Nathan <nhess@baumer.com>
2024-05-09 10:28:44 +02:00
Debbie Martin
882238116e arch: arm: cortex-r: Add compiler tuning for Cortex-R82
Change the GCC toolchain configuration to make use of the Cortex-R82
target. When Cortex-R82 was added as a GCC toolchain option, the GCC
version of the Zephyr SDK did not support Cortex-R82 tuning. Zephyr was
therefore compiled compiled for the Armv8.4-A architecture. Since Zephyr
SDK 0.15.0 (which updated GCC from 10.3.0 to 12.1.0) coupled with Zephyr
3.2, the Cortex-R82 target is supported.

The Armv8-R AArch64 architecture does not support the EL3 exception level.
EL3 support is therefore made conditional on Armv8-R vs Armv8-A.

Signed-off-by: Debbie Martin <Debbie.Martin@arm.com>
2024-05-07 17:57:05 -04:00
Daniel Leung
b69d2486fe kernel: rename Z_KERNEL_STACK_BUFFER to K_KERNEL_STACK_BUFFER
Simple rename to align the kernel naming scheme. This is being
used throughout the tree, especially in the architecture code.
As this is not a private API internal to kernel, prefix it
appropriately with K_.

Signed-off-by: Daniel Leung <daniel.leung@intel.com>
2024-03-27 19:27:10 -04:00
Daniel Leung
6ea749de52 arch: rename arch_start_cpu() to arch_cpu_start()
Rename arch_start_cpu() to arch_cpu_start() so it belongs to
the "cpu" namespace.

Signed-off-by: Daniel Leung <daniel.leung@intel.com>
2024-03-25 09:58:35 +00:00
Daniel Leung
3664ed64c3 arch: move arch_interface.h under zephyr/arch
arch_interface.h is for architecture and should not be
under sys/. So move it under include/zephyr/arch/.

Signed-off-by: Daniel Leung <daniel.leung@intel.com>
2024-03-25 09:58:35 +00:00
Daniel Leung
3a1784432b arch: arm64: correct a comment on CONFIG_ARM64_STACK_PROTECTION
There is a #endif comment which was incorrectly marked with
CONFIG_HW_STACK_PROTECTION instead of
CONFIG_ARM64_STACK_PROTECTION, which is used at #if.
So update it.

Signed-off-by: Daniel Leung <daniel.leung@intel.com>
2024-03-11 08:16:06 -04:00
Radoslaw Koppel
08070fbf1f arch: arm, arm64: Remove zephyr prefix from linker includes
This fix removes the zephyr/ prefix from linker included files.
With this prefix the build works only for Ninja and not for
other build tools.

Linking in Zephyr / CMake:
 - Ninja invokes linking directly from <build>.
 - Make invokes linking form <build>/zephyr.

The linker default uses cwd for looking up INCLUDE directives if not found
in list of includes.
Zephyr always adds <build>/zephyr as link include using CMake,
and this is passed to ld as -L<build>/zephyr therefore using
INCLUDE isr_tables_swi.ld ensures it will be correctly found in all cases.

Signed-off-by: Radosław Koppel <radoslaw.koppel@nordicsemi.no>
2024-02-05 12:32:46 +01:00
Radoslaw Koppel
adaacefc20 arch: arm, arm64: Disable swi_tables.ld file when not required
This commit removes the need of swi_tables.ld file if the
ISR table generator is not configured to use it.

Signed-off-by: Radosław Koppel <radoslaw.koppel@nordicsemi.no>
2024-02-05 12:32:46 +01:00
Radoslaw Koppel
1ff24b34eb arch: arm: Update to support local ISR declaration
This commit updates the arm and arm64 architecture files
to support the new ISR handlers creation parser.

Signed-off-by: Radosław Koppel <radoslaw.koppel@nordicsemi.no>
2024-02-02 19:49:36 +01:00
honglin leng
c9208b3864 arm64: smp: Fix cache operations in the SMP
The arm64_cpu_boot_params will be read on other cores

call sys_cache_data_flush_range flush the data from the cache to RAM.

This ensures that other cores can access the correct data.

Signed-off-by: honglin leng <a909204013@gmail.com>
2024-01-29 09:26:51 +01:00
Anas Nashif
7d3b6c6a40 arch: smp: make flush_fpu_ipi a common, optional interfaces
The interface to flush fpu is not unique to one architecture, make it a
generic, optional interface that can be implemented (and overriden) by a
platform.

Signed-off-by: Anas Nashif <anas.nashif@intel.com>
2024-01-09 10:00:17 +01:00
Anas Nashif
3f75373584 arch: introduce arch_secondary_cpu_init
Different architecture are doing this in custom ways and using different
naming conventions, unify this interface and make it part of the arch
implementation for SMP.

Signed-off-by: Anas Nashif <anas.nashif@intel.com>
2024-01-09 10:00:17 +01:00
Anas Nashif
37f427a5c7 arch: introduce arch_smp_init
Introduce a new arch interface for intializing smp.

Signed-off-by: Anas Nashif <anas.nashif@intel.com>
2024-01-09 10:00:17 +01:00
Anas Nashif
044e2d6bff arch: make CONFIG_EXCEPTION_DEBUG cross arch config
Define CONFIG_EXCEPTION_DEBUG globally and remove architecture specific
definition of config.

Signed-off-by: Anas Nashif <anas.nashif@intel.com>
2023-12-14 09:32:27 +01:00
Anas Nashif
4a73fb1b92 x86: add CODE_UNREACHABLE after z_cstart
Add CODE_UNREACHABLE to be consistent with all other architectures.

Signed-off-by: Anas Nashif <anas.nashif@intel.com>
2023-12-11 18:23:52 -05:00
Anas Nashif
5a77fc239f arch: arm: z_arm64_prep_c -> z_prep_c
Rename to use common naming for z_prep_c applied to all architectures.

Signed-off-by: Anas Nashif <anas.nashif@intel.com>
2023-12-11 18:23:52 -05:00
Anas Nashif
ca3839ca38 arch: arm64: rename expection header
Rename exception header and use the same name as all architecture ports.

Signed-off-by: Anas Nashif <anas.nashif@intel.com>
2023-12-11 18:22:40 -05:00
Mykola Kvach
c4ffadb0b6 arch: arm64: avoid invalidating of RO mem after mem map
The Cortex ARM documentation states that the DC IVAC instruction
requires write access permission to the virtual address (VA);
otherwise, it may generate a permission fault.

Therefore, it is needed to avoid invalidating read-only memory
after the memory map operation.

This issue has been produced by commit c9b534c.
This commit resolves the issue #64758.

Signed-off-by: Mykola Kvach <mykola_kvach@epam.com>
2023-11-09 10:19:12 +00:00
Charlie Xiong
314b72f332 arch: arm64: Re-init HCR_EL2 in z_arm64_el2_init
HCR_EL2 is configured to certain value by some
loaders such as Uboot on some arm64 boards(such as roc_rk3568_pc),
When HCR_EL2.TGE, HCR_EL2.AMO and HCR_EL2.IMO bits are
set to 1, some unpredictable behaviors may occur during
zephyr boot. So we clear these bits to avoid it.

Signed-off-by: Charlie Xiong <1981639884@qq.com>
2023-11-06 10:14:20 +01:00
Jaxson Han
7904c6f0f3 arch: arm64: Use voting lock for multi-core boot race condition
The exclusive load/store instructions don't work well when MMU and cache
are disabled on some cores e.g. Cortex-A72. Change it to voting lock[1]
to select the primary core when multi-cores boot simultaneously.

The voting lock has reasonable but minimal requirements on the memory
system.

[1] https://www.kernel.org/doc/html/next/arch/arm/vlocks.html

Signed-off-by: Jaxson Han <jaxson.han@arm.com>
2023-10-20 15:09:34 +02:00
Carlo Caione
85c4111002 memory-attr: Rationalize _MASK and _GET(x) macros
Let's make this official: we use the suffix `_MASK` for the define
carrying the GENMASK for the attributes, and the suffix `_GET(x)` for
the actual macro extracting the attributes.

Signed-off-by: Carlo Caione <ccaione@baylibre.com>
2023-09-22 09:27:57 +02:00
Jaxson Han
0928830409 arch: arm64: Enable stack guard for v8R
Enable stack guard for v8R which is backed by MPU.

Signed-off-by: Jaxson Han <jaxson.han@arm.com>
2023-09-22 09:25:12 +02:00
Jaxson Han
5878898328 arch: arm64: Add stack guard support for MPU
To make the stack guard works well, clean and refine the MPU code. To
save the MPU regions (the number of MPU regions are limited), we choose
to remove the guard region. Comparing to add an individual region to
guard the stack, removing the guard region can at least save 2 regions
per core.

Similarly with userspace, the stack guard will leverage the dynamic
regions switching mechanism which means we need a region switch during
the context switch. Otherwise, the other option is using stack guard
region, but this is very limited since the number of MPU regions is
limited.

Signed-off-by: Jaxson Han <jaxson.han@arm.com>
2023-09-22 09:25:12 +02:00
Jaxson Han
d3ec98806d arch: arm64: Refactor the stack relevant macros
Refactor the stack relevant macros to prepare to introduce the stack
guard. Also add comments about the changes related to stack layout.

Signed-off-by: Jaxson Han <jaxson.han@arm.com>
2023-09-22 09:25:12 +02:00
Jaxson Han
d4dd47b9c5 arch: arm64: Add stack check at z_arm64_fatal_error
Add the stack check function z_arm64_stack_corruption_check at
z_arm64_fatal_error to handle the stack overflow triggered by the
hardware region.

Signed-off-by: Jaxson Han <jaxson.han@arm.com>
2023-09-22 09:25:12 +02:00
Jaxson Han
1fe9a06d8b arch: arm64: Kconfig: Introduce ARM64_STACK_PROTECTION
Introduce the ARM64_STACK_PROTECTION config. This option leverages the
MMU or MPU to cause a system fatal error if the bounds of the current
process stack are overflowed. This is done by preceding all stack areas
with a fixed guard region. The config depends on MPU for now since MMU
stack protection is not ready.

Signed-off-by: Jaxson Han <jaxson.han@arm.com>
2023-09-22 09:25:12 +02:00
Jaxson Han
1ac3d1cc5e arch: arm64: thread: Clean thread arch when creating thread
Clean the thread->arch during the arch_new_thread to avoid unexpected
behavior. If the thread struct is allocated from heap or in stack, the
data in thread->arch might be dirty.

Signed-off-by: Jaxson Han <jaxson.han@arm.com>
2023-09-22 09:25:12 +02:00