Assembly implementation for z_early_memset() and z_early_memcpy().
Otherwise the compiler will happily replace our C code with a direct
call to memset/memcpy which kind of defeats the purpose.
Signed-off-by: Nicolas Pitre <npitre@baylibre.com>
We need those simple alternatives to be used during early boot when the
MMU is not yet enabled. However they don't have to be the slowest they
can be. Those functions are mainly used to clear .bss sections and copy
.data to final destination when doing XIP, etc. Therefore it is very
likely for provided pointers to be 64-bit aligned. Let's optimize for
that case.
Signed-off-by: Nicolas Pitre <npitre@baylibre.com>
Commit f7e11649fd ("arch/arm64/mmu: fix page table reference
counting") missed a case where the freeing of a table wasn't propagated
properly to all domains. To fix this, the page freeing logic was removed
from set_mapping() and a del_mapping() was created instead, to be usedby
both by remove_map() and globalize_table().
A test covering this case should definitely be created but that'll come
later. Proper operation was verified through manual debug log
inspection for now.
Signed-off-by: Nicolas Pitre <npitre@baylibre.com>
This code was never formally tested before... and without the preceding
commit it obviously didn't work either.
Signed-off-by: Nicolas Pitre <npitre@baylibre.com>
Existing code confused table usage and table reference counts together.
This obviously doesn't work. A table with one reference to it and one
populated PTE is not the same as a table with 2 references to it and
no PTe in use.
So split the two concepts and adjust the code accordingly. A page needs
to have its PTE usage count drop to zero before the last reference is
released. When both counts are 0 then the page is free.
Signed-off-by: Nicolas Pitre <npitre@baylibre.com>
Move this to a call in the init process. arch_* calls are no services
and should be called consistently during initialization.
Place it between PRE_KERNEL_1 and PRE_KERNEL_2 as some drivers
initialized in PRE_KERNEL_2 might depend on SMP being setup.
Signed-off-by: Anas Nashif <anas.nashif@intel.com>
Platforms that support IPIs allow them to be broadcast via the
new arch_sched_broadcast_ipi() routine (replacing arch_sched_ipi()).
Those that also allow IPIs to be directed to specific CPUs may
use arch_sched_directed_ipi() to do so.
As the kernel has the capability to track which CPUs may need an IPI
(see CONFIG_IPI_OPTIMIZE), this commit updates the signalling of
tracked IPIs to use the directed version if supported; otherwise
they continue to use the broadcast version.
Platforms that allow directed IPIs may see a significant reduction
in the number of IPI related ISRs when CONFIG_IPI_OPTIMIZE is
enabled and the number of CPUs increases. These platforms can be
identified by the Kconfig option CONFIG_ARCH_HAS_DIRECTED_IPIS.
Signed-off-by: Peter Mitsis <peter.mitsis@intel.com>
Make `struct arch_esf` compulsory for all architectures by
declaring it in the `arch_interface.h` header.
After this commit, the named struct `z_arch_esf_t` is only used
internally to generate offsets, and is slated to be removed
from the `arch_interface.h` header in the future.
Signed-off-by: Yong Cong Sin <ycsin@meta.com>
Fix the dependencies of `CONFIG_EXCEPTION_STACK_TRACE`:
- Architecture-specific Kconfig, i.e.
`X86_EXCEPTION_STACK_TRACE`, will be enabled automatically
when all the dependencies are met.
- `EXCEPTION_STACK_TRACE` depends on architecture-specific
Kconfig to be enabled.
- The stack trace implementations should be compiled only if
user enables `CONFIG_EXCEPTION_STACK_TRACE`.
Signed-off-by: Yong Cong Sin <ycsin@meta.com>
Currently, the stack trace in ARM64 implementation depends on
frame pointer Kconfigs combo to be enabled. Create a dedicated
Kconfig for that instead, so that it is consistent with x86 and
riscv, and update the source accordingly.
Signed-off-by: Yong Cong Sin <ycsin@meta.com>
In some cases, the `fp` will never be `NULL` and the stack
unwinding can go on and on forever, limit the max depth so that
this will not happen.
Signed-off-by: Yong Cong Sin <ycsin@meta.com>
Namespaced the generated headers with `zephyr` to prevent
potential conflict with other headers.
Introduce a temporary Kconfig `LEGACY_GENERATED_INCLUDE_PATH`
that is enabled by default. This allows the developers to
continue the use of the old include paths for the time being
until it is deprecated and eventually removed. The Kconfig will
generate a build-time warning message, similar to the
`CONFIG_TIMER_RANDOM_GENERATOR`.
Updated the includes path of in-tree sources accordingly.
Most of the changes here are scripted, check the PR for more
info.
Signed-off-by: Yong Cong Sin <ycsin@meta.com>
Selecting `CONFIG_SYMTAB` will
enable the symtab generation which will be used in the
stack trace to print the function name of the return
address.
Added `arm64` to the `arch.common.stack_unwind.symtab` test.
Signed-off-by: Yong Cong Sin <ycsin@meta.com>
Change the GCC toolchain configuration to make use of the Cortex-R82
target. When Cortex-R82 was added as a GCC toolchain option, the GCC
version of the Zephyr SDK did not support Cortex-R82 tuning. Zephyr was
therefore compiled compiled for the Armv8.4-A architecture. Since Zephyr
SDK 0.15.0 (which updated GCC from 10.3.0 to 12.1.0) coupled with Zephyr
3.2, the Cortex-R82 target is supported.
The Armv8-R AArch64 architecture does not support the EL3 exception level.
EL3 support is therefore made conditional on Armv8-R vs Armv8-A.
Signed-off-by: Debbie Martin <Debbie.Martin@arm.com>
Simple rename to align the kernel naming scheme. This is being
used throughout the tree, especially in the architecture code.
As this is not a private API internal to kernel, prefix it
appropriately with K_.
Signed-off-by: Daniel Leung <daniel.leung@intel.com>
arch_interface.h is for architecture and should not be
under sys/. So move it under include/zephyr/arch/.
Signed-off-by: Daniel Leung <daniel.leung@intel.com>
There is a #endif comment which was incorrectly marked with
CONFIG_HW_STACK_PROTECTION instead of
CONFIG_ARM64_STACK_PROTECTION, which is used at #if.
So update it.
Signed-off-by: Daniel Leung <daniel.leung@intel.com>
This fix removes the zephyr/ prefix from linker included files.
With this prefix the build works only for Ninja and not for
other build tools.
Linking in Zephyr / CMake:
- Ninja invokes linking directly from <build>.
- Make invokes linking form <build>/zephyr.
The linker default uses cwd for looking up INCLUDE directives if not found
in list of includes.
Zephyr always adds <build>/zephyr as link include using CMake,
and this is passed to ld as -L<build>/zephyr therefore using
INCLUDE isr_tables_swi.ld ensures it will be correctly found in all cases.
Signed-off-by: Radosław Koppel <radoslaw.koppel@nordicsemi.no>
This commit removes the need of swi_tables.ld file if the
ISR table generator is not configured to use it.
Signed-off-by: Radosław Koppel <radoslaw.koppel@nordicsemi.no>
This commit updates the arm and arm64 architecture files
to support the new ISR handlers creation parser.
Signed-off-by: Radosław Koppel <radoslaw.koppel@nordicsemi.no>
The arm64_cpu_boot_params will be read on other cores
call sys_cache_data_flush_range flush the data from the cache to RAM.
This ensures that other cores can access the correct data.
Signed-off-by: honglin leng <a909204013@gmail.com>
The interface to flush fpu is not unique to one architecture, make it a
generic, optional interface that can be implemented (and overriden) by a
platform.
Signed-off-by: Anas Nashif <anas.nashif@intel.com>
Different architecture are doing this in custom ways and using different
naming conventions, unify this interface and make it part of the arch
implementation for SMP.
Signed-off-by: Anas Nashif <anas.nashif@intel.com>
The Cortex ARM documentation states that the DC IVAC instruction
requires write access permission to the virtual address (VA);
otherwise, it may generate a permission fault.
Therefore, it is needed to avoid invalidating read-only memory
after the memory map operation.
This issue has been produced by commit c9b534c.
This commit resolves the issue #64758.
Signed-off-by: Mykola Kvach <mykola_kvach@epam.com>
HCR_EL2 is configured to certain value by some
loaders such as Uboot on some arm64 boards(such as roc_rk3568_pc),
When HCR_EL2.TGE, HCR_EL2.AMO and HCR_EL2.IMO bits are
set to 1, some unpredictable behaviors may occur during
zephyr boot. So we clear these bits to avoid it.
Signed-off-by: Charlie Xiong <1981639884@qq.com>
The exclusive load/store instructions don't work well when MMU and cache
are disabled on some cores e.g. Cortex-A72. Change it to voting lock[1]
to select the primary core when multi-cores boot simultaneously.
The voting lock has reasonable but minimal requirements on the memory
system.
[1] https://www.kernel.org/doc/html/next/arch/arm/vlocks.html
Signed-off-by: Jaxson Han <jaxson.han@arm.com>
Let's make this official: we use the suffix `_MASK` for the define
carrying the GENMASK for the attributes, and the suffix `_GET(x)` for
the actual macro extracting the attributes.
Signed-off-by: Carlo Caione <ccaione@baylibre.com>
To make the stack guard works well, clean and refine the MPU code. To
save the MPU regions (the number of MPU regions are limited), we choose
to remove the guard region. Comparing to add an individual region to
guard the stack, removing the guard region can at least save 2 regions
per core.
Similarly with userspace, the stack guard will leverage the dynamic
regions switching mechanism which means we need a region switch during
the context switch. Otherwise, the other option is using stack guard
region, but this is very limited since the number of MPU regions is
limited.
Signed-off-by: Jaxson Han <jaxson.han@arm.com>
Refactor the stack relevant macros to prepare to introduce the stack
guard. Also add comments about the changes related to stack layout.
Signed-off-by: Jaxson Han <jaxson.han@arm.com>
Add the stack check function z_arm64_stack_corruption_check at
z_arm64_fatal_error to handle the stack overflow triggered by the
hardware region.
Signed-off-by: Jaxson Han <jaxson.han@arm.com>
Introduce the ARM64_STACK_PROTECTION config. This option leverages the
MMU or MPU to cause a system fatal error if the bounds of the current
process stack are overflowed. This is done by preceding all stack areas
with a fixed guard region. The config depends on MPU for now since MMU
stack protection is not ready.
Signed-off-by: Jaxson Han <jaxson.han@arm.com>
Clean the thread->arch during the arch_new_thread to avoid unexpected
behavior. If the thread struct is allocated from heap or in stack, the
data in thread->arch might be dirty.
Signed-off-by: Jaxson Han <jaxson.han@arm.com>
Accessing mem before mmu or mpu init will cause a cache coherence issue.
To avoid such a problem, move the safe exception stack init function
after the mmu or mpu is initiated.
Also change the data section attribute from INNER_SHAREABLE to
OUTER_SHAREABLE. Otherwise there will be a cache coherence issue during
the memory regions switch. Because we are using background region to do
the regions switch, and the default background region is
OUTER_SHAREABLE, if we use INNER_SHAREABLE as the foreground region,
then we have to flush all cache regions to make sure the cached values
are right. However, flushing all regions is too heavy, so we set
OUTER_SHAREABLE to fix this issue.
Signed-off-by: Jaxson Han <jaxson.han@arm.com>
Dom0less is Xen mode without privileged domain. All guests are created
according to hypervisor device tree configuration on boot. Thus, there
is no Dom0 with console daemon, that usually manages console output
from domains.
Zephyr OS contains 2 serial drivers related to Xen hypervisor: regular
with console shared page and consoleio-based. The first one is for
setups with console daemon and usually was used for Zephyr DomU guests.
The second one previously was used only for Zephyr Dom0 and had
corresponding Kconfig options. But consoleio is also used as interface
for DomU output on Dom0less setups and should be configurable without
XEN_DOM0 option.
Add corresponding XEN_DOM0LESS config to Xen Kconfig files and proper
dependencies in serial drivers.
Signed-off-by: Dmytro Firsov <dmytro_firsov@epam.com>
Co-authored-by: Oleksandr Tyshchenko <oleksandr_tyshchenko@epam.com>
This is the final step in making the `zephyr,memory-attr` property
actually useful.
The problem with the current implementation is that `zephyr,memory-attr`
is an enum type, this is making very difficult to use that to actually
describe the memory capabilities. The solution proposed in this PR is to
use the `zephyr,memory-attr` property as an OR-ed bitmask of memory
attributes.
With the change proposed in this PR it is possible in the DeviceTree to
mark the memory regions with a bitmask of attributes by using the
`zephyr,memory-attr` property. This property and the related memory
region can then be retrieved at run-time by leveraging a provided helper
library or the usual DT helpers.
The set of general attributes that can be specified in the property are
defined and explained in
`include/zephyr/dt-bindings/memory-attr/memory-attr.h` (the list can be
extended when needed).
For example, to mark a memory region in the DeviceTree as volatile,
non-cacheable, out-of-order:
mem: memory@10000000 {
compatible = "mmio-sram";
reg = <0x10000000 0x1000>;
zephyr,memory-attr = <( DT_MEM_VOLATILE |
DT_MEM_NON_CACHEABLE |
DT_MEM_OOO )>;
};
The `zephyr,memory-attr` property can also be used to set
architecture-specific custom attributes that can be interpreted at run
time. This is leveraged, among other things, to create MPU regions out
of DeviceTree defined memory regions on ARM, for example:
mem: memory@10000000 {
compatible = "mmio-sram";
reg = <0x10000000 0x1000>;
zephyr,memory-region = "NOCACHE_REGION";
zephyr,memory-attr = <( DT_ARM_MPU(ATTR_MPU_RAM_NOCACHE) )>;
};
See `include/zephyr/dt-bindings/memory-attr/memory-attr-mpu.h` to see
how an architecture can define its own special memory attributes (in
this case ARM MPU).
The property can also be used to set custom software-specific
attributes. For example we can think of marking a memory region as
available to be used for memory allocation (not yet implemented):
mem: memory@10000000 {
compatible = "mmio-sram";
reg = <0x10000000 0x1000>;
zephyr,memory-attr = <( DT_MEM_NON_CACHEABLE |
DT_MEM_SW_ALLOCATABLE )>;
};
Or maybe we can leverage the property to specify some alignment
requirements for the region:
mem: memory@10000000 {
compatible = "mmio-sram";
reg = <0x10000000 0x1000>;
zephyr,memory-attr = <( DT_MEM_CACHEABLE |
DT_MEM_SW_ALIGN(32) )>;
};
The conventional and recommended way to deal and manage with memory
regions marked with attributes is by using the provided `mem-attr`
helper library by enabling `CONFIG_MEM_ATTR` (or by using the usual DT
helpers).
When this option is enabled the list of memory regions and their
attributes are compiled in a user-accessible array and a set of
functions is made available that can be used to query, probe and act on
regions and attributes, see `include/zephyr/mem_mgmt/mem_attr.h`
Note that the `zephyr,memory-attr` property is only a descriptive
property of the capabilities of the associated memory region, but it
does not result in any actual setting for the memory to be set. The
user, code or subsystem willing to use this information to do some work
(for example creating an MPU region out of the property) must use either
the provided `mem-attr` library or the usual DeviceTree helpers to
perform the required work / setting.
Signed-off-by: Carlo Caione <ccaione@baylibre.com>
Add Xen domctl API implementation for Zephyr as control domain.
Previously Zephyr OS was used as unprivileged Xen domain (Domain-U),
but it also can be used as lightweight Xen control domain (Domain-0).
To implement such fuctionality additional Xen interfaces are needed.
One of them is Xen domain controls (domctls) - it allows to create,
configure and manage Xen domains.
Also, used it as a possibility to update files copyright and licenses
identifiers in touched files.
Signed-off-by: Dmytro Firsov <dmytro_firsov@epam.com>
Xen-related Kconfig options were highly dependand on BOARD/SOC xenvm.
It is not correct because Xen support may be used on any board and SoC.
So, Kconfig structure was refactored, now CONFIG_XEN is located in
arch/ directory (same as in Linux kernel) and can be selected for
any Cortex-A arm64 setup (no other platforms are currently supported).
Also remove confusion in Domain 0 naming: Domain-0, initial domain,
Dom0, privileged domain etc. Now all options related to Xen Domain 0
will be controlled by CONFIG_XEN_DOM0.
Signed-off-by: Dmytro Firsov <dmytro_firsov@epam.com>
This header is private and included only in architecture code, no need for
it to be in the top of the public include directory.
Note: This might move to a more private location later. For now just
cleaning up the obvious issues.
Signed-off-by: Anas Nashif <anas.nashif@intel.com>
