Commit graph

2393 commits

Author SHA1 Message Date
Ioannis Glaropoulos
73288490f6 arm: cortex_m: log EXC_RETURN value in fatal.c
If CONFIG_EXTRA_EXCEPTION_INFO is enabled, log
the value of EXC_RETURN in the fault handler.

Signed-off-by: Ioannis Glaropoulos <Ioannis.Glaropoulos@nordicsemi.no>
2021-02-02 17:58:58 -05:00
Ioannis Glaropoulos
cafe04558c arm: cortex_m: make lazy FP stacking enabling dynamic
Under FPU sharing mode, any thread is allowed to generate
a Floating Point context (use FP registers in FP instructions),
regardless of whether threads are pre-tagged with K_FP_REGS
option when they are created.

When building with MPU stack guard feature enabled,
a large MPU stack guard is required to catch stack
overflows, if lazy FP stacking is enabled. When lazy
FP stacking is not enabled, a default 32 byte guard is
sufficient.

If lazy stacking is enabled by default, all threads may
potentially generate FP context, so they would need to
program a large MPU guard, carved out of their reserved
stack memory.

To avoid this memory waste, we modify the behavior, and make
lazy stacking a dynamically enabled feature, implemented as
follows:
- threads that are not pre-tagged with K_FP_REGS, and have
not generated an FP context use a default MPU guard and disable
lazy stacking. As long as the threads do not have an active FP
context, they won't stack FP registers, anyway, on ISRs and
exceptions, while they will benefit from reserving a small
MPU guard size
- as soon as a thread starts using FP registers, ISR might
temporarily experience some increased ISR latency due to lazy
stacking being disabled. This will be the case until the next
context switch, where the threads that have active FP context
will be tagged with K_FP_REGS, enable lazy stacking, and
program a wide MPU guard.

The implementation is a tradeoff between performance (ISR
latency) and memory consumption.

Note that when MPU STACK GUARD feature is not enabled, lazy
FP stacking is always activated.

Signed-off-by: Ioannis Glaropoulos <Ioannis.Glaropoulos@nordicsemi.no>
2021-02-02 17:58:58 -05:00
Ioannis Glaropoulos
2642eb28bf arm: cortex_m: force FP context stacking by default
For the standard multi-theading builds, we will
enforce FP context stacking only when FPU_SHARING
is set. For the single-threading use case we enable
context stacking by default.

Signed-off-by: Ioannis Glaropoulos <Ioannis.Glaropoulos@nordicsemi.no>
2021-02-02 17:58:58 -05:00
Ioannis Glaropoulos
56dd787627 arm: cortex_m: skip clearing CONTROL if this is done at boot
If CONTROL register is done in reset.S we can skip
clearing the FPCA when enabling the floating point
support, to save a few instructions. The CONTROL
register is cleared right after boot, if the symbol
CONFIG_INIT_ARCH_HW_AT_BOOT is enabled.

Signed-off-by: Ioannis Glaropoulos <Ioannis.Glaropoulos@nordicsemi.no>
2021-02-02 17:58:58 -05:00
Nicolas Pitre
7fcf5519d0 aarch64: mmu: cleanups and fixes
Major changes:

- move related functions together
- optimize add_map() not to walk the page tables *twice* on
  every loop
- properly handle leftover size when a range is already mapped
- don't overwrite existing mappings by default
- return an error when the mapping fails

and make the code clearer overall.

Signed-off-by: Nicolas Pitre <npitre@baylibre.com>
2021-01-28 20:24:30 -05:00
Daniel Leung
0d099bdd54 linker: remove asterisk from IRQ/ISR section name macro
Both _IRQ_VECTOR_TABLE_SECTION_NAME and _SW_ISR_TABLE_SECTION_NAME
are defined with asterisk at the end in an attempt to include
all related symbols in the linker script. However, these two
macros are also being used in the source code to specify
the destination sections for variables. Asterisks in the name
results in older GCC (4.x) complaining about those asterisks.
So create new macros for use in linker script, and keep
the names asterisk free.

Fixes #29936

Signed-off-by: Daniel Leung <daniel.leung@intel.com>
2021-01-26 16:24:11 -05:00
Stephanos Ioannidis
f769a03081 arch: arm: aarch32: Fix interrupt nesting
In the current interrupt nesting implementation, if an ISR is
interrupted while executing inside a branch, the lr_svc register will
be corrupted, and the branch of the interrupted ISR will exit to the
return address of the final branch of the interrupting ISR, which may
or may not correspond to the intended return address.

This commit fixes the aforementioned bug by storing the lr_svc register
in the stack at the ISR entry, and restoring its value before exiting
the ISR.

For more details, refer to the issue #30517.

Signed-off-by: Stephanos Ioannidis <root@stephanos.io>
2021-01-26 06:17:15 -05:00
Stephanos Ioannidis
c00169daba arch: arm: aarch32: Fix exception exit failures
This commit fixes the following bugs in the AArch32 z_arm_exc_exit
routine:

1. Invalid return address when calling `z_arm_pendsv` from the
   exception-specific mode

2. Caller-saved register is referenced after a call to `z_arm_pendsv`

For more details, refer to the issue #31511.

Signed-off-by: Stephanos Ioannidis <root@stephanos.io>
2021-01-26 06:17:15 -05:00
Stephanos Ioannidis
d86fdb2154 arch: arm: aarch32: Update stale references to _IntExit
This commit updates the stale references to the `_IntExit` function in
the in-line documentation.

Signed-off-by: Stephanos Ioannidis <root@stephanos.io>
2021-01-26 06:17:15 -05:00
Volodymyr Babchuk
cd86ec2655 aarch64: add ability to generate image header
Image header is compatible with Linux aarch64 boot protocol,
so zephyr can be booted with U-boot or Xen loader.

Signed-off-by: Volodymyr Babchuk <volodymyr_babchuk@epam.com>
2021-01-24 13:59:55 -05:00
Daniel Leung
d3218ca515 debug: coredump: remove z_ prefix for stuff used outside subsys
This removes the z_ prefix those (functions, enums, etc.) that
are being used outside the coredump subsys. This aligns better
with the naming convention.

Signed-off-by: Daniel Leung <daniel.leung@intel.com>
2021-01-21 22:08:59 -05:00
Carlo Caione
57f7e31017 drivers: PSCI: Add driver and subsystem
Firmware implementing the PSCI functions described in ARM document
number ARM DEN 0022A ("Power State Coordination Interface System
Software on ARM processors") can be used by Zephyr to initiate various
CPU-centric power operations.

It is needed for virtualization, it is used to coordinate OSes and
hypervisors and it provides the functions used for SMP bring-up such as
CPU_ON and CPU_OFF.

A new PSCI driver is introduced to setup a proper subsystem used to
communicate with the PSCI firmware, implementing the basic operations:
get_version, cpu_on, cpu_off and affinity_info.

The current implementation only supports PSCI 0.2 and PSCI 1.0

The PSCI conduit (SMC or HVC) is setup reading the corresponding
property in the DTS node.

Signed-off-by: Carlo Caione <ccaione@baylibre.com>
2021-01-18 19:06:53 +01:00
Carlo Caione
c5b898743a aarch64: Fix alignment fault on z_bss_zero()
Using newlibc with AArch64 is causing an alignement fault in
z_bss_zero() when the code is run on real hardware (on QEMU the problem
is not reproducible).

The main cause is that the memset() function exported by newlibc is
using 'DC ZVA' to zero out memory.

While this is often a nice optimization, this is causing the issue on
AArch64 because memset() is being used before the MMU is enabled, and
when the MMU is disabled all data accesses will be treated as
Device_nGnRnE.

This is a problem because quoting from the ARM reference manual: "If the
memory region being zeroed is any type of Device memory, then DC ZVA
generates an Alignment fault which is prioritized in the same way as
other alignment faults that are determined by the memory type".

newlibc tries to be a bit smart about this reading the DCZID_EL0
register before deciding whether using 'DC ZVA' or not. While this is a
good idea for code running in EL0, currently the Zephyr kernel is
running in EL1. This means that the value of the DCZID_EL0 register is
actually retrieved from the HCR_EL2.TDZ bit, that is always 0 because
EL2 is not currently supported / enabled. So the 'DC ZVA' instruction is
unconditionally used in the newlibc memset() implementation.

The "standard" solution for this case is usually to use a different
memset routine to be specifically used for two cases: (1) against IO
memory or (2) against normal memory but with MMU disabled (which means
all memory is considered device memory for data accesses).

To fix this issue in Zephyr we avoid calling memset() when clearing the
bss, and instead we use a simple loop to zero the memory region.

Signed-off-by: Carlo Caione <ccaione@baylibre.com>
2021-01-14 13:37:47 -08:00
Ioannis Glaropoulos
c6c14724ba arch: arm: cortex_m: fix stack overflow error detection
In rare cases when a thread may overflow its stack, the
core will not report a Stacking Error. This is the case
when a large stack array is created, making the PSP cross
beyond the stack guard; in this case a MemManage fault
won't cause a stacking error (but only a Data Access
Violation error). We fix the fault handling logic so
such errors are reported as stack overflows and not as
generic CPU exceptions.

Signed-off-by: Ioannis Glaropoulos <Ioannis.Glaropoulos@nordicsemi.no>
2021-01-14 12:35:47 +01:00
Ioannis Glaropoulos
202c2fde54 arch: arm: cortex_m: do not read MMFAR if MMARVALID is not set
When the MMARVALID bit is not set, do not read the MMFAR
register to get the fault address in a MemManage fault.
This change prevents the fault handler to erroneously
assume MMFAR contains a valid address.

Signed-off-by: Ioannis Glaropoulos <Ioannis.Glaropoulos@nordicsemi.no>
2021-01-14 12:35:47 +01:00
Julien Massot
d3345dd54d arch: arm: Add Cortex-R7 support
Pass the correct -mcpu flags to the compiler when building for the
Cortex-R7.

Signed-off-by: Julien Massot <julien.massot@iot.bzh>
2021-01-13 15:04:43 +01:00
Carlo Caione
e710d36f77 aarch64: mmu: Enable CONFIG_MMU
Enable CONFIG_MMU for AArch64 and add the new arch_mem_map() required
function.

Signed-off-by: Carlo Caione <ccaione@baylibre.com>
2021-01-12 06:51:09 -05:00
Carlo Caione
6a3401d6be aarch64: mmu: Fix variable types
Before hooking up the MMU driver code to the Zephyr MMU core code it's
better to match the expected variable types of the two parts.

Signed-off-by: Carlo Caione <ccaione@baylibre.com>
2021-01-12 06:51:09 -05:00
Carlo Caione
0a0061d901 aarch64: mmu: Do not assume a single set of pagetables is used
The MMU code is currently assuming that Zephyr only uses one single set
of page tables shared by kernel and user threads. This could possibly be
not longer true in the future when multiple set of page tables can be
present and swapped at run-time.

With this patch a new arm_mmu_ptables struct is introduced that is used
to host a buffer pointing to the memory region containing the page
tables and the helper variables used to manage the page tables. This new
struct is then used by the ARM64 MMU code instead of assuming that the
kernel page tables are the only ones present.

Signed-off-by: Carlo Caione <ccaione@baylibre.com>
2021-01-12 06:51:09 -05:00
Carlo Caione
1a4a73da97 aarch64: mmu: Makes memory mapping functions more generic
The ARM64 MMU code used to create the page tables is strictly tied to
the custom arm_mmu_region struct. To be able to hook up this code to the
Zephyr MMU APIs we need to make it more generic.

This patch makes the mapping function more generic and creates a new
helper function add_arm_mmu_region() to map the regions defined by the
old arm_mmu_region structs using this new generic function.

Signed-off-by: Carlo Caione <ccaione@baylibre.com>
2021-01-12 06:51:09 -05:00
Carlo Caione
2581009c3e aarch64: mmu: Move xlat tables to one single array
In the current code the base xlat table is a standalone array. This is
done because we know at compile time the size of this table so we can
allocate the correct size and save a bit of memory. All the other xlat
tables are statically allocated in a different array with full size.

With this patch we move all the page tables in one single array,
including the base table. This is probably going to waste a bit of space
but it makes easier to:

- have all the page tables mapped in one single contiguous memory region
  instead of having to take care of two different arrays in two
  different locations
- duplicate the page tables more quickly if we need to
- use a pre-allocated space to host the page tables
- use a pre-computed set of page tables saved in a contiguous memory
  region

Signed-off-by: Carlo Caione <ccaione@baylibre.com>
2021-01-12 06:51:09 -05:00
Carlo Caione
1d68c48786 aarch64: mmu: Fix typo in mask definition
Fix typo in mask definition (s/UPPER/LOWER)

Signed-off-by: Carlo Caione <ccaione@baylibre.com>
2021-01-06 08:18:27 -06:00
Carlo Caione
181088600e aarch64: mmu: Avoid creating a new table when not needed
In the current MMU code a new table is created when mapping a memory
region that is overlapping with a block already mapped. The problem is
that the new table is created also when the new and old mappings have
the same attributes.

To avoid using a new table when not needed the attributes of the two
mappings are compared before creating the new table.

Signed-off-by: Carlo Caione <ccaione@baylibre.com>
2021-01-04 23:53:04 -08:00
Wojciech Sipak
56c06e852b arch: arm: cortex_r: disable ECC on TCMs
This commit adds possibility to disable ECC in Tightly Coupled
Memory in Cortex-R.
Linker scripts places stacks in this memory and marks it as
.noinit section. With ECC enabled, stack read accesses without
previous write result in Data Abort Exception.

Signed-off-by: Wojciech Sipak <wsipak@antmicro.com>
2020-12-27 18:16:00 +01:00
Peng Fan
cca070c80a arch: arm64: mmu: support using MT_NS attribute
According to CONFIG_ARMV8_A_NS, using MT_SECURE or MT_NS, to simplify
code change, use MT_DEFAULT_SECURE_STATE instead

Signed-off-by: Peng Fan <peng.fan@nxp.com>
2020-12-17 08:08:00 -05:00
Peng Fan
0d0f168460 arm: aarch64: Kconfig: introduce ARMV8_A_NS
Add new Kconfig entry ARMV8_A_NS for zephyr runs in normal world usage.

Signed-off-by: Peng Fan <peng.fan@nxp.com>
2020-12-17 08:08:00 -05:00
Luke Starrett
430952f0b2 arch: arm64: GICv2/v3 handling causes abort on spurious interrupt
In _isr_wrapper, the interrupt ID read from the GIC is blindly used to
index into _sw_isr_table, which is only sized based on CONFIG_NUM_IRQ.

It is possible for both GICv2 and GICv3 to return 1023 for a handful
of scenarios, the simplest of which is a level sensitive interrupt
which has subsequently become de-asserted.  Borrowing from the Linux
GIC implementation, a read that returns an interrupt ID of 1023 is
simply ignored.

Minor collateral changes to gic.h to group !_ASMLANGUAGE content
together to allow this header to be used in assembler files.

Signed-off-by: Luke Starrett <luke.starrett@gmail.com>
2020-12-16 08:46:03 -05:00
Carlo Caione
0d1a6dc27f intc: gic: Use SYS_INIT instead of custom init function
The GIC interrupt controller driver is using a custom init function
called directly from the prep_c function. For consistency move that to
use SYS_INIT.

Signed-off-by: Carlo Caione <ccaione@baylibre.com>
2020-12-11 10:17:27 -05:00
Anas Nashif
c10d4b377d power: move z_pm_save_idle_exit prototype to power.h
Maintain power prototypes in power.h instead of kernel and arch headers.

Signed-off-by: Anas Nashif <anas.nashif@intel.com>
2020-12-09 15:18:29 -05:00
Anas Nashif
e0f3833bf7 power: remove SYS_ and sys_ prefixes
Remove SYS_ and sys_ from all PM related functions and defines.

Signed-off-by: Anas Nashif <anas.nashif@intel.com>
2020-12-09 15:18:29 -05:00
Anas Nashif
dd931f93a2 power: standarize PM Kconfigs and cleanup
- Remove SYS_ prefix
- shorten POWER_MANAGEMENT to just PM
- DEVICE_POWER_MANAGEMENT -> PM_DEVICE

and use PM_ as the prefix for all PM related Kconfigs

Signed-off-by: Anas Nashif <anas.nashif@intel.com>
2020-12-09 15:18:29 -05:00
Carlo Caione
7e36bd31fe arch: aarch64: Use SP_EL0 instead of SP_ELx
ARM64 is currently using SP_ELx as stack pointer for kernel and threads
because everything is running in EL1. If support for EL0 is required, it
is necessary to switch to use SP_EL0 instead, that is the only stack
pointer that can be accessed at all exception levels by threads.

While it is not required to keep using SP_EL0 also during the
exceptions, the current code implementation makes it easier to use the
same stack pointer as the one used by threads also during the
exceptions.

This patch moves the code from using SP_ELx to SP_EL0 and fill in the
missing entries in the vector table.

Signed-off-by: Carlo Caione <ccaione@baylibre.com>
2020-12-04 08:13:42 -05:00
Alexandre Bourdiol
4cf1d4380e arch: arm: aarch32:cortex_m: timing.c: cortex M7 may need DWT unlock
On Cortex M7, we need to check the optional presence of
Lock Access Register (LAR) which is indicated in
Lock Status Register (LSR).
When present, a special access token must be written to unlock DWT
registers.

Signed-off-by: Alexandre Bourdiol <alexandre.bourdiol@st.com>
2020-12-02 10:58:08 +02:00
Peng Fan
346ecb2a5a arch: arm64: correct vector_table alignment
The 2K alignment assembler directives should be under
'SECTION_SUBSEC_FUNC(exc_vector_table,_vector_table_section,_vector_table)'
Otherwise the _vector_table is actually 0x80 bytes aligned.

Signed-off-by: Peng Fan <peng.fan@nxp.com>
2020-12-01 10:04:59 -06:00
Krzysztof Chruscinski
3ed8083dc1 kernel: Cleanup logger setup in kernel files
Most of kernel files where declaring os module without providing
log level. Because of that default log level was used instead of
CONFIG_KERNEL_LOG_LEVEL.

Signed-off-by: Krzysztof Chruscinski <krzysztof.chruscinski@nordicsemi.no>
2020-11-27 09:56:34 -05:00
Carlo Caione
47ebde30b9 aarch64: error: Handle software-generated fatal exceptions
Introduce a new SVC call ID to trigger software-generated CPU
exceptions.

Signed-off-by: Carlo Caione <ccaione@baylibre.com>
2020-11-25 12:02:11 +02:00
Håkon Øye Amundsen
2ce570b03f arch: arm: clear SPLIM registers before z_platform_init
Allow z_platform_init to perform stack operations.

Signed-off-by: Håkon Øye Amundsen <haakon.amundsen@nordicsemi.no>
2020-11-24 20:53:49 +02:00
Andrew Boie
5a58ad508c arch: mem protect Kconfig cleanups
Adds a new CONFIG_MPU which is set if an MPU is enabled. This
is a menuconfig will some MPU-specific options moved
under it.

MEMORY_PROTECTION and SRAM_REGION_PERMISSIONS have been merged.
This configuration depends on an MMU or MPU. The protection
test is updated accordingly.

Signed-off-by: Andrew Boie <andrew.p.boie@intel.com>
2020-11-18 08:02:08 -05:00
Andrew Boie
00cdb597ff arm: de-couple MPU code from k_mem_partition
k_mem_partition is part of the CONFIG_USERSPACE abstraction,
but some older MPU code was depending on it even if user mode
isn't enabled. Use a new structure z_arm_mpu_partition instead,
which will insulate this code from any changes to the core
kernel definition of k_mem_partition.

The logic in z_arm_configure_dynamic_mpu_regions has been
adjusted to copy the necessary information out of the
memory domain instead of passing the addresses of the domain
structures directly to the lower-level MPU code.

Signed-off-by: Andrew Boie <andrew.p.boie@intel.com>
2020-11-18 08:02:08 -05:00
Carlo Caione
f095e2fd05 arch: arm64: mmu: Rework defines
Every time I try to decode all the defines in this driver what I get is
only a huge headache. This patch:

- adds a few sensible comments
- remove the redundant defines
- rename the defines to be more self-explanatory
- reorder the defines
- try to make sense of some mysterious derived values

Signed-off-by: Carlo Caione <ccaione@baylibre.com>
2020-11-17 19:04:25 -05:00
Carlo Caione
96f574c7a4 aarch64: Use macro-generated absolute symbols for the ESF
As done already for other structs, use the macro-generated offsets when
referencing register in the ESF.

Signed-off-by: Carlo Caione <ccaione@baylibre.com>
2020-11-17 18:59:23 -05:00
Carlo Caione
daa94e5e59 aarch64: Remove redundant init_stack_frame
The init_stack_frame is the same as the the ESF. No need to have two
separate structs. Consolidate everything into one single struct and make
register entries explicit.

Signed-off-by: Carlo Caione <ccaione@baylibre.com>
2020-11-17 18:59:23 -05:00
Carlo Caione
a7d94b003e aarch64: Use absolute symbols for the callee saved registers
Use GEN_OFFSET_SYM macro to genarate absolute symbols for the
_callee_saved struct and use these new symbols in the assembly code.

Signed-off-by: Carlo Caione <ccaione@baylibre.com>
2020-11-17 18:59:23 -05:00
Carlo Caione
666974015e aarch64: error: Introduce CONFIG_EXCEPTION_DEBUG
Introduce CONFIG_EXCEPTION_DEBUG to discard exception debug strings and
code when not needed.

Signed-off-by: Carlo Caione <ccaione@baylibre.com>
2020-11-17 18:52:45 -05:00
Carlo Caione
2683a1ed97 aarch64: error: Enable recoverable errors
For some kind of faults we want to be able to put in action some
corrective actions and keep executing the code.

Signed-off-by: Carlo Caione <ccaione@baylibre.com>
2020-11-17 18:52:45 -05:00
Carlo Caione
a054e424e4 aarch64: error: Beautify error printing
Make the printing of errors a bit more descriptive and print the FAR_ELn
register only when strictly required.

Signed-off-by: Carlo Caione <ccaione@baylibre.com>
2020-11-17 18:52:45 -05:00
Carlo Caione
6d05c57781 arch: aarch64: Branch SError vector table entry
Each vector table entry has 128-bytes to host the vector code. This is
not always enough and in general it's better to branch to the actual
exception handler elsewhere in memory.

Move the SError entry to a branched code.

Signed-off-by: Carlo Caione <ccaione@baylibre.com>
2020-11-17 18:52:45 -05:00
Wentong Wu
bfc7785da0 arch: arm: push ssf to thread privileged stack to complete stack frame
Pushes the seventh argument named ssf to thread's privileged
stack to follow below syscall prototype.

uintptr_t z_mrsh_xx(uintptr_t arg0, uintptr_t arg1, uintptr_t arg2,
		    uintptr_t arg3, uintptr_t arg4, void *more, void *ssf)

Fixes: #29386.

Signed-off-by: Wentong Wu <wentong.wu@intel.com>
2020-11-12 17:12:38 -05:00
Daniel Leung
11e6b43090 tracing: roll thread switch in/out into thread stats functions
Since the tracing of thread being switched in/out has the same
instrumentation points, we can roll the tracing function calls
into the one for thread stats gathering functions.
This avoids duplicating code to call another function.

Signed-off-by: Daniel Leung <daniel.leung@intel.com>
2020-11-11 23:55:49 -05:00
Daniel Leung
9be37553ee timing: do not repeatedly do init()/start()/stop()
We should not be initializing/starting/stoping timing functions
multiple times. So this changes how the timing functions are
structured to allow only one initialization, only start when
stopped, and only stop when started.

Signed-off-by: Daniel Leung <daniel.leung@intel.com>
2020-11-11 23:55:49 -05:00