Commit graph

3814 commits

Author SHA1 Message Date
Wayne Ren
05a669c568 arch: arc: fix the bug where regular irq comes out in user mode
if USERSPACE is configured, it needs to record the user/kernel mode
of interrupted thread, because the switch of aux_sec_k_sp/aux_user_sp
depends on the aux_irq_act's U bit.

Signed-off-by: Wayne Ren <wei.ren@synopsys.com>
2020-02-05 08:49:10 -08:00
Andrei Emeltchenko
9987c2e2f9 arch: xtensa: Use reset-vector.S in booloader code
Use BOOTLOADER definition to separate bootloader code. This allows to
use the same file reset-vector.S when building bootloader and when
CONFIG_XTENSA_RESET_VECTOR is enabled.

Signed-off-by: Andrei Emeltchenko <andrei.emeltchenko@intel.com>
2020-02-05 10:43:25 -05:00
Johan Hedberg
8183a7fd29 arch: xtensa: Add support for Intel Apollolake
Add the necessary architecture changes for Intel Apollolake.

Signed-off-by: Johan Hedberg <johan.hedberg@intel.com>
2020-02-05 10:43:25 -05:00
Andy Ross
5b85d6da6a arch/x86_64: Poison instruction pointer of running threads
There was a bug where double-dispatch of a single thread on multiple
SMP CPUs was possible.  This can be mind-bending to diagnose, so when
CONFIG_ASSERT is enabled add an extra instruction to __resume (the
shared code path for both interupt return and context switch) that
poisons the shared RIP of the now-running thread with a recognizable
invalid value.

Now attempts to run the thread again will crash instantly with a
discoverable cookie in their instruction pointer, and this will remain
true until it gets a new RIP at the next interrupt or switch.

This is under CONFIG_ASSERT because it meets the same design goals of
"a cheap test for impossible situations", not because it's part of the
assertion framework.

Signed-off-by: Andy Ross <andrew.j.ross@intel.com>
2020-02-03 09:31:56 -05:00
Carlo Caione
0958673ee1 arch: arm64: Enable shared IRQ line for UART
Enable the shared IRQ for the UART line and enable the remaining tasks
that depends on a separated declaration of the TX/RX/Err/... IRQs.

Signed-off-by: Carlo Caione <ccaione@baylibre.com>
2020-02-01 08:08:43 -05:00
Carlo Caione
3cdead707d arch: arm64: Fix cmsis_rtos tests
The cmsis_rtos tests are failing because the stack size used by CMSIS is
too small. Customize the stack size for the aarch64 architecture and
re-enable the tests.

Signed-off-by: Carlo Caione <ccaione@baylibre.com>
2020-02-01 08:08:43 -05:00
Carlo Caione
3aef85458d arch: arm64: Dump registers content on fatal error
Extend the ESF structure and dump the most important registers in the
error exception handler.

Signed-off-by: Carlo Caione <ccaione@baylibre.com>
2020-02-01 08:08:43 -05:00
Carlo Caione
6531d1c649 arch: arm64: Enable config option to switch from EL3 to EL1
ARMv8-A SoCs enter EL3 after reset. Add a new config option
(CONFIG_SWITCH_TO_EL1) to switch from EL3 to EL1 at boot and default it
to 'y'.

Signed-off-by: Carlo Caione <ccaione@baylibre.com>
2020-02-01 08:08:43 -05:00
Carlo Caione
0d5e011be4 arch: arm64: Enable tracing
Enable the tracing hooks on the ARM64 platform and white-list the
tracing tests.

Signed-off-by: Carlo Caione <ccaione@baylibre.com>
2020-02-01 08:08:43 -05:00
Carlo Caione
528319bff7 arch: arm64: Support all the ELn
While QEMU's Cortex-A53 emulation by default only emulates a CPU in EL1,
other QEMU forks (for example the QEMU released by Xilinx) and real
hardware starts in EL3.

To support all the ELn we introduce a macro to identify at run-time the
Exception Level and take the correct actions.

Signed-off-by: Carlo Caione <ccaione@baylibre.com>
2020-02-01 08:08:43 -05:00
Carlo Caione
868264b8b4 tests: benchmarks: Add ARM64 case
To be able to pass the unit test we need to add a set of defines for the
ARM64 architecture. Fix this.

Signed-off-by: Carlo Caione <ccaione@baylibre.com>
2020-02-01 08:08:43 -05:00
Carlo Caione
87d8a035dd arch: arm64: Support aarch64-gcc compiler
To be able to successfully compile the kernel for the ARM64 architecture
we have to tweak the compiler-related files to be able to use the
AArch64 GCC compiler.

Signed-off-by: Carlo Caione <ccaione@baylibre.com>
2020-02-01 08:08:43 -05:00
Carlo Caione
1be0c05311 arch: arm64: Introduce ARM64 (AArch64) architecture
Introduce the basic ARM64 architecture support.

A new CONFIG_ARM64 symbol is introduced for the new architecture and new
cmake / Kconfig files are added to switch between ARM and ARM64.

Signed-off-by: Carlo Caione <ccaione@baylibre.com>
2020-02-01 08:08:43 -05:00
Ioannis Glaropoulos
8345434a24 arch: arm: mpu: no dynamic MPU re-program in swap if not required
Dynamic MPU regions are used in build configurations with User
mode or MPU-based stack-overflow guards. If these features are
disabled, we skip calling the ARM function for re-programming
the MPU peripheral during context-switch. We also skip doing
this when jumping to main thread (although this brings limited
performace gain as it is called once in the boot cycle)

Signed-off-by: Ioannis Glaropoulos <Ioannis.Glaropoulos@nordicsemi.no>
2020-01-31 21:38:35 +01:00
Øyvind Rønningstad
350f184d98 arch: common: Delete isr_tables.ld which was a copy of intlist.ld
Refer directly to intlist.ld in zephyr_linker_sources()

Signed-off-by: Øyvind Rønningstad <oyvind.ronningstad@nordicsemi.no>
2020-01-30 14:19:14 -05:00
Sebastian Bøe
7d3fa1c2c8 gen_isr_tables: Improve error message
Users are reportedly not able to understand how to debug the following
error message from gen_isr_tables:

gen_isr_tables.py: multiple registrations at table_index 8 for irq 8

Debugging issues these kinds of issues is difficult so we need to give
users as much information as possible.

To make it clearer that it could be an abuse of the 'IRQ_CONNECT' API
that is causing the issue we add this to the error message.

Signed-off-by: Sebastian Bøe <sebastian.boe@nordicsemi.no>
2020-01-29 14:21:00 -08:00
Ulf Magnusson
cf89ba33ea global: Fix up leading/trailing blank lines in files
To make the updated test in
https://github.com/zephyrproject-rtos/ci-tools/pull/121 clean, though it
only checks modified files.

Signed-off-by: Ulf Magnusson <Ulf.Magnusson@nordicsemi.no>
2020-01-27 17:41:55 -06:00
Øyvind Rønningstad
05f0d85b6a extensions.cmake: Replace TEXT_START with ROM_START
In zephyr_linker_sources().
This is done since the point of the location is to place things at given
offsets. This can only be done consistenly if the linker code is placed
into the _first_ section.

All uses of TEXT_START are replaced with ROM_START.

ROM_START is only supported in some arches, as some arches have several
custom sections before text. These don't currently have ROM_START or
TEXT_START available, but that could be added with a bit of refactoring
in their linker script.

No SORT_KEYs are changed.

This also fixes an error introduced when TEXT_START was added, where
TEXT_SECTION_OFFSET was applied to riscv's common linker.ld instead of
to openisa_rv32m1's specific linker.ld.

Signed-off-by: Øyvind Rønningstad <oyvind.ronningstad@nordicsemi.no>
2020-01-23 03:22:59 -08:00
Ioannis Glaropoulos
5fcbe4e5d5 arch: arm: cortex-m: move openocd sections after _vector_end
openocd linker sections are not supposed to be part of the
vector table sections. Place the sections after we define
the _vector_end linker symbol.

Signed-off-by: Ioannis Glaropoulos <Ioannis.Glaropoulos@nordicsemi.no>
2020-01-23 03:22:59 -08:00
Andy Ross
86430d8d46 kernel: arch: Clarify output switch handle requirements in arch_switch
The original intent was that the output handle be written through the
pointer in the second argument, though not all architectures used that
scheme.  As it turns out, that write is becoming a synchronization
signal, so it's no longer optional.

Clarify the documentation in arch_switch() about this requirement, and
add an instruction to the x86_64 context switch to implement it as
original envisioned.

Signed-off-by: Andy Ross <andrew.j.ross@intel.com>
2020-01-21 14:47:52 -08:00
Anas Nashif
22b95a2047 base: add error checking macros
Define there options for runtime error handling:
- assert on all errors (ASSERT_ON_ERRORS)
- no runtime checks (no asserts, no runtime error handling)
  (NO_RUNTIME_CHECKS)
- full runtime error handling (the default) (RUNTIME_ERROR_CHECKS)

Signed-off-by: Anas Nashif <anas.nashif@intel.com>
2020-01-20 17:19:54 -05:00
Stephanos Ioannidis
c4e6bd2d7c arch: arm: cortex_m: Add CPU DWT feature symbol
This commit adds a Kconfig symbol for specifying whether the SoC
implements the CPU DWT feature.

The Data Watchpoint and Trace (DWT) is an optional debug unit for the
Cortex-M family cores (except ARMv6-M; i.e. M0 and M0+) that provides
watchpoints, data tracing and system profiling capabilities.

Signed-off-by: Stephanos Ioannidis <root@stephanos.io>
2020-01-20 14:05:47 +01:00
Andrew Boie
e34f1cee06 x86: implement kernel page table isolation
Implement a set of per-cpu trampoline stacks which all
interrupts and exceptions will initially land on, and also
as an intermediate stack for privilege changes as we need
some stack space to swap page tables.

Set up the special trampoline page which contains all the
trampoline stacks, TSS, and GDT. This page needs to be
present in the user page tables or interrupts don't work.

CPU exceptions, with KPTI turned on, are treated as interrupts
and not traps so that we have IRQs locked on exception entry.

Add some additional macros for defining IDT entries.

Add special handling of locore text/rodata sections when
creating user mode page tables on x86-64.

Restore qemu_x86_64 to use KPTI, and remove restrictions on
enabling user mode on x86-64.

Signed-off-by: Andrew Boie <andrew.p.boie@intel.com>
2020-01-17 16:17:39 -05:00
Ulf Magnusson
4e85006ba4 dts: Rename generated_dts_board*.{h,conf} to devicetree*.{h,conf}
generated_dts_board.h is pretty redundant and confusing as a name. Call
it devicetree.h instead.

dts.h would be another option, but DTS stands for "devicetree source"
and is the source code format, so it's a bit confusing too.

The replacement was done by grepping for 'generated_dts_board' and
'GENERATED_DTS_BOARD'.

Two build diagram and input-output SVG files were updated as well, along
with misc. documentation.

hal_ti, mcuboot, and ci-tools updates are included too, in the west.yml
update.

Signed-off-by: Ulf Magnusson <Ulf.Magnusson@nordicsemi.no>
2020-01-17 17:57:59 +01:00
Andrew Boie
2690c9e550 x86: move some per-cpu initialization to C
No reason we need to stay in assembly domain once we have
GS and a stack set up.

Signed-off-by: Andrew Boie <andrew.p.boie@intel.com>
2020-01-13 16:35:10 -05:00
Andrew Boie
a594ca7c8f kernel: cleanup and formally define CPU start fn
The "key" parameter is legacy, remove it.

Add a typedef for the expected function pointer type.

Signed-off-by: Andrew Boie <andrew.p.boie@intel.com>
2020-01-13 16:35:10 -05:00
Andrew Boie
808cca0efb x86: disable usermode on 64-bit unless no meltdown
KPTI is still work-in-progress on x86_64. Don't allow
user mode to be enabled unless the SOC/board configuration
indicates that the CPU in use is invulnerable to meltdown
attacks.

Signed-off-by: Andrew Boie <andrew.p.boie@intel.com>
2020-01-13 16:35:10 -05:00
Andrew Boie
4fcf28ef25 x86: mitigate swapgs Spectre V1 attacks
See CVE-2019-1125. We mitigate this by adding an 'lfence'
upon interrupt/exception entry after the decision has been
made whether it's necessary to invoke 'swapgs' or not.

Only applies to x86_64, 32-bit doesn't use swapgs.

Signed-off-by: Andrew Boie <andrew.p.boie@intel.com>
2020-01-13 16:35:10 -05:00
Andrew Boie
3d80208025 x86: implement user mode on 64-bit
- In early boot, enable the syscall instruction and set up
  necessary MSRs
- Add a hook to update page tables on context switch
- Properly initialize thread based on whether it will
  start in user or supervisor mode
- Add landing function for system calls to execute the
  desired handler
- Implement arch_user_string_nlen()
- Implement logic for dropping a thread down to user mode
- Reserve per-CPU storage space for user and privilege
  elevation stack pointers, necessary for handling syscalls
  when no free registers are available
- Proper handling of gs register considerations when
  transitioning privilege levels

Kernel page table isolation (KPTI) is not yet implemented.

Signed-off-by: Andrew Boie <andrew.p.boie@intel.com>
2020-01-13 16:35:10 -05:00
Andrew Boie
07c278382a x86: remove retpoline code
This code:

1) Doesn't work
2) Hasn't ever been enabled by default
3) We mitigate Spectre V2 via Extended IBRS anyway

Signed-off-by: Andrew Boie <andrew.p.boie@intel.com>
2020-01-13 16:35:10 -05:00
Andrew Boie
077b587447 x86: implement hw-based oops for both variants
We use a fixed value of 32 as the way interrupts/exceptions
are setup in x86_64's locore.S do not lend themselves to
Kconfig configuration of the vector to use.

HW-based kernel oops is now permanently on, there's no reason
to make it optional that I can see.

Default vectors for IPI and irq offload adjusted to not
collide.

Signed-off-by: Andrew Boie <andrew.p.boie@intel.com>
2020-01-13 16:35:10 -05:00
Andrew Boie
708d5f7922 x86: don't use privilege stack areas as a guard
This is causing problems, as if we create a thread in
a system call we will *not* be using the kernel page
tables if CONFIG_KPTI=n.

Just don't fiddle with this page's permissions; we don't
need it as a guard area anyway since we have a stack
guard placed immediately before it, and this page
is unused if user mode isn't active.

Signed-off-by: Andrew Boie <andrew.p.boie@intel.com>
2020-01-13 16:35:10 -05:00
Andrew Boie
06c4207602 x86: add CONFIG_X86_USERSPACE for Intel64
Hidden config to select dependencies.

Signed-off-by: Andrew Boie <andrew.p.boie@intel.com>
2020-01-13 16:35:10 -05:00
Andrew Boie
edc14e50ad x86: up-level speculative attack mitigations
These are now part of the common Kconfig and we
build spec_ctrl.c for all.

Signed-off-by: Andrew Boie <andrew.p.boie@intel.com>
2020-01-13 16:35:10 -05:00
Andrew Boie
c71e66e2a5 x86: add system call functions for 64-bit
Nothing too fancy here, we try as much as possible to
use the same register layout as the C calling convention.

Signed-off-by: Andrew Boie <andrew.p.boie@intel.com>
2020-01-13 16:35:10 -05:00
Andrew Boie
7f82b99ad4 x86: up-level some user mode functions
These are now common code, all are related to user mode
threads. The rat's nest of ifdefs in ia32's arch_new_thread
has been greatly simplified, there is now just one hook
if user mode is turned on.

Signed-off-by: Andrew Boie <andrew.p.boie@intel.com>
2020-01-13 16:35:10 -05:00
Andrew Boie
7ea958e0dd x86: optimize locations of psp and thread ptables
z_x86_thread_page_tables_get() now works for both user
and supervisor threads, returning the kernel page tables
in the latter case. This API has been up-leveled to
a common header.

The per-thread privilege elevation stack initial stack
pointer, and the per-thread page table locations are no
longer computed from other values, and instead are stored
in thread->arch.

A problem where the wrong page tables were dumped out
on certain kinds of page faults has been fixed.

Signed-off-by: Andrew Boie <andrew.p.boie@intel.com>
2020-01-13 16:35:10 -05:00
Andrew Boie
e45c6eeebc x86: expose APIs for dumping MMU entry flags
Add two new non-static APIs for dumping out the
page table entries for a specified memory address,
and move to the main MMU code. Has debugging uses
when trying to figure out why memory domains are not
set up correctly.

Signed-off-by: Andrew Boie <andrew.p.boie@intel.com>
2020-01-13 16:35:10 -05:00
Andrew Boie
7c293831c6 x86: add support for 64-bit thread ptables
Slightly different layout since the top-lebel PML4
is page-sized and must be page-aligned.

Signed-off-by: Andrew Boie <andrew.p.boie@intel.com>
2020-01-13 16:35:10 -05:00
Andrew Boie
fc589d7279 x86: implement 64-bit exception recovery logic
The esf has a different set of members on 64-bit.

Signed-off-by: Andrew Boie <andrew.p.boie@intel.com>
2020-01-13 16:35:10 -05:00
Andrew Boie
ded0185eb8 x86: add GDT descriptors for user mode
These are arranged in the particular order required
by the syscall/sysret instructions.

Signed-off-by: Andrew Boie <andrew.p.boie@intel.com>
2020-01-13 16:35:10 -05:00
Andrew Boie
692fda47fc x86: use MSRs for %gs
We don't need to set up GDT data descriptors for setting
%gs. Instead, we use the x86 MSRs to set GS_BASE and
KERNEL_GS_BASE.

We don't currently allow user mode to set %gs on its own,
but later on if we do, we have everything set up to issue
'swapgs' instructions on syscall or IRQ.

Unused entries in the GDT have been removed.

Signed-off-by: Andrew Boie <andrew.p.boie@intel.com>
2020-01-13 16:35:10 -05:00
Andrew Boie
3256e9e00b x86: use BIT() macros for cr0/cr4 bits
Easier to establish correspondence with the technical
manuals.

Signed-off-by: Andrew Boie <andrew.p.boie@intel.com>
2020-01-13 16:35:10 -05:00
Andrew Boie
10d033ebf0 x86: enable recoverable exceptions on 64-bit
These were previously assumed to always be fatal.
We can't have the faulting thread's XMM registers
clobbered, so put the SIMD/FPU state onto the stack
as well. This is fairly large (512 bytes) and the
execption stack is already uncomfortably small, so
increase to 2K.

Signed-off-by: Andrew Boie <andrew.p.boie@intel.com>
2020-01-13 16:35:10 -05:00
Stephanos Ioannidis
bc8524eb82 arch: arm: Rewrite Cortex-R reset vector function.
This commit addresses the following issues:

1. Add a new Kconfig configuration for specifying Dual-redundant Core
   Lock-step (DCLS) processor topology.

2. Register initialisation is only required when Dual-redundant Core
   Lock-step (DCLS) is implemented in hardware. This initialisation is
   required on DCLS only because the architectural registers are in an
   indeterminate state after reset and therefore the initial register
   state of the two parallel executing cores are not guaranteed to be
   identical, which can lead to DCCM detecting it as a hardware fault.
   A conditional compilation check for this hardware configuration
   using the newly added CONFIG_CPU_HAS_DCLS flag has been added.

3. The existing CPU register initialisation code did not take into
   account the banked registers for every execution mode. The new
   implementation ensures that all architectural registers of every
   mode are initialised.

4. Add VFP register initialisation for when floating-point support is
   enabled and the core is configured in DCLS topology. This
   initialisation sequence is required for the same reason given in
   the first issue.

5. Add provision for platform-specific initialisation on Cortex-R
   using PLATFORM_SPECIFIC_INIT config and z_platform_init function.

6. Remove seemingly pointless and inadequately defined STACK_MARGIN.
   Not only does it violate the 8-byte stack alignment rule, it does
   not provide any form of real stack protection.

Signed-off-by: Stephanos Ioannidis <root@stephanos.io>
2020-01-10 10:34:17 +01:00
Anas Nashif
26cecf74c4 arc: remove old macro used for event logger
Leftover macro from the kernel event logger which was replaced by
generic tracing infrastructure.

Signed-off-by: Anas Nashif <anas.nashif@intel.com>
2020-01-09 11:21:19 -05:00
Daniel Leung
017a8eea4f xtensa: fix atomic_cas reporting value swapped even when not
The atomic_cas function was using incorrect register when determining
whether value was swapped. The swapping instruction s32c1i in
atomic_cas stores the value at memory location in register a4
regardless of whether swapping is done. In this case, the register a4
should be used to determine whether a swap is done. However, register
a3 (containing the oldValue as function argument) is used instead.
Since register a5 contains the old value at address loaded before
the swapping instruction, a3 and a5 contain the same value.
Since a3 == a5 is always true in this case, the function will always
return 1 even though values are not swapped. So fix it by using
the correct register.

Also, in case the value is not swapped, it jumps to where it returns
zero instead of loading from memory and comparing again.
The function was simply looping until swapping was done, which did not
align with the API where it would return 0 when swapping is not done
(regardless whether the memory location contains the old value or not).

Signed-off-by: Daniel Leung <daniel.leung@intel.com>
2020-01-08 19:57:05 -05:00
Ioannis Glaropoulos
a07cb30d18 arch: arm: cortex-m: implement support for dynamic direct interrupts
This commits implements the support for dynamic direct
interrupts for the ARM Cortex-M architecture, and exposes
the support to the user as an ARM-only API.

Signed-off-by: Ioannis Glaropoulos <Ioannis.Glaropoulos@nordicsemi.no>
2020-01-08 10:15:09 -08:00
Ioannis Glaropoulos
33048680b0 arch: arm: introduce support for dynamic direct interrupts
With this commit we add support for Dynamic Direct interrupts
for the ARM Cortex-M architecture. For that we introduce a new,
user-enabled, Kconfig symbol, DYNAMIC_DIRECT_INTERRUPTS.

Signed-off-by: Ioannis Glaropoulos <Ioannis.Glaropoulos@nordicsemi.no>
2020-01-08 10:15:09 -08:00
Stephanos Ioannidis
916a8eb3a5 arch: arm: Fix Cortex-R power management interrupt handling
The system power management handling code in the '_isr_wrapper' enables
interrupts by executing the 'cpsie i' instruction, which causes a
system crash on the Cortex-R devices because the Cortex-R arch port
does not support nested interrupts at this time.

This commit restricts the interrupt state manipulations in the system
power management code to the Cortex-M arch, in order to prevent
interrupt nesting on other AArch32 family archs (only Cortex-R for
now).

Signed-off-by: Stephanos Ioannidis <root@stephanos.io>
2020-01-08 15:20:40 +01:00