if USERSPACE is configured, it needs to record the user/kernel mode
of interrupted thread, because the switch of aux_sec_k_sp/aux_user_sp
depends on the aux_irq_act's U bit.
Signed-off-by: Wayne Ren <wei.ren@synopsys.com>
Use BOOTLOADER definition to separate bootloader code. This allows to
use the same file reset-vector.S when building bootloader and when
CONFIG_XTENSA_RESET_VECTOR is enabled.
Signed-off-by: Andrei Emeltchenko <andrei.emeltchenko@intel.com>
There was a bug where double-dispatch of a single thread on multiple
SMP CPUs was possible. This can be mind-bending to diagnose, so when
CONFIG_ASSERT is enabled add an extra instruction to __resume (the
shared code path for both interupt return and context switch) that
poisons the shared RIP of the now-running thread with a recognizable
invalid value.
Now attempts to run the thread again will crash instantly with a
discoverable cookie in their instruction pointer, and this will remain
true until it gets a new RIP at the next interrupt or switch.
This is under CONFIG_ASSERT because it meets the same design goals of
"a cheap test for impossible situations", not because it's part of the
assertion framework.
Signed-off-by: Andy Ross <andrew.j.ross@intel.com>
Enable the shared IRQ for the UART line and enable the remaining tasks
that depends on a separated declaration of the TX/RX/Err/... IRQs.
Signed-off-by: Carlo Caione <ccaione@baylibre.com>
The cmsis_rtos tests are failing because the stack size used by CMSIS is
too small. Customize the stack size for the aarch64 architecture and
re-enable the tests.
Signed-off-by: Carlo Caione <ccaione@baylibre.com>
ARMv8-A SoCs enter EL3 after reset. Add a new config option
(CONFIG_SWITCH_TO_EL1) to switch from EL3 to EL1 at boot and default it
to 'y'.
Signed-off-by: Carlo Caione <ccaione@baylibre.com>
While QEMU's Cortex-A53 emulation by default only emulates a CPU in EL1,
other QEMU forks (for example the QEMU released by Xilinx) and real
hardware starts in EL3.
To support all the ELn we introduce a macro to identify at run-time the
Exception Level and take the correct actions.
Signed-off-by: Carlo Caione <ccaione@baylibre.com>
To be able to pass the unit test we need to add a set of defines for the
ARM64 architecture. Fix this.
Signed-off-by: Carlo Caione <ccaione@baylibre.com>
To be able to successfully compile the kernel for the ARM64 architecture
we have to tweak the compiler-related files to be able to use the
AArch64 GCC compiler.
Signed-off-by: Carlo Caione <ccaione@baylibre.com>
Introduce the basic ARM64 architecture support.
A new CONFIG_ARM64 symbol is introduced for the new architecture and new
cmake / Kconfig files are added to switch between ARM and ARM64.
Signed-off-by: Carlo Caione <ccaione@baylibre.com>
Dynamic MPU regions are used in build configurations with User
mode or MPU-based stack-overflow guards. If these features are
disabled, we skip calling the ARM function for re-programming
the MPU peripheral during context-switch. We also skip doing
this when jumping to main thread (although this brings limited
performace gain as it is called once in the boot cycle)
Signed-off-by: Ioannis Glaropoulos <Ioannis.Glaropoulos@nordicsemi.no>
Users are reportedly not able to understand how to debug the following
error message from gen_isr_tables:
gen_isr_tables.py: multiple registrations at table_index 8 for irq 8
Debugging issues these kinds of issues is difficult so we need to give
users as much information as possible.
To make it clearer that it could be an abuse of the 'IRQ_CONNECT' API
that is causing the issue we add this to the error message.
Signed-off-by: Sebastian Bøe <sebastian.boe@nordicsemi.no>
In zephyr_linker_sources().
This is done since the point of the location is to place things at given
offsets. This can only be done consistenly if the linker code is placed
into the _first_ section.
All uses of TEXT_START are replaced with ROM_START.
ROM_START is only supported in some arches, as some arches have several
custom sections before text. These don't currently have ROM_START or
TEXT_START available, but that could be added with a bit of refactoring
in their linker script.
No SORT_KEYs are changed.
This also fixes an error introduced when TEXT_START was added, where
TEXT_SECTION_OFFSET was applied to riscv's common linker.ld instead of
to openisa_rv32m1's specific linker.ld.
Signed-off-by: Øyvind Rønningstad <oyvind.ronningstad@nordicsemi.no>
openocd linker sections are not supposed to be part of the
vector table sections. Place the sections after we define
the _vector_end linker symbol.
Signed-off-by: Ioannis Glaropoulos <Ioannis.Glaropoulos@nordicsemi.no>
The original intent was that the output handle be written through the
pointer in the second argument, though not all architectures used that
scheme. As it turns out, that write is becoming a synchronization
signal, so it's no longer optional.
Clarify the documentation in arch_switch() about this requirement, and
add an instruction to the x86_64 context switch to implement it as
original envisioned.
Signed-off-by: Andy Ross <andrew.j.ross@intel.com>
Define there options for runtime error handling:
- assert on all errors (ASSERT_ON_ERRORS)
- no runtime checks (no asserts, no runtime error handling)
(NO_RUNTIME_CHECKS)
- full runtime error handling (the default) (RUNTIME_ERROR_CHECKS)
Signed-off-by: Anas Nashif <anas.nashif@intel.com>
This commit adds a Kconfig symbol for specifying whether the SoC
implements the CPU DWT feature.
The Data Watchpoint and Trace (DWT) is an optional debug unit for the
Cortex-M family cores (except ARMv6-M; i.e. M0 and M0+) that provides
watchpoints, data tracing and system profiling capabilities.
Signed-off-by: Stephanos Ioannidis <root@stephanos.io>
Implement a set of per-cpu trampoline stacks which all
interrupts and exceptions will initially land on, and also
as an intermediate stack for privilege changes as we need
some stack space to swap page tables.
Set up the special trampoline page which contains all the
trampoline stacks, TSS, and GDT. This page needs to be
present in the user page tables or interrupts don't work.
CPU exceptions, with KPTI turned on, are treated as interrupts
and not traps so that we have IRQs locked on exception entry.
Add some additional macros for defining IDT entries.
Add special handling of locore text/rodata sections when
creating user mode page tables on x86-64.
Restore qemu_x86_64 to use KPTI, and remove restrictions on
enabling user mode on x86-64.
Signed-off-by: Andrew Boie <andrew.p.boie@intel.com>
generated_dts_board.h is pretty redundant and confusing as a name. Call
it devicetree.h instead.
dts.h would be another option, but DTS stands for "devicetree source"
and is the source code format, so it's a bit confusing too.
The replacement was done by grepping for 'generated_dts_board' and
'GENERATED_DTS_BOARD'.
Two build diagram and input-output SVG files were updated as well, along
with misc. documentation.
hal_ti, mcuboot, and ci-tools updates are included too, in the west.yml
update.
Signed-off-by: Ulf Magnusson <Ulf.Magnusson@nordicsemi.no>
KPTI is still work-in-progress on x86_64. Don't allow
user mode to be enabled unless the SOC/board configuration
indicates that the CPU in use is invulnerable to meltdown
attacks.
Signed-off-by: Andrew Boie <andrew.p.boie@intel.com>
See CVE-2019-1125. We mitigate this by adding an 'lfence'
upon interrupt/exception entry after the decision has been
made whether it's necessary to invoke 'swapgs' or not.
Only applies to x86_64, 32-bit doesn't use swapgs.
Signed-off-by: Andrew Boie <andrew.p.boie@intel.com>
- In early boot, enable the syscall instruction and set up
necessary MSRs
- Add a hook to update page tables on context switch
- Properly initialize thread based on whether it will
start in user or supervisor mode
- Add landing function for system calls to execute the
desired handler
- Implement arch_user_string_nlen()
- Implement logic for dropping a thread down to user mode
- Reserve per-CPU storage space for user and privilege
elevation stack pointers, necessary for handling syscalls
when no free registers are available
- Proper handling of gs register considerations when
transitioning privilege levels
Kernel page table isolation (KPTI) is not yet implemented.
Signed-off-by: Andrew Boie <andrew.p.boie@intel.com>
This code:
1) Doesn't work
2) Hasn't ever been enabled by default
3) We mitigate Spectre V2 via Extended IBRS anyway
Signed-off-by: Andrew Boie <andrew.p.boie@intel.com>
We use a fixed value of 32 as the way interrupts/exceptions
are setup in x86_64's locore.S do not lend themselves to
Kconfig configuration of the vector to use.
HW-based kernel oops is now permanently on, there's no reason
to make it optional that I can see.
Default vectors for IPI and irq offload adjusted to not
collide.
Signed-off-by: Andrew Boie <andrew.p.boie@intel.com>
This is causing problems, as if we create a thread in
a system call we will *not* be using the kernel page
tables if CONFIG_KPTI=n.
Just don't fiddle with this page's permissions; we don't
need it as a guard area anyway since we have a stack
guard placed immediately before it, and this page
is unused if user mode isn't active.
Signed-off-by: Andrew Boie <andrew.p.boie@intel.com>
Nothing too fancy here, we try as much as possible to
use the same register layout as the C calling convention.
Signed-off-by: Andrew Boie <andrew.p.boie@intel.com>
These are now common code, all are related to user mode
threads. The rat's nest of ifdefs in ia32's arch_new_thread
has been greatly simplified, there is now just one hook
if user mode is turned on.
Signed-off-by: Andrew Boie <andrew.p.boie@intel.com>
z_x86_thread_page_tables_get() now works for both user
and supervisor threads, returning the kernel page tables
in the latter case. This API has been up-leveled to
a common header.
The per-thread privilege elevation stack initial stack
pointer, and the per-thread page table locations are no
longer computed from other values, and instead are stored
in thread->arch.
A problem where the wrong page tables were dumped out
on certain kinds of page faults has been fixed.
Signed-off-by: Andrew Boie <andrew.p.boie@intel.com>
Add two new non-static APIs for dumping out the
page table entries for a specified memory address,
and move to the main MMU code. Has debugging uses
when trying to figure out why memory domains are not
set up correctly.
Signed-off-by: Andrew Boie <andrew.p.boie@intel.com>
We don't need to set up GDT data descriptors for setting
%gs. Instead, we use the x86 MSRs to set GS_BASE and
KERNEL_GS_BASE.
We don't currently allow user mode to set %gs on its own,
but later on if we do, we have everything set up to issue
'swapgs' instructions on syscall or IRQ.
Unused entries in the GDT have been removed.
Signed-off-by: Andrew Boie <andrew.p.boie@intel.com>
These were previously assumed to always be fatal.
We can't have the faulting thread's XMM registers
clobbered, so put the SIMD/FPU state onto the stack
as well. This is fairly large (512 bytes) and the
execption stack is already uncomfortably small, so
increase to 2K.
Signed-off-by: Andrew Boie <andrew.p.boie@intel.com>
This commit addresses the following issues:
1. Add a new Kconfig configuration for specifying Dual-redundant Core
Lock-step (DCLS) processor topology.
2. Register initialisation is only required when Dual-redundant Core
Lock-step (DCLS) is implemented in hardware. This initialisation is
required on DCLS only because the architectural registers are in an
indeterminate state after reset and therefore the initial register
state of the two parallel executing cores are not guaranteed to be
identical, which can lead to DCCM detecting it as a hardware fault.
A conditional compilation check for this hardware configuration
using the newly added CONFIG_CPU_HAS_DCLS flag has been added.
3. The existing CPU register initialisation code did not take into
account the banked registers for every execution mode. The new
implementation ensures that all architectural registers of every
mode are initialised.
4. Add VFP register initialisation for when floating-point support is
enabled and the core is configured in DCLS topology. This
initialisation sequence is required for the same reason given in
the first issue.
5. Add provision for platform-specific initialisation on Cortex-R
using PLATFORM_SPECIFIC_INIT config and z_platform_init function.
6. Remove seemingly pointless and inadequately defined STACK_MARGIN.
Not only does it violate the 8-byte stack alignment rule, it does
not provide any form of real stack protection.
Signed-off-by: Stephanos Ioannidis <root@stephanos.io>
The atomic_cas function was using incorrect register when determining
whether value was swapped. The swapping instruction s32c1i in
atomic_cas stores the value at memory location in register a4
regardless of whether swapping is done. In this case, the register a4
should be used to determine whether a swap is done. However, register
a3 (containing the oldValue as function argument) is used instead.
Since register a5 contains the old value at address loaded before
the swapping instruction, a3 and a5 contain the same value.
Since a3 == a5 is always true in this case, the function will always
return 1 even though values are not swapped. So fix it by using
the correct register.
Also, in case the value is not swapped, it jumps to where it returns
zero instead of loading from memory and comparing again.
The function was simply looping until swapping was done, which did not
align with the API where it would return 0 when swapping is not done
(regardless whether the memory location contains the old value or not).
Signed-off-by: Daniel Leung <daniel.leung@intel.com>
This commits implements the support for dynamic direct
interrupts for the ARM Cortex-M architecture, and exposes
the support to the user as an ARM-only API.
Signed-off-by: Ioannis Glaropoulos <Ioannis.Glaropoulos@nordicsemi.no>
With this commit we add support for Dynamic Direct interrupts
for the ARM Cortex-M architecture. For that we introduce a new,
user-enabled, Kconfig symbol, DYNAMIC_DIRECT_INTERRUPTS.
Signed-off-by: Ioannis Glaropoulos <Ioannis.Glaropoulos@nordicsemi.no>
The system power management handling code in the '_isr_wrapper' enables
interrupts by executing the 'cpsie i' instruction, which causes a
system crash on the Cortex-R devices because the Cortex-R arch port
does not support nested interrupts at this time.
This commit restricts the interrupt state manipulations in the system
power management code to the Cortex-M arch, in order to prevent
interrupt nesting on other AArch32 family archs (only Cortex-R for
now).
Signed-off-by: Stephanos Ioannidis <root@stephanos.io>
The SRAM address and size are currently available as both
DT_SRAM_{BASE_ADDRESS,SIZE} and as CONFIG_SRAM_{BASE_ADDRESS,SIZE} (via
the Kconfig preprocessor).
Use the CONFIG_SRAM_* versions everywhere, and remove generation of the
DT_SRAM_* versions from gen_defines.py.
The Kconfig symbols currently depend on 'ARC || ARM || NIOS2 || X86'.
Not sure why, so I removed it.
It looks like no configuration files set CONFIG_SRAM_* at the moment, so
another option might be to use the DT_* symbols everywhere instead. Some
Kconfig.defconfig.series files add defaults to them though.
Also improve the help texts for CONFIG_SRAM_* to say that they normally
come from devicetree rather than configuration files.
Signed-off-by: Ulf Magnusson <Ulf.Magnusson@nordicsemi.no>
This is no longer needed, since all in-tree platforms are only using
the standard mstatus formats. Remove it to avoid the complexity.
Signed-off-by: Olof Johansson <olof@lixom.net>
Same deal as in commit 41713244b3 ("kconfig: Remove '# Hidden' comments
on promptless symbols"). I forgot to do a case-insensitive search.
Signed-off-by: Ulf Magnusson <Ulf.Magnusson@nordicsemi.no>
This commit enables the CMSIS-Core(R) processor interface driver for
the Cortex-R platforms by default.
The CMSIS-Core component provides a set of standard interface functions
to control the Cortex-R series processor cores and will be required by
the arch port as well as other CMSIS library components (e.g. CMSIS-DSP
and CMSIS-NN).
Signed-off-by: Stephanos Ioannidis <root@stephanos.io>
How prompts work is better documented nowadays, and these comments might
not be that helpful if you don't know.
There are lots promptless symbols that don't have a comment.
Also fix up some comments in arch/Kconfig that seem misplaced/redundant,
and clean up some whitespace (no blank line after a comment makes it
look like it only applies to the symbol directly after it to me).
Signed-off-by: Ulf Magnusson <Ulf.Magnusson@nordicsemi.no>
Same deal as in commit bd6e04411e ("kconfig: Clean up header comments
and make them consistent") and commit 1f38ea77ba ("kconfig: Clean up
'config FOO' (two spaces) definitions"), for some newly-introduced
stuff.
Signed-off-by: Ulf Magnusson <Ulf.Magnusson@nordicsemi.no>
A single menu within an if like
if FOO
menu "blah"
...
endmenu
endif
can be replaced with
menu "blah"
depends on FOO
...
endmenu
Fix up all existing instances.
Also remove redundant extra menus underneath 'menuconfig' symbols.
'menuconfig' already creates a menu.
Also remove the menu in arch/arm/core/aarch32/Kconfig around the
"Floating point ABI" choice. The choice depends on FLOAT, which depends
on CPU_HAS_CPU, so remove the 'depends on CPU_HAS_FPU' too.
Piggyback removing a redundant 'default n' for BME280.
Signed-off-by: Ulf Magnusson <Ulf.Magnusson@nordicsemi.no>
Out-of-tree code can still be using the old file locations. Introduce
header shims to include the headers from the new correct location and
print a warning message.
Add also a new Kconfig symbol to suppress such warning.
The shim will go away after two releases, so make sure to adapt your
application for the new locations.
Signed-off-by: Carlo Caione <ccaione@baylibre.com>
Before introducing the code for ARM64 (AArch64) we need to relocate the
current ARM code to a new AArch32 sub-directory. For now we can assume
that no code is shared between ARM and ARM64.
There are no functional changes. The code is moved to the new location
and the file paths are fixed to reflect this change.
Signed-off-by: Carlo Caione <ccaione@baylibre.com>
to its own linker file snippet so snippets can be placed before it.
Using zephyr_linker_sources().
Signed-off-by: Øyvind Rønningstad <oyvind.ronningstad@nordicsemi.no>
This adds the necessary bits to build the Xtensa HAL as
a module, and removes the bits to use the HAL built with
the Zephyr SDK.
Signed-off-by: Daniel Leung <daniel.leung@intel.com>
To be able to define main() in C++ code we need to have its
prototype defined somewhere visibly. Otherwise name mangling
will prevent the linker from finding it.
Zephyr assumes a void main(void) prototype and therefore
this will be the prototype after renaming:
void zephyr_app_main(void);
Signed-off-by: Alberto Escolar Piedras <alpi@oticon.com>
Runtime stack traces (at least as currently implemented)
don't work on x86_64 normally as RBP is treated as a general-
purpose register. Depend on CONFIG_NO_OPTIMIZATIONS to enable
this on 64-bit.
Signed-off-by: Andrew Boie <andrew.p.boie@intel.com>
qemu_x86_64 will exit the emulator on a fatal system error,
like qemu_x86 already does.
Improves CI times when tests fail since sanitycheck will not
need to wait for the timeout to expire.
Signed-off-by: Andrew Boie <andrew.p.boie@intel.com>
We now dump more information for less common cases,
and this is now centralized code for 32-bit/64-bit.
All of this code is now correctly wrapped around
CONFIG_EXCEPTION_DEBUG. Some cruft and unused defines
removed.
Signed-off-by: Andrew Boie <andrew.p.boie@intel.com>
We need a size_t and not a u32_t for partition sizes,
for 64-bit compatibility.
Additionally, app_memdomain.h was also casting the base
address to a u32_t instead of a uintptr_t.
Signed-off-by: Andrew Boie <andrew.p.boie@intel.com>
Remove leading/trailing blank lines in .c, .h, .py, .rst, .yml, and
.yaml files.
Will avoid failures with the new CI test in
https://github.com/zephyrproject-rtos/ci-tools/pull/112, though it only
checks changed files.
Move the 'target-notes' target in boards/xtensa/odroid_go/doc/index.rst
to get rid of the trailing blank line there. It was probably misplaced.
Signed-off-by: Ulf Magnusson <Ulf.Magnusson@nordicsemi.no>
This is causing problems, as if we create a thread in
a system call we will *not* be using the kernel page
tables if CONFIG_KPTI=n, resulting in a crash when
the later call to copy_page_tables() tries to initialize
the PDPT (which is in the same page as the privilege
stack).
Just don't fiddle with this page's permissions; we don't
need it as a guard area anyway since we have a stack
guard placed immediately before it, and this page
is unused if user mode isn't active.
Signed-off-by: Andrew Boie <andrew.p.boie@intel.com>
Bool symbols implicitly default to 'n'.
A 'default n' can make sense e.g. in a Kconfig.defconfig file, if you
want to override a 'default y' on the base definition of the symbol. It
isn't used like that on any of these symbols though.
Also replace some
config
prompt "foo"
bool/int
with the more common shorthand
config
bool/int "foo"
See the 'Style recommendations and shorthands' section in
https://docs.zephyrproject.org/latest/guides/kconfig/index.html.
Signed-off-by: Ulf Magnusson <Ulf.Magnusson@nordicsemi.no>
Fix misspellings in docs (and Kconfig and headers processed into docs)
missed during regular reviews.
Signed-off-by: David B. Kinder <david.b.kinder@intel.com>
When we build without support for user mode, we do not need
a large number of MPU regions, so we should not allow having
MPU_GAP_FILLING unset. This would allow PRIV code execute from
SRAM, which is an unnecessary compromise on ARMv8-M builds
without USERSPACE support. We update the Kconfig dependencies
and add a sentence for clarification.
Signed-off-by: Ioannis Glaropoulos <Ioannis.Glaropoulos@nordicsemi.no>
For some reason, some users have been facing a bizarre issue
in which the -m32 option was not being passed to the linker
by cmake when building for the POSIX arch as a 32bit target,
even though the option was actually supported.
Instead of using zephyr_ld_options() which checks if an
option is supported and drops it otherwise, use
zephyr_link_libraries()
Signed-off-by: Alberto Escolar Piedras <alpi@oticon.com>
It's found that in nsim_hs_smp, sometimes the cpu
doesn't response inter-core interrupt after executing sleep
instruction.
It may be a bug of nsim, but needs more time to
investigate the root of this issue.
This commit is a workround for this, as nsim is just an
instruction simulator, no direct impact.
Signed-off-by: Wayne Ren <wei.ren@synopsys.com>
* necessary fixes after commit 11bd67db where ipi interrupt is used
to notify other cores to do a thread switch if necessary
* then for arc, it's needed to ignore swap_ok and check whether thread
switch is needed in the exit of irq handling.
Signed-off-by: Wayne Ren <wei.ren@synopsys.com>
dyn_reg_info has MPU_DYNAMIC_REGION_AREAS_NUM elements, just changing
the if check to be greater equal to this number to avoid access
MPU_DYNAMIC_REGION_AREAS_NUM element causing an out-of-bounds write.
CID: 205648
Fixes#20487
Signed-off-by: Flavio Ceolin <flavio.ceolin@intel.com>
The races are believed to be resolved with the patch to
irq_offload(). Allow the MMU to be turned on and enable
it for qemu_x86_64.
Signed-off-by: Andrew Boie <andrew.p.boie@intel.com>
Move PLATFORM_SPECIFIC_INIT declaration from Cortex-M Kconfig to the
ARM arch Kconfig in order to make it available for all ARM variants.
The rationale is that there is really no good reason why
platform-specific initialisation should be a Cortex-M-specific feature
and that Cortex-R port is expected to utilise this in a near future.
Signed-off-by: Stephanos Ioannidis <root@stephanos.io>
This commit inlines the direct ISR functions that were previously
implemented in irq_manage.c, since the PR #20119 resolved the circular
dependency between arch.h and kernel_structs.h described in the issue
#3056.
Signed-off-by: Stephanos Ioannidis <root@stephanos.io>
This commit inlines arch_isr_direct_header function that was previously
placed in irq_manage.c for no good reason (possibly in relation to the
FIXME for #3056).
In addition, since the PR #20119 resolved the header circular
dependency issue described in the issue #3056, this commit removes the
references to it in the code.
The reason for not inlining _arch_is_direct_pm as the #3056 FIXME
suggests is that there is little to gain from doing so and there still
exists circular dependency for the headers required by this function
(#20119 only addresses kernel_structs.h, which is required for _current
and _kernel, which, in turn, is required for handling interrupt nesting
in many architectures; in fact, Cortex-A and Cortex-R port will require
it as well).
Signed-off-by: Stephanos Ioannidis <root@stephanos.io>
Mark the old time conversion APIs deprecated, leave compatibility
macros in place, and replace all usage with the new API.
Signed-off-by: Andy Ross <andrew.j.ross@intel.com>
Promote the private z_arch_* namespace, which specifies
the interface between the core kernel and the
architecture code, to a new top-level namespace named
arch_*.
This allows our documentation generation to create
online documentation for this set of interfaces,
and this set of interfaces is worth treating in a
more formal way anyway.
Signed-off-by: Andrew Boie <andrew.p.boie@intel.com>
These are not part of the generic kernel to
architecture interface, rename appropriately to
reflect they are ARC-specific.
Signed-off-by: Andrew Boie <andrew.p.boie@intel.com>
This API was only created to facilitate testing of kernel
objects in IRQ context, never for actual applications.
Signed-off-by: Andrew Boie <andrew.p.boie@intel.com>
We allow the run-time, full paritioning of the SRAM space by the
ARMv8-M MPU driver to be an optional feature.
Signed-off-by: Ioannis Glaropoulos <Ioannis.Glaropoulos@nordicsemi.no>
This commit moves the function mpu_configure_regions(.) from
arm_mpu_v7_internal.h to arm_mpu.c. The function is to be used
by the both ARMv7-M MPU driver, as well as the ARMv8-M MPU
driver (when it behaves like the ARMv7-M driver).
Signed-off-by: Ioannis Glaropoulos <Ioannis.Glaropoulos@nordicsemi.no>
We introduce MPU_GAP_FILLING Kconfig option that instructs
the MPU driver to enforce a full SRAM partitioning, when it
programs the dynamic MPU regions (user thread stack, PRIV stack
guard and application memory domains) at context-switch. We
allow this to be configurable, in order to increase the number
of MPU regions available for application memory domain programming.
This option is introduced in arch/Kconfig, as it is expected
to serve as a cross-ARCH symbol. The option can be set by the
user during build configuration.
By not enforcing full partition, we may leave part of kernel
SRAM area covered only by the default ARM memory map. This
is fine for User Mode, since the background ARM map does not
allow nPRIV access at all. The difference is that kernel code
will be able to attempt fetching instructions from kernel SRAM
area without this leading directly to a MemManage exception.
Since this does not compromize User Mode, we make the skipping
of full partitioning the default behavior for the ARMv8-M MPU
driver. The application developer may be able to overwrite this.
In the wake of this change we update the macro definitions in
arm_core_mpu_dev.h that derive the maximum number of MPU regions
for application memory domains.
Signed-off-by: Ioannis Glaropoulos <Ioannis.Glaropoulos@nordicsemi.no>
Some code for unwinding stacks and z_x86_fatal_error()
now in a common C file, suitable for both modes.
Signed-off-by: Andrew Boie <andrew.p.boie@intel.com>
When compiling the components under the arch directory, the compiler
include paths for arch and kernel private headers need to be specified.
This was previously done by adding 'zephyr_library_include_directories'
to CMakeLists.txt file for every component under the arch directory,
and this resulted in a significant amount of duplicate code.
This commit uses the CMake 'include_directories' command in the root
CMakeLists.txt to simplify specification of the private header include
paths for all the arch components.
Signed-off-by: Stephanos Ioannidis <root@stephanos.io>
This commit refactors kernel and arch headers to establish a boundary
between private and public interface headers.
The refactoring strategy used in this commit is detailed in the issue
This commit introduces the following major changes:
1. Establish a clear boundary between private and public headers by
removing "kernel/include" and "arch/*/include" from the global
include paths. Ideally, only kernel/ and arch/*/ source files should
reference the headers in these directories. If these headers must be
used by a component, these include paths shall be manually added to
the CMakeLists.txt file of the component. This is intended to
discourage applications from including private kernel and arch
headers either knowingly and unknowingly.
- kernel/include/ (PRIVATE)
This directory contains the private headers that provide private
kernel definitions which should not be visible outside the kernel
and arch source code. All public kernel definitions must be added
to an appropriate header located under include/.
- arch/*/include/ (PRIVATE)
This directory contains the private headers that provide private
architecture-specific definitions which should not be visible
outside the arch and kernel source code. All public architecture-
specific definitions must be added to an appropriate header located
under include/arch/*/.
- include/ AND include/sys/ (PUBLIC)
This directory contains the public headers that provide public
kernel definitions which can be referenced by both kernel and
application code.
- include/arch/*/ (PUBLIC)
This directory contains the public headers that provide public
architecture-specific definitions which can be referenced by both
kernel and application code.
2. Split arch_interface.h into "kernel-to-arch interface" and "public
arch interface" divisions.
- kernel/include/kernel_arch_interface.h
* provides private "kernel-to-arch interface" definition.
* includes arch/*/include/kernel_arch_func.h to ensure that the
interface function implementations are always available.
* includes sys/arch_interface.h so that public arch interface
definitions are automatically included when including this file.
- arch/*/include/kernel_arch_func.h
* provides architecture-specific "kernel-to-arch interface"
implementation.
* only the functions that will be used in kernel and arch source
files are defined here.
- include/sys/arch_interface.h
* provides "public arch interface" definition.
* includes include/arch/arch_inlines.h to ensure that the
architecture-specific public inline interface function
implementations are always available.
- include/arch/arch_inlines.h
* includes architecture-specific arch_inlines.h in
include/arch/*/arch_inline.h.
- include/arch/*/arch_inline.h
* provides architecture-specific "public arch interface" inline
function implementation.
* supersedes include/sys/arch_inline.h.
3. Refactor kernel and the existing architecture implementations.
- Remove circular dependency of kernel and arch headers. The
following general rules should be observed:
* Never include any private headers from public headers
* Never include kernel_internal.h in kernel_arch_data.h
* Always include kernel_arch_data.h from kernel_arch_func.h
* Never include kernel.h from kernel_struct.h either directly or
indirectly. Only add the kernel structures that must be referenced
from public arch headers in this file.
- Relocate syscall_handler.h to include/ so it can be used in the
public code. This is necessary because many user-mode public codes
reference the functions defined in this header.
- Relocate kernel_arch_thread.h to include/arch/*/thread.h. This is
necessary to provide architecture-specific thread definition for
'struct k_thread' in kernel.h.
- Remove any private header dependencies from public headers using
the following methods:
* If dependency is not required, simply omit
* If dependency is required,
- Relocate a portion of the required dependencies from the
private header to an appropriate public header OR
- Relocate the required private header to make it public.
This commit supersedes #20047, addresses #19666, and fixes#3056.
Signed-off-by: Stephanos Ioannidis <root@stephanos.io>
Use this short header style in all Kconfig files:
# <description>
# <copyright>
# <license>
...
Also change all <description>s from
# Kconfig[.extension] - Foo-related options
to just
# Foo-related options
It's clear enough that it's about Kconfig.
The <description> cleanup was done with this command, along with some
manual cleanup (big letter at the start, etc.)
git ls-files '*Kconfig*' | \
xargs sed -i -E '1 s/#\s*Kconfig[\w.-]*\s*-\s*/# /'
Signed-off-by: Ulf Magnusson <Ulf.Magnusson@nordicsemi.no>
Clean up space errors and use a consistent style throughout the Kconfig
files. This makes reading the Kconfig files more distraction-free, helps
with grepping, and encourages the same style getting copied around
everywhere (meaning another pass hopefully won't be needed).
Go for the most common style:
- Indent properties with a single tab, including for choices.
Properties on choices work exactly the same syntactically as
properties on symbols, so not sure how the no-indentation thing
happened.
- Indent help texts with a tab followed by two spaces
- Put a space between 'config' and the symbol name, not a tab. This
also helps when grepping for definitions.
- Do '# A comment' instead of '#A comment'
I tweaked Kconfiglib a bit to find most of the stuff.
Some help texts were reflowed to 79 columns with 'gq' in Vim as well,
though not all, because I was afraid I'd accidentally mess up
formatting.
Signed-off-by: Ulf Magnusson <Ulf.Magnusson@nordicsemi.no>
In the cortex-r port we are currently using GIC as a fake cascade
controller hooked to a fake parent IRQ #0. And in gic_init() we use
IRQ_CONNECT() to connect this dummy IRQ.
Unfortunately this value is shifted and offset when calling
irq_set_priority_next_level() that tries to set the IRQ priority on a
value of 0xffffffff.
This value is offset again in gic_irq_set_priority() that actually sets
the priority on the PPI #31.
Fix this avoiding to set any priority for IRQ #0.
Signed-off-by: Carlo Caione <ccaione@baylibre.com>
Existed already in commit 8ddf82cf70 ("First commit"). Has never been
used.
Found with a script.
Signed-off-by: Ulf Magnusson <Ulf.Magnusson@nordicsemi.no>
Existed already in commit 8ddf82c ("First commit"). Has never been
used.
Found with a script.
Also remove some pointless menus that have no visible symbols in them.
Signed-off-by: Ulf Magnusson <Ulf.Magnusson@nordicsemi.no>
Adding r0 to the clobber list in the inline ASM block of
z_arch_switch_to_main_thread(). This instructs assembler
to not use r0 to store ASM expression operands, e.g. in
the subsequent instruction, msr PSR %1.
We also do a minor optimization with the clearing of R1
before jumping to main.
Signed-off-by: Ioannis Glaropoulos <Ioannis.Glaropoulos@nordicsemi.no>
Add a common definition for NUM_IRQS in arch/arm/core/Kconfig and
arch/riscv/Kconfig. That way, the type doesn't have to be given for
NUM_IRQS in all the Kconfig.defconfig files.
Trying to get rid of unnecessary "full" symbol definitions in
Kconfig.defconfig files, to make the organization clearer. It can also
help with finding unused symbols.
Signed-off-by: Ulf Magnusson <Ulf.Magnusson@nordicsemi.no>
There are two set of code supporting x86_64: x86_64 using x32 ABI,
and x86 long mode, and this consolidates both into one x86_64
architecture and SoC supporting truly 64-bit mode.
() Removes the x86_64:x32 architecture and SoC, and replaces
them with the existing x86 long mode arch and SoC.
() Replace qemu_x86_64 with qemu_x86_long as qemu_x86_64.
() Updates samples and tests to remove reference to
qemu_x86_long.
() Renames CONFIG_X86_LONGMODE to CONFIG_X86_64.
Signed-off-by: Daniel Leung <daniel.leung@intel.com>
Define FP_FPU_DA in arch/arc/Kconfig to make it always available. That
way, the Kconfig.defconfig definitions can skip the type, making them
incomplete if the base definition of the symbol disappears. That makes
the organization easier to understand and errors easier to spot.
Signed-off-by: Ulf Magnusson <Ulf.Magnusson@nordicsemi.no>
Define CPU_EM4* and CPU_EM6 in arch/arc/Kconfig to make them always
available. That way, the Kconfig.defconfig definitions can skip the
type, making them incomplete if the base definition of the symbol
disappears. That makes the organization easier to understand and errors
easier to spot.
The help texts were taken from
https://gcc.gnu.org/onlinedocs/gcc/ARC-Options.html. Help texts for
invisible symbols can be checked in the menuconfig too if you go into
show-all mode, so they're better than adding a comment.
Signed-off-by: Ulf Magnusson <Ulf.Magnusson@nordicsemi.no>
posix_soc_if.h is meant to be a private header between
the POSIX ARCH, SOC, and maybe boards,
it should not contain definitions meant to be used directly
by the kernel or app.
Some definitions were placed here due to a dependency moebius
loop.
Unravel that by removing all header dependencies in posix_soc_if.h,
move those definitions out to a more logical place,
and while we are here reduce the amount of users of
irq_offload.h in POSIX arch related code
Signed-off-by: Alberto Escolar Piedras <alpi@oticon.com>
The page tables to use are now stored in the cpuboot struct.
For the first CPU, we set to the flat page tables, and then
update later in z_x86_prep_c() once the runtime tables have
been generated.
For other CPUs, by the time we get to z_arch_start_cpu()
the runtime tables are ready do go, and so we just install
them directly.
Signed-off-by: Andrew Boie <andrew.p.boie@intel.com>
- Bring in CONFIG_X86_MMU and some related defines to
common X86 Kconfig
- Don't set ARCH_HAS_USERSPACE for intel64 yet when
X86_MMU is enabled
- Uplevel x86_mmu.c to common code
- Add logic for handling PML4 table and generating PDPTs
- move z_x86_paging_init() to common kernel_arch_func.h
- Uplevel inclusion of mmustructs.h to common x86 arch.h,
both need it for memory domain defines
Signed-off-by: Andrew Boie <andrew.p.boie@intel.com>
Program text, rodata, and data need different MMU
permissions. Split out rodata and data from the program
text, updating the linker script appropriately.
Region size symbols added to the linker script, so these
can later be used with MMU_BOOT_REGION().
Signed-off-by: Andrew Boie <andrew.p.boie@intel.com>
Same deal as in commit 7fdb525754 ("kconfig: Use 'default' instead of
'def_bool' in Kconfig.defconfig files"), but I hacked Kconfiglib to also
find cases where the type is given separately as e.g.
config FOO
int
default 3
Motivation (from a note in
https://docs.zephyrproject.org/latest/guides/kconfig/index.html):
For a symbol defined in multiple locations (e.g., in a
Kconfig.defconfig file in Zephyr), it is best to only give the
symbol type for the "base" definition of the symbol, and to use
'default' (instead of 'def_<type>' value) for the remaining
definitions. That way, if the base definition of the symbol is
removed, the symbol ends up without a type, which generates a
warning that points to the other definitions. That makes the extra
definitions easier to discover and remove.
It's also nice if 'def_bool' and the like turn into a semi-reliable flag
that the symbol is only defined in Kconfig.defconfig files. That might
be a sign that things could be cleaned up.
Will do a separate pass later to remove some symbols only defined in
Kconfig.defconfig files.
Signed-off-by: Ulf Magnusson <Ulf.Magnusson@nordicsemi.no>
We add an ARM internal API which allows the kernel to
infer the execution mode we are going to return after
the current exception.
Signed-off-by: Ioannis Glaropoulos <Ioannis.Glaropoulos@nordicsemi.no>
We introduce a Kconfig option to signify whether
an Architecture has the capability of detecting
whether execution is, currently, in a nested
exception.
Signed-off-by: Ioannis Glaropoulos <Ioannis.Glaropoulos@nordicsemi.no>
We re-implement the z_arch_is_in_isr function
so it aligns with the implementation for other
ARCHEs, i.e. returning false whenever any IRQ
or system exception is active.
Signed-off-by: Ioannis Glaropoulos <Ioannis.Glaropoulos@nordicsemi.no>
This commit refactors and cleans up __fault, so the function
- reduces to supplying MSP, PSP, and EXC_RETURN to the C
function for fault handling
- simplifies itself, removing conditional
implementation, i.e. based on ARM Secure firmware,
The reason for that is simple: it is much better to write the
fault handling in C instead of assembly, so we really do only
what is strictly required, in assembly.
Therefore, the commit refactors the z_arm_fault() function
as well, organizing better the different functional blocks,
that is:
- unlocking interrupts
- retriving ESF
- asserting for HW errors
- printing additional error logs
The refactoring unifies the way the ESF is retrieved for the
different Cortex-M variants and security execution states.
Signed-off-by: Ioannis Glaropoulos <Ioannis.Glaropoulos@nordicsemi.no>
Add some documentation for ARM-specific function
z_do_kernel_oops, stating clearly that it is only
invoked inside SVC context. We also comment on
the validity of the supplied ESF.
Signed-off-by: Ioannis Glaropoulos <Ioannis.Glaropoulos@nordicsemi.no>
We add a useful inline comment in the SVC handler (written in
assembly), which identifies one of the function return points
a bit more clearly.
Signed-off-by: Ioannis Glaropoulos <Ioannis.Glaropoulos@nordicsemi.no>
Replace:
dt_chosen_reg_addr
dt_chosen_reg_size
dt_node_reg_addr
dt_node_reg_size
with:
dt_chosen_reg_addr_int
dt_chosen_reg_size_int
dt_chosen_reg_addr_hex
dt_chosen_reg_size_hex
dt_node_reg_addr_int
dt_node_reg_size_int
dt_node_reg_addr_hex
dt_node_reg_size_hex
So that we get the proper formatted string for the type of symbol.
Signed-off-by: Kumar Gala <kumar.gala@linaro.org>
"brk" is a break-point instruction which among other things
halts ARC core. As compared to pure halt (which is "flag 1" for ARC)
it is much more convenient as it might be executed from either
secure mode or normal mode (with SecureShield enabled), while "flag"
instruction will raise privilege violation exception if SecureShield
is enabled and we're in "normal" mode.
Signed-off-by: Alexey Brodkin <abrodkin@synopsys.com>
Unused since commit 6fd6b7e50a ("xtensa: remove legacy arch
implementation").
Found with a script.
Signed-off-by: Ulf Magnusson <Ulf.Magnusson@nordicsemi.no>
Unused since commit 6fd6b7e50a ("xtensa: remove legacy arch
implementation").
Found with a script.
Signed-off-by: Ulf Magnusson <Ulf.Magnusson@nordicsemi.no>
Duplicate definitions elsewhere have been removed.
A couple functions which are defined by the arch interface
to be non-inline, but were implemented inline by native_posix
and intel64, have been moved to non-inline.
Some missing conditional compilation for z_arch_irq_offload()
has been fixed, as this is an optional feature.
Some massaging of native_posix headers to get everything
in the right scope.
Signed-off-by: Andrew Boie <andrew.p.boie@intel.com>
* implement DIRECT IRQ support both for normal irq and fast irq.
* add separate interrupt stack for fast irq and use CONFIG_ARC_
_FIRQ_STACK to control it. This will bring shortest interrupt
latency for fast irq.
* note that scheduing in DIRECT IRQ is not supported.
Signed-off-by: Wayne Ren <wei.ren@synopsys.com>
Unused after commit 71ce8ceb18 ("kernel: consolidate error handling
code").
Found with a script.
Signed-off-by: Ulf Magnusson <Ulf.Magnusson@nordicsemi.no>
The PDPT was moved to the stack area since it has alignment
requirements, but never removed from here.
Signed-off-by: Andrew Boie <andrew.p.boie@intel.com>
The intel64 switch implementation doesn't actually use a switch handle
per se, just the raw thread struct pointers which get stored into the
handle field. This works fine for normally initialized threads, but
when switching out of a dummy thread at initialization, nothing has
initialized that field and the code was dumping registers into the
bottom of memory through the resulting NULL pointer.
Fix this by skipping the load of the field value and just using an
offset instead to get the struct address, which is actually slightly
faster anyway (a SUB immediate instruction vs. the load).
Actually for extra credit we could even move the switch_handle field
to the top of the thread struct and eliminate the instruction
entirely, though if we did that it's probably worth adding some
conditional code to make the switch_handle field disappear entirely.
Signed-off-by: Andy Ross <andrew.j.ross@intel.com>
This commit updates all references to HAS_CMSIS to use HAS_CMSIS_CORE
instead. With the changes introduced to allow multiple CMSIS variants
to be specified, the latter is semantically equivalent to the former.
For more details, see issue #19717.
Signed-off-by: Stephanos Ioannidis <root@stephanos.io>
In this commit we implement the assembly functions in userspace.S
- z_arm_userspace_enter()
- z_arm_do_syscall()
- z_arch_user_string_nlen()
for ARMv6-M and ARMv8-M Baseline architecture. We "inline" the
implementation for Baseline, along with the Mainline (ARMv7-M)
implementation, i.e. we rework only what is required to build
for Baseline Cortex-M.
Signed-off-by: Ioannis Glaropoulos <Ioannis.Glaropoulos@nordicsemi.no>
In this commit we implement the assembly functions in
swap_helper.S, namely
- z_arm_pendsv()
- z_arm_svc()
for ARMv6-M and ARMv8-M Baseline architecture. We "inline" the
implementation for Baseline, along with the Mainline (ARMv7-M)
implementation, i.e. we rework only what is required to build
for Baseline Cortex-M.
Signed-off-by: Ioannis Glaropoulos <Ioannis.Glaropoulos@nordicsemi.no>
We do not support HW Stack protection capabilities in
Cortex-M Baseline CPUs (unless they have built-in stack
overflow detection capability). We adapt the Kconfig
option to reflect this.
Signed-off-by: Ioannis Glaropoulos <Ioannis.Glaropoulos@nordicsemi.no>
Remove the
# Omit prompt to signify a "hidden" option
comments that appear on some symbols. They seem to have been copy-pasted
at random, as there are lots of promptless symbols that don't have them
(that's confusing in itself, because it might give the idea that the
ones with comments are special in some way).
I suspect those comments wouldn't have helped me much if I didn't know
Kconfig either. There's a lot more Kconfig documentation now too, e.g.
https://docs.zephyrproject.org/latest/guides/kconfig/index.html.
Keep some comments that give more information than the symbol having no
prompt.
Also do some minor drive-by cleanup.
Signed-off-by: Ulf Magnusson <Ulf.Magnusson@nordicsemi.no>
This is an optional feature and no logic for it should
be present unless CONFIG_IRQ_OFFLOAD is enabled.
Signed-off-by: Andrew Boie <andrew.p.boie@intel.com>
Line up everything nicely, add leading '0x' to hex
addresses, and remove redundant newlines. Add
whitespace between the register name and contents
so the contents can be easily selected from a terminal.
Signed-off-by: Andrew Boie <andrew.p.boie@intel.com>
The struct definitions for pdpt, pd, and pt entries has been
removed:
- Bitfield ordering in a struct is implementation dependent,
it can be right-to-left or left-to-right
- The two different structures for page directory entries were
not being used consistently, or when the type of the PDE
was unknown
- Anonymous structs/unions are GCC extensions
Instead these are now u64_t, with bitwise operations used to
get/set fields.
A new set of inline functions for fetcing various page table
structures has been implemented, replacing the older macros.
Signed-off-by: Andrew Boie <andrew.p.boie@intel.com>
This hasn't been necessary since we dropped support for 32-bit
non-PAE page tables. Replace it with u64_t and scrub any
unnecessary casts left behind.
Signed-off-by: Andrew Boie <andrew.p.boie@intel.com>
This will be used for both 32-bit and 64-bit mode.
This header gets pulled in by x86's arch/cpu.h, so put
it in include/arch/x86/.
Signed-off-by: Andrew Boie <andrew.p.boie@intel.com>
Some assembly simplifications, to make code common for ARMv6
and ARMv7 architecture.
We can use ldrb, directly for reading the SVC encoding; this
removes the need for ANDing the result with 0xff right below.
We remove an immediate value of 0 from an str instruction, as
it's redundant.
Signed-off-by: Ioannis Glaropoulos <Ioannis.Glaropoulos@nordicsemi.no>
Add more documentation and inline explanatory comments in
assembly sources swap_helper.S and userspace.S and remove
redundant/wrong documentation when applicable.
Signed-off-by: Ioannis Glaropoulos <Ioannis.Glaropoulos@nordicsemi.no>
ARM user space requires ARM_MPU. We can, therefore,
remove the unnecessary #ifdef CONFIG_ARM_MPU blocks
in userspace.S. In addition, we do minor refactoring
in z_arm_userspace_enter(), and z_arm_pendsv(), and
z_arm_svc(), aiming at reducing the push/pop overhead
as much as possible.
Signed-off-by: Ioannis Glaropoulos <Ioannis.Glaropoulos@nordicsemi.no>
include/sys/arch_inlines.h will contain all architecture APIs
that are used by public inline functions and macros,
with implementations deriving from include/arch/cpu.h.
kernel/include/arch_interface.h will contain everything
else, with implementations deriving from
arch/*/include/kernel_arch_func.h.
Instances of duplicate documentation for these APIs have been
removed; implementation details have been left in place.
Signed-off-by: Andrew Boie <andrew.p.boie@intel.com>
Set the NXE bit in the EFER MSR so that the NX bit can
be set in page tables. Otherwise, the NX bit is treated
as reserved and leads to a fault if set.
Signed-off-by: Andrew Boie <andrew.p.boie@intel.com>
arch/arm/core is shared between Cortex-M and Cortex-R, so
enhance the file description headers accordingly.
Signed-off-by: Ioannis Glaropoulos <Ioannis.Glaropoulos@nordicsemi.no>
A clean-up commit that removes unnecessary inclusions from
assembly files in arm/core and arm/core/cortex_m. It also
ogranizes the inclusions based on the following order and
set of rules:
- never include kernel_structs.h
- include toolchain.h and linker/sections.h in all ASM files
- include offsets-short.h, if ASM accesses offset constants
- include arch/cpu.h, if ASM accesses CMSIS constants
(defined locally in include/arch/arm)
- include file-specific headers, if needed (e.g. vector-table.h)
Signed-off-by: Ioannis Glaropoulos <Ioannis.Glaropoulos@nordicsemi.no>
after recent changes in zephyr's fault handling, e.g. use log
to repace printk, it requires more stack to exception handling, or
the stack overflow may happen and crash the system.
this commit adds a kconfig option for exception stack size with
a larger default size.
Signed-off-by: Wayne Ren <wei.ren@synopsys.com>
The POSIX ARCH delegates some of the tasks which normally
are taken care of by the ARCH to the SOC or BOARD levels.
To avoid changes in the kernel-arch IF propagating into
the arch-soc and arch-board interfaces (which would break
off-tree posix boards) isolate them.
Also move arch inlined functions into the arch.h header,
and out from the headers which specify the posix arch-soc
and arch-board interfaces.
Signed-off-by: Alberto Escolar Piedras <alpi@oticon.com>
arch/cpu.h and kernel_arch_func.h are expected to define different
functions, per the architecture interface.
Signed-off-by: Andrew Boie <andrew.p.boie@intel.com>
There's no compelling reason why this should be inline unlike all
other arches, it's a large function, called exactly once.
Signed-off-by: Andrew Boie <andrew.p.boie@intel.com>
The specification for these arch APIs is to have them inline,
and the bodies were just oneliners calling another function
anyway.
Signed-off-by: Andrew Boie <andrew.p.boie@intel.com>
Add in-line documentation describing the process of register
preservation and exception handling on Cortex-R.
Signed-off-by: Stephanos Ioannidis <root@stephanos.io>
The interrupt exit and swap service routines for Cortex-R
unnecessarily preserve r0 and lr registers when making function calls
using bl instruction.
In case of _IntExit in exc_exit.S, the r0 register containing the
caller mode is preserved at the top, and the lr register can safely be
assumed to have been saved into the system mode stack by the interrupt
service routine.
In case of __svc in swap_helper.S, since the function saves lr to the
system mode stack at the top and exits through _IntExit, it is not
necessary to preserve lr register when executing bl instructions.
Signed-off-by: Stephanos Ioannidis <root@stephanos.io>
It's possible to have multiple processors configured without using the
SMP scheduler, so don't make definitions dependent on CONFIG_SMP.
Signed-off-by: Charles E. Youse <charles.youse@intel.com>
In non-SMP MP situations, the interrupt stacks might not exist, so
do not assume they do. Instead, initialize the TSS IST1 from the
cpuboot[] vector (meaning, on APs, the stack from z_arch_start_cpu).
Eliminates redundancy at the same time.
Signed-off-by: Charles E. Youse <charles.youse@intel.com>
This is the Wrong Thing(tm) with SMP enabled. Previously this
worked because interrupts would be re-enabled in the interrupt
entry sequence, but this is no longer the case.
Signed-off-by: Charles E. Youse <charles.youse@intel.com>
was ignoring the rest of the expression, though the effect was
harmless (including unreachable code in some builds).
Signed-off-by: Charles E. Youse <charles.youse@intel.com>
Trivial change to the Kconfig: the first 32 vectors are reserved,
so it's not possible to have 256 IRQ vectors. Change max to 224.
Signed-off-by: Charles E. Youse <charles.youse@intel.com>
Add duplicate per-CPU data structures (x86_cpuboot, tss, stacks, etc.)
for up to 4 total CPUs, add code in locore and z_arch_start_cpu().
The test board, qemu_x86_long, now defaults to 2 CPUs.
Signed-off-by: Charles E. Youse <charles.youse@intel.com>
Take a dummy first argument, so that the BSP entry point (z_x86_prep_c)
has the same signature as the AP entry point (smp_init_top).
Signed-off-by: Charles E. Youse <charles.youse@intel.com>
A new 'struct x86_cpuboot' is created as well as an instance called
'x86_cpuboot[]' which contains per-CPU boot data (initial stack,
entry function/arg, selectors, etc.). The locore now consults this
table to set up per-CPU registers, etc. during early boot.
Also, rename tss.c to cpu.c as its scope is growing.
Signed-off-by: Charles E. Youse <charles.youse@intel.com>
There's no need to qualify the 64-bit CS/DS selectors, and the GS and
TR selectors are renamed CPU0_GS and CPU0_TR as they are CPU-specific.
Signed-off-by: Charles E. Youse <charles.youse@intel.com>
In some places the code was being overly pedantic; e.g., there is no
need to load our own 32-bit descriptors because the loader's are fine
for our purposes. We can defer loading our own segments until 64-bit.
The sequence is re-ordered to faciliate code sharing between the BSP
and APs when SMP is enabled (all BSP-specific operations occur before
the per-CPU initialization).
Signed-off-by: Charles E. Youse <charles.youse@intel.com>
This is really just to facilitate CPU bootstrap code between
the BSP and the APs, moving the clear operation out of the way.
Signed-off-by: Charles E. Youse <charles.youse@intel.com>
In the general case, the local APIC can't be treated as a normal device
with a single boot-time initialization - on SMP systems, each CPU must
initialize its own. Hence the initialization proper is separated from
the device-driver initialization, and said initialization is called
from the early startup-assembly code when appropriate.
Signed-off-by: Charles E. Youse <charles.youse@intel.com>
The 32-bit and 64-bit assembly startup sequences share quite a
bunch of common code, so it's factored out into one file to avoid
repeating ourselves (and potentially falling out of sync).
Signed-off-by: Charles E. Youse <charles.youse@intel.com>
The linker script was missing symbols that defined the boundaries
of kernel memory segments (_image_rom_end, etc.). These are added
so that core/memmap.c can properly account for those segments.
Signed-off-by: Charles E. Youse <charles.youse@intel.com>
Elevate the previously 32-bit-only z_x86_prep_c() function to common
code, so both 32-bit and 64-bit arches now enter the kernel this way.
Minor changes to prep_c.c to make it build with the SMP scheduler on.
Signed-off-by: Charles E. Youse <charles.youse@intel.com>
And set qemu_x86_long board to build with CONFIG_SMP=y by default.
Apparently two benchmark tests - latency_measure and sys_kernel -
do not work with the SMP scheduler, so those tests are disabled.
Signed-off-by: Charles E. Youse <charles.youse@intel.com>
This patch is a preparatory step in enabling the MMU in
long mode; no steps are taken to implement long mode support.
We introduce struct x86_page_tables, which represents the
top-level data structure for page tables:
- For 32-bit, this will contain a four-entry page directory
pointer table (PDPT)
- For 64-bit, this will (eventually) contain a page map level 4
table (PML4)
In either case, this pointer value is what gets programmed into
CR3 to activate a set of page tables. There are extra bits in
CR3 to set for long mode, we'll get around to that later.
This abstraction will allow us to use the same APIs that work
with page tables in either mode, rather than hard-coding that
the top level data structure is a PDPT.
z_x86_mmu_validate() has been re-written to make it easier to
add another level of paging for long mode, to support 2MB
PDPT entries, and correctly validate regions which span PDPTE
entries.
Some MMU-related APIs moved out of 32-bit x86's arch.h into
mmustructs.h.
Signed-off-by: Andrew Boie <andrew.p.boie@intel.com>
This patch re-namespaces global variables and functions
that are used only within the arch/arm/ code to be
prefixed with z_arm_.
Some instances of CamelCase have been corrected.
Signed-off-by: Andrew Boie <andrew.p.boie@intel.com>
* the old codes may not save the caller saved regs correctly,
e.g. r7- r12. Because the sys call entry is called in the form
of static inline function. The compiler optimizations may not save
all the caller saved regs.
* new codes use the irq stack frame as the sys call frame and gurantee
all the called saved regs are pushed and popped correctly.
* the side effect of new codes are more stack operations and a little
overhead.
Signed-off-by: Wayne Ren <wei.ren@synopsys.com>
This makes it clearer that this is an API that is expected
to be implemented at the architecture level.
Signed-off-by: Andrew Boie <andrew.p.boie@intel.com>
The main and idle threads, and their associated stacks,
were being referenced in various parts of the kernel
with no central definition. Expose these in kernel_internal.h
and namespace with z_ appropriately.
The main and idle threads were being defined statically,
with another variable exposed to contain their pointer
value. This wastes a bit of memory and isn't accessible
to user threads anyway, just expose the actual thread
objects.
Redundance MAIN_STACK_SIZE and IDLE_STACK_SIZE defines
in init.c removed, just use the Kconfigs they derive
from.
Signed-off-by: Andrew Boie <andrew.p.boie@intel.com>
This metric shows when the system first enters an idle
state, which has already been recorded in the arch-
independent implementation of the idle thread.
Only x86 was doing this.
Signed-off-by: Andrew Boie <andrew.p.boie@intel.com>
This is part of the core kernel -> architecture interface and
has been renamed z_arch_kernel_init().
Signed-off-by: Andrew Boie <andrew.p.boie@intel.com>
z_set_thread_return_value is part of the core kernel -> arch
interface and has been renamed to z_arch_thread_return_value_set.
z_set_thread_return_value_with_data renamed to
z_thread_return_value_set_with_data for consistency.
Signed-off-by: Andrew Boie <andrew.p.boie@intel.com>
k_cpu_idle() and k_cpu_atomic_idle() were being directly
implemented by arch code.
Rename these implementations to z_arch_cpu_idle() and
z_arch_cpu_atomic_idle(), and call them from new inline
function definitions in kernel.h.
Signed-off-by: Andrew Boie <andrew.p.boie@intel.com>
This is part of the core kernel -> architecture interface
and is appropriately renamed z_arch_is_in_isr().
References from test cases changed to k_is_in_isr().
Signed-off-by: Andrew Boie <andrew.p.boie@intel.com>
This is part of the core kernel -> architecture interface
and should have a leading prefix z_arch_.
Signed-off-by: Andrew Boie <andrew.p.boie@intel.com>
Global variables related to timing information have been
renamed to be prefixed with z_arch, with naming arranged
in increasing order of specificity.
Signed-off-by: Andrew Boie <andrew.p.boie@intel.com>
ACPI is predominantly x86, and only currently implemented on x86,
but it is employed on other architectures, so rename accordingly.
Signed-off-by: Charles E. Youse <charles.youse@intel.com>
Simple naming change, since MULTIBOOT is clear enough by itself and
"namespacing" it to X86 is unnecessary and/or inappropriate.
Signed-off-by: Charles E. Youse <charles.youse@intel.com>
x86 has more complex memory maps than most Zephyr targets. A mechanism
is introduced here to manage such a map, and some methods are provided
to populate it (e.g., Multiboot).
The x86_info tool is extended to display memory map data.
Signed-off-by: Charles E. Youse <charles.youse@intel.com>
Originally, the multiboot info struct was copied in the early assembly
language code. This code is moved to a C function in multiboot.c for
two reasons:
1. It's about to get more complicated, as we want the ability to use
a multiboot-provided memory map if available, and
2. this will faciliate its sharing between 32- and 64-bit subarches.
Signed-off-by: Charles E. Youse <charles.youse@intel.com>
Implement a simple ACPI parser with enough functionality to
enumerate CPU cores and determine their local APIC IDs.
Signed-off-by: Charles E. Youse <charles.youse@intel.com>
* In ARC, pop reg ==> sp=sp-4; *sp= b; The original codes have bug that
the save of ilink (st ilink [sp]) will crash the interruptted stack's
content. This commit fixes this bug and makes the codes easier to
understand
Signed-off-by: Wayne Ren <wei.ren@synopsys.com>
Various C and Assembly modules
make function calls to z_sys_trace_*. These merely call
corresponding functions sys_trace_*. This commit
is to simplify these by making direct function calls
to the sys_trace_* functions from these modules.
Subsequently, the z_sys_trace_* functions are removed.
Signed-off-by: Mrinal Sen <msen@oticon.com>
- Remove redundant inclusions in irq_init.c
- Remove comment about thread_abort function,
which does not belong in this file (probably
left-out during code refactoring)
- Include arm cmsis.h only under #ifdef CONFIG_ARM
Signed-off-by: Ioannis Glaropoulos <Ioannis.Glaropoulos@nordicsemi.no>
This used to be part of the "restore always" set of registers because
__swap was expected to return a value. No longer required, so RAX is
moved to the volatile registers and we save a few cycles occasionally.
Signed-off-by: Charles E. Youse <charles.youse@intel.com>
A space is allocated in the TSS for per-CPU variables. At present,
this is only a 'struct _cpu *' to find the _kernel CPU struct. The
locore routines are rewritten to find _current and _nested via this
pointer rather than referencing the _kernel global directly.
This is obviously in preparation for SMP support.
Signed-off-by: Charles E. Youse <charles.youse@intel.com>
This function call was erroneously inserted between the instruction
that set the Z flag and the instruction that tested the Z flag. The
call is moved up a few instructions where it can't junk CPU state.
Signed-off-by: Charles E. Youse <charles.youse@intel.com>
Declare the 64-bit TSS as a struct, and define the instance in C.
Add a data segment selector that overlaps the TSS and keep that
loaded in GS so we can access the TSS via a segment-override prefix.
Signed-off-by: Charles E. Youse <charles.youse@intel.com>
This is moved from arch/x86/include/ia32/kernel_arch_func.h to the
common header arch/x86/include/kernel_arch_func.h so it can be shared.
Signed-off-by: Charles E. Youse <charles.youse@intel.com>
This is largely a conceptual change rather than an actual change.
Instead of using an array of interrupt stacks (one for each IRQ
nesting level), we use one interrupt stack and subdivide it. The
effect is the same, but this is more in line with the Zephyr model
of one ISR stack per CPU (as reflected in init.c).
Signed-off-by: Charles E. Youse <charles.youse@intel.com>
Like its 32-bit sibling, the 64-bit code should EOI inline rather than
invoking a function. Defeats the performance advantages of x2APIC.
Signed-off-by: Charles E. Youse <charles.youse@intel.com>
The boot time measurement sample was giving bogus values on x86: an
assumption was made that the system timer is in sync with the CPU TSC,
which is not the case on most x86 boards.
Boot time measurements are no longer permitted unless the timer source
is the local APIC. To avoid issues of TSC scaling, the startup datum
has been forced to 0, which is in line with the ARM implementation
(which is the only other platform which supports this feature).
Cleanups along the way:
As the datum is now assumed zero, some variables are removed and
calculations simplified. The global variables involved in boot time
measurements are moved to the kernel.h header rather than being
redeclared in every place they are referenced. Since none of the
measurements actually use 64-bit precision, the samples are reduced
to 32-bit quantities.
In addition, this feature has been enabled in long mode.
Fixes: #19144
Signed-off-by: Charles E. Youse <charles.youse@intel.com>
There are not enough bits in k_thread.thread_state with SMP enabled,
and the field is (should be) private to the scheduler, anyway. So
move state bits to the _thread_arch where they belong.
While we're at it, refactor some offset data w/r/t _thread_arch
because it can be shared between 32- and 64-bit subarches.
Signed-off-by: Charles E. Youse <charles.youse@intel.com>
k_thread.thread_state (or rather, _thread_base.thread_state) should be
private to the kernel/scheduler, so flags previously stored there are
moved to _thread_arch where the belong.
Signed-off-by: Charles E. Youse <charles.youse@intel.com>
when enable CONFIG_CUSTOM_SECTION_ALIGN, it need less alignment
memory for image rom region. But that needs carefully configure
MPU region and sub-regions(ARMv7-M) to cover this feature.
Fixes: #17337.
Signed-off-by: Wentong Wu <wentong.wu@intel.com>
The GNU ARM Embedded "8-2019-q3-update" toolchain
erroneously uses "typeof" instead of "__typeof__".
To work around this we define typeof to be able to
support it.
This reverts commit 01a71eae3d.
Signed-off-by: Ioannis Glaropoulos <Ioannis.Glaropoulos@nordicsemi.no>
We don't need to save the ABI caller-save registers here, because
we don't preempt threads from nested IRQ contexts.
Signed-off-by: Charles E. Youse <charles.youse@intel.com>
This is a naive implementation which does "eager" context switching
for floating-point context, which, of course, introduces performance
concerns. Other approaches have security concerns, SMP implications,
and impact the x86 arch and Zephyr project as a whole. Discussion is
needed, so punting with the straightforward solution for now.
Signed-off-by: Charles E. Youse <charles.youse@intel.com>
Fleshed out z_arch_esf_t and added code to build this frame when
exceptions occur. Created a separate small stack for exceptions and
shifted the initialization code to use this instead of the IRQ stack.
Moved IRQ stack(s) to irq.c.
Signed-off-by: Charles E. Youse <charles.youse@intel.com>
The IRQ_OFFLOAD_VECTOR config option is also moved to the arch level,
as it is shared between both 32- and 64-bit subarches.
Signed-off-by: Charles E. Youse <charles.youse@intel.com>
Using the arch Kconfig here, instead of kernel/Kconfig. Intel64 with
the SysV ABI requires some pretty big stacks. These 4K-8K defaults
are arguably a bit small, but the Zephyr defaults are REALLY too small.
Signed-off-by: Charles E. Youse <charles.youse@intel.com>
First "complete" version of Intel64 support for x86. Compilation of
apps for supported boards (read: up_squared) with CONFIG_X86_LONGMODE=y
is now working. Booting, device drivers, interrupts, scheduling, etc.
appear to be functioning properly. Beware that this is ALHPA quality,
not ready for production use, but the port has advanced far enough that
it's time to start working through the test suite and samples, fleshing
out any missing features, and squashing bugs.
Signed-off-by: Charles E. Youse <charles.youse@intel.com>
Widen the integer to pointer size before conversion, to make
explicit the intent (and silence the compiler warning). Also
fix a minor bug involving a duplicate (and thus dead) store.
Signed-off-by: Charles E. Youse <charles.youse@intel.com>
This patch adds basic build infrastructure, definitions, a linker
script, etc. to use the Zephyr and 0.10.1 SDK to build a 64-bit
ELF binary suitable for use with GRUB to minimally bootstrap an
Apollo Lake (e.g., UpSquared) board. The resulting binary can hardly
be called a Zephyr kernel as it is lacking most of the glue logic,
but it is a starting point to flesh those out in the x86 tree.
The "kernel" builds with a few harmless warnings, both with GCC from
the Zephyr SDK and with ICC (which is currently being worked on in
a separate branch). These warnings are either related to pointer size
differences (since this is an LP64 build) and/or dummy functions
that will be replaced with working versions shortly.
Signed-off-by: Charles E. Youse <charles.youse@intel.com>
Use different headers for kernel_arch_{func,thread}.h when in
CONFIG_X86_LONGMODE, and add placeholders for Intel64 versions.
Signed-off-by: Charles E. Youse <charles.youse@intel.com>
Some definitions may be shared between subarchitectures, so refactor
accordingly. The definitions are also modified to separate bits. A
placeholder is created for the Intel64 definitions.
Signed-off-by: Charles E. Youse <charles.youse@intel.com>
The IA32 and Intel64 subarchitectures will generate different offset
symbols, so they are refactored. No functional change.
Signed-off-by: Charles E. Youse <charles.youse@intel.com>
The _irq_to_interrupt_vector[] array shouldn't be accessed directly,
as there is a macro for this.
Signed-off-by: Charles E. Youse <charles.youse@intel.com>
