Commit graph

301 commits

Author SHA1 Message Date
Andrew Boie a594ca7c8f kernel: cleanup and formally define CPU start fn
The "key" parameter is legacy, remove it.

Add a typedef for the expected function pointer type.

Signed-off-by: Andrew Boie <andrew.p.boie@intel.com>
2020-01-13 16:35:10 -05:00
Andrew Boie 3d80208025 x86: implement user mode on 64-bit
- In early boot, enable the syscall instruction and set up
  necessary MSRs
- Add a hook to update page tables on context switch
- Properly initialize thread based on whether it will
  start in user or supervisor mode
- Add landing function for system calls to execute the
  desired handler
- Implement arch_user_string_nlen()
- Implement logic for dropping a thread down to user mode
- Reserve per-CPU storage space for user and privilege
  elevation stack pointers, necessary for handling syscalls
  when no free registers are available
- Proper handling of gs register considerations when
  transitioning privilege levels

Kernel page table isolation (KPTI) is not yet implemented.

Signed-off-by: Andrew Boie <andrew.p.boie@intel.com>
2020-01-13 16:35:10 -05:00
Andrew Boie 077b587447 x86: implement hw-based oops for both variants
We use a fixed value of 32 as the way interrupts/exceptions
are setup in x86_64's locore.S do not lend themselves to
Kconfig configuration of the vector to use.

HW-based kernel oops is now permanently on, there's no reason
to make it optional that I can see.

Default vectors for IPI and irq offload adjusted to not
collide.

Signed-off-by: Andrew Boie <andrew.p.boie@intel.com>
2020-01-13 16:35:10 -05:00
Andrew Boie 7f82b99ad4 x86: up-level some user mode functions
These are now common code, all are related to user mode
threads. The rat's nest of ifdefs in ia32's arch_new_thread
has been greatly simplified, there is now just one hook
if user mode is turned on.

Signed-off-by: Andrew Boie <andrew.p.boie@intel.com>
2020-01-13 16:35:10 -05:00
Andrew Boie 7ea958e0dd x86: optimize locations of psp and thread ptables
z_x86_thread_page_tables_get() now works for both user
and supervisor threads, returning the kernel page tables
in the latter case. This API has been up-leveled to
a common header.

The per-thread privilege elevation stack initial stack
pointer, and the per-thread page table locations are no
longer computed from other values, and instead are stored
in thread->arch.

A problem where the wrong page tables were dumped out
on certain kinds of page faults has been fixed.

Signed-off-by: Andrew Boie <andrew.p.boie@intel.com>
2020-01-13 16:35:10 -05:00
Andrew Boie 692fda47fc x86: use MSRs for %gs
We don't need to set up GDT data descriptors for setting
%gs. Instead, we use the x86 MSRs to set GS_BASE and
KERNEL_GS_BASE.

We don't currently allow user mode to set %gs on its own,
but later on if we do, we have everything set up to issue
'swapgs' instructions on syscall or IRQ.

Unused entries in the GDT have been removed.

Signed-off-by: Andrew Boie <andrew.p.boie@intel.com>
2020-01-13 16:35:10 -05:00
Andrew Boie 3256e9e00b x86: use BIT() macros for cr0/cr4 bits
Easier to establish correspondence with the technical
manuals.

Signed-off-by: Andrew Boie <andrew.p.boie@intel.com>
2020-01-13 16:35:10 -05:00
Andrew Boie 2b67ca8ac9 x86: improve exception debugging
We now dump more information for less common cases,
and this is now centralized code for 32-bit/64-bit.
All of this code is now correctly wrapped around
CONFIG_EXCEPTION_DEBUG. Some cruft and unused defines
removed.

Signed-off-by: Andrew Boie <andrew.p.boie@intel.com>
2019-12-17 11:39:22 -08:00
Andrew Boie 4f77c2ad53 kernel: rename z_arch_ to arch_
Promote the private z_arch_* namespace, which specifies
the interface between the core kernel and the
architecture code, to a new top-level namespace named
arch_*.

This allows our documentation generation to create
online documentation for this set of interfaces,
and this set of interfaces is worth treating in a
more formal way anyway.

Signed-off-by: Andrew Boie <andrew.p.boie@intel.com>
2019-11-07 15:21:46 -08:00
Andrew Boie 0b27cacabd x86: up-level page fault handling
Now in common code for 32-bit and 64-bit.

Signed-off-by: Andrew Boie <andrew.p.boie@intel.com>
2019-11-07 11:19:38 -05:00
Andrew Boie 9fb89ccf32 x86: consolidate some fatal error code
Some code for unwinding stacks and z_x86_fatal_error()
now in a common C file, suitable for both modes.

Signed-off-by: Andrew Boie <andrew.p.boie@intel.com>
2019-11-06 17:50:34 -08:00
Andrew Boie 64f6e2ac6b x86: consolidate STACK_ROUND_* definition
There was no definition for 64-bit.

Signed-off-by: Andrew Boie <andrew.p.boie@intel.com>
2019-11-06 17:50:34 -08:00
Stephanos Ioannidis 2d7460482d headers: Refactor kernel and arch headers.
This commit refactors kernel and arch headers to establish a boundary
between private and public interface headers.

The refactoring strategy used in this commit is detailed in the issue

This commit introduces the following major changes:

1. Establish a clear boundary between private and public headers by
  removing "kernel/include" and "arch/*/include" from the global
  include paths. Ideally, only kernel/ and arch/*/ source files should
  reference the headers in these directories. If these headers must be
  used by a component, these include paths shall be manually added to
  the CMakeLists.txt file of the component. This is intended to
  discourage applications from including private kernel and arch
  headers either knowingly and unknowingly.

  - kernel/include/ (PRIVATE)
    This directory contains the private headers that provide private
   kernel definitions which should not be visible outside the kernel
   and arch source code. All public kernel definitions must be added
   to an appropriate header located under include/.

  - arch/*/include/ (PRIVATE)
    This directory contains the private headers that provide private
   architecture-specific definitions which should not be visible
   outside the arch and kernel source code. All public architecture-
   specific definitions must be added to an appropriate header located
   under include/arch/*/.

  - include/ AND include/sys/ (PUBLIC)
    This directory contains the public headers that provide public
   kernel definitions which can be referenced by both kernel and
   application code.

  - include/arch/*/ (PUBLIC)
    This directory contains the public headers that provide public
   architecture-specific definitions which can be referenced by both
   kernel and application code.

2. Split arch_interface.h into "kernel-to-arch interface" and "public
  arch interface" divisions.

  - kernel/include/kernel_arch_interface.h
    * provides private "kernel-to-arch interface" definition.
    * includes arch/*/include/kernel_arch_func.h to ensure that the
     interface function implementations are always available.
    * includes sys/arch_interface.h so that public arch interface
     definitions are automatically included when including this file.

  - arch/*/include/kernel_arch_func.h
    * provides architecture-specific "kernel-to-arch interface"
     implementation.
    * only the functions that will be used in kernel and arch source
     files are defined here.

  - include/sys/arch_interface.h
    * provides "public arch interface" definition.
    * includes include/arch/arch_inlines.h to ensure that the
     architecture-specific public inline interface function
     implementations are always available.

  - include/arch/arch_inlines.h
    * includes architecture-specific arch_inlines.h in
     include/arch/*/arch_inline.h.

  - include/arch/*/arch_inline.h
    * provides architecture-specific "public arch interface" inline
     function implementation.
    * supersedes include/sys/arch_inline.h.

3. Refactor kernel and the existing architecture implementations.

  - Remove circular dependency of kernel and arch headers. The
   following general rules should be observed:

    * Never include any private headers from public headers
    * Never include kernel_internal.h in kernel_arch_data.h
    * Always include kernel_arch_data.h from kernel_arch_func.h
    * Never include kernel.h from kernel_struct.h either directly or
     indirectly. Only add the kernel structures that must be referenced
     from public arch headers in this file.

  - Relocate syscall_handler.h to include/ so it can be used in the
   public code. This is necessary because many user-mode public codes
   reference the functions defined in this header.

  - Relocate kernel_arch_thread.h to include/arch/*/thread.h. This is
   necessary to provide architecture-specific thread definition for
   'struct k_thread' in kernel.h.

  - Remove any private header dependencies from public headers using
   the following methods:

    * If dependency is not required, simply omit
    * If dependency is required,
      - Relocate a portion of the required dependencies from the
       private header to an appropriate public header OR
      - Relocate the required private header to make it public.

This commit supersedes #20047, addresses #19666, and fixes #3056.

Signed-off-by: Stephanos Ioannidis <root@stephanos.io>
2019-11-06 16:07:32 -08:00
Daniel Leung b7eb04b300 x86: consolidate x86_64 architecture, SoC and boards
There are two set of code supporting x86_64: x86_64 using x32 ABI,
and x86 long mode, and this consolidates both into one x86_64
architecture and SoC supporting truly 64-bit mode.

() Removes the x86_64:x32 architecture and SoC, and replaces
   them with the existing x86 long mode arch and SoC.
() Replace qemu_x86_64 with qemu_x86_long as qemu_x86_64.
() Updates samples and tests to remove reference to
   qemu_x86_long.
() Renames CONFIG_X86_LONGMODE to CONFIG_X86_64.

Signed-off-by: Daniel Leung <daniel.leung@intel.com>
2019-10-25 17:57:55 -04:00
Andrew Boie e1f2292895 x86: intel64: set page tables for every CPU
The page tables to use are now stored in the cpuboot struct.
For the first CPU, we set to the flat page tables, and then
update later in z_x86_prep_c() once the runtime tables have
been generated.

For other CPUs, by the time we get to z_arch_start_cpu()
the runtime tables are ready do go, and so we just install
them directly.

Signed-off-by: Andrew Boie <andrew.p.boie@intel.com>
2019-10-24 12:48:45 -07:00
Andrew Boie f6e82ea1bd x86: generate runtime 64-bit page tables
- Bring in CONFIG_X86_MMU and some related defines to
  common X86 Kconfig
- Don't set ARCH_HAS_USERSPACE for intel64 yet when
  X86_MMU is enabled
- Uplevel x86_mmu.c to common code
- Add logic for handling PML4 table and generating PDPTs
- move z_x86_paging_init() to common kernel_arch_func.h
- Uplevel inclusion of mmustructs.h to common x86 arch.h,
  both need it for memory domain defines

Signed-off-by: Andrew Boie <andrew.p.boie@intel.com>
2019-10-24 12:48:45 -07:00
Andrew Boie 979b17f243 kernel: activate arch interface headers
Duplicate definitions elsewhere have been removed.

A couple functions which are defined by the arch interface
to be non-inline, but were implemented inline by native_posix
and intel64, have been moved to non-inline.

Some missing conditional compilation for z_arch_irq_offload()
has been fixed, as this is an optional feature.

Some massaging of native_posix headers to get everything
in the right scope.

Signed-off-by: Andrew Boie <andrew.p.boie@intel.com>
2019-10-21 10:13:38 -07:00
Andrew Boie f82beb984d x86: remove unused thread arch member
The PDPT was moved to the stack area since it has alignment
requirements, but never removed from here.

Signed-off-by: Andrew Boie <andrew.p.boie@intel.com>
2019-10-19 20:40:26 -07:00
Andrew Boie 24958f30d9 x86: move z_x86_early_serial_init()
This works with long mode as well, uplevel to
common kernel_arch_func.h.

Signed-off-by: Andrew Boie <andrew.p.boie@intel.com>
2019-10-15 09:00:49 -07:00
Andrew Boie e3ab43580c x86: move mmustructs.h
This will be used for both 32-bit and 64-bit mode.
This header gets pulled in by x86's arch/cpu.h, so put
it in include/arch/x86/.

Signed-off-by: Andrew Boie <andrew.p.boie@intel.com>
2019-10-14 11:49:39 -07:00
Andrew Boie 8ffff144ea kernel: add architecture interface headers
include/sys/arch_inlines.h will contain all architecture APIs
that are used by public inline functions and macros,
with implementations deriving from include/arch/cpu.h.

kernel/include/arch_interface.h will contain everything
else, with implementations deriving from
arch/*/include/kernel_arch_func.h.

Instances of duplicate documentation for these APIs have been
removed; implementation details have been left in place.

Signed-off-by: Andrew Boie <andrew.p.boie@intel.com>
2019-10-11 13:30:46 -07:00
Andrew Boie 7f715bd113 intel64: add inline definition of z_arch_switch()
This just calls the assembly code. Needed to be
inline per the architecture API.

Signed-off-by: Andrew Boie <andrew.p.boie@intel.com>
2019-10-09 09:14:18 -04:00
Andrew Boie 54506b5a9b arches: fix z_arch_is_in_isr() defintion
For some reason these were implemented as macros when they
should be inline functions.

Signed-off-by: Andrew Boie <andrew.p.boie@intel.com>
2019-10-09 09:14:18 -04:00
Charles E. Youse f7cfb4303b arch/x86: do not assume MP means SMP
It's possible to have multiple processors configured without using the
SMP scheduler, so don't make definitions dependent on CONFIG_SMP.

Signed-off-by: Charles E. Youse <charles.youse@intel.com>
2019-10-07 19:46:55 -04:00
Charles E. Youse 643661cb44 arch/x86: declare z_x86_prep_c() in kernel_arch_func.h
And remove the ad hoc prototype in cpu.c for Intel64.

Signed-off-by: Charles E. Youse <charles.youse@intel.com>
2019-10-07 19:46:55 -04:00
Charles E. Youse 5abab591c2 arch/x86: (Intel64) make z_arch_start_cpu() synchronous
Don't leave z_arch_start_cpu() until the target CPU has been started.

Signed-off-by: Charles E. Youse <charles.youse@intel.com>
2019-10-07 19:46:55 -04:00
Charles E. Youse 66510db98c arch/x86: (Intel64) add scheduler IPI support
Add z_arch_sched_ipi() and such to enable scheduler IPIs when SMP.

Signed-off-by: Charles E. Youse <charles.youse@intel.com>
2019-10-07 19:46:55 -04:00
Charles E. Youse 3eb1a8b59a arch/x86: (Intel64) implement SMP support
Add duplicate per-CPU data structures (x86_cpuboot, tss, stacks, etc.)
for up to 4 total CPUs, add code in locore and z_arch_start_cpu().

The test board, qemu_x86_long, now defaults to 2 CPUs.

Signed-off-by: Charles E. Youse <charles.youse@intel.com>
2019-10-07 19:46:55 -04:00
Charles E. Youse f9eaee35b8 arch/x86: (Intel64) use per-CPU parameter struct for CPU startup
A new 'struct x86_cpuboot' is created as well as an instance called
'x86_cpuboot[]' which contains per-CPU boot data (initial stack,
entry function/arg, selectors, etc.). The locore now consults this
table to set up per-CPU registers, etc. during early boot.

Also, rename tss.c to cpu.c as its scope is growing.

Signed-off-by: Charles E. Youse <charles.youse@intel.com>
2019-10-07 19:46:55 -04:00
Charles E. Youse edf5761c83 arch/x86: (Intel64) rename kernel segment constants
There's no need to qualify the 64-bit CS/DS selectors, and the GS and
TR selectors are renamed CPU0_GS and CPU0_TR as they are CPU-specific.

Signed-off-by: Charles E. Youse <charles.youse@intel.com>
2019-10-07 19:46:55 -04:00
Charles E. Youse 25a7cc1136 arch/x86: (Intel64) add missing linker symbols
The linker script was missing symbols that defined the boundaries
of kernel memory segments (_image_rom_end, etc.). These are added
so that core/memmap.c can properly account for those segments.

Signed-off-by: Charles E. Youse <charles.youse@intel.com>
2019-10-07 19:46:55 -04:00
Charles E. Youse 8d97750eef arch/x86: (Intel64) add z_arch_curr_cpu() to enable CONFIG_SMP=y
And set qemu_x86_long board to build with CONFIG_SMP=y by default.
Apparently two benchmark tests - latency_measure and sys_kernel -
do not work with the SMP scheduler, so those tests are disabled.

Signed-off-by: Charles E. Youse <charles.youse@intel.com>
2019-10-07 19:46:55 -04:00
Andrew Boie f0ddbd7eee x86: abstract toplevel page table pointer
This patch is a preparatory step in enabling the MMU in
long mode; no steps are taken to implement long mode support.

We introduce struct x86_page_tables, which represents the
top-level data structure for page tables:

- For 32-bit, this will contain a four-entry page directory
  pointer table (PDPT)
- For 64-bit, this will (eventually) contain a page map level 4
  table (PML4)

In either case, this pointer value is what gets programmed into
CR3 to activate a set of page tables. There are extra bits in
CR3 to set for long mode, we'll get around to that later.

This abstraction will allow us to use the same APIs that work
with page tables in either mode, rather than hard-coding that
the top level data structure is a PDPT.

z_x86_mmu_validate() has been re-written to make it easier to
add another level of paging for long mode, to support 2MB
PDPT entries, and correctly validate regions which span PDPTE
entries.

Some MMU-related APIs moved out of 32-bit x86's arch.h into
mmustructs.h.

Signed-off-by: Andrew Boie <andrew.p.boie@intel.com>
2019-10-04 15:53:49 -07:00
Andrew Boie f6fb634b89 kernel: rename kernel_arch_init()
This is part of the core kernel -> architecture interface and
has been renamed z_arch_kernel_init().

Signed-off-by: Andrew Boie <andrew.p.boie@intel.com>
2019-09-30 15:25:55 -04:00
Andrew Boie 4ad9f687df kernel: rename thread return value functions
z_set_thread_return_value is part of the core kernel -> arch
interface and has been renamed to z_arch_thread_return_value_set.

z_set_thread_return_value_with_data renamed to
z_thread_return_value_set_with_data for consistency.

Signed-off-by: Andrew Boie <andrew.p.boie@intel.com>
2019-09-30 15:25:55 -04:00
Andrew Boie 07525a3d54 kernel: add arch interface for idle functions
k_cpu_idle() and k_cpu_atomic_idle() were being directly
implemented by arch code.

Rename these implementations to z_arch_cpu_idle() and
z_arch_cpu_atomic_idle(), and call them from new inline
function definitions in kernel.h.

Signed-off-by: Andrew Boie <andrew.p.boie@intel.com>
2019-09-30 15:25:55 -04:00
Andrew Boie e1ec59f9c2 kernel: renamespace z_is_in_isr()
This is part of the core kernel -> architecture interface
and is appropriately renamed z_arch_is_in_isr().

References from test cases changed to k_is_in_isr().

Signed-off-by: Andrew Boie <andrew.p.boie@intel.com>
2019-09-30 15:25:55 -04:00
Andrew Boie 61901ccb4c kernel: rename z_new_thread()
This is part of the core kernel -> architecture interface
and should have a leading prefix z_arch_.

Signed-off-by: Andrew Boie <andrew.p.boie@intel.com>
2019-09-30 15:25:55 -04:00
Charles E. Youse 66a2ed2360 arch/x86: (Intel64) move RAX to volatile register set
This used to be part of the "restore always" set of registers because
__swap was expected to return a value.  No longer required, so RAX is
moved to the volatile registers and we save a few cycles occasionally.

Signed-off-by: Charles E. Youse <charles.youse@intel.com>
2019-09-23 17:50:09 -07:00
Charles E. Youse 074ce889fb arch/x86: (Intel64) migrate from __swap to z_arch_switch()
The latter primitive is required for SMP.

Signed-off-by: Charles E. Youse <charles.youse@intel.com>
2019-09-23 17:50:09 -07:00
Charles E. Youse 32fc239aa2 arch/x86: (Intel64) use TSS for per-CPU variables
A space is allocated in the TSS for per-CPU variables. At present,
this is only a 'struct _cpu *' to find the _kernel CPU struct. The
locore routines are rewritten to find _current and _nested via this
pointer rather than referencing the _kernel global directly.

This is obviously in preparation for SMP support.

Signed-off-by: Charles E. Youse <charles.youse@intel.com>
2019-09-23 17:50:09 -07:00
Charles E. Youse e4d5ab363c arch/x86: (Intel64) define TSS in C, not assembly
Declare the 64-bit TSS as a struct, and define the instance in C.
Add a data segment selector that overlaps the TSS and keep that
loaded in GS so we can access the TSS via a segment-override prefix.

Signed-off-by: Charles E. Youse <charles.youse@intel.com>
2019-09-23 17:50:09 -07:00
Charles E. Youse a1afde043c arch/x86: share declaration of _interrupt_stack
This is moved from arch/x86/include/ia32/kernel_arch_func.h to the
common header arch/x86/include/kernel_arch_func.h so it can be shared.

Signed-off-by: Charles E. Youse <charles.youse@intel.com>
2019-09-23 17:50:09 -07:00
Charles E. Youse a95c94cfe2 arch/x86/ia32: move IA32 thread state to _thread_arch
There are not enough bits in k_thread.thread_state with SMP enabled,
and the field is (should be) private to the scheduler, anyway. So
move state bits to the _thread_arch where they belong.

While we're at it, refactor some offset data w/r/t _thread_arch
because it can be shared between 32- and 64-bit subarches.

Signed-off-by: Charles E. Youse <charles.youse@intel.com>
2019-09-20 14:31:18 -04:00
Charles E. Youse a224998355 arch/x86/intel64: do not use thread_state for arch data
k_thread.thread_state (or rather, _thread_base.thread_state) should be
private to the kernel/scheduler, so flags previously stored there are
moved to _thread_arch where the belong.

Signed-off-by: Charles E. Youse <charles.youse@intel.com>
2019-09-20 14:31:18 -04:00
Charles E. Youse a5eea17dda arch/x86: add SSE floating-point to Intel64 subarch
This is a naive implementation which does "eager" context switching
for floating-point context, which, of course, introduces performance
concerns. Other approaches have security concerns, SMP implications,
and impact the x86 arch and Zephyr project as a whole. Discussion is
needed, so punting with the straightforward solution for now.

Signed-off-by: Charles E. Youse <charles.youse@intel.com>
2019-09-15 11:33:47 +08:00
Charles E. Youse cdb9ac3895 arch/x86: Add exception reporting code for Intel64
Fleshed out z_arch_esf_t and added code to build this frame when
exceptions occur. Created a separate small stack for exceptions and
shifted the initialization code to use this instead of the IRQ stack.

Moved IRQ stack(s) to irq.c.

Signed-off-by: Charles E. Youse <charles.youse@intel.com>
2019-09-15 11:33:47 +08:00
Charles E. Youse 4ddaa59a89 arch/x86: initial Intel64 support
First "complete" version of Intel64 support for x86. Compilation of
apps for supported boards (read: up_squared) with CONFIG_X86_LONGMODE=y
is now working. Booting, device drivers, interrupts, scheduling, etc.
appear to be functioning properly. Beware that this is ALHPA quality,
not ready for production use, but the port has advanced far enough that
it's time to start working through the test suite and samples, fleshing
out any missing features, and squashing bugs.

Signed-off-by: Charles E. Youse <charles.youse@intel.com>
2019-09-15 11:33:47 +08:00
Charles E. Youse 34307a54f0 arch/x86: initial Intel64 bootstrap framework
This patch adds basic build infrastructure, definitions, a linker
script, etc. to use the Zephyr and 0.10.1 SDK to build a 64-bit
ELF binary suitable for use with GRUB to minimally bootstrap an
Apollo Lake (e.g., UpSquared) board. The resulting binary can hardly
be called a Zephyr kernel as it is lacking most of the glue logic,
but it is a starting point to flesh those out in the x86 tree.

The "kernel" builds with a few harmless warnings, both with GCC from
the Zephyr SDK and with ICC (which is currently being worked on in
a separate branch). These warnings are either related to pointer size
differences (since this is an LP64 build) and/or dummy functions
that will be replaced with working versions shortly.

Signed-off-by: Charles E. Youse <charles.youse@intel.com>
2019-09-15 11:33:47 +08:00
Charles E. Youse c58b28ab0a arch/x86: add placeholders for Intel64 headers
Use different headers for kernel_arch_{func,thread}.h when in
CONFIG_X86_LONGMODE, and add placeholders for Intel64 versions.

Signed-off-by: Charles E. Youse <charles.youse@intel.com>
2019-09-15 11:33:47 +08:00
Charles E. Youse 5e10d590c6 arch/x86: refactor kernel_arch_data.h
Some definitions may be shared between subarchitectures, so refactor
accordingly. The definitions are also modified to separate bits. A
placeholder is created for the Intel64 definitions.

Signed-off-by: Charles E. Youse <charles.youse@intel.com>
2019-09-15 11:33:47 +08:00
Charles E. Youse 6767563f94 arch/x86: remove support for IAMCU ABI
This ABI is no longer required by any targets and is deprecated.

Signed-off-by: Charles E. Youse <charles.youse@intel.com>
2019-09-07 10:07:42 -04:00
Peter Bigot 324203f79b arch/x86: rearrange for standard use of extern "C"
Consistently place C++ use of extern "C" after all include directives,
within the negative branch of _ASMLANGUAGE if used.

Background from issue #17997:

Declarations that use C linkage should be placed within extern "C"
so the language linkage is correct when the header is included by
a C++ compiler.

Similarly #include directives should be outside the extern "C" to
ensure the language-specific default linkage is applied to any
declarations provided by the included header.

See: https://en.cppreference.com/w/cpp/language/language_linkage
Signed-off-by: Peter Bigot <peter.bigot@nordicsemi.no>
2019-08-20 00:49:15 +02:00
Andrew Boie 02629b69b5 x86: add prep_c function
Assembly language start code will enter here, which sets up
early kernel initialization and then calls z_cstart() when
finished.

Signed-off-by: Andrew Boie <andrew.p.boie@intel.com>
2019-08-07 12:50:53 -07:00
Andrew Boie c3b3aafaec x86: generate page tables at runtime
Removes very complex boot-time generation of page tables
with a much simpler runtime generation of them at bootup.

For those x86 boards that enable the MMU in the defconfig,
set the number of page pool pages appropriately.

The MMU_RUNTIME_* flags have been removed. They were an
artifact of the old page table generation and did not
correspond to any hardware state.

Signed-off-by: Andrew Boie <andrew.p.boie@intel.com>
2019-08-07 12:50:53 -07:00
Andrew Boie 0add92523c x86: use a struct to specify stack layout
Makes the code that defines stacks, and code referencing
areas within the stack object, much clearer.

Signed-off-by: Andrew Boie <andrew.p.boie@intel.com>
2019-08-05 13:25:50 +02:00
Andrew Boie 8014e075f4 x86: use per-thread page tables
Previously, context switching on x86 with memory protection
enabled involved walking the page tables, de-configuring all
the partitions in the outgoing thread's memory domain, and
then configuring all the partitions in the incoming thread's
domain, on a global set of page tables.

We now have a much faster design. Each thread has reserved in
its stack object a number of pages to store page directories
and page tables pertaining to the system RAM area. Each
thread also has a toplevel PDPT which is configured to use
the per-thread tables for system RAM, and the global tables
for the rest of the address space.

The result of this is on context switch, at most we just have
to update the CR3 register to the incoming thread's PDPT.

The x86_mmu_api test was making too many assumptions and has
been adjusted to work with the new design.

Signed-off-by: Andrew Boie <andrew.p.boie@intel.com>
2019-08-05 13:25:50 +02:00
Andrew Boie 8915e41b7b userspace: adjust arch memory domain interface
The current API was assuming too much, in that it expected that
arch-specific memory domain configuration is only maintained
in some global area, and updates to domains that are not currently
active have no effect.

This was true when all memory domain state was tracked in page
tables or MPU registers, but no longer works when arch-specific
memory management information is stored in thread-specific areas.

This is needed for: #13441 #13074 #15135

Signed-off-by: Andrew Boie <andrew.p.boie@intel.com>
2019-08-05 13:25:50 +02:00
Andrew Boie fcd2c14500 x86: add functions to get/set page tables
Wrapper to assembly code working with CR3 register.

Signed-off-by: Andrew Boie <andrew.p.boie@intel.com>
2019-08-05 13:25:50 +02:00
Andrew Boie ea201b206f x86: add debug functions for dumping page tables
These turned out to be quite useful when debugging MMU
issues, commit them to the tree. The output format is
virtually the same as gen_mmu_x86.py's verbose output.

Signed-off-by: Andrew Boie <andrew.p.boie@intel.com>
2019-08-05 13:25:50 +02:00
Andrew Boie 26dccaabcb x86: reserve room for per-thread page tables
Currently page tables have to be re-computed in
an expensive operation on context switch. Here we
reserve some room in the page tables such that
we can have per-thread page table data, which will
be much simpler to update on context switch at
the expense of memory.

Signed-off-by: Andrew Boie <andrew.p.boie@intel.com>
2019-08-05 13:25:50 +02:00
Andrew Boie 76310f6896 x86: make guard pages ro instead of non-present
Has the same effect of catching stack overflows, but
makes debugging with GDB simpler since we won't get
errors when inspecting such regions. Making these
areas non-present was more than we needed, read-only
is sufficient.

Signed-off-by: Andrew Boie <andrew.p.boie@intel.com>
2019-08-05 13:25:50 +02:00
Andrew Boie bd709c7322 x86: support very early printk() if desired
Adapted from similar code in the x86_64 port.
Useful when debugging boot problems on actual x86
hardware if a JTAG isn't handy or feasible.

Turn this on for qemu_x86.

Signed-off-by: Andrew Boie <andrew.p.boie@intel.com>
2019-08-02 00:29:21 -07:00
Andrew Boie 96571a8c40 kernel: rename NANO_ESF
This is now called z_arch_esf_t, conforming to our naming
convention.

This needs to remain a typedef due to how our offset generation
header mechanism works.

Signed-off-by: Andrew Boie <andrew.p.boie@intel.com>
2019-07-25 15:06:58 -07:00
Charles E. Youse e96c178e93 arch/x86: refactor offsets_short_arch.h
The current version is 32-bit specific, so move it to ia32/
and add a layer of indirection via an arch-level header file.

Signed-off-by: Charles E. Youse <charles.youse@intel.com>
2019-07-03 20:01:17 -04:00
Charles E. Youse 820ea28f87 arch/x86: move kernel_arch_func.h to ia32/
Refactoring 32- and 64-bit subarchitectures, so this file is moved
to ia32/ and a new "redirector" header file is introduced.

Signed-off-by: Charles E. Youse <charles.youse@intel.com>
2019-07-03 20:01:17 -04:00
Charles E. Youse f40fe36ca6 arch/x86: refactor kernel_arch_thread.h
This data is subarchitecture-specific, so move it to ia32/
and add a layer of indirection at the architecture level.

Signed-off-by: Charles E. Youse <charles.youse@intel.com>
2019-07-03 20:01:17 -04:00
Charles E. Youse aa6d5b43f2 arch/x86: refactor kernel_arch_data.h
Some of this is 32-bit specific, some applies to all subarchitectures.
A preliminary attempt is made to refactor and place 32-bit-specific
portions in ia32/kernel_arch_data.h.

Signed-off-by: Charles E. Youse <charles.youse@intel.com>
2019-07-03 20:01:17 -04:00
Charles E. Youse 0fb9d3450b arch/x86: move exception.h to ia32/exception.h
This file is currently 32-bit specific. Move it and references to it.

Signed-off-by: Charles E. Youse <charles.youse@intel.com>
2019-07-03 20:01:17 -04:00
Charles E. Youse 3ff2746857 arch/x86: eliminate cache_private.h
This file merely declares external functions referenced only
by ia32/cache.c, so the declarations are inlined instead.

Signed-off-by: Charles E. Youse <charles.youse@intel.com>
2019-07-03 20:01:17 -04:00
Charles E. Youse 589b86f534 arch/x86: remove swapstk.h and references to it
This file was used to generate offsets for host tools that are no
longer in use, so it's removed and the offsets are no longer generated.

Signed-off-by: Charles E. Youse <charles.youse@intel.com>
2019-07-03 20:01:17 -04:00
Charles E. Youse b4316fef48 arch/x86: eliminate arch/x86/include/asm_inline.h
Over time, this has been reduced to a few functions dealing solely
with floating-point support, referenced only from core/ia32/float.c.
Thus they are moved into that file and the header is eliminated.

Signed-off-by: Charles E. Youse <charles.youse@intel.com>
2019-07-03 20:01:17 -04:00
Charles E. Youse 7c2d7d7b69 arch/x86: move arch/x86/include/mmustructs.h to ia32/mmustructs.h
For now, only the 32-bit subarchitecture supports memory protection.

Signed-off-by: Charles E. Youse <charles.youse@intel.com>
2019-07-03 20:01:17 -04:00
Charles E. Youse dff016b53c arch/x86: move include/arch/x86/arch.h to ia32/arch.h
Making room for the Intel64 subarch in this tree. This header is
32-bit specific and so it's relocated, and references rewritten
to find it in its new location.

Signed-off-by: Charles E. Youse <charles.youse@intel.com>
2019-07-02 19:30:00 -04:00
Charles E. Youse 8a8e6a1e52 arch/x86: merge asm_inline_gcc.h with asm_inline.h
This pattern exists in both the include/arch/x86 and arch/x86/include
trees. This indirection is historic and unnecessary, as all supported
toolchains for x86 support gas/gcc-style inline assembly.

Signed-off-by: Charles E. Youse <charles.youse@intel.com>
2019-07-02 19:30:00 -04:00
Anas Nashif a2fd7d70ec cleanup: include/: move misc/util.h to sys/util.h
move misc/util.h to sys/util.h and
create a shim for backward-compatibility.

No functional changes to the headers.
A warning in the shim can be controlled with CONFIG_COMPAT_INCLUDES.

Related to #16539

Signed-off-by: Anas Nashif <anas.nashif@intel.com>
2019-06-27 22:55:49 -04:00
Anas Nashif ee9dd1a54a cleanup: include/: move misc/dlist.h to sys/dlist.h
move misc/dlist.h to sys/dlist.h and
create a shim for backward-compatibility.

No functional changes to the headers.
A warning in the shim can be controlled with CONFIG_COMPAT_INCLUDES.

Related to #16539

Signed-off-by: Anas Nashif <anas.nashif@intel.com>
2019-06-27 22:55:49 -04:00
Charles E. Youse 4bdbd879ef arch/x86: remove old PRINTK() debugging macro
This appears to date all the way back to the initial import
and is used in exactly one place if DEBUG is on. Removed.

Signed-off-by: Charles E. Youse <charles.youse@intel.com>
2019-06-24 07:46:24 -07:00
Charles E. Youse 2835c22985 arch/x86: used fixed initial EFLAGS on thread creation
Previously the existing EFLAGS was used as a base which was
then manipulated accordingly. This is unnecessary as the bits
preserved contain no useful state related to the new thread.

Signed-off-by: Charles E. Youse <charles.youse@intel.com>
2019-06-24 07:46:24 -07:00
Charles E. Youse 01b572937b arch/x86: remove CR4_PAE_DISABLE mask
This is never used. The value is incorrect, in any case.

Signed-off-by: Charles E. Youse <charles.youse@intel.com>
2019-06-24 07:46:24 -07:00
Anas Nashif f2cb20c772 docs: fix misspelling across the tree
Found a few annoying typos and figured I better run script and
fix anything it can find, here are the results...

Signed-off-by: Anas Nashif <anas.nashif@intel.com>
2019-06-19 15:34:13 -05:00
Charles E. Youse e9f6cb2594 drivers/timer/loapic_timer.c: migrate to new local APIC accessors
More clearly differentiate MVIC vs. APIC timer code, and use new APIC
accessors in include/drivers/loapic.h. Remove extraneous comments, and
other light cleanup work.

This driver is in need of a serious overhaul -- despite appearing to
have support for TICKLESS_KERNEL and DEVICE_POWER_MANAGEMENT, bitrot
has taken its toll and the driver will not build with these enabled.
These should be removed or made to work... but not in this patch.

Old x2APIC-related accessors in kernel_arch_func.h are eliminated.

Signed-off-by: Charles E. Youse <charles.youse@intel.com>
2019-06-08 15:41:36 -04:00
Charles E. Youse 0fe4e1b3a8 arch/x86: x2APIC support is not specific to jailhouse
Simple renaming and Kconfig reorganization. Choice of local APIC
access method isn't specific to the Jailhouse hypervisor.

Signed-off-by: Charles E. Youse <charles.youse@intel.com>
2019-06-08 15:41:36 -04:00
Charles E. Youse 0e166fa2a8 arch/x86: move MSR definitions to include/arch/x86/msr.h
Light reorganization. All MSR definitions and manipulation functions
are consolidated into one header. The names are changed to use an
X86_* prefix instead of IA32_* which is misleading/incorrect.

Signed-off-by: Charles E. Youse <charles.youse@intel.com>
2019-06-08 15:41:36 -04:00
Ioannis Glaropoulos fd97e44011 arch: remove unused tracing_arch.h
The API in tracing_arch.h is not used by the
kernel so we remove it.

Signed-off-by: Ioannis Glaropoulos <Ioannis.Glaropoulos@nordicsemi.no>
2019-05-13 14:04:15 -07:00
Flavio Ceolin 4f99a38b06 arch: all: Remove not used struct _caller_saved
The struct _caller_saved is not used. Most architectures put
automatically the registers onto stack, in others architectures the
exception code does it.

Signed-off-by: Flavio Ceolin <flavio.ceolin@intel.com>
2019-04-18 12:24:56 -07:00
Flavio Ceolin d61c679d43 arch: all: Remove legacy code
The struct _kernel_ach exists only because ARC' s port needed it, in
all other ports this was defined as an empty struct. Turns out that
this struct is not required even for ARC anymore, this is a legacy
code from nanokernel time.

Signed-off-by: Flavio Ceolin <flavio.ceolin@intel.com>
2019-04-18 12:24:56 -07:00
Andrew Boie 4e5c093e66 kernel: demote K_THREAD_STACK_BUFFER() to private
This macro is slated for complete removal, as it's not possible
on arches with an MPU stack guard to know the true buffer bounds
without also knowing the runtime state of its associated thread.

As removing this completely would be invasive to where we are
in the 1.14 release, demote to a private kernel Z_ API instead.
The current way that the macro is being used internally will
not cause any undue harm, we just don't want any external code
depending on it.

The final work to remove this (and overhaul stack specification in
general) will take place in 1.15 in the context of #14269

Fixes: #14766

Signed-off-by: Andrew Boie <andrew.p.boie@intel.com>
2019-04-05 16:10:02 -04:00
Patrik Flykt 7c0a245d32 arch: Rename reserved function names
Rename reserved function names in arch/ subdirectory. The Python
script gen_priv_stacks.py was updated to follow the 'z_' prefix
naming.

Signed-off-by: Patrik Flykt <patrik.flykt@intel.com>
2019-04-03 17:31:00 -04:00
Flavio Ceolin b80c3d9c77 arch: x86: Remove not used fp struct
The legacy struct s_coopFloatReg was never being used, though it was
an empty struct (not wasting space), some symbols were being generate
for it.

Nevertheless, neither C99 nor C11 allow empty structs, so this
was also a violation to the C standards.

Signed-off-by: Flavio Ceolin <flavio.ceolin@intel.com>
Signed-off-by: Anas Nashif <anas.nashif@intel.com>
2019-04-03 12:06:31 -04:00
Flavio Ceolin 3306a5862f arch: x86: Add missing U for unsigned constants
Add U for integer constants to avoid an implicit cast.

MISRA-C rule 10.1

Signed-off-by: Flavio Ceolin <flavio.ceolin@intel.com>
2019-03-26 22:06:45 -04:00
Flavio Ceolin 39a50f6392 arch: x86: Use proper essential types in operands
MISRA defines a serie of essential types, boolean, signed/unsigned
integers, float, ... and operations must respect these essential types.

MISRA-C rule 10.1

Signed-off-by: Flavio Ceolin <flavio.ceolin@intel.com>
2019-03-26 14:31:29 -04:00
Andrew Boie 50d72ed9c9 x86: implement eager FP save/restore
Speculative execution side channel attacks can read the
entire FPU/SIMD register state on affected Intel Core
processors, see CVE-2018-3665.

We now have two options for managing floating point
context between threads on x86: CONFIG_EAGER_FP_SHARING
and CONFIG_LAZY_FP_SHARING.

The mitigation is to unconditionally save/restore these
registers on context switch, instead of the lazy sharing
algorithm used by CONFIG_LAZY_FP_SHARING.

Signed-off-by: Andrew Boie <andrew.p.boie@intel.com>
2019-03-11 20:36:55 -07:00
Patrik Flykt 4344e27c26 all: Update reserved function names
Update reserved function names starting with one underscore, replacing
them as follows:
   '_k_' with 'z_'
   '_K_' with 'Z_'
   '_handler_' with 'z_handl_'
   '_Cstart' with 'z_cstart'
   '_Swap' with 'z_swap'

This renaming is done on both global and those static function names
in kernel/include and include/. Other static function names in kernel/
are renamed by removing the leading underscore. Other function names
not starting with any prefix listed above are renamed starting with
a 'z_' or 'Z_' prefix.

Function names starting with two or three leading underscores are not
automatcally renamed since these names will collide with the variants
with two or three leading underscores.

Various generator scripts have also been updated as well as perf,
linker and usb files. These are
   drivers/serial/uart_handlers.c
   include/linker/kobject-text.ld
   kernel/include/syscall_handler.h
   scripts/gen_kobject_list.py
   scripts/gen_syscall_header.py

Signed-off-by: Patrik Flykt <patrik.flykt@intel.com>
2019-03-11 13:48:42 -04:00
Andrew Boie f093285345 x86: modify MMU APIs for multiple page tables
Current set of APIs and macros assumed that only one set
of page tables would ever be in use.

Signed-off-by: Andrew Boie <andrew.p.boie@intel.com>
2019-02-14 12:46:36 -05:00
Andrew Boie 2d9bbdf5f3 x86: remove support for non-PAE page tables
PAE tables introduce the NX bit which is very desirable
from a security perspetive, back in 1995.

PAE tables are larger, but we are not targeting x86 memory
protection for RAM constrained devices.

Remove the old style 32-bit tables to make the x86 port
easier to maintain.

Renamed some verbosely named data structures, and fixed
incorrect number of entries for the page directory
pointer table.

Signed-off-by: Andrew Boie <andrew.p.boie@intel.com>
2019-02-05 20:51:21 -08:00
Flavio Ceolin 4b35dd2628 misra: Fixes for MISRA-C rule 8.2
In C90 was introduced function prototype, that allows argument types
to be checked against parameter types, though it is not necessary
specify names for the parameters. MISRA-C requires names for function
prototype parameters, it claims that names can provide useful
information regarding the function interface.

MISRA-C rule 8.2

Signed-off-by: Flavio Ceolin <flavio.ceolin@intel.com>
2018-12-07 09:06:34 -05:00
Ioannis Glaropoulos 39bf24a9bd kernel: expose k_mem_partition_attr_t outside User mode
This commit exposes k_mem_partition_attr_t outside User Mode, so
we can use struct k_mem_partition for defining memory partitions
outside the scope of user space (for example, to describe thread
stack guards or no-cacheable MPU regions). A requirement is that
the Zephyr build supports Memory protection. To signify this, a
new hidden, all-architecture Kconfig symbol is defined (MPU). In
the wake of exposing k_mem_partition_attr_t, the commit exposes
the MPU architecture-specific access permission attribute macros
outside the User space context (for all ARCHs), so they can be
used in a more generic way.

Signed-off-by: Ioannis Glaropoulos <Ioannis.Glaropoulos@nordicsemi.no>
2018-12-05 15:15:07 -05:00
Flavio Ceolin 46715faa5c kernel: Remove _IntLibInit function
There were many platforms where this function was doing nothing. Just
merging its functionality with _PrepC function.

Signed-off-by: Flavio Ceolin <flavio.ceolin@intel.com>
2018-11-28 14:59:10 -08:00
Andrew Boie 7bac15f2ff x86: add dynamic interrupt support
If dynamic interrupts are enabled, a set of trampoline stubs
are generated which transfer control to a common dynamic
interrupt handler function, which then looks up the proper
handler and parameter and then executes the interrupt.

Based on the prior x86 dynamic interrupt implementation which
was removed from the kernel some time ago, and adapted to
changes in the common interrupt handling code, build system,
and IDT generation tools.

An alternative approach could be to read the currently executing
vector out of the APIC, but this is a much slower operation.

Signed-off-by: Andrew Boie <andrew.p.boie@intel.com>
2018-11-10 11:01:22 -05:00