Commit graph

1902 commits

Author SHA1 Message Date
Andrew Boie
708d5f7922 x86: don't use privilege stack areas as a guard
This is causing problems, as if we create a thread in
a system call we will *not* be using the kernel page
tables if CONFIG_KPTI=n.

Just don't fiddle with this page's permissions; we don't
need it as a guard area anyway since we have a stack
guard placed immediately before it, and this page
is unused if user mode isn't active.

Signed-off-by: Andrew Boie <andrew.p.boie@intel.com>
2020-01-13 16:35:10 -05:00
Andrew Boie
06c4207602 x86: add CONFIG_X86_USERSPACE for Intel64
Hidden config to select dependencies.

Signed-off-by: Andrew Boie <andrew.p.boie@intel.com>
2020-01-13 16:35:10 -05:00
Andrew Boie
edc14e50ad x86: up-level speculative attack mitigations
These are now part of the common Kconfig and we
build spec_ctrl.c for all.

Signed-off-by: Andrew Boie <andrew.p.boie@intel.com>
2020-01-13 16:35:10 -05:00
Andrew Boie
c71e66e2a5 x86: add system call functions for 64-bit
Nothing too fancy here, we try as much as possible to
use the same register layout as the C calling convention.

Signed-off-by: Andrew Boie <andrew.p.boie@intel.com>
2020-01-13 16:35:10 -05:00
Andrew Boie
7f82b99ad4 x86: up-level some user mode functions
These are now common code, all are related to user mode
threads. The rat's nest of ifdefs in ia32's arch_new_thread
has been greatly simplified, there is now just one hook
if user mode is turned on.

Signed-off-by: Andrew Boie <andrew.p.boie@intel.com>
2020-01-13 16:35:10 -05:00
Andrew Boie
7ea958e0dd x86: optimize locations of psp and thread ptables
z_x86_thread_page_tables_get() now works for both user
and supervisor threads, returning the kernel page tables
in the latter case. This API has been up-leveled to
a common header.

The per-thread privilege elevation stack initial stack
pointer, and the per-thread page table locations are no
longer computed from other values, and instead are stored
in thread->arch.

A problem where the wrong page tables were dumped out
on certain kinds of page faults has been fixed.

Signed-off-by: Andrew Boie <andrew.p.boie@intel.com>
2020-01-13 16:35:10 -05:00
Andrew Boie
e45c6eeebc x86: expose APIs for dumping MMU entry flags
Add two new non-static APIs for dumping out the
page table entries for a specified memory address,
and move to the main MMU code. Has debugging uses
when trying to figure out why memory domains are not
set up correctly.

Signed-off-by: Andrew Boie <andrew.p.boie@intel.com>
2020-01-13 16:35:10 -05:00
Andrew Boie
7c293831c6 x86: add support for 64-bit thread ptables
Slightly different layout since the top-lebel PML4
is page-sized and must be page-aligned.

Signed-off-by: Andrew Boie <andrew.p.boie@intel.com>
2020-01-13 16:35:10 -05:00
Andrew Boie
fc589d7279 x86: implement 64-bit exception recovery logic
The esf has a different set of members on 64-bit.

Signed-off-by: Andrew Boie <andrew.p.boie@intel.com>
2020-01-13 16:35:10 -05:00
Andrew Boie
ded0185eb8 x86: add GDT descriptors for user mode
These are arranged in the particular order required
by the syscall/sysret instructions.

Signed-off-by: Andrew Boie <andrew.p.boie@intel.com>
2020-01-13 16:35:10 -05:00
Andrew Boie
692fda47fc x86: use MSRs for %gs
We don't need to set up GDT data descriptors for setting
%gs. Instead, we use the x86 MSRs to set GS_BASE and
KERNEL_GS_BASE.

We don't currently allow user mode to set %gs on its own,
but later on if we do, we have everything set up to issue
'swapgs' instructions on syscall or IRQ.

Unused entries in the GDT have been removed.

Signed-off-by: Andrew Boie <andrew.p.boie@intel.com>
2020-01-13 16:35:10 -05:00
Andrew Boie
3256e9e00b x86: use BIT() macros for cr0/cr4 bits
Easier to establish correspondence with the technical
manuals.

Signed-off-by: Andrew Boie <andrew.p.boie@intel.com>
2020-01-13 16:35:10 -05:00
Andrew Boie
10d033ebf0 x86: enable recoverable exceptions on 64-bit
These were previously assumed to always be fatal.
We can't have the faulting thread's XMM registers
clobbered, so put the SIMD/FPU state onto the stack
as well. This is fairly large (512 bytes) and the
execption stack is already uncomfortably small, so
increase to 2K.

Signed-off-by: Andrew Boie <andrew.p.boie@intel.com>
2020-01-13 16:35:10 -05:00
Ulf Magnusson
bd9962d8d9 kconfig: Remove '# hidden' comments on promptless symbols
Same deal as in commit 41713244b3 ("kconfig: Remove '# Hidden' comments
on promptless symbols"). I forgot to do a case-insensitive search.

Signed-off-by: Ulf Magnusson <Ulf.Magnusson@nordicsemi.no>
2020-01-03 11:38:40 +01:00
Andrew Boie
1b45de8010 x86: fix stack traces on x86_64
Runtime stack traces (at least as currently implemented)
don't work on x86_64 normally as RBP is treated as a general-
purpose register. Depend on CONFIG_NO_OPTIMIZATIONS to enable
this on 64-bit.

Signed-off-by: Andrew Boie <andrew.p.boie@intel.com>
2019-12-18 08:15:54 -05:00
Andrew Boie
7e014f6b30 x86: up-level QEMU arch_system_halt()
qemu_x86_64 will exit the emulator on a fatal system error,
like qemu_x86 already does.

Improves CI times when tests fail since sanitycheck will not
need to wait for the timeout to expire.

Signed-off-by: Andrew Boie <andrew.p.boie@intel.com>
2019-12-17 18:56:54 -08:00
Andrew Boie
2b67ca8ac9 x86: improve exception debugging
We now dump more information for less common cases,
and this is now centralized code for 32-bit/64-bit.
All of this code is now correctly wrapped around
CONFIG_EXCEPTION_DEBUG. Some cruft and unused defines
removed.

Signed-off-by: Andrew Boie <andrew.p.boie@intel.com>
2019-12-17 11:39:22 -08:00
Andrew Boie
a824821b86 kernel: fix k_mem_partition data types
We need a size_t and not a u32_t for partition sizes,
for 64-bit compatibility.

Additionally, app_memdomain.h was also casting the base
address to a u32_t instead of a uintptr_t.

Signed-off-by: Andrew Boie <andrew.p.boie@intel.com>
2019-12-12 14:48:42 -08:00
Kumar Gala
24ae1b1aa7 include: Fix use of <misc/FOO.h> -> <sys/FOO.h>
Fix #include <misc/FOO.h> as misc/FOO.h has been deprecated and
should be #include <sys/FOO.h>.

Signed-off-by: Kumar Gala <kumar.gala@linaro.org>
2019-12-10 08:39:37 -05:00
Flavio Ceolin
4afa181831 x86: fatal: Explicitely include syscall header
This file is using definitions from ia32/syscall.h, just properly
including this file.

Signed-off-by: Flavio Ceolin <flavio.ceolin@intel.com>
2019-12-09 12:47:15 -05:00
Andrew Boie
9849c7d3e7 x86: don't use privilege stack areas as a guard
This is causing problems, as if we create a thread in
a system call we will *not* be using the kernel page
tables if CONFIG_KPTI=n, resulting in a crash when
the later call to copy_page_tables() tries to initialize
the PDPT (which is in the same page as the privilege
stack).

Just don't fiddle with this page's permissions; we don't
need it as a guard area anyway since we have a stack
guard placed immediately before it, and this page
is unused if user mode isn't active.

Signed-off-by: Andrew Boie <andrew.p.boie@intel.com>
2019-12-09 12:40:31 -05:00
Ulf Magnusson
87e917a925 kconfig: Remove redundant 'default n' and 'prompt' properties
Bool symbols implicitly default to 'n'.

A 'default n' can make sense e.g. in a Kconfig.defconfig file, if you
want to override a 'default y' on the base definition of the symbol. It
isn't used like that on any of these symbols though.

Also replace some

    config
    	prompt "foo"
    	bool/int

with the more common shorthand

    config
    	bool/int "foo"

See the 'Style recommendations and shorthands' section in
https://docs.zephyrproject.org/latest/guides/kconfig/index.html.

Signed-off-by: Ulf Magnusson <Ulf.Magnusson@nordicsemi.no>
2019-12-09 16:14:50 +01:00
Andrew Boie
1c97851726 x86: enable MMU on 64-bit with SMP
The races are believed to be resolved with the patch to
irq_offload(). Allow the MMU to be turned on and enable
it for qemu_x86_64.

Signed-off-by: Andrew Boie <andrew.p.boie@intel.com>
2019-11-08 15:16:43 -08:00
Stephanos Ioannidis
9695763f5f arch: x86: Inline direct ISR functions.
This commit inlines the direct ISR functions that were previously
implemented in irq_manage.c, since the PR #20119 resolved the circular
dependency between arch.h and kernel_structs.h described in the issue
#3056.

Signed-off-by: Stephanos Ioannidis <root@stephanos.io>
2019-11-08 15:50:23 +01:00
Andrew Boie
4f77c2ad53 kernel: rename z_arch_ to arch_
Promote the private z_arch_* namespace, which specifies
the interface between the core kernel and the
architecture code, to a new top-level namespace named
arch_*.

This allows our documentation generation to create
online documentation for this set of interfaces,
and this set of interfaces is worth treating in a
more formal way anyway.

Signed-off-by: Andrew Boie <andrew.p.boie@intel.com>
2019-11-07 15:21:46 -08:00
Andrew Boie
0b27cacabd x86: up-level page fault handling
Now in common code for 32-bit and 64-bit.

Signed-off-by: Andrew Boie <andrew.p.boie@intel.com>
2019-11-07 11:19:38 -05:00
Andrew Boie
7d1ae023f8 x86: enable stack overflow detection on 64-bit
CONFIG_HW_STACK_PROTECTION is now available.

Signed-off-by: Andrew Boie <andrew.p.boie@intel.com>
2019-11-06 17:50:34 -08:00
Andrew Boie
9fb89ccf32 x86: consolidate some fatal error code
Some code for unwinding stacks and z_x86_fatal_error()
now in a common C file, suitable for both modes.

Signed-off-by: Andrew Boie <andrew.p.boie@intel.com>
2019-11-06 17:50:34 -08:00
Andrew Boie
64f6e2ac6b x86: consolidate STACK_ROUND_* definition
There was no definition for 64-bit.

Signed-off-by: Andrew Boie <andrew.p.boie@intel.com>
2019-11-06 17:50:34 -08:00
Stephanos Ioannidis
7bcfdadf81 arch: Simplify private header include path configuration.
When compiling the components under the arch directory, the compiler
include paths for arch and kernel private headers need to be specified.

This was previously done by adding 'zephyr_library_include_directories'
to CMakeLists.txt file for every component under the arch directory,
and this resulted in a significant amount of duplicate code.

This commit uses the CMake 'include_directories' command in the root
CMakeLists.txt to simplify specification of the private header include
paths for all the arch components.

Signed-off-by: Stephanos Ioannidis <root@stephanos.io>
2019-11-06 16:07:32 -08:00
Stephanos Ioannidis
2d7460482d headers: Refactor kernel and arch headers.
This commit refactors kernel and arch headers to establish a boundary
between private and public interface headers.

The refactoring strategy used in this commit is detailed in the issue

This commit introduces the following major changes:

1. Establish a clear boundary between private and public headers by
  removing "kernel/include" and "arch/*/include" from the global
  include paths. Ideally, only kernel/ and arch/*/ source files should
  reference the headers in these directories. If these headers must be
  used by a component, these include paths shall be manually added to
  the CMakeLists.txt file of the component. This is intended to
  discourage applications from including private kernel and arch
  headers either knowingly and unknowingly.

  - kernel/include/ (PRIVATE)
    This directory contains the private headers that provide private
   kernel definitions which should not be visible outside the kernel
   and arch source code. All public kernel definitions must be added
   to an appropriate header located under include/.

  - arch/*/include/ (PRIVATE)
    This directory contains the private headers that provide private
   architecture-specific definitions which should not be visible
   outside the arch and kernel source code. All public architecture-
   specific definitions must be added to an appropriate header located
   under include/arch/*/.

  - include/ AND include/sys/ (PUBLIC)
    This directory contains the public headers that provide public
   kernel definitions which can be referenced by both kernel and
   application code.

  - include/arch/*/ (PUBLIC)
    This directory contains the public headers that provide public
   architecture-specific definitions which can be referenced by both
   kernel and application code.

2. Split arch_interface.h into "kernel-to-arch interface" and "public
  arch interface" divisions.

  - kernel/include/kernel_arch_interface.h
    * provides private "kernel-to-arch interface" definition.
    * includes arch/*/include/kernel_arch_func.h to ensure that the
     interface function implementations are always available.
    * includes sys/arch_interface.h so that public arch interface
     definitions are automatically included when including this file.

  - arch/*/include/kernel_arch_func.h
    * provides architecture-specific "kernel-to-arch interface"
     implementation.
    * only the functions that will be used in kernel and arch source
     files are defined here.

  - include/sys/arch_interface.h
    * provides "public arch interface" definition.
    * includes include/arch/arch_inlines.h to ensure that the
     architecture-specific public inline interface function
     implementations are always available.

  - include/arch/arch_inlines.h
    * includes architecture-specific arch_inlines.h in
     include/arch/*/arch_inline.h.

  - include/arch/*/arch_inline.h
    * provides architecture-specific "public arch interface" inline
     function implementation.
    * supersedes include/sys/arch_inline.h.

3. Refactor kernel and the existing architecture implementations.

  - Remove circular dependency of kernel and arch headers. The
   following general rules should be observed:

    * Never include any private headers from public headers
    * Never include kernel_internal.h in kernel_arch_data.h
    * Always include kernel_arch_data.h from kernel_arch_func.h
    * Never include kernel.h from kernel_struct.h either directly or
     indirectly. Only add the kernel structures that must be referenced
     from public arch headers in this file.

  - Relocate syscall_handler.h to include/ so it can be used in the
   public code. This is necessary because many user-mode public codes
   reference the functions defined in this header.

  - Relocate kernel_arch_thread.h to include/arch/*/thread.h. This is
   necessary to provide architecture-specific thread definition for
   'struct k_thread' in kernel.h.

  - Remove any private header dependencies from public headers using
   the following methods:

    * If dependency is not required, simply omit
    * If dependency is required,
      - Relocate a portion of the required dependencies from the
       private header to an appropriate public header OR
      - Relocate the required private header to make it public.

This commit supersedes #20047, addresses #19666, and fixes #3056.

Signed-off-by: Stephanos Ioannidis <root@stephanos.io>
2019-11-06 16:07:32 -08:00
Ulf Magnusson
bd6e04411e kconfig: Clean up header comments and make them consistent
Use this short header style in all Kconfig files:

    # <description>

    # <copyright>
    # <license>

    ...

Also change all <description>s from

    # Kconfig[.extension] - Foo-related options

to just

    # Foo-related options

It's clear enough that it's about Kconfig.

The <description> cleanup was done with this command, along with some
manual cleanup (big letter at the start, etc.)

    git ls-files '*Kconfig*' | \
        xargs sed -i -E '1 s/#\s*Kconfig[\w.-]*\s*-\s*/# /'

Signed-off-by: Ulf Magnusson <Ulf.Magnusson@nordicsemi.no>
2019-11-04 17:31:27 -05:00
Daniel Leung
b7eb04b300 x86: consolidate x86_64 architecture, SoC and boards
There are two set of code supporting x86_64: x86_64 using x32 ABI,
and x86 long mode, and this consolidates both into one x86_64
architecture and SoC supporting truly 64-bit mode.

() Removes the x86_64:x32 architecture and SoC, and replaces
   them with the existing x86 long mode arch and SoC.
() Replace qemu_x86_64 with qemu_x86_long as qemu_x86_64.
() Updates samples and tests to remove reference to
   qemu_x86_long.
() Renames CONFIG_X86_LONGMODE to CONFIG_X86_64.

Signed-off-by: Daniel Leung <daniel.leung@intel.com>
2019-10-25 17:57:55 -04:00
Andrew Boie
abf5157fdd REVERTME: x86: disable X86_MMU if MP_NUM_CPUS != 1
We are working through some SMP issues with this enabled,
prevent this for now.

Signed-off-by: Andrew Boie <andrew.p.boie@intel.com>
2019-10-24 12:48:45 -07:00
Andrew Boie
e1f2292895 x86: intel64: set page tables for every CPU
The page tables to use are now stored in the cpuboot struct.
For the first CPU, we set to the flat page tables, and then
update later in z_x86_prep_c() once the runtime tables have
been generated.

For other CPUs, by the time we get to z_arch_start_cpu()
the runtime tables are ready do go, and so we just install
them directly.

Signed-off-by: Andrew Boie <andrew.p.boie@intel.com>
2019-10-24 12:48:45 -07:00
Andrew Boie
f6e82ea1bd x86: generate runtime 64-bit page tables
- Bring in CONFIG_X86_MMU and some related defines to
  common X86 Kconfig
- Don't set ARCH_HAS_USERSPACE for intel64 yet when
  X86_MMU is enabled
- Uplevel x86_mmu.c to common code
- Add logic for handling PML4 table and generating PDPTs
- move z_x86_paging_init() to common kernel_arch_func.h
- Uplevel inclusion of mmustructs.h to common x86 arch.h,
  both need it for memory domain defines

Signed-off-by: Andrew Boie <andrew.p.boie@intel.com>
2019-10-24 12:48:45 -07:00
Andrew Boie
4c0d044863 x86: mmustructs: use Z_STRUCT_SECTION_ITERABLE()
This does the right thing for arches with 8-byte words.

Signed-off-by: Andrew Boie <andrew.p.boie@intel.com>
2019-10-24 12:48:45 -07:00
Andrew Boie
46540f4bb1 intel64: don't set global flag in ptables
May cause issues when runtime page tables are installed.

Signed-off-by: Andrew Boie <andrew.p.boie@intel.com>
2019-10-24 12:48:45 -07:00
Andrew Boie
cdd721db3b locore: organize data by type
Program text, rodata, and data need different MMU
permissions. Split out rodata and data from the program
text, updating the linker script appropriately.

Region size symbols added to the linker script, so these
can later be used with MMU_BOOT_REGION().

Signed-off-by: Andrew Boie <andrew.p.boie@intel.com>
2019-10-24 12:48:45 -07:00
Andrew Boie
55cb961878 x86: arm: rename some functions
z_arch_ is only for those APIs in the arch interface.
Other stuff needs to be renamed.

Signed-off-by: Andrew Boie <andrew.p.boie@intel.com>
2019-10-21 10:13:38 -07:00
Andrew Boie
979b17f243 kernel: activate arch interface headers
Duplicate definitions elsewhere have been removed.

A couple functions which are defined by the arch interface
to be non-inline, but were implemented inline by native_posix
and intel64, have been moved to non-inline.

Some missing conditional compilation for z_arch_irq_offload()
has been fixed, as this is an optional feature.

Some massaging of native_posix headers to get everything
in the right scope.

Signed-off-by: Andrew Boie <andrew.p.boie@intel.com>
2019-10-21 10:13:38 -07:00
Andrew Boie
f82beb984d x86: remove unused thread arch member
The PDPT was moved to the stack area since it has alignment
requirements, but never removed from here.

Signed-off-by: Andrew Boie <andrew.p.boie@intel.com>
2019-10-19 20:40:26 -07:00
Andy Ross
8bc3b6f673 arch/x86/intel64: Fix assumption with dummy threads
The intel64 switch implementation doesn't actually use a switch handle
per se, just the raw thread struct pointers which get stored into the
handle field.  This works fine for normally initialized threads, but
when switching out of a dummy thread at initialization, nothing has
initialized that field and the code was dumping registers into the
bottom of memory through the resulting NULL pointer.

Fix this by skipping the load of the field value and just using an
offset instead to get the struct address, which is actually slightly
faster anyway (a SUB immediate instruction vs. the load).

Actually for extra credit we could even move the switch_handle field
to the top of the thread struct and eliminate the instruction
entirely, though if we did that it's probably worth adding some
conditional code to make the switch_handle field disappear entirely.

Signed-off-by: Andy Ross <andrew.j.ross@intel.com>
2019-10-19 12:09:32 -07:00
Andrew Boie
9c76d28bee x86: intel64: fatal: minor formatting improvements
Line up everything nicely, add leading '0x' to hex
addresses, and remove redundant newlines. Add
whitespace between the register name and contents
so the contents can be easily selected from a terminal.

Signed-off-by: Andrew Boie <andrew.p.boie@intel.com>
2019-10-15 09:00:49 -07:00
Andrew Boie
24958f30d9 x86: move z_x86_early_serial_init()
This works with long mode as well, uplevel to
common kernel_arch_func.h.

Signed-off-by: Andrew Boie <andrew.p.boie@intel.com>
2019-10-15 09:00:49 -07:00
Andrew Boie
fdd3ba896a x86: intel64: set the WP bit for paging
Otherwise, supervisor mode can write to read-only areas,
failing tests that check this.

Signed-off-by: Andrew Boie <andrew.p.boie@intel.com>
2019-10-15 09:00:49 -07:00
Andrew Boie
31620b90e2 x86: refactor mmustructs.h
The struct definitions for pdpt, pd, and pt entries has been
removed:

 - Bitfield ordering in a struct is implementation dependent,
   it can be right-to-left or left-to-right
 - The two different structures for page directory entries were
   not being used consistently, or when the type of the PDE
   was unknown
 - Anonymous structs/unions are GCC extensions

Instead these are now u64_t, with bitwise operations used to
get/set fields.

A new set of inline functions for fetcing various page table
structures has been implemented, replacing the older macros.

Signed-off-by: Andrew Boie <andrew.p.boie@intel.com>
2019-10-14 11:49:39 -07:00
Andrew Boie
ab4d647e6d x86: mmu: get rid of x86_page_entry_data_t typedef
This hasn't been necessary since we dropped support for 32-bit
non-PAE page tables. Replace it with u64_t and scrub any
unnecessary casts left behind.

Signed-off-by: Andrew Boie <andrew.p.boie@intel.com>
2019-10-14 11:49:39 -07:00
Andrew Boie
e3ab43580c x86: move mmustructs.h
This will be used for both 32-bit and 64-bit mode.
This header gets pulled in by x86's arch/cpu.h, so put
it in include/arch/x86/.

Signed-off-by: Andrew Boie <andrew.p.boie@intel.com>
2019-10-14 11:49:39 -07:00
Andrew Boie
8ffff144ea kernel: add architecture interface headers
include/sys/arch_inlines.h will contain all architecture APIs
that are used by public inline functions and macros,
with implementations deriving from include/arch/cpu.h.

kernel/include/arch_interface.h will contain everything
else, with implementations deriving from
arch/*/include/kernel_arch_func.h.

Instances of duplicate documentation for these APIs have been
removed; implementation details have been left in place.

Signed-off-by: Andrew Boie <andrew.p.boie@intel.com>
2019-10-11 13:30:46 -07:00