From 714e37fb78081ea31049546a58314b67423da5db Mon Sep 17 00:00:00 2001 From: Andrew Boie Date: Thu, 15 Aug 2019 16:10:49 -0700 Subject: [PATCH] doc: add more details about memory domains We now more throroughly discuss memory domains, thread resource pools, and automatic memory domains. Signed-off-by: Andrew Boie --- doc/reference/usermode/index.rst | 3 +- doc/reference/usermode/memory_domain.rst | 352 ++++++++++++++++-- doc/reference/usermode/usermode_sharedmem.rst | 75 ---- 3 files changed, 316 insertions(+), 114 deletions(-) delete mode 100644 doc/reference/usermode/usermode_sharedmem.rst diff --git a/doc/reference/usermode/index.rst b/doc/reference/usermode/index.rst index 275bd94a4d5..e850ca9af4f 100644 --- a/doc/reference/usermode/index.rst +++ b/doc/reference/usermode/index.rst @@ -182,9 +182,8 @@ for execution after the kernel starts: .. toctree:: :maxdepth: 2 + memory_domain.rst kernelobjects.rst syscalls.rst - memory_domain.rst mpu_stack_objects.rst mpu_userspace.rst - usermode_sharedmem.rst diff --git a/doc/reference/usermode/memory_domain.rst b/doc/reference/usermode/memory_domain.rst index af729316c27..cdcf537a036 100644 --- a/doc/reference/usermode/memory_domain.rst +++ b/doc/reference/usermode/memory_domain.rst @@ -1,40 +1,316 @@ .. _memory_domain: -Memory Domain -############# +Memory Protection Design +######################## -The memory domain APIs are used by unprivileged threads to share data to -the threads in the same memory domain and protect sensitive data from threads -outside their domain. Memory domains are not only used for improving security, -but are also useful for debugging (unexpected access would cause an exception). +Zephyr's memory protection design is geared towards microcontrollers with MPU +(Memory Protection Unit) hardware. We do support some architectures which have +a paged MMU (Memory Management Unit), but in that case the MMU is used like +an MPU with an identity page table. -Since architectures generally have constraints on how many partitions can be -defined, and the size/alignment of each partition, users may need to group -related data together using linker sections. +All of the discussion below will be using MPU terminology; systems with MMUs +can be considered to have an MPU with an unlimited number of programmable +regions. -.. contents:: - :local: - :depth: 2 +There are a few different levels on how memory access is configured when +Zephyr memory protection features are enabled, which we will describe here: -Concepts -******** +Boot Time Memory Configuration +****************************** -A memory domain contains some number of memory partitions. -A memory partition is a memory region (might be RAM, peripheral registers, -or flash, for example) with specific attributes (access permission, e.g. -privileged read/write, unprivileged read-only, or execute never). -Memory partitions are defined by a set of underlying MPU regions -or MMU tables. A thread belongs to a single memory domain at -any point in time but a memory domain may contain multiple threads. -Threads in the same memory domain have the same access permissions -to the memory partitions belonging to the memory domain. New threads -will inherit any memory domain configuration from the parent thread. +This is the configuration of the MPU after the kernel has started up. It should +contain the following: -Implementation +- Any configuration of memory regions which need to have special caching or + write-back policies for basic hardware and driver function. Note that most + MPUs have the concept of a default memory access policy map, which can be + enabled as a "background" mapping for any area of memory that doesn't + have an MPU region configuring it. It is strongly recommended to use this + to maximize the number of available MPU regions for the end user. On + ARMv7-M/ARMv8-M this is called the System Address Map, other CPUs may + have similar capabilities. + +- A read-only, executable region or regions for program text and ro-data, that + is accessible to user mode. This could be further sub-divided into a + read-only region for ro-data, and a read-only, executable region for text, but + this will require an additional MPU region. This is required so that + threads running in user mode can read ro-data and fetch instructions. + +- Depending on configuration, user-accessible read-write regions to support + extra features like GCOV, HEP, etc. + +Assuming there is a background map which allows supervisor mode to access any +memory it needs, and regions are defined which grant user mode access to +text/ro-data, this is sufficient for the boot time configuration. + +Hardware Stack Overflow +*********************** + +``CONFIG_HW_STACK_PROTECTION`` is an optional feature which detects stack +buffer overflows when the system is running in supervisor mode. This +catches issues when the entire stack buffer has overflowed, and not +individual stack frames, use compiler-assisted CONFIG_STACK_CANARIES for that. + +Like any crash in supervisor mode, no guarantees can be made about the overall +health of the system after a supervisor mode stack overflow, and any instances +of this should be treated as a serious error. However it's still very useful to +know when these overflows happen, as without robust detection logic the system +will either crash in mysterious ways or behave in an undefined manner when the +stack buffer overflows. + +Some systems implement this feature by creating at runtime a 'guard' MPU region +which is set to be read-only and is at either the beginning or immediately +preceding the supervisor mode stack buffer. If the stack overflows an +exception will be generated. + +This feature is optional and is not required to catch stack overflows in user +mode; disabling this may free 1-2 MPU regions depending on the MPU design. + +Other systems may have dedicated CPU support for catching stack overflows +and no extra MPU regions will be required. + +Thread Stack +************ + +Any thread running in user mode will need access to its own stack buffer. +On context switch into a user mode thread, a dedicated MPU region will be +programmed with the bounds of the stack buffer. A thread exceeding its stack +buffer will start pushing data onto memory it doesn't have access to and a +memory access violation exception will be generated. + +Thread Resource Pools +********************* + +A small subset of kernel APIs, invoked as system calls, require heap memory +allocations. This memory is used only by the kernel and is not accessible +directly by user mode. In order to use these system calls, invoking threads +must assign themselves to a resource pool, which is a k_mem_pool object. +Memory is drawn from a thread's resource pool using :c:func:`z_thread_malloc()` +and freed with :c:func:`k_free()`. + +The APIs which use resource pools are as follows, with any alternatives +noted for users who do not want heap allocations within their application: + + - :c:func:`k_stack_alloc_init()` sets up a k_stack with its storage + buffer allocated out of a resource pool instead of a buffer provided by the + user. An alternative is to declare k_stacks that are automatically + initialized at boot with :c:macro:`K_STACK_DEFINE()`, or to initialize the + k_stack in supervisor mode with :c:func:`k_stack_init()`. + + - :c:func:`k_pipe_alloc_init()` sets up a k_pipe object with its + storage buffer allocated out of a resource pool instead of a buffer provided + by the user. An alternative is to declare k_pipes that are automatically + initialized at boot with :c:macro:`K_PIPE_DEFINE()`, or to initialize the + k_pipe in supervisor mode with :c:func:`k_pipe_init()`. + + - :c:func:`k_msgq_alloc_init()` sets up a k_msgq object with its + storage buffer allocated out of a resource pool instead of a buffer provided + by the user. An alternative is to declare a k_msgq that is automatically + initialized at boot with :c:macro:`K_MSGQ_DEFINE()`, or to initialize the + k_msgq in supervisor mode with :c:func:`k_msgq_init()`. + + - :c:func:`k_poll()` when invoked from user mode, needs to make a kernel-side + copy of the provided events array while waiting for an event. This copy is + freed when :c:func:`k_poll()` returns for any reason. + + - :c:func:`k_queue_alloc_prepend()` and :c:func:`k_queue_alloc_append()` + allocate a container structure to place the data in, since the internal + bookkeeping information that defines the queue cannot be placed in the + memory provided by the user. + + - :c:func:`k_object_alloc()` allows for entire kernel objects to be + dynamically allocated at runtime and a usable pointer to them returned to + the caller. + +The relevant API is :c:func:`k_thread_resource_pool_assign()` which assigns +a k_mem_pool to draw these allocations from for the target thread. + +If the system heap is enabled, then the system heap may be used with +:c:func:`k_thread_system_pool_assign()`, but it is preferable for different +logical applications running on the system to have their own pools. + +Memory Domains ************** +The kernel ensures that any user thread will have access to its own stack +buffer, plus program text and read-only data. The memory domain APIs are the +way to grant access to additional blocks of memory to a user thread. + +Conceptually, a memory domain is a collection of some number of memory +partitions. The maximum number of memory partitions in a domain +is limited by the number of available MPU regions. This is why it is important +to minimize the number of boot-time MPU regions. + +Memory domains are *not* intended to control access to memory from supervisor +mode. In some cases this may be unavoidable; for example some architectures do +not allow for the definition of regions which are read-only to user mode but +read-write to supervisor mode. A great deal of care must be taken when working +with such regions to not unintentionally cause the kernel to crash when +accessing such a region. + +Memory domain APIs are only available to supervisor mode. The only control +user mode has over memory domains is that any user thread's child threads +will automatically become members of the parent's domain. + +Memory Partitions +================= + +Each memory partition consists of a memory address, a size, +and access attributes. It is intended that memory partitions are used to +control access to system memory. Defining memory partitions are subject +to the following constraints: + +- The partition must represent a memory region that can be programmed by + the underlying memory management hardware, and needs to conform to any + underlying hardware constraints. For example, many MPU-based systems require + that partitions be sized to some power of two, and aligned to their own + size. For MMU-based systems, the partition must be aligned to a page and + the size some multiple of the page size. + +- Partitions within the same memory domain may not overlap each other. There is + no notion of precedence among partitions within a memory domain. Partitions + within a memory domain are assumed to have a higher precedence than any + boot-time memory regions, however whether a memory domain partition can + overlap a boot-time memory region is architecture specific. + +- The same partition may be specified in multiple memory domains. For example + there may be a shared memory area that multiple domains grant access to. + +- Care must be taken in determining what memory to expose in a partition. + It is not appropriate to provide direct user mode access to any memory + containing private kernel data. + +- Memory domain partitions are intended to control access to system RAM. + Configuration of memory partitions which do not correspond to RAM + may not be supported by the architecture; this is true for MMU-based systems. + +There are two ways to define memory partitions: either manually or +automatically. + +Manual Memory Partitions +------------------------ + +The following code declares a global array buf, and then declares +a read-write partition for it which may be added to a domain: + +.. code-block:: c + + u8_t __aligned(32) buf[32]; + + K_MEM_PARTITION_DEFINE(my_partition, buf, sizeof(buf), + K_MEM_PARTITION_P_RW_U_RW); + +This does not scale particularly well when we are trying to contain multiple +objects spread out across several C files into a single partition. + +Automatic Memory Partitions +--------------------------- + +Automatic memory partitions are created by the build system. All globals +which need to be placed inside a partition are tagged with their destination +partition. The build system will then coalesce all of these into a single +contiguous block of memory, zero any BSS variables at boot, and define +a memory partition of appropriate base address and size which contains all +the tagged data. + +Automatic memory partitions are only configured as read-write +regions. They are defined with :c:macro:`K_APPMEM_PARTITION_DEFINE()`. +Global variables are then routed to this partition using +:c:macro:`K_APP_DMEM()` for initialized data and :c:macro:`K_APP_BMEM()` for +BSS. + +.. code-block:: c + + #include + + /* Declare a k_mem_partition "my_partition" that is read-write to + * user mode. Note that we do not specify a base address or size. + */ + K_APPMEM_PARTITION_DEFINE(my_partition); + + /* The global variable var1 will be inside the bounds of my_partition + * and be initialized with 37 at boot. + */ + K_APP_DMEM(my_partition) int var1 = 37; + + /* The global variable var2 will be inside the bounds of my_partition + * and be zeroed at boot size K_APP_BMEM() was used, indicating a BSS + * variable. + */ + K_APP_BMEM(my_partition) int var2; + +The build system will ensure that the base address of my_partition will +be properly aligned, and the total size of the region conforms to the memory +management hardware requirments, adding padding if necessary. + +If multiple partitions are being created, a variadic preprocessor macro can be +used as provided in ``app_macro_support.h``: + +.. code-block:: c + + FOR_EACH(K_APPMEM_PARTITION_DEFINE, part0, part1, part2); + +There are some kernel objects which are defined by macros and take an argument +for a destination section. A good example of these are sys_mem_pools, which +are heap objects. The destination section name for an automatic partition +can be obtained with :c:macro:`K_APP_DMEM_SECTION()` and +:c:macro:`K_APP_BMEM_SECTION()` respectively for initialized data and BSS: + +.. code-block:: c + + /* Declare automatic memory partition foo_partition */ + K_APPMEM_PARTITION_DEFINE(foo_partition); + + /* Section argument for the destination section obtained via + * K_APP_DMEM_SECTION() + */ + SYS_MEM_POOL_DEFINE(foo_pool, NULL, BLK_SIZE_MIN, BLK_SIZE_MAX, + BLK_NUM_MAX, BLK_ALIGN, + K_APP_DMEM_SECTION(foo_partition)); + +Automatic Partitions for Static Library Globals +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +The build-time logic for setting up automatic memory partitions is in +``scripts/gen_app_partitions.py``. If a static library is linked into Zephyr, +it is possible to route all the globals in that library to a specific +memory partition with the ``--library`` argument. + +For example, if the Newlib C library is enabled, the Newlib globals all need +to be placed in ``z_libc_partition``. The invocation of the script in the +top-level ``CMakeLists.txt`` adds the following: + +.. code-block:: none + + gen_app_partitions.py ... --library libc.a z_libc_partition .. + +There is no support for expressing this in the project-level configuration +or build files; the toplevel ``CMakeLists.txt`` must be edited. + +Pre-defined Memory Partitions +----------------------------- + +There are a few memory partitions which are pre-defined by the system: + + - ``z_malloc_partition`` - This partition contains the system-wide pool of + memory used by libc malloc(). Due to possible starvation issues, it is + not recommended to draw heap memory from a global pool, instead + it is better to define various sys_mem_pool objects and assign them + to specific memory domains. + + - ``z_libc_partition`` - Contains globals required by the C library and runtime. + Required if using newlib. Required if using minimal libc with + ``CONFIG_STACK_CANARIES`` enabled. + +Library-specific partitions are listed in ``include/app_memory/partitions.h``. +For example, to use the MBEDTLS library from user mode, the +``k_mbedtls_partition`` must be added to the domain. + +Memory Domain Usage +=================== + Create a Memory Domain -====================== +---------------------- A memory domain is defined using a variable of type :c:type:`struct k_mem_domain`. It must then be initialized by calling @@ -49,7 +325,7 @@ The following code defines and initializes an empty memory domain. k_mem_domain_init(&app0_domain, 0, NULL); Add Memory Partitions into a Memory Domain -========================================== +------------------------------------------ There are two ways to add memory partitions into a memory domain. @@ -97,20 +373,22 @@ memory domain one by one. The maximum number of memory partitions is limited by the maximum number of MPU regions or the maximum number of MMU tables. -Add Threads into a Memory Domain -================================ +Memory Domain Assignment +------------------------ -Adding threads into a memory domain grants threads permission to access -the memory partitions in the memory domain. - -The following code shows how to add threads into a memory domain. +Any thread may join a memory domain, and any memory domain may have multiple +threads assigned to it. Threads are assigned to memory domains with an API +call: .. code-block:: c k_mem_domain_add_thread(&app0_domain, app_thread_id); +In addition, if a thread is a member of a memory domain, and it creates a +child thread, that thread will belong to the domain as well. + Remove a Memory Partition from a Memory Domain -============================================== +---------------------------------------------- The following code shows how to remove a memory partition from a memory domain. @@ -124,7 +402,7 @@ that matches the given parameter and removes that partition from the memory domain. Remove a Thread from the Memory Domain -====================================== +-------------------------------------- The following code shows how to remove a thread from the memory domain. @@ -133,7 +411,7 @@ The following code shows how to remove a thread from the memory domain. k_mem_domain_remove_thread(app_thread_id); Destroy a Memory Domain -======================= +----------------------- The following code shows how to destroy a memory domain. @@ -142,7 +420,7 @@ The following code shows how to destroy a memory domain. k_mem_domain_destroy(&app0_domain); Available Partition Attributes -============================== +------------------------------ When defining a partition, we need to set access permission attributes to the partition. Since the access control of memory partitions relies on diff --git a/doc/reference/usermode/usermode_sharedmem.rst b/doc/reference/usermode/usermode_sharedmem.rst deleted file mode 100644 index 75dbfa7f6e9..00000000000 --- a/doc/reference/usermode/usermode_sharedmem.rst +++ /dev/null @@ -1,75 +0,0 @@ -.. _usermode_sharedmem: - -Application Shared Memory -######################### - -.. note:: - - In this document, we will cover the basic usage of enabling shared - memory using a template around app_memory subsystem. - -Overview -******** - -The use of subsystem app_memory in userspace allows control of -shared memory between threads. The foundation of the implementation -consists of memory domains and partitions. Memory partitions are created -and used in the definition of variable to group them into a -common space. The memory partitions are linked to domains -that are then assigned to a thread. The process allows selective -access to memory from a thread and sharing of memory between two -threads by assigning a partition to two different domains. By using -the shared memory template, code to protect memory can be used -on different platform without the application needing to implement -specific handlers for each platform. Note the developer should understand -the hardware limitations in context to the maximum number of memory -partitions available to a thread. Specifically processors with MPU's -cannot support the same number of partitions as a MMU. - -This specific implementation adds a wrapper to simplify the programmers -task of using the app_memory subsystem through the use of macros and -a python script to generate the linker script. The linker script provides -the proper alignment for processors requiring power of two boundaries. -Without the wrapper, a developer is required to implement custom -linker scripts for each processor in the project. - -The general usage is as follows. Include app_memory/app_memdomain.h -in the userspace source file. Mark the variable to be placed in -a memory partition. The two markers are for data and bss respectively: -K_APP_DMEM(id) and K_APP_BMEM(id). The id is used as the partition name. -The resulting section name can be seen in the linker.map as -"data_smem_id" and "data_smem_idb". - -To create a k_mem_partition, call the macro K_APPMEM_PARTITION_DEFINE(part0) -where "part0" is the name then used to refer to that partition. The -standard memory domain APIs may be used to add it to domains; the declared -name is a k_mem_partition symbol. - -Example: - -.. code-block:: c - - /* create partition at top of file outside functions */ - K_APPMEM_PARTITION_DEFINE(part0); - /* create domain */ - struct k_mem_domain dom0; - /* assign variables to the domain */ - K_APP_DMEM(part0) int var1; - K_APP_BMEM(part0) static volatile int var2; - - int main() - { - k_mem_domain_init(&dom0, 0, NULL) - k_mem_domain_add_partition(&dom0, part0); - k_mem_domain_add_thread(&dom0, k_current_get()); - ... - } - -If multiple partitions are being created, a variadic -preprocessor macro can be used as provided in -app_macro_support.h: - -.. code-block:: c - - FOR_EACH(K_APPMEM_PARTITION_DEFINE, part0, part1, part2); -