doc: add more details about memory domains

We now more throroughly discuss memory domains, thread
resource pools, and automatic memory domains.

Signed-off-by: Andrew Boie <andrew.p.boie@intel.com>

doc/reference/usermode/index.rst

@@ -182,9 +182,8 @@ for execution after the kernel starts:
 .. toctree::
     :maxdepth: 2
 
+    memory_domain.rst
     kernelobjects.rst
     syscalls.rst
-    memory_domain.rst
     mpu_stack_objects.rst
     mpu_userspace.rst
-    usermode_sharedmem.rst

doc/reference/usermode/memory_domain.rst

@@ -1,40 +1,316 @@
 .. _memory_domain:
 
-Memory Domain
+Memory Protection Design
-#############
+########################
 
-The memory domain APIs are used by unprivileged threads to share data to
+Zephyr's memory protection design is geared towards microcontrollers with MPU
-the threads in the same memory domain and protect sensitive data from threads
+(Memory Protection Unit) hardware. We do support some architectures which have
-outside their domain. Memory domains are not only used for improving security,
+a paged MMU (Memory Management Unit), but in that case the MMU is used like
-but are also useful for debugging (unexpected access would cause an exception).
+an MPU with an identity page table.
 
-Since architectures generally have constraints on how many partitions can be
+All of the discussion below will be using MPU terminology; systems with MMUs
-defined, and the size/alignment of each partition, users may need to group
+can be considered to have an MPU with an unlimited number of programmable
-related data together using linker sections.
+regions.
 
-.. contents::
+There are a few different levels on how memory access is configured when
-    :local:
+Zephyr memory protection features are enabled, which we will describe here:
-    :depth: 2
 
-Concepts
+Boot Time Memory Configuration
-********
+******************************
 
-A memory domain contains some number of memory partitions.
+This is the configuration of the MPU after the kernel has started up. It should
-A memory partition is a memory region (might be RAM, peripheral registers,
+contain the following:
-or flash, for example) with specific attributes (access permission, e.g.
-privileged read/write, unprivileged read-only, or execute never).
-Memory partitions are defined by a set of underlying MPU regions
-or MMU tables. A thread belongs to a single memory domain at
-any point in time but a memory domain may contain multiple threads.
-Threads in the same memory domain have the same access permissions
-to the memory partitions belonging to the memory domain. New threads
-will inherit any memory domain configuration from the parent thread.
 
-Implementation
+- Any configuration of memory regions which need to have special caching or
+  write-back policies for basic hardware and driver function. Note that most
+  MPUs have the concept of a default memory access policy map, which can be
+  enabled as a "background" mapping for any area of memory that doesn't
+  have an MPU region configuring it. It is strongly recommended to use this
+  to maximize the number of available MPU regions for the end user. On
+  ARMv7-M/ARMv8-M this is called the System Address Map, other CPUs may
+  have similar capabilities.
+
+- A read-only, executable region or regions for program text and ro-data, that
+  is accessible to user mode. This could be further sub-divided into a
+  read-only region for ro-data, and a read-only, executable region for text, but
+  this will require an additional MPU region. This is required so that
+  threads running in user mode can read ro-data and fetch instructions.
+
+- Depending on configuration, user-accessible read-write regions to support
+  extra features like GCOV, HEP, etc.
+
+Assuming there is a background map which allows supervisor mode to access any
+memory it needs, and regions are defined which grant user mode access to
+text/ro-data, this is sufficient for the boot time configuration.
+
+Hardware Stack Overflow
+***********************
+
+``CONFIG_HW_STACK_PROTECTION`` is an optional feature which detects stack
+buffer overflows when the system is running in supervisor mode. This
+catches issues when the entire stack buffer has overflowed, and not
+individual stack frames, use compiler-assisted CONFIG_STACK_CANARIES for that.
+
+Like any crash in supervisor mode, no guarantees can be made about the overall
+health of the system after a supervisor mode stack overflow, and any instances
+of this should be treated as a serious error. However it's still very useful to
+know when these overflows happen, as without robust detection logic the system
+will either crash in mysterious ways or behave in an undefined manner when the
+stack buffer overflows.
+
+Some systems implement this feature by creating at runtime a 'guard' MPU region
+which is set to be read-only and is at either the beginning or immediately
+preceding the supervisor mode stack buffer.  If the stack overflows an
+exception will be generated.
+
+This feature is optional and is not required to catch stack overflows in user
+mode; disabling this may free 1-2 MPU regions depending on the MPU design.
+
+Other systems may have dedicated CPU support for catching stack overflows
+and no extra MPU regions will be required.
+
+Thread Stack
+************
+
+Any thread running in user mode will need access to its own stack buffer.
+On context switch into a user mode thread, a dedicated MPU region will be
+programmed with the bounds of the stack buffer. A thread exceeding its stack
+buffer will start pushing data onto memory it doesn't have access to and a
+memory access violation exception will be generated.
+
+Thread Resource Pools
+*********************
+
+A small subset of kernel APIs, invoked as system calls, require heap memory
+allocations. This memory is used only by the kernel and is not accessible
+directly by user mode. In order to use these system calls, invoking threads
+must assign themselves to a resource pool, which is a k_mem_pool object.
+Memory is drawn from a thread's resource pool using :c:func:`z_thread_malloc()`
+and freed with :c:func:`k_free()`.
+
+The APIs which use resource pools are as follows, with any alternatives
+noted for users who do not want heap allocations within their application:
+
+ - :c:func:`k_stack_alloc_init()` sets up a k_stack with its storage
+   buffer allocated out of a resource pool instead of a buffer provided by the
+   user. An alternative is to declare k_stacks that are automatically
+   initialized at boot with :c:macro:`K_STACK_DEFINE()`, or to initialize the
+   k_stack in supervisor mode with :c:func:`k_stack_init()`.
+
+ - :c:func:`k_pipe_alloc_init()` sets up a k_pipe object with its
+   storage buffer allocated out of a resource pool instead of a buffer provided
+   by the user. An alternative is to declare k_pipes that are automatically
+   initialized at boot with :c:macro:`K_PIPE_DEFINE()`, or to initialize the
+   k_pipe in supervisor mode with :c:func:`k_pipe_init()`.
+
+ - :c:func:`k_msgq_alloc_init()` sets up a k_msgq object with its
+   storage buffer allocated out of a resource pool instead of a buffer provided
+   by the user. An alternative is to declare a k_msgq that is automatically
+   initialized at boot with :c:macro:`K_MSGQ_DEFINE()`, or to initialize the
+   k_msgq in supervisor mode with :c:func:`k_msgq_init()`.
+
+ - :c:func:`k_poll()` when invoked from user mode, needs to make a kernel-side
+   copy of the provided events array while waiting for an event. This copy is
+   freed when :c:func:`k_poll()` returns for any reason.
+
+ - :c:func:`k_queue_alloc_prepend()` and :c:func:`k_queue_alloc_append()`
+   allocate a container structure to place the data in, since the internal
+   bookkeeping information that defines the queue cannot be placed in the
+   memory provided by the user.
+
+ - :c:func:`k_object_alloc()` allows for entire kernel objects to be
+   dynamically allocated at runtime and a usable pointer to them returned to
+   the caller.
+
+The relevant API is :c:func:`k_thread_resource_pool_assign()` which assigns
+a k_mem_pool to draw these allocations from for the target thread.
+
+If the system heap is enabled, then the system heap may be used with
+:c:func:`k_thread_system_pool_assign()`, but it is preferable for different
+logical applications running on the system to have their own pools.
+
+Memory Domains
 **************
 
+The kernel ensures that any user thread will have access to its own stack
+buffer, plus program text and read-only data. The memory domain APIs are the
+way to grant access to additional blocks of memory to a user thread.
+
+Conceptually, a memory domain is a collection of some number of memory
+partitions. The maximum number of memory partitions in a domain
+is limited by the number of available MPU regions. This is why it is important
+to minimize the number of boot-time MPU regions.
+
+Memory domains are *not* intended to control access to memory from supervisor
+mode. In some cases this may be unavoidable; for example some architectures do
+not allow for the definition of regions which are read-only to user mode but
+read-write to supervisor mode. A great deal of care must be taken when working
+with such regions to not unintentionally cause the kernel to crash when
+accessing such a region.
+
+Memory domain APIs are only available to supervisor mode. The only control
+user mode has over memory domains is that any user thread's child threads
+will automatically become members of the parent's domain.
+
+Memory Partitions
+=================
+
+Each memory partition consists of a memory address, a size,
+and access attributes. It is intended that memory partitions are used to
+control access to system memory. Defining memory partitions are subject
+to the following constraints:
+
+- The partition must represent a memory region that can be programmed by
+  the underlying memory management hardware, and needs to conform to any
+  underlying hardware constraints. For example, many MPU-based systems require
+  that partitions be sized to some power of two, and aligned to their own
+  size. For MMU-based systems, the partition must be aligned to a page and
+  the size some multiple of the page size.
+
+- Partitions within the same memory domain may not overlap each other. There is
+  no notion of precedence among partitions within a memory domain.  Partitions
+  within a memory domain are assumed to have a higher precedence than any
+  boot-time memory regions, however whether a memory domain partition can
+  overlap a boot-time memory region is architecture specific.
+
+- The same partition may be specified in multiple memory domains. For example
+  there may be a shared memory area that multiple domains grant access to.
+
+- Care must be taken in determining what memory to expose in a partition.
+  It is not appropriate to provide direct user mode access to any memory
+  containing private kernel data.
+
+- Memory domain partitions are intended to control access to system RAM.
+  Configuration of memory partitions which do not correspond to RAM
+  may not be supported by the architecture; this is true for MMU-based systems.
+
+There are two ways to define memory partitions: either manually or
+automatically.
+
+Manual Memory Partitions
+------------------------
+
+The following code declares a global array buf, and then declares
+a read-write partition for it which may be added to a domain:
+
+.. code-block:: c
+
+    u8_t __aligned(32) buf[32];
+
+    K_MEM_PARTITION_DEFINE(my_partition, buf, sizeof(buf),
+                           K_MEM_PARTITION_P_RW_U_RW);
+
+This does not scale particularly well when we are trying to contain multiple
+objects spread out across several C files into a single partition.
+
+Automatic Memory Partitions
+---------------------------
+
+Automatic memory partitions are created by the build system. All globals
+which need to be placed inside a partition are tagged with their destination
+partition. The build system will then coalesce all of these into a single
+contiguous block of memory, zero any BSS variables at boot, and define
+a memory partition of appropriate base address and size which contains all
+the tagged data.
+
+Automatic memory partitions are only configured as read-write
+regions. They are defined with :c:macro:`K_APPMEM_PARTITION_DEFINE()`.
+Global variables are then routed to this partition using
+:c:macro:`K_APP_DMEM()` for initialized data and :c:macro:`K_APP_BMEM()` for
+BSS.
+
+.. code-block:: c
+
+    #include <app_memory/app_memdomain.h>
+
+    /* Declare a k_mem_partition "my_partition" that is read-write to
+     * user mode. Note that we do not specify a base address or size.
+     */
+    K_APPMEM_PARTITION_DEFINE(my_partition);
+
+    /* The global variable var1 will be inside the bounds of my_partition
+     * and be initialized with 37 at boot.
+     */
+    K_APP_DMEM(my_partition) int var1 = 37;
+
+    /* The global variable var2 will be inside the bounds of my_partition
+     * and be zeroed at boot size K_APP_BMEM() was used, indicating a BSS
+     * variable.
+     */
+    K_APP_BMEM(my_partition) int var2;
+
+The build system will ensure that the base address of my_partition will
+be properly aligned, and the total size of the region conforms to the memory
+management hardware requirments, adding padding if necessary.
+
+If multiple partitions are being created, a variadic preprocessor macro can be
+used as provided in ``app_macro_support.h``:
+
+.. code-block:: c
+
+    FOR_EACH(K_APPMEM_PARTITION_DEFINE, part0, part1, part2);
+
+There are some kernel objects which are defined by macros and take an argument
+for a destination section. A good example of these are sys_mem_pools, which
+are heap objects. The destination section name for an automatic partition
+can be obtained with :c:macro:`K_APP_DMEM_SECTION()` and
+:c:macro:`K_APP_BMEM_SECTION()` respectively for initialized data and BSS:
+
+.. code-block:: c
+
+    /* Declare automatic memory partition foo_partition */
+    K_APPMEM_PARTITION_DEFINE(foo_partition);
+
+    /* Section argument for the destination section obtained via
+     * K_APP_DMEM_SECTION()
+     */
+    SYS_MEM_POOL_DEFINE(foo_pool, NULL, BLK_SIZE_MIN, BLK_SIZE_MAX,
+                        BLK_NUM_MAX, BLK_ALIGN,
+    			K_APP_DMEM_SECTION(foo_partition));
+
+Automatic Partitions for Static Library Globals
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+The build-time logic for setting up automatic memory partitions is in
+``scripts/gen_app_partitions.py``. If a static library is linked into Zephyr,
+it is possible to route all the globals in that library to a specific
+memory partition with the ``--library`` argument.
+
+For example, if the Newlib C library is enabled, the Newlib globals all need
+to be placed in ``z_libc_partition``. The invocation of the script in the
+top-level ``CMakeLists.txt`` adds the following:
+
+.. code-block:: none
+
+    gen_app_partitions.py ... --library libc.a z_libc_partition ..
+
+There is no support for expressing this in the project-level configuration
+or build files; the toplevel ``CMakeLists.txt`` must be edited.
+
+Pre-defined Memory Partitions
+-----------------------------
+
+There are a few memory partitions which are pre-defined by the system:
+
+ - ``z_malloc_partition`` - This partition contains the system-wide pool of
+   memory used by libc malloc(). Due to possible starvation issues, it is
+   not recommended to draw heap memory from a global pool, instead
+   it is better to define various sys_mem_pool objects and assign them
+   to specific memory domains.
+
+ - ``z_libc_partition`` - Contains globals required by the C library and runtime.
+   Required if using newlib. Required if using minimal libc with
+   ``CONFIG_STACK_CANARIES`` enabled.
+
+Library-specific partitions are listed in ``include/app_memory/partitions.h``.
+For example, to use the MBEDTLS library from user mode, the
+``k_mbedtls_partition`` must be added to the domain.
+
+Memory Domain Usage
+===================
+
 Create a Memory Domain
-======================
+----------------------
 
 A memory domain is defined using a variable of type
 :c:type:`struct k_mem_domain`. It must then be initialized by calling
@@ -49,7 +325,7 @@ The following code defines and initializes an empty memory domain.
     k_mem_domain_init(&app0_domain, 0, NULL);
 
 Add Memory Partitions into a Memory Domain
-==========================================
+------------------------------------------
 
 There are two ways to add memory partitions into a memory domain.
 
@@ -97,20 +373,22 @@ memory domain one by one.
     The maximum number of memory partitions is limited by the maximum
     number of MPU regions or the maximum number of MMU tables.
 
-Add Threads into a Memory Domain
+Memory Domain Assignment
-================================
+------------------------
 
-Adding threads into a memory domain grants threads permission to access
+Any thread may join a memory domain, and any memory domain may have multiple
-the memory partitions in the memory domain.
+threads assigned to it. Threads are assigned to memory domains with an API
-
+call:
-The following code shows how to add threads into a memory domain.
 
 .. code-block:: c
 
     k_mem_domain_add_thread(&app0_domain, app_thread_id);
 
+In addition, if a thread is a member of a memory domain, and it creates a
+child thread, that thread will belong to the domain as well.
+
 Remove a Memory Partition from a Memory Domain
-==============================================
+----------------------------------------------
 
 The following code shows how to remove a memory partition from a memory
 domain.
@@ -124,7 +402,7 @@ that matches the given parameter and removes that partition from the
 memory domain.
 
 Remove a Thread from the Memory Domain
-======================================
+--------------------------------------
 
 The following code shows how to remove a thread from the memory domain.
 
@@ -133,7 +411,7 @@ The following code shows how to remove a thread from the memory domain.
     k_mem_domain_remove_thread(app_thread_id);
 
 Destroy a Memory Domain
-=======================
+-----------------------
 
 The following code shows how to destroy a memory domain.
 
@@ -142,7 +420,7 @@ The following code shows how to destroy a memory domain.
     k_mem_domain_destroy(&app0_domain);
 
 Available Partition Attributes
-==============================
+------------------------------
 
 When defining a partition, we need to set access permission attributes
 to the partition. Since the access control of memory partitions relies on

doc/reference/usermode/usermode_sharedmem.rst

@@ -1,75 +0,0 @@
-.. _usermode_sharedmem:
-
-Application Shared Memory
-#########################
-
-.. note::
-
-   In this document, we will cover the basic usage of enabling shared
-   memory using a template around app_memory subsystem.
-
-Overview
-********
-
-The use of subsystem app_memory in userspace allows control of
-shared memory between threads.  The foundation of the implementation
-consists of memory domains and partitions. Memory partitions are created
-and used in the definition of variable to group them into a
-common space.  The memory partitions are linked to domains
-that are then assigned to a thread.  The process allows selective
-access to memory from a thread and sharing of memory between two
-threads by assigning a partition to two different domains.  By using
-the shared memory template, code to protect memory can be used
-on different platform without the application needing to implement
-specific handlers for each platform.  Note the developer should understand
-the hardware limitations in context to the maximum number of memory
-partitions available to a thread.  Specifically processors with MPU's
-cannot support the same number of partitions as a MMU.
-
-This specific implementation adds a wrapper to simplify the programmers
-task of using the app_memory subsystem through the use of macros and
-a python script to generate the linker script. The linker script provides
-the proper alignment for processors requiring power of two boundaries.
-Without the wrapper, a developer is required to implement custom
-linker scripts for each processor in the project.
-
-The general usage is as follows. Include app_memory/app_memdomain.h
-in the userspace source file.  Mark the variable to be placed in
-a memory partition.  The two markers are for data and bss respectively:
-K_APP_DMEM(id) and K_APP_BMEM(id).  The id is used as the partition name.
-The resulting section name can be seen in the linker.map as
-"data_smem_id" and "data_smem_idb".
-
-To create a k_mem_partition, call the macro K_APPMEM_PARTITION_DEFINE(part0)
-where "part0" is the name then used to refer to that partition. The
-standard memory domain APIs may be used to add it to domains; the declared
-name is a k_mem_partition symbol.
-
-Example:
-
-.. code-block:: c
-
-            /* create partition at top of file outside functions */
-            K_APPMEM_PARTITION_DEFINE(part0);
-            /* create domain */
-            struct k_mem_domain dom0;
-            /* assign variables to the domain */
-            K_APP_DMEM(part0) int var1;
-            K_APP_BMEM(part0) static volatile int var2;
-
-            int main()
-            {
-                    k_mem_domain_init(&dom0, 0, NULL)
-                    k_mem_domain_add_partition(&dom0, part0);
-                    k_mem_domain_add_thread(&dom0, k_current_get());
-                    ...
-            }
-
-If multiple partitions are being created, a variadic
-preprocessor macro can be used as provided in
-app_macro_support.h:
-
-.. code-block:: c
-
- FOR_EACH(K_APPMEM_PARTITION_DEFINE, part0, part1, part2);
-