doc: document python build scripts

We have a collection of python scripts that are part of our build
system.  This PR collects docstring comments added to these scripts into
a summary document.  Previous references to just the script name in
other documentation are updated to point to this build tool
documentation.

Some of the scripts needed an update to be processed (via include
directives) consistently.

Signed-off-by: David B. Kinder <david.b.kinder@intel.com>
Signed-off-by: Anas Nashif <anas.nashif@intel.com>

doc/guides/build/index.rst

@@ -74,9 +74,9 @@ Offset generation
    *offsets.h* (by *gen_offset_header.py*) facilitates this.
 
 System call boilerplate
-   The *gen_syscall_header.py*, *parse_syscalls.py* and
+   The *gen_syscall.py* and *parse_syscalls.py*  scripts work
-   *gen_syscall_header.py* scripts work together to bind potential
+   together to bind potential system call functions with their
-   system call functions with their implementations.
+   implementations.
 
 .. figure:: build-build-phase-1.svg
     :align: center
@@ -153,3 +153,93 @@ target. This is accomplished in a straightforward manner with *objdump*.
     :alt: Zephyr's build final stage
     :figclass: align-center
     :width: 80%
+
+
+.. _build_system_scripts:
+
+Supporting Scripts and Tools
+****************************
+
+The following is a detailed description of the scripts used during the build process.
+
+.. _gen_syscalls.py:
+
+:zephyr_file:`scripts/gen_syscalls.py`
+========================================
+
+.. include:: ../../../scripts/gen_syscalls.py
+   :start-after: """
+   :end-before: """
+
+.. _gen_kobject_list.py:
+
+:zephyr_file:`scripts/gen_kobject_list.py`
+==========================================
+
+.. include:: ../../../scripts/gen_kobject_list.py
+   :start-after: """
+   :end-before: """
+
+.. _gen_offset_header.py:
+
+:zephyr_file:`scripts/gen_offset_header.py`
+===========================================
+
+.. include:: ../../../scripts/gen_offset_header.py
+   :start-after: """
+   :end-before: """
+
+.. _parse_syscalls.py:
+
+:zephyr_file:`scripts/parse_syscalls.py`
+========================================
+
+
+.. include:: ../../../scripts/parse_syscalls.py
+   :start-after: """
+   :end-before: """
+
+.. _gen_priv_stacks.py:
+
+:zephyr_file:`scripts/gen_priv_stacks.py`
+==========================================
+
+.. include:: ../../../scripts/gen_priv_stacks.py
+   :start-after: """
+   :end-before: """
+
+.. _gen_idt.py:
+
+:zephyr_file:`arch/x86/gen_idt.py`
+==================================
+
+.. include:: ../../../arch/x86/gen_idt.py
+   :start-after: """
+   :end-before: """
+
+.. _gen_gdt.py:
+
+:zephyr_file:`arch/x86/gen_gdt.py`
+===================================
+
+.. include:: ../../../arch/x86/gen_gdt.py
+   :start-after: """
+   :end-before: """
+
+.. _gen_relocate_app.py:
+
+:zephyr_file:`scripts/gen_relocate_app.py`
+===========================================
+
+.. include:: ../../../scripts/gen_relocate_app.py
+   :start-after: """
+   :end-before: """
+
+.. _process_gperf.py:
+
+:zephyr_file:`scripts/process_gperf.py`
+========================================
+
+.. include:: ../../../scripts/process_gperf.py
+   :start-after: """
+   :end-before: """

doc/guides/code-relocation.rst

@@ -7,8 +7,8 @@ Overview
 ********
 This feature will allow relocating .text, .rodata, .data, and .bss sections from
 required files and place them in the required memory region. The memory region
-and file are given to this python script in the form of a string. This script is
+and file are given to the :ref:`gen_relocate_app.py` script in the form
-always invoked from inside cmake.
+of a string. This script is always invoked from inside cmake.
 
 This script provides a robust way to re-order the memory contents without
 actually having to modify the code.  In simple terms this script will do the job
@@ -16,7 +16,7 @@ of ``__attribute__((section("name")))`` for a bunch of files together.
 
 Details
 *******
-The memory region and file are given to this python script in the form of a string.
+The memory region and file are given to the :ref:`gen_relocate_app.py` script in the form of a string.
 
 An example of such a string is:
 ``SRAM2:/home/xyz/zephyr/samples/hello_world/src/main.c,SRAM1:/home/xyz/zephyr/samples/hello_world/src/main2.c``

doc/guides/index.rst

@@ -9,6 +9,7 @@ User and Developer Guides
    getting-help.rst
    beyond-GSG.rst
    bluetooth/index.rst
+   build/index
    c_library
    ../README.rst
    coccinelle.rst
@@ -30,4 +31,3 @@ User and Developer Guides
    tracing/index
    west/index
    optimizations/index
-   build/index

doc/reference/kernel/other/interrupts.rst

@@ -380,12 +380,12 @@ x86 Details
 The x86 architecture has a special type of vector table called the Interrupt
 Descriptor Table (IDT) which must be laid out in a certain way per the x86
 processor documentation.  It is still fundamentally a vector table, and the
-gen_idt tool uses the .intList section to create it. However, on APIC-based
+:ref:`gen_idt.py` tool uses the .intList section to create it. However, on APIC-based
 systems the indexes in the vector table do not correspond to the IRQ line. The
 first 32 vectors are reserved for CPU exceptions, and all remaining vectors (up
 to index 255) correspond to the priority level, in groups of 16. In this
 scheme, interrupts of priority level 0 will be placed in vectors 32-47, level 1
-48-63, and so forth. When the gen_idt tool is constructing the IDT, when it
+48-63, and so forth. When the :ref:`gen_idt.py` tool is constructing the IDT, when it
 configures an interrupt it will look for a free vector in the appropriate range
 for the requested priority level and set the handler there.
 
@@ -400,7 +400,7 @@ ISR is placed directly in the IDT.
 
 On systems where the position in the vector table corresponds to the
 interrupt's priority level, the interrupt controller needs to know at
-runtime what vector is associated with an IRQ line. gen_idt additionally
+runtime what vector is associated with an IRQ line. :ref:`gen_idt.py` additionally
 creates an _irq_to_interrupt_vector array which maps an IRQ line to its
 configured vector in the IDT. This is used at runtime by :c:macro:`IRQ_CONNECT`
 to program the IRQ-to-vector association in the interrupt controller.

doc/reference/usermode/kernelobjects.rst

@@ -51,7 +51,7 @@ call APIs, several conditions must be met on how the kernel object is declared:
 
 * The object must be declared as a top-level global at build time, such that it
   appears in the ELF symbol table. It is permitted to declare kernel objects
-  with static scope. The post-build script ``gen_kobject_list.py`` scans the
+  with static scope. The post-build script :ref:`gen_kobject_list.py` scans the
   generated ELF file to find kernel objects and places their memory addresses
   in a special table of kernel object metadata.  Kernel objects may be members
   of arrays or embedded within other data structures.
@@ -66,7 +66,7 @@ Kernel objects that are found but do not meet the above conditions will not be
 included in the generated table that is used to validate kernel object pointers
 passed in from user mode.
 
-The debug output of the ``gen_kobject_list.py`` script may be useful when
+The debug output of the :ref:`gen_kobject_list.py` script may be useful when
 debugging why some object was unexpectedly not being tracked. This
 information will be printed if the script is run with the ``--verbose`` flag,
 or if the build system is invoked with verbose output.
@@ -98,7 +98,7 @@ the access control aspects of the permission system are not enforced.
 Implementation Details
 ======================
 
-The ``gen_kobject_list.py`` script is a post-build step which finds all the
+The :ref:`gen_kobject_list.py` script is a post-build step which finds all the
 valid kernel object instances in the binary. It accomplishes this by parsing
 the DWARF debug information present in the generated ELF file for the kernel.
 

doc/reference/usermode/mpu_userspace.rst

@@ -8,7 +8,7 @@ set of stacks.  These stacks exist in a 1:1 relationship with each thread stack
 defined in the system.  The privileged stacks are created as a part of the
 build process.
 
-A post-build script ``gen_priv_stacks.py`` scans the generated
+A post-build script :ref:`gen_priv_stacks.py` scans the generated
 ELF file and finds all of the thread stack objects.  A set of privileged
 stacks, a lookup table, and a set of helper functions are created and added
 to the image.

doc/reference/usermode/syscalls.rst

@@ -28,7 +28,7 @@ All system calls have the following components:
 
 * A **C prototype** for the API, declared in some header under ``include/`` and
   prefixed with :c:macro:`__syscall`.  This prototype is never implemented
-  manually, instead it gets created by the ``scripts/gen_syscalls.py`` script.
+  manually, instead it gets created by the :ref:`gen_syscalls.py` script.
   What gets generated is an inline function which either calls the
   implementation function directly (if called from supervisor mode) or goes
   through privilege elevation and validation steps (if called from user
@@ -57,8 +57,8 @@ supervisor mode. For example, to initialize a semaphore:
 
 The :c:macro:`__syscall` attribute is very special. To the C compiler, it
 simply expands to 'static inline'. However to the post-build
-``parse_syscalls.py`` script, it indicates that this API is a system call.
+:ref:`parse_syscalls.py` script, it indicates that this API is a system call.
-The ``parse_syscalls.py`` script does some parsing of the function prototype,
+The :ref:`parse_syscalls.py` script does some parsing of the function prototype,
 to determine the data types of its return value and arguments, and has some
 limitations:
 
@@ -118,7 +118,7 @@ Implementation Details
 ======================
 
 Declaring an API with :c:macro:`__syscall` causes some code to be generated in
-C and header files by ``scripts/gen_syscalls.py``, all of which can be found in
+C and header files by the :ref:`gen_syscalls.py` script, all of which can be found in
 the project out directory under ``include/generated/``:
 
 * The system call is added to the enumerated type of system call IDs,

scripts/gen_kobject_list.py

@@ -19,7 +19,7 @@ validate accesses to kernel objects to make the following assertions:
 
     - The calling thread has sufficient permissions on the object
 
-For more details see the "Kernel Objects" section in the documentation.
+For more details see the :ref:`kernelobjects` section in the documentation.
 
 The zephyr build generates an intermediate ELF binary, zephyr_prebuilt.elf,
 which this script scans looking for kernel objects by examining the DWARF

scripts/gen_offset_header.py

@@ -5,6 +5,13 @@
 # SPDX-License-Identifier: Apache-2.0
 #
 
+"""
+This script scans a specified object file and generates a header file
+that defined macros for the offsets of various found structure members
+(particularly symbols ending with ``_OFFSET`` or ``_SIZEOF``), primarily
+intended for use in assembly code.
+"""
+
 from elftools.elf.elffile import ELFFile
 from elftools.elf.sections import SymbolTableSection
 import argparse

scripts/gen_relocate_app.py

@@ -5,22 +5,32 @@
 # SPDX-License-Identifier: Apache-2.0
 #
 
-# This script will relocate .text, .rodata, .data and .bss sections from required files
+"""
-# and places it in the required memory region. This memory region and file
+This script will relocate .text, .rodata, .data and .bss sections from required files
-# are given to this python script in the form of a string.
+and places it in the required memory region. This memory region and file
-# Example of such a string would be:
+are given to this python script in the form of a string.
-# SRAM2:/home/xyz/zephyr/samples/hello_world/src/main.c,\
+
-# SRAM1:/home/xyz/zephyr/samples/hello_world/src/main2.c
+Example of such a string would be::
-# To invoke this script:
+
-# python3 gen_relocate_app.py -i input_string -o generated_linker -c generated_code
+   SRAM2:/home/xyz/zephyr/samples/hello_world/src/main.c,\
-# Configuration that needs to be sent to the python script.
+   SRAM1:/home/xyz/zephyr/samples/hello_world/src/main2.c
-# if the memory is like SRAM1/SRAM2/CCD/AON then place full object in
+
-# the sections
+To invoke this script::
-# if the memory type is appended with _DATA / _TEXT/ _RODATA/ _BSS only the
+
-# selected memory is placed in the required memory region. Others are
+   python3 gen_relocate_app.py -i input_string -o generated_linker -c generated_code
-# ignored.
+
-# NOTE: multiple regions can be appended together like SRAM2_DATA_BSS
+Configuration that needs to be sent to the python script.
-# this will place data and bss inside SRAM2
+
+- If the memory is like SRAM1/SRAM2/CCD/AON then place full object in
+  the sections
+- If the memory type is appended with _DATA / _TEXT/ _RODATA/ _BSS only the
+  selected memory is placed in the required memory region. Others are
+  ignored.
+
+Multiple regions can be appended together like SRAM2_DATA_BSS
+this will place data and bss inside SRAM2.
+"""
+
 
 import sys
 import argparse