doc: document python build scripts

We have a collection of python scripts that are part of our build
system.  This PR collects docstring comments added to these scripts into
a summary document.  Previous references to just the script name in
other documentation are updated to point to this build tool
documentation.

Some of the scripts needed an update to be processed (via include
directives) consistently.

Signed-off-by: David B. Kinder <david.b.kinder@intel.com>
Signed-off-by: Anas Nashif <anas.nashif@intel.com>
This commit is contained in:
David B. Kinder 2019-05-31 15:39:39 -07:00 committed by Anas Nashif
commit 17299f0734
10 changed files with 142 additions and 35 deletions

View file

@ -74,9 +74,9 @@ Offset generation
*offsets.h* (by *gen_offset_header.py*) facilitates this.
System call boilerplate
The *gen_syscall_header.py*, *parse_syscalls.py* and
*gen_syscall_header.py* scripts work together to bind potential
system call functions with their implementations.
The *gen_syscall.py* and *parse_syscalls.py* scripts work
together to bind potential system call functions with their
implementations.
.. figure:: build-build-phase-1.svg
:align: center
@ -153,3 +153,93 @@ target. This is accomplished in a straightforward manner with *objdump*.
:alt: Zephyr's build final stage
:figclass: align-center
:width: 80%
.. _build_system_scripts:
Supporting Scripts and Tools
****************************
The following is a detailed description of the scripts used during the build process.
.. _gen_syscalls.py:
:zephyr_file:`scripts/gen_syscalls.py`
========================================
.. include:: ../../../scripts/gen_syscalls.py
:start-after: """
:end-before: """
.. _gen_kobject_list.py:
:zephyr_file:`scripts/gen_kobject_list.py`
==========================================
.. include:: ../../../scripts/gen_kobject_list.py
:start-after: """
:end-before: """
.. _gen_offset_header.py:
:zephyr_file:`scripts/gen_offset_header.py`
===========================================
.. include:: ../../../scripts/gen_offset_header.py
:start-after: """
:end-before: """
.. _parse_syscalls.py:
:zephyr_file:`scripts/parse_syscalls.py`
========================================
.. include:: ../../../scripts/parse_syscalls.py
:start-after: """
:end-before: """
.. _gen_priv_stacks.py:
:zephyr_file:`scripts/gen_priv_stacks.py`
==========================================
.. include:: ../../../scripts/gen_priv_stacks.py
:start-after: """
:end-before: """
.. _gen_idt.py:
:zephyr_file:`arch/x86/gen_idt.py`
==================================
.. include:: ../../../arch/x86/gen_idt.py
:start-after: """
:end-before: """
.. _gen_gdt.py:
:zephyr_file:`arch/x86/gen_gdt.py`
===================================
.. include:: ../../../arch/x86/gen_gdt.py
:start-after: """
:end-before: """
.. _gen_relocate_app.py:
:zephyr_file:`scripts/gen_relocate_app.py`
===========================================
.. include:: ../../../scripts/gen_relocate_app.py
:start-after: """
:end-before: """
.. _process_gperf.py:
:zephyr_file:`scripts/process_gperf.py`
========================================
.. include:: ../../../scripts/process_gperf.py
:start-after: """
:end-before: """

View file

@ -7,8 +7,8 @@ Overview
********
This feature will allow relocating .text, .rodata, .data, and .bss sections from
required files and place them in the required memory region. The memory region
and file are given to this python script in the form of a string. This script is
always invoked from inside cmake.
and file are given to the :ref:`gen_relocate_app.py` script in the form
of a string. This script is always invoked from inside cmake.
This script provides a robust way to re-order the memory contents without
actually having to modify the code. In simple terms this script will do the job
@ -16,7 +16,7 @@ of ``__attribute__((section("name")))`` for a bunch of files together.
Details
*******
The memory region and file are given to this python script in the form of a string.
The memory region and file are given to the :ref:`gen_relocate_app.py` script in the form of a string.
An example of such a string is:
``SRAM2:/home/xyz/zephyr/samples/hello_world/src/main.c,SRAM1:/home/xyz/zephyr/samples/hello_world/src/main2.c``

View file

@ -9,6 +9,7 @@ User and Developer Guides
getting-help.rst
beyond-GSG.rst
bluetooth/index.rst
build/index
c_library
../README.rst
coccinelle.rst
@ -30,4 +31,3 @@ User and Developer Guides
tracing/index
west/index
optimizations/index
build/index

View file

@ -380,12 +380,12 @@ x86 Details
The x86 architecture has a special type of vector table called the Interrupt
Descriptor Table (IDT) which must be laid out in a certain way per the x86
processor documentation. It is still fundamentally a vector table, and the
gen_idt tool uses the .intList section to create it. However, on APIC-based
:ref:`gen_idt.py` tool uses the .intList section to create it. However, on APIC-based
systems the indexes in the vector table do not correspond to the IRQ line. The
first 32 vectors are reserved for CPU exceptions, and all remaining vectors (up
to index 255) correspond to the priority level, in groups of 16. In this
scheme, interrupts of priority level 0 will be placed in vectors 32-47, level 1
48-63, and so forth. When the gen_idt tool is constructing the IDT, when it
48-63, and so forth. When the :ref:`gen_idt.py` tool is constructing the IDT, when it
configures an interrupt it will look for a free vector in the appropriate range
for the requested priority level and set the handler there.
@ -400,7 +400,7 @@ ISR is placed directly in the IDT.
On systems where the position in the vector table corresponds to the
interrupt's priority level, the interrupt controller needs to know at
runtime what vector is associated with an IRQ line. gen_idt additionally
runtime what vector is associated with an IRQ line. :ref:`gen_idt.py` additionally
creates an _irq_to_interrupt_vector array which maps an IRQ line to its
configured vector in the IDT. This is used at runtime by :c:macro:`IRQ_CONNECT`
to program the IRQ-to-vector association in the interrupt controller.

View file

@ -51,7 +51,7 @@ call APIs, several conditions must be met on how the kernel object is declared:
* The object must be declared as a top-level global at build time, such that it
appears in the ELF symbol table. It is permitted to declare kernel objects
with static scope. The post-build script ``gen_kobject_list.py`` scans the
with static scope. The post-build script :ref:`gen_kobject_list.py` scans the
generated ELF file to find kernel objects and places their memory addresses
in a special table of kernel object metadata. Kernel objects may be members
of arrays or embedded within other data structures.
@ -66,7 +66,7 @@ Kernel objects that are found but do not meet the above conditions will not be
included in the generated table that is used to validate kernel object pointers
passed in from user mode.
The debug output of the ``gen_kobject_list.py`` script may be useful when
The debug output of the :ref:`gen_kobject_list.py` script may be useful when
debugging why some object was unexpectedly not being tracked. This
information will be printed if the script is run with the ``--verbose`` flag,
or if the build system is invoked with verbose output.
@ -98,7 +98,7 @@ the access control aspects of the permission system are not enforced.
Implementation Details
======================
The ``gen_kobject_list.py`` script is a post-build step which finds all the
The :ref:`gen_kobject_list.py` script is a post-build step which finds all the
valid kernel object instances in the binary. It accomplishes this by parsing
the DWARF debug information present in the generated ELF file for the kernel.

View file

@ -8,7 +8,7 @@ set of stacks. These stacks exist in a 1:1 relationship with each thread stack
defined in the system. The privileged stacks are created as a part of the
build process.
A post-build script ``gen_priv_stacks.py`` scans the generated
A post-build script :ref:`gen_priv_stacks.py` scans the generated
ELF file and finds all of the thread stack objects. A set of privileged
stacks, a lookup table, and a set of helper functions are created and added
to the image.

View file

@ -28,7 +28,7 @@ All system calls have the following components:
* A **C prototype** for the API, declared in some header under ``include/`` and
prefixed with :c:macro:`__syscall`. This prototype is never implemented
manually, instead it gets created by the ``scripts/gen_syscalls.py`` script.
manually, instead it gets created by the :ref:`gen_syscalls.py` script.
What gets generated is an inline function which either calls the
implementation function directly (if called from supervisor mode) or goes
through privilege elevation and validation steps (if called from user
@ -57,8 +57,8 @@ supervisor mode. For example, to initialize a semaphore:
The :c:macro:`__syscall` attribute is very special. To the C compiler, it
simply expands to 'static inline'. However to the post-build
``parse_syscalls.py`` script, it indicates that this API is a system call.
The ``parse_syscalls.py`` script does some parsing of the function prototype,
:ref:`parse_syscalls.py` script, it indicates that this API is a system call.
The :ref:`parse_syscalls.py` script does some parsing of the function prototype,
to determine the data types of its return value and arguments, and has some
limitations:
@ -118,7 +118,7 @@ Implementation Details
======================
Declaring an API with :c:macro:`__syscall` causes some code to be generated in
C and header files by ``scripts/gen_syscalls.py``, all of which can be found in
C and header files by the :ref:`gen_syscalls.py` script, all of which can be found in
the project out directory under ``include/generated/``:
* The system call is added to the enumerated type of system call IDs,

View file

@ -19,7 +19,7 @@ validate accesses to kernel objects to make the following assertions:
- The calling thread has sufficient permissions on the object
For more details see the "Kernel Objects" section in the documentation.
For more details see the :ref:`kernelobjects` section in the documentation.
The zephyr build generates an intermediate ELF binary, zephyr_prebuilt.elf,
which this script scans looking for kernel objects by examining the DWARF

View file

@ -5,6 +5,13 @@
# SPDX-License-Identifier: Apache-2.0
#
"""
This script scans a specified object file and generates a header file
that defined macros for the offsets of various found structure members
(particularly symbols ending with ``_OFFSET`` or ``_SIZEOF``), primarily
intended for use in assembly code.
"""
from elftools.elf.elffile import ELFFile
from elftools.elf.sections import SymbolTableSection
import argparse

View file

@ -5,22 +5,32 @@
# SPDX-License-Identifier: Apache-2.0
#
# This script will relocate .text, .rodata, .data and .bss sections from required files
# and places it in the required memory region. This memory region and file
# are given to this python script in the form of a string.
# Example of such a string would be:
# SRAM2:/home/xyz/zephyr/samples/hello_world/src/main.c,\
# SRAM1:/home/xyz/zephyr/samples/hello_world/src/main2.c
# To invoke this script:
# python3 gen_relocate_app.py -i input_string -o generated_linker -c generated_code
# Configuration that needs to be sent to the python script.
# if the memory is like SRAM1/SRAM2/CCD/AON then place full object in
# the sections
# if the memory type is appended with _DATA / _TEXT/ _RODATA/ _BSS only the
# selected memory is placed in the required memory region. Others are
# ignored.
# NOTE: multiple regions can be appended together like SRAM2_DATA_BSS
# this will place data and bss inside SRAM2
"""
This script will relocate .text, .rodata, .data and .bss sections from required files
and places it in the required memory region. This memory region and file
are given to this python script in the form of a string.
Example of such a string would be::
SRAM2:/home/xyz/zephyr/samples/hello_world/src/main.c,\
SRAM1:/home/xyz/zephyr/samples/hello_world/src/main2.c
To invoke this script::
python3 gen_relocate_app.py -i input_string -o generated_linker -c generated_code
Configuration that needs to be sent to the python script.
- If the memory is like SRAM1/SRAM2/CCD/AON then place full object in
the sections
- If the memory type is appended with _DATA / _TEXT/ _RODATA/ _BSS only the
selected memory is placed in the required memory region. Others are
ignored.
Multiple regions can be appended together like SRAM2_DATA_BSS
this will place data and bss inside SRAM2.
"""
import sys
import argparse