From 17299f0734e84a9a443e4c79ffa1275167f70f15 Mon Sep 17 00:00:00 2001 From: "David B. Kinder" Date: Fri, 31 May 2019 15:39:39 -0700 Subject: [PATCH] doc: document python build scripts We have a collection of python scripts that are part of our build system. This PR collects docstring comments added to these scripts into a summary document. Previous references to just the script name in other documentation are updated to point to this build tool documentation. Some of the scripts needed an update to be processed (via include directives) consistently. Signed-off-by: David B. Kinder Signed-off-by: Anas Nashif --- doc/guides/build/index.rst | 96 ++++++++++++++++++++++- doc/guides/code-relocation.rst | 6 +- doc/guides/index.rst | 2 +- doc/reference/kernel/other/interrupts.rst | 6 +- doc/reference/usermode/kernelobjects.rst | 6 +- doc/reference/usermode/mpu_userspace.rst | 2 +- doc/reference/usermode/syscalls.rst | 8 +- scripts/gen_kobject_list.py | 2 +- scripts/gen_offset_header.py | 7 ++ scripts/gen_relocate_app.py | 42 ++++++---- 10 files changed, 142 insertions(+), 35 deletions(-) diff --git a/doc/guides/build/index.rst b/doc/guides/build/index.rst index 9c1e4c2a4e3..04bef94fea1 100644 --- a/doc/guides/build/index.rst +++ b/doc/guides/build/index.rst @@ -74,9 +74,9 @@ Offset generation *offsets.h* (by *gen_offset_header.py*) facilitates this. System call boilerplate - The *gen_syscall_header.py*, *parse_syscalls.py* and - *gen_syscall_header.py* scripts work together to bind potential - system call functions with their implementations. + The *gen_syscall.py* and *parse_syscalls.py* scripts work + together to bind potential system call functions with their + implementations. .. figure:: build-build-phase-1.svg :align: center @@ -153,3 +153,93 @@ target. This is accomplished in a straightforward manner with *objdump*. :alt: Zephyr's build final stage :figclass: align-center :width: 80% + + +.. _build_system_scripts: + +Supporting Scripts and Tools +**************************** + +The following is a detailed description of the scripts used during the build process. + +.. _gen_syscalls.py: + +:zephyr_file:`scripts/gen_syscalls.py` +======================================== + +.. include:: ../../../scripts/gen_syscalls.py + :start-after: """ + :end-before: """ + +.. _gen_kobject_list.py: + +:zephyr_file:`scripts/gen_kobject_list.py` +========================================== + +.. include:: ../../../scripts/gen_kobject_list.py + :start-after: """ + :end-before: """ + +.. _gen_offset_header.py: + +:zephyr_file:`scripts/gen_offset_header.py` +=========================================== + +.. include:: ../../../scripts/gen_offset_header.py + :start-after: """ + :end-before: """ + +.. _parse_syscalls.py: + +:zephyr_file:`scripts/parse_syscalls.py` +======================================== + + +.. include:: ../../../scripts/parse_syscalls.py + :start-after: """ + :end-before: """ + +.. _gen_priv_stacks.py: + +:zephyr_file:`scripts/gen_priv_stacks.py` +========================================== + +.. include:: ../../../scripts/gen_priv_stacks.py + :start-after: """ + :end-before: """ + +.. _gen_idt.py: + +:zephyr_file:`arch/x86/gen_idt.py` +================================== + +.. include:: ../../../arch/x86/gen_idt.py + :start-after: """ + :end-before: """ + +.. _gen_gdt.py: + +:zephyr_file:`arch/x86/gen_gdt.py` +=================================== + +.. include:: ../../../arch/x86/gen_gdt.py + :start-after: """ + :end-before: """ + +.. _gen_relocate_app.py: + +:zephyr_file:`scripts/gen_relocate_app.py` +=========================================== + +.. include:: ../../../scripts/gen_relocate_app.py + :start-after: """ + :end-before: """ + +.. _process_gperf.py: + +:zephyr_file:`scripts/process_gperf.py` +======================================== + +.. include:: ../../../scripts/process_gperf.py + :start-after: """ + :end-before: """ diff --git a/doc/guides/code-relocation.rst b/doc/guides/code-relocation.rst index 8d9b2f13f27..37289d8b00a 100644 --- a/doc/guides/code-relocation.rst +++ b/doc/guides/code-relocation.rst @@ -7,8 +7,8 @@ Overview ******** This feature will allow relocating .text, .rodata, .data, and .bss sections from required files and place them in the required memory region. The memory region -and file are given to this python script in the form of a string. This script is -always invoked from inside cmake. +and file are given to the :ref:`gen_relocate_app.py` script in the form +of a string. This script is always invoked from inside cmake. This script provides a robust way to re-order the memory contents without actually having to modify the code. In simple terms this script will do the job @@ -16,7 +16,7 @@ of ``__attribute__((section("name")))`` for a bunch of files together. Details ******* -The memory region and file are given to this python script in the form of a string. +The memory region and file are given to the :ref:`gen_relocate_app.py` script in the form of a string. An example of such a string is: ``SRAM2:/home/xyz/zephyr/samples/hello_world/src/main.c,SRAM1:/home/xyz/zephyr/samples/hello_world/src/main2.c`` diff --git a/doc/guides/index.rst b/doc/guides/index.rst index 22b484dc3a6..6a859f2019c 100644 --- a/doc/guides/index.rst +++ b/doc/guides/index.rst @@ -9,6 +9,7 @@ User and Developer Guides getting-help.rst beyond-GSG.rst bluetooth/index.rst + build/index c_library ../README.rst coccinelle.rst @@ -30,4 +31,3 @@ User and Developer Guides tracing/index west/index optimizations/index - build/index diff --git a/doc/reference/kernel/other/interrupts.rst b/doc/reference/kernel/other/interrupts.rst index 6f078668990..61dbdb89bae 100644 --- a/doc/reference/kernel/other/interrupts.rst +++ b/doc/reference/kernel/other/interrupts.rst @@ -380,12 +380,12 @@ x86 Details The x86 architecture has a special type of vector table called the Interrupt Descriptor Table (IDT) which must be laid out in a certain way per the x86 processor documentation. It is still fundamentally a vector table, and the -gen_idt tool uses the .intList section to create it. However, on APIC-based +:ref:`gen_idt.py` tool uses the .intList section to create it. However, on APIC-based systems the indexes in the vector table do not correspond to the IRQ line. The first 32 vectors are reserved for CPU exceptions, and all remaining vectors (up to index 255) correspond to the priority level, in groups of 16. In this scheme, interrupts of priority level 0 will be placed in vectors 32-47, level 1 -48-63, and so forth. When the gen_idt tool is constructing the IDT, when it +48-63, and so forth. When the :ref:`gen_idt.py` tool is constructing the IDT, when it configures an interrupt it will look for a free vector in the appropriate range for the requested priority level and set the handler there. @@ -400,7 +400,7 @@ ISR is placed directly in the IDT. On systems where the position in the vector table corresponds to the interrupt's priority level, the interrupt controller needs to know at -runtime what vector is associated with an IRQ line. gen_idt additionally +runtime what vector is associated with an IRQ line. :ref:`gen_idt.py` additionally creates an _irq_to_interrupt_vector array which maps an IRQ line to its configured vector in the IDT. This is used at runtime by :c:macro:`IRQ_CONNECT` to program the IRQ-to-vector association in the interrupt controller. diff --git a/doc/reference/usermode/kernelobjects.rst b/doc/reference/usermode/kernelobjects.rst index 55cfb1d163f..f493d32ac4e 100644 --- a/doc/reference/usermode/kernelobjects.rst +++ b/doc/reference/usermode/kernelobjects.rst @@ -51,7 +51,7 @@ call APIs, several conditions must be met on how the kernel object is declared: * The object must be declared as a top-level global at build time, such that it appears in the ELF symbol table. It is permitted to declare kernel objects - with static scope. The post-build script ``gen_kobject_list.py`` scans the + with static scope. The post-build script :ref:`gen_kobject_list.py` scans the generated ELF file to find kernel objects and places their memory addresses in a special table of kernel object metadata. Kernel objects may be members of arrays or embedded within other data structures. @@ -66,7 +66,7 @@ Kernel objects that are found but do not meet the above conditions will not be included in the generated table that is used to validate kernel object pointers passed in from user mode. -The debug output of the ``gen_kobject_list.py`` script may be useful when +The debug output of the :ref:`gen_kobject_list.py` script may be useful when debugging why some object was unexpectedly not being tracked. This information will be printed if the script is run with the ``--verbose`` flag, or if the build system is invoked with verbose output. @@ -98,7 +98,7 @@ the access control aspects of the permission system are not enforced. Implementation Details ====================== -The ``gen_kobject_list.py`` script is a post-build step which finds all the +The :ref:`gen_kobject_list.py` script is a post-build step which finds all the valid kernel object instances in the binary. It accomplishes this by parsing the DWARF debug information present in the generated ELF file for the kernel. diff --git a/doc/reference/usermode/mpu_userspace.rst b/doc/reference/usermode/mpu_userspace.rst index 614e1edef79..df1d74013a6 100644 --- a/doc/reference/usermode/mpu_userspace.rst +++ b/doc/reference/usermode/mpu_userspace.rst @@ -8,7 +8,7 @@ set of stacks. These stacks exist in a 1:1 relationship with each thread stack defined in the system. The privileged stacks are created as a part of the build process. -A post-build script ``gen_priv_stacks.py`` scans the generated +A post-build script :ref:`gen_priv_stacks.py` scans the generated ELF file and finds all of the thread stack objects. A set of privileged stacks, a lookup table, and a set of helper functions are created and added to the image. diff --git a/doc/reference/usermode/syscalls.rst b/doc/reference/usermode/syscalls.rst index 3ac0e2a5481..a1665248eb2 100644 --- a/doc/reference/usermode/syscalls.rst +++ b/doc/reference/usermode/syscalls.rst @@ -28,7 +28,7 @@ All system calls have the following components: * A **C prototype** for the API, declared in some header under ``include/`` and prefixed with :c:macro:`__syscall`. This prototype is never implemented - manually, instead it gets created by the ``scripts/gen_syscalls.py`` script. + manually, instead it gets created by the :ref:`gen_syscalls.py` script. What gets generated is an inline function which either calls the implementation function directly (if called from supervisor mode) or goes through privilege elevation and validation steps (if called from user @@ -57,8 +57,8 @@ supervisor mode. For example, to initialize a semaphore: The :c:macro:`__syscall` attribute is very special. To the C compiler, it simply expands to 'static inline'. However to the post-build -``parse_syscalls.py`` script, it indicates that this API is a system call. -The ``parse_syscalls.py`` script does some parsing of the function prototype, +:ref:`parse_syscalls.py` script, it indicates that this API is a system call. +The :ref:`parse_syscalls.py` script does some parsing of the function prototype, to determine the data types of its return value and arguments, and has some limitations: @@ -118,7 +118,7 @@ Implementation Details ====================== Declaring an API with :c:macro:`__syscall` causes some code to be generated in -C and header files by ``scripts/gen_syscalls.py``, all of which can be found in +C and header files by the :ref:`gen_syscalls.py` script, all of which can be found in the project out directory under ``include/generated/``: * The system call is added to the enumerated type of system call IDs, diff --git a/scripts/gen_kobject_list.py b/scripts/gen_kobject_list.py index b545546b77d..0e3d7320ae7 100755 --- a/scripts/gen_kobject_list.py +++ b/scripts/gen_kobject_list.py @@ -19,7 +19,7 @@ validate accesses to kernel objects to make the following assertions: - The calling thread has sufficient permissions on the object -For more details see the "Kernel Objects" section in the documentation. +For more details see the :ref:`kernelobjects` section in the documentation. The zephyr build generates an intermediate ELF binary, zephyr_prebuilt.elf, which this script scans looking for kernel objects by examining the DWARF diff --git a/scripts/gen_offset_header.py b/scripts/gen_offset_header.py index 89b152b1e0e..82152fbb42f 100755 --- a/scripts/gen_offset_header.py +++ b/scripts/gen_offset_header.py @@ -5,6 +5,13 @@ # SPDX-License-Identifier: Apache-2.0 # +""" +This script scans a specified object file and generates a header file +that defined macros for the offsets of various found structure members +(particularly symbols ending with ``_OFFSET`` or ``_SIZEOF``), primarily +intended for use in assembly code. +""" + from elftools.elf.elffile import ELFFile from elftools.elf.sections import SymbolTableSection import argparse diff --git a/scripts/gen_relocate_app.py b/scripts/gen_relocate_app.py index 4afa7b55d32..6c349488138 100644 --- a/scripts/gen_relocate_app.py +++ b/scripts/gen_relocate_app.py @@ -5,22 +5,32 @@ # SPDX-License-Identifier: Apache-2.0 # -# This script will relocate .text, .rodata, .data and .bss sections from required files -# and places it in the required memory region. This memory region and file -# are given to this python script in the form of a string. -# Example of such a string would be: -# SRAM2:/home/xyz/zephyr/samples/hello_world/src/main.c,\ -# SRAM1:/home/xyz/zephyr/samples/hello_world/src/main2.c -# To invoke this script: -# python3 gen_relocate_app.py -i input_string -o generated_linker -c generated_code -# Configuration that needs to be sent to the python script. -# if the memory is like SRAM1/SRAM2/CCD/AON then place full object in -# the sections -# if the memory type is appended with _DATA / _TEXT/ _RODATA/ _BSS only the -# selected memory is placed in the required memory region. Others are -# ignored. -# NOTE: multiple regions can be appended together like SRAM2_DATA_BSS -# this will place data and bss inside SRAM2 +""" +This script will relocate .text, .rodata, .data and .bss sections from required files +and places it in the required memory region. This memory region and file +are given to this python script in the form of a string. + +Example of such a string would be:: + + SRAM2:/home/xyz/zephyr/samples/hello_world/src/main.c,\ + SRAM1:/home/xyz/zephyr/samples/hello_world/src/main2.c + +To invoke this script:: + + python3 gen_relocate_app.py -i input_string -o generated_linker -c generated_code + +Configuration that needs to be sent to the python script. + +- If the memory is like SRAM1/SRAM2/CCD/AON then place full object in + the sections +- If the memory type is appended with _DATA / _TEXT/ _RODATA/ _BSS only the + selected memory is placed in the required memory region. Others are + ignored. + +Multiple regions can be appended together like SRAM2_DATA_BSS +this will place data and bss inside SRAM2. +""" + import sys import argparse