This adds support to generate SPDX 2.2 tag-value documents via the
new west spdx command. The CMake file-based APIs are leveraged to
create relationships from source files to the corresponding
generated build files. SPDX-License-Identifier comments in source
files are scanned and filled into the SPDX documents.
Before `west build` is run, a specific file must be created in the
build directory so that the CMake API reply will run. This can be
done by running:
west spdx --init -d BUILD_DIR
After `west build` is run, SPDX generation is then activated by
calling `west spdx`; currently this requires passing the build
directory as a parameter again:
west spdx -d BUILD_DIR
This will generate three SPDX documents in `BUILD_DIR/spdx/`:
1) `app.spdx`: This contains the bill-of-materials for the
application source files used for the build.
2) `zephyr.spdx`: This contains the bill-of-materials for the
specific Zephyr source code files that are used for the build.
3) `build.spdx`: This contains the bill-of-materials for the built
output files.
Each file in the bill-of-materials is scanned, so that its hashes
(SHA256 and SHA1) can be recorded, along with any detected licenses
if an `SPDX-License-Identifier` appears in the file.
SPDX Relationships are created to indicate dependencies between
CMake build targets; build targets that are linked together; and
source files that are compiled to generate the built library files.
`west spdx` can be called with optional parameters for further
configuration:
* `-n PREFIX`: specifies a prefix for the Document Namespaces that
will be included in the generated SPDX documents. See SPDX spec 2.2
section 2.5 at
https://spdx.github.io/spdx-spec/2-document-creation-information/.
If -n is omitted, a default namespace will be generated according
to the default format described in section 2.5 using a random UUID.
* `-s SPDX_DIR`: specifies an alternate directory where the SPDX
documents should be written. If not specified, they will be saved
in `BUILD_DIR/spdx/`.
* `--analyze-includes`: in addition to recording the compiled
source code files (e.g. `.c`, `.S`) in the bills-of-materials, if
this flag is specified, `west spdx` will attempt to determine the
specific header files that are included for each `.c` file. This
will take longer, as it performs a dry run using the C compiler
for each `.c` file (using the same arguments that were passed to it
for the actual build).
* `--include-sdk`: if `--analyze-includes` is used, then adding
`--include-sdk` will create a fourth SPDX document, `sdk.spdx`,
which will list any header files included from the SDK.
Signed-off-by: Steve Winslow <steve@swinslow.net>
Support this when debugging also.
Tweak the style for brevity also while we're here by propertizing the
supports_nogui method, etc.
Signed-off-by: Martí Bolívar <marti.bolivar@nordicsemi.no>
This makes it easier for runners to check if the binary has thread
info support turned on, allowing automatic configuration of the
underlying tool to support threads, if possible.
Signed-off-by: Martí Bolívar <marti.bolivar@nordicsemi.no>
Set options that are definitely true or false to True or False in the
options dict. Add a 'getboolean' method that also allows a fallback to
False in case the option is not mentioned in .config due to unmet
dependencies. This allows calling code to just ask about the option
they are interested in, even if the .config file doesn't mention the
option at all.
Propagate this to users within the runners package and 'west sign',
taking advantage of the new build_conf property.
Rename the 'bcfg' internal variable in sign.py to 'build_conf' to
match other source files that use BuildConfiguration instances, to
make it easier to grep for users.
Signed-off-by: Martí Bolívar <marti.bolivar@nordicsemi.no>
Instead of mocking out the BuildConfiguration class, just create its
input file and let the real class do the work.
This in turn exposes a bug in the way the board name is being pulled
out of the BuildConfiguration, which we fix to keep the tests passing.
Signed-off-by: Martí Bolívar <marti.bolivar@nordicsemi.no>
This is really verbose, and I doubt anyone cares unless there is a
problem. Keep it around when run as 'west -v flash' to allow for
debugging, though.
Signed-off-by: Martí Bolívar <marti.bolivar@nordicsemi.no>
Defer loading .config until we really need it, when we are flashing a
binary. Pre-emptively loading it is wasted effort if we're flashing a
.hex, which has been the default behavior when possible since
dcaabb860f ("west: runners: jlink: prefer .hex over .bin").
Signed-off-by: Martí Bolívar <marti.bolivar@nordicsemi.no>
Put it all in one log message rather than splitting it up.
This makes it look cleaner now that each log message is prefixed with
'runners.link:'.
Signed-off-by: Martí Bolívar <marti.bolivar@nordicsemi.no>
JLink versions like 'V7.0a' do not conform to PEP 440 version
conventions; the 'a' part is used by PEP 440 compliant versions for
alphas. It gets parsed to a legacy type by the packaging library,
which always is treated as a lower value when compared with a
conforming version string.
To fix, get the version from the shared library distributed with the
JLink tools. This has the side benefit of making the code work on
Windows. That's merely a nice to have for -nogui 1 detection for now,
but will be essential in the next commit.
Reported-by: Jake Mercer <jake.mercer@civica.co.uk>
Signed-off-by: Martí Bolívar <marti.bolivar@nordicsemi.no>
Make it easier to get a BuildConfiguration from runner code.
Stash the result so it only has to be computed once.
Signed-off-by: Martí Bolívar <marti.bolivar@nordicsemi.no>
This option has existed since the beginning of the runners package,
which greatly predates the way DT is used in zephyr right now. It
never really worked the way I wanted it to but it's too much work to
fix it now. Try to improve the help a bit at least while I'm looking
at it again.
Signed-off-by: Martí Bolívar <marti.bolivar@nordicsemi.no>
Copy a fix from test_nrfjprog.py to the other runner test suites. The
current code will enter an infinite recursion if you hit the path
where os.path.isfile is called, since it's been patched to
os_path_isfile_patch in the calling context. The fix is to cache the
'real' version in the parent scope and call it directly as a fallback.
Signed-off-by: Martí Bolívar <marti.bolivar@nordicsemi.no>
Today, there is a build target is added for each runner: flash, debug,
debugserver, attach.
And those runners will have a dependency to Zephyr logical target that
is built before invoking `west <runner>`.
This design has some flaws, mainly that additional dependencies directly
on the target will not be built when running `west <runner>` directly.
That generator expressions cannot be used for the DEPENDS argument.
Instead, the build target `<runner>` will not have any dependencies, and
will raise a build error if a dependency is added to the target.
Due to how `add_dependencies()` work, this must be done as a build time
check, and not configure time check.
`west <runner>` will invoke a build before executing the runner, and
this way ensure the build target is up-to-date, which again removes the
need for a dedicated `west_<runner>_target`.
It also minimizes the risk of developer errors, as developers no longer
need to consider the need for adding additional dependencies.
If a custom target is part of the default `all` build, then it's ensured
to be up-to-date.
Fixes: Issue reported on slack.
Signed-off-by: Torsten Rasmussen <Torsten.Rasmussen@nordicsemi.no>
The commit fixes problem with nrfjprog runner for west flash,
that has been introduced with latest changes to BuildConfiguration
class.
Signed-off-by: Dominik Ermel <dominik.ermel@nordicsemi.no>
The ImgtoolSigner.sign() is passed a BuildConfiguration object instance
that holds all of Kconfig options that have been defined for a build;
yet it has been reading the .config file once again, with the
load_dot_config, to check some of these options.
The commit replaces the code that has been using the load_dot_config
obtained data with the code that uses the BuildConfiguration object.
Signed-off-by: Dominik Ermel <dominik.ermel@nordicsemi.no>
The body of load_dot_config method has been reimplemented in
BuildConfiguration.get(), replacing the previous code.
Signed-off-by: Dominik Ermel <dominik.ermel@nordicsemi.no>
The BuildConfiguration.path attribute has been added that stores path
to .config file which has been used as source for options collected to
BuildCOnfiguration object.
Signed-off-by: Dominik Ermel <dominik.ermel@nordicsemi.no>
PR #33746 introduced changes to the devicetree python file
that requires changes in the python code that imports the
devicetree module.
This was omitted in the west sign command implementation.
Signed-off-by: Mikkel Jakobsen <mikkel.aunsbjerg@prevas.dk>
We are now in the process of extracting edtlib and dtlib into a
standalone source code library that we intend to share with other
projects.
Links related to the work making this standalone:
https://pypi.org/project/devicetree/https://python-devicetree.readthedocs.io/en/latest/https://github.com/zephyrproject-rtos/python-devicetree
This standalone repo includes the same features as what we have in
Zephyr, but in its own 'devicetree' python package with PyPI
integration, etc.
To avoid making this a hard fork, move the code that's being made
standalone around in Zephyr into a new scripts/dts/python-devicetree
subdirectory, and handle the package and sys.path changes in the
various places in the tree that use it.
From now on, it will be possible to update the standalone repository
by just recursively copying scripts/dts/python-devicetree's contents
into it and committing the results.
This is an interim step; do NOT 'pip install devicetree' yet.
The code in the zephyr repository is still the canonical location.
(In the long term, people will get the devicetree package from PyPI
just like they do the 'yaml' package today, but that won't happen for
the foreseeable future.)
This commit is purely intended to avoid a hard fork for the standalone
code, and no functional changes besides the package structure and
location of the code itself are expected.
Signed-off-by: Martí Bolívar <marti.bolivar@nordicsemi.no>
The jlink runner performs a version check which is skipped on
Windows. If running inside WSL we also need to skip.
Signed-off-by: Pieter De Gendt <pieter.degendt@basalte.be>
Allow a custom (remote) gdb host passed as a debug argument
which tries to connect to an existing one instead of creating one.
This also allows to run the gdb server outside of a WSL
in Windows and the debugger inside of the WSL environment.
Signed-off-by: Pieter De Gendt <pieter.degendt@basalte.be>
The nrfjprog utility is not capable of flashing a hex file which
affects the flash memories of both coprocessors of the nRF53 family of
SoCs.
However, the user is capable of creating such a hex file using the
HEX_FILES_TO_MERGE build system variable.
An example use case is to build a bluetooth controller application for
the network core, then use the zephyr.hex file in that build directory
as the HEX_FILES_TO_MERGE argument for a separate Bluetooth
application build targeting the app core.
Work around this by detecting the situation and doing the right thing
by splitting the hex file back up again, even if thats a bit awkward.
Splitting the hex into app and network core components allows them to
be flashed separately. This is the only way we can get the job done
with nrfjprog.
This is arguably nicer since there's just one 'west flash' invocation.
At least in the use case named above, you wouldn't need to rebuild the
controller application very often, so this is a simpler user workflow.
Signed-off-by: Martí Bolívar <marti.bolivar@nordicsemi.no>
We need all available runners to be defined as subclasses of
runners.core.ZephyrBinaryRunner in order to be able to look them up by
name at runtime. We do this by importing them from runners.__init__.
This process periodically fails when some runner or other cannot be
imported, usually because it is trying to import something outside of
stdlib and not handling ImportError.
Rather than letting this bring down the entire Python process, catch
and log the error. Sort the list again while we're here.
Signed-off-by: Martí Bolívar <marti.bolivar@nordicsemi.no>
Runners may not import anything outside of stdlib unless they handle
the possible ImportError.
Apply this rule to the bossac runner. The general pattern is to have a
MISSING_REQUIREMENTS global that we handle at the beginning of
do_run(), but in this case this isn't a missing pip package, but a
misconfiguration. Just log a warning; it won't happen in practice.
Signed-off-by: Martí Bolívar <marti.bolivar@nordicsemi.no>
Runners can (and many should) depend on the devicetree to decide what
to do, especially as it relates to the flash layout. Make it so that
they do not have to manipulate sys.path to get a hold of the edtlib
directory by doing it in one place, before importing any of them.
Signed-off-by: Martí Bolívar <marti.bolivar@nordicsemi.no>
Add support for including a board-specific config.yaml file
in the pyocd flash command. Similary to openOCD, the config
file is placed in the board directory under support/.
Signed-off-by: Ioannis Glaropoulos <Ioannis.Glaropoulos@nordicsemi.no>
Update test to account for new default parameters that allow
use image smaller than flash device capacity.
Signed-off-by: Jose Alberto Meza <jose.a.meza.arellano@intel.com>
Update the jlink runner to prefer flashing .hex files instead of .bin.
This can increase programming speed in cases where there are large
amounts of padding in an application.
Signed-off-by: Jordan Yates <jordan.yates@data61.csiro.au>
Zephyr testcases(not SOF case) not use kernel DSP driver to load image
on ADSP board, thus do not need signing with xman. So add a input
'--no-manifest' to specify signing without xman in image. If use DSP
driver load image, we should not specify this.
Signed-off-by: Jian Kang <jianx.kang@intel.com>
Signed-off-by: Anas Nashif <anas.nashif@intel.com>
Handle a semantic versioning case used by Kitware that is not
supported by the packaging module we're using for version comparisons.
Signed-off-by: Martí Bolívar <marti.bolivar@nordicsemi.no>
There are use cases where --use-elf is given and the hex file does not
exist. Handle them.
Fixes: #31944
Signed-off-by: Martí Bolívar <marti.bolivar@nordicsemi.no>
Commit 3204554841 (" scripts: runners: error on missing
non-elf outputs") created the possibility of None bin_file and
hex_file attributes in the RunnerConfig without updating pyocd
appropriately. Fix that.
Fixes: #31921
Signed-off-by: Martí Bolívar <marti.bolivar@nordicsemi.no>
We can't trust that a python file is executable on Windows, regardless
of what the mode bits say.
When we find that imgtool is a .py file, run 'python imgtool.py'
instead of 'imgtool.py' on that platform.
Fixes: #31876
Signed-off-by: Martí Bolívar <marti.bolivar@nordicsemi.no>
Have it return a list, which is prepended to the final arguments list
passed to subprocess. No behavioral changes expected, but this enables
making it contain multiple strings instead of just one.
Signed-off-by: Martí Bolívar <marti.bolivar@nordicsemi.no>
The RunnerConfig class stores the locations of the Zephyr output files
in various formats (elf, hex, bin). A longstanding issue with the
representation is that these might not exist if the corresponding
Kconfig options are not set. For example, if
CONFIG_BUILD_OUTPUT_BIN=n, there is no .bin file.
Change this so the type system knows these are Optional[str], not str.
Fix the runners that use non-ELF outputs so they check for the
existence of the relevant file before using it, mostly using a new
ZephyrBinaryRunner.ensure_output helper.
I'm not going to bother with checking for the ELF file itself; that's
always there as far as I can tell.
Signed-off-by: Martí Bolívar <marti.bolivar@nordicsemi.no>
If we do not have the configuration path via cmake, then check if the
configuration path was provided on the command line and use it instead,
if both fail, bail out and die.
Signed-off-by: Anas Nashif <anas.nashif@intel.com>
rimage signing tool requires the path to the configuration directory.
This can be fetched using cmake cache if we are building with a module
that has rimage integrated like the sof module, however, we should be
able to sign images that were built standalone and without SOF, so
support this new option to provide the location of the configuration
files for the tool.
Signed-off-by: Anas Nashif <anas.nashif@intel.com>
Calling cmake/pristine.cmake now requires SOURCE_DIR and BINARY_DIR as
arguments.
This ensures that pristine.cmake can evaluate if pristine is requested
on in-source builds, and bail out in such case with an error message.
All uses of `pristine.cmake` has been updated to use the new arguments.
Signed-off-by: Torsten Rasmussen <Torsten.Rasmussen@nordicsemi.no>
