Add a --coverage-per-test mode that collects an isolated coverage
artifact for every Ztest test case and post-processes them into a
test-to-code coverage matrix, answering "which tests covered line X of
file foo.c".
The mode builds on CONFIG_ZTEST_COVERAGE_PER_TEST. On platforms that
support semihosting (ARM/RISC-V/Xtensa under QEMU, e.g. mps2/*) the
transport is switched to semihosting so per-test data is written straight
to the host as "<gcda>@@<suite>.<test>", avoiding the large amount of
serial traffic per-test dumping would otherwise generate; other platforms
fall back to the console transport. The required Kconfig is injected into
the build overlay.
Each test is captured by invoking gcov directly -- one
"gcov --stdout --json-format" per test over its gcda set, materialized in
a temporary object tree with symlinks to the shared gcno -- and its JSON
is converted to an lcov tracefile. This is far cheaper than a full lcov
capture per test; lcov capture remains a fallback for non-GCC toolchains.
To keep the end-of-run aggregation proportional to the number of
instances rather than the number of tests, each instance's per-test
tracefiles are collapsed into a single untagged union, and the aggregate
matrix is built by merging the per-instance matrices. Function records
are omitted from the tracefiles because gcov reports per-function counts
and end lines inconsistently across per-test dumps, which would fail
lcov's merge consistency checks; line and branch coverage are unaffected.
Per-test mode requires the lcov coverage tool (the matrix relies on lcov
TN records; gcovr has no equivalent) and this is validated at
argument-parsing time. Output is one <scenario>.<test>.info per test
under each build's coverage/tests/ directory plus a test_matrix.json with
by_line ({file:{line:[tests]}}) and by_test views.
Covered by unit tests for the tracefile conversion, tracefile union,
matrix build and merge, and the tagged-dump discovery and parsing
helpers.
Assisted-by: Claude:claude-opus-4-8
Signed-off-by: Anas Nashif <anas.nashif@intel.com>
With --coverage-per-instance, run_coverage_instance() generates a
coverage report for every test instance, and each generate() call
emitted an info-level '<FORMAT> report generated: ...' line. On a run
with many instances this floods the console with one line per instance.
Log the per-instance report messages at debug level and keep only the
final aggregated report at info level, so the console stays readable
while the detail is still available with -vv.
Assisted-by: Claude:claude-opus-4-8
Signed-off-by: Anas Nashif <anas.nashif@intel.com>
When --gcov-tool was not given, coverage relied on ZEPHYR_SDK_INSTALL_DIR
being exported and then hardcoded the x86_64-zephyr-elf gcov path. That
fails when the variable is not in the environment (even though the build
itself resolves the SDK via CMake), and the hardcoded architecture is
wrong for non-x86 targets.
Each Zephyr build already records the resolved gcov binary as CMAKE_GCOV
in its CMakeCache.txt, which for the Zephyr SDK points at the SDK's cross
gcov matching the toolchain. Read it back and use it so the gcov tool is
detected automatically without any environment setup and matches the
gcno/gcda data for any target architecture.
Keep ZEPHYR_SDK_INSTALL_DIR as a fallback, sourced from the environment
or, failing that, from a build's CMakeCache.txt, before erroring out.
Assisted-by: Claude:claude-opus-4-8
Signed-off-by: Anas Nashif <anas.nashif@intel.com>
choose_gcov_tool() called .endswith() directly on the result of
os.environ.get("ZEPHYR_TOOLCHAIN_VARIANT"). When the variable is not
exported in the environment, get() returns None and the coverage run
aborts with:
AttributeError: 'NoneType' object has no attribute 'endswith'
Provide an empty-string default so an unset variant safely falls
through to the system gcov / Zephyr SDK gcov branches instead of
crashing.
Signed-off-by: Anas Nashif <anas.nashif@intel.com>
Some simulators resolve their executable at CMake configure time via
find_program() instead of being known during test planning, and the
binary that gets used can depend on the build configuration. Arm FVP is
the motivating case: Corstone-300 selects a different model depending on
the Ethos-U NPU, so armfvp cannot be listed in SUPPORTED_SIMS_WITH_EXEC
and checked up front. As a result, when the FVP binary was not installed
Twister still considered the test runnable, attempted to run nothing, and
reported a misleading status.
Inspect CMakeCache after the build instead: add a SIM_PROGRAM_CMAKE_VARS
mapping of simulator name to the cache variable recording its resolved
executable. When the variable is absent or holds a *-NOTFOUND value
(e.g. ARMFVP:FILEPATH=ARMFVP-NOTFOUND), mark the instance as NOTRUN with
a clear reason and propagate that status to its test cases, so the test
is no longer reported as runnable.
This check is done in the post_build stage, which now always runs after a
successful build; the --post-build-checks option only gates the optional
build-failing checks, which are skipped inside post_build when it is not
set. This keeps the not-run determination separate from gather_metrics,
where it did not belong.
Assisted-by: Claude:claude-opus-4-8
Signed-off-by: Anas Nashif <anas.nashif@intel.com>
Compiler-reported paths may contain "." or "..", causing the same header
to be treated as multiple files. This substantially increases ownership,
license, hash, and relationship processing during --analyze-includes.
Normalize paths before deduplication. In a test build pulling in LVGL,
this reduced the include set from 50,000+ entries to ~1,200.
Signed-off-by: Benjamin Cabé <benjamin@zephyrproject.org>
domains.py is imported during west command discovery, before the
missing dependencies are installed. Since jsonschema is not a west
dependency, this broke `west update`. Wrap the import in a try-except
and only fail when a domains.yaml is actually parsed.
Signed-off-by: Pieter De Gendt <pieter.degendt@basalte.be>
Extend the `SEMIHOST` data dump function with support for multiple dumps
per application run, such as when the test application is intentionally
testing reboot paths.
Instead of the final `filename.gcda` file being overwritten on each
boot, create a unique `filename.gcda.{X}` file per boot, which are then
merged back into `filename.gcda` by `coverage.py`. The merging step is
required as the coverage tools expect specific paths for the GCDA files
derived from the source filename, they are not searching through the
folders for any filenames containing 'gcda'.
Signed-off-by: Jordan Yates <jordan@embeint.com>
Support using a different toolchain for a specific platform.
A platform can now set preferred_toolchain in the yaml file and this
toolchain will be used instead of the the default. This allows us to
test multiple toolchains and have some platforms build using the
preferred toolchain in CI. This applies only when running twister,
everything behaves as usual when building with west.
Signed-off-by: Anas Nashif <anas.nashif@intel.com>
Clean up Twister OpenOCD runner by removing
deprecated cmsis_dap_serial argument and
replacing with adapter serial
Signed-off-by: Shankar Ramasamy <shankar.ramasamy@microchip.com>
Processing the raw coverage data into the output `html` can take a
significant amount of time. Add a display for the duration, similar to
the timers for building the testsuite list and test build/execute times.
Example output:
```
INFO - Built testsuite list in 0.04 seconds
...
INFO - 16 of 16 executed test configurations passed (100.00%), 0 \
built (not run), 0 failed, 0 errored, with no warnings in \
25.85 seconds.
...
INFO - Coverage data processed in 11.45 seconds
```
Signed-off-by: Jordan Yates <jordan@embeint.com>
Use suite id instead of name as the key in the testsuites
registry. With --detailed-test-id enabled, the suite name
includes the path, which caused a false duplicate error when a
required application was passed via -T on the command line and
was also listed in the required_application section of another
test.
Duplicated scenarios are reported as an error regardless of
whether --detailed-test-id is used.
Update the twister unit tests.
Signed-off-by: Grzegorz Chwierut <grzegorz.chwierut@nordicsemi.no>
Enrich the Build profile with per-target compile flags and defines and
the key build inputs (board, arch, toolchain), mirroring the
Build-profile output of projects such as the Linux kernel.
Signed-off-by: Benjamin Cabé <benjamin@zephyrproject.org>
Emit the SPDX 3.0.1 Build profile, recording how the image and each
target was built. The profile is omitted when no build information is
available.
Signed-off-by: Benjamin Cabé <benjamin@zephyrproject.org>
Gather the parsed toolchain, CMake file-API and per-target metadata
into the SBOM graph so serializers have the build details they need.
Signed-off-by: Benjamin Cabé <benjamin@zephyrproject.org>
Parse the CMake file-API toolchains-v1 reply and cmake index object so
the SPDX 3.0 Build profile can describe the build toolchain.
Signed-off-by: Benjamin Cabé <benjamin@zephyrproject.org>
Replace the hand-written __init__ boilerplate in the CMake file-API
data classes with @dataclass field declarations, matching the style
already used elsewhere in zspdx (e.g. model.py). This makes the
module easier to extend with new data classes going forward.
No behavioral change: field defaults, __repr__ output, and the
zero-argument construction pattern used by cmakefileapijson.py are
all preserved.
Signed-off-by: Benjamin Cabé <benjamin@zephyrproject.org>
Ownership was decided while mutating the set it was tested against, so a
relationship's endpoints pulled in further relationships and the build
graph cascaded into every document. Decide ownership from a snapshot of
the document's own elements instead.
Signed-off-by: Benjamin Cabé <benjamin@zephyrproject.org>
Assisted-by: Claude:opus-4.8
Every relationship is also stored on its "from" element, so iterating the
graph-wide list in addition to components and files emitted each one
twice. Drop the redundant pass.
Signed-off-by: Benjamin Cabé <benjamin@zephyrproject.org>
Assisted-by: Claude:opus-4.8
Deduplicating against the output list with "not in" ran the model's deep
__eq__ for every comparison and dominated serialization time. Track the
gathered element IDs in a set and skip by ID instead.
Signed-off-by: Benjamin Cabé <benjamin@zephyrproject.org>
Assisted-by: Claude:opus-4.8
As there are multiple custom nrf* runners which needs dev-id,
handle them in general way for future.
Signed-off-by: Piotr Kosycarz <piotr.kosycarz@nordicsemi.no>
Before this change, whenever a Gtest failed the runner failed reporting
that `instance.reason` was a `NoneType` and couldn't be concatenated
to string.
This PR fix that by adding a reason to the Gtest instance whenever
it failed. Additionally, it covers the edge case where Gtest
started but it has never finished executing the test
Signed-off-by: Martin Moya <moyamartin1@gmail.com>
When loading a saved test plan with --load-tests/-F, get_platform()
returns None for any platform name in the file that is not part of the
current build. TestInstance() then dereferences platform.name and fails
with an opaque "AttributeError: 'NoneType' object has no attribute
'name'", which gives no hint about which entry is at fault.
Raise a TwisterRuntimeError that names the file, the unknown platform
and the test instead, so a stale or mismatched test plan is easy to
diagnose.
Assisted-by: AI (GitHub Copilot)
Signed-off-by: Bill Waters <bill.waters@infineon.com>
- fix mplab_ipe runner to use the board by serial number
- fix twister handler to pass dev_id to mplab_ipe runner
- fix map file generation to include MPLAB boards
Signed-off-by: Shankar Ramasamy <shankar.ramasamy@microchip.com>
Rename functions, class attributes, and local variables from camelCase
to snake_case across the zspdx library and west spdx command. This is
a mechanical rename with no functional changes.
Signed-off-by: Benjamin Cabé <benjamin@zephyrproject.org>
Explicitly open SPDX text output files with UTF-8 encoding to avoid
invalid UTF-8 content from platform-default encoding.
Signed-off-by: David Nepožitek <david@nepozitek.cz>
Add an SPDX 3.0 (JSON-LD) serializer built on the spdx-python-model
package and wire it into makeSPDX, so that running west spdx with
--spdx-version 3.0 produces SPDX 3.0 output. One JSON-LD document is
written per SBOM document (app, zephyr, build, modules-deps); each
emits its packages, files and relationships and declares conformance
to the Core, Software and SimpleLicensing profiles.
- Add zspdx.serializers.spdx3.SPDX3Serializer.
- Register SPDX_VERSION_3_0 in the supported versions list.
- Add the spdx-python-model dependency.
- Add spdx3-validate and validate the generated SPDX 3.0 documents in
the software_bill_of_materials test.
- Generalize the west spdx command description and document SPDX 3.0
usage.
Signed-off-by: Benjamin Cabé <benjamin@zephyrproject.org>
7b543c84ac moved the testcase status/reason finalization into
Console.handle() and added an `else` branch for the NONE status that
logged "Patterns left unmatched" and set tc.reason accordingly. Because
handle() runs once per console line and self.status stays NONE until all
patterns have matched, that branch fired on every line of a normal run.
This produced repeated bogus "ERROR - Patterns left unmatched" output on
any regex-matching test and left tc.reason set to "Patterns left
unmatched" even on tests that subsequently passed (e.g.
samples/synchronization reported reason="Patterns left unmatched" while
status="passed").
Signed-off-by: Anas Nashif <anas.nashif@intel.com>
Platforms marked with `twister: false` were dropped entirely before being
added to the platform list, so any test referencing them (platform_allow,
integration_platforms, etc.) failed platform-existence verification and
twister aborted on the unidentified platform.
Keep such platforms in the platform list so references to them still
resolve, but never add them as default platforms nor assign them tests.
This allows disabling a platform in CI without having to scrub every
test that references it.
Signed-off-by: Anas Nashif <anas.nashif@intel.com>
Assisted-by: Claude
Add type annotations in harness and testcase classes for clearer APIs and
better static checking.
Refine Console harness testcase status handling so FAIL/ERROR states and
reasons are propagated, and report unmatched patterns when execution ends
without a definitive pass/fail outcome.
Signed-off-by: Lukasz Fundakowski <lukasz.fundakowski@nordicsemi.no>
Add a new "post_build" operation to the twister build pipeline that runs
checks against the freshly built artifacts before metrics gathering. The
build stage hands off to post_build, which fails the instance (ERROR,
missing cases BLOCKed) if any check reports a problem.
The first check, check_no_nested_git_repos, walks the build directory for
any nested git repository (a .git directory or file) and fails the build
if found. A test or platform should never clone a git tree into its build
directory; enforcing this per-build attributes the failure to the specific
test instance and works for local runs as well as CI.
The checks are gated behind a new --post-build-checks option and are
disabled by default, as they add a build-directory walk per build. The
option is enabled in the twister CI workflow (TWISTER_COMMON), which also
drops the previous CI-only "find .git" step.
Added unit tests for the new operation, the check, and the option gating,
and updated the existing build-op process cases for the new transition.
Ran: ZEPHYR_BASE=$PWD python -m pytest
scripts/tests/twister/test_runner.py (144 passed).
Assisted-by: Claude:claude-opus-4-8
Signed-off-by: Anas Nashif <anas.nashif@intel.com>
Generated source files that live under the build directory (e.g.
zephyr/isr_tables.c, zephyr/misc/generated/configs.c) were being listed
under the "app" build package (the one that produces libapp.a) even
though they are not sources of libapp.a.
All build-target components share the build directory as their base_dir,
so findOwningComponent() resolved ownership purely by path prefix and
returned whichever build target happened to be iterated first ("app")
for every file under the build directory. The CMake codemodel reports
these generated files with absolute build-tree paths, so the previous
path-only search could not tell the targets apart.
Carry the build target that referenced each source through the pending
queue and let findOwningComponent() prefer that target when the file
lives under the build directory. Generated files now land in the
component that actually generates them (e.g. configs.c -> zephyr,
isr_tables.c -> zephyr_final) instead of the app package.
Also normalize non-absolute source paths joined from the codemodel so a
relative build-tree path cannot lexically alias the app sources
directory.
Fixes#108005
Assisted-by: Claude:opus-4.8
Signed-off-by: Benjamin Cabé <benjamin@zephyrproject.org>
Rework the SBOM build pipeline to populate the format-agnostic
zspdx.model and emit SPDX 2.x through SPDX2Serializer instead of the
legacy writer:
- Rewrite the walker to build an SBOMGraph of components, files and
relationships.
- Refactor the scanner to scan the SBOMGraph.
- Route makeSPDX through the SPDX 2.x serializer.
- Remove the now-unused legacy datatypes and writer modules.
No user-visible change to the SPDX 2.x output beyond the new
generation path.
Assisted-by: Claude:opus-4.8
Signed-off-by: Benjamin Cabé <benjamin@zephyrproject.org>
Add a format-agnostic serializer framework (zspdx.serializers) and an
SPDX 2.x tag-value serializer
(zspdx.serializers.spdx2.SPDX2Serializer) that consumes the
zspdx.model SBOMGraph. It is not yet wired into the SBOM build path;
that switch is done in a follow-up change.
Mark the legacy zspdx.writer functions as deprecated.
Assisted-by: Claude:opus-4.8
Signed-off-by: Benjamin Cabé <benjamin@zephyrproject.org>
Add zspdx.model, a set of format-agnostic dataclasses (SBOMGraph,
SBOMDocument, SBOMComponent, SBOMFile, SBOMRelationship and the
ComponentPurpose/RelationshipType enums) meant to back both SPDX 2.x
and SPDX 3.0 serialization in later changes.
Mark the legacy SPDX 2.x structures in zspdx.datatypes as deprecated;
they are kept for backward compatibility during the migration.
Assisted-by: Claude:opus-4.8
Signed-off-by: Benjamin Cabé <benjamin@zephyrproject.org>
Previously --shuffle-tests required --subset to be set, which
limited shuffling to subset-based runs. Remove that restriction
so test execution order can be randomized in any run.
When shuffling with a random seed across multiple subsets, not
all tests are guaranteed to be covered. Add a warning suggesting
a fixed seed via --shuffle-tests-seed in that case.
Signed-off-by: Grzegorz Chwierut <grzegorz.chwierut@nordicsemi.no>
When a ZTEST_P test is dispatched with multiple parameter values the
console output previously repeated the same test name for every
invocation, making it impossible to tell which parameter value caused
a failure at a glance.
Format a unique display name for each parameterized invocation using
the same bracket convention as Google Test:
test_name[instance_name/N]
where N is the 0-based index, or a custom label returned by the
optional name_cb callback:
test_name[instance_name/custom_label]
The display name is stored in a 128-byte buffer inside the existing
z_ztest_param_ctx struct and is used only by TC_START() and
Z_TC_END_RESULT() inside run_test(); the base test->name is unchanged
everywhere else (stats, verbose summary).
Non-parameterized tests are not affected: the display_name buffer is
cleared before each plain run_test() call.
The Twister Test harness test_case_start_pattern is relaxed from the
character-class [a-zA-Z0-9_-]+ to \S+ so it can parse the new
[instance/N] suffix; the end-of-test pattern already used \S* and
requires no change.
Before:
START - test_param_index
PASS - test_param_index in 0.000 seconds
START - test_param_index
PASS - test_param_index in 0.000 seconds
After:
START - test_param_index[idx_check/0]
PASS - test_param_index[idx_check/0] in 0.000 seconds
START - test_param_index[idx_check/1]
PASS - test_param_index[idx_check/1] in 0.000 seconds
Assisted-by: GitHub Copilot:claude-sonnet-4-5
Signed-off-by: Anas Nashif <anas.nashif@intel.com>
Extract XUnitXMLReport and JsonReport classes from the monolithic
Reporting class so that each class has a single, well-defined
responsibility. Move build error parsing methods
(_parse_cmake_build_failure, _parse_build_failure, get_detailed_reason)
to a dedicated class.
Rename xunit_report() to XUnitXMLReport.create() and
xunit_report_suites() to XUnitXMLReport.create_with_all_testsuites()
to better reflect their purpose. Add type annotations to new class
constructors and key methods.
Update Reporting.reports() to instantiate and delegate to the new
classes instead of calling methods on self directly.
Signed-off-by: Lukasz Fundakowski <lukasz.fundakowski@nordicsemi.no>
Prevent reuse to trip on "... SPDX-License-Identifier: tag." and
complain about "License may not be allowed:License file for 'tag.'"
Signed-off-by: Benjamin Cabé <benjamin@zephyrproject.org>
Do not have different sorting based on mode, always sort the same and
run all tests in the same subsets or runnings. A test failing on
multiple platforms should not cause majority of runners to fail.
Also adapt related tests.
Signed-off-by: Anas Nashif <anas.nashif@intel.com>
Keep zephyr/edt.pickle when cleaning artifacts for test-only
packages.
Add edt.pickle to the single-domain and multi-domain allow-lists
used by cleanup_device_testing_artifacts(). Update the related
Twister unit tests to match the new retained artifact lists.
Some runners need edt.pickle at flash time to resolve the code
partition from the EDT, so removing it breaks flashing from
packaged device-testing artifacts.
Signed-off-by: Hake Huang <hake.huang@nxp.com>
Do not raise exception when exe file was not found,
as it is related with build failure. Allow processing
of exception handling in build stage to keep build.log
in twister report output.
Signed-off-by: Grzegorz Chwierut <grzegorz.chwierut@nordicsemi.no>
The ``west.log`` module is deprecated. Drop it from every module under
``scripts/pylib/zspdx/`` and switch to standard Python
``logging.getLogger(__name__)`` instead. The reusable
``forward_logging_to_west()`` bridge wired up in ``spdx.py``routes
records emitted under the ``zspdx`` logger tree back to the command's
``self.dbg/inf/wrn/err/die``.
Call-site mapping:
log.dbg -> _logger.debug
log.inf -> _logger.info
log.wrn -> _logger.warning
log.err -> _logger.error
``except ... as e: log.err(..., str(e))`` pairs are simplified to
``except ...: _logger.exception(...)`` which already includes the
exception text and traceback.
Signed-off-by: Pieter De Gendt <pieter.degendt@basalte.be>
Assisted-by: Claude:claude-opus-4.7
Add support to build required_applications for a given test
automatically. Users no longer need to manually specify them
on the command line with -T and -s options.
If a required application is not found in testsuites, Twister
searches for it in the same directory as the tests.yaml file.
A new `path` field in required_applications and required_devices
allows specifying an explicit search directory.
Rename the 'name' field to 'application' in required_applications
to align the API with required_devices which already uses
'application'. The old 'name' field is kept for backward
compatibility but marked as deprecated with a runtime warning.
Signed-off-by: Grzegorz Chwierut <grzegorz.chwierut@nordicsemi.no>
