Raise the minimum required CMake version from 3.20.0 (documented as
3.20.5) to 3.28.0, which is satisfied by the CMake 3.28.3 package
shipped in the Ubuntu 24.04 LTS repositories. Ubuntu 24.04 is the
current Ubuntu LTS release targeted by the Zephyr getting started
guide, and by the time of the next Zephyr release, Ubuntu 22.04 will be
within months of its end of standard support. Users of distributions
shipping an older CMake can use the Kitware APT repository or a
pip-installed CMake, as the documentation already suggests.
Raising the floor to 3.28 unlocks a range of modern CMake features for
the build system, among which the cmake_file_api() command (3.27),
file(COPY_FILE) (3.21), block()/endblock() (3.25), and allows removal
of several version-conditional workarounds.
The tree-wide cmake_minimum_required() occurrences in samples, tests
and boards are updated accordingly, together with the documentation
and the sysbuild CMake presets. The IAR C-STAT integration keeps its
own higher requirement (4.1.0).
Assisted-by: Claude:fable-5
Signed-off-by: Benjamin Cabé <benjamin@zephyrproject.org>
Pytest and Ctest each duplicated an almost identical run() skeleton and a
byte-identical _update_test_status(); the two differed only in the tool
name used in reason strings, the exception type run() catches, and
whether pytest's PYTHONPATH env dependencies are injected.
Introduce a ReportHarness(Script) base that owns run() and
_update_test_status(). Subclasses now declare a report_tool label and a
run_exception type, and override the _run_report_command() hook when they
need to alter how the tool is invoked (Pytest injects env dependencies).
The triple of ExitCodes meaning "the tool errored" becomes the module
constant REPORT_ERROR_EXIT_CODES. Script keeps its own script-oriented
run()/_update_test_status(); Bsim is unaffected.
Assisted-by: Claude:claude-opus-4-8
Signed-off-by: Anas Nashif <anas.nashif@intel.com>
A formal review request is not always possible. GitHub rejects one for
anybody without collaborator status, so an area maintainer or collaborator
listed in MAINTAINERS.yml who lacks push access was silently dropped, and
the request as a whole may be refused for reasons the script cannot
anticipate. In both cases the people responsible for the changed code
never learned the PR existed.
Fall back to asking them by name. Users who cannot be added to the review
request are @mentioned in a comment requesting their review, which
notifies them regardless of their permissions. This covers both the
non-collaborator case and whatever the review request could not
accommodate, including the tail dropped when a rejected bulk request is
retried with fewer candidates.
The comment is maintained in place rather than appended: it carries a
hidden MENTION_MARKER, so a re-run finds its own previous comment and
edits it only when the set of people actually changed. Re-running over a
PR therefore neither duplicates the comment nor adds timeline noise.
Users who removed themselves from the review request are excluded, and
the self-removal test now runs before the collaborator test so that an
opt-out is never routed to the mention path instead. Deleted accounts are
still skipped outright, there being nobody to mention. A failure to post
the comment is logged and swallowed, like the review request itself, so
that a repository without comment write access still completes its run.
Assisted-by: Claude Opus 4.8 <noreply@anthropic.com>
Signed-off-by: Anas Nashif <anas.nashif@intel.com>
MAX_REVIEWERS (15) drove two behaviours: the candidate list was truncated
to the remaining vacancy, and a PR already holding that many reviewers
discarded its candidate list entirely in favour of the primary area's
maintainers. Both predate the tiered candidate ordering, and neither
still earns its keep.
The cap was never GitHub's. GitHub documents no limit on how many
reviewers a pull request may carry, and PRs with more than 15 individual
reviewers requested (no teams involved) are observable in this very
repository, so the number was self-imposed. Its only effect was to drop
people who were legitimately responsible for part of the change.
The overflow strategy was worse than redundant. It compared
MAX_REVIEWERS against a set that unions submitted reviews with pending
review requests, so a well-reviewed PR could take the restricted path
with no pending requests at all and plenty of room to add more. Now that
maintainers are ordered ahead of collaborators, the ordering already does
what the strategy was reaching for.
Drop the cap and request everyone eligible. This also retires
_add_reviewers()'s primary_maintainers and extra_reviewers parameters,
which were referenced only from the removed branch; additional_reviews is
folded into the candidate list by _build_reviewer_candidates and needs no
separate path. Should GitHub refuse the full set anyway, the call is
retried once with the first REVIEWER_RETRY_BATCH candidates, which the
tier ordering makes the maintainers, so a rejection cannot leave the PR
with no reviewers at all.
Signed-off-by: Anas Nashif <anas.nashif@intel.com>
Assisted-by: Claude Opus 4.8 <noreply@anthropic.com>
Every reviewer tier is derived from the areas a changed file maps to, so a
file that matches no area in MAINTAINERS.yml contributes nobody. A PR
made up only of such files ends up with an empty candidate list, and the
review request is skipped entirely: nobody is asked to look at it.
Feed unmatched files into the same heuristics already used for
under-covered areas.
The heuristic pass now triggers on the union of under-covered-area files
and orphaned files, so a mixed PR still walks only the files that are
actually unstaffed, and a fully covered PR still issues no extra query.
Heuristic picks stay appended after the maintainer and collaborator
tiers, so they only ever fill leftover review slots.
Assisted-by: Claude Opus 4.8 <noreply@anthropic.com>
Signed-off-by: Anas Nashif <anas.nashif@intel.com>
Some areas in MAINTAINERS.yml name no maintainers, or only a handful of
collaborators. When a PR touches such an area, the reviewer candidate
list built from MAINTAINERS.yml alone is too thin to attract a meaningful
review, and nobody responsible for the change is requested.
Supplement these areas heuristically. An area is treated as
under-covered when it names no maintainers, or at most
THIN_AREA_COLLABORATORS (2) collaborators. For every such area two
sources are tried in order, and the result is appended after the
maintainer and collaborator tiers so that it only ever fills leftover
review slots.
First, GitHub's own suggestions for the pull request
(PullRequest.suggestedReviewers), which GitHub derives from both commit
history and past review comments. That is the better-informed signal and
costs a single query instead of one per changed file, so suggestions
GitHub flagged as authors of the changed code are ranked ahead of
comment-only ones. The field is only available over GraphQL, so this
issues a raw query through PyGithub's requester, which reuses the
existing token and session; no new dependency is needed. Every failure
path (transport error, GraphQL errors, unexpected shape, null list)
degrades to an empty list, so a bad response can only cost the
suggestion, never the run.
Second, when GitHub suggests nobody, the recent contributors to the
changed files the under-covered areas own, read from the commit history.
This fallback is not redundant: sampling a dozen open PRs showed GitHub
returning suggestions for only about a third of them, with no visible
correlation to how many reviewers were already requested. The commit
walk also stays scoped to the files that are actually short of reviewers,
whereas GitHub's suggestions cover the PR as a whole. History is read
through the GitHub API rather than a local 'git log'/'git blame': the API
resolves commits to real GitHub logins (author emails cannot be mapped to
logins reliably) and does not depend on the CI checkout having full
history.
In both cases the PR author and everyone already on the candidate list
are excluded, so these last-resort slots are not spent on reviewers the
PR already has, and the commit walk is bounded by
HISTORY_COMMITS_PER_FILE, MAX_HISTORY_FILES, and MAX_HISTORY_REVIEWERS to
keep the API cost predictable.
Signed-off-by: Anas Nashif <anas.nashif@intel.com>
Assisted-by: Claude Opus 4.8 <noreply@anthropic.com>
The reviewer candidate list was built by iterating touched areas in
descending weight order and appending each area's maintainers immediately
followed by its collaborators. _add_reviewers() then truncates the
filtered list to the remaining reviewer vacancy (MAX_REVIEWERS minus the
existing reviewer count).
On a fresh PR that touches many areas, the collaborators of the
highest-weight areas were reached before the maintainers of the
lower-weight areas, so the review-request slots could be filled with
collaborators while the maintainers of several touched areas were never
requested.
Build the candidate list in two tiers instead: every touched area's
maintainers first (in descending weight order), plus the manifest /
MAINTAINERS.yml-change and deferred-file-group maintainers, followed by
all area and path-specific collaborators. Truncation to the vacancy now
drops collaborators before it drops any maintainer. Set-derived inputs
are sorted so candidate ordering is deterministic.
Signed-off-by: Anas Nashif <anas.nashif@intel.com>
With no tests found when pointing to a directory with no tests, which is
not wrong, we were exiting with 1, when twister was crashing because of
bad syntax or when not able to load a python module, same thing.
Now fail early when there are schema errors or other issues, and fail
with a warning when no tests are found, which should be fine given that
a warning is emitted, but not an error.
Signed-off-by: Anas Nashif <anas.nashif@intel.com>
Twister exits with an error when --enable-valgrind is given but the
valgrind executable is not on PATH. The blackbox test calls twister
in-process, so that exit propagated as SystemExit and failed the test
on hosts without valgrind installed.
Add a requires_tool() helper to the blackbox conftest that builds a
skipif marker from shutil.which(), and apply it to the valgrind
parametrization. The baseline case without the option is unaffected and
still runs everywhere.
Assisted-by: Claude:opus-4.8
Signed-off-by: Anas Nashif <anas.nashif@intel.com>
Replace the importlib-based module loader pattern used across blackbox
tests with direct calls to twister_main(), and overhaul the shared
conftest.py: rename fixtures for clarity, isolate logging state via a
dedicated _reset_logging() helper and autouse fixture, add helper
functions read_testplan(), active_testcases(), and active_testsuites(),
and introduce session-scoped hello_world_elf and out_path fixtures.
Add pytest markers fast, build, and run. Rename test_filename_mock to
TEST_FILENAME_MOCK and resolve TEST_DATA through os.path.realpath() to
avoid symlink mismatches. Delete test_addon.py and restructure
test_config.py and test_coverage.py into focused classes grouped by
feature.
Assisted-by: Claude:opus-4.8
Signed-off-by: Anas Nashif <anas.nashif@intel.com>
Add process-level caches to TwisterEnv.check_zephyr_version() and
get_toolchain() so repeated twister_main() calls within the same
process skip slow git/cmake subprocess re-runs, and add a module-level
cache to generate_platforms() keyed on the board/soc/arch root tuples.
This significantly speeds up in-process invocations such as the
blackbox test suite.
Assisted-by: Claude:opus-4.8
Signed-off-by: Anas Nashif <anas.nashif@intel.com>
When the Robot Framework executable is not found, subprocess
raised an unhandled FileNotFoundError that aborted the entire
Twister run instead of just the affected test.
Catch the exception in the Robot harness, mark the instance as
skipped with a descriptive reason, and continue so remaining
tests still execute.
Signed-off-by: Grzegorz Chwierut <grzegorz.chwierut@nordicsemi.no>
Now that we are building with multiple toolchains some failures will only
happen with one toolchain.
So, also print the toolchain used, so it is easier to identify the issue.
And update the twister blackbox tests reference data.
Signed-off-by: Alberto Escolar Piedras <alberto.escolar.piedras@nordicsemi.no>
Twister picks exactly one toolchain per platform, so a board that is
buildable with several toolchains only ever gets coverage for one of
them. The only way to build a test more than once was the test scenario
level integration_toolchains option, which cannot express "build
everything on this platform with both GCC and Clang".
Add a build_toolchains option listing the toolchains every test assigned
to a platform should be built with. Twister then creates one test
instance per toolchain, each in its own build directory. The option can
be set in the board configuration, or per platform in the twister
configuration file, which lets multi-toolchain coverage be enabled for
CI only while local runs keep building each test once. An empty list in
the twister configuration disables it for a platform that requests it.
The toolchain resolution chain moves into get_toolchains(), keeping the
existing precedence, with build_toolchains taking effect just below a
scenario's integration_toolchains.
Toolchain is also added to the platform_key deduplication key. That key
deduplicates platforms, not toolchains, and without this every extra
toolchain of a scenario using platform_key is discarded as already
covered.
Assisted-by: Claude:claude-opus-4-8
Signed-off-by: Anas Nashif <anas.nashif@intel.com>
Some QEMU tests need a host-side process running for the guest to talk to
(a virtiofsd daemon, an ivshmem reader, a network tap). Model this as a
first-class "sidecar": a testsuite field orthogonal to the harness, so a
test keeps its normal harness (ztest, console, ...) and just declares a
sidecar, and twister may also attach one itself.
This commit adds only the plumbing and the base concept:
- A Sidecar base class with configure()/setup()/teardown() and a
SidecarImporter registry (empty here; concrete sidecars register in
later commits), in twisterlib/sidecar.py.
- The `sidecar:` testsuite field (config_parser, TestSuite) and the
per-instance sidecar/dtc_overlay attributes (TestInstance).
- The runner wraps handler.handle() with the sidecar's setup/teardown and
skips the run cleanly if setup() reports the host side is unavailable.
Assisted-by: Claude:claude-opus-4-8
Signed-off-by: Anas Nashif <anas.nashif@intel.com>
ManifestStrategy.analyze() returned set(manifest_files) - the set of
consumed filenames - as the calls list when west.yml changed but no
project revision differed (e.g. an unrelated field or formatting
edit). TestPlanOrchestrator.run() iterates calls and accesses
call.full_run, so a plain filename string raised:
AttributeError: 'str' object has no attribute 'full_run'
This affects any PR that touches west.yml without bumping a project
revision, independent of the actual manifest change. Return an empty
calls list instead, and add regression tests covering both the
no-revision-change and revision-change paths.
Assisted-by: Claude:claude-sonnet-5
Signed-off-by: Minyuan Xue <minyuan_xue@realsil.com.cn>
Add scripts/gen_test_matrix_dashboard.py, a standalone tool that turns a
Twister per-test coverage matrix (test_matrix.json) into a self-contained,
interactive HTML dashboard. It has no dependency on Twister and can be run
on any previously generated matrix.
The dashboard shows, per test, how many files and lines it covers
alongside how many lines it covers uniquely -- lines no other test
reaches -- with sortable columns so redundant tests (zero unique) and
load-bearing tests stand out. Selecting a test lists the files it covers
with per-file line and unique-line counts, and a file lookup shows which
tests cover each line, highlighting uniquely-covered lines. All derived
metrics are computed in the browser from the matrix inlined into the page.
Assisted-by: Claude:claude-opus-4-8
Signed-off-by: Anas Nashif <anas.nashif@intel.com>
Add a --coverage-per-test mode that collects an isolated coverage
artifact for every Ztest test case and post-processes them into a
test-to-code coverage matrix, answering "which tests covered line X of
file foo.c".
The mode builds on CONFIG_ZTEST_COVERAGE_PER_TEST. On platforms that
support semihosting (ARM/RISC-V/Xtensa under QEMU, e.g. mps2/*) the
transport is switched to semihosting so per-test data is written straight
to the host as "<gcda>@@<suite>.<test>", avoiding the large amount of
serial traffic per-test dumping would otherwise generate; other platforms
fall back to the console transport. The required Kconfig is injected into
the build overlay.
Each test is captured by invoking gcov directly -- one
"gcov --stdout --json-format" per test over its gcda set, materialized in
a temporary object tree with symlinks to the shared gcno -- and its JSON
is converted to an lcov tracefile. This is far cheaper than a full lcov
capture per test; lcov capture remains a fallback for non-GCC toolchains.
To keep the end-of-run aggregation proportional to the number of
instances rather than the number of tests, each instance's per-test
tracefiles are collapsed into a single untagged union, and the aggregate
matrix is built by merging the per-instance matrices. Function records
are omitted from the tracefiles because gcov reports per-function counts
and end lines inconsistently across per-test dumps, which would fail
lcov's merge consistency checks; line and branch coverage are unaffected.
Per-test mode requires the lcov coverage tool (the matrix relies on lcov
TN records; gcovr has no equivalent) and this is validated at
argument-parsing time. Output is one <scenario>.<test>.info per test
under each build's coverage/tests/ directory plus a test_matrix.json with
by_line ({file:{line:[tests]}}) and by_test views.
Covered by unit tests for the tracefile conversion, tracefile union,
matrix build and merge, and the tagged-dump discovery and parsing
helpers.
Assisted-by: Claude:claude-opus-4-8
Signed-off-by: Anas Nashif <anas.nashif@intel.com>
Some simulators resolve their executable at CMake configure time via
find_program() instead of being known during test planning, and the
binary that gets used can depend on the build configuration. Arm FVP is
the motivating case: Corstone-300 selects a different model depending on
the Ethos-U NPU, so armfvp cannot be listed in SUPPORTED_SIMS_WITH_EXEC
and checked up front. As a result, when the FVP binary was not installed
Twister still considered the test runnable, attempted to run nothing, and
reported a misleading status.
Inspect CMakeCache after the build instead: add a SIM_PROGRAM_CMAKE_VARS
mapping of simulator name to the cache variable recording its resolved
executable. When the variable is absent or holds a *-NOTFOUND value
(e.g. ARMFVP:FILEPATH=ARMFVP-NOTFOUND), mark the instance as NOTRUN with
a clear reason and propagate that status to its test cases, so the test
is no longer reported as runnable.
This check is done in the post_build stage, which now always runs after a
successful build; the --post-build-checks option only gates the optional
build-failing checks, which are skipped inside post_build when it is not
set. This keeps the not-run determination separate from gather_metrics,
where it did not belong.
Assisted-by: Claude:claude-opus-4-8
Signed-off-by: Anas Nashif <anas.nashif@intel.com>
Clean up Twister OpenOCD runner by removing
deprecated cmsis_dap_serial argument and
replacing with adapter serial
Signed-off-by: Shankar Ramasamy <shankar.ramasamy@microchip.com>
Use suite id instead of name as the key in the testsuites
registry. With --detailed-test-id enabled, the suite name
includes the path, which caused a false duplicate error when a
required application was passed via -T on the command line and
was also listed in the required_application section of another
test.
Duplicated scenarios are reported as an error regardless of
whether --detailed-test-id is used.
Update the twister unit tests.
Signed-off-by: Grzegorz Chwierut <grzegorz.chwierut@nordicsemi.no>
Add a new optional boolean field defer-to-other-areas to file-groups in
MAINTAINERS.yml. When set to true it signals that the files in that
group are not the sole responsibility of the enclosing area: if another
area also covers the same files without the flag, that other area's
maintainer takes precedence as the PR assignee and the deferred area's
maintainers are added as reviewers instead. When no non-deferred area
covers the files, the deferral has no effect and the area is used for
assignment as normal.
Typical use-case: the Clock Control subsystem owns generic driver
infrastructure under drivers/clock_control/, but a platform area also
covers the platform-specific files there. The subsystem maintainer
adds those files to a deferred file-group so the platform maintainer
becomes the assignee while the subsystem maintainer is still notified
as a reviewer.
Assisted-by: GitHub Copilot:claude-sonnet-4.6
Signed-off-by: Anas Nashif <anas.nashif@intel.com>
Before this change, whenever a Gtest failed the runner failed reporting
that `instance.reason` was a `NoneType` and couldn't be concatenated
to string.
This PR fix that by adding a reason to the Gtest instance whenever
it failed. Additionally, it covers the edge case where Gtest
started but it has never finished executing the test
Signed-off-by: Martin Moya <moyamartin1@gmail.com>
When loading a saved test plan with --load-tests/-F, get_platform()
returns None for any platform name in the file that is not part of the
current build. TestInstance() then dereferences platform.name and fails
with an opaque "AttributeError: 'NoneType' object has no attribute
'name'", which gives no hint about which entry is at fault.
Raise a TwisterRuntimeError that names the file, the unknown platform
and the test instead, so a stale or mismatched test plan is easy to
diagnose.
Assisted-by: AI (GitHub Copilot)
Signed-off-by: Bill Waters <bill.waters@infineon.com>
Platforms marked with `twister: false` were dropped entirely before being
added to the platform list, so any test referencing them (platform_allow,
integration_platforms, etc.) failed platform-existence verification and
twister aborted on the unidentified platform.
Keep such platforms in the platform list so references to them still
resolve, but never add them as default platforms nor assign them tests.
This allows disabling a platform in CI without having to scrub every
test that references it.
Signed-off-by: Anas Nashif <anas.nashif@intel.com>
Assisted-by: Claude
Add a new "post_build" operation to the twister build pipeline that runs
checks against the freshly built artifacts before metrics gathering. The
build stage hands off to post_build, which fails the instance (ERROR,
missing cases BLOCKed) if any check reports a problem.
The first check, check_no_nested_git_repos, walks the build directory for
any nested git repository (a .git directory or file) and fails the build
if found. A test or platform should never clone a git tree into its build
directory; enforcing this per-build attributes the failure to the specific
test instance and works for local runs as well as CI.
The checks are gated behind a new --post-build-checks option and are
disabled by default, as they add a build-directory walk per build. The
option is enabled in the twister CI workflow (TWISTER_COMMON), which also
drops the previous CI-only "find .git" step.
Added unit tests for the new operation, the check, and the option gating,
and updated the existing build-op process cases for the new transition.
Ran: ZEPHYR_BASE=$PWD python -m pytest
scripts/tests/twister/test_runner.py (144 passed).
Assisted-by: Claude:claude-opus-4-8
Signed-off-by: Anas Nashif <anas.nashif@intel.com>
Add scripts/tests/ci/test_set_assignees.py with 50 tests covering the
pure-logic and lightly-stateful helpers in set_assignees.py.
Heavy third-party dependencies (PyGithub, west, get_maintainer) are
stubbed out in sys.modules before the SUT is imported so the suite
runs without a GitHub token or a full Zephyr build environment.
Coverage:
load_areas - includes/excludes areas based on
files/files-regex keys
set_or_empty - missing key, None value, empty list
_diff_area_entry - maintainer/collaborator/label/files/status changes
compare_areas - added, removed, changed, unchanged areas
update_size_xs_label
- XS qualification criteria (commits, additions,
deletions, manifest files)
- stale label removal with and without dry-run
_pick_assignees - non-platform priority, platform fallback, author
exclusion, meta-only area, zero-count skip,
empty-maintainer skip, all-maintainers fallback,
zero num_files divide-by-zero guard
setup_logging - verbose=0/1/2/3 log level mapping
Assisted-by: GitHub Copilot:claude-sonnet-4.6
Signed-off-by: Anas Nashif <anas.nashif@intel.com>
Previously --shuffle-tests required --subset to be set, which
limited shuffling to subset-based runs. Remove that restriction
so test execution order can be randomized in any run.
When shuffling with a random seed across multiple subsets, not
all tests are guaranteed to be covered. Add a warning suggesting
a fixed seed via --shuffle-tests-seed in that case.
Signed-off-by: Grzegorz Chwierut <grzegorz.chwierut@nordicsemi.no>
Add a new BoilerplateFilter consuming strategy that runs after the
existing IgnoreStrategy. The filter inspects the actual diff of each
changed file and silently consumes those whose entire change set is
non-substantive, i.e. made up exclusively of:
- Whitespace adjustments and blank-line additions/removals
(detected via 'git diff -w --ignore-blank-lines').
- SPDX-License-Identifier and SPDX-FileCopyrightText header lines.
- Copyright notice lines.
- Lines consisting solely of comment delimiter characters
(/* */ // # *).
A file is only consumed when every added/removed line in its diff falls
into one of those categories. A single substantive line (changed
statement, macro, declaration) causes the file to pass through to
downstream strategies unchanged so normal test selection proceeds.
The filter is a no-op when no git commit range is available (e.g. when
--modified-files is used without --commits), because the diff cannot be
computed in that case.
Also add 14 unit tests in scripts/tests/ci/test_test_plan_v2.py covering
the helper methods and the analyze() interface, and update the developer
documentation in doc/develop/test/ci_test_plan.rst with a new
'Boilerplate filter' section and a table row for the strategy.
Assisted-by: GitHub Copilot:claude-opus-4.6
Signed-off-by: Anas Nashif <anas.nashif@intel.com>
Do not have different sorting based on mode, always sort the same and
run all tests in the same subsets or runnings. A test failing on
multiple platforms should not cause majority of runners to fail.
Also adapt related tests.
Signed-off-by: Anas Nashif <anas.nashif@intel.com>
Keep zephyr/edt.pickle when cleaning artifacts for test-only
packages.
Add edt.pickle to the single-domain and multi-domain allow-lists
used by cleanup_device_testing_artifacts(). Update the related
Twister unit tests to match the new retained artifact lists.
Some runners need edt.pickle at flash time to resolve the code
partition from the EDT, so removing it breaks flashing from
packaged device-testing artifacts.
Signed-off-by: Hake Huang <hake.huang@nxp.com>
Add support to build required_applications for a given test
automatically. Users no longer need to manually specify them
on the command line with -T and -s options.
If a required application is not found in testsuites, Twister
searches for it in the same directory as the tests.yaml file.
A new `path` field in required_applications and required_devices
allows specifying an explicit search directory.
Rename the 'name' field to 'application' in required_applications
to align the API with required_devices which already uses
'application'. The old 'name' field is kept for backward
compatibility but marked as deprecated with a runtime warning.
Signed-off-by: Grzegorz Chwierut <grzegorz.chwierut@nordicsemi.no>
Extend test_check_build_or_run() to verify both existing and
non-existing QEMU_BIN_PATH values.
The previous test used an empty QEMU_BIN_PATH value, which only
covered the falsy branch and did not exercise os.path.exists().
Add explicit checks for valid and invalid non-empty paths to
improve coverage of the QEMU_BIN_PATH validation logic.
Signed-off-by: Gobi M <gobi@aerlync.com>
Add a new testsuite YAML option `build` that allows a test
scenario to skip its own build step and instead use the build
artifacts from the first entry in `required_applications`.
Default is true; to skip building, set `build: false`.
This is useful for test scenarios that only run a harness (e.g.,
pytest, bsim) against an image built by another scenario, without
needing their own compiled binary.
Signed-off-by: Grzegorz Chwierut <grzegorz.chwierut@nordicsemi.no>
Add a parametrized pytest in scripts/tests/twister/ that exercises
apply_filters() for a testsuite whose integration_platforms list mixes
default and non-default platforms, asserting the documented platform
scope under both -G (integration mode) and the default mode.
The non-default path (no -G) silently regressed once in #32835 by
dropping integration_platforms from the per-testsuite scope; this test
pins both branches so a similar regression is caught in CI.
The fixture testsuite lives in a separate testsuites_integration/ tree
so that the shared all_testsuites_dict fixture, which calls
add_testsuites() before platforms are loaded, is not affected by the
new fixture's verify_platforms_existence() requirement.
Fixes#33247
Signed-off-by: Andrii Anoshyn <anoshyn.andrii@gmail.com>
Remove no-name-in-module lint, which we attempt to disable but keeps
coming back with no easy way to subpress.
Signed-off-by: Anas Nashif <anas.nashif@intel.com>
Rename test fixture files from test_sample.yaml to test_data.yaml and
update all references across twister unit and blackbox tests to use
test_data.yaml.
Signed-off-by: Anas Nashif <anas.nashif@intel.com>
Rework the sysbuild artifact relocation approach based on
reviewer feedback: instead of rewriting domains.yaml with
relative paths at package time, keep the original absolute
paths and rebase them transparently in Domains.from_file()
when the file has been moved to a different location.
This makes the fix generic - it works for any consumer of
Domains.from_file() (west flash, twister, pytest-twister-
harness), not just the --package-artifacts path.
Changes:
- domains.py: update from_file() to detect when build_dir
stored in domains.yaml differs from the file's actual
directory and rebase all domain paths accordingly.
Remove the now-unused _resolve_build_dir() helper and
get_domains_with_build_dirs_relative_to() method.
- runner.py: remove _sanitize_domains_file() and its call
site; path fixup is now handled by Domains.from_file().
- test_domains.py: replace the mocked __init__ approach
with end-to-end assertions; add a 'relocated artifacts'
test case that exercises the new rebasing logic.
- test_runner.py: remove test_projectbuilder_sanitize_
domains_file() and references to the deleted method.
Signed-off-by: Hake Huang <hake.huang@nxp.com>
Introduce a Script harness base class to enable running
shell-script-based tests through Twister, primarily
targeting BabbleSim test scenarios. The class provides
subprocess execution, output streaming, and log handling
that Pytest, Ctest, and Bsim now share through inheritance.
Signed-off-by: Grzegorz Chwierut <grzegorz.chwierut@nordicsemi.no>
Add dt_label_compat_enabled() to filter targets that embed a node
label matching a target compatible string.
This helper function will be useful to consider zephyr,mapped-partition
compatible nodes that are now used instead of fixed-partition compatible
nodes for memory mapped flash partitions.
Signed-off-by: Etienne Carriere <etienne.carriere@st.com>
Allow multi-DUT test scenarios to run on native_sim simulator.
For non-device handlers, create placeholder reserved_duts entries
using CompoundHardwareData so pytest harness can find DUT configs.
Multi-DUT on native_sim is only allowed when all required devices
use the same platform (no cross-platform multi-DUT in simulation).
Signed-off-by: Grzegorz Chwierut <grzegorz.chwierut@nordicsemi.no>
Replace plain dict usage for required_application elements with
a typed RequiredApplication dataclass for improved type safety
Signed-off-by: Grzegorz Chwierut <grzegorz.chwierut@nordicsemi.no>
Add support for flashing a different application to a required
DUT in multi-DUT testing. When a required_device entry specifies
an 'application' field, twister matches it against the
required_applications build dirs and assigns the correct
build_dir to the reserved DUT before flashing.
Move the required applications readiness check from
process_tasks() into ProjectBuilder.process() at the 'run'
stage. This ensures that building is not blocked by the
required_applications mechanism. Mutual
dependencies between instances is also allowed.
Signed-off-by: Grzegorz Chwierut <grzegorz.chwierut@nordicsemi.no>
Main:
- Add HardwareReservationManager class and move DUT reservation
logic from HardwareMap to improve separation of concerns
- Add 'required_devices' structure to testsuite YAML schema to
define multiple device requirements for a test
- Add option to reserve multiple devices, filtered by platform
and fixtures. Fixtures can have an extra parameter (after ':')
that must match across the main DUT and required devices
Other:
- Add HarnessConfig dataclass to provide structured access to
harness configuration data instead of raw dict access
- Change 'fixture' parameter in harness_config from string to
list of strings for more flexibility when matching DUTs with
multiple fixture requirements
- Move conversion of short platform names to full target names
from hardware map into TestPlan module. This ensures platform
names are resolved before test discovery
- Add unit tests for HardwareReservationManager
- Align existing tests with new dataclass structures
Signed-off-by: Grzegorz Chwierut <grzegorz.chwierut@nordicsemi.no>
Parametrized tests in gtest are shown as the following example in a log:
[ FAILED ] FooTest/FooTypedTest/0.TypedTestFoo, where TypeParam = <type>
current regex of the gtest harness didn't detect such test so it marks
this as pass or that the result wasn't found. This commit extends the
regex to detect such behavior and adds unit test to verify that.
Signed-off-by: Martin Moya <moyamartin1@gmail.com>
Moved all `sys.path.insert` from twister's modules
to `twisterlib.__init__.py` to have only one place where all paths
are added.
Signed-off-by: Łukasz Fundakowski <lukasz.fundakowski@nordicsemi.no>
