Twister picks exactly one toolchain per platform, so a board that is
buildable with several toolchains only ever gets coverage for one of
them. The only way to build a test more than once was the test scenario
level integration_toolchains option, which cannot express "build
everything on this platform with both GCC and Clang".
Add a build_toolchains option listing the toolchains every test assigned
to a platform should be built with. Twister then creates one test
instance per toolchain, each in its own build directory. The option can
be set in the board configuration, or per platform in the twister
configuration file, which lets multi-toolchain coverage be enabled for
CI only while local runs keep building each test once. An empty list in
the twister configuration disables it for a platform that requests it.
The toolchain resolution chain moves into get_toolchains(), keeping the
existing precedence, with build_toolchains taking effect just below a
scenario's integration_toolchains.
Toolchain is also added to the platform_key deduplication key. That key
deduplicates platforms, not toolchains, and without this every extra
toolchain of a scenario using platform_key is discarded as already
covered.
Assisted-by: Claude:claude-opus-4-8
Signed-off-by: Anas Nashif <anas.nashif@intel.com>
Some QEMU tests need a host-side process running for the guest to talk to
(a virtiofsd daemon, an ivshmem reader, a network tap). Model this as a
first-class "sidecar": a testsuite field orthogonal to the harness, so a
test keeps its normal harness (ztest, console, ...) and just declares a
sidecar, and twister may also attach one itself.
This commit adds only the plumbing and the base concept:
- A Sidecar base class with configure()/setup()/teardown() and a
SidecarImporter registry (empty here; concrete sidecars register in
later commits), in twisterlib/sidecar.py.
- The `sidecar:` testsuite field (config_parser, TestSuite) and the
per-instance sidecar/dtc_overlay attributes (TestInstance).
- The runner wraps handler.handle() with the sidecar's setup/teardown and
skips the run cleanly if setup() reports the host side is unavailable.
Assisted-by: Claude:claude-opus-4-8
Signed-off-by: Anas Nashif <anas.nashif@intel.com>
Support using a different toolchain for a specific platform.
A platform can now set preferred_toolchain in the yaml file and this
toolchain will be used instead of the the default. This allows us to
test multiple toolchains and have some platforms build using the
preferred toolchain in CI. This applies only when running twister,
everything behaves as usual when building with west.
Signed-off-by: Anas Nashif <anas.nashif@intel.com>
Add support to build required_applications for a given test
automatically. Users no longer need to manually specify them
on the command line with -T and -s options.
If a required application is not found in testsuites, Twister
searches for it in the same directory as the tests.yaml file.
A new `path` field in required_applications and required_devices
allows specifying an explicit search directory.
Rename the 'name' field to 'application' in required_applications
to align the API with required_devices which already uses
'application'. The old 'name' field is kept for backward
compatibility but marked as deprecated with a runtime warning.
Signed-off-by: Grzegorz Chwierut <grzegorz.chwierut@nordicsemi.no>
Add a new testsuite YAML option `build` that allows a test
scenario to skip its own build step and instead use the build
artifacts from the first entry in `required_applications`.
Default is true; to skip building, set `build: false`.
This is useful for test scenarios that only run a harness (e.g.,
pytest, bsim) against an image built by another scenario, without
needing their own compiled binary.
Signed-off-by: Grzegorz Chwierut <grzegorz.chwierut@nordicsemi.no>
Introduce a Script harness base class to enable running
shell-script-based tests through Twister, primarily
targeting BabbleSim test scenarios. The class provides
subprocess execution, output streaming, and log handling
that Pytest, Ctest, and Bsim now share through inheritance.
Signed-off-by: Grzegorz Chwierut <grzegorz.chwierut@nordicsemi.no>
Main:
- Add HardwareReservationManager class and move DUT reservation
logic from HardwareMap to improve separation of concerns
- Add 'required_devices' structure to testsuite YAML schema to
define multiple device requirements for a test
- Add option to reserve multiple devices, filtered by platform
and fixtures. Fixtures can have an extra parameter (after ':')
that must match across the main DUT and required devices
Other:
- Add HarnessConfig dataclass to provide structured access to
harness configuration data instead of raw dict access
- Change 'fixture' parameter in harness_config from string to
list of strings for more flexibility when matching DUTs with
multiple fixture requirements
- Move conversion of short platform names to full target names
from hardware map into TestPlan module. This ensures platform
names are resolved before test discovery
- Add unit tests for HardwareReservationManager
- Align existing tests with new dataclass structures
Signed-off-by: Grzegorz Chwierut <grzegorz.chwierut@nordicsemi.no>
in some multi-core platform, `west flash` can not support all cores.
`west debug` is supported to test secondary core, add this to twister
such as with
`--west-flash-cmd=debug` in twister command
or with
`--west-flash-cmd: debug` in hwmap.yaml
and same with pytest
command line setting will overrid the one in hwmap
Signed-off-by: Hake Huang <hake.huang@nxp.com>
Co-authored-by: Grzegorz Chwierut <grzegorz.chwierut@nordicsemi.no>
Convert the DUT class from a traditional class to a dataclass to enable
proper serialization and support for future multi-device testing with
pytest-harness.
Key changes:
- Migrated DUT class to use `dataclass` decorator with proper type hints
- Renamed `baud` property to `serial_baud` for consistency
- Updated hardware map schema to support both `baud` (legacy) and
`serial_baud` fields for backward compatibility
- Updated tests
Signed-off-by: Grzegorz Chwierut <grzegorz.chwierut@nordicsemi.no>
Add support for boards to specify flash_before in their board YAML files,
which is then used by pytest and shell harnesses. This is needed for
Synopsys ARC development boards (hsdk, hsdk/arc_hsdk/2cores, hsdk4xd
and iotdk) where the USB serial port disconnects during flashing.
Signed-off-by: Mohamed Moawad <moawad@synopsys.com>
This feature enables sharing of built applications between test
scenarios, allowing tests to access build artifacts from other
applications.
Signed-off-by: Grzegorz Chwierut <grzegorz.chwierut@nordicsemi.no>
Extend the testcase schema to provide a way to communicate that a given
test is expected to reboot during execution. The generic harness takes
that information and suppresses `already STARTED` warnings if the
restart is expected.
Signed-off-by: Jordan Yates <jordan@embeint.com>
Add option to force integration mode on a defined list of tests, for
example tests for sample that are identified with 'sample.'.
A sample per definition shall be tested on documented and supported
platforms listed in the sample documentation. Samples shall not be used
as tests to verify functionality of a feature on all available plaforms
in Zephyr.
To still allow testing on platforms not listed in the doc, and when such
platforms are covered by the provided filter, we should still be able to
build/run the tests on those platforms (not directly listed in
integration_platforms).
We detect a sample by its test identifier, i.e., it starts with 'sample.'.
Signed-off-by: Anas Nashif <anas.nashif@intel.com>
Allow user to add shell commands in testcase/sample yaml file
alongside the harness_config like in the console harness.
Signed-off-by: Grzegorz Chwierut <grzegorz.chwierut@nordicsemi.no>
Introduce a new harness based on pytest that does basic shell command
handling. The harness is enabeld using:
harness: shell
and expects a file with parameters in the form:
test_shell_harness:
- command: "kernel version"
expected: "Zephyr version .*"
- ...
Multiple commands and their expected output can be tested.
Signed-off-by: Anas Nashif <anas.nashif@intel.com>
Extend Twister Harness 'recording' feature to allow multiple
regular expression patterns to extract different types of records
from test output.
Add 'merge' recording mode to collect all extracted data fields
into a single record object of the test instance.
Export to CSV file now takes all field names occurred in the collected
records, sort it alphabetically, and then use it for columns instead of
using only the first record's fields. This is done to address possible
situation when records have different set of fields.
Adjust Twister documentation and test suite to the above changes.
Signed-off-by: Dmitrii Golovanov <dmitrii.golovanov@intel.com>
Added integration_toolchains to allow building/testing with multiple
toolchains available in the environment.
This changes the output structure and adds another level in the path
under twister_out signifying the toolchain used. The toolchain used
(variant) is also part of the json output now.
Signed-off-by: Anas Nashif <anas.nashif@intel.com>
add vendor_allow and vendor_exclude to the list of filters. The vendor
data is maintained in the board.yml files.
Signed-off-by: Anas Nashif <anas.nashif@intel.com>
This change introduces the ability in twister to select which
emulation/simulation tool to use on the command line.
If none is specified, it will select the first in the list.
Signed-off-by: Wilfried Chauveau <wilfried.chauveau@arm.com>
Extended hardware map to share a single board between variants.
To run tests for different variants on the same board
without re-configuring the hardware map file for each variant,
one can use a `platform` atribute as a list of names.
Signed-off-by: Grzegorz Chwierut <grzegorz.chwierut@nordicsemi.no>
We now use hwmv2 to list boards instead of relying on twister specific
config files.
One yaml files (twister.yaml for now) will have all the data needed for
all possible targets and variations of a board reusing most of the data
where possible and variations can override the top level data.
Twister keeps track of 'aliases' of boards and identifies that for
example native_sim is the same as native_sim/native, so either names
will be possible in both test yaml files or on the command line,
however, the reporting will always use the full name, so no there is no
confusion about what is being tested/built.
Signed-off-by: Anas Nashif <anas.nashif@intel.com>
The current version of scipts do not consider OOT boards use cases and
the tests with robot now are strict to only one robot file, which is
not realistic for real environment. This address those issues and allow
multiple testsuits at command line and lists at tests entries. It add
another test parameter to allow configure robotframework options.
Fixes: #74563
Signed-off-by: Gerson Fernando Budke <gerson.budke@ossystems.com.br>
Extend Twister Harness recording feature to allow selected data fields,
extracted from the log by a regular expression, to be parsed into JSON
objects and eventually reported in `twister.json` as `recording` list
property of the test suite.
With this extension, log records can convey layered data structures
passed from a test image as summary results, traces, statistics, etc.
This extension also allows flexible recording structure: a test image
can output different types of data records incapsulated into a fixed
set of fields, so `recording.csv` file columns are respected, whereas
some of the columns keep strings with json-encoded semi-structured data.
Signed-off-by: Dmitrii Golovanov <dmitrii.golovanov@intel.com>
More complex platforms require sysbuild to use always, even for
such "simple" samples like hello_world. Such platforms can have
`sysbuild: true` entry in their board_name.yaml used by twister.
Using such entry will tell twister, that sysbuild must always be used
on a given platform.
Twister is aligned to have information about need of sysbuild at
instance (platform + suite) level (was only at suite level before).
Instance.sysbuild is true whenever a test suite or a platform requires
sysbuild.
Twister pytest unit tests are aligned with changes.
Signed-off-by: Maciej Perkowski <Maciej.Perkowski@nordicsemi.no>
Initial integration with renode-test was introduced in commit bdf02ff,
which added support for calling the `renode-test` command from both west
and twister.
This commit removes the custom run_renode_test target used for running
Robot tests with the `west build` command and makes twister call
`renode-test` directly instead.
Signed-off-by: Michał Szprejda <mszprejda@antmicro.com>
Removed Harness's repeat field.
Added to schemas in e722db14ad,
together with the rest of harness_config.
Added to Harness when it was created.
Now present only in one sample -
mec15xxevb_assy6853 power management sample.
Could not find definitive proof whether it has ever been used.
Signed-off-by: Lukasz Mrugala <lukaszx.mrugala@intel.com>
Add option to flash board before attach serial.
Current implementation performs the following sequence:
1. Open serial port to listen to board log output
2. Flash device
In case of ESP32 where it uses the same serial port
for both operations, flashing needs to come first.
This PR adds a twister option named --flash-before
which enables the process above, allowing tests to be
performed properly.
Signed-off-by: Lucas Tamborrino <lucas.tamborrino@espressif.com>
This commit adds an enum to the `arch` field of the Twister platform
schema. This helps better filter boards for testcases which use
architecture-based filters, and helps maintain uniformity in naming
convetion.
Signed-off-by: Filip Kokosinski <fkokosinski@antmicro.com>
Rather than duplicating the same schema, define it only once and reuse
for both common section and tests section.
Signed-off-by: Anas Nashif <anas.nashif@intel.com>
To make possible to build bsim tests by Twister, it is necessary to
copy executables to BabbleSim bin directory.
Signed-off-by: Piotr Golyzniak <metody159@gmail.com>
This gives us the flexibility to exclude certain platforms'
simulator that's known to fail from running in Twister, but
allows real hardware to be tested.
Signed-off-by: Yong Cong Sin <ycsin@meta.com>
The Console Harness is able to parse its log with patterns to compose
extracted fields into records in 'recording.csv' file in the test's build
directory. This feature allows to extract custom test results like
performance counters.
With this change the extracted records are also written into 'twister.json'
as a part of each test suite object. This makes easier to store
all the data collected by the test for its further processing.
Other improvements:
- compile parsing pattern only once instead of at each input line;
- quote fields in '.csv' to avoid unexpected field separators;
- make 'regex' a required schema field of 'harness_config';
- Twister documentation update.
Signed-off-by: Dmitrii Golovanov <dmitrii.golovanov@intel.com>
Added pytest_dut_scope keyword under harness_config section.
New keyword is used to determine the scope of dut and shell
fixtures in pytest-twister-harness plugin.
Signed-off-by: Grzegorz Chwierut <grzegorz.chwierut@nordicsemi.no>
Add a new option --vendor which allows filtering by vendors tracked in
the board/platform yaml file. The vendor string is compatible with DTS
and is what we have in dts/bindings/vendor-prefixes.txt.
Providing multiple vendors is also supported.
Signed-off-by: Anas Nashif <anas.nashif@intel.com>
Allow to specify a list of pytest directories, files or subtests
with pytest_root keyword in test yaml.
Signed-off-by: Grzegorz Chwierut <grzegorz.chwierut@nordicsemi.no>
Enable the possibility for boards to implement a custom `run` target in
its board.cmake to run any arbitrary commands. This is helpful for devs
who would like to add support for proprietary simulator to their boards
that can't be upstreamed.
Signed-off-by: Yong Cong Sin <ycsin@meta.com>
Add support for calling the `renode-test` command from west and twister.
Enable running Robot Framework tests suites in Renode.
Signed-off-by: Michał Szprejda <mszprejda@antmicro.com>
Signed-off-by: Mateusz Hołenko <mholenko@antmicro.com>
This allows to write both, lists or strings in the YAML file.
Unfortunately pykwalify schema language does not support multi-type
fields. JSON schema or Marshmallow could be used instead.
Signed-off-by: Gerard Marull-Paretas <gerard.marull@nordicsemi.no>
Use explicit flash command execution timeout at DeviceHandler
instead of the hardcoded value of 60 sec.
When a HW platform executes test cases right after the flash command,
the test case remaining timeout is affected by how much time the flash
operation consumes. Some simulation platforms need to spend significant
amount of time on each 'flash' cycle, usually adding the same delay on
each test case.
This improvement adds two new command line options and device map fields:
--device-flash-timeout - for the flash operation timeout
('flash-timeout' device map field).
--device-flash-with-test - to indicate that the platform flash
command also runs a test case, so the overall timeout should be
calculated as a sum of the flash timeout and the current test case
timeout to receive all console output from the platform
('flash-with-test' device map field).
The device map field values override command line values for the
particular platform where configured.
Default behavior is backward compatible: flash operation fixed timeout
is 60 sec. not including the test case timeout.
Signed-off-by: Dmitrii Golovanov <dmitrii.golovanov@intel.com>
