Removed Harness's repeat field.
Added to schemas in e722db14ad,
together with the rest of harness_config.
Added to Harness when it was created.
Now present only in one sample -
mec15xxevb_assy6853 power management sample.
Could not find definitive proof whether it has ever been used.
Signed-off-by: Lukasz Mrugala <lukaszx.mrugala@intel.com>
Add option to flash board before attach serial.
Current implementation performs the following sequence:
1. Open serial port to listen to board log output
2. Flash device
In case of ESP32 where it uses the same serial port
for both operations, flashing needs to come first.
This PR adds a twister option named --flash-before
which enables the process above, allowing tests to be
performed properly.
Signed-off-by: Lucas Tamborrino <lucas.tamborrino@espressif.com>
This commit adds an enum to the `arch` field of the Twister platform
schema. This helps better filter boards for testcases which use
architecture-based filters, and helps maintain uniformity in naming
convetion.
Signed-off-by: Filip Kokosinski <fkokosinski@antmicro.com>
Rather than duplicating the same schema, define it only once and reuse
for both common section and tests section.
Signed-off-by: Anas Nashif <anas.nashif@intel.com>
To make possible to build bsim tests by Twister, it is necessary to
copy executables to BabbleSim bin directory.
Signed-off-by: Piotr Golyzniak <metody159@gmail.com>
This gives us the flexibility to exclude certain platforms'
simulator that's known to fail from running in Twister, but
allows real hardware to be tested.
Signed-off-by: Yong Cong Sin <ycsin@meta.com>
The Console Harness is able to parse its log with patterns to compose
extracted fields into records in 'recording.csv' file in the test's build
directory. This feature allows to extract custom test results like
performance counters.
With this change the extracted records are also written into 'twister.json'
as a part of each test suite object. This makes easier to store
all the data collected by the test for its further processing.
Other improvements:
- compile parsing pattern only once instead of at each input line;
- quote fields in '.csv' to avoid unexpected field separators;
- make 'regex' a required schema field of 'harness_config';
- Twister documentation update.
Signed-off-by: Dmitrii Golovanov <dmitrii.golovanov@intel.com>
Added pytest_dut_scope keyword under harness_config section.
New keyword is used to determine the scope of dut and shell
fixtures in pytest-twister-harness plugin.
Signed-off-by: Grzegorz Chwierut <grzegorz.chwierut@nordicsemi.no>
Add a new option --vendor which allows filtering by vendors tracked in
the board/platform yaml file. The vendor string is compatible with DTS
and is what we have in dts/bindings/vendor-prefixes.txt.
Providing multiple vendors is also supported.
Signed-off-by: Anas Nashif <anas.nashif@intel.com>
Allow to specify a list of pytest directories, files or subtests
with pytest_root keyword in test yaml.
Signed-off-by: Grzegorz Chwierut <grzegorz.chwierut@nordicsemi.no>
Enable the possibility for boards to implement a custom `run` target in
its board.cmake to run any arbitrary commands. This is helpful for devs
who would like to add support for proprietary simulator to their boards
that can't be upstreamed.
Signed-off-by: Yong Cong Sin <ycsin@meta.com>
Add support for calling the `renode-test` command from west and twister.
Enable running Robot Framework tests suites in Renode.
Signed-off-by: Michał Szprejda <mszprejda@antmicro.com>
Signed-off-by: Mateusz Hołenko <mholenko@antmicro.com>
This allows to write both, lists or strings in the YAML file.
Unfortunately pykwalify schema language does not support multi-type
fields. JSON schema or Marshmallow could be used instead.
Signed-off-by: Gerard Marull-Paretas <gerard.marull@nordicsemi.no>
Use explicit flash command execution timeout at DeviceHandler
instead of the hardcoded value of 60 sec.
When a HW platform executes test cases right after the flash command,
the test case remaining timeout is affected by how much time the flash
operation consumes. Some simulation platforms need to spend significant
amount of time on each 'flash' cycle, usually adding the same delay on
each test case.
This improvement adds two new command line options and device map fields:
--device-flash-timeout - for the flash operation timeout
('flash-timeout' device map field).
--device-flash-with-test - to indicate that the platform flash
command also runs a test case, so the overall timeout should be
calculated as a sum of the flash timeout and the current test case
timeout to receive all console output from the platform
('flash-with-test' device map field).
The device map field values override command line values for the
particular platform where configured.
Default behavior is backward compatible: flash operation fixed timeout
is 60 sec. not including the test case timeout.
Signed-off-by: Dmitrii Golovanov <dmitrii.golovanov@intel.com>
Add support test levels and the ability to assign a specific test to one
or more levels. Using command line options of twister it is then possible
to select a level and just execute the tests included in this level.
Additionally, a test configuration allows definiing level
dependencies and additional inclusion of tests into a specific level if
the test itself does not have this information already.
In the configuration file you can include complete components using
regular expressions and you can specify which test level to import from
the same file, making management of levels easier.
To help with testing outside of upstream CI infrastructure, additional
options are available in the configuration file, which can be hosted
locally. As of now, those options are available:
- Ability to ignore default platforms as defined in board definitions
(Those are mostly emulation platforms used to run tests in upstream
CI)
- Option to specify your own list of default platforms overriding what
upstream defines.
- Ability to override build_onl_all options used in some testscases.
This will treat tests or sample as any other just build for default
platforms you specify in the configuation file or on the command line.
Signed-off-by: Anas Nashif <anas.nashif@intel.com>
This is an implementation of issue #48334 and adds support for
specifying additional config and device tree overlays through fields in
the testcase.yaml file, which is more readable than having to cram these
in to `extra_args`.
Consider this example which shows the original and new ways to add
config and DT overlays:
```
common:
extra_args: "CONF_FILE=a.conf;b.conf
DTC_OVERLAY_FILE=w.overlay;x.overlay OVERLAY_CONFIG=e.conf
UNRELATED=abc"
tests:
utilities.base64:
extra_conf_files:
- "c.conf"
- "d.conf"
extra_overlay_confs:
- "extra_overlay.conf"
extra_dtc_overlay_files:
- "y.overlay"
- "z.overlay"
extra_configs:
- CONFIG_SAMPLE=y
tags: base64
type: unit
```
The new fields are `extra_conf_files`, `extra_overlay_confs,
`extra_dtc_overlay_files`. Files specified in these sections are appended
to any `CONF_FILE`, `OVERLAY_CONFIG`, or `DTC_OVERLAY_FILE` fields in
`extra_args`, causing the following args being passed in to
`self.run_cmake` at `runner.py:850`:
```
['-DUNRELATED=abc',
'-DCONF_FILE=a.conf;b.conf;c.conf;d.conf',
'-DDTC_OVERLAY_FILE=w.overlay;x.overlay;y.overlay;z.overlay',
'-DOVERLAY_CONFIG=e.conf extra_overlay.conf '
'<build_dir>/twister/testsuite_extra.conf']
```
These fields can be used in the common or scenario-specific YAML
sections and will be merged in order of least to most specific:
1. config files extracted from common's extra_args
2. files listed in common's {extra_conf_files or extra_overlay_confs
or extra_dtc_overlay_files}
3. config files extracted from test scenario's extra_args
4. files listed in test scenario's {extra_conf_files or
extra_overlay_confs or extra_dtc_overlay_files}
Specifying these files in extra_args now triggers a deprecation warning,
as the direct YAML fields are preferred for readability. They will still
function for now but support will be dropped in the future. One
testcase.yaml
(`zephyr/tests/cmake/overlays/var_expansions/testcase.yaml`) is
converted to use the new fields. A follow-up PR will convert the
remaining files to the new format.
Signed-off-by: Tristan Honscheid <honscheid@google.com>
Implementation ported from TwisterV2.
- quarantine handled by separate module
- multiple yaml allowed from args: --quarantine-list
- scenarios, platforms, architectures keywords in
quarantine yaml are optional, if not given - means take it all
Signed-off-by: Grzegorz Chwierut <grzegorz.chwierut@nordicsemi.no>
Adds an option to inform twister a testsuite should only be built and
run for platforms with unique sets of attributes. This enables
for example keying on unique (arch, simulation) platforms to run the test
suite on.
The most common usage may be test suites configured to run once per
(arch, simulation) pair as being enough. Additional information about
platforms may enable running a test once per hardware IP block or once
per soc family or soc avoiding duplicated effort in building and running
tests when once suffices.
Signed-off-by: Tom Burdick <thomas.burdick@intel.com>
Have boards define what binaries should be kept for device testing. The
purpose of this is to reduce the amount of artifacts and just keep the
binaries needed for flashing a device, reducing the size of the final
payload.
A board can have twister keep multiple binaries needed for testing when
--prep-artifacts-for-testing option is used.
Signed-off-by: Anas Nashif <anas.nashif@intel.com>
Fix all thruthy errors detected by yamllint:
yamllint -f parsable -c .yamllint $( find -regex '.*\.y[a]*ml' ) | \
grep '(truthy)'
This only accepts true/false for boolean properties. Seems like python
takes all sort of formats:
https://github.com/yaml/pyyaml/blob/master/lib/yaml/constructor.py#L224-L235
But the current specs only mention "true" or "false"
https://yaml.org/spec/1.2.2/#10212-boolean
Which is the standard yamllint config.
Excluding codeconv and workflow files, as some are using yes/no instead
in the respective documentation.
Signed-off-by: Fabio Baltieri <fabiobaltieri@google.com>
Provide the information as part of the board yaml file, this way we will
know what binary to check for.
Signed-off-by: Anas Nashif <anas.nashif@intel.com>
Add support for building with sysbuild using twister, via the "sysbuild"
yaml property in testsuites. This will currently disable Kconfig and
devicetree filtering.
Signed-off-by: Daniel DeGrasse <daniel.degrasse@nxp.com>
Instead of relying on runtime filter to limit scope to emulation
platforms, use the type attribute for each platform and do the filtering
very early on. This will speed things up for tests where we only run on
emulation platforms.
Signed-off-by: Anas Nashif <anas.nashif@intel.com>
Some complex testsuite use macros and other mean to define testcases
which makes them unparseable. This ends up in a descripency between the
testplan and the test results.
Allow adding the testcases into the yaml and use those instead of
parsing the C files.
Signed-off-by: Anas Nashif <anas.nashif@intel.com>
TestCase in the old terminology is the test application, but we have
just freed up TestSuite, so use TestSuite here to signify the test
application which consists of many test scenarios and testcases.
Signed-off-by: Anas Nashif <anas.nashif@intel.com>
The hardware map feature can be used with serial pty mode in
this change. And also add an runner_params option for passing
more parameters from hardware map file to west runner. Note
that you need to create this map file manually because it
cannot be scanned out correctly due to pty is not a physical
HW device existing in system.
And also update doc/develop/test/twister.rst for the usage.
Signed-off-by: Enjia Mai <enjia.mai@intel.com>
Twister allows us to control maximum execution time for each
test with timeout value in test .yaml configuration. This
helps us to prevent slow tests from stopping by timeout.
However it's hard to choose test timeout for all platforms
as some platforms are naturally slow. It could be a HW board with
power-efficient but slow CPU or simulation platform which
can perform instruction accurate simulation but does it slowly.
As we don't want to increase test timeout infinitely to meet
the needs of the slowest platform let's introduce
platform-specific test timeout management. It's implemented as
an optional 'timeout_multiplier' field in board .yaml
configuration. Setting it to some value multiplies each test
timeout by this value. By that we can increase timeouts only
for the platforms where it's required.
Signed-off-by: Eugeniy Paltsev <Eugeniy.Paltsev@synopsys.com>
Signed-off-by: Evgeniy Paltsev <PaltsevEvgeniy@gmail.com>
Support new keyword in the testcase.yaml to allow filtering based on
enabled modules. This is useful when you only whitelist a limited set of
modules in your project and want to only run those samples that have
their dependencies satisfied and those modules available.
The option allows adding multiple modules, all of which need to be
enabled in the manifest.
Signed-off-by: Anas Nashif <anas.nashif@intel.com>
Each board needs to have a maintainer to allow automatic assignment and
adding reviewers when pull requests concerning the platform are being
submitted.
This is now a placeholder so we can start adding maintainers early, the
data is not being used anywhere yet.
Signed-off-by: Anas Nashif <anas.nashif@intel.com>
Change the "yes/no" value to "true/false".
Otherwise pykwalify 1.8.0 will report below errors:
- Value: 'no' for required keyword must be a boolean:
Path: '/mapping/common'>
- Value: 'yes' for required keyword must be a boolean:
Path: '/mapping/sample/mapping/name'>
Signed-off-by: Shao Ming <ming.shao@intel.com>
Fixes#38046
Document the changes in twister.rst
Add baud rate to debug message in twister
Add baud parameter to twister's add_device function
Set the twister baud rate from input parameters
Use 115200 as the default baud rate if not specified
Add baud to the hwmap-schema.yaml file
Add --device-serial-baud to twister arguments
Fix compliance issues
Fix mistake in parameter name from device-baud to device-serial-baud
Refactoring of the code in orded to simplify the logic
and clear multiple definitions of the default baud rate.
Signed-off-by: Maciej Perkowski <Maciej.Perkowski@nordicsemi.no>
Signed-off-by: Dennis Ruffer <daruffer@gmail.com>
1. Add a class of Pytest(Harness) which can handle pytest script
in harness.py
2. Use running_dir to store current test directory which be
used in pytest scrpt.
3. Add usage of this feature into zephyr doc.
Signed-off-by: YouhuaX Zhu <youhuax.zhu@intel.com>
Add arm fvp emulator in order to use ninja run or west build -t run.
Add armfvp in order to run twister.
Set env ARMFVP_BIN_PATH before using it,
e.g. export ARMFVP_BIN_PATH=<path/to/fvp/dir>
NOTE: ARMFVP_BIN_PATH is the dir path.
Signed-off-by: Jaxson Han <jaxson.han@arm.com>
Adds feature allowing to use yaml file with dictionaries defining
tests to be quarantined (extra arg "--quarantine-list FILENAME").
The dictionaries are validated according to the proper schema
and loaded.
A flat list is created containing quarantined configurations
(configuration = platform + scenario). Configurations under quarantine
are skipped and get "Quarantine" as a reason in the results reports.
A "comment" can be added to a quarantine entry in the quarantine yaml
with more details (e.g. issue #) and it will be also added to
the report.
The status of tests under quarantine can be verify if
`--quarantine-verify` is used in addition to
"--quarantine-list FILENAME". Using these args will make twister skip
all tests which are not on the quarantine list.
Signed-off-by: Maciej Perkowski <Maciej.Perkowski@nordicsemi.no>
Move yaml schemas into one single location. We have quite a few schema
used across the tree, one common place will make them easier to find and
reference.
Signed-off-by: Anas Nashif <anas.nashif@intel.com>
