Fix `--device-flash-with-test` command line options check logic
to be effective with `--device-testing`.
Remove `--device-flash-timeout` check for `--device-testing`
presence as marginal.
Signed-off-by: Dmitrii Golovanov <dmitrii.golovanov@intel.com>
hardware was referenced as return value, without being declared. When the
program goes into the 'except' path, hardware is unbound, but returned,
which throws an error:
`UnboundLocalError`: local variable 'hardware' referenced before assignment
Signed-off-by: Manoel Brunnen <manoel.brunnen@ithinx.io>
Extended parsing of report.xml file - produced by pytest-twister-harness
plugin. Now testcases are properly extracted and added to twister
report. Added unit tests.
Signed-off-by: Grzegorz Chwierut <grzegorz.chwierut@nordicsemi.no>
We have many tests that are marked as slow that do not run in CI and any
known workflow, to isolate them and be able to run them on their own,
add a new option --enable-slow-only which causes only those tests to build
and execute.
Signed-off-by: Anas Nashif <anas.nashif@intel.com>
Add DeviceAbstract class to default imports from pytest-twister-harness
package to simplify importing DUT package, when creating tests.
Signed-off-by: Grzegorz Chwierut <grzegorz.chwierut@nordicsemi.no>
Use the natural sort of list when generating a hardware map. The
list is sorted with a serial port as a key. When more than 10 ports
are active and some of devices use more than one port, the ports
of one device may be listed in wrong sequence, which may cause
futher problems with selecting the right port for listening to
the device.
Signed-off-by: Grzegorz Chwierut <grzegorz.chwierut@nordicsemi.no>
Add protection timeout for pytest subprocess, to avoid situation of
suspending whole Twister in case of internal pytest test problem.
Co-authored-by: Grzegorz Chwierut <grzegorz.chwierut@nordicsemi.no>
Signed-off-by: Piotr Golyzniak <piotr.golyzniak@nordicsemi.no>
Added waiting for prompt instead of calling time.sleep,
to fix issue with longer setting up platforms with TF-M
enabled. Wrapped iter_stdout generator, to avoid issues
when generator is created and called multiple times
in test scenarios.
Fixes: #58747
Signed-off-by: Grzegorz Chwierut <grzegorz.chwierut@nordicsemi.no>
When calling twister with --coverage and no platform specified on the
command line, set the the value of coverage platforms to an empty list.
Fixes#57534
Signed-off-by: Anas Nashif <anas.nashif@intel.com>
testinstance.py contains a string literal comparison to the
word 'platform', which has been misspelled as 'plaform'.
The change is verified by a test expansion.
Signed-off-by: Lukasz Mrugala <lukaszx.mrugala@intel.com>
Making the necessary changes to enable the new pytest plugin.
By default Twister should work without the pytest-twister-harness
plugin installed. To achieve this, each time Twister calls pytest,
the PYTHONPATH environment variable is expanded and the
`-p twister_harness.plugin` option is added to the pytest command.
Co-authored-by: Piotr Golyzniak <piotr.golyzniak@nordicsemi.no>
Signed-off-by: Grzegorz Chwierut <grzegorz.chwierut@nordicsemi.no>
Adding pytest plugin dedicated to running pytest tests in Zephyr
project. This plugin provides a dut fixture which allows to handle
bidirectional communication with the device under test. This version
of plugin can be used for tests dedicated to real hardware, QEMU and
native_posix simulator.
Co-authored-by: Lukasz Fundakowski <lukasz.fundakowski@nordicsemi.no>
Co-authored-by: Grzegorz Chwierut <grzegorz.chwierut@nordicsemi.no>
Co-authored-by: Katarzyna Giadla <katarzyna.giadla@nordicsemi.no>
Signed-off-by: Piotr Golyzniak <piotr.golyzniak@nordicsemi.no>
Add support for calling the `renode-test` command from west and twister.
Enable running Robot Framework tests suites in Renode.
Signed-off-by: Michał Szprejda <mszprejda@antmicro.com>
Signed-off-by: Mateusz Hołenko <mholenko@antmicro.com>
Current implementation of ConfigurationError causes
an exception to be thrown.
It is because a PosixPath is being concatenated with an str with a '+'
operator, which is not permitted.
The path in question is now cast to an str.
Additional test has been added to test_twister.py to track regressions.
Signed-off-by: Lukasz Mrugala <lukaszx.mrugala@intel.com>
For the arch's that have standard toolchains thare are supported
across the arch set them in twister rather than in <BOARD>.yaml.
Signed-off-by: Kumar Gala <kumar.gala@intel.com>
Some 3P logic (Pigweed for example) is already heavily invested in
gTest as a testing framework. Adding the `gtest` harness enables
running the existing 3P tests on various Zephyr platforms and configs.
Signed-off-by: Yuval Peress <peress@google.com>
Simics converts newline characters when writing console output to
stdout, so we need to remove them as a suffix string. Otherwise the
console harness fails to match expected PASS/FAIL/RunID string patterns
and twister tests timeout.
Signed-off-by: Maureen Helm <maureen.helm@intel.com>
After generating a hardware-map a platform field is marked
as 'unknown'. If it is not changed, do not process them
when running --device-testing.
Signed-off-by: Grzegorz Chwierut <grzegorz.chwierut@nordicsemi.no>
In some cases CMake doesn't create build directory and in those cases
Twister should create this directory by itself.
Signed-off-by: Piotr Golyzniak <piotr.golyzniak@nordicsemi.no>
We have been clearning the list of testcases in every iteration over the
symbols ending up with only one testcase in the instance.
Signed-off-by: Anas Nashif <anas.nashif@intel.com>
Add a new option to Twister that allows to load alternative
configuration files, `--alt-config-root`. This new option takes an
arbitrary number of root folders where alternative configuration files
can be stored. Twister will check if a test configuration file exists in
any of the alternative test configuration folders. For example, given
`$test_root/tests/foo/testcase.yaml`, Twister will use
`$alt_config_root/tests/foo/testcase.yaml` if it exists.
This feature can be useful if an out-of-tree project needs to run
upstream tests in different configurations, potentially not available
upstream (e.g. an out-of-tree board, or Kconfig setting).
Note that overlaying has been discarded because we can't easily delete
fields from the original configuration file, something that in certain
cases could be required. Current approach may lead to some dupplication,
but guarantees full control of the test configuration.
Signed-off-by: Gerard Marull-Paretas <gerard.marull@nordicsemi.no>
Make twister to raise error by default whenever duplicated test ids
are found. Remove redundant option to list duplicates.
Signed-off-by: Maciej Perkowski <Maciej.Perkowski@nordicsemi.no>
Don't override a test case name check failed result by the following
RunID check failed result to report clearly about test case mismatch
situation as well as more specifically when the same test case appears
with unexpected RunID.
Signed-off-by: Dmitrii Golovanov <dmitrii.golovanov@intel.com>
Fix suite-name-check bypassed at DeviceHandler on flash errors.
If the flash timeout is smaller than needed, but test images are
actually flashed successfully and appropriate tests executed,
then results from the previous test case were taken because
the flash error status makes the suite-name-check bypassed.
Signed-off-by: Dmitrii Golovanov <dmitrii.golovanov@intel.com>
We can now have situations like this when it comes to lists in Twister
config:
```yaml
common:
platform_exclude: foo
tests:
my.test:
platform_exclude: bar
```
```yaml
common:
platform_exclude: foo
tests:
my.test:
platform_exclude:
- bar
- baz
```
```yaml
common:
platform_exclude:
- foo
- bar
tests:
my.test:
platform_exclude: baz
```
```yaml
common:
platform_exclude:
- foo
- bar
tests:
my.test:
platform_exclude:
- baz
- bazz
```
This patch fixes handling of cases (2), (3) and (4).
Signed-off-by: Gerard Marull-Paretas <gerard.marull@nordicsemi.no>
This patch handles mixed types between common/tests sections in the
configuration, e.g. str+list, list+str, str+str.
Signed-off-by: Gerard Marull-Paretas <gerard.marull@nordicsemi.no>
Since pykwalify does not support multi-type fields, we are forced to use
the 'any' type. This is used for fields that allow sets/lists written as
either YAML lists or space-separated strings. This patch checks for
str/list Python type, and raises a configuration error if not true so
that user obtains better error messages in case the configuration file
contains invalid format.
Signed-off-by: Gerard Marull-Paretas <gerard.marull@nordicsemi.no>
Twister configuration files have a few fields defined with the "set"
type. The set needs to be specified as a space-separated list of
entries, for example:
```yaml
plaform_allow: boardx boardy
```
This patch allows users to write these fields as YAML lists, for
example:
```yaml
platform_allow:
- boardx
- boardy
```
The list gets converted to a Python set, therfore the same exact
behavior is kept in Twister.
In some cases like in the example above, using lists allows to have more
readable configuration files, specially if the list is long. It also
helps on diffs, provided we have one item per line.
Signed-off-by: Gerard Marull-Paretas <gerard.marull@nordicsemi.no>
A change to the scope selection rules was introduced by #52715
but it doesn't comply with the description in the code. What's more,
the change introduces major issues i.e. non deterministic scope
selection, especially in a CI environment. More context at #57595
Signed-off-by: Maciej Perkowski <Maciej.Perkowski@nordicsemi.no>
Twister does not handle out-of-tree boards automatically. Modules like
e.g. example-application that define their own board root folder, are
required to inform twister via `--board-root`. In contrast, west is able
to to this automatically. This patch makes twister a bit smarter by
automatically filling the board root list with module paths as well.
Signed-off-by: Gerard Marull-Paretas <gerard.marull@nordicsemi.no>
Add option to shuffle tests (randomly) before they are split into subsets.
This allow to fairly distribute tests across subsets,
so that long tests are not crowded in a single group.
This also allows to detect unwanted dependencies in test execution order.
Also, added option to provide custom seed to random generator
used to shuffle tests.
This will allow to reproduce certain test order if needed.
Used seed is printed at console.
Signed-off-by: Piotr Kosycarz <piotr.kosycarz@nordicsemi.no>
The initshell sections in the linker scripts where associated with theo
old shell code. The old shell code has been removed for some time so
remove references to initshell in the linker scripts and size_calc.py
Signed-off-by: Kumar Gala <kumar.gala@intel.com>
Convert following construction:
elif self.options.device_serial or self.options.device_serial_pty:
if self.options.device_serial:
...
else
...
to the construction:
elif self.options.device_serial:
...
else self.options.device_serial_pty:
...
Signed-off-by: Andrei Emeltchenko <andrei.emeltchenko@intel.com>
Use flash_timeout and flash_with_test options not only for
serial console but also for device_serial_pty.
Signed-off-by: Andrei Emeltchenko <andrei.emeltchenko@intel.com>
compliance script runs pylink on size_calc.py and reports:
C0325:Unnecessary parens after '=' keyword (superfluous-parens)
Fix warning by remove unnecessary parens.
Signed-off-by: Kumar Gala <kumar.gala@intel.com>
`TestInstance.get_elf_file()` does not include the binary name used for
unit testing (`testbinary`) in its search, causing unit tests to not
complete successfully. This does not cause tests to fail and thus goes
unnoticed.
Update the glob expressions used in `get_elf_file()` to also look for
files called `testbinary`
Signed-off-by: Tristan Honscheid <honscheid@google.com>
If we are doing a sysbuild get_elf_file needs to return the default
domain's build_dir. The standard build_dir will not any ELF builds
in it so will error out.
Signed-off-by: Kumar Gala <kumar.gala@intel.com>
This is usually used only when doing size calculation, but now it is
being used for extracting symbols from the ELF for testing purposes and
some issues arise with multiple ELF files found.
Simplify the code a bit to make it easy to maintain a lists of things to
exclude.
Signed-off-by: Anas Nashif <anas.nashif@intel.com>
The DeviceHandler class method monitor_serial() has time-of-check
to time-of-use (TOCTOU) race condition possible on its serial port
which might be closed and its file descriptor cleared in between
of isOpen() and is_waiting status checks.
The issue is alleviated with controlled exception handling.
Signed-off-by: Dmitrii Golovanov <dmitrii.golovanov@intel.com>
Fixes an issue when DeviceHandler awaited for a suitable DUT infinitely.
It was possible if the device map has no devices with required platform,
or no devices with fixture needed for the test case, or devices without
serial connection. A trivial case is a typo in platform name.
Signed-off-by: Dmitrii Golovanov <dmitrii.golovanov@intel.com>
For ztest twister would parse the symbol information that was
generated as part of the build (zephyr.symbols). However the format
of the zephyr.symbols files is highly dependant on the toolchain.
Move to using pyelf to parse the symbol information directly from
zephyr.elf instead so that this works regardless of toolchain.
Signed-off-by: Kumar Gala <kumar.gala@intel.com>
On native platforms we typically have CONFIG_BUILD_OUTPUT_EXE set so
we end up with both zephyr.elf and zephyr.exe. This would cause a
call to get_elf_file() to fail. There isn't any reason to error out
since zephyr.elf and zephyr.exe should be identical. So allow it
if the platform type is native.
Signed-off-by: Kumar Gala <kumar.gala@intel.com>