Add support for calling the `renode-test` command from west and twister.
Enable running Robot Framework tests suites in Renode.
Signed-off-by: Michał Szprejda <mszprejda@antmicro.com>
Signed-off-by: Mateusz Hołenko <mholenko@antmicro.com>
Current implementation of ConfigurationError causes
an exception to be thrown.
It is because a PosixPath is being concatenated with an str with a '+'
operator, which is not permitted.
The path in question is now cast to an str.
Additional test has been added to test_twister.py to track regressions.
Signed-off-by: Lukasz Mrugala <lukaszx.mrugala@intel.com>
For the arch's that have standard toolchains thare are supported
across the arch set them in twister rather than in <BOARD>.yaml.
Signed-off-by: Kumar Gala <kumar.gala@intel.com>
Some 3P logic (Pigweed for example) is already heavily invested in
gTest as a testing framework. Adding the `gtest` harness enables
running the existing 3P tests on various Zephyr platforms and configs.
Signed-off-by: Yuval Peress <peress@google.com>
Simics converts newline characters when writing console output to
stdout, so we need to remove them as a suffix string. Otherwise the
console harness fails to match expected PASS/FAIL/RunID string patterns
and twister tests timeout.
Signed-off-by: Maureen Helm <maureen.helm@intel.com>
After generating a hardware-map a platform field is marked
as 'unknown'. If it is not changed, do not process them
when running --device-testing.
Signed-off-by: Grzegorz Chwierut <grzegorz.chwierut@nordicsemi.no>
In some cases CMake doesn't create build directory and in those cases
Twister should create this directory by itself.
Signed-off-by: Piotr Golyzniak <piotr.golyzniak@nordicsemi.no>
We have been clearning the list of testcases in every iteration over the
symbols ending up with only one testcase in the instance.
Signed-off-by: Anas Nashif <anas.nashif@intel.com>
Add a new option to Twister that allows to load alternative
configuration files, `--alt-config-root`. This new option takes an
arbitrary number of root folders where alternative configuration files
can be stored. Twister will check if a test configuration file exists in
any of the alternative test configuration folders. For example, given
`$test_root/tests/foo/testcase.yaml`, Twister will use
`$alt_config_root/tests/foo/testcase.yaml` if it exists.
This feature can be useful if an out-of-tree project needs to run
upstream tests in different configurations, potentially not available
upstream (e.g. an out-of-tree board, or Kconfig setting).
Note that overlaying has been discarded because we can't easily delete
fields from the original configuration file, something that in certain
cases could be required. Current approach may lead to some dupplication,
but guarantees full control of the test configuration.
Signed-off-by: Gerard Marull-Paretas <gerard.marull@nordicsemi.no>
Make twister to raise error by default whenever duplicated test ids
are found. Remove redundant option to list duplicates.
Signed-off-by: Maciej Perkowski <Maciej.Perkowski@nordicsemi.no>
Don't override a test case name check failed result by the following
RunID check failed result to report clearly about test case mismatch
situation as well as more specifically when the same test case appears
with unexpected RunID.
Signed-off-by: Dmitrii Golovanov <dmitrii.golovanov@intel.com>
Fix suite-name-check bypassed at DeviceHandler on flash errors.
If the flash timeout is smaller than needed, but test images are
actually flashed successfully and appropriate tests executed,
then results from the previous test case were taken because
the flash error status makes the suite-name-check bypassed.
Signed-off-by: Dmitrii Golovanov <dmitrii.golovanov@intel.com>
We can now have situations like this when it comes to lists in Twister
config:
```yaml
common:
platform_exclude: foo
tests:
my.test:
platform_exclude: bar
```
```yaml
common:
platform_exclude: foo
tests:
my.test:
platform_exclude:
- bar
- baz
```
```yaml
common:
platform_exclude:
- foo
- bar
tests:
my.test:
platform_exclude: baz
```
```yaml
common:
platform_exclude:
- foo
- bar
tests:
my.test:
platform_exclude:
- baz
- bazz
```
This patch fixes handling of cases (2), (3) and (4).
Signed-off-by: Gerard Marull-Paretas <gerard.marull@nordicsemi.no>
This patch handles mixed types between common/tests sections in the
configuration, e.g. str+list, list+str, str+str.
Signed-off-by: Gerard Marull-Paretas <gerard.marull@nordicsemi.no>
Since pykwalify does not support multi-type fields, we are forced to use
the 'any' type. This is used for fields that allow sets/lists written as
either YAML lists or space-separated strings. This patch checks for
str/list Python type, and raises a configuration error if not true so
that user obtains better error messages in case the configuration file
contains invalid format.
Signed-off-by: Gerard Marull-Paretas <gerard.marull@nordicsemi.no>
Twister configuration files have a few fields defined with the "set"
type. The set needs to be specified as a space-separated list of
entries, for example:
```yaml
plaform_allow: boardx boardy
```
This patch allows users to write these fields as YAML lists, for
example:
```yaml
platform_allow:
- boardx
- boardy
```
The list gets converted to a Python set, therfore the same exact
behavior is kept in Twister.
In some cases like in the example above, using lists allows to have more
readable configuration files, specially if the list is long. It also
helps on diffs, provided we have one item per line.
Signed-off-by: Gerard Marull-Paretas <gerard.marull@nordicsemi.no>
A change to the scope selection rules was introduced by #52715
but it doesn't comply with the description in the code. What's more,
the change introduces major issues i.e. non deterministic scope
selection, especially in a CI environment. More context at #57595
Signed-off-by: Maciej Perkowski <Maciej.Perkowski@nordicsemi.no>
Twister does not handle out-of-tree boards automatically. Modules like
e.g. example-application that define their own board root folder, are
required to inform twister via `--board-root`. In contrast, west is able
to to this automatically. This patch makes twister a bit smarter by
automatically filling the board root list with module paths as well.
Signed-off-by: Gerard Marull-Paretas <gerard.marull@nordicsemi.no>
Add option to shuffle tests (randomly) before they are split into subsets.
This allow to fairly distribute tests across subsets,
so that long tests are not crowded in a single group.
This also allows to detect unwanted dependencies in test execution order.
Also, added option to provide custom seed to random generator
used to shuffle tests.
This will allow to reproduce certain test order if needed.
Used seed is printed at console.
Signed-off-by: Piotr Kosycarz <piotr.kosycarz@nordicsemi.no>
The initshell sections in the linker scripts where associated with theo
old shell code. The old shell code has been removed for some time so
remove references to initshell in the linker scripts and size_calc.py
Signed-off-by: Kumar Gala <kumar.gala@intel.com>
Convert following construction:
elif self.options.device_serial or self.options.device_serial_pty:
if self.options.device_serial:
...
else
...
to the construction:
elif self.options.device_serial:
...
else self.options.device_serial_pty:
...
Signed-off-by: Andrei Emeltchenko <andrei.emeltchenko@intel.com>
Use flash_timeout and flash_with_test options not only for
serial console but also for device_serial_pty.
Signed-off-by: Andrei Emeltchenko <andrei.emeltchenko@intel.com>
compliance script runs pylink on size_calc.py and reports:
C0325:Unnecessary parens after '=' keyword (superfluous-parens)
Fix warning by remove unnecessary parens.
Signed-off-by: Kumar Gala <kumar.gala@intel.com>
`TestInstance.get_elf_file()` does not include the binary name used for
unit testing (`testbinary`) in its search, causing unit tests to not
complete successfully. This does not cause tests to fail and thus goes
unnoticed.
Update the glob expressions used in `get_elf_file()` to also look for
files called `testbinary`
Signed-off-by: Tristan Honscheid <honscheid@google.com>
If we are doing a sysbuild get_elf_file needs to return the default
domain's build_dir. The standard build_dir will not any ELF builds
in it so will error out.
Signed-off-by: Kumar Gala <kumar.gala@intel.com>
This is usually used only when doing size calculation, but now it is
being used for extracting symbols from the ELF for testing purposes and
some issues arise with multiple ELF files found.
Simplify the code a bit to make it easy to maintain a lists of things to
exclude.
Signed-off-by: Anas Nashif <anas.nashif@intel.com>
The DeviceHandler class method monitor_serial() has time-of-check
to time-of-use (TOCTOU) race condition possible on its serial port
which might be closed and its file descriptor cleared in between
of isOpen() and is_waiting status checks.
The issue is alleviated with controlled exception handling.
Signed-off-by: Dmitrii Golovanov <dmitrii.golovanov@intel.com>
Fixes an issue when DeviceHandler awaited for a suitable DUT infinitely.
It was possible if the device map has no devices with required platform,
or no devices with fixture needed for the test case, or devices without
serial connection. A trivial case is a typo in platform name.
Signed-off-by: Dmitrii Golovanov <dmitrii.golovanov@intel.com>
For ztest twister would parse the symbol information that was
generated as part of the build (zephyr.symbols). However the format
of the zephyr.symbols files is highly dependant on the toolchain.
Move to using pyelf to parse the symbol information directly from
zephyr.elf instead so that this works regardless of toolchain.
Signed-off-by: Kumar Gala <kumar.gala@intel.com>
On native platforms we typically have CONFIG_BUILD_OUTPUT_EXE set so
we end up with both zephyr.elf and zephyr.exe. This would cause a
call to get_elf_file() to fail. There isn't any reason to error out
since zephyr.elf and zephyr.exe should be identical. So allow it
if the platform type is native.
Signed-off-by: Kumar Gala <kumar.gala@intel.com>
Use explicit flash command execution timeout at DeviceHandler
instead of the hardcoded value of 60 sec.
When a HW platform executes test cases right after the flash command,
the test case remaining timeout is affected by how much time the flash
operation consumes. Some simulation platforms need to spend significant
amount of time on each 'flash' cycle, usually adding the same delay on
each test case.
This improvement adds two new command line options and device map fields:
--device-flash-timeout - for the flash operation timeout
('flash-timeout' device map field).
--device-flash-with-test - to indicate that the platform flash
command also runs a test case, so the overall timeout should be
calculated as a sum of the flash timeout and the current test case
timeout to receive all console output from the platform
('flash-with-test' device map field).
The device map field values override command line values for the
particular platform where configured.
Default behavior is backward compatible: flash operation fixed timeout
is 60 sec. not including the test case timeout.
Signed-off-by: Dmitrii Golovanov <dmitrii.golovanov@intel.com>
Rather than having to add 'armclang' to every <BOARD>.yaml, just list
it as a generally supported toolchain if the architecture is 'arm'.
Signed-off-by: Kumar Gala <kumar.gala@intel.com>
When a test start, mark it as such, if we do not get any results due to
a timeout, it will be finally marked as failed. Previously such tests
causing a freeze were marked as blocked (or not run), which is not
exactly right.
Signed-off-by: Anas Nashif <anas.nashif@intel.com>
Skips due to overflows on integration_platforms during integretion
mode should be treated as errors.
Signed-off-by: Maciej Perkowski <Maciej.Perkowski@nordicsemi.no>
`self.instances[filtered_instance.name]` and `filtered_instance`
are already referancing the same object.
Signed-off-by: Maciej Perkowski <Maciej.Perkowski@nordicsemi.no>
If canonical zephyr base is already ended by trailing slash, then it
should not be added redundantly.
Signed-off-by: Piotr Golyzniak <piotr.golyzniak@nordicsemi.no>
In some cases built binary files have different names than "zephyr.hex"
files, and for those cases, information about names of file to keep
should be taken from runners.yaml file.
Moreover if in runners.yaml file exist absolute files paths to binary
files, then they should be changed to relative ones.
Signed-off-by: Piotr Golyzniak <piotr.golyzniak@nordicsemi.no>
Currently in binary handler and qemu handler we have status `failed`
in case of test timeout, but in device handler we have status `error`
This not only adds inconsistency between handlers, but also prevents
us from usage test retry functionality for the runs on HW.
Fix timeout status by changing it to `failed` instead of `error`.
Signed-off-by: Eugeniy Paltsev <Eugeniy.Paltsev@synopsys.com>
Signed-off-by: Evgeniy Paltsev <PaltsevEvgeniy@gmail.com>
In many of test specifications yaml files (testcase.yaml, sample.yaml)
section `platform_allow` is added. This change allows to test some
scenarios on platforms, that are not added yet to platform allow
list (or are not going to be added for some reasons).
Signed-off-by: Grzegorz Chwierut <grzegorz.chwierut@nordicsemi.no>
Fix how DeviceHandler class does 'serial leftover' cleanup
with loss of test output observed on fast console connections,
e.g. over telnet: either 1 sec. of output might be missed,
or up to 1000 bytes/chars, whatever happens first.
Signed-off-by: Dmitrii Golovanov <dmitrii.golovanov@intel.com>
We are wrongly claiming no console output, yet console output is there
and the failure is just a regular timeout, i.e. the test did not
complete within allocated time.
Signed-off-by: Anas Nashif <anas.nashif@intel.com>
