Commit graph

822 commits

Author SHA1 Message Date
Grzegorz Chwierut
57c27cd736 twister: pytest: Added fixture for mcumgr with unittests
Added mcumgr fixture to pytest-twister-harness. Added unittests for
new fixture.

Signed-off-by: Grzegorz Chwierut <grzegorz.chwierut@nordicsemi.no>
2023-07-17 16:23:59 -04:00
Dmitrii Golovanov
7afeefd7dd twister: Fix flashing timeout command line options check
Fix `--device-flash-with-test` command line options check logic
to be effective with `--device-testing`.

Remove `--device-flash-timeout` check for `--device-testing`
presence as marginal.

Signed-off-by: Dmitrii Golovanov <dmitrii.golovanov@intel.com>
2023-07-17 15:57:03 -04:00
Manoel Brunnen
368b0e0d16 twister: Initialise unbound 'hardware' variable
hardware was referenced as return value, without being declared. When the
program goes into the 'except' path, hardware is unbound, but returned,
which throws an error:

`UnboundLocalError`: local variable 'hardware' referenced before assignment

Signed-off-by: Manoel Brunnen <manoel.brunnen@ithinx.io>
2023-07-10 13:57:21 +02:00
Grzegorz Chwierut
4045bab1b7 twister: pytest: Parse report file to get testcases
Extended parsing of report.xml file - produced by pytest-twister-harness
plugin. Now testcases are properly extracted and added to twister
report. Added unit tests.

Signed-off-by: Grzegorz Chwierut <grzegorz.chwierut@nordicsemi.no>
2023-06-25 02:44:49 -04:00
Anas Nashif
b8fc1c3607 twister: run slow tests only
We have many tests that are marked as slow that do not run in CI and any
known workflow, to isolate them and be able to run them on their own,
add a new option --enable-slow-only which causes only those tests to build
and execute.

Signed-off-by: Anas Nashif <anas.nashif@intel.com>
2023-06-22 17:14:02 -04:00
Grzegorz Chwierut
732dd4843d twister: pytest: Simplify interface to pytest-twister-harness
Add DeviceAbstract class to default imports from pytest-twister-harness
package to simplify importing DUT package, when creating tests.

Signed-off-by: Grzegorz Chwierut <grzegorz.chwierut@nordicsemi.no>
2023-06-20 14:46:55 -04:00
Grzegorz Chwierut
c37deeb0c4 twister: Use natural sort when generating hardware map
Use the natural sort of list when generating a hardware map. The
list is sorted with a serial port as a key. When more than 10 ports
are active and some of devices use more than one port, the ports
of one device may be listed in wrong sequence, which may cause
futher problems with selecting the right port for listening to
the device.

Signed-off-by: Grzegorz Chwierut <grzegorz.chwierut@nordicsemi.no>
2023-06-17 07:54:46 -04:00
Piotr Golyzniak
4f77883043 scripts: twister: add timeout for pytest process
Add protection timeout for pytest subprocess, to avoid situation of
suspending whole Twister in case of internal pytest test problem.

Co-authored-by: Grzegorz Chwierut <grzegorz.chwierut@nordicsemi.no>
Signed-off-by: Piotr Golyzniak <piotr.golyzniak@nordicsemi.no>
2023-06-02 18:55:27 -04:00
Grzegorz Chwierut
ab5b48beed twister: pytest: Wrap iter_stdout
Added waiting for prompt instead of calling time.sleep,
to fix issue with longer setting up platforms with TF-M
enabled. Wrapped iter_stdout generator, to avoid issues
when generator is created and called multiple times
in test scenarios.
Fixes: #58747

Signed-off-by: Grzegorz Chwierut <grzegorz.chwierut@nordicsemi.no>
2023-06-02 18:55:27 -04:00
Anas Nashif
dae79cefaa twister: coverage: set coverage platforms default to an empty list
When calling twister with --coverage and no platform specified on the
command line, set the the value of coverage platforms to an empty list.

Fixes #57534

Signed-off-by: Anas Nashif <anas.nashif@intel.com>
2023-05-30 06:30:39 -04:00
Lukasz Mrugala
164c26ff18 scripts: Typo in the word 'platform' fix
testinstance.py contains a string literal comparison to the
word 'platform', which has been misspelled as 'plaform'.

The change is verified by a test expansion.

Signed-off-by: Lukasz Mrugala <lukaszx.mrugala@intel.com>
2023-05-29 14:34:33 -04:00
Keith Short
dfa992c596 twister: Remove newline suffix in BinaryHandler
Update the code that removes newline suffix in BinaryHandler for
compatibility with python 3.8.

Fixes: https://github.com/zephyrproject-rtos/zephyr/issues/58335

Signed-off-by: Keith Short <keithshort@google.com>
2023-05-26 14:59:03 -04:00
Grzegorz Chwierut
f1f305f4ae scripts: twister: adaptation for pytest plugin
Making the necessary changes to enable the new pytest plugin.
By default Twister should work without the pytest-twister-harness
plugin installed. To achieve this, each time Twister calls pytest,
the PYTHONPATH environment variable is expanded and the
`-p twister_harness.plugin` option is added to the pytest command.

Co-authored-by: Piotr Golyzniak <piotr.golyzniak@nordicsemi.no>
Signed-off-by: Grzegorz Chwierut <grzegorz.chwierut@nordicsemi.no>
2023-05-26 13:52:15 -04:00
Piotr Golyzniak
8c4bfcf324 scripts: add pytest plugin
Adding pytest plugin dedicated to running pytest tests in Zephyr
project. This plugin provides a dut fixture which allows to handle
bidirectional communication with the device under test. This version
of plugin can be used for tests dedicated to real hardware, QEMU and
native_posix simulator.

Co-authored-by: Lukasz Fundakowski <lukasz.fundakowski@nordicsemi.no>
Co-authored-by: Grzegorz Chwierut <grzegorz.chwierut@nordicsemi.no>
Co-authored-by: Katarzyna Giadla <katarzyna.giadla@nordicsemi.no>
Signed-off-by: Piotr Golyzniak <piotr.golyzniak@nordicsemi.no>
2023-05-26 13:52:15 -04:00
Michał Szprejda
bdf02ff5d6 Twister: Add integration with renode-test
Add support for calling the `renode-test` command from west and twister.
Enable running Robot Framework tests suites in Renode.

Signed-off-by: Michał Szprejda <mszprejda@antmicro.com>
Signed-off-by: Mateusz Hołenko <mholenko@antmicro.com>
2023-05-26 09:43:49 -04:00
Lukasz Mrugala
6ce37948a0 scripts: Twister ConfigurationError Fix
Current implementation of ConfigurationError causes
an exception to be thrown.
It is because a PosixPath is being concatenated with an str with a '+'
operator, which is not permitted.
The path in question is now cast to an str.
Additional test has been added to test_twister.py to track regressions.

Signed-off-by: Lukasz Mrugala <lukaszx.mrugala@intel.com>
2023-05-26 05:50:20 -04:00
Kumar Gala
0c5ff8ce43 twister: set default supported toolchains in twister
For the arch's that have standard toolchains thare are supported
across the arch set them in twister rather than in <BOARD>.yaml.

Signed-off-by: Kumar Gala <kumar.gala@intel.com>
2023-05-26 05:48:59 -04:00
Anas Nashif
6feacfa243 twister: improve new ztest case detection
Address cases where a suite and its testcases are defined in different
files.

Signed-off-by: Anas Nashif <anas.nashif@intel.com>
2023-05-25 04:44:11 -04:00
Anas Nashif
164c887ec1 twister: fix testcase detection
Do not reset initial list of testcases in cases where we do not use
ztest.

Signed-off-by: Anas Nashif <anas.nashif@intel.com>
2023-05-25 04:44:11 -04:00
Yuval Peress
64c41022e2 twister: Add support for gTest as a harness
Some 3P logic (Pigweed for example) is already heavily invested in
gTest as a testing framework. Adding the `gtest` harness enables
running the existing 3P tests on various Zephyr platforms and configs.

Signed-off-by: Yuval Peress <peress@google.com>
2023-05-24 19:39:10 -04:00
Maureen Helm
6a6d05b941 twister: Remove newline suffix in BinaryHandler
Simics converts newline characters when writing console output to
stdout, so we need to remove them as a suffix string. Otherwise the
console harness fails to match expected PASS/FAIL/RunID string patterns
and twister tests timeout.

Signed-off-by: Maureen Helm <maureen.helm@intel.com>
2023-05-24 13:14:27 -04:00
Grzegorz Chwierut
243a9dfba4 scripts: twister: skip unknown platform from hardware map
After generating a hardware-map a platform field is marked
as 'unknown'. If it is not changed, do not process them
when running --device-testing.
Signed-off-by: Grzegorz Chwierut <grzegorz.chwierut@nordicsemi.no>
2023-05-22 11:24:46 -04:00
Piotr Golyzniak
e8c3baa4a1 scripts: twister: create build dir for logs
In some cases CMake doesn't create build directory and in those cases
Twister should create this directory by itself.

Signed-off-by: Piotr Golyzniak <piotr.golyzniak@nordicsemi.no>
2023-05-22 07:03:10 -04:00
Anas Nashif
729a795aef twister: fix testcase parsing from binary
We have been clearning the list of testcases in every iteration over the
symbols ending up with only one testcase in the instance.

Signed-off-by: Anas Nashif <anas.nashif@intel.com>
2023-05-18 07:05:15 -04:00
Gerard Marull-Paretas
21b9a1c823 twister: allow loading external configuration
Add a new option to Twister that allows to load alternative
configuration files, `--alt-config-root`. This new option takes an
arbitrary number of root folders where alternative configuration files
can be stored. Twister will check if a test configuration file exists in
any of the alternative test configuration folders. For example, given
`$test_root/tests/foo/testcase.yaml`, Twister will use
`$alt_config_root/tests/foo/testcase.yaml` if it exists.

This feature can be useful if an out-of-tree project needs to run
upstream tests in different configurations, potentially not available
upstream (e.g. an out-of-tree board, or Kconfig setting).

Note that overlaying has been discarded because we can't easily delete
fields from the original configuration file, something that in certain
cases could be required. Current approach may lead to some dupplication,
but guarantees full control of the test configuration.

Signed-off-by: Gerard Marull-Paretas <gerard.marull@nordicsemi.no>
2023-05-16 07:25:52 -04:00
Maciej Perkowski
164ad1065f twister: Raise error when duplicates found on default
Make twister to raise error by default whenever duplicated test ids
are found. Remove redundant option to list duplicates.

Signed-off-by: Maciej Perkowski <Maciej.Perkowski@nordicsemi.no>
2023-05-16 06:21:25 -04:00
Dmitrii Golovanov
7617046c67 twister: Add expected and detected test case names to debug log
Log expected and detected test case names for the test suite name
check debug.

Signed-off-by: Dmitrii Golovanov <dmitrii.golovanov@intel.com>
2023-05-12 10:03:52 +02:00
Dmitrii Golovanov
8ee23dcdb8 twister: Don't override test name check with RunID check
Don't override a test case name check failed result by the following
RunID check failed result to report clearly about test case mismatch
situation as well as more specifically when the same test case appears
with unexpected RunID.

Signed-off-by: Dmitrii Golovanov <dmitrii.golovanov@intel.com>
2023-05-12 10:03:52 +02:00
Dmitrii Golovanov
14ba39c50e twister: Fix suite-name-check bypass on flash error
Fix suite-name-check bypassed at DeviceHandler on flash errors.

If the flash timeout is smaller than needed, but test images are
actually flashed successfully and appropriate tests executed,
then results from the previous test case were taken because
the flash error status makes the suite-name-check bypassed.

Signed-off-by: Dmitrii Golovanov <dmitrii.golovanov@intel.com>
2023-05-12 10:03:52 +02:00
Gerard Marull-Paretas
76b91c700d twister: fix handling of mixed str/list configurations
We can now have situations like this when it comes to lists in Twister
config:

```yaml
common:
  platform_exclude: foo
tests:
  my.test:
    platform_exclude: bar
```

```yaml
common:
  platform_exclude: foo
tests:
  my.test:
    platform_exclude:
      - bar
      - baz
```

```yaml
common:
  platform_exclude:
    - foo
    - bar
tests:
  my.test:
    platform_exclude: baz
```

```yaml
common:
  platform_exclude:
    - foo
    - bar
tests:
  my.test:
    platform_exclude:
      - baz
      - bazz
```

This patch fixes handling of cases (2), (3) and (4).

Signed-off-by: Gerard Marull-Paretas <gerard.marull@nordicsemi.no>
2023-05-11 09:17:44 -04:00
Gerard Marull-Paretas
a91620f5bb twister: deprecate space-separated lists
Deprecate in favor of native YAML lists.

Signed-off-by: Gerard Marull-Paretas <gerard.marull@nordicsemi.no>
2023-05-10 09:52:37 +02:00
Gerard Marull-Paretas
0db74702c3 twister: handle mixed common/tests types
This patch handles mixed types between common/tests sections in the
configuration, e.g. str+list, list+str, str+str.

Signed-off-by: Gerard Marull-Paretas <gerard.marull@nordicsemi.no>
2023-05-10 09:52:37 +02:00
Gerard Marull-Paretas
d3b8b8d9db twister: improve diagnostics for 'any' set/list types
Since pykwalify does not support multi-type fields, we are forced to use
the 'any' type. This is used for fields that allow sets/lists written as
either YAML lists or space-separated strings. This patch checks for
str/list Python type, and raises a configuration error if not true so
that user obtains better error messages in case the configuration file
contains invalid format.

Signed-off-by: Gerard Marull-Paretas <gerard.marull@nordicsemi.no>
2023-05-10 09:52:37 +02:00
Gerard Marull-Paretas
560a72171e twister: allow list input for set config fields
Twister configuration files have a few fields defined with the "set"
type. The set needs to be specified as a space-separated list of
entries, for example:

```yaml
plaform_allow: boardx boardy
```

This patch allows users to write these fields as YAML lists, for
example:

```yaml
platform_allow:
  - boardx
  - boardy
```

The list gets converted to a Python set, therfore the same exact
behavior is kept in Twister.

In some cases like in the example above, using lists allows to have more
readable configuration files, specially if the list is long. It also
helps on diffs, provided we have one item per line.

Signed-off-by: Gerard Marull-Paretas <gerard.marull@nordicsemi.no>
2023-05-10 09:52:37 +02:00
Maciej Perkowski
3bf7f83f21 twister: Bring back scope selection rule using platform_allow
A change to the scope selection rules was introduced by #52715
but it doesn't comply with the description in the code. What's more,
the change introduces major issues i.e. non deterministic scope
selection, especially in a CI environment. More context at #57595

Signed-off-by: Maciej Perkowski <Maciej.Perkowski@nordicsemi.no>
2023-05-08 11:18:44 +02:00
Gerard Marull-Paretas
baeee77ba6 twister: automatically add module board roots
Twister does not handle out-of-tree boards automatically. Modules like
e.g. example-application that define their own board root folder, are
required to inform twister via `--board-root`. In contrast, west is able
to to this automatically. This patch makes twister a bit smarter by
automatically filling the board root list with module paths as well.

Signed-off-by: Gerard Marull-Paretas <gerard.marull@nordicsemi.no>
2023-05-02 13:25:12 -04:00
Piotr Kosycarz
7892f40a52 twister: add options to shuffle tests across subsets
Add option to shuffle tests (randomly) before they are split into subsets.
This allow to fairly distribute tests across subsets,
so that long tests are not crowded in a single group.

This also allows to detect unwanted dependencies in test execution order.

Also, added option to provide custom seed to random generator
used to shuffle tests.
This will allow to reproduce certain test order if needed.
Used seed is printed at console.

Signed-off-by: Piotr Kosycarz <piotr.kosycarz@nordicsemi.no>
2023-05-01 06:12:59 -04:00
Kumar Gala
47cf00058e fb: cfb: Use TYPE_SECTION macros for cfb_fonts
Clean up cfb_fonts to utilize TYPE_SECTION macros for handling sections.

Signed-off-by: Kumar Gala <kumar.gala@intel.com>
2023-04-19 17:14:54 +02:00
Kumar Gala
5b5fbc4c11 shell: remove dead linker section for shell
The initshell sections in the linker scripts where associated with theo
old shell code.  The old shell code has been removed for some time so
remove references to initshell in the linker scripts and size_calc.py

Signed-off-by: Kumar Gala <kumar.gala@intel.com>
2023-04-19 17:14:37 +02:00
Andrei Emeltchenko
29dc4b68ec twister: Cleanup redundant if / else
Convert following construction:

elif self.options.device_serial or self.options.device_serial_pty:
 if self.options.device_serial:
   ...
 else
   ...

to the construction:

elif self.options.device_serial:
  ...
else self.options.device_serial_pty:
  ...

Signed-off-by: Andrei Emeltchenko <andrei.emeltchenko@intel.com>
2023-04-19 03:27:08 -04:00
Andrei Emeltchenko
186d706c73 twister: Use flash options also for device_serial_pty
Use flash_timeout and flash_with_test options not only for
serial console but also for device_serial_pty.

Signed-off-by: Andrei Emeltchenko <andrei.emeltchenko@intel.com>
2023-04-19 03:27:08 -04:00
Kumar Gala
63ed1d977a twister: size_calc: Fix compliance warning in script
compliance script runs pylink on size_calc.py and reports:

C0325:Unnecessary parens after '=' keyword (superfluous-parens)

Fix warning by remove unnecessary parens.

Signed-off-by: Kumar Gala <kumar.gala@intel.com>
2023-04-19 02:38:46 -04:00
Tristan Honscheid
bfd6177631 twister: Include unit test testbinary in ELF search space
`TestInstance.get_elf_file()` does not include the binary name used for
unit testing (`testbinary`) in its search, causing unit tests to not
complete successfully. This does not cause tests to fail and thus goes
unnoticed.

Update the glob expressions used in `get_elf_file()` to also look for
files called `testbinary`

Signed-off-by: Tristan Honscheid <honscheid@google.com>
2023-04-13 19:56:26 -04:00
Andrei Emeltchenko
b9f880a141 twister: trivial: Correct license year
Use correct year in license.

Signed-off-by: Andrei Emeltchenko <andrei.emeltchenko@intel.com>
2023-04-13 11:05:47 -04:00
Kumar Gala
ae23da81f2 twister: make get_elf_file sysbuild aware
If we are doing a sysbuild get_elf_file needs to return the default
domain's build_dir.  The standard build_dir will not any ELF builds
in it so will error out.

Signed-off-by: Kumar Gala <kumar.gala@intel.com>
2023-04-11 20:33:29 -04:00
Anas Nashif
d8126be557 twister: ignore remapped ELF file when getting ELF name
This is usually used only when doing size calculation, but now it is
being used for extracting symbols from the ELF for testing purposes and
some issues arise with multiple ELF files found.

Simplify the code a bit to make it easy to maintain a lists of things to
exclude.

Signed-off-by: Anas Nashif <anas.nashif@intel.com>
2023-04-11 14:01:39 +02:00
Dmitrii Golovanov
7efec71a85 twister: Alleviate DeviceHandler serial port TOCTOU
The DeviceHandler class method monitor_serial() has time-of-check
to time-of-use (TOCTOU) race condition possible on its serial port
which might be closed and its file descriptor cleared in between
of isOpen() and is_waiting status checks.
The issue is alleviated with controlled exception handling.

Signed-off-by: Dmitrii Golovanov <dmitrii.golovanov@intel.com>
2023-04-11 09:34:57 +02:00
Dmitrii Golovanov
0d2ba9c793 twister: Fix DeviceHandler hangs without suitable DUT
Fixes an issue when DeviceHandler awaited for a suitable DUT infinitely.
It was possible if the device map has no devices with required platform,
or no devices with fixture needed for the test case, or devices without
serial connection. A trivial case is a typo in platform name.

Signed-off-by: Dmitrii Golovanov <dmitrii.golovanov@intel.com>
2023-04-11 09:34:41 +02:00
Kumar Gala
c09e8fae23 twister: Use pyelf to extract symbol information
For ztest twister would parse the symbol information that was
generated as part of the build (zephyr.symbols).  However the format
of the zephyr.symbols files is highly dependant on the toolchain.

Move to using pyelf to parse the symbol information directly from
zephyr.elf instead so that this works regardless of toolchain.

Signed-off-by: Kumar Gala <kumar.gala@intel.com>
2023-04-08 18:37:52 +02:00
Kumar Gala
53bae96793 twister: allow get_elf_file() to be called on native platforms
On native platforms we typically have CONFIG_BUILD_OUTPUT_EXE set so
we end up with both zephyr.elf and zephyr.exe.  This would cause a
call to get_elf_file() to fail.  There isn't any reason to error out
since zephyr.elf and zephyr.exe should be identical.  So allow it
if the platform type is native.

Signed-off-by: Kumar Gala <kumar.gala@intel.com>
2023-04-08 18:37:52 +02:00