Re-arrange Twister command line options for test plan reporting
to a dedicated group with mutally-exclusive options to reflect
actual implementation and its dry-run execution mode.
Signed-off-by: Dmitrii Golovanov <dmitrii.golovanov@intel.com>
To specify the serial number, JLink expects either one argument
('--dev-id=xxx') or two (e.g. '--dev-id' 'xxx'), but it can not deal
with a single one that is '--dev-id xxx'.
The problem has been introduced (or just made visible?) by commit
5ee4284320 (twister: runner: j-link: use
dev-id instead of SelectEmuBySN) in PR #76931.
How to reproduce:
1. Create a HW configuration map, e.g.:
```
$ cat zephyr-hw-map-nrf52840dk-1.yml
- connected: true
id: '683517317'
platform: nrf52840dk/nrf52840
product: nRF52840 DK 1
runner: jlink
serial: /dev/ttyACM-nrf-dk-1
```
2. Run test `logging.dictionary` with Twister:
```
$ west twister --platform nrf52840dk/nrf52840 --device-testing \
--hardware-map zephyr-hw-map-nrf52840dk-1.yml -s logging.dictionary
```
3. The build will fail, and the `twister_harness.log` contains:
```
10:21:24.375:DEBUG:twister_harness.device.factory: Get device type
"hardware"
10:21:24.375:DEBUG:twister_harness.device.hardware_adapter: Opening
serial connection for /dev/ttyACM-nrf-dk-1
10:21:24.376:DEBUG:twister_harness.device.hardware_adapter: Flashing
device 683517317
10:21:24.376:DEBUG:twister_harness.device.hardware_adapter: Flashing
command: <snip>/bin/west flash --skip-rebuild --build-dir
twister-out/<snip>/tests/subsys/logging/dictionary/logging.dictionary
--runner jlink '--dev-id 683517317'
10:21:24.590:ERROR:twister_harness.device.hardware_adapter: Could not
flash device 683517317
10:21:24.592:DEBUG:twister_harness.device.hardware_adapter: Closed
serial connection for /dev/ttyACM-nrf-dk-1
```
(note the '--dev-id 683517317' part)
4. Running the stated `west flash` command shows the following error
message:
```
-- west flash: using runner jlink
FATAL ERROR: runner jlink received unknown arguments: ['--dev-id
683517317']
```
Signed-off-by: Reto Schneider <reto.schneider@husqvarnagroup.com>
Readline method sometimes receives more lines in buffer.
Split them to avoid misinterpreting data in harness module.
Signed-off-by: Grzegorz Chwierut <grzegorz.chwierut@nordicsemi.no>
Use pytest-args parameters provided in command line after
parameters taken from yaml file. The last occurence is
considered by argparse, so parameters from command line
will be used.
Fixes: #77319
Signed-off-by: Grzegorz Chwierut <grzegorz.chwierut@nordicsemi.no>
The gtest 1.15 has slightly different output from the original one that
was used to set up the harness. With this change we support both output
formats.
Fixes#72318
Signed-off-by: Yuval Peress <peress@google.com>
The serial number for debugger selection over USB
can be selected with the dev-id. This change
reflects also more the workflow of west flash
with J-Link.
The usage of SelectEmuBySN breaks the support for
J-Link over IP with twister.
Signed-off-by: Michael Arnold <marnold@baumer.com>
Extend pytest-args from configuration yaml file with args
from command line instead of overwriting.
Signed-off-by: Grzegorz Chwierut <grzegorz.chwierut@nordicsemi.no>
Fix confusing `TypeError: 'type' object is not subscriptable`
at _get_installed_packages() if twister runs on python version < 3.9
Signed-off-by: Dmitrii Golovanov <dmitrii.golovanov@intel.com>
Improve DUT selection at DeviceHandler: for each DUT it counts
how many test instances have been failed on it during the current
twister execution, so the next available DUT will be chosen
ordering the eligible DUTs by less failures occured so far.
The new selection mechanism should increase chances to retry failed
tests on different DUTs, for instance to resolve ploblems when some
DUTs have connectivity or HW issues slowing down test plan execution,
or even block the execution when only one test suite runs whereas
the same first DUT candidate in the list is not working and others
were not chosen.
Signed-off-by: Dmitrii Golovanov <dmitrii.golovanov@intel.com>
Change Twister PyTest plugin's test finalizing sequence to release
the DUT it is used as the very last operation, after the Test Instance
status becomes fully updated from the execution results.
This also fix a race condition possible when pytest plugin releases
the DUT and it becomes acquired by another test while the current test
is not yet finalized completely.
Signed-off-by: Dmitrii Golovanov <dmitrii.golovanov@intel.com>
Twister DeviceHandler - add test failure counter for how many
test instances have been failed on each DUT (Device Under Test)
when it executes the current test plan.
Output DUT falure counter summary at the end of Twister run.
Signed-off-by: Dmitrii Golovanov <dmitrii.golovanov@intel.com>
Fix Twister DeviceHandler exit on SerialException when
it connects to the serial device in 'flash before' mode.
Signed-off-by: Dmitrii Golovanov <dmitrii.golovanov@intel.com>
Several improvements at Twister DeviceHandler when it releases
current DUT (Device Under Test):
- release the exact DUT which is used for the test instance
instead of all configured DUTs which happened to have
the same serial device configured.
- Twister PyTtest harness plugin adjustment to the above.
- additional debug logging to track DUT waiting/retain/release.
Signed-off-by: Dmitrii Golovanov <dmitrii.golovanov@intel.com>
Updated the Pytest-Harness methods to capture and log error messages from
the subprocess, when pytest command fails.
Ensured that error messages are logged with `--inline-log` option
and placed in twister log reports.
Signed-off-by: Grzegorz Chwierut <grzegorz.chwierut@nordicsemi.no>
While the option to inform pytest about which platform a test is for
(platform being the board, such as mps2/an385) is available, it wasn't
being used. This patch fixes that, by building a pytest command which
includes the platform.
Signed-off-by: Ederson de Souza <ederson.desouza@intel.com>
Add a new fixture, `unlaunched_dut`, which is basically the `dut`
one, but without launching the device - thus also not building the
application. It is useful for tests who need a finer control of
the building of the application, before the dut can be launched
to run it.
It will be used on a future patch, which will use it to enable LLEXT EDK
tests.
Signed-off-by: Ederson de Souza <ederson.desouza@intel.com>
Fix after #71401 merge at Twister Reporting:
* filter status value type mismatch;
* tesplan.json generation incorrect error logs.
Signed-off-by: Dmitrii Golovanov <dmitrii.golovanov@intel.com>
Incorrect status assignments were previously just warnings in logs.
Now they are elevated to errors.
Signed-off-by: Lukasz Mrugala <lukaszx.mrugala@intel.com>
Various different Statuses were joined into a single class,
TwisterStatus. This change anticipates further streamlining
of the Twister's approach to Status.
Code guarding Twister's Properties was shortened to a
value check only.
QEMUOutputStatus was left separate, as doubts were cast
whether it should remain a status. Leaving it separate makes
its removal easier.
Signed-off-by: Lukasz Mrugala <lukaszx.mrugala@intel.com>
TestInstanceStatus of TIMEOUT and FLASH were never really used.
They were checked for, but never assigned.
Signed-off-by: Lukasz Mrugala <lukaszx.mrugala@intel.com>
Now statuses are not just a str that can be easily mistyped
or assigned wrong. Now they are an Enum.
Signed-off-by: Lukasz Mrugala <lukaszx.mrugala@intel.com>
This change is removing some deprecation warnings
which for some reason causing failing tests with
Python 3.12 on CI #76877.
Also, it fixes warnings from pytest like:
PytestCollectionWarning: cannot collect test class 'TestPlan'
because it has a __init__ constructor
(from: scripts/tests/twister/test_testplan.py)
Signed-off-by: Lukasz Fundakowski <lukasz.fundakowski@nordicsemi.no>
When presenting an example of how to run a failing test case with
`west build`, provide the source dir path relative to the current
working directory, not the zephyr root directory.
Signed-off-by: Jordan Yates <jordan@embeint.com>
Read CONFIG_SHELL_PROMPT_UART from config file and use them
in shell fixture in pytest-harness package.
Signed-off-by: Grzegorz Chwierut <grzegorz.chwierut@nordicsemi.no>
Moved helper methods from tests/boot/with_mcumgr to pytest-harness
package. It can be reused by other tests.
Signed-off-by: Grzegorz Chwierut <grzegorz.chwierut@nordicsemi.no>
Twister scans C-files to find testcases that are implemented
using ZTest framework. Also runs scanning of Elf files
after building. Skip scanning files if it is not required.
Signed-off-by: Grzegorz Chwierut <grzegorz.chwierut@nordicsemi.no>
The default base timeout for pytest is statically set by
the TwisterHarnessConfig class to be 60 seconds. However,
sometimes it takes longer than 60s before the app starts
to run, especially on emulator/simulator where it takes
quite some time to start. So pass the test timeout as
the base timeout via pytest command line argument.
Signed-off-by: Daniel Leung <daniel.leung@intel.com>
Disable branch coverage for the `__ASSERT` family of macros. Covering
all of the assertion branches by definition means triggering the
assertion, which can be either challenging or impossible to exercise,
and in either case results in the immediate termination of the test.
Signed-off-by: Jordan Yates <jordan@embeint.com>
Multiple values for `--exclude-branches-by-pattern` will result in only
the last value taking effect. Resolve this by merging all the provided
regex patterns into a single pattern with the `|` operator.
Signed-off-by: Jordan Yates <jordan@embeint.com>
The gTest harness asssumed that the lines end with the test name, but
some gTest implementations include the test duration in the line. Update
both the tests and regex to allow this and also avoid capturing
characters into the `test_name` that cannot be valid test name chars.
Fixes#72318
Signed-off-by: Yuval Peress <peress@google.com>
The current version of scipts do not consider OOT boards use cases and
the tests with robot now are strict to only one robot file, which is
not realistic for real environment. This address those issues and allow
multiple testsuits at command line and lists at tests entries. It add
another test parameter to allow configure robotframework options.
Fixes: #74563
Signed-off-by: Gerson Fernando Budke <gerson.budke@ossystems.com.br>
The flag won't work with sysbuild since there is no way to
reliably tell to a parser which data came from which image.
fixes: #74092
Signed-off-by: Maciej Perkowski <Maciej.Perkowski@nordicsemi.no>
Execute 'zephyr.exe' in application build directory as 'current working
directory' (cwd). This makes sure that native_sim specific drivers (like
flash simulator with file backend in 'flash.bin') are using unique context
for external resources with relative paths.
This fixes executing native_sim tests in twister with flash simulator.
Previously a shared 'flash.bin' was used for all executed 'zephyr.exe'
processes in twister. After this patch a unique 'flash.bin' file is used
for each tested sample, since those 'flash.bin' is placed in application
build directory instead of twister root directory.
Signed-off-by: Marcin Niestroj <m.niestroj@emb.dev>
Stop Twister if there are too many backup copies of the output
directory already.
Before this fix, Twister silently kept artifacts from the last run,
unless `--clobber-output` was explicitly given.
Signed-off-by: Dmitrii Golovanov <dmitrii.golovanov@intel.com>
Extend the coverage tool to handle applications that generate multiple
gcov dumps in a single execution. This can happen when the application
calls `sys_reboot`.
Handling multiple dumps enables coverage testing of exception handlers.
Signed-off-by: Jordan Yates <jordan@embeint.com>
Pass Twister pytest plugin's log output, as well as output from
a test image running with pytest, up to Twister log irregardless
of the current verbosity level set at Twister.
This allows to collect full test output and also ensures that
recording data embedded in the test log is passed to the Twister
pytest harness for export.
Signed-off-by: Dmitrii Golovanov <dmitrii.golovanov@intel.com>
Extend Twister Harness recording feature to allow selected data fields,
extracted from the log by a regular expression, to be parsed into JSON
objects and eventually reported in `twister.json` as `recording` list
property of the test suite.
With this extension, log records can convey layered data structures
passed from a test image as summary results, traces, statistics, etc.
This extension also allows flexible recording structure: a test image
can output different types of data records incapsulated into a fixed
set of fields, so `recording.csv` file columns are respected, whereas
some of the columns keep strings with json-encoded semi-structured data.
Signed-off-by: Dmitrii Golovanov <dmitrii.golovanov@intel.com>
More complex platforms require sysbuild to use always, even for
such "simple" samples like hello_world. Such platforms can have
`sysbuild: true` entry in their board_name.yaml used by twister.
Using such entry will tell twister, that sysbuild must always be used
on a given platform.
Twister is aligned to have information about need of sysbuild at
instance (platform + suite) level (was only at suite level before).
Instance.sysbuild is true whenever a test suite or a platform requires
sysbuild.
Twister pytest unit tests are aligned with changes.
Signed-off-by: Maciej Perkowski <Maciej.Perkowski@nordicsemi.no>
Currently we launch qemu (well, "ninja run" usually) using Popen and
request stdout and stderr to be redirected into a pipe. However we never
read that pipe so the information is not captured.
Instead log directly into files that can be inspected after a failed
to to find out why qemu run failed.
Note that this is really only useful in cases where qemu either fails to
launch or crashes.
Regular test data is still handled via the qemu fifo.
Signed-off-by: Juha Kuikka <juha.kuikka@gmail.com>
we use reason for a failure to indicate state and then set the status
later and reason for the failure, in case of the failure is taken from
the handler status. Clean this up by setting status and reason coming
from the handler very early, so we do not have to go through replacing
meaning later.
Signed-off-by: Anas Nashif <anas.nashif@intel.com>
When dumping logs from the `--inline-logs` option, remove any coverage
information that may be contained in those logs. Coverage dumps are
unrelated to any test failures and make it harder to find the failing
test information. In extreme cases the relevant information is lost due
to terminal scrollback limits.
If the raw dump information is required, it is still present in the
original `handler.log` file.
Signed-off-by: Jordan Yates <jordan@embeint.com>
New Twister option `--footprint-report` is introduced to collect and
write detailed memory footprint results for symbols as an additional
JSON file. By default, the new option is disabled.
The new option implies and extends `--create-rom-ram-report`, so there
are three choices: 'ROM', 'RAM', and 'all' to select what memory area
symbols to report in `twister_footprint.json`.
In case of the custom report name, or per-platform report, it is always
composed with the rightmost '_footprint.json' suffix.
The memory footprint report has similar structure as `twister.json`
and compelements it having reduced set of test suite properties:
- instead of `testcases` it contains `footprint` object with
`rom.json` and `ram.json` artifacts embedded there;
- other properites are limited to represent only the essential test
suite context, thus to allow further data processing consistently
and independently from the `twister.json`.
- 'filtered' test instances are not included into the footprint report.
Signed-off-by: Dmitrii Golovanov <dmitrii.golovanov@intel.com>
When Twister composes resulting twister.json reports, add optional
filtering by a test instance resulting status and/or its individual
properties to be allowed/denied on output to the JSON file.
This internal feature is introduced to facilitate JSON reports with
a custom data schema.
Signed-off-by: Dmitrii Golovanov <dmitrii.golovanov@intel.com>
