Fix after #71401 merge at Twister Reporting:
* filter status value type mismatch;
* tesplan.json generation incorrect error logs.
Signed-off-by: Dmitrii Golovanov <dmitrii.golovanov@intel.com>
Incorrect status assignments were previously just warnings in logs.
Now they are elevated to errors.
Signed-off-by: Lukasz Mrugala <lukaszx.mrugala@intel.com>
Various different Statuses were joined into a single class,
TwisterStatus. This change anticipates further streamlining
of the Twister's approach to Status.
Code guarding Twister's Properties was shortened to a
value check only.
QEMUOutputStatus was left separate, as doubts were cast
whether it should remain a status. Leaving it separate makes
its removal easier.
Signed-off-by: Lukasz Mrugala <lukaszx.mrugala@intel.com>
TestInstanceStatus of TIMEOUT and FLASH were never really used.
They were checked for, but never assigned.
Signed-off-by: Lukasz Mrugala <lukaszx.mrugala@intel.com>
Now statuses are not just a str that can be easily mistyped
or assigned wrong. Now they are an Enum.
Signed-off-by: Lukasz Mrugala <lukaszx.mrugala@intel.com>
This change is removing some deprecation warnings
which for some reason causing failing tests with
Python 3.12 on CI #76877.
Also, it fixes warnings from pytest like:
PytestCollectionWarning: cannot collect test class 'TestPlan'
because it has a __init__ constructor
(from: scripts/tests/twister/test_testplan.py)
Signed-off-by: Lukasz Fundakowski <lukasz.fundakowski@nordicsemi.no>
When presenting an example of how to run a failing test case with
`west build`, provide the source dir path relative to the current
working directory, not the zephyr root directory.
Signed-off-by: Jordan Yates <jordan@embeint.com>
Read CONFIG_SHELL_PROMPT_UART from config file and use them
in shell fixture in pytest-harness package.
Signed-off-by: Grzegorz Chwierut <grzegorz.chwierut@nordicsemi.no>
Moved helper methods from tests/boot/with_mcumgr to pytest-harness
package. It can be reused by other tests.
Signed-off-by: Grzegorz Chwierut <grzegorz.chwierut@nordicsemi.no>
Twister scans C-files to find testcases that are implemented
using ZTest framework. Also runs scanning of Elf files
after building. Skip scanning files if it is not required.
Signed-off-by: Grzegorz Chwierut <grzegorz.chwierut@nordicsemi.no>
The default base timeout for pytest is statically set by
the TwisterHarnessConfig class to be 60 seconds. However,
sometimes it takes longer than 60s before the app starts
to run, especially on emulator/simulator where it takes
quite some time to start. So pass the test timeout as
the base timeout via pytest command line argument.
Signed-off-by: Daniel Leung <daniel.leung@intel.com>
Disable branch coverage for the `__ASSERT` family of macros. Covering
all of the assertion branches by definition means triggering the
assertion, which can be either challenging or impossible to exercise,
and in either case results in the immediate termination of the test.
Signed-off-by: Jordan Yates <jordan@embeint.com>
Multiple values for `--exclude-branches-by-pattern` will result in only
the last value taking effect. Resolve this by merging all the provided
regex patterns into a single pattern with the `|` operator.
Signed-off-by: Jordan Yates <jordan@embeint.com>
The gTest harness asssumed that the lines end with the test name, but
some gTest implementations include the test duration in the line. Update
both the tests and regex to allow this and also avoid capturing
characters into the `test_name` that cannot be valid test name chars.
Fixes#72318
Signed-off-by: Yuval Peress <peress@google.com>
The current version of scipts do not consider OOT boards use cases and
the tests with robot now are strict to only one robot file, which is
not realistic for real environment. This address those issues and allow
multiple testsuits at command line and lists at tests entries. It add
another test parameter to allow configure robotframework options.
Fixes: #74563
Signed-off-by: Gerson Fernando Budke <gerson.budke@ossystems.com.br>
The flag won't work with sysbuild since there is no way to
reliably tell to a parser which data came from which image.
fixes: #74092
Signed-off-by: Maciej Perkowski <Maciej.Perkowski@nordicsemi.no>
Execute 'zephyr.exe' in application build directory as 'current working
directory' (cwd). This makes sure that native_sim specific drivers (like
flash simulator with file backend in 'flash.bin') are using unique context
for external resources with relative paths.
This fixes executing native_sim tests in twister with flash simulator.
Previously a shared 'flash.bin' was used for all executed 'zephyr.exe'
processes in twister. After this patch a unique 'flash.bin' file is used
for each tested sample, since those 'flash.bin' is placed in application
build directory instead of twister root directory.
Signed-off-by: Marcin Niestroj <m.niestroj@emb.dev>
Stop Twister if there are too many backup copies of the output
directory already.
Before this fix, Twister silently kept artifacts from the last run,
unless `--clobber-output` was explicitly given.
Signed-off-by: Dmitrii Golovanov <dmitrii.golovanov@intel.com>
Extend the coverage tool to handle applications that generate multiple
gcov dumps in a single execution. This can happen when the application
calls `sys_reboot`.
Handling multiple dumps enables coverage testing of exception handlers.
Signed-off-by: Jordan Yates <jordan@embeint.com>
Pass Twister pytest plugin's log output, as well as output from
a test image running with pytest, up to Twister log irregardless
of the current verbosity level set at Twister.
This allows to collect full test output and also ensures that
recording data embedded in the test log is passed to the Twister
pytest harness for export.
Signed-off-by: Dmitrii Golovanov <dmitrii.golovanov@intel.com>
Extend Twister Harness recording feature to allow selected data fields,
extracted from the log by a regular expression, to be parsed into JSON
objects and eventually reported in `twister.json` as `recording` list
property of the test suite.
With this extension, log records can convey layered data structures
passed from a test image as summary results, traces, statistics, etc.
This extension also allows flexible recording structure: a test image
can output different types of data records incapsulated into a fixed
set of fields, so `recording.csv` file columns are respected, whereas
some of the columns keep strings with json-encoded semi-structured data.
Signed-off-by: Dmitrii Golovanov <dmitrii.golovanov@intel.com>
More complex platforms require sysbuild to use always, even for
such "simple" samples like hello_world. Such platforms can have
`sysbuild: true` entry in their board_name.yaml used by twister.
Using such entry will tell twister, that sysbuild must always be used
on a given platform.
Twister is aligned to have information about need of sysbuild at
instance (platform + suite) level (was only at suite level before).
Instance.sysbuild is true whenever a test suite or a platform requires
sysbuild.
Twister pytest unit tests are aligned with changes.
Signed-off-by: Maciej Perkowski <Maciej.Perkowski@nordicsemi.no>
Currently we launch qemu (well, "ninja run" usually) using Popen and
request stdout and stderr to be redirected into a pipe. However we never
read that pipe so the information is not captured.
Instead log directly into files that can be inspected after a failed
to to find out why qemu run failed.
Note that this is really only useful in cases where qemu either fails to
launch or crashes.
Regular test data is still handled via the qemu fifo.
Signed-off-by: Juha Kuikka <juha.kuikka@gmail.com>
we use reason for a failure to indicate state and then set the status
later and reason for the failure, in case of the failure is taken from
the handler status. Clean this up by setting status and reason coming
from the handler very early, so we do not have to go through replacing
meaning later.
Signed-off-by: Anas Nashif <anas.nashif@intel.com>
When dumping logs from the `--inline-logs` option, remove any coverage
information that may be contained in those logs. Coverage dumps are
unrelated to any test failures and make it harder to find the failing
test information. In extreme cases the relevant information is lost due
to terminal scrollback limits.
If the raw dump information is required, it is still present in the
original `handler.log` file.
Signed-off-by: Jordan Yates <jordan@embeint.com>
New Twister option `--footprint-report` is introduced to collect and
write detailed memory footprint results for symbols as an additional
JSON file. By default, the new option is disabled.
The new option implies and extends `--create-rom-ram-report`, so there
are three choices: 'ROM', 'RAM', and 'all' to select what memory area
symbols to report in `twister_footprint.json`.
In case of the custom report name, or per-platform report, it is always
composed with the rightmost '_footprint.json' suffix.
The memory footprint report has similar structure as `twister.json`
and compelements it having reduced set of test suite properties:
- instead of `testcases` it contains `footprint` object with
`rom.json` and `ram.json` artifacts embedded there;
- other properites are limited to represent only the essential test
suite context, thus to allow further data processing consistently
and independently from the `twister.json`.
- 'filtered' test instances are not included into the footprint report.
Signed-off-by: Dmitrii Golovanov <dmitrii.golovanov@intel.com>
When Twister composes resulting twister.json reports, add optional
filtering by a test instance resulting status and/or its individual
properties to be allowed/denied on output to the JSON file.
This internal feature is introduced to facilitate JSON reports with
a custom data schema.
Signed-off-by: Dmitrii Golovanov <dmitrii.golovanov@intel.com>
Make sure Twster DeviceHandler serial-pty process is terminated
with all its remaining children to avoid Twister hanging on it
infinitely.
The reolved issue occurs sometimes, for example when serial-pty
script is used for serial port tunneling over network.
Signed-off-by: Dmitrii Golovanov <dmitrii.golovanov@intel.com>
Filtered testcases are removed by default from Twister tests.
Older functionality is preserved via a new Twister flag:
--report-filtered.
Old tests were adjusted and a new test for that flag added.
Signed-off-by: Lukasz Mrugala <lukaszx.mrugala@intel.com>
Allow twister fixtures to contain extra information, which can be used for
test suite configuration. The extra information can be appended to existing
fixtures separated by a colon (i.e. <fixture>:<configuration>).
This is especially useful for the pytest harness, where a fixture of a
given type may need to refer to an instance of a particular piece of host
hardware needed by the pytest suite (e.g. a network interface, a UART, or a
CAN interface connected to the device under test).
Signed-off-by: Henrik Brix Andersen <hebad@vestas.com>
Pass the list of supported twister fixtures for a given platform to pytest
via DeviceConfig. This allows for the pytest suites to use knowledge of the
fixtures for test suite configuration.
Signed-off-by: Henrik Brix Andersen <hebad@vestas.com>
When something goes wrong with execution, due to twister issues and
bugs, do not continue with execution and abort the process and report
and return an error code.
Fixes#72807
Signed-off-by: Anas Nashif <anas.nashif@intel.com>
Read default domain from domains.yaml file and update
paths to proper build directory. It fixes native and qemu
pytest scenarios, when application is build with sysbuild.
Signed-off-by: Grzegorz Chwierut <grzegorz.chwierut@nordicsemi.no>
Remove the deprecated uart_mux and gsm_mux modules and all of
their configurations/dependencies across zephyr.
Optimally uart_mux and gsm_mux would be removed in their own
respective PRs, but the two modules are directly coupled, so
to preserve bisectability, they must be removed together.
Signed-off-by: Bjarki Arge Andreasen <bjarki@arge-andreasen.me>
Initial integration with renode-test was introduced in commit bdf02ff,
which added support for calling the `renode-test` command from both west
and twister.
This commit removes the custom run_renode_test target used for running
Robot tests with the `west build` command and makes twister call
`renode-test` directly instead.
Signed-off-by: Michał Szprejda <mszprejda@antmicro.com>
The --flash-before flag allows devices to be flashed before the serial
connection is established. However, the implementation was incomplete and
only worked if the port address already existed at the start of the run.
This is incompatible with devices that implement the USB in software
(eg: USB-CDC).
This commit fixes the implementation to delay setting up the connection
until after the device is flashed, and to retry the connection for two
seconds after flashing to give the device time to enumerate as a USB
device.
Signed-off-by: Mike Szczys <mike@golioth.io>
Terminate QEMU earlier if the test finishes and no output data is
received for 1 second. This new timeout operates in parallel with the
global test timeout.
For coverage testing, this can reduce the time spent running individual
tests by up to 29 seconds, while still giving the full 30 extra seconds
to dump gcov data if needed.
Signed-off-by: Jordan Yates <jordan@embeint.com>
In the case where a test causes the test executor to crash, the stderr
is currently lost, making it hard to debug failures. This changes it
so that the process' stderr gets captured to 'handler_stderr.log' for
inspection.
Signed-off-by: Benjamin Gwin <bgwin@google.com>
PR #72100 broke execution of unit type tests. This commit fixes
it by bringing back the "old" logic for selecting a binary path
for unit type tests.
Signed-off-by: Maciej Perkowski <Maciej.Perkowski@nordicsemi.no>
Fix default command line options check dependency on `--device-testing`
when it is provided to run Twister and turn device command options
as mandatory.
Fixes after changes introduced with #72399.
Signed-off-by: Dmitrii Golovanov <dmitrii.golovanov@intel.com>
Store Twister command line options in twister.json report as
`environment.options` object. It allows to keep the actual
execution context for test results analysis and history comparison.
A new command line option `--report-all-options` enables to report all
command line options applied, including these set as default values.
Signed-off-by: Dmitrii Golovanov <dmitrii.golovanov@intel.com>
in many platforms, before the debuger flash, the former testsuite
many already executed, so we will see some old testsuite name.
and we should not compare them directly, instead if we compare in
reversed order them we can avoid such mis-judge.
Signed-off-by: Hake Huang <hake.huang@oss.nxp.com>
Fix issue 72083. Update path to zephyr.exe binary using
default domain from domains.yaml file.
Signed-off-by: Grzegorz Chwierut <grzegorz.chwierut@nordicsemi.no>
Added a switch that show failed/error report from the last run.
Default shows all items found. However, you can specify the number of items
(e.g. --report-summary 15).
It also works well with the --outdir switch
Signed-off-by: Kamil Paszkiet <kamilx.paszkiet@intel.com>
Extend Pytest Harness to support 'recording' feature to parse
test log by a regular expression and collect as records the same
way as Console Harness do.
Signed-off-by: Dmitrii Golovanov <dmitrii.golovanov@intel.com>
