When pytest is executed, it only writes logging messages to the twister
harness, and does not include information about any test assertion
failures. This information is printed to stdout, so simply write
the contents stdout to the harness log.
Remove the log file arguments previously passed to pytest, as these are
no longer needed.
Signed-off-by: Daniel DeGrasse <ddegrasse@tenstorrent.com>
In `--coverage-per-instance` mode for 'gcovr' merge individual
coverage.json reports into the aggregate code coverage report
instead of processing .gcda data files again.
Signed-off-by: Dmitrii Golovanov <dmitrii.golovanov@intel.com>
With this change, the coverage data (GCOV dump) is extracted from the
test log files after each of the test instance parallel execution,
instead of post processing all the logs at the end of the Twister run.
The new `--coverage-per-instance` mode extends Twister coverage operations
to report coverage statistics on each test instance execution individually
in addition to the default reporting mode which aggregates data to one
report with all the test instances in the current scope of the Twister run.
The split mode allows to identify precisely what amount of code coverage
each test suite provides and to analyze its contribution to the overall
test plan's coverage. Each test configuration's output directory will have
its own coverage report and data files, so the overall disk space and
the total execution time increase.
Another new `--disable-coverage-aggregation` option allows to execute
only the `--coverage-per-instance` mode when the aggregate coverage
report for the whole Twister run scope is not needed.
Signed-off-by: Dmitrii Golovanov <dmitrii.golovanov@intel.com>
Group Twister code coverage command line arguments for better usability.
Add more details in descriptions.
Signed-off-by: Dmitrii Golovanov <dmitrii.golovanov@intel.com>
In situations where a test won't be run because there is a gap
in the hardware support, add the gap information to the "reason"
field to aid in debugging test cases.
With this change, the reason field for the error in test case
will show the gap:
e.g. reason field now updated with gaps
"No hardware support {'usbd'} but is one of the integration platforms"
Signed-off-by: David Leach <david.leach@nxp.com>
Actually using the error count before clearing it
would probably be more useful.
ExecutionCounter's error count was set to 0
before being used to lower the done count properly.
This should fix in-progress overcounting,
where Twister claimed to execute more tests than planned.
Signed-off-by: Lukasz Mrugala <lukaszx.mrugala@intel.com>
Refactor Reporting to use custom ReportingJSONEncoder and encode
all pathlib.Path derived instances there which are possible
e.g. in 'environment.options' received from command line.
Fixes: #83823
Signed-off-by: Dmitrii Golovanov <dmitrii.golovanov@intel.com>
Device adapter strips all whitespace from output lines causing test
failures when matching default shell prompt "uart:~$ " with trailing
space. Update _handle_device_output to only strip line endings (\r\n)
while preserving whitespace required for prompt detection.
A testcase sample.pytest.shell.no_vt100 was added to verify prompt
matching works with CONFIG_SHELL_VT100_COLORS disabled.
Signed-off-by: Thomas Günther <thomas.guenther@limatica.com>
This commit removes all references to the `xtools` toolchain variant in the
twister scripts.
Note that the `xtools` toolchain variant has been deprecated since Zephyr
v3.3.0 and now removed.
Signed-off-by: Stephanos Ioannidis <root@stephanos.io>
Extend Twister Harness 'recording' feature to allow multiple
regular expression patterns to extract different types of records
from test output.
Add 'merge' recording mode to collect all extracted data fields
into a single record object of the test instance.
Export to CSV file now takes all field names occurred in the collected
records, sort it alphabetically, and then use it for columns instead of
using only the first record's fields. This is done to address possible
situation when records have different set of fields.
Adjust Twister documentation and test suite to the above changes.
Signed-off-by: Dmitrii Golovanov <dmitrii.golovanov@intel.com>
not force host variant if environment is setting something different in
the case of posix arch.
Fixes#83766
Signed-off-by: Anas Nashif <anas.nashif@intel.com>
Added integration_toolchains to allow building/testing with multiple
toolchains available in the environment.
This changes the output structure and adds another level in the path
under twister_out signifying the toolchain used. The toolchain used
(variant) is also part of the json output now.
Signed-off-by: Anas Nashif <anas.nashif@intel.com>
using "dt_node_prop_enabled" to filter out supported platforms
1. check node existing.
2. check prop in node.
3. check the prop is True.
Signed-off-by: Hake Huang <hake.huang@oss.nxp.com>
Pass a test failure reason text from harness to the instance
to convey more details on the failure.
Signed-off-by: Dmitrii Golovanov <dmitrii.golovanov@intel.com>
Set missing TestCase statuses when a test under the BinaryHandler failed
(crashed), so remaining 'STARTED' and 'NONE' are now 'BLOCK' instead.
Signed-off-by: Dmitrii Golovanov <dmitrii.golovanov@intel.com>
Pretty print XML reports with tab identation on levels for better
readability and ease text diff there as well.
Signed-off-by: Dmitrii Golovanov <dmitrii.golovanov@intel.com>
The testcase.classname properties in `twister_report.xml` and
`twister_suite_report.xml` now contain the full testsuite name
(without its path prefix) instead of just two its dot-separated
leftmost components. This way `twister_report.xml` testsuite container
has no duplicate testcase elements with the same `classname` and `name`
properties executing with --no-detailed-test-id for a project with same
testcase names in its 'sibling' suites, for example`tests/kernel/sleep`.
Signed-off-by: Dmitrii Golovanov <dmitrii.golovanov@intel.com>
Extend `--no-detailed-test-id` command line option: in addition to its
current behavior to exclude from a test Suite name its configuration path
prefix, also don't prefix each Ztest Case name with its Scenario name.
For example: 'kernel.common.timing' Scenario name, the same Suite name,
and 'sleep.usleep' test Case (where 'sleep' is its Ztest suite name
and 'usleep' is Ztest test name.
This way both TestSuite and TestCase names follow the same principle
having no parent object name prefix.
There is no information loss in Twister reports with this naming:
TestSuite is a container object for its TestCases, whereas TestSuite
has its configuration path as a property.
Signed-off-by: Dmitrii Golovanov <dmitrii.golovanov@intel.com>
Milliseconds are logged as is, without leading zeros. This means that
following log content is produced, which is not aligned by columns:
12:54:27.972:INFO:root: Test message
12:54:27.992:INFO:root: Test message
12:54:28.12:INFO:root: Test message
12:54:28.32:INFO:root: Test message
12:54:28.52:INFO:root: Test message
12:54:28.73:INFO:root: Test message
12:54:28.93:INFO:root: Test message
12:54:28.113:INFO:root: Test message
12:54:28.133:INFO:root: Test message
Change milliseconds logging to include leading zeroes. This allows to print
nicely aligned log messages:
12:54:27.972:INFO:root: Test message
12:54:27.992:INFO:root: Test message
12:54:28.012:INFO:root: Test message
12:54:28.032:INFO:root: Test message
12:54:28.052:INFO:root: Test message
12:54:28.073:INFO:root: Test message
12:54:28.093:INFO:root: Test message
12:54:28.113:INFO:root: Test message
12:54:28.133:INFO:root: Test message
Additionally it is much easier to configure programs simplifying
timestamped log browsing (like in logview-mode for Emacs), as it is more
standard and does not require custom regex matching.
Signed-off-by: Marcin Niestroj <m.niestroj@emb.dev>
Pass additional args to test binary. Twister
passes it only for native_sim.
Fixes#82463
Signed-off-by: Grzegorz Chwierut <grzegorz.chwierut@nordicsemi.no>
Use with statement to open stdout/stderr and resolve current warning
when running twister:
ResourceWarning: unclosed file....
Signed-off-by: Anas Nashif <anas.nashif@intel.com>
In case of unknown platforms in filters such as plaform_allow,
integration_plaforms or platform_exclude, report an error and exit.
Signed-off-by: Anas Nashif <anas.nashif@intel.com>
Add documentation for the get_filtered_output method in both the Python
implementation and the Integration with pytest chapter. The method is used
to filter command outputs by removing prompts and log messages, making it
easier to process shell command results.
Signed-off-by: Thomas Günther <thomas.guenther@limatica.com>
removed traling whitespace
in CI, we may need to quit if there is any failure
to save time, so add this --quit-on-failure so that
any failure will quit the test.
Signed-off-by: Hake Huang <hake.huang@oss.nxp.com>
There is an incorrect logic that causes twister with
--device-testing to skip all tests on Windows. Correct
the logical condition.
Signed-off-by: Dat Nguyen Duy <dat.nguyenduy@nxp.com>
bsim requires a special setup for it to be usuable and run by twister.
If this environment is not available, we should not error when the
platform is an integration platform.
Most developers and users of twister will not have the babblesim setup
and reporting errors is misleading and confusing. So if such env. is not
available, just filter as usual and do not error.
Signed-off-by: Anas Nashif <anas.nashif@intel.com>
Support for space-separated lists was deprecated a long time ago
a91620f5bb so it is time to remove
support for them.
Any project that has not migrated can still use
`scripts/utils/twister_to_list.py` to automatically migrate twister
files.
Signed-off-by: Gerard Marull-Paretas <gerard@teslabs.com>
update test cases for twister
1. test_errors.py add protection.
2. test_handlers.py change call to status
3. test_testsuite.py change call to status
The log traces for TwisterException objects only.
And the stack trace output follow the same rules for all exceptions,
but StatusAttributeError with its dedicated handlers.
Signed-off-by: Hake Huang <hake.huang@oss.nxp.com>
This is line is breaking in some CI setups since the recent ruff fixup
with:
ModuleNotFoundError: No module named 'domains'
Try and fixit by explicitly adding the path before the include, all
other instances have it already.
Signed-off-by: Fabio Baltieri <fabiobaltieri@google.com>
This fixes ruff linting error UP007,
where one would use Optional or Union
instead of X | Y notation.
Signed-off-by: Lukasz Mrugala <lukaszx.mrugala@intel.com>
Fixes ruff linting errors UP030, UP031, UP032,
which make you use format specifiers (fstrings)
instead of printf formatting or str.format().
Signed-off-by: Lukasz Mrugala <lukaszx.mrugala@intel.com>
This fixes ruff linting error F541,
where an fstring is used when a normal
string would suffice.
Signed-off-by: Lukasz Mrugala <lukaszx.mrugala@intel.com>
