Commit graph

286 commits

Author SHA1 Message Date
Maciej Perkowski
61e9de5ec4 twister: Minor quarantine fixes
Statuses of quarantined test cases were not updated properly.
Quarantined tests shouldn't cause errors in integration mode.
Removed obsolate filter type.

Signed-off-by: Maciej Perkowski <maciej.perkowski@nordicsemi.no>
2025-06-10 10:22:51 -04:00
Maciej Perkowski
49595c7309 twister: Adjust status for quarantined instances
Skipped status fits quarantined items better than filtered.
Filtered tests are by default removed from reports, which
shouldn't be the case for quarantined tests.
Adjust tests unit and blackbox tests accordingly.

Signed-off-by: Maciej Perkowski <maciej.perkowski@nordicsemi.no>
2025-06-05 09:34:13 +02:00
Mateusz Junkier
04b49fadab twister: harness_config extended by test retries in testcase.yaml
Extend Twister to allow defining repeat counts
for specific scenarios by harness_config.

Signed-off-by: Mateusz Junkier <mateusz.junkier@intel.com>
2025-05-13 03:20:10 +02:00
Luca Burelli
021d90b641 tests: twister_blackbox: fix vendor filter test
Commit 6dc27a4 ("boards: align vendor entry in board.yml files")
unknowingly broke Twister blackbox tests by correctly placing Atom
targets under the 'intel' vendor. Since this was not previously the
case, the Twister blackbox vendor filter test for 'intel' was actually
expecting 'qemu_x86/atom' to be filtered out as "Not a selected vendor
platform."

This commit updates test_filter.py with the new vendor flags.

Signed-off-by: Luca Burelli <l.burelli@arduino.cc>
2025-05-03 17:49:29 +02:00
Lukasz Mrugala
d698209647 twister: Remove C object buildlist from -i test
The disparity in C objects built causes
the test to fail.
This removes those lines from the log
to inline log comparison.

Let's disable it in the relevant test.

Signed-off-by: Lukasz Mrugala <lukaszx.mrugala@intel.com>
2025-04-08 22:59:12 +02:00
Dmitrii Golovanov
067126abcd twister: harness: fix Ztest case summary pattern
Fix missing match on Ztest test case summary log entries, so even
if the test case 'END' log entry is missed or corrupted, its status
will be updated from the Ztest summary log entries, if present.

Signed-off-by: Dmitrii Golovanov <dmitrii.golovanov@intel.com>
2025-04-08 11:45:08 +02:00
Anas Nashif
19c6240b68 tests: twister: disable test_inline_logs temporarily
This test fails on older python versions (3.10) and only on CI.
Disabling it while we investigate. The test itself verifies inline logs
options, so the functionality test is not impacted.

Tracked in #87769

Signed-off-by: Anas Nashif <anas.nashif@intel.com>
2025-03-29 07:44:46 -04:00
Lukasz Mrugala
6ef4db88dd twister: Remove unrecognised sections test
The section lists that govern whether the ELF
section is recognised or not are out of date.

A programming bug has rendered the unrecognised
section check unrunnable for years.

Thus, the practically dead code of the
unrecognised section check is removed
in this commit.

Signed-off-by: Lukasz Mrugala <lukaszx.mrugala@intel.com>
2025-03-24 07:53:01 +01:00
Lukasz Fundakowski
d45396a6bb twister: Cleanup twister loggers
Removed logger.setLevel(...) from all twister modules, becasue
there should be only one place where twister loggers are configured,
and the log level should be set in that place.
This should be done in twister_main.py in setup_logging function.

Signed-off-by: Lukasz Fundakowski <lukasz.fundakowski@nordicsemi.no>
2025-03-19 09:08:00 -04:00
Lukasz Mrugala
ffe72a49bf scripts: Add invariant log closing
If BB tests failed, crashed, etc., log handles
could linger and crash test cleanup.
Adding log closing to a `finally` section
should prevent those issues.

Loggers should not be set up as globals,
as it makes testing much harder.
Running multiple Twisters may cause for the
logfiles to be still "in use".

When exiting main, close all logfiles
and remove their handlers from all loggers.
Do that for conftest as well.

Signed-off-by: Lukasz Mrugala <lukaszx.mrugala@intel.com>
2025-03-13 16:53:55 +00:00
Alberto Escolar Piedras
cb53e40ff9 drivers uart_native_posix: rename to native_pty and support N instances
Rename the driver from uart_native_posix to uart_native_pty.
Including renaming the DTS compatible, and kconfig options, deprecating
the old ones.

And refactor the driver, generalizing it, so we can have any number of
instances.
Note that, unfortunately generalizing to N instances cannot be done
without a degree of backwards compatibility breakage: This driver was
born with all its configuration and selection of the instances based on
kconfig.
When the driver was made to use DT, it was done in a way that required
both DT and kconfig needing to manually coherently enable the 2nd UART.
This has now been fixed, which it means only DT is used to decide how
many instances are avaliable, and UART_NATIVE_POSIX_PORT_1_ENABLE is
just ignored.

Including:
* Deprecate UART_NATIVE_WAIT_PTS_READY_ENABLE: the options is always on
  now as it has no practical drawbacks.
* Deprecate UART_NATIVE_POSIX_PORT_1_ENABLE: DTS intanciation defines it
  being available now.
* Rename a few functions and in general shorten pseudo-tty/pseudo-
  terminal to PTY instead of PTTY.

Signed-off-by: Alberto Escolar Piedras <alberto.escolar.piedras@nordicsemi.no>
2025-03-11 18:54:02 +01:00
Alberto Escolar Piedras
4338122248 drivers entropy: fake_entropy_native_posix rename to _native_sim
Rename the driver files, binding and kconfig options, while deprecating
the old binding and kconfig options.

Uses in tree are replaced.

Signed-off-by: Alberto Escolar Piedras <alberto.escolar.piedras@nordicsemi.no>
2025-03-08 03:38:13 +01:00
Anas Nashif
a7e2846a94 twister: setup logging per process
Setup logging per process to fix issue on both mac and windows where
handlers are not available to the processes.

Fixes #86237

Signed-off-by: Anas Nashif <anas.nashif@intel.com>
2025-02-28 14:53:38 +01:00
Benjamin Cabé
9ca11f3e9c twister: allowlist build_info.yml and zephyr.dts
These files can be useful to troubleshoot a test that's failing,
and they're really small, too.

Signed-off-by: Benjamin Cabé <benjamin@zephyrproject.org>
2025-02-20 15:08:50 +00:00
Grzegorz Swiderski
bb8b059e23 twister: Account for board & SoC extensions
Problem
-------

Board & SoC extensions are used to define out-of-tree board variants or
SoC qualifiers. When a board is extended, it has multiple directories
associated with it (each with its own `board.yml`), where twister should
be able to find additional platform files to support these qualifiers.
Currently, this doesn't work, because twister only traverses the primary
BOARD_DIR and ignores the rest.

The fix would've been trivial in the case of "legacy" platform files,
i.e. those of the form `<normalized_board_target>.yaml`, but it's less
straightforward for the newly introduced `twister.yaml` format.

A `twister.yaml` file contains platform configuration that can be shared
by multiple board targets and tweaked for specific targets by using the
top-level `variants` key. Normally, there is at most one `twister.yaml`
per board, but the file isn't necessarily unique to one board. Instead,
it's unique to one directory, which may define multiple boards (as is
the case with e.g. `boards/qemu/x86/`).

With extensions in the picture, the goal is to initialize platforms when
given multiple `twister.yaml` per board. The OOT files are expected to
only provide information about OOT board targets, without being able to
override in-tree targets (same principle as in the Zephyr build system).

Solution
--------

The `twister.yaml` handling is broken up into multiple passes - first
loading all the files, then splitting the `variants` keys apart from the
shared configuration, before constructing the Platform instances.

The purpose of the split is to treat the variant information as global,
instead of making unnecessary or faulty assumptions about locality.
Remember that the build system can derive board target names not only
from `board.yml`, but from `soc.yml` too. Considering that any board may
end up using an OOT-extended SoC (and hence multiple `soc.yml` files),
not every board target can be said to belong to some board dir.

Unlike the variant data, the remaining top-level config is still rooted
to the primary BOARD_DIR and inherited by the extension dirs from there.
This is quite intuitive in most imagined cases, but there is a caveat:
if a `twister.yaml` resides in an extension dir, then it is allowed to
have a top-level config of its own, but it will be silently ignored.
This is to support corner cases where, much like how a single board dir
can define multiple boards, a single board dir can also extend multiple
boards, or even do both. In those cases, the primary BOARD_DIR rule
should make it unambiguous which config belongs to which board, even if
it may seem counter-intuitive at first.

For concrete examples of what this means, please see the newly added
platform unit tests.

As part of these functional changes, a good chunk of logic is moved out
of `TestPlan.add_configurations()` into a new function in `platform.py`.
This is because recombining the top-level and variant configs requires
direct manipulation of the loaded YAML contents, which would be improper
to do outside of the module responsible for encapsulating this data.

Signed-off-by: Grzegorz Swiderski <grzegorz.swiderski@nordicsemi.no>
2025-02-14 21:01:33 +01:00
Grzegorz Chwierut
33b202e295 twister: blackbox: fix test_report_summary after extending reason
Updated test_report_summary to match new string with detailed
reason of build failure.

Signed-off-by: Grzegorz Chwierut <grzegorz.chwierut@nordicsemi.no>
2025-02-13 16:42:23 +01:00
Anas Nashif
d80e3f7687 twister: harness: introduce shell harness
Introduce a new harness based on pytest that does basic shell command
handling. The harness is enabeld using:

harness: shell

and expects a file with parameters in the form:

test_shell_harness:
- command: "kernel version"
  expected: "Zephyr version .*"
- ...

Multiple commands and their expected output can be tested.

Signed-off-by: Anas Nashif <anas.nashif@intel.com>
2025-02-08 08:13:46 +01:00
Dmitrii Golovanov
f93f82f160 twister: coverage: Data collection and reporting per-test instance
With this change, the coverage data (GCOV dump) is extracted from the
test log files after each of the test instance parallel execution,
instead of post processing all the logs at the end of the Twister run.

The new `--coverage-per-instance` mode extends Twister coverage operations
to report coverage statistics on each test instance execution individually
in addition to the default reporting mode which aggregates data to one
report with all the test instances in the current scope of the Twister run.
The split mode allows to identify precisely what amount of code coverage
each test suite provides and to analyze its contribution to the overall
test plan's coverage. Each test configuration's output directory will have
its own coverage report and data files, so the overall disk space and
the total execution time increase.

Another new `--disable-coverage-aggregation` option allows to execute
only the `--coverage-per-instance` mode when the aggregate coverage
report for the whole Twister run scope is not needed.

Signed-off-by: Dmitrii Golovanov <dmitrii.golovanov@intel.com>
2025-01-30 18:29:08 +01:00
Stephanos Ioannidis
0e6f6cfb12 twister: Remove 'xtools' toolchain variant references
This commit removes all references to the `xtools` toolchain variant in the
twister scripts.

Note that the `xtools` toolchain variant has been deprecated since Zephyr
v3.3.0 and now removed.

Signed-off-by: Stephanos Ioannidis <root@stephanos.io>
2025-01-17 10:50:07 +01:00
Dmitrii Golovanov
b4d22420e9 twister: harness: recording: Allow multiple patterns
Extend Twister Harness 'recording' feature to allow multiple
regular expression patterns to extract different types of records
from test output.

Add 'merge' recording mode to collect all extracted data fields
into a single record object of the test instance.

Export to CSV file now takes all field names occurred in the collected
records, sort it alphabetically, and then use it for columns instead of
using only the first record's fields. This is done to address possible
situation when records have different set of fields.

Adjust Twister documentation and test suite to the above changes.

Signed-off-by: Dmitrii Golovanov <dmitrii.golovanov@intel.com>
2025-01-16 22:38:51 +01:00
Anas Nashif
11e656bb6a twister: support testing multiple toolchain variants
Added integration_toolchains to allow building/testing with multiple
toolchains available in the environment.

This changes the output structure and adds another level in the path
under twister_out signifying the toolchain used. The toolchain used
(variant) is also part of the json output now.

Signed-off-by: Anas Nashif <anas.nashif@intel.com>
2025-01-08 12:58:59 +01:00
Dmitrii Golovanov
05c167e0b8 twister: ztest: Optimize logging and comment special cases
Add more comments and optimize logging on Ztest tracing.

Signed-off-by: Dmitrii Golovanov <dmitrii.golovanov@intel.com>
2025-01-08 06:07:35 +01:00
Dmitrii Golovanov
26e0aa268f twister: handlers: Pass harness reason to instance
Pass a test failure reason text from harness to the instance
to convey more details on the failure.

Signed-off-by: Dmitrii Golovanov <dmitrii.golovanov@intel.com>
2024-12-17 11:37:40 +00:00
Dmitrii Golovanov
ea3f105f82 twister: binaryhandler: Fix None status on test fail
Set missing TestCase statuses when a test under the BinaryHandler failed
(crashed), so remaining 'STARTED' and 'NONE' are now 'BLOCK' instead.

Signed-off-by: Dmitrii Golovanov <dmitrii.golovanov@intel.com>
2024-12-17 11:37:40 +00:00
Dmitrii Golovanov
b69a8d1deb twister: ztest: short test case names on --no-detailed-test-id
Extend `--no-detailed-test-id` command line option: in addition to its
current behavior to exclude from a test Suite name its configuration path
prefix, also don't prefix each Ztest Case name with its Scenario name.

For example: 'kernel.common.timing' Scenario name, the same Suite name,
and 'sleep.usleep' test Case (where 'sleep' is its Ztest suite name
and 'usleep' is Ztest test name.

This way both TestSuite and TestCase names follow the same principle
having no parent object name prefix.

There is no information loss in Twister reports with this naming:
TestSuite is a container object for its TestCases, whereas TestSuite
has its configuration path as a property.

Signed-off-by: Dmitrii Golovanov <dmitrii.golovanov@intel.com>
2024-12-16 08:35:34 +01:00
Anas Nashif
82f7857a94 twister: debug: change skipped -> filtered
Change debug message to say filtered instead of skipped.

Signed-off-by: Anas Nashif <anas.nashif@intel.com>
2024-12-13 09:12:59 +01:00
Grzegorz Chwierut
c4d884cc65 twister: Pass extra test args to pytest
Pass additional args to test binary. Twister
passes it only for native_sim.
Fixes #82463

Signed-off-by: Grzegorz Chwierut <grzegorz.chwierut@nordicsemi.no>
2024-12-12 10:20:55 -05:00
Gerard Marull-Paretas
714e193340 scripts: twister: drop support for space-separated lists
Support for space-separated lists was deprecated a long time ago
a91620f5bb so it is time to remove
support for them.

Any project that has not migrated can still use
`scripts/utils/twister_to_list.py` to automatically migrate twister
files.

Signed-off-by: Gerard Marull-Paretas <gerard@teslabs.com>
2024-12-04 14:14:53 -05:00
Hake Huang
afba775304 twister: test: update test case
update test cases for twister
1. test_errors.py add protection.
2. test_handlers.py change call to status
3. test_testsuite.py change call to status

The log traces for TwisterException objects only.
And the stack trace output follow the same rules for all exceptions,
but StatusAttributeError with its dedicated handlers.

Signed-off-by: Hake Huang <hake.huang@oss.nxp.com>
2024-12-04 02:03:33 +01:00
Anas Nashif
440659878d twister: do not create overlays/run_id for filtered cases
Avoid creating directories and files for instances that are filtered.

Signed-off-by: Anas Nashif <anas.nashif@intel.com>
2024-12-04 02:02:52 +01:00
Lukasz Mrugala
22a875e490 scripts: Fix twisterlib for ruff - SIM115
This fixes ruff linting error SIM115,
where files were opened without the use of
a context handler.

Signed-off-by: Lukasz Mrugala <lukaszx.mrugala@intel.com>
2024-11-29 15:29:31 +01:00
Lukasz Mrugala
0c6fd60407 scripts: Fix twisterlib for ruff - UP015
This fixes ruff linting error UP015,
where open() has specified unnecessary modes.

Signed-off-by: Lukasz Mrugala <lukaszx.mrugala@intel.com>
2024-11-29 15:29:31 +01:00
Lukasz Mrugala
9b128a24e2 scripts: Fix twisterlib for ruff - B028
This fixes ruff linting error B028,
where warnings.warn() lacked explicit
stacklevel parameter.

Signed-off-by: Lukasz Mrugala <lukaszx.mrugala@intel.com>
2024-11-29 15:29:31 +01:00
Dmitrii Golovanov
113875d53c twister: testplan: Detect duplicates on --no-detailed-test-id
Detect duplicate TestSuites on load, and raise error when it happens
with `--no-detailed-test-id` option which shortens TestSuite name
excluding the test project path prefix, thus increasing chances
for duplicates. Without this check, only the last duplicated
test configuration was selected while others silently ignored.

Signed-off-by: Dmitrii Golovanov <dmitrii.golovanov@intel.com>
2024-11-27 08:19:39 +01:00
Dmitrii Golovanov
89466f9d7e twister: testplan: Fix duplicate scenarios error reporting
Fix duplicate test scenario error reporting to show paths to all
twister.yaml configuration files where these duplicates were found.

Signed-off-by: Dmitrii Golovanov <dmitrii.golovanov@intel.com>
2024-11-27 08:19:39 +01:00
Pieter De Gendt
bf2db7afc0 python: Format and sort imports
ruff check --select I001 --fix applied to all python files that had
this as only issue.

Signed-off-by: Pieter De Gendt <pieter.degendt@basalte.be>
2024-11-25 10:07:13 +01:00
Wilfried Chauveau
7cc70f01f1 scripts: twisterlib: Enable multiple simulator support in twister
This change introduces the ability in twister to select which
emulation/simulation tool to use on the command line.

If none is specified, it will select the first in the list.

Signed-off-by: Wilfried Chauveau <wilfried.chauveau@arm.com>
2024-11-25 08:31:28 +01:00
Dmitrii Golovanov
e11aecaed5 twister: fix Ztest C++ test names extraction from ELF
Fix Ztest test function name extraction from ELF symbols
for C++ compiled binaries where symbol names need additional
'demangling' to match with corresponding test names.

The `c++filt` utility (part of binutils) is called for
demangling when it is needed.

Twister test suite extension and adjustment.

Signed-off-by: Dmitrii Golovanov <dmitrii.golovanov@intel.com>
2024-11-22 08:26:59 -05:00
Dmitrii Golovanov
a72244f2d0 twister: ztest: harness: Fix missed TestCase statuses
Fix a problem of Ztest suite names not taken into account by Twister
to identify a TestCase, so in some situations a Ztest test's status
was not assigned to the proper TestCase and it remains 'None'
whereas the actual status value lost, eventually the resulting total
execution counters not correct.

The issue was observed in these situations:
 * Ztest application with multiple test suites having same test names.
 * Ztest suite is 'skipped' entirely on execution with all its tests.

The proposed solution extends Twister test case name for Ztest to
include Ztest suite name, so the resulting identifier looks like:
   `<test_scenario_name>.<ztest_suite_name>.<ztest_name>`

The above naming scheme now requires ztest_suite_name part to be
provided for `--sub-test` command line option.

Testcase identifiers in twister.json and testplan.json will also
include ztest_suite_name component.

The Twister Ztest(Test) Harness is improved to track all state changes
known from the test application's log for Ztest suites and test cases,
so now it parses log output from a Ztest application more scurpulously.
Regular expressions to match log records are extended and optimized
to compile them only once and, in some cases, fixed (suite summary).

Signed-off-by: Dmitrii Golovanov <dmitrii.golovanov@intel.com>
2024-11-22 08:26:59 -05:00
Pieter De Gendt
f05deb1aa4 python: Format trivial files where only newlines were missing
Apply formatting on files that only needed adding newlines.

Signed-off-by: Pieter De Gendt <pieter.degendt@basalte.be>
2024-11-21 20:10:51 +01:00
Grzegorz Chwierut
e0bd7e7c87 twister: Allow sharing hardware platform between variants
Extended hardware map to share a single board between variants.
To run tests for different variants on the same board
without re-configuring the hardware map file for each variant,
one can use a `platform` atribute as a list of names.

Signed-off-by: Grzegorz Chwierut <grzegorz.chwierut@nordicsemi.no>
2024-11-21 14:48:03 +01:00
Anas Nashif
cfc232fa32 twister: stats: fix platform stats
Fix wrong reporting about where testcases were executed.

We have:

INFO    - 1130 of 1130 executed test cases passed (100.00%) on 0 out of
total 860 platforms (0.00%).

which is obviously wrong.

Signed-off-by: Anas Nashif <anas.nashif@intel.com>
2024-11-16 13:29:23 -05:00
Anas Nashif
049b243b51 twister: stats: use anytree to create summary
Do not create the tree structure manually, use anytree instead.

Signed-off-by: Anas Nashif <anas.nashif@intel.com>
2024-11-12 18:05:16 -06:00
Anas Nashif
a43a67e3a3 twister: stats: fix suite statistics
suite stats were not correct, a mixup between skipped and filtered
suites was leading to inconsistent numbers. This is now fixed.

Signed-off-by: Anas Nashif <anas.nashif@intel.com>
2024-11-12 18:05:16 -06:00
Anas Nashif
b578fa5b2a twister: stats: more renames skipped -> filtered
Additional variable renames leading to confusion between skipped and
filtered.

Signed-off-by: Anas Nashif <anas.nashif@intel.com>
2024-11-12 18:05:16 -06:00
Anas Nashif
d69d14aac6 twister: stats: skipped_configs -> filtered_configs
Rename variables leading to confusion between skipped and filtered
suites.

Signed-off-by: Anas Nashif <anas.nashif@intel.com>
2024-11-12 18:05:16 -06:00
Anas Nashif
61aefee3ad twister: stats: skipped testcases are part of selected group
Count skipped cases as part of selected group in stats.

Signed-off-by: Anas Nashif <anas.nashif@intel.com>
2024-11-12 18:05:16 -06:00
Anas Nashif
eddb0b77a3 twisters: tests: adapt twister testsuite
Adapt tests for new changes in twister.

Signed-off-by: Anas Nashif <anas.nashif@intel.com>
2024-11-12 18:05:16 -06:00
Dmitrii Golovanov
7e1cd18be4 twister: Fix NOTRUN status
Fix Twister TestCase statuses left not assigned ('NONE') in these 'NOTRUN'
situations:

 * a test suite has `integration platform` which is not available to run.

 * `--cmake-only` execution.

Signed-off-by: Dmitrii Golovanov <dmitrii.golovanov@intel.com>
2024-11-06 09:52:25 -08:00
Lukasz Mrugala
9dc0af55e2 scripts: twister: Fix NOTRUN in test_only
When using the --build-only into --test-only
Twister setup, NOTRUN statuses were not properly rerun.

Now they are properly run again if runnable.

Signed-off-by: Lukasz Mrugala <lukaszx.mrugala@intel.com>
2024-10-31 14:18:17 -05:00