Statuses of quarantined test cases were not updated properly.
Quarantined tests shouldn't cause errors in integration mode.
Removed obsolate filter type.
Signed-off-by: Maciej Perkowski <maciej.perkowski@nordicsemi.no>
Skipped status fits quarantined items better than filtered.
Filtered tests are by default removed from reports, which
shouldn't be the case for quarantined tests.
Adjust tests unit and blackbox tests accordingly.
Signed-off-by: Maciej Perkowski <maciej.perkowski@nordicsemi.no>
Commit 6dc27a4 ("boards: align vendor entry in board.yml files")
unknowingly broke Twister blackbox tests by correctly placing Atom
targets under the 'intel' vendor. Since this was not previously the
case, the Twister blackbox vendor filter test for 'intel' was actually
expecting 'qemu_x86/atom' to be filtered out as "Not a selected vendor
platform."
This commit updates test_filter.py with the new vendor flags.
Signed-off-by: Luca Burelli <l.burelli@arduino.cc>
The disparity in C objects built causes
the test to fail.
This removes those lines from the log
to inline log comparison.
Let's disable it in the relevant test.
Signed-off-by: Lukasz Mrugala <lukaszx.mrugala@intel.com>
Fix missing match on Ztest test case summary log entries, so even
if the test case 'END' log entry is missed or corrupted, its status
will be updated from the Ztest summary log entries, if present.
Signed-off-by: Dmitrii Golovanov <dmitrii.golovanov@intel.com>
This test fails on older python versions (3.10) and only on CI.
Disabling it while we investigate. The test itself verifies inline logs
options, so the functionality test is not impacted.
Tracked in #87769
Signed-off-by: Anas Nashif <anas.nashif@intel.com>
The section lists that govern whether the ELF
section is recognised or not are out of date.
A programming bug has rendered the unrecognised
section check unrunnable for years.
Thus, the practically dead code of the
unrecognised section check is removed
in this commit.
Signed-off-by: Lukasz Mrugala <lukaszx.mrugala@intel.com>
Removed logger.setLevel(...) from all twister modules, becasue
there should be only one place where twister loggers are configured,
and the log level should be set in that place.
This should be done in twister_main.py in setup_logging function.
Signed-off-by: Lukasz Fundakowski <lukasz.fundakowski@nordicsemi.no>
If BB tests failed, crashed, etc., log handles
could linger and crash test cleanup.
Adding log closing to a `finally` section
should prevent those issues.
Loggers should not be set up as globals,
as it makes testing much harder.
Running multiple Twisters may cause for the
logfiles to be still "in use".
When exiting main, close all logfiles
and remove their handlers from all loggers.
Do that for conftest as well.
Signed-off-by: Lukasz Mrugala <lukaszx.mrugala@intel.com>
Rename the driver from uart_native_posix to uart_native_pty.
Including renaming the DTS compatible, and kconfig options, deprecating
the old ones.
And refactor the driver, generalizing it, so we can have any number of
instances.
Note that, unfortunately generalizing to N instances cannot be done
without a degree of backwards compatibility breakage: This driver was
born with all its configuration and selection of the instances based on
kconfig.
When the driver was made to use DT, it was done in a way that required
both DT and kconfig needing to manually coherently enable the 2nd UART.
This has now been fixed, which it means only DT is used to decide how
many instances are avaliable, and UART_NATIVE_POSIX_PORT_1_ENABLE is
just ignored.
Including:
* Deprecate UART_NATIVE_WAIT_PTS_READY_ENABLE: the options is always on
now as it has no practical drawbacks.
* Deprecate UART_NATIVE_POSIX_PORT_1_ENABLE: DTS intanciation defines it
being available now.
* Rename a few functions and in general shorten pseudo-tty/pseudo-
terminal to PTY instead of PTTY.
Signed-off-by: Alberto Escolar Piedras <alberto.escolar.piedras@nordicsemi.no>
Rename the driver files, binding and kconfig options, while deprecating
the old binding and kconfig options.
Uses in tree are replaced.
Signed-off-by: Alberto Escolar Piedras <alberto.escolar.piedras@nordicsemi.no>
Setup logging per process to fix issue on both mac and windows where
handlers are not available to the processes.
Fixes#86237
Signed-off-by: Anas Nashif <anas.nashif@intel.com>
These files can be useful to troubleshoot a test that's failing,
and they're really small, too.
Signed-off-by: Benjamin Cabé <benjamin@zephyrproject.org>
Problem
-------
Board & SoC extensions are used to define out-of-tree board variants or
SoC qualifiers. When a board is extended, it has multiple directories
associated with it (each with its own `board.yml`), where twister should
be able to find additional platform files to support these qualifiers.
Currently, this doesn't work, because twister only traverses the primary
BOARD_DIR and ignores the rest.
The fix would've been trivial in the case of "legacy" platform files,
i.e. those of the form `<normalized_board_target>.yaml`, but it's less
straightforward for the newly introduced `twister.yaml` format.
A `twister.yaml` file contains platform configuration that can be shared
by multiple board targets and tweaked for specific targets by using the
top-level `variants` key. Normally, there is at most one `twister.yaml`
per board, but the file isn't necessarily unique to one board. Instead,
it's unique to one directory, which may define multiple boards (as is
the case with e.g. `boards/qemu/x86/`).
With extensions in the picture, the goal is to initialize platforms when
given multiple `twister.yaml` per board. The OOT files are expected to
only provide information about OOT board targets, without being able to
override in-tree targets (same principle as in the Zephyr build system).
Solution
--------
The `twister.yaml` handling is broken up into multiple passes - first
loading all the files, then splitting the `variants` keys apart from the
shared configuration, before constructing the Platform instances.
The purpose of the split is to treat the variant information as global,
instead of making unnecessary or faulty assumptions about locality.
Remember that the build system can derive board target names not only
from `board.yml`, but from `soc.yml` too. Considering that any board may
end up using an OOT-extended SoC (and hence multiple `soc.yml` files),
not every board target can be said to belong to some board dir.
Unlike the variant data, the remaining top-level config is still rooted
to the primary BOARD_DIR and inherited by the extension dirs from there.
This is quite intuitive in most imagined cases, but there is a caveat:
if a `twister.yaml` resides in an extension dir, then it is allowed to
have a top-level config of its own, but it will be silently ignored.
This is to support corner cases where, much like how a single board dir
can define multiple boards, a single board dir can also extend multiple
boards, or even do both. In those cases, the primary BOARD_DIR rule
should make it unambiguous which config belongs to which board, even if
it may seem counter-intuitive at first.
For concrete examples of what this means, please see the newly added
platform unit tests.
As part of these functional changes, a good chunk of logic is moved out
of `TestPlan.add_configurations()` into a new function in `platform.py`.
This is because recombining the top-level and variant configs requires
direct manipulation of the loaded YAML contents, which would be improper
to do outside of the module responsible for encapsulating this data.
Signed-off-by: Grzegorz Swiderski <grzegorz.swiderski@nordicsemi.no>
Updated test_report_summary to match new string with detailed
reason of build failure.
Signed-off-by: Grzegorz Chwierut <grzegorz.chwierut@nordicsemi.no>
Introduce a new harness based on pytest that does basic shell command
handling. The harness is enabeld using:
harness: shell
and expects a file with parameters in the form:
test_shell_harness:
- command: "kernel version"
expected: "Zephyr version .*"
- ...
Multiple commands and their expected output can be tested.
Signed-off-by: Anas Nashif <anas.nashif@intel.com>
With this change, the coverage data (GCOV dump) is extracted from the
test log files after each of the test instance parallel execution,
instead of post processing all the logs at the end of the Twister run.
The new `--coverage-per-instance` mode extends Twister coverage operations
to report coverage statistics on each test instance execution individually
in addition to the default reporting mode which aggregates data to one
report with all the test instances in the current scope of the Twister run.
The split mode allows to identify precisely what amount of code coverage
each test suite provides and to analyze its contribution to the overall
test plan's coverage. Each test configuration's output directory will have
its own coverage report and data files, so the overall disk space and
the total execution time increase.
Another new `--disable-coverage-aggregation` option allows to execute
only the `--coverage-per-instance` mode when the aggregate coverage
report for the whole Twister run scope is not needed.
Signed-off-by: Dmitrii Golovanov <dmitrii.golovanov@intel.com>
This commit removes all references to the `xtools` toolchain variant in the
twister scripts.
Note that the `xtools` toolchain variant has been deprecated since Zephyr
v3.3.0 and now removed.
Signed-off-by: Stephanos Ioannidis <root@stephanos.io>
Extend Twister Harness 'recording' feature to allow multiple
regular expression patterns to extract different types of records
from test output.
Add 'merge' recording mode to collect all extracted data fields
into a single record object of the test instance.
Export to CSV file now takes all field names occurred in the collected
records, sort it alphabetically, and then use it for columns instead of
using only the first record's fields. This is done to address possible
situation when records have different set of fields.
Adjust Twister documentation and test suite to the above changes.
Signed-off-by: Dmitrii Golovanov <dmitrii.golovanov@intel.com>
Added integration_toolchains to allow building/testing with multiple
toolchains available in the environment.
This changes the output structure and adds another level in the path
under twister_out signifying the toolchain used. The toolchain used
(variant) is also part of the json output now.
Signed-off-by: Anas Nashif <anas.nashif@intel.com>
Pass a test failure reason text from harness to the instance
to convey more details on the failure.
Signed-off-by: Dmitrii Golovanov <dmitrii.golovanov@intel.com>
Set missing TestCase statuses when a test under the BinaryHandler failed
(crashed), so remaining 'STARTED' and 'NONE' are now 'BLOCK' instead.
Signed-off-by: Dmitrii Golovanov <dmitrii.golovanov@intel.com>
Extend `--no-detailed-test-id` command line option: in addition to its
current behavior to exclude from a test Suite name its configuration path
prefix, also don't prefix each Ztest Case name with its Scenario name.
For example: 'kernel.common.timing' Scenario name, the same Suite name,
and 'sleep.usleep' test Case (where 'sleep' is its Ztest suite name
and 'usleep' is Ztest test name.
This way both TestSuite and TestCase names follow the same principle
having no parent object name prefix.
There is no information loss in Twister reports with this naming:
TestSuite is a container object for its TestCases, whereas TestSuite
has its configuration path as a property.
Signed-off-by: Dmitrii Golovanov <dmitrii.golovanov@intel.com>
Pass additional args to test binary. Twister
passes it only for native_sim.
Fixes#82463
Signed-off-by: Grzegorz Chwierut <grzegorz.chwierut@nordicsemi.no>
Support for space-separated lists was deprecated a long time ago
a91620f5bb so it is time to remove
support for them.
Any project that has not migrated can still use
`scripts/utils/twister_to_list.py` to automatically migrate twister
files.
Signed-off-by: Gerard Marull-Paretas <gerard@teslabs.com>
update test cases for twister
1. test_errors.py add protection.
2. test_handlers.py change call to status
3. test_testsuite.py change call to status
The log traces for TwisterException objects only.
And the stack trace output follow the same rules for all exceptions,
but StatusAttributeError with its dedicated handlers.
Signed-off-by: Hake Huang <hake.huang@oss.nxp.com>
This fixes ruff linting error SIM115,
where files were opened without the use of
a context handler.
Signed-off-by: Lukasz Mrugala <lukaszx.mrugala@intel.com>
Detect duplicate TestSuites on load, and raise error when it happens
with `--no-detailed-test-id` option which shortens TestSuite name
excluding the test project path prefix, thus increasing chances
for duplicates. Without this check, only the last duplicated
test configuration was selected while others silently ignored.
Signed-off-by: Dmitrii Golovanov <dmitrii.golovanov@intel.com>
Fix duplicate test scenario error reporting to show paths to all
twister.yaml configuration files where these duplicates were found.
Signed-off-by: Dmitrii Golovanov <dmitrii.golovanov@intel.com>
This change introduces the ability in twister to select which
emulation/simulation tool to use on the command line.
If none is specified, it will select the first in the list.
Signed-off-by: Wilfried Chauveau <wilfried.chauveau@arm.com>
Fix Ztest test function name extraction from ELF symbols
for C++ compiled binaries where symbol names need additional
'demangling' to match with corresponding test names.
The `c++filt` utility (part of binutils) is called for
demangling when it is needed.
Twister test suite extension and adjustment.
Signed-off-by: Dmitrii Golovanov <dmitrii.golovanov@intel.com>
Fix a problem of Ztest suite names not taken into account by Twister
to identify a TestCase, so in some situations a Ztest test's status
was not assigned to the proper TestCase and it remains 'None'
whereas the actual status value lost, eventually the resulting total
execution counters not correct.
The issue was observed in these situations:
* Ztest application with multiple test suites having same test names.
* Ztest suite is 'skipped' entirely on execution with all its tests.
The proposed solution extends Twister test case name for Ztest to
include Ztest suite name, so the resulting identifier looks like:
`<test_scenario_name>.<ztest_suite_name>.<ztest_name>`
The above naming scheme now requires ztest_suite_name part to be
provided for `--sub-test` command line option.
Testcase identifiers in twister.json and testplan.json will also
include ztest_suite_name component.
The Twister Ztest(Test) Harness is improved to track all state changes
known from the test application's log for Ztest suites and test cases,
so now it parses log output from a Ztest application more scurpulously.
Regular expressions to match log records are extended and optimized
to compile them only once and, in some cases, fixed (suite summary).
Signed-off-by: Dmitrii Golovanov <dmitrii.golovanov@intel.com>
Extended hardware map to share a single board between variants.
To run tests for different variants on the same board
without re-configuring the hardware map file for each variant,
one can use a `platform` atribute as a list of names.
Signed-off-by: Grzegorz Chwierut <grzegorz.chwierut@nordicsemi.no>
Fix wrong reporting about where testcases were executed.
We have:
INFO - 1130 of 1130 executed test cases passed (100.00%) on 0 out of
total 860 platforms (0.00%).
which is obviously wrong.
Signed-off-by: Anas Nashif <anas.nashif@intel.com>
suite stats were not correct, a mixup between skipped and filtered
suites was leading to inconsistent numbers. This is now fixed.
Signed-off-by: Anas Nashif <anas.nashif@intel.com>
Fix Twister TestCase statuses left not assigned ('NONE') in these 'NOTRUN'
situations:
* a test suite has `integration platform` which is not available to run.
* `--cmake-only` execution.
Signed-off-by: Dmitrii Golovanov <dmitrii.golovanov@intel.com>
When using the --build-only into --test-only
Twister setup, NOTRUN statuses were not properly rerun.
Now they are properly run again if runnable.
Signed-off-by: Lukasz Mrugala <lukaszx.mrugala@intel.com>
