Use csv instead of a plain split to split extra arguments for extra west
flags, this allows correct passing of arguments that are internally
escaped in quotes, such as arugments containing commas.
Signed-off-by: Fabio Baltieri <fabiobaltieri@google.com>
Refactored twister_main.py module to simplify the code and system tests.
Removed the need to patch `sys.argv` in blackbox tests.
Signed-off-by: Lukasz Fundakowski <lukasz.fundakowski@nordicsemi.no>
Currently twister requires west to flash a board, either directly using
west-flash or indirectly through the cmake flash target.
Add an option to specify a custom flash script, this is passed a
build-dir and board-id flags so it can be used to implement custom
flashing scripts in a system that does not use west.
Signed-off-by: Fabio Baltieri <fabiobaltieri@google.com>
Multiple handlers were setting status/reason in slightly different ways,
but the code was mostly trying to do the same. Put this code in the
handler class and implement in one place.
This fixes an issue where in some cases qemu failures were not handled
correctly and status of cases was not updated, leading to multiple
failures.
Signed-off-by: Anas Nashif <anas.nashif@intel.com>
Honor the --flash-before option when using a serial-pty script with
--device-serial-pty <script> and start the script only after
flashing the device.
Signed-off-by: Tim Pambor <tim.pambor@codewrights.de>
Drop the logic to use the generator "flash" target, that is using west
anyway, always call flash directly anyway so that those targets can be
deprecated.
Signed-off-by: Fabio Baltieri <fabiobaltieri@google.com>
Use --test-pattern to filter scenarios using regular expressions, for
example:
./scripts/twister --test-pattern '^kernel.semaphore.*' -v
will run all those test using this identifier pattern, and nothing else.
Multiple patterns are possible.
Signed-off-by: Anas Nashif <anas.nashif@intel.com>
Make this option default and reduce amount of details displayed by
twister.
This options has been enabled in CI for a while now with the intention
of making it default.
Signed-off-by: Anas Nashif <anas.nashif@intel.com>
mock has been available in Python standard lib since Python 3.3, remove
the unnecessary dependency to `mock` pip package.
Signed-off-by: Benjamin Cabé <benjamin@zephyrproject.org>
Somehow this file had no exclude rules in .ruff-excludes.toml. Make it
compliant going forward.
Signed-off-by: Benjamin Cabé <benjamin@zephyrproject.org>
If test configuration is not quarantined and `--quarantine-verify`
mode is enabled, these should be filtered and not attached
to reports.
Proposed change applies this approach.
Signed-off-by: Katarzyna Giądła <katarzyna.giadla@nordicsemi.no>
If test plan is used, memory usage metrics are overridden.
Result of this operation is missing memory footprint.
This change removes redundand metrics update and fixes
described inconvenience.
Signed-off-by: Katarzyna Giądła <katarzyna.giadla@nordicsemi.no>
Statuses of quarantined test cases were not updated properly.
Quarantined tests shouldn't cause errors in integration mode.
Removed obsolate filter type.
Signed-off-by: Maciej Perkowski <maciej.perkowski@nordicsemi.no>
Skipped status fits quarantined items better than filtered.
Filtered tests are by default removed from reports, which
shouldn't be the case for quarantined tests.
Adjust tests unit and blackbox tests accordingly.
Signed-off-by: Maciej Perkowski <maciej.perkowski@nordicsemi.no>
Commit 6dc27a4 ("boards: align vendor entry in board.yml files")
unknowingly broke Twister blackbox tests by correctly placing Atom
targets under the 'intel' vendor. Since this was not previously the
case, the Twister blackbox vendor filter test for 'intel' was actually
expecting 'qemu_x86/atom' to be filtered out as "Not a selected vendor
platform."
This commit updates test_filter.py with the new vendor flags.
Signed-off-by: Luca Burelli <l.burelli@arduino.cc>
The disparity in C objects built causes
the test to fail.
This removes those lines from the log
to inline log comparison.
Let's disable it in the relevant test.
Signed-off-by: Lukasz Mrugala <lukaszx.mrugala@intel.com>
Fix missing match on Ztest test case summary log entries, so even
if the test case 'END' log entry is missed or corrupted, its status
will be updated from the Ztest summary log entries, if present.
Signed-off-by: Dmitrii Golovanov <dmitrii.golovanov@intel.com>
This test fails on older python versions (3.10) and only on CI.
Disabling it while we investigate. The test itself verifies inline logs
options, so the functionality test is not impacted.
Tracked in #87769
Signed-off-by: Anas Nashif <anas.nashif@intel.com>
The section lists that govern whether the ELF
section is recognised or not are out of date.
A programming bug has rendered the unrecognised
section check unrunnable for years.
Thus, the practically dead code of the
unrecognised section check is removed
in this commit.
Signed-off-by: Lukasz Mrugala <lukaszx.mrugala@intel.com>
Removed logger.setLevel(...) from all twister modules, becasue
there should be only one place where twister loggers are configured,
and the log level should be set in that place.
This should be done in twister_main.py in setup_logging function.
Signed-off-by: Lukasz Fundakowski <lukasz.fundakowski@nordicsemi.no>
If BB tests failed, crashed, etc., log handles
could linger and crash test cleanup.
Adding log closing to a `finally` section
should prevent those issues.
Loggers should not be set up as globals,
as it makes testing much harder.
Running multiple Twisters may cause for the
logfiles to be still "in use".
When exiting main, close all logfiles
and remove their handlers from all loggers.
Do that for conftest as well.
Signed-off-by: Lukasz Mrugala <lukaszx.mrugala@intel.com>
Rename the driver from uart_native_posix to uart_native_pty.
Including renaming the DTS compatible, and kconfig options, deprecating
the old ones.
And refactor the driver, generalizing it, so we can have any number of
instances.
Note that, unfortunately generalizing to N instances cannot be done
without a degree of backwards compatibility breakage: This driver was
born with all its configuration and selection of the instances based on
kconfig.
When the driver was made to use DT, it was done in a way that required
both DT and kconfig needing to manually coherently enable the 2nd UART.
This has now been fixed, which it means only DT is used to decide how
many instances are avaliable, and UART_NATIVE_POSIX_PORT_1_ENABLE is
just ignored.
Including:
* Deprecate UART_NATIVE_WAIT_PTS_READY_ENABLE: the options is always on
now as it has no practical drawbacks.
* Deprecate UART_NATIVE_POSIX_PORT_1_ENABLE: DTS intanciation defines it
being available now.
* Rename a few functions and in general shorten pseudo-tty/pseudo-
terminal to PTY instead of PTTY.
Signed-off-by: Alberto Escolar Piedras <alberto.escolar.piedras@nordicsemi.no>
Rename the driver files, binding and kconfig options, while deprecating
the old binding and kconfig options.
Uses in tree are replaced.
Signed-off-by: Alberto Escolar Piedras <alberto.escolar.piedras@nordicsemi.no>
Setup logging per process to fix issue on both mac and windows where
handlers are not available to the processes.
Fixes#86237
Signed-off-by: Anas Nashif <anas.nashif@intel.com>
These files can be useful to troubleshoot a test that's failing,
and they're really small, too.
Signed-off-by: Benjamin Cabé <benjamin@zephyrproject.org>
Problem
-------
Board & SoC extensions are used to define out-of-tree board variants or
SoC qualifiers. When a board is extended, it has multiple directories
associated with it (each with its own `board.yml`), where twister should
be able to find additional platform files to support these qualifiers.
Currently, this doesn't work, because twister only traverses the primary
BOARD_DIR and ignores the rest.
The fix would've been trivial in the case of "legacy" platform files,
i.e. those of the form `<normalized_board_target>.yaml`, but it's less
straightforward for the newly introduced `twister.yaml` format.
A `twister.yaml` file contains platform configuration that can be shared
by multiple board targets and tweaked for specific targets by using the
top-level `variants` key. Normally, there is at most one `twister.yaml`
per board, but the file isn't necessarily unique to one board. Instead,
it's unique to one directory, which may define multiple boards (as is
the case with e.g. `boards/qemu/x86/`).
With extensions in the picture, the goal is to initialize platforms when
given multiple `twister.yaml` per board. The OOT files are expected to
only provide information about OOT board targets, without being able to
override in-tree targets (same principle as in the Zephyr build system).
Solution
--------
The `twister.yaml` handling is broken up into multiple passes - first
loading all the files, then splitting the `variants` keys apart from the
shared configuration, before constructing the Platform instances.
The purpose of the split is to treat the variant information as global,
instead of making unnecessary or faulty assumptions about locality.
Remember that the build system can derive board target names not only
from `board.yml`, but from `soc.yml` too. Considering that any board may
end up using an OOT-extended SoC (and hence multiple `soc.yml` files),
not every board target can be said to belong to some board dir.
Unlike the variant data, the remaining top-level config is still rooted
to the primary BOARD_DIR and inherited by the extension dirs from there.
This is quite intuitive in most imagined cases, but there is a caveat:
if a `twister.yaml` resides in an extension dir, then it is allowed to
have a top-level config of its own, but it will be silently ignored.
This is to support corner cases where, much like how a single board dir
can define multiple boards, a single board dir can also extend multiple
boards, or even do both. In those cases, the primary BOARD_DIR rule
should make it unambiguous which config belongs to which board, even if
it may seem counter-intuitive at first.
For concrete examples of what this means, please see the newly added
platform unit tests.
As part of these functional changes, a good chunk of logic is moved out
of `TestPlan.add_configurations()` into a new function in `platform.py`.
This is because recombining the top-level and variant configs requires
direct manipulation of the loaded YAML contents, which would be improper
to do outside of the module responsible for encapsulating this data.
Signed-off-by: Grzegorz Swiderski <grzegorz.swiderski@nordicsemi.no>
Updated test_report_summary to match new string with detailed
reason of build failure.
Signed-off-by: Grzegorz Chwierut <grzegorz.chwierut@nordicsemi.no>
Introduce a new harness based on pytest that does basic shell command
handling. The harness is enabeld using:
harness: shell
and expects a file with parameters in the form:
test_shell_harness:
- command: "kernel version"
expected: "Zephyr version .*"
- ...
Multiple commands and their expected output can be tested.
Signed-off-by: Anas Nashif <anas.nashif@intel.com>
With this change, the coverage data (GCOV dump) is extracted from the
test log files after each of the test instance parallel execution,
instead of post processing all the logs at the end of the Twister run.
The new `--coverage-per-instance` mode extends Twister coverage operations
to report coverage statistics on each test instance execution individually
in addition to the default reporting mode which aggregates data to one
report with all the test instances in the current scope of the Twister run.
The split mode allows to identify precisely what amount of code coverage
each test suite provides and to analyze its contribution to the overall
test plan's coverage. Each test configuration's output directory will have
its own coverage report and data files, so the overall disk space and
the total execution time increase.
Another new `--disable-coverage-aggregation` option allows to execute
only the `--coverage-per-instance` mode when the aggregate coverage
report for the whole Twister run scope is not needed.
Signed-off-by: Dmitrii Golovanov <dmitrii.golovanov@intel.com>
This commit removes all references to the `xtools` toolchain variant in the
twister scripts.
Note that the `xtools` toolchain variant has been deprecated since Zephyr
v3.3.0 and now removed.
Signed-off-by: Stephanos Ioannidis <root@stephanos.io>
Extend Twister Harness 'recording' feature to allow multiple
regular expression patterns to extract different types of records
from test output.
Add 'merge' recording mode to collect all extracted data fields
into a single record object of the test instance.
Export to CSV file now takes all field names occurred in the collected
records, sort it alphabetically, and then use it for columns instead of
using only the first record's fields. This is done to address possible
situation when records have different set of fields.
Adjust Twister documentation and test suite to the above changes.
Signed-off-by: Dmitrii Golovanov <dmitrii.golovanov@intel.com>
Added integration_toolchains to allow building/testing with multiple
toolchains available in the environment.
This changes the output structure and adds another level in the path
under twister_out signifying the toolchain used. The toolchain used
(variant) is also part of the json output now.
Signed-off-by: Anas Nashif <anas.nashif@intel.com>
Pass a test failure reason text from harness to the instance
to convey more details on the failure.
Signed-off-by: Dmitrii Golovanov <dmitrii.golovanov@intel.com>
Set missing TestCase statuses when a test under the BinaryHandler failed
(crashed), so remaining 'STARTED' and 'NONE' are now 'BLOCK' instead.
Signed-off-by: Dmitrii Golovanov <dmitrii.golovanov@intel.com>
Extend `--no-detailed-test-id` command line option: in addition to its
current behavior to exclude from a test Suite name its configuration path
prefix, also don't prefix each Ztest Case name with its Scenario name.
For example: 'kernel.common.timing' Scenario name, the same Suite name,
and 'sleep.usleep' test Case (where 'sleep' is its Ztest suite name
and 'usleep' is Ztest test name.
This way both TestSuite and TestCase names follow the same principle
having no parent object name prefix.
There is no information loss in Twister reports with this naming:
TestSuite is a container object for its TestCases, whereas TestSuite
has its configuration path as a property.
Signed-off-by: Dmitrii Golovanov <dmitrii.golovanov@intel.com>
Pass additional args to test binary. Twister
passes it only for native_sim.
Fixes#82463
Signed-off-by: Grzegorz Chwierut <grzegorz.chwierut@nordicsemi.no>
Support for space-separated lists was deprecated a long time ago
a91620f5bb so it is time to remove
support for them.
Any project that has not migrated can still use
`scripts/utils/twister_to_list.py` to automatically migrate twister
files.
Signed-off-by: Gerard Marull-Paretas <gerard@teslabs.com>
update test cases for twister
1. test_errors.py add protection.
2. test_handlers.py change call to status
3. test_testsuite.py change call to status
The log traces for TwisterException objects only.
And the stack trace output follow the same rules for all exceptions,
but StatusAttributeError with its dedicated handlers.
Signed-off-by: Hake Huang <hake.huang@oss.nxp.com>
This fixes ruff linting error SIM115,
where files were opened without the use of
a context handler.
Signed-off-by: Lukasz Mrugala <lukaszx.mrugala@intel.com>
