Adds an option to inform twister a testsuite should only be built and
run for platforms with unique sets of attributes. This enables
for example keying on unique (arch, simulation) platforms to run the test
suite on.
The most common usage may be test suites configured to run once per
(arch, simulation) pair as being enough. Additional information about
platforms may enable running a test once per hardware IP block or once
per soc family or soc avoiding duplicated effort in building and running
tests when once suffices.
Signed-off-by: Tom Burdick <thomas.burdick@intel.com>
We want to be able to have platform or architecture extra configs
without having to duplicate a whole section of the test specification.
This adds support for namespacing of extra configs, for example:
arch:nios2:CONFIG_SYS_CLOCK_TICKS_PER_SEC=1000
or
platform:qemu_x86:CONFIG_FOO=y
Signed-off-by: Anas Nashif <anas.nashif@intel.com>
Previous commit dealing with cleanup (c304db56f1) was not taking
--prep-artifacts-for-testing into consideration. Fix this and make sure
we do cleanup when this option is enabled.
Signed-off-by: Anas Nashif <anas.nashif@intel.com>
Have boards define what binaries should be kept for device testing. The
purpose of this is to reduce the amount of artifacts and just keep the
binaries needed for flashing a device, reducing the size of the final
payload.
A board can have twister keep multiple binaries needed for testing when
--prep-artifacts-for-testing option is used.
Signed-off-by: Anas Nashif <anas.nashif@intel.com>
When re-generating hardware map for DKs with more than 1 serial port (more
enries in the hardware map), then we keep enfo only about the last entry.
For boards like nrf5340 it works, because the last serial port is used,
but for nrf52840 with two ports, the first one is important, so after
re-generating the hardware map it does not work.
The solution is to keep all entries after re-generating the hardware map.
Signed-off-by: Grzegorz Chwierut <grzegorz.chwierut@nordicsemi.no>
When scanning for available elfs, twister discards ones with `_pre`
in their name, to not include prebuilds like 'zephyr_pre0.elf`.
However, the scan looks at the whole test name. If _pre is in the name
(e.g. _precision) it would be wrongly discarded ending with twister
reporting error that no elfs were found. This commit makes the discard
to look only at the last part of the name (i.e. not including test name
but only a lowest level, where zephyr.elf etc are located.)
Signed-off-by: Maciej Perkowski <Maciej.Perkowski@nordicsemi.no>
When using llvm we need to set the gcov-tool to "llvm-cov gcov" but
the lcov tool is incapable of passing arguments to the gcov-tool. i.e.
the following cannot work:
$ lcov --gcov-tool "llvm-cov gcov" ...
Instead, create a symlink to llvm-cov prefixed as `gcov` which by the
documentation of llvm-cov will alias to `llvm-cov gcov` subcommand.
Signed-off-by: Yuval Peress <peress@google.com>
This commit expands Twister's CLI so that arguments passed after
a double dash (`--`) get forwarded to the ztest test executable when
run. When built for native_posix, ztest provides a useful CLI that
allows filtering individual suites and tests, adjusting timing settings,
and controlling the shuffling seed. Currently there is no easy way to
use these (with the exception of `--seed`) without needing to dig out
the resulting `zephyr.exe` binary from the build dir and invoke it
manually.
### Examples
Run a specific ztest suite only (useful when writing tests and you don't
want to run a long testcase in its entirety)
```
$ scripts/twister \
-p native_posix \
-s
zephyr/tests/lib/cmsis_dsp/complexmath/libraries.cmsis_dsp.complexmath \
-- -test=complexmath_q31::*
```
Unrecognized arguments that precede the double dash will result in an
error message:
```
$ scripts/twister \
-p native_posix \
--foobar 123 \
-- -test=basic_math_q7::*
...
Unrecognized arguments found: '--foobar 123'. Use -- to delineate extra
arguments for test binary or pass -h for help.
```
Signed-off-by: Tristan Honscheid <honscheid@google.com>
Ignore unit tests and EFI binaries when calculating footprints with
--show-footprint, or we will get a build error.
Signed-off-by: Anas Nashif <anas.nashif@intel.com>
The --snr (nrfjprog) --id (jlink) and --board-id (pyocd) options were
deprecated a long time ago in favor of --dev-id. It is time to remove
them.
Signed-off-by: Gerard Marull-Paretas <gerard.marull@nordicsemi.no>
This commit calculate memory footprint from build.log and
proposes an alternative approach to #2465.
Signed-off-by: Katarzyna Giądła <katarzyna.giadla@nordicsemi.no>
non-simulation platforms set key to 'na', we need to check for that
instead of just checking for existance of key.
Fixes#52595
Signed-off-by: Anas Nashif <anas.nashif@intel.com>
If a default platform is not able to run for whatever reason, treat it
as any other platform. Default platforms shall ALWAYS be runnable in CI,
or else we are just re-building for no good reason.
Signed-off-by: Anas Nashif <anas.nashif@intel.com>
the ser.in_wait sometime meet issue like serial handler error.
This is because the serial port is in reset status.
add final protection to retry later
error log:
Traceback (most recent call last):
File "/usr/lib/python3.10/threading.py", line 1016, in _bootstrap_inner
self.run()
File "/usr/lib/python3.10/threading.py", line 953, in run
self._target(*self._args, **self._kwargs)
File "zephyr/scripts/pylib/twister/twisterlib/handlers.py", line 345,
in monitor_serial
if not ser.in_waiting:
File "~/python3.10/site-packages/serial/serialposix.py", line 549,
in in_waiting
s = fcntl.ioctl(self.fd, TIOCINQ, TIOCM_zero_str)
OSError: [Errno 9] Bad file descriptor
Signed-off-by: Hake Huang <hake.huang@oss.nxp.com>
add 2 choices for '--runtime-artifact-cleanup' option,
'passing' to delete artifacts of passing tests,
'all' to delete artifacts of both passing tests
and failing tests
keep testcase_extra.config, it contains some extra Kconfig
needed for the tests and they is only generated first time
when run retry failed
Signed-off-by: Cong Nguyen Huu <cong.nguyenhuu@nxp.com>
Convert domains.py to use basic python logging module implementation to
remove west logging dependency. This removes hard dependency on west from
twister.
Fixes#51731
Signed-off-by: Daniel DeGrasse <daniel.degrasse@nxp.com>
With the patch a relative path between a test and west's project top
directory is always calculated instead of being relative to zephyr
base. The test names and results structur change by addition of one
level. This way the names/structure conventon for tests from zephyr and
modules is alligned.
For external tests ".." parts are replaced with "external".
This allows to maintain unified folder structure also for results of
external tests.
Without this patch the folder/name allignment is lost for
external and module test becaming depandant on a way how
twister was called.
.
Signed-off-by: Maciej Perkowski <Maciej.Perkowski@nordicsemi.no>
Version is not enough, we need the run date but also the commit date, to
ease reporting and use in dashboards.
Signed-off-by: Anas Nashif <anas.nashif@intel.com>
In some scenarios we might end up with a null version value in the json
file, if the version can't be determined, set it to Unkown instead.
Signed-off-by: Anas Nashif <anas.nashif@intel.com>
When twister is set to retry any failures, count the number of retries
and record the number in the json file. This will help us identify
unstable tests or tests requiring attention.
Signed-off-by: Anas Nashif <anas.nashif@intel.com>
Update twister to support running QEMU platforms with sysbuild, by parsing
domains.yaml and executing the "run" target of the default application.
This will allow twister to test QEMU targets with sysbuild. It is assumed
that QEMU targets will add any external images they need in the build
phase
Signed-off-by: Daniel DeGrasse <daniel.degrasse@nxp.com>
Since twister can make use of domains.py as well, refactor west's use
this file so that domains.py can be moved to a generic library folder.
Signed-off-by: Daniel DeGrasse <daniel.degrasse@nxp.com>
When running twister -vv many blank lines would be logged.
Fix handing of empty lines from the reader thread. Real blank likes will
be b"\n" not b"".
Change open call to use with, and logger to use %s to fix pylint
warnings.
Signed-off-by: Jeremy Bettis <jbettis@google.com>
Modify the regex used to merge overlays specified for
test samples in twister tests.
Overlays without any prefixes will be merged and passed to cmake.
Overlays with prefixes won't be merged but passed to cmake as they are.
Signed-off-by: Piotr Węgliński <piotr.weglinski@nordicsemi.no>
The logic to drain the leftover logs of previous test should be
put before the monitor_serial(). It was misplaced when re-adding
this logic back.
Signed-off-by: Ming Shao <ming.shao@intel.com>
In BinaryHandler we process input from simulator with decode('utf-8')
to convert it to string. decode has strict error handling by default -
so it raises UnicodeDecodeError exception if it can't decode input
binary sequence.
So if test start to print some junk to uart console we get
UnicodeDecodeError exception which cause the whole twister crash.
To fix that switch decode to less strict error handling when it
just replace undecoded binary sequence with unicode replacement
character.
Signed-off-by: Eugeniy Paltsev <Eugeniy.Paltsev@synopsys.com>
Signed-off-by: Evgeniy Paltsev <PaltsevEvgeniy@gmail.com>
On windows, some board manufacturers have name that is "Microsoft"
instead of real its name. Need add it to manufacturer list.
Product information is missing. we can assign it "unknown" and
then add missing informations by hand.
Signed-off-by: Cong Nguyen Huu <cong.nguyenhuu@nxp.com>
Currently, support for Twister on Windows is only build.
The only thing that needs to be ported is that select()
cannot wait on file descriptors on Windows. Therefore
the serial monitor function needs to be reworked to
support both OSes.
Signed-off-by: Cong Nguyen Huu <cong.nguyenhuu@nxp.com>
Add support for building with sysbuild using twister, via the "sysbuild"
yaml property in testsuites. This will currently disable Kconfig and
devicetree filtering.
Signed-off-by: Daniel DeGrasse <daniel.degrasse@nxp.com>
Due to the diverse coding styles and lack of preprocessing when
scanning for test cases, there were many unintended combinations
of yaml test scenarios and C test functions with the regex-based
test case discovery, which caused an inaccurate test plan and test
result stats.
As the new ztest fx is used, the test cases of a test instance can
be accurately determined via zephyr.symbols file.
Because the zephyr.symbols file is not available until after build,
test cases determination is placed right after the build stage of
the pipeline and before the runtime test collection.
For those test instances that don't go through the build stage,
such as those with "Skip filtered" (statically filtered) reason
before entering the pipeline, they are not affected.
This patch also adjust the stats logic because the zephyr.symbols
file is generated after build. So ExecutionCounter update is split
and some must be postponed until test execution pipeline is completed.
Some concepts:
A test instance = a yaml scenario + a platform
"Test instance" and "test config" are synonyms in twister, unfortunately
excessive IMHO...
A test instance can be filtered or skipped in 3 ways.
Let's define "runtime" as "after entering the execution pipeline".
1) statically filtered (before runtime)
Such test instance is filtered by analyzing the yaml and never
enters the execution pipeline.
2) cmake filtered (runtime)
Such test instance enters pipeline and is filtered at cmake stage.
3) build skipped (also runtime)
Such test instance enters pipeline and is skipped at build stage.
All the test instances that enter the execution pipeline will go
through the report stage, where ExecutionCounter is further updated.
The meaning of the fields of ExecutionCounter are:
.skipped_configs = static filtered + cmake filtered + build skipped
.skipped_runtime = cmake filtered + build skipped
.skipped_filter = static filtered
.done = instances that enter the execution pipeline
.passed = instances that actually executed and passed
Definition of the overall stats:
total_complete = .done + .skipped_filter
total = yaml test scenario * applicable platforms
complete percentage = total_complete / total
pass rate = passed / (total - skipped_configs)
Signed-off-by: Ming Shao <ming.shao@intel.com>
This commit removes the stale reference to `k_mem_pool_area` for the
memory pool API, which was removed in the v2.5.0 release.
Signed-off-by: Stephanos Ioannidis <stephanos.ioannidis@nordicsemi.no>
This logic was added before but seems to be missed during the
twister refactoring.
Some tests can cause serial leftover logs buffered somewhere.
Such leftover can interfere with the next test case because the
serial log monitoring thread is started before a board is flashed.
And the monitoring thread can be fooled by such leftover logs and
make incorrect judgement of the test result.
A simple ser.flush() is not enough to eliminate such leftovers.
So add explicit readline() to drain such logs which ensures a clean
serial context for the case that follows.
An example from reel board captured with this patch:
leftover log of previous test: b'.287 seconds\r\n'
leftover log of previous test: b' - SKIP - [...test_coredump_backend]...
leftover log of previous test: b'\r\n'
leftover log of previous test: b'------ TESTSUITE SUMMARY END ------\r\n'
leftover log of previous test: b'\r\n'
leftover log of previous test: b'=====================================...
leftover log of previous test: b'RunID: 4e93757ad...53dcab9f0f5c6\r\n'
leftover log of previous test: b'PROJECT EXECUTION SUCCESSFUL\r\n
Signed-off-by: Ming Shao <ming.shao@intel.com>
Twister detects FLASH overflow and skips tests that trigger that
condition by default, but sometimes images are just on the flash size
limit and then overflows at a later stage when imgtool adds a trailer,
which gets detected as a build fail and fails the run.
This detects the imgtool flash overflow together with the normal build
ones, causing the build to be skipped on imgtool flash overflow as well.
Signed-off-by: Fabio Baltieri <fabiobaltieri@google.com>
A workaround to exclude boards from twister. Other solutions do not work,
setting twister fields to false in the board yaml file is not enough,
given that this board has many associated samples and tests, so it can't
be disabled on its own.
Signed-off-by: Anas Nashif <anas.nashif@intel.com>
Follow-up: #41301
This commit is a rework and cleanup of the tools handling in Zephyr
CMake build system.
Instead of directly loading code a CMake modules for tool lookup, the
host tools now follows the CMake `find_package()` pattern for finding
programs / tools in module mode.
This makes it more clear which modules are responsible for finding tools
and which modules provides build integration / features.
The following tools can now be found using `find_package()`:
- Zephyr-sdk : find_package(Zephyr-sdk <version>)
- Generic host tools: find_package(HostTools)
This further allows us to decouple the `verify-toolchain` CMake script
part required by `twister` into a tool lookup module and a dedicated
CMake script which utilizes the lookup module.
Signed-off-by: Torsten Rasmussen <Torsten.Rasmussen@nordicsemi.no>
`handlers.py` treats the `psutil` package as optional and quietly prints
a message to stdout if it is not present, but it is actually a hard
requirement for the base `Handler` class. This try/except block is bad
as it hides the requirement until a timeout occurs and the handler
attempts to use `psutil` in the terminate method, which will crash if
the module happens to be missing.
This PR removes the try/except guard so missing `psutil` will cause
twister to immediately fail. `psutil` is already specified in the
`requirements.txt` file, so this should only affect users who
unwittingly have an outdated/incorrect environment.
Also update the Github Action for twister tests to install from
requirements files.
Signed-off-by: Tristan Honscheid <honscheid@google.com>
stty operates on the file descriptor from stdin, but we want to operate
on stdout. If stdin and stdout are not the same tty then stty prints an
error.
Redirect stdout to stdin for stty calls.
Tested by running twister -T tests/subsys/shell </dev/null
Signed-off-by: Jeremy Bettis <jbettis@google.com>
With verbose output enabled, twister logs the cmake arguments as a
python array whenever it invokes cmake to generate a build system.
This is clunky output for manual reproduction, because it forces the
developer to manually join the arguments together for consumption by
the shell.
As an enhancement for this use case, use the standard library
shlex.join() function to produce a string that can be copy/pasted
directly into the shell on POSIX platforms.
We should use this function instead of str.join, because it correctly
handles things like shell quoting for options with spaces in them.
This function is not available on Windows, so we fall back on behavior
that is similar to the old one there.
The main difference on Windows is that the output now includes the
path to cmake as well, and not just its arguments. (This is the same
on POSIX, and is intended to help debug if multiple cmakes are
installed on the same machine.)
We are running cmake from a couple of different places, so doing this
cleanly requires adding a shared module with the platform abstraction.
Signed-off-by: Martí Bolívar <marti.bolivar@nordicsemi.no>
esp32 platforms can overflow their dram0_1_seg so add that section name
to the overflow check in twister. We were seeing this with the
samples/arch/smp/pi/sample.smp.pi test and would get:
zephyr_pre0.elf section `noinit' will not fit in region `dram0_1_seg'
region `dram0_1_seg' overflowed by 17456 bytes
Fixes#49164
Signed-off-by: Kumar Gala <galak@kernel.org>
This adds an extra layer to match the revision part where this
needs to be of permitted patterns as described in.
cmake/modules/extensions.cmake. Without this, the matching
may produce undesirable results. For example, if there is
a file named qemu_x86_tiny_1.conf, the testplan will create
a board name qemu_x86@tiny.1 (which is definitely not correct).
Signed-off-by: Daniel Leung <daniel.leung@intel.com>
