This commit expands Twister's CLI so that arguments passed after
a double dash (`--`) get forwarded to the ztest test executable when
run. When built for native_posix, ztest provides a useful CLI that
allows filtering individual suites and tests, adjusting timing settings,
and controlling the shuffling seed. Currently there is no easy way to
use these (with the exception of `--seed`) without needing to dig out
the resulting `zephyr.exe` binary from the build dir and invoke it
manually.
### Examples
Run a specific ztest suite only (useful when writing tests and you don't
want to run a long testcase in its entirety)
```
$ scripts/twister \
-p native_posix \
-s
zephyr/tests/lib/cmsis_dsp/complexmath/libraries.cmsis_dsp.complexmath \
-- -test=complexmath_q31::*
```
Unrecognized arguments that precede the double dash will result in an
error message:
```
$ scripts/twister \
-p native_posix \
--foobar 123 \
-- -test=basic_math_q7::*
...
Unrecognized arguments found: '--foobar 123'. Use -- to delineate extra
arguments for test binary or pass -h for help.
```
Signed-off-by: Tristan Honscheid <honscheid@google.com>
Ignore unit tests and EFI binaries when calculating footprints with
--show-footprint, or we will get a build error.
Signed-off-by: Anas Nashif <anas.nashif@intel.com>
This commit calculate memory footprint from build.log and
proposes an alternative approach to #2465.
Signed-off-by: Katarzyna Giądła <katarzyna.giadla@nordicsemi.no>
add 2 choices for '--runtime-artifact-cleanup' option,
'passing' to delete artifacts of passing tests,
'all' to delete artifacts of both passing tests
and failing tests
keep testcase_extra.config, it contains some extra Kconfig
needed for the tests and they is only generated first time
when run retry failed
Signed-off-by: Cong Nguyen Huu <cong.nguyenhuu@nxp.com>
When twister is set to retry any failures, count the number of retries
and record the number in the json file. This will help us identify
unstable tests or tests requiring attention.
Signed-off-by: Anas Nashif <anas.nashif@intel.com>
Update twister to support running QEMU platforms with sysbuild, by parsing
domains.yaml and executing the "run" target of the default application.
This will allow twister to test QEMU targets with sysbuild. It is assumed
that QEMU targets will add any external images they need in the build
phase
Signed-off-by: Daniel DeGrasse <daniel.degrasse@nxp.com>
Modify the regex used to merge overlays specified for
test samples in twister tests.
Overlays without any prefixes will be merged and passed to cmake.
Overlays with prefixes won't be merged but passed to cmake as they are.
Signed-off-by: Piotr Węgliński <piotr.weglinski@nordicsemi.no>
Add support for building with sysbuild using twister, via the "sysbuild"
yaml property in testsuites. This will currently disable Kconfig and
devicetree filtering.
Signed-off-by: Daniel DeGrasse <daniel.degrasse@nxp.com>
Due to the diverse coding styles and lack of preprocessing when
scanning for test cases, there were many unintended combinations
of yaml test scenarios and C test functions with the regex-based
test case discovery, which caused an inaccurate test plan and test
result stats.
As the new ztest fx is used, the test cases of a test instance can
be accurately determined via zephyr.symbols file.
Because the zephyr.symbols file is not available until after build,
test cases determination is placed right after the build stage of
the pipeline and before the runtime test collection.
For those test instances that don't go through the build stage,
such as those with "Skip filtered" (statically filtered) reason
before entering the pipeline, they are not affected.
This patch also adjust the stats logic because the zephyr.symbols
file is generated after build. So ExecutionCounter update is split
and some must be postponed until test execution pipeline is completed.
Some concepts:
A test instance = a yaml scenario + a platform
"Test instance" and "test config" are synonyms in twister, unfortunately
excessive IMHO...
A test instance can be filtered or skipped in 3 ways.
Let's define "runtime" as "after entering the execution pipeline".
1) statically filtered (before runtime)
Such test instance is filtered by analyzing the yaml and never
enters the execution pipeline.
2) cmake filtered (runtime)
Such test instance enters pipeline and is filtered at cmake stage.
3) build skipped (also runtime)
Such test instance enters pipeline and is skipped at build stage.
All the test instances that enter the execution pipeline will go
through the report stage, where ExecutionCounter is further updated.
The meaning of the fields of ExecutionCounter are:
.skipped_configs = static filtered + cmake filtered + build skipped
.skipped_runtime = cmake filtered + build skipped
.skipped_filter = static filtered
.done = instances that enter the execution pipeline
.passed = instances that actually executed and passed
Definition of the overall stats:
total_complete = .done + .skipped_filter
total = yaml test scenario * applicable platforms
complete percentage = total_complete / total
pass rate = passed / (total - skipped_configs)
Signed-off-by: Ming Shao <ming.shao@intel.com>
Twister detects FLASH overflow and skips tests that trigger that
condition by default, but sometimes images are just on the flash size
limit and then overflows at a later stage when imgtool adds a trailer,
which gets detected as a build fail and fails the run.
This detects the imgtool flash overflow together with the normal build
ones, causing the build to be skipped on imgtool flash overflow as well.
Signed-off-by: Fabio Baltieri <fabiobaltieri@google.com>
With verbose output enabled, twister logs the cmake arguments as a
python array whenever it invokes cmake to generate a build system.
This is clunky output for manual reproduction, because it forces the
developer to manually join the arguments together for consumption by
the shell.
As an enhancement for this use case, use the standard library
shlex.join() function to produce a string that can be copy/pasted
directly into the shell on POSIX platforms.
We should use this function instead of str.join, because it correctly
handles things like shell quoting for options with spaces in them.
This function is not available on Windows, so we fall back on behavior
that is similar to the old one there.
The main difference on Windows is that the output now includes the
path to cmake as well, and not just its arguments. (This is the same
on POSIX, and is intended to help debug if multiple cmakes are
installed on the same machine.)
We are running cmake from a couple of different places, so doing this
cleanly requires adding a shared module with the platform abstraction.
Signed-off-by: Martí Bolívar <marti.bolivar@nordicsemi.no>
esp32 platforms can overflow their dram0_1_seg so add that section name
to the overflow check in twister. We were seeing this with the
samples/arch/smp/pi/sample.smp.pi test and would get:
zephyr_pre0.elf section `noinit' will not fit in region `dram0_1_seg'
region `dram0_1_seg' overflowed by 17456 bytes
Fixes#49164
Signed-off-by: Kumar Gala <galak@kernel.org>
Escape quotes provided in extra arguments on the command line.
example: ... -x "CONFIG_COMPILER_OPT=\"-fanalyzer\""
Fixes#46382
Signed-off-by: Anas Nashif <anas.nashif@intel.com>
Add option to prepare for device testing by only building and storing
the needed data that can then later be executed and run on DUT using
--test-only.
This adds --prep-artifacts-for-testing which is the same as --build-only
and --device-testing and other options that were needed to make this
operational and produce the right binaries and files.
Signed-off-by: Anas Nashif <anas.nashif@intel.com>
