Take an instance that has failed and not whatever was last processed in
the for loop.
Also, change instructions to use `west twister` rather than
`./scripts/twister` to be consistent.
Signed-off-by: Anas Nashif <anas.nashif@intel.com>
If errors are found and tests are failing, print a brief summary showing
the top 10 items failing with instructions on how to reproduce either
with twister or west.
Signed-off-by: Anas Nashif <anas.nashif@intel.com>
Disables allowing the python argparse library from automatically
shortening command line arguments, this prevents issues whereby
a new command is added and code that wrongly uses the shortened
command of an existing argument which is the same as the new
command being added will silently change script behaviour.
Signed-off-by: Jamie McCrae <jamie.mccrae@nordicsemi.no>
When twister is started from make, use the information in MAKEFLAGS to
participate in the job server started by make.
When twister is running standalone, start a jobserver to share with
child processes. If using the kitware version of ninja, then this will
prevent ninja+twister from starting more processes than XX when twister
is run with -jXX.
There is no implementation for windows yet.
Signed-off-by: Jeremy Bettis <jbettis@google.com>
This reverts commit 21eb27c5c0.
This change has been a source of much trouble and ends up preventing us
from tracking test results across tree. It uses the repo name as the
namespace, and that is not always the same and does not have to be
called 'zephyr', depending on where you are running and in which
environment.
Signed-off-by: Anas Nashif <anas.nashif@intel.com>
Improves the message for tests being excluded using the platform key,
describes the key already found and the platform being used to run the test
in its place.
Signed-off-by: Tom Burdick <thomas.burdick@intel.com>
Implementation ported from TwisterV2.
- quarantine handled by separate module
- multiple yaml allowed from args: --quarantine-list
- scenarios, platforms, architectures keywords in
quarantine yaml are optional, if not given - means take it all
Signed-off-by: Grzegorz Chwierut <grzegorz.chwierut@nordicsemi.no>
Some hardware boards are supported in qemu as simulation platforms. If
we are testing with the real hardware, make sure we treat this as a
device and not as a simulator. First priority when initializing a
handler is to check if device testing is requested, then we can handle
the rest without --device-testing.
Signed-off-by: Anas Nashif <anas.nashif@intel.com>
Adds an option to inform twister a testsuite should only be built and
run for platforms with unique sets of attributes. This enables
for example keying on unique (arch, simulation) platforms to run the test
suite on.
The most common usage may be test suites configured to run once per
(arch, simulation) pair as being enough. Additional information about
platforms may enable running a test once per hardware IP block or once
per soc family or soc avoiding duplicated effort in building and running
tests when once suffices.
Signed-off-by: Tom Burdick <thomas.burdick@intel.com>
We want to be able to have platform or architecture extra configs
without having to duplicate a whole section of the test specification.
This adds support for namespacing of extra configs, for example:
arch:nios2:CONFIG_SYS_CLOCK_TICKS_PER_SEC=1000
or
platform:qemu_x86:CONFIG_FOO=y
Signed-off-by: Anas Nashif <anas.nashif@intel.com>
Previous commit dealing with cleanup (c304db56f1) was not taking
--prep-artifacts-for-testing into consideration. Fix this and make sure
we do cleanup when this option is enabled.
Signed-off-by: Anas Nashif <anas.nashif@intel.com>
Have boards define what binaries should be kept for device testing. The
purpose of this is to reduce the amount of artifacts and just keep the
binaries needed for flashing a device, reducing the size of the final
payload.
A board can have twister keep multiple binaries needed for testing when
--prep-artifacts-for-testing option is used.
Signed-off-by: Anas Nashif <anas.nashif@intel.com>
When re-generating hardware map for DKs with more than 1 serial port (more
enries in the hardware map), then we keep enfo only about the last entry.
For boards like nrf5340 it works, because the last serial port is used,
but for nrf52840 with two ports, the first one is important, so after
re-generating the hardware map it does not work.
The solution is to keep all entries after re-generating the hardware map.
Signed-off-by: Grzegorz Chwierut <grzegorz.chwierut@nordicsemi.no>
When scanning for available elfs, twister discards ones with `_pre`
in their name, to not include prebuilds like 'zephyr_pre0.elf`.
However, the scan looks at the whole test name. If _pre is in the name
(e.g. _precision) it would be wrongly discarded ending with twister
reporting error that no elfs were found. This commit makes the discard
to look only at the last part of the name (i.e. not including test name
but only a lowest level, where zephyr.elf etc are located.)
Signed-off-by: Maciej Perkowski <Maciej.Perkowski@nordicsemi.no>
When using llvm we need to set the gcov-tool to "llvm-cov gcov" but
the lcov tool is incapable of passing arguments to the gcov-tool. i.e.
the following cannot work:
$ lcov --gcov-tool "llvm-cov gcov" ...
Instead, create a symlink to llvm-cov prefixed as `gcov` which by the
documentation of llvm-cov will alias to `llvm-cov gcov` subcommand.
Signed-off-by: Yuval Peress <peress@google.com>
This commit expands Twister's CLI so that arguments passed after
a double dash (`--`) get forwarded to the ztest test executable when
run. When built for native_posix, ztest provides a useful CLI that
allows filtering individual suites and tests, adjusting timing settings,
and controlling the shuffling seed. Currently there is no easy way to
use these (with the exception of `--seed`) without needing to dig out
the resulting `zephyr.exe` binary from the build dir and invoke it
manually.
### Examples
Run a specific ztest suite only (useful when writing tests and you don't
want to run a long testcase in its entirety)
```
$ scripts/twister \
-p native_posix \
-s
zephyr/tests/lib/cmsis_dsp/complexmath/libraries.cmsis_dsp.complexmath \
-- -test=complexmath_q31::*
```
Unrecognized arguments that precede the double dash will result in an
error message:
```
$ scripts/twister \
-p native_posix \
--foobar 123 \
-- -test=basic_math_q7::*
...
Unrecognized arguments found: '--foobar 123'. Use -- to delineate extra
arguments for test binary or pass -h for help.
```
Signed-off-by: Tristan Honscheid <honscheid@google.com>
Ignore unit tests and EFI binaries when calculating footprints with
--show-footprint, or we will get a build error.
Signed-off-by: Anas Nashif <anas.nashif@intel.com>
The --snr (nrfjprog) --id (jlink) and --board-id (pyocd) options were
deprecated a long time ago in favor of --dev-id. It is time to remove
them.
Signed-off-by: Gerard Marull-Paretas <gerard.marull@nordicsemi.no>
This commit calculate memory footprint from build.log and
proposes an alternative approach to #2465.
Signed-off-by: Katarzyna Giądła <katarzyna.giadla@nordicsemi.no>
non-simulation platforms set key to 'na', we need to check for that
instead of just checking for existance of key.
Fixes#52595
Signed-off-by: Anas Nashif <anas.nashif@intel.com>
If a default platform is not able to run for whatever reason, treat it
as any other platform. Default platforms shall ALWAYS be runnable in CI,
or else we are just re-building for no good reason.
Signed-off-by: Anas Nashif <anas.nashif@intel.com>
the ser.in_wait sometime meet issue like serial handler error.
This is because the serial port is in reset status.
add final protection to retry later
error log:
Traceback (most recent call last):
File "/usr/lib/python3.10/threading.py", line 1016, in _bootstrap_inner
self.run()
File "/usr/lib/python3.10/threading.py", line 953, in run
self._target(*self._args, **self._kwargs)
File "zephyr/scripts/pylib/twister/twisterlib/handlers.py", line 345,
in monitor_serial
if not ser.in_waiting:
File "~/python3.10/site-packages/serial/serialposix.py", line 549,
in in_waiting
s = fcntl.ioctl(self.fd, TIOCINQ, TIOCM_zero_str)
OSError: [Errno 9] Bad file descriptor
Signed-off-by: Hake Huang <hake.huang@oss.nxp.com>
add 2 choices for '--runtime-artifact-cleanup' option,
'passing' to delete artifacts of passing tests,
'all' to delete artifacts of both passing tests
and failing tests
keep testcase_extra.config, it contains some extra Kconfig
needed for the tests and they is only generated first time
when run retry failed
Signed-off-by: Cong Nguyen Huu <cong.nguyenhuu@nxp.com>
With the patch a relative path between a test and west's project top
directory is always calculated instead of being relative to zephyr
base. The test names and results structur change by addition of one
level. This way the names/structure conventon for tests from zephyr and
modules is alligned.
For external tests ".." parts are replaced with "external".
This allows to maintain unified folder structure also for results of
external tests.
Without this patch the folder/name allignment is lost for
external and module test becaming depandant on a way how
twister was called.
.
Signed-off-by: Maciej Perkowski <Maciej.Perkowski@nordicsemi.no>
Version is not enough, we need the run date but also the commit date, to
ease reporting and use in dashboards.
Signed-off-by: Anas Nashif <anas.nashif@intel.com>
In some scenarios we might end up with a null version value in the json
file, if the version can't be determined, set it to Unkown instead.
Signed-off-by: Anas Nashif <anas.nashif@intel.com>
When twister is set to retry any failures, count the number of retries
and record the number in the json file. This will help us identify
unstable tests or tests requiring attention.
Signed-off-by: Anas Nashif <anas.nashif@intel.com>
Update twister to support running QEMU platforms with sysbuild, by parsing
domains.yaml and executing the "run" target of the default application.
This will allow twister to test QEMU targets with sysbuild. It is assumed
that QEMU targets will add any external images they need in the build
phase
Signed-off-by: Daniel DeGrasse <daniel.degrasse@nxp.com>
When running twister -vv many blank lines would be logged.
Fix handing of empty lines from the reader thread. Real blank likes will
be b"\n" not b"".
Change open call to use with, and logger to use %s to fix pylint
warnings.
Signed-off-by: Jeremy Bettis <jbettis@google.com>
Modify the regex used to merge overlays specified for
test samples in twister tests.
Overlays without any prefixes will be merged and passed to cmake.
Overlays with prefixes won't be merged but passed to cmake as they are.
Signed-off-by: Piotr Węgliński <piotr.weglinski@nordicsemi.no>
The logic to drain the leftover logs of previous test should be
put before the monitor_serial(). It was misplaced when re-adding
this logic back.
Signed-off-by: Ming Shao <ming.shao@intel.com>
In BinaryHandler we process input from simulator with decode('utf-8')
to convert it to string. decode has strict error handling by default -
so it raises UnicodeDecodeError exception if it can't decode input
binary sequence.
So if test start to print some junk to uart console we get
UnicodeDecodeError exception which cause the whole twister crash.
To fix that switch decode to less strict error handling when it
just replace undecoded binary sequence with unicode replacement
character.
Signed-off-by: Eugeniy Paltsev <Eugeniy.Paltsev@synopsys.com>
Signed-off-by: Evgeniy Paltsev <PaltsevEvgeniy@gmail.com>
On windows, some board manufacturers have name that is "Microsoft"
instead of real its name. Need add it to manufacturer list.
Product information is missing. we can assign it "unknown" and
then add missing informations by hand.
Signed-off-by: Cong Nguyen Huu <cong.nguyenhuu@nxp.com>
Currently, support for Twister on Windows is only build.
The only thing that needs to be ported is that select()
cannot wait on file descriptors on Windows. Therefore
the serial monitor function needs to be reworked to
support both OSes.
Signed-off-by: Cong Nguyen Huu <cong.nguyenhuu@nxp.com>
