Commit graph

286 commits

Author SHA1 Message Date
Lukasz Mrugala
ff1857af55 scripts: twister: Enchance TestCase/Instance info and presentation
ExecutionCounter has been expanded and now hold i.a.
more information on the statuses of TestCases.
This information is now incorporated in relevant summaries
- runner.py and reports.py.
Layout of those was changed to present that
and previous information in a clear and concise way.

TestInstance execution counter now is more intuitive.
Instances filtered out before running are no longer included there.
Retries now properly reset the counter.

TestCases with None and other incorrect final statuses
are logged as errors, but do not
exit Twister with a nonzero exit code.
This is because None statuses, although incorrect,
are currently common.

Inconsistent spacing in ERROR and FAILED fixed.

Signed-off-by: Lukasz Mrugala <lukaszx.mrugala@intel.com>

scripts: Dmitri fix

Fix of a problem noticed by Dmitri

Removed unnecessary additional spaces when
printing FAILED and ERROR status.
Now TwisterStatus.get_color is used more.

Signed-off-by: Lukasz Mrugala <lukaszx.mrugala@intel.com>
2024-10-31 09:24:18 -05:00
Reto Schneider
743761d7d1 scripts: Fix CMake spelling
As per its creators, CMake is written with a capital "M".

The initial reason for this change is that I want Twister to print
"ERROR : CMake build failure" instead of "ERROR : Cmake build failure".

Signed-off-by: Reto Schneider <reto.schneider@husqvarnagroup.com>
2024-10-30 16:32:24 -05:00
Anas Nashif
e90559f0da twister: fix platform filter when loading plan from file
if alias or shorthand name is provided on the command line, we need
convert this to complete target name for the filters to work.

Fixes #80332

Signed-off-by: Anas Nashif <anas.nashif@intel.com>
2024-10-26 13:41:55 +02:00
Anas Nashif
1c3b47e9ef twister: test udpates for new board handling
Updated tests for new board handling.

Signed-off-by: Anas Nashif <anas.nashif@intel.com>
2024-10-22 18:31:09 -04:00
Mara Furland
7ac41a1b24 scripts: twister: set Python3_EXECUTABLE when calling cmake
When twister calls cmake, set Python3_EXECUTABLE so things work as expected
when using non-global python

Signed-off-by: Mara Furland <mara@fur.land>
2024-10-18 14:15:13 +02:00
Lukasz Mrugala
82548410bd scripts: twister: NOTRUN status
Adds a new NOTRUN status, which indicates
that a test was successfully built, but
not run on account of being not
runnable in given test instance.

Signed-off-by: Lukasz Mrugala <lukaszx.mrugala@intel.com>
2024-10-17 09:46:17 +02:00
Hake Huang
33fe360bfc tests: twister: runner add runner test
add test for platform in extra_args

Signed-off-by: Hake Huang <hake.huang@oss.nxp.com>
2024-10-15 04:09:16 -04:00
Dmitrii Golovanov
27456ed2b5 twister: blackbox: coverage: fix matching pattern
Fix the expected pattern randomly matched to coverage.json contents
ordered differently at test_coverage_basedir().

Signed-off-by: Dmitrii Golovanov <dmitrii.golovanov@intel.com>
2024-10-10 04:19:56 -04:00
Dmitrii Golovanov
2ddab56bfd scripts: twister: Fix trailing CR/LF at BinaryHandler logs
Fix trailing `\\r\\n` (escaped CR/LF) didn't cut off because of rstrip()
removed by #58338, so the CR/LF suffix was never found as the actual line
end was `\\r\\n\n`.

Add ANSI code sequence to `test_handlers` Twister unit test.

Signed-off-by: Dmitrii Golovanov <dmitrii.golovanov@intel.com>
2024-09-27 14:30:26 +01:00
Benedikt Schmidt
fe3287a9ac scripts: dts: extract pickled EDT generation
Separate the pickled EDT generation from the C-Macro header
generation in gen_defines.py to have a more clear responsibility
of the scripts in the DTS parsing process.

Signed-off-by: Benedikt Schmidt <benedikt.schmidt@embedded-solutions.at>
2024-09-25 13:46:32 -05:00
Alberto Escolar Piedras
833f7da7ee scripts/tests/twister_blackbox: Don't test on native_posix
native_posix is now deprecated.
Building this sample in both native_sim and native_posix does not
improve coverage for the sample or subsystem but doubles CI time.
As anyhow native_posix will be removed all together in
2 releases, let's remove it already for this sample.

Signed-off-by: Alberto Escolar Piedras <alberto.escolar.piedras@nordicsemi.no>
2024-09-25 04:04:36 -04:00
Lukasz Mrugala
558c74be04 scripts: twister: decouple debug and verbosity
Currently, debug logging in the console and verbosity
are tightly coupled - verbosity of level 2 and higher
enables logging at the debug level.

This change introduces a separate Twister flag
responsible for controlling the debug logging,
while leaving the rest of verbosity unchanged.

This allows for controlling the verbosity on
both logging levels, according to one's needs.

Signed-off-by: Lukasz Mrugala <lukaszx.mrugala@intel.com>
2024-09-20 11:07:48 +02:00
Lukasz Mrugala
418b1e0e21 scripts: twister: Add TestCase status printing
When at verbosity 1, we print out the status of TestInstances.
This makes it harder to notice changes at TestCase level,
which require perusing the logs.

This adds TestCase status and reason printing
if verbosity level is 2 or more.
Reason printing is suppressed if the reason is empty or None.

Signed-off-by: Lukasz Mrugala <lukaszx.mrugala@intel.com>
2024-09-17 15:04:04 -04:00
Wilfried Chauveau
6194159608 scripts: twister: handlers: Enable BinaryHandler to run sysbuilt tests
Running sysbuilt tests fails because of missing "run" target.

This adds the default domain context to the command.

Signed-off-by: Wilfried Chauveau <wilfried.chauveau@arm.com>
2024-09-17 09:13:26 -05:00
Lukasz Mrugala
14a72b5158 scripts: Plug TwisterStatus type gaps
Some dict.get() calls did not use a TwisterStatus
as a default value, thus using a NoneType where
TwisterStatus should appear.

Signed-off-by: Lukasz Mrugala <lukaszx.mrugala@intel.com>
2024-09-16 20:18:09 +02:00
Hake Huang
7f63faa4f5 twister: hwmap: add script parameter support
for pre/post/post_flash script, add timeout as param

Signed-off-by: Hake Huang <hake.huang@oss.nxp.com>
2024-09-16 10:03:53 +02:00
Pieter De Gendt
d2bae91803 scripts: pylib: twister: Pass testsuite name to cmake
Add a CMake argument with the current testsuite name.

Signed-off-by: Pieter De Gendt <pieter.degendt@basalte.be>
2024-09-12 10:04:05 +02:00
Wilfried Chauveau
30dc833cab scripts: twister: Use args rather than injecting members
This changes how some arguments are set in the `Handler`s.
`options`, `generator_cmd` and `suite_name_check` are now passed as
arguments to the constructor rather than injected from an other module.

Signed-off-by: Wilfried Chauveau <wilfried.chauveau@arm.com>
2024-09-12 10:03:44 +02:00
Wilfried Chauveau
6d9e5df401 scripts: twister: TwisterEnv is always used with an options parameter
Adjusting the type hints and code accordingly.

Signed-off-by: Wilfried Chauveau <wilfried.chauveau@arm.com>
2024-09-12 10:03:44 +02:00
Wilfried Chauveau
deeec33934 scripts: twister: Solve deprecation notice when running tests
CONF_FILE, DTC_OVERLAY_FILE and OVERLAY_CONFIG are deprecated but still
used by the tests causing warnings when running them.

This adds a test_data specific to validate the emission of the warning,
and removes the offending args from the other test_data files.

Signed-off-by: Wilfried Chauveau <wilfried.chauveau@arm.com>
2024-09-12 10:03:44 +02:00
Wilfried Chauveau
28add0cd36 scripts: twister: check for deprecation notice for space-separated lists
Space separated lists are deprecated but this notice is not checked for.
extract_fields_from_arg_list also converts lists back to space-separated
lists causing a warning on get_scenario

Signed-off-by: Wilfried Chauveau <wilfried.chauveau@arm.com>
2024-09-12 10:03:44 +02:00
Lukasz Mrugala
345eb8e94a scripts: twister: Elevate Status Error
Status errors previously logged an error, but didn't fail the running test.
This commit changes that
 and introduces a new StatusAttributeError to use there.

One test is modified so it follows proper status form.
One test for the new error has been added.

Status errors now will properly mark the Instance as ERROR
and not run TestCases as SKIP.
This necessitated some code layout changes in runner.py

Signed-off-by: Lukasz Mrugala <lukaszx.mrugala@intel.com>
2024-09-09 09:28:37 +02:00
Anas Nashif
d145db01b4 twister: setup soc/arch roots based on module settings
Get data for various roots from modules and do not hardcode location
when retrieving list of boards in testplan.

Fixes #71761

Signed-off-by: Anas Nashif <anas.nashif@intel.com>
2024-08-30 11:46:41 -04:00
Grzegorz Chwierut
892b25aece twister: tests: Add unit test for readline method in handler
Check if lines from serial are processed one by one
in case when was received in one buffer from redline method.

Signed-off-by: Grzegorz Chwierut <grzegorz.chwierut@nordicsemi.no>
2024-08-24 07:15:17 -04:00
Grzegorz Chwierut
fcf17dd17e twister: pytest: Change the order or pytest-args
Use pytest-args parameters provided in command line after
parameters taken from yaml file. The last occurence is
considered by argparse, so parameters from command line
will be used.

Fixes: #77319

Signed-off-by: Grzegorz Chwierut <grzegorz.chwierut@nordicsemi.no>
2024-08-23 15:50:19 +01:00
Yuval Peress
bfb893a0c5 gtest: Add support for gtest 1.15.0
The gtest 1.15 has slightly different output from the original one that
was used to set up the harness. With this change we support both output
formats.

Fixes #72318

Signed-off-by: Yuval Peress <peress@google.com>
2024-08-23 09:54:41 +02:00
Michael Arnold
5ee4284320 twister: runner: j-link: use dev-id instead of SelectEmuBySN
The serial number for debugger selection over USB
can be selected with the dev-id. This change
reflects also more the workflow of west flash
with J-Link.
The usage of SelectEmuBySN breaks the support for
J-Link over IP with twister.

Signed-off-by: Michael Arnold <marnold@baumer.com>
2024-08-20 14:52:48 -04:00
Grzegorz Chwierut
7cca59c81a twister: pytest: Allow using pytest-args from command file and yaml
Extend pytest-args from configuration yaml file with args
from command line instead of overwriting.

Signed-off-by: Grzegorz Chwierut <grzegorz.chwierut@nordicsemi.no>
2024-08-19 10:00:47 -04:00
Alberto Escolar Piedras
05f5390b92 twister tests: Use native_sim instead of native_posix
native_posix is deprecated. native_sim should be used in its place.

Signed-off-by: Alberto Escolar Piedras <alberto.escolar.piedras@nordicsemi.no>
2024-08-16 09:21:08 +01:00
Dmitrii Golovanov
a4cb802d54 twister: DeviceHandler improve DUT selection
Improve DUT selection at DeviceHandler: for each DUT it counts
how many test instances have been failed on it during the current
twister execution, so the next available DUT will be chosen
ordering the eligible DUTs by less failures occured so far.

The new selection mechanism should increase chances to retry failed
tests on different DUTs, for instance to resolve ploblems when some
DUTs have connectivity or HW issues slowing down test plan execution,
or even block the execution when only one test suite runs whereas
the same first DUT candidate in the list is not working and others
were not chosen.

Signed-off-by: Dmitrii Golovanov <dmitrii.golovanov@intel.com>
2024-08-15 10:15:41 +01:00
Dmitrii Golovanov
c5e8ac7cbf twister: DeviceHandler count test failures on DUTs
Twister DeviceHandler - add test failure counter for how many
test instances have been failed on each DUT (Device Under Test)
when it executes the current test plan.
Output DUT falure counter summary at the end of Twister run.

Signed-off-by: Dmitrii Golovanov <dmitrii.golovanov@intel.com>
2024-08-15 10:15:41 +01:00
Dmitrii Golovanov
bd6e1a2da4 twister: DeviceHandler make DUT counter increment atomic
Twister DeviceHandler - make DUT use counter increment
operation atomic.

Signed-off-by: Dmitrii Golovanov <dmitrii.golovanov@intel.com>
2024-08-15 10:15:41 +01:00
Dmitrii Golovanov
44852d05ec twister: Improve DUT release at DeviceHandler
Several improvements at Twister DeviceHandler when it releases
current DUT (Device Under Test):

 - release the exact DUT which is used for the test instance
   instead of all configured DUTs which happened to have
   the same serial device configured.

 - Twister PyTtest harness plugin adjustment to the above.

 - additional debug logging to track DUT waiting/retain/release.

Signed-off-by: Dmitrii Golovanov <dmitrii.golovanov@intel.com>
2024-08-15 10:15:41 +01:00
Lukasz Mrugala
212f48c146 scripts: twister: Unify Twister Statuses
Various different Statuses were joined into a single class,
TwisterStatus. This change anticipates further streamlining
of the Twister's approach to Status.

Code guarding Twister's Properties was shortened to a
value check only.

QEMUOutputStatus was left separate, as doubts were cast
whether it should remain a status. Leaving it separate makes
its removal easier.

Signed-off-by: Lukasz Mrugala <lukaszx.mrugala@intel.com>
2024-08-12 19:23:07 -04:00
Lukasz Mrugala
6f452e81f7 scripts: twister: Remove unused statuses
TestInstanceStatus of TIMEOUT and FLASH were never really used.
They were checked for, but never assigned.

Signed-off-by: Lukasz Mrugala <lukaszx.mrugala@intel.com>
2024-08-12 19:23:07 -04:00
Lukasz Mrugala
5c6c44a247 scripts: twister: Isolate statuses into a separate class
Now statuses are not just a str that can be easily mistyped
or assigned wrong. Now they are an Enum.

Signed-off-by: Lukasz Mrugala <lukaszx.mrugala@intel.com>
2024-08-12 19:23:07 -04:00
Anas Nashif
a9babb5a83 twister: tests: use platforms that do not depend on a HAL
Do not depend on platforms that need a HAL. This should speed things up
and should resolve issues where runner did not have enough space to deal
with all HALs.
t
Signed-off-by: Anas Nashif <anas.nashif@intel.com>
2024-08-06 08:47:01 -04:00
Grzegorz Chwierut
663786d0a4 twister: Skip scanning non-ztest tests in Twister
Twister scans C-files to find testcases that are implemented
using ZTest framework. Also runs scanning of Elf files
after building. Skip scanning files if it is not required.

Signed-off-by: Grzegorz Chwierut <grzegorz.chwierut@nordicsemi.no>
2024-08-02 03:31:24 -04:00
Yuval Peress
1b51740fa3 twister: Fix gTest harness
The gTest harness asssumed that the lines end with the test name, but
some gTest implementations include the test duration in the line. Update
both the tests and regex to allow this and also avoid capturing
characters into the `test_name` that cannot be valid test name chars.

Fixes #72318

Signed-off-by: Yuval Peress <peress@google.com>
2024-07-03 08:52:13 +02:00
Gerson Fernando Budke
768b8bbca3 scripts: west robot & simulation: Fix OOT
The current version of scipts do not consider OOT boards use cases and
the tests with robot now are strict to only one robot file, which is
not realistic for real environment. This address those issues and allow
multiple testsuits at command line and lists at tests entries. It add
another test parameter to allow configure robotframework options.

Fixes: #74563

Signed-off-by: Gerson Fernando Budke <gerson.budke@ossystems.com.br>
2024-07-01 16:10:35 -04:00
Jordan Yates
243eb36b18 everywhere: reindent .overlay files with tabs
`checkpatch.pl` requires that dts sources are indented with tabs,
fix all the spaces that slipped in while checkpatch wasn't watching.

Signed-off-by: Jordan Yates <jordan@embeint.com>
2024-06-26 15:59:44 +02:00
Dmitrii Golovanov
35e313f9e5 twister: recording: Allow JSON data fields
Extend Twister Harness recording feature to allow selected data fields,
extracted from the log by a regular expression, to be parsed into JSON
objects and eventually reported in `twister.json` as `recording` list
property of the test suite.

With this extension, log records can convey layered data structures
passed from a test image as summary results, traces, statistics, etc.

This extension also allows flexible recording structure: a test image
can output different types of data records incapsulated into a fixed
set of fields, so `recording.csv` file columns are respected, whereas
some of the columns keep strings with json-encoded semi-structured data.

Signed-off-by: Dmitrii Golovanov <dmitrii.golovanov@intel.com>
2024-06-14 09:41:52 -04:00
Maciej Perkowski
da1066fa4b twister: Add sysbuild boolean to platform definitions
More complex platforms require sysbuild to use always, even for
such "simple" samples like hello_world. Such platforms can have
`sysbuild: true` entry in their board_name.yaml used by twister.
Using such entry will tell twister, that sysbuild must always be used
on a given platform.
Twister is aligned to have information about need of sysbuild at
instance (platform + suite) level (was only at suite level before).
Instance.sysbuild is true whenever a test suite or a platform requires
sysbuild.
Twister pytest unit tests are aligned with changes.

Signed-off-by: Maciej Perkowski <Maciej.Perkowski@nordicsemi.no>
2024-06-13 16:46:25 -04:00
Anas Nashif
eaafd30b56 twister: cleanup how we capture results of handlers
we use reason for a failure to indicate state and then set the status
later and reason for the failure, in case of the failure is taken from
the handler status. Clean this up by setting status and reason coming
from the handler very early, so we do not have to go through replacing
meaning later.

Signed-off-by: Anas Nashif <anas.nashif@intel.com>
2024-06-12 21:12:13 -04:00
Henrik Brix Andersen
331b404185 twister: hardwaremap: avoid exceptions when generating hardware map
Avoid exceptions when generating persistent hardware maps on Linux with no
devices available.

Signed-off-by: Henrik Brix Andersen <hebad@vestas.com>
2024-06-06 07:50:51 -04:00
Dmitrii Golovanov
4dc2a98d0c twister: Ensure serial-pty process termination
Make sure Twster DeviceHandler serial-pty process is terminated
with all its remaining children to avoid Twister hanging on it
infinitely.
The reolved issue occurs sometimes, for example when serial-pty
script is used for serial port tunneling over network.

Signed-off-by: Dmitrii Golovanov <dmitrii.golovanov@intel.com>
2024-06-05 17:38:42 -05:00
Lukasz Mrugala
5f4d330db6 scripts: twister: Do not report filtered test instances by default
Filtered testcases are removed by default from Twister tests.
Older functionality is preserved via a new Twister flag:
--report-filtered.
Old tests were adjusted and a new test for that flag added.

Signed-off-by: Lukasz Mrugala <lukaszx.mrugala@intel.com>
2024-06-05 01:38:47 -07:00
Henrik Brix Andersen
a7d55cfa13 twister: harness: pytest: list available fixtures in device config
Pass the list of supported twister fixtures for a given platform to pytest
via DeviceConfig. This allows for the pytest suites to use knowledge of the
fixtures for test suite configuration.

Signed-off-by: Henrik Brix Andersen <hebad@vestas.com>
2024-06-04 21:27:35 -05:00
Anas Nashif
d877d29913 twister: abort on execution and retun 0 on exceptions
When something goes wrong with execution, due to twister issues and
bugs, do not continue with execution and abort the process and report
and return an error code.

Fixes #72807

Signed-off-by: Anas Nashif <anas.nashif@intel.com>
2024-06-04 07:58:31 +02:00
Guennadi Liakhovetski
024bd41efb llext: xtensa: add support for the xt-clang toolchain
To build LLEXT images using the xt-clang toolchain from Cadence
linker flags have to be set similar to other toolchains. Add the
missing cmake files.

Signed-off-by: Guennadi Liakhovetski <guennadi.liakhovetski@linux.intel.com>
2024-05-29 10:49:40 -07:00