Commit graph

280 commits

Author SHA1 Message Date
Andy Ross
e05d0132fd scripts/twister: Kill qemus with SIGKILL, not SIGTERM
On qemu_cortex_a53 with icount enabled, the qemu process will spin
hard at the end of its run (common with icount), but will also ignore
the SIGTERM twister is sending to terminate the (successful!) test
run.  The resulting zombie spinning processes end up accumulating and
poisoning the test run.

Just use a hard KILL instead of trying to be a good citizen (and in
lieu of fixing the upstream qemu bug).  I can't see why this should be
a problem.

Signed-off-by: Andy Ross <andyross@google.com>
2022-08-18 10:19:19 +02:00
Enjia Mai
a2160efed3 twister: fix the hanging up on an exception happens during west flash
Fix the missing logic of setting the flash_error flag when
caught the exceptions. And we should let the serial thread
can be run first if flash error. Otherwise the twister will
hang up while an exception happens on closing serial, after
the #47820 got merged. This mostly happens on frdm_k64f.

Fixes #49086

Signed-off-by: Enjia Mai <enjia.mai@intel.com>
2022-08-17 15:24:13 -05:00
Enjia Mai
e33fd73546 soc: xtensa: add support for specifying the service port
Add specifying the port for the remote cavstool remote service.
Ex.

    west flash --remote-host {host}:{port} \
               --pty {host}:{port}

Specify the port is optional when running west.

And another major change is now we have to specify the remote log
service by --pty to make it output the log message immediately.
Previously it will output directly. Why we make this change is
because this is more close to the behavior we use west flash.

Fixes. #46865

Signed-off-by: Enjia Mai <enjia.mai@intel.com>
2022-08-16 22:51:53 -07:00
Guo Lixin
0ea346009f twister: avoid monitor serial when flash error
Currently, twister keeps on monitoring serial output until timeout
even if we have a flash error. And the error reason will be reported
as 'No Console Output(Timeout)', which is not accurate.
So add a flash_error flag to control if we need to monitor the output.

Signed-off-by: Guo Lixin <lixinx.guo@intel.com>
2022-08-13 08:52:03 -04:00
Anas Nashif
f28a8d2240 twister: fix bug retrieving subtests
During the recent overhaul, some code was left retrieving cases the old
way.

Fixes #48897

Signed-off-by: Anas Nashif <anas.nashif@intel.com>
2022-08-12 17:39:35 +02:00
Carles Cufi
a580b3d174 west: Fix handling of modules in the boards command
The boards command was not properly using the zephyr_module
functionality to obtain the board roots of all modules. Fix that by
moving the functionality required to the core zephyr_module file and
reuse it from external scripts.

Signed-off-by: Carles Cufi <carles.cufi@nordicsemi.no>
2022-08-09 16:07:51 +02:00
Chen Peng1
7f105cea87 twister: support board@revision in platform filter
support to use board@revision as platform filter when running
twister, like "twister -p nucleo_f030r8@1 ...".

Signed-off-by: Chen Peng1 <peng1.chen@intel.com>
2022-08-09 08:45:55 -04:00
Piotr Golyzniak
4484f68d02 twister: verify platform names from quarantine
Add verification of platform name from quarantine list.

Fixes: #43450

Signed-off-by: Piotr Golyzniak <piotr.golyzniak@nordicsemi.no>
2022-08-01 09:29:03 -04:00
Anas Nashif
c0317fba1f twister: handler quotes in extra_args
Escape quotes provided in extra arguments on the command line.

example: ... -x "CONFIG_COMPILER_OPT=\"-fanalyzer\""

Fixes #46382

Signed-off-by: Anas Nashif <anas.nashif@intel.com>
2022-07-29 05:58:15 -04:00
Piotr Golyzniak
7ce914b7f4 twister: verify all platform names
Fix bug in verify_platforms_existence method - make it possible to
verify all platform names from list - not only first.

Fixes: #48321

Signed-off-by: Piotr Golyzniak <piotr.golyzniak@nordicsemi.no>
2022-07-26 20:42:20 -04:00
Stephanos Ioannidis
0f9588e51f scripts: twisterlib: Fix default value for --ninja argument
This commit makes the `--ninja` argument default to `true` when neither
`-k` nor `--make` is specified (i.e. when Make generator is not
selected).

Signed-off-by: Stephanos Ioannidis <root@stephanos.io>
2022-07-26 20:17:59 -04:00
Stephanos Ioannidis
1f81d2cd97 scripts: twisterlib: Fix --short-build-path argument handling
When `--short-build-path` argument is specified to the twister, the
following error message is displayed because the Python argparse module
does not allow specifying a mutually exclusive argument as required:

  ValueError: mutually exclusive arguments must be optional

This commit removes the `required` parameter when adding the `--ninja`
argument, which is a mutually exclusive argument, and adds a manual
check to validate this condition instead.

Signed-off-by: Stephanos Ioannidis <root@stephanos.io>
2022-07-26 20:17:59 -04:00
Bartosz Bilas
3322aad03b scripts: pylib: twister: twisterlib: fix typo
s/enviornment/environment

Signed-off-by: Bartosz Bilas <b.bilas@grinn-global.com>
2022-07-19 11:41:54 -04:00
Jason Wright
c6c7dde969 scripts: better logging for --coverage-formats option
Adds additional log messages for coverage reports
beyond HTML format.

Signed-off-by: Jason Wright <jwright@synchron.com>
2022-07-18 15:03:13 -04:00
Jason Wright
60c4d20548 scripts: restrict --coverage-formats option to gcovr
Generates an error if --coverage-formats option is supplied without
coverage tool being set to gcovr.

Signed-off-by: Jason Wright <jwright@synchron.com>
2022-07-18 15:03:13 -04:00
Jason Wright
786f28f162 scripts: add --coverage-formats option for use with gcovr
Adds optional twister flag to support gcovr output report formats.

Signed-off-by: Jason Wright <jwright@synchron.com>
2022-07-18 15:03:13 -04:00
Chen Peng1
e7088743df scripts: twister: only load connected hardwares.
only load hardwares with connected is true in
hardware map files.

Signed-off-by: Chen Peng1 <peng1.chen@intel.com>
2022-07-13 06:18:08 -04:00
Chen Peng1
1d5c37ab96 script: twister: fix hardware runner_params parsing.
The specific hardware instance is already got from
device_is_available function, we don't need to
iterate all hardwares again.

Signed-off-by: Chen Peng1 <peng1.chen@intel.com>
2022-07-13 06:18:08 -04:00
Anas Nashif
a2fbb783dd twister: fix new ztest parsing
Somehow this change was dropped (rebase?), this is needed to parse new
ztest test correctly, otherwise will have many tests missing from the
plan.

Signed-off-by: Anas Nashif <anas.nashif@intel.com>
2022-07-13 10:13:28 +02:00
Anas Nashif
5f146dcaf9 twister: prep for device testing
Add option to prepare for device testing by only building and storing
the needed data that can then later be executed and run on DUT using
--test-only.

This adds --prep-artifacts-for-testing which is the same as --build-only
and --device-testing and other options that were needed to make this
operational and produce the right binaries and files.

Signed-off-by: Anas Nashif <anas.nashif@intel.com>
2022-07-13 10:13:28 +02:00
Anas Nashif
68a8613a92 twister: fix --build-only argument help
Minor fix to the --build-only option help.

Signed-off-by: Anas Nashif <anas.nashif@intel.com>
2022-07-13 10:13:28 +02:00
Anas Nashif
398e9902b8 twister: rename argument group: serial -> device
using serial as the group name is misleading, so call it 'device' which
is what all those options about.

Signed-off-by: Anas Nashif <anas.nashif@intel.com>
2022-07-13 10:13:28 +02:00
Anas Nashif
e1e43015ec twister: also parse cpp files in app root
Fix wildcard to also scan cpp files in root folder of test application.

Fixes #47220

Signed-off-by: Anas Nashif <anas.nashif@intel.com>
2022-07-12 23:24:41 -04:00
Anas Nashif
9e3d697f15 twister: fix calculation of testsuite names
We were omitting one element of the path by mistake.
Also fix some naming issues in the testsuite.

Signed-off-by: Anas Nashif <anas.nashif@intel.com>
2022-07-05 15:07:35 -04:00
Anas Nashif
1463f4bdc5 twister: rework testsuite parsing
Move testsuite setup and parsing to the testsuite class. Simplify
reading data from yaml.

Signed-off-by: Anas Nashif <anas.nashif@intel.com>
2022-07-05 12:06:27 -04:00
Anas Nashif
4789f07a8f twister: fixups to various files
Various fixups, will be folded in correct commits eventually.

Signed-off-by: Anas Nashif <anas.nashif@intel.com>
2022-07-05 12:06:27 -04:00
Anas Nashif
087f1e4a63 twister: fixed various pylint issues
Fix various pylint issues.

Signed-off-by: Anas Nashif <anas.nashif@intel.com>
2022-07-05 12:06:27 -04:00
Anas Nashif
3d1971dee3 twister: adapt and fix testsuite tests
Re-add testsuite tests, include file scanning.

Signed-off-by: Anas Nashif <anas.nashif@intel.com>
2022-07-05 12:06:27 -04:00
Anas Nashif
1a6948ac26 twister: move file scanning out of class
This is all static functions that can maintained outside of the class
and called only in one place, so move this to the testsuite file.

Signed-off-by: Anas Nashif <anas.nashif@intel.com>
2022-07-05 12:06:27 -04:00
Anas Nashif
9437e6d963 twister: fix twister testsuite
Make testsuite work again.

Signed-off-by: Anas Nashif <anas.nashif@intel.com>
2022-07-05 12:06:27 -04:00
Anas Nashif
08caa71de7 twister: testplan: get options directly
Get options from passed environment class instead of using command line
options to the init function.

Signed-off-by: Anas Nashif <anas.nashif@intel.com>
2022-07-05 12:06:27 -04:00
Anas Nashif
91b2cce64e twister: fix subset processing
Process subsets was done in the wrong step, so make it apply and
generarte subsets correctly.

Signed-off-by: Anas Nashif <anas.nashif@intel.com>
2022-07-05 12:06:27 -04:00
Anas Nashif
79d7d0188b twister: make ninja the default
Using ninja is up to 50% faster, make it the default and add an option
to build with Make.

Signed-off-by: Anas Nashif <anas.nashif@intel.com>
2022-07-05 12:06:27 -04:00
Anas Nashif
7149173627 twister: simplify handler options
Depend on global options instead of passing them around for each class.

Signed-off-by: Anas Nashif <anas.nashif@intel.com>
2022-07-05 12:06:27 -04:00
Anas Nashif
f2b5a6674a twister: move harness class to class folder
harness class moved along size other classes.

Signed-off-by: Anas Nashif <anas.nashif@intel.com>
2022-07-05 12:06:27 -04:00
Anas Nashif
57dbaa3da3 twister: move more code from main script
Move code from twister script into respective classes.

Signed-off-by: Anas Nashif <anas.nashif@intel.com>
2022-07-05 12:06:27 -04:00
Anas Nashif
f539f1dc48 twister: move size report into size calc class
Move from main script into size_calc class.

Signed-off-by: Anas Nashif <anas.nashif@intel.com>
2022-07-05 12:06:27 -04:00
Anas Nashif
806722cdbf twister: move device testing code
Move device handling code into hardware map class.

Signed-off-by: Anas Nashif <anas.nashif@intel.com>
2022-07-05 12:06:27 -04:00
Anas Nashif
f7d501447d twister: move cleanup and classification of code
Move code out of the main script and use global options.

Signed-off-by: Anas Nashif <anas.nashif@intel.com>
2022-07-05 12:06:27 -04:00
Anas Nashif
396b8574eb twister: testplan: move calls to json loading/filter into class
Move all code related to instance filtering and loading into the class
instead of having it in the main script. Simplify option passing.

Signed-off-by: Anas Nashif <anas.nashif@intel.com>
2022-07-05 12:06:27 -04:00
Anas Nashif
f85d1eb4e1 twister: rework testplan and main twister script
Move all testplan related code to the testplan class and cleanup main
twister script.

Signed-off-by: Anas Nashif <anas.nashif@intel.com>
2022-07-05 12:06:27 -04:00
Anas Nashif
b8735b3300 twister: more cleanup to option passing
Use global options rather than passing them individually.

Signed-off-by: Anas Nashif <anas.nashif@intel.com>
2022-07-05 12:06:27 -04:00
Anas Nashif
facc685ae9 twister: move runner code into runner class
Move all code related to test execution into runner class.

Signed-off-by: Anas Nashif <anas.nashif@intel.com>
2022-07-05 12:06:27 -04:00
Anas Nashif
20f257a97d twister: move execution code out of testplan
Move all code related to execution into runner class.

Signed-off-by: Anas Nashif <anas.nashif@intel.com>
2022-07-05 12:06:27 -04:00
Anas Nashif
d43e03e19e twister: isolate testplan class
Move testplan class into own file and rename the original twisterlib.

Signed-off-by: Anas Nashif <anas.nashif@intel.com>
2022-07-05 12:06:27 -04:00
Anas Nashif
96b9ff6b5e twister: move runner classes out
Move runners into own file.

Signed-off-by: Anas Nashif <anas.nashif@intel.com>
2022-07-05 12:06:27 -04:00
Anas Nashif
e5c2e797c9 twister: move test instance class out
Move test instance class into own file.

Signed-off-by: Anas Nashif <anas.nashif@intel.com>
2022-07-05 12:06:27 -04:00
Anas Nashif
adfa6c7777 twister: move coverage classes out
Move coverage related classes out into own file.

Signed-off-by: Anas Nashif <anas.nashif@intel.com>
2022-07-05 12:06:27 -04:00
Anas Nashif
78d9dab7f3 twister: cleanup imports and move size classes out
More python import optimizations and cleanup and move size_calc class
out into own file.

Signed-off-by: Anas Nashif <anas.nashif@intel.com>
2022-07-05 12:06:27 -04:00
Anas Nashif
75eee4b1f3 twister: move platform class out
Move both platform and config parser into own file.

Signed-off-by: Anas Nashif <anas.nashif@intel.com>
2022-07-05 12:06:27 -04:00