Detect duplicate TestSuites on load, and raise error when it happens
with `--no-detailed-test-id` option which shortens TestSuite name
excluding the test project path prefix, thus increasing chances
for duplicates. Without this check, only the last duplicated
test configuration was selected while others silently ignored.
Signed-off-by: Dmitrii Golovanov <dmitrii.golovanov@intel.com>
Fix duplicate test scenario error reporting to show paths to all
twister.yaml configuration files where these duplicates were found.
Signed-off-by: Dmitrii Golovanov <dmitrii.golovanov@intel.com>
This change introduces the ability in twister to select which
emulation/simulation tool to use on the command line.
If none is specified, it will select the first in the list.
Signed-off-by: Wilfried Chauveau <wilfried.chauveau@arm.com>
Fix Ztest test function name extraction from ELF symbols
for C++ compiled binaries where symbol names need additional
'demangling' to match with corresponding test names.
The `c++filt` utility (part of binutils) is called for
demangling when it is needed.
Twister test suite extension and adjustment.
Signed-off-by: Dmitrii Golovanov <dmitrii.golovanov@intel.com>
Fix a problem of Ztest suite names not taken into account by Twister
to identify a TestCase, so in some situations a Ztest test's status
was not assigned to the proper TestCase and it remains 'None'
whereas the actual status value lost, eventually the resulting total
execution counters not correct.
The issue was observed in these situations:
* Ztest application with multiple test suites having same test names.
* Ztest suite is 'skipped' entirely on execution with all its tests.
The proposed solution extends Twister test case name for Ztest to
include Ztest suite name, so the resulting identifier looks like:
`<test_scenario_name>.<ztest_suite_name>.<ztest_name>`
The above naming scheme now requires ztest_suite_name part to be
provided for `--sub-test` command line option.
Testcase identifiers in twister.json and testplan.json will also
include ztest_suite_name component.
The Twister Ztest(Test) Harness is improved to track all state changes
known from the test application's log for Ztest suites and test cases,
so now it parses log output from a Ztest application more scurpulously.
Regular expressions to match log records are extended and optimized
to compile them only once and, in some cases, fixed (suite summary).
Signed-off-by: Dmitrii Golovanov <dmitrii.golovanov@intel.com>
Extended hardware map to share a single board between variants.
To run tests for different variants on the same board
without re-configuring the hardware map file for each variant,
one can use a `platform` atribute as a list of names.
Signed-off-by: Grzegorz Chwierut <grzegorz.chwierut@nordicsemi.no>
Fix wrong reporting about where testcases were executed.
We have:
INFO - 1130 of 1130 executed test cases passed (100.00%) on 0 out of
total 860 platforms (0.00%).
which is obviously wrong.
Signed-off-by: Anas Nashif <anas.nashif@intel.com>
suite stats were not correct, a mixup between skipped and filtered
suites was leading to inconsistent numbers. This is now fixed.
Signed-off-by: Anas Nashif <anas.nashif@intel.com>
Fix Twister TestCase statuses left not assigned ('NONE') in these 'NOTRUN'
situations:
* a test suite has `integration platform` which is not available to run.
* `--cmake-only` execution.
Signed-off-by: Dmitrii Golovanov <dmitrii.golovanov@intel.com>
When using the --build-only into --test-only
Twister setup, NOTRUN statuses were not properly rerun.
Now they are properly run again if runnable.
Signed-off-by: Lukasz Mrugala <lukaszx.mrugala@intel.com>
ExecutionCounter has been expanded and now hold i.a.
more information on the statuses of TestCases.
This information is now incorporated in relevant summaries
- runner.py and reports.py.
Layout of those was changed to present that
and previous information in a clear and concise way.
TestInstance execution counter now is more intuitive.
Instances filtered out before running are no longer included there.
Retries now properly reset the counter.
TestCases with None and other incorrect final statuses
are logged as errors, but do not
exit Twister with a nonzero exit code.
This is because None statuses, although incorrect,
are currently common.
Inconsistent spacing in ERROR and FAILED fixed.
Signed-off-by: Lukasz Mrugala <lukaszx.mrugala@intel.com>
scripts: Dmitri fix
Fix of a problem noticed by Dmitri
Removed unnecessary additional spaces when
printing FAILED and ERROR status.
Now TwisterStatus.get_color is used more.
Signed-off-by: Lukasz Mrugala <lukaszx.mrugala@intel.com>
As per its creators, CMake is written with a capital "M".
The initial reason for this change is that I want Twister to print
"ERROR : CMake build failure" instead of "ERROR : Cmake build failure".
Signed-off-by: Reto Schneider <reto.schneider@husqvarnagroup.com>
if alias or shorthand name is provided on the command line, we need
convert this to complete target name for the filters to work.
Fixes#80332
Signed-off-by: Anas Nashif <anas.nashif@intel.com>
Adds a new NOTRUN status, which indicates
that a test was successfully built, but
not run on account of being not
runnable in given test instance.
Signed-off-by: Lukasz Mrugala <lukaszx.mrugala@intel.com>
Fix trailing `\\r\\n` (escaped CR/LF) didn't cut off because of rstrip()
removed by #58338, so the CR/LF suffix was never found as the actual line
end was `\\r\\n\n`.
Add ANSI code sequence to `test_handlers` Twister unit test.
Signed-off-by: Dmitrii Golovanov <dmitrii.golovanov@intel.com>
Separate the pickled EDT generation from the C-Macro header
generation in gen_defines.py to have a more clear responsibility
of the scripts in the DTS parsing process.
Signed-off-by: Benedikt Schmidt <benedikt.schmidt@embedded-solutions.at>
native_posix is now deprecated.
Building this sample in both native_sim and native_posix does not
improve coverage for the sample or subsystem but doubles CI time.
As anyhow native_posix will be removed all together in
2 releases, let's remove it already for this sample.
Signed-off-by: Alberto Escolar Piedras <alberto.escolar.piedras@nordicsemi.no>
Currently, debug logging in the console and verbosity
are tightly coupled - verbosity of level 2 and higher
enables logging at the debug level.
This change introduces a separate Twister flag
responsible for controlling the debug logging,
while leaving the rest of verbosity unchanged.
This allows for controlling the verbosity on
both logging levels, according to one's needs.
Signed-off-by: Lukasz Mrugala <lukaszx.mrugala@intel.com>
When at verbosity 1, we print out the status of TestInstances.
This makes it harder to notice changes at TestCase level,
which require perusing the logs.
This adds TestCase status and reason printing
if verbosity level is 2 or more.
Reason printing is suppressed if the reason is empty or None.
Signed-off-by: Lukasz Mrugala <lukaszx.mrugala@intel.com>
Running sysbuilt tests fails because of missing "run" target.
This adds the default domain context to the command.
Signed-off-by: Wilfried Chauveau <wilfried.chauveau@arm.com>
Some dict.get() calls did not use a TwisterStatus
as a default value, thus using a NoneType where
TwisterStatus should appear.
Signed-off-by: Lukasz Mrugala <lukaszx.mrugala@intel.com>
This changes how some arguments are set in the `Handler`s.
`options`, `generator_cmd` and `suite_name_check` are now passed as
arguments to the constructor rather than injected from an other module.
Signed-off-by: Wilfried Chauveau <wilfried.chauveau@arm.com>
CONF_FILE, DTC_OVERLAY_FILE and OVERLAY_CONFIG are deprecated but still
used by the tests causing warnings when running them.
This adds a test_data specific to validate the emission of the warning,
and removes the offending args from the other test_data files.
Signed-off-by: Wilfried Chauveau <wilfried.chauveau@arm.com>
Space separated lists are deprecated but this notice is not checked for.
extract_fields_from_arg_list also converts lists back to space-separated
lists causing a warning on get_scenario
Signed-off-by: Wilfried Chauveau <wilfried.chauveau@arm.com>
Status errors previously logged an error, but didn't fail the running test.
This commit changes that
and introduces a new StatusAttributeError to use there.
One test is modified so it follows proper status form.
One test for the new error has been added.
Status errors now will properly mark the Instance as ERROR
and not run TestCases as SKIP.
This necessitated some code layout changes in runner.py
Signed-off-by: Lukasz Mrugala <lukaszx.mrugala@intel.com>
Get data for various roots from modules and do not hardcode location
when retrieving list of boards in testplan.
Fixes#71761
Signed-off-by: Anas Nashif <anas.nashif@intel.com>
Check if lines from serial are processed one by one
in case when was received in one buffer from redline method.
Signed-off-by: Grzegorz Chwierut <grzegorz.chwierut@nordicsemi.no>
Use pytest-args parameters provided in command line after
parameters taken from yaml file. The last occurence is
considered by argparse, so parameters from command line
will be used.
Fixes: #77319
Signed-off-by: Grzegorz Chwierut <grzegorz.chwierut@nordicsemi.no>
The gtest 1.15 has slightly different output from the original one that
was used to set up the harness. With this change we support both output
formats.
Fixes#72318
Signed-off-by: Yuval Peress <peress@google.com>
The serial number for debugger selection over USB
can be selected with the dev-id. This change
reflects also more the workflow of west flash
with J-Link.
The usage of SelectEmuBySN breaks the support for
J-Link over IP with twister.
Signed-off-by: Michael Arnold <marnold@baumer.com>
Extend pytest-args from configuration yaml file with args
from command line instead of overwriting.
Signed-off-by: Grzegorz Chwierut <grzegorz.chwierut@nordicsemi.no>
Improve DUT selection at DeviceHandler: for each DUT it counts
how many test instances have been failed on it during the current
twister execution, so the next available DUT will be chosen
ordering the eligible DUTs by less failures occured so far.
The new selection mechanism should increase chances to retry failed
tests on different DUTs, for instance to resolve ploblems when some
DUTs have connectivity or HW issues slowing down test plan execution,
or even block the execution when only one test suite runs whereas
the same first DUT candidate in the list is not working and others
were not chosen.
Signed-off-by: Dmitrii Golovanov <dmitrii.golovanov@intel.com>
Twister DeviceHandler - add test failure counter for how many
test instances have been failed on each DUT (Device Under Test)
when it executes the current test plan.
Output DUT falure counter summary at the end of Twister run.
Signed-off-by: Dmitrii Golovanov <dmitrii.golovanov@intel.com>
