this fixes ruff linting error UP006,
where old typing types are still used
instead of newer, PEP585-compliant ones.
Signed-off-by: Lukasz Mrugala <lukaszx.mrugala@intel.com>
This fixes ruff linting error E741,
where a variable name is ambiguous
or hard to read (e.g. l).
Signed-off-by: Lukasz Mrugala <lukaszx.mrugala@intel.com>
This fixes ruff linting error SIM1101,
where a builtin, like any() or all()
has been reimplemented.
Signed-off-by: Lukasz Mrugala <lukaszx.mrugala@intel.com>
This fixes ruff linting error SIM114,
where there are multiple arms in an
if-elif-else construct with the same result.
Signed-off-by: Lukasz Mrugala <lukaszx.mrugala@intel.com>
This fixes ruff linting error SIM201,
where not is used instead of a more
appropriate comparison operator.
SIM202 exclusion was removed, as it
didn't apply to anything.
Signed-off-by: Lukasz Mrugala <lukaszx.mrugala@intel.com>
This fixes ruff linting error SIM300,
where constant values are on the left
side of an expression.
Signed-off-by: Lukasz Mrugala <lukaszx.mrugala@intel.com>
This fixes the ruff linting error SIM401,
where if-else construction is used instead
of dict.get().
Signed-off-by: Lukasz Mrugala <lukaszx.mrugala@intel.com>
This fixes errors E402 - module import
not at top of file.
One error in testplan.py left in,
as it is a part of a obscure 2020 fix
that would need to be re-tested.
Signed-off-by: Lukasz Mrugala <lukaszx.mrugala@intel.com>
#79174 incorectly interpreted the condition. If no simulator is found
if cannot be "renode" and therefore robot should be considered unsupported.
Signed-off-by: Wilfried Chauveau <wilfried.chauveau@arm.com>
add vendor_allow and vendor_exclude to the list of filters. The vendor
data is maintained in the board.yml files.
Signed-off-by: Anas Nashif <anas.nashif@intel.com>
Detect duplicate TestSuites on load, and raise error when it happens
with `--no-detailed-test-id` option which shortens TestSuite name
excluding the test project path prefix, thus increasing chances
for duplicates. Without this check, only the last duplicated
test configuration was selected while others silently ignored.
Signed-off-by: Dmitrii Golovanov <dmitrii.golovanov@intel.com>
Fix duplicate test scenario error reporting to show paths to all
twister.yaml configuration files where these duplicates were found.
Signed-off-by: Dmitrii Golovanov <dmitrii.golovanov@intel.com>
This change introduces the ability in twister to select which
emulation/simulation tool to use on the command line.
If none is specified, it will select the first in the list.
Signed-off-by: Wilfried Chauveau <wilfried.chauveau@arm.com>
Fix Ztest test function name extraction from ELF symbols
for C++ compiled binaries where symbol names need additional
'demangling' to match with corresponding test names.
The `c++filt` utility (part of binutils) is called for
demangling when it is needed.
Twister test suite extension and adjustment.
Signed-off-by: Dmitrii Golovanov <dmitrii.golovanov@intel.com>
Fix a problem of Ztest suite names not taken into account by Twister
to identify a TestCase, so in some situations a Ztest test's status
was not assigned to the proper TestCase and it remains 'None'
whereas the actual status value lost, eventually the resulting total
execution counters not correct.
The issue was observed in these situations:
* Ztest application with multiple test suites having same test names.
* Ztest suite is 'skipped' entirely on execution with all its tests.
The proposed solution extends Twister test case name for Ztest to
include Ztest suite name, so the resulting identifier looks like:
`<test_scenario_name>.<ztest_suite_name>.<ztest_name>`
The above naming scheme now requires ztest_suite_name part to be
provided for `--sub-test` command line option.
Testcase identifiers in twister.json and testplan.json will also
include ztest_suite_name component.
The Twister Ztest(Test) Harness is improved to track all state changes
known from the test application's log for Ztest suites and test cases,
so now it parses log output from a Ztest application more scurpulously.
Regular expressions to match log records are extended and optimized
to compile them only once and, in some cases, fixed (suite summary).
Signed-off-by: Dmitrii Golovanov <dmitrii.golovanov@intel.com>
Extended hardware map to share a single board between variants.
To run tests for different variants on the same board
without re-configuring the hardware map file for each variant,
one can use a `platform` atribute as a list of names.
Signed-off-by: Grzegorz Chwierut <grzegorz.chwierut@nordicsemi.no>
Fix wrong reporting about where testcases were executed.
We have:
INFO - 1130 of 1130 executed test cases passed (100.00%) on 0 out of
total 860 platforms (0.00%).
which is obviously wrong.
Signed-off-by: Anas Nashif <anas.nashif@intel.com>
suite stats were not correct, a mixup between skipped and filtered
suites was leading to inconsistent numbers. This is now fixed.
Signed-off-by: Anas Nashif <anas.nashif@intel.com>
Do not report status issues as errors, very confusing and developer end
up looking at the wrong thing, instead, treat those as warnings and
count them and report them at the end.
Signed-off-by: Anas Nashif <anas.nashif@intel.com>
It is confusing to report filtered testcases as testcases that were
selected but not exexuted. If they are filtered, then there should not
be considered as selected.
Signed-off-by: Anas Nashif <anas.nashif@intel.com>
custom simulator needs an executable defined, or else we will have tests
marked runnable with nothing to run them which would result in an
error/warning.
Signed-off-by: Anas Nashif <anas.nashif@intel.com>
Fix Twister TestCase statuses left not assigned ('NONE') in these 'NOTRUN'
situations:
* a test suite has `integration platform` which is not available to run.
* `--cmake-only` execution.
Signed-off-by: Dmitrii Golovanov <dmitrii.golovanov@intel.com>
Twister shall now verify that the user does not
use an obsolete Python version. If user's Python
is deemed too old, it will raise a relevant error.
This check will also happen when running Twister
via west.
Signed-off-by: Lukasz Mrugala <lukaszx.mrugala@intel.com>
When using the --build-only into --test-only
Twister setup, NOTRUN statuses were not properly rerun.
Now they are properly run again if runnable.
Signed-off-by: Lukasz Mrugala <lukaszx.mrugala@intel.com>
ExecutionCounter has been expanded and now hold i.a.
more information on the statuses of TestCases.
This information is now incorporated in relevant summaries
- runner.py and reports.py.
Layout of those was changed to present that
and previous information in a clear and concise way.
TestInstance execution counter now is more intuitive.
Instances filtered out before running are no longer included there.
Retries now properly reset the counter.
TestCases with None and other incorrect final statuses
are logged as errors, but do not
exit Twister with a nonzero exit code.
This is because None statuses, although incorrect,
are currently common.
Inconsistent spacing in ERROR and FAILED fixed.
Signed-off-by: Lukasz Mrugala <lukaszx.mrugala@intel.com>
scripts: Dmitri fix
Fix of a problem noticed by Dmitri
Removed unnecessary additional spaces when
printing FAILED and ERROR status.
Now TwisterStatus.get_color is used more.
Signed-off-by: Lukasz Mrugala <lukaszx.mrugala@intel.com>
As per its creators, CMake is written with a capital "M".
The initial reason for this change is that I want Twister to print
"ERROR : CMake build failure" instead of "ERROR : Cmake build failure".
Signed-off-by: Reto Schneider <reto.schneider@husqvarnagroup.com>
Set soc_root using Path to avoid wrong generated path in the list of
soc_roots and other roots read from module.yml file
Fixes#80531
Signed-off-by: Anas Nashif <anas.nashif@intel.com>
Cleanup notrun tests with the -M option. This is used in the weekly
build and fails because devices will run out of space.
Signed-off-by: Anas Nashif <anas.nashif@intel.com>
Fixes: #69548
Support extending an existing board with new board variants.
This commit introduces the following changes to allow a board to be
extended out-of-tree.
The board yaml schema is extended to support an extend field which
will be used to identify the board to be extended.
A board 'plank' can be extended like this:
> board:
> extend: plank
> variants:
> - name: ext
> qualifier: soc1
For the rest of the build system this means that there is no longer a
single board directory.
The existing CMake variable BOARD_DIR is kept and reference the
directory which defines the board.
A new CMake variable BOARD_DIRECTORIES provides a list of all
directories which defines board targets for the board.
This means the directory which defines the board as well as all
directories that extends the board.
Signed-off-by: Torsten Rasmussen <Torsten.Rasmussen@nordicsemi.no>
if alias or shorthand name is provided on the command line, we need
convert this to complete target name for the filters to work.
Fixes#80332
Signed-off-by: Anas Nashif <anas.nashif@intel.com>
--device-testing implies runnable tests, this was the case before the
rework and was missed in one spot, so instead do that directly in the
argument parser.
Fixes#80428
Signed-off-by: Anas Nashif <anas.nashif@intel.com>
