Test suites names are not being handled uniformly for tests not
in zephyr tree. Their names depend on -T arg used in twister's CLI.
The newly added options allow to select if twister should add paths
to suite names. This is needed if test plans are to be used for tests
outside of zephyr tree.
Signed-off-by: Maciej Perkowski <Maciej.Perkowski@nordicsemi.no>
Domain module was not yet covered by explicit unit tests.
This commit adds a new test file to cover that gap.
It achieves 100% coverage at the time of creation.
Signed-off-by: Lukasz Mrugala <lukaszx.mrugala@intel.com>
Twister allows us to control maximum execution time for each test
with timeout value in test's .yaml configuration and
platform level timeout multiplier which allows us to tweak
timeout value for specific platform.
However, sometimes we want to additionally adjust tests timeouts
when running twister. This is especially useful in case of
simulation platform as simulation time may depend on the host
speed & load, we may select different (i.e. cycle accurate but
slower one) simulation method, etc...
Let's introduce global (twister-level) timeout multiplier option.
Signed-off-by: Eugeniy Paltsev <Eugeniy.Paltsev@synopsys.com>
Signed-off-by: Evgeniy Paltsev <PaltsevEvgeniy@gmail.com>
There is clearly an issue with current timeout value
timeout into mocked_instance as this value is used only
once before this mock happen, so this mock is actually useless
and we always use default value. Fix that.
Signed-off-by: Eugeniy Paltsev <Eugeniy.Paltsev@synopsys.com>
Signed-off-by: Evgeniy Paltsev <PaltsevEvgeniy@gmail.com>
Added an exception to scl.py module, to not process an empty
yaml file, because of error from pykwalify. New exception
is handled when processing quarantine files, to allow
an empty file.
Signed-off-by: Grzegorz Chwierut <grzegorz.chwierut@nordicsemi.no>
After the handler.py module has been split up more,
we can update the relevant tests.
100% coverage achieved.
Signed-off-by: Lukasz Mrugala <lukaszx.mrugala@intel.com>
Expansion of the tests related to handlers.py
Achieves over 85% coverage.
Implemented most of golowanow's suggestions.
Signed-off-by: Lukasz Mrugala <lukaszx.mrugala@intel.com>
Allow to specify a list of pytest directories, files or subtests
with pytest_root keyword in test yaml.
Signed-off-by: Grzegorz Chwierut <grzegorz.chwierut@nordicsemi.no>
A commit breaking tests slipped past CI checks.
This commit fixes broken tests in test_testsuite.py.
Signed-off-by: Lukasz Mrugala <lukaszx.mrugala@intel.com>
Add support handling quotes for configuration options in extra args by
escaping them properly instead of removing the quotes altogether. For
other options in extra_args quotes are removes as usual.
Add similar support in west build command also.
Add a unit test to check this functionality.
Signed-off-by: Chaitanya Tata <Chaitanya.Tata@nordicsemi.no>
This should increase the coverage of the testsuite.py module
up to 100%.
Some tests have been commented out, as the code tested there is
incorrect. Thus the real increase is 99%.
Signed-off-by: Lukasz Mrugala <lukaszx.mrugala@intel.com>
In addition to the white-box testing and unit tests,
we would like to have some black-box testing for the
console commands of twister.
This serves as a Proof of Concept for future expansion.
Special dummy tests are made for blackbox testing.
Signed-off-by: Lukasz Mrugala <lukaszx.mrugala@intel.com>
While the code already includes some provisions to allow running outside
of a git checkout, attempting that right now causes a failure to generate
the XML report, as the JSON report creates a null `zephyr_version` field.
The reason for that is that the original code doesn't set `self.version`
when the subprocess returns a non-zero status (and also doesn't log.)
Instead of having to set `self.version` for all failure cases, initialize
it to the failure state, and log if it hasn't changed from that.
Signed-off-by: Diego Elio Pettenò <flameeyes@meta.com>
If size is given then twister prnits the size information for the ELF
but returns one which if used in automation fails the command.
Return zero as the command is successful. Also, fix the unit test case.
Signed-off-by: Chaitanya Tata <Chaitanya.Tata@nordicsemi.no>
As we aim to enhance our test coverage, environment module
is a target-rich environment. Some errors in the original
module were discovered and tests regarding these cases were
commented out until they are fixed.
Signed-off-by: Lukasz Mrugala <lukaszx.mrugala@intel.com>
Here we achieve 100% coverage for the quarantine module,
thanks to unit tests for QuarantineElement and QuarantineData dataclasses.
Implemented PR suggestions of gchwier.
Signed-off-by: Lukasz Mrugala <lukaszx.mrugala@intel.com>
Extended parsing of report.xml file - produced by pytest-twister-harness
plugin. Now testcases are properly extracted and added to twister
report. Added unit tests.
Signed-off-by: Grzegorz Chwierut <grzegorz.chwierut@nordicsemi.no>
testinstance.py contains a string literal comparison to the
word 'platform', which has been misspelled as 'plaform'.
The change is verified by a test expansion.
Signed-off-by: Lukasz Mrugala <lukaszx.mrugala@intel.com>
Current implementation of ConfigurationError causes
an exception to be thrown.
It is because a PosixPath is being concatenated with an str with a '+'
operator, which is not permitted.
The path in question is now cast to an str.
Additional test has been added to test_twister.py to track regressions.
Signed-off-by: Lukasz Mrugala <lukaszx.mrugala@intel.com>
Some 3P logic (Pigweed for example) is already heavily invested in
gTest as a testing framework. Adding the `gtest` harness enables
running the existing 3P tests on various Zephyr platforms and configs.
Signed-off-by: Yuval Peress <peress@google.com>
Tests dedicated for choosing binaries basing on platform.binaries and
runners.yaml file. Additional tests for paths sanitizing.
Signed-off-by: Piotr Golyzniak <piotr.golyzniak@nordicsemi.no>
Add support test levels and the ability to assign a specific test to one
or more levels. Using command line options of twister it is then possible
to select a level and just execute the tests included in this level.
Additionally, a test configuration allows definiing level
dependencies and additional inclusion of tests into a specific level if
the test itself does not have this information already.
In the configuration file you can include complete components using
regular expressions and you can specify which test level to import from
the same file, making management of levels easier.
To help with testing outside of upstream CI infrastructure, additional
options are available in the configuration file, which can be hosted
locally. As of now, those options are available:
- Ability to ignore default platforms as defined in board definitions
(Those are mostly emulation platforms used to run tests in upstream
CI)
- Option to specify your own list of default platforms overriding what
upstream defines.
- Ability to override build_onl_all options used in some testscases.
This will treat tests or sample as any other just build for default
platforms you specify in the configuation file or on the command line.
Signed-off-by: Anas Nashif <anas.nashif@intel.com>
Twister allows filtering based on kconfigs and dts, however the
filtration is a part of the cmake stage, i.e. the stage has to pass
first and then twister checks if required properties are available.
This causes problems, when the full cmake stage is unable to pass.
If so, other filtration methods had to be used, e.g. platform_allow.
The commit modifies the twister workflow:
if a test defines filters based on kconfig/dts first a cmake package
helper script is used to extract dt and/or kconfigs and if the
conditions are fulfilled it proceeds to a regular cmake stage.
If not, test is skipped.
Co-authored-by: Daniel DeGrasse <daniel.degrasse@nxp.com>
Signed-off-by: Maciej Perkowski <Maciej.Perkowski@nordicsemi.no>
This is an implementation of issue #48334 and adds support for
specifying additional config and device tree overlays through fields in
the testcase.yaml file, which is more readable than having to cram these
in to `extra_args`.
Consider this example which shows the original and new ways to add
config and DT overlays:
```
common:
extra_args: "CONF_FILE=a.conf;b.conf
DTC_OVERLAY_FILE=w.overlay;x.overlay OVERLAY_CONFIG=e.conf
UNRELATED=abc"
tests:
utilities.base64:
extra_conf_files:
- "c.conf"
- "d.conf"
extra_overlay_confs:
- "extra_overlay.conf"
extra_dtc_overlay_files:
- "y.overlay"
- "z.overlay"
extra_configs:
- CONFIG_SAMPLE=y
tags: base64
type: unit
```
The new fields are `extra_conf_files`, `extra_overlay_confs,
`extra_dtc_overlay_files`. Files specified in these sections are appended
to any `CONF_FILE`, `OVERLAY_CONFIG`, or `DTC_OVERLAY_FILE` fields in
`extra_args`, causing the following args being passed in to
`self.run_cmake` at `runner.py:850`:
```
['-DUNRELATED=abc',
'-DCONF_FILE=a.conf;b.conf;c.conf;d.conf',
'-DDTC_OVERLAY_FILE=w.overlay;x.overlay;y.overlay;z.overlay',
'-DOVERLAY_CONFIG=e.conf extra_overlay.conf '
'<build_dir>/twister/testsuite_extra.conf']
```
These fields can be used in the common or scenario-specific YAML
sections and will be merged in order of least to most specific:
1. config files extracted from common's extra_args
2. files listed in common's {extra_conf_files or extra_overlay_confs
or extra_dtc_overlay_files}
3. config files extracted from test scenario's extra_args
4. files listed in test scenario's {extra_conf_files or
extra_overlay_confs or extra_dtc_overlay_files}
Specifying these files in extra_args now triggers a deprecation warning,
as the direct YAML fields are preferred for readability. They will still
function for now but support will be dropped in the future. One
testcase.yaml
(`zephyr/tests/cmake/overlays/var_expansions/testcase.yaml`) is
converted to use the new fields. A follow-up PR will convert the
remaining files to the new format.
Signed-off-by: Tristan Honscheid <honscheid@google.com>
This reverts commit 21eb27c5c0.
This change has been a source of much trouble and ends up preventing us
from tracking test results across tree. It uses the repo name as the
namespace, and that is not always the same and does not have to be
called 'zephyr', depending on where you are running and in which
environment.
Signed-off-by: Anas Nashif <anas.nashif@intel.com>
Fix all thruthy errors detected by yamllint:
yamllint -f parsable -c .yamllint $( find -regex '.*\.y[a]*ml' ) | \
grep '(truthy)'
This only accepts true/false for boolean properties. Seems like python
takes all sort of formats:
https://github.com/yaml/pyyaml/blob/master/lib/yaml/constructor.py#L224-L235
But the current specs only mention "true" or "false"
https://yaml.org/spec/1.2.2/#10212-boolean
Which is the standard yamllint config.
Excluding codeconv and workflow files, as some are using yes/no instead
in the respective documentation.
Signed-off-by: Fabio Baltieri <fabiobaltieri@google.com>
Add test cases verifing that test id for external and module tests is
not --testsuite-root dependant.
Signed-off-by: Maciej Perkowski <Maciej.Perkowski@nordicsemi.no>
With the patch a relative path between a test and west's project top
directory is always calculated instead of being relative to zephyr
base. The test names and results structur change by addition of one
level. This way the names/structure conventon for tests from zephyr and
modules is alligned.
For external tests ".." parts are replaced with "external".
This allows to maintain unified folder structure also for results of
external tests.
Without this patch the folder/name allignment is lost for
external and module test becaming depandant on a way how
twister was called.
.
Signed-off-by: Maciej Perkowski <Maciej.Perkowski@nordicsemi.no>
From docs "If a component is an absolute path, all previous
components are thrown away and joining continues from the
absolute path component." This commit fixes the args and
corrects the expected value.
https://docs.python.org/3/library/os.path.html#os.path.join
Signed-off-by: Maciej Perkowski <Maciej.Perkowski@nordicsemi.no>
