twister: docs: Update twister docs with quarantine
Docs updated with changes proposed in #52179 Signed-off-by: Grzegorz Chwierut <grzegorz.chwierut@nordicsemi.no>
This commit is contained in:
parent
6809c05d09
commit
0a7a61f15c
7 changed files with 27 additions and 17 deletions
|
@ -902,14 +902,15 @@ using an external J-Link probe. The "probe_id" keyword overrides the
|
|||
Quarantine
|
||||
++++++++++
|
||||
|
||||
Twister allows using user-defined yaml files defining the list of tests to be put
|
||||
under quarantine. Such tests will be skipped and marked accordingly in the output
|
||||
reports. This feature is especially useful when running larger test suits, where
|
||||
a failure of one test can affect the execution of other tests (e.g. putting the
|
||||
physical board in a corrupted state).
|
||||
Twister allows user to provide onfiguration files defining a list of tests or
|
||||
platforms to be put under quarantine. Such tests will be skipped and marked
|
||||
accordingly in the output reports. This feature is especially useful when
|
||||
running larger test suits, where a failure of one test can affect the execution
|
||||
of other tests (e.g. putting the physical board in a corrupted state).
|
||||
|
||||
To use the quarantine feature one has to add the argument
|
||||
``--quarantine-list <PATH_TO_QUARANTINE_YAML>`` to a twister call.
|
||||
Multiple quarantine files can be used.
|
||||
The current status of tests on the quarantine list can also be verified by adding
|
||||
``--quarantine-verify`` to the above argument. This will make twister skip all tests
|
||||
which are not on the given list.
|
||||
|
@ -920,22 +921,33 @@ to put under quarantine. In addition, an optional entry "comment" can be used, w
|
|||
some more details can be given (e.g. link to a reported issue). These comments will also
|
||||
be added to the output reports.
|
||||
|
||||
When quarantining a class of tests or many scenarios in a single testsuite or
|
||||
when dealing with multiple issues within a subsystem, it is possible to use
|
||||
regular expressions, for example, **kernel.*** would quarantine
|
||||
all kernel tests.
|
||||
|
||||
An example of entries in a quarantine yaml::
|
||||
|
||||
- scenarios:
|
||||
- sample.basic.helloworld
|
||||
platforms:
|
||||
- all
|
||||
comment: "Link to the issue: https://github.com/zephyrproject-rtos/zephyr/pull/33287"
|
||||
|
||||
- scenarios:
|
||||
- kernel.common
|
||||
- kernel.common.misra
|
||||
- kernel.common.(misra|tls)
|
||||
- kernel.common.nano64
|
||||
platforms:
|
||||
- qemu_cortex_m3
|
||||
- .*_cortex_.*
|
||||
- native_posix
|
||||
|
||||
To exclude a platform, use the following syntax::
|
||||
|
||||
- platforms:
|
||||
- qemu_x86
|
||||
comment: "broken qemu"
|
||||
|
||||
Additionally you can quarantine entire architectures or a specific simulator for executing tests.
|
||||
|
||||
Running in Tests in Random Order
|
||||
********************************
|
||||
Enable ZTEST framework's :kconfig:option:`CONFIG_ZTEST_SHUFFLE` config option to
|
||||
|
|
|
@ -40,7 +40,7 @@ class QuarantineElement:
|
|||
platforms: list[str] = field(default_factory=list)
|
||||
architectures: list[str] = field(default_factory=list)
|
||||
simulations: list[str] = field(default_factory=list)
|
||||
comment: str = 'under quarantine'
|
||||
comment: str = 'NA'
|
||||
|
||||
def __post_init__(self):
|
||||
# If there is no entry in filters then take all possible values.
|
||||
|
|
|
@ -753,7 +753,7 @@ class TestPlan:
|
|||
instance.testsuite.id, plat.name, plat.arch, plat.simulation
|
||||
)
|
||||
if matched_quarantine and not self.options.quarantine_verify:
|
||||
instance.add_filter(matched_quarantine, Filters.QUARENTINE)
|
||||
instance.add_filter("Quarantine: " + matched_quarantine, Filters.QUARENTINE)
|
||||
if not matched_quarantine and self.options.quarantine_verify:
|
||||
instance.add_filter("Not under quarantine", Filters.QUARENTINE)
|
||||
|
||||
|
|
|
@ -25,19 +25,19 @@ sequence:
|
|||
type: seq
|
||||
sequence:
|
||||
- type: str
|
||||
- unique: True
|
||||
- unique: true
|
||||
"architectures":
|
||||
required: false
|
||||
type: seq
|
||||
sequence:
|
||||
- type: str
|
||||
- unique: True
|
||||
- unique: true
|
||||
"simulations":
|
||||
required: false
|
||||
type: seq
|
||||
sequence:
|
||||
- type: str
|
||||
- unique: True
|
||||
- unique: true
|
||||
"comment":
|
||||
type: str
|
||||
required: false
|
||||
|
|
|
@ -1,4 +1,3 @@
|
|||
|
||||
- platforms:
|
||||
- demo_board_3
|
||||
comment: "all on board_3"
|
||||
|
|
|
@ -1,4 +1,3 @@
|
|||
|
||||
- scenarios:
|
||||
- test_(a|c).check_2
|
||||
architectures:
|
||||
|
|
|
@ -332,6 +332,6 @@ def test_quarantine(class_testplan, platforms_list, test_data,
|
|||
else:
|
||||
if testname in expected_val:
|
||||
assert instance.status == 'filtered'
|
||||
assert instance.reason == expected_val[testname]
|
||||
assert instance.reason == "Quarantine: " + expected_val[testname]
|
||||
else:
|
||||
assert not instance.status
|
||||
|
|
Loading…
Add table
Add a link
Reference in a new issue