twister: Add quarantine feature

Adds feature allowing to use yaml file with dictionaries defining
tests to be quarantined (extra arg "--quarantine-list FILENAME").
The dictionaries are validated according to the proper schema
and loaded.

A flat list is created containing quarantined configurations
(configuration = platform + scenario). Configurations under quarantine
are skipped and get "Quarantine" as a reason in the results reports.

A "comment" can be added to a quarantine entry in the quarantine yaml
with more details (e.g. issue #) and it will be also added to
the report.

The status of tests under quarantine can be verify if
`--quarantine-verify` is used in addition to
"--quarantine-list FILENAME". Using these args will make twister skip
all tests which are not on the quarantine list.

Signed-off-by: Maciej Perkowski <Maciej.Perkowski@nordicsemi.no>
This commit is contained in:
Maciej Perkowski 2021-03-11 13:18:33 +01:00 committed by Anas Nashif
commit 9c6dfce0c0
3 changed files with 97 additions and 3 deletions

View file

@ -0,0 +1,31 @@
#
# Schema to validate a YAML file providing the list of configurations
# under quarantine
#
# We load this with pykwalify
# (http://pykwalify.readthedocs.io/en/unstable/validation-rules.html),
# a YAML structure validator, to validate the YAML files that provide
# a list of configurations (scenarios + platforms) under quarantine
#
type: seq
matching: all
sequence:
- type: map
required: yes
matching: all
mapping:
"scenarios":
type: seq
required: true
sequence:
- type: str
- unique: True
"platforms":
required: true
type: seq
sequence:
- type: str
- unique: True
"comment":
type: str
required: false