twister: Add quarantine feature
Adds feature allowing to use yaml file with dictionaries defining tests to be quarantined (extra arg "--quarantine-list FILENAME"). The dictionaries are validated according to the proper schema and loaded. A flat list is created containing quarantined configurations (configuration = platform + scenario). Configurations under quarantine are skipped and get "Quarantine" as a reason in the results reports. A "comment" can be added to a quarantine entry in the quarantine yaml with more details (e.g. issue #) and it will be also added to the report. The status of tests under quarantine can be verify if `--quarantine-verify` is used in addition to "--quarantine-list FILENAME". Using these args will make twister skip all tests which are not on the quarantine list. Signed-off-by: Maciej Perkowski <Maciej.Perkowski@nordicsemi.no>
This commit is contained in:
parent
8bee027ec4
commit
9c6dfce0c0
3 changed files with 97 additions and 3 deletions
31
scripts/schemas/twister/quarantine-schema.yaml
Normal file
31
scripts/schemas/twister/quarantine-schema.yaml
Normal file
|
@ -0,0 +1,31 @@
|
|||
#
|
||||
# Schema to validate a YAML file providing the list of configurations
|
||||
# under quarantine
|
||||
#
|
||||
# We load this with pykwalify
|
||||
# (http://pykwalify.readthedocs.io/en/unstable/validation-rules.html),
|
||||
# a YAML structure validator, to validate the YAML files that provide
|
||||
# a list of configurations (scenarios + platforms) under quarantine
|
||||
#
|
||||
type: seq
|
||||
matching: all
|
||||
sequence:
|
||||
- type: map
|
||||
required: yes
|
||||
matching: all
|
||||
mapping:
|
||||
"scenarios":
|
||||
type: seq
|
||||
required: true
|
||||
sequence:
|
||||
- type: str
|
||||
- unique: True
|
||||
"platforms":
|
||||
required: true
|
||||
type: seq
|
||||
sequence:
|
||||
- type: str
|
||||
- unique: True
|
||||
"comment":
|
||||
type: str
|
||||
required: false
|
Loading…
Add table
Add a link
Reference in a new issue