michaelh/zephyr
1
0
Fork 0
Code Releases Activity
zephyr/scripts/sanity_chk/testcase-schema.yaml

225 lines
5.4 KiB
YAML
Raw Permalink Normal View History

sanitycheck: validate YAML files w/ schemas files This introduces an schema-based YAML validation process when loading any YAML file, before doing any operations on them. An exception will be raised at SanityConfigParser() if the file fails to verify with the given schema. Schemas are defined for the platform files in board///*.yaml and for the (sample|testcase).yaml files. The verification is done using the pykwalify python library. If not installed, a warning is printed and the verification schema is skipped. At some point, we might want to force it being installed. The verification library is made a separate module (scl.py) so it can be easily imported by others. Signed-off-by: Inaky Perez-Gonzalez <inaky.perez-gonzalez@intel.com>
2017-07-24 19:24:35 +02:00
#
# Schema to validate a YAML file describing a Zephyr test platform
#
# We load this with pykwalify
# (http://pykwalify.readthedocs.io/en/unstable/validation-rules.html),
# a YAML structure validator, to validate the YAML files that describe
# Zephyr test platforms
#
# The original spec comes from Zephyr's sanitycheck script
#
type: map
mapping:
sanitycheck: add extra_configs for testing with multiple values Support new keywords in testcase.yaml that would allow us to inject configuration options to be merged with default configuration instead of having to provide a prj.conf for each variant of the test which is very difficult to keep in sync. Sanitycheck script will create an overlay file that is merged during the build process. This is now done using the extra_configs option which is a yaml list of option with the values, for example: extra_configs: - CONFIG_XXXX=y - CONFIG_YYYY=y With this option we can have multiple tests that for example run on hardware with different values. This type of testing is good on HW but it does not make sense to be built in normal sanitycheck operation because it will be just rebuilding the same code with different values. Signed-off-by: Anas Nashif <anas.nashif@intel.com>
2017-10-04 22:14:27 +02:00
"common":
type: map
required: no
mapping:
"arch_exclude":
type: str
required: no
"arch_whitelist":
type: str
required: no
"build_only":
type: bool
required: no
"build_on_all":
type: bool
required: no
"depends_on":
type: str
required: no
"extra_args":
type: str
required: no
"extra_sections":
type: str
required: no
"filter":
type: str
required: no
tests: add a new harness keyword to schema This keyword would mean that a special harness is needed to run the tests sucessfully. This can be as simple as a loopback wiring or a complete hardware test setup for sensor and IO testing. It is free form initially and would be changed to be an enum once we have more values in place. Signed-off-by: Anas Nashif <anas.nashif@intel.com>
2017-12-08 16:17:57 +01:00
"harness":
type: str
required: no
sanitycheck: support harness_config keyword in schema Add harness_config to schema with basic options for now to support console harness. Signed-off-by: Anas Nashif <anas.nashif@intel.com>
2017-12-25 00:27:21 +01:00
"harness_config":
type: map
required: no
mapping:
"type":
type: str
sanitycheck: Add fixture option for external hardware dependecy Add a new option as fixture in harness configurations for utilizing sanitycheck to identify test cases that require external hardware such as sensor, ble, networking for validation. The config will be added to yaml files with unique fixture name to identify each hardware and allow automation to trigger test execution on setup having the specific fixture enabled. Also, remove the default required for type and regex configs that is not essential in case of ztest based test cases. Signed-off-by: Praful Swarnakar <praful.swarnakar@intel.com>
2018-09-12 23:28:28 +02:00
required: no
"fixture":
type: str
required: no
sanitycheck: support harness_config keyword in schema Add harness_config to schema with basic options for now to support console harness. Signed-off-by: Anas Nashif <anas.nashif@intel.com>
2017-12-25 00:27:21 +01:00
"ordered":
type: bool
required: no
"repeat":
type: int
required: no
"regex":
type: seq
sanitycheck: Add fixture option for external hardware dependecy Add a new option as fixture in harness configurations for utilizing sanitycheck to identify test cases that require external hardware such as sensor, ble, networking for validation. The config will be added to yaml files with unique fixture name to identify each hardware and allow automation to trigger test execution on setup having the specific fixture enabled. Also, remove the default required for type and regex configs that is not essential in case of ztest based test cases. Signed-off-by: Praful Swarnakar <praful.swarnakar@intel.com>
2018-09-12 23:28:28 +02:00
required: no
sanitycheck: support harness_config keyword in schema Add harness_config to schema with basic options for now to support console harness. Signed-off-by: Anas Nashif <anas.nashif@intel.com>
2017-12-25 00:27:21 +01:00
sequence:
- type: str
sanitycheck: Complete overhaul and job handling rework A complete overhaul of the sanitycheck script and how we build and run tests. This new version of sanitycheck uses python for job distribution and drop use of Make. In addition to the move to python threading library, the following has been changed: - All handlers now run in parallel, meaning that any simulator will run in parallel and when testing on multiple devices (using --device-testing) the tests are run in parallel. - Lexicial filtering (using the filter keyword in yaml files) is now evaluated at runtime and is no long being pre-processed. This will allow us to immediately start executing tests and skip the wait time that was needed for filtering. - Device testing now supports multiple devices connected at the same time and is managed using a hardware map that needs to be generated and maintained for every test environment. (using --generate-hardware-map option). - Reports are not long stored in the Zephyr tree and instead stored in the output directory where all build artifacts are generated. - Each tested target now has a junit report in the output directory. - Recording option for performance data and other metrics is now available. This will allow us to record the output from the console and store the data for later processing. For example benchmark data can be captured and uploaded to a tracking server. - Test configurations (or instances) are no longer being sorted, this will help with balancing the load when we run sanitycheck on multiple hosts (as we do in CI). And many other cleanups and improvements... Signed-off-by: Anas Nashif <anas.nashif@intel.com>
2019-06-22 17:04:10 +02:00
"record":
type: map
required: no
mapping:
"regex":
type: str
required: no
sanitycheck: add extra_configs for testing with multiple values Support new keywords in testcase.yaml that would allow us to inject configuration options to be merged with default configuration instead of having to provide a prj.conf for each variant of the test which is very difficult to keep in sync. Sanitycheck script will create an overlay file that is merged during the build process. This is now done using the extra_configs option which is a yaml list of option with the values, for example: extra_configs: - CONFIG_XXXX=y - CONFIG_YYYY=y With this option we can have multiple tests that for example run on hardware with different values. This type of testing is good on HW but it does not make sense to be built in normal sanitycheck operation because it will be just rebuilding the same code with different values. Signed-off-by: Anas Nashif <anas.nashif@intel.com>
2017-10-04 22:14:27 +02:00
"min_ram":
type: int
required: no
"min_flash":
type: int
required: no
"platform_exclude":
type: str
required: no
"platform_whitelist":
type: str
required: no
"tags":
type: str
tests: tags should not be required Tags can be set in the common section, some tests also might choose not to set any tags. Signed-off-by: Anas Nashif <anas.nashif@intel.com>
2018-02-24 03:23:42 +01:00
required: no
sanitycheck: add extra_configs for testing with multiple values Support new keywords in testcase.yaml that would allow us to inject configuration options to be merged with default configuration instead of having to provide a prj.conf for each variant of the test which is very difficult to keep in sync. Sanitycheck script will create an overlay file that is merged during the build process. This is now done using the extra_configs option which is a yaml list of option with the values, for example: extra_configs: - CONFIG_XXXX=y - CONFIG_YYYY=y With this option we can have multiple tests that for example run on hardware with different values. This type of testing is good on HW but it does not make sense to be built in normal sanitycheck operation because it will be just rebuilding the same code with different values. Signed-off-by: Anas Nashif <anas.nashif@intel.com>
2017-10-04 22:14:27 +02:00
"timeout":
type: int
required: no
"toolchain_exclude":
type: str
required: no
"toolchain_whitelist":
type: str
required: no
"type":
type: str
enum: [ 'unit' ]
"skip":
type: bool
required: no
"slow":
type: bool
required: no
sanitycheck: validate YAML files w/ schemas files This introduces an schema-based YAML validation process when loading any YAML file, before doing any operations on them. An exception will be raised at SanityConfigParser() if the file fails to verify with the given schema. Schemas are defined for the platform files in board///*.yaml and for the (sample|testcase).yaml files. The verification is done using the pykwalify python library. If not installed, a warning is printed and the verification schema is skipped. At some point, we might want to force it being installed. The verification library is made a separate module (scl.py) so it can be easily imported by others. Signed-off-by: Inaky Perez-Gonzalez <inaky.perez-gonzalez@intel.com>
2017-07-24 19:24:35 +02:00
# The sample descriptor, if present
"sample":
type: map
required: no
mapping:
"name":
type: str
required: yes
"description":
type: str
required: no
# The list of testcases -- IDK why this is a sequence of
# maps maps, shall just be a sequence of maps
# maybe it is just an artifact?
"tests":
sanitycheck: refactor data parsing in script Simplify parsing of yaml structures and remove usage of cp which was for the ConfigParser used for ini files. Signed-off-by: Anas Nashif <anas.nashif@intel.com>
2017-12-05 21:08:26 +01:00
type: map
matching-rule: "any"
mapping:
sanitycheck: validate YAML files w/ schemas files This introduces an schema-based YAML validation process when loading any YAML file, before doing any operations on them. An exception will be raised at SanityConfigParser() if the file fails to verify with the given schema. Schemas are defined for the platform files in board///*.yaml and for the (sample|testcase).yaml files. The verification is done using the pykwalify python library. If not installed, a warning is printed and the verification schema is skipped. At some point, we might want to force it being installed. The verification library is made a separate module (scl.py) so it can be easily imported by others. Signed-off-by: Inaky Perez-Gonzalez <inaky.perez-gonzalez@intel.com>
2017-07-24 19:24:35 +02:00
# The key for the testname is any, so
# regex;(([a-zA-Z0-9_]+)) for this to work, note below we
# make it required: no
regex;(([a-zA-Z0-9_]+)):
type: map
# has to be not-required, otherwise the parser gets
# confused and things it never found it
required: no
mapping:
"arch_exclude":
type: str
required: no
"arch_whitelist":
type: str
required: no
"build_only":
type: bool
required: no
"build_on_all":
type: bool
required: no
"depends_on":
type: str
required: no
"extra_args":
type: str
required: no
sanitycheck: add extra_configs for testing with multiple values Support new keywords in testcase.yaml that would allow us to inject configuration options to be merged with default configuration instead of having to provide a prj.conf for each variant of the test which is very difficult to keep in sync. Sanitycheck script will create an overlay file that is merged during the build process. This is now done using the extra_configs option which is a yaml list of option with the values, for example: extra_configs: - CONFIG_XXXX=y - CONFIG_YYYY=y With this option we can have multiple tests that for example run on hardware with different values. This type of testing is good on HW but it does not make sense to be built in normal sanitycheck operation because it will be just rebuilding the same code with different values. Signed-off-by: Anas Nashif <anas.nashif@intel.com>
2017-10-04 22:14:27 +02:00
"extra_configs":
type: seq
required: no
sequence:
- type: str
sanitycheck: validate YAML files w/ schemas files This introduces an schema-based YAML validation process when loading any YAML file, before doing any operations on them. An exception will be raised at SanityConfigParser() if the file fails to verify with the given schema. Schemas are defined for the platform files in board///*.yaml and for the (sample|testcase).yaml files. The verification is done using the pykwalify python library. If not installed, a warning is printed and the verification schema is skipped. At some point, we might want to force it being installed. The verification library is made a separate module (scl.py) so it can be easily imported by others. Signed-off-by: Inaky Perez-Gonzalez <inaky.perez-gonzalez@intel.com>
2017-07-24 19:24:35 +02:00
"extra_sections":
type: str
required: no
"filter":
type: str
required: no
tests: add a new harness keyword to schema This keyword would mean that a special harness is needed to run the tests sucessfully. This can be as simple as a loopback wiring or a complete hardware test setup for sensor and IO testing. It is free form initially and would be changed to be an enum once we have more values in place. Signed-off-by: Anas Nashif <anas.nashif@intel.com>
2017-12-08 16:17:57 +01:00
"harness":
type: str
required: no
sanitycheck: support harness_config keyword in schema Add harness_config to schema with basic options for now to support console harness. Signed-off-by: Anas Nashif <anas.nashif@intel.com>
2017-12-25 00:27:21 +01:00
"harness_config":
type: map
required: no
mapping:
"type":
type: str
sanitycheck: Add fixture option for external hardware dependecy Add a new option as fixture in harness configurations for utilizing sanitycheck to identify test cases that require external hardware such as sensor, ble, networking for validation. The config will be added to yaml files with unique fixture name to identify each hardware and allow automation to trigger test execution on setup having the specific fixture enabled. Also, remove the default required for type and regex configs that is not essential in case of ztest based test cases. Signed-off-by: Praful Swarnakar <praful.swarnakar@intel.com>
2018-09-12 23:28:28 +02:00
required: no
"fixture":
type: str
required: no
sanitycheck: support harness_config keyword in schema Add harness_config to schema with basic options for now to support console harness. Signed-off-by: Anas Nashif <anas.nashif@intel.com>
2017-12-25 00:27:21 +01:00
"ordered":
type: bool
required: no
"repeat":
type: int
required: no
"regex":
type: seq
sanitycheck: Add fixture option for external hardware dependecy Add a new option as fixture in harness configurations for utilizing sanitycheck to identify test cases that require external hardware such as sensor, ble, networking for validation. The config will be added to yaml files with unique fixture name to identify each hardware and allow automation to trigger test execution on setup having the specific fixture enabled. Also, remove the default required for type and regex configs that is not essential in case of ztest based test cases. Signed-off-by: Praful Swarnakar <praful.swarnakar@intel.com>
2018-09-12 23:28:28 +02:00
required: no
sanitycheck: support harness_config keyword in schema Add harness_config to schema with basic options for now to support console harness. Signed-off-by: Anas Nashif <anas.nashif@intel.com>
2017-12-25 00:27:21 +01:00
sequence:
- type: str
sanitycheck: Complete overhaul and job handling rework A complete overhaul of the sanitycheck script and how we build and run tests. This new version of sanitycheck uses python for job distribution and drop use of Make. In addition to the move to python threading library, the following has been changed: - All handlers now run in parallel, meaning that any simulator will run in parallel and when testing on multiple devices (using --device-testing) the tests are run in parallel. - Lexicial filtering (using the filter keyword in yaml files) is now evaluated at runtime and is no long being pre-processed. This will allow us to immediately start executing tests and skip the wait time that was needed for filtering. - Device testing now supports multiple devices connected at the same time and is managed using a hardware map that needs to be generated and maintained for every test environment. (using --generate-hardware-map option). - Reports are not long stored in the Zephyr tree and instead stored in the output directory where all build artifacts are generated. - Each tested target now has a junit report in the output directory. - Recording option for performance data and other metrics is now available. This will allow us to record the output from the console and store the data for later processing. For example benchmark data can be captured and uploaded to a tracking server. - Test configurations (or instances) are no longer being sorted, this will help with balancing the load when we run sanitycheck on multiple hosts (as we do in CI). And many other cleanups and improvements... Signed-off-by: Anas Nashif <anas.nashif@intel.com>
2019-06-22 17:04:10 +02:00
"record":
type: map
required: no
mapping:
"regex":
type: str
required: no
sanitycheck: validate YAML files w/ schemas files This introduces an schema-based YAML validation process when loading any YAML file, before doing any operations on them. An exception will be raised at SanityConfigParser() if the file fails to verify with the given schema. Schemas are defined for the platform files in board///*.yaml and for the (sample|testcase).yaml files. The verification is done using the pykwalify python library. If not installed, a warning is printed and the verification schema is skipped. At some point, we might want to force it being installed. The verification library is made a separate module (scl.py) so it can be easily imported by others. Signed-off-by: Inaky Perez-Gonzalez <inaky.perez-gonzalez@intel.com>
2017-07-24 19:24:35 +02:00
"min_ram":
type: int
required: no
"min_flash":
type: int
required: no
"platform_exclude":
type: str
required: no
"platform_whitelist":
type: str
required: no
"tags":
type: str
sanitycheck: add extra_configs for testing with multiple values Support new keywords in testcase.yaml that would allow us to inject configuration options to be merged with default configuration instead of having to provide a prj.conf for each variant of the test which is very difficult to keep in sync. Sanitycheck script will create an overlay file that is merged during the build process. This is now done using the extra_configs option which is a yaml list of option with the values, for example: extra_configs: - CONFIG_XXXX=y - CONFIG_YYYY=y With this option we can have multiple tests that for example run on hardware with different values. This type of testing is good on HW but it does not make sense to be built in normal sanitycheck operation because it will be just rebuilding the same code with different values. Signed-off-by: Anas Nashif <anas.nashif@intel.com>
2017-10-04 22:14:27 +02:00
required: no
sanitycheck: validate YAML files w/ schemas files This introduces an schema-based YAML validation process when loading any YAML file, before doing any operations on them. An exception will be raised at SanityConfigParser() if the file fails to verify with the given schema. Schemas are defined for the platform files in board///*.yaml and for the (sample|testcase).yaml files. The verification is done using the pykwalify python library. If not installed, a warning is printed and the verification schema is skipped. At some point, we might want to force it being installed. The verification library is made a separate module (scl.py) so it can be easily imported by others. Signed-off-by: Inaky Perez-Gonzalez <inaky.perez-gonzalez@intel.com>
2017-07-24 19:24:35 +02:00
"timeout":
type: int
required: no
"toolchain_exclude":
type: str
required: no
"toolchain_whitelist":
type: str
required: no
"type":
type: str
enum: [ 'unit' ]
"skip":
type: bool
required: no
"slow":
type: bool
required: no
Copy permalink