twister: Platform key for test suites
Adds an option to inform twister a testsuite should only be built and run for platforms with unique sets of attributes. This enables for example keying on unique (arch, simulation) platforms to run the test suite on. The most common usage may be test suites configured to run once per (arch, simulation) pair as being enough. Additional information about platforms may enable running a test once per hardware IP block or once per soc family or soc avoiding duplicated effort in building and running tests when once suffices. Signed-off-by: Tom Burdick <thomas.burdick@intel.com>
This commit is contained in:
parent
b36a31fd7a
commit
38385ed3ff
5 changed files with 64 additions and 0 deletions
|
@ -388,6 +388,25 @@ harness: <string>
|
|||
Usually pertains to external dependency domains but can be anything such as
|
||||
console, sensor, net, keyboard, Bluetooth or pytest.
|
||||
|
||||
platform_key: <list of platform attributes>
|
||||
Often a test needs to only be built and run once to qualify as passing.
|
||||
Imagine a library of code that depends on the platform architecture where
|
||||
passing the test on a single platform for each arch is enough to qualify the
|
||||
tests and code as passing. The platform_key attribute enables doing just
|
||||
that.
|
||||
|
||||
For example to key on (arch, simulation) to ensure a test is run once
|
||||
per arch and simulation (as would be most common)::
|
||||
|
||||
platform_key:
|
||||
- arch
|
||||
- simulation
|
||||
|
||||
Adding platform (board) attributes to include things such as soc name,
|
||||
soc family, and perhaps sets of IP blocks implementing each peripheral
|
||||
interface would enable other interesting uses. For example, this could enable
|
||||
building and running SPI tests once for eacn unique IP block.
|
||||
|
||||
harness_config: <harness configuration options>
|
||||
Extra harness configuration options to be used to select a board and/or
|
||||
for handling generic Console with regex matching. Config can announce
|
||||
|
|
|
@ -33,6 +33,7 @@ class TwisterConfigParser:
|
|||
"platform_type": {"type": "list", "default": []},
|
||||
"platform_exclude": {"type": "set"},
|
||||
"platform_allow": {"type": "set"},
|
||||
"platform_key": {"type": "list", "default": []},
|
||||
"toolchain_exclude": {"type": "set"},
|
||||
"toolchain_allow": {"type": "set"},
|
||||
"filter": {"type": "str"},
|
||||
|
|
|
@ -361,6 +361,9 @@ structure in the main Zephyr tree: boards/<arch>/<board_name>/""")
|
|||
help="Upon test failure, print relevant log data to stdout "
|
||||
"instead of just a path to it.")
|
||||
|
||||
parser.add_argument("--ignore-platform-key", action="store_true",
|
||||
help="Do not filter based on platform key")
|
||||
|
||||
parser.add_argument(
|
||||
"-j", "--jobs", type=int,
|
||||
help="Number of jobs for building, defaults to number of CPU threads, "
|
||||
|
|
|
@ -570,6 +570,7 @@ class TestPlan:
|
|||
runnable = (self.options.device_testing or self.options.filter == 'runnable')
|
||||
force_toolchain = self.options.force_toolchain
|
||||
force_platform = self.options.force_platform
|
||||
ignore_platform_key = self.options.ignore_platform_key
|
||||
emu_filter = self.options.emulation_only
|
||||
|
||||
logger.debug("platform filter: " + str(platform_filter))
|
||||
|
@ -616,6 +617,8 @@ class TestPlan:
|
|||
|
||||
logger.info("Building initial testsuite list...")
|
||||
|
||||
keyed_tests = {}
|
||||
|
||||
for ts_name, ts in self.testsuites.items():
|
||||
|
||||
if ts.build_on_all and not platform_filter:
|
||||
|
@ -642,6 +645,7 @@ class TestPlan:
|
|||
platform_scope = list(filter(lambda item: item.name in ts.platform_allow, \
|
||||
self.platforms))
|
||||
|
||||
|
||||
# list of instances per testsuite, aka configurations.
|
||||
instance_list = []
|
||||
for plat in platform_scope:
|
||||
|
@ -746,6 +750,31 @@ class TestPlan:
|
|||
if plat.only_tags and not set(plat.only_tags) & ts.tags:
|
||||
instance.add_filter("Excluded tags per platform (only_tags)", Filters.PLATFORM)
|
||||
|
||||
# platform_key is a list of unique platform attributes that form a unique key a test
|
||||
# will match against to determine if it should be scheduled to run. A key containing a
|
||||
# field name that the platform does not have will filter the platform.
|
||||
#
|
||||
# A simple example is keying on arch and simulation to run a test once per unique (arch, simulation) platform.
|
||||
if not ignore_platform_key and hasattr(ts, 'platform_key') and len(ts.platform_key) > 0:
|
||||
# form a key by sorting the key fields first, then fetching the key fields from plat if they exist
|
||||
# if a field does not exist the test is still scheduled on that platform as its undeterminable.
|
||||
key_fields = sorted(set(ts.platform_key))
|
||||
key = [getattr(plat, key_field) for key_field in key_fields]
|
||||
has_all_fields = True
|
||||
for key_field in key_fields:
|
||||
if key_field is None:
|
||||
has_all_fields = False
|
||||
if has_all_fields:
|
||||
key.append(ts.name)
|
||||
key = tuple(key)
|
||||
keyed_test = keyed_tests.get(key)
|
||||
if keyed_test is not None:
|
||||
instance.add_filter(f"Excluded test already covered by platform_key {key} given fields {key_fields}", Filters.TESTSUITE)
|
||||
else:
|
||||
keyed_tests[key] = {'plat': plat, 'ts': ts}
|
||||
else:
|
||||
instance.add_filter(f"Excluded platform missing key fields demanded by test {key_fields}", Filters.PLATFORM)
|
||||
|
||||
test_configuration = ".".join([instance.platform.name,
|
||||
instance.testsuite.id])
|
||||
# skip quarantined tests
|
||||
|
|
|
@ -115,6 +115,12 @@ mapping:
|
|||
required: false
|
||||
sequence:
|
||||
- type: str
|
||||
"platform_key":
|
||||
required: false
|
||||
type: seq
|
||||
matching: "all"
|
||||
sequence:
|
||||
- type: str
|
||||
"tags":
|
||||
type: str
|
||||
required: false
|
||||
|
@ -272,6 +278,12 @@ mapping:
|
|||
required: false
|
||||
sequence:
|
||||
- type: str
|
||||
"platform_key":
|
||||
required: false
|
||||
type: seq
|
||||
matching: "all"
|
||||
sequence:
|
||||
- type: str
|
||||
"tags":
|
||||
type: str
|
||||
required: false
|
||||
|
|
Loading…
Add table
Add a link
Reference in a new issue