twister: Platform key for test suites

Adds an option to inform twister a testsuite should only be built and
run for platforms with unique sets of attributes. This enables
for example keying on unique (arch, simulation) platforms to run the test
suite on.

The most common usage may be test suites configured to run once per
(arch, simulation) pair as being enough. Additional information about
platforms may enable running a test once per hardware IP block or once
per soc family or soc avoiding duplicated effort in building and running
tests when once suffices.

Signed-off-by: Tom Burdick <thomas.burdick@intel.com>
This commit is contained in:
Tom Burdick 2022-11-15 11:49:06 -06:00 committed by Anas Nashif
commit 38385ed3ff
5 changed files with 64 additions and 0 deletions

View file

@ -388,6 +388,25 @@ harness: <string>
Usually pertains to external dependency domains but can be anything such as
console, sensor, net, keyboard, Bluetooth or pytest.
platform_key: <list of platform attributes>
Often a test needs to only be built and run once to qualify as passing.
Imagine a library of code that depends on the platform architecture where
passing the test on a single platform for each arch is enough to qualify the
tests and code as passing. The platform_key attribute enables doing just
that.
For example to key on (arch, simulation) to ensure a test is run once
per arch and simulation (as would be most common)::
platform_key:
- arch
- simulation
Adding platform (board) attributes to include things such as soc name,
soc family, and perhaps sets of IP blocks implementing each peripheral
interface would enable other interesting uses. For example, this could enable
building and running SPI tests once for eacn unique IP block.
harness_config: <harness configuration options>
Extra harness configuration options to be used to select a board and/or
for handling generic Console with regex matching. Config can announce

View file

@ -33,6 +33,7 @@ class TwisterConfigParser:
"platform_type": {"type": "list", "default": []},
"platform_exclude": {"type": "set"},
"platform_allow": {"type": "set"},
"platform_key": {"type": "list", "default": []},
"toolchain_exclude": {"type": "set"},
"toolchain_allow": {"type": "set"},
"filter": {"type": "str"},

View file

@ -361,6 +361,9 @@ structure in the main Zephyr tree: boards/<arch>/<board_name>/""")
help="Upon test failure, print relevant log data to stdout "
"instead of just a path to it.")
parser.add_argument("--ignore-platform-key", action="store_true",
help="Do not filter based on platform key")
parser.add_argument(
"-j", "--jobs", type=int,
help="Number of jobs for building, defaults to number of CPU threads, "

View file

@ -570,6 +570,7 @@ class TestPlan:
runnable = (self.options.device_testing or self.options.filter == 'runnable')
force_toolchain = self.options.force_toolchain
force_platform = self.options.force_platform
ignore_platform_key = self.options.ignore_platform_key
emu_filter = self.options.emulation_only
logger.debug("platform filter: " + str(platform_filter))
@ -616,6 +617,8 @@ class TestPlan:
logger.info("Building initial testsuite list...")
keyed_tests = {}
for ts_name, ts in self.testsuites.items():
if ts.build_on_all and not platform_filter:
@ -642,6 +645,7 @@ class TestPlan:
platform_scope = list(filter(lambda item: item.name in ts.platform_allow, \
self.platforms))
# list of instances per testsuite, aka configurations.
instance_list = []
for plat in platform_scope:
@ -746,6 +750,31 @@ class TestPlan:
if plat.only_tags and not set(plat.only_tags) & ts.tags:
instance.add_filter("Excluded tags per platform (only_tags)", Filters.PLATFORM)
# platform_key is a list of unique platform attributes that form a unique key a test
# will match against to determine if it should be scheduled to run. A key containing a
# field name that the platform does not have will filter the platform.
#
# A simple example is keying on arch and simulation to run a test once per unique (arch, simulation) platform.
if not ignore_platform_key and hasattr(ts, 'platform_key') and len(ts.platform_key) > 0:
# form a key by sorting the key fields first, then fetching the key fields from plat if they exist
# if a field does not exist the test is still scheduled on that platform as its undeterminable.
key_fields = sorted(set(ts.platform_key))
key = [getattr(plat, key_field) for key_field in key_fields]
has_all_fields = True
for key_field in key_fields:
if key_field is None:
has_all_fields = False
if has_all_fields:
key.append(ts.name)
key = tuple(key)
keyed_test = keyed_tests.get(key)
if keyed_test is not None:
instance.add_filter(f"Excluded test already covered by platform_key {key} given fields {key_fields}", Filters.TESTSUITE)
else:
keyed_tests[key] = {'plat': plat, 'ts': ts}
else:
instance.add_filter(f"Excluded platform missing key fields demanded by test {key_fields}", Filters.PLATFORM)
test_configuration = ".".join([instance.platform.name,
instance.testsuite.id])
# skip quarantined tests

View file

@ -115,6 +115,12 @@ mapping:
required: false
sequence:
- type: str
"platform_key":
required: false
type: seq
matching: "all"
sequence:
- type: str
"tags":
type: str
required: false
@ -272,6 +278,12 @@ mapping:
required: false
sequence:
- type: str
"platform_key":
required: false
type: seq
matching: "all"
sequence:
- type: str
"tags":
type: str
required: false