michaelh/zephyr
1
0
Fork 0
Code Releases Activity

twister: ztest: harness: Fix missed TestCase statuses

Browse source 
Fix a problem of Ztest suite names not taken into account by Twister
to identify a TestCase, so in some situations a Ztest test's status
was not assigned to the proper TestCase and it remains 'None'
whereas the actual status value lost, eventually the resulting total
execution counters not correct.

The issue was observed in these situations:
 * Ztest application with multiple test suites having same test names.
 * Ztest suite is 'skipped' entirely on execution with all its tests.

The proposed solution extends Twister test case name for Ztest to
include Ztest suite name, so the resulting identifier looks like:
   `<test_scenario_name>.<ztest_suite_name>.<ztest_name>`

The above naming scheme now requires ztest_suite_name part to be
provided for `--sub-test` command line option.

Testcase identifiers in twister.json and testplan.json will also
include ztest_suite_name component.

The Twister Ztest(Test) Harness is improved to track all state changes
known from the test application's log for Ztest suites and test cases,
so now it parses log output from a Ztest application more scurpulously.
Regular expressions to match log records are extended and optimized
to compile them only once and, in some cases, fixed (suite summary).

Signed-off-by: Dmitrii Golovanov <dmitrii.golovanov@intel.com>
This commit is contained in:
Dmitrii Golovanov 2024-10-17 09:02:35 +02:00 • committed by Anas Nashif
commit a72244f2d0
18 changed files with 254 additions and 110 deletions
Download patch file Download diff file Expand all files Collapse all files

11
scripts/pylib/twister/twisterlib/runner.py
View file

@ -1,6 +1,6 @@
# vim: set syntax=python ts=4 :
#
# Copyright (c) 20180-2022 Intel Corporation
# Copyright (c) 2018-2024 Intel Corporation
# Copyright 2022 NXP
# SPDX-License-Identifier: Apache-2.0
@ -1108,13 +1108,16 @@ class ProjectBuilder(FilterBuilder):
matches = new_ztest_unit_test_regex.findall(sym.name)
if matches:
for m in matches:
# new_ztest_suite = m[0] # not used for now
new_ztest_suite = m[0]
if new_ztest_suite not in self.instance.testsuite.ztest_suite_names:
logger.warning(f"Unexpected Ztest suite '{new_ztest_suite}' "
f"not present in: {self.instance.testsuite.ztest_suite_names}")
test_func_name = m[1].replace("test_", "", 1)
testcase_id = f"{yaml_testsuite_name}.{test_func_name}"
testcase_id = f"{yaml_testsuite_name}.{new_ztest_suite}.{test_func_name}"
detected_cases.append(testcase_id)
if detected_cases:
logger.debug(f"{', '.join(detected_cases)} in {elf_file}")
logger.debug(f"Detected Ztest cases: [{', '.join(detected_cases)}] in {elf_file}")
tc_keeper = {tc.name: {'status': tc.status, 'reason': tc.reason} for tc in self.instance.testcases}
self.instance.testcases.clear()
self.instance.testsuite.testcases.clear()