michaelh/zephyr
1
0
Fork 0
Code Releases Activity

twister: refine the twister test plan generation

Browse source 
Due to the diverse coding styles and lack of preprocessing when
scanning for test cases, there were many unintended combinations
of yaml test scenarios and C test functions with the regex-based
test case discovery, which caused an inaccurate test plan and test
result stats.

As the new ztest fx is used, the test cases of a test instance can
be accurately determined via zephyr.symbols file.

Because the zephyr.symbols file is not available until after build,
test cases determination is placed right after the build stage of
the pipeline and before the runtime test collection.

For those test instances that don't go through the build stage,
such as those with "Skip filtered" (statically filtered) reason
before entering the pipeline, they are not affected.

This patch also adjust the stats logic because the zephyr.symbols
file is generated after build. So ExecutionCounter update is split
and some must be postponed until test execution pipeline is completed.

Some concepts:

A test instance = a yaml scenario + a platform
"Test instance" and "test config" are synonyms in twister, unfortunately
excessive IMHO...

A test instance can be filtered or skipped in 3 ways.
Let's define "runtime" as "after entering the execution pipeline".

1) statically filtered (before runtime)
   Such test instance is filtered by analyzing the yaml and never
   enters the execution pipeline.
2) cmake filtered (runtime)
   Such test instance enters pipeline and is filtered at cmake stage.
3) build skipped (also runtime)
   Such test instance enters pipeline and is skipped at build stage.

All the test instances that enter the execution pipeline will go
through the report stage, where ExecutionCounter is further updated.

The meaning of the fields of ExecutionCounter are:

 .skipped_configs = static filtered + cmake filtered + build skipped
 .skipped_runtime = cmake filtered + build skipped
 .skipped_filter = static filtered
 .done = instances that enter the execution pipeline
 .passed = instances that actually executed and passed

Definition of the overall stats:
total_complete = .done + .skipped_filter
total = yaml test scenario * applicable platforms
complete percentage = total_complete / total
pass rate = passed / (total - skipped_configs)

Signed-off-by: Ming Shao <ming.shao@intel.com>
This commit is contained in:
Ming Shao 2022-08-29 18:22:50 +08:00 • committed by Anas Nashif
commit 94f2d2437f
2 changed files with 81 additions and 23 deletions
Download patch file Download diff file Expand all files Collapse all files

2
subsys/testsuite/ztest/include/zephyr/ztest_test_new.h
View file

@ -336,7 +336,7 @@ void ztest_test_skip(void);
static void _##suite##_##fn##_wrapper(void *data); \
static void suite##_##fn( \
COND_CODE_1(use_fixture, (struct suite##_fixture *fixture), (void))); \
static STRUCT_SECTION_ITERABLE(ztest_unit_test, z_ztest_unit_test_##suite##_##fn) = { \
static STRUCT_SECTION_ITERABLE(ztest_unit_test, z_ztest_unit_test__##suite##__##fn) = { \
.test_suite_name = STRINGIFY(suite), \
.name = STRINGIFY(fn), \
.test = (_##suite##_##fn##_wrapper), \