doc: document best practices for main.c suite declaration
With the introduction of --list-tests to sanitycheck, a few rules have been added to how to declare test suites. Signed-off-by: Inaky Perez-Gonzalez <inaky.perez-gonzalez@intel.com>
This commit is contained in:
parent
65e96eefea
commit
850b218e73
1 changed files with 136 additions and 1 deletions
|
@ -41,7 +41,7 @@ prj.conf
|
|||
:language: text
|
||||
:linenos:
|
||||
|
||||
src/main.c
|
||||
src/main.c (see :ref:`best practices <main_c_bp>`)
|
||||
|
||||
.. literalinclude:: ../../../samples/testing/integration/src/main.c
|
||||
:language: c
|
||||
|
@ -148,6 +148,141 @@ and are used to decide whether a test failed or passed by verifying whether an
|
|||
interaction with an object occurred, and if required, to assert the order of
|
||||
that interaction.
|
||||
|
||||
.. _main_c_bp:
|
||||
|
||||
Best practices for declaring the test suite
|
||||
===========================================
|
||||
|
||||
*sanitycheck* and other validation tools need to obtain the list of
|
||||
subcases that a Zephyr *ztest* test image will expose.
|
||||
|
||||
.. admonition:: Rationale
|
||||
|
||||
This all is for the purpose of traceability. It's not enough to
|
||||
have only a semaphore test project. We also need to show that we
|
||||
have testpoints for all APIs and functionality, and we trace back
|
||||
to documentation of the API, and functional requirements.
|
||||
|
||||
The idea is that test reports show results for every sub-testcase
|
||||
as passed, failed, blocked, or skipped. Reporting on only the
|
||||
high-level test project level, particularly when tests do too
|
||||
many things, is too vague.
|
||||
|
||||
Here is a generic template for a test showing the expected use of
|
||||
:func:`ztest_test_suite`:
|
||||
|
||||
.. code-block:: C
|
||||
|
||||
#include <ztest.h>
|
||||
|
||||
extern void test_sometest1(void);
|
||||
extern void test_sometest2(void);
|
||||
#ifndef CONFIG_WHATEVER /* Conditionally skip test_sometest3 */
|
||||
void test_sometest3(void)
|
||||
{
|
||||
ztest_test_skip();
|
||||
}
|
||||
#else
|
||||
extern void test_sometest3(void);
|
||||
#endif
|
||||
extern void test_sometest4(void);
|
||||
...
|
||||
|
||||
void test_main(void)
|
||||
{
|
||||
ztest_test_suite(common,
|
||||
ztest_unit_test(test_sometest1),
|
||||
ztest_unit_test(test_sometest2),
|
||||
ztest_unit_test(test_sometest3),
|
||||
ztest_unit_test(test_sometest4),
|
||||
);
|
||||
ztest_run_test_suite(common);
|
||||
}
|
||||
|
||||
For *sanitycheck* to parse source files and create a list of subcases,
|
||||
the declarations of :func:`ztest_test_suite` must follow a few rules:
|
||||
|
||||
- one declaration per line
|
||||
|
||||
- conditional execution by using :func:`ztest_test_skip`
|
||||
|
||||
What to avoid:
|
||||
|
||||
- packing multiple testcases in one source file
|
||||
|
||||
.. code-block:: C
|
||||
|
||||
void test_main(void)
|
||||
{
|
||||
#ifdef TEST_feature1
|
||||
ztest_test_suite(feature1,
|
||||
ztest_unit_test(test_1a),
|
||||
ztest_unit_test(test_1b),
|
||||
ztest_unit_test(test_1c),
|
||||
ztest_run_test_suite(feature1);
|
||||
#endif
|
||||
|
||||
#ifdef TEST_feature2
|
||||
ztest_test_suite(feature2,
|
||||
ztest_unit_test(test_2a),
|
||||
ztest_unit_test(test_2b),
|
||||
ztest_run_test_suite(feature2);
|
||||
#endif
|
||||
}
|
||||
|
||||
|
||||
- Do not use ``#if``
|
||||
|
||||
.. code-block:: C
|
||||
|
||||
ztest_test_suite(common,
|
||||
ztest_unit_test(test_sometest1),
|
||||
ztest_unit_test(test_sometest2),
|
||||
#ifdef CONFIG_WHATEVER
|
||||
ztest_unit_test(test_sometest3),
|
||||
#endif
|
||||
ztest_unit_test(test_sometest4),
|
||||
...
|
||||
|
||||
- Do not add comments on lines with a call to :func:`ztest_unit_test`:
|
||||
|
||||
.. code-block:: C
|
||||
|
||||
ztest_test_suite(common,
|
||||
ztest_unit_test(test_sometest1),
|
||||
ztest_unit_test(test_sometest2) /* will fail */,
|
||||
/* will fail! */ ztest_unit_test(test_sometest3),
|
||||
ztest_unit_test(test_sometest4),
|
||||
...
|
||||
|
||||
- Do not define multiple definitions of unit / user unit test case per
|
||||
line
|
||||
|
||||
|
||||
.. code-block:: C
|
||||
|
||||
ztest_test_suite(common,
|
||||
ztest_unit_test(test_sometest1), ztest_unit_test(test_sometest2),
|
||||
ztest_unit_test(test_sometest3),
|
||||
ztest_unit_test(test_sometest4),
|
||||
...
|
||||
|
||||
|
||||
Other questions:
|
||||
|
||||
- Why not pre-scan with CPP and then parse? or post scan the ELF file?
|
||||
|
||||
If C pre-processing or building fails because of any issue, then we
|
||||
won't be able to tell the subcases.
|
||||
|
||||
- Why not declare them in the YAML testcase description?
|
||||
|
||||
A separate testcase description file would be harder to maintain
|
||||
than just keeping the information in the test source files
|
||||
themselves -- only one file to update when changes are made
|
||||
eliminates duplication.
|
||||
|
||||
|
||||
API reference
|
||||
*************
|
||||
|
||||
|
|
Loading…
Add table
Add a link
Reference in a new issue