doc: document best practices for main.c suite declaration

With the introduction of --list-tests to sanitycheck, a few rules have
been added to how to declare test suites.

Signed-off-by: Inaky Perez-Gonzalez <inaky.perez-gonzalez@intel.com>
This commit is contained in:
Inaky Perez-Gonzalez 2018-05-03 10:05:55 -07:00 committed by Anas Nashif
commit 850b218e73

View file

@ -41,7 +41,7 @@ prj.conf
:language: text
:linenos:
src/main.c
src/main.c (see :ref:`best practices <main_c_bp>`)
.. literalinclude:: ../../../samples/testing/integration/src/main.c
:language: c
@ -148,6 +148,141 @@ and are used to decide whether a test failed or passed by verifying whether an
interaction with an object occurred, and if required, to assert the order of
that interaction.
.. _main_c_bp:
Best practices for declaring the test suite
===========================================
*sanitycheck* and other validation tools need to obtain the list of
subcases that a Zephyr *ztest* test image will expose.
.. admonition:: Rationale
This all is for the purpose of traceability. It's not enough to
have only a semaphore test project. We also need to show that we
have testpoints for all APIs and functionality, and we trace back
to documentation of the API, and functional requirements.
The idea is that test reports show results for every sub-testcase
as passed, failed, blocked, or skipped. Reporting on only the
high-level test project level, particularly when tests do too
many things, is too vague.
Here is a generic template for a test showing the expected use of
:func:`ztest_test_suite`:
.. code-block:: C
#include <ztest.h>
extern void test_sometest1(void);
extern void test_sometest2(void);
#ifndef CONFIG_WHATEVER /* Conditionally skip test_sometest3 */
void test_sometest3(void)
{
ztest_test_skip();
}
#else
extern void test_sometest3(void);
#endif
extern void test_sometest4(void);
...
void test_main(void)
{
ztest_test_suite(common,
ztest_unit_test(test_sometest1),
ztest_unit_test(test_sometest2),
ztest_unit_test(test_sometest3),
ztest_unit_test(test_sometest4),
);
ztest_run_test_suite(common);
}
For *sanitycheck* to parse source files and create a list of subcases,
the declarations of :func:`ztest_test_suite` must follow a few rules:
- one declaration per line
- conditional execution by using :func:`ztest_test_skip`
What to avoid:
- packing multiple testcases in one source file
.. code-block:: C
void test_main(void)
{
#ifdef TEST_feature1
ztest_test_suite(feature1,
ztest_unit_test(test_1a),
ztest_unit_test(test_1b),
ztest_unit_test(test_1c),
ztest_run_test_suite(feature1);
#endif
#ifdef TEST_feature2
ztest_test_suite(feature2,
ztest_unit_test(test_2a),
ztest_unit_test(test_2b),
ztest_run_test_suite(feature2);
#endif
}
- Do not use ``#if``
.. code-block:: C
ztest_test_suite(common,
ztest_unit_test(test_sometest1),
ztest_unit_test(test_sometest2),
#ifdef CONFIG_WHATEVER
ztest_unit_test(test_sometest3),
#endif
ztest_unit_test(test_sometest4),
...
- Do not add comments on lines with a call to :func:`ztest_unit_test`:
.. code-block:: C
ztest_test_suite(common,
ztest_unit_test(test_sometest1),
ztest_unit_test(test_sometest2) /* will fail */,
/* will fail! */ ztest_unit_test(test_sometest3),
ztest_unit_test(test_sometest4),
...
- Do not define multiple definitions of unit / user unit test case per
line
.. code-block:: C
ztest_test_suite(common,
ztest_unit_test(test_sometest1), ztest_unit_test(test_sometest2),
ztest_unit_test(test_sometest3),
ztest_unit_test(test_sometest4),
...
Other questions:
- Why not pre-scan with CPP and then parse? or post scan the ELF file?
If C pre-processing or building fails because of any issue, then we
won't be able to tell the subcases.
- Why not declare them in the YAML testcase description?
A separate testcase description file would be harder to maintain
than just keeping the information in the test source files
themselves -- only one file to update when changes are made
eliminates duplication.
API reference
*************