d888e012ed
A Kconfig boolean is added to allow users to provide their own output strings when running tests via ztest. This allows changing e.g. the PASS/FAIL/SKIPPED strings, add counters, change separators, and similar. A test using the feature and relevant documentation is added. Signed-off-by: Torstein Grindvik <torstein.grindvik@nordicsemi.no>
362 lines
11 KiB
ReStructuredText
362 lines
11 KiB
ReStructuredText
.. _test-framework:
|
|
|
|
Test Framework
|
|
###############
|
|
|
|
The Zephyr Test Framework (Ztest) provides a simple testing framework intended
|
|
to be used during development. It provides basic assertion macros and a generic
|
|
test structure.
|
|
|
|
The framework can be used in two ways, either as a generic framework for
|
|
integration testing, or for unit testing specific modules.
|
|
|
|
Quick start - Integration testing
|
|
*********************************
|
|
|
|
A simple working base is located at :zephyr_file:`samples/testing/integration`. Just
|
|
copy the files to ``tests/`` and edit them for your needs. The test will then
|
|
be automatically built and run by the sanitycheck script. If you are testing
|
|
the **bar** component of **foo**, you should copy the sample folder to
|
|
``tests/foo/bar``. It can then be tested with::
|
|
|
|
./scripts/sanitycheck -s tests/foo/bar/test-identifier
|
|
|
|
|
|
In the example above ``tests/foo/bar`` signifies the path to the test and the
|
|
``test-identifier`` references a test defined in the :file:`testcase.yaml` file.
|
|
|
|
To run all tests defined in a test project, run::
|
|
|
|
./scripts/sanitycheck -T tests/foo/bar/
|
|
|
|
The sample contains the following files:
|
|
|
|
CMakeLists.txt
|
|
|
|
.. literalinclude:: ../../../samples/testing/integration/CMakeLists.txt
|
|
:language: CMake
|
|
:linenos:
|
|
|
|
testcase.yaml
|
|
|
|
.. literalinclude:: ../../../samples/testing/integration/testcase.yaml
|
|
:language: yaml
|
|
:linenos:
|
|
|
|
prj.conf
|
|
|
|
.. literalinclude:: ../../../samples/testing/integration/prj.conf
|
|
:language: text
|
|
:linenos:
|
|
|
|
src/main.c (see :ref:`best practices <main_c_bp>`)
|
|
|
|
.. literalinclude:: ../../../samples/testing/integration/src/main.c
|
|
:language: c
|
|
:linenos:
|
|
|
|
.. contents::
|
|
:depth: 1
|
|
:local:
|
|
:backlinks: top
|
|
|
|
|
|
|
|
A test case project may consist of multiple sub-tests or smaller tests that
|
|
either can be testing functionality or APIs. Functions implementing a test
|
|
should follow the guidelines below:
|
|
|
|
* Test cases function names should be prefix with **test_**
|
|
* Test cases should be documented using doxygen
|
|
* Test function names should be unique within the section or component being
|
|
tested
|
|
|
|
|
|
An example can be seen below::
|
|
|
|
/**
|
|
* @brief Test Asserts
|
|
*
|
|
* This test verifies the zassert_true macro.
|
|
*/
|
|
static void test_assert(void)
|
|
{
|
|
zassert_true(1, "1 was false");
|
|
}
|
|
|
|
|
|
The above test is then enabled as part of the testsuite using::
|
|
|
|
ztest_unit_test(test_assert)
|
|
|
|
|
|
Listing Tests
|
|
=============
|
|
|
|
Tests (test projects) in the Zephyr tree consist of many testcases that run as
|
|
part of a project and test similar functionality, for example an API or a
|
|
feature. The ``sanitycheck`` script can parse the testcases in all
|
|
test projects or a subset of them, and can generate reports on a granular
|
|
level, i.e. if cases have passed or failed or if they were blocked or skipped.
|
|
|
|
Sanitycheck parses the source files looking for test case names, so you
|
|
can list all kernel test cases, for example, by entering::
|
|
|
|
sanitycheck --list-tests -T tests/kernel
|
|
|
|
Skipping Tests
|
|
==============
|
|
|
|
Special- or architecture-specific tests cannot run on all
|
|
platforms and architectures, however we still want to count those and
|
|
report them as being skipped. Because the test inventory and
|
|
the list of tests is extracted from the code, adding
|
|
conditionals inside the test suite is sub-optimal. Tests that need
|
|
to be skipped for a certain platform or feature need to explicitly
|
|
report a skip using :c:func:`ztest_test_skip()`. If the test runs,
|
|
it needs to report either a pass or fail. For example::
|
|
|
|
#ifdef CONFIG_TEST1
|
|
void test_test1(void)
|
|
{
|
|
zassert_true(1, "true");
|
|
}
|
|
#else
|
|
void test_test1(void)
|
|
{
|
|
ztest_test_skip();
|
|
}
|
|
#endif
|
|
|
|
|
|
void test_main(void)
|
|
{
|
|
ztest_test_suite(common,
|
|
ztest_unit_test(test_test1),
|
|
ztest_unit_test(test_test2)
|
|
);
|
|
ztest_run_test_suite(common);
|
|
}
|
|
|
|
Quick start - Unit testing
|
|
**************************
|
|
|
|
Ztest can be used for unit testing. This means that rather than including the
|
|
entire Zephyr OS for testing a single function, you can focus the testing
|
|
efforts into the specific module in question. This will speed up testing since
|
|
only the module will have to be compiled in, and the tested functions will be
|
|
called directly.
|
|
|
|
Since you won't be including basic kernel data structures that most code
|
|
depends on, you have to provide function stubs in the test. Ztest provides
|
|
some helpers for mocking functions, as demonstrated below.
|
|
|
|
In a unit test, mock objects can simulate the behavior of complex real objects
|
|
and are used to decide whether a test failed or passed by verifying whether an
|
|
interaction with an object occurred, and if required, to assert the order of
|
|
that interaction.
|
|
|
|
.. _main_c_bp:
|
|
|
|
Best practices for declaring the test suite
|
|
===========================================
|
|
|
|
*sanitycheck* and other validation tools need to obtain the list of
|
|
subcases that a Zephyr *ztest* test image will expose.
|
|
|
|
.. admonition:: Rationale
|
|
|
|
This all is for the purpose of traceability. It's not enough to
|
|
have only a semaphore test project. We also need to show that we
|
|
have testpoints for all APIs and functionality, and we trace back
|
|
to documentation of the API, and functional requirements.
|
|
|
|
The idea is that test reports show results for every sub-testcase
|
|
as passed, failed, blocked, or skipped. Reporting on only the
|
|
high-level test project level, particularly when tests do too
|
|
many things, is too vague.
|
|
|
|
Here is a generic template for a test showing the expected use of
|
|
:func:`ztest_test_suite`:
|
|
|
|
.. code-block:: C
|
|
|
|
#include <ztest.h>
|
|
|
|
extern void test_sometest1(void);
|
|
extern void test_sometest2(void);
|
|
#ifndef CONFIG_WHATEVER /* Conditionally skip test_sometest3 */
|
|
void test_sometest3(void)
|
|
{
|
|
ztest_test_skip();
|
|
}
|
|
#else
|
|
extern void test_sometest3(void);
|
|
#endif
|
|
extern void test_sometest4(void);
|
|
...
|
|
|
|
void test_main(void)
|
|
{
|
|
ztest_test_suite(common,
|
|
ztest_unit_test(test_sometest1),
|
|
ztest_unit_test(test_sometest2),
|
|
ztest_unit_test(test_sometest3),
|
|
ztest_unit_test(test_sometest4)
|
|
);
|
|
ztest_run_test_suite(common);
|
|
}
|
|
|
|
For *sanitycheck* to parse source files and create a list of subcases,
|
|
the declarations of :func:`ztest_test_suite` must follow a few rules:
|
|
|
|
- one declaration per line
|
|
|
|
- conditional execution by using :func:`ztest_test_skip`
|
|
|
|
What to avoid:
|
|
|
|
- packing multiple testcases in one source file
|
|
|
|
.. code-block:: C
|
|
|
|
void test_main(void)
|
|
{
|
|
#ifdef TEST_feature1
|
|
ztest_test_suite(feature1,
|
|
ztest_unit_test(test_1a),
|
|
ztest_unit_test(test_1b),
|
|
ztest_unit_test(test_1c)
|
|
);
|
|
ztest_run_test_suite(feature1);
|
|
#endif
|
|
|
|
#ifdef TEST_feature2
|
|
ztest_test_suite(feature2,
|
|
ztest_unit_test(test_2a),
|
|
ztest_unit_test(test_2b)
|
|
);
|
|
ztest_run_test_suite(feature2);
|
|
#endif
|
|
}
|
|
|
|
|
|
- Do not use ``#if``
|
|
|
|
.. code-block:: C
|
|
|
|
ztest_test_suite(common,
|
|
ztest_unit_test(test_sometest1),
|
|
ztest_unit_test(test_sometest2),
|
|
#ifdef CONFIG_WHATEVER
|
|
ztest_unit_test(test_sometest3),
|
|
#endif
|
|
ztest_unit_test(test_sometest4),
|
|
...
|
|
|
|
- Do not add comments on lines with a call to :func:`ztest_unit_test`:
|
|
|
|
.. code-block:: C
|
|
|
|
ztest_test_suite(common,
|
|
ztest_unit_test(test_sometest1),
|
|
ztest_unit_test(test_sometest2) /* will fail */,
|
|
/* will fail! */ ztest_unit_test(test_sometest3),
|
|
ztest_unit_test(test_sometest4),
|
|
...
|
|
|
|
- Do not define multiple definitions of unit / user unit test case per
|
|
line
|
|
|
|
|
|
.. code-block:: C
|
|
|
|
ztest_test_suite(common,
|
|
ztest_unit_test(test_sometest1), ztest_unit_test(test_sometest2),
|
|
ztest_unit_test(test_sometest3),
|
|
ztest_unit_test(test_sometest4),
|
|
...
|
|
|
|
|
|
Other questions:
|
|
|
|
- Why not pre-scan with CPP and then parse? or post scan the ELF file?
|
|
|
|
If C pre-processing or building fails because of any issue, then we
|
|
won't be able to tell the subcases.
|
|
|
|
- Why not declare them in the YAML testcase description?
|
|
|
|
A separate testcase description file would be harder to maintain
|
|
than just keeping the information in the test source files
|
|
themselves -- only one file to update when changes are made
|
|
eliminates duplication.
|
|
|
|
|
|
API reference
|
|
*************
|
|
|
|
Running tests
|
|
=============
|
|
|
|
.. doxygengroup:: ztest_test
|
|
:project: Zephyr
|
|
|
|
Assertions
|
|
==========
|
|
|
|
These macros will instantly fail the test if the related assertion fails.
|
|
When an assertion fails, it will print the current file, line and function,
|
|
alongside a reason for the failure and an optional message. If the config
|
|
option:`CONFIG_ZTEST_ASSERT_VERBOSE` is 0, the assertions will only print the
|
|
file and line numbers, reducing the binary size of the test.
|
|
|
|
Example output for a failed macro from
|
|
``zassert_equal(buf->ref, 2, "Invalid refcount")``:
|
|
|
|
.. code-block:: none
|
|
|
|
Assertion failed at main.c:62: test_get_single_buffer: Invalid refcount (buf->ref not equal to 2)
|
|
Aborted at unit test function
|
|
|
|
.. doxygengroup:: ztest_assert
|
|
:project: Zephyr
|
|
|
|
Mocking
|
|
=======
|
|
|
|
These functions allow abstracting callbacks and related functions and
|
|
controlling them from specific tests. You can enable the mocking framework by
|
|
setting :option:`CONFIG_ZTEST_MOCKING` to "y" in the configuration file of the
|
|
test. The amount of concurrent return values and expected parameters is
|
|
limited by :option:`CONFIG_ZTEST_PARAMETER_COUNT`.
|
|
|
|
Here is an example for configuring the function ``expect_two_parameters`` to
|
|
expect the values ``a=2`` and ``b=3``, and telling ``returns_int`` to return
|
|
``5``:
|
|
|
|
.. literalinclude:: mocking.c
|
|
:language: c
|
|
:linenos:
|
|
|
|
.. doxygengroup:: ztest_mock
|
|
:project: Zephyr
|
|
|
|
Customizing Test Output
|
|
***********************
|
|
The way output is presented when running tests can be customized.
|
|
An example can be found in :zephyr_file:`tests/ztest/custom_output`.
|
|
|
|
Customization is enabled by setting :option:`CONFIG_ZTEST_TC_UTIL_USER_OVERRIDE` to "y"
|
|
and adding a file :file:`tc_util_user_override.h` with your overrides.
|
|
|
|
Add the line ``zephyr_include_directories(my_folder)`` to
|
|
your project's :file:`CMakeLists.txt` to let Zephyr find your header file during builds.
|
|
|
|
See the file :zephyr_file:`subsys/testsuite/include/tc_util.h` to see which macros and/or defines can be overridden.
|
|
These will be surrounded by blocks such as::
|
|
|
|
#ifndef SOMETHING
|
|
#define SOMETHING <default implementation>
|
|
#endif /* SOMETHING */
|