This commit updates all in-tree code to use `CONFIG_CPP` instead of
`CONFIG_CPLUSPLUS`, which is now deprecated.
Signed-off-by: Stephanos Ioannidis <stephanos.ioannidis@nordicsemi.no>
Accurate timekeeping is something that is often taken for granted.
However, reliability of timekeeping code is critical for most core
and subsystem code. Furthermore, Many higher-level timekeeping
utilities in Zephyr work off of ticks but there is no way to modify
ticks directly which would require either unnecessary delays in
test code or non-ideal compromises in test coverage.
Since timekeeping is so critical, there should be as few barriers
to testing timekeeping code as possible, while preserving
integrity of the kernel's public interface.
With this, we expose `sys_clock_tick_set()` as a system call only
when `CONFIG_ZTEST` is set, declared within the ztest framework.
Signed-off-by: Chris Friedt <cfriedt@meta.com>
Test summary can add a lot of noise to the logs when debugging
a specific test using `-test` argument.
Add control to turn this off.
Signed-off-by: Al Semjonovs <asemjonovs@google.com>
Some missing features for getting coverage data for unit tests:
- Setting the unit_testing board to have coverage support and native
application.
- Fixing the CONFIG_COVERAGE check
Signed-off-by: Yuval Peress <peress@google.com>
The heap size for serial dump of gcov data is currently
defined by MALLOC_MAX_HEAP_SIZE, which cannot be
adjusted by kconfig. This commit adds a new kconfig
option, CONFIG_COVERAGE_GCOV_HEAP_SIZE, which retains
the behavior or MALLOC_MAX_HEAP_SIZE by default.
Signed-off-by: Jason Wright <jwright@synchron.com>
We get compiler warnings on testcases that set CONFIG_MP_NUM_CPUS=1
and platforms that have CONFIG_SMP=y. Qualify the code so its only
built if CONFIG_SMP && (CONFIG_MP_NUM_CPUS > 1).
Signed-off-by: Kumar Gala <kumar.gala@intel.com>
Move the function in the `subsys/testsuite/ztest/src/ztest_mock.c` files.
This is motivated by the fact that there is others re-implementation of
`*printk` functions using libc counterparts in the `ztest_mock.c` file.
Signed-off-by: Théo Battrel <theo.battrel@nordicsemi.no>
For test_1cpu_start/test_1cpu_stop make the code only build if
CONFIG_SMP and move to using arch_num_cpus() for runtime loops
and CONFIG_MP_MAX_NUM_CPUS for array decleration.
Signed-off-by: Kumar Gala <kumar.gala@intel.com>
3000 milliseconds may not always be enough time for all 1cpu type
tests to finish on all platforms. Making the CPU hold time
configurable allows for additional flexibility.
Signed-off-by: Peter Mitsis <peter.mitsis@intel.com>
Change for loops of the form:
for (i = 0; i < CONFIG_MP_NUM_CPUS; i++)
...
to
unsigned int num_cpus = arch_num_cpus();
for (i = 0; i < num_cpus; i++)
...
We do the call outside of the for loop so that it only happens once,
rather than on every iteration.
Signed-off-by: Kumar Gala <kumar.gala@intel.com>
Adding the new Kconfig (enabled by default) to make a failed assumption
mark the final result as failed. This change has the following benefits
which have been asked for by the Zephyr community:
1. A failed assumption does not go silent. In this example, the failed
assumption will still mark the test as skipped, but the final result
will be to mark the full test run as failed. This would allow
blocking the CI when an assume fails.
2. Normal test skipping via the ztest_test_skip() is unaffected by this
change. Those tests will be marked as skipped, but the binary will
still pass.
Signed-off-by: Yuval Peress <peress@google.com>
This fixes the compiler warning "comparison of integer expressions of
different signedness: 'int' and 'unsigned int' [-Werror=sign-compare]"
Signed-off-by: Benjamin Gwin <bgwin@google.com>
Update fff.h to the pending PR upstream which allows for using a custom
function signature. This enables the use of C++ std::function as the
mock.
Signed-off-by: Yuval Peress <peress@google.com>
Due to the diverse coding styles and lack of preprocessing when
scanning for test cases, there were many unintended combinations
of yaml test scenarios and C test functions with the regex-based
test case discovery, which caused an inaccurate test plan and test
result stats.
As the new ztest fx is used, the test cases of a test instance can
be accurately determined via zephyr.symbols file.
Because the zephyr.symbols file is not available until after build,
test cases determination is placed right after the build stage of
the pipeline and before the runtime test collection.
For those test instances that don't go through the build stage,
such as those with "Skip filtered" (statically filtered) reason
before entering the pipeline, they are not affected.
This patch also adjust the stats logic because the zephyr.symbols
file is generated after build. So ExecutionCounter update is split
and some must be postponed until test execution pipeline is completed.
Some concepts:
A test instance = a yaml scenario + a platform
"Test instance" and "test config" are synonyms in twister, unfortunately
excessive IMHO...
A test instance can be filtered or skipped in 3 ways.
Let's define "runtime" as "after entering the execution pipeline".
1) statically filtered (before runtime)
Such test instance is filtered by analyzing the yaml and never
enters the execution pipeline.
2) cmake filtered (runtime)
Such test instance enters pipeline and is filtered at cmake stage.
3) build skipped (also runtime)
Such test instance enters pipeline and is skipped at build stage.
All the test instances that enter the execution pipeline will go
through the report stage, where ExecutionCounter is further updated.
The meaning of the fields of ExecutionCounter are:
.skipped_configs = static filtered + cmake filtered + build skipped
.skipped_runtime = cmake filtered + build skipped
.skipped_filter = static filtered
.done = instances that enter the execution pipeline
.passed = instances that actually executed and passed
Definition of the overall stats:
total_complete = .done + .skipped_filter
total = yaml test scenario * applicable platforms
complete percentage = total_complete / total
pass rate = passed / (total - skipped_configs)
Signed-off-by: Ming Shao <ming.shao@intel.com>
Some testsuites dump lots of output very fast where some systems are not
able to capture the complete output from the tests. Add a slight delay
between each test in the suite.
New kconfig CONFIG_ZTEST_TEST_DELAY_MS is added to ztest.
Signed-off-by: Anas Nashif <anas.nashif@intel.com>
The GCC/GCOV version over 12 has slight format change of the gcno
and gcda. Make some adaption in the gcov dump function to fix the
code coverage report.
Mainly two places change:
1. Added the checksum in the struct gcov_info. This fix the crash
in qemu_x86, and mps2_an385 when run with --coverage.
2. Adjust the GCOV_TAG_FUNCTION_LENGTH accroding to gcov-io.h. It's
length unit is caculated by bytes now.
Fixes#50255.
Fixes#50257.
Signed-off-by: Enjia Mai <enjia.mai@intel.com>
When CONFIG_ZTEST_SHUFFLE is enabled and ASSERTS are disabled
`start_pos` becomes an unused variable leading to a compile error.
Cleaned-up shuffling algorithm to not need a `start_pos` check.
Signed-off-by: Al Semjonovs <asemjonovs@google.com>
On GICv3, when we send an IPI interrupt, aff3, aff2 and aff1 should
be assigned a value corespond to a PE for which interrupt will be
generated. target_list only corresponds to aff0.
On real hardware, aff3, aff2, aff1 and aff0 should be treated as a
whole to determine a PE.
Signed-off-by: Huifeng Zhang <Huifeng.Zhang@arm.com>
This commit removes the requirement for msg parameter in zassert_*
and zassume_* macros. It will allow to use them without sending NULL
as the last parameter.
The default messages look like:
void* beauty_ptr = NULL;
zassert_not_null(beauty_ptr);
=>
Assertion failed at main.c:20: suite_test: beauty_ptr is NULL
void* ugly_ptr = (void*)0xbaadf00d;
zassume_is_null(ugly_ptr);
=>
Assumption failed at main.c:23: suite_test: ugly_ptr is not NULL
int apple = 3;
int tomato = 9;
zassume_equal(apple, tomato);
=>
Assumption failed at main.c:27: suite_test: apple not equal to tomato
Signed-off-by: Michał Barnaś <mb@semihalf.com>
This commit adds intermediate macros that will allow to use the
zassert and zassume macros without setting the msg parameter as NULL
if it isn't used.
Signed-off-by: Michał Barnaś <mb@semihalf.com>
Use XOSHIRO random number generator if target has entropy generator.
Some entropy generators may have limitations (e.g. only thread context)
which would conflict with ztress usage.
Added Kconfig.defconfig for testsuite.
Signed-off-by: Krzysztof Chruscinski <krzysztof.chruscinski@nordicsemi.no>
Added a workaround to call random generator once during the
initialization. It is done to handle XOSHIRO generator limitation
which performs initialization in the first sys_rand32_get call.
And for some entropy generators it cannot be done from an interrupt
context and it happens if k_timer context is used which expires first.
Signed-off-by: Krzysztof Chruscinski <krzysztof.chruscinski@nordicsemi.no>
As of today <zephyr/zephyr.h> is 100% equivalent to <zephyr/kernel.h>.
This patch proposes to then include <zephyr/kernel.h> instead of
<zephyr/zephyr.h> since it is more clear that you are including the
Kernel APIs and (probably) nothing else. <zephyr/zephyr.h> sounds like a
catch-all header that may be confusing. Most applications need to
include a bunch of other things to compile, e.g. driver headers or
subsystem headers like BT, logging, etc.
The idea of a catch-all header in Zephyr is probably not feasible
anyway. Reason is that Zephyr is not a library, like it could be for
example `libpython`. Zephyr provides many utilities nowadays: a kernel,
drivers, subsystems, etc and things will likely grow. A catch-all header
would be massive, difficult to keep up-to-date. It is also likely that
an application will only build a small subset. Note that subsystem-level
headers may use a catch-all approach to make things easier, though.
NOTE: This patch is **NOT** removing the header, just removing its usage
in-tree. I'd advocate for its deprecation (add a #warning on it), but I
understand many people will have concerns.
Signed-off-by: Gerard Marull-Paretas <gerard.marull@nordicsemi.no>
When CONFIG_KERNEL_COHERENCE=y, it's illegal to place shared
data(like ztress_context_data in this test case) on the stack,
because that memory is incoherent with respect to other CPUs
in the system.
In this specific case, it will cause PC register to load invalid
handler function pointer and crash the Xtensa processor.
Make it static.
Signed-off-by: Ming Shao <ming.shao@intel.com>
Current ztress design supports both timer context and thread
context. The timer context always uses the first element of
several global array variables like exec_cnt[] and backoff[].
But progress_timeout() is using the last element of exec_cnt[]
to calculate the progress of timer context. It is not right.
Signed-off-by: Ming Shao <ming.shao@intel.com>
Tests with KERNEL enabled perform their cleanup logic after the suite's
after and test rules are executed. Unittests should do the same.
Signed-off-by: Yuval Peress <peress@google.com>
Updates the implementation when KERNEL is available to safely bail on
the test when the test calls fail, skip, or pass during invalid test
phases. Print a detailed message, and skip all other tests. The test
run will be marked as failed.
Signed-off-by: Yuval Peress <peress@google.com>
Move the function used for printing the phase name up so its available
for both unittest and KERNEL mode of tests.
Signed-off-by: Yuval Peress <peress@google.com>
When not running in setup, before, or in the test. Calling fail, skip,
or pass is invalid and should be considered an error. Properly handle
these cases by printing a more detailed error message and bailing the
process.
Signed-off-by: Yuval Peress <peress@google.com>
When KERNEL was not defined (unittest), the call to init_testing was
used to set a longjump target using 'stack_fail'. When triggered,
this was actually causing a segfault, because longjmp is only valid
if going directly up the stack. Since init_testing returned, it was
no longer on the stack. Instead, that logic MUST be inlined.
Signed-off-by: Yuval Peress <peress@google.com>
The current zassert macro uses implicit conversion to boolean which
has implication on analysis tools like clang-tidy-14. This add an
aditional step to create a boolean value for the evaluation instead
use the string direct which allows run analysis tool without this
warning/error.
Signed-off-by: Gerson Fernando Budke <nandojve@gmail.com>
It is frequent to find variable definitions like this:
```c
static const struct device *dev = DEVICE_DT_GET(...)
```
That is, module level variables that are statically initialized with a
device reference. Such value is, in most cases, never changed meaning
the variable can also be declared as const (immutable). This patch
constifies all such cases.
Signed-off-by: Gerard Marull-Paretas <gerard.marull@nordicsemi.no>
This adds the bits to pin the GCOV sections to the pinned
sections so they can be accessed during boot when demand
paging is enabled. Or else accessing them would result in
page faults.
Signed-off-by: Daniel Leung <daniel.leung@intel.com>
This adds an ALIGN statement to the rodata linker snippet
for coverage. Without this, sometimes the section is not
aligned, but __init_array_start indicates an aligned
address, resulting in incorrect function pointers.
So align it.
Signed-off-by: Daniel Leung <daniel.leung@intel.com>
This commit introduces a dedicated unit testing board.
Today, a dedicated Zephyr unit testing scheme exists but is different
from how a Zephyr build generally works.
For example Kconfig is not possible, resulting on various different
hacks to pass Kconfig settings from test cases / testcase.yaml through
CMake to the code.
Some directly as compile definitions, some as header files with forced
inclusion on sources, some with wrapper flags which again results in
different define being enabled. There is even cases where a second
forced header inclusion undefines previous defines.
Unit test often does a manual check for the right boards, like this:
> if (NOT BOARD STREQUAL unit_testing)
> message(FATAL_ERROR "This project can only be used with...")
> endif()
Introducing a dedicated unit_testing board under `tests/root` allows
us to use Kconfig in unit test samples, and thus proper `prj.conf` and
extra Kconfig fragments.
Generation of autoconf.h so the overall architecture follows regular
Zephyr builds.
Proper and uniform error messages when invalid board is selected.
The unit_testing board and arch is located under: `subsys/testsuite` so
that it is only available when find_package(Zephyr COMPONENTS unittest)
is used, and not available for regular Zephyr builds.
Kconfig generates autoconf.h which is applied as compile flag to
test binary which means that kconfig defines placed in ztest.h can now
be removed.
Signed-off-by: Torsten Rasmussen <Torsten.Rasmussen@nordicsemi.no>
Add a detailed docstring to the zassume() macro that describes what it does
and how its usage is different from zassert.
Signed-off-by: Aaron Massey <aaronmassey@google.com>
Add duration stats at both suite level and unit test
level into test summary. The duration is at second
level. Since the new ztest fx can execute a test suite
for multiple times, the worst/longest test duration is
collected.
Note that even a skipped test can have a duration greater
than 0 because the skip operation itself is not free.
Signed-off-by: Ming Shao <ming.shao@intel.com>
Previously, unit test duration is collected within the
TC_START and Z_TC_END_RESULT macros. With existing tests,
the TC_START macro can be invoked by both the ztest fx
and the tests themselves. And the TC_START macro definition
went lengths to avoid the interference when it is invoked
within a unit test. This commit decouple the time collection
and the TC_STRAT/Z_TC_END_RESULT macros to fix this issue.
Now only the (old) ztest framework is responsible for the
test duration measure. The test duration stats of new ztest
fx is different from this btw.
Signed-off-by: Ming Shao <ming.shao@intel.com>
Add test summary after all test suites finish running.
The summary can be one-line or verbose, which is configured
with CONFIG_ZTEST_VERBOSE_SUMMARY. The one-line summary covers
overall suite stats. The verbose summary covers each test
function within the suite besides the one-line summary.
The new ztest output ultimately go through the printk. If
printk go through the logging subsystem, there may be log
messages dropped. And if log_panic is invoked, log messages
can be flushed in a mess. So several explicit log flush
are used when printing summary to ensure no content is lost
and content is in good shape.
Some macros are shared between old and new ztests. Such as
TC_START_PRINT and TC_END_PRINT. The are defined accordingly.
Signed-off-by: Ming Shao <ming.shao@intel.com>
Support verbose or one-line summary at test suite level.
Support verbose or no output at test function level.
Totally 4 combinations configurable:
- function verbose + suite verbose
- function verbose + suite oneline
- no function output + suite verbose
- no function output + suite oneline
Signed-off-by: Ming Shao <ming.shao@intel.com>
Updates the ztest_test_fail() function to allow failures in setup.
When executed, a failed assert will fail every test in the suite owning
the setup function. This was verified by adding a suite which asserts
in the setup function and has a test that should pass. During
exeuction, ztest marks the test as failing.
In order to verify exection I also added 2 new APIs:
- ZTEST_EXPECT_FAIL(suite_name, test_name)
- ZTEST_EXPECT_SKIP(suite_name, test_name)
Signed-off-by: Yuval Peress <peress@google.com>
In scenarios where test_main is overridden ztest_run_all
may be invoked multiple times leading to the verify check to
fail inadvertently.
Signed-off-by: Al Semjonovs <asemjonovs@google.com>
If the test is run with the config NO_OPTIMIZATIONS enabled then the
stack size usage increases by around 80% for ARM platforms.
Increase the stack size used in test cases that enables building with no
optimizations for ARM.
Update description on TEST_ARM_CORTEX_M since it was outdated and said
it was only used for a single purpose.
Fixes: #47930Fixes: #47929Fixes: #47855
Signed-off-by: Joakim Andersson <joakim.andersson@nordicsemi.no>
