Added test command line arguments to filter
which tests are executed. Filtered tests should follow
suiteA::test1,suiteB::test2 format.
Signed-off-by: Al Semjonovs <asemjonovs@google.com>
Adds few missing zephyr/ prefixes to leftover #include statements that
either got added recently or were using double quote format.
Signed-off-by: Fabio Baltieri <fabiobaltieri@google.com>
Update the new API to use K_USER as the flags for both
CONFIG_USERSPACE and CONFIG_TEST_USERSPACE. Also, fix the linker
script to properly include the suites, tests, and rules.
Fixes#44108
Signed-off-by: Yuval Peress <peress@google.com>
output of ztest is not easy to ready with many testsuites running in
sequence, make it a bit more readable.
Signed-off-by: Anas Nashif <anas.nashif@intel.com>
Added macro to test that a value falls within a defined range, including
the provided end points.
Signed-off-by: Jordan Yates <jordan.yates@data61.csiro.au>
Enable ZTEST_DO_THE_SHUFFLE to shuffle the order tests are ran.
Additional configs ZTEST_DO_THE_SHUFFLE_SUITE_REPEAT_COUNT
ZTEST_DO_THE_SHUFFLE_TEST_REPEAT_COUNT specify the number of times
the test or suite is executed.
Signed-off-by: Al Semjonovs <asemjonovs@google.com>
In order to bring consistency in-tree, migrate all subsystems code to
the new prefix <zephyr/...>. Note that the conversion has been scripted,
refer to zephyrproject-rtos#45388 for more details.
Signed-off-by: Gerard Marull-Paretas <gerard.marull@nordicsemi.no>
The ztest unittest module relies on some headers that fake the real
ones, e.g. <zephyr/arch/cpu.h>. When new header location was introduced,
their path was not updates. This patch fixes this problem and provides
ztest with both, legacy and new include paths so that it can keep
working on both scenarios.
Signed-off-by: Gerard Marull-Paretas <gerard.marull@nordicsemi.no>
Update the new API to use K_USER as the flags for both
CONFIG_USERSPACE and CONFIG_TEST_USERSPACE. Also, fix the linker
script to properly include the suites, tests, and rules.
Fixes#44108
Signed-off-by: Yuval Peress <peress@google.com>
define test run id and make it available to ztest. This will be used to
verify we are evaluating the same test we have just built and flashed
and that we are not looking at some old output.
Existing code in ztest will use this, and twister will define the run
id.
Signed-off-by: Anas Nashif <anas.nashif@intel.com>
Make sure that the test rules' `before` function runs before the
suite's. This allows the suite to override any defaults set by the
rule.
Signed-off-by: Yuval Peress <peress@google.com>
The idle thread got an index suffix in #23536 to make it easier to
identify different idle threads on different cores. This looks out of
place on single-core devices when the idle thread is listed next to
other kernel threads, such as main.
Remove the idle thread index on single-core platforms, and replace all
references to this format in tests and documentation.
Signed-off-by: Trond Einar Snekvik <Trond.Einar.Snekvik@nordicsemi.no>
Delay start of threads and timer to ensure that setup
is completed. Especially, vital on multiple CPUs.
Signed-off-by: Krzysztof Chruscinski <krzysztof.chruscinski@nordicsemi.no>
The `handle_signal()` function in the new ztest API (`ztest_new.c`) uses
an array of strings to lookup a friendly name for each test phase, but
the array only has three elements despite there being six test phases.
This can lead to an out-of-bounds issue. Replace the array with a helper
function and switch statement.
Signed-off-by: Tristan Honscheid <honscheid@google.com>
This change introduced some weird failures on x86 that will take some
time to untangle, and wasn't a particularly important feature to
merge. Revert for now.
This reverts commit adc901aa6a.
Signed-off-by: Andy Ross <andrew.j.ross@intel.com>
Some hardware has "interesting" configuration like asymmetric default
interrupt masking (the intel_adsp devices in particular, but x86's
IO-APIC driver has tripped over this in the past too) that needs
special treatment if you want to run something on "core 1"
specifically, and 1cpu test cases pretty much by definition are going
to have been written without SMP details in mind. Switch the logic
around a tiny bit such that these test cases always run on CPU ID zero
explicitly.
Actually in practice this was ALMOST guaranteed to be true already,
because test setup and happens serially, having been started on the
main thread, which starts on CPU 0 by definition. Then the test
teardown aborts all the spawned threads that might have been running
on CPUs 1+, so those reach idle, and the next test case starts
syncronously on the same thread (and thus CPU) where it started. But
nonetheless that wasn't actually enforced, and we've found at least
one simulation environment where timing conspires to break things.
Signed-off-by: Andy Ross <andrew.j.ross@intel.com>
Some SOCs cannot be flashed reliably in low power modes. If
CONFIG_ZTEST_NO_YIELD is selected, do not yield to the idle thread after
testsuite completes, so that the SOC will not enter low power mode.
Signed-off-by: Daniel DeGrasse <daniel.degrasse@nxp.com>
Add ztest_test_skip() support to non-kernel tests by implementing
long jump buffer and TC_SKIP result collection.
Signed-off-by: Thomas Ebert Hansen <thoh@oticon.com>
When system clock is set globally for the test which is executed
on various qemu platforms it may happen that clock is set too
high compared to CPU clock frequency. In that case test may stuck.
Added check and test skipping in that case.
Signed-off-by: Krzysztof Chruscinski <krzysztof.chruscinski@nordicsemi.no>
Fix an issue where a test's suite name was misspelled and thus the
test never ran because it wasn't associated with a suite. Example:
ZTEST_SUITE(my_suite, NULL, NULL, NULL, NULL, NULL);
ZTEST(myy_suite, test)
{
}
The above will no longer pass since `myy_suite` with a double `y` will
fail the verification.
Signed-off-by: Yuval Peress <peress@google.com>
Use UTIL_CAT which resolves input arguments instead of direct
preprocessor concatenation.
Signed-off-by: Krzysztof Chruscinski <krzysztof.chruscinski@nordicsemi.no>
Flushing logs here is especially vital if LOG_PRINTK is enabled.
In that case printk goes to log and without flushing after all
tests are run you will not get the final report out.
Signed-off-by: Krzysztof Chruscinski <krzysztof.chruscinski@nordicsemi.no>
A lot of tests need to be able to get their current tid and do some
action with it. It makes sense for the `before` function/rule to be
able to run in the same thread as the test. Note that the `after`
function does not run in the same thread because we need to guarantee
that it will run.
Signed-off-by: Yuval Peress <peress@google.com>
Using macro which will resolve complex SUITE_NAME. Without
it wrong name was assigned when SUITE_NAME was consists of
concatenated defines.
Signed-off-by: Krzysztof Chruscinski <krzysztof.chruscinski@nordicsemi.no>
Add maximum timeout used for conversion to Kconfig. Option is used
to determine which conversion algorithm to use: faster but overflowing
earlier or slower without early overflow.
Signed-off-by: Krzysztof Chruscinski <krzysztof.chruscinski@nordicsemi.no>
Fix initial thread priority and move starting of the
test timer after threads initialization to ensure that
it does not expire before threads are initiated.
Signed-off-by: Krzysztof Chruscinski <krzysztof.chruscinski@nordicsemi.no>
Currently, ztress is not supporting SMP, mainly because of
cpu load calculation algorithm. It may be re-enabled in the
future.
Signed-off-by: Krzysztof Chruscinski <krzysztof.chruscinski@nordicsemi.no>
1. Test suites in prior ztest serve no purpose other than logical
ordering of tests into a named-group. Move the construct of setup and
teardown into the test suite and away from individual tests.
Additionally, add the constructs of before/after to the test suites.
This model more closely resembels other testing frameworks such as gTest
and Junit.
2. Test can be added to a suite by using ZTEST() or ZTEST_F() where _F
stands for fixture. In the case where _F is used, the argument `this`
will be provided with the type `struct suite_name##_fixture*`. Again,
this models other modern testing frameworks and allows the test to
directly access the already set up data related to the test suite.
3. Add the concept of test rules (from Junit). Rules are similar to the
before/after functions of the test suites but are global and run on all
suites. An example of a test rule can be to check that nothing was
logged to ERROR. The rule can cause the test to fail if anything was
logged to ERROR during an integration test. Another example would be a
rule that verifies that tests ran within some defined timeout.
Signed-off-by: Yuval Peress <peress@google.com>
Added framework for concurrency testing. Framework setup multiple
priority contexts and executes user handlers in that context. Test
terminates after certain number of repetitions or preemptions or
when timeout occurs. It can also be aborted by the user.
Framework can be used for testing resiliency to preemptions.
Signed-off-by: Krzysztof Chruscinski <krzysztof.chruscinski@nordicsemi.no>
Add guard to not pend until all logs are flushed when default
logging thread is disabled. In that case, logging has no control
where logs are processed and sleeping may not lead to flushing
log data. That may result in test hanging.
Signed-off-by: Krzysztof Chruscinski <krzysztof.chruscinski@nordicsemi.no>
This changes both k_mem_domain_add_partition() and
k_mem_domain_remove_partition() to return errors instead of
asserting when errors are encountered. This gives the application
chance to recover.
The arch_mem_domain_parition_add()/_remove() will be modified
later together with all the other arch_mem_domain_*() changes
since the architecture code for partition addition and removal
functions usually cannot be separately changed.
Signed-off-by: Daniel Leung <daniel.leung@intel.com>
When deferred mode is used, logging is using lowest priority
thread to process the logs. When test cases are performed one
by one processor never reaches lowest priority thread until
whole suite is completed. Added flushing after each test case.
Feature is optional (by default enabled).
CONFIG_TEST_LOGGING_FLUSH_AFTER_TEST=n to disable.
Signed-off-by: Krzysztof Chruscinski <krzysztof.chruscinski@nordicsemi.no>
This change adds `k_cycle_get_64()` on platforms that
support a 64-bit cycle counter.
The interface functions `arch_k_cycle_get_64()` and
`sys_clock_cycle_get_64()` are also introduced.
Fixes#39934
Signed-off-by: Christopher Friedt <chrisfriedt@gmail.com>
Introduce a weak implementation of test_main() which calls:
* ztest_run_registered_test_suites(NULL);
* ztest_verify_all_registered_test_suites_ran();
This will attempt to run all registered test suites and verify that
they each ran.
Signed-off-by: Yuval Peress <peress@chromium.org>
Add new functionality to ztest to improve test modularity. The two
primary new entry points are:
* ztest_register_test_suite
* ztest_run_registered_test_suites
When registering a new test suite, users provide the name as well as
an optional predicate used to filter the tests for each run. Using NULL
as the predicate ensures that the test is run exactly once (after which
it is automatically filtered from future runs).
Calls to ztest_run_registered_test_suites take a state pointer as an
argument. This allows the the pragma functions to decide whether the
test should be run.
The biggest benefit of this system (other than the ability to filter
tests and maintain a larger test state) is the ability to better
modularize the test source code. Instead of all the various tests
having to coordinate and the main function having to know which tests
to run, each source file manages registering its own test
suite and handling the conditions for running the suite.
Signed-off-by: Yuval Peress <peress@chromium.org>
According to the rule MISRAC-2012 21.4.a the standard header
file <setjmp.h> shall not be used. Suppress it, because it raises
violation in a testcode, not in a runtime code.
Tag suppresses reporting of violation for the current file,
starting from the line where the suppression is located.
It is a deliberate deviation.
Found as a coding guideline violation (MISRA R21.4.a) by static
coding scanning tool.
Signed-off-by: Maksim Masalski <maksim.masalski@intel.com>
