1. Test suites in prior ztest serve no purpose other than logical
ordering of tests into a named-group. Move the construct of setup and
teardown into the test suite and away from individual tests.
Additionally, add the constructs of before/after to the test suites.
This model more closely resembels other testing frameworks such as gTest
and Junit.
2. Test can be added to a suite by using ZTEST() or ZTEST_F() where _F
stands for fixture. In the case where _F is used, the argument `this`
will be provided with the type `struct suite_name##_fixture*`. Again,
this models other modern testing frameworks and allows the test to
directly access the already set up data related to the test suite.
3. Add the concept of test rules (from Junit). Rules are similar to the
before/after functions of the test suites but are global and run on all
suites. An example of a test rule can be to check that nothing was
logged to ERROR. The rule can cause the test to fail if anything was
logged to ERROR during an integration test. Another example would be a
rule that verifies that tests ran within some defined timeout.
Signed-off-by: Yuval Peress <peress@google.com>
Because RISC-V arch (M extension) doesn't trigger exception for
division-by-zero, this test can't support RISC-V and is disabled in
RISC-V.
Signed-off-by: Jim Shu <cwshu@andestech.com>
Introduce a weak implementation of test_main() which calls:
* ztest_run_registered_test_suites(NULL);
* ztest_verify_all_registered_test_suites_ran();
This will attempt to run all registered test suites and verify that
they each ran.
Signed-off-by: Yuval Peress <peress@chromium.org>
Add new functionality to ztest to improve test modularity. The two
primary new entry points are:
* ztest_register_test_suite
* ztest_run_registered_test_suites
When registering a new test suite, users provide the name as well as
an optional predicate used to filter the tests for each run. Using NULL
as the predicate ensures that the test is run exactly once (after which
it is automatically filtered from future runs).
Calls to ztest_run_registered_test_suites take a state pointer as an
argument. This allows the the pragma functions to decide whether the
test should be run.
The biggest benefit of this system (other than the ability to filter
tests and maintain a larger test state) is the ability to better
modularize the test source code. Instead of all the various tests
having to coordinate and the main function having to know which tests
to run, each source file manages registering its own test
suite and handling the conditions for running the suite.
Signed-off-by: Yuval Peress <peress@chromium.org>
Issue #38643
Introduce a more powerful and well established mocking framework
into Zephyr. It also allows running the actual FFF tests using the
zephyr SDK and ztest framework to ensure compatibility.
As per TSC meeting, the fff.h header was directly added to
subsys/testsuite/include/. As per the guidelines, the file is exactly
the same as it is in FFF's library, but re-styled with clang-format.
The west.yml entry was added using the "ci" group and filtered by
default. (note that the tests will break until the CI actually
specifies that the group is needed).
Signed-off-by: Yuval Peress <peress@google.com>
Fix the trigger_fault_access. The fix is similar to cortex m.
Fix the trigger_fault_divide_zero. The fix is similar to qemu, i.e.
skip this testcase, because FVP would not trigger the exception for
dividing zero either.
Signed-off-by: Jaxson Han <jaxson.han@arm.com>
Move to CMake 3.20.0.
At the Toolchain WG it was decided to move to CMake 3.20.0.
The main reason for increasing CMake version is better toolchain
support.
Better toolchain support is added in the following CMake versions:
- armclang, CMake 3.15
- Intel oneAPI, CMake 3.20
- IAR, CMake 3.15 and 3.20
Signed-off-by: Torsten Rasmussen <Torsten.Rasmussen@nordicsemi.no>
The Cortex-R platforms may no use XIP which means userspace code with
have read and execute access to some of the RAM so that it can execute
code. However, userspace will not have access the the protected
read/write parts of RAM so it is safe to use the Cortex-M version of
trigger_fault_access for this test.
Signed-off-by: Bradley Bolen <bbolen@lexmark.com>
On the Cortex M0+, M0, M23 the div-by-zero is not mapped
as an hardware exception but triggers a HardFault.
The test case is skipped for any ARMV6_M_ARMV8_M_BASELINE.
Signed-off-by: Francois Ramu <francois.ramu@st.com>
This compile test should be checking if a symbol has been defined,
otherwise it is using the kconfig value directly. This fixes a warning
../src/main.c:115:37: warning: "CONFIG_SOC_QEMU_ARC" is not defined,
evaluates to 0 [-Wundef]
when using the -Wundef flag.
Signed-off-by: Bradley Bolen <bbolen@lexmark.com>
qemu arc has limited support for DIV/REM, no DIVZERO exception
throw, so we need to skip trigger_fault_divide_zero test case.
Signed-off-by: Watson Zeng <zhiwei@synopsys.com>
Put the testcase test_catch_assert_in_isr() to execute last, to prevent
it affects other test cases. Because when we caught an assert failure
in the ISR handler, it cannot be guaranteed that all the current
program status would be recovered.
Fixes#34844.
Signed-off-by: Enjia Mai <enjiax.mai@intel.com>
Make the divide instruction not being optimized of the ztest example
test case. Do this because it will be replaced by invalid opcode
exception instruction generated by GCC.
Fixes#33403.
Signed-off-by: Enjia Mai <enjiax.mai@intel.com>
em_starterkit has ICCM at 0x0 address, access to 0x0
address doesn't generate any exception, so we access
to 0xFFFFFFFF address instead to trigger exception.
Signed-off-by: Watson Zeng <zhiwei@synopsys.com>
When running the test on Cortex-M platforms, use access to
_current to trigger a memory access fault, as this address
is guaranteed to be in kernel ram and will not trigger
SecureFault exceptions.
Signed-off-by: Ioannis Glaropoulos <Ioannis.Glaropoulos@nordicsemi.no>
This functions is being called across the tree, no reason why it should
not be a public API.
The current usage violates a few MISRA rules.
Signed-off-by: Anas Nashif <anas.nashif@intel.com>
The quirky behavior when accessing of 0x0 address is related to
exact ARC SoCs/boards and not to entire ARC EM CPUs family. So
check for exact SoCs configs (CONFIG_SOC_ARC_IOT and CONFIG_SOC_NSIM)
instead of checking against CONFIG_CPU_ARCEM.
Signed-off-by: Eugeniy Paltsev <Eugeniy.Paltsev@synopsys.com>
Minor fixes to the sample output in the README file,
reflecting the correct error reason (3: oops) instead
of reason 0 (generic CPU error).
Signed-off-by: Ioannis Glaropoulos <Ioannis.Glaropoulos@nordicsemi.no>
test_catch_z_oops test case should not run in user mode,
since Z_OOPS needs to be called in supervisor mode, otherwise
we will be getting a reguser user memory access error that is
irrelevant to the test case (and simply is triggered by Z_OOPS
accessing the thread's syscall frame pointer).
In addition to that, we fix the argument of Z_OOPS, because, it
was triggering a null-pointer dereferencing, which results in an
error thrown before Z_OOPS is even executed (if null-pointer
exception is caught).
We also need to generate a dummy thread->syscall_frame, that
Z_OOPS implementation will access. We fix this to
_image_ram_start, because this memory is always part of the
image memory and is always available to supervisor mode. So,
accessing it won't trigger a security fault, for example.
Signed-off-by: Ioannis Glaropoulos <Ioannis.Glaropoulos@nordicsemi.no>
The test case suggests that "For most arch which support
userspace, derefencing NULL pointer will be caught by
exception.". This is certainly not true for Cortex-M, where
read access to address 0x0 is generally allowed. The reason
the test had been passing was either 1) because in many
Cortex-M platforms, including QEMU, address 0x0 is unmapped,
or 2) GCC is generating an undefining instruction. However,
now that we have activated the null pointer derefrencing
detection, we may end up with two exceptions and the test
would fail.
Change illegal access to something outside the mapped memory
area, e.g. 0xFFFFFFFF.
Signed-off-by: Ioannis Glaropoulos <Ioannis.Glaropoulos@nordicsemi.no>
Enlarge the test thread stack size, to fix test case fail
in some board which need more stack size for testing, when
CONFIG_EXCEPTION_DEBUG is enabled.
Signed-off-by: Enjia Mai <enjiax.mai@intel.com>
Fix issue #31339 and #31419, test case fail due to access NULL did not
trigger a fatal error in some platform such as nsim_em and iotdk.
Signed-off-by: Enjia Mai <enjiax.mai@intel.com>
This commit is a followup to PR #25808 which updates the tests to
ensure the REQUIRED keyword is also used for the ZephyrUnitest package.
This provides a better error message when building with CMake and
forgetting ZEPHYR_BASE or not registering Zephyr in the CMake package
registry.
Signed-off-by: Torsten Rasmussen <Torsten.Rasmussen@nordicsemi.no>
... because it is (required).
This makes a difference when building with CMake and forgetting
ZEPHYR_BASE or not registering Zephyr in the CMake package registry.
In this particular case, REQUIRED turns this harmless looking log
statement:
-- Could NOT find Zephyr (missing: Zephyr_DIR)
-- The C compiler identification is GNU 9.3.0
-- The CXX compiler identification is GNU 9.3.0
-- Check for working C compiler: /usr/bin/cc
-- ...
-- ...
-- ...
-- Detecting CXX compile features
-- Detecting CXX compile features - done
CMake Error at CMakeLists.txt:8 (target_sources):
Cannot specify sources for target "app" which is not built by
this project.
... into this louder, clearer, faster and (last but not least) final
error:
CMake Error at CMakeLists.txt:5 (find_package):
Could not find a package configuration file provided by "Zephyr" with
any of the following names:
ZephyrConfig.cmake
zephyr-config.cmake
Add the installation prefix of "Zephyr" to CMAKE_PREFIX_PATH or set
"Zephyr_DIR" to a directory containing one of the above files. If
"Zephyr" provides a separate development package or SDK, be sure it
has been installed.
-- Configuring incomplete, errors occurred!
Signed-off-by: Marc Herbert <marc.herbert@intel.com>
The following commit:
commit 407b49b35c (refs/bisect/bad)
Author: Torsten Rasmussen <Torsten.Rasmussen@nordicsemi.no>
Date: Wed Feb 12 15:00:46 2020 +0100
cmake: use find_package to locate Zephyr
breaks as we don't find the ZephyrUnittest package. For now revert to
the old means until a proper fix can be made.
Signed-off-by: Kumar Gala <kumar.gala@linaro.org>
Using find_package to locate Zephyr.
Old behavior was to use $ENV{ZEPHYR_BASE} for inclusion of boiler plate
code.
Whenever an automatic run of CMake happend by the build system / IDE
then it was required that ZEPHYR_BASE was defined.
Using ZEPHYR_BASE only to locate the Zephyr package allows CMake to
cache the base variable and thus allowing subsequent invocation even
if ZEPHYR_BASE is not set in the environment.
It also removes the risk of strange build results if a user switchs
between different Zephyr based project folders and forgetting to reset
ZEPHYR_BASE before running ninja / make.
Signed-off-by: Torsten Rasmussen <Torsten.Rasmussen@nordicsemi.no>
A Kconfig boolean is added to allow users to provide their own
output strings when running tests via ztest.
This allows changing e.g. the PASS/FAIL/SKIPPED strings,
add counters, change separators, and similar.
A test using the feature and relevant documentation is added.
Signed-off-by: Torstein Grindvik <torstein.grindvik@nordicsemi.no>
This commit changes zassert_mem_equal as a macro instead
of a function implementation.
In the previous implementation when an assertion fails
the location inside ztest_assert.h file was displayed.
This modification displays the location where zassert_mem_equal
was used.
Signed-off-by: Radoslaw Koppel <radoslaw.koppel@nordicsemi.no>
Update the files which contain no license information with the
'Apache-2.0' SPDX license identifier. Many source files in the tree are
missing licensing information, which makes it harder for compliance
tools to determine the correct license.
By default all files without license information are under the default
license of Zephyr, which is Apache version 2.
Signed-off-by: Anas Nashif <anas.nashif@intel.com>
Move test related code and the testsuite away from tests/ and make it a
proper subsystem.
The way tests were integrate in the tree was not obvious and actual
tests were intermixed with the testsuite code.
This will allow us to have trees with the testcode and without the
samples by just remove the folders tests/ and samples, needed for
isolating actual code from test/sample code.
Signed-off-by: Anas Nashif <anas.nashif@intel.com>
This was never a long-term solution, more of a gross hack
to get test cases working until we could figure out a good
end-to-end solution for memory domains that generated
appropriate linker sections. Now that we have this with
the app shared memory feature, and have converted all tests
to remove it, delete this feature.
To date all userspace APIs have been tagged as 'experimental'
which sidesteps deprecation policies.
Signed-off-by: Andrew Boie <andrew.p.boie@intel.com>
* Newlib now defines a special z_newlib_partition containing
all globals relevant to newlib. Most of these are in libc.a
with a heap tracking variable in newlib's hooks.
* Both C libraries now expose a k_mem_partition containing the
bounds of the malloc heap arena. Threads that want to use
libc malloc() will need to add this to their memory domain.
* z_newlib_get_heap_bounds has been removed, in favor of the
memory partition for the heap arena
* ztest now includes the C library partitions in its memory
domain.
* The mem_alloc test now runs in user mode to prove that this
all works for both C libraries.
Signed-off-by: Andrew Boie <andrew.p.boie@intel.com>
If CONFIG_APP_SHARED_MEM is enabled, ztest will set up a
default memory domain with a single partition on startup.
Test case globals may be added to the partition via
ZTEST_BMEM/ZTEST_DMEM macros, add their own partitions,
or leave the domain and join another one.
Signed-off-by: Andrew Boie <andrew.p.boie@intel.com>
The public APIs for application shared memory are now
properly documented and conform to zephyr naming
conventions.
Signed-off-by: Andrew Boie <andrew.p.boie@intel.com>
Move to latest cmake version with many bug fixes and enhancements.
Signed-off-by: Anas Nashif <anas.nashif@intel.com>
Signed-off-by: Carles Cufi <carles.cufi@nordicsemi.no>
