CMake will translate from the toolchain variant 'gccarmemb' to
'gnuarmemb', but sanitycheck does not. This causes inconsistent and
therefore confusing behaviour between CMake and sanitycheck.
Until gccarmemb is dropped support for, do the same translation with
sanitycheck.
Signed-off-by: Sebastian Bøe <sebastian.boe@nordicsemi.no>
Add a new option as fixture in harness configurations for utilizing
sanitycheck to identify test cases that require external hardware
such as sensor, ble, networking for validation. The config will be
added to yaml files with unique fixture name to identify each hardware
and allow automation to trigger test execution on setup having the
specific fixture enabled. Also, remove the default required for type and
regex configs that is not essential in case of ztest based test cases.
Signed-off-by: Praful Swarnakar <praful.swarnakar@intel.com>
New shell support features like:
- multi-instance
- command tree
- static and dynamic commands
- multiline
- help print function
- smart tab (autocompletion)
- meta-keys
- history, wildcards etc.
- generic transport (initially, uart present)
Signed-off-by: Jakub Rzeszutko <jakub.rzeszutko@nordicsemi.no>
Signed-off-by: Krzysztof Chruscinski <krzysztof.chruscinski@nordicsemi.no>
Signed-off-by: Piotr Zięcik <piotr.ziecik@nordicsemi.no>
Do not close console after PASS is reported, wait a bit for any
remaining messages from the tests, sometimes we have faults that need to
be parsed.
This now works for Qemu handler, support for other handlers to follow.
Fixes#9646
Signed-off-by: Anas Nashif <anas.nashif@intel.com>
multiprocessing.cpu_count() already returns the number of threads
(not cores), no need to multiply it by 2.
Signed-off-by: Oleg Zhurakivskyy <oleg.zhurakivskyy@intel.com>
Summary: revised attempt at addressing issue 6290. The
following provides an alternative to using
CONFIG_APPLICATION_MEMORY by compartmentalizing data into
Memory Domains. Dependent on MPU limitations, supports
compartmentalized Memory Domains for 1...N logical
applications. This is considered an initial attempt at
designing flexible compartmentalized Memory Domains for
multiple logical applications and, with the provided python
script and edited CMakeLists.txt, provides support for power
of 2 aligned MPU architectures.
Overview: The current patch uses qualifiers to group data into
subsections. The qualifier usage allows for dynamic subsection
creation and affords the developer a large amount of flexibility
in the grouping, naming, and size of the resulting partitions and
domains that are built on these subsections. By additional macro
calls, functions are created that help calculate the size,
address, and permissions for the subsections and enable the
developer to control application data in specified partitions and
memory domains.
Background: Initial attempts focused on creating a single
section in the linker script that then contained internally
grouped variables/data to allow MPU/MMU alignment and protection.
This did not provide additional functionality beyond
CONFIG_APPLICATION_MEMORY as we were unable to reliably group
data or determine their grouping via exported linker symbols.
Thus, the resulting decision was made to dynamically create
subsections using the current qualifier method. An attempt to
group the data by object file was tested, but found that this
broke applications such as ztest where two object files are
created: ztest and main. This also creates an issue of grouping
the two object files together in the same memory domain while
also allowing for compartmenting other data among threads.
Because it is not possible to know a) the name of the partition
and thus the symbol in the linker, b) the size of all the data
in the subsection, nor c) the overall number of partitions
created by the developer, it was not feasible to align the
subsections at compile time without using dynamically generated
linker script for MPU architectures requiring power of 2
alignment.
In order to provide support for MPU architectures that require a
power of 2 alignment, a python script is run at build prior to
when linker_priv_stacks.cmd is generated. This script scans the
built object files for all possible partitions and the names given
to them. It then generates a linker file (app_smem.ld) that is
included in the main linker.ld file. This app_smem.ld allows the
compiler and linker to then create each subsection and align to
the next power of 2.
Usage:
- Requires: app_memory/app_memdomain.h .
- _app_dmem(id) marks a variable to be placed into a data
section for memory partition id.
- _app_bmem(id) marks a variable to be placed into a bss
section for memory partition id.
- These are seen in the linker.map as "data_smem_id" and
"data_smem_idb".
- To create a k_mem_partition, call the macro
app_mem_partition(part0) where "part0" is the name then used to
refer to that partition. This macro only creates a function and
necessary data structures for the later "initialization".
- To create a memory domain for the partition, the macro
app_mem_domain(dom0) is called where "dom0" is the name then
used for the memory domain.
- To initialize the partition (effectively adding the partition
to a linked list), init_part_part0() is called. This is followed
by init_app_memory(), which walks all partitions in the linked
list and calculates the sizes for each partition.
- Once the partition is initialized, the domain can be
initialized with init_domain_dom0(part0) which initializes the
domain with partition part0.
- After the domain has been initialized, the current thread
can be added using add_thread_dom0(k_current_get()).
- The code used in ztests ans kernel/init has been added under
a conditional #ifdef to isolate the code from other tests.
The userspace test CMakeLists.txt file has commands to insert
the CONFIG_APP_SHARED_MEM definition into the required build
targets.
Example:
/* create partition at top of file outside functions */
app_mem_partition(part0);
/* create domain */
app_mem_domain(dom0);
_app_dmem(dom0) int var1;
_app_bmem(dom0) static volatile int var2;
int main()
{
init_part_part0();
init_app_memory();
init_domain_dom0(part0);
add_thread_dom0(k_current_get());
...
}
- If multiple partitions are being created, a variadic
preprocessor macro can be used as provided in
app_macro_support.h:
FOR_EACH(app_mem_partition, part0, part1, part2);
or, for multiple domains, similarly:
FOR_EACH(app_mem_domain, dom0, dom1);
Similarly, the init_part_* can also be used in the macro:
FOR_EACH(init_part, part0, part1, part2);
Testing:
- This has been successfully tested on qemu_x86 and the
ARM frdm_k64f board. It compiles and builds power of 2
aligned subsections for the linker script on the 96b_carbon
boards. These power of 2 alignments have been checked by
hand and are viewable in the zephyr.map file that is
produced during build. However, due to a shortage of
available MPU regions on the 96b_carbon board, we are unable
to test this.
- When run on the 96b_carbon board, the test suite will
enter execution, but each individaul test will fail due to
an MPU FAULT. This is expected as the required number of
MPU regions exceeds the number allowed due to the static
allocation. As the MPU driver does not detect this issue,
the fault occurs because the data being accessed has been
placed outside the active MPU region.
- This now compiles successfully for the ARC boards
em_starterkit_em7d and em_starterkit_em7d_v22. However,
as we lack ARC hardware to run this build on, we are unable
to test this build.
Current known issues:
1) While the script and edited CMakeLists.txt creates the
ability to align to the next power of 2, this does not
address the shortage of available MPU regions on certain
devices (e.g. 96b_carbon). In testing the APB and PPB
regions were commented out.
2) checkpatch.pl lists several issues regarding the
following:
a) Complex macros. The FOR_EACH macros as defined in
app_macro_support.h are listed as complex macros needing
parentheses. Adding parentheses breaks their
functionality, and we have otherwise been unable to
resolve the reported error.
b) __aligned() preferred. The _app_dmem_pad() and
_app_bmem_pad() macros give warnings that __aligned()
is preferred. Prior iterations had this implementation,
which resulted in errors due to "complex macros".
c) Trailing semicolon. The macro init_part(name) has
a trailing semicolon as the semicolon is needed for the
inlined macro call that is generated when this macro
expands.
Update: updated to alternative CONFIG_APPLCATION_MEMORY.
Added config option CONFIG_APP_SHARED_MEM to enable a new section
app_smem to contain the shared memory component. This commit
seperates the Kconfig definition from the definition used for the
conditional code. The change is in response to changes in the
way the build system treats definitions. The python script used
to generate a linker script for app_smem was also midified to
simplify the alignment directives. A default linker script
app_smem.ld was added to remove the conditional includes dependency
on CONFIG_APP_SHARED_MEM. By addining the default linker script
the prebuild stages link properly prior to the python script running
Signed-off-by: Joshua Domagalski <jedomag@tycho.nsa.gov>
Signed-off-by: Shawn Mosley <smmosle@tycho.nsa.gov>
Add new kwyboard to board definition to allow blacklisting boards. This
is needed when a board is broken causing CI to fail without a fix in
sight.
Add:
sanitycheck: false
to the board yaml file to disable the board. By default, the value is
set to true.
Signed-off-by: Anas Nashif <anas.nashif@intel.com>
sanitycheck not printing QEMU console in some cases where a crash
happens and when state is not set correctly.
Fixes#9061
Signed-off-by: Anas Nashif <anas.nashif@intel.com>
The two handlers were doing pretty much the same with minor differences,
unify them into one single handler BinaryHandler that will be able to
handle additional targets.
Signed-off-by: Anas Nashif <anas.nashif@intel.com>
refactor all add_goal routines in to one single function that can handle
multiple platforms. Move everything to one single add_goal function.
Signed-off-by: Anas Nashif <anas.nashif@intel.com>
For native builds it does not make much sense to calculate
the size of the suposed RAM or ROM, or to check that new
unexpected sections did not appear.
So let's save the time and not do it.
Signed-off-by: Alberto Escolar Piedras <alpi@oticon.com>
Added a new command line options to sanitycheck:
--enable-coverage which will compile for native_posix
with CONFIG_COVERAGE set, and unit tests accordingly.
+
Now -C --coverage implies also --enable-coverage.
Background:
After 608778a4de
it is possible to add Kconfig options from command
line during the cmake invocation.
So we can use it to set CONFIG_COVERAGE for the native_posix
target when we need to instead of relaying on it always
being compiled with coverage enabled.
Signed-off-by: Alberto Escolar Piedras <alpi@oticon.com>
VERBOSE if set to any value enables verbose build output,
setting to 0 does not have the intended effect.
Signed-off-by: Andrew Boie <andrew.p.boie@intel.com>
In verbose mode (-v) progress of running test cases number out of total
number is not printed. Thus, if there are many test cases, it is
impossible to follow the progress. This commit adds printing of current
test case and total numbers to the verbose output.
Signed-off-by: Ruslan Mstoi <ruslan.mstoi@intel.com>
Export tests to a file with Section, subsection and identifier as
columns making it easy to import testcases into test management system.
Signed-off-by: Anas Nashif <anas.nashif@intel.com>
This section got added in commit 470349c25a ("Bluetooth: settings:
Add support for per-submodule handlers"), but sanitycheck didn't know
about it and was whining.
Signed-off-by: Andy Ross <andrew.j.ross@intel.com>
Ninja and GNU make don't play well with each other. Both try to start
enough processes to keep the system's CPUs busy, resulting in an
O(N^2) system load in the number of processors.
Long term, Ninja seems likely to support the GNU make jobserver
mechanism for sharing access to parallelism. But for now we can get
90% of the way there with a simple hack: just run ninja in serial mode
with -j1. Sanitycheck when run in non-trivial circumstances has
PLENTY of parallelism just from the number of test cases.
One interesting note is that even with -j1, system loads under ninja
are rather higher. That may be because of significant work done in
the (serial) makefiles that dilutes the parallelism of the eventual
build, or possibly because ninja itself is multithreaded in its setup
code. So I tweaked the number of jobs down to keep the load roughly
where it is with make.
With this change, I see no difference in behavior or system load, and a
~24% improvement in runtime.
Signed-off-by: Andy Ross <andrew.j.ross@intel.com>
- Some tests start with test_, some do not, so make sure we parse both.
- Parse skipped tests
- Improve handling of test case identifier
- Handle Exceptions in device handler
Signed-off-by: Anas Nashif <anas.nashif@intel.com>
We are passing global arguments from one level to the next when those
variables are available globally. Reduce the arguments and remove unused
arguments as well.
Signed-off-by: Anas Nashif <anas.nashif@intel.com>
init class in one place, no need to duplicate all class members in every
subclass.
run_log is not needed in the handler class.
Signed-off-by: Anas Nashif <anas.nashif@intel.com>
Parse the test results and create a test report with more granular
results that can be imported to into test management/reporting system.
Signed-off-by: Anas Nashif <anas.nashif@intel.com>
This will allow us to run sanitycheck on real devices and get reporting
out of it the same way we do that with Qemu.
To use this, run sanitycheck with the following new options:
scripts/sanitycheck --device-testing --device-serial /dev/ttyACM0 -p
frdm_k64f -T tests/crypto/
--device-serial denotes the serial device the board is connected to.
This needs to be accessible by the user running sanitycheck. You can
run this on one board only at a time, the board is specified using the
--platform option.
This was tested with only a few boards, some board will not work
because how they reset the serial device during flashing.
Signed-off-by: Anas Nashif <anas.nashif@intel.com>
After removing the -T linker warning for the POSIX arch
we can, and should, also treat its linker warnings as errors
Signed-off-by: Alberto Escolar Piedras <alpi@oticon.com>
This packs a few improvements:
- Add a smarter regex that will catch multiple combinations of
ztest_[user_]unit_test[_setup_teardown](NAME[, setup, teardown])
as well as single liners like:
ztest_test_suite(mutex_complex, ztest_user_unit_test(TESTNAME));
- Limit how much we look forward in suite_regex -- we don't have to
look past the first argument, otherwise we consume too much and the
loopup at suite_regex_match.start() will start too late.
- Remove include_regex, unused
- Fix the path where we warn about matches in achtung_regexes--it
needed a few decodes and to use error() vs the unexistant warning()
- Cleanup the path to produce the subcase names, doing the decode and
the purging of any test_ prefix in scan_path().
Signed-off-by: Inaky Perez-Gonzalez <inaky.perez-gonzalez@intel.com>
This addresses issue #7146.
The current helptext does not state that the directory
will be deleted.
Signed-off-by: Håkon Øye Amundsen <haakon.amundsen@nordicsemi.no>
Parse all yaml file and create a list of declared testcases. This does
list the individual tests inside test projects, not only the projects
containing the tests, for example:
$ sanitycheck --list-tests -T tests/net/socket/
- net.socket.udp.send_recv_2_sock
- net.socket.udp.v4_sendto_recvfrom
- net.socket.udp.v6_sendto_recvfrom
- net.socket.udp.v4_bind_sendto
- net.socket.udp.v6_bind_sendto
- net.socket.getaddrinfo_ok
- net.socket.getaddrinfo_no_host
- net.socket.tcp.v4_send_recv
- net.socket.tcp.v6_send_recv
- net.socket.tcp.v4_sendto_recvfrom
- net.socket.tcp.v6_sendto_recvfrom
- net.socket.tcp.v4_sendto_recvfrom_null_dest
- net.socket.tcp.v6_sendto_recvfrom_null_dest
13 total.
Signed-off-by: Anas Nashif <anas.nashif@intel.com>
This parses the tests that run within a test project/application from
the source code and gives us a view of what was run, skipped and what
was blocked due to early termination of the test.
Signed-off-by: Anas Nashif <anas.nashif@intel.com>
If we have some error parsing a testcase or other files we treat these
as errors and will exit before continuing on building other tests.
Signed-off-by: Kumar Gala <kumar.gala@linaro.org>
In many cases we are calling cmake twice with the same options, first to
build, and then we do the same thing when we want to call 'make run'.
Just call cmake once.
Signed-off-by: Anas Nashif <anas.nashif@intel.com>
When running large set of tests we always get a huge list of footprint
changes that mask the test results making them impossible to see on the
screen. Show the footprint results only on-demand and not on every
build.
Signed-off-by: Anas Nashif <anas.nashif@intel.com>
With this commit it is possible to add priority to sent or received
network packets. So user is able to send or receive higher priority
packets faster than lower level packets.
The traffic class support is activated by CONFIG_NET_TC_COUNT option.
The TC support uses work queues to separate the traffic. The
priority of the work queue thread specifies the ordering of the
network traffic. Each work queue thread handles traffic to one specific
work queue. Note that you should not enable traffic classes unless
you really need them by your application. Each TC thread needs
stack so this feature requires more memory.
It is possible to disable transmit traffic class support and keep the
receive traffic class support, or vice versa. If both RX and TX traffic
classes are enabled, then both will use the same number of queues
defined by CONFIG_NET_TC_COUNT option.
Fixes#6588
Signed-off-by: Jukka Rissanen <jukka.rissanen@linux.intel.com>
Move IP address settings from net_if to separate structs.
This is needed for VLAN support.
Signed-off-by: Jukka Rissanen <jukka.rissanen@linux.intel.com>
