The change adds an optional capability to kill process by PID in the
BinaryHandler. At the moment proc.terminate only closed the make
process, also closing the stdin/stdout, but this is impossible to detect
in .NET.
Signed-off-by: Dawid Wojciechowski <dwojciechowski@internships.antmicro.com>
Signed-off-by: Jan Kowalewski <jkowalewski@antmicro.com>
Signed-off-by: Piotr Zierhoffer <pzierhoffer@antmicro.com>
Extend the "slow" flag on tests to inhibit building too. The original
assumption was that building would be fast but running slow, but now
We have tests using a component (OpenThread) that wants to pull and
build software from github for every app.
Signed-off-by: Andy Ross <andrew.j.ross@intel.com>
The segment size computation has value, but it's actually a very slow
process that parallizes poorly. It seems to be bound by the Python
GIL doing the parsing, so never sees more than about 150% of CPU in
use even on wildly parallel systems. And it takes about 75-80
seconds, which is 15-20% of the entire runtime of the test on that
box!
And the only "failure" case this can detect (unexpected sections in
the output file) is now a duplicate of the orphan section warning
we've since enalbled at the linker level.
This defaults to enabling the test to preserve behavior (as I don't
know where all the existing users of the size report are to change
them), but long term we might consider making "disabled" the default
and switching this to an --enable flag.
Signed-off-by: Andy Ross <andrew.j.ross@intel.com>
When running valgrind in sanitycheck, use the suppression file
and dump the log in its own file.
Signed-off-by: Alberto Escolar Piedras <alpi@oticon.com>
If the test process returns an error (return code != 0),
it should be considered a failure, even if the handler
considers the test passed.
But this is not done when we are killing (SIGTERM) the processes,
as then the return code is not necessarily sensible.
Without this, for example, when running with valgrind, all valgrind
errors are missed.
Signed-off-by: Alberto Escolar Piedras <alpi@oticon.com>
In addition to Kconfig values from generated .config, also parse and
filter based on configs generated by device-tree.
Signed-off-by: Anas Nashif <anas.nashif@intel.com>
There's no current need for this and it makes work items
declared with K_WORK_DEFINE() inaccessible to user mode.
Signed-off-by: Andrew Boie <andrew.p.boie@intel.com>
Fix false passed on localized error message in make invocation.
Fixes#8348
Signed-off-by: Bobby Noelte <b0661n0e17e@gmail.com>
Signed-off-by: Anas Nashif <anas.nashif@intel.com>
We should leave ZEPHYR_BASE alone and create the test output wherever
we are running.
This will support running sanitycheck on test roots other than zephyr's
main git repo.
Signed-off-by: Anas Nashif <anas.nashif@intel.com>
right now we assume we are running the tests inside the Zephyr tree and
things do not work well when someone tries to use sanitycheck on tests
outside the tree with a different test root. Cleanup the name handling
and support running outside of Zephyr.
Signed-off-by: Anas Nashif <anas.nashif@intel.com>
Set COVERAGE option using an overlay instead of piping it through CMAKE
which breaks dependecies, i.e. if someone wants to disable that option
on the application level.
Signed-off-by: Anas Nashif <anas.nashif@intel.com>
Support specifying a test fixture attached to a board to filter in/out
tests that require this fixture to be able to run a test.
Signed-off-by: Anas Nashif <anas.nashif@intel.com>
Allow the user to disable the "unrecognized section" test. I can see
multiple use-cases for disabling the test.
If orphan sections exist and are dynamically or unpredictably named
the unrecognized section test will fail.
If out-of-tree sections exist, one might want to temporarily disable
the "unrecognized section" test until one has made it recognized.
The test is disabled through a CLI flag.
Signed-off-by: Sebastian Bøe <sebastian.boe@nordicsemi.no>
We now place the linker directives for the SW ISR table
in the common linker scripts, instead of repeating it
everywhere.
The table will be placed in RAM if dynamic interrupts are
enabled.
A dedicated section is used, as this data must not move
in between build phases.
Signed-off-by: Andrew Boie <andrew.p.boie@intel.com>
New Qemu is emitting binary charachters which is disturbing generation
of XML, so make sure we only log printable charachters.
Signed-off-by: Anas Nashif <anas.nashif@intel.com>
This puts the priviledged stack at the end of RAM.
This combines PR #10507 and #10542.
Fixes#10473Fixes#10474Fixes#10515
Signed-off-by: Adithya Baglody <adithya.nagaraj.baglody@intel.com>
Signed-off-by: Daniel Leung <daniel.leung@intel.com>
Some boards depend on environment variables so we want to make sure we
do not attempt to build boards requiring additional setup.
Add the section below into the board YAML file, sanitycheck will check
the environment and will only run tests on that board if the variables
are defined.
env:
- VAR1
- VAR2
Signed-off-by: Anas Nashif <anas.nashif@intel.com>
To be able to collect all coverage reports from the "native"
applications in the first CI instance, let's also place the
nrf52_bsim testcases first. The order now is:
unit_testing
native_posix
*_bsim
anything else
Signed-off-by: Alberto Escolar Piedras <alpi@oticon.com>
run tests based on sub-testcases, for example:
sanitycheck --sub-test net.app.app_tcp4_client_hostname_fail
will run all the tests in tests/net/app/net.app.
Useful for re-running tests that are reported in testrail and in reports
generated by sanitycheck.
Signed-off-by: Anas Nashif <anas.nashif@intel.com>
Do not check for board/toolchain combination and if a certain toolchain
is configured for a board, run with whatever toolchain we have.
Signed-off-by: Anas Nashif <anas.nashif@intel.com>
CMake will translate from the toolchain variant 'gccarmemb' to
'gnuarmemb', but sanitycheck does not. This causes inconsistent and
therefore confusing behaviour between CMake and sanitycheck.
Until gccarmemb is dropped support for, do the same translation with
sanitycheck.
Signed-off-by: Sebastian Bøe <sebastian.boe@nordicsemi.no>
Add a new option as fixture in harness configurations for utilizing
sanitycheck to identify test cases that require external hardware
such as sensor, ble, networking for validation. The config will be
added to yaml files with unique fixture name to identify each hardware
and allow automation to trigger test execution on setup having the
specific fixture enabled. Also, remove the default required for type and
regex configs that is not essential in case of ztest based test cases.
Signed-off-by: Praful Swarnakar <praful.swarnakar@intel.com>
New shell support features like:
- multi-instance
- command tree
- static and dynamic commands
- multiline
- help print function
- smart tab (autocompletion)
- meta-keys
- history, wildcards etc.
- generic transport (initially, uart present)
Signed-off-by: Jakub Rzeszutko <jakub.rzeszutko@nordicsemi.no>
Signed-off-by: Krzysztof Chruscinski <krzysztof.chruscinski@nordicsemi.no>
Signed-off-by: Piotr Zięcik <piotr.ziecik@nordicsemi.no>
Do not close console after PASS is reported, wait a bit for any
remaining messages from the tests, sometimes we have faults that need to
be parsed.
This now works for Qemu handler, support for other handlers to follow.
Fixes#9646
Signed-off-by: Anas Nashif <anas.nashif@intel.com>
multiprocessing.cpu_count() already returns the number of threads
(not cores), no need to multiply it by 2.
Signed-off-by: Oleg Zhurakivskyy <oleg.zhurakivskyy@intel.com>
Summary: revised attempt at addressing issue 6290. The
following provides an alternative to using
CONFIG_APPLICATION_MEMORY by compartmentalizing data into
Memory Domains. Dependent on MPU limitations, supports
compartmentalized Memory Domains for 1...N logical
applications. This is considered an initial attempt at
designing flexible compartmentalized Memory Domains for
multiple logical applications and, with the provided python
script and edited CMakeLists.txt, provides support for power
of 2 aligned MPU architectures.
Overview: The current patch uses qualifiers to group data into
subsections. The qualifier usage allows for dynamic subsection
creation and affords the developer a large amount of flexibility
in the grouping, naming, and size of the resulting partitions and
domains that are built on these subsections. By additional macro
calls, functions are created that help calculate the size,
address, and permissions for the subsections and enable the
developer to control application data in specified partitions and
memory domains.
Background: Initial attempts focused on creating a single
section in the linker script that then contained internally
grouped variables/data to allow MPU/MMU alignment and protection.
This did not provide additional functionality beyond
CONFIG_APPLICATION_MEMORY as we were unable to reliably group
data or determine their grouping via exported linker symbols.
Thus, the resulting decision was made to dynamically create
subsections using the current qualifier method. An attempt to
group the data by object file was tested, but found that this
broke applications such as ztest where two object files are
created: ztest and main. This also creates an issue of grouping
the two object files together in the same memory domain while
also allowing for compartmenting other data among threads.
Because it is not possible to know a) the name of the partition
and thus the symbol in the linker, b) the size of all the data
in the subsection, nor c) the overall number of partitions
created by the developer, it was not feasible to align the
subsections at compile time without using dynamically generated
linker script for MPU architectures requiring power of 2
alignment.
In order to provide support for MPU architectures that require a
power of 2 alignment, a python script is run at build prior to
when linker_priv_stacks.cmd is generated. This script scans the
built object files for all possible partitions and the names given
to them. It then generates a linker file (app_smem.ld) that is
included in the main linker.ld file. This app_smem.ld allows the
compiler and linker to then create each subsection and align to
the next power of 2.
Usage:
- Requires: app_memory/app_memdomain.h .
- _app_dmem(id) marks a variable to be placed into a data
section for memory partition id.
- _app_bmem(id) marks a variable to be placed into a bss
section for memory partition id.
- These are seen in the linker.map as "data_smem_id" and
"data_smem_idb".
- To create a k_mem_partition, call the macro
app_mem_partition(part0) where "part0" is the name then used to
refer to that partition. This macro only creates a function and
necessary data structures for the later "initialization".
- To create a memory domain for the partition, the macro
app_mem_domain(dom0) is called where "dom0" is the name then
used for the memory domain.
- To initialize the partition (effectively adding the partition
to a linked list), init_part_part0() is called. This is followed
by init_app_memory(), which walks all partitions in the linked
list and calculates the sizes for each partition.
- Once the partition is initialized, the domain can be
initialized with init_domain_dom0(part0) which initializes the
domain with partition part0.
- After the domain has been initialized, the current thread
can be added using add_thread_dom0(k_current_get()).
- The code used in ztests ans kernel/init has been added under
a conditional #ifdef to isolate the code from other tests.
The userspace test CMakeLists.txt file has commands to insert
the CONFIG_APP_SHARED_MEM definition into the required build
targets.
Example:
/* create partition at top of file outside functions */
app_mem_partition(part0);
/* create domain */
app_mem_domain(dom0);
_app_dmem(dom0) int var1;
_app_bmem(dom0) static volatile int var2;
int main()
{
init_part_part0();
init_app_memory();
init_domain_dom0(part0);
add_thread_dom0(k_current_get());
...
}
- If multiple partitions are being created, a variadic
preprocessor macro can be used as provided in
app_macro_support.h:
FOR_EACH(app_mem_partition, part0, part1, part2);
or, for multiple domains, similarly:
FOR_EACH(app_mem_domain, dom0, dom1);
Similarly, the init_part_* can also be used in the macro:
FOR_EACH(init_part, part0, part1, part2);
Testing:
- This has been successfully tested on qemu_x86 and the
ARM frdm_k64f board. It compiles and builds power of 2
aligned subsections for the linker script on the 96b_carbon
boards. These power of 2 alignments have been checked by
hand and are viewable in the zephyr.map file that is
produced during build. However, due to a shortage of
available MPU regions on the 96b_carbon board, we are unable
to test this.
- When run on the 96b_carbon board, the test suite will
enter execution, but each individaul test will fail due to
an MPU FAULT. This is expected as the required number of
MPU regions exceeds the number allowed due to the static
allocation. As the MPU driver does not detect this issue,
the fault occurs because the data being accessed has been
placed outside the active MPU region.
- This now compiles successfully for the ARC boards
em_starterkit_em7d and em_starterkit_em7d_v22. However,
as we lack ARC hardware to run this build on, we are unable
to test this build.
Current known issues:
1) While the script and edited CMakeLists.txt creates the
ability to align to the next power of 2, this does not
address the shortage of available MPU regions on certain
devices (e.g. 96b_carbon). In testing the APB and PPB
regions were commented out.
2) checkpatch.pl lists several issues regarding the
following:
a) Complex macros. The FOR_EACH macros as defined in
app_macro_support.h are listed as complex macros needing
parentheses. Adding parentheses breaks their
functionality, and we have otherwise been unable to
resolve the reported error.
b) __aligned() preferred. The _app_dmem_pad() and
_app_bmem_pad() macros give warnings that __aligned()
is preferred. Prior iterations had this implementation,
which resulted in errors due to "complex macros".
c) Trailing semicolon. The macro init_part(name) has
a trailing semicolon as the semicolon is needed for the
inlined macro call that is generated when this macro
expands.
Update: updated to alternative CONFIG_APPLCATION_MEMORY.
Added config option CONFIG_APP_SHARED_MEM to enable a new section
app_smem to contain the shared memory component. This commit
seperates the Kconfig definition from the definition used for the
conditional code. The change is in response to changes in the
way the build system treats definitions. The python script used
to generate a linker script for app_smem was also midified to
simplify the alignment directives. A default linker script
app_smem.ld was added to remove the conditional includes dependency
on CONFIG_APP_SHARED_MEM. By addining the default linker script
the prebuild stages link properly prior to the python script running
Signed-off-by: Joshua Domagalski <jedomag@tycho.nsa.gov>
Signed-off-by: Shawn Mosley <smmosle@tycho.nsa.gov>
Add new kwyboard to board definition to allow blacklisting boards. This
is needed when a board is broken causing CI to fail without a fix in
sight.
Add:
sanitycheck: false
to the board yaml file to disable the board. By default, the value is
set to true.
Signed-off-by: Anas Nashif <anas.nashif@intel.com>
sanitycheck not printing QEMU console in some cases where a crash
happens and when state is not set correctly.
Fixes#9061
Signed-off-by: Anas Nashif <anas.nashif@intel.com>
The two handlers were doing pretty much the same with minor differences,
unify them into one single handler BinaryHandler that will be able to
handle additional targets.
Signed-off-by: Anas Nashif <anas.nashif@intel.com>
refactor all add_goal routines in to one single function that can handle
multiple platforms. Move everything to one single add_goal function.
Signed-off-by: Anas Nashif <anas.nashif@intel.com>
For native builds it does not make much sense to calculate
the size of the suposed RAM or ROM, or to check that new
unexpected sections did not appear.
So let's save the time and not do it.
Signed-off-by: Alberto Escolar Piedras <alpi@oticon.com>
Added a new command line options to sanitycheck:
--enable-coverage which will compile for native_posix
with CONFIG_COVERAGE set, and unit tests accordingly.
+
Now -C --coverage implies also --enable-coverage.
Background:
After 608778a4de
it is possible to add Kconfig options from command
line during the cmake invocation.
So we can use it to set CONFIG_COVERAGE for the native_posix
target when we need to instead of relaying on it always
being compiled with coverage enabled.
Signed-off-by: Alberto Escolar Piedras <alpi@oticon.com>
VERBOSE if set to any value enables verbose build output,
setting to 0 does not have the intended effect.
Signed-off-by: Andrew Boie <andrew.p.boie@intel.com>
In verbose mode (-v) progress of running test cases number out of total
number is not printed. Thus, if there are many test cases, it is
impossible to follow the progress. This commit adds printing of current
test case and total numbers to the verbose output.
Signed-off-by: Ruslan Mstoi <ruslan.mstoi@intel.com>
Export tests to a file with Section, subsection and identifier as
columns making it easy to import testcases into test management system.
Signed-off-by: Anas Nashif <anas.nashif@intel.com>
This section got added in commit 470349c25a ("Bluetooth: settings:
Add support for per-submodule handlers"), but sanitycheck didn't know
about it and was whining.
Signed-off-by: Andy Ross <andrew.j.ross@intel.com>
Ninja and GNU make don't play well with each other. Both try to start
enough processes to keep the system's CPUs busy, resulting in an
O(N^2) system load in the number of processors.
Long term, Ninja seems likely to support the GNU make jobserver
mechanism for sharing access to parallelism. But for now we can get
90% of the way there with a simple hack: just run ninja in serial mode
with -j1. Sanitycheck when run in non-trivial circumstances has
PLENTY of parallelism just from the number of test cases.
One interesting note is that even with -j1, system loads under ninja
are rather higher. That may be because of significant work done in
the (serial) makefiles that dilutes the parallelism of the eventual
build, or possibly because ninja itself is multithreaded in its setup
code. So I tweaked the number of jobs down to keep the load roughly
where it is with make.
With this change, I see no difference in behavior or system load, and a
~24% improvement in runtime.
Signed-off-by: Andy Ross <andrew.j.ross@intel.com>
- Some tests start with test_, some do not, so make sure we parse both.
- Parse skipped tests
- Improve handling of test case identifier
- Handle Exceptions in device handler
Signed-off-by: Anas Nashif <anas.nashif@intel.com>
We are passing global arguments from one level to the next when those
variables are available globally. Reduce the arguments and remove unused
arguments as well.
Signed-off-by: Anas Nashif <anas.nashif@intel.com>
init class in one place, no need to duplicate all class members in every
subclass.
run_log is not needed in the handler class.
Signed-off-by: Anas Nashif <anas.nashif@intel.com>
Parse the test results and create a test report with more granular
results that can be imported to into test management/reporting system.
Signed-off-by: Anas Nashif <anas.nashif@intel.com>
This will allow us to run sanitycheck on real devices and get reporting
out of it the same way we do that with Qemu.
To use this, run sanitycheck with the following new options:
scripts/sanitycheck --device-testing --device-serial /dev/ttyACM0 -p
frdm_k64f -T tests/crypto/
--device-serial denotes the serial device the board is connected to.
This needs to be accessible by the user running sanitycheck. You can
run this on one board only at a time, the board is specified using the
--platform option.
This was tested with only a few boards, some board will not work
because how they reset the serial device during flashing.
Signed-off-by: Anas Nashif <anas.nashif@intel.com>
After removing the -T linker warning for the POSIX arch
we can, and should, also treat its linker warnings as errors
Signed-off-by: Alberto Escolar Piedras <alpi@oticon.com>
This packs a few improvements:
- Add a smarter regex that will catch multiple combinations of
ztest_[user_]unit_test[_setup_teardown](NAME[, setup, teardown])
as well as single liners like:
ztest_test_suite(mutex_complex, ztest_user_unit_test(TESTNAME));
- Limit how much we look forward in suite_regex -- we don't have to
look past the first argument, otherwise we consume too much and the
loopup at suite_regex_match.start() will start too late.
- Remove include_regex, unused
- Fix the path where we warn about matches in achtung_regexes--it
needed a few decodes and to use error() vs the unexistant warning()
- Cleanup the path to produce the subcase names, doing the decode and
the purging of any test_ prefix in scan_path().
Signed-off-by: Inaky Perez-Gonzalez <inaky.perez-gonzalez@intel.com>
This addresses issue #7146.
The current helptext does not state that the directory
will be deleted.
Signed-off-by: Håkon Øye Amundsen <haakon.amundsen@nordicsemi.no>
Parse all yaml file and create a list of declared testcases. This does
list the individual tests inside test projects, not only the projects
containing the tests, for example:
$ sanitycheck --list-tests -T tests/net/socket/
- net.socket.udp.send_recv_2_sock
- net.socket.udp.v4_sendto_recvfrom
- net.socket.udp.v6_sendto_recvfrom
- net.socket.udp.v4_bind_sendto
- net.socket.udp.v6_bind_sendto
- net.socket.getaddrinfo_ok
- net.socket.getaddrinfo_no_host
- net.socket.tcp.v4_send_recv
- net.socket.tcp.v6_send_recv
- net.socket.tcp.v4_sendto_recvfrom
- net.socket.tcp.v6_sendto_recvfrom
- net.socket.tcp.v4_sendto_recvfrom_null_dest
- net.socket.tcp.v6_sendto_recvfrom_null_dest
13 total.
Signed-off-by: Anas Nashif <anas.nashif@intel.com>
This parses the tests that run within a test project/application from
the source code and gives us a view of what was run, skipped and what
was blocked due to early termination of the test.
Signed-off-by: Anas Nashif <anas.nashif@intel.com>
If we have some error parsing a testcase or other files we treat these
as errors and will exit before continuing on building other tests.
Signed-off-by: Kumar Gala <kumar.gala@linaro.org>
In many cases we are calling cmake twice with the same options, first to
build, and then we do the same thing when we want to call 'make run'.
Just call cmake once.
Signed-off-by: Anas Nashif <anas.nashif@intel.com>
When running large set of tests we always get a huge list of footprint
changes that mask the test results making them impossible to see on the
screen. Show the footprint results only on-demand and not on every
build.
Signed-off-by: Anas Nashif <anas.nashif@intel.com>
With this commit it is possible to add priority to sent or received
network packets. So user is able to send or receive higher priority
packets faster than lower level packets.
The traffic class support is activated by CONFIG_NET_TC_COUNT option.
The TC support uses work queues to separate the traffic. The
priority of the work queue thread specifies the ordering of the
network traffic. Each work queue thread handles traffic to one specific
work queue. Note that you should not enable traffic classes unless
you really need them by your application. Each TC thread needs
stack so this feature requires more memory.
It is possible to disable transmit traffic class support and keep the
receive traffic class support, or vice versa. If both RX and TX traffic
classes are enabled, then both will use the same number of queues
defined by CONFIG_NET_TC_COUNT option.
Fixes#6588
Signed-off-by: Jukka Rissanen <jukka.rissanen@linux.intel.com>
Move IP address settings from net_if to separate structs.
This is needed for VLAN support.
Signed-off-by: Jukka Rissanen <jukka.rissanen@linux.intel.com>
For those still using old variable ZEPHYR_GCC_VARIANT.
raise an error if the variable is not defined.
Signed-off-by: Anas Nashif <anas.nashif@intel.com>
Add the STM32 ccm_bss, ccm_noinit, abd ccm_data sections
to the list of allowed sections so the sanity check passes.
Signed-off-by: Erwin Rol <erwin@erwinrol.com>
We want to support other toolchain not based on GCC, so the variable is
confusing, use ZEPHYR_TOOLCHAIN_VARIANT instead.
Signed-off-by: Anas Nashif <anas.nashif@intel.com>
It is important that it is easy to reproduce CI issues locally. Using
the same sanitycheck options locally and in CI helps in this
regard. Specifically, Ninja and Make can produce different results and
therefore the default generator should be the same for sanitycheck and
shippable.
This patch makes four changes:
The sanitycheck option '--make' is introduced to allow specifying Make
as a generator.
CI no longer passes the option '--ninja' to sanitycheck.
Sanitycheck defaults to using Ninja.
Sanitycheck documents the --ninja option as deprecated.
Signed-off-by: Sebastian Bøe <sebastian.boe@nordicsemi.no>
When generating coverage report:
* Also generate branch coverage
* If the report is properly generated tell user where to find it
* Handle case in which no ztests are run (it would crash before)
Signed-off-by: Alberto Escolar Piedras <alpi@oticon.com>
Aligned sanitycheck command line options in doc
with the actual ones.
+
Minor fix in --coverage description.
Signed-off-by: Alberto Escolar Piedras <alpi@oticon.com>
sanitycheck: Compile unit tests with coverage enabled always
+ run also first unit tests together with native_posix
shippable: also include unit_testing coverage into report to
codecov
Signed-off-by: Alberto Escolar Piedras <alpi@oticon.com>
This patch adds app_pad and priv_stacks to the rw section list so that
tests which validate the sections found in the binary pass when run on
systems which contain MPUs.
Signed-off-by: Andy Gross <andy.gross@linaro.org>
Rather than having handler generate differently named log files,
generate the same for all handlers, so qemu, native and unit will now
generate handler.log.
This will simplify things and reporting will be consistent. Also, this
fixes a bug where we only included qemu.log in generated reports.
Signed-off-by: Anas Nashif <anas.nashif@intel.com>
When building the testcase path, replace symlinks in path
with the realpath both in the testcase root and in ZEPHYR_BASE
For more info see issue #5772Fixes: #5772
Signed-off-by: Alberto Escolar Piedras <alpi@oticon.com>
Changed the default behavior for when we run sanitycheck to match the -R
flag that turns on assertions. Introduced a --disable-asserts option if
we desire to explicitly turn of asserts.
This matches behavior that our CI builds have been doing and addresses
part of #5726.
Signed-off-by: Kumar Gala <kumar.gala@linaro.org>
Add 2 classes, one to handle the current TestCase scenario, and one more
for handling generic Console with regex matching.
Signed-off-by: Anas Nashif <anas.nashif@intel.com>
Some boards are supported natively by qemu. This option will allow us to
run tests using those platforms directly without having to go via a
dedicated qemu board definition.
Signed-off-by: Anas Nashif <anas.nashif@intel.com>
if we are using command line platform filter, no need to list every
other platform as excluded, we know that already. Show only the
discards that apply to the selected platforms on the command line
Signed-off-by: Anas Nashif <anas.nashif@intel.com>
When we load tests from a file, we do not have the discarded list, this
would have been done already when saving the test file.
Signed-off-by: Anas Nashif <anas.nashif@intel.com>
When checking if a testcase passed or failed, allow there
to be prefixes or postfixes in the line, around
PROJECT EXECUTION SUCCESSFUL/FAILED
Signed-off-by: Alberto Escolar Piedras <alpi@oticon.com>
CONFIG_ASSERT is being set by cmake, so it is not possible to filter
using the generated config, add this as a standalone filter.
Signed-off-by: Anas Nashif <anas.nashif@intel.com>
Generating coverage data is split over two CI jobs which means the
service will need to merge results and reports wrong coverage data when
only 1 job is finished. This puts the native_posix board first making
sure we run on the first job and generate data in one place.
Signed-off-by: Anas Nashif <anas.nashif@intel.com>
Extend exception handling to cover not just YAML loading, but any
error while accessing parsed data too. That may catch e.g. schema
mismatch errors (for folks who don't have pykwalify installed, which
is optional). So, now error will be logged, but processing of other
tests will continue.
For example, I had a local, uncommitted test which wasn't converted
per 23f81eeb42 and caused:
Traceback (most recent call last):
File "./scripts/sanitycheck", line 2456, in <module>
main()
File "./scripts/sanitycheck", line 2324, in main
options.outdir, options.coverage)
File "./scripts/sanitycheck", line 1445, in __init__
for name in parsed_data.tests.keys():
AttributeError: 'list' object has no attribute 'keys'
Signed-off-by: Paul Sokolovsky <paul.sokolovsky@linaro.org>
We have been passing around options from one function to the next making
it very difficult to add a new option easily and requiring changes to
man function prototypes.
This declated the parsed command line options global and renames args to
options. args is being used elsewhere and this was confusing.
Signed-off-by: Anas Nashif <anas.nashif@intel.com>
This keyword would mean that a special harness is needed to run the
tests sucessfully. This can be as simple as a loopback wiring or a
complete hardware test setup for sensor and IO testing. It is free form
initially and would be changed to be an enum once we have more values in
place.
Signed-off-by: Anas Nashif <anas.nashif@intel.com>
We have now different runners/handlers, so avoid using qemu terminology
for the generic classes and for generic usage.
Signed-off-by: Anas Nashif <anas.nashif@intel.com>
Passing ARCH during the build process is something from the past and
samples/tests should not do that, remove it here to catch any
violations.
Signed-off-by: Anas Nashif <anas.nashif@intel.com>
Since the move to YAML format and the change in how we define default
platforms this is no longer needed as we are able to set multiple
default platforms per architecture and not using a list based on
priority anymore.
Fixes#4445
Signed-off-by: Anas Nashif <anas.nashif@intel.com>
The section terminology was relevant with the ini syntax, with yaml we
can call this a test and avoid confusion and make the code more
readable.
Signed-off-by: Anas Nashif <anas.nashif@intel.com>
Simplify parsing of yaml structures and remove usage of cp which was for
the ConfigParser used for ini files.
Signed-off-by: Anas Nashif <anas.nashif@intel.com>
It is supported to add give extra flags to the linker from the
commandline like this:
cmake -DEXTRA_LDFLAGS=-Lmy_dir path
But unfortunately this was broken during the CMake
migration. Interestingly, the reason that it was broken is that KBuild
was also partially broken. KBuild would pass on EXTRA_LDFLAGS when
object files were linked together into built-in.o files, but it would
not use EXTRA_LDFLAGS for the final link into an elf file.
This patch fixes EXTRA_LDFLAGS.
Signed-off-by: Sebastian Bøe <sebastian.boe@nordicsemi.no>
This can be used by other handlers and is defined in the main Handler
class. Qemu is just an implementer.
Signed-off-by: Anas Nashif <anas.nashif@intel.com>
This makes piped output work as the user expects. And looking at the
piped output is the only way to use sanitycheck normally because
of #4603.
Signed-off-by: Paul Sokolovsky <paul.sokolovsky@linaro.org>
sanitycheck was incorrectly documenting that --extra-args would pass
on it's input unchanged to Make.
In reality --extra-args acts as a way to define extra CMake cache
entries. The key-value entries will be prefixed with -D before being
passed to CMake.
E.g
"sanitycheck -x=USE_CCACHE=0"
will translate to
"cmake -DUSE_CCACHE=0"
Signed-off-by: Sebastian Boe <sebastian.boe@nordicsemi.no>
We only had a few hundred tests run when sanitycheck was first written,
and printing out the reasoning why tests were skipped seemed reasonable
at the time. Now that we are running tens of thousands of tests, this
is too much information.
The dump of what tests were skipped and why now requires two instances
of --verbose on the command line.
Signed-off-by: Andrew Boie <andrew.p.boie@intel.com>
It's possible to declare static threads that start up as K_USER,
but these threads can't do much since they start with permissions on
no kernel objects other than their own thread object.
Rather than do some run-time synchronization to have some other thread
grant the necessary permissions, we introduce macros
to conveniently assign object permissions to these threads when they
are brought up at boot by the kernel. The tables generated here
are constant and live in ROM when possible.
Example usage:
K_THREAD_DEFINE(my_thread, STACK_SIZE, my_thread_entry,
NULL, NULL, NULL, 0, K_USER, K_NO_WAIT);
K_THREAD_ACCESS_GRANT(my_thread, &my_sem, &my_mutex, &my_pipe);
Signed-off-by: Andrew Boie <andrew.p.boie@intel.com>
Support new keywords in testcase.yaml that would allow us to inject
configuration options to be merged with default configuration instead of
having to provide a prj.conf for each variant of the test which is very
difficult to keep in sync. Sanitycheck script will create an overlay
file that is merged during the build process.
This is now done using the extra_configs option which is a yaml list of
option with the values, for example:
extra_configs:
- CONFIG_XXXX=y
- CONFIG_YYYY=y
With this option we can have multiple tests that for example run on
hardware with different values. This type of testing is good on HW but
it does not make sense to be built in normal sanitycheck operation
because it will be just rebuilding the same code with different values.
Signed-off-by: Anas Nashif <anas.nashif@intel.com>
because we do not use ini files anymore, to avoid confusion, rename this
to be yamlfile, which is the format we use for testcases now.
Signed-off-by: Anas Nashif <anas.nashif@intel.com>
All system calls made from userspace which involve pointers to kernel
objects (including device drivers) will need to have those pointers
validated; userspace should never be able to crash the kernel by passing
it garbage.
The actual validation with _k_object_validate() will be in the system
call receiver code, which doesn't exist yet.
- CONFIG_USERSPACE introduced. We are somewhat far away from having an
end-to-end implementation, but at least need a Kconfig symbol to
guard the incoming code with. Formal documentation doesn't exist yet
either, but will appear later down the road once the implementation is
mostly finalized.
- In the memory region for RAM, the data section has been moved last,
past bss and noinit. This ensures that inserting generated tables
with addresses of kernel objects does not change the addresses of
those objects (which would make the table invalid)
- The DWARF debug information in the generated ELF binary is parsed to
fetch the locations of all kernel objects and pass this to gperf to
create a perfect hash table of their memory addresses.
- The generated gperf code doesn't know that we are exclusively working
with memory addresses and uses memory inefficently. A post-processing
script process_gperf.py adjusts the generated code before it is
compiled to work with pointer values directly and not strings
containing them.
- _k_object_init() calls inserted into the init functions for the set of
kernel object types we are going to support so far
Issue: ZEP-2187
Signed-off-by: Andrew Boie <andrew.p.boie@intel.com>
make this consistent with flash size check. This issue caused platforms
with 8k to be completelty ignored.
Signed-off-by: Anas Nashif <anas.nashif@intel.com>
This introduces an schema-based YAML validation process when loading
any YAML file, before doing any operations on them. An exception will
be raised at SanityConfigParser() if the file fails to verify with the
given schema.
Schemas are defined for the platform files in board///*.yaml and for
the (sample|testcase).yaml files. The verification is done using the
pykwalify python library. If not installed, a warning is printed and
the verification schema is skipped. At some point, we might want to
force it being installed.
The verification library is made a separate module (scl.py) so it can
be easily imported by others.
Signed-off-by: Inaky Perez-Gonzalez <inaky.perez-gonzalez@intel.com>
AFAIK an ini file system was ported to a yaml file system. But some
ini file references still remain.
This patch changes all ini file mentions into yaml.
Signed-off-by: Sebastian Bøe <sebastian.boe@nordicsemi.no>
Upcoming memory protection features will be placing some additional
constraints on kernel objects:
- They need to reside in memory owned by the kernel and not the
application
- Certain kernel object validation schemes will require some run-time
initialization of all kernel objects before they can be used.
Per Ben these initializer macros were never intended to be public. It is
not forbidden to use them, but doing so requires care: the memory being
initialized must reside in kernel space, and extra runtime
initialization steps may need to be peformed before they are fully
usable as kernel objects. In particular, kernel subsystems or drivers
whose objects are already in kernel memory may still need to use these
macros if they define kernel objects as members of a larger data
structure.
It is intended that application developers instead use the
K_<object>_DEFINE macros, which will automatically put the object in the
right memory and add them to a section which can be iterated over at
boot to complete initiailization.
There was no K_WORK_DEFINE() macro for creating struct k_work objects,
this is now added.
k_poll_event and k_poll_signal are intended to be instatiated from
application memory and have not been changed.
Signed-off-by: Andrew Boie <andrew.p.boie@intel.com>
If the depends_on has more than one item we need to match all of those
dependencies in the supported list.
Signed-off-by: Kumar Gala <kumar.gala@linaro.org>
Some testcases can only be built with certain toolchains. Instead of
using filters, add support for toolchain keyword which enables
whitelisting and exclusion.
Signed-off-by: Anas Nashif <anas.nashif@intel.com>
Put the results of the config-sanitycheck into their own log so we can
see warnings from that stage of the build.
Signed-off-by: Kumar Gala <kumar.gala@linaro.org>
The build_on_all tag in synchronisation sample was resetting the
supplied arguemnt for filtering platforms.
Signed-off-by: Anas Nashif <anas.nashif@intel.com>
This commit changes the syntax of the testcase files and changes the
behaviour and configuration of the sanitycheck script.
To avoid having multiple files with different syntax for boards,
samples, tests; this change unifies the syntax and uses YAML instead of
INI.
We maintain the current keywords used in the old syntax and maintain the
flexibility of adding tests with different configuration by using YAML
list configuration. On top of that, the following features are added:
- We now scan for board configurations in the boards directory and look
for a YAML file describing a board and how it should be tested. This
eliminates the need for listing boards per architecture in a special ini
file under scripts/.
- We define hardware information charachterstics in the board YAML file
that helps identifying if a certain test should run on that board or
not. For example, we can specify the available RAM in the board and
filter tests that would require more RAM than the board can handle.
- Boards can be set as default for testing meaning that we always run a
test case (build and run of possible) when sanitycheck is called without
any arguments. Previously this was done only by selecting the first
board defined for a specific architecture.
- Tests can be configured to run on all possible boards, this is to make
sure we always build some basic tests for all boards to catch issues
with the core kernel features.
Signed-off-by: Anas Nashif <anas.nashif@intel.com>
Moving the net_buf_pool objects to a dedicated area lets us access
them by array offset into this area instead of directly by pointer.
This helps reduce the size of net_buf objects by 4 bytes.
Signed-off-by: Johan Hedberg <johan.hedberg@intel.com>
This patch changes the config target to use the config-sanitycheck
target. The config-sanitycheck target gets not only the Kconfig
options, but also the DTS generated options.
Signed-off-by: Andy Gross <andy.gross@linaro.org>
classname seems to be expected by some parsers, so use it in the report,
the name now has the same value, we can add some more information once
we have meaningful metadata for the testcases with more details.
Signed-off-by: Anas Nashif <anas.nashif@intel.com>
This new options makes it possible to only run a subset of the tests
which will allow us to run sanitycheck on multiple hosts and merge the
results into one report. This way we do not need to worry about
selecting specific architectures to be run on a certain host.
The option accepts a string value: x/y where x is the subset and y is
the total, so if we specify --subset 1/5, we will only run the first
fifth of the total tests, --subset 5/5 would only select the last fifth.
To get consistent results, the testcase instance list is now ordered,
to avoid duplications and have full coverage.
Signed-off-by: Anas Nashif <anas.nashif@intel.com>
Generate a test report using junit/xunit format with all details and
logs that can be published on the web to show results from a sanitycheck
run.
Output is stored in scripts/sanity_chk/ alongside the CSV file.
Change-Id: I5ea6f409c1f86f408eeae870b90a953e71046da9
Signed-off-by: Anas Nashif <anas.nashif@intel.com>
Now that k_poll landed in the kernel, it's worth using it to save
memory and reduce the number of threads at runtime.
Such switch has been first done in bluetooth (see hci_core.c and conn.c
in subsys/bluetooth/host). Since network interfaces kind of follows the
same design for sending data, it was then easy to copy the same change as
in bluetooth.
Change-Id: I7f9734b88ac818284bbabaedc946b4765b905ebb
Signed-off-by: Tomasz Bursztyka <tomasz.bursztyka@linux.intel.com>
By convention Kconfigs are in all uppercase, but not strictly
required to be so. Currently the Xtensa build defines
CONFIG_${XTENSA_CORE} which can have lowercase letters. This
is marked for cleanup in ZEP-1711.
Issue: ZEP-1711
Change-Id: Ia901ea1688202129f1629001a77f498522859f14
Signed-off-by: Andrew Boie <andrew.p.boie@intel.com>
When current directory is a symbolic link or in a symbolic link then os.getcwd
return the real path leading to os.path.relpath returning not so nice relative
path.
Change-Id: I3e24d3e6f1fe7185992f64803d3728e9588d67e1
Signed-off-by: Mazen NEIFER <mazen@nestwave.com>
This completely breaks all the cc-option checks. It's not clear
why this was done for everyone running sanitycheck.
Change-Id: I3f944423869ccb15fab29c28130820a93cd94917
Signed-off-by: Andrew Boie <andrew.p.boie@intel.com>
In scripts/Makefile.qemu, the target qemu is already an alias for taget run.
In scripts/Makefile.xt{,sc}-run there is no target qemu but a target run.
Change-Id: Ib9dce3561ef9367b17d30070354e709df4d7e528
Signed-off-by: Mazen NEIFER <mazen@nestwave.com>
Many users set USE_CCACHE=1 in their envionment. Sanitycheck was
disregarding that and placing USE_CCACHE=0 in the generated
Makefile unless --ccache was added to the command line every time
it was invoked.
Issue: ZEP-1663
Change-Id: Id3c1379f5039d4d2f4a3ae01d2ec8d0dd216f523
Signed-off-by: Andrew Boie <andrew.p.boie@intel.com>
Values now spaced 4 apart since we commonly build thousands of
tests.
Percentage complete indicator added.
Change-Id: Ia7454dc1e8cc0701dc74dc8a4f6e7a37706524e8
Signed-off-by: Andrew Boie <andrew.p.boie@intel.com>
These were causing some issues in the old kernel, but the unified
kernel no longer uses these.
Issue: ZEP-513
Change-Id: I87216565231cd244886fbffe4b4d420d1687b245
Signed-off-by: Andrew Boie <andrew.p.boie@intel.com>
This had been erroneously reported as a build error.
Issue: ZEP-1460
Change-Id: Idddaf833011beb1165bd766dd115c021c5b90cdf
Signed-off-by: Andrew Boie <andrew.p.boie@intel.com>
When running a reduced set of tests with --platform-limit or using the
default of 1 we build the configuration (make initconfig) for all
platforms, although we are only interested in a limited set. On my
machine this is 100s of build time for configurations we are never going
to use.
This reduces the number of builds to what we really need and speeds up
sanitycheck runtime overall.
Might need a few more optimisations.
Change-Id: I813fabf1453f19ce7a83b28e4792b5c80616d936
Signed-off-by: Anas Nashif <anas.nashif@intel.com>
Currently CI plays tricks with `tee` to capture the output and post
process it. This makes it more complex to handle and easier for it to
fail in ways it should not.
So add a simple log capture option sanity check that mirrors the
stdout output, removing color encoding for errors.
Change-Id: I6de0b6cfe4da9c289f537979545dddbcd49cf834
Signed-off-by: Inaky Perez-Gonzalez <inaky.perez-gonzalez@intel.com>
This reverts commit 4e2cf33d9d.
The change does not work correct if we have a filter and specify a
specific platform, such as:
./scripts/sanitycheck -p 96b_nitrogen -i -s samples/hello_world/test_singlethread
Change-Id: I43a4f25ed25c8fe5379cda23b6dbfa8f9ae5cb0d
Signed-off-by: Kumar Gala <kumar.gala@linaro.org>
When running a reduced set of tests with --platform-limit or using the
default of 1 we build the configuration (make initconfig) for all
platforms, although we are only interested in a limited set. On my
machine this is 100s of build time for configurations we are never going
to use.
This reduces the number of builds to what we really need and speeds up
sanitycheck runtime overall.
Might need a few more optimisations.
Change-Id: I53d6bacf66045dde7baabbaa40b4fbfc4a01782a
Signed-off-by: Anas Nashif <anas.nashif@intel.com>
The macro defines the stack as usual, but if user has
enabled net shell (CONFIG_NET_SHELL), then additional
information about the stack is stored in net_shell
linker section. The information in the net_shell linker
section is then used to print information about the
stacks in the networking sub-system.
Change-Id: Ic6e9f16a73a192b9a59d32a6d0070322382f98bd
Signed-off-by: Jukka Rissanen <jukka.rissanen@linux.intel.com>
No need to store the nbr pool data into a dedicated section
as we have only one nbr pool defined.
Change-Id: I6f2afcce57b5f588878496bf085567b938e32c80
Signed-off-by: Jukka Rissanen <jukka.rissanen@linux.intel.com>
We missed that section, so sanitycheck was failing.
Also putting the right section in the relevant array.
Change-Id: I361ffa1707c01f0d3c870a7bc36a86ada66af1d2
Signed-off-by: Tomasz Bursztyka <tomasz.bursztyka@linux.intel.com>
Without these, test case will fail miserably.
Change-Id: Ib46d167b8782c33362ac6edb9d8c50abc25908c0
Signed-off-by: Tomasz Bursztyka <tomasz.bursztyka@linux.intel.com>
Added new option to make sanitycheck script error on deprecation
warnings, to use:
sanitycheck --error-on-deprecations
For example, to find usage of legacy APIs in master, run the following:
sanitycheck -e legacy --error-on-deprecations
Change-Id: Ib83c266c8357475840dea03d62ceefed72b73f27
Signed-off-by: Anas Nashif <anas.nashif@intel.com>
When running --only-failed make sure last_sanity.csv exists.
Change-Id: I862529a58f337987c37da83fd5541dd4f42b3391
Signed-off-by: Anas Nashif <anas.nashif@intel.com>
Enhance the linker script so that all statically-defined kernel
objects of a given type are co-located.
Note 1: This capability is needed so that static kernel objects
can be added to the associated object tracing list. (Some kernel
object types are already co-located to permit the kernel to do
other initialzation on statically-defined objects.)
Note 2: A follow-up commit is needed to ensure statically-
defined kernel objects are placed in the sections referenced
in the linker script.
Note 3: The legacy script info remains for now to allow
applications using the microkernel or nanokernel to continue
to build.
Change-Id: I40d831f9e183fb121c950e30fa8298b6d529375b
Signed-off-by: Allan Stephens <allan.stephens@windriver.com>
Ability to use Zephyr shell by multiple modules simultaneously, each
module for its own usage.
Old shell implementation enabled the user to call only one module
commands, not all of the modules simultaneously.
Change-Id: I0ef8fa2fd190b7490c44fe91d1016363258302c9
Signed-off-by: Yael Avramovich <yael.avramovich@intel.com>
Signed-off-by: Anas Nashif <anas.nashif@intel.com>
Add a ccache option to allows enabling the use of ccache for builds
generated by sanitycheck.
Change-Id: Ia997ed29f38a3d2d71ba143456e6fcbc5946d56e
Signed-off-by: Kumar Gala <kumar.gala@linaro.org>
commit 018b831a36 keeps another elf file in the
outdir which confused the sanitycheck script.
Change-Id: I87a9bcee6afe6526ecc24f1f5d9cf38101247ce2
Signed-off-by: Anas Nashif <anas.nashif@intel.com>
This fix sanitycheck for apps that specify unified kernel.
Change-Id: I3f1c4d5a566f6833e56bbda097cb497ccdbfbec9
Signed-off-by: Szymon Janc <szymon.janc@codecoup.pl>
Python module argparse has a feature to load arguments from a file.
This commit enables such feature on sanitycheck script so files
prefixed by + (plus sign) are loaded during argument parsing stage
and included as arguments to such script.
Jira: ZEP-122
Change-Id: Ibb0a6e8464218a4f58ce4044a5a0860dc5b32480
Signed-off-by: Genaro Saucedo Tejada <genaro.saucedo.tejada@intel.com>
This will make it possible for us to optimize the list of tests we run, for
example, we could exclude footprint tests from the main run because those are
run as part of the footprint checks later in the CI job.
Use like this: sanitycheck -e footprint
Change-Id: I4e7a3aa6fac2ba1c9c99b356f08459da97fda777
Signed-off-by: Anas Nashif <anas.nashif@intel.com>
This misfeature can crop up at development time as the linker scripts
are modified, not just as a regression in known tests (c.f. ZEP-955).
So it makes more sense to put it in a check_link_map script inovked
for every build. Remove it from sanitycheck.
Change-Id: I9c2f47a26a3bc03170b895545476350341e91170
Signed-off-by: Andy Ross <andrew.j.ross@intel.com>
If the -C flag is given to sanitycheck, generate gcov files for unit
tests and render them with lcov.
Signed-off-by: Jaakko Hannikainen <jaakko.hannikainen@intel.com>
Change-Id: Ic25eae6a3cfc2c45595bd6aa235df2c483aaf6ec
This commit allows building tests using the ztest framework without
including Zephyr. This can be used to enable unit testing single
functions, even static ones.
Origin: Original
Change-Id: Ib7e84f4bd9bbbf158b9a19edaf6540f28e47259f
Signed-off-by: Jaakko Hannikainen <jaakko.hannikainen@intel.com>
Signed-off-by: Anas Nashif <anas.nashif@intel.com>
Due to the memory pool structure only static declaration of
memory pool is possible.
Change-Id: I4797ed88fd2ac3b7812ff26e552e1745611c4575
Signed-off-by: Dmitriy Korovkin <dmitriy.korovkin@windriver.com>
In 1.0 you could set only one callback on the whole gpio controller. It
was impossible for another sub-system to add another callback, without
overwritting an existing one.
Such API has been obsolete for a long time and no one is using it
anymore. Thus removing it entirely.
Change-Id: I6a17fd99373dc6cef1fa2ebb421e992412d5015e
Signed-off-by: Tomasz Bursztyka <tomasz.bursztyka@linux.intel.com>
In some cases, particularly when developing experimental features, it is
useful to be able to pass parameters to 'make' globally for all
sanity checks.
Change-Id: I19eddf222a74064cf6f22ec641fa6b4ff72992e7
Signed-off-by: Andrew Boie <andrew.p.boie@intel.com>
If defconfig files exist under boards/ that don't correspond to
the set of platforms defined in the .ini files under
scripts/sanity_chk/arches, complain about this. We should be at
least compiling all our supported platforms in CI runs.
At the moment it flags 5 boards. Future patches should either add
these boards to the appropriate arch .ini files or remove them
from the kernel.
Change-Id: Id825d6c6bd32089d9f1bead11bc0765685e3771a
Signed-off-by: Andrew Boie <andrew.p.boie@intel.com>
Added support for multiple toolchain usage.
Every arch contains a list of supported toolchain.
Each board can override the supported toolchain list.
If the board is not supported in the current toolchain,
will be discarded.
The current toolchain is defined by ZEPHYR_GCC_VARIANT.
Added support for toolchain ISSM and ZEPHYR.
Change-Id: I31e9b39ba01f6e9bdc4557702428cd09e05f492a
Jira: ZEP-592
Signed-off-by: Javier B Perez <javier.b.perez.hernandez@intel.com>
Added support in the sanitycheck to use env variable
value in the filter.
Change-Id: I76388dc04557dcd31e651d23ae8edf96a8fe2474
Jira: ZEP-592
Signed-off-by: Javier B Perez <javier.b.perez.hernandez@intel.com>
Sanity check script was not handling non utf-8 characters as a result
it crashed and didn't kill qemu and test used to hang.
Now tests fail upon untranslatable bytes.
JIRA: ZEP-673
Change-Id: I885a185e0481083501fce15cd51412bfdd26a7d7
Signed-off-by: Genaro Saucedo Tejada <genaro.saucedo.tejada@intel.com>
sanitycheck is set up to error out on any compiler warnings.
However in the case of deprecated APIs, we may feel compelled
to still test them while they remain in-tree. Don't fail the
build if deprecated APIs are used during sanity runs.
Change-Id: Ic3196896689706d308117e693ab720713d67d698
Signed-off-by: Andrew Boie <andrew.p.boie@intel.com>
Some tests individually enable assertions; this forces them on
for all tests. Intended for use with daily builds. Disregard
footprint or benchmark data with this enabled.
Change-Id: I7a0822432d5cab7f3ab0767faf214fde536cd68e
Signed-off-by: Andrew Boie <andrew.p.boie@intel.com>
We will require 6 variables to be defined by SOC-specific
linker script; these values in turn can be pulled from
defines in layout.h.
To help position code correctly we define two new ELF sections
for this arch, 'reset' and 'exceptions'.
Change-Id: Idffbd53895945b7d0ec0aac281e5bf7c85b4b2c2
Signed-off-by: Andrew Boie <andrew.p.boie@intel.com>
Similar to '==' but compiles the RHS into a regular expression
and attempts to match whatever is in the environment against it.
Change-Id: I7a03452ef88d067b62661d14dc6f42273de436fa
Signed-off-by: Andrew Boie <andrew.p.boie@intel.com>
The old 'config_whitelist' directive in testcase.ini has been removed.
We use the new expr_parser module to parse a 'filter' directive which
is a boolean expression. This gives a great deal more flexibility
in how tests can be filtered.
To keep the tree bisectable, use of config_whitelist in testcase.ini
converted to the new expression language.
Change-Id: I0617319818c5559c0f0569d2fa73d09b681cac51
Signed-off-by: Andrew Boie <andrew.p.boie@intel.com>
Many sub-systems might require to set a callback on different pins.
Thus enabling it via changing the API.
It is also possible to retrieve private-data in the callback handler
using CONTAINER_OF() macro (include/misc/util.h).
Former API is still available, and is emulated through the new one.
Using both should not be a problem as it's using new API calls.
However, it's now better to start using the new API.
Change-Id: Id16594202905976cc524775d1cd3592b54a84514
Signed-off-by: Tomasz Bursztyka <tomasz.bursztyka@linux.intel.com>
Signed-off-by: Johan Hedberg <johan.hedberg@intel.com>
() The test names are getting longer so adds more padding.
() Aligns the numerical size numbers for easier visual
comparison when scrolling in shell.
Change-Id: Id218a641e9d5dffb77bf813ffbd828c48d14020f
Signed-off-by: Daniel Leung <daniel.leung@intel.com>
Add the last path information in the testcase root to the
test name, so that there is much less of chance of name
collisions and the reports come out with the same names
as before.
Change-Id: I75ed6ee96251058dd2547e57be31db3c2517a497
Signed-off-by: Andrew Boie <andrew.p.boie@intel.com>
A recent tree change moved testcases to both the tests/ and
samples/ directory. The default testcase root was just made
the root of the Zephyr tree, which could lead to potentially
long scan times as every single directory in-tree was checked.
For example, it could be very time-consuming to scan everything
under the sanity-out/ dir if -n is used and --all was used in
a previous run.
Now we allow multiple testcase roots to be supplied on the
command line, and by default only scan under tests/ and
samples/.
Change-Id: I8dac747835b87801474e08c236913e04db64f87b
Signed-off-by: Andrew Boie <andrew.p.boie@intel.com>
This adds the ability for sanitycheck to use arbitrary report file
for size comparison. This is useful when optimizing binary size.
For example, a report can be generated with master head, and used
as base for comparison.
Change-Id: Id8c61146c498ba62c4cc77a9ef7d390c0fb11487
Signed-off-by: Daniel Leung <daniel.leung@intel.com>
Instead of running nm/objdump and parsing the results one testcase
at a time, do these in parallel to speed up sanity check a bit.
The sanitycheck now requires at least Python 3.2 to run because of
the usage of concurrent.futures.
This reduces build time (--all -b) from 4:30 to 3:08 on 24C/48T
Xeon-E5.
Change-Id: I8e7c1efb2f473c7f2b65658f8ed9a101ed091eea
Signed-off-by: Daniel Leung <daniel.leung@intel.com>
The "initlevel" section is being metioned twice. It does not affect
the final size calculation, but just remove it just for reader's
sanity.
Change-Id: I0e7d6e82c730631d2c83ca09dde0f675805840f3
Signed-off-by: Daniel Leung <daniel.leung@intel.com>
If the arch ini file doesn't have a specific block for the platform,
just assume all defaults.
Change-Id: Id6ab1ede8936bdd6006a5c4fd2f9a4f6aaf42478
Signed-off-by: Andrew Boie <andrew.p.boie@intel.com>
It is an unfortunate fact of life that our CI machines are often
severly underpowered virtual machines that take quite some time
to run sanitycheck --all. The only other option was to just use
the default test case filtering semantics which chose one platform
per arch to test.
Now it is possible to specify N platforms per arch with the new
--platform-limit option, allowing greater flexibility on how many
tests will be run.
The counter-intuitive use of '--platform all' or '--platform default'
is no longer supported as these use-cases are covered by other
command line options.
Change-Id: I45b0050df52e1a22e75534a3876d89e3c77698c9
Signed-off-by: Andrew Boie <andrew.p.boie@intel.com>
These don't have necessary symbol information to determine if the
kernel is XIP or not. Fail instead of giving bogus data.
Change-Id: I87f6eeb5983f5275929e5b8d448a054b83e23d8f
Signed-off-by: Andrew Boie <andrew.p.boie@intel.com>
Python 2 isn't supported anymore and doesn't get fixes/updates.
For the long-term maintenance health of sanitycheck, convert to
use Python 3 interpreter, which mostly involved changes to how
strings are handled.
Change-Id: Ic42f5b2328001f04d876fd650986c4433968a76b
Signed-off-by: Andrew Boie <andrew.p.boie@intel.com>
Specify the kernel being tested instead of relying on parsing
Makefile for KERNEL_TYPE.
This is needed for testcases that build for multiple kernels.
Change-Id: Ic4df2fa66eee4cd498955aee2b0da80a99d35e30
Signed-off-by: Anas Nashif <anas.nashif@intel.com>
We have test cases in both samples and tests directories,
so cover both by scannining the root directory.
Change-Id: I5258d11b23ced2cc4fc88af4d40269310442dc9f
Signed-off-by: Anas Nashif <anas.nashif@intel.com>
Some tests take a very long time to complete, add a directive in
testcase.ini to mark them as such. Unless --enable-slow is passed
on the command line, these tests are not executed, only compiled.
app_kernel and test_sha256 marked as slow test cases. It appears
test_sha256 was being skipped earlier due to platform_whitelist line
which has been removed; the test should be able to run on all
microkernel-supporting boards. Same with test_aes, which completes
very quickly and is not marked as 'slow'.
Change-Id: I39ec8212f3fa10122ff786c10b6659d22bae64e3
Signed-off-by: Andrew Boie <andrew.p.boie@intel.com>
This reverts commit e3c2477c52f792ec972f9715ac9a810e52bc9929.
This was done for windows build support. We can disable this for
windows individually instead of disabling it for everyone.
Change-Id: I8bd322b024c0471f89e83a41589c03783c0f9893
Signed-off-by: Anas Nashif <anas.nashif@intel.com>
Updates the sanitycheck script to identify the "init_array" section
as a valid section in the image.
Change-Id: I55c20909d3734e68e8ec319e1abde94da93665a1
Signed-off-by: Peter Mitsis <peter.mitsis@windriver.com>
When an app fail to run, windows show a pop-up with the errors.
To avoid this pop-up we disable this feature by default
Change-Id: If21f3aaf37555052874ee8e29310e955124d2cb0
Signed-off-by: Louise Mendoza <yonattan.a.louise.mendoza@intel.com>
When config_whitelist is used in a testcase that has extra
arguments, those were not set correctly and the test case build would
fail in some cases.
Change-Id: Idb8a4f07d29beefe62de932afa7cc0986e8dd1d4
Signed-off-by: Anas Nashif <anas.nashif@intel.com>
In case we want to skip a test for whatever reason without having to change
the other filters or deleting the ini file.
Change-Id: I8af527b1c56b8a2f395cb9ca336162233f150c2e
Signed-off-by: Anas Nashif <anas.nashif@intel.com>
This patch enables defining microkernel events within source code.
This is similar to other private kernel object patches.
The test has been modified a little bit due to the fact that
the event ID is now a memory address, instead of numeric ID.
Change-Id: Ie3c8d4f4e459d9c631e50bb242cf7a05ca8ea82c
Signed-off-by: Daniel Leung <daniel.leung@intel.com>
Signed-off-by: Anas Nashif <anas.nashif@intel.com>
When building a sinle application, use the compiler option checking
macro, disable macro if running sanity checks to speed up build
time.
Change-Id: I63b17f4bd9fdff9030f1b2550ec8b05ee0bc2df2
Signed-off-by: Anas Nashif <anas.nashif@intel.com>
Originally, the config_whitelist was done on the platform level
ignoring any config options in the testcase. This change generates
the defconfig for all relevant testcases that have config_whitelist
and applies filters based on the testcase defconfig.
If we have many testcases using config_whitelist it might be a bit
slower, but it is more accurate.
Change-Id: Ie1e871e85f05196d04419d6cf9a87f9927bfbfd1
Signed-off-by: Anas Nashif <anas.nashif@intel.com>
XIP is currently implemented in Zephyr by copying only the RW
sections from flash into RAM. For these sections, if the VMA spacing
is different than the LMA spacing the sections in RAM will not be
at the right addresses. Warn about this situation.
Change-Id: I59a55fc27703103e37a7f0bbc34f236047621e0e
Signed-off-by: Andrew Boie <andrew.p.boie@intel.com>
- Several recently added binary sections were not being taken
into consideration, unknown sections now trigger test case
failure
- Use same VMA/LMA terminology that objdump uses and show both
addresses
Change-Id: I641eef64aaed6612f62e5aa092f66baaa2797c70
Signed-off-by: Andrew Boie <andrew.p.boie@intel.com>
Fixes an issue if CONFIG_KERNEL_BIN_NAME is customized.
There isn't a mechanism to parse the project defconfig, but
as there should be exactly one .elf artifact in the outdir
just glob it.
Change-Id: Ib3d517006dd13ef6e3e10ea3a8e975c287344d56
Signed-off-by: Andrew Boie <andrew.p.boie@intel.com>
The sizing code was merged with the sanitycheck script. A new
-z option has been added to simply measure and report sizes
to stdout instead of running any tests.
Change-Id: I1450e3b08e3bd0030d2df00190da884621962d01
Signed-off-by: Andrew Boie <andrew.p.boie@intel.com>
This shows all changes in size, good or bad. Useful for testing
size optimizations.
Change-Id: I47124b64f7d751120af666a4b7c7efd4bbed4ac8
Signed-off-by: Andrew Boie <andrew.p.boie@intel.com>
This policy was in place for the old sanity checks, was simply
overlooked.
Change-Id: Id021a669b35622f9e9cba23b4417c5d0b7afc15c
Signed-off-by: Andrew Boie <andrew.p.boie@intel.com>
-C/--clean is now -n/--no-clean. Made this change since the build system
can't always do incremental rebuild correctly depending on what was
changed (such as Make variables).
Change-Id: I199e505693479184b2f197ddb584ade5eca1704a
Signed-off-by: Andrew Boie <andrew.p.boie@intel.com>
The online help ./scripts/sanitycheck --help describes usage.
Most users will simply want to run with no arguments.
Change-Id: Icedbbfc22599a64a6e3dbbb808ff3276db06f2e0
Signed-off-by: Andrew Boie <andrew.p.boie@intel.com>
