Commit graph

450 commits

Author SHA1 Message Date
Anas Nashif cf21f5f10b sanitycheck: enhance toolchain filtering
Do not run filters on platforms not supported by toolchain.

Signed-off-by: Anas Nashif <anas.nashif@intel.com>
2017-06-22 09:22:48 -04:00
Anas Nashif a792a3d410 sanitycheck: support testcases in yaml instead of ini
This commit changes the syntax of the testcase files and changes the
behaviour and configuration of the sanitycheck script.

To avoid having multiple files with different syntax for boards,
samples, tests; this change unifies the syntax and uses YAML instead of
INI.

We maintain the current keywords used in the old syntax and maintain the
flexibility of adding tests with different configuration by using YAML
list configuration. On top of that, the following features are added:

- We now scan for board configurations in the boards directory and look
for a YAML file describing a board and how it should be tested. This
eliminates the need for listing boards per architecture in a special ini
file under scripts/.

- We define hardware information charachterstics in the board YAML file
that helps identifying if a certain test should run on that board or
not. For example, we can specify the available RAM in the board and
filter tests that would require more RAM than the board can handle.

- Boards can be set as default for testing meaning that we always run a
test case (build and run of possible) when sanitycheck is called without
any arguments. Previously this was done only by selecting the first
board defined for a specific architecture.

- Tests can be configured to run on all possible boards, this is to make
sure we always build some basic tests for all boards to catch issues
with the core kernel features.

Signed-off-by: Anas Nashif <anas.nashif@intel.com>
2017-06-21 20:56:53 -04:00
David B. Kinder 29963c30d1 scripts: sanitycheck: more spelliing fixes
PR #515 got merged before I could submit some additional spelling
fixes.

Signed-off-by: David B. Kinder <david.b.kinder@intel.com>
2017-06-16 17:15:57 -04:00
Johan Hedberg 9703927f84 net: buf: Move net_buf_pool objects to dedicated linker area
Moving the net_buf_pool objects to a dedicated area lets us access
them by array offset into this area instead of directly by pointer.
This helps reduce the size of net_buf objects by 4 bytes.

Signed-off-by: Johan Hedberg <johan.hedberg@intel.com>
2017-06-16 17:05:06 +03:00
Paul Sokolovsky ff70add7ab scripts: sanitycheck: Clean up help text.
Fixes a typo, punctuation, and a bit of old info about test paths.

Signed-off-by: Paul Sokolovsky <paul.sokolovsky@linaro.org>
2017-06-15 23:04:12 -04:00
Andy Gross 25309ab51a scripts: sanitycheck: Switch config target
This patch changes the config target to use the config-sanitycheck
target.  The config-sanitycheck target gets not only the Kconfig
options, but also the DTS generated options.

Signed-off-by: Andy Gross <andy.gross@linaro.org>
2017-06-15 16:14:59 -05:00
Anas Nashif e6fcc0105e sanitycheck: remove color output from xml
Signed-off-by: Anas Nashif <anas.nashif@intel.com>
2017-05-17 09:10:42 -05:00
Anas Nashif 0605fa33ea sanitycheck: junit: add total duration of run
Signed-off-by: Anas Nashif <anas.nashif@intel.com>
2017-05-15 15:54:50 -04:00
Anas Nashif 202d1308ae sanitycheck: add classname to testcase
classname seems to be expected by some parsers, so use it in the report,
the name now has the same value, we can add some more information once
we have meaningful metadata for the testcases with more details.

Signed-off-by: Anas Nashif <anas.nashif@intel.com>
2017-05-15 15:54:50 -04:00
Anas Nashif 035799fe45 sanitycheck: add subsets support
This new options makes it possible to only run a subset of the tests
which will allow us to run sanitycheck on multiple hosts and merge the
results into one report. This way we do not need to worry about
selecting specific architectures to be run on a certain host.

The option accepts a string value: x/y where x is the subset and y is
the total, so if we specify --subset 1/5, we will only run the first
fifth of the total tests, --subset 5/5 would only select the last fifth.

To get consistent results, the testcase instance list is now ordered,
to avoid duplications and have full coverage.

Signed-off-by: Anas Nashif <anas.nashif@intel.com>
2017-05-15 15:54:50 -04:00
Anas Nashif accc8eb6f5 sanitycheck: include qemu log when failure is not a build issue
When tests run in Qemu fail, we want to see the log from Qemu, not the
successful build log.

Signed-off-by: Anas Nashif <anas.nashif@intel.com>
2017-05-01 17:48:24 -04:00
Anas Nashif b3311edff6 sanitycheck: support xunit report
Generate a test report using junit/xunit format with all details and
logs that can be published on the web to show results from a sanitycheck
run.

Output is stored in scripts/sanity_chk/ alongside the CSV file.

Change-Id: I5ea6f409c1f86f408eeae870b90a953e71046da9
Signed-off-by: Anas Nashif <anas.nashif@intel.com>
2017-04-26 21:37:33 -04:00
Andrew Boie c09b4b84fc sanitycheck: catch ValueErrors when evaulating expressions
Change-Id: Iae4eb0b242427b98dd6515676c024f307446e49d
Signed-off-by: Andrew Boie <andrew.p.boie@intel.com>
2017-04-18 20:51:48 +00:00
Paul Sokolovsky cb2ae52fae scripts: sanitycheck: Typo fixes in docstring.
Change-Id: I219d9f6af972b4090536fe83b6b93e6b7191572a
Signed-off-by: Paul Sokolovsky <paul.sokolovsky@linaro.org>
2017-03-13 18:06:05 +03:00
Tomasz Bursztyka e38a9e8b9c net: Switch from per-iface tx thread to a unique k_poll triggered one
Now that k_poll landed in the kernel, it's worth using it to save
memory and reduce the number of threads at runtime.

Such switch has been first done in bluetooth (see hci_core.c and conn.c
in subsys/bluetooth/host). Since network interfaces kind of follows the
same design for sending data, it was then easy to copy the same change as
in bluetooth.

Change-Id: I7f9734b88ac818284bbabaedc946b4765b905ebb
Signed-off-by: Tomasz Bursztyka <tomasz.bursztyka@linux.intel.com>
2017-03-09 20:33:49 +02:00
Anas Nashif 6d72e63f81 sanitycheck: add _k_queue_area to recognised sections
Signed-off-by: Anas Nashif <anas.nashif@intel.com>
Change-Id: I2b5084e651d57ffeada38327c774af6471b43efc
2017-02-27 19:52:58 -05:00
Andrew Boie 1e4e68b52a sanitycheck: only disable tryrun when using SDK
Change-Id: I8018db62408240208b6f0b5e8f6f5ddf96d497f2
Signed-off-by: Andrew Boie <andrew.p.boie@intel.com>
2017-02-13 11:39:03 -08:00
Andrew Boie 60823c2325 sanitycheck: fix defconfig regex
By convention Kconfigs are in all uppercase, but not strictly
required to be so. Currently the Xtensa build defines
CONFIG_${XTENSA_CORE} which can have lowercase letters. This
is marked for cleanup in ZEP-1711.

Issue: ZEP-1711
Change-Id: Ia901ea1688202129f1629001a77f498522859f14
Signed-off-by: Andrew Boie <andrew.p.boie@intel.com>
2017-02-13 11:39:03 -08:00
Mazen NEIFER 8f1dd3a6e5 sanitycheck: Fixed displayed log when current directory is a symbolic link.
When current directory is a symbolic link or in a symbolic link then os.getcwd
return the real path leading to os.path.relpath returning not so nice relative
path.

Change-Id: I3e24d3e6f1fe7185992f64803d3728e9588d67e1
Signed-off-by: Mazen NEIFER <mazen@nestwave.com>
2017-02-13 08:04:27 -08:00
Andrew Boie e536b270f0 sanitycheck: don't disable tryrun
This completely breaks all the cc-option checks. It's not clear
why this was done for everyone running sanitycheck.

Change-Id: I3f944423869ccb15fab29c28130820a93cd94917
Signed-off-by: Andrew Boie <andrew.p.boie@intel.com>
2017-02-13 08:04:27 -08:00
Mazen NEIFER 7f046378eb sanitycheck: support other simulators (xt-run and/or xtsc) than QEMU.
In scripts/Makefile.qemu, the target qemu is already an alias for taget run.
In scripts/Makefile.xt{,sc}-run there is no target qemu but a target run.

Change-Id: Ib9dce3561ef9367b17d30070354e709df4d7e528
Signed-off-by: Mazen NEIFER <mazen@nestwave.com>
2017-02-13 08:04:27 -08:00
Andrew Boie f7a6e28a3c sanitycheck: don't override environment for USE_CCACHE
Many users set USE_CCACHE=1 in their envionment. Sanitycheck was
disregarding that and placing USE_CCACHE=0 in the generated
Makefile unless --ccache was added to the command line every time
it was invoked.

Issue: ZEP-1663
Change-Id: Id3c1379f5039d4d2f4a3ae01d2ec8d0dd216f523
Signed-off-by: Andrew Boie <andrew.p.boie@intel.com>
2017-02-02 13:09:00 -08:00
Andrew Boie 7a992aebac sanitycheck: improve terse output
Values now spaced 4 apart since we commonly build thousands of
tests.

Percentage complete indicator added.

Change-Id: Ia7454dc1e8cc0701dc74dc8a4f6e7a37706524e8
Signed-off-by: Andrew Boie <andrew.p.boie@intel.com>
2017-01-18 18:47:38 +00:00
Andrew Boie 18ba153300 build: remove obsolete sections from linker scripts
These were causing some issues in the old kernel, but the unified
kernel no longer uses these.

Issue: ZEP-513
Change-Id: I87216565231cd244886fbffe4b4d420d1687b245
Signed-off-by: Andrew Boie <andrew.p.boie@intel.com>
2017-01-18 18:47:17 +00:00
Anas Nashif bfcdfaf941 sanitycheck: riscv: add vector to recognised sections
Change-Id: I7f1692eb1bd51b8efc16919780aebaf890b8872c
Signed-off-by: Anas Nashif <anas.nashif@intel.com>
2017-01-13 20:00:16 +00:00
Andrew Boie 822b08711f sanitycheck: properly identify when QEMU crashes out
This had been erroneously reported as a build error.

Issue: ZEP-1460
Change-Id: Idddaf833011beb1165bd766dd115c021c5b90cdf
Signed-off-by: Andrew Boie <andrew.p.boie@intel.com>
2017-01-11 14:03:39 +00:00
Andrew Boie d01535cbb6 sanitycheck: don't explode if log data can't be read
Issue: ZEP-723
Change-Id: I5d2b2e756fed7e2906da1e84c2da332682893660
Signed-off-by: Andrew Boie <andrew.p.boie@intel.com>
2017-01-11 14:03:33 +00:00
Anas Nashif 6d4ff2303e sanitycheck: reduce number of unnecessary configuration builds
When running a reduced set of tests with --platform-limit or using the
default of 1 we build the configuration (make initconfig) for all
platforms, although we are only interested in a limited set. On my
machine this is 100s of build time for configurations we are never going
to use.
This reduces the number of builds to what we really need and speeds up
sanitycheck runtime overall.

Might need a few more optimisations.

Change-Id: I813fabf1453f19ce7a83b28e4792b5c80616d936
Signed-off-by: Anas Nashif <anas.nashif@intel.com>
2017-01-05 02:05:58 +00:00
Inaky Perez-Gonzalez 9a36cb6456 sanitycheck: add --log-file to log everything to a file too
Currently CI plays tricks with `tee` to capture the output and post
process it. This makes it more complex to handle and easier for it to
fail in ways it should not.

So add a simple log capture option sanity check that mirrors the
stdout output, removing color encoding for errors.

Change-Id: I6de0b6cfe4da9c289f537979545dddbcd49cf834
Signed-off-by: Inaky Perez-Gonzalez <inaky.perez-gonzalez@intel.com>
2017-01-03 19:21:31 +00:00
Kumar Gala 60a079cb2c Revert "sanitycheck: reduce number of unnecessary configuration builds"
This reverts commit 4e2cf33d9d.

The change does not work correct if we have a filter and specify a
specific platform, such as:

./scripts/sanitycheck -p 96b_nitrogen -i -s samples/hello_world/test_singlethread

Change-Id: I43a4f25ed25c8fe5379cda23b6dbfa8f9ae5cb0d
Signed-off-by: Kumar Gala <kumar.gala@linaro.org>
2016-12-30 09:43:48 -06:00
Anas Nashif 4e2cf33d9d sanitycheck: reduce number of unnecessary configuration builds
When running a reduced set of tests with --platform-limit or using the
default of 1 we build the configuration (make initconfig) for all
platforms, although we are only interested in a limited set. On my
machine this is 100s of build time for configurations we are never going
to use.
This reduces the number of builds to what we really need and speeds up
sanitycheck runtime overall.

Might need a few more optimisations.

Change-Id: I53d6bacf66045dde7baabbaa40b4fbfc4a01782a
Signed-off-by: Anas Nashif <anas.nashif@intel.com>
2016-12-28 19:51:43 +00:00
Anas Nashif 5312754988 sanitycheck: remove kernel keywords
Change-Id: I076491f58147ffc70d84b62e1d78b4ae419add7b
Signed-off-by: Anas Nashif <anas.nashif@intel.com>
2016-12-28 18:38:46 +00:00
Jukka Rissanen 8e84e36f14 net: Declare a macro to define stacks used by networking
The macro defines the stack as usual, but if user has
enabled net shell (CONFIG_NET_SHELL), then additional
information about the stack is stored in net_shell
linker section. The information in the net_shell linker
section is then used to print information about the
stacks in the networking sub-system.

Change-Id: Ic6e9f16a73a192b9a59d32a6d0070322382f98bd
Signed-off-by: Jukka Rissanen <jukka.rissanen@linux.intel.com>
2016-12-02 12:41:14 +02:00
Jukka Rissanen dbb3988ee6 net: Remove nbr_data section as it is useless
No need to store the nbr pool data into a dedicated section
as we have only one nbr pool defined.

Change-Id: I6f2afcce57b5f588878496bf085567b938e32c80
Signed-off-by: Jukka Rissanen <jukka.rissanen@linux.intel.com>
2016-12-02 12:40:27 +02:00
Tomasz Bursztyka b56c4df869 sanitycheck: Add support for section net_l2_data
We missed that section, so sanitycheck was failing.
Also putting the right section in the relevant array.

Change-Id: I361ffa1707c01f0d3c870a7bc36a86ada66af1d2
Signed-off-by: Tomasz Bursztyka <tomasz.bursztyka@linux.intel.com>
2016-12-02 12:40:23 +02:00
Tomasz Bursztyka 11f572c9e1 sanitycheck: Recognize native IP stack specific sections
Without these, test case will fail miserably.

Change-Id: Ib46d167b8782c33362ac6edb9d8c50abc25908c0
Signed-off-by: Tomasz Bursztyka <tomasz.bursztyka@linux.intel.com>
2016-12-02 12:40:13 +02:00
Anas Nashif e3febe9aa5 sanitycheck: allow error on deprecation warnings
Added new option to make sanitycheck script error on deprecation
warnings, to use:

sanitycheck --error-on-deprecations

For example, to find usage of legacy APIs in master, run the following:

sanitycheck -e legacy --error-on-deprecations

Change-Id: Ib83c266c8357475840dea03d62ceefed72b73f27
Signed-off-by: Anas Nashif <anas.nashif@intel.com>
2016-12-01 21:06:07 +00:00
Andrew Boie 52fef67c13 sanitycheck: allow for extra binary sections in testcase.ini
Change-Id: I9e47a58f3f32ba093f7916af111a289b620c30d5
Signed-off-by: Andrew Boie <andrew.p.boie@intel.com>
2016-11-30 14:14:10 +00:00
Anas Nashif d9616b9400 sanitycheck: abort if last_sanity.csv not found
When running --only-failed make sure last_sanity.csv exists.

Change-Id: I862529a58f337987c37da83fd5541dd4f42b3391
Signed-off-by: Anas Nashif <anas.nashif@intel.com>
2016-11-29 13:41:00 -05:00
Andrew Boie 4187822d08 sanitycheck: build for unified kernel only
Change-Id: I3bd744b10341bc7292fb0670faf217933efe9edc
Signed-off-by: Andrew Boie <andrew.p.boie@intel.com>
2016-11-04 18:32:15 +00:00
Allan Stephens 3f5c74c922 unified: Add linker script support for all kernel object types
Enhance the linker script so that all statically-defined kernel
objects of a given type are co-located.

Note 1: This capability is needed so that static kernel objects
can be added to the associated object tracing list. (Some kernel
object types are already co-located to permit the kernel to do
other initialzation on statically-defined objects.)

Note 2: A follow-up commit is needed to ensure statically-
defined kernel objects are placed in the sections referenced
in the linker script.

Note 3: The legacy script info remains for now to allow
applications using the microkernel or nanokernel to continue
to build.

Change-Id: I40d831f9e183fb121c950e30fa8298b6d529375b
Signed-off-by: Allan Stephens <allan.stephens@windriver.com>
2016-11-02 21:56:26 +00:00
Yael Avramovich d5db35204a console: shell: Support multiple modules
Ability to use Zephyr shell by multiple modules simultaneously, each
module for its own usage.

Old shell implementation enabled the user to call only one module
commands, not all of the modules simultaneously.

Change-Id: I0ef8fa2fd190b7490c44fe91d1016363258302c9
Signed-off-by: Yael Avramovich <yael.avramovich@intel.com>
Signed-off-by: Anas Nashif <anas.nashif@intel.com>
2016-11-02 00:23:18 +00:00
Kumar Gala d5719a64c6 sanitycheck: Add support for ccache
Add a ccache option to allows enabling the use of ccache for builds
generated by sanitycheck.

Change-Id: Ia997ed29f38a3d2d71ba143456e6fcbc5946d56e
Signed-off-by: Kumar Gala <kumar.gala@linaro.org>
2016-10-28 17:04:27 +00:00
Anas Nashif f8dcad4a5b sanitycheck: filter out the prebuilt.elf introduced by Makefile cleanup
commit 018b831a36 keeps another elf file in the
outdir which confused the sanitycheck script.

Change-Id: I87a9bcee6afe6526ecc24f1f5d9cf38101247ce2
Signed-off-by: Anas Nashif <anas.nashif@intel.com>
2016-10-27 18:30:27 -04:00
Szymon Janc 50b9317832 sanitycheck: Add support for unified kernel
This fix sanitycheck for apps that specify unified kernel.

Change-Id: I3f1c4d5a566f6833e56bbda097cb497ccdbfbec9
Signed-off-by: Szymon Janc <szymon.janc@codecoup.pl>
2016-10-25 19:01:28 +00:00
Genaro Saucedo Tejada 28bba92dc0 sanity: Load arguments from files
Python module argparse has a feature to load arguments from a file.
This commit enables such feature on sanitycheck script so files
prefixed by + (plus sign) are loaded during argument parsing stage
and included as arguments to such script.

Jira: ZEP-122

Change-Id: Ibb0a6e8464218a4f58ce4044a5a0860dc5b32480
Signed-off-by: Genaro Saucedo Tejada <genaro.saucedo.tejada@intel.com>
2016-10-25 13:05:36 +00:00
Anas Nashif dfa86e2aca sanitycheck: provide option to exclude tags
This will make it possible for us to optimize the list of tests we run, for
example, we could exclude footprint tests from the main run because those are
run as part of the footprint checks later in the CI job.

Use like this: sanitycheck -e footprint

Change-Id: I4e7a3aa6fac2ba1c9c99b356f08459da97fda777
Signed-off-by: Anas Nashif <anas.nashif@intel.com>
2016-10-24 22:55:13 +00:00
Andy Ross 8d801a5e27 sanitycheck: Remove linker VMA/LMA offset checking
This misfeature can crop up at development time as the linker scripts
are modified, not just as a regression in known tests (c.f. ZEP-955).
So it makes more sense to put it in a check_link_map script inovked
for every build.  Remove it from sanitycheck.

Change-Id: I9c2f47a26a3bc03170b895545476350341e91170
Signed-off-by: Andy Ross <andrew.j.ross@intel.com>
2016-10-05 09:48:54 +00:00
Jaakko Hannikainen 54c90bcbd5 tests: Add gcov support
If the -C flag is given to sanitycheck, generate gcov files for unit
tests and render them with lcov.

Signed-off-by: Jaakko Hannikainen <jaakko.hannikainen@intel.com>
Change-Id: Ic25eae6a3cfc2c45595bd6aa235df2c483aaf6ec
2016-09-30 21:17:42 +00:00
Jaakko Hannikainen ca505f8452 ztest: Add native building support
This commit allows building tests using the ztest framework without
including Zephyr. This can be used to enable unit testing single
functions, even static ones.

Origin: Original

Change-Id: Ib7e84f4bd9bbbf158b9a19edaf6540f28e47259f
Signed-off-by: Jaakko Hannikainen <jaakko.hannikainen@intel.com>
Signed-off-by: Anas Nashif <anas.nashif@intel.com>
2016-09-30 21:17:39 +00:00
Dmitriy Korovkin 3c426888a1 unified: Implement memory pools
Due to the memory pool structure only static declaration of
memory pool is possible.

Change-Id: I4797ed88fd2ac3b7812ff26e552e1745611c4575
Signed-off-by: Dmitriy Korovkin <dmitriy.korovkin@windriver.com>
2016-09-20 22:04:46 +00:00
Tomasz Bursztyka a6cf6038e3 gpio: Remove obsolete API 1.0 callback mechanism
In 1.0 you could set only one callback on the whole gpio controller. It
was impossible for another sub-system to add another callback, without
overwritting an existing one.

Such API has been obsolete for a long time and no one is using it
anymore. Thus removing it entirely.

Change-Id: I6a17fd99373dc6cef1fa2ebb421e992412d5015e
Signed-off-by: Tomasz Bursztyka <tomasz.bursztyka@linux.intel.com>
2016-09-14 10:51:09 +00:00
Andrew Boie ba61200a8f sanitycheck: allow extra arguments to be passed to the build
In some cases, particularly when developing experimental features, it is
useful to be able to pass parameters to 'make' globally for all
sanity checks.

Change-Id: I19eddf222a74064cf6f22ec641fa6b4ff72992e7
Signed-off-by: Andrew Boie <andrew.p.boie@intel.com>
2016-09-06 22:44:20 +00:00
Andrew Boie 05e3d7f893 sanitycheck: complain loudly about missing platforms
If defconfig files exist under boards/ that don't correspond to
the set of platforms defined in the .ini files under
scripts/sanity_chk/arches, complain about this. We should be at
least compiling all our supported platforms in CI runs.

At the moment it flags 5 boards. Future patches should either add
these boards to the appropriate arch .ini files or remove them
from the kernel.

Change-Id: Id825d6c6bd32089d9f1bead11bc0765685e3771a
Signed-off-by: Andrew Boie <andrew.p.boie@intel.com>
2016-09-06 16:32:03 +00:00
Javier B Perez 4b554baa87 sanitycheck: support for multiple toolchain
Added support for multiple toolchain usage.
Every arch contains a list of supported toolchain.
Each board can override the supported toolchain list.
If the board is not supported in the current toolchain,
will be discarded.

The current toolchain is defined by ZEPHYR_GCC_VARIANT.
Added support for toolchain ISSM and ZEPHYR.

Change-Id: I31e9b39ba01f6e9bdc4557702428cd09e05f492a
Jira: ZEP-592
Signed-off-by: Javier B Perez <javier.b.perez.hernandez@intel.com>
2016-08-30 21:44:24 +00:00
Javier B Perez 7941454b70 sanitycheck: filter: add support to use env variables
Added support in the sanitycheck to use env variable
value in the filter.

Change-Id: I76388dc04557dcd31e651d23ae8edf96a8fe2474
Jira: ZEP-592
Signed-off-by: Javier B Perez <javier.b.perez.hernandez@intel.com>
2016-08-29 14:43:01 +00:00
Genaro Saucedo Tejada 2968b3644c sanity: non utf-8 characters are tolerated
Sanity check script was not handling non utf-8 characters as a result
it crashed and didn't kill qemu and test used to hang.

Now tests fail upon untranslatable bytes.

JIRA: ZEP-673

Change-Id: I885a185e0481083501fce15cd51412bfdd26a7d7
Signed-off-by: Genaro Saucedo Tejada <genaro.saucedo.tejada@intel.com>
2016-08-10 14:31:57 +00:00
Andrew Boie 12244a3a8e sanitycheck: allow deprecated APIs
sanitycheck is set up to error out on any compiler warnings.
However in the case of deprecated APIs, we may feel compelled
to still test them while they remain in-tree. Don't fail the
build if deprecated APIs are used during sanity runs.

Change-Id: Ic3196896689706d308117e693ab720713d67d698
Signed-off-by: Andrew Boie <andrew.p.boie@intel.com>
2016-07-28 21:48:23 +00:00
Andrew Boie 5512105562 sanitycheck: add -R to build all tests with assertions
Some tests individually enable assertions; this forces them on
for all tests. Intended for use with daily builds. Disregard
footprint or benchmark data with this enabled.

Change-Id: I7a0822432d5cab7f3ab0767faf214fde536cd68e
Signed-off-by: Andrew Boie <andrew.p.boie@intel.com>
2016-07-20 23:41:05 +00:00
Andrew Boie 3b9302158c nios2: set up common linker script for XIP and non-XIP
We will require 6 variables to be defined by SOC-specific
linker script; these values in turn can be pulled from
defines in layout.h.

To help position code correctly we define two new ELF sections
for this arch, 'reset' and 'exceptions'.

Change-Id: Idffbd53895945b7d0ec0aac281e5bf7c85b4b2c2
Signed-off-by: Andrew Boie <andrew.p.boie@intel.com>
2016-06-10 15:22:41 +00:00
Andrew Boie 7a87f9f500 expr_parser: support ':' operator
Similar to '==' but compiles the RHS into a regular expression
and attempts to match whatever is in the environment against it.

Change-Id: I7a03452ef88d067b62661d14dc6f42273de436fa
Signed-off-by: Andrew Boie <andrew.p.boie@intel.com>
2016-06-02 23:10:07 +00:00
Andrew Boie 3ea7892410 sanitycheck: allow for more expressive filtering in testcase.ini
The old 'config_whitelist' directive in testcase.ini has been removed.
We use the new expr_parser module to parse a 'filter' directive which
is a boolean expression. This gives a great deal more flexibility
in how tests can be filtered.

To keep the tree bisectable, use of config_whitelist in testcase.ini
converted to the new expression language.

Change-Id: I0617319818c5559c0f0569d2fa73d09b681cac51
Signed-off-by: Andrew Boie <andrew.p.boie@intel.com>
2016-05-04 22:57:39 +00:00
Tomasz Bursztyka fea1c49ba2 gpio: Improve the public API to handle multi callbacks
Many sub-systems might require to set a callback on different pins.
Thus enabling it via changing the API.

It is also possible to retrieve private-data in the callback handler
using CONTAINER_OF() macro (include/misc/util.h).

Former API is still available, and is emulated through the new one.
Using both should not be a problem as it's using new API calls.
However, it's now better to start using the new API.

Change-Id: Id16594202905976cc524775d1cd3592b54a84514
Signed-off-by: Tomasz Bursztyka <tomasz.bursztyka@linux.intel.com>
Signed-off-by: Johan Hedberg <johan.hedberg@intel.com>
2016-04-22 22:15:12 -04:00
Daniel Leung 00525c2acf sanitycheck: align output for easier visual comparison
() The test names are getting longer so adds more padding.
() Aligns the numerical size numbers for easier visual
   comparison when scrolling in shell.

Change-Id: Id218a641e9d5dffb77bf813ffbd828c48d14020f
Signed-off-by: Daniel Leung <daniel.leung@intel.com>
2016-04-12 12:00:47 +00:00
Andrew Boie 14ac902633 sanitycheck: fix test names to be same as before
Add the last path information in the testcase root to the
test name, so that there is much less of chance of name
collisions and the reports come out with the same names
as before.

Change-Id: I75ed6ee96251058dd2547e57be31db3c2517a497
Signed-off-by: Andrew Boie <andrew.p.boie@intel.com>
2016-04-08 13:31:53 -07:00
Andrew Boie 3d348715d8 sanitycheck: be smarter about scanning for test cases
A recent tree change moved testcases to both the tests/ and
samples/ directory. The default testcase root was just made
the root of the Zephyr tree, which could lead to potentially
long scan times as every single directory in-tree was checked.
For example, it could be very time-consuming to scan everything
under the sanity-out/ dir if -n is used and --all was used in
a previous run.

Now we allow multiple testcase roots to be supplied on the
command line, and by default only scan under tests/ and
samples/.

Change-Id: I8dac747835b87801474e08c236913e04db64f87b
Signed-off-by: Andrew Boie <andrew.p.boie@intel.com>
2016-04-08 20:09:18 +00:00
Daniel Leung 7f8501090c sanitycheck: add ability to use arbitrary report for comparison
This adds the ability for sanitycheck to use arbitrary report file
for size comparison. This is useful when optimizing binary size.
For example, a report can be generated with master head, and used
as base for comparison.

Change-Id: Id8c61146c498ba62c4cc77a9ef7d390c0fb11487
Signed-off-by: Daniel Leung <daniel.leung@intel.com>
2016-04-08 20:09:11 +00:00
Daniel Leung 6b17007b56 sanitycheck: parallelize binary size calculations
Instead of running nm/objdump and parsing the results one testcase
at a time, do these in parallel to speed up sanity check a bit.

The sanitycheck now requires at least Python 3.2 to run because of
the usage of concurrent.futures.

This reduces build time (--all -b) from 4:30 to 3:08 on 24C/48T
Xeon-E5.

Change-Id: I8e7c1efb2f473c7f2b65658f8ed9a101ed091eea
Signed-off-by: Daniel Leung <daniel.leung@intel.com>
2016-04-08 14:50:20 +00:00
Daniel Leung 1bdc6f8b22 sanitycheck: fix initlevel section being mentioned twice
The "initlevel" section is being metioned twice. It does not affect
the final size calculation, but just remove it just for reader's
sanity.

Change-Id: I0e7d6e82c730631d2c83ca09dde0f675805840f3
Signed-off-by: Daniel Leung <daniel.leung@intel.com>
2016-04-07 00:56:41 +00:00
Andrew Boie e57a1e5622 sanitycheck: don't require arch.ini blocks for every platform
If the arch ini file doesn't have a specific block for the platform,
just assume all defaults.

Change-Id: Id6ab1ede8936bdd6006a5c4fd2f9a4f6aaf42478
Signed-off-by: Andrew Boie <andrew.p.boie@intel.com>
2016-03-23 15:35:38 +00:00
Andrew Boie 821d832f6d sanitycheck: allow N platforms per arch to be run
It is an unfortunate fact of life that our CI machines are often
severly underpowered virtual machines that take quite some time
to run sanitycheck --all. The only other option was to just use
the default test case filtering semantics which chose one platform
per arch to test.

Now it is possible to specify N platforms per arch with the new
--platform-limit option, allowing greater flexibility on how many
tests will be run.

The counter-intuitive use of '--platform all' or '--platform default'
is no longer supported as these use-cases are covered by other
command line options.

Change-Id: I45b0050df52e1a22e75534a3876d89e3c77698c9
Signed-off-by: Andrew Boie <andrew.p.boie@intel.com>
2016-03-23 15:35:38 +00:00
Andrew Boie 8f0211dcf3 sanitycheck: fail on footprint analysis of stripped ELFs
These don't have necessary symbol information to determine if the
kernel is XIP or not. Fail instead of giving bogus data.

Change-Id: I87f6eeb5983f5275929e5b8d448a054b83e23d8f
Signed-off-by: Andrew Boie <andrew.p.boie@intel.com>
2016-03-03 10:24:06 +00:00
Andrew Boie 08ce5a5da7 sanitycheck: use Python 3
Python 2 isn't supported anymore and doesn't get fixes/updates.
For the long-term maintenance health of sanitycheck, convert to
use Python 3 interpreter, which mostly involved changes to how
strings are handled.

Change-Id: Ic42f5b2328001f04d876fd650986c4433968a76b
Signed-off-by: Andrew Boie <andrew.p.boie@intel.com>
2016-02-24 17:10:39 +00:00
Andrew Boie 411a97c5d0 sanitycheck: correctly treat linker warnings as errors
Change-Id: I2fc82fe83d91121d2228251d9fba8a61ba33b63e
Signed-off-by: Andrew Boie <andrew.p.boie@intel.com>
2016-02-24 17:08:56 +00:00
Anas Nashif a7c4aa2e8a sanitychecks: add new keyword 'kernel' to testcase.ini
Specify the kernel being tested instead of relying on parsing
Makefile for KERNEL_TYPE.

This is needed for testcases that build for multiple kernels.

Change-Id: Ic4df2fa66eee4cd498955aee2b0da80a99d35e30
Signed-off-by: Anas Nashif <anas.nashif@intel.com>
2016-02-11 02:23:26 +00:00
Anas Nashif d93cb67109 scan for testcases in root to cover samples and tests
We have test cases in both samples and tests directories,
so cover both by scannining the root directory.

Change-Id: I5258d11b23ced2cc4fc88af4d40269310442dc9f
Signed-off-by: Anas Nashif <anas.nashif@intel.com>
2016-02-11 02:23:00 +00:00
Andrew Boie 6bb087c511 sanitycheck: add testcase.ini field for slow tests
Some tests take a very long time to complete, add a directive in
testcase.ini to mark them as such. Unless --enable-slow is passed
on the command line, these tests are not executed, only compiled.

app_kernel and test_sha256 marked as slow test cases. It appears
test_sha256 was being skipped earlier due to platform_whitelist line
which has been removed; the test should be able to run on all
microkernel-supporting boards. Same with test_aes, which completes
very quickly and is not marked as 'slow'.

Change-Id: I39ec8212f3fa10122ff786c10b6659d22bae64e3
Signed-off-by: Andrew Boie <andrew.p.boie@intel.com>
2016-02-10 22:12:55 +00:00
Anas Nashif c0ddcb7a49 Revert "Change the try-run option as disable by default."
This reverts commit e3c2477c52f792ec972f9715ac9a810e52bc9929.

This was done for windows build support. We can disable this for
windows individually instead of disabling it for everyone.

Change-Id: I8bd322b024c0471f89e83a41589c03783c0f9893
Signed-off-by: Anas Nashif <anas.nashif@intel.com>
2016-02-05 20:25:30 -05:00
Peter Mitsis a74d5d63f1 test c++: Recognize init_array as a valid section
Updates the sanitycheck script to identify the "init_array" section
as a valid section in the image.

Change-Id: I55c20909d3734e68e8ec319e1abde94da93665a1
Signed-off-by: Peter Mitsis <peter.mitsis@windriver.com>
2016-02-05 20:25:22 -05:00
Anas Nashif c740608ce5 Use BOARD instead of PLATFORM_CONFIG
Change-Id: I14db0087cd705df0db8a911071bc3949b5c79314
Signed-off-by: Anas Nashif <anas.nashif@intel.com>
2016-02-05 20:25:11 -05:00
Louise Mendoza 988e9eda1f Change the try-run option as disable by default.
When an app fail to run, windows show a pop-up with the errors.
To avoid this pop-up we disable this feature by default

Change-Id: If21f3aaf37555052874ee8e29310e955124d2cb0
Signed-off-by: Louise Mendoza <yonattan.a.louise.mendoza@intel.com>
2016-02-05 20:25:10 -05:00
Anas Nashif 8ea9d02cdf sanity: add extra arguments when creating defconfigs
When config_whitelist is used in a testcase that has extra
arguments, those were not set correctly and the test case build would
fail in some cases.

Change-Id: Idb8a4f07d29beefe62de932afa7cc0986e8dd1d4
Signed-off-by: Anas Nashif <anas.nashif@intel.com>
2016-02-05 20:24:43 -05:00
Andrew Boie 829c056260 sanitycheck: show absolute size deltas
Change-Id: I0de8446b32ca2def2baceab4fa837a022c82db84
Signed-off-by: Andrew Boie <andrew.p.boie@intel.com>
2016-02-05 20:24:36 -05:00
Anas Nashif 2bd99bcbdf sanitychecks: add skip keyword for skipping tests unconditionally
In case we want to skip a test for whatever reason without having to change
the other filters or deleting the ini file.

Change-Id: I8af527b1c56b8a2f395cb9ca336162233f150c2e
Signed-off-by: Anas Nashif <anas.nashif@intel.com>
2016-02-05 20:24:35 -05:00
Daniel Leung 851c6f8f60 microkernel: introduce private event objects
This patch enables defining microkernel events within source code.
This is similar to other private kernel object patches.

The test has been modified a little bit due to the fact that
the event ID is now a memory address, instead of numeric ID.

Change-Id: Ie3c8d4f4e459d9c631e50bb242cf7a05ca8ea82c
Signed-off-by: Daniel Leung <daniel.leung@intel.com>
Signed-off-by: Anas Nashif <anas.nashif@intel.com>
2016-02-05 20:24:35 -05:00
Anas Nashif 6f8d628d77 re-enable compile time option checking and make it conditional
When building a sinle application, use the compiler option checking
macro, disable macro if running sanity checks to speed up build
time.

Change-Id: I63b17f4bd9fdff9030f1b2550ec8b05ee0bc2df2
Signed-off-by: Anas Nashif <anas.nashif@intel.com>
2016-02-05 20:24:28 -05:00
Anas Nashif 2cf0df018d sanity: build testcase defconfigs for config_whitelist
Originally, the config_whitelist was done on the platform level
ignoring any config options in the testcase. This change generates
the defconfig for all relevant testcases that have config_whitelist
and applies filters based on the testcase defconfig.
If we have many testcases using config_whitelist it might be a bit
slower, but it is more accurate.

Change-Id: Ie1e871e85f05196d04419d6cf9a87f9927bfbfd1
Signed-off-by: Anas Nashif <anas.nashif@intel.com>
2016-02-05 20:24:28 -05:00
Anas Nashif 30d1387c5c sanity: add platform/arch exclude functionality
Change-Id: I8af20c04ba8b0872d5196aa166d2479bfa048275
Signed-off-by: Anas Nashif <anas.nashif@intel.com>
2016-02-05 20:24:28 -05:00
Andrew Boie 9882dcd3d2 sanitycheck: look for misaligned XIP VMA/LMA addresses
XIP is currently implemented in Zephyr by copying only the RW
sections from flash into RAM. For these sections, if the VMA spacing
is different than the LMA spacing the sections in RAM will not be
at the right addresses. Warn about this situation.

Change-Id: I59a55fc27703103e37a7f0bbc34f236047621e0e
Signed-off-by: Andrew Boie <andrew.p.boie@intel.com>
2016-02-05 20:24:27 -05:00
Andrew Boie 73b4ee69b7 sanitycheck: binary size calculation improvements
- Several recently added binary sections were not being taken
  into consideration, unknown sections now trigger test case
  failure
- Use same VMA/LMA terminology that objdump uses and show both
  addresses

Change-Id: I641eef64aaed6612f62e5aa092f66baaa2797c70
Signed-off-by: Andrew Boie <andrew.p.boie@intel.com>
2016-02-05 20:24:27 -05:00
Andrew Boie 5d4eb7812b sanitycheck: fix issues if alternate binary name is used
Fixes an issue if CONFIG_KERNEL_BIN_NAME is customized.
There isn't a mechanism to parse the project defconfig, but
as there should be exactly one .elf artifact in the outdir
just glob it.

Change-Id: Ib3d517006dd13ef6e3e10ea3a8e975c287344d56
Signed-off-by: Andrew Boie <andrew.p.boie@intel.com>
2016-02-05 20:24:22 -05:00
Andrew Boie b391e66edf sanitycheck: fix -O with relative paths
Change-Id: If4bcfb20113344bd62bcf33ccc594ac9b7c7a7e1
Signed-off-by: Andrew Boie <andrew.p.boie@intel.com>
2016-02-05 20:15:34 -05:00
Andrew Boie 4ef16c5314 sanitycheck: properly report defconfig build errors
Change-Id: I9122f9c1d8232caa110dd6459c5f7cf2a079e36b
Signed-off-by: Andrew Boie <andrew.p.boie@intel.com>
2016-02-05 20:15:33 -05:00
Andrew Boie bbd670c9fc sanitycheck: add -z option for 'truesize' functionality
The sizing code was merged with the sanitycheck script. A new
-z option has been added to simply measure and report sizes
to stdout instead of running any tests.

Change-Id: I1450e3b08e3bd0030d2df00190da884621962d01
Signed-off-by: Andrew Boie <andrew.p.boie@intel.com>
2016-02-05 20:15:24 -05:00
Andrew Boie ea7928f642 sanitycheck: add --all-deltas option
This shows all changes in size, good or bad. Useful for testing
size optimizations.

Change-Id: I47124b64f7d751120af666a4b7c7efd4bbed4ac8
Signed-off-by: Andrew Boie <andrew.p.boie@intel.com>
2016-02-05 20:15:24 -05:00
Anas Nashif 427cdd36ec sanity: add new line when emitting an error
Change-Id: I4075dbe3706b45e926e9ab4d0f620fe262aaa705
Signed-off-by: Anas Nashif <anas.nashif@intel.com>
2016-02-05 20:15:19 -05:00
Andrew Boie d1b3d54a94 sanitycheck: treat warnings as errors
This policy was in place for the old sanity checks, was simply
overlooked.

Change-Id: Id021a669b35622f9e9cba23b4417c5d0b7afc15c
Signed-off-by: Andrew Boie <andrew.p.boie@intel.com>
2016-02-05 20:14:45 -05:00
Andrew Boie ae9e7f7bca sanitycheck: default to cleaning outdir
-C/--clean is now -n/--no-clean. Made this change since the build system
can't always do incremental rebuild correctly depending on what was
changed (such as Make variables).

Change-Id: I199e505693479184b2f197ddb584ade5eca1704a
Signed-off-by: Andrew Boie <andrew.p.boie@intel.com>
2016-02-05 20:14:45 -05:00
Andrew Boie 4b18247749 sanitycheck: print elapsed time
Change-Id: I5e18edaf9074f0166582fc51da81bb8978c9038b
Signed-off-by: Andrew Boie <andrew.p.boie@intel.com>
2016-02-05 20:14:44 -05:00
Andrew Boie 6acbe63aea initial commit for next-gen sanity checks
The online help ./scripts/sanitycheck --help describes usage.
Most users will simply want to run with no arguments.

Change-Id: Icedbbfc22599a64a6e3dbbb808ff3276db06f2e0
Signed-off-by: Andrew Boie <andrew.p.boie@intel.com>
2016-02-05 20:14:42 -05:00