A complete overhaul of the sanitycheck script and how we build and run
tests. This new version of sanitycheck uses python for job distribution
and drop use of Make.
In addition to the move to python threading library, the following has
been changed:
- All handlers now run in parallel, meaning that any simulator will run
in parallel and when testing on multiple devices (using
--device-testing) the tests are run in parallel.
- Lexicial filtering (using the filter keyword in yaml files) is now
evaluated at runtime and is no long being pre-processed. This will allow
us to immediately start executing tests and skip the wait time that was
needed for filtering.
- Device testing now supports multiple devices connected at the same
time and is managed using a hardware map that needs to be generated and
maintained for every test environment. (using --generate-hardware-map
option).
- Reports are not long stored in the Zephyr tree and instead stored in
the output directory where all build artifacts are generated.
- Each tested target now has a junit report in the output directory.
- Recording option for performance data and other metrics is now
available. This will allow us to record the output from the console and
store the data for later processing. For example benchmark data can be
captured and uploaded to a tracking server.
- Test configurations (or instances) are no longer being sorted, this
will help with balancing the load when we run sanitycheck on multiple
hosts (as we do in CI).
And many other cleanups and improvements...
Signed-off-by: Anas Nashif <anas.nashif@intel.com>
Promote a handy and often-overlooked sys.exit() feature: Passing it a
string (or any other non-int object) prints it to stderr and exits with
status 1.
See the documentation at
https://docs.python.org/3/library/sys.html#sys.exit.
This indirectly prints some errors to stderr that previously went to
stdout.
Signed-off-by: Ulf Magnusson <Ulf.Magnusson@nordicsemi.no>
Fixes these pylint warnings:
scripts/sanity_chk/ini2yaml.py:25:22: R1719: The if expression can
be replaced with 'test' (simplifiable-if-expression)
scripts/sanity_chk/expr_parser.py:208:15: R1719: The if expression
can be replaced with 'bool(test)' (simplifiable-if-expression)
scripts/sanity_chk/expr_parser.py:210:15: R1719: The if expression
can be replaced with 'bool(test)' (simplifiable-if-expression)
Also replace a redundant re.compile().match() with re.match(). compile()
doesn't help when re-compiling the regular expression each time through.
(compile() often doesn't help much in general, because the 're' module
caches compiled regexes.)
Signed-off-by: Ulf Magnusson <Ulf.Magnusson@nordicsemi.no>
Reported by pylint's 'bad-whitespace' warning.
Not gonna enable this warning in the CI check, because it flags stuff
like deliberately aligning assignments and gets too cultish. Just a
cleanup pass.
For whatever reason, the common convention in Python is to skip spaces
around '=' when passing keyword arguments and giving default arguments:
f(x=3, y=4)
def f(x, y=8):
...
Signed-off-by: Ulf Magnusson <Ulf.Magnusson@nordicsemi.no>
This board and SoC was discontinued some time ago and is currently not
maintained in the zephyr tree.
Remove all associated configurations and variants from the tree.
Signed-off-by: Anas Nashif <anas.nashif@intel.com>
Any fatal error will print "ZEPHYR FATAL ERROR" now, so
we don't have to maintain a set of strings in the
sanitycheck harness.py
Signed-off-by: Andrew Boie <andrew.p.boie@intel.com>
The way sanitycheck did its ordered regexes is that it would test
every regex against every line, and store the matching lines and their
regexes in an OrderedDict and check that they happened in the right
order.
That's wrong, because it disallows matching against a line that
previously appeared (and should have been ignored) in the input
stream. The watchdog sample is the best illustration: the first boot
will (by definition) contain all the output already, but the regex has
to match against a line from the SECOND boot and not the same one it
saw earlier.
Do this the simple way: keep a counter of which regex we're trying to
apply next and increment it on a match. This is faster too as we only
need to check one pattern per line.
Signed-off-by: Andy Ross <andrew.j.ross@intel.com>
The Harness handlers for tests were parsing the realtime stream out of
qemu pipes by recompiling and executing every regex for every line (!)
of output from the simulator. That's a significant CPU load, and it's
(1) in a separate thread not tracked by the JOBS limit and (2)
happening at the worst possible time and contending with the qemu
process for host CPU cycles that it needs to hit its (real world)
timer targets on time.
Compile them just once, please.
Signed-off-by: Andy Ross <andrew.j.ross@intel.com>
Update the files which contain no license information with the
'Apache-2.0' SPDX license identifier. Many source files in the tree are
missing licensing information, which makes it harder for compliance
tools to determine the correct license.
By default all files without license information are under the default
license of Zephyr, which is Apache version 2.
Signed-off-by: Anas Nashif <anas.nashif@intel.com>
We have not been counting samples in reports. This change lists tests
associated with sample code which in many cases is just verifying output
from the sample and counts as 1 test.
Signed-off-by: Anas Nashif <anas.nashif@intel.com>
Discovered with pylint3.
Use the placeholder name '_' for unproblematic unused variables. It's
what I'm used to, and pylint knows not to flag it.
Python tip:
for i in range(n):
some_list.append(0)
can be replaced with
some_list += n*[0]
Similarly, 3*'\t' gives '\t\t\t'.
(Relevant here because pylint flagged the loop index as unused.)
To do integer division in Python 3, use // instead of /.
Signed-off-by: Ulf Magnusson <Ulf.Magnusson@nordicsemi.no>
Make sure we capture data from gcov and do not timeout before all the
data has been captured. Also report on incomplete data capture.
Signed-off-by: Anas Nashif <anas.nashif@intel.com>
Some boards depend on environment variables so we want to make sure we
do not attempt to build boards requiring additional setup.
Add the section below into the board YAML file, sanitycheck will check
the environment and will only run tests on that board if the variables
are defined.
env:
- VAR1
- VAR2
Signed-off-by: Anas Nashif <anas.nashif@intel.com>
Add a new option as fixture in harness configurations for utilizing
sanitycheck to identify test cases that require external hardware
such as sensor, ble, networking for validation. The config will be
added to yaml files with unique fixture name to identify each hardware
and allow automation to trigger test execution on setup having the
specific fixture enabled. Also, remove the default required for type and
regex configs that is not essential in case of ztest based test cases.
Signed-off-by: Praful Swarnakar <praful.swarnakar@intel.com>
We have been dropping lines after finding a fault which resulted in
missing information in the log. Make sure we continue and only report
failure at the end of the execution.
Signed-off-by: Anas Nashif <anas.nashif@intel.com>
Do not close console after PASS is reported, wait a bit for any
remaining messages from the tests, sometimes we have faults that need to
be parsed.
This now works for Qemu handler, support for other handlers to follow.
Fixes#9646
Signed-off-by: Anas Nashif <anas.nashif@intel.com>
Since a distributions name for the ply package may change, the only
constant name which can be used for a recommendation is the name on
PyPi which happens to be "ply".
Signed-off-by: Daniel Egger <daniel@eggers-club.de>
Add new kwyboard to board definition to allow blacklisting boards. This
is needed when a board is broken causing CI to fail without a fix in
sight.
Add:
sanitycheck: false
to the board yaml file to disable the board. By default, the value is
set to true.
Signed-off-by: Anas Nashif <anas.nashif@intel.com>
Fail in tests where we have an OOPS or a panic. Right now and in many
cases we continue and test case might be reported as PASS.
Cases that have the tag ignore_faults will ignore those faults (cases
that are testing faults for example).
Signed-off-by: Anas Nashif <anas.nashif@intel.com>
- Some tests start with test_, some do not, so make sure we parse both.
- Parse skipped tests
- Improve handling of test case identifier
- Handle Exceptions in device handler
Signed-off-by: Anas Nashif <anas.nashif@intel.com>
Parse the test results and create a test report with more granular
results that can be imported to into test management/reporting system.
Signed-off-by: Anas Nashif <anas.nashif@intel.com>
Allow for any toolchain variants to be useable with sanitycheck without
the need for them to be registered.
Signed-off-by: Anas Nashif <anas.nashif@intel.com>
Add 2 classes, one to handle the current TestCase scenario, and one more
for handling generic Console with regex matching.
Signed-off-by: Anas Nashif <anas.nashif@intel.com>
Some boards are supported natively by qemu. This option will allow us to
run tests using those platforms directly without having to go via a
dedicated qemu board definition.
Signed-off-by: Anas Nashif <anas.nashif@intel.com>
This keyword would mean that a special harness is needed to run the
tests sucessfully. This can be as simple as a loopback wiring or a
complete hardware test setup for sensor and IO testing. It is free form
initially and would be changed to be an enum once we have more values in
place.
Signed-off-by: Anas Nashif <anas.nashif@intel.com>
when running multiple instances of sanitycheck, allow placing the
parsetab.py in a customer location that can be set using an environment
variable.
export PARSETAB_DIR=/tmp/
run sanitycheck and the parsetab.py will be placed in /tmp/.
Fixes#4513
Signed-off-by: Anas Nashif <anas.nashif@intel.com>
Simplify parsing of yaml structures and remove usage of cp which was for
the ConfigParser used for ini files.
Signed-off-by: Anas Nashif <anas.nashif@intel.com>
Support new keywords in testcase.yaml that would allow us to inject
configuration options to be merged with default configuration instead of
having to provide a prj.conf for each variant of the test which is very
difficult to keep in sync. Sanitycheck script will create an overlay
file that is merged during the build process.
This is now done using the extra_configs option which is a yaml list of
option with the values, for example:
extra_configs:
- CONFIG_XXXX=y
- CONFIG_YYYY=y
With this option we can have multiple tests that for example run on
hardware with different values. This type of testing is good on HW but
it does not make sense to be built in normal sanitycheck operation
because it will be just rebuilding the same code with different values.
Signed-off-by: Anas Nashif <anas.nashif@intel.com>
This commit changes the syntax of the testcase files and changes the
behaviour and configuration of the sanitycheck script.
To avoid having multiple files with different syntax for boards,
samples, tests; this change unifies the syntax and uses YAML instead of
INI.
We maintain the current keywords used in the old syntax and maintain the
flexibility of adding tests with different configuration by using YAML
list configuration. On top of that, the following features are added:
- We now scan for board configurations in the boards directory and look
for a YAML file describing a board and how it should be tested. This
eliminates the need for listing boards per architecture in a special ini
file under scripts/.
- We define hardware information charachterstics in the board YAML file
that helps identifying if a certain test should run on that board or
not. For example, we can specify the available RAM in the board and
filter tests that would require more RAM than the board can handle.
- Boards can be set as default for testing meaning that we always run a
test case (build and run of possible) when sanitycheck is called without
any arguments. Previously this was done only by selecting the first
board defined for a specific architecture.
- Tests can be configured to run on all possible boards, this is to make
sure we always build some basic tests for all boards to catch issues
with the core kernel features.
Signed-off-by: Anas Nashif <anas.nashif@intel.com>
- board name olimex_stm32_e407
- CPU STM32F407ZGT6 Cortex M4
- LED/BUTTON support
- Console on USART1 with 8n1 115200 baud
Signed-off-by: Erwin Rol <erwin@erwinrol.com>
Add necessary board files, pinmux and device tree in order to have a
usable debug console.
Origin: Original
Change-Id: I43a9d278c3f2c936a714263626722f630367b663
Signed-off-by: Florian Vaussard <florian.vaussard@heig-vd.ch>
The stm32f4discovery was incorrectly named and should have been
stm32f4_disco. Also added 96b_carbon_nrf51 that was missing from the
list.
Signed-off-by: Kumar Gala <kumar.gala@linaro.org>
Add configuration, pinmux, dts and documentation for the STM32L496G
Discovery board based on the STM32L496AG SoC.
Signed-off-by: Neil Armstrong <narmstrong@baylibre.com>
Now that we can specify what toolchain is intended for each
SOC, enable some more SOCs to be built.
A full sanitycheck run will require the installation of both
RF-2016.4 and RG-2016.4 releases.
Signed-off-by: Andrew Boie <andrew.p.boie@intel.com>
Add necessary board files, pinmux and device tree in order to have a
usable debug console.
Origin: Original
Change-Id: I280320700352fd36a544c03f4e57d2eeec2449e5
Signed-off-by: Florian Vaussard <florian.vaussard@heig-vd.ch>
This core configuration was removed from the tree since it cannot
implement irq_offload().
Remove an orphaned block in xtesna.ini.
Signed-off-by: Andrew Boie <andrew.p.boie@intel.com>
Add configuration, dts and documentation for the Nucleo L432KC board
based on the STM32L432KC SoC.
Signed-off-by: Neil Armstrong <narmstrong@baylibre.com>
This commit provides support for disco_l475_iot1 board
Pinmux driver is provided with initial support definitions
Change-Id: I17b637a8ba0b033014969eca8fffe76319c47c52
Signed-off-by: Erwan Gouriou <erwan.gouriou@linaro.org>
In order to allow the use of such board, a very preliminar port was
developed. It consists of board files, as well as pinmux, uart, gpio,
spi drivers and device tree files.
Change-Id: I5753064e39e0b023cf4481744c176de26d8dbebb
Signed-off-by: Gustavo Denardin <gustavo.denardin@gmail.com>
Signed-off-by: Kumar Gala <kumar.gala@linaro.org>
CC3220SF_LAUNCHXL effectively replaces the CC3200_LAUNCHXL,
with support for the CC3220SF SoC, which is an update for
the CC3200 SoC.
This is supported by the Texas Instruments CC3220 SDK.
Jira: ZEP-1958
Change-Id: I2484d3ee87b7f909c783597d95128f2b45db36f2
Signed-off-by: Gil Pitney <gil.pitney@linaro.org>
Adds initial support and documentation for the frdm_kw41z board.
- Configures the kw41 to use the 32 MHz external oscillator on the board
to generate a 40 MHz system clock. The clock settings match the MCUX
SDK hello_world example project.
- Provides pinmux settings for the uart, i2c, LEDs, and switches
- Enables pinmux, gpio, uart, and i2c driver instances
- Configures the fxos8700 accelerometer/magnetometer driver
Jira: ZEP-1390
Change-Id: I025a0eae3d380eaf90b02683acf5c592e2204a2e
Signed-off-by: Maureen Helm <maureen.helm@nxp.com>
This reverts commit 9fb0c2af5c.
SDK 0.9 has now been released.
Change-Id: I676b6f0e31ab48fde3dda41b681abf53964ea9f9
Signed-off-by: Andrew Boie <andrew.p.boie@intel.com>
The 0.9 SDK is not released yet!!
This reverts commit 7995e6207c.
Change-Id: I550d1aa27ba7a06ff2cda09496ceb92645a17460
Signed-off-by: Andrew Boie <andrew.p.boie@intel.com>
ARM's Cortex-M Prototyping System (MPS2) is a board with an FPGA that
can be programmed with different 'SoCs'. To use these in Zephyr we need
a set of board files for each variant.
This adds a board for a variant which implements a Cortex-M3 CPU; the
naming of this matches that used for the Zephyr SoC (which is itself
based on ARM's documentation nomenclature).
Change-Id: Ie02a67a03016b8aeee31e3694f0edbcc37f9ee64
Signed-off-by: Jon Medhurst <tixy@linaro.org>
Signed-off-by: Vincenzo Frascino <vincenzo.frascino@linaro.org>
Added sanity args to be used by the compare footprint.
Please notice that compare footprint use sanitycheck to
build footprint apps for base commit and for current commit.
This in order to collect the information and generate the diff.
Change-Id: I6f86bbfa020999b3f26e93a608fc7b34ee508e10
Signed-off-by: Javier B Perez <javier.b.perez.hernandez@intel.com>
With the appearance of the nRF52840 IC a new Preview Development Kit
(PDK) board has been introduced. This patch adds basic support for this
new board.
JIRA: ZEP-1418
Change-Id: If5845e75312ec756b968e595e5dc31c4c9624be2
Signed-off-by: Vinayak Chettimada <vinayak.kariappa.chettimada@nordicsemi.no>
Signed-off-by: Carles Cufi <carles.cufi@nordicsemi.no>
Add board support for the Nucleo64 L476RG development board.
Change-Id: Ibb5424bc936c67a5d96855617202136d7dea772c
Signed-off-by: Fabien Parent <fparent@baylibre.com>
Signed-off-by: Kumar Gala <kumar.gala@linaro.org>
Signed-off-by: Erwan Gouriou <erwan.gouriou@linaro.org>
Update file to catch any regressions and major changes during
stabilisation period.
Change-Id: Id30dd1827034b96c5478c78f9c388384f51bcbec
Signed-off-by: Anas Nashif <anas.nashif@intel.com>
Add board support for ARM V2M Beetle platform.
ARM V2M Beetle board is build around the ARM Beetle Cortex-M3
based processor.
The support has been tested in nanokernel mode with the bringup
application that will be pushed with a future patch.
Jira: ZEP-1245
Change-Id: Ib05a40c072f10149e692283177387cf2cfe32f66
Signed-off-by: Vincenzo Frascino <vincenzo.frascino@linaro.org>
Signed-off-by: Kumar Gala <kumar.gala@linaro.org>
The Quark SE C1000 BLE Core is a nRF51822-QFAA, with 16kB of RAM and
256kB of flash. The configuration is otherwise similar to the Arduino
101 BLE, except that the UART RTS pin is the same as that used by
nrf51_pca10028.
Change-Id: I88cb18876bdde65abcf9a499894f70802046c824
Signed-off-by: Johan Hedberg <johan.hedberg@intel.com>
Added Kconfig and makefiles to be able to build a Zephyr application
on Linux/gcc, and load via OpenOCD.
Validated by running the hello world, and philosophers microkernel
samples, and stepping through the code in gdb.
Jira: ZEP-1109
Change-Id: If5d3e7b1a8ecf5ecf6a00f147742b3bc5716190f
Signed-off-by: Gil Pitney <gil.pitney@linaro.org>
Now that these platforms pass sanitycheck add them into the list of
platforms we support
Change-Id: If559c80e107505e6b98f81c0e94b9862618b1735
Signed-off-by: Kumar Gala <kumar.gala@linaro.org>
