Create support for muxed UARTs which are attached to a real
UART and which use GSM 07.10 muxing protocol to create virtual
channels that can be run on top of the real UART.
Signed-off-by: Jukka Rissanen <jukka.rissanen@linux.intel.com>
When retrying failed tests, make sure we keep old results and only
update those tests that were retried.
Also, remove duplication of code for creating the reports and make the
report function a bit more generic.
sanitycheck.xml is now listing or tests, not only the test application.
Signed-off-by: Anas Nashif <anas.nashif@intel.com>
The "net stacks" shell command support was just removed, but
the net_stacks linker section was left around.
Signed-off-by: Jukka Rissanen <jukka.rissanen@linux.intel.com>
We are not propagting those options all the way to the device handler
now, fix this by using the suite member in the handler which has those
already.
Signed-off-by: Anas Nashif <anas.nashif@intel.com>
Do not declare VERBOSE as global, instead pass verbosity as argument.
Also get rid of options as global and fix coverage class to not use
global option variable.
Signed-off-by: Anas Nashif <anas.nashif@intel.com>
We do not use those arg files anymore, so remove them to avoid
confusion and for the sake of cleanup.
Signed-off-by: Anas Nashif <anas.nashif@intel.com>
We rename the nRF51 Dev Kit board target (nrf51_pca10028)
to nrf51dk_nrf51422. We update all associated references
in the supportive documentation and all nRF51-related
cofigurations and overlay files in the samples and tests
in the tree.
Signed-off-by: Ioannis Glaropoulos <Ioannis.Glaropoulos@nordicsemi.no>
We were parsing random FAIL messages from the output of test runs ad
testcases and capturing them in the xml output. Now we only parse the
name if it starts with test_.
Fixes#21162
Signed-off-by: Anas Nashif <anas.nashif@intel.com>
Add the following functions to allow filtering based on device tree
dt_compat_enabled(compat) - Returns true if a device tree node
compatible matches 'compat' and the node is enabled.
dt_alias_exists(alias) - Returns true if a device tree node exists with
'alias' and the node is enabled.
dt_compat_enabled_with_alias - Returns true if a device tree node
compatible matches 'compat' and the node has 'alias' and the node is
enabled.
Signed-off-by: Kumar Gala <kumar.gala@linaro.org>
A complete overhaul of the sanitycheck script and how we build and run
tests. This new version of sanitycheck uses python for job distribution
and drop use of Make.
In addition to the move to python threading library, the following has
been changed:
- All handlers now run in parallel, meaning that any simulator will run
in parallel and when testing on multiple devices (using
--device-testing) the tests are run in parallel.
- Lexicial filtering (using the filter keyword in yaml files) is now
evaluated at runtime and is no long being pre-processed. This will allow
us to immediately start executing tests and skip the wait time that was
needed for filtering.
- Device testing now supports multiple devices connected at the same
time and is managed using a hardware map that needs to be generated and
maintained for every test environment. (using --generate-hardware-map
option).
- Reports are not long stored in the Zephyr tree and instead stored in
the output directory where all build artifacts are generated.
- Each tested target now has a junit report in the output directory.
- Recording option for performance data and other metrics is now
available. This will allow us to record the output from the console and
store the data for later processing. For example benchmark data can be
captured and uploaded to a tracking server.
- Test configurations (or instances) are no longer being sorted, this
will help with balancing the load when we run sanitycheck on multiple
hosts (as we do in CI).
And many other cleanups and improvements...
Signed-off-by: Anas Nashif <anas.nashif@intel.com>
Promote a handy and often-overlooked sys.exit() feature: Passing it a
string (or any other non-int object) prints it to stderr and exits with
status 1.
See the documentation at
https://docs.python.org/3/library/sys.html#sys.exit.
This indirectly prints some errors to stderr that previously went to
stdout.
Signed-off-by: Ulf Magnusson <Ulf.Magnusson@nordicsemi.no>
Fixes these pylint warnings:
scripts/sanity_chk/ini2yaml.py:25:22: R1719: The if expression can
be replaced with 'test' (simplifiable-if-expression)
scripts/sanity_chk/expr_parser.py:208:15: R1719: The if expression
can be replaced with 'bool(test)' (simplifiable-if-expression)
scripts/sanity_chk/expr_parser.py:210:15: R1719: The if expression
can be replaced with 'bool(test)' (simplifiable-if-expression)
Also replace a redundant re.compile().match() with re.match(). compile()
doesn't help when re-compiling the regular expression each time through.
(compile() often doesn't help much in general, because the 're' module
caches compiled regexes.)
Signed-off-by: Ulf Magnusson <Ulf.Magnusson@nordicsemi.no>
Reported by pylint's 'bad-whitespace' warning.
Not gonna enable this warning in the CI check, because it flags stuff
like deliberately aligning assignments and gets too cultish. Just a
cleanup pass.
For whatever reason, the common convention in Python is to skip spaces
around '=' when passing keyword arguments and giving default arguments:
f(x=3, y=4)
def f(x, y=8):
...
Signed-off-by: Ulf Magnusson <Ulf.Magnusson@nordicsemi.no>
This board and SoC was discontinued some time ago and is currently not
maintained in the zephyr tree.
Remove all associated configurations and variants from the tree.
Signed-off-by: Anas Nashif <anas.nashif@intel.com>
Any fatal error will print "ZEPHYR FATAL ERROR" now, so
we don't have to maintain a set of strings in the
sanitycheck harness.py
Signed-off-by: Andrew Boie <andrew.p.boie@intel.com>
The way sanitycheck did its ordered regexes is that it would test
every regex against every line, and store the matching lines and their
regexes in an OrderedDict and check that they happened in the right
order.
That's wrong, because it disallows matching against a line that
previously appeared (and should have been ignored) in the input
stream. The watchdog sample is the best illustration: the first boot
will (by definition) contain all the output already, but the regex has
to match against a line from the SECOND boot and not the same one it
saw earlier.
Do this the simple way: keep a counter of which regex we're trying to
apply next and increment it on a match. This is faster too as we only
need to check one pattern per line.
Signed-off-by: Andy Ross <andrew.j.ross@intel.com>
The Harness handlers for tests were parsing the realtime stream out of
qemu pipes by recompiling and executing every regex for every line (!)
of output from the simulator. That's a significant CPU load, and it's
(1) in a separate thread not tracked by the JOBS limit and (2)
happening at the worst possible time and contending with the qemu
process for host CPU cycles that it needs to hit its (real world)
timer targets on time.
Compile them just once, please.
Signed-off-by: Andy Ross <andrew.j.ross@intel.com>
Update the files which contain no license information with the
'Apache-2.0' SPDX license identifier. Many source files in the tree are
missing licensing information, which makes it harder for compliance
tools to determine the correct license.
By default all files without license information are under the default
license of Zephyr, which is Apache version 2.
Signed-off-by: Anas Nashif <anas.nashif@intel.com>
We have not been counting samples in reports. This change lists tests
associated with sample code which in many cases is just verifying output
from the sample and counts as 1 test.
Signed-off-by: Anas Nashif <anas.nashif@intel.com>
Discovered with pylint3.
Use the placeholder name '_' for unproblematic unused variables. It's
what I'm used to, and pylint knows not to flag it.
Python tip:
for i in range(n):
some_list.append(0)
can be replaced with
some_list += n*[0]
Similarly, 3*'\t' gives '\t\t\t'.
(Relevant here because pylint flagged the loop index as unused.)
To do integer division in Python 3, use // instead of /.
Signed-off-by: Ulf Magnusson <Ulf.Magnusson@nordicsemi.no>
Make sure we capture data from gcov and do not timeout before all the
data has been captured. Also report on incomplete data capture.
Signed-off-by: Anas Nashif <anas.nashif@intel.com>
Some boards depend on environment variables so we want to make sure we
do not attempt to build boards requiring additional setup.
Add the section below into the board YAML file, sanitycheck will check
the environment and will only run tests on that board if the variables
are defined.
env:
- VAR1
- VAR2
Signed-off-by: Anas Nashif <anas.nashif@intel.com>
Add a new option as fixture in harness configurations for utilizing
sanitycheck to identify test cases that require external hardware
such as sensor, ble, networking for validation. The config will be
added to yaml files with unique fixture name to identify each hardware
and allow automation to trigger test execution on setup having the
specific fixture enabled. Also, remove the default required for type and
regex configs that is not essential in case of ztest based test cases.
Signed-off-by: Praful Swarnakar <praful.swarnakar@intel.com>
We have been dropping lines after finding a fault which resulted in
missing information in the log. Make sure we continue and only report
failure at the end of the execution.
Signed-off-by: Anas Nashif <anas.nashif@intel.com>
Do not close console after PASS is reported, wait a bit for any
remaining messages from the tests, sometimes we have faults that need to
be parsed.
This now works for Qemu handler, support for other handlers to follow.
Fixes#9646
Signed-off-by: Anas Nashif <anas.nashif@intel.com>
Since a distributions name for the ply package may change, the only
constant name which can be used for a recommendation is the name on
PyPi which happens to be "ply".
Signed-off-by: Daniel Egger <daniel@eggers-club.de>
Add new kwyboard to board definition to allow blacklisting boards. This
is needed when a board is broken causing CI to fail without a fix in
sight.
Add:
sanitycheck: false
to the board yaml file to disable the board. By default, the value is
set to true.
Signed-off-by: Anas Nashif <anas.nashif@intel.com>
Fail in tests where we have an OOPS or a panic. Right now and in many
cases we continue and test case might be reported as PASS.
Cases that have the tag ignore_faults will ignore those faults (cases
that are testing faults for example).
Signed-off-by: Anas Nashif <anas.nashif@intel.com>
- Some tests start with test_, some do not, so make sure we parse both.
- Parse skipped tests
- Improve handling of test case identifier
- Handle Exceptions in device handler
Signed-off-by: Anas Nashif <anas.nashif@intel.com>
Parse the test results and create a test report with more granular
results that can be imported to into test management/reporting system.
Signed-off-by: Anas Nashif <anas.nashif@intel.com>
Allow for any toolchain variants to be useable with sanitycheck without
the need for them to be registered.
Signed-off-by: Anas Nashif <anas.nashif@intel.com>
Add 2 classes, one to handle the current TestCase scenario, and one more
for handling generic Console with regex matching.
Signed-off-by: Anas Nashif <anas.nashif@intel.com>
Some boards are supported natively by qemu. This option will allow us to
run tests using those platforms directly without having to go via a
dedicated qemu board definition.
Signed-off-by: Anas Nashif <anas.nashif@intel.com>
This keyword would mean that a special harness is needed to run the
tests sucessfully. This can be as simple as a loopback wiring or a
complete hardware test setup for sensor and IO testing. It is free form
initially and would be changed to be an enum once we have more values in
place.
Signed-off-by: Anas Nashif <anas.nashif@intel.com>
when running multiple instances of sanitycheck, allow placing the
parsetab.py in a customer location that can be set using an environment
variable.
export PARSETAB_DIR=/tmp/
run sanitycheck and the parsetab.py will be placed in /tmp/.
Fixes#4513
Signed-off-by: Anas Nashif <anas.nashif@intel.com>
Simplify parsing of yaml structures and remove usage of cp which was for
the ConfigParser used for ini files.
Signed-off-by: Anas Nashif <anas.nashif@intel.com>
Support new keywords in testcase.yaml that would allow us to inject
configuration options to be merged with default configuration instead of
having to provide a prj.conf for each variant of the test which is very
difficult to keep in sync. Sanitycheck script will create an overlay
file that is merged during the build process.
This is now done using the extra_configs option which is a yaml list of
option with the values, for example:
extra_configs:
- CONFIG_XXXX=y
- CONFIG_YYYY=y
With this option we can have multiple tests that for example run on
hardware with different values. This type of testing is good on HW but
it does not make sense to be built in normal sanitycheck operation
because it will be just rebuilding the same code with different values.
Signed-off-by: Anas Nashif <anas.nashif@intel.com>
This commit changes the syntax of the testcase files and changes the
behaviour and configuration of the sanitycheck script.
To avoid having multiple files with different syntax for boards,
samples, tests; this change unifies the syntax and uses YAML instead of
INI.
We maintain the current keywords used in the old syntax and maintain the
flexibility of adding tests with different configuration by using YAML
list configuration. On top of that, the following features are added:
- We now scan for board configurations in the boards directory and look
for a YAML file describing a board and how it should be tested. This
eliminates the need for listing boards per architecture in a special ini
file under scripts/.
- We define hardware information charachterstics in the board YAML file
that helps identifying if a certain test should run on that board or
not. For example, we can specify the available RAM in the board and
filter tests that would require more RAM than the board can handle.
- Boards can be set as default for testing meaning that we always run a
test case (build and run of possible) when sanitycheck is called without
any arguments. Previously this was done only by selecting the first
board defined for a specific architecture.
- Tests can be configured to run on all possible boards, this is to make
sure we always build some basic tests for all boards to catch issues
with the core kernel features.
Signed-off-by: Anas Nashif <anas.nashif@intel.com>
- board name olimex_stm32_e407
- CPU STM32F407ZGT6 Cortex M4
- LED/BUTTON support
- Console on USART1 with 8n1 115200 baud
Signed-off-by: Erwin Rol <erwin@erwinrol.com>
Add necessary board files, pinmux and device tree in order to have a
usable debug console.
Origin: Original
Change-Id: I43a9d278c3f2c936a714263626722f630367b663
Signed-off-by: Florian Vaussard <florian.vaussard@heig-vd.ch>
The stm32f4discovery was incorrectly named and should have been
stm32f4_disco. Also added 96b_carbon_nrf51 that was missing from the
list.
Signed-off-by: Kumar Gala <kumar.gala@linaro.org>
Add configuration, pinmux, dts and documentation for the STM32L496G
Discovery board based on the STM32L496AG SoC.
Signed-off-by: Neil Armstrong <narmstrong@baylibre.com>
Now that we can specify what toolchain is intended for each
SOC, enable some more SOCs to be built.
A full sanitycheck run will require the installation of both
RF-2016.4 and RG-2016.4 releases.
Signed-off-by: Andrew Boie <andrew.p.boie@intel.com>
Add necessary board files, pinmux and device tree in order to have a
usable debug console.
Origin: Original
Change-Id: I280320700352fd36a544c03f4e57d2eeec2449e5
Signed-off-by: Florian Vaussard <florian.vaussard@heig-vd.ch>
This core configuration was removed from the tree since it cannot
implement irq_offload().
Remove an orphaned block in xtesna.ini.
Signed-off-by: Andrew Boie <andrew.p.boie@intel.com>
Add configuration, dts and documentation for the Nucleo L432KC board
based on the STM32L432KC SoC.
Signed-off-by: Neil Armstrong <narmstrong@baylibre.com>
This commit provides support for disco_l475_iot1 board
Pinmux driver is provided with initial support definitions
Change-Id: I17b637a8ba0b033014969eca8fffe76319c47c52
Signed-off-by: Erwan Gouriou <erwan.gouriou@linaro.org>
In order to allow the use of such board, a very preliminar port was
developed. It consists of board files, as well as pinmux, uart, gpio,
spi drivers and device tree files.
Change-Id: I5753064e39e0b023cf4481744c176de26d8dbebb
Signed-off-by: Gustavo Denardin <gustavo.denardin@gmail.com>
Signed-off-by: Kumar Gala <kumar.gala@linaro.org>
CC3220SF_LAUNCHXL effectively replaces the CC3200_LAUNCHXL,
with support for the CC3220SF SoC, which is an update for
the CC3200 SoC.
This is supported by the Texas Instruments CC3220 SDK.
Jira: ZEP-1958
Change-Id: I2484d3ee87b7f909c783597d95128f2b45db36f2
Signed-off-by: Gil Pitney <gil.pitney@linaro.org>
Adds initial support and documentation for the frdm_kw41z board.
- Configures the kw41 to use the 32 MHz external oscillator on the board
to generate a 40 MHz system clock. The clock settings match the MCUX
SDK hello_world example project.
- Provides pinmux settings for the uart, i2c, LEDs, and switches
- Enables pinmux, gpio, uart, and i2c driver instances
- Configures the fxos8700 accelerometer/magnetometer driver
Jira: ZEP-1390
Change-Id: I025a0eae3d380eaf90b02683acf5c592e2204a2e
Signed-off-by: Maureen Helm <maureen.helm@nxp.com>
