Cleanup fixture processing and allow ztest testcases to support
harness_config with fixture definition.
Signed-off-by: Anas Nashif <anas.nashif@intel.com>
We were parsing random FAIL messages from the output of test runs ad
testcases and capturing them in the xml output. Now we only parse the
name if it starts with test_.
Fixes#21162
Signed-off-by: Anas Nashif <anas.nashif@intel.com>
A complete overhaul of the sanitycheck script and how we build and run
tests. This new version of sanitycheck uses python for job distribution
and drop use of Make.
In addition to the move to python threading library, the following has
been changed:
- All handlers now run in parallel, meaning that any simulator will run
in parallel and when testing on multiple devices (using
--device-testing) the tests are run in parallel.
- Lexicial filtering (using the filter keyword in yaml files) is now
evaluated at runtime and is no long being pre-processed. This will allow
us to immediately start executing tests and skip the wait time that was
needed for filtering.
- Device testing now supports multiple devices connected at the same
time and is managed using a hardware map that needs to be generated and
maintained for every test environment. (using --generate-hardware-map
option).
- Reports are not long stored in the Zephyr tree and instead stored in
the output directory where all build artifacts are generated.
- Each tested target now has a junit report in the output directory.
- Recording option for performance data and other metrics is now
available. This will allow us to record the output from the console and
store the data for later processing. For example benchmark data can be
captured and uploaded to a tracking server.
- Test configurations (or instances) are no longer being sorted, this
will help with balancing the load when we run sanitycheck on multiple
hosts (as we do in CI).
And many other cleanups and improvements...
Signed-off-by: Anas Nashif <anas.nashif@intel.com>
Reported by pylint's 'bad-whitespace' warning.
Not gonna enable this warning in the CI check, because it flags stuff
like deliberately aligning assignments and gets too cultish. Just a
cleanup pass.
For whatever reason, the common convention in Python is to skip spaces
around '=' when passing keyword arguments and giving default arguments:
f(x=3, y=4)
def f(x, y=8):
...
Signed-off-by: Ulf Magnusson <Ulf.Magnusson@nordicsemi.no>
Any fatal error will print "ZEPHYR FATAL ERROR" now, so
we don't have to maintain a set of strings in the
sanitycheck harness.py
Signed-off-by: Andrew Boie <andrew.p.boie@intel.com>
The way sanitycheck did its ordered regexes is that it would test
every regex against every line, and store the matching lines and their
regexes in an OrderedDict and check that they happened in the right
order.
That's wrong, because it disallows matching against a line that
previously appeared (and should have been ignored) in the input
stream. The watchdog sample is the best illustration: the first boot
will (by definition) contain all the output already, but the regex has
to match against a line from the SECOND boot and not the same one it
saw earlier.
Do this the simple way: keep a counter of which regex we're trying to
apply next and increment it on a match. This is faster too as we only
need to check one pattern per line.
Signed-off-by: Andy Ross <andrew.j.ross@intel.com>
The Harness handlers for tests were parsing the realtime stream out of
qemu pipes by recompiling and executing every regex for every line (!)
of output from the simulator. That's a significant CPU load, and it's
(1) in a separate thread not tracked by the JOBS limit and (2)
happening at the worst possible time and contending with the qemu
process for host CPU cycles that it needs to hit its (real world)
timer targets on time.
Compile them just once, please.
Signed-off-by: Andy Ross <andrew.j.ross@intel.com>
Update the files which contain no license information with the
'Apache-2.0' SPDX license identifier. Many source files in the tree are
missing licensing information, which makes it harder for compliance
tools to determine the correct license.
By default all files without license information are under the default
license of Zephyr, which is Apache version 2.
Signed-off-by: Anas Nashif <anas.nashif@intel.com>
We have not been counting samples in reports. This change lists tests
associated with sample code which in many cases is just verifying output
from the sample and counts as 1 test.
Signed-off-by: Anas Nashif <anas.nashif@intel.com>
Discovered with pylint3.
Use the placeholder name '_' for unproblematic unused variables. It's
what I'm used to, and pylint knows not to flag it.
Python tip:
for i in range(n):
some_list.append(0)
can be replaced with
some_list += n*[0]
Similarly, 3*'\t' gives '\t\t\t'.
(Relevant here because pylint flagged the loop index as unused.)
To do integer division in Python 3, use // instead of /.
Signed-off-by: Ulf Magnusson <Ulf.Magnusson@nordicsemi.no>
Make sure we capture data from gcov and do not timeout before all the
data has been captured. Also report on incomplete data capture.
Signed-off-by: Anas Nashif <anas.nashif@intel.com>
We have been dropping lines after finding a fault which resulted in
missing information in the log. Make sure we continue and only report
failure at the end of the execution.
Signed-off-by: Anas Nashif <anas.nashif@intel.com>
Do not close console after PASS is reported, wait a bit for any
remaining messages from the tests, sometimes we have faults that need to
be parsed.
This now works for Qemu handler, support for other handlers to follow.
Fixes#9646
Signed-off-by: Anas Nashif <anas.nashif@intel.com>
Fail in tests where we have an OOPS or a panic. Right now and in many
cases we continue and test case might be reported as PASS.
Cases that have the tag ignore_faults will ignore those faults (cases
that are testing faults for example).
Signed-off-by: Anas Nashif <anas.nashif@intel.com>
- Some tests start with test_, some do not, so make sure we parse both.
- Parse skipped tests
- Improve handling of test case identifier
- Handle Exceptions in device handler
Signed-off-by: Anas Nashif <anas.nashif@intel.com>
Parse the test results and create a test report with more granular
results that can be imported to into test management/reporting system.
Signed-off-by: Anas Nashif <anas.nashif@intel.com>
Add 2 classes, one to handle the current TestCase scenario, and one more
for handling generic Console with regex matching.
Signed-off-by: Anas Nashif <anas.nashif@intel.com>
