doc: sanitycheck: document all new features
Update documentation with new features. Signed-off-by: Anas Nashif <anas.nashif@intel.com>
This commit is contained in:
Anas Nashif
parent
2f0be4334b
commit
ace5609a6b
1 changed files with 114 additions and 154 deletions
|
|
@ -1,33 +1,34 @@
|
||||||
|
|
||||||
.. _sanitycheck_script:
|
.. _sanitycheck_script:
|
||||||
|
|
||||||
Zephyr Sanity Tests
|
Sanity Tests
|
||||||
###################
|
#############
|
||||||
|
|
||||||
This script scans for the set of unit test applications in the git repository
|
This script scans for the set of unit test applications in the git repository
|
||||||
and attempts to execute them. By default, it tries to build each test
|
and attempts to execute them. By default, it tries to build each test
|
||||||
case on boards marked as default in the board definition file.
|
case on boards marked as default in the board definition file.
|
||||||
|
|
||||||
The default options will build the majority of the tests on a defined set of
|
The default options will build the majority of the tests on a defined set of
|
||||||
boards and will run in an emulated environment (QEMU) if available for the
|
boards and will run in an emulated environment if available for the
|
||||||
architecture or configuration being tested.
|
architecture or configuration being tested.
|
||||||
|
|
||||||
In normal use, sanitycheck runs a limited set of kernel tests (inside
|
In normal use, sanitycheck runs a limited set of kernel tests (inside
|
||||||
QEMU). Because of its limited text execution coverage, sanitycheck
|
an emulator). Because of its limited text execution coverage, sanitycheck
|
||||||
cannot guarantee local changes will succeed in the full build
|
cannot guarantee local changes will succeed in the full build
|
||||||
environment, but it does sufficient testing by building samples and
|
environment, but it does sufficient testing by building samples and
|
||||||
tests for different boards and different configurations to help keep the
|
tests for different boards and different configurations to help keep the
|
||||||
full code tree buildable.
|
complete code tree buildable.
|
||||||
|
|
||||||
When using (at least) one ``-v`` option, sanitycheck's console output
|
When using (at least) one ``-v`` option, sanitycheck's console output
|
||||||
shows for every test how the test is run (qemu, native_posix, etc.) or
|
shows for every test how the test is run (qemu, native_posix, etc.) or
|
||||||
whether the image is just built. There are a few reasons why sanitycheck
|
whether the binary was just built. There are a few reasons why sanitycheck
|
||||||
only builds a test and doesn't run it:
|
only builds a test and doesn't run it:
|
||||||
|
|
||||||
- The test is marked as ``build_only: true`` in its ``.yaml``
|
- The test is marked as ``build_only: true`` in its ``.yaml``
|
||||||
configuration file.
|
configuration file.
|
||||||
- The test configuration has defined a ``harness`` but you don't have
|
- The test configuration has defined a ``harness`` but you don't have
|
||||||
it or haven't set it up.
|
it or haven't set it up.
|
||||||
|
- The target device is not connected and not available for flashing
|
||||||
- You or some higher level automation invoked sanitycheck with
|
- You or some higher level automation invoked sanitycheck with
|
||||||
``--build-only``.
|
``--build-only``.
|
||||||
|
|
||||||
|
|
@ -49,139 +50,12 @@ all possible tests using the following options:
|
||||||
$ ./scripts/sanitycheck --all --enable-slow
|
$ ./scripts/sanitycheck --all --enable-slow
|
||||||
|
|
||||||
This will build for all available boards and run all applicable tests in
|
This will build for all available boards and run all applicable tests in
|
||||||
a simulated (QEMU) environment.
|
a simulated (for example QEMU) environment.
|
||||||
|
|
||||||
The sanitycheck script accepts the following optional arguments:
|
The list of command line options supported by sanitycheck can be viewed using::
|
||||||
|
|
||||||
|
$ ./scripts/sanitycheck --help
|
||||||
|
|
||||||
-h, --help Show the complete and most up-to-date help message
|
|
||||||
and exit.
|
|
||||||
-p PLATFORM, --platform PLATFORM
|
|
||||||
Platform filter for testing. This option may be used
|
|
||||||
multiple times. Testcases will only be built/run on
|
|
||||||
the platforms specified. If this option is not used,
|
|
||||||
then platforms marked as default in the platform
|
|
||||||
metadata file will be chosen to build and test.
|
|
||||||
-a ARCH, --arch ARCH Arch filter for testing. Takes precedence over
|
|
||||||
--platform. If unspecified, test all arches. Multiple
|
|
||||||
invocations are treated as a logical 'or' relationship
|
|
||||||
-t TAG, --tag TAG Specify tags to restrict which tests to run by tag
|
|
||||||
value. Default is to not do any tag filtering.
|
|
||||||
Multiple invocations are treated as a logical 'or'
|
|
||||||
relationship
|
|
||||||
-e EXCLUDE_TAG, --exclude-tag EXCLUDE_TAG
|
|
||||||
Specify tags of tests that should not run. Default is
|
|
||||||
to run all tests with all tags.
|
|
||||||
-f, --only-failed Run only those tests that failed the previous sanity
|
|
||||||
check invocation.
|
|
||||||
-c CONFIG, --config CONFIG
|
|
||||||
Specify platform configuration values filtering. This
|
|
||||||
can be specified two ways: <config>=<value> or just
|
|
||||||
<config>. The defconfig for all platforms will be
|
|
||||||
checked. For the <config>=<value> case, only match
|
|
||||||
defconfig that have that value defined. For the
|
|
||||||
<config> case, match defconfig that have that value
|
|
||||||
assigned to any value. Prepend a '!' to invert the
|
|
||||||
match.
|
|
||||||
-s TEST, --test TEST Run only the specified test cases. These are named by
|
|
||||||
<path to test project relative to --testcase-
|
|
||||||
root>/<testcase.yaml section name>
|
|
||||||
-l, --all Build/test on all platforms. Any --platform arguments
|
|
||||||
ignored.
|
|
||||||
-o TESTCASE_REPORT, --testcase-report TESTCASE_REPORT
|
|
||||||
Output a CSV spreadsheet containing results of the
|
|
||||||
test run
|
|
||||||
-d DISCARD_REPORT, --discard-report DISCARD_REPORT
|
|
||||||
Output a CSV spreadsheet showing tests that were
|
|
||||||
skipped and why
|
|
||||||
--compare-report COMPARE_REPORT
|
|
||||||
Use this report file for size comparison
|
|
||||||
-B SUBSET, --subset SUBSET
|
|
||||||
Only run a subset of the tests, 1/4 for running the
|
|
||||||
first 25%, 3/5 means run the 3rd fifth of the total.
|
|
||||||
This option is useful when running a large number of
|
|
||||||
tests on different hosts to speed up execution time.
|
|
||||||
-N, --ninja Use the Ninja generator with CMake
|
|
||||||
-y, --dry-run Create the filtered list of test cases, but don't
|
|
||||||
actually run them. Useful if you're just interested in
|
|
||||||
--discard-report
|
|
||||||
--list-tags list all tags in selected tests
|
|
||||||
--list-tests list all tests.
|
|
||||||
--detailed-report FILENAME
|
|
||||||
Generate a junit report with detailed testcase
|
|
||||||
results.
|
|
||||||
-r, --release Update the benchmark database with the results of this
|
|
||||||
test run. Intended to be run by CI when tagging an
|
|
||||||
official release. This database is used as a basis for
|
|
||||||
comparison when looking for deltas in metrics such as
|
|
||||||
footprint
|
|
||||||
-w, --warnings-as-errors
|
|
||||||
Treat warning conditions as errors
|
|
||||||
-v, --verbose Emit debugging information, call multiple times to
|
|
||||||
increase verbosity
|
|
||||||
-i, --inline-logs Upon test failure, print relevant log data to stdout
|
|
||||||
instead of just a path to it
|
|
||||||
--log-file FILENAME log also to file
|
|
||||||
-m, --last-metrics Instead of comparing metrics from the last --release,
|
|
||||||
compare with the results of the previous sanity check
|
|
||||||
invocation
|
|
||||||
-u, --no-update do not update the results of the last run of the
|
|
||||||
sanity checks
|
|
||||||
-F FILENAME, --load-tests FILENAME
|
|
||||||
Load list of tests to be run from file.
|
|
||||||
-E FILENAME, --save-tests FILENAME
|
|
||||||
Save list of tests to be run to file.
|
|
||||||
-b, --build-only Only build the code, do not execute any of it in QEMU
|
|
||||||
-j JOBS, --jobs JOBS Number of cores to use when building, defaults to
|
|
||||||
number of CPUs * 2
|
|
||||||
--device-testing Test on device directly. Specify the serial device to
|
|
||||||
use with the --device-serial option.
|
|
||||||
--device-serial DEVICE_SERIAL
|
|
||||||
Serial device for accessing the board (e.g.,
|
|
||||||
/dev/ttyACM0)
|
|
||||||
--show-footprint Show footprint statistics and deltas since last
|
|
||||||
release.
|
|
||||||
-H FOOTPRINT_THRESHOLD, --footprint-threshold FOOTPRINT_THRESHOLD
|
|
||||||
When checking test case footprint sizes, warn the user
|
|
||||||
if the new app size is greater then the specified
|
|
||||||
percentage from the last release. Default is 5. 0 to
|
|
||||||
warn on any increase on app size
|
|
||||||
-D, --all-deltas Show all footprint deltas, positive or negative.
|
|
||||||
Implies --footprint-threshold=0
|
|
||||||
-O OUTDIR, --outdir OUTDIR
|
|
||||||
Output directory for logs and binaries. This directory
|
|
||||||
will be deleted unless '--no-clean' is set.
|
|
||||||
-n, --no-clean Do not delete the outdir before building. Will result
|
|
||||||
in faster compilation since builds will be incremental
|
|
||||||
-T TESTCASE_ROOT, --testcase-root TESTCASE_ROOT
|
|
||||||
Base directory to recursively search for test cases.
|
|
||||||
All testcase.yaml files under here will be processed.
|
|
||||||
May be called multiple times. Defaults to the
|
|
||||||
'samples' and 'tests' directories in the Zephyr tree.
|
|
||||||
-A BOARD_ROOT, --board-root BOARD_ROOT
|
|
||||||
Directory to search for board configuration files. All
|
|
||||||
.yaml files in the directory will be processed.
|
|
||||||
-z SIZE, --size SIZE Don't run sanity checks. Instead, produce a report to
|
|
||||||
stdout detailing RAM/ROM sizes on the specified
|
|
||||||
filenames. All other command line arguments ignored.
|
|
||||||
-S, --enable-slow Execute time-consuming test cases that have been
|
|
||||||
marked as 'slow' in testcase.yaml. Normally these are
|
|
||||||
only built.
|
|
||||||
-R, --enable-asserts Build all test cases with assertions enabled. (This is
|
|
||||||
the default)
|
|
||||||
--disable-asserts Build all test cases with assertions disabled.
|
|
||||||
-Q, --error-on-deprecations
|
|
||||||
Error on deprecation warnings.
|
|
||||||
-x EXTRA_ARGS, --extra-args EXTRA_ARGS
|
|
||||||
Extra CMake cache entries to define when building test
|
|
||||||
cases. May be called multiple times. The key-value
|
|
||||||
entries will be prefixed with -D before being passed
|
|
||||||
to CMake. E.g "sanitycheck -x=USE_CCACHE=0" will
|
|
||||||
translate to "cmake -DUSE_CCACHE=0" which will
|
|
||||||
ultimately disable ccache.
|
|
||||||
--enable-coverage Enable code coverage when building unit tests and when
|
|
||||||
targeting the native_posix board
|
|
||||||
-C, --coverage Generate coverage report for unit tests, and tests and
|
|
||||||
samples run in native_posix. Implies --enable-coverage.
|
|
||||||
|
|
||||||
|
|
||||||
Board Configuration
|
Board Configuration
|
||||||
|
|
@ -199,20 +73,30 @@ required for best test coverage for this specific board:
|
||||||
|
|
||||||
.. code-block:: yaml
|
.. code-block:: yaml
|
||||||
|
|
||||||
identifier: tinytile
|
identifier: frdm_k64f
|
||||||
name: tinyTILE
|
name: NXP FRDM-K64F
|
||||||
type: mcu
|
type: mcu
|
||||||
arch: x86
|
arch: arm
|
||||||
toolchain:
|
toolchain:
|
||||||
- zephyr
|
- zephyr
|
||||||
- issm
|
- gnuarmemb
|
||||||
ram: 52
|
- xtools
|
||||||
flash: 192
|
supported:
|
||||||
testing:
|
- arduino_gpio
|
||||||
default: true
|
- arduino_i2c
|
||||||
ignore_tags:
|
- netif:eth
|
||||||
- net
|
- adc
|
||||||
- bluetooth
|
- i2c
|
||||||
|
- nvs
|
||||||
|
- spi
|
||||||
|
- gpio
|
||||||
|
- usb_device
|
||||||
|
- watchdog
|
||||||
|
- hwinfo
|
||||||
|
- can
|
||||||
|
- pwm
|
||||||
|
testing:
|
||||||
|
default: true
|
||||||
|
|
||||||
|
|
||||||
identifier:
|
identifier:
|
||||||
|
|
@ -421,6 +305,13 @@ harness_config: <harness configuration options>
|
||||||
type: <one_line|multi_line> (required)
|
type: <one_line|multi_line> (required)
|
||||||
Depends on the regex string to be matched
|
Depends on the regex string to be matched
|
||||||
|
|
||||||
|
|
||||||
|
record: <recording options>
|
||||||
|
|
||||||
|
regex: <expression> (required)
|
||||||
|
Any string that the particular test case prints to record test
|
||||||
|
results.
|
||||||
|
|
||||||
regex: <expression> (required)
|
regex: <expression> (required)
|
||||||
Any string that the particular test case prints to confirm test
|
Any string that the particular test case prints to confirm test
|
||||||
runs as expected.
|
runs as expected.
|
||||||
|
|
@ -548,15 +439,84 @@ Beside being able to run tests in QEMU and other simulated environments,
|
||||||
sanitycheck supports running most of the tests on real devices and produces
|
sanitycheck supports running most of the tests on real devices and produces
|
||||||
reports for each run with detailed FAIL/PASS results.
|
reports for each run with detailed FAIL/PASS results.
|
||||||
|
|
||||||
To use this feature, run sanitycheck with the following new options::
|
|
||||||
|
Executing tests on a single device
|
||||||
|
===================================
|
||||||
|
|
||||||
|
To use this feature on a single connected device, run sanitycheck with
|
||||||
|
the following new options::
|
||||||
|
|
||||||
scripts/sanitycheck --device-testing --device-serial /dev/ttyACM0 -p \
|
scripts/sanitycheck --device-testing --device-serial /dev/ttyACM0 -p \
|
||||||
frdm_k64f -T tests/kernel
|
frdm_k64f -T tests/kernel
|
||||||
|
|
||||||
The ``--device-serial`` option denotes the serial device the board is connected to.
|
The ``--device-serial`` option denotes the serial device the board is connected to.
|
||||||
This needs to be accessible by the user running sanitycheck. You can run this on
|
This needs to be accessible by the user running sanitycheck. You can run this on
|
||||||
only one board at a time, specified using the
|
only one board at a time, specified using the ``--platform`` option.
|
||||||
``--platform`` option.
|
|
||||||
|
|
||||||
|
Executing tests on multiple devices
|
||||||
|
===================================
|
||||||
|
|
||||||
|
To build and execute tests on multiple devices connected to the host PC, a
|
||||||
|
hardware map needs to be created with all connected devices and their
|
||||||
|
details such as the serial device and their IDs if available. Run the following
|
||||||
|
command to produce the hardware map::
|
||||||
|
|
||||||
|
./scripts/sanitycheck --generate-hardware-map map.yml
|
||||||
|
|
||||||
|
The generated hardware map file (map.yml) will have the list of connected
|
||||||
|
devices, for example::
|
||||||
|
|
||||||
|
- available: true
|
||||||
|
id: OSHW000032254e4500128002ab98002784d1000097969900
|
||||||
|
platform: unknown
|
||||||
|
product: DAPLink CMSIS-DAP
|
||||||
|
runner: pyocd
|
||||||
|
serial: /dev/cu.usbmodem146114202
|
||||||
|
- available: true
|
||||||
|
id: 000683759358
|
||||||
|
platform: unknown
|
||||||
|
product: J-Link
|
||||||
|
runner: unknown
|
||||||
|
serial: /dev/cu.usbmodem0006837593581
|
||||||
|
|
||||||
|
|
||||||
|
Any options marked as 'unknown' need to be changed and set with the correct
|
||||||
|
values, in the above example both the platform names and the runners need to be
|
||||||
|
replaced with the correct values corresponding to the connected hardware. In
|
||||||
|
this example we are using a reel_board and an nrf52840_pca10056::
|
||||||
|
|
||||||
|
- available: true
|
||||||
|
id: OSHW000032254e4500128002ab98002784d1000097969900
|
||||||
|
platform: reel_board
|
||||||
|
product: DAPLink CMSIS-DAP
|
||||||
|
runner: pyocd
|
||||||
|
serial: /dev/cu.usbmodem146114202
|
||||||
|
- available: true
|
||||||
|
id: 000683759358
|
||||||
|
platform: nrf52840_pca10056
|
||||||
|
product: J-Link
|
||||||
|
runner: nrfjprog
|
||||||
|
serial: /dev/cu.usbmodem0006837593581
|
||||||
|
|
||||||
|
If the map file already exists, then new entries are added and existing entries
|
||||||
|
will be updated. This way you can use one single master hardware map and update
|
||||||
|
it for every run to get the correct serial devices and status of the devices.
|
||||||
|
|
||||||
|
With the hardware map ready, you can run any tests by pointing to the map
|
||||||
|
file::
|
||||||
|
|
||||||
|
./scripts/sanitycheck --device-testing --hardware-map map.yml -T samples/hello_world/
|
||||||
|
|
||||||
|
The above command will result in sanitycheck building tests for the platforms
|
||||||
|
defined in the hardware map and subsequently flashing and running the tests
|
||||||
|
on those platforms.
|
||||||
|
|
||||||
|
.. note::
|
||||||
|
|
||||||
|
Currently only boards with support for both pyocd and nrfjprog are supported
|
||||||
|
with the hardware map features. Boards that require other runners to flash the
|
||||||
|
Zephyr binary are still work in progress.
|
||||||
|
|
||||||
To produce test reports, use the ``--detailed-report FILENAME`` option which will
|
To produce test reports, use the ``--detailed-report FILENAME`` option which will
|
||||||
generate an XML file using the JUNIT syntax. This file can be used to generate
|
generate an XML file using the JUNIT syntax. This file can be used to generate
|
||||||
|
|
|
||||||
Loading…
Add table
Add a link
Reference in a new issue