In some cases we were not handling return code correctly and evaluating
timeouts as a pass. Report failure correctly.
Fixes#26065
Signed-off-by: Wentong Wu <wentong.wu@intel.com>
For iterable areas defined with Z_STRUCT_SECTION_ITERABLE(),
the corresponding output section in the linker script is just
boilerplate. Add macros to make these definitions simpler.
Unfortunately, we have a fair number of iterable sections not
defined with Z_STRUCT_SECTION_ITERABLE(), this patch does not
address this.
The output sections are all named <struct name>_area, update
sanitylib.py with this.
sys_sem with no userspace, and k_lifo/k_fifo are special cases
where different data types that are all equivalent need to be
put in the same iterable area. Add
Z_STRUCT_SECTION_ITERABLE_ALTERNATE() for this special case.
Signed-off-by: Andrew Boie <andrew.p.boie@intel.com>
Prior to addition of schema validation running sanitycheck
--generate-hw-map would preserve the value of any field in an existing
map that it didn't update. These fields now cause a parsing error.
Add notes as an optional key where information relevant to the board
can be added to provide useful context (such as which serial port
corresponds to the console, or why a non-standard runner is required).
Signed-off-by: Peter Bigot <peter.bigot@nordicsemi.no>
probe_id is used as an optional key that overrides id as the source
of board_id when configured to use a runner that is not the default.
Signed-off-by: Peter Bigot <peter.bigot@nordicsemi.no>
If a test instance from .yaml file was skipped due to being on
a filtered list then there are no entries about it in the
sanitycheck_report.xml for a given platform. This commit fills
instance.results with skipped test cases also for skipped instances so
those test cases are visible when the full report is generated. Tested
for ./tests/kernel/mem_protect/userspace on nrf52dk_nrf52832 where
userspace.gap_filling is filtered out.
Signed-off-by: Maciej Perkowski <Maciej.Perkowski@nordicsemi.no>
It is now possible to add a list of fixture a platform supports which is
matched to testcases requesting fixtures to be avaiable to be able to
run.
For example:
- available: true
connected: true
id: 0240000026334e450015400f5e0e000b4eb1000097969900
platform: frdm_k64f
product: DAPLink CMSIS-DAP
runner: pyocd
serial: /dev/ttyACM9
fixtures:
- gpio_loopback
Fixes#24943
Signed-off-by: Anas Nashif <anas.nashif@intel.com>
Cleanup fixture processing and allow ztest testcases to support
harness_config with fixture definition.
Signed-off-by: Anas Nashif <anas.nashif@intel.com>
This fix creates class variables SAMPLE_FILENAME & TESTCASE_FILENAME
in Class TestSuite and remove the hardcoded sample.yaml & testcase.yaml
from add_testcases function. This makes testing for sanitylib script
easier so that shippable do not detect the test_data for sanitylib
as actual zephyr testcases.
Signed-off-by: Aastha Grover <aastha.grover@intel.com>
Add cpu time for QEMUHandler because the guest virtual time
in QEMU icount mode isn't host time and it's maintained by
counting guest instructions, we use QEMU process CPU time to
mostly simulate the time of guest OS.
Signed-off-by: Wentong Wu <wentong.wu@intel.com>
Sometimes QEMU can't handle SIGTERM signal correctly and just
run as normal, in that case kill -9 QEMU process directly and
leave sanitycheck judge the testing result by console output.
For failures caused by other reasons, terminate all releated
processes to avoid CI waits 2 hours and reports timeout.
Signed-off-by: Wentong Wu <wentong.wu@intel.com>
We don't need sanitycheck's use of EDT to report warnings, we'll get
them from the build system. So the warnings are just duplication and
noise, thus lets always suppress them.
Signed-off-by: Kumar Gala <kumar.gala@linaro.org>
Abort if we find tests that are being skipped incorrectly. Tests should
be skipped using ztest_test_skip().
Signed-off-by: Anas Nashif <anas.nashif@intel.com>
While testing sanitycheck classes, found some wrongly initialisized
variables which caused issues, fixed those..
All Test* classes should now be ignored by pytest and should not be
treated as pytest code.
Signed-off-by: Anas Nashif <anas.nashif@intel.com>
In some cases terminate (SIGTERM) is not enough and we have to kill the
proces (SIGKILL), otherwise the process is stuck forever and taking
hours to complete and finally timeout in CI.
Signed-off-by: Anas Nashif <anas.nashif@intel.com>
In some cases we were missing the reason and reporting 'Unknown',
check for failure and set the reason.
Signed-off-by: Anas Nashif <anas.nashif@intel.com>
Add option --report-suffix to append custom string to all generated
files. This is going to be useful for generating results for a specific
version, i.e. --report-suffix zephyr-v2.2.0-1814-ge737761d23
Signed-off-by: Anas Nashif <anas.nashif@intel.com>
This option prefers serial device names which are stable
across device plug/un-plug on platforms that support it (currently
just Linux, via /dev/serial/by-id).
This feature is opt-in as not all manufacturers include the
appropriate metadata for udev to generate unique names for their
devices.
Signed-off-by: Martí Bolívar <marti.bolivar@nordicsemi.no>
Create support for muxed UARTs which are attached to a real
UART and which use GSM 07.10 muxing protocol to create virtual
channels that can be run on top of the real UART.
Signed-off-by: Jukka Rissanen <jukka.rissanen@linux.intel.com>
When retrying failed tests, make sure we keep old results and only
update those tests that were retried.
Also, remove duplication of code for creating the reports and make the
report function a bit more generic.
sanitycheck.xml is now listing or tests, not only the test application.
Signed-off-by: Anas Nashif <anas.nashif@intel.com>
The "net stacks" shell command support was just removed, but
the net_stacks linker section was left around.
Signed-off-by: Jukka Rissanen <jukka.rissanen@linux.intel.com>
We are not propagting those options all the way to the device handler
now, fix this by using the suite member in the handler which has those
already.
Signed-off-by: Anas Nashif <anas.nashif@intel.com>
Do not declare VERBOSE as global, instead pass verbosity as argument.
Also get rid of options as global and fix coverage class to not use
global option variable.
Signed-off-by: Anas Nashif <anas.nashif@intel.com>
We do not use those arg files anymore, so remove them to avoid
confusion and for the sake of cleanup.
Signed-off-by: Anas Nashif <anas.nashif@intel.com>
We rename the nRF51 Dev Kit board target (nrf51_pca10028)
to nrf51dk_nrf51422. We update all associated references
in the supportive documentation and all nRF51-related
cofigurations and overlay files in the samples and tests
in the tree.
Signed-off-by: Ioannis Glaropoulos <Ioannis.Glaropoulos@nordicsemi.no>
We were parsing random FAIL messages from the output of test runs ad
testcases and capturing them in the xml output. Now we only parse the
name if it starts with test_.
Fixes#21162
Signed-off-by: Anas Nashif <anas.nashif@intel.com>
Add the following functions to allow filtering based on device tree
dt_compat_enabled(compat) - Returns true if a device tree node
compatible matches 'compat' and the node is enabled.
dt_alias_exists(alias) - Returns true if a device tree node exists with
'alias' and the node is enabled.
dt_compat_enabled_with_alias - Returns true if a device tree node
compatible matches 'compat' and the node has 'alias' and the node is
enabled.
Signed-off-by: Kumar Gala <kumar.gala@linaro.org>
