restore how --only-failed works by allow rebuilds in case of build
errors, however, do not rebuild when --retry-failed is used, which is a
CI usecase and nothing would change in the second iteration.
Fixes#26685
Signed-off-by: Anas Nashif <anas.nashif@intel.com>
According to our compliance checks, we have 'useless-suppression'
and 'unused-import' on our hands:
W0611: Unused CLoader imported from yaml as Loader
W0611: Unused Loader imported from yaml
I0021: Useless suppression of 'unsubscriptable-object'
Signed-off-by: Martí Bolívar <marti.bolivar@nordicsemi.no>
Use the C LibYAML parser if available, rather than the Python parser. It
is much faster.
This is a clean and rebased version of PR #20210 by
Ulf Magnusson <Ulf.Magnusson@nordicsemi.no>
Co-authored-by: Ulf Magnusson <Ulf.Magnusson@nordicsemi.no>
Signed-off-by: Anas Nashif <anas.nashif@intel.com>
Deal with qemu crashes and do not register pass status as the default.
In case we do not have a status, report unknown.
This now captures different states from qemu that were not captured
before.
Fixes#26679
Signed-off-by: Anas Nashif <anas.nashif@intel.com>
Recent ST boards embed the new ST-Link probe V3.
It is advertised as "STLINK-V3", update sanitylmib to take it
into account.
In handle function, it is proposed to treat it separately as current
"STM32 STLink" as processing might differ in next future (hla_serial
deprecation).
Signed-off-by: Erwan Gouriou <erwan.gouriou@linaro.org>
Some devices have multiple serial ports, which show as duplicates in the
hardware map. This doesn't work well when using --generate-hardware-map
to regenerate an existing map. E.g. nrf5340pdk_nrf5340_cpuapp by default
prints to the 3rd of 3 devices. If it shows up on /dev/ttyACM0,
/dev/ttyACM1, and /dev/ttyACM2, printout will come on /dev/ttyACM2.
nrf9160dk_nrf9160 also provides 3 devices, but prints to the 1st.
This patch sorts the devices by the serial port and matches multiple
duplicate entries one-to-one to retain the ordering. This way, the
correct device can be given the platform name and the others can be kept
as "unknown" so that --device-testing understands correctly
1) that there is only one device (not 3)
2) where the serial output will come.
Signed-off-by: Øyvind Rønningstad <oyvind.ronningstad@nordicsemi.no>
Modifying discard_report function to fix the check for
self.discards dictionary in try block. self.discards
is None is dead code which didnot check if the dictionary was
empty. Changed it to if not self.discards.
Signed-off-by: Aastha Grover <aastha.grover@intel.com>
- Report build errors as errors, not test failures
- Do not try and build/run tests with build failures
- Fix issue with empty reports when running --only-failed
- Report build errors in the detailed and target reports
Signed-off-by: Anas Nashif <anas.nashif@intel.com>
While testing the warnings in scan_file() format is not the same
everytime we run tests
Added sorted with reverse flag so that warnings are same for
every run
Signed-off-by: Spoorthy Priya Yerabolu <spoorthy.priya.yerabolu@intel.com>
If we pass --west-flash with no additional argument that was getting
treated the same as if it wasn't passed at all.
Change the check to be based on 'None' to handle the bare --west-flash
case properly.
Signed-off-by: Kumar Gala <kumar.gala@linaro.org>
classname in the junit report was not being used correctly since the
beginning. We had the targetname in the classname and put all results in
1 single testsuite.
The new output add 1 single testsuite for each platform and simplifies
the classname so that tests of the same classname can be grouped in
output, producing readable reports.
The current output can be seen here:
https://testing.zephyrproject.org/daily_tests/zephyr-v2.2.0-2533-gdbbf834605/report/index.html
This should much better once we have this change in.
Signed-off-by: Anas Nashif <anas.nashif@intel.com>
In some cases we were not handling return code correctly and evaluating
timeouts as a pass. Report failure correctly.
Fixes#26065
Signed-off-by: Wentong Wu <wentong.wu@intel.com>
For iterable areas defined with Z_STRUCT_SECTION_ITERABLE(),
the corresponding output section in the linker script is just
boilerplate. Add macros to make these definitions simpler.
Unfortunately, we have a fair number of iterable sections not
defined with Z_STRUCT_SECTION_ITERABLE(), this patch does not
address this.
The output sections are all named <struct name>_area, update
sanitylib.py with this.
sys_sem with no userspace, and k_lifo/k_fifo are special cases
where different data types that are all equivalent need to be
put in the same iterable area. Add
Z_STRUCT_SECTION_ITERABLE_ALTERNATE() for this special case.
Signed-off-by: Andrew Boie <andrew.p.boie@intel.com>
Prior to addition of schema validation running sanitycheck
--generate-hw-map would preserve the value of any field in an existing
map that it didn't update. These fields now cause a parsing error.
Add notes as an optional key where information relevant to the board
can be added to provide useful context (such as which serial port
corresponds to the console, or why a non-standard runner is required).
Signed-off-by: Peter Bigot <peter.bigot@nordicsemi.no>
probe_id is used as an optional key that overrides id as the source
of board_id when configured to use a runner that is not the default.
Signed-off-by: Peter Bigot <peter.bigot@nordicsemi.no>
If a test instance from .yaml file was skipped due to being on
a filtered list then there are no entries about it in the
sanitycheck_report.xml for a given platform. This commit fills
instance.results with skipped test cases also for skipped instances so
those test cases are visible when the full report is generated. Tested
for ./tests/kernel/mem_protect/userspace on nrf52dk_nrf52832 where
userspace.gap_filling is filtered out.
Signed-off-by: Maciej Perkowski <Maciej.Perkowski@nordicsemi.no>
It is now possible to add a list of fixture a platform supports which is
matched to testcases requesting fixtures to be avaiable to be able to
run.
For example:
- available: true
connected: true
id: 0240000026334e450015400f5e0e000b4eb1000097969900
platform: frdm_k64f
product: DAPLink CMSIS-DAP
runner: pyocd
serial: /dev/ttyACM9
fixtures:
- gpio_loopback
Fixes#24943
Signed-off-by: Anas Nashif <anas.nashif@intel.com>
Cleanup fixture processing and allow ztest testcases to support
harness_config with fixture definition.
Signed-off-by: Anas Nashif <anas.nashif@intel.com>
This fix creates class variables SAMPLE_FILENAME & TESTCASE_FILENAME
in Class TestSuite and remove the hardcoded sample.yaml & testcase.yaml
from add_testcases function. This makes testing for sanitylib script
easier so that shippable do not detect the test_data for sanitylib
as actual zephyr testcases.
Signed-off-by: Aastha Grover <aastha.grover@intel.com>
Add cpu time for QEMUHandler because the guest virtual time
in QEMU icount mode isn't host time and it's maintained by
counting guest instructions, we use QEMU process CPU time to
mostly simulate the time of guest OS.
Signed-off-by: Wentong Wu <wentong.wu@intel.com>
Sometimes QEMU can't handle SIGTERM signal correctly and just
run as normal, in that case kill -9 QEMU process directly and
leave sanitycheck judge the testing result by console output.
For failures caused by other reasons, terminate all releated
processes to avoid CI waits 2 hours and reports timeout.
Signed-off-by: Wentong Wu <wentong.wu@intel.com>
We don't need sanitycheck's use of EDT to report warnings, we'll get
them from the build system. So the warnings are just duplication and
noise, thus lets always suppress them.
Signed-off-by: Kumar Gala <kumar.gala@linaro.org>
Abort if we find tests that are being skipped incorrectly. Tests should
be skipped using ztest_test_skip().
Signed-off-by: Anas Nashif <anas.nashif@intel.com>
While testing sanitycheck classes, found some wrongly initialisized
variables which caused issues, fixed those..
All Test* classes should now be ignored by pytest and should not be
treated as pytest code.
Signed-off-by: Anas Nashif <anas.nashif@intel.com>
In some cases terminate (SIGTERM) is not enough and we have to kill the
proces (SIGKILL), otherwise the process is stuck forever and taking
hours to complete and finally timeout in CI.
Signed-off-by: Anas Nashif <anas.nashif@intel.com>
In some cases we were missing the reason and reporting 'Unknown',
check for failure and set the reason.
Signed-off-by: Anas Nashif <anas.nashif@intel.com>
Add option --report-suffix to append custom string to all generated
files. This is going to be useful for generating results for a specific
version, i.e. --report-suffix zephyr-v2.2.0-1814-ge737761d23
Signed-off-by: Anas Nashif <anas.nashif@intel.com>
This option prefers serial device names which are stable
across device plug/un-plug on platforms that support it (currently
just Linux, via /dev/serial/by-id).
This feature is opt-in as not all manufacturers include the
appropriate metadata for udev to generate unique names for their
devices.
Signed-off-by: Martí Bolívar <marti.bolivar@nordicsemi.no>
Create support for muxed UARTs which are attached to a real
UART and which use GSM 07.10 muxing protocol to create virtual
channels that can be run on top of the real UART.
Signed-off-by: Jukka Rissanen <jukka.rissanen@linux.intel.com>
When retrying failed tests, make sure we keep old results and only
update those tests that were retried.
Also, remove duplication of code for creating the reports and make the
report function a bit more generic.
sanitycheck.xml is now listing or tests, not only the test application.
Signed-off-by: Anas Nashif <anas.nashif@intel.com>
The "net stacks" shell command support was just removed, but
the net_stacks linker section was left around.
Signed-off-by: Jukka Rissanen <jukka.rissanen@linux.intel.com>
We are not propagting those options all the way to the device handler
now, fix this by using the suite member in the handler which has those
already.
Signed-off-by: Anas Nashif <anas.nashif@intel.com>
Do not declare VERBOSE as global, instead pass verbosity as argument.
Also get rid of options as global and fix coverage class to not use
global option variable.
Signed-off-by: Anas Nashif <anas.nashif@intel.com>
