If we already have recorded the reason during the qemu run, do not
override it at the final stage of status processing.
Signed-off-by: Anas Nashif <anas.nashif@intel.com>
below command line need has fixture configured in map.yml,
if not, it shall be skipped
twister -p frdm_k64f --device-testing \
--hardware-map map.yml -T tests/subsys/modbus/ -vv
add harness check with testcase harness by default
will fix this issue
Fixing: #46635
Signed-off-by: Hake Huang <hake.huang@oss.nxp.com>
Option --runtime-artifact-cleanup/-M allows to clean build artifacts
during runtime to save space usage. However, if one wanted to combine
it with --build-only and then --test-only, they would get an error.
This commit expands the list of files stored with -M with files
required to made --test-only also to work afterwards. Such change can
be useful for setups where building is done on one node and then
minimal amount of artifacts are transfered to another one.
Signed-off-by: Maciej Perkowski <Maciej.Perkowski@nordicsemi.no>
add_missing_case_status is more appropriate for this function.
We are adding missing status, not cases.
Signed-off-by: Anas Nashif <anas.nashif@intel.com>
Add a filter type to distinguish between testsuite filters that are
maintained in the testcase.yaml file and those provided on the command
line.
The issue was that when something is excluded deliberately on the command
line twister would report an error if that filter would filter out an
integration platform. We do not want that, because the filtering is
being done by the caller knowing that some tests would be not run, even
if those are integration platforms.
Signed-off-by: Anas Nashif <anas.nashif@intel.com>
Some tests can cause serial leftover logs buffered on board.
A simple ser.flush() is not enough. So add explicit readline()
to drain such logs which prepares a clean serial context for
the case that follows.
Signed-off-by: Ming Shao <ming.shao@intel.com>
The fallback testcase, which is the scenario name from the yaml file is
added if we did not discover any testcases via parsing. It can be
removed if we discover testcaes at runtime or otherwise it is added as
skipped...
Signed-off-by: Anas Nashif <anas.nashif@intel.com>
Status was set for testcases after a timeout on flash, which does not
abort the process and continues with test execution and eventually
passes the test.
Setting status on the testcases was done too early and before we have
executed the tests.
Also increase the flash timeout to 60s.
Fixes#45845
Signed-off-by: Anas Nashif <anas.nashif@intel.com>
Do not set status as blocked on testsuites which passed. The sub cases
with no status will get a processed later.
Fixes#45845
Signed-off-by: Anas Nashif <anas.nashif@intel.com>
Add a pty parameter from twister to intel_adsp when west-flash,
to tell the intel_adsp runner to change the way it outputs the
log.
Signed-off-by: Enjia Mai <enjia.mai@intel.com>
At the end of DeviceHandler.handle(), the execution time
should be recorded no matter what the harness.state is.
This reflects how much time has been spent on the device.
And it is also used later to determine if a test has been
attempted on a real device or just build-only.
Signed-off-by: Ming Shao <ming.shao@intel.com>
in device test when harness is Test, if there is no console output,
the self.state is None, as at this time self.instance.state
is "pass", as the last step cmake is "pass". So it will report "pass",
but actually there is no console output issue
you can easily reproduce this issue by set
line='' at the begining of the Test.handle function
fixing: #45942
Signed-off-by: Hake Huang <hake.huang@oss.nxp.com>
Colorama, by default, strips out the color escape characters when the
output is redirected, and this may have an unintended consequence of
disabling color outputs when using a utility such as `tee` and in the
CI runners that redirect the stdout and stderr console outputs.
This commit adds a new command line option called `--force-color` to
always force the ANSI color escape sequence output even when the output
is redirected.
Signed-off-by: Stephanos Ioannidis <root@stephanos.io>
This commit adds a call to the Colorama initialisation function during
the module execution so that ANSI color sequences are properly
converted to the relevant Win32 API calls on the Windows.
Note that the init function needs to be called per module, hence it is
also called in the `twisterlib.py`.
Signed-off-by: Stephanos Ioannidis <root@stephanos.io>
Serial is not required, in some cases it might be set to null or we
might be using serial_pty, so sort existing map file based on ID
instead.
Fixes#45713
Signed-off-by: Anas Nashif <anas.nashif@intel.com>
For some boards, like up_squared and ehl_crb, they need do a
power off/on operation and use bios to load and execute zephyr
test binary, during the power off, it may meet this SerialException,
but the serial object still works well after power on, we can continue
to get all results from serial, so we don't need to close it,
it's ok to ignore this exception.
BTW, even if we ignore this exception here, it doesn't influence twister
function, because if it was really caused by disconnection, the test would
still be judged as failed due to timeout and serial would be closed later
in main thread, it doesn't impact results.
On the contrary, close serial directly in the monitor serial thread, it
will cause later tests failed due to mismatch errors.
Signed-off-by: Chen Peng1 <peng1.chen@intel.com>
Instead of relying on runtime filter to limit scope to emulation
platforms, use the type attribute for each platform and do the filtering
very early on. This will speed things up for tests where we only run on
emulation platforms.
Signed-off-by: Anas Nashif <anas.nashif@intel.com>
For CI and for the condensed version of the report, consider built tests
as passed and do not mark them as skipped. Tests that are built only
will be reported as skipped in the full and suite reports.
Signed-off-by: Anas Nashif <anas.nashif@intel.com>
Filtered tests are now not being reported by default to allow generation
of reports that are easier to parse and work with in different tools.
The complete filtered set of tests is still available in the json output
for review and verification.
Signed-off-by: Anas Nashif <anas.nashif@intel.com>
We have been generating reports with the platform being used as the
testsuite, ie. all testcases would land under one single big testsuite,
which is very difficult to use or read.
This change uses testsuites as intended and does that for each defined
testsuite (or scenario). Platforms are added as properties among other
things.
Signed-off-by: Anas Nashif <anas.nashif@intel.com>
When loading tests from json file we want to reset the reason for those
tests that are to be run and remove whatever we had when the test was
only built.
Signed-off-by: Anas Nashif <anas.nashif@intel.com>
we mark built only tests as skipped, however, the type was still the
same as other skipped tests, set this to 'built' to indicate the tests
was only built.
Signed-off-by: Anas Nashif <anas.nashif@intel.com>
Remove some legacy code and set status and failure/skip reason directly
without set_state and get_state.
Signed-off-by: Anas Nashif <anas.nashif@intel.com>
Handle build errors due to overflows correctly and do not report those
as failures if not requested.
Signed-off-by: Anas Nashif <anas.nashif@intel.com>
When we have a build failure or anything that would prevent execution of
the test, set the status for all subcases accordignly.
Signed-off-by: Anas Nashif <anas.nashif@intel.com>
Report misconfigued testsuites and subcase parsing issues instead of
erroring out. This should help us fix those tests and then enforce
accurate result capturing.
Signed-off-by: Anas Nashif <anas.nashif@intel.com>
Remove unused arguments that became obsolete through the refactoring.
- filter_status
- only_failed
- json_report
Signed-off-by: Anas Nashif <anas.nashif@intel.com>
Capture failures per testcase so that we can include the output in
reports for only failed testcases and not include the whole log which
can be overwhelming.
Signed-off-by: Anas Nashif <anas.nashif@intel.com>
Removed tests for dropped functionality.
Renamed classes and files to reflect new structure.
As we refactor the reporting infrastructure, tests will follow once we
have everything in place.
Signed-off-by: Anas Nashif <anas.nashif@intel.com>
Some complex testsuite use macros and other mean to define testcases
which makes them unparseable. This ends up in a descripency between the
testplan and the test results.
Allow adding the testcases into the yaml and use those instead of
parsing the C files.
Signed-off-by: Anas Nashif <anas.nashif@intel.com>
We are now able to capture individual case duration and can display this
as part of the reports instead of using the overall test instance
execution time.
Signed-off-by: Anas Nashif <anas.nashif@intel.com>
Now we can add the TestCase class which will hold all information about
individual testcases, their status and why they are not passing, etc.
Test cases are parsed initially and testsuite is populated with the list
of testcases it contsins. This list is duplicate for each instance
(scenario) and used to capture test case data.
Signed-off-by: Anas Nashif <anas.nashif@intel.com>
More renaming and change to variable names to make it less confusing.
Especially, renames of tc to become ts where applicable.
Signed-off-by: Anas Nashif <anas.nashif@intel.com>
TestCase in the old terminology is the test application, but we have
just freed up TestSuite, so use TestSuite here to signify the test
application which consists of many test scenarios and testcases.
Signed-off-by: Anas Nashif <anas.nashif@intel.com>
Change terminology and fix usage based on the current hierarchy. A
testsuite is each test application we have in the tree. The top class is
where we detect all the test suites and create the testplan to be run
later.
Content of testplan will have to split out, so it only deals with the
plan and not the overall execution of tests. This step will be done at a
later point.
Signed-off-by: Anas Nashif <anas.nashif@intel.com>
