in device test when harness is Test, if there is no console output,
the self.state is None, as at this time self.instance.state
is "pass", as the last step cmake is "pass". So it will report "pass",
but actually there is no console output issue
you can easily reproduce this issue by set
line='' at the begining of the Test.handle function
fixing: #45942
Signed-off-by: Hake Huang <hake.huang@oss.nxp.com>
Colorama, by default, strips out the color escape characters when the
output is redirected, and this may have an unintended consequence of
disabling color outputs when using a utility such as `tee` and in the
CI runners that redirect the stdout and stderr console outputs.
This commit adds a new command line option called `--force-color` to
always force the ANSI color escape sequence output even when the output
is redirected.
Signed-off-by: Stephanos Ioannidis <root@stephanos.io>
Update the new API to use K_USER as the flags for both
CONFIG_USERSPACE and CONFIG_TEST_USERSPACE. Also, fix the linker
script to properly include the suites, tests, and rules.
Fixes#44108
Signed-off-by: Yuval Peress <peress@google.com>
This is a simple script which snapshots open bugs in the
zephyrproject-rtos/zephyr repository using the GitHub REST API.
It relies on the 'github' module, which is a de-facto standard for
accessing this API in Python:
https://pygithub.readthedocs.io
We are already using this package in scripts/release/bug_bash.py, so
this is not a new dependency, even though it's a third party package
not available in the standard library.
The resulting bugs are stored in the standard library's 'pickle'
format, as a list of github.Issue.Issue objects. For more on pickle,
see:
https://docs.python.org/3/library/pickle.html
I am choosing pickle because it is standard, easy to use, and we are
already using it in the build system to store the edtlib.EDT object
created by gen_defines.py. This is also therefore not a new
dependency (and even if it were, it's in the standard library).
Signed-off-by: Martí Bolívar <marti.bolivar@nordicsemi.no>
This is meant to be a place where we can store generic zephyr wrappers
around the de-facto standard github API for python.
The first 'customer' will be a script that snapshots our open bugs at a
particular point in time, which will be added in a later patch.
I think it's useful to factor this file out of there from the
beginning just to keep things clean, even though I don't have a second
customer in mind at the moment.
Signed-off-by: Martí Bolívar <marti.bolivar@nordicsemi.no>
Signed-off-by: Stephanos Ioannidis <root@stephanos.io>
This commit adds a call to the Colorama initialisation function during
the module execution so that ANSI color sequences are properly
converted to the relevant Win32 API calls on the Windows.
Signed-off-by: Stephanos Ioannidis <root@stephanos.io>
This commit adds a call to the Colorama initialisation function during
the module execution so that ANSI color sequences are properly
converted to the relevant Win32 API calls on the Windows.
Signed-off-by: Stephanos Ioannidis <root@stephanos.io>
This commit adds a call to the Colorama initialisation function during
the module execution so that ANSI color sequences are properly
converted to the relevant Win32 API calls on the Windows.
Signed-off-by: Stephanos Ioannidis <root@stephanos.io>
This commit adds a call to the Colorama initialisation function during
the module execution so that ANSI color sequences are properly
converted to the relevant Win32 API calls on the Windows.
Note that the init function needs to be called per module, hence it is
also called in the `twisterlib.py`.
Signed-off-by: Stephanos Ioannidis <root@stephanos.io>
Iterate on 3 main namespace to avoid collision in identifiers sharing
the same name at the top level when creating the diff.
Fixes#44940
Signed-off-by: Anas Nashif <anas.nashif@intel.com>
Provide a way to specify the openocd target script handle used to
describe the core target.
In most target scripts '_TARGETNAME' is used, but it can happen that
'_TARGETNAME.foo' or '_CHIPNAME.bar' is used, specially on SoCs subject
to multicore.
Today, this option is required to enable rtos thread awareness,
using '$_TARGETNAME configure -rtos Zephyr' command, which is generated
by this runner.
It could be useful for similar needs in future.
Signed-off-by: Erwan Gouriou <erwan.gouriou@linaro.org>
The builtin list function `.sort()` sorts the list in-place and returns
None. As this is an invalid type for iteration, use the builtin `sorted`
function, which returns a sorted copy of the list, which we can iterate
over.
Signed-off-by: Jordan Yates <jordan.yates@data61.csiro.au>
The flag --testcase-root has been renamed to --testsuite-root in
15bc98eb50 twister: rename TestCase -> TestSuite
Fix the --help text to reflect the change.
Signed-off-by: Fabio Baltieri <fabiobaltieri@google.com>
Use a path relative to the script itself to determine the ZEPHYR_BASE path
instead of relying on the environment variable.
This allows for modules to use the Zephyr kconfiglib without having
ZEPHYR_BASE set in the environment.
Signed-off-by: Henrik Brix Andersen <henrik@brixandersen.dk>
This commit fixes the broken links to the Contribution Guidelines in
the Compliance Check script.
Signed-off-by: Stephanos Ioannidis <root@stephanos.io>
Serial is not required, in some cases it might be set to null or we
might be using serial_pty, so sort existing map file based on ID
instead.
Fixes#45713
Signed-off-by: Anas Nashif <anas.nashif@intel.com>
For some boards, like up_squared and ehl_crb, they need do a
power off/on operation and use bios to load and execute zephyr
test binary, during the power off, it may meet this SerialException,
but the serial object still works well after power on, we can continue
to get all results from serial, so we don't need to close it,
it's ok to ignore this exception.
BTW, even if we ignore this exception here, it doesn't influence twister
function, because if it was really caused by disconnection, the test would
still be judged as failed due to timeout and serial would be closed later
in main thread, it doesn't impact results.
On the contrary, close serial directly in the monitor serial thread, it
will cause later tests failed due to mismatch errors.
Signed-off-by: Chen Peng1 <peng1.chen@intel.com>
Instead of relying on runtime filter to limit scope to emulation
platforms, use the type attribute for each platform and do the filtering
very early on. This will speed things up for tests where we only run on
emulation platforms.
Signed-off-by: Anas Nashif <anas.nashif@intel.com>
Add dt_chosen_has_compat kconfig helper function. This function checks
if a given `chosen` node has a provided compatible string in its
compatible list. Returns "y" if compatible string is present, and "n"
otherwise.
Signed-off-by: Daniel DeGrasse <daniel.degrasse@nxp.com>
When updating `sys.path` to allow importing the pickled edtlib instance,
add the path to the front of `sys.path`, not the end. This ensures that
the `devicetree.edtlib` module that is imported is the one relative
to the files being run, not some other version which may exist on the
path.
Signed-off-by: Jordan Yates <jordan.yates@data61.csiro.au>
Now that we have partially oerhauled twister in the tree, modify actions
with new options:
- not reporting filtered tests is now default
- output is json, not csv
Signed-off-by: Anas Nashif <anas.nashif@intel.com>
For CI and for the condensed version of the report, consider built tests
as passed and do not mark them as skipped. Tests that are built only
will be reported as skipped in the full and suite reports.
Signed-off-by: Anas Nashif <anas.nashif@intel.com>
Filtered tests are now not being reported by default to allow generation
of reports that are easier to parse and work with in different tools.
The complete filtered set of tests is still available in the json output
for review and verification.
Signed-off-by: Anas Nashif <anas.nashif@intel.com>
This was forgotten from an early overhaul, fix this as
connected_hardware attribute does not exist anymore.
Signed-off-by: Anas Nashif <anas.nashif@intel.com>
We have been generating reports with the platform being used as the
testsuite, ie. all testcases would land under one single big testsuite,
which is very difficult to use or read.
This change uses testsuites as intended and does that for each defined
testsuite (or scenario). Platforms are added as properties among other
things.
Signed-off-by: Anas Nashif <anas.nashif@intel.com>
When loading tests from json file we want to reset the reason for those
tests that are to be run and remove whatever we had when the test was
only built.
Signed-off-by: Anas Nashif <anas.nashif@intel.com>
we mark built only tests as skipped, however, the type was still the
same as other skipped tests, set this to 'built' to indicate the tests
was only built.
Signed-off-by: Anas Nashif <anas.nashif@intel.com>
Remove some legacy code and set status and failure/skip reason directly
without set_state and get_state.
Signed-off-by: Anas Nashif <anas.nashif@intel.com>
Handle build errors due to overflows correctly and do not report those
as failures if not requested.
Signed-off-by: Anas Nashif <anas.nashif@intel.com>
When we have a build failure or anything that would prevent execution of
the test, set the status for all subcases accordignly.
Signed-off-by: Anas Nashif <anas.nashif@intel.com>
twister now exports json instead of csv, so we make the testplan read
the json data and generate the consolidated plan for the workflow.
Signed-off-by: Anas Nashif <anas.nashif@intel.com>
Report misconfigued testsuites and subcase parsing issues instead of
erroring out. This should help us fix those tests and then enforce
accurate result capturing.
Signed-off-by: Anas Nashif <anas.nashif@intel.com>
Remove unused arguments that became obsolete through the refactoring.
- filter_status
- only_failed
- json_report
Signed-off-by: Anas Nashif <anas.nashif@intel.com>
Capture failures per testcase so that we can include the output in
reports for only failed testcases and not include the whole log which
can be overwhelming.
Signed-off-by: Anas Nashif <anas.nashif@intel.com>
Removed tests for dropped functionality.
Renamed classes and files to reflect new structure.
As we refactor the reporting infrastructure, tests will follow once we
have everything in place.
Signed-off-by: Anas Nashif <anas.nashif@intel.com>
Some complex testsuite use macros and other mean to define testcases
which makes them unparseable. This ends up in a descripency between the
testplan and the test results.
Allow adding the testcases into the yaml and use those instead of
parsing the C files.
Signed-off-by: Anas Nashif <anas.nashif@intel.com>
We are now able to capture individual case duration and can display this
as part of the reports instead of using the overall test instance
execution time.
Signed-off-by: Anas Nashif <anas.nashif@intel.com>
