Remove some legacy code and set status and failure/skip reason directly
without set_state and get_state.
Signed-off-by: Anas Nashif <anas.nashif@intel.com>
Handle build errors due to overflows correctly and do not report those
as failures if not requested.
Signed-off-by: Anas Nashif <anas.nashif@intel.com>
When we have a build failure or anything that would prevent execution of
the test, set the status for all subcases accordignly.
Signed-off-by: Anas Nashif <anas.nashif@intel.com>
twister now exports json instead of csv, so we make the testplan read
the json data and generate the consolidated plan for the workflow.
Signed-off-by: Anas Nashif <anas.nashif@intel.com>
Report misconfigued testsuites and subcase parsing issues instead of
erroring out. This should help us fix those tests and then enforce
accurate result capturing.
Signed-off-by: Anas Nashif <anas.nashif@intel.com>
Remove unused arguments that became obsolete through the refactoring.
- filter_status
- only_failed
- json_report
Signed-off-by: Anas Nashif <anas.nashif@intel.com>
Capture failures per testcase so that we can include the output in
reports for only failed testcases and not include the whole log which
can be overwhelming.
Signed-off-by: Anas Nashif <anas.nashif@intel.com>
Removed tests for dropped functionality.
Renamed classes and files to reflect new structure.
As we refactor the reporting infrastructure, tests will follow once we
have everything in place.
Signed-off-by: Anas Nashif <anas.nashif@intel.com>
Some complex testsuite use macros and other mean to define testcases
which makes them unparseable. This ends up in a descripency between the
testplan and the test results.
Allow adding the testcases into the yaml and use those instead of
parsing the C files.
Signed-off-by: Anas Nashif <anas.nashif@intel.com>
We are now able to capture individual case duration and can display this
as part of the reports instead of using the overall test instance
execution time.
Signed-off-by: Anas Nashif <anas.nashif@intel.com>
Now we can add the TestCase class which will hold all information about
individual testcases, their status and why they are not passing, etc.
Test cases are parsed initially and testsuite is populated with the list
of testcases it contsins. This list is duplicate for each instance
(scenario) and used to capture test case data.
Signed-off-by: Anas Nashif <anas.nashif@intel.com>
More renaming and change to variable names to make it less confusing.
Especially, renames of tc to become ts where applicable.
Signed-off-by: Anas Nashif <anas.nashif@intel.com>
TestCase in the old terminology is the test application, but we have
just freed up TestSuite, so use TestSuite here to signify the test
application which consists of many test scenarios and testcases.
Signed-off-by: Anas Nashif <anas.nashif@intel.com>
Change terminology and fix usage based on the current hierarchy. A
testsuite is each test application we have in the tree. The top class is
where we detect all the test suites and create the testplan to be run
later.
Content of testplan will have to split out, so it only deals with the
plan and not the overall execution of tests. This step will be done at a
later point.
Signed-off-by: Anas Nashif <anas.nashif@intel.com>
We now have all the logic for generating data suitable for reports in
one single place (json). The junit reports and in the future other
reports will be using this as input and generate reports using the json
data.
Signed-off-by: Anas Nashif <anas.nashif@intel.com>
An obsolete feature that is no longer being used due to the amount of
data that is needed per release and the fact it becomes stale very fast.
Instead, point to files directly for comparison.
Signed-off-by: Anas Nashif <anas.nashif@intel.com>
Drop CSV generation which was the standard for sharing information since
the early days and instead use json with more content to share
information across sessions and for report generation.
Signed-off-by: Anas Nashif <anas.nashif@intel.com>
Refine structure and accomodate more data, this report will serve as the
source for a testplan, reports and retries.
Signed-off-by: Anas Nashif <anas.nashif@intel.com>
Specify why something was skipped (either filtered or through a ztest
case) and do not report testcases that are only being built as passed,
mark those tests as skipped with the reason specified as "Build only".
Keeping --no-skipped-report for now, although it has no impact.
Signed-off-by: Anas Nashif <anas.nashif@intel.com>
Many option cancel out very already on,. so lets do this as part of the
option processing function first before we create workspace.
Signed-off-by: Anas Nashif <anas.nashif@intel.com>
Cleanup main() and move logging code into own function.
This reduces clutter in main function and simplifies workflow.
Signed-off-by: Anas Nashif <anas.nashif@intel.com>
Adding LOG_BACKEND_MOCK_OUTPUT_DEFAULT &
LOG_BACKEND_MOCK_OUTPUT_SYST to UNDEF_KCONFIG_WHITELIST as Kconfig
options generated by a template under testcases are not
identified by Compliance check by default.
Signed-off-by: Aastha Grover <aastha.grover@intel.com>
The hardware map feature can be used with serial pty mode in
this change. And also add an runner_params option for passing
more parameters from hardware map file to west runner. Note
that you need to create this map file manually because it
cannot be scanned out correctly due to pty is not a physical
HW device existing in system.
And also update doc/develop/test/twister.rst for the usage.
Signed-off-by: Enjia Mai <enjia.mai@intel.com>
All includes are now prefixed with <zephyr/...>, even though the old
include paths can still be used when CONFIG_LEGACY_INCLUDE_PATH=y, an
option still enabled by default. Migrating large projects can be tedious
and time consuming. This patch provides a script that can be used to
migrate any Zephyr-based project to the new include path. It is used
like this:
python $ZEPHYR_BASE/scripts/utils/migrate_includes.py \
-p path/to/zephyr-based-project
Signed-off-by: Gerard Marull-Paretas <gerard.marull@nordicsemi.no>
Adjust get_mem_region to not return region when address == end
as there will be nothing to read there. Also, a subsequent region
may have that address as a start address and would be a more appropriate
selection.
Signed-off-by: Mark Holden <mholden@fb.com>
Add a runner to "flash" and "debug" Cyclone V SoC FPGA Development Kit
the runner is based on OpenOCD and GDB
Signed-off-by: Esteban Valverde <esteban.valverde.vega@intel.com>
Twister should be partially usable even without West installed, however,
Twister unconditionally tries to import it when initializing to check
for west projects and crashes if it is not there.
This PR changes the behavior to return None if the west imports failed,
which appears to be more in line with what was intended. This allows
Twister to proceed and successfully run tests in a non-west environment.
There is no impact on environments that do have west installed.
Fixes#45355
Signed-off-by: Tristan Honscheid <honscheid@google.com>
This updates the MIPI Sys-T collateral generation to include log
modules in the XML file. This allows the parser to show where
each log message is coming from.
Note that the addition of Name propety in sys:Client is to clarify
that this collateral is from Zephyr.
Also note he addition of Mask property in syst:Guid is for parser
to match this collateral to any incoming messages, as the module
IDs are encoded in the set bits in the GUID. Without the mask,
the parser will discard any messages with non-zero module IDs.
This is based on the observation of the reference parser,
systprint, where pseudo GUIDs are generated from the module/unit
IDs (origin unit) in the incoming messages for matching with
collaterals.
Signed-off-by: Daniel Leung <daniel.leung@intel.com>
Update the script so that it handles generates files using the
<zephyr/...> include prefix.
Signed-off-by: Gerard Marull-Paretas <gerard.marull@nordicsemi.no>
Update the script so that it handles generates files using the
<zephyr/...> include prefix.
Signed-off-by: Gerard Marull-Paretas <gerard.marull@nordicsemi.no>
When doing global changes, like typo fixes or header changes, keep
default scope and do not build each test for each platform.
Signed-off-by: Anas Nashif <anas.nashif@intel.com>
Twister allows us to control maximum execution time for each
test with timeout value in test .yaml configuration. This
helps us to prevent slow tests from stopping by timeout.
However it's hard to choose test timeout for all platforms
as some platforms are naturally slow. It could be a HW board with
power-efficient but slow CPU or simulation platform which
can perform instruction accurate simulation but does it slowly.
As we don't want to increase test timeout infinitely to meet
the needs of the slowest platform let's introduce
platform-specific test timeout management. It's implemented as
an optional 'timeout_multiplier' field in board .yaml
configuration. Setting it to some value multiplies each test
timeout by this value. By that we can increase timeouts only
for the platforms where it's required.
Signed-off-by: Eugeniy Paltsev <Eugeniy.Paltsev@synopsys.com>
Signed-off-by: Evgeniy Paltsev <PaltsevEvgeniy@gmail.com>
CSV file is expanded to store run_id and the corresponding test
requires a modification.
Signed-off-by: Maciej Perkowski <Maciej.Perkowski@nordicsemi.no>
