Remove unused arguments that became obsolete through the refactoring.
- filter_status
- only_failed
- json_report
Signed-off-by: Anas Nashif <anas.nashif@intel.com>
Capture failures per testcase so that we can include the output in
reports for only failed testcases and not include the whole log which
can be overwhelming.
Signed-off-by: Anas Nashif <anas.nashif@intel.com>
Removed tests for dropped functionality.
Renamed classes and files to reflect new structure.
As we refactor the reporting infrastructure, tests will follow once we
have everything in place.
Signed-off-by: Anas Nashif <anas.nashif@intel.com>
Some complex testsuite use macros and other mean to define testcases
which makes them unparseable. This ends up in a descripency between the
testplan and the test results.
Allow adding the testcases into the yaml and use those instead of
parsing the C files.
Signed-off-by: Anas Nashif <anas.nashif@intel.com>
We are now able to capture individual case duration and can display this
as part of the reports instead of using the overall test instance
execution time.
Signed-off-by: Anas Nashif <anas.nashif@intel.com>
Now we can add the TestCase class which will hold all information about
individual testcases, their status and why they are not passing, etc.
Test cases are parsed initially and testsuite is populated with the list
of testcases it contsins. This list is duplicate for each instance
(scenario) and used to capture test case data.
Signed-off-by: Anas Nashif <anas.nashif@intel.com>
More renaming and change to variable names to make it less confusing.
Especially, renames of tc to become ts where applicable.
Signed-off-by: Anas Nashif <anas.nashif@intel.com>
TestCase in the old terminology is the test application, but we have
just freed up TestSuite, so use TestSuite here to signify the test
application which consists of many test scenarios and testcases.
Signed-off-by: Anas Nashif <anas.nashif@intel.com>
Change terminology and fix usage based on the current hierarchy. A
testsuite is each test application we have in the tree. The top class is
where we detect all the test suites and create the testplan to be run
later.
Content of testplan will have to split out, so it only deals with the
plan and not the overall execution of tests. This step will be done at a
later point.
Signed-off-by: Anas Nashif <anas.nashif@intel.com>
We now have all the logic for generating data suitable for reports in
one single place (json). The junit reports and in the future other
reports will be using this as input and generate reports using the json
data.
Signed-off-by: Anas Nashif <anas.nashif@intel.com>
An obsolete feature that is no longer being used due to the amount of
data that is needed per release and the fact it becomes stale very fast.
Instead, point to files directly for comparison.
Signed-off-by: Anas Nashif <anas.nashif@intel.com>
Drop CSV generation which was the standard for sharing information since
the early days and instead use json with more content to share
information across sessions and for report generation.
Signed-off-by: Anas Nashif <anas.nashif@intel.com>
Refine structure and accomodate more data, this report will serve as the
source for a testplan, reports and retries.
Signed-off-by: Anas Nashif <anas.nashif@intel.com>
Specify why something was skipped (either filtered or through a ztest
case) and do not report testcases that are only being built as passed,
mark those tests as skipped with the reason specified as "Build only".
Keeping --no-skipped-report for now, although it has no impact.
Signed-off-by: Anas Nashif <anas.nashif@intel.com>
The hardware map feature can be used with serial pty mode in
this change. And also add an runner_params option for passing
more parameters from hardware map file to west runner. Note
that you need to create this map file manually because it
cannot be scanned out correctly due to pty is not a physical
HW device existing in system.
And also update doc/develop/test/twister.rst for the usage.
Signed-off-by: Enjia Mai <enjia.mai@intel.com>
Twister allows us to control maximum execution time for each
test with timeout value in test .yaml configuration. This
helps us to prevent slow tests from stopping by timeout.
However it's hard to choose test timeout for all platforms
as some platforms are naturally slow. It could be a HW board with
power-efficient but slow CPU or simulation platform which
can perform instruction accurate simulation but does it slowly.
As we don't want to increase test timeout infinitely to meet
the needs of the slowest platform let's introduce
platform-specific test timeout management. It's implemented as
an optional 'timeout_multiplier' field in board .yaml
configuration. Setting it to some value multiplies each test
timeout by this value. By that we can increase timeouts only
for the platforms where it's required.
Signed-off-by: Eugeniy Paltsev <Eugeniy.Paltsev@synopsys.com>
Signed-off-by: Evgeniy Paltsev <PaltsevEvgeniy@gmail.com>
A unique run_id is now also added to the twister.csv file. This
file is used to store meta-data, in particular for `--test-only`
option. The run_id is then reused in `--test-only` calls to verify
if the id from readout matches the expected one.
fixes: #45278
Signed-off-by: Maciej Perkowski <Maciej.Perkowski@nordicsemi.no>
Support new keyword in the testcase.yaml to allow filtering based on
enabled modules. This is useful when you only whitelist a limited set of
modules in your project and want to only run those samples that have
their dependencies satisfied and those modules available.
The option allows adding multiple modules, all of which need to be
enabled in the manifest.
Signed-off-by: Anas Nashif <anas.nashif@intel.com>
Set the same encoding for collected CMake output and file where this
output will be saved.
Signed-off-by: Piotr Golyzniak <piotr.golyzniak@nordicsemi.no>
Set run id (md5 of instance and a random number) for each test and match
what was set during build with the output on the screen. This will
verify that we are evaluating the same test we have uploaded and not
some previous output on the screen.
Signed-off-by: Anas Nashif <anas.nashif@intel.com>
Decision about whether test should be considered as ztest should be made
after detection ztest testcase - not after detection ztest test suite.
Fixes: #44397
Signed-off-by: Piotr Golyzniak <piotr.golyzniak@nordicsemi.no>
New ztest API is available in those places:
subsys/testsuite/ztest/src/ztest_new.c
subsys/testsuite/ztest/include/ztest_test_new.h
This requires also changes in Twister code during parsing c files to
find properly test suites and testcases.
Signed-off-by: Piotr Golyzniak <piotr.golyzniak@nordicsemi.no>
Extend scope of searching src directory with defined test source code.
In some cases for optimization reason src direcotry is placed in parent
directory (in relation to testcase.yaml file placement). This changes
allow to detect such situations.
Signed-off-by: Piotr Golyzniak <piotr.golyzniak@nordicsemi.no>
Test wirtten in ztest framework should print in output information about
test suite name defined in test's source code (as a first argument of
ztest_test_suite() function). This changes make it possible to find such
test names in c files. Next they are used during test execution to
verify if performed test was performed properly (verification can base
not only on existance of "PROJECT EXECUTION SUCCESSFUL" info).
Signed-off-by: Piotr Golyzniak <piotr.golyzniak@nordicsemi.no>
Modify serial and serial_pty option availability verification in DTU
object in device_is_available method to avoid situation when empty string
passed as serial in hardware_map.yaml cause hang up Twister in this
place.
Fixes: #41169
Signed-off-by: Piotr Golyzniak <piotr.golyzniak@nordicsemi.no>
falures found during valgrind execution were treated as normal test
failures, so the logger would display a link to handler.log and not to
valgrind.log, leading to some confusion.
the return code is now handled in a correct sequence.
Signed-off-by: Mikkel Jakobsen <mikkel.aunsbjerg@escolifesciences.com>
if --enable-valgrind is selected but no valgrind executable can be
found in PATH, exit with an error
Signed-off-by: Mikkel Jakobsen <mikkel.aunsbjerg@escolifesciences.com>
Allow retries on build errors, this is now done using
--retry-build-errors. This option is useful when for example build
failures are caused by licensing issues or intermittent network issues.
Signed-off-by: Anas Nashif <anas.nashif@intel.com>
Do not wait till the end to update counters, do the counting at
realitime and reuse data for on-screen reporting.
Add a counter summary function for debugging.
This patch changes how we count and report skipped tests and the total
now has the skipped tests included.
Signed-off-by: Anas Nashif <anas.nashif@intel.com>
Adds operation "gather_footprint" after the build stage
with minor refactoring. Before footprints were collected
only at the very end of the twister workflow
and there were situations were elf files were not available any more
(for qemu and --runtime-artifact-cleanup used).
Signed-off-by: Maciej Perkowski <Maciej.Perkowski@nordicsemi.no>
Git describe should work in eg. shallow clones when there are no tags.
Has been fixed in cmake previously, but not in twister.
Signed-off-by: Andreas Pettersson <andreaspettersson95@gmail.com>
When running twister with --only-failed option, based on last xml
result, the new skipped count should be the last skipped count
plus current skipped count.
Signed-off-by: Chen Peng1 <peng1.chen@intel.com>
Passing the zephyr build directory to the script run by
--device-serial-pty, it makes the feature of the device serial
pty more flexible.
Signed-off-by: Enjia Mai <enjia.mai@intel.com>
With --no-skipped-report, twister will skip over all tests marked as
skipped (filtered) in the junit report.
This is useful for CI where have 1000s of filtered tests that appear in
the report and in some cases cause tools parsing the output to fail or
provide incomplete results.
Fixes#38179
Signed-off-by: Anas Nashif <anas.nashif@intel.com>
Commit 28b2e55321 introduced a regression
into twister workflow, breaking memory footprint calculation.
The commit changed name of produced .elf files.
There is no more zephyr_prebuilt.elf and instead we have
zephyr_pre0.elf and zephyr_pre1.elf. This commit fixes a filter
telling which files to exclude when calculating memory footprints,
to match the current status.
Signed-off-by: Maciej Perkowski <Maciej.Perkowski@nordicsemi.no>
Sometimes custom script fails, but we can't see the
error information in twister log, because the twister
uses _ to ignore the stderr.
Signed-off-by: Jingru Wang <jingru@synopsys.com>
