When pytest scenario fails, then 'handler.log' is printed.
Changed to print 'twister_harness.log' that is a log from
pytest execution. That file tells much more when test fails.
Signed-off-by: Grzegorz Chwierut <grzegorz.chwierut@nordicsemi.no>
Apply a workaround for an issue introduced with bc8b7dd
for code coverage collection at qemu_x86 when gcov data
was broken because of optimized out memcpy() in write_u64()
static inline function, so gcovr parser fails with
"gcovr.gcov_parser.NegativeHits: Got negative hit value
in gcov line 'branch 1 taken -1'".
Add gcovr command line option to ignore such errors (since v.6.0):
"--gcov-ignore-parse-errors=negative_hits.warn_once_per_file"
Signed-off-by: Dmitrii Golovanov <dmitrii.golovanov@intel.com>
Added pytest_dut_scope keyword under harness_config section.
New keyword is used to determine the scope of dut and shell
fixtures in pytest-twister-harness plugin.
Signed-off-by: Grzegorz Chwierut <grzegorz.chwierut@nordicsemi.no>
Do not break processing when incorrect coverage data are received.
Instead, report failures but still process valid data.
Include coverage processing status within twister retrun code.
Signed-off-by: Piotr Kosycarz <piotr.kosycarz@nordicsemi.no>
When coverage is enabled, gather console logs longer
to receive gcov data which are send after test output.
Signed-off-by: Piotr Kosycarz <piotr.kosycarz@nordicsemi.no>
Very verbose log message without too much context that should be a debug
message rather than a info().
Signed-off-by: Anas Nashif <anas.nashif@intel.com>
An urforseen error was introduce with #62713. When "--device-testing"
or "--filter runnable" is used in CLI, twister will skip tests which
cannot be executed (not just built) on a given platform. If a given
platform is among integration platforms it will cause an error.
However, it shouldn't be the case. Such skip is intentional and
shouldn't be an error. Fixed by assigning an existing Filter.CMD_LINE,
which is exempted from being turned to error.
fixes: #63844
Signed-off-by: Maciej Perkowski <Maciej.Perkowski@nordicsemi.no>
Reset TestInstance status when ProjectBuilder.run() is called
to run the test after its successful build, so the build status
will not propagate as the run status if the run fails.
Signed-off-by: Dmitrii Golovanov <dmitrii.golovanov@intel.com>
Fix the Twister Console harness ordered 'multi_line' and 'one_line'
pattern matching to treat the ztest as failed when not all of the
expected patterns were found in the console output, but the ztest
application itself reports 'PROJECT EXECUTION SUCCESSFUL'.
Signed-off-by: Dmitrii Golovanov <dmitrii.golovanov@intel.com>
When using a quarantine file with more than 512 unique
entries, than time of matching quarantine increases
significantly. This is because regexp cache size is
512. Add precompiled regexp entries to the quarantine
as a fix.
Signed-off-by: Grzegorz Chwierut <grzegorz.chwierut@nordicsemi.no>
All handlers except QEMU were right-stripping console lines before
checking them. In certain cases where the regex is expecting the
whitespace, this could make the test fail on qemu when it is passing
on something else. Change QEMUHandler to us .rstrip instead of .strip()
Signed-off-by: Øyvind Rønningstad <oyvind.ronningstad@nordicsemi.no>
In PR #62947 a timeout variable was replaced with a method,
but this timeout was used in another place to start
pytest subprogram. This commit provides a fix.
Signed-off-by: Grzegorz Chwierut <grzegorz.chwierut@nordicsemi.no>
Test suites names are not being handled uniformly for tests not
in zephyr tree. Their names depend on -T arg used in twister's CLI.
The newly added options allow to select if twister should add paths
to suite names. This is needed if test plans are to be used for tests
outside of zephyr tree.
Signed-off-by: Maciej Perkowski <Maciej.Perkowski@nordicsemi.no>
"Path" is a path (relative to zephyr) to a directory with test suite's
definition. Such entry helps to locate a given test.
Signed-off-by: Maciej Perkowski <Maciej.Perkowski@nordicsemi.no>
In test-only mode with a hardware map, all tests with fixtures were
skipped, even when the hardware map has appropriate fixtures defined.
TestPlan.load_from_file had the check for fixtures missing, but actually
best is that the TestInstance checks the fixtures in the hardware map.
Signed-off-by: Manoel Brunnen <manoel.brunnen@ithinx.io>
Twister allows us to control maximum execution time for each test
with timeout value in test's .yaml configuration and
platform level timeout multiplier which allows us to tweak
timeout value for specific platform.
However, sometimes we want to additionally adjust tests timeouts
when running twister. This is especially useful in case of
simulation platform as simulation time may depend on the host
speed & load, we may select different (i.e. cycle accurate but
slower one) simulation method, etc...
Let's introduce global (twister-level) timeout multiplier option.
Signed-off-by: Eugeniy Paltsev <Eugeniy.Paltsev@synopsys.com>
Signed-off-by: Evgeniy Paltsev <PaltsevEvgeniy@gmail.com>
Currently, when a Robot test is run, the test case status value is
reported as `None`, which causes the XML to contain information about
the test being skipped due to misconfiguration.
This commit fixes the value reported in Twister XML test result by
assigning the test result to the `status` variable of a testcase
instance after the test has finished running.
Signed-off-by: Franciszek Pindel <fpindel@internships.antmicro.com>
Signed-off-by: Piotr Zierhoffer <pzierhoffer@antmicro.com>
This is coming from devicetree and corrosponds to what we have in the
dts/bindings/vendor-prefixes.txt file.
This will allow for static filtering, especially with twister, i.e. no
need to build anything to know the vendor of a board
All of the vendor data was extracted automatically from the devicetree,
so some platforms might not have the right vendor or no vendor at all
right now, we need to fix some of the DTS information or do this
manually to get this 100% correct. But we are close.
Signed-off-by: Anas Nashif <anas.nashif@intel.com>
Add a new option --vendor which allows filtering by vendors tracked in
the board/platform yaml file. The vendor string is compatible with DTS
and is what we have in dts/bindings/vendor-prefixes.txt.
Providing multiple vendors is also supported.
Signed-off-by: Anas Nashif <anas.nashif@intel.com>
Skips are turned to errors on integration platforms. However, if
a user (or a CI) selects toolchain other than zephyr, unexpected
errors due to skips can appear. With this commit skips due to
toolchain incompatibilities are not treated as errors.
Signed-off-by: Maciej Perkowski <Maciej.Perkowski@nordicsemi.no>
If coverage is provided by a different platform not in the integration
list, it should not be reported as error.
Signed-off-by: Anas Nashif <anas.nashif@intel.com>
Remove dead and erroneous (DUT is not a dict) if branch. Simply use the
self.options.platform list. Which is filled by either the hardware map
or the '--platform' option.
This change addresses issue #62723 and #62560.
Signed-off-by: Manoel Brunnen <manoel.brunnen@ithinx.io>
Before only with --integration such skips were changed to errors.
This hinders issues when twister is call with --all flag in CI
and new skips on integration platforms are not caught. Then those
skips cause errors in other PRs when --integration was used.
Signed-off-by: Maciej Perkowski <Maciej.Perkowski@nordicsemi.no>
The value of emulation_platforms is a bool corresponding to
--emulation-only CLI option. Therefore, it should be registered
as Filters.CMD_LINE. Without it, twister raises errors when an
integration platform is descoped due to this option.
Signed-off-by: Maciej Perkowski <Maciej.Perkowski@nordicsemi.no>
Added an exception to scl.py module, to not process an empty
yaml file, because of error from pykwalify. New exception
is handled when processing quarantine files, to allow
an empty file.
Signed-off-by: Grzegorz Chwierut <grzegorz.chwierut@nordicsemi.no>
Simple refactoring aiming to reduce the average method length.
Minor corrections of the handlers module, removing wholly unused
variables, etc.
Signed-off-by: Lukasz Mrugala <lukaszx.mrugala@intel.com>
Allow to specify a list of pytest directories, files or subtests
with pytest_root keyword in test yaml.
Signed-off-by: Grzegorz Chwierut <grzegorz.chwierut@nordicsemi.no>
Previously, only the main src/ directories was scanned for
ZTest testsuite names and tests. This allows us to place
test sources in subdirectories as well.
Signed-off-by: Christopher Friedt <cfriedt@meta.com>
Add support handling quotes for configuration options in extra args by
escaping them properly instead of removing the quotes altogether. For
other options in extra_args quotes are removes as usual.
Add similar support in west build command also.
Add a unit test to check this functionality.
Signed-off-by: Chaitanya Tata <Chaitanya.Tata@nordicsemi.no>
Remove log.py and by this logging configuration from
pytest-harness-plugin. Instead use pytest options for setting verbosity
and format of printed logs. Thanks to this management of verbosity
level can be easier for users.
Signed-off-by: Piotr Golyzniak <piotr.golyzniak@nordicsemi.no>
Select and place common code of three adapters (HardwareAdapter,
BinaryAdapterBase and QemuAdapter) into basic DeviceAdapter class.
Introduce new way of reading device output by run separate thread which
try to read device output and place it into internal python queue.
Thanks to this, now it is possible to create readline method for all
adapters, which can be unblock when timeout occur.
Collect all common steps which have to be done before setup device in
launch method. The same was done for teardown operations which were
placed into close method.
Additionally some protection mechanisms were introduced to prevent for
undesirable side-effects when user could try to launch to already
launched device or try to send some data to disconnected device.
iter_stdout method was replaced by two new methods: readline and
readlines. To make it possible to remove all read output from internal
buffer (queue), clear_buffer method was introduced.
Also unit tests were rewritten to work properly with current version
of adapters.
Signed-off-by: Piotr Golyzniak <piotr.golyzniak@nordicsemi.no>
Add possibility of passing timeouts from Twister to pytest and unify
various timeouts used in adapters to point to one main timeout.
Signed-off-by: Piotr Golyzniak <piotr.golyzniak@nordicsemi.no>
When parsing test names from symbol names, use a non-greedy match to
avoid parsing test names with __ in them.
The symbol looks like z_ztest_unit_test__SUITE__TEST. It is more likely
that a test name will contain __ than a suite name, so now something
z_ztest_unit_test__a_b__c_d__e_f will be parsed as suite:a_b
test:c_d__e_f, whereas it used to be parsed as suite:a_b__c_d test:e_f
Signed-off-by: Jeremy Bettis <jbettis@google.com>
If a test doesn't start with test_, twister currently emits an error:
suite_name.test_name: Unknown status 'started'
Following the suggestion by Maksim Masalski on PR#21191, change the
regex to exclude spaces in the test name, and remove the check that the
test_ prefix exists.
See also #21162
Signed-off-by: Jeremy Bettis <jbettis@google.com>
While the code already includes some provisions to allow running outside
of a git checkout, attempting that right now causes a failure to generate
the XML report, as the JSON report creates a null `zephyr_version` field.
The reason for that is that the original code doesn't set `self.version`
when the subprocess returns a non-zero status (and also doesn't log.)
Instead of having to set `self.version` for all failure cases, initialize
it to the failure state, and log if it hasn't changed from that.
Signed-off-by: Diego Elio Pettenò <flameeyes@meta.com>
When using the hardware map and testing on device, print out the name of
the dut (from the hardware map) on the script and report it in the json
output as well.
Signed-off-by: Anas Nashif <anas.nashif@intel.com>
If size is given then twister prnits the size information for the ELF
but returns one which if used in automation fails the command.
Return zero as the command is successful. Also, fix the unit test case.
Signed-off-by: Chaitanya Tata <Chaitanya.Tata@nordicsemi.no>
Let's not provide default toolchains for 'arc' intentionally: some
targets can't be built with GNU toolchain ("zephyr", "cross-compile",
"xtools" options) and for some targets we haven't provided MWDT
compiler / linker options in corresponding SoC file in Zephyr, so
these targets can't be built with ARC MWDT toolchain
("arcmwdt" option) by Zephyr build system.
Instead for 'arc' we rely on 'toolchain' option in board yaml
configuration.
Signed-off-by: Eugeniy Paltsev <Eugeniy.Paltsev@synopsys.com>
Signed-off-by: Evgeniy Paltsev <PaltsevEvgeniy@gmail.com>
