Fixtures in pytest-twister-harness plugin are moved to one
file to simplify adding new fixtures in the future - no
need to add pytest_plugins entry and register asserts.
Moved also shell fixture from sample dir, because that
fixture can be reused in new tests.
Signed-off-by: Grzegorz Chwierut <grzegorz.chwierut@nordicsemi.no>
If pylib twister harness reads bytes that can't be decoded to Unicode,
it would raise an exception at a separate "serial reader" process, that
would fail without any warning.
This patch uses "errors='replace'" parameter for Python str.decode()
method, so that this doesn't happen.
Signed-off-by: Ederson de Souza <ederson.desouza@intel.com>
Previously, only the main src/ directories was scanned for
ZTest testsuite names and tests. This allows us to place
test sources in subdirectories as well.
Signed-off-by: Christopher Friedt <cfriedt@meta.com>
Add support handling quotes for configuration options in extra args by
escaping them properly instead of removing the quotes altogether. For
other options in extra_args quotes are removes as usual.
Add similar support in west build command also.
Add a unit test to check this functionality.
Signed-off-by: Chaitanya Tata <Chaitanya.Tata@nordicsemi.no>
Add tests which use pytester to call pytest with pytest-twister-harness
plugin on previously defined test. This gives a possibility to test
plugin in end-to-end variant.
Signed-off-by: Piotr Golyzniak <piotr.golyzniak@nordicsemi.no>
According to new adapters API add new tests which verify their
correctness. Introduce some minor naming changes to describe better
their role.
Signed-off-by: Piotr Golyzniak <piotr.golyzniak@nordicsemi.no>
Add better description of available options and add actions which
should help to prevent from passing invalid options to plugin.
Signed-off-by: Piotr Golyzniak <piotr.golyzniak@nordicsemi.no>
Remove log.py and by this logging configuration from
pytest-harness-plugin. Instead use pytest options for setting verbosity
and format of printed logs. Thanks to this management of verbosity
level can be easier for users.
Signed-off-by: Piotr Golyzniak <piotr.golyzniak@nordicsemi.no>
Create helper class Shell which can be used in tests which require
interaction with shell application.
Signed-off-by: Piotr Golyzniak <piotr.golyzniak@nordicsemi.no>
Move helper.py file content to utils.py file in device directory,
where those helper functions are used.
Signed-off-by: Piotr Golyzniak <piotr.golyzniak@nordicsemi.no>
Divide dut fixture into three smaller fixtures to make it easier
to user to change dut fixture scope or to write her/his own dut
fixture. Add initialize_log_files method to make it possible to
initialize log files per each test.
Signed-off-by: Piotr Golyzniak <piotr.golyzniak@nordicsemi.no>
Pass pytest test name to handler.log header to make it easier to
analyze handler.log content.
Signed-off-by: Piotr Golyzniak <piotr.golyzniak@nordicsemi.no>
Add readlines_until method to DeviceAdapter class to simplify using
adapters in tests by give user possibility to read output from device
until following conditions:
- provided regex was found
- provided number of lines was already read
- timeout was occurred
Signed-off-by: Piotr Golyzniak <piotr.golyzniak@nordicsemi.no>
Introduce changes into FifoHandler class to make it more safe
(especially to avoid blocking all program when FIFO queue is not opened
by other application).
Signed-off-by: Piotr Golyzniak <piotr.golyzniak@nordicsemi.no>
Select and place common code of three adapters (HardwareAdapter,
BinaryAdapterBase and QemuAdapter) into basic DeviceAdapter class.
Introduce new way of reading device output by run separate thread which
try to read device output and place it into internal python queue.
Thanks to this, now it is possible to create readline method for all
adapters, which can be unblock when timeout occur.
Collect all common steps which have to be done before setup device in
launch method. The same was done for teardown operations which were
placed into close method.
Additionally some protection mechanisms were introduced to prevent for
undesirable side-effects when user could try to launch to already
launched device or try to send some data to disconnected device.
iter_stdout method was replaced by two new methods: readline and
readlines. To make it possible to remove all read output from internal
buffer (queue), clear_buffer method was introduced.
Also unit tests were rewritten to work properly with current version
of adapters.
Signed-off-by: Piotr Golyzniak <piotr.golyzniak@nordicsemi.no>
Change name of SimulatorAdapterBase class to more meaningful like
BinaryAdapterBase. This changes is also made to be more align with
Twister terminology.
Signed-off-by: Piotr Golyzniak <piotr.golyzniak@nordicsemi.no>
Add possibility of passing timeouts from Twister to pytest and unify
various timeouts used in adapters to point to one main timeout.
Signed-off-by: Piotr Golyzniak <piotr.golyzniak@nordicsemi.no>
PytestAssertRewriteWarning is raised when pytest-twister-harness
plugin is called from twister. Call register method to get rid
of that warning
Signed-off-by: Grzegorz Chwierut <grzegorz.chwierut@nordicsemi.no>
When parsing test names from symbol names, use a non-greedy match to
avoid parsing test names with __ in them.
The symbol looks like z_ztest_unit_test__SUITE__TEST. It is more likely
that a test name will contain __ than a suite name, so now something
z_ztest_unit_test__a_b__c_d__e_f will be parsed as suite:a_b
test:c_d__e_f, whereas it used to be parsed as suite:a_b__c_d test:e_f
Signed-off-by: Jeremy Bettis <jbettis@google.com>
If a test doesn't start with test_, twister currently emits an error:
suite_name.test_name: Unknown status 'started'
Following the suggestion by Maksim Masalski on PR#21191, change the
regex to exclude spaces in the test name, and remove the check that the
test_ prefix exists.
See also #21162
Signed-off-by: Jeremy Bettis <jbettis@google.com>
While the code already includes some provisions to allow running outside
of a git checkout, attempting that right now causes a failure to generate
the XML report, as the JSON report creates a null `zephyr_version` field.
The reason for that is that the original code doesn't set `self.version`
when the subprocess returns a non-zero status (and also doesn't log.)
Instead of having to set `self.version` for all failure cases, initialize
it to the failure state, and log if it hasn't changed from that.
Signed-off-by: Diego Elio Pettenò <flameeyes@meta.com>
When using the hardware map and testing on device, print out the name of
the dut (from the hardware map) on the script and report it in the json
output as well.
Signed-off-by: Anas Nashif <anas.nashif@intel.com>
If size is given then twister prnits the size information for the ELF
but returns one which if used in automation fails the command.
Return zero as the command is successful. Also, fix the unit test case.
Signed-off-by: Chaitanya Tata <Chaitanya.Tata@nordicsemi.no>
Let's not provide default toolchains for 'arc' intentionally: some
targets can't be built with GNU toolchain ("zephyr", "cross-compile",
"xtools" options) and for some targets we haven't provided MWDT
compiler / linker options in corresponding SoC file in Zephyr, so
these targets can't be built with ARC MWDT toolchain
("arcmwdt" option) by Zephyr build system.
Instead for 'arc' we rely on 'toolchain' option in board yaml
configuration.
Signed-off-by: Eugeniy Paltsev <Eugeniy.Paltsev@synopsys.com>
Signed-off-by: Evgeniy Paltsev <PaltsevEvgeniy@gmail.com>
Enable the possibility for boards to implement a custom `run` target in
its board.cmake to run any arbitrary commands. This is helpful for devs
who would like to add support for proprietary simulator to their boards
that can't be upstreamed.
Signed-off-by: Yong Cong Sin <ycsin@meta.com>
We have been dealing with missing and multiple binaries the same way and
both would result in a build error, which is not accurate. multiple
binaries in the build directory are fine, we just need to pick the right
one for parsing.
If we get no binaries, raise an exception and report failure, however,
if we have multiple binaries, filter intermediate artifacts out and
parse what remains.
qemu binaries generated after a run are also being filtered here. Those
are not build artificats and appear only after running in qemu. However
they have been causing issues on retries.
Signed-off-by: Anas Nashif <anas.nashif@intel.com>
When no/too many elf files are detected after a build a BuildError
exception is raised. However, it was not being counted as an issue
with a test. With the patch satuses of tests' with such exception
are reported as errors. Whithout it, twister finished without
reported errors and was getting green CI checks.
Signed-off-by: Maciej Perkowski <Maciej.Perkowski@nordicsemi.no>
When using twister to generate coverage with the coverage tool lcov,
allow using --coverage-formats to pick if you want lcov or html,
defaulting to both. Picking html will also use lcov, since that is
required for geninfo. This will allow callers to avoid the potentially
slow and disk intensive html reports if they only wanted the lcov info
file.
Signed-off-by: Jeremy Bettis <jbettis@google.com>
Fixes Issue #60522 by guarding member access against None parameters.
Shortens the init by using the if...else assignment.
Fixes two typos.
Signed-off-by: Lukasz Mrugala <lukaszx.mrugala@intel.com>
Fix `--device-flash-with-test` command line options check logic
to be effective with `--device-testing`.
Remove `--device-flash-timeout` check for `--device-testing`
presence as marginal.
Signed-off-by: Dmitrii Golovanov <dmitrii.golovanov@intel.com>
hardware was referenced as return value, without being declared. When the
program goes into the 'except' path, hardware is unbound, but returned,
which throws an error:
`UnboundLocalError`: local variable 'hardware' referenced before assignment
Signed-off-by: Manoel Brunnen <manoel.brunnen@ithinx.io>
Extended parsing of report.xml file - produced by pytest-twister-harness
plugin. Now testcases are properly extracted and added to twister
report. Added unit tests.
Signed-off-by: Grzegorz Chwierut <grzegorz.chwierut@nordicsemi.no>
We have many tests that are marked as slow that do not run in CI and any
known workflow, to isolate them and be able to run them on their own,
add a new option --enable-slow-only which causes only those tests to build
and execute.
Signed-off-by: Anas Nashif <anas.nashif@intel.com>
Add DeviceAbstract class to default imports from pytest-twister-harness
package to simplify importing DUT package, when creating tests.
Signed-off-by: Grzegorz Chwierut <grzegorz.chwierut@nordicsemi.no>
Use the natural sort of list when generating a hardware map. The
list is sorted with a serial port as a key. When more than 10 ports
are active and some of devices use more than one port, the ports
of one device may be listed in wrong sequence, which may cause
futher problems with selecting the right port for listening to
the device.
Signed-off-by: Grzegorz Chwierut <grzegorz.chwierut@nordicsemi.no>
Add protection timeout for pytest subprocess, to avoid situation of
suspending whole Twister in case of internal pytest test problem.
Co-authored-by: Grzegorz Chwierut <grzegorz.chwierut@nordicsemi.no>
Signed-off-by: Piotr Golyzniak <piotr.golyzniak@nordicsemi.no>
Added waiting for prompt instead of calling time.sleep,
to fix issue with longer setting up platforms with TF-M
enabled. Wrapped iter_stdout generator, to avoid issues
when generator is created and called multiple times
in test scenarios.
Fixes: #58747
Signed-off-by: Grzegorz Chwierut <grzegorz.chwierut@nordicsemi.no>
When calling twister with --coverage and no platform specified on the
command line, set the the value of coverage platforms to an empty list.
Fixes#57534
Signed-off-by: Anas Nashif <anas.nashif@intel.com>
testinstance.py contains a string literal comparison to the
word 'platform', which has been misspelled as 'plaform'.
The change is verified by a test expansion.
Signed-off-by: Lukasz Mrugala <lukaszx.mrugala@intel.com>
Making the necessary changes to enable the new pytest plugin.
By default Twister should work without the pytest-twister-harness
plugin installed. To achieve this, each time Twister calls pytest,
the PYTHONPATH environment variable is expanded and the
`-p twister_harness.plugin` option is added to the pytest command.
Co-authored-by: Piotr Golyzniak <piotr.golyzniak@nordicsemi.no>
Signed-off-by: Grzegorz Chwierut <grzegorz.chwierut@nordicsemi.no>
Adding pytest plugin dedicated to running pytest tests in Zephyr
project. This plugin provides a dut fixture which allows to handle
bidirectional communication with the device under test. This version
of plugin can be used for tests dedicated to real hardware, QEMU and
native_posix simulator.
Co-authored-by: Lukasz Fundakowski <lukasz.fundakowski@nordicsemi.no>
Co-authored-by: Grzegorz Chwierut <grzegorz.chwierut@nordicsemi.no>
Co-authored-by: Katarzyna Giadla <katarzyna.giadla@nordicsemi.no>
Signed-off-by: Piotr Golyzniak <piotr.golyzniak@nordicsemi.no>
