When something goes wrong with execution, due to twister issues and
bugs, do not continue with execution and abort the process and report
and return an error code.
Fixes#72807
Signed-off-by: Anas Nashif <anas.nashif@intel.com>
Read default domain from domains.yaml file and update
paths to proper build directory. It fixes native and qemu
pytest scenarios, when application is build with sysbuild.
Signed-off-by: Grzegorz Chwierut <grzegorz.chwierut@nordicsemi.no>
Remove the deprecated uart_mux and gsm_mux modules and all of
their configurations/dependencies across zephyr.
Optimally uart_mux and gsm_mux would be removed in their own
respective PRs, but the two modules are directly coupled, so
to preserve bisectability, they must be removed together.
Signed-off-by: Bjarki Arge Andreasen <bjarki@arge-andreasen.me>
Initial integration with renode-test was introduced in commit bdf02ff,
which added support for calling the `renode-test` command from both west
and twister.
This commit removes the custom run_renode_test target used for running
Robot tests with the `west build` command and makes twister call
`renode-test` directly instead.
Signed-off-by: Michał Szprejda <mszprejda@antmicro.com>
The --flash-before flag allows devices to be flashed before the serial
connection is established. However, the implementation was incomplete and
only worked if the port address already existed at the start of the run.
This is incompatible with devices that implement the USB in software
(eg: USB-CDC).
This commit fixes the implementation to delay setting up the connection
until after the device is flashed, and to retry the connection for two
seconds after flashing to give the device time to enumerate as a USB
device.
Signed-off-by: Mike Szczys <mike@golioth.io>
Terminate QEMU earlier if the test finishes and no output data is
received for 1 second. This new timeout operates in parallel with the
global test timeout.
For coverage testing, this can reduce the time spent running individual
tests by up to 29 seconds, while still giving the full 30 extra seconds
to dump gcov data if needed.
Signed-off-by: Jordan Yates <jordan@embeint.com>
In the case where a test causes the test executor to crash, the stderr
is currently lost, making it hard to debug failures. This changes it
so that the process' stderr gets captured to 'handler_stderr.log' for
inspection.
Signed-off-by: Benjamin Gwin <bgwin@google.com>
PR #72100 broke execution of unit type tests. This commit fixes
it by bringing back the "old" logic for selecting a binary path
for unit type tests.
Signed-off-by: Maciej Perkowski <Maciej.Perkowski@nordicsemi.no>
Fix default command line options check dependency on `--device-testing`
when it is provided to run Twister and turn device command options
as mandatory.
Fixes after changes introduced with #72399.
Signed-off-by: Dmitrii Golovanov <dmitrii.golovanov@intel.com>
Store Twister command line options in twister.json report as
`environment.options` object. It allows to keep the actual
execution context for test results analysis and history comparison.
A new command line option `--report-all-options` enables to report all
command line options applied, including these set as default values.
Signed-off-by: Dmitrii Golovanov <dmitrii.golovanov@intel.com>
in many platforms, before the debuger flash, the former testsuite
many already executed, so we will see some old testsuite name.
and we should not compare them directly, instead if we compare in
reversed order them we can avoid such mis-judge.
Signed-off-by: Hake Huang <hake.huang@oss.nxp.com>
Fix issue 72083. Update path to zephyr.exe binary using
default domain from domains.yaml file.
Signed-off-by: Grzegorz Chwierut <grzegorz.chwierut@nordicsemi.no>
Added a switch that show failed/error report from the last run.
Default shows all items found. However, you can specify the number of items
(e.g. --report-summary 15).
It also works well with the --outdir switch
Signed-off-by: Kamil Paszkiet <kamilx.paszkiet@intel.com>
Extend Pytest Harness to support 'recording' feature to parse
test log by a regular expression and collect as records the same
way as Console Harness do.
Signed-off-by: Dmitrii Golovanov <dmitrii.golovanov@intel.com>
Refactor Twister 'recording' feature moving it from Handler class
to TestInstance class and enable it also for other Harness child
classes other than Console.
Signed-off-by: Dmitrii Golovanov <dmitrii.golovanov@intel.com>
Using `erase` with west-flash was blocked as it was
messing with sysbuild. With #69748 the issue is fixed, hence
'erase' is no longer blocked.
Remove obsolete twister test
Signed-off-by: Maciej Perkowski <Maciej.Perkowski@nordicsemi.no>
Fix Twister to generate platform reports (`--platform-reports`)
only once for each platform instead of repeating it multiple times
for each instance executed.
Signed-off-by: Dmitrii Golovanov <dmitrii.golovanov@intel.com>
Disable the branch coverage calculations on the `LOG_*` family of macros.
Branch misses are due to the implementation of `Z_LOG2` and cannot be
reasonably covered in library code. The internal paths through the
`Z_LOG2` macro are not of interest when inspecting files that happen to
use the Zephyr logging API.
Signed-off-by: Jordan Yates <jordan@embeint.com>
sometime we need use quarantine at runtime when --test-only
add this will help to skip problematic cases in test cycle
to reenable the quarantined case you need add
--quarantine-list quarantine.yaml --quarantine-verify
Signed-off-by: Hake Huang <hake.huang@oss.nxp.com>
Current find_v2_boards() has a lot of unnecessarry fluff.
We can make a new version of that function without it
given its use in TestPlan.add_configurations().
Signed-off-by: Lukasz Mrugala <lukaszx.mrugala@intel.com>
Sped up Twister via name prefix loading.
If the only thing influencing the platform selection
is the platform filter (-p), then we only load *.yaml
files that start the same way as those selected platforms.
We split the platform name to the first '/' or '@'.
Signed-off-by: Lukasz Mrugala <lukaszx.mrugala@intel.com>
Spell checking tools do not recognize "iff", replace with "if and only if".
See https://en.wikipedia.org/wiki/If_and_only_if
Signed-off-by: Pieter De Gendt <pieter.degendt@basalte.be>
Removed Harness's repeat field.
Added to schemas in e722db14ad,
together with the rest of harness_config.
Added to Harness when it was created.
Now present only in one sample -
mec15xxevb_assy6853 power management sample.
Could not find definitive proof whether it has ever been used.
Signed-off-by: Lukasz Mrugala <lukaszx.mrugala@intel.com>
esp32 platforms can overflow its dram0_0_seg and dram0_1_seg.
So update current dram section to meet both cases.
Signed-off-by: Sylvio Alves <sylvio.alves@espressif.com>
Removed following fields from relevant Handlers,
as they were unused in code:
* Handler's state
* Handler's generator
* BinaryHandler's call_west_flash
* QEMUHandler's results
* QEMUWinHandler's results
Signed-off-by: Lukasz Mrugala <lukaszx.mrugala@intel.com>
Pytest harness in Twister tries to get custom parameters to
the pytest harnesses. It is required although those do not exist
(e.g. Twister uses devices without hw map). This change checks
if the custom parameters to the pytest harnesses exists.
Fixes#71817
Signed-off-by: Katarzyna Giądła <katarzyna.giadla@nordicsemi.no>
Make the --list-tests and --tests-tree options outputs
respect the --tag and --exclude-tag options,
so that only the tests for the specified tags are listed.
Also update the TestPlan report testcases for this change.
Signed-off-by: Declan Snyder <declan.snyder@nxp.com>
If the src_dir_path is a child of testsuite_path,
some c files might be scanned twice.
To prevent that, we check for parentage.
Signed-off-by: Lukasz Mrugala <lukaszx.mrugala@intel.com>
Add error handling for 'gather_metrics' stage and separate
build issues from 'scripts/footprint/size_report' issues.
Signed-off-by: Dmitrii Golovanov <dmitrii.golovanov@intel.com>
In some cases, Twister would report twice the amount
of testcases run than expected, with half of them
lacking any status.
This change fixes erroneous ELF testcase name extraction,
which deleted every instance of 'test_' in the name,
rather than just the first.
Signed-off-by: Lukasz Mrugala <lukaszx.mrugala@intel.com>
Fail a test run gracefully when an unknown test level
is requested to execute with the current test plan.
Signed-off-by: Dmitrii Golovanov <dmitrii.golovanov@intel.com>
Use 'runner_params' specified in hardware map YAML file. This allows to
configure custom params (like openocd's adapter configuration for
FT232-based probe) when used with pytest harness.
Signed-off-by: Marcin Niestroj <m.niestroj@emb.dev>
Some NXP boards are not detected by twister when creating hardware map,
because manufacture name is NXP instead of NXP Semiconductors expected
by twister.
Fix it by adding NXP to manufacturers list.
Signed-off-by: Michal Smola <michal.smola@nxp.com>
Improve Twister command line arguments for memory footprint:
* group and order footprint arguments meaningfully,
* clearer help descriptions,
* resolve logical inconsistences for combinations of arguments,
in particular:
`--last-metrics` now forces `--enable-size-report`,
`--show-footprint` now controls only detailed log output of
footprint deltas in comparison modes.
* align twister tests accordingly.
Signed-off-by: Dmitrii Golovanov <dmitrii.golovanov@intel.com>
When sending commands containing something like "AT+CFUN" the expected
regex was not escaped and as a result such regex never matched what has
been echoed over serial.
Escape all commands with re.escape() so that echoed shell commands are
always matched, even when they contain special characters (like '+' in the
example above).
Signed-off-by: Marcin Niestroj <m.niestroj@emb.dev>
ANSI color is great for viewing output in a terminal, but currently it
also gets written out to the `handler.log` file. Text editors usually
don't render these, resulting in a hard-to-read file with a lot of
gibberish interspered. This commit strips ANSI sequences from lines
before writing them to the handler log file. This change does not
affect what is printed in Twister's console output, so one would still
see the colors there.
Signed-off-by: Tristan Honscheid <honscheid@google.com>
Winodws user may use the `\` as path, but in twister we use the common `/`
as path separated, to avoid the mis-use, convert it to `/` in twister
first normpath, and then replace the os.sep
tested by:
For Linux Like:
west twister -p disco_l475_iot1 -s samples/hello_world/...
For Windows:
west twister -p disco_l475_iot1 -s samples\hello_world\...
fixing: #70310
Signed-off-by: Hake Huang <hake.huang@oss.nxp.com>
Add option to flash board before attach serial.
Current implementation performs the following sequence:
1. Open serial port to listen to board log output
2. Flash device
In case of ESP32 where it uses the same serial port
for both operations, flashing needs to come first.
This PR adds a twister option named --flash-before
which enables the process above, allowing tests to be
performed properly.
Signed-off-by: Lucas Tamborrino <lucas.tamborrino@espressif.com>
For using --device-testing with --device-serial or --device-serial-pty
it is necessary to provide the platform to be used for testing.
Not specifying a platform or specifying more than one is an error.
Signed-off-by: Luca Burelli <l.burelli@arduino.cc>
tfm_integration samples fail on Windows on mps2_an521 platform,
because output from ninja process running QEMU freezes in re-build
phase.
Fix it by implementing named pipes to read output from QEMU process
directly using os.open and os.read methods.
Signed-off-by: Michal Smola <michal.smola@nxp.com>
Follow-up: fe25709a9c twister: add unit_testing soc and board
Moving the soc.yml and board.yml to subsys/testsuite tree.
Introduce subsys/testsuite/arch/archs.yml with unit_testing arch.
Update the Zephyr CMake module unittest.cmake to include HWMv2 and
reorder includes to follow same order as zephyr_default.cmake.
Signed-off-by: Torsten Rasmussen <Torsten.Rasmussen@nordicsemi.no>
The new hwmv2 platform names have "/" and "@" in
their names, we need to replace those with "_".
Otherwise the harness will produce a FileNotFoundError
exception when trying to create the executable.
Signed-off-by: Alberto Escolar Piedras <alberto.escolar.piedras@nordicsemi.no>
