Measure both cmake and make/ninja build times and log them in debug mode
and additionally put the result in the json report for tracking build
times.
Cleanup what build/cmake function return and remove unused keys in the
result map.
Remove some excessive logging of launched jobs or default platforms that
gets in the way when in debug mode.
Signed-off-by: Anas Nashif <anas.nashif@intel.com>
Test suites names are not being handled uniformly for tests not
in zephyr tree. Their names depend on -T arg used in twister's CLI.
The newly added options allow to select if twister should add paths
to suite names. This is needed if test plans are to be used for tests
outside of zephyr tree.
Signed-off-by: Maciej Perkowski <Maciej.Perkowski@nordicsemi.no>
In test-only mode with a hardware map, all tests with fixtures were
skipped, even when the hardware map has appropriate fixtures defined.
TestPlan.load_from_file had the check for fixtures missing, but actually
best is that the TestInstance checks the fixtures in the hardware map.
Signed-off-by: Manoel Brunnen <manoel.brunnen@ithinx.io>
When using the hardware map and testing on device, print out the name of
the dut (from the hardware map) on the script and report it in the json
output as well.
Signed-off-by: Anas Nashif <anas.nashif@intel.com>
We have been dealing with missing and multiple binaries the same way and
both would result in a build error, which is not accurate. multiple
binaries in the build directory are fine, we just need to pick the right
one for parsing.
If we get no binaries, raise an exception and report failure, however,
if we have multiple binaries, filter intermediate artifacts out and
parse what remains.
qemu binaries generated after a run are also being filtered here. Those
are not build artificats and appear only after running in qemu. However
they have been causing issues on retries.
Signed-off-by: Anas Nashif <anas.nashif@intel.com>
Extended parsing of report.xml file - produced by pytest-twister-harness
plugin. Now testcases are properly extracted and added to twister
report. Added unit tests.
Signed-off-by: Grzegorz Chwierut <grzegorz.chwierut@nordicsemi.no>
testinstance.py contains a string literal comparison to the
word 'platform', which has been misspelled as 'plaform'.
The change is verified by a test expansion.
Signed-off-by: Lukasz Mrugala <lukaszx.mrugala@intel.com>
Making the necessary changes to enable the new pytest plugin.
By default Twister should work without the pytest-twister-harness
plugin installed. To achieve this, each time Twister calls pytest,
the PYTHONPATH environment variable is expanded and the
`-p twister_harness.plugin` option is added to the pytest command.
Co-authored-by: Piotr Golyzniak <piotr.golyzniak@nordicsemi.no>
Signed-off-by: Grzegorz Chwierut <grzegorz.chwierut@nordicsemi.no>
Add support for calling the `renode-test` command from west and twister.
Enable running Robot Framework tests suites in Renode.
Signed-off-by: Michał Szprejda <mszprejda@antmicro.com>
Signed-off-by: Mateusz Hołenko <mholenko@antmicro.com>
Some 3P logic (Pigweed for example) is already heavily invested in
gTest as a testing framework. Adding the `gtest` harness enables
running the existing 3P tests on various Zephyr platforms and configs.
Signed-off-by: Yuval Peress <peress@google.com>
`TestInstance.get_elf_file()` does not include the binary name used for
unit testing (`testbinary`) in its search, causing unit tests to not
complete successfully. This does not cause tests to fail and thus goes
unnoticed.
Update the glob expressions used in `get_elf_file()` to also look for
files called `testbinary`
Signed-off-by: Tristan Honscheid <honscheid@google.com>
If we are doing a sysbuild get_elf_file needs to return the default
domain's build_dir. The standard build_dir will not any ELF builds
in it so will error out.
Signed-off-by: Kumar Gala <kumar.gala@intel.com>
This is usually used only when doing size calculation, but now it is
being used for extracting symbols from the ELF for testing purposes and
some issues arise with multiple ELF files found.
Simplify the code a bit to make it easy to maintain a lists of things to
exclude.
Signed-off-by: Anas Nashif <anas.nashif@intel.com>
On native platforms we typically have CONFIG_BUILD_OUTPUT_EXE set so
we end up with both zephyr.elf and zephyr.exe. This would cause a
call to get_elf_file() to fail. There isn't any reason to error out
since zephyr.elf and zephyr.exe should be identical. So allow it
if the platform type is native.
Signed-off-by: Kumar Gala <kumar.gala@intel.com>
When a test start, mark it as such, if we do not get any results due to
a timeout, it will be finally marked as failed. Previously such tests
causing a freeze were marked as blocked (or not run), which is not
exactly right.
Signed-off-by: Anas Nashif <anas.nashif@intel.com>
Some hardware boards are supported in qemu as simulation platforms. If
we are testing with the real hardware, make sure we treat this as a
device and not as a simulator. First priority when initializing a
handler is to check if device testing is requested, then we can handle
the rest without --device-testing.
Signed-off-by: Anas Nashif <anas.nashif@intel.com>
We want to be able to have platform or architecture extra configs
without having to duplicate a whole section of the test specification.
This adds support for namespacing of extra configs, for example:
arch:nios2:CONFIG_SYS_CLOCK_TICKS_PER_SEC=1000
or
platform:qemu_x86:CONFIG_FOO=y
Signed-off-by: Anas Nashif <anas.nashif@intel.com>
When scanning for available elfs, twister discards ones with `_pre`
in their name, to not include prebuilds like 'zephyr_pre0.elf`.
However, the scan looks at the whole test name. If _pre is in the name
(e.g. _precision) it would be wrongly discarded ending with twister
reporting error that no elfs were found. This commit makes the discard
to look only at the last part of the name (i.e. not including test name
but only a lowest level, where zephyr.elf etc are located.)
Signed-off-by: Maciej Perkowski <Maciej.Perkowski@nordicsemi.no>
Ignore unit tests and EFI binaries when calculating footprints with
--show-footprint, or we will get a build error.
Signed-off-by: Anas Nashif <anas.nashif@intel.com>
This commit calculate memory footprint from build.log and
proposes an alternative approach to #2465.
Signed-off-by: Katarzyna Giądła <katarzyna.giadla@nordicsemi.no>
non-simulation platforms set key to 'na', we need to check for that
instead of just checking for existance of key.
Fixes#52595
Signed-off-by: Anas Nashif <anas.nashif@intel.com>
If a default platform is not able to run for whatever reason, treat it
as any other platform. Default platforms shall ALWAYS be runnable in CI,
or else we are just re-building for no good reason.
Signed-off-by: Anas Nashif <anas.nashif@intel.com>
When twister is set to retry any failures, count the number of retries
and record the number in the json file. This will help us identify
unstable tests or tests requiring attention.
Signed-off-by: Anas Nashif <anas.nashif@intel.com>
Currently, support for Twister on Windows is only build.
The only thing that needs to be ported is that select()
cannot wait on file descriptors on Windows. Therefore
the serial monitor function needs to be reworked to
support both OSes.
Signed-off-by: Cong Nguyen Huu <cong.nguyenhuu@nxp.com>
