When run by a makefile twister might fail to decode the text results of
commands, based upon the environment settings of the system. With this
change escape characters are removed before the string is passed to the
json decoder.
Signed-off-by: Benedikt Schmidt <benedikt.schmidt@embedded-solutions.at>
When using both test-only and retry-failed flags, all tests would be
added to the test queue regardless if they failed or not. This commit
updates the logic to process test-only runs similar as other runs,
by adding tests to the queue with the 'run' operation directly.
Signed-off-by: Andreas Vibeto <andreas.vibeto@nordicsemi.no>
Create a "global" gen_defines.py option and edtlib.EDT constructor
kwarg that turns edtlib-specific warnings into errors. This applies to
edtlib-specific warnings only. Warnings that are just dupes of dtc
warnings are not affected.
Use it from twister to increase DT testing coverage in upstream zephyr.
Signed-off-by: Martí Bolívar <marti.bolivar@nordicsemi.no>
Make correction on the terms "intput" appearing
in sciripts/pylib/twister/twisterlib.py to "input" for all.
This type of misspellings may cause another unnoticeable misspellings,
when propagated by code completions.
(assuming that the "intput" was intended
to have the opposite meaning of "output",
which also appear on the same areas)
Signed-off-by: Kentaro Sugimoto <tarotene@gmail.com>
This function takes a node label (not a label property) and returns
true if a node with such label exists and the node is enabled.
Signed-off-by: Andrzej Głąbek <andrzej.glabek@nordicsemi.no>
When integration mode is on, any skips on integration_platforms are
treated as errors. This patch adds an exemption for quarantined tests.
They will stay as skipped.
Signed-off-by: Maciej Perkowski <Maciej.Perkowski@nordicsemi.no>
If "Cmake build failure" is detected test instance get status "error".
Despite this in final report this error is counted as failure. It can
be fix, by set proper results of each testcase in instance from None
to "BLOCK" after found Cmake error. After this fix, error is counted
properly in final report.
Fixes#37140
Signed-off-by: Piotr Golyzniak <piotr.golyzniak@nordicsemi.no>
In dt_label_with_parent_compat_enabled, it maybe that there is
no node matching expected label.
In this case don't generate error, but return False.
Signed-off-by: Erwan Gouriou <erwan.gouriou@linaro.org>
sometimes ninja fails to launch qemu, the sub-thread
can't read anything from qemu pipeline, then the
corresponding testcase will timeout. Then the
sub-thread will get blocked if it call join()
set terminate as the Handler's method, then
Handler's children class can call it
Signed-off-by: Jingru Wang <jingru@synopsys.com>
Function used for filtering "dt_compat_enabled_with_label" was not
working as expected as it was not taking into account that we're
looking for a children/parent combination:
Provided "compat" with enabled status should be the one of the parent
of the node matching given label.
Function is then reworked to take this into account.
And to make it's usage clear:
- function name is changed to be clearer on the intention
- args order is reversed to be more logical wrt the intention
Users of the function are also updated to take the change into
account.
Fixes#36093
Signed-off-by: Erwan Gouriou <erwan.gouriou@linaro.org>
when generating a new map, set the status of the device and whether it
is connected or now. The available key is runtime only and is not part
of the map anymore.
Adapt documentation and remove available as a key in the generated map.
Fixes#35341
Signed-off-by: Anas Nashif <anas.nashif@intel.com>
On ARC some platforms utilize ICCM/DCCM for their "flash" and "ram"
storage. So we might get errors about overflowing one of these regions.
So treat them similar to how we treat FLASH and SRAM.
Signed-off-by: Kumar Gala <kumar.gala@linaro.org>
1. Add a class of Pytest(Harness) which can handle pytest script
in harness.py
2. Use running_dir to store current test directory which be
used in pytest scrpt.
3. Add usage of this feature into zephyr doc.
Signed-off-by: YouhuaX Zhu <youhuax.zhu@intel.com>
All skips for platforms listed in the .yaml's integration_platforms
will be treated as errors in the integration mode ('-G/--integration').
This feature serves to guarantee a proper testing scope in CI as no
integration platform will be skipped silently.
Fixes#33874
Signed-off-by: Maciej Perkowski <Maciej.Perkowski@nordicsemi.no>
With the previous logic memory footprint can only be saved in a report
if a given test/sample was fully executed and passed (built and run),
hence build-only tests were not providing these metrics. This commit
modifies the logic so that it is enough to have the build successful
to be able to get the memory footprint.
Signed-off-by: Maciej Perkowski <Maciej.Perkowski@nordicsemi.no>
Upon discovery of a sample/testcase.yaml file, twister will disregard
further subfolders. Given the folder structure below, the testcase.yaml
will currently not be discovered:
samples/sample1/
samples/sample1/sample.yaml
samples/sample1/tests/
samples/sample1/tests/testcase.yaml
This is desirable to allow placing test code closer to sample code.
Signed-off-by: Jorgen Kvalvaag <jorgen.kvalvaag@nordicsemi.no>
Add arm fvp emulator in order to use ninja run or west build -t run.
Add armfvp in order to run twister.
Set env ARMFVP_BIN_PATH before using it,
e.g. export ARMFVP_BIN_PATH=<path/to/fvp/dir>
NOTE: ARMFVP_BIN_PATH is the dir path.
Signed-off-by: Jaxson Han <jaxson.han@arm.com>
Adds feature allowing to use yaml file with dictionaries defining
tests to be quarantined (extra arg "--quarantine-list FILENAME").
The dictionaries are validated according to the proper schema
and loaded.
A flat list is created containing quarantined configurations
(configuration = platform + scenario). Configurations under quarantine
are skipped and get "Quarantine" as a reason in the results reports.
A "comment" can be added to a quarantine entry in the quarantine yaml
with more details (e.g. issue #) and it will be also added to
the report.
The status of tests under quarantine can be verify if
`--quarantine-verify` is used in addition to
"--quarantine-list FILENAME". Using these args will make twister skip
all tests which are not on the quarantine list.
Signed-off-by: Maciej Perkowski <Maciej.Perkowski@nordicsemi.no>
In CI we typically build for "default" platforms. However some
testcases that set "platform_allow" may have no overlap with the default
platforms and thus thoses testcases will not get built.
Change the platform selection logic in these cases to set the platforms
to the list of allowed platforms ("platform_allow") in the testcase.
Signed-off-by: Kumar Gala <kumar.gala@linaro.org>
We are now in the process of extracting edtlib and dtlib into a
standalone source code library that we intend to share with other
projects.
Links related to the work making this standalone:
https://pypi.org/project/devicetree/https://python-devicetree.readthedocs.io/en/latest/https://github.com/zephyrproject-rtos/python-devicetree
This standalone repo includes the same features as what we have in
Zephyr, but in its own 'devicetree' python package with PyPI
integration, etc.
To avoid making this a hard fork, move the code that's being made
standalone around in Zephyr into a new scripts/dts/python-devicetree
subdirectory, and handle the package and sys.path changes in the
various places in the tree that use it.
From now on, it will be possible to update the standalone repository
by just recursively copying scripts/dts/python-devicetree's contents
into it and committing the results.
This is an interim step; do NOT 'pip install devicetree' yet.
The code in the zephyr repository is still the canonical location.
(In the long term, people will get the devicetree package from PyPI
just like they do the 'yaml' package today, but that won't happen for
the foreseeable future.)
This commit is purely intended to avoid a hard fork for the standalone
code, and no functional changes besides the package structure and
location of the code itself are expected.
Signed-off-by: Martí Bolívar <marti.bolivar@nordicsemi.no>
Remove dead code from twisterlib.py which is a copy of code
from twister script. This redundancy causes pylint errors in
compliance checks.
Signed-off-by: Maciej Perkowski <Maciej.Perkowski@nordicsemi.no>
The function `handle()` from class `Test(Harness)` had most of its
code being the direct copy of the code from
`Harness.process_test(line)`. This patch replaces the copied code
with the call for `process_test(line)` and removes repetition of
this function call.
Signed-off-by: Maciej Perkowski <Maciej.Perkowski@nordicsemi.no>
Use edt.label2node in dt_compat_enabled_with_label DT filter.
Signed-off-by: Johann Fischer <johann.fischer@nordicsemi.no>
Co-authored-by: Martí Bolívar <marti.bolivar@nordicsemi.no>
The patch improves handling runners errors from flashing. It
add thread termination in case of process returning non 0 exit
code. Before, even though the error was returned,
a thread was not terminated and test ended
only after timeout termination. The patch also adds information about
the failure reason to the reports
Signed-off-by: Maciej Perkowski <Maciej.Perkowski@nordicsemi.no>
The function returns true if the node with the referenced
node label is compatible and enabled.
Signed-off-by: Johann Fischer <johann.fischer@nordicsemi.no>
Add execution time for testing result of each ztest testcase as:
START - test_sem_multi_take_timeout_diff_sem
PASS - test_sem_multi_take_timeout_diff_sem in 2.54 seconds
Fix#32137.
Signed-off-by: Enjia Mai <enjiax.mai@intel.com>
Logic for selecting non-default platforms listed in
integration_platforms did not work. Functionality is restored in this
change.
Fixes#32835
Signed-off-by: Anas Nashif <anas.nashif@intel.com>
Originally added in 7733b94224.
This filter is not well-formed. It's meant to match nodes like
/leds/led_0 in this DTS:
/ {
aliases {
led0 = &led0;
};
leds {
compatible = "gpio-leds";
led0: led_0 {
gpios = <...>;
label = "LED 0";
};
};
};
Uses look like this:
filter: dt_compat_enabled_with_alias("gpio-leds", "led0")
But notice how the led_0 node doesn't have compatible "gpio-leds";
it's actually the *parent* node that has that compatible.
Replace this with a new filter, dt_enabled_alias_with_parent_compat(),
which is used like this:
filter: dt_enabled_alias_with_parent_compat("led0", "gpio-leds")
This has a name and argument order that makes the meaning of the
filter clearer.
Replace in-tree users with the new filter.
Deprecate the old filter and warn about its use using the standard
logging module.
Signed-off-by: Martí Bolívar <marti.bolivar@nordicsemi.no>
CPython is sometimes described as "single threaded" due to the GIL,
but the interpreter will still "preemptively" switch between threads
(the details seem poorly documented).
So the time between checking whether acquire is 1 and decrementing the
count could result in more than one thread seeing an "available"
device, and more than one test being run (simultaneously, on the same
physical device!). We have a big herd of threads all polling for
this, so in a large test run this would happen maybe one time out of
20-30 tries.
Use a lock. Also remove the very similar looking
DUT.get_available_device() method, which had the same bug but appears
to be dead code.
Fixes#32679
Signed-off-by: Andy Ross <andrew.j.ross@intel.com>
Use EXTRA_GEN_DEFINES_ARGS to error out on deprecated devicetree
properties when warnings are treated as errors.
Signed-off-by: Martí Bolívar <marti.bolivar@nordicsemi.no>
sometimes a test instance hasn't been boot up to execute successfully,
so the results is an empty dictionary, it will not be added into final
twister_report.xml file.
in order to include it, fill the results as BLOCK.
Signed-off-by: peng1 chen <peng1.chen@intel.com>
Currently duplicated testcases are only determined by the classname.
This results in testcases not beeing added to the twister.xml if the
testcases are from the same class (e.g. sample.yaml), but with different
names.
The changed check if the combination of classname and testname
already exists.
Signed-off-by: Sebastian Wezel <sebastian.wezel@nordicsemi.no>
Include coverage support for device testing. It works by switching
capture_coverage to True (as it would meet coverage data start
pattern). Then we continue tot read coverage data until we get
coverage data end pattern. Otherwise, after receiving test result
pattern, we close serial console and do not get time to find start
coverage data pattern.
Signed-off-by: Andrei Emeltchenko <andrei.emeltchenko@intel.com>
Fix twister output. "Test suite" used previously did not correspond
to what was counted (test scenarios). A test scenario is an entry from
testcase/sample.yaml under "tests:", which specify configuration used to
build a test. Changes in docs will follow later on. Also replace the
amount of preselectd platforms with the amount of platforms that
were not filtered out and actually tested.
Signed-off-by: Maciej Perkowski <Maciej.Perkowski@nordicsemi.no>
Followup commit on comment in #32003.
Using pathlib to avoid mixing of `/` and `\` if twister someday will be
able to run on Windows.
Signed-off-by: Torsten Rasmussen <Torsten.Rasmussen@nordicsemi.no>
Fix possible race when determining available devices. Find available
device and return it in the same operation to avoid the race.
Fixes#31769
Signed-off-by: Anas Nashif <anas.nashif@intel.com>
Fixes: #30713
Now using the common script `verify-toolchain.cmake` for verifying
toolchain settings.
Signed-off-by: Torsten Rasmussen <Torsten.Rasmussen@nordicsemi.no>
As of today we unconditionally call 'stty sane' that leads to
lots of error messages
'stty: standard input: Inappropriate ioctl for device'
when output isn't connected to tty (for example when we launch
twister in Jenkins)
Fix that by check if output is really connected to tty before
'stty sane' call.
Signed-off-by: Eugeniy Paltsev <Eugeniy.Paltsev@synopsys.com>
This is always the same, so we might as well just remove it for now and
save some space since it is the default anyways. Type can be added later
when it provides more value.
Signed-off-by: Anas Nashif <anas.nashif@intel.com>
Logic for scope of platforms to be checked was changed and this options
now only applies to limited scope, fix this and reset scope when a
testcase declares it needs to build on available platforms.
Signed-off-by: Anas Nashif <anas.nashif@intel.com>
