With --retry-failed, add_tasks_to_queue() resets the status of a failed
instance before queueing it again, but leaves the per-testcase statuses
from the failed attempt in place.
When the first attempt fails before any output is seen (for example
"Timeout during flashing" on a hardware target), every testcase is
marked blocked. On the retry, only cases that print a result are
updated. Cases that never report, such as ztest cases compiled out of
the image for the current configuration, keep the stale blocked status
and reason, so twister.json shows blocked testcases inside a suite whose
final status is passed. A clean run without a retry reports no status
for those same cases.
Clear the status, reason, duration and output of every testcase after
removing the previous attempt from the status counts, so a retried run
reports the same testcase results as a run that passed first time.
Assisted-by: AI (GitHub Copilot)
Signed-off-by: Bill Waters <bill.waters@infineon.com>
When MAINTAINERS.yml changes, only the maintainers of each changed area
were added as reviewers, looked up in the checked-out MAINTAINERS.yml.
Collaborators were skipped, as were people added by the PR and members
of areas that only exist in the PR's version of the file.
Add every maintainer and collaborator of each changed area, taken from
both the base and the updated MAINTAINERS.yml, so the whole area is
notified, including people being added or removed.
Tested with pytest scripts/tests/ci/test_set_assignees.py.
Assisted-by: Copilot:claude-opus-5.5
Signed-off-by: Emil Gydesen <emil.gydesen@nordicsemi.no>
A test case can be declared static ZTEST(), which makes its statistics
object static too; many Bluetooth tests do so. The source scan matches
only lines that start with ZTEST, so it misses those cases: --list-tests
leaves them out, --sub-test cannot select them, and they are known only
from the runtime output. On main this concerns 369 test cases in 22 test
directories.
Accept an optional static before the macro in both test case patterns.
Tested with the twister unit tests and with --list-tests over
tests/bluetooth.
Assisted-by: Claude:claude-opus-5-5
Signed-off-by: Johan Hedberg <johan.hedberg@silabs.com>
Bluetooth Classic tests on native_sim need linked virtual controllers
and one Zephyr instance per device. Without host-side support every
test carries shell scripts that allocate TCP ports, start and stop the
controllers and launch the peers, and that depend on the layout of the
twister output directory.
The "bumble" sidecar provisions this from sidecar_config, so a test
keeps its normal harness and ships no scripts. The controllers bind
kernel-chosen ports and report them to the sidecar, since a port
picked up front can be taken by a parallel worker before it is bound.
The handler watches the guest only, so the sidecar reads the Ztest
results the peers printed and reports the cases they ran, and the
peers share the scenario's timeout with the guest. On a host without
Bumble the test is built but not run.
Based on Anas Nashif's proof of concept for the pre-merge sidecar
framework (nashif/zephyr commit 099427868f7).
Tested on native_sim with Bumble 0.0.220: the acl_conn test passes
with both of its cases reported, also as three instances run in
parallel, and is only built when Bumble is missing.
Assisted-by: Claude:claude-fable-5.1
Signed-off-by: Johan Hedberg <johan.hedberg@silabs.com>
ESP32-P4, C5, C6, C61 and H2 name their RAM region sram0_0_seg, which
the overflow pattern did not match, so a too large image was reported
as a build error instead of an overflow skip like on the other ESP32
SoCs.
Signed-off-by: Benjamin Cabé <benjamin@zephyrproject.org>
Assisted-by: Claude:opus-5.5
Check for which installed Ninja and make versions and CMake arguments
the job server becomes a named pipe, that such a job server keeps one
token per job across a claimed job and advertises the pipe in MAKEFLAGS
without passing file descriptors, and that it falls back to file
descriptors or cleans up when the pipe cannot be used or created.
Assisted-by: Claude:claude-opus-5-5
Signed-off-by: Johan Hedberg <johan.hedberg@silabs.com>
Ninja 1.13 became a GNU make job server client, but on POSIX it accepts
the job server only as a named pipe. Twister hands out a pair of file
descriptors, so Ninja warns "Ignoring jobserver: Pipe-based protocol is
not supported!" and every build twister runs in parallel starts its own
full set of jobs: a load average of 150 and 11 GB of anonymous memory
for tests/bluetooth on native_sim on a 24-thread machine.
Create the job server as a named pipe when the build tool takes that
form (Ninja from 1.13, GNU make from 4.4) and no make on PATH is older
than GNU make 4.4: such a make stops with an error on a named pipe, and
native builds run make below Ninja. Otherwise, and when extra CMake
arguments select the tools, the file descriptors are passed as before.
Tested with Ninja 1.13.2 and GNU make 4.4.1: without ccache the run
above takes 425 to 438 s instead of 477 s and peaks at 4.4 GB.
Assisted-by: Claude:claude-opus-5-5
Signed-off-by: Johan Hedberg <johan.hedberg@silabs.com>
_get_info_from_build_failure() reports which region overflowed by reading
the line before the one carrying "error: ld returned":
if "error: ld returned" in line and "overflowed by" in lines[i - 1]:
At i == 0 that is lines[-1], the last line of the log. That line does not
even come from the build: JsonReport.create() appends the handler's
stderr to the build log before writing it to twister.json, which is what
this tool reads. When it happens to mention an overflow, the function
returns straight away with no region and no build step, hiding the real
overflow further down the log.
Read the previous line only when there is one, as commit 30beaec88f
("twister: reports: don't look past the start of the build log") did for
the same expression in JsonReport._parse_build_failure(); that commit
left this tool for a change of its own.
Add the first unit tests for the tool, covering the region and
build-step cases and the first-line one.
I ran the new cases against the parser directly rather than under pytest,
and did not run the tool on a real twister.json.
Assisted-by: Claude:claude-opus-5
Signed-off-by: Leo Camus <leo.camus23@gmail.com>
The twister Test harness attributes every Ztest suite/case marker it
parses to the current run. When a device restarts mid-run -- because it
reboots on its own, or because console output was captured from a
previously-programmed image while the board was still being flashed (the
default flash_before=False opens the serial port before flashing) -- the
markers seen before the restart are left pending. They never reach a
terminal status, so they surface as phantom 'started' cases and produce
spurious 'already STARTED' warnings once the real run re-emits the same
markers.
Discard that stale state when a restart is detected, keyed on two signals
that Ztest emits itself rather than on any application artifact:
- A start marker for a suite or case that is already running. Ztest emits
these itself, so whenever there is stale state to discard the evidence
of the restart is already present, with no dependency on any Kconfig.
This replaces the 'already STARTED' warnings the harness logged for
these restarts (729 across an 1841-log device-run replay corpus) with a
correct recovery.
- The "RunID started:" marker that TC_PRINT_RUNID_START prints at the
start of every run. A second one seen while markers are still pending
proves the image restarted, so it catches a restart that never re-emits
a marker that is still pending. It is emitted on every boot when a run
id is set, and nothing is printed otherwise.
A suite that ends before starting again is a legitimate repeat and is
left alone, as is any suite declaring expect_reboot.
An earlier revision used the Zephyr boot banner for the second signal.
The banner is an application artifact and is optional; the RunID marker
is part of the test output and does not depend on CONFIG_BOOT_BANNER.
Assisted-by: AI (GitHub Copilot)
Signed-off-by: Bill Waters <bill.waters@infineon.com>
Unit cases for the identity the walker derives for Zephyr: which of a
release tag and the VERSION file names the version, how a pre-release
qualifier maps onto the CPE 'update' field, and what the purl is pinned
to for a fork, an unconfirmed revision, or a checkout recording no
remote. None of those states is reachable from the application test,
which only ever sees the tree it runs in.
That test keeps what a build can show: both Zephyr packages report this
tree's version and carry the matching CPE.
Assisted-by: Claude:opus-5
Signed-off-by: Benjamin Cabé <benjamin@zephyrproject.org>
Match the complete name GoogleTest prints and derive the internal test
name from it by replacing "/" with ".", so no index or generator name
can be lost. This supports TEST, TEST_F, TYPED_TEST, TYPED_TEST_P and
TEST_P, each with a default or custom name and, for TEST_P, with or
without an instantiation prefix.
Fixes: ba480d810b ("twister: add support for gtest parametrized tests")
Signed-off-by: Mario Jaun <mario.jaun@zuehlke.com>
Check that two builds with different namespace prefixes define no
element under the same IRI, that document IDs are full IRIs under the
prefix, and that every imported element is defined by the sibling
document its locationHint names.
Assisted-by: Claude:opus-5.5
Signed-off-by: Benjamin Cabé <benjamin@zephyrproject.org>
When the command line argument for twister is used to treat
warnings as errors, will also set the same option for treating
deprecated Kconfigs as errors, this allows CI compliance to ensure
that the API lifecycle process is followed
Signed-off-by: Jamie McCrae <jamie.mccrae@nordicsemi.no>
The handlers fill in a reason of their own, such as the exit code,
only when the instance has none. The JSON report, however, writes a
placeholder reason, "Unknown Instance status", for every suite that
has not run, which is every suite in a test plan saved before
running, and loading that plan with --load-tests restores the
placeholder along with the unset status. A run that then fails
keeps the placeholder over the exit code, which is how CI reports
a QEMU that could not start.
Write no reason for a suite that has not run, do not restore one
for such a suite from a loaded plan, and clear the reason together
with the status before every run so that neither a loaded plan nor
an earlier retry iteration can leak one into the new result.
Assisted-by: Claude:claude-fable-5-1
Signed-off-by: Anas Nashif <anas.nashif@intel.com>
The QEMU monitor thread opens the console fifos as QEMU's
counterpart, and each of those opens blocks until QEMU opens the
other end. When the run command fails to start QEMU at all, the
thread sits in open() with nothing to wait for while the handler
joins it for the whole test timeout: a QEMU binary that cannot be
executed costs the full timeout per test, several minutes in CI,
before being reported as a timeout.
Once the run command has exited, give the thread a moment to drain
what QEMU left in the pipe. If it is still around after that it is
blocked on the fifos, so connect to them in QEMU's place and
disconnect again: the opens return, the thread reads EOF and
finishes at once. If QEMU never wrote its pid file, drop the EOF
reason the thread recorded so the exit code names the failure.
Assisted-by: Claude:claude-fable-5-1
Signed-off-by: Anas Nashif <anas.nashif@intel.com>
The QEMU handler captures the run command's stderr to qemu.stderr, a
file nothing reads: neither the inline log printed for a failure nor
the JSON report look at it. When QEMU cannot start at all, because
the binary is not found, refuses an option or fails to load a
library, the console log stays empty and twister prints build.log,
a log of a build that succeeded, and records nothing about why the
test did not run.
Capture that stderr as handler_stderr.log instead, the file the
binary handler already uses and that both the inline log and the
JSON report read. The run command's stdout stays in qemu.stdout.
The inline log printed for a failure also appends handler_stderr.log
to a non-empty console log, as the JSON report already does, so a
console log that stops short comes with the stderr that explains it.
Assisted-by: Claude:claude-fable-5-1
Signed-off-by: Anas Nashif <anas.nashif@intel.com>
Replace scripts/kconfig/hardened.csv with a distributed YAML database.
The CSV format could not carry any explanation for its recommendations,
had no validation (a malformed line was silently dropped), no notion of
profiles, no ownership story, and no way to extend it out of tree.
The database now lives where the code lives: the central
scripts/kconfig/hardening.yaml defines the hardening profiles (plus
rules for top-level Kconfig symbols), and per-subsystem hardening.yaml
fragments next to the relevant Kconfig files carry the rules, so each
rule is reviewed by the maintainers of the code it hardens. Every rule
has a required rationale and optional references, supports exact-value
and integer min/max recommendations (STACK_POINTER_RANDOM is now
'>=100' instead of '==100'), and belongs to a profile: 'base' (memory
protection and inherently insecure features) or 'strict' (extends base
with debug/observability feature removal). The default profile is
'strict', matching the rule set the CSV checked.
hardenconfig is restructured around a serializable HardeningReport
model so results can be consumed programmatically, and gains:
- a Rationale column in the report
- profile selection:
west build -t hardenconfig -- -DHARDENCONFIG_PROFILE=base
- HARDENCONFIG_SHOW_ALL to also list PASS/NA results
- HARDENCONFIG_STRICT to exit non-zero on failures, for CI use
- HARDENCONFIG_JSON to write machine-readable results, including which
database file each rule came from
- HARDENCONFIG_EXTRA_SOURCES to layer out-of-tree database files that
may add profiles/rules and override in-tree ones
The EXPERIMENTAL/DEPRECATED/NOT_SECURE marker symbol detection is
preserved unchanged.
Three CSV entries are intentionally not carried over because they were
redundant: BT_CONN_DISABLE_SECURITY and BT_OOB_DATA_FIXED can only be
enabled together with BT_TESTING, and TIMER_RANDOM_GENERATOR together
with TEST_RANDOM_GENERATOR, all of which the database already
recommends disabling for the same profiles.
Behavioral fix: the CSV's DEBUG_COREDUMP line used '=' instead of ','
and was silently ignored, so CONFIG_DEBUG_COREDUMP was never actually
checked; it now is.
Also add tabulate to requirements-base.txt: hardenconfig has always
imported it, but it was only pulled in via other requirements files.
Assisted-by: Claude:fable-5
Signed-off-by: Benjamin Cabé <benjamin@zephyrproject.org>
Introduce the foundation for replacing scripts/kconfig/hardened.csv with
a schema-validated, distributed YAML hardening database:
- scripts/schemas/hardening-schema.yaml: JSON Schema (Draft 2020-12) for
the database files. Rules are keyed by Kconfig symbol name (so a
duplicate entry within a file is a duplicate YAML key, which the
loader rejects), require a rationale, and recommend either an exact
value or an integer min/max constraint. Rules belong to named profiles
which may extend one another.
- scripts/kconfig/hardeninglib.py: shared loader used by both the
hardenconfig build target and the compliance check, so parsing and
evaluation semantics can never drift between the two. The database is
distributed: a central file defines the profiles, while rule
fragments named hardening.yaml are discovered next to the subsystem,
driver, architecture and SoC Kconfig files they relate to, keeping
each rule under its own maintainers' review. The loader handles
fragment discovery, schema validation, duplicate-rule rejection,
value normalization to .config-style strings, profile resolution with
'extends' cycle detection, and merging of additional out-of-tree
database files.
- scripts/tests/hardenconfig/: pytest suite covering schema
accept/reject cases, fragment discovery and merge rules, profile
resolution, rule evaluation against a fixture Kconfig tree, value
normalization and extra-source override, wired into the scripts tests
workflow.
Assisted-by: Claude:fable-5
Signed-off-by: Benjamin Cabé <benjamin@zephyrproject.org>
Commit 11c210cec9 ("twister: match platform extra args against
aliases") replaced the substring match of platform:<selector>:<arg>
against the platform name with an exact match against its aliases.
Selectors that name only part of a board target, such as
platform:nrf54l:shell_remote_SNIPPET=nordic-flpr, no longer apply. For
example, shell.shell_remote.no_multithreading on
nrf54l15dk/nrf54l15/cpuapp builds the remote image for the application
core and fails with "No IPC node found".
Match the selector as a substring of the platform name as before, and
additionally as an exact alias so a default board target still selects
its default revision. Add a test case for a partial selector.
Assisted-by: Claude:claude-opus-5-5
Signed-off-by: Anas Nashif <anas.nashif@intel.com>
Verify platform extra_args selectors match a default board target alias
and do not apply it to a non-default board revision.
Signed-off-by: Hake Huang <hake.huang@nxp.com>
Add the ability to configure additional runner parameters.
For instance the Raspberry Pi boards require a specific openocd version
with rp2xxx driver support.
Signed-off-by: Joel Guittet <joelguittet@gmail.com>
Sixteen checks in the SPDX application test assert only that the
serializers render a graph correctly: the document header, DESCRIBES
relationships, package ID format, the role comment on a dependency
package, the supplier and purl derived from a remote URL, and the
VARIANT_OF/DEPENDENCY_OF edges between a module's sources and its
dependency. None of them need a build, yet all were gated behind one --
several carried pytest.skip guards precisely because they could not
assume what a build had produced.
Move them to scripts/tests/zspdx, where a fixture drives the walker's
module half directly and each check knows exactly what the graph holds.
Coverage widens on the way: the SPDX 2.x checks now run against 2.2 as
well as 2.3, the document checks gain SPDX 3.x equivalents, and the
supplier mapping is exercised on both of its branches instead of only
the Zephyr namespace.
What stays is what only a build can show: the '-sources' comments come
from setup_zephyr_component(), which needs a west workspace and the
CMake file-API, the utility-target check needs real build targets, and
the tool creator's version is read from the tree.
Assisted-by: Claude:opus-5
Signed-off-by: Benjamin Cabé <benjamin@zephyrproject.org>
zspdx has no unit tests today: everything it emits is only covered by the
SBOM application test, which needs a full Zephyr build to run. Add a
pytest suite that builds an SBOM graph directly and runs the serializers
over it, so serializer behaviour can be covered without a build.
The suite starts with what a module dependency component produces: the
locator classification in the shared model, and the external references
that reach the SPDX 2.x tag-value and SPDX 3.x JSON-LD output. Shared
fixtures serialize a graph and parse every document back, parametrized
over each supported spec version. The SPDX 3.1 leg skips until a
released spdx-python-model carries its bindings, matching the
application test, which does not exercise 3.1 either.
It joins the existing pylib workflow, which already hosts the
scripts/tests suites for code under scripts/pylib. spdx-python-model
moves into the actions requirements, since the SPDX 3.x tests need it.
Assisted-by: Claude:opus-5
Signed-off-by: Benjamin Cabé <benjamin@zephyrproject.org>
This reverts commit 9fa96059da.
Treating deprecated Kconfig symbols as errors breaks main CI on several
boards, drivers and module tests that still enable them. Revert until
those are migrated.
Assisted-by: Claude:opus-5.5
Signed-off-by: Benjamin Cabé <benjamin@zephyrproject.org>
When the command line argument for twister is used to treat
warnings as errors, will also set the same option for treating
deprecated Kconfigs as errors, this allows CI compliance to ensure
that the API lifecycle process is followed
Signed-off-by: Jamie McCrae <jamie.mccrae@nordicsemi.no>
A hardware map entry can set probe_id to override the id passed to the
flash runner, for example when the board is flashed through an external
J-Link probe. The schema accepts the key and the device handler uses
hardware.probe_id when it is set, but HardwareMap.load() does not read
the key, so probe_id is always None and the flash runner gets the id.
The key has not been read since the hardware map entries were turned
into objects in commit 531fe89e80 ("sanitycheck: use multiprcoessing
instead of threads").
Read probe_id when loading the hardware map.
Checked the flash command twister generates for a native_sim/native/64
build with --device-testing and a pyocd hardware map entry that sets
both keys: it now uses probe_id. Not run on hardware.
Signed-off-by: YunHung Hua <oliverhua.us@gmail.com>
Allow a hardware-map entry to set run_with_fixture_only.
Such DUTs are excluded from tests that do not request
fixtures, while fixture tests remain eligible when their
requested fixtures match the DUT.
Add schema validation and unit coverage for map loading
and DUT selection.
Signed-off-by: Hake Huang <hake.huang@nxp.com>
The twister source scanner extracts ZTEST() testcase names with the
regex [a-zA-Z0-9_]+. For cases whose name is built with the ##
token-paste operator, e.g.
ZTEST(dma_m2m_loop, test_dma##idx##_m2m_loop)
the regex stops at the first # and captures "test_dma", which after the
"test_" prefix is stripped registers a phantom subcase "<suite>.dma".
That case never runs (the real, fully expanded cases such as
test_dma0_m2m_loop are only produced at build time), so every run of
tests/drivers/dma/loop_transfer and other macro-generated ztest suites
reports it as "None status".
Add a negative lookahead so a testcase name immediately followed by a
word character or # is not matched. Token-pasted names are therefore
ignored by the static scan and no longer create phantom subcases; the
real cases continue to be discovered from the harness output at runtime.
Add a scan_file unit test covering a suite that mixes a literal ZTEST
case with token-pasted, macro-generated cases, and extend the
get_all_testsuites expectation with the new fixture's literal case.
Assisted-by: AI (GitHub Copilot)
Signed-off-by: Bill Waters <bill.waters@infineon.com>
Add unit tests that build a CTF trace from the tracing subsystem's own
TSDL metadata, so the event ids and field layouts they exercise are the
ones a build emits rather than a copy of them that can drift. They cover
the lane routing, that records keep the time they happened, that a trace
ending mid-run keeps its running thread, that payload strings such as a
syscall name reach the output, and that a metadata mismatch is an error
while a half-written trailing record is not.
Run them in the misc. pylib workflow, beside the build_helpers tests it
already runs out of scripts/tests. Its path filter gains subsys/tracing
so that a change to the TSDL metadata, which is the input these tests are
built from, reruns them; it was listed for push only. MAINTAINERS.yml
gains the new directory so it lands in the tracing area, as the script it
tests already does.
Signed-off-by: Ugo Marchand <marchand.ugo@orange.fr>
Assisted-by: Claude:claude-opus-5
A single GitHub review request accepts at most 15 users, but
_request_reviews() sent the whole candidate list in one call and, when
GitHub rejected it, retried with only the first 15 and gave up on the
rest. Those left over were merely mentioned in a comment, and only a
later run of the script - where the first 15 are filtered out as
existing reviewers - added them. That is why a PR received a subset of
its reviewers on the first run and more on the second.
Split the candidates into batches of REVIEWER_REQUEST_BATCH (15) and
issue one request per batch, so every eligible reviewer is added on the
first run. A rejected batch is now retried one user at a time, so a
single bad candidate (deleted account, lost collaborator status) no
longer costs the rest of its batch their review request; only the users
that still fail are mentioned instead.
Also delete the reviewer-mention comment once a run has nobody left to
mention, so the PR stops claiming that people could not be requested
after they have been.
Verified with scripts/tests/ci/test_set_assignees.py, extended with
tests for the batching, the per-user fallback and the comment cleanup.
Signed-off-by: Emil Gydesen <emil.gydesen@nordicsemi.no>
Tests that trigger faults on purpose set ignore_faults, which disables
the generic ">>> ZEPHYR FATAL ERROR" crash detection entirely. When
such a test then crashes for real (e.g. testing.ztest.error_hook), the
crash went unrecognized and the run was reported with a twister-side
artifact such as "unexpected eof" or a timeout.
Ztest's fatal-error and assert hooks already make the distinction the
harness needs: they print "Fatal error was unexpected, aborting..." or
"Assert failed was unexpected, aborting..." right before halting when
the fault was not declared as expected by the test. Several tests with
custom hooks print the same message, some in lower case.
Match these markers case-insensitively in Harness.process_test() and
set harness.fault even when fail_on_fault is off, so the handlers
report such runs as "Fault detected while running test" like any
other crash. The hooks' expected-fault messages ("Fatal error
expected as part of test case.") do not contain the markers and are
unaffected.
Assisted-by: Claude:claude-fable-5
Signed-off-by: Anas Nashif <anas.nashif@intel.com>
When a test crashes with an unexpected fatal error the target halts
and never prints a verdict, so the QEMU monitor kept reading until
the full per-test timeout expired and the run was reported as
"Timeout". The timeout was a twister artifact, not the failure: the
console already contained the ">>> ZEPHYR FATAL ERROR" banner. Other
simulators exit on such a crash and got reported differently again
(e.g. "failed (rc=2)" on mps4 FVP), so the same crash produced
inconsistent results across platforms.
The harness already tracks unexpected fatal errors in harness.fault
(tests that fault on purpose opt out with ignore_faults), and any run
with a recorded fault is already doomed to fail once a verdict
arrives. Use that flag in the handlers:
- The QEMU monitor loops (POSIX and Windows) record the failure as
"Fault detected while running test" as soon as the fault banner is
seen and switch to the short post-status grace period, capturing
the rest of the crash dump instead of waiting out the timeout. A
subsequent EOF no longer overwrites the recorded fault reason with
"unexpected eof".
- BinaryHandler's output loop stops after the same grace period once
a fault is seen, covering simulators such as Arm FVP.
- Handler._update_instance_info() prefers the fault over the derived
"Timeout" or "Exited with <rc>" reasons, so every handler
(including device testing) reports such a crash the same way.
The reason string is shared as FAULT_REASON in constants.py so the
harness and the handlers cannot drift apart.
Assisted-by: Claude:claude-fable-5
Signed-off-by: Anas Nashif <anas.nashif@intel.com>
SizeCalculator takes the footprint numbers twister reports from the last
"Memory region" table in build.log. Everything between finding that table
and the four numbers it ends up with is string handling, and none of it
was covered: the module sat at 16% under the twister suite, with no test
file of its own.
These drive that path with build logs written by the test: finding the
table (and the last one when there are two, and not a line that merely
mentions it), taking its lines, splitting the columns, converting the
units, and the four numbers that come out. The ELF analysis is stood in
for, since none of this is about the ELF.
Coverage of the module goes from 16% to 67%.
Signed-off-by: Egemen Tuncarslan <egementuncarslan@hotmail.com>
Assisted-by: Claude:claude-opus-5
_parse_build_failure decides why a link failed from the line before the
one carrying "error: ld returned":
elif "error: ld returned" in line:
if last_warning:
return last_warning
elif "overflowed by" in lines[i - 1]:
return "ld.bfd: region overflowed"
elif "ld.bfd: warning: " in lines[i - 1]:
return "ld.bfd:" + lines[i - 1].split("ld.bfd:", 1)[-1]
At i == 0 that is lines[-1], the last line of the log. So a log whose
first line is the linker error is described by its last line -- and that
line need not even come from the same file, since JsonReport.create()
appends the handler's stderr to the build log before parsing it.
The same failure then gets reported two ways, depending only on whether
anything was printed before the linker error:
[ld error, "ninja: build stopped.", "...overflowed by 4096 bytes"]
-> "ld.bfd: region overflowed"
["[1/2] Building ...", ld error, "ninja: build stopped.",
"...overflowed by 4096 bytes"]
-> "/usr/bin/ld.bfd: error: ld returned 1 exit status"
Nothing precedes the first line, so the answer is the line itself, which
is already what the function returns when the previous line says nothing
in particular.
scripts/ci/twister_report_analyzer.py reads lines[i - 1] the same way,
parsing the same linker output for the CI job summary. It is a separate
tool with no tests of its own, in a directory this change does not
otherwise touch, so it is left for a change of its own.
Signed-off-by: Egemen Tuncarslan <egementuncarslan@hotmail.com>
Assisted-by: Claude:claude-opus-5
reports.py had no test file. It is 528 statements -- the JSON and XUnit
report writers, and the parsers that turn a build log into the short
reason twister puts in its reports -- and the twister suite covered 10%
of it, all of that incidental to other tests.
This covers the parts that need no fixtures: ReportStatus, the JSON
encoder, process_log, get_detailed_reason, and the two build-failure
parsers against the shapes of log they are written for. Coverage of the
module goes from 10% to 21%.
Signed-off-by: Egemen Tuncarslan <egementuncarslan@hotmail.com>
Assisted-by: Claude:claude-opus-5
Space-separated lists stopped being split in 714e193340, so "a b" is now
taken verbatim and silently matches nothing. Fail loudly for keys whose
members name a tag, platform, feature or toolchain. extra_args and
extra_configs are exempt: their Kconfig values may contain spaces.
Signed-off-by: Flavio Ceolin <flavio@hubble.com>
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Cover the four cases of TwisterEnv._setup_ccache(): the variables it
sets for a normal run, the two settings it leaves alone when they are
already in the environment, and the coverage build it skips.
os.environ is patched for the duration so the test neither depends on
nor leaks ccache settings.
Assisted-by: Claude:opus-5
Signed-off-by: Benjamin Cabé <benjamin@zephyrproject.org>
A sidecar could only discover a missing host tool in setup(), just
before executing the test image. The virtiofs sidecar then skipped
the instance at run time and logged a warning on every host without
a virtiofsd binary, which is noise on CI runners where the daemon is
simply not installed.
Introduce Sidecar.host_ready(), consulted from check_runnable() on a
configured sidecar at test-plan time. When the host lacks what the
sidecar needs, the instance is demoted to build-only, exactly like a
platform whose simulator executable is not installed, instead of the
gap surfacing as a late skip plus a warning. The virtiofs sidecar
implements the hook by checking that a virtiofsd binary was resolved;
the setup() guard remains as a safety net for forced runs.
Document the new lifecycle step and add unit tests for host_ready()
and for its wiring in check_runnable().
Assisted-by: Claude:claude-fable-5
Signed-off-by: Anas Nashif <anas.nashif@intel.com>
BoilerplateFilter uses a partial regular expression match when
classifying changed lines. This can cause substantive lines beginning
with comment delimiter characters, such as #define directives, to be
classified as boilerplate and consumed.
Match the complete changed line instead, while preserving support for
comment-prefixed SPDX and copyright lines.
Add regression coverage for delimiter-prefixed substantive changes and
verify that such files are not consumed.
Assisted-by: ChatGPT:GPT-5.6 Sol
Signed-off-by: Jeewoong Kim <dnd414@gmail.com>
On long runs the scrolling console output makes it hard to tell what
twister is actually doing: what is queued, which pipeline stage every
job is in, and which tests already failed and why. Add a
--console-monitor option that replaces the normal output with a live,
top-style full-screen dashboard for the duration of the run.
The monitoring core (twisterlib/runmonitor.py) is transport-free: the
worker processes emit one plain-dict event per pipeline op
start/completion over a multiprocessing.Queue (put_nowait, dropped on
any error, so monitoring can never stall or fail the pipeline), and a
drain thread in the main process folds them into a thread-safe state
model with one row per test instance, initialized from the test plan
so statically filtered instances are visible too. The curses UI
(twisterlib/consolemonitor.py) runs in a thread of the main process
and reads that state in-process; other frontends (e.g. a web view)
can be layered on the same core later.
The dashboard shows overall progress with counters and an ETA, the
in-flight jobs with their current pipeline stage, and a scrollable
instance table with filter tabs. f jumps to the failures view, /
starts an incremental search over instance names and failure reasons,
and arrows or j/k plus Enter open a per-instance detail view: the
pipeline stage timeline, the failing test cases with their reasons,
and scrollable log tails (l switches between the available logs).
While the UI owns the terminal, console log handlers are detached
(everything still lands in twister.log) and restored when it exits;
the workers' single-line progress ticker is skipped; the handlers'
`stty sane` reset after local runs is suppressed and simulator
processes get /dev/null as stdin so nothing can knock the terminal
out of curses mode - the UI also re-asserts its terminal modes on
every poll tick as a backstop. These worker-side accommodations are
keyed to a shared UI-active flag (a module-level multiprocessing
Value inherited by the forked workers - deliberately not an options
attribute, since options travels inside pickled TestInstance objects
through the manager queues where a Value cannot go), so they all
revert the moment the UI exits.
Leaving the dashboard always hands the terminal back cleanly: q
mid-run resumes the normal console output (including the progress
ticker) while the run continues; when the run finishes the dashboard
stays up for inspecting failures and q then writes the reports and
exits as usual; Ctrl-C aborts the run, shuts the dashboard down
without waiting for a keypress, and skips any retry iterations. The
option requires an interactive terminal and curses, and is ignored
with a warning otherwise (e.g. in CI).
Assisted-by: Claude:claude-fable-5
Signed-off-by: Anas Nashif <anas.nashif@intel.com>
Harness.process_test() failed a run on a kernel fatal error only when
two conditions held: the console line had to be exactly equal to
"ZEPHYR FATAL ERROR", and the fault had to be seen before the test
printed "PROJECT EXECUTION SUCCESSFUL". Neither holds in practice.
z_fatal_error() prints ">>> ZEPHYR FATAL ERROR %d: %s on CPU %d",
normally through LOG_ERR and therefore behind a timestamp and a module
prefix, so no console line ever compares equal to the marker. The
Console harness has always matched it as a substring and worked; the
Test and Ztest harnesses rely on process_test() alone, so for them
fault detection never triggered at all. The existing unit test passed
the bare marker as the whole line, which is why this went unnoticed.
Match the marker as a substring, move the check ahead of the RUN_PASSED
branch, and withdraw a PASS that was recorded earlier in the output.
Tests that fault on purpose are unaffected: they opt out with
ignore_faults, which clears fail_on_fault. Drop the now duplicated
check in Console.handle(), which process_test() covers.
Signed-off-by: Anas Nashif <anas.nashif@intel.com>
The QEMU monitor thread's _thread_update_instance_info() stamped the
instance reason with "Unknown Error" whenever the harness provided no
reason, regardless of the resulting status. Passing harnesses supply
no reason, so every passing QEMU test instance silently carried
reason="Unknown Error". The classic console output only prints the
reason for failures, which kept this invisible, but trying to view
status using other means reveals this issue.
Only fall back to "Unknown Error" when the status is actually a
failure (failed/error) and no more specific reason is known. For
non-failure results clear the reason instead of keeping it, so a
stale reason from an earlier retry iteration cannot stick to a
now-passing instance.
Signed-off-by: Anas Nashif <anas.nashif@intel.com>
The script only ever adds to a pull request: labels, review requests and
an assignee accumulate across runs. There is no way to correct one that
was staffed badly, or to re-apply the current MAINTAINERS.yml to a pull
request that was opened before an area changed hands.
Add --reset, which works out the new staffing first and then, just
before applying it, undoes what an earlier run decided: it withdraws the
review requests, removes every assignee and strips the area labels that
MAINTAINERS.yml defines, leaving the pull request labelled, reviewed and
assigned as if it had just been opened.
Doing this late rather than on entry matters twice over. Every skip
condition -- draft, closed, more than MAX_FILES changed files -- has had
its say by then, so a reset cannot strip a pull request the run goes on
to decline. And the reviewers about to be requested again are known, so
their existing request is left in place instead of being withdrawn and
re-sent, which would notify them twice.
Four things survive the reset:
- Reviewers who submitted a review or commented on the pull request or
one of its review threads. They are involved by their own action,
and dropping their request would either lose that or notify them
again. This covers approvals and changes-requested reviews
regardless of the verdict.
- Team review requests. The script only ever requests individuals, so
a team was requested by hand and a reset has no basis to withdraw it.
- Size labels, which update_size_labels recomputes and prunes on every
run anyway; removing them here would double-remove.
- Labels the script does not own, such as 'bug' or backport labels.
They are not derived from MAINTAINERS.yml, so a reset has no basis
for second-guessing whoever added them.
People who removed themselves from the review request are not re-added:
_add_reviewers already refuses to request them, and the reset's own
removals are made by the token account, so they do not read as
self-removals either.
Assisted-by: Claude:claude-opus-5
Signed-off-by: Anas Nashif <anas.nashif@intel.com>
A pull request matching more than MAX_LABELS (10) area labels had its
label step skipped entirely, on the grounds that labelling a change with
a dozen areas says nothing. The result was worse: the broadest pull
requests, the ones hardest to route, ended up with no area label at all.
Rank the collected labels by how much of the pull request the areas
carrying them cover, and apply the MAX_LABELS highest-ranked ones
instead, so the same limit now caps how many labels are applied rather
than deciding whether any are. A label's rank is the sum of the file
weights of every touched area carrying it, falling back to the plain
file count for areas weighted 0. Those weights are zeroed for assignee
selection -- meta-areas, files that only matched CMakeLists.txt,
repeated matches of one Platform area -- but such an area still
describes what the pull request changes, and without the fallback a
documentation-only pull request would rank 'area: Documentation' last
and drop it first. Ties break by name to keep the outcome
deterministic.
Size labels take no part in this. They are computed for the pull
request alone and update_size_labels has already pruned the stale ones,
so they neither count towards the limit -- ten area labels plus a size
label is not an overflow -- nor can be dropped by it.
The limit governs what a run applies; no label is ever taken off the
pull request, whether a human or an earlier run put it there. --reset
is the way to clear those.
Assisted-by: Claude:claude-opus-5
Signed-off-by: Anas Nashif <anas.nashif@intel.com>
Changed files that MAINTAINERS.yml cannot staff (files in under-covered
areas, files in no area at all) fall back to two heuristics: GitHub's own
suggestedReviewers, then the recent contributors to those files. Both
ran whenever such a file was present, however well staffed the rest of
the pull request already was, so a single orphaned file was enough to
append lower-confidence names to a review request that already held the
people responsible for the change.
Skip both heuristics once the touched areas have yielded
MIN_TOTAL_REVIEWERS (3) candidates other than the author, who cannot
review their own pull request. The count is taken across all areas, so
two thinly staffed areas that together cover the pull request no longer
trigger the fallback either.
Also limit the commit walk to the last HISTORY_MAX_AGE_DAYS (365) days.
A contributor who has not touched a file within a year is unlikely to
still be the right person to ask, and on long-lived files the unbounded
history filled every heuristic slot with people who have moved on. The
window is applied by GitHub via the 'since' parameter, so it also cuts
the API cost of the walk.
Assisted-by: Claude:claude-opus-5
Signed-off-by: Anas Nashif <anas.nashif@intel.com>
An area that names no maintainers, or at most two collaborators, is
considered under-covered: it cannot staff a review on its own, so recent
Git contributors to its files are added as heuristic reviewers.
That test misfires on meta-areas. They match large parts of the tree by
design and name few people on purpose, so a documentation- or
samples-only pull request would walk the history of every file it
touches and request reviews from people on top of the maintainers and
collaborators who already own the area.
Skip areas carrying 'meta: true' when collecting under-covered areas.
Files matched only by a meta-area no longer feed the heuristic; files
that a thin non-meta area also claims still do, as do orphaned files
that match no area at all.
Assisted-by: Claude:claude-opus-5
Signed-off-by: Anas Nashif <anas.nashif@intel.com>
Meta-areas, i.e. areas that cut across the whole tree instead of owning
one subsystem, were hardcoded as name lists in two CI scripts:
set_assignees.py (META_AREAS) and sync_maintainer_tests.py
(_META_AREAS). The two lists had drifted apart and both had gone stale:
they named "Release Notes" and "Boards", neither of which is an area in
MAINTAINERS.yml anymore.
Move the classification into MAINTAINERS.yml itself as an optional
per-area 'meta: true' key, so it lives in one place next to the area it
describes and new meta-areas can be added without touching any script.
get_maintainer.py validates the key as a boolean, exposes it as
Area.meta and prints it alongside the other area fields.
Both scripts now read that key instead of their own list. As the two
lists are unified, membership changes slightly: Benchmarks is no longer
weighted when picking a pull request assignee, and Documentation and
Release no longer get auto-generated 'tests' entries.
Assisted-by: Claude:claude-opus-5
Signed-off-by: Anas Nashif <anas.nashif@intel.com>
