The tool has caused some hard to detect failures when used
concurrently with the SOF Linux driver to observe logs. This
has been rootcaused to the code to handle and send IPC messages
that has been active even in logs-only mode. In vast majority
of cases, the SOF kernel driver is faster to act and handles
the IPC interrupts. But occasionally the cavstool.py timing was
just right and it managed to ack a message before Linux driver.
Fix the problem by not handling IPC messages when the tool
is run in logs-only mode.
Reported-by: Peter Ujfalusi <peter.ujfalusi@linux.intel.com>
Signed-off-by: Kai Vehmanen <kai.vehmanen@linux.intel.com>
* Adds a default hook and init function for cavstool.
* Adds an optional padding on flush feature to ensure all data is written.
* Fixes an error in cavstool.py for correctly wrapping the ring buffer.
* The test case now ensures wraps and flushes occur numerous times.
Signed-off-by: Tom Burdick <thomas.burdick@intel.com>
Because we use INADDR_ANY by default now, specifying a net interface
is no longer needed. Remove it and remove the dependency of the
netifaces python package.
Fixes. #48584
Signed-off-by: Enjia Mai <enjia.mai@intel.com>
Make the cavstool service can start with specifying the port
optionally by:
cavstool.py -s {host}:{log port} --req-port {req port}
OR
cavstool.py -s {host} --log-port {log port} --req-port {req port}
And we can also specify the network interface instead:
cavstool.py -i {network iface}
If the server address or the network interface is not specified,
it will use INADDR_ANY by default.
Signed-off-by: Enjia Mai <enjia.mai@intel.com>
Uses a macro with a define flag to enable register dumps on the DSP
side. On the python side a simple booling flag.
The default disabled both debug flags and makes the tests
considerably quieter.
Signed-off-by: Tom Burdick <thomas.burdick@intel.com>
Change some behaviors when --log-only is passed in order to add support
for DSP power state: D3. This makes it possible to keep `cavstool
--log-only` running permanently in the "real-world" use case where the
Linux kernel loads the firmware and powers down audio when not in use.
More specifically:
- Do not disable D3 when using --log-only.
- Wait forever for the FW at boot time.
- Check live status when an invalid IPC is received and wait forever if
not alive.
Signed-off-by: Marc Herbert <marc.herbert@intel.com>
Add a new timeout_s parameter that can also be 'None' = infinite.
No functional change. Required to add future support for DSP power
state: D3
Signed-off-by: Marc Herbert <marc.herbert@intel.com>
1. Improve the firmware transfer reliability by fixing the misuse
of the socket. Fix the most frequent occurence of the common `recv()`
bug described here:
https://docs.python.org/3/howto/sockets.html#using-a-socket
The longer term fix is to switch to a higher level API like Python
Remote Objects.
2. Not rely on the client's command to disconnect. Previously we
rely on the SIGINT to send stop_command to the server, but it does
not work well in some environments. Refine the whole logic and the
sever disconnect service by checking if the client is alive or not.
These changes make the client-server-based cavstool more stable.
Fixes#46864
Signed-off-by: Enjia Mai <enjia.mai@intel.com>
A full second is too much, and since "Ack local interrupt before
processing IPC" patch, it doesn't seem necessary. This whole second
wait would break tests that log too much content, as the firmware
is already running during the reset - and the host script wouldn't read
the contents, thus some of the logs would be overwritten.
Signed-off-by: Ederson de Souza <ederson.desouza@intel.com>
It seems a race can happen between the host acknowledging a DONE
interrupt and the DSP being able to mark some IPC as done. To avoid
this, the host script now always acknowledges the DONE interrupt
_before_ processing the IPC message.
Signed-off-by: Ederson de Souza <ederson.desouza@intel.com>
They may be disabled in some environments (such as Gentoo based
developer mode on Chromebooks). Use extended regular expressions
instead.
Signed-off-by: Ederson de Souza <ederson.desouza@intel.com>
When using more than one core on cavs25, some hangs were made
"persistent", as it seems not all cores were being properly reset, thus
compromising tests - if a test hangs for any reason, subsequent tests
that were not restrict to a single CPU would also fail.
This patch mitigates these issues by two changes:
- Closely mimics SOF Linux driver way of loading the firmware. So,
explicit stall and reset (and "unstall" and "unreset") of cores, with
appropriate checks that states have been reached;
- More generous sleep before resetting the stream.
Also, the status of ADSPCS (Audio DSP Control and Status) register is
logged more thoroughly to aid debugging in case new issues arise.
Signed-off-by: Ederson de Souza <ederson.desouza@intel.com>
The name change for cavstool.py has also broken the backward
compatibility for SOF testing. Rename cavstool_server.py back
to cavstool.py. Keep the functionality still as same as the previous
one.
And also update the documentation of it.
Signed-off-by: Enjia Mai <enjia.mai@intel.com>
Make the intel_adsp west runner starting to work on all the
intel_adsp boards. Changes include:
1. Make the cavstool.py work as a service in remote host
ADSP board and rename it to cavstool_server.py.
2. Active the runner and adds a common board.cmake file to
specify the default signing key for cavs boards.
Signed-off-by: Enjia Mai <enjia.mai@intel.com>
Adds a log backend that maintains a ringbuffer in coordination
with cAVS HDA.
The DMA channel is expected to be given some time after the logger
starts so a seperate step to initialize the dma channel is required.
Signed-off-by: Tom Burdick <thomas.burdick@intel.com>
Adds a header only low level driver for HDA streams along with smoke
tests to ensure basic host in and out stream functionality.
The tests require host side interaction. In cavstool a new HDAStream
class encapsulates somewhat a single stream and its registers. This
is manipulated in the tests using IPC with the Host ensuring that a
specific order of operations is done.
This low level driver allows testing certain hardware configurations
and flows with easy to use register dump debugging. It is not
intended to be the end API an application might use. That would be
a DMA driver using this.
Signed-off-by: Tom Burdick <thomas.burdick@intel.com>
Add a fairly simple test of the IPM-over-IPC driver. This hits all
the code, but works by implementing the host side of the protocol
partially in the C test code. The message is sent with an initial
payload, and then IPC commands from the firmware copy the data over
into the "inbox" region to simulate data being sent via the host.
Then we make sure it lands correctly as if the host driver had done it
directly.
This requries a new command in the cavstool script that will copy a
word from the "outbox" region to the "inbox" region (both are just
different SRAM windows, conceptually no different than the way the
script is already managing log output), but no significant surgery.
Signed-off-by: Andy Ross <andrew.j.ross@intel.com>
Now that we have easy access to code on the host, it's trivial to
check the clock against host timestamps with high precision.
Signed-off-by: Andy Ross <andrew.j.ross@intel.com>
On cAVS 1.8 (specifically) there seems to be a propagation delay on
the IPC registers. Hitting the TDA register to signal DONE too soon
after clearing the interrupt via TDR can cause the interrupt to be
dropped. Merely polling for it to read back correctly isn't
sufficient, we need an actual sleep here.
(The behavior that a message won't send while an existing message is
in progress is actually a hardware feature that is new with 1.8. My
guess is it's a little glitchy in its first version.)
Signed-off-by: Andy Ross <andrew.j.ross@intel.com>
On cAVS before 2.5, core power was controlled by the host. Add a
command to the cavstool.py script to allow us to do that under test
command so we can exercise multiprocessor startup/shutdown outside of
SOF.
Signed-off-by: Andy Ross <andrew.j.ross@intel.com>
As Zephyr begins to absorb drivers for these platforms that had
previously been managed by the SOF app, there's a need for a rapid
board-specific smoke test to use during development.
This starts with the smp_boot_delay test (itself a unit test for a
SOF-derived feature) and adds a host IPC case (that needs to match
code in cavstool.py on the other side of the PCI bus!).
It will grow more features over time as needed.
Signed-off-by: Andy Ross <andrew.j.ross@intel.com>
Failures to boot are not exclusive to the cavstool.py, they can happen
with the kernel driver and --log-only too. For such a situation this
commit adds a useful delay and these two log lines (before the mmap
crashes eventually):
INFO:cavs-fw:Waiting for firmware handoff, FW_STATUS = 0x81000012
WARNING:cavs-fw:Load failed? FW_STATUS = 0x1006701
Signed-off-by: Marc Herbert <marc.herbert@intel.com>
Audio users want logging too. This restores feature parity since the
older scripts were removed in commit
cd5302fa00 ("boards/intel_adsp_cavs15: Remove ancient tooling")
Signed-off-by: Marc Herbert <marc.herbert@intel.com>
Don't "crash" when passing no argument at all.
Log exceptions, adding for instance the second line:
ERROR:cavs-fw:Could not map device in sysfs; run as root?
ERROR:cavs-fw:[Errno 13] Permission denied: \
'/sys/bus/pci/devices/0000:00:0e.0/power/control'
Signed-off-by: Marc Herbert <marc.herbert@intel.com>
There is a hardware bug with stream reset, it won't stop the stream.
I thought it was limited to just the early versions, but it turns out
that 2.5 has it too in some variants. Which is frustrating, because
the TGL chromebook I have doesn't like the START clear. But all
systems work with an extra delay between them, so do that.
Signed-off-by: Andy Ross <andrew.j.ross@intel.com>
This platform was stale for a long time and got a little left behind.
Basic OS stuff was working but secondary core bringup didn't. It has
a slightly different set of choices from the "weird hardware
quirks" menu:
+ Like cAVS 1.5, it boots from a ROM that needs a short delay after
power-up before it can receive the startup IDC.
+ But, like 2.5 and unlike 1.5, it doesn't start running until the
PWRCTL bit for the core gets set by DSP software (1.5 gets launched
by the host). So the delay needed to move down a bit.
+ It wants that PWRCTL bit to be set last, after CLKCTL enables the
clock. (Which makes sense I guess: EE classes always tell you hold
circuits in reset while an initial clock propagates). Not sure why
it was in the reverse order originally; this way works for
everything.
+ The ROM likes to scribble on the interrupt controller and mask its
own IDC interrupts after we've already set it up. They have to be
unmasked. We had code to do this already, thinking it was a
workaround for legecy SOF code (that we never actually located).
Now I'm thinking it was this behavior all along being detected by
SOF's more extensive hardware CI. Take out the test and do it
always, it's like nine instructions.
+ The host/loader-side behavior is a mix of 1.5 and 2.5. It won't
actually start the secondary cores under host command, but it does
need to see bits set for them in ADSPCS for the DSP-initiated
power-up to work (2.5 would just ignore all but core 0's bits).
+ Also, like 1.5, it needs the host DMA stream to be explicitly
stopped (and not just reset) or else further loads will be unstable.
Note that the loader changes now require more logic than just "1.5 or
not", so the platform detection has been enhanced to fully categorize
the device based on PCI ID (not quite: we don't have any 2.0 platform
hardware, so I left that alone for now).
Signed-off-by: Andy Ross <andrew.j.ross@intel.com>
There's desire for a "log only" mode like the older adsplog script
had. Add a few other quality of life command line options too. And
catch the most obvious user errors to print a message instead of a
stack dump.
Signed-off-by: Andy Ross <andrew.j.ross@intel.com>
The existing scripting for these platforms has gotten a little stale.
The loader had bifurcated into a v15 and v25 variant, both of which
lived in the cavs15 board directory. Building off Shao Ming's
excellent (if somewhat surprisingly committed) rework to unify
unchanged parts of the scripts, let's finish the job.
This adds a "cavstool.py" script with the following advantages:
+ It's just one script for everything, with a single unified load
process that works reliably on both 1.5 and 1.8+ hardware.
+ It runs on all cAVS platforms (with a compatible kernel, those
requirements haven't changed)
+ It automatically emitts logging synchronously after loading,
eliminating the race between adsplog.py and cavs-fw.py where you
could see logging from a previous test run.
+ It automatically detects and unloads a linux kernel module managing
the same device (even if SOF has renamed the module again, heh).
+ Timings have been tuned up in general, it's about 2 seconds faster
to get to first log output now.
Signed-off-by: Andy Ross <andrew.j.ross@intel.com>
