IMX fsl_micfil driver hang because DMA is not available yet

Question

I'm working on an embedded device based on an NXP i.MX8M mini SoC. It is running Linux based on NXP's "hardknott" Yocto recipe: https://source.codeaurora.org/external/imx/imx-manifest/tree/imx-5.10.52-2.1.0.xml?h=imx-linux-hardknott

Here's the bitbake script for the kernel it is using: https://source.codeaurora.org/external/imx/meta-imx/tree/meta-bsp/recipes-kernel/linux?h=hardknott-5.10.52-2.1.0

I believe it is pulling the kernel sources from here: https://source.codeaurora.org/external/imx/linux-imx/tree/?h=lf-5.10.y

Our device has a digital microphone that is being driven by a modified version of NXP's fsl_micfil.c driver: https://source.codeaurora.org/external/imx/linux-imx/tree/sound/soc/fsl/fsl_micfil.c?h=lf-5.10.y

Our modifications are:

• In fsl_micfil_hw_params, where their code does "enable channels" (around line 1592), we've changed it to only enable the channel we care about instead of all of them.

  We changed:

  ret = regmap_update_bits(micfil->regmap, REG_MICFIL_CTRL1,
                           0xFF, ((1 << channels) - 1));
  

  To:

  ret = regmap_update_bits(micfil->regmap, REG_MICFIL_CTRL1,
                           0xFF, 1 << 6);
  

• At the start of fsl_micfil_probe (around line 2202), we add code to pull a GPIO line (the mic's enable line) low. If GPIO isn't available at the time, we return -EPROBE_DEFER in order to try again later

• Further down in fsl_micfil_probe (around line 2334), we change the DMA channel from 0 to 6 (to align with our enable change).

  We changed:

  micfil->dma_params_rx.addr = res->start + REG_MICFIL_DATACH0;
  

  To:

  micfil->dma_params_rx.addr = res->start + REG_MICFIL_DATACH6;
  

That having been described, here's the problem.

Our application is trying to read audio from this microphone (we can test it with the arecord command).

We have found that if we launch our application immediately after Linux boots, the device hangs. We need to power-cycle it to recover.

If we wait about 60 seconds after bootup, however, we see the following messages appear on the console:

[   60.386133] imx-sdma 302c0000.dma-controller: firmware found.
[   60.392010] imx-sdma 302c0000.dma-controller: loaded firmware 4.6

If we launch our application after these messages appear, everything works fine.

I looked through the implementation of the imx-sdma driver: https://source.codeaurora.org/external/imx/linux-imx/tree/drivers/dma/imx-sdma.c?h=lf-5.10.y

The driver loads some kind of firmware when it starts up. The first message ("firmware found") is presented by the probe function. The second ("loaded firmware") is presented by one of the two power management resume functions (sdma_resume or sdma_runtime_resume).

So it appears that the problem is that this DMA driver is not loading until the system has been running for about 60 seconds. I assume this is due to some kind of lazy initialization that waits until some kernel driver requires access. And my problem is happening because my driver requires access and is initializing before the DMA driver has loaded.

So, what's a good fix for this? I assume I need to add something to the fsl_micfil driver to request startup of the DMA driver. But I don't know how to do this and I don't actually know if that is the correct fix for this problem.

What would you suggest as a proper fix for this bug?

(FWIW, we made very similar changes to an older version of the kernel - based on Yocto's "sumo" branch, running kernel 4.14, and there are no problems there. So clearly something changed between kernel 4.14 and 5.10 that our code needs to deal with, but I don't know what that might be.)

Answer 1

I eventually figured out the reason (but not the best solution) for the problem.

As I wrote in the original message, the micfil driver uses imx-sdma, which tries to load firmware from the file system.

The "Sumo" (kernel 4.14) branch has no problem with this. At the time the driver starts, the root file system has been mounted (read-only), so it has no problem loading its firmware.

With the "Hardknott" (kernel 5.10) and "Kirkstone" (kernel 5.15) branches, however, the SDMA driver tries to load very very early in the boot sequence - before any file systems have mounted. So direct loading fails. It then posts an event via udev, to request that user-space code provide the firmware via sysfs. But the driver appears to be starting so early (within 10-15 ms, according to dmesg) that udev isn't able to queue up the event. I'm not sure about the specific reason why but debugging shows that the event is never delivered to user space.

The kernel side of the udev request times out after 60 seconds, at which point the driver retries the request, which succeeds because all the file systems are fully mounted by that point.

An ideal fix would be to modify the imx-sdma driver so it doesn't start until after the file system mounts, so it can directly load its firmware. Or at minimum, after udev is ready and able to deliver the firmware-load event to user space. I let NXP know about this issue, so maybe they'll fix it in a future release of their driver.

Until that happens, my workaround is to change the kernel configuration so the imx-sdma driver is compiled as a loadable module. This means it can't load and start until after the root file system is available, at which point it has no problem finding its firmware. dmesg shows that this happens about 6s into the boot sequence - later than I'd prefer, but early enough that no application software has started running, so nothing hangs or crashes.