A headless Linux NFS fileserver has been running slow for the last couple days (based on subjective reports from users). I checked journalctl did not see any relevant errors.
However, when I connected a monitor, I was greeted by a screen full of these errors:
failed command: WRITE FPDMA QUEUED
Here's a photo:
What are some suggested next steps? Should I just replace the disk?
Only one disk is having issues, according to what is visible here. But it's not obvious whether it's the drive, or the cable, or the controller.
I would first reboot, to reset the hardware. It could just be that the controller was temporarily confused and literally turning it off and on again would help.
If the error comes back after rebooting, then I would plug that drive into a different SATA port, and if necessary plug some other drive into that port. If the error is still on the same port, then the problem is with the controller.
But if the error "moves" to the new port, then it's either the cable or the drive. At this point I would replace the SATA cable. If the error goes away, then it was a bad cable. If you still have the error, then it's the drive.
Another thing you can try (which works remotely) is to run smartctl -a to see if the drive is reporting any errors, and perhaps smartctl -t short to run a self-test on it.
In my case it revealed that the WRITE FPDMA QUEUED failures were due to an ICRC (interface CRC) error, which means the data was corrupted between the drive and the controller, so the disk itself is ok and it's either the SATA cable, or the circuitry at either end that the SATA cable plugs into.
While I'm no expert, presumably in this case the command was retried and eventually got through the SATA cable without being corrupted, resulting in a functioning system but with a very slow disk due to all the retries.
