Dmesg is full of I/O errors on Ubuntu 18.04 VM, physical disks and host are fine

Question

I have an Ubuntu 18.04 virtual machine on a Windows Server 2016 host (Dell PowerEdge R530, Intel Xeon E5-2620 @ 2.1GHz, 96GB RAM), running in VirtualBox 6.1.10. The host system has 2x2TB HDDs in RAID 1 via a PERC H730 Mini controller. The VM runs an Elasticsearch node and some other software used to process incoming data. It has two virtual disks connected to it (155GB, 1.5TB), but it's only the 2nd one that's acting up. It was fine for several months, but recently it has started throwing I/O errors seemingly randomly.

Here's some dmesg output from yesterday:

[Mon May 10 20:37:04 2021] ata4.00: exception Emask 0x0 SAct 0x0 SErr 0x0 action 0x0
[Mon May 10 20:37:04 2021] ata4.00: irq_stat 0x40000000
[Mon May 10 20:37:04 2021] ata4.00: failed command: WRITE DMA EXT
[Mon May 10 20:37:04 2021] ata4.00: cmd 35/00:40:00:09:41/00:05:16:00:00/e0 tag 21 dma 688128 out
                                   res 41/10:40:00:09:41/00:05:16:00:00/e0 Emask 0x81 (invalid argument)
[Mon May 10 20:37:04 2021] ata4.00: status: { DRDY ERR }
[Mon May 10 20:37:04 2021] ata4.00: error: { IDNF }
[Mon May 10 20:37:04 2021] ata4.00: configured for UDMA/133
[Mon May 10 20:37:04 2021] ata4: EH complete
[Mon May 10 20:37:04 2021] ata4.00: exception Emask 0x0 SAct 0x0 SErr 0x0 action 0x0
[Mon May 10 20:37:04 2021] ata4.00: irq_stat 0x40000000
[Mon May 10 20:37:04 2021] ata4.00: failed command: WRITE DMA EXT
[Mon May 10 20:37:04 2021] ata4.00: cmd 35/00:c0:40:0e:41/00:02:16:00:00/e0 tag 15 dma 360448 out
                                   res 41/10:c0:40:0e:41/00:02:16:00:00/e0 Emask 0x81 (invalid argument)
[Mon May 10 20:37:04 2021] ata4.00: status: { DRDY ERR }
[Mon May 10 20:37:04 2021] ata4.00: error: { IDNF }
[Mon May 10 20:37:04 2021] ata4.00: configured for UDMA/133
[Mon May 10 20:37:04 2021] ata4: EH complete
[Mon May 10 20:37:04 2021] ata4.00: exception Emask 0x0 SAct 0x0 SErr 0x0 action 0x0
[Mon May 10 20:37:04 2021] ata4.00: irq_stat 0x40000000
[Mon May 10 20:37:04 2021] ata4.00: failed command: WRITE DMA EXT
[Mon May 10 20:37:04 2021] ata4.00: cmd 35/00:40:00:11:41/00:05:16:00:00/e0 tag 31 dma 688128 out
                                   res 41/10:40:00:11:41/00:05:16:00:00/e0 Emask 0x81 (invalid argument)
[Mon May 10 20:37:04 2021] ata4.00: status: { DRDY ERR }
[Mon May 10 20:37:04 2021] ata4.00: error: { IDNF }
[Mon May 10 20:37:04 2021] ata4.00: configured for UDMA/133
[Mon May 10 20:37:04 2021] ata4: EH complete
[Mon May 10 20:37:04 2021] ata4.00: exception Emask 0x0 SAct 0x0 SErr 0x0 action 0x0
[Mon May 10 20:37:04 2021] ata4.00: irq_stat 0x40000000
[Mon May 10 20:37:04 2021] ata4.00: failed command: WRITE DMA EXT
[Mon May 10 20:37:04 2021] ata4.00: cmd 35/00:40:00:09:41/00:05:16:00:00/e0 tag 7 dma 688128 out
                                   res 41/10:40:00:09:41/00:05:16:00:00/e0 Emask 0x81 (invalid argument)
[Mon May 10 20:37:04 2021] ata4.00: status: { DRDY ERR }
[Mon May 10 20:37:04 2021] ata4.00: error: { IDNF }
[Mon May 10 20:37:04 2021] ata4.00: configured for UDMA/133
[Mon May 10 20:37:04 2021] ata4: EH complete
[Mon May 10 20:37:04 2021] ata4.00: exception Emask 0x0 SAct 0x0 SErr 0x0 action 0x0
[Mon May 10 20:37:04 2021] ata4.00: irq_stat 0x40000000
[Mon May 10 20:37:04 2021] ata4.00: failed command: WRITE DMA EXT
[Mon May 10 20:37:04 2021] ata4.00: cmd 35/00:c0:40:0e:41/00:02:16:00:00/e0 tag 15 dma 360448 out
                                   res 41/10:c0:40:0e:41/00:02:16:00:00/e0 Emask 0x81 (invalid argument)
[Mon May 10 20:37:04 2021] ata4.00: status: { DRDY ERR }
[Mon May 10 20:37:04 2021] ata4.00: error: { IDNF }
[Mon May 10 20:37:04 2021] ata4.00: configured for UDMA/133
[Mon May 10 20:37:04 2021] ata4: EH complete
[Mon May 10 20:37:04 2021] ata4.00: exception Emask 0x0 SAct 0x0 SErr 0x0 action 0x0
[Mon May 10 20:37:04 2021] ata4.00: irq_stat 0x40000000
[Mon May 10 20:37:04 2021] ata4.00: failed command: WRITE DMA EXT
[Mon May 10 20:37:04 2021] ata4.00: cmd 35/00:40:00:11:41/00:05:16:00:00/e0 tag 23 dma 688128 out
                                   res 41/10:40:00:11:41/00:05:16:00:00/e0 Emask 0x81 (invalid argument)
[Mon May 10 20:37:04 2021] ata4.00: status: { DRDY ERR }
[Mon May 10 20:37:04 2021] ata4.00: error: { IDNF }
[Mon May 10 20:37:04 2021] ata4.00: configured for UDMA/133
[Mon May 10 20:37:04 2021] ata4: EH complete
[Mon May 10 20:37:04 2021] ata4.00: exception Emask 0x0 SAct 0x0 SErr 0x0 action 0x0
[Mon May 10 20:37:04 2021] ata4.00: irq_stat 0x40000000
[Mon May 10 20:37:04 2021] ata4.00: failed command: WRITE DMA EXT
[Mon May 10 20:37:04 2021] ata4.00: cmd 35/00:40:00:09:41/00:05:16:00:00/e0 tag 0 dma 688128 out
                                   res 41/10:40:00:09:41/00:05:16:00:00/e0 Emask 0x81 (invalid argument)
[Mon May 10 20:37:04 2021] ata4.00: status: { DRDY ERR }
[Mon May 10 20:37:04 2021] ata4.00: error: { IDNF }
[Mon May 10 20:37:04 2021] ata4.00: configured for UDMA/133
[Mon May 10 20:37:04 2021] ata4: EH complete
[Mon May 10 20:37:04 2021] ata4.00: exception Emask 0x0 SAct 0x0 SErr 0x0 action 0x0
[Mon May 10 20:37:04 2021] ata4.00: irq_stat 0x40000000
[Mon May 10 20:37:04 2021] ata4.00: failed command: WRITE DMA EXT
[Mon May 10 20:37:04 2021] ata4.00: cmd 35/00:c0:40:0e:41/00:02:16:00:00/e0 tag 8 dma 360448 out
                                   res 41/10:c0:40:0e:41/00:02:16:00:00/e0 Emask 0x81 (invalid argument)
[Mon May 10 20:37:04 2021] ata4.00: status: { DRDY ERR }
[Mon May 10 20:37:04 2021] ata4.00: error: { IDNF }
[Mon May 10 20:37:04 2021] ata4.00: configured for UDMA/133
[Mon May 10 20:37:04 2021] ata4: EH complete
[Mon May 10 20:37:04 2021] ata4.00: exception Emask 0x0 SAct 0x0 SErr 0x0 action 0x0
[Mon May 10 20:37:04 2021] ata4.00: irq_stat 0x40000000
[Mon May 10 20:37:04 2021] ata4.00: failed command: WRITE DMA EXT
[Mon May 10 20:37:04 2021] ata4.00: cmd 35/00:40:00:11:41/00:05:16:00:00/e0 tag 16 dma 688128 out
                                   res 41/10:40:00:11:41/00:05:16:00:00/e0 Emask 0x81 (invalid argument)
[Mon May 10 20:37:04 2021] ata4.00: status: { DRDY ERR }
[Mon May 10 20:37:04 2021] ata4.00: error: { IDNF }
[Mon May 10 20:37:04 2021] ata4.00: configured for UDMA/133
[Mon May 10 20:37:04 2021] ata4: EH complete
[Mon May 10 20:37:04 2021] ata4.00: exception Emask 0x0 SAct 0x0 SErr 0x0 action 0x0
[Mon May 10 20:37:04 2021] ata4.00: irq_stat 0x40000000
[Mon May 10 20:37:04 2021] ata4.00: failed command: WRITE DMA EXT
[Mon May 10 20:37:04 2021] ata4.00: cmd 35/00:40:00:09:41/00:05:16:00:00/e0 tag 31 dma 688128 out
                                   res 41/10:40:00:09:41/00:05:16:00:00/e0 Emask 0x81 (invalid argument)
[Mon May 10 20:37:04 2021] ata4.00: status: { DRDY ERR }
[Mon May 10 20:37:04 2021] ata4.00: error: { IDNF }
[Mon May 10 20:37:04 2021] ata4.00: configured for UDMA/133
[Mon May 10 20:37:04 2021] ata4: EH complete
[Mon May 10 20:37:04 2021] ata4.00: exception Emask 0x0 SAct 0x0 SErr 0x0 action 0x0
[Mon May 10 20:37:04 2021] ata4.00: irq_stat 0x40000000
[Mon May 10 20:37:04 2021] ata4.00: failed command: WRITE DMA EXT
[Mon May 10 20:37:04 2021] ata4.00: cmd 35/00:c0:40:0e:41/00:02:16:00:00/e0 tag 7 dma 360448 out
                                   res 41/10:c0:40:0e:41/00:02:16:00:00/e0 Emask 0x81 (invalid argument)
[Mon May 10 20:37:04 2021] ata4.00: status: { DRDY ERR }
[Mon May 10 20:37:04 2021] ata4.00: error: { IDNF }
[Mon May 10 20:37:04 2021] ata4.00: configured for UDMA/133
[Mon May 10 20:37:04 2021] ata4: EH complete
[Mon May 10 20:37:04 2021] ata4.00: exception Emask 0x0 SAct 0x0 SErr 0x0 action 0x0
[Mon May 10 20:37:04 2021] ata4.00: irq_stat 0x40000000
[Mon May 10 20:37:04 2021] ata4.00: failed command: WRITE DMA EXT
[Mon May 10 20:37:04 2021] ata4.00: cmd 35/00:40:00:11:41/00:05:16:00:00/e0 tag 17 dma 688128 out
                                   res 41/10:40:00:11:41/00:05:16:00:00/e0 Emask 0x81 (invalid argument)
[Mon May 10 20:37:04 2021] ata4.00: status: { DRDY ERR }
[Mon May 10 20:37:04 2021] ata4.00: error: { IDNF }
[Mon May 10 20:37:04 2021] ata4.00: configured for UDMA/133
[Mon May 10 20:37:04 2021] ata4: EH complete
[Mon May 10 20:37:04 2021] ata4.00: exception Emask 0x0 SAct 0x0 SErr 0x0 action 0x0
[Mon May 10 20:37:04 2021] ata4.00: irq_stat 0x40000000
[Mon May 10 20:37:04 2021] ata4.00: failed command: WRITE DMA EXT
[Mon May 10 20:37:04 2021] ata4.00: cmd 35/00:40:00:09:41/00:05:16:00:00/e0 tag 25 dma 688128 out
                                   res 41/10:40:00:09:41/00:05:16:00:00/e0 Emask 0x81 (invalid argument)
[Mon May 10 20:37:04 2021] ata4.00: status: { DRDY ERR }
[Mon May 10 20:37:04 2021] ata4.00: error: { IDNF }
[Mon May 10 20:37:04 2021] ata4.00: configured for UDMA/133
[Mon May 10 20:37:04 2021] ata4: EH complete
[Mon May 10 20:37:04 2021] ata4.00: exception Emask 0x0 SAct 0x0 SErr 0x0 action 0x0
[Mon May 10 20:37:04 2021] ata4.00: irq_stat 0x40000000
[Mon May 10 20:37:04 2021] ata4.00: failed command: WRITE DMA EXT
[Mon May 10 20:37:04 2021] ata4.00: cmd 35/00:c0:40:0e:41/00:02:16:00:00/e0 tag 2 dma 360448 out
                                   res 41/10:c0:40:0e:41/00:02:16:00:00/e0 Emask 0x81 (invalid argument)
[Mon May 10 20:37:04 2021] ata4.00: status: { DRDY ERR }
[Mon May 10 20:37:04 2021] ata4.00: error: { IDNF }
[Mon May 10 20:37:04 2021] ata4.00: configured for UDMA/133
[Mon May 10 20:37:04 2021] ata4: EH complete
[Mon May 10 20:37:04 2021] ata4.00: exception Emask 0x0 SAct 0x0 SErr 0x0 action 0x0
[Mon May 10 20:37:04 2021] ata4.00: irq_stat 0x40000000
[Mon May 10 20:37:04 2021] ata4.00: failed command: WRITE DMA EXT
[Mon May 10 20:37:04 2021] ata4.00: cmd 35/00:40:00:11:41/00:05:16:00:00/e0 tag 10 dma 688128 out
                                   res 41/10:40:00:11:41/00:05:16:00:00/e0 Emask 0x81 (invalid argument)
[Mon May 10 20:37:04 2021] ata4.00: status: { DRDY ERR }
[Mon May 10 20:37:04 2021] ata4.00: error: { IDNF }
[Mon May 10 20:37:04 2021] ata4.00: configured for UDMA/133
[Mon May 10 20:37:04 2021] ata4: EH complete
[Mon May 10 20:37:04 2021] ata4.00: exception Emask 0x0 SAct 0x0 SErr 0x0 action 0x0
[Mon May 10 20:37:04 2021] ata4.00: irq_stat 0x40000000
[Mon May 10 20:37:04 2021] ata4.00: failed command: WRITE DMA EXT
[Mon May 10 20:37:04 2021] ata4.00: cmd 35/00:40:00:09:41/00:05:16:00:00/e0 tag 18 dma 688128 out
                                   res 41/10:40:00:09:41/00:05:16:00:00/e0 Emask 0x81 (invalid argument)
[Mon May 10 20:37:04 2021] ata4.00: status: { DRDY ERR }
[Mon May 10 20:37:04 2021] ata4.00: error: { IDNF }
[Mon May 10 20:37:04 2021] ata4.00: configured for UDMA/133
[Mon May 10 20:37:04 2021] sd 3:0:0:0: [sdb] tag#18 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE
[Mon May 10 20:37:04 2021] sd 3:0:0:0: [sdb] tag#18 Sense Key : Illegal Request [current]
[Mon May 10 20:37:04 2021] sd 3:0:0:0: [sdb] tag#18 Add. Sense: Logical block address out of range
[Mon May 10 20:37:04 2021] sd 3:0:0:0: [sdb] tag#18 CDB: Write(10) 2a 00 16 41 09 00 00 05 40 00
[Mon May 10 20:37:04 2021] blk_update_request: I/O error, dev sdb, sector 373360896 op 0x1:(WRITE) flags 0x4000 phys_seg 168 prio class 0
[Mon May 10 20:37:04 2021] ata4: EH complete
[Mon May 10 20:37:04 2021] ata4.00: exception Emask 0x0 SAct 0x0 SErr 0x0 action 0x0
[Mon May 10 20:37:04 2021] ata4.00: irq_stat 0x40000000
[Mon May 10 20:37:04 2021] ata4.00: failed command: WRITE DMA EXT
[Mon May 10 20:37:04 2021] ata4.00: cmd 35/00:c0:40:0e:41/00:02:16:00:00/e0 tag 1 dma 360448 out
                                   res 41/10:c0:40:0e:41/00:02:16:00:00/e0 Emask 0x81 (invalid argument)
[Mon May 10 20:37:04 2021] ata4.00: status: { DRDY ERR }
[Mon May 10 20:37:04 2021] ata4.00: error: { IDNF }
[Mon May 10 20:37:04 2021] ata4.00: configured for UDMA/133
[Mon May 10 20:37:04 2021] sd 3:0:0:0: [sdb] tag#1 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE
[Mon May 10 20:37:04 2021] sd 3:0:0:0: [sdb] tag#1 Sense Key : Illegal Request [current]
[Mon May 10 20:37:04 2021] sd 3:0:0:0: [sdb] tag#1 Add. Sense: Logical block address out of range
[Mon May 10 20:37:04 2021] sd 3:0:0:0: [sdb] tag#1 CDB: Write(10) 2a 00 16 41 0e 40 00 02 c0 00
[Mon May 10 20:37:04 2021] blk_update_request: I/O error, dev sdb, sector 373362240 op 0x1:(WRITE) flags 0x0 phys_seg 88 prio class 0
[Mon May 10 20:37:04 2021] EXT4-fs warning (device sdb1): ext4_end_bio:315: I/O error 10 writing to inode 10095089 (offset 8388608 size 1871872 starting block 46670368)
[Mon May 10 20:37:04 2021] Buffer I/O error on device sdb1, logical block 46669856
[Mon May 10 20:37:04 2021] Buffer I/O error on device sdb1, logical block 46669857
[Mon May 10 20:37:04 2021] Buffer I/O error on device sdb1, logical block 46669858
[Mon May 10 20:37:04 2021] Buffer I/O error on device sdb1, logical block 46669859
[Mon May 10 20:37:04 2021] Buffer I/O error on device sdb1, logical block 46669860
[Mon May 10 20:37:04 2021] Buffer I/O error on device sdb1, logical block 46669861
[Mon May 10 20:37:04 2021] Buffer I/O error on device sdb1, logical block 46669862
[Mon May 10 20:37:04 2021] Buffer I/O error on device sdb1, logical block 46669863
[Mon May 10 20:37:04 2021] Buffer I/O error on device sdb1, logical block 46669864
[Mon May 10 20:37:04 2021] Buffer I/O error on device sdb1, logical block 46669865
[Mon May 10 20:37:04 2021] ata4: EH complete
[Mon May 10 20:37:04 2021] ata4.00: exception Emask 0x0 SAct 0x0 SErr 0x0 action 0x0
[Mon May 10 20:37:04 2021] ata4.00: irq_stat 0x40000000
[Mon May 10 20:37:04 2021] ata4.00: failed command: WRITE DMA EXT
[Mon May 10 20:37:04 2021] ata4.00: cmd 35/00:40:00:11:41/00:05:16:00:00/e0 tag 7 dma 688128 out
                                   res 41/10:40:00:11:41/00:05:16:00:00/e0 Emask 0x81 (invalid argument)
[Mon May 10 20:37:04 2021] ata4.00: status: { DRDY ERR }
[Mon May 10 20:37:04 2021] ata4.00: error: { IDNF }
[Mon May 10 20:37:04 2021] ata4.00: configured for UDMA/133
[Mon May 10 20:37:04 2021] sd 3:0:0:0: [sdb] tag#7 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE
[Mon May 10 20:37:04 2021] sd 3:0:0:0: [sdb] tag#7 Sense Key : Illegal Request [current]
[Mon May 10 20:37:04 2021] sd 3:0:0:0: [sdb] tag#7 Add. Sense: Logical block address out of range
[Mon May 10 20:37:04 2021] sd 3:0:0:0: [sdb] tag#7 CDB: Write(10) 2a 00 16 41 11 00 00 05 40 00
[Mon May 10 20:37:04 2021] blk_update_request: I/O error, dev sdb, sector 373362944 op 0x1:(WRITE) flags 0x4000 phys_seg 168 prio class 0
[Mon May 10 20:37:04 2021] ata4: EH complete
[Mon May 10 20:37:04 2021] ata4.00: exception Emask 0x0 SAct 0x0 SErr 0x0 action 0x0
[Mon May 10 20:37:04 2021] ata4.00: irq_stat 0x40000000
[Mon May 10 20:37:04 2021] ata4.00: failed command: WRITE DMA EXT
[Mon May 10 20:37:04 2021] ata4.00: cmd 35/00:08:40:16:41/00:01:16:00:00/e0 tag 13 dma 135168 out
                                   res 41/10:08:40:16:41/00:01:16:00:00/e0 Emask 0x81 (invalid argument)
[Mon May 10 20:37:04 2021] ata4.00: status: { DRDY ERR }
[Mon May 10 20:37:04 2021] ata4.00: error: { IDNF }
[Mon May 10 20:37:04 2021] ata4.00: configured for UDMA/133
[Mon May 10 20:37:04 2021] ata4: EH complete
[Mon May 10 20:37:05 2021] ata4.00: exception Emask 0x0 SAct 0x0 SErr 0x0 action 0x0
[Mon May 10 20:37:05 2021] ata4.00: irq_stat 0x40000000
[Mon May 10 20:37:05 2021] ata4.00: failed command: WRITE DMA EXT
[Mon May 10 20:37:05 2021] ata4.00: cmd 35/00:08:40:16:41/00:01:16:00:00/e0 tag 30 dma 135168 out
                                   res 41/10:08:40:16:41/00:01:16:00:00/e0 Emask 0x81 (invalid argument)
[Mon May 10 20:37:05 2021] ata4.00: status: { DRDY ERR }
[Mon May 10 20:37:05 2021] ata4.00: error: { IDNF }
[Mon May 10 20:37:05 2021] ata4.00: configured for UDMA/133
[Mon May 10 20:37:05 2021] ata4: EH complete
[Mon May 10 20:37:05 2021] ata4.00: exception Emask 0x0 SAct 0x0 SErr 0x0 action 0x0
[Mon May 10 20:37:05 2021] ata4.00: irq_stat 0x40000000
[Mon May 10 20:37:05 2021] ata4.00: failed command: WRITE DMA EXT
[Mon May 10 20:37:05 2021] ata4.00: cmd 35/00:08:40:16:41/00:01:16:00:00/e0 tag 5 dma 135168 out
                                   res 41/10:08:40:16:41/00:01:16:00:00/e0 Emask 0x81 (invalid argument)
[Mon May 10 20:37:05 2021] ata4.00: status: { DRDY ERR }
[Mon May 10 20:37:05 2021] ata4.00: error: { IDNF }
[Mon May 10 20:37:05 2021] ata4.00: configured for UDMA/133
[Mon May 10 20:37:05 2021] ata4: EH complete
[Mon May 10 20:37:05 2021] ata4.00: exception Emask 0x0 SAct 0x0 SErr 0x0 action 0x0
[Mon May 10 20:37:05 2021] ata4.00: irq_stat 0x40000000
[Mon May 10 20:37:05 2021] ata4.00: failed command: WRITE DMA EXT
[Mon May 10 20:37:05 2021] ata4.00: cmd 35/00:08:40:16:41/00:01:16:00:00/e0 tag 28 dma 135168 out
                                   res 41/10:08:40:16:41/00:01:16:00:00/e0 Emask 0x81 (invalid argument)
[Mon May 10 20:37:05 2021] ata4.00: status: { DRDY ERR }
[Mon May 10 20:37:05 2021] ata4.00: error: { IDNF }
[Mon May 10 20:37:05 2021] ata4.00: configured for UDMA/133
[Mon May 10 20:37:05 2021] ata4: EH complete
[Mon May 10 20:37:05 2021] ata4.00: exception Emask 0x0 SAct 0x0 SErr 0x0 action 0x0
[Mon May 10 20:37:05 2021] ata4.00: irq_stat 0x40000000
[Mon May 10 20:37:05 2021] ata4.00: failed command: WRITE DMA EXT
[Mon May 10 20:37:05 2021] ata4.00: cmd 35/00:08:40:16:41/00:01:16:00:00/e0 tag 19 dma 135168 out
                                   res 41/10:08:40:16:41/00:01:16:00:00/e0 Emask 0x81 (invalid argument)
[Mon May 10 20:37:05 2021] ata4.00: status: { DRDY ERR }
[Mon May 10 20:37:05 2021] ata4.00: error: { IDNF }
[Mon May 10 20:37:05 2021] ata4.00: configured for UDMA/133
[Mon May 10 20:37:05 2021] ata4: EH complete
[Mon May 10 20:37:05 2021] ata4.00: exception Emask 0x0 SAct 0x0 SErr 0x0 action 0x0
[Mon May 10 20:37:05 2021] ata4.00: irq_stat 0x40000000
[Mon May 10 20:37:05 2021] ata4.00: failed command: WRITE DMA EXT
[Mon May 10 20:37:05 2021] ata4.00: cmd 35/00:08:40:16:41/00:01:16:00:00/e0 tag 11 dma 135168 out
                                   res 41/10:08:40:16:41/00:01:16:00:00/e0 Emask 0x81 (invalid argument)
[Mon May 10 20:37:05 2021] ata4.00: status: { DRDY ERR }
[Mon May 10 20:37:05 2021] ata4.00: error: { IDNF }
[Mon May 10 20:37:05 2021] ata4.00: configured for UDMA/133
[Mon May 10 20:37:05 2021] sd 3:0:0:0: [sdb] tag#11 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE
[Mon May 10 20:37:05 2021] sd 3:0:0:0: [sdb] tag#11 Sense Key : Illegal Request [current]
[Mon May 10 20:37:05 2021] sd 3:0:0:0: [sdb] tag#11 Add. Sense: Logical block address out of range
[Mon May 10 20:37:05 2021] sd 3:0:0:0: [sdb] tag#11 CDB: Write(10) 2a 00 16 41 16 40 00 01 08 00
[Mon May 10 20:37:05 2021] blk_update_request: I/O error, dev sdb, sector 373364288 op 0x1:(WRITE) flags 0x0 phys_seg 33 prio class 0
[Mon May 10 20:37:05 2021] EXT4-fs warning (device sdb1): ext4_end_bio:315: I/O error 10 writing to inode 10095089 (offset 8388608 size 1871872 starting block 46670569)
[Mon May 10 20:37:05 2021] ata4: EH complete
[Mon May 10 20:37:05 2021] JBD2: Detected IO errors while flushing file data on sdb1-8

HD Sentinel reports that both disks are fine, and I don't see anything of concern in the SMART data. Diagnostics via iDRAC didn't find anything, the RAID controller and its battery are good.

Everything Google brings up points to a physical HDD fault or a Hyper-V issue, but this is a virtual disk and Hyper-V is disabled. Maybe it has something to do with VirtualBox's settings? It ran fine with the same settings for several months. Maybe it's the controller, after all?

Some pointers on how to continue hunting down the source of this issue would be much appreciated.

Which host OS are you using? – shodanshok May 11 '21 at 13:18 — shodanshok, May 11 '21 at 13:18
@shodanshok Windows Server 2016 – Marcell Bernát May 11 '21 at 13:48 — Marcell Bernát, May 11 '21 at 13:48

score 1 · Accepted Answer · answered Jun 18 '21 at 12:38

I checked and tried everything shodanshok recommended (thanks again!), but it looks like the problem ended up being the VDI file itself. My best guess is that it was somehow corrupted.

I created a new VDI and moved everything to it during a scheduled maintenance window. Before this, the errors mentioned in my question occurred almost every day, but now it's been more than 3 weeks since the last error, so I truly believe this fixed the problem.

score 0 · Answer 2 · answered May 11 '21 at 21:55

When reported in a guest OS, such errors generally means that the host is extremely slow to process incoming IO request. So slow that the guest OS iSCSI layer times out after ~30 seconds, reporting the aforementioned errors.

A very slow host can be the result of:

a RAID rebuild
a (too much) aggressive running patrol read
a physical issue with the underlying disks
an heavy, running host task (for example Windows update)

Please check your HDD health from within the host OS, preferably by querying the PERC adapter (ie: via iDRAC or DELL's OpenManage tools).

Dmesg is full of I/O errors on Ubuntu 18.04 VM, physical disks and host are fine

2 Answers2