1

When NVME controller sends data into FPGA. lba is not in order. we got trouble about getting lba with correct order from nvme source disk.

I am working with NVMe about the project using Samsung SSD 970 PRO 512GB. In our project, we use FPGA to communicate with NVMe devices but we got trouble with our process as bellow:

1. FPGA having FIFO buffer mapped with DMAP2P works as destination/from address for NVMe controller to read/write.
2. NVMe driver sends command to NVMe controller, requests a read command with transfer size 4096 lba (2MB).
3. When controller sends data into FPGA. lba is not in order. we got trouble about getting lba with correct order from nvme source disk. (1)

I have investigated about "Dataset Management (DSM)", with "Sequential Request" bit[6] inside dword 13 for each read/write command. I confirmed DSM is supported in our NVMe device (Samsung 970 Pro), but look like it does not have any effected to order of lba transfer in a NVMe read command.

Please help me to clarify what is "Sequential Request" option inside each NVMe read command, as inside NVMe specification described "command is part of a sequential read that includes multiple Read commands". Is it able to change lba transfer order in a NVMe read command? As our expect (1) receive lba in order from an NVMe read command (with large of lba), is it possible? What can we do for our expectation?

vocsiphuong
  • 11
  • 2
  • What do mean by writing "lba is not in order"? Note that DSM is advisory and firmware might silently ignore it. – BeginEnd Jul 08 '19 at 21:43
  • BeginEnd thank for your reply. "lba is not in order" i mean when i send a NVMe read command with size=4096 lba, destination is FPGA FIFO buffer. The data is received by FPGA is not in order, it look like lba 1 -> 2 -> 3 -> 5 -> 40 -> 41 -> 6 -> 7 -> 8 -> 20. – vocsiphuong Jul 09 '19 at 01:19

1 Answers1

0

NVMe devices are quite complicated devices and shouldn't be considered as raw FLASH storage. If you request to read 2MB of data that is stored in 4096 LBA you can end up reading several physical chips. Each chip might be divided internally into smaller logical elements. Each whole device and each chip on its own might be doing some house keeping during this big-and-long-READ-request so some LBA's might be read faster than others. This is totally vendor-dependent. If whole request have to be serialized this will end up in huge delays and huge amount of memory that NVMe device should have to "buffer" all requests. So I deeply doubt that there is something in NVMe spec that forbids out-of-order data transfers. More, until whole read is reported as completed by NVMe device you should not assume any valid data in given memory range. If you want to serialize LBA reads you should divide your long request into 4096 separate requests and maybe also set "Sequential Request" bit in each.

BeginEnd
  • 334
  • 2
  • 11
  • Thank for your advice! Because the transfer speed will be slow down when divide my long request into 4096 separate requests, maybe i will try handling buffer by physical RAM instead of FIFO. – vocsiphuong Jul 10 '19 at 16:41