1

I am developing an application, where I have large data that's continuously been write into ram, and I am trying to quickly read the data out from ram, and write it into NVMe SSD, after write is completed, I re queue ram space to allow it being written.

My current issue is that the data write continuously in an indefinitely time, so eventually my RAM is not large enough to host all the data, it has to be readout and store into disk. Due to the size of data, the write speed criteria is high ( 1.5G/s).

I often see mmap be mentioned to improve I/O efficiency, but mostly read efficiency because it prevents copy large data to DRAM. So I wonder in my use case, is it beneficial to use mmap to map my SSD directly as virtual memory and directly write to it be faster than standard fwrite ?

user2386301
  • 412
  • 1
  • 5
  • 18
  • 1
    Try it and see. Carefully benchmark each aspect of your operations separately then together. Note that if you write continuously and indefinitely at 1.5GB/s, you won't only run out of RAM, you'll run out of disk too... – Mark Setchell Mar 27 '19 at 07:57

1 Answers1

3

The problem with mmap for write is that you do not know when the write is completed or even started, this may increase the number of writes the drive has to do to an LBA since the memory got written into but not in a complete 4KB chunk, the write was issued to the disk and then the data was written again to memory and now the page needs to be written again.

If you want it simple your best bet is to use an O_DIRECT file and use the write syscall or aio. If you want the best speed you can use SPDK to get raw access to the NVMe device without the kernel interfering and having a fully zero-copy write.

SPDK is a driver for NVMe devices completely in user-space. This means that you use a kernel driver to map the PCIe BAR to user-space, tell SPDK to attach to the NVMe device and now you can issue raw NVMe commands to the device without any copy. The big advantage over just a plain mmap is that you have full control on what IOs are done, in what order and how many commands are in-flight at a time and their sizes. It means more work for the application but it really gives you the ultimate control and the best performance.

Baruch Even
  • 2,581
  • 1
  • 17
  • 23
  • Hi Baruch, I often heard about people talking to SPDK, what's the exactly role that SPDK provide? is underlying it use mmap to access the NVMe device and map it to user space? If so, what's the benefit over using SPDK instead of just call mmap ? – user2386301 Apr 15 '19 at 16:48
  • SPDK actually maps the PCI BAR to user-space and then communicates the NVMe protocol directly with the hardware without switching to the kernel so you remove a lot of the syscall switching latency and can get raw performance, you also eliminate all memory copies for extra benefit. The cost is that you take exclusive access to the drive and no one else can use it in parallel and you lose the filesystem organization and need to implement your own order on the on-disk data. – Baruch Even Jun 26 '19 at 17:01