3

we have a very latency sensitive application, in the sense that latency spikes on reads are very very bad.

I’ve tested XFS and ext4, and writing O_ASYNC to the file and then fdatasync() at the end can lead to 1 second or more spikes in read latency!

i then tried O_SYNC and I got much more stable read latencies but writing to the file is extremely slow.

So, I tried writing O_ASYNC and syncing every 5 megabytes written to the file, and its fast and the read latencies are reasonably stable as well.

However, 30 minutes I can still get a read that takes a second or more.

If you built latency sensitive applications on Linux, how did YOU deal with working with the filesystem, or did you just not use it at all and mount the device as a RAW device?

Bibhas Debnath
  • 14,559
  • 17
  • 68
  • 96
Michael Xu
  • 557
  • 1
  • 5
  • 14
  • What's the underlying hardware? Disk or flash? – Fred Foo Mar 11 '11 at 14:28
  • ssd. not spinning disk. this is my first time working with an app that has such close tie-ins with the filesystem performance wise. i guess, im just curious about different ways of guaranteeing read latencies...... i'll look into real time filesystems for more about this... – Michael Xu Mar 11 '11 at 14:43
  • How much data do you read and at what rate? Is all data in an existing file, or is that file being read while another process writes it? – Maxim Egorushkin Mar 11 '11 at 14:51
  • the read pattern is all random. (this means that i'm constantly paging in data from disk). the writes are coming in over the network. – Michael Xu Mar 11 '11 at 14:52

1 Answers1

0

A filesystem always adds a small amount of latency, so for a really latency sensitive application I would consider bypassing the file system by using the raw device or opening the file with O_DIRECT bypassing the OS caches.

Other tricks for latency on SSD storage are:

  • use the inherent parallelism in modern SSDs by reading/writing from multiple threads. The latency will not decrease, but if the real issue are IOPS this helps.
  • Check the alignment of the filesystem on the SSD. If this is not done right, you half the performance and double the latency.
  • Check the disk scheduling algorithms. Linux "noop" scheduler is usually the best for SSD based storage.
  • Use better SSD hardware, e.g. PCIe based SSDs that usually provide a much lower latency

That said: A read time from a second simply doesn't sound right. Either the load/utilization is so high that the hardware simply can't cope with it and more/better hardware would be the best solution or something else is terribly wrong.

dmeister
  • 34,704
  • 19
  • 73
  • 95