-1

I've heard it said that a 512byte data transfer over fibre channel is 10us so would an 8k (16x the data) take 160us? or is much of that 10us one time transmit overhead?

I'm interested in comparing NFS to FC. Over NFS I'm seeing about 350us for 8K with out jumbo frames. I figure it could be taken down lower than 350us with some modifications. (and it seems like most of that is from the TCP layer. I'm only looking at the NFS server and on the server only about 50us is from NFS layer, using dtrace for the measurements. I don't know the client side NFS costs but if it's similar, that would mean the bulk is from TCP).

EEAA
  • 109,363
  • 18
  • 175
  • 245
Kyle Hailey
  • 275
  • 3
  • 11
  • this is like comparing apples to the trucks oranges are delivered in. FC is a much more complete spec, NFS can work over a modem at one end of the spectrum or over 100Gbs lines, even over Infiniband - it's a protocol not a full system so there's literally no comparison to be had I'm afraid. Why don't you ask a different type of question about the same basic scenario such as "will this FC box (xxxx) be faster than this NFS/NAS box (yyyy) in this situation (zzzz)?", much more likely to get a sensible answer. – Chopper3 May 23 '11 at 15:00
  • Agreed that equipment is key. My question is about the best equipment. The fastest implied speed for NFS I have ever seen is 200us on an Oracle benchmark, but this was stated the difference between two benchmarks not the actual measured speed. Running tests in what some say is the fast NFS lab in the world in Santa Clara with AMD an Netapp we got 350us for 8k transfers but we weren't using jumbo frames. Jumbo frames should pull us down a bit farther, say for for arguments sake 300us. If FC is 160us then the difference is 140us per 8K which is a a small difference for all the NFS advantages. – Kyle Hailey May 23 '11 at 15:21

1 Answers1

1

This is a very subjective question which is vendor and configuration dependent. According to the Wikipedia article the maximum data size for a FC frame is 2112 bytes., thus it will take 4 frames to send you 8K packet. If you have two HBA's configured to multiplex then you can send your frames in parallel thus cutting your transfer sizes theoretically in half.

Why would you want to use NFS over FC? NFS sits at layer 7 in the OSI model, whereas FC is might fit into layer 3 or 4, even though it really does not fit the model. Just from that perspective there is a lot more protocol overhead for NFS vs FC.

Comparing FC to NFS is like trying to compare an apple to broccoli. An apples to pears comparison might be FC vs ATAoE. Both have very light wight protocols which sit on top of layer 2, though FC is more tightly bound/optimized into layer 2. Both protocols have their error correction built in and do not require on a higher level protocol like TCP/IP.

I think the bigger question is what are you trying to accomplish?

Red Tux
  • 2,074
  • 13
  • 14
  • NFS and FC support a similar role in infrastructure - serving disk space in a flexible, extensible, fault tolerant manner. NFS is easier to set up, cheaper to buy and easier to administer. NFS is passing FC up in throughput. Ethernet is at 10G where as FC is at 8G. Sure 16G just came out two weeks ago at one company, but when do you think you'll see in a data center? 40GE is targeted for next year and 100Ge the year after. AFAIK the highest FC planned is 32G. So the big difference is speed - latency for reads and speed of fail over. Speed of fail over is important but how important is 350us. – Kyle Hailey May 23 '11 at 14:46
  • Again that depends on what you are trying to accomplish. For example in the 350us latency of NFS you might be able to get a round trip packet or two. True the framing may be smaller, but but for smaller data transactions FC may prove to be substantially faster even though the top end speed is not the same. In addition NFS is for file systems, not block devices which means things like Clustered file systems will not work over NFS directly. – Red Tux May 24 '11 at 03:53
  • This is what it comes down to me - what's the difference in 8k latency over FC compared to NFS, on the best equipment. If NFS is 350us and FC is 160us then it's just 190us which is minimal. Sure I can give scenarios, as a database guy, where this would double the transaction time which might not be acceptable but those cases are extreme and there are ways to achieve even better performance for less capex. If NFS can be taken down to 200us then the difference is getting to within standard error. I'm guessing FC can get to 100us and NFS to 200us which if true is pretty minimal. WIll test. TBD. – Kyle Hailey May 25 '11 at 04:52
  • I think NFS is the best choice for you because that's what you want, not to understand the fundamental differences and use cases for each protocol. I wish you luck in your implementation. – Red Tux May 25 '11 at 14:50