1

I am trying to figure out the optimal way to share storage with multiple VMs on the same box. I am aware of a range of options: NFS, iSCSI, fibrechannel, bridged, etc, but I am unclear how they fit together or are mutually exclusive.

Here are a few statments of what I understand to be true based on what I have read so far, but an not certain of. If someone could either affirm or correct my understanding, that would be great. At this time I am thinking about using Proxmox (KVM+ZFS), but if other hypervisors have important differences with respect to these statements, please do explain.

True or False?:

  1. For sharing storage devices between VMs on the same host, I can used bridged networking to maximize speed.

  2. With bridged networking, no physical network hardware is being used, instead the CPU is acting as a virtual NIC, which is faster because the limit is the speed of the system bus, rather than of ethernet/fibrechannel etc.

  3. For the reason stated above, there is no benefit to using something fancy like fibrechannel to share storage with VMs, except the elimination of a small amount of CPU overhead. Using a fibrechannel card would be needlessly creating an extra step for VM interaction with the hosted storage.

  4. The speed and complexity of setup of iSCSI vs NFS is something that varies a lot between hypervisors, so this choice should be made once I have everything else decided.

Stonecraft
  • 313
  • 3
  • 4
  • 18
  • Don't expect the software switches to be faster than hardware. And I am not sure how guest networking relates to host storage in your case. – eckes May 20 '17 at 23:47
  • The host will manage several devices (probably zpools, but I haven't decided that yet). Those devices will then be shared to VMs on that same host. Are you saying it would be better to have multiple network controllers in one box, and have a really short ethernet cable going from controller to another? – Stonecraft May 21 '17 at 00:23
  • 1
    No I am just correcting your statement about the fast software switch, they are not so fast in all cases. However the access to the storage won't go over network anyway. – eckes May 21 '17 at 02:01
  • So this is what I don't get.. even if I share the storage volumes via NFS, it's not actually going over the network, and thus is not limited by anything about my network card, be it ethernet, fibrechannel, etc. Right? Sorry if I'm being dense here, but this really confuses me. – Stonecraft May 21 '17 at 02:13
  • 1
    Yes NFS goes over the network, it depends however on who speaks NFS. Most often the storage protocols are spoken by the host. The guests see block devices. Especially if you plan to make them available to multiple guests. You can also make the guests speak the network protocols, but then you have for example no acceleration for iscsi in virtual NICs. When you share a block device the host has more potential for caching. – eckes May 21 '17 at 06:44
  • So there are no obstacles to having multiple guests simultaneously access the same volume as a block device? I was thinking that the block device thing was like passthrough in that it could only be used by one OS at a time. So, if I can make a an Ext4-formatted ZVOL and have multiple guests all happily using it at once, that's great. – Stonecraft May 21 '17 at 08:19
  • Is this what I want? https://en.wikipedia.org/wiki/Clustered_file_system – Stonecraft May 21 '17 at 08:26
  • 1
    Oh now I understand, I was assuming you had a special reason to share a volume. So if you want to share a volume between multiple guest you need coordination by a Cluster filesystem or failover Cluster to manage this. If all you want is to share files between multiple guest then a network filesystem mounted inside multiple guests is the more leightweigt solution. (Iscsi and SAN/FC would all require coordination). It depends a bit on how you manage your hosts how you would actually serve the NFS filesystem. This can be done by the host but it's rather unusual. – eckes May 21 '17 at 12:35
  • I don't see any reason why I couldn't just mount the top level directory of the volume in multiple guests, but I do want the absolute best performance I can get. Which takes me back to my original questions: If I share the top level directory of the volume via NFS to other guests on the same physical box, is the network hardware of my computer actually involved? – Stonecraft May 21 '17 at 17:34
  • 1
    If you have an NFS Server on your host and use a virtual switch/software bridge then no network hardware is involved. – eckes May 21 '17 at 18:07

0 Answers0