26

Can NFS be reasonably used on production servers as a means of connecting a compute server to a storage server, assuming the connection is over a LAN 1Gbe or 10Gbe connection?

There's obviously some network overhead and NFS seems particularly slower with writes if you have sync mode enabled. Otherwise it seems reasonably lightweight and able to scale from what I can tell, but I have little experience with it personally. Am I wrong?

The problem is I have a server right now that acts as both the storage and web server but I'm going to end up needing to split the two likely in the future, and considering some requests need to pass through the web application layer for authentication before initializing the file transfer, it gets kind of tricky with this software. A network fs mount is the simplest option I just.. don't know if that's a good one.

I also plan to try and utilize local caching with NFS which should improve performance a good bit, but I'm not sure if that's enough.

As far as alternatives, there's only iSCSI that I'm aware of as a real competitor, and most people seem to recommend NFS over any of the other lesser known ones.

Makoto
  • 371
  • 3
  • 6
  • 8
    If you'll be maintaining the NFS storage yourself (and not some high availability NFS-as-a-service) and uptime is important, you may want to look into how to handle downtime for the NFS storage. Doing simple failover between two NFS servers can be tricky. Although more complicated, we moved to GlusterFS to better handle storage failures. – sbrattla Aug 16 '20 at 18:55
  • 5
    Do you have any reason to believe that NFS *wouldn't* be reaonable, based on facts or expertise? – womble Aug 17 '20 at 00:44
  • 4
    So long as you keep in mind that NFS is a best-effort storage stack - it's fine. Don't rely on it for anything a) high-volume, b) mission-critical. *Can* it handle high-volume and/or mission-critical services? Yep. And it *likely* will. Right up until it *doesn't*. – warren Aug 17 '20 at 19:05
  • Just as a short follow up, I've set up our new server infrastructure using an NFS mount as described above. Had some issues getting jumbo frames enabled, but things still seem to be performing quite well. I have a 256GB fscache partition on our NVMe drive to help performance and all seems to be working as expected. So, pushing out around 2TB of data transfer per day right now over a 1Gbps line without any noticeable drop in performance. I'm certain fscache is definitely helping with that, but otherwise I'm relieved everything is working well and appreciate everyone here's help! – Makoto Sep 30 '20 at 11:19

4 Answers4

30

NFS is fine, barring some specific other criteria are met, namely:

  • The systems involved are both able to use NFS natively. Windows doesn't count here, it kind of works, but it's got a lot of quirks and is often a pain to work with when dealing with NFS in a cross-platform environment (and if it's just Windows, use SMB3, it eliminates most of the other issues with NFS). Note that on the client side, this means kernel-level support, because a user-level implementation either has to deal with the efficiency issues inherent in using something like FUSE, or it has to be linked directly into the application that needs to access the share.
  • You've properly verified how the NFS client handles an NFS server restart. This includes both the OS itself (which should be fine in most cases), and the software that will be accessing the share. In particular, special care is needed on some client platforms when the software using the share holds files open for extended periods of time, as not all NFS client implementations gracefully handle server restarts by explicitly remounting and revalidating locks and file handles like they should (which leads to all kinds of issues for the client software). Note that you should recheck this any time any part of the stack is either upgraded or reconfigured.
  • You're willing to set up proper user/group ID mapping. This is big, because without it you either need to mirror the UID/GID mappings between the systems (doable, but I'd be wary of setting up SSO against an internal network for an internet facing system) or you end up with potentially serious security implications (namely, what you see on one system for permissions does not match what you see on others).
  • You're operating over a secured network link, or are willing to properly set up authentication for the share. Without auth, anybody on the link can access it (and a malicious client can easily side-step the basic UNIX discretionary access controls).

Assuming you meet all those criteria, and you have a reasonably fast network, you should be fine. Also, if you can run jumbo frames, do so, they help a lot for any network filesystem or networked block storage.

Austin Hemmelgarn
  • 2,295
  • 9
  • 15
  • 2
    I'd like to add that the Windows NFS client has completely broken a couple of my NFS shares over the last year. – ScottishTapWater Aug 18 '20 at 08:29
  • 1
    NFS on Windows is licensed and very poorly maintained code. There's a reason why MSFT doesn't support NFS in production for majority of their "selling" scenarios like SQL Server, Exchange, Hyper-V and so on. – RiGiD5 Aug 28 '20 at 18:32
18

NFS is absolutely OK and is preferred over iSCSI due to the fact NFS is much easier to manage, share and backup.

BaronSamedi1958
  • 13,676
  • 1
  • 21
  • 53
  • 3
    That's too strong a statement to be unconditionally true. Some use cases NFS is completely unsuited for but iSCSI does fine -- if you need true POSIX semantics for an application you're running, for example. NFS is sorta, kinda POSIX-y; whereas if you can run your own filesystem, you can have exactly the semantics you need. (Of course, needing to use multi-master-safe block-layer filesystems is a serious constraint, but those that exist have been built to work with heavy-duty applications like Oracle RAC; see f/e GFS2) – Charles Duffy Aug 17 '20 at 21:07
  • 2
    OP asked an exact question and got his answer. This isn’t a discussion forum and OP didn’t ask “NFS Vs Block” in general. – BaronSamedi1958 Aug 18 '20 at 06:08
  • "A production web server" is nothing remotely like an exact specification. We don't know what datastores that web server is using and what their requirements on underlying storage semantics are. We don't know whether there's an intent to support active/active database backends, etc. – Charles Duffy Aug 18 '20 at 15:37
  • 1
    I’m not here to argue. There’s always reddit. – BaronSamedi1958 Aug 18 '20 at 15:54
  • 2
    Thank you for the feedback, I should have provided more information in my original post. When I say linking to a storage server, I mean for storing static files (user uploaded content) - media files, images, user avatars, things of that nature. Definitely not running MySQL or any other database server over NFS, that will be handled on the compute server on its native NVMe storage device. Additionally small static assets like images will be cached over CloudFlare's CDN as needed, so really only larger files (like videos) will be actively served directly from the server itself. – Makoto Aug 18 '20 at 23:45
12

We've been using NFS for years to attach our SAN to our VMware ESXi servers, running hundreds of VMs on it. No trouble at all.

The bottleneck is rather the storage system than the network protocol.

The network connection should be fast enough of course, meaning 10Gb Ethernet or fibre. We don't even bother with a separate storage network anymore.

Gerald Schneider
  • 23,274
  • 8
  • 57
  • 89
  • Disk I/O is usually what ends up killing performance, especially when people try to do performance heavy tasks against systems not really designed well for performance. – MaQleod Aug 19 '20 at 00:12
  • 1
    @MaQleod and latency. There's always someone who thinks cross-country NFS will work fine. – Wayne Conrad Aug 19 '20 at 04:39
6

iSCSI might be a bit faster...

https://www.vmware.com/content/dam/digitalmarketing/vmware/en/pdf/techpaper/storage_protocol_comparison-white-paper.pdf

https://www.hyper-v.io/whos-got-bigger-balls-testing-nfs-vs-iscsi-performance-part-3-test-results/

...but NFS like any other network redirector (SMB3, AFS/AFP etc) allows concurrent multi-access which is tricky with iSCSI or other block protocols.

https://forums.starwindsoftware.com/viewtopic.php?f=5&t=1392

NISMO1968
  • 897
  • 1
  • 6
  • 15