9

We are having an issue that happens intermittently during a code deployment. NFS caches the files as they are read, but if a file is read during a code deploy it stays in a dirty state as if the file wasn't changed during the deploy. The only way we can alleviate this issue is by clearing the NFS cache after the deploy.

Our webserver returns blank pages for all requests until the NFS cache is cleared.

Is there a setting or does anyone have a suggestion on how to fix this?

We are running NFS v3 at the moment.

Michael
  • 333
  • 2
  • 3
  • 7
  • What happens when you manually open a newly deployed file on a web server before clearing the NFS caches? Do you see the old or updated version. How do you deploy, @rsync@ or anything else? What are the mount options? AFAIK it is not supposed to behave like that. – Dima Chubarov Oct 22 '14 at 06:27
  • Well a newly deployed file shows the new file. We deploy with a `git pull` -- Mount options: nfs rsize=32768,wsize=32768,timeo=30,retrans=10,intr,noatime,soft,async,nodev – Michael Oct 22 '14 at 19:27
  • Since Linux NFS client has close-to-open cache consistency you need to make sure that the files are closed and reopened by the webserver as soon as the deployment is complete. – Dima Chubarov Oct 23 '14 at 06:35
  • Restart the webserver every time? – Michael Oct 23 '14 at 19:31
  • Sorry to ask dumb questions, but both server and client are NTP sync'ed, yes? – MadHatter Oct 23 '14 at 19:36
  • @MadHatter yes :) it's sync'ed – Michael Oct 23 '14 at 20:16
  • Need to restart the server process unless your application can be triggered to close open files & clear application caches. Many web frameworks that have caching functionality have something like that, e.g. `symfony cc` or `drush cc all` – Dima Chubarov Oct 24 '14 at 13:47
  • I don't know that it will matter much, but what OS and revision are the NFS server and client? – Dan Pritts Oct 28 '14 at 15:40
  • I second Dan Pritts comment - We really need to know the OS version of the NFS-server and the NFS-client (and the NFS version helps) – Signal15 Oct 30 '14 at 14:09

3 Answers3

8

Exists different options to cache file attribute on a NFS mount (client side):

NOAC
Use the noac mount option to achieve attribute cache coherence among multiple clients. Almost every file system operation checks file attribute information. The client keeps this information cached for a period of time to reduce network and server load. When noac is in effect, a client’s file attribute cache is disabled, so each operation that needs to check a file’s attributes is forced to go back to the server. This permits a client to see changes to a file very quickly, at the cost of many extra network operations.

lookupcache=none
If the client ignores its cache and validates every application lookup request with the server, that client can immediately detect when a new directory entry has been either created or removed by another client. You can specify this behavior using lookupcache=none. The extra NFS requests needed if the client does not cache directory entries can exact a performance penalty. Disabling lookup caching should result in less of a performance penalty than using noac, and has no effect on how the NFS client caches the attributes of files.

actimeo=n
Using actimeo sets all of acregmin, acregmax, acdirmin, and acdirmax to the same value.

  • acregmin=n, The minimum time (in seconds) that the NFS client caches attributes of a regular file before it requests fresh attribute information from a server. If this option is not specified, the NFS client uses a 3-second minimum.
  • acregmax=n, The maximum time (in seconds) that the NFS client caches attributes of a regular file before it requests fresh attribute information from a server. If this option is not specified, the NFS client uses a 60-second maximum.
  • acdirmin=n, The minimum time (in seconds) that the NFS client caches attributes of a directory before it requests fresh attribute information from a server. If this option is not specified, the NFS
    client uses a 30-second minimum.
  • acdirmax=n, The maximum time (in seconds) that the NFS client caches attributes of a directory before it requests fresh attribute information from a server. If this option is not specified, the NFS
    client uses a 60-second maximum.

Taken from the NFS man page.

I hope this help.

Federico Sierra
  • 3,589
  • 1
  • 20
  • 26
1

you only need to set acdirmax=1. By default the NFS mount uses Close-To-Open cache consistency. This means it will check the modification time and permissions of the file every time it opens a file, regardless of cached data. Issues occur if you add new files to a directory or overwrite files with new versions.

When files are overwritten on server A, just after server B read that same directory, the lookup cache on server B will hold a stale copy of these files. So when we try to access attributes of a file, it will send a request to the fileserver, but it will use the filehandle that is being stored in the cache, eventhough it was replaced by a new file with a different handle. Since the file with the old handle is gone it will report “File not Found”

So the solution to this problem would be to invalidate the lookup cache for a specific directory. Disabling the lookup cache completely is not an option, as it will overload the file server with NFS requests. The lookup cache checks if the cache is still valid by checking the timestamp of the directory that contains the file. If this is in the attribute cache, it will use it from cache. So to resolve your issues you should make sure that the attribute cache, specific for directories, has the lowest cache time possible.

This can be done using mountoptions : acdirmax=1

Franky_GT
  • 11
  • 1
1

You're specifying a lot of NFS options, most of which (rsize/wsize/soft) are not needed because they are already (usually) the defaults (in Linux);

NFS Mount Options rsize=32768,wsize=32768,timeo=30,retrans=10,intr,noatime,soft,async,nodev

However, if you are running Linux, you should probably look into setting the following NFS options;

actimeo=3 (sets acreg[max|min],acdir[max|min], can be lowered to 1 if you're still seeing speed problems)

lookupcache=none (ignores cache completely, default is "all")

noac (Disables attribute caching)

This blog post seems to have more information

Mark Lakata
  • 245
  • 2
  • 6
Signal15
  • 952
  • 7
  • 29