-1

I have a netapp server exportting NFS share (NFS3).
I am monitoring the fpolicy on the netapp server, this means, I listen to all File read operation of the volume (and share..).

From a CentOS machine, I mount that NFS and perform File read (cat) operation.
In the first time I will do "cat", in the netapp I can see that there was a FILE READ event.

But then if I do additional "cat filename", I dont get the event FILE READ counter to increase.
If I got to a different machine, and mount the NFS Share, and do cat, I will see the counter increase by one.
I assume - the NFS Client is having internal cache mechanism.

I want to disable the NFS Client Cache.

I mounted the NFS with the following command: 

mount -o lookupcache=none,actimeo=0,noac 10.10.10.1:/vol/vol0_unix /mnt/nfs1

Notice the lookupcache=none,actimeo=0,noac options - taken from link.

Are there additional NFS Client cache mechanism i am missing ?

My NFS Client is: Linux CENTOS 6.4 2.6.32-358.el6.x86_64

Machine NFS Version:

[ilan@DevCore-Centos6 nfs1]$ rpm -qa|grep -i nfs
nfs-utils-1.2.3-36.el6.x86_64
nfs-utils-lib-1.1.5-6.el6.x86_64
nfs4-acl-tools-0.3.3-6.el6.x86_64
[ilan@DevCore-Centos6 nfs1]$

I Assume that cachefilesd is not running by default.

Community
  • 1
ilansch
  • 179
  • 2
  • 13
  • 5
    *I want to disable the NFS Client Cache.* Why? What *problem* are you trying to solve? This looks like an XY problem. See http://meta.stackexchange.com/a/66378 – Andrew Henle Feb 10 '17 at 12:33
  • My actual problem is that I am monitoring FILE READ events on the Netapp, There are many NFS Client (different users BUT on the SAME Redhat machine) that performing FILE READ operations on the netapp server, and I only get SOME of the FILE READ events, I need to get all the FILE READ events, I dont care if its from different users or the same user. – ilansch Feb 12 '17 at 08:41
  • Sorry, this question has gone way past the sensible point. You've asked a question, and had an answer (two, in fact) - but you have some weird business need to outguess the linux kernel's VM subsystem. that you won't tell us, no matter how many times we ask. I'm voting to close as "unclear what you're asking". – MadHatter Feb 12 '17 at 20:39

2 Answers2

4

I think that your problem is that the cacheing that's going on is not specifically NFS-related. Here's what I see on the network when I do an md5sum of a small NFS-mounted file for the second time in five seconds:

12:29:27.615255 IP6 2a02:xxxx:6049:11:76d4:35ff:fe8d:39ec.1003 > 2a02:xxxx:6049:11:7271:bcff:feac:445a.2049: Flags [P.], seq 3028:3236, ack 34857, win 1432, options [nop,nop,TS val 453693630 ecr 2105657370], length 208: NFS request xid 3660266482 204 getattr fh 0,2/53
12:29:27.617595 IP6 2a02:xxxx:6049:11:7271:bcff:feac:445a.2049 > 2a02:xxxx:6049:11:76d4:35ff:fe8d:39ec.1003: Flags [P.], seq 34857:35153, ack 3236, win 500, options [nop,nop,TS val 2105661555 ecr 453693630], length 296: NFS reply xid 3660266482 reply ok 292 getattr NON 3 ids 0/-2006101416 sz 1610612736
12:29:27.617632 IP6 2a02:xxxx:6049:11:76d4:35ff:fe8d:39ec.1003 > 2a02:xxxx:6049:11:7271:bcff:feac:445a.2049: Flags [.], ack 35153, win 1432, options [nop,nop,TS val 453693632 ecr 2105661555], length 0

All the kernel has done is check that the file hasn't been modified, with a getattr (because it's correctly not cacheing any attributes, but instead going back to the server for fresh ones). Then it knows it the file is unmodified, so it can use the pages that are still in-core that contain the content. This is exactly what it would do with a local file as well, if I understand correctly.

So it's not that you want to turn off NFS cacheing so much as you want to modify file cacheing in the VM subsystem so that it specifically doesn't apply to NFS-mounted content. This may be quite a lot more difficult, because you're not trying to turn off something that's already there, you're trying to introduce a whole new distinction that probably isn't there right now, if you follow me.

I think I agree with Andrew Henle that it's sounding like an XY problem. Perhaps you should tell us what you are really trying to do.

Edit: you assert that the same is not true for local file access. However, I think you're wrong. I've connected a HDD to my desktop to which only one process has access, and from that I have twice in a row md5summed a file while using iostat to monitor traffic to and from the physical device.

First time:

Device:         rrqm/s   wrqm/s     r/s     w/s    rkB/s    wkB/s avgrq-sz avgqu-sz   await r_await w_await  svctm  %util
sdc               0.00     0.00    0.80    0.00    49.60     0.00   124.00     0.08   95.75   95.75    0.00  94.00   7.52

Note the read traffic (rkB/s) from the drive; now the second time:

sdc               0.00     0.20    0.00    0.40     0.00     2.40    12.00     0.00    1.00    0.00    1.00   1.00   0.04

so it looks like there, too, the VM subsystem is getting in the way. So at the moment I see no reason to think you aren't seeing on your netapp all the traffic that clients correctly generate, and I ask again, what are you actually trying to do here? Why is it that you "must disable [cacheing], and send the actual FILE READ from the redhat"? Because at the moment, it really looks like a non-problem.

MadHatter
  • 79,770
  • 20
  • 184
  • 232
  • he's trying to test fpolicy, a data feed of who accesses what data on the Netapp. – Basil Feb 12 '17 at 01:21
  • Seems that one would be okay with allowing a cache to operate in that case. It's hardly a real-world scenario for an NFS client to have to read from the server for every single file operation on the same file. I agree that it's not at all clear what OP wants to test here. – Spooler Feb 12 '17 at 02:00
  • @Basil by your own admission (first line of your answer), that's an assumption on your part. You may be right, but I'd really like the OP to clarify. – MadHatter Feb 12 '17 at 07:00
  • My actual problem is that I am monitoring FILE READ events on the Netapp, There are many NFS Client (different users BUT on the SAME Redhat machine) that performing FILE READ operations on the netapp server, and I only get SOME of the FILE READ events, I need to get all the FILE READ events, I dont care if its from different users or the same user. – ilansch Feb 12 '17 at 08:41
  • @ilansch I think I've shown above that not every client-side file access in fact generates a full file read. Why do you think, then, that you are not seeing all of those that are actually generated? – MadHatter Feb 12 '17 at 08:58
  • Hi, Thanks for answer, first of all I am not here to argue or something like that - I appreciate your help very much. second, You wrote "it specifically doesn't apply to NFS-mounted content", When I perform 2 cat operations on the same file (from same user or different users), locally in a certain filesystem on the CentOS machine, the CentOS kernel will have 2 FILE READ events, only on a mounted drive it acts with 1 actual FILE READ. – ilansch Feb 12 '17 at 08:58
  • Dont take my word, as I dont take others and I double check everything myself, Once I have proved the problem is not NFS Server (NetApp) caching (almost believe I did), And it is the client side caching, Whether NFS client or cache is some sort of internal kernel read cache mechanism that applies on specific type of mounts, I must disable it, and send the actual FILE READ from the redhat. – ilansch Feb 12 '17 at 09:05
  • @ilansch see my edit above. Please don't worry about coming across as argumentative, you aren't; I hope I don't, either. What I'm trying to do is drill down into the problem, to understand it better. – MadHatter Feb 12 '17 at 09:13
  • There are many client that use the same NFS mount in a specific redhat machine,(different users), They perform READ FILE operations on servers (via NFS), the problem is not every read operation will invoke FILE READ on server operating system, Why ? because as we are seeing - not all clients are sending FILE READ, some are having internal caching mechanism, So i disable the NFS cache, now I need to see how to deal with other caching mechanism. – ilansch Feb 12 '17 at 09:52
  • 2
    **Why do you care?** I'm not trying to be a pain about this, but it's really difficult to understand the underlying issue when you'll only tell us about a symptom - which isn't a symptom, it's correct behaviour. – MadHatter Feb 12 '17 at 10:07
  • I mean, on my Netapp share there are 2 files, I cat each file 2 times, each time from different user with different mount - on same machine. the wireshark tells me that only 2 READ operations were sent via NFS protocol. as we suspect, the server actually does get only 2 file read requests. – ilansch Feb 12 '17 at 10:36
  • 1
    @ilansch Doesn't that mean that you've confirmed that there is no problem with your monitoring? – Håkan Lindqvist Feb 12 '17 at 10:39
  • @ilansch OK, try this: `cat` your file once, check for the `READ` operation on the NFS server. `cat` it again, check for the `READ` operation on the server - from your description, you won't see another `READ` on the server. Then, as root on the Red Hat NFS *client*, run `echo 3 > /proc/sys/vm/drop_caches`. That will drop *all* cached data on the client. Then `cat` your file again. If you see a `READ` operation on the server after dropping all cached data, your file data was likely being held in the [page cache](https://en.wikipedia.org/wiki/Page_cache) on the client. – Andrew Henle Feb 12 '17 at 14:30
  • @ilansch You can experiment with dropping different portions of cached data on the server and see how the caching affects NFS operations that actually get sent to the server. See http://unix.stackexchange.com/questions/17936/setting-proc-sys-vm-drop-caches-to-clear-cache for some explanations. – Andrew Henle Feb 12 '17 at 14:31
  • @HåkanLindqvist there is no problem - my question is how to enforce the sent of file read from nfs client every time. – ilansch Feb 12 '17 at 15:24
  • OK, the problem is at page cache, when i run echo 3 > /proc/sys/vm/drop_caches, if i invoke cat operation after the flush, i see it on server, so is it possible to disable page cache on NFS file systems ? – ilansch Feb 12 '17 at 15:58
  • 1
    @ilansch I think you could clarify things a whole deal by actually explaining your end goal, a goal which probably ties in with why you are not satisfied despite also saying "there is no problem". Less reads due to caching should be a win-win situation (better performance, less network traffic, less load on the server side), why do you want to disable the cache? Example: Is your goal some form of file access auditing rather than monitoring NFS events? – Håkan Lindqvist Feb 12 '17 at 17:58
  • Read my previous comments on what I need to achieve.I audit file operation on the server. if the nfs client does not send file read due to some sort of cache, i need to disable it. Even if i will not impl it, because we understand the pro/conin client side cache on VM page cache, i still want to know if its do-able – ilansch Feb 12 '17 at 19:40
3

Assuming you're testing fpolicy and are simply looking for a way to reliably generate some traffic, I would skip the OS NFS stack completely and just use a python script using libnfs or something. That you have complete, documented control over when NFS operations are generated. The OS isn't designed to do this, and a quick google tells me that while it should work, it's not obvious and certainly not the default behaviour.

Basil
  • 8,851
  • 3
  • 38
  • 73
  • Phyton script for what? NFS file sharing? – ilansch Feb 08 '17 at 19:58
  • For generating traffic for testing fpolicy on the Netapp. Rather than letting the OS decide when to do an NFS operation, this would allow you to force the issue. – Basil Feb 10 '17 at 12:22