2

I got a simple AWS cloud using nginx servers (EC2s) and NFS drive to store CMS files (EFS). Amazon has famous credits and I ran out of credits on a very low traffic cloud. After investigating it turns out that EC2 is flooding EFS with metadata requests. I just don't know how and why?

All the files in total are 8GB on NFS, they're all PHP files. Static files are offloaded to S3. The traffic between EC2 and EFS is about 1MB per second, however, metadata traffic is 5MB per second! That comes to between 100-300MB per minute of metadata traffic only!!!

After checking nfsstat and dumping the traffic it turns out that 90% of that traffic is GETATTR request and response.

But observing EC2, the processors on are almost asleep so processing 5MB per second sounds surreal. Not mentioning 5MB metadata for 1MB transfer.

I'd like to do some more search but my knowledge and Google has ended so don't know where from here. Tried to utilise lsof and nsfstats and can't find what in world is needing that much of metadata. Any help would be much appreciated.

EDIT: If you're looking for a solution for running out of AWS EFS credits because of this or similar reasons - EFS credits are based on the space you're taking. The solution (from AWS support team) is to create dummy files to increase your credit balance. In my case adding 50GB of "empty" taken space did the trick via command:

dd if=/dev/zero of=/path/to/efs/dummyfile bs=1G count=10

Above will create 10GB "empty" file on EFS that will increase your credits

Ralf
  • 23
  • 3
  • Is your socket for communicating with the php daemon also on NFS or just the files? – David Aug 25 '17 at 06:05
  • Is `find` running on the front end for any reason? – Cedric Knight Aug 25 '17 at 07:06
  • @david no NFS stores only files there's no logic on NFS all processing is on EC2 server – Ralf Aug 26 '17 at 08:43
  • @cedric do you mean find as unix find? No there's barely anything going on on the server and nginx. The more I look at it I suspect php-fpm socket asking constantly for file permissions but I can't figure out how to read those metadata packets to find out what they are. They're unreadable on wire shark :( – Ralf Aug 26 '17 at 08:47
  • Yes, I meant GNU `find` utility. Something like `mlocate.updatedb` uses it and generates a lot of `stat()` calls for a filesystem. `iotop`, `iostat`, 'lsof' and `strace` can all be useful for looking at filesystem activity. Have you tried `tcpdump` instead of wireshark? – Cedric Knight Aug 26 '17 at 12:36
  • Checking iotop there's barely anything happening. I can't find if it tracks metadata transfer though or only data transfer. lsof definitely doesn't show anything and yes, I did tcpdump and just read it via wireshark to see if I can read metadata but I can't :( – Ralf Aug 27 '17 at 23:43

0 Answers0