4

I've got a Mac OS X 10.6.4 Snow Leopard Server file server (AFP) which has been running out of storage space on the boot volume for the past few weeks. It takes about two days for the remaining 42GB on the 80GB boot volume to be eaten up, even though a sudo du -chsx -I dev / still shows only 29GB used.

I've run into this in the past w/a Linux server who's Apache logs were deleted after N days, but the log was still kept open by Apache, causing the storage to not be freed. I had been able to track it down relatively easily in that case w/a sudo lsof, but I'm not easily finding the culprit in this case (being a file server, there are a ton of open files and sockets). How can I sort lsof output by file size (and show sized in a human friendly format) so I can find the culprit?

The server has 3GB of RAM. After being up 4hrs, Activity Monitor shows 700MB free, 1.5GB inactive, and 200GB of VM. mds has the largest VM usage at 1.8GB, AppleFileServer in 2nd place w/500MB, and everything else is using 10MB-75MB of VM. That said, /private/var/vm is only 128MB.

Rebooting the system clears the issue, hence my belief that it's free storage space that's still being held open by some process or processes.

Any other hypothesis, suggestions, etc., greatly appreciated.

morgant
  • 1,470
  • 6
  • 23
  • 33

3 Answers3

18

You can find the biggest open files with:

sudo lsof -s | awk '$5 == "REG"' | sort -n -r -k 7,7 | head -n 50

This will list the regular files (not pipes, sockets, etc) sort by size in descending order, and take the top 50.

You might also look at what processes have the most files open, with something like

sudo lsof | awk '$5 == "REG" {freq[$2]++ ; names[$2] = $1 ;} END {for (pid in freq) print freq[pid], names[pid], pid ; }' | sort -n -r -k 1,1
Ben Jencks
  • 1,361
  • 8
  • 13
  • Thanks, those two work perfectly. Will still need to dig and explore to determine what the culprit is, but those are much needed tools. – morgant Nov 29 '10 at 22:11
  • FYI, the second awk command is taken almost verbatim from the "Word Sorting" page of the GNU awk manual. – Ben Jencks Nov 29 '10 at 23:18
  • Those two examples were enough to get my head around the syntax w/the `awk` man page to tweak them as necessary. I've found the culprit: lots (30, at this time) of 1.2GB files like `/private/var/folders/zz/zzzivhrRnAmviuee+++++++++++/-Tmp-/termIdFile.XXXXXX` which are open by `mds` (Spotlight), but result in a file-not-found error when trying to list/stat them. So, I just need to determine why `mds` is holding onto them. – morgant Nov 30 '10 at 16:11
2

I'd look at sudo lsof | grep deleted

James L
  • 6,025
  • 1
  • 22
  • 26
  • That doesn't return any results. The man page for `lsof` 4.82 notes that "(deleted)" will be appended under Solaris 10 and later if using the "-X" option (which isn't even available on OS X). The man page also implies that under Linux there's a "DEL" type for deleted maps, but `sudo lsof -s | awk '$5 == "REG"'` doesn't pull up anything. – morgant Nov 29 '10 at 22:20
  • 1
    Ah sorry, it works on Linux – James L Nov 29 '10 at 22:34
  • I'll certainly try it again when the volume is nearly full to see if maybe there just weren't any culprits at the time I ran `sudo lsof | grep deleted`. – morgant Nov 29 '10 at 23:00
  • I've found the culprits: lots of files like `/private/var/folders/zz/zzzivhrRnAmviuee+++++++++++/-Tmp-/termIdFile.XXXXXX` being held open by `mds` (Spotlight), but which are no longer files on disk. Unfortunately, they neither have "deleted" appended to their description by `lsof`, nor have a type of "DEL". I still appreciate the suggestion. – morgant Nov 30 '10 at 16:08
1

sudo lsof +L1 will show deleted files. For speed, combine this with -Pn.

e.g. lsof -Pn +L1

scottsome
  • 111
  • 1