4

I have some filenames (session files) that can't be written ("No space left on device"). Other filenames (same directory) are fine. Disk is not full. The filesystem is ext3

PHP is giving an error when trying to create the files, but the error can be reproduced on the command line:

# less /path/to/session_data/sess_u2q1pfelfr0jof3mp38jb2eaj1
/path/to/session_data/sess_u2q1pfelfr0jof3mp38jb2eaj1: No such file or directory
# touch /path/to/session_data/sess_u2q1pfelfr0jof3mp38jb2eaj1
touch: cannot touch `/path/to/session_data/sess_u2q1pfelfr0jof3mp38jb2eaj1': No space left on device
# touch /path/to/session_data/sess_u2q1pfelfr0jof3mp38jb2eaj0
# less /path/to/session_data/sess_u2q1pfelfr0jof3mp38jb2eaj0
# ls -al /path/to/session_data/sess_u2q1pfelfr0jof3mp38jb2eaj1
ls: /path/to/session_data/sess_u2q1pfelfr0jof3mp38jb2eaj1: No such file or directory

Note the different filenames. There doesn't seem to be any pattern, but it does affect certain filenames only, and they don't seem to be ever able to be written to.

To possibly complicate things further, this is an OpenVZ server, but you can't write to/create those filenames from either the virtual server or the hardware node.

There are a lot of files in that directory (18,456,002 at the time of writing), but there is no inode problem, and the disk is definitely not full.

As requested, df outputs:

[root@web1 session_data]# df -h
Filesystem            Size  Used Avail Use% Mounted on
/dev/simfs            2.7T  2.1T  495G  81% /
/dev/root             2.7T  2.1T  495G  81% /path/to/session_data
[root@web1 session_data]# df -i
Filesystem            Inodes   IUsed   IFree IUse% Mounted on
/dev/simfs           726761472 22843560 703917912    4% /
/dev/root            726761472 22843560 703917912    4% /path/to/session_data

(I've edited the path to the session data, but it really is mounted separately like that, to enable sharing of session data between multiple virtual servers)

Rob Ferrer
  • 171
  • 1
  • 7
  • 2
    `session.save_path` from `php.ini`, `df -h` and `df -i`, please. – GioMac Sep 17 '13 at 12:24
  • Did you mean "there is no inode problem" – jftuga Sep 17 '13 at 12:33
  • yes, sorry - there is no inode problem – Rob Ferrer Sep 17 '13 at 12:41
  • I've added the DF output. I hope I'm replying to the right comment - This is my first time on serverfault! – Rob Ferrer Sep 17 '13 at 12:46
  • Welcome to the serverfault. Looks like php is not cleaning up sessions. What is your filesystem? – GioMac Sep 17 '13 at 12:59
  • Thanks. We do our own session garbage collection (to allow us to keep relevant ones for longer than empty ones). The filesystem is ext3 (I do mention that). I'm aware there are a lot of session files (something we are looking to address), but it's not hit any limits (that I know of), and it's only certain filenames! – Rob Ferrer Sep 17 '13 at 13:02
  • There is a subdirectory limit of 32k and file limit is far from 13 mln. – GioMac Sep 17 '13 at 13:17
  • It's definitely not related to the filename. – GioMac Sep 17 '13 at 13:18
  • it may not be the filename itself, but once there is a problem creating a certain file, I am completely unable to create a file with that name (although other names are completely fine). I'm doing a file count now for you. – Rob Ferrer Sep 17 '13 at 13:21
  • edited to add number of files (18456002 at time of writing) – Rob Ferrer Sep 17 '13 at 14:10
  • Maybe quota? what shows repquota and quota commands? – Andrei Mikhaltsov Sep 17 '13 at 14:15
  • I have specifically disabled any quotas. running `quota` gives no output, and `repquota` says "Bad number of arguments". – Rob Ferrer Sep 17 '13 at 14:21
  • My most bizarre similar experience was years ago when some PHP application caused lots and lots of semaphores and shared memory segments to be created, but not ever destroyed. The system started to misbehave in very weird ways, at least under that user privileges. See `ipcs` output. Also, see if `dmesg` returns any filesystem related errors. – Janne Pikkarainen Sep 17 '13 at 14:26
  • Thanks Janne. I looked at semaphores. There are some, but not the sort of volume that would cause the volume of errors. There's nothing in dmesg. I have had a suggestion from another source that it could be something to do with the HTree used by dir_index. My theory is that it could be filename hashes clashing (which would explain why it only affects certain filenames). Not easy to test though. – Rob Ferrer Sep 17 '13 at 14:36

1 Answers1

4

Ext3 does hashing on file/dir-names.
With so many many files in one directory I wouldn't be surprised if you are running in hash-collision issues.
Having all the filenames of the same length and have the same structure makes the likelihood of a collision even worse.

If there is a collision the filename is effectively unusable, as you noticed.

I have no further experience with this sort of thing so I can't give you any more detailed advice.
(We ran into a similar problem in the office once and the resident Linux guru used some tunefs magic to make the problem go away. I never got the exact details what he did. Just that it had to do with hash-collisions on the filenames.)

You can apparently set the hashing behavior via tunefs. There is also a ext3 extension called "dirindex" that really helps dealing with directories containing so many files.

Tonny
  • 6,332
  • 1
  • 18
  • 31
  • Thanks, it's good to hear the hash-collision theory from another source. I'd love to hear something definite from someone, but in the meantime we'll look at reducing the number of files in the directory (we found our garbage collection wasn't functioning as it should have been, which is a start). – Rob Ferrer Sep 17 '13 at 15:10
  • I hadn't seen your comment about the hashing when I wrote my answer. I can get more info from our Linux guy, but he is on holiday at the moment. Another 2 weeks before he is back. As your are debugging: Maybe change the paths to /sessiondata// or something like that. Is usually rather trivial to implement and greatly reduces the amount of files per directory (at the expense of more directories and a slight overhead on generating the paths). Of course that is assuming you don't need to re-generate those paths in 500 different places. – Tonny Sep 17 '13 at 18:04
  • I've thought about that, but that would mean losing the existing session files, which means customers losing their baskets (which they complain about). We found a problem in our garbage collection which we've now fixed which should drastically reduce the number of files. Hopefully that should fix it for now, until we get a chance to overhaul the session handling. – Rob Ferrer Sep 17 '13 at 19:20