4

I call shmget many times in my program, to get shm of of average size 85840 bytes. I obtain about 32771 shm ok, then shmget doesn't return a shm, but the error: "No space left on device".

I have increased the kernel limits to:

$ sysctl -A|grep shm
kernel.shmmax = 33554432
kernel.shmall = 1677721600
kernel.shmmni = 409600

But still get the issue. Why?

Do I have to put something into /etc/security/limits.conf too? I only have "user - nofile 1000000" because the program also opens about as many files as shms.

This is the output of free

$ free
          total       used       free     shared    buffers     cached
Mem:       8150236    7261676     888560          0     488100    3270792
-/+ buffers/cache:    3502784    4647452 
Swap:     12287992     554692   11733300

And ipcs

$ ipcs -lm                                                                         

------ Shared Memory Limits --------
max number of segments = 409600
max seg size (kbytes) = 1638400
max total shared memory (kbytes) = 6710886400
min seg size (bytes) = 1

Since I assume shm is capable of being swapped out, there should be enough space.

j13r
  • 163
  • 1
  • 1
  • 7

4 Answers4

3

Use ipcs -l to check the limits actually in force, and ipcs -a and ipcs -m to see what is in use, so you can compare the output. Look at the nattch column: are there segments with no processes attached that were not removed when processes exited (which normally means the program crashed)? ipcrm can clear them, although if this is a test machine, a reboot is quicker (and will make sure your changes to limits are picked up).

Your kernel parameters seem odd. In particular, shmall is a count of pages, not bytes, and 4kB is the default page size (run getconf PAGESIZE to check what you are using). How many terabytes of RAM do you have?

Now, you say you get about 32771 shared memory segments, which is also about 32768 (or 2 to the 15) which suggests a signed 16-bit int is the limiting factor. And what kernel are you running (as this will have its own limits)? The two may be related.

Kalle Richter
  • 268
  • 6
  • 18
ramruma
  • 2,740
  • 1
  • 15
  • 8
  • 1
    Setting SHMMAX to something huge is fairly common practice. Oracle, for example, recommends settings it to 4294967296 on 32-bit systems, iirc, which is a lot more than the 1677721600 above. It doesn't hurt to set SHMALL and SHMMAX to values that are higher than the available amount of memory. The value Oracle recommends for SHMALL is 1073741824, which, in pages of 4KiB, is the same as recommended for SHMMAX. – wzzrd Mar 19 '12 at 11:24
  • PAGESIZE is 4kb, I use Linux 3.0.22. Interesting point about the base2 coincidence, I'll investigate on that. Thanks @ramruma. – j13r Mar 19 '12 at 12:52
  • I added the output of ipcs -l, I don't think that's the problem. – j13r Mar 19 '12 at 13:05
  • Wouldn't a limiting factor involving a signed int always mean that the amount of shared segments will be *lower* than 32768? Or am I misunderstanding you? – wzzrd Mar 19 '12 at 14:22
  • Oracle has always had the SHMMAX recommended setting wrong. – kmarsh Sep 13 '18 at 17:59
3

It turns out shmmni is limited to 32768 in the kernel:

#define IPCMNI 32768  /* <= MAX_INT limit for ipc arrays (including sysctl changes) */

in the file ...version.../include/linux/ipc.h.

So short of recompiling the kernel, that is the hard limit on the number of shared memory segments.

Kalle Richter
  • 268
  • 6
  • 18
ramruma
  • 2,740
  • 1
  • 15
  • 8
  • That seems to be it. I'll try recompiling the kernel – j13r Mar 21 '12 at 13:02
  • Here, they talk about that it's the limit, http://www.tebyan.net/index.aspx?pid=31159&BookID=23749&PageIndex=136&Language=3 and that you can basically just change it: http://us.generation-nt.com/answer/patch-1-3-ipc-rename-ipcmni-ipcmni-max-help-198345051.html I think kernel-developers are just afraid of carrying a too-large array around. – j13r Mar 21 '12 at 13:03
  • Hi I have same problem as this. How to fix it? I don't understand above solution. It means modify file ipc.ch and then recompile kernel? instruction in detail please. thanks for help. – KentZhou Apr 07 '14 at 12:53
0

Just do:

sysctl -a | grep shm

and find your kernel.shmmni entry. Double that entry value in your /etc/sysctl.conf.

mforsetti
  • 2,666
  • 2
  • 16
  • 20
Nikos
  • 1
0

As shmget() allocates a new shared memory segment, and you seem to use a lot of them (considering your limits.conf), isn't possible you are using too many shared memory segments? I do not have a lot of experience with calling shmget(), but it seems to me the number of possible open files (1000000) is larger than the amount of allowed shared memory segments (SHMMNI), which is 409600.

wzzrd
  • 10,409
  • 2
  • 35
  • 47