0

I have an Apache 2.4.23 configuration with mod_proxy/mod_proxy_balancer/mod_slotmem_shm/mod_lbmethod_byrequests enabled.

Furthermore, I have multiple VirtualHost listening on different ports with a setup like this (PORT and COMPONENT are unique):

Listen PORT

<VirtualHost *:PORT>
    Include "eib/conf/default-proxy-params"

    ErrorLog "logs/PORT-COMPONENT-error.log"
    CustomLog "logs/PORT-COMPONENT-access.log" common

    <Proxy balancer://COMPONENT/>
            BalancerMember https://HOST1:PORT
            BalancerMember https://HOST2:PORT
            ProxySet lbmethod=byrequests
    </Proxy>

    ProxyPass / balancer://COMPONENT/ lbmethod=byrequests nofailover=Off maxattempts=3
    ProxyPassReverse / balancer://COMPONENT/
</VirtualHost>

When I have only one VirtualHost entry in my config, everything works fine. As soon as I add an another VirtualHost, Apache crashes with a configuration failed error:

[Wed Oct 12 21:59:38.211829 2016] [slotmem_shm:error] [pid 4129010:tid 1] (24)Too many open files: AH02611: create: apr_shm_create(/apache/logs/slotmem-shm-pf3f0916c.shm) failed
[Wed Oct 12 21:59:38.211927 2016] [:emerg] [pid 4129010:tid 1] AH00020: Configuration Failed, exiting

I have checked that ulimit is set to unlimited for number of files, so this can not be the problem. It seems that there is a collision between slotmem files created for BalancerMemebers.

When I start with one single VirtualHost only, everything will work as expected and I get the following slotmem files created:

slotmem-shm-p99964586.shm
slotmem-shm-p99964586_COMPONENT.shm

One seems to be unique (_COMPONENT) for the virtual host, but the second one looks like it could cause name collisions.

Does someone have an idea how to solve this problem?

ivicaa
  • 111
  • 6
  • Can you add the output of the following commands to your post please: `apachectl -M`, and with both vhosts in place: `apachectl -S` – Unbeliever Oct 12 '16 at 20:35
  • I've appended the output to the post. Moreover, I configured a similar setup on Debian. It worked fine. It could be an AIX specific problem. – ivicaa Oct 12 '16 at 20:47
  • I think you must be correct. Your modules list looks completely standard. I would try not loading `slotmem_shm_module`. Also you should convert all your auth directives to apache v2.4 ones and comment out the loading of the `access_compat_module`. Also you could comment out the proxy modules you don't need. But overall it looks fine. – Unbeliever Oct 12 '16 at 20:54
  • slotmem_shm_module is required by mod_proxy_balancer, hence I have to include it. – ivicaa Oct 12 '16 at 20:56
  • Yes, sorry, I meant as a test more than anything else, but I was running out of characters in the comment. Would be interesting to see if a more simple setup worked. But this looks like a problem that you may need to take the the Apache httpd dev mailing lists. – Unbeliever Oct 12 '16 at 21:00
  • I just noticed there is a mod_slotmem_plain module which is using plain memory. Will mod_proxy_balancer work with this provider too? What would be the drawback? Only possible to share memory within the process? How would this affect the balancer? – ivicaa Oct 12 '16 at 21:47
  • The load balancing info would not be coordinated across multiple processes with slotmem_plain. – covener Oct 12 '16 at 22:48
  • I'd suggest running startup under truss -f ... apachectl ... -X to confirm which system call fals (shm_open, mmap, ???) – covener Oct 12 '16 at 22:48
  • I believe it's definitively AIX specific. When it comes to IPC stuff, it's getting worse. Could you please post `ipcs -a` with one virtual host so that we could know what's actually going on in the SHM? – Peter Zhabin Oct 12 '16 at 22:58
  • @covener: Thanks to your hint with truss I was able to solve the problem! Thanks a lot! – ivicaa Oct 13 '16 at 14:39

2 Answers2

1

praise and honour to covener! Using 

truss -f apachectl -X

I was able to trace the error to the shmat call 

6226262:    45089225: shmat(39845990, 0x00000000, 0)    Err#24 EMFILE

man shmat on AIX reveals the secret:

An extended shmat capability is available. If an environment variable EXTSHM=ON is defined then processes executing in that environment will be able to create and attach more than eleven shared memory segments.

ivicaa
  • 111
  • 6
1

I had a similar error on MacOS running an Apache compiled from source and it turned out the error actually is quite descriptive if you know what it means. There is apparently some tooling around the shared memory management (on both Linux and OS X).

In my case running:

ipcs -m

gave me some output like:

IPC status from <running system> as of Fri Feb  1 15:43:45 CET 2019
T     ID     KEY        MODE       OWNER    GROUP
Shared Memory:
m  65536 0x52043973 --rw-------     root    wheel
m 393217 0xe3046b9b --rw-------  myuser1    staff

... followed by say 20 more. I tried to free this up by doing:

ipcrm -m 393217

after that the error didn't occur and apache started. I hope this helps anyone, it should work on both Linux and MacOS and likely on *nix.

Vincent Gerris
  • 131
  • 3