1

In the last few months we're experiencing a major issue in our production php servers:

The httpd running on our linux servers hangs and not responding to new requests (up to request timeout) until we manually restart it.

We've documented this issue quite a bit, and I can point out a few major investigation directions we want to follow, but before starting the research i wanted to ask the community if someone else might be experiencing a similar problem.

we are running 4 instances of linux servers, with version: linux- 2.6.32-71, red had 4.4.4-13
and httpd versions: 2.2.15

the php versions we are running are 5.5.15 and 5.5.10 (2 servers on each version).

we have detected a few suspicious errors in the logs, but notably couldnt find a specific error near the crashes :

  1. Servers seems busy, (you may need to increase StartServers, or Min/MaxSpareServers) spawning X children, there are Y idle and Z total children

  2. mod_wsgi Initializing Python

  3. mod_wsgi Terminating Python
  4. mod_wsgi Cleanup interpeter: ''
  5. mod_wsgi: Destroying interpeters
  6. KeyError(NUMBER) in ignored
    this is a php server, why is it throwing python errors?

  7. SIGHUP received. Attempting to restart - * glibc detected * /usr/bin/httpd: corrupted double-linked list: ADDRESS ***

  8. Connection reset by peer: core_output_filter: writing data to network

    • We noticed that "/var/log/httpd/error_log" and "/var/log/httpd/access_log" removed every time the error occur. We are having a really hard time with this subject, and would really appriciate it if someone can help shed some light on this matter.

Thank you very much!

user2033370
  • 11
  • 1
  • 1
    you should post the exact error messages, possibly redacting only personal data. with what you did write its somehow unclear what the error logs actually show. (for me at least) https://github.com/moby/moby/issues/34885 looks like a starting point for one of your issues. the 8th point: either my english is too bad (non-nativ here) or you might want to respell the "we noticed that..." sentence, i dont understand it. if you mean the 2 logfiles are empty everytime you get the error this seems really bad and might be a reason to check for Hardware and/or non apache related issues as well. – Dennis Nolte May 16 '18 at 11:49
  • thanks for the response. what i meant is that the log files are non existent, its not that they are empty. after we restart the apahce they are created again – user2033370 May 16 '18 at 14:47
  • so you might want to find out where apache writes to in this situation, strace/gdb on start, or lsof and /proc fs when the process already runs. this might give you a hint that the file gets deleted by something, where you then need to find that something. – Dennis Nolte May 17 '18 at 07:37

0 Answers0