13

Our application runs in a Docker container on AWS:

  • Operating system: Ubuntu 14.04.2 LTS (Trusty Tahr)
  • Nginx version: nginx/1.4.6 (Ubuntu)
  • Memcached version: memcached 1.4.14
  • PHP version: PHP 5.5.9-1ubuntu4.11 (cli) (built: Jul 2 2015 15:23:08)
  • System Memory: 7.5 GB

We get blank pages and a 404 Error less frequently. While checking the logs, I found that the php-child process is killed and it seems that memory is mostly used by memcache and php-fpm process and very low free memory.

memcache is configured to use 2 GB memory.

Here is php www.conf

pm = dynamic
pm.max_children = 30
pm.start_servers = 9
pm.min_spare_servers = 4
pm.max_spare_servers = 14
rlimit_files = 131072
rlimit_core = unlimited

Error logs

/var/log/nginx/php5-fpm.log
[29-Jul-2015 14:37:09] WARNING: [pool www] child 259 exited on signal 11 (SIGSEGV - core dumped) after 1339.412219 seconds from start

/var/log/nginx/error.log

2015/07/29 14:37:09 [error] 141#0: *2810 recv() failed (104: Connection reset by peer) while reading response header from upstream, client: x.x.x.x, server: _, request: "GET /suggestions/business?q=Selectfrom HTTP/1.1", upstream: "fastcgi://127.0.0.1:9000", host: "example.com", referrer: "http://example.com/"

/var/log/nginx/php5-fpm.log
[29-Jul-2015 14:37:09] NOTICE: [pool www] child 375 started


/var/log/nginx/php5-fpm.log:[29-Jul-2015 14:37:56] WARNING: [pool www] child 290 exited on signal 11 (SIGSEGV - core dumped) after 1078.606356 seconds from start

Coredump

Core was generated by php-fpm: pool www.Program terminated with signal SIGSEGV, Segmentation fault.#0  0x00007f41ccaea13a in memcached_io_readline(memcached_server_st*, char*, unsigned long, unsigned long&) () from /usr/lib/x86_64-linux-gnu/libmemcached.so.10

dmesg

[Wed Jul 29 14:26:15 2015] php5-fpm[12193]: segfault at 7f41c9e8e2da ip 00007f41ccaea13a sp 00007ffcc5730ce0 error 4 in libmemcached.so.10.0.0[7f41ccad2000+32000]
[Wed Jul 29 14:28:26 2015] php5-fpm[12211]: segfault at 7f41c966b2da ip 00007f41ccaea13a sp 00007ffcc5730ce0 error 4 in libmemcached.so.10.0.0[7f41ccad2000+32000]
[Wed Jul 29 14:29:16 2015] php5-fpm[12371]: segfault at 7f41c9e972da ip 00007f41ccaea13a sp 00007ffcc5730b70 error 4 in libmemcached.so.10.0.0[7f41ccad2000+32000]
[Wed Jul 29 14:35:36 2015] php5-fpm[12469]: segfault at 7f41c96961e9 ip 00007f41ccaea13a sp 00007ffcc5730ce0 error 4 in libmemcached.so.10.0.0[7f41ccad2000+32000]
[Wed Jul 29 14:35:43 2015] php5-fpm[12142]: segfault at 7f41c9e6c2bd ip 00007f41ccaea13a sp 00007ffcc5730b70 error 4 in libmemcached.so.10.0.0[7f41ccad2000+32000]
[Wed Jul 29 14:37:07 2015] php5-fpm[11917]: segfault at 7f41c9dd22bd ip 00007f41ccaea13a sp 00007ffcc5730ce0 error 4 in libmemcached.so.10.0.0[7f41ccad2000+32000]
[Wed Jul 29 14:37:54 2015] php5-fpm[12083]: segfault at 7f41c9db72bd ip 00007f41ccaea13a sp 00007ffcc5730ce0 error 4 in libmemcached.so.10.0.0[7f41ccad2000+32000]
Peter Mortensen
  • 30,738
  • 21
  • 105
  • 131
jobycxa
  • 321
  • 1
  • 3
  • 6
  • You are using docker-compose or starting directly with the docker-machine? If possible let us know the commands or settings to start the services. – Gullit Miranda Jul 29 '15 at 13:12
  • Some other user reported compatibility issue with PHP 5.5 and memcache extension. Any workaroud for fixing this issue. https://github.com/yiisoft/yii2/issues/7100 – user1960200 Jul 31 '15 at 05:27
  • I have changed os to debian and it fixed the issue for now. – jobycxa Aug 05 '15 at 11:01
  • What do you mean by *"memcache"*? Do you mean *"memory cache"*? Or *"Memcached"*? Or something else? – Peter Mortensen Jun 06 '21 at 00:11

10 Answers10

16

While googling for this same issue, and trying hard to find a solution that was not related to sessions (because I have ruled that out) nor to bad PHP code (because I have several websites running precisely the same version of WordPress, and none have issues... except for one), I came upon an answer telling that a possible solution did involve removing some buggy extension (usually memcache/d, but could be something else).

Since I had this same site working flawlessly on one Ubuntu server, when switching to a newer server, I immediately suspected that it was the migration from PHP 5.5 to 7 that caused the problem. It was just strange because no other website was affected. Then I remembered that another thing was different on this new server: I had also installed New Relic. This is both an extension and a small server that runs in the background and sends a lot of analytics data to New Relic for processing. Allegedly, it's a PHP 5 extension, but, surprisingly, it loads well on PHP 7, too.

Now here comes the tricky bit. At some point, I had installed W3 Total Cache for the WordPress installation of that particular website. Subsequently, I saw that the performance of that server was so stellar that W3TC was unnecessary, and simply stuck to a much simpler configuration. So I could uninstall W3TC. That's all very nice, but... I forgot that I had turned New Relic on W3TC, too (allegedly, it adds some extra analytics data to be sent to New Relic). When uninstalling W3TC, probably there was 'something' left on the New Relic configuration in my server which was still attempting to send data through the W3TC interface (assuming that W3TC has an interface... I really have no idea how it works at that level), and, because that specific bit of code was missing, the php_fpm handler for that website would fail... some of the time. Not all the time, because I'm assuming that, in most cases, nginx was sending static pages back. Or maybe php_fpm, set to 'recycle' after 100 calls or so, would crash-on-stop. Whatever exactly was happening, it was definitely related to New Relic — as soon as I removed the New Relic extension from PHP, that website went back to working normally.

Because this is such a specific scenario, I'm just writing this as an answer, in the remote chance that someone in the future googles for the exact problem.

Gwyneth Llewelyn
  • 796
  • 1
  • 11
  • 27
  • 4
    Thanks for the great write-up! I had a very similar situation, but disabling the newrelic module did not fix the problem for me. After digging a bit more, I found that the Opcache module can be a common cause of conflict. After disabling that one I'm not seeing any more segfaults, and New Relic is back on. I had also recently installed the latest ionCube loader extension for php 7, which is another zend extension that may have been the cause of the conflict, but I don't have the option to disable that one, so I'm not able to test that. – Evan Mattson Oct 22 '16 at 11:16
  • 1
    Weirdly specifically the same as my problem - upgraded WP form php5 to php7 with W3TC and New Relic. Removed PHP new relic and all is well! – Robert Egginton Dec 13 '16 at 23:03
  • 1
    It's actually -most probably- a problem in de serializer for php-memcached. That extention is not really buggy but the default serializer messes up from time to time. It's very hard to find others with the same segfaults. you can fix this by installing php5-msgpack and add this in memcached.ini : `memcached.serializer=msgpack` – Glenn Plas Apr 17 '17 at 18:30
  • 2
    Forgot to mention that this only happens when opcache is enabled. like Evan states. also, these 2 in your ini file should also help `opcache.validate_permission=0` , `opcache.validate_root=0` – Glenn Plas Apr 17 '17 at 18:41
  • Thanks for this hint. It helped by by know that the issue is related to php extensions in general.. it was zend debug issue in my case. – Abdullah Nov 27 '17 at 06:03
  • Disabling newrelic whilst I'm not using it worked for me `newrelic.enabled=0` in `/usr/local/etc/php/conf.d/newrelic.ini` – Dominic Pixie Jun 27 '20 at 00:36
6

In my case it was related to zend debug/xdebug. It forwards some TCP packets to the IDE (PhpStorm), that was not listening on this port (debugging was off). The solution is to either disable these extensions or enable debug listening on the debugging port.

Peter Mortensen
  • 30,738
  • 21
  • 105
  • 131
Abdullah
  • 5,445
  • 10
  • 41
  • 48
3

I had this problem after installing xdebug, adding some properties to /etc/php/7.1/fpm/php.ini and restarting nginx. This is running on a Homestead Laravel box.

Simply restarting the php7.1-fpm service solved it for me.

Gudlaugsson
  • 31
  • 2
2

It can happen if PHP is unable to write the session information to a file. By default it is /var/lib/php/session. You can change it by using configuration session_save_path.

phpMyAdmin having problems on nginx and php-fpm on RHEL 6

Peter Mortensen
  • 30,738
  • 21
  • 105
  • 131
zainengineer
  • 13,289
  • 6
  • 38
  • 28
2

In my case it was Xdebug. After uninstalling it, it got back to normal.

Peter Mortensen
  • 30,738
  • 21
  • 105
  • 131
Mr Coder
  • 8,169
  • 5
  • 45
  • 74
1

In my case, it was caused by the New Relic PHP Agent. Therefore, for a specific function that caused a crash, I added this code to disable New Relic:

if (function_exists('newrelic_ignore_transaction')) {
    newrelic_ignore_transaction();
}

Refer to: https://discuss.newrelic.com/t/how-to-disable-a-specific-transaction-in-php-agent/42384/2

Peter Mortensen
  • 30,738
  • 21
  • 105
  • 131
Irfan Kamil
  • 71
  • 1
  • 3
1

In our case it was caused by Guzzle + New Relic. In the New Relic Agent changelog they've mentioned that in version 7.3 there was some Guzzle fix, but even using the 8.0 didn't work, so there is still something wrong. In our case this was happening only in two of our scripts that were using Guzzle. We found that there are two solutions:

  1. Set newrelic.guzzle.enabled = false in newrelic.ini. You will lose data in the External Services tab this way, but you might not need it anyway.
  2. Downgrade New Relic Agent to version 6.x that somehow also works
  3. If you are reading this when they've released something newer than version 8.0, you could also try to update New Relic Agent to the latest and maybe they fixed that
Peter Mortensen
  • 30,738
  • 21
  • 105
  • 131
labm0nkey
  • 626
  • 1
  • 5
  • 9
0

In my case I had deactivated the buffering function ob_start("buffer"); in my code ;)

Peter Mortensen
  • 30,738
  • 21
  • 105
  • 131
J. Doe
  • 119
  • 1
  • 8
0

A possible problem is PHP 7.3 + Xdebug. Please change Xdebug 2.7.0beta1 to Xdebug 2.7.0rc1 or the latest version of Xdebug.

Peter Mortensen
  • 30,738
  • 21
  • 105
  • 131
alexfort
  • 21
  • 1
  • 2
    Hey there. Given the age of this question, this answer likely won't be helpful *unless you have a firm reason for thinking this is the exact solution*, in which case, please elaborate on why you believe the above will solve OP's issue. – MyStackRunnethOver Feb 01 '19 at 22:23
0

For some reason, when I remove profile from my xdebug.ini modes, it fixes it for me.

i.e. change

xdebug.mode=debug,develop,profile

to

xdebug.mode=debug,develop
Mike P. Sinn
  • 126
  • 1
  • 1
  • 4