30

This may be better suited to server fault but I thought I'd ask here first.

We have a file that is prepended to every PHP file on our servers using auto-prepend that contains a class called Bootstrap that we use for autoloading, environment detection, etc. It's all working fine.

However, when there is an "OUT OF MEMORY" error directly preceding (i.e., less than a second or even at the same time) a request to another file on the same server, one of three things happens:

  1. Our code for checking if(class_exists('Bootstrap'), which we used to wrap the class definition when we first got this error, returns true, meaning that class has already been declared despite this being the auto-prepend file.

  2. We get a "cannot redeclare class Bootstrap" error from our auto-prepended file, meaning that class_exists('Bootstrap') returned false but it somehow was still declared.

  3. The file is not prepended at all, leading to a one-time fatal error for files that depend on it.

We could, of course, try to fix the out of memory issues since those seem to be causing the other errors, but for various reasons, they are unfixable in our setup or very difficult to fix. But that's beside the point - it seems to me that this is a bug in PHP with some sort of memory leak causing issues with the auto-prepend directive.

This is more curiosity than anything since this rarely happens (maybe once a week on our high-traffic servers). But I'd like to know - why is this happening, and what can we do to fix it?

We're running FreeBSD 9.2 with PHP 5.4.19.

EDIT: A few things we've noticed while trying to fix this over the past few months:

  • It seems to only happen on our secure servers. The out of memory issues are predominantly on our secure servers (they're usually from our own employees trying to download too much data), so it could just be a coincidence, but it deserves pointing out

  • The dump of get_declared_classes when we have this issue contains classes that are not used on the page that is triggering the error. For example, the output of $_SERVER says the person is on xyz.com, but one of the declared classes is only used in abc.com, which is where the out of memory issues usually originate from.

  • All of this leads me to believe that PHP is not doing proper end-of-cycle garbage collection after getting an out of memory error, which causes the Bootstrap class to either be entirely or partly in memory on the next page request if it's soon enough after the error. I'm not familiar enough with PHP garbage collection to actually act on this, but I think this is most likely the issue.

timiTao
  • 1,417
  • 3
  • 20
  • 34
jraede
  • 6,846
  • 5
  • 29
  • 32
  • 1
    Could be. Sounds like a hard edge case to hunt down. I'd use [dtrace](http://www.php.net/manual/en/features.dtrace.dtrace.php) to log (a) the compile of your prepend, (b) what GC is doing, and (c) error conditions. For the GC bit, you'll probably need to write your own [static probe](http://blog.experimentalworks.net/2008/12/dtracing-php/). Also, if you have APC running, I'd disable it for these diagnostics. – bishop Dec 18 '13 at 21:55
  • Thanks. Now comes the hard part of convincing ourselves that this is a big enough issue to delve into the interpreter and gc in order to fix. :-D – jraede Dec 18 '13 at 22:15
  • Are you using any kind of opcode cache there on those servers? Are you using php as a module or as (fast)cgi? Is your _server_ out of memory, or is a php script just hitting the limit? – Wrikken Dec 18 '13 at 22:25
  • No cache. Cgi. And it's just hitting the `memory_limit` ini setting as far as I can tell. – jraede Dec 18 '13 at 22:45
  • Straight up mod_cgi? If so, switch to fastcgi with process isolation and see if still get it. – bishop Dec 18 '13 at 23:06
  • We're still trying to figure out how to trigger it when we want to - I think we need to simultaneously load a page that gets the out of memory error while we hammer another page on the same server. But I'll try to test with fastcgi and get back to you. – jraede Dec 18 '13 at 23:40
  • My mistake, we're using fastcgi w/ nginx. – jraede Dec 19 '13 at 17:21
  • I know this is a two year old question ... but alternate theory: file system locking issue preventing timely open of the prepend, cascading into out of memory due to race condition on opening that file. – virmaior May 02 '15 at 14:06
  • That's the best guess I've seen yet. Unfortunately I can't duplicate it anymore since I've left that company, but this will probably always bug me. – jraede Jul 03 '15 at 03:00
  • This sounds very convoluted to prepend a file to every php-file by some built-in php magic. Shouldn't it be easier to not use that feature at all if it's so error prone? – greenone83 Sep 10 '20 at 20:14

3 Answers3

1

You might not be able to "fix" the problem without fixing the out of memory issue. Without knowing the framework you're using, I'll just go down the list of areas that come to mind.

You stated "they're usually from our own employees trying to download too much data". I would start there, as it could be the biggest/loudest opportunity for optimizations, a few idea come to mind.

  • if the data being downloaded is files, perhaps you could use streams to chunk the reads, to a constant size, so the memory is not gobbled up on big downloads.

  • can you do download queueing, throttling.

  • if the data is coming from a database, besides optimizing your queries, you could rate limit them, reduce the result set sizes and ideally move such workloads to a dedicated environment, with mirrored data.

  • ensure your code is releasing file pointers and database connections responsibly, leaving it to PHP teardown, could result in delayed garbage collection and a sort of cascading effect, in high traffic situations.

Other low hanging fruit when it comes to memory limits

  • you are running php 5.4.19, if your software permits it, consider updating to more resent version "PHP 5.4 has not been patched since 2015" besides PHP 7 comes with a whole slew of performance improvements.

  • if you have a client side application involved monitor it's xhr and overall network activity, look for excessive polling and hanging connections.

  • as for your autoloader, based on your comment "The dump of get_declared_classes when we have this issue contains classes that are not used on the page that is triggering the error" you may want to check the implementation, to make sure it's not loading some sort of bundled class cache, if you are using composer, dump-autoload might be helpful.

  • sessions, I've seen some applications load files based on cookies and sessions, if you have such a setup, I would audit that logic and ensure there are no sticky sessions loading unneeded resources.

It's clear from your question you are running a multi-tenency server. Without proper stats it hard to be more specific, but I would think it's clear the issue is not a PHP issue, as it seems to be somewhat isolated, based on your description.

Proper Debugging and Profiling

I would suggest installing a PHP profiler, even for a short time, new relic is pretty good. You will be able to see exactly what is going on, and have the data to fix the right problem. I think they have a free trial, which should get you pointed in the right direction... There are others too, but their names escape me at the moment.

rexfordkelly
  • 1,623
  • 10
  • 15
0

Even if class_exists returns false, it would never return true if an interface of the same name exists. However, you cannot declare an interface and class of the same name.

Try running class_exists('Bootstrap') && interface_exists('Bootstrap') to make sure you do not redeclare.

Ozzy
  • 10,285
  • 26
  • 94
  • 138
  • If that were the case it would have happened every time. The issue I described happens intermittently, even on the same page. I've since left that company, but regardless, we didn't use interfaces. But thanks for the idea – jraede Feb 10 '15 at 13:39
0

Did you have a look at __autoload function?

I believe that you could workaround this issue by creating some function like that in your code:

function __autoload($className)
{
    if (\file_exists($className . '.php')) 
        include_once($className . '.php');
    else 
        eval('class ' . $className . ' { function __call($method, $args) { return false; } }');
}

If you have a file called Bootstrap.php with class Bootstrap declared inside it, PHP will automatically load file, otherwise declare a ghost class that could handle any function call inside it, avoiding any error messages. Note that for ghost function I used __call magic method.

Bruno Natali
  • 13
  • 3
  • Eval is actually bad pattern. There would be much better to use proper auto-loading tool. For example that one which was included in composer. – Lukáš Klíma Feb 04 '21 at 13:37
  • I agree with you @Arziel, but composer actually not have capability to handle nonexistent classes and it`s function. I called eval() to build an unbreakable code. – Bruno Natali Feb 09 '21 at 17:06