1

I'm trying to understand memory usage of some php processes. I've tried using both get_memory_usage() and pmap, but the results seem to be off by about one order of magnitude. I've tried with both memory_get_usage() and memory_get_usage(true), as well as memory_get_peak_usage(true), but even with memory_get_peak_usage(true) (the largest of all three varieties) there is still a huge with what is reported via pmap.

More specifically, calling memory_get_peak_usage(true) every minute within my php script returns values ranging between 1.75MB and 3.5MB, whereas a typical result of pmap -d PID gives something like:

...

b7839000       4 r---- 0000000000008000 0ca:00060 libcrypt-2.11.1.so
b783a000       4 rw--- 0000000000009000 0ca:00060 libcrypt-2.11.1.so
b783b000     156 rw--- 0000000000000000 000:00000   [ anon ]
b7864000       8 rw--- 0000000000000000 000:00000   [ anon ]
b7867000      12 r-x-- 0000000000000000 0ca:00060 libgpg-error.so.0.4.0
b786a000       4 r---- 0000000000002000 0ca:00060 libgpg-error.so.0.4.0
b786b000       4 rw--- 0000000000003000 0ca:00060 libgpg-error.so.0.4.0
b786c000       4 r---- 0000000000000000 000:00000   [ anon ]
b786d000      16 rw--- 0000000000000000 000:00000   [ anon ]
b7871000     108 r-x-- 0000000000000000 0ca:00060 ld-2.11.1.so
b788c000       4 r---- 000000000001a000 0ca:00060 ld-2.11.1.so
b788d000       4 rw--- 000000000001b000 0ca:00060 ld-2.11.1.so
bffc7000     136 rw--- 0000000000000000 000:00000   [ stack ]
f57fe000       4 r-x-- 0000000000000000 000:00000   [ anon ]
mapped: 32740K    writeable/private: 13116K    shared: 28K

If I understand correctly, the writeable/private figure is the most relevant figure because it is the memory used exclusively by the process. Nearly 13MB is a far cry from the amount reported by memory_get_peak_usage(true). Can someone please explain the discrepancy?

KingCrunch
  • 128,817
  • 21
  • 151
  • 173
robguinness
  • 16,266
  • 14
  • 55
  • 65

2 Answers2

3

My understanding is that memory_get_peak_usage() will return the amount of memory being used by your script. So that's not including the overhead of PHP.

Returns the peak of memory, in bytes, that's been allocated to your PHP script.

You can reduce the memory that PHP uses by compiling in fewer extensions.

PHP itself is a large application, especially with the default extensions. We (as php developers) get to write simple code like simplexml_load_string() and watch the magic happen, but the code powering that magic is kicking around somewhere in memory. Looking at the output of phpinfo() will reveal just how many extensions are installed. Inside PHP sits the code that will take your PHP and convert it into OPCODEs, then the code that executes those opcodes. Each instance of PHP as your executing it will carry that overhead.

This memory usage is clearly non-trivial, but to be expected. If you had to write all the code to process incoming GET/POST/FILES manage XML, file, and stream magic, etc. The memory usage would add up quickly.

As a result a common performance technique is to remove all unneeded extensions to shrink the executable size. For memory constrained servers (which is most of the loaded ones) removing a few extensions and dropping memory usage from 9 to 7 megs results in a bunch more apache workers running.

This overhead isn't shared because each execution really is a separate copy of PHP executing. There alternates available using the thread safe builds, but not all PHP extensions are thread safe.

Removing Extensions Take a look at the functions you're using. mysqli_*? xml_*? etc. Also take a look at how PHP is being built, here's my configure line:

Configure Command => './configure' '--with-apxs2=/usr/local/apache2/bin/apxs' '--with-mysql=mysqlnd' '--with-gd' '--enable-soap' '--with-libxml-dir=/usr/lib/' '--with-mysql-sock=/tmp' '--with-tidy' '--with-jpeg-dir=/usr/lib/' '--with-xsl' '--with-curl' '--with-zlib' '--enable-gd-native-ttf' '--with-openssl' '--with-mcrypt' '--with-pdo-mysql=mysqlnd' '--with-mysqli=mysqlnd' '--with-bz2' '--enable-bcmath'

If I was shaving memory down I would start by building PHP with ./configure --disable-all (to remove default extensions), then explicitly adding only the extensions I new I was using. I'm not using libxml for my code anymore, nor soap so I can drop those bits. I can outsource my tidy work to a gearman worker and command line, so I'll pull that as well. Then running though my code (having unit testing would be fantastic here) and seeing what broke. Re-building PHP with the required extension, rinse and repeat. Clearly don't do this in production or you're going to have a Bad Time.

preinheimer
  • 3,712
  • 20
  • 34
  • I think there is some truth to what you're saying, but shouldn't the overhead be shared by *all* PHP processes? If I understand the output of pmap, the writeable/private references only the memory space used exclusively by the process. Furthermore, I have ~40 of these processes running concurrently, and the total memory used (judging from top) is roughly 275-300MB. This means each process using 7-7.5MB. That's still a long way from what is reported by memory_get_peak_usage(). Where is this other memory being used? In linux, each activity that is using resources is identified as a process. – robguinness Feb 14 '13 at 15:12
  • Sorry, had to break this up into two comments. Just wanted to add: Where are the processes that are contributing to this "overhead"? If I run ps aux | grep php I see only ~40 instances of my script. Is this overhead hiding in some other processes, and if so, which ones? – robguinness Feb 14 '13 at 15:14
  • extended post with more details – preinheimer Feb 14 '13 at 15:36
  • Ok, thanks! Now it's making more sense. I still don't understand why there is no more functions in PHP to give a more comprehensive picture of memory usage. So if I understand you now correctly, my only choice is to re-compile php with fewer extensions. I can't simply comment some extensions out via php.ini file (as has been suggested in some posts). Any guidance on which ones I can safely remove? Obviously, this depends on my application, but should I basically go through the output of phpinfo() and start putting obvious ones on the chopping block? – robguinness Feb 14 '13 at 15:43
  • extended post with more details. also: some funny – preinheimer Feb 14 '13 at 16:02
  • Thanks again. FYI, today I re-compiled php with "./configure --disable-all" and gave it a test drive. Running a trivial script (php -r 'sleep(90);') resulted in a hugely lower memory footprint (pmap reported 1828KB for writeable/private). This can be compared to the memory used by the same script if run by my standard php executable: 11268K. That's more than 6 times less memory! Now I will begin to add back in the needed extensions.. – robguinness Feb 15 '13 at 14:55
  • I'm leaning towards accepting this answer because it really helped point me towards a good method to reduce the memory footprint of my application. But I'd still like to address the original intent of the question, which was to *understand* the output of memory_get_usage() (and pmap). You say that "that memory_get_peak_usage() will return the amount of memory being used by your script", but I still don't understand how memory used by the "script" is different from memory used by the php application which is processing the script. Is there some line within the output of pmap that would... – robguinness Feb 15 '13 at 17:18
  • ...correspond to the memory "used by the script"? Most of the lines seem to correspond to various extensions, except the first three lines which list "php5" as the mapping. Can I assume everything nearly everything else (e.g. [anon] and [stack]) are specific to the script that is being executed? – robguinness Feb 15 '13 at 17:19
  • Ok...things were going in a good direction, but now I'm thoroughly confused! I managed to get my script running with only 5 extensions enabled (mysql, curl, pcntl, posix, and json). I checked out the memory usage with pmap. Still looking good: about half to 1/3 of what was being used early. Then, I check out the resources being used via top. Surprise! The "bare bones" php process is using roughly the same amount of memory as the ones with the "bloated" php configuration. Even more disconcerting, it's using VASTLY more CPU than before. Even 15-20 times the earlier amount! What's going on here?? – robguinness Feb 15 '13 at 20:09
  • Finally, I have a version of PHP that doesn't use huge amounts of CPU, but the memory usage is still as high as before (as reported by top). Any other ideas how I can get it reduced? I have mysql and curl loaded as shared libraries. – robguinness Feb 19 '13 at 20:49
0

Bytes vs Bits are a factor of 10 out. Hence 1.3 MBytes = 13MBits

Dave
  • 3,280
  • 2
  • 22
  • 40
  • Thanks. Actually 1 byte = 8 bits, but I double-checked this. According to the manual for pmap (http://linux.die.net/man/1/pmap), memory usage is reported in kilobytes. memory_get_peak_usage() outputs usage/allocation in bytes, which I've then converted to megabytes using a function I stole from somewhere [$exp = floor(log($bytes) / log(1024)); $bytes/pow(1024, floor($exp));]...after double-checking some values using Google's handy disk storage converter. So I don't think that's the culprit. Although, it's suspicious how close the discrepancy is. – robguinness Feb 14 '13 at 14:55
  • Yeah i didn't look up the exact figure I always work on factor of 10 for simplicity I can remember that :) base 8 not my strong forte I'm afraid. Convert bytes to kb just do bytes/1024 and then do kb /1024 to get mb no need to mess around with that overly complex. equation to get the mb figure :) – Dave Feb 14 '13 at 15:03
  • 13116K = 12.8 mb which is closer to the base 8 conversion than the base 10 one I initially put. I'll place money on one of the figures being returned in bits not bytes even though it tells you something different! – Dave Feb 14 '13 at 15:04