2

I have a python script base daemon (listen on network for client connection) which is using lots of memory, its 23G almost. following is my pmap output:

[root@example ~]# pmap -x 9766 | grep anon
0000000001f64000   31140   31140   31140 rw---    [ anon ]
0000000003dcd000 24265388 24265220 24265220 rw---    [ anon ]
000000308d875000      16      12      12 rw---    [ anon ]
000000308e409000     184       0       0 rw---    [ anon ]
0000003659a18000       8       0       0 rw---    [ anon ]
0000003989021000       4       4       4 rw---    [ anon ]
000000398998f000      20      16      16 rw---    [ anon ]
0000003989c19000      16       4       4 rw---    [ anon ]
000000398d017000       8       0       0 rw---    [ anon ]
0000003e49c1e000       4       4       4 rw---    [ anon ]
0000003e4a1df000      16      16      16 rw---    [ anon ]
0000003e4a585000       8       0       0 rw---    [ anon ]
0000003e4d02b000       4       0       0 rw---    [ anon ]
00007fc7d8000000     132       8       8 rw---    [ anon ]
00007fc7d8021000   65404       0       0 -----    [ anon ]
00007fc7e0000000     132       8       8 rw---    [ anon ]
00007fc7e0021000   65404       0       0 -----    [ anon ]
00007fc7e6bfe000       4       0       0 -----    [ anon ]
00007fc7e6bff000   10240      12      12 rw---    [ anon ]
00007fc7e75ff000       4       0       0 -----    [ anon ]
00007fc7e7600000   10240    2048    2048 rw---    [ anon ]
00007fc7e8000000     132       8       8 rw---    [ anon ]
00007fc7e8021000   65404       0       0 -----    [ anon ]
00007fc7ec000000     132       8       8 rw---    [ anon ]
00007fc7ec021000   65404       0       0 -----    [ anon ]
00007fc7f0000000     132       8       8 rw---    [ anon ]
00007fc7f0021000   65404       0       0 -----    [ anon ]
00007fc7f4179000    3076    3076    3076 rw---    [ anon ]
00007fc7f44b8000    1452    1440    1440 rw---    [ anon ]
00007fc7f466a000     908     884     884 rw---    [ anon ]
00007fc7f47f1000       4       0       0 -----    [ anon ]
00007fc7f47f2000   10240      12      12 rw---    [ anon ]
00007fc7f51f2000       4       0       0 -----    [ anon ]
00007fc7f51f3000   10240      12      12 rw---    [ anon ]
00007fc7f5bf3000       4       0       0 -----    [ anon ]
00007fc7f5bf4000   10240      12      12 rw---    [ anon ]
00007fc7f8503000     260     256     256 rw---    [ anon ]
00007fc7f8b56000     520     512     512 rw---    [ anon ]
00007fc7f8ddb000     260     256     256 rw---    [ anon ]
00007fc7f903d000     260     256     256 rw---    [ anon ]
00007fc7f9495000     520     512     512 rw---    [ anon ]
00007fc7f972c000    1292    1284    1284 rw---    [ anon ]
00007fc7fa08f000     260     256     256 rw---    [ anon ]
00007fc7fa4d9000     260     256     256 rw---    [ anon ]
00007fc7fb360000     260     256     256 rw---    [ anon ]
00007fc7fbfd6000     260     256     256 rw---    [ anon ]
00007fc801ea8000     520     512     512 rw---    [ anon ]
00007fc801f5c000     540     532     532 rw---    [ anon ]
00007fc8023ae000      60      60      60 rw---    [ anon ]
00007fc8023c5000       4       4       4 rw---    [ anon ]
00007fc8023c6000       4       4       4 rwx--    [ anon ]
00007fc8023c7000       4       4       4 rw---    [ anon ]
00007fffd45ff000       4       4       0 r-x--    [ anon ]
ffffffffff600000       4       0       0 r-x--    [ anon ]

I have notice it is growing, thank my system has 64G memory so it is still surviving but i am afraid it will crash once it reach to max point.

0000000003dcd000 24265388 24265220 24265220 rw---    [ anon ]

is above output looks normal? I am not expert but i need suggestion to know what is going on, How do i clean dirty memories?

following memory usage:

[root@example ~]# free -m
             total       used       free     shared    buffers     cached
Mem:         64389      46304      18085         22        242      12892
-/+ buffers/cache:      33170      31219
Swap:         1027          0       1027
Satish
  • 16,544
  • 29
  • 93
  • 149

1 Answers1

5

Looking at this it looks like most of your usage is here:

0000000003dcd000 24265388 24265220 24265220 rw--- [ anon ]

This is likely Python run-time's heap. It's impossible to guess what's in it by looking at pmap output in this case.

I recommend you attach the gdb debugger to this process. This gives you direct access to the process memory, and dump this segment to a file to examine it.

First you need to find the memory offset start and end (pmap doesn't give you that as far as I can see). You can dump smaps for the process for this:

cat /procs/9766/smaps

Find the corresponding block that starts at 0000000003dcd000. Should read something like:

03dcd000 -070be000 rw-p 00000000 00:00 0 Size: 60 kB Rss: 60 kB Pss: 51 kB Shared_Clean: 0 kB Shared_Dirty: 12 kB Private_Clean: 0 kB Private_Dirty: 48 kB Referenced: 60 kB Anonymous: 60 kB AnonHugePages: 0 kB Swap: 0 kB KernelPageSize: 4 kB MMUPageSize: 4 kB

Here the start/end of the data segment is 0x03dcd000 / 0x070be000

Attach the debugger:

gdb -p 9766

Then dump this memory to a file:

dump binary memory /root/swap_problem_raw 0x03dcd000 0x070be000

You can then view this file in a hex editor to understand what's going on. It should hopefully give you a clue what kind of data is being passed around if/when it's clear text. I currently use vim and works ok.

Summary information for /proc/$id/smaps can be found in proc man page.

Burak Cetin
  • 62
  • 4
  • I did `dump binary memory` command but it created empty file. how it is possible, if it is holding lots of data. – Satish Apr 16 '15 at 01:21
  • this is what i am seeing in smap `03860000-c7eb1000 rw-p 00000000 00:00 0` see last digit `0` that means it is empty – Satish Apr 16 '15 at 01:37
  • Here's what those figures mean: 3860000-c7eb1000 : process address offset for this segement rw-p : permissions for this segment (p for private) 00000000 : offset into the file (none in here) 00:00 : device version (none in here) 0 : inode on the device So, the size is the difference between the start and end. It's possible your offset has changed, have you restarted the process? (See your man page entry for "proc" for details on this.) – Burak Cetin Apr 16 '15 at 09:08
  • Yes we do restart to prevent 'OOM', I find new pid and follow your steps but everytime that location is empty. – Satish Apr 16 '15 at 13:11
  • Ok, at this point I'd try various things. You can try a different file path, you can try to dump another segment, try to access things like threads information or other misc information using other debugger commands (e.g. info threads). This is to eliminate possibilities like permissions failures. It sounds like your debugger can't access that part of memory, hence the empty file. This may be because the user you're doing this with doesn't have perms to access the other processes memory space. But impossible to say for me. Are you using the root user? – Burak Cetin Apr 16 '15 at 14:51
  • Yes, I am `root` and GDB version `7.2-75.el6` – Satish Apr 16 '15 at 14:57
  • Can you access _any_ part of the process memory using the debugger and the dump commands? – Burak Cetin Apr 16 '15 at 17:19
  • Yes, i tried other memory location of same process and it dump `100M` of data in that `raw` file. so gdb is working but i don't know why that location is empty which is show in `pmap` is that because of `[anon]` ? – Satish Apr 16 '15 at 17:24
  • Cool, hope you find something useful. – Burak Cetin Apr 17 '15 at 09:33
  • I analyze code but it's all hex and some logs style text, I am not seeing any script related code? Is that because I'm using pyc extension, precompiled Python – Satish Apr 17 '15 at 13:17
  • You wouldn't necessarily see code in this dump. This is the data segment used inside the python runtime. The idea is that you see data in clear text and that gives you a clue what it's. If this isn't the case then I recommend you look into other memory usage/leak techniques. – Burak Cetin Apr 20 '15 at 09:34