Any idea of why mod_wsgi creates a coredump in Apache httpd?

Question

I went through the troubleshooting of mod_wsgi but cannot find an answer for my case of segmentation fault. I get the following coredump when the module mod_wsgi is integrated in my Apache httpd server. The server without mod_wsgi works well.

Apache httpd: 2.2.22
mod_wsgi: 3.3
Python: 2.6.7

Any idea what causes the coredump? Is there a thing or a work-around I could try?

The core dump:

Program terminated with signal 11, Segmentation fault.
#0  0x00007fe06c39d206 in wsgi_python_init () from /remote/projects1/pdrtke/install/httpd-2.2.22/modules/mod_wsgi.so
#1  0x00007fe06c3aadb5 in wsgi_hook_child_init () from /remote/projects1/pdrtke/install/httpd-2.2.22/modules/mod_wsgi.so
#2  0x00000000004424db in ap_run_child_init ()
#3  0x000000000047ea35 in child_main ()
#4  0x000000000047ef26 in make_child ()
#5  0x000000000047f198 in perform_idle_server_maintenance ()
#6  0x000000000047f67b in ap_mpm_run ()
#7  0x0000000000429361 in main ()

The httpd binary, compiled from source. (I ran configure --prefix=..., that's all)

> file httpd                                                                                                                                                                                
    httpd: ELF 64-bit LSB executable, x86-64, version 1 (SYSV), for GNU/Linux 2.6.4, dynamically linked (uses shared libs), not stripped
> ldd httpd
    linux-vdso.so.1 =>  (0x00007fffdc5ff000)
    libm.so.6 => /lib64/libm.so.6 (0x00007f33ef7fe000)
    libaprutil-1.so.0 => /remote/projects1/pdrtke/install/httpd-2.2.22/lib/libaprutil-1.so.0 (0x00007f33ef5d4000)
    libexpat.so.1 => /lib64/libexpat.so.1 (0x00007f33ef3aa000)
    libapr-1.so.0 => /remote/projects1/pdrtke/install/httpd-2.2.22/lib/libapr-1.so.0 (0x00007f33ef172000)
    librt.so.1 => /lib64/librt.so.1 (0x00007f33eef69000)
    libcrypt.so.1 => /lib64/libcrypt.so.1 (0x00007f33eed2e000)
    libpthread.so.0 => /lib64/libpthread.so.0 (0x00007f33eeb11000)
    libdl.so.2 => /lib64/libdl.so.2 (0x00007f33ee90d000)
    libc.so.6 => /lib64/libc.so.6 (0x00007f33ee5af000)
    /lib64/ld-linux-x86-64.so.2 (0x00007f33efa54000)

The module WSGI:

> file mod_wsgi.so       
    mod_wsgi.so: ELF 64-bit LSB shared object, x86-64, version 1 (SYSV), dynamically linked, not stripped
> ldd mod_wsgi.so
    linux-vdso.so.1 =>  (0x00007fffb8f0e000)
    libpython2.6.so.1.0 => /usr/lib64/libpython2.6.so.1.0 (0x00007f4c6dd87000)
    libpthread.so.0 => /lib64/libpthread.so.0 (0x00007f4c6db69000)
    libdl.so.2 => /lib64/libdl.so.2 (0x00007f4c6d965000)
    libutil.so.1 => /lib64/libutil.so.1 (0x00007f4c6d762000)
    libm.so.6 => /lib64/libm.so.6 (0x00007f4c6d50b000)
    libc.so.6 => /lib64/libc.so.6 (0x00007f4c6d1ad000)
    /lib64/ld-linux-x86-64.so.2 (0x00007f4c6e37b000)

The Python executable:

> file python
    python: ELF 64-bit LSB executable, x86-64, version 1 (SYSV), for GNU/Linux 2.6.4, dynamically linked (uses shared libs), not stripped
> ldd python
    linux-vdso.so.1 =>  (0x00007fff6a1ff000)
    libpython2.6.so.1.0 => /softntools/opt/Python-2.6/bin/../lib/libpython2.6.so.1.0 (0x00007f14730fc000)
    libpthread.so.0 => /lib64/libpthread.so.0 (0x00007f1472edf000)
    libdl.so.2 => /lib64/libdl.so.2 (0x00007f1472cdb000)
    libutil.so.1 => /lib64/libutil.so.1 (0x00007f1472ad8000)
    libm.so.6 => /lib64/libm.so.6 (0x00007f1472882000)
    libc.so.6 => /lib64/libc.so.6 (0x00007f1472524000)
    /lib64/ld-linux-x86-64.so.2 (0x00007f14733b0000)

score 2 · Accepted Answer · answered Jun 27 '12 at 07:06

In fact, we found the issue, it was a dependency issue:

mod_wsgi.so used a specific version of libpython2.6.so.1.0

ldd mod_wsgi.so libpython2.6.so.1.0 => /usr/lib64/libpython2.6.so.1.0 (0x00007f4c6dd87000)

vs. a different libpython2.6.so.1.0 used by the python binary itself.

> ldd python
    libpython2.6.so.1.0 => /softntools/opt/Python-2.6/bin/../lib/libpython2.6.so.1.0 (0x00007f14730fc000)

Even though these were the same files names, these files didn't have the same size

> ls -l /softntools/opt/Python-2.6/bin/../lib/libpython2.6.so.1.0

gave 3710590 bytes

> ls -l /usr/lib64/libpython2.6.so.1.0                                                                                                                                                         3:33PM
    -rw-r--r-- 1 root root 1594904 May  5  2010 /usr/lib64/libpython2.6.so.1.0

What I did to solve the issue was to recompile mod_wsgi by changing the LD_RUN_PATH variable to point to /softntools/opt/Python-2.6/lib/ and now it works.

score 0 · Answer 2 · answered Jun 26 '12 at 09:57

Rebuild everything with debugging symbols and get a better backtrace. There's obviously a bug somewhere, but without a proper backtrace you don't really have a hope of finding it, or even getting someone else to fix it for you (unless you really luck out and someone who has recently had the exact same problem happens to be available to answer).

Any idea of why mod_wsgi creates a coredump in Apache httpd?

2 Answers2