Apache mod_wsgi crashing

Question

I'm trying to set up ReviewBoard and I'm running into an issue where mod_wsgi seems to be crashing. I get an error message like this:

AH00052: child pid 44359 exit signal Segmentation fault (11)

I tried to debug using GDB as stated here: https://code.google.com/p/modwsgi/wiki/DebuggingTechniques#Debugging_Crashes_With_GDB. I'm not having any luck. When I attach gdb and hit the URL that crashes, the only info I get is this:

Program received signal SIGSEGV, Segmentation fault.
[Switching to Thread 0x7f170ab51700 (LWP 45017)]
0x00007f17159f957f in __strlen_sse42 () from /lib64/libc.so.6
Missing separate debuginfos, use: debuginfo-install expat-2.0.1-11.el6_2.x86_64 keyutils-libs-1.4-5.el6.x86_64 krb5-libs-1.10.3-37.el6_6.x86_64 libcom_err-1.42.8-1.0.2.el6.x86_64 libffi-3.0.5-3.2.el6.x86_64 libgcc-4.4.7-11.el6.x86_64 libselinux-2.0.94-5.8.el6.x86_64 libstdc++-4.4.7-11.el6.x86_64 libuuid-2.17.2-12.18.0.1.el6.x86_64 openssl-1.0.1e-30.el6_6.8.x86_64 zlib-1.2.3-29.el6.x86_64
(gdb)

I'm running Apache 2.4.12, mod_wsgi 4.4.11 and Python 2.6.6 on Oracle Enterprise Linux 6. I'm pretty stuck here, because this is beyond anything I've had to do before.

EDIT: Obtained some more info:

Thread 2 (Thread 0x7fc77b65a700 (LWP 46498)):
#0  __strlen_sse42 () at ../sysdeps/x86_64/multiarch/strlen-sse4.S:32
No locals.
#1  0x00007fc78302ce4b in PyString_FromString (str=0x0) at Objects/stringobject.c:116
        size = <value optimized out>
        op = <value optimized out>
#2  0x00007fc783090811 in PyModule_AddStringConstant (m=<module at remote 0x7fc7759b56a8>, name=0x7fc76af55ec5 "XML_ERROR_UNDECLARING_PREFIX", value=<value optimized out>) at Python/modsupport.c:636
        o = <value optimized out>
#3  0x00007fc76af5508d in initpyexpat () at /usr/src/debug/Python-2.6.6/Modules/pyexpat.c:2041
        m = <module at remote 0x7fc7759a6fd8>
        d = <value optimized out>
        errmod_name = <value optimized out>
        errors_module = <module at remote 0x7fc7759b56a8>
        modelmod_name = <value optimized out>
        model_module = <module at remote 0x7fc7759b5718>
        sys_modules = <value optimized out>
        capi = {magic = 0x0, size = 0, MAJOR_VERSION = 0, MINOR_VERSION = 0, MICRO_VERSION = 0, ErrorString = 0, GetErrorCode = 0, GetErrorColumnNumber = 0, GetErrorLineNumber = 0, Parse = 0, ParserCreate_MM = 0, ParserFree = 0, 
          SetCharacterDataHandler = 0, SetCommentHandler = 0, SetDefaultHandlerExpand = 0, SetElementHandler = 0, SetNamespaceDeclHandler = 0, SetProcessingInstructionHandler = 0, SetUnknownEncodingHandler = 0, SetUserData = 0}
        capi_object = <value optimized out>
#4  0x00007fc78308d865 in _PyImport_LoadDynamicModule (name=0x7fc77b631500 "pyexpat", pathname=0x7fc77b630430 "/usr/lib64/python2.6/lib-dynload/pyexpat.so", fp=<value optimized out>) at Python/importdl.c:53
        m = <value optimized out>
        lastdot = <value optimized out>
        shortname = <value optimized out>
        packagecontext = 0x0
        oldcontext = 0x0
        p = 0x7fc76af54ad0 <initpyexpat>
#5  0x00007fc78308b8a5 in import_submodule (mod=None, subname=0x7fc77b631500 "pyexpat", fullname=0x7fc77b631500 "pyexpat") at Python/import.c:2592
        buf = "/usr/lib64/python2.6/lib-dynload/pyexpat.so\000\000e.so\000-py2.6.egg/pyexpat.pyc\000e.so\000yc\000e.so\000y2.6.egg/pyexpat.pyc\000e.so\000.so\000\307\177\000\000om\376\202\307\177\000\000\020\005c{\307\177\000\000\020R1\203\307\177\000\000\001\000\000\000\000\000\000\000\251\262\r\203\307\177\000\000\251\262\r\203\307\177\000\000+x\376\202\307\177\000\000\220\026%\002\000\000\000\000\020R1\203\307\177\000\000\030\000\000\000\060\000\000\000\340\005c{\307\177\000\000$\000\000\000\307\177\000\000\t\000\000\000\000\000\000\000"...
        fp = 0x7fc775a970e0
        path = <value optimized out>
        loader = 0x0
        fdp = <value optimized out>
        modules = 
    {'reviewboard.diffviewer.django': None, 'django.utils.deprecation': <module at remote 0x7fc774dddb08>, 'reviewboard.diffviewer.parser': <module at remote 0x7fc775496520>, 'pygments.filters': <module at remote 0x7fc775b5c050>, 'haystack.signals': <module at remote 0x7fc774f69948>, 'django.dispatch.threading': None, 'docutils.utils.warnings': None, 'pprint': <module at remote 0x7fc774600408>, 'django.middleware.gzip': <module at remote 0x7fc774ba8ad0>, 'django.dispatch.weakref': None, 'paramiko.proxy': <module at remote 0x7fc7756aa520>, 'django.utils.log': <module at remote 0x7fc774b2a750>, 'paramiko.select': None, 'reviewboard.admin.haystack': None, 'SocketServer': <module at remote 0x7fc774b9dcc8>, 'django.contrib.admin.decimal': None, 'xml.etree.sys': None, 'django.test.contextlib': None, 'shlex': <module at remote 0x7fc775550d00>, 'django.utils.http': <module at remote 0x7fc774a31948>, 'djblets.util.datetime': None, 'django.conf.urls.django': None, 'json.sre_constants': None, 'django.core.serializers': <mod...(truncated)
        m = <value optimized out>
#6  0x00007fc78308bb24 in load_next (mod=None, altmod=None, p_name=<value optimized out>, buf=0x7fc77b631500 "pyexpat", p_buflen=0x7fc77b6314f8) at Python/import.c:2412
        name = 0x7fc775a52b64 "pyexpat"
        dot = <value optimized out>
        len = 7
        p = 0x7fc77b631500 "pyexpat"
        result = <value optimized out>
#7  0x00007fc78308c17d in import_module_level (name=0x0, globals=<value optimized out>, locals=<value optimized out>, fromlist=['__doc__'], level=<value optimized out>) at Python/import.c:2133
        buf = "pyexpat\000\004\000\000\000\000\000\000\000\340\315\301u\307\177\000\000\260\367\277u\307\177\000\000\017\000\000\000\000\000\000\000p\352\230u\307\177\000\000\260\001{\207\307\177\000\000\345,\001\203\307\177\000\000p#c{\307\177\000\000t\000\000\000\000\000\000\000\004\000\000\000\000\000\000\000\250\351\b\203\307\177\000\000ponent(\001\000\000\000t\t\000\000\000universali\002\000\000\000i\005\000\000\000(\001\000\000\000t\n\000\000\000__import__t\006\000\000\000Writerc\000\000\000\000\000\000\000\000\001\000\000\000B\000\000\000s_\000\000\000e\000\000Z\001\000d\000\000Z\002\000d\001\000Z\003\000d\002\000Z\004\000d\003\000\204\000\000Z\005\000d\b\000Z\a\000"...

It seems to be related to pyexpat, and I did read about those issues here: https://code.google.com/p/modwsgi/wiki/IssuesWithExpatLibrary. However, I'm running Python 2.6.6, and it says there that those issues only apply to versions of Python prior to 2.5.

I thought that was just some sort of warning listing out some libraries. I guess it's actually asking me to install some stuff eh? I haven't used gdb in probably 10 years, so I'm clumsily trying to figure it out again. — cloudwalker, Apr 30 '15 at 14:07
Ok, so after installing those I still don't get much. I get the following now: Program received signal SIGSEGV, Segmentation fault. [Switching to Thread 0x7f4881ebb700 (LWP 45991)] __strlen_sse42 () at ../sysdeps/x86_64/multiarch/strlen-sse4.S:32 32 pcmpeqb (%rdi), %xmm1 — cloudwalker, Apr 30 '15 at 14:27
The expat problem can hit newer versions if Python was installed such that it was told to use system Python and that was different version to what Apache was told to use. So is rarer that it can occur now, but still can occur. — Graham Dumpleton, Apr 30 '15 at 21:25
Yeah, you are quite correct. I updated the system expat version to match what Python was running and it did fix the issue. I just didn't try that before since it said that it was only an issue for 2.5 or below. Thanks! — cloudwalker, May 05 '15 at 01:30

score 1 · Answer 1 · answered Jun 21 '16 at 15:21

You've already diagnosed the answer yourself. It is the same problem I was having.

Using Oracle Enterprise Linux 6 which has /lib64/libexpat.so.1.5.2
Using Oracle HTTP Server 12c (essentially a rebranded Apache), which has <install path>/ohs/lib/libexpat.so.0.5.0

Like you, I did a backtrace of the Apache httpd coredump. Apache crashed in __strlen_ss43 while calling initpyexpat, as part of loading the Python module pyexpat, which was being loaded because ReviewBoard needed it.

As you linked, it's a known issue caused by the Python pyexpat module being compiled for one version of libexpat, but an incompatible version is loaded.

On my system, the pyexpat module (installed as a dependency of ReviewBoard) is compiled and linked against the system's /lib64/libexpat.so.1.5.2. However, Apache is configured to load libexpat.so.0.5.0.

The linked article say that the problem is fixed in Python 2.6, but not the Python 2.6 in OEL 6:

$ ldd /usr/lib64/python2.6/lib-dynload/pyexpat.so
    linux-vdso.so.1 =>  (0x00007fff0c7de000)
    libexpat.so.1 => /lib64/libexpat.so.1 (0x00007f0f27dc0000)
    ...

The pyexpat module here isn't name prefixed as the linked article says, it's just using the system-wide libexpat, so anything will do.

I found that I do not use anything in the Apache setup which requires libexpat, except for ReviewBoard. So I am justified in removing/replacing the libexpat.so links in the Apache install directory with the one that pyexpat was compiled for:

# cd <install path>/ohs/lib
# mkdir disabled
# mv libexpat* disabled
# ln -s /lib64/libexpat.so.1 libexpat.so.0

Apache mod_wsgi crashing

1 Answers1