5

On a new Ubuntu server, our Apache2 mod_perl2 system randomly fails when more than 8 clients connect. Once it fails, it won't recover -- all further fast-CGI activities fail, and Apache must be restarted. The error is very peculiar: it says there are syntax errors in standard Perl modules that it's trying to reload. Clearly this is wrong since it already loaded them many times before, and these are standard Perl modules.

Here is the Perl stack trace we get:

    [Thu Oct 04 18:33:09.222314 2018] [perl:error] [pid 11191:tid 139899223140096] [client 10.100.1.61:38632] failed to resolve handler `Che\
mAxonServer': syntax error at /usr/lib/x86_64-linux-gnu/perl/5.22/IO/Handle.pm line 65, near "croak "usage: $class->new()""
syntax error at /usr/lib/x86_64-linux-gnu/perl/5.22/IO/Handle.pm line 73, near "croak "usage: $class->new_from_fd(FD, MODE)""
syntax error at /usr/lib/x86_64-linux-gnu/perl/5.22/IO/Handle.pm line 99, near "croak "IO::Handle: bad open mode: $mode""
syntax error at /usr/lib/x86_64-linux-gnu/perl/5.22/IO/Handle.pm line 104, near "croak 'usage: $io->fdopen(FD, MODE)'"
syntax error at /usr/lib/x86_64-linux-gnu/perl/5.22/IO/Handle.pm line 123, near "croak 'usage: $io->close()'"
syntax error at /usr/lib/x86_64-linux-gnu/perl/5.22/IO/Handle.pm line 137, near "croak 'usage: $io->opened()'"
syntax error at /usr/lib/x86_64-linux-gnu/perl/5.22/IO/Handle.pm line 142, near "croak 'usage: $io->fileno()'"
syntax error at /usr/lib/x86_64-linux-gnu/perl/5.22/IO/Handle.pm line 147, near "croak 'usage: $io->getc()'"
syntax error at /usr/lib/x86_64-linux-gnu/perl/5.22/IO/Handle.pm line 152, near "croak 'usage: $io->eof()'"
syntax error at /usr/lib/x86_64-linux-gnu/perl/5.22/IO/Handle.pm line 157, near "croak 'usage: $io->print(ARGS)'"
/usr/lib/x86_64-linux-gnu/perl/5.22/IO/Handle.pm has too many errors.
Compilation failed in require at /usr/lib/x86_64-linux-gnu/perl/5.22/IO/Seekable.pm line 9.
BEGIN failed--compilation aborted at /usr/lib/x86_64-linux-gnu/perl/5.22/IO/Seekable.pm line 9.
Compilation failed in require at /usr/lib/x86_64-linux-gnu/perl/5.22/IO/File.pm line 11.
BEGIN failed--compilation aborted at /usr/lib/x86_64-linux-gnu/perl/5.22/IO/File.pm line 11.
Compilation failed in require at /usr/share/perl/5.22/IO/Compress/Base.pm line 11.
BEGIN failed--compilation aborted at /usr/share/perl/5.22/IO/Compress/Base.pm line 11.
Compilation failed in require at /usr/share/perl/5.22/IO/Compress/RawDeflate.pm line 9.
BEGIN failed--compilation aborted at /usr/share/perl/5.22/IO/Compress/RawDeflate.pm line 9.
Compilation failed in require at /usr/share/perl/5.22/IO/Compress/Gzip.pm line 11.
BEGIN failed--compilation aborted at /usr/share/perl/5.22/IO/Compress/Gzip.pm line 11.
Compilation failed in require at /emi/src/trunk/public_html_chemaxon/fcgi/ChemAxonServer.pm line 29.
BEGIN failed--compilation aborted at /emi/src/trunk/public_html_chemaxon/fcgi/ChemAxonServer.pm line 29.
Compilation failed in require at (eval 3) line 2.

Further attempts to use the system result in repeated errors, like this:

[Thu Oct 04 18:33:09.514512 2018] [perl:error] [pid 11191:tid 139899072071424] [client 10.100.1.61:38686] failed to resolve handler `ChemAxonServer': Attempt to reload ChemAxonServer.pm aborted.\nCompilation failed in require at (eval 9) line 2.\n

As far as we can tell, there are no significant differences between the old and new servers:

Ubuntu 14.04 Apache 2.4.18

I can only guess that it's hitting some resource limit, like memory or files, and once that happens it can't recompile the modules, and assumes that compiler's error message is real, when in reality the compiler is hitting a resource limit. But this is just a guess.

Thanks, Craig

Craig James
  • 51
  • 1
  • Craig, I couldn't find the question in your post, could you be more explicit about what question you're asking? – LuRsT Oct 04 '18 at 20:13
  • What mpm are you using? IMO using anything but prefork is a disaster with Perl. But regardless of mpm, you should be compiling all the modules you will be loading at startup, not during a request. – ysth Oct 07 '18 at 20:11
  • 1
    LuRsT -- Sorry, I thought the question was pretty clear. mod_perl2 gets corrupted in some way by ordinary use, and can't recover. It requires restarting Apache. That's not acceptable behavior for a web server. What is making this happen? Is it a serious bug in mod_perl2, or are we doing something stupid? – Craig James Oct 08 '18 at 21:03
  • ysth -- We used all standard modules from Debian. According to Apache documentation, that means we're using the "event" mpm. – Craig James Oct 08 '18 at 21:12
  • We solved this problem by using PerlModule in the config file to preload/compile the modules. This, I believe, matches ysth's comment above that everything should be compile at startup rather than when the specific Perl program is loaded. The problem disappeared entirely; we could run 100 requests simultaneously, and never found a problem. – Craig James Oct 08 '18 at 21:18

0 Answers0