0

For the past few years we have been running a custom, pre-forking, HTTP server written in C++, libevent and embedded Lua (LuaJit using the exact same LuaJit shared library that openresty, an Nginx package, uses). It's got almost the same architecture as Nginx just by coincidence.

Most of the business functionality of this server is in the Lua script "handlers" and when we saw Nginx supported Lua we thought we'd move our code base over to Nginx and take advantage of all the community support that Nginx has. We got all the code moved over and our testing shows everything working, as far as the functionality being preserved, however....

We have run into a problem. The latency for an HTTP request on the old custom, C++ server was typically 2ms round trip. The new server based on Nginx is up around 60ms. The weird thing is, our internal measurements of the actual Lua code executing (using Lua os.clock() to time code regions) shows the two server versions are virtually identical as far as Lua execution speed.

Using various profilers doesn't show any extremely large area of slowness on the Nginx code. pcre is the highest with about 6% of the time spent.

One more thing. The setup we have has a Proxy Nginx server talking to our Nginx Lua Server, both on the same machine, talking over 127.0.0.1, so it's LAN => Nginx Proxy => Nginx Lua Server. 

ldd nginx
    linux-vdso.so.1 =>  (0x00007ffc02ff3000)
    libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0 (0x00007f88d0b46000)
    libcrypt.so.1 => /lib/x86_64-linux-gnu/libcrypt.so.1 (0x00007f88d090f000)
    libluajit-5.1.so.2 => /d1/apps/mt3/openresty/luajit/lib/libluajit-5.1.so.2 (0x00007f88d069a000)
    libm.so.6 => /lib/x86_64-linux-gnu/libm.so.6 (0x00007f88d0418000)
    libpcre.so.3 => /lib/x86_64-linux-gnu/libpcre.so.3 (0x00007f88d01db000)
    libssl.so.1.0.0 => /usr/lib/x86_64-linux-gnu/libssl.so.1.0.0 (0x00007f88cff79000)
    libcrypto.so.1.0.0 => /usr/lib/x86_64-linux-gnu/libcrypto.so.1.0.0 (0x00007f88cfb81000)
    libdl.so.2 => /lib/x86_64-linux-gnu/libdl.so.2 (0x00007f88cf97d000)
    libz.so.1 => /lib/x86_64-linux-gnu/libz.so.1 (0x00007f88cf765000)
    libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007f88cf3da000)
    /lib64/ld-linux-x86-64.so.2 (0x00007f88d0d72000)
    libgcc_s.so.1 => /lib/x86_64-linux-gnu/libgcc_s.so.1 (0x00007f88cf1c4000)

Anyone got any ideas?

Walt Howard
  • 7,676
  • 1
  • 16
  • 11
  • Which version of Lua is being embedded? It seems it can use luajit or standard lua 5.1. In reality it's probably got something to do with the interface between lua and nginx. Is it using the luajit ffi? – hookenz Feb 09 '16 at 19:28
  • 1
    I bet your problem doesn't have anything to do with Lua. – user253751 Feb 09 '16 at 19:33
  • I used that exact same, physical libluajit-5.1.so.2, linked it to the old code, and it performed at 2ms. So it's not the LuaJit. – Walt Howard Feb 09 '16 at 20:06
  • Careful with os.clock() on Linux. It returns consumed cpu time, and not wall time. If there was some sleep or i/o waits, you won't see it with os.clock() on Linux. – Vlad Feb 09 '16 at 20:19
  • Interesting, we found an Nginx setting in the proxy Nginx that fixed the problem. proxy_set_header Connection ""; Go figger. – Walt Howard Feb 09 '16 at 21:51

0 Answers0