1

I have inherited some Lua code that communicates with a server via HTTP Request/Response. This has been running in the field for several years but I recently noticed some hangs in the timeout of the HTTP requests from the system. Each individual message has the capability to define its own timeout, but if none is set a default timeout of 30 seconds is used. I noticed on a system a couple of weeks ago that there was a hang on a timeout for about 15 minutes before it recovered and processing continued. But I am currently looking at a system that has been hung for well over 3 hours on a 30 second timeout. Here is the setup for the requests:

local socket = require "socket"
local http = require "socket.http"
local ltn12 = require "ltn12"
local ssl = require "ssl"
local try = socket.try
local protect = socket.protect
...
function serverapi.http_request(request, timeout)
    ... (local variable setup and logging)
    socket.TIMEOUT = timeout
    socket.http.TIMEOUT = timeout

    result, status_code, content = socket.http.request {
        url = request.url,
        method = request.method,
        headers = request.header,
        source = ltn12.source.string(request_body),
        sink = ltn12.sink.table(response_body),
    }

    ... (receive response and process)

I should note that the hangs are erratic in terms of the messages types they hang on. So it is not consistently with one message. And, as I said, this has been deployed and running in the field for several years.

Anyone have any ideas here...? Even if it's just a way to help debug what's going on. I don't even know how to get any kind of logging in what's going on after the request is sent and it's waiting for timeout.

Thanks

hjpotter92
  • 78,589
  • 36
  • 144
  • 183
linsek
  • 3,334
  • 9
  • 42
  • 55
  • What does the hang look like exactly? Where in the request is it hanging? Initial connection? Sending of the request? Waiting on the response to start? Waiting on the response to finish? – Etan Reisner May 09 '14 at 17:24
  • There is a log message immediately before the call to `socket.http.request` that logs the timeout and one immediately after printing the status code received. So the hang must be in that call. – linsek May 09 '14 at 17:43
  • 1
    can you capture the requests with wireshark then play them back to see if you get the same ones to hang? can you attach a debugger to the process? – Oliver May 09 '14 at 19:56

2 Answers2

0

I've had similar issues, and the socket library is old, last updated in 2007? I fix a bug in socket/http.lua, but it still occasionally hangs so that wasn't it. I tried luacurl and it also hangs, so it may be a problem at the compiled code level. I'm on Windows. nodejs has more momentum these days so I'm not sure how much I want to pursue Lua unless support improves.

while string.find(line, "^%s") do
    value = value .. line
    line, err = sock:receive() -- was line = sock:receive()
    if err then return nil, err end
end
Keith Mashinter
  • 209
  • 3
  • 3
0

I am still facing timeout issues for my system, I am trying to bring down the timeout to 30 sec but no luck for now but in the process have found some interesting articles and approaches. Hopes one works for you. my approach might seem a bit more kong biased and this is because my search was more kong specific.

=======

Approach 1

DNS Resolution can be taking time, switching to IP based can reduce the time taken

=======

Approach 2

We used HTTP and HTTPS timeout and it didn't work for us, refer to article no 2

=======

Approach 3

We are currently using the kong framework and am modifying the default timeout and keepalive parameters but it doesn't seem to work as well.

=======

Approach 4

changing underlying nginx configuration

Naman Lashkari
  • 1
  • 2