1

(This question replaces this one, hopefully with better information.)

I've got three servers that I'm gonna call Alice, Charlie, and Harry (Harry being the server's actual nickname, so as not to confuse myself). They all talk to each other to perform quite a complicated authentication flow:

  • A client performs a sequence of three requests through Alice to Harry.
  • On the third one, Harry makes a call to Charlie.
  • Charlie is prone to timeouts during periods of heavy traffic. If it does, Harry returns a 503 with a Retry-After header to Alice.
  • Harry is returning a 503, I have confirmed that in its own logs.
  • Alice is not receiving a 503 but a 500, and without the header.
  • Alice's other clients (which I'm not in control of) treat a 500 the same as other errors, which is what I'm ultimately trying to fix.

Some extra information, such as I have been able to ascertain:

  • Alice proxies calls to Harry using RestClient, which uses Net::HTTP under the hood.
  • Using Net::HTTP directly gives the same result.
  • It's not environment specific; I have had this problem both in Production and Development.
  • I have been trying to simulate Alice using Postman, but haven't had any luck yet; Charlie's traffic is quieter at the moment, and I can't force or simulate a timeout, so so far I've only been getting successful 200 responses from Harry.
  • Fixing Charlie's timeouts would obviously be ideal, but I'm not in control of Charlie's hardware either.

Is there something I can change about Alice so it properly detects Harry's 503?

Or, is it possible that something about Harry is changing its 503 to a 500 after it's returned and logged?

Here's Alice's code for that third call, if it's likely to help, but nothing's jumping out at me; I have been wondering if RestClient or Net::HTTP has some configuration that I don't know about.

http_verb = :post
args = [ # actually constructed differently, but this is a reasonable mock up
  'https://api.harry/path?sso_token=token',
  '',
  {
    'content_type' => 'application/json',
    'accept' => '*/*',
    # some other app-specific headers
  }
]

RestClient.send(http_verb, *args) do |response, request, result, &block|
  # `result` is present and has a 500 code at the start of this block if Harry returns a 503.
  @status_code = result.present? ? result.code : :internal_server_error
  cors.merge!( response.headers.slice(:access_control_allow_origin, :access_control_request_method) )
  @body = response.body
end
PJSCopeland
  • 2,818
  • 1
  • 26
  • 40
  • 1
    If you try `RestClient::Request.execute` and then rescue `RestClient::Exception` see what the code is on the error. Its likely something RestClient is doing, and not Net::HTTP - from experience we've seen RestClient do some kind of weird stuff dealing with errors. – Justin Oct 03 '19 at 21:00
  • Except, as I said – "Using `Net::HTTP` directly gives the same result." – PJSCopeland Oct 03 '19 at 22:06
  • 1
    I tried `Net::HTTP` against this mock response http://www.mocky.io/v2/5d97a0563b00001200c31785 and am getting 503 correctly: `Net::HTTP.start("www.mocky.io", 80) { |http| p http.get("/v2/5d97a0563b00001200c31785") }`. Can you try, too? I doubt that the problem is in the library. – Matouš Borák Oct 04 '19 at 19:54
  • @BoraMa that returns a 503, yes. It did seem somewhat unlikely to me as well, but I'm a bit stumped as to what else might be causing it. – PJSCopeland Oct 06 '19 at 23:46
  • @PJSCopeland I'd probably try to use Wireshark (or a similar tool) to capture the traffic and find out whether the correct status is returned from the server. At least you'd narrow the problem down a bit. You might need this while capturing: https://www.comparitech.com/net-admin/decrypt-ssl-with-wireshark/ – Matouš Borák Oct 07 '19 at 08:47

1 Answers1

0

Turns out it's something way simpler and more obvious than anyone thought.

A separate error was raised in the middleware responsible for returning the 503. As with any other exception, this got rendered as a 500.

The thing that was causing it was a line that was supposed to tell the client to wait five seconds and try again:

response.headers['Retry-After'] = 5

... some middleware component was complaining of undefined method 'each' on 5:Fixnum, because it was expecting an Array where it wasn't a String; it started working when we changed 5 to '5'.

PJSCopeland
  • 2,818
  • 1
  • 26
  • 40