4

I need to find out a way to smoothly manage server stalls when I have been reading data from it. I wrote the piece of code below:

def listener():    
    resp = requests.get(someurl, stream=True)
    if resp.status_code == 200:
        for line in resp.iter_lines():            
            if line:
                do_something_with_the_line
                print(result)

price_thread = threading.Thread(target=listener, name="StreamingThread", args=[])
trade_thread.start()

The code works well until a stall of the server occur (the API provider suggests a stall occur when no "lines" are received in 10 seconds).

How can I implement this in my code? In other words I would try to recall the listener method without exiting the price_thread thread when a stall occur.

Thanks in advance

AntoG
  • 833
  • 1
  • 10
  • 16
  • Does the API server break the underlying socket in the event of a timeout (i.e. no data is sent for 10 seconds+) or you should break it off and reestablish a new connection? – zwer Apr 06 '18 at 14:14
  • From the documentation: _"No data has been received from the stream for more than 10 seconds. **Re-connection**: There is a re-connection rate limit in place that is enforced. Clients whose re-connection attempts exceed this limit will receive HTTP 429 error responses._". I would say I have to reestablish the connection. – AntoG Apr 06 '18 at 14:24

1 Answers1

5

You can use the timeout property to ensure that your connection breaks off after no data is received in the specified amount of time, and then respawn the connection:

def listener():
    while True:
        try:
            resp = requests.get(someurl, stream=True, timeout=10)
            if resp.status_code == 200:
                for line in resp.iter_lines():
                    if line:
                        # do_something_with_the_line
                        pass
            elif resp.status_code == 429:
                    print("Too many reconnects, exiting.")
                    break
            else:
                print("Unhandled status `{}` retreived, exiting.".format(resp.status_code))
                break
        except requests.exceptions.Timeout:
            pass  # we'll ignore timeout errors and reconnect
        except requests.exceptions.RequestException as e:
            print("Request exception `{}`, exiting".format(e))
            break

You may also add a reconnect counter instead of having the while True loop indefinitely.

zwer
  • 24,943
  • 3
  • 48
  • 66
  • Thanks. With _# we'll ignore timeout errors and reconnect_ you mean calling back the `requests.get` method or call back `resp.iter_lines()`? – AntoG Apr 06 '18 at 15:20
  • @AntoG - at that point the connection will be left for scrapping, a new one will be spawned in the next `while True` iteration. – zwer Apr 06 '18 at 15:22
  • you are right, silly question. I'll give a try and let you know. – AntoG Apr 06 '18 at 15:25