Remote end closed connection without response for a python post request in apache airflow job

Question

I am doing a python service call in the apache airflow job, the POST call is straight forward, but after 5 minutes , it fails with this error "Remote end closed connection without response".

res = requests.post(url, verify=False, auth=token) //The server responds in 5 to 10 minutes after it process few items

Note : The same call works fine using the "CURL" command. Received response after 8 minutes consistently

I am trying to understand who is closing the connection? Is it Client side, Server side or Apache airflow? As CURL command works fine, the server side seems to be fine. Do I need to change my client side python request call timeout etc or Apache airflow side changes?

score 3 · Answer 1 · answered Aug 16 '22 at 00:45

I made it work using TCPKeepAliveAdapter. Here is the code now,

import requests
from requests_toolbelt.adapters.socket_options import TCPKeepAliveAdapter

session = requests.Session()
keep_alive = TCPKeepAliveAdapter(idle=120, count=120, interval=60)//We are telling server that am alive.After 2 minutes it starts pinging server 120 times with 1 minute interval.
session.mount("https://", keep_alive)
res = session.post(url, verify=False, auth=token))

Remote end closed connection without response for a python post request in apache airflow job

1 Answers1