3

I keep getting '[Errno 101] Connection timed out' when I tried to call an API using Python on Airflow that provided by Google Cloud Composer.

Here's my code:

r1 = requests.post(
    url1,
    params=request1_params,
    headers=request1_headers
)

a_token = r1.json().get('access_token')

And the response when I tried to trigger DAG is this:

[2021-10-04 01:21:33,056] {taskinstance.py:1152} ERROR - HTTPSConnectionPool(host='xxx.com', port=443): Max retries exceeded with url: /sf/xxx/token?grant_type=password&username=???%40???&password=???%2???&client_id=????&client_secret=???? (Caused by NewConnectionError('<urllib3.connection.HTTPSConnection object at 0x7f43a2958860>: Failed to establish a new connection: [Errno 110] Connection timed out',)) Traceback (most recent call last): File "/opt/python3.6/lib/python3.6/site-packages/urllib3/connection.py", line 170, in _new_conn (self._dns_host, self.port), self.timeout, **extra_kw File "/opt/python3.6/lib/python3.6/site-packages/urllib3/util/connection.py", line 96, in create_connection raise err File "/opt/python3.6/lib/python3.6/site-packages/urllib3/util/connection.py", line 86, in create_connection sock.connect(sa) TimeoutError: [Errno 110] Connection timed out

I've read some articles that you need to include proxy when sending requests, but what proxy should I used?

Any help is appreciated!

edit: I've found out that the problem is on the airflow that I am using. It says that private network need NAT configured. So, after installed few libraries and setup NAT, it now can send requests to API.

Thank you for all the help.

Yohandy
  • 41
  • 4
  • If I were in your spot, I'd try and make a connection to the target server outside of Airflow/Composer and validate that the underlying request is working. If it is not, then it is not an airflow issue. – Kolban Oct 04 '21 at 04:44
  • I did try to call the function and it worked. So, the only possible reason is because of the airflow. Although, I haven't tried it on local machine but reading some implementation from the Internet, it should be working. – Yohandy Oct 04 '21 at 06:03
  • What exactly do you want to achieve? Are you running this from GCP UI or from your local machine? How are you authenticating yourself? This is your whole code? Please run the dag again with the following changes and post the output using [this snippet](https://pastebin.com/d06u3MBW) – PjoterS Oct 04 '21 at 10:57

1 Answers1

1

Most probably this has nothing to do with proxy but rather a malformed request. This looks like a part of an OAuth flow, but any particular reason for not using operator? E.g. https://airflow.apache.org/docs/apache-airflow-providers-http/stable/operators.html#simplehttpoperator , you can then set the the timeout via extra_options as those are from requests lib https://github.com/apache/airflow/blob/main/airflow/providers/http/operators/http.py#L58

redvg
  • 56
  • 4