2

I am using the requests module to query a relatively large PostgRES database (~14 GB), accessible via a PostgREST API, hosted on a simple server set up by a peer.

When I make simple API calls, such as

requests.get('https://api.nycdb.info/real_property_legals?bbl=eq.4043210050')

I get a 200 response and everything works fine. However, when I do more computationally intensive searches (such as text searches), I get 504 responses the vast majority of the time.

This is an example of an API call that almost always returns 504 time-outs:

requests.get('https://api.nycdb.info/real_property_parties?address1=@@."MADISON"')

I suspect this is because the server is not responding fast enough, due to its simple processing power and the size of the database. But, I look to the community for help here.

Is there a way to increase the amount of time a request waits for a response before a 504 response is provided? What else can I do to try to improve the reliability of more complicated queries?

Andrew Feierman
  • 128
  • 1
  • 10

2 Answers2

2

You can add ,timeout= where seconds is a float. (See pydoc requests.request for available opts which can be supplied directly to pydoc.get().) There may also be limits on the API server side in quick case you may need to bust down your query into bite size chunks.

Paul Doom
  • 146
  • 3
1

If you read the docs for requests, you will see that requests.get does not timeout by default.

In addition, a 504 is an error produced by a gateway server.

So, what is actually happening, is that your PostGREST server is timeing out in its communication with your postgres server.

You cannot stop this by changing your request.get call, you must reconfigure one or both of your servers instead.

(In other words, this is not a python or a requests issue)

djspoulter
  • 41
  • 2