0

I have created a Lambda function that uses awswrangler data api to read in data from an RDS Serverless Aurora PostgreSQL Database from a query. The query contains a conditional that is a list of IDs. If the query has less then 1K ids it works great, if over 1K I get this message:

Maximum BadRequestException retries reached for query

An example query is:

"""select * from serverlessDB where column_name in %s""" % ids_list

I adjusted the serverless RDS instance to force scaling as well as increased the concurrency on the lambda function. Is there a way fix this issue?

John Rotenstein
  • 241,921
  • 22
  • 380
  • 470
Prof. Falken
  • 499
  • 6
  • 21
  • Is this the MySQL or PostgreSQL version of Serverless Aurora? Are you certain about the 1000 items threshold? e.g. 999 works but 1000+ doesn't? – Noel Llevares Nov 01 '21 at 19:07
  • @NoelLlevares PostgreSQL (I will edit the question)and my tests actually contained 980 ids and 1150 ids. – Prof. Falken Nov 01 '21 at 19:13

1 Answers1

0

With PostgreSQL, 1000+ items in a WHERE IN clause should be just fine.

I believe you are running into a Data API limit.

There isn't a fixed upper limit on the number of parameter sets. However, the maximum size of the HTTP request submitted through the Data API is 4 MiB. If the request exceeds this limit, the Data API returns an error and doesn't process the request. This 4 MiB limit includes the size of the HTTP headers and the JSON notation in the request. Thus, the number of parameter sets that you can include depends on a combination of factors, such as the size of the SQL statement and the size of each parameter set.

The response size limit is 1 MiB. If the call returns more than 1 MiB of response data, the call is terminated.

The maximum number of requests per second is 1,000./

Either your request exceeds 4 MB or the response of your query exceeds 1 MB. I suggest you split your query into multiple smaller queries.

Reference: https://docs.aws.amazon.com/AmazonRDS/latest/AuroraUserGuide/data-api.html

Noel Llevares
  • 15,018
  • 3
  • 57
  • 81
  • Thanks for pointing this out in the docs Noel. I will need to set up a test to know the true size of the request/response but given the error I am definitely exceeding it. My other option would be to not go serverless. To be clear though, this is an aurora serverless issue and not a aws wrangler issue correct? – Prof. Falken Nov 02 '21 at 13:52