2

Issue: Our Redshift Java SDK queries return ValidationException: software.amazon.awssdk.services.redshiftdata.model.ValidationException: Cannot process query string larger than 100kB

Details:

  • Our system may generate many SQL SELECTs connected together via SQL UNIONs with sorting rules on UNIONs level - that's reason why we have reached 100kB query size limit.
  • We understand that having SQL query over 100 KB smells by poor queries design and we are going to reduce those queries size.
  • We need to understand better this Redshift infrastructure limit before applying changes and constraints in business logic.
  • I have not met any similar error while executing the same SQL query via standard SQL client connected to Redshift engine.
  • We prefer keep using SDK (instead of JDBC, CLI) due to various reasons.

Question: Is query size 100 KB soft limit constraint configurable within our cluster/SDK or is it hard limit set on Amazon side?

Zbynek R
  • 53
  • 9

2 Answers2

2

Redshift's query size limit is 16MB - https://docs.aws.amazon.com/redshift/latest/dg/c_redshift-sql.html

I think you need to focus on the SDK.

Bill Weiner
  • 8,835
  • 2
  • 7
  • 18
1

Finally there is constraint regarding Redshift Data API - The maximum query statement size is 100 KB.

Source: https://docs.aws.amazon.com/redshift/latest/mgmt/data-api.html#data-api-calling-considerations

Zbynek R
  • 53
  • 9
  • How did you overcome this limitation? I am trying to perform an update that is big and 100 KB seems really small to me. – pitchblack408 Jul 25 '23 at 17:21