My spark job accessing AWS dynamodb, under high load shows this error:
ERROR: Failed to refresh endpoint : RequestCanceled: request context canceled
caused by: context deadline exceeded
Any idea about this error?
Asked
Active
Viewed 251 times
1
Maria Ines Parnisari
- 16,584
- 9
- 85
- 130
Sowanth
- 51
- 4
-
Very likely due to throttling on the DynamoDB side, have you checked DDB metrics? – Leeroy Hannigan Jan 26 '23 at 14:01
-
Thanks for your response @LeeHannigan. Yes, I monitored the DDB metrics. But not having any severe throttling issues based on dynamic WCU/RCU min-max values. We are handling the DynamoDB Throttling exception and doing the auto scaling options too. – Sowanth Jan 31 '23 at 14:38