0

I'm trying to copy indexes between AWS Opensearch(Elastic engine v6.8) to AWS Opensearch (Opensearch engine v2.3).

I'm using elasticdump to copy the indexes.

I got the foloowing error while in one of the indexes:

Error Emitted => {"Message":"Request size exceeded 104857600 bytes"}

dump ended with error (get phase) => REQUEST_ENTITY_TOO_LARGE: {"Message":"Request size exceeded 104857600 bytes"}

I read that in AWS Opensearch it's limit to 100MB per wrrting to the Opensearch.

I tried to reduce the limit to 10 (using --limit) and still failed at the same point. Only when I set the limit to 1 it's works.

the question if someone can help me to understand if there is any parameter that I missing or I need a combination of parameters to get it works.

Thanks!!

SomeOneDudOrg
  • 11
  • 1
  • It's weird because elasticdump usually handles pretty well the read/write sequencing, I've been using it to replicate multi-TB indexes without any issues. Do you have really large documents in your index? – Val Feb 13 '23 at 11:59
  • Hey @val yes - I also have multi indexes that total amount is 20TB... edit: the issue is that the AWS is limiting the write to 100MB per operation/batch and the default of --limit is 100 docs per operation, the elasticdump is not split the batch if it's more than 100MB – SomeOneDudOrg Feb 13 '23 at 12:24
  • How big is your biggest document? – Val Feb 13 '23 at 13:04

0 Answers0