I am loading a JSON file with 6.5 GB size which is stored in S3 bucket and loading it into Open search using AWS Lambda. I tried to increase the timeout limit to 15 minutes but still the data doesn't get processed.
Is there another way to do it ?
Asked
Active
Viewed 763 times
0
-
A 6.5GB Json file? That's the problem. Json files shouldn't be more than a few MB. A few KB is normal. – Jeremy Thompson May 09 '22 at 04:27
-
This JSON file, does it contain one big JSON document, or does it contain lots of smaller documents? – JoeNyland Jun 14 '22 at 09:53
-
I am doing something similar. What has worked so far is that I have to pre-process the JSON before sending it in. What I did was use the _bulk endpoint, then before sending JSON, you have to format it like the documentation states. The format is something called NDJSON, but there is an example in the intro: https://opensearch.org/docs/latest/opensearch/index-data/ Since I have a rather large json file, I am doing this through a Node JS process and you have to send data as binary I think too. In curl use -d, in Node JS at least we use Buffer.from. – bearsworth Jun 22 '22 at 02:17