0

I am in a real pain with Elastic App Search, the indexing limit for bulk index is 100 according to the docs: https://www.elastic.co/guide/en/app-search/current/limits.html

I was trying to create all the promises and then do promise.all(allPromises), but it's failing to index everything, and the response of when this fails still return 200, and you have to loop over: res.data (all the 100 documents array), and look if they have error field.

Is there any solution to index lot of document fast? Because indexing 1 million with loop to await between every 100 batch size query is extremely slow.

eliezra236
  • 547
  • 1
  • 4
  • 16

1 Answers1

0

Unfortunately the limit of 100 makes it a slow operation. We are indexing 1.1 million documents and our solution was to slice that our into ten segments and run ten processes in parallel to decrease the time that it takes.

Since reading from the index is very quick we have a separate job that validates the information in App Search matches our source data and flags anything out of order. So I don't check for errors on a full import or update so that part goes as fast as possible. We only see around a 1% failure rate on bulk imports.

I should note that part of the reason for the second piece is we have on occasion found the search index can get out of alignment with the documents and fields we have, so validation seemed like a good idea.

Jeremy Rusnak
  • 1