0

While trying to index around 20000 records (each record has around 100kb size) using python's bulk API, I saw that max write queue size consumed is 6. So are my following understanding about bulk indexing correct (with default configuration of 500 chunks and 100 mb chunk size)?

  1. Bulk API will send 40 requests (20000/500) to ES.
  2. Each bulk request is persisted atomically i.e all 500 chunks or none.
  3. If active thread is busy, then bulk request will be pooled in write queue as one whole object.
Divakar Patil
  • 323
  • 2
  • 11

1 Answers1

1
  1. That's correct
  2. No, each bulk operation is independent, it's definitely possible that some go through and some fail. You need to make sure that your client code handles such cases and retries failed bulk operations.
  3. All bulk operations are handled by the write thread pool. If several bulk operations arrive at the same time, then some of them will be handled (depending on the number of cores on the node receiving the bulk operation) and the ones that can't be handled will be added to the queue, whose size is 10000 (which means that 10000 indexing operations can be queued before being processed). Once that queue is filled, indexing operations will start to be rejected (HTTP 429)
Val
  • 207,596
  • 13
  • 358
  • 360
  • For 3rd point, so if we send 500 chunks in a bulk request, will 500 operations be added in the queue or only one? – Divakar Patil Mar 22 '21 at 09:16
  • 1
    Only one operation gets added, then each single operation will be dispatched to the right shards on the right nodes. – Val Mar 22 '21 at 10:32