1

I'm monitoring the elasticsearch cluster while bulk importing data.

So when I run curl localhost:9200/_cat/thread_pool?v

it shows:

host            ip           bulk.active bulk.queue bulk.rejected index.active index.queue index.rejected search.active search.queue search.rejected 
xye@OptiPlex  127.0.0.1            8          6             0            0           0              0             0            0               0

I'm current running 5 scripts concurrently to import the data and the bulk size is 500 and I also increased the bulk queue size too 300.

So I'm wondering what bulk.queue 6 means. Is it just 6 documents in one bulk operation or it is actually 6 bulk requests which is 6 * 500 = 3000 documents?

Laurel
  • 5,965
  • 14
  • 31
  • 57
milodky
  • 443
  • 2
  • 7
  • 18

1 Answers1

2

bulk.queue represents the number of bulk requests in the queue and not the number of documents. index.queue would represent the number of documents to be indexed present in the index queue.

bittusarkar
  • 6,247
  • 3
  • 30
  • 50
  • So if I set the bulk queue very large, is it possible to avoid EsRejectedExecutionException? – milodky Jan 19 '16 at 22:46
  • is bulk.rejected also represent # documents or # bulk operations? – Shai M. Mar 26 '17 at 08:35
  • @milodky I know this is very late but setting the bulk queue to a very large value has its own problems. You may run into memory issues trying to keep so many bulk requests in memory at the same time. – bittusarkar May 02 '17 at 16:32
  • @ShaiM. No, bulk.rejected represents the number of bulk request failures. – bittusarkar May 02 '17 at 16:33