0

I'm doing a benchmark with crate and insert a lot of records at the same time. It seems like I hit some limit (queue capacity 50) and I didn't find how to change the configuration.

Exception in thread Thread-1:

Traceback (most recent call last): File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/threading.py", line 810, in __bootstrap_inner self.run() File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/threading.py", line 763, in run self.__target(*self.__args, **self.__kwargs) File "createdata.py", line 60, in worker cursor.execute(ins, params) File "/Users/jodok/sandbox/crate-demo/amsterdam/pyenv/lib/python2.7/site-packages/crate/client/cursor.py", line 48, in execute self._result = self.connection.client.sql(sql, parameters) File "/Users/jodok/sandbox/crate-demo/amsterdam/pyenv/lib/python2.7/site-packages/crate/client/http.py", line 190, in sql content = self._json_request('POST', self.sql_path, data=data) File "/Users/jodok/sandbox/crate-demo/amsterdam/pyenv/lib/python2.7/site-packages/crate/client/http.py", line 345, in _json_request self._raise_for_status(response) File "/Users/jodok/sandbox/crate-demo/amsterdam/pyenv/lib/python2.7/site-packages/crate/client/http.py", line 331, in _raise_for_status raise ProgrammingError(error.get('message', '')) ProgrammingError: SQLActionException[RemoteTransportException[[nuc2][inet[/192.168.42.72:4300]][bulk/shard]]; nested: EsRejectedExecutionException[rejected execution (queue capacity 50) on org.elasticsearch.action.support.replication.TransportShardReplicationOperationAction$AsyncShardOperationAction$1@23c7247f]; ]

gtonic
  • 2,295
  • 1
  • 24
  • 32
Jodok Batlogg
  • 373
  • 1
  • 9

1 Answers1

0

bulk inserts are using the bulk threadpool, so add this to your crate.yml configuration file to change its size:

threadpool.bulk.queue_size: 100

but with the current master this shouldn't be needed anymore, because crate is retrying the current bulk request on such queue size rejecting issues now.

Sebastian Utz
  • 719
  • 3
  • 9