issue while loading data in cassandra using dsbulk

Question

I’m facing issue while loading data into table from .csv file using dsbulk. I get like below in the errorlog.

Caused by: com.datastax.driver.core.exceptions.OperationTimedOutException: [/10.0.126.13:9042] Timed out waiting for server response

This environment is our POC environment of 3 nodes with 8 CPUs and 64G memory. And as per my observation when I run dsbulk command it eats up all the CPUs on the server and memory consumption goes high too.

If you can give me pointer to fine tune dsbulk by which cpu usage/memory consumption can be reduced. If this operation slows down and if I get manageable performance im ok with it.

score 3 · Answer 1 · answered Mar 22 '19 at 07:59

3

You can specify the --executor.maxPerSecond option to limit the number of operations per second. See the documentation for DSBulk.

Also you can try to tune the batching options, like, --batch.maxBatchStatements.

And it's also recommended to run DSBulk from a separate machine to prevent it influence the DSE's performance. (that's common advice for all load testing, etc.)

answered Mar 22 '19 at 07:59

Alex Ott

80,552
8
87
132

im trying to use --batch.maxBatchStatements but it says invalid option and errors out. Also if i trying to run this command from other machine what should be the ipaddress under -h ? – v parkar Mar 22 '19 at 19:40
What version of DSBulk? – Alex Ott Mar 22 '19 at 19:42
latest version is 1.3.3: http://downloads.datastax.com/dsbulk/dsbulk-1.3.3.tar.gz – Alex Ott Mar 23 '19 at 19:29

score 0 · Answer 2 · answered Mar 23 '19 at 20:51

0

thank you all for help I was able to resolve this issue by downloading latest version of debulk and setting batch size to 5000.

answered Mar 23 '19 at 20:51

v parkar

39
2

issue while loading data in cassandra using dsbulk

2 Answers2