Druid Tuning Configuration

Question

iam beginner using druid and kafka.

I want to create interactive data realtime kafka-druid. I still confuse what tuning should i change of this configuration?

Thank in advance

Answer 1

It depends:

• on your size of kafka's messages
• on your producer's capacity and the rate of messages produced to Kafka
• on your Kafka's servers, partitioning and lots of other factors.
• on your druid's deployment model (single or cluster)
• and ....

But the most important thing I see missing here is your task count, which means the amount of parallel processing (and since yours is Kafka, it means parallel consumers). Increase it and make sure your Druid host (or your middle manager host if it is a cluster) has the adequate cores for your tasks. And make sure in your middle-manager you have increased the total number of available tasks: druid.worker.capacity