Batch size in Storm Trident

Question

I would like to know how to set BATCH SIZE (example I want a batch to have 10000 records) in TRIDENT. I have the below configuration in my code and this fetches approximately 250 records per batch from Kafka. Can I increase this to 10000*1024? or is there a way to set No of records per batch.

TridentKafkaConfig tridentKafkaConfig = new TridentKafkaConfig(hosts, topic, consumerGroupId);
tridentKafkaConfig.scheme = new SchemeAsMultiScheme(new XScheme()); 
tridentKafkaConfig.fetchSizeBytes = 1024*1024;

Answer 1

You can increase the batch size by changing "tridentKafkaConfig.fetchSizeBytes" property. Also, batch size is related with number of brokers and number of partitions. For example, if you have 2 brokers and 3 partitions for each broker that means the total count of partition is 6. By this way, the batch size equals to tridentKafkaConfig.fetchSizeBytes X total partition count. if we assume that the tridentKafkaConfig.fetchSizeBytes is 1024X1024, the batch size equals to 6 MB.(3x2x1024x1024)bytes