Many factors contribute the overall performance of the Clients (Producer/Consumer) connected to a KAFKA Broker. First of all, I am not sure how you are running your consumer instances, whether 4 instances running on 4 separate servers or 4 instances through any IDE tool for loading test per se. You can better clarify here. Also, how is your consumer implementation look like. Is it just reading from the topic and writing it into a console or doing full blown business functionality connected to any of the backend systems. Kindly confirm.
Default Partitioner:
If a key exists and the default partitioner is used, Kafka will hash the key and use
the result to map the message to a specific partition. The mapping of keys to partitions is consistent only as long as the number of partitions in a topic does not change.
You can change this behaviour implementing a Customer Partitioner
Dynamic consumers:
You can't increase the consumers dynamically based on the throughput, unless you have a multi-threaded consumers implemented. You can read more about Java Executor Service ref: https://dzone.com/articles/kafka-consumer-and-multi-threading. Your consumer implementation must be having something as follows. So you should have a counter of number of records polled, and if it is more than the threshold you are after then you can instantiate the ExecutorService to add up more instances.
private List executors = new ArrayList() ;
@Override
public void run(String... args) throws Exception {
Runtime.getRuntime().addShutdownHook(new Thread() {
@Override
public void run() {
executors.forEach(exe -> {
exe.shutdown();
try {
if (!exe.awaitTermination(10000, TimeUnit.MILLISECONDS)) {
exe.shutdownNow();
}
} catch (InterruptedException e) {
exe.shutdownNow();
}
int instances = <<number of instances>>;
ExecutorService executor = Executors.newFixedThreadPool(instances);
for (int i=0; i < instances; i++) {
executor.execute(<<Consumer Implemenation class>>);
executors.add(executor);
}
}