0

Can we use both partitioning and consumer group for same topic. So we want to create a topic which have partitioning and then create multiple consumers to it and out of them 2-3 will be generic listening to all messages(needs to be in a consumer group so that message in not processed multiple times) and then one consumer for specific partion.

Is partitioning and consumer are mutually exclusive?

Abhishek bhutra
  • 1,400
  • 1
  • 11
  • 29

2 Answers2

1

With the high level consumer API you can´t pin a consumer instance to a particular partition but there is nothing preventing you from having a set of consumers using the high level API and another set of consumers using the simple API for the same topic.

With this you could have a simple consumer consuming from a specific partition and a set of high level consumers in a consumer group consuming messages across all partitions.

Lundahl
  • 6,434
  • 1
  • 35
  • 37
1

High Level Consumer for kafka 0.8.x doesn't allow you to specify partition. It reads the data from all partitions and does complex failure detection and rebalancing. Probably it will be supported in future versions according to the API redesign - https://cwiki.apache.org/confluence/display/KAFKA/Consumer+Client+Re-Design#ConsumerClientRe-Design-Allowmanualpartitionassignment.

If you need to read from specific topic and partition - use SimpleConsumer(it requires manual leader/offset/exclusivity/error handling).

Or you can use high level consumer and filter the data by partition(if you accept the overhead).

Also as an option you can redesign your topics to write data to separate topic instead of 'specific partition'.

Anatoly Deyneka
  • 1,238
  • 8
  • 13
  • Hi, thanks, thats my understanding. I wanted to confirm though. If you accept overhead - you mean that consume all messages in all consumers and discard if it should not be there? – Abhishek bhutra Aug 17 '15 at 13:38
  • right, 'consume all messages in all partitions and discard if it should not be there' – Anatoly Deyneka Aug 18 '15 at 08:20