How to set topic name in KafkaListener by code?

Question

I am writing the listener for kafka topic and using @kafkaListener annotation to do the same. Till now i hard-coded the topic name (topic1) and it was working fine.Here was a working code :-

@Component
public class KafkaConsumer {

@Autowired
private KafkaProperties kafkaProps;


@Autowired
@Qualifier("CustomObjectMapper")
private ObjectMapper objectMapper;

@KafkaListener(topics = "topic1", containerFactory = "createPokafkaListenerContainerFactory")
public void CreatePoListener(PurchaseOrder po, Acknowledgment ack)
    throws JsonProcessingException {

    LOG.info("Received message for po create from kafka topic {} is {}",
        kafkaProps.getOmsCreateTopicName(), objectMapper
            .writerWithDefaultPrettyPrinter().writeValueAsString(po));

    ack.acknowledge();


}

}

Now, when i am trying to change the code to get the topic name from code, it is not working. Code which i tried to get the value from code after following this page in stack-overflow (How to pass dynamic topic name to @kafkalistener(topics from environment variable) is :-

@Component
public class KafkaConsumer {

@Autowired
private KafkaProperties kafkaProps;

public KafkaProperties getKafkaProps() {
    return kafkaProps;
}

@Autowired
@Qualifier("CustomObjectMapper")
private ObjectMapper objectMapper;

@KafkaListener(topics = "#{__listener.kafkaProps.getOmsCreateTopicName()}", containerFactory = "omsCreatePokafkaListenerContainerFactory")
    public void CreatePoListener(PurchaseOrder po, Acknowledgment ack)
    throws JsonProcessingException {

    LOG.info("Received message for po create from kafka topic {} is {}",
        kafkaProps.getOmsCreateTopicName(), objectMapper
            .writerWithDefaultPrettyPrinter().writeValueAsString(po));

    ack.acknowledge();


}

}

When i am trying to bring the server, it is throwing the error :-

Initialization of bean failed; nested exception is org.springframework.beans.factory.BeanExpressionException: Expression parsing failed; nested exception is org.springframework.expression.spel.SpelEvaluationException: EL1008E: Property or field '__listener' cannot be found on object of type 'org.springframework.beans.factory.config.BeanExpressionContext' - maybe not public?

can someone help me here what i am doing wrong here ?

What version are you using? That feature was added in 2.1.2. — Gary Russell, Dec 03 '19 at 14:53
@GaryRussell if you are asking about spring-kafka, we are using 1.1.3.RELEASE. let me increase the version and try it again — Neer1009, Dec 03 '19 at 15:05
Yes I mean spring-kafka; the current release is 2.3.3. 1.1.x has not been supported for a long time. — Gary Russell, Dec 03 '19 at 15:36
@GaryRussell Since this is little old project and we are using spring 4.3.x version. And i guess 2.1.2 is supported by spring 5. I am getting no class def found error due to compatibility issue between Spring 4 and Spring-kafka-2.1.x. Is there any way i can achieve this in spring 4 keeping spring-kafka 1.1.x ? — Neer1009, Dec 03 '19 at 16:42

score 2 · Accepted Answer · answered Dec 03 '19 at 16:52

2

1.1.x is no longer supported; if you can't upgrade beyond Spring Framework 4.3, you should upgrade to 1.3.10, which is the latest version for use with Spring 4.3; it has a much simpler and reliable threading model than 1.1.x, thanks to KIP-62.

While you can't use __listener with 1.3.x, you can use a SpEL expression to get the topic directly from the KafkaProperties bean:

@KafkaListener(topics = "#{kafkaProps.omsCreateTopicName}" ...)

answered Dec 03 '19 at 16:52

Gary Russell

166,535
14
146
179

Thanks Gary. I made the suggested changes (upgrading 1.1.x to 1.3.10) and use SpEl as described above, but i am getting this error :- "Initialization of bean failed; nested exception is org.springframework.beans.factory.BeanExpressionException: Expression parsing failed; nested exception is org.springframework.expression.spel.SpelEvaluationException: EL1008E: Property or field 'kafkaProps' cannot be found on object of type 'org.springframework.beans.factory.config.BeanExpressionContext' - maybe not public?" – Neer1009 Dec 04 '19 at 04:54
Given that you have `@Autowired private KafkaProperties kafkaProps;` in your question, I assumed you have such a bean in your application context. If it has a different bean name, you must use the proper bean name in the expression. – Gary Russell Dec 04 '19 at 04:59
I changed the variable name from kafkaProps to kafkaProperties and it started working. Thanks a lot for your help. – Neer1009 Dec 05 '19 at 07:12

score 0 · Answer 2 · edited Sep 03 '20 at 13:09

0

In Consumer class, you need to make below changes :

@Autowired
public KafkaProperties kafkaProps;

@KafkaListener(topics = "#{kafkaConsumer.kafkaProps.getOmsCreateTopicName()}"

It worked for me.

edited Sep 03 '20 at 13:09

Arghya Sadhu

41,002
9
78
107

answered Sep 03 '20 at 11:52

Himi-Hash

31
3

How to set topic name in KafkaListener by code?

2 Answers2