3

I've been trying to do some searching (Google, Slack, Stack) and have yet to find an answer. We have some applications that are written using Spring Cloud Streams and were interested in swapping the back end from Kafka to Pulsar. Spring currently doesn't have any native support for Kafka, however it appears that pulsar provides the ability to use the Kafka API to communicate with pulsar directly (https://pulsar.apache.org/docs/en/adaptors-kafka).

I'm wondering if anybody has gone down the route of trying to use this replacement for the Kafka-clients library in the context of Spring cloud messaging.

An alternative valid approach, of course, is to just re-write the code - but I wanted to turn to the community to see if anybody has gone down this road as of yet.

Thanks

Jeef
  • 26,861
  • 21
  • 78
  • 156
  • Spring Cloud Stream currently does not provide a binder implementation for Apache Pulsar. The binder API is pretty open so that you can build a binder for Pulsar if need be. The integration with kafka API's sound interesting, haven't explored that either. – sobychacko Nov 22 '19 at 17:34
  • The wrapper looks very limited; the kafka-clients version mentioned on that page (0.10.2.1) is extremely old. In any case, since they don't support the consumer method `void subscribe(Collection topics, ConsumerRebalanceListener callback)` (as well as many others), it looks like a non-starter to me. – Gary Russell Nov 22 '19 at 18:07
  • 1
    These are already supported in master and will be released soon in 2.5 – Matteo Merli Nov 24 '19 at 05:34

1 Answers1

1

While not exactly what you're asking for, I've tried to integrate Pulsar with Spring-Kafka using the shaded library org.apache.pulsar:pulsar-client-kafka:2.5.0 and it blows-up with this stack trace:

Caused by: java.lang.UnsupportedOperationException: null
at org.apache.kafka.clients.consumer.KafkaConsumer.partitionsFor(KafkaConsumer.java:650) ~[pulsar-client-kafka-2.5.0.jar:2.5.0]
at org.springframework.kafka.listener.AbstractMessageListenerContainer.checkTopics(AbstractMessageListenerContainer.java:312) ~[spring-kafka-2.2.10.RELEASE.jar:2.2.10.RELEASE]
at org.springframework.kafka.listener.ConcurrentMessageListenerContainer.doStart(ConcurrentMessageListenerContainer.java:136) ~[spring-kafka-2.2.10.RELEASE.jar:2.2.10.RELEASE]
at org.springframework.kafka.listener.AbstractMessageListenerContainer.start(AbstractMessageListenerContainer.java:292) ~[spring-kafka-2.2.10.RELEASE.jar:2.2.10.RELEASE]
at org.springframework.kafka.config.KafkaListenerEndpointRegistry.startIfNecessary(KafkaListenerEndpointRegistry.java:311) ~[spring-kafka-2.2.10.RELEASE.jar:2.2.10.RELEASE]
at org.springframework.kafka.config.KafkaListenerEndpointRegistry.start(KafkaListenerEndpointRegistry.java:255) ~[spring-kafka-2.2.10.RELEASE.jar:2.2.10.RELEASE]
at org.springframework.context.support.DefaultLifecycleProcessor.doStart(DefaultLifecycleProcessor.java:182) ~[spring-context-5.1.10.RELEASE.jar:5.1.10.RELEASE]

This specific error can be bypassed (for these specific library versions) using a custom implementation of AbstractMessageListenerContainer.checkTopics but you'll likely run into more issues.

acarroll
  • 31
  • 4