Multithreading with pulsar message listeners

Question

I'm fairly new to java message listeners and apache pulsar. Assume that I've maintained a listener like so,

private MessageListener<byte[]> generateListener() {
        MessageListener<byte[]> listener = (consumer, respMsg) -> {
            String respM = new String(respMsg.getValue(), StandardCharsets.UTF_8);
            logger.info(respM);
            consumer.acknowledgeAsync(respMsg);
        };
        return listener;
    }

And a Consumer instance like so,

Consumer<byte[]> c = consumerBuilder.messageListener(generateListener()).topic(topicName).subscriptionName("Consumer-" + i).subscribeAsync().get();

What I would like to know is how multiple incoming messages would be handled by this listener? Will each message be handled in a seperate thread as in the case of JMS listeners? If so, then how can I configure the number of threads to use - is it by using the ClientBuilder.listenerThreads() property?

Is there a need to maintain multiple listener objects respective to each consumer, when maintaining multiple consumers i.e, something like this -

consumerBuilder.clone().messageListener(generateListener()).topic(topicName).subscriptionName("Consumer-" + i).subscribeAsync() ?

I asked a similar question a while ago on Stack - check here: https://stackoverflow.com/questions/67171146/apache-pulsar-async-consumer-setup-completable-future — user521990, Jun 26 '21 at 07:16

score 1 · Accepted Answer · answered Nov 04 '20 at 21:03

The ClientBuilder#listenerThreads method allows configuring the size of an internal thread-pool that will be shared and used across all Consumers or Readers that will be created from that client.

Pulsar Client will give you the guarantee that a MessageListener for a single consumer will always be invoked by the same thread, i.e that the provided MessageListener don't need to be thread-safe. So, YES it is prefered to use a dedicated MessageListener object per Consumer.

Note that this also ensure ordering.

So basically, if you only use a single Consumer then you can keep the listenerThreads to 1 (that is the default).

Here is a complete example that can used to observe the behaviour :

public class PulsarConsumerListenerExample {

    public static void main(String[] args) throws PulsarClientException {

        int numListenerThread = 2;

        PulsarClient client = PulsarClient
                .builder()
                .serviceUrl("pulsar://localhost:6650")
                .listenerThreads(numListenerThread)
                .build();

        final List<Consumer<?>> consumers = new ArrayList<>();
        for (int i = 0; i < numListenerThread; i++) {
            consumers.add(createConsumerWithLister(client, "my-topic", "my-subscription", "C" + i));
        }

        Runtime.getRuntime().addShutdownHook(new Thread(() -> {
            for (Consumer<?> consumer : consumers) {
                try {
                    consumer.close();
                } catch (PulsarClientException e) {
                    e.printStackTrace();
                }
            }
        }));
    }

    private static Consumer<String> createConsumerWithLister(final PulsarClient client,
                                                             final String topic,
                                                             final String subscription,
                                                             final String consumerName) throws PulsarClientException {
        return client.newConsumer(Schema.STRING)
            .topic(topic)
            .consumerName(consumerName)
            .subscriptionName(subscription)
            .subscriptionMode(SubscriptionMode.Durable)
            .subscriptionType(SubscriptionType.Failover)
            .subscriptionInitialPosition(SubscriptionInitialPosition.Earliest)
            .messageListener((MessageListener<String>) (consumer, msg) -> {
                System.out.printf(
                    "[%s/%s]Message received: key=%s, value=%s, topic=%s, id=%s%n",
                    consumerName,
                    Thread.currentThread().getName(),
                    msg.getKey(),
                    msg.getValue(),
                    msg.getTopicName(),
                    msg.getMessageId().toString());
                consumer.acknowledgeAsync(msg);
            })
            .subscribe();
    }
}

Thanks for the response. In that case, to consume huge messages with good throughput would it be better to maintain multiple consumers with shared subscription on the same topic? But wouldn't creating multiples consumers create unnecessary overhead? So is there no option to asynchronously consume message with multiple threads? — Ivak, Nov 05 '20 at 12:46
Regarding the fact to have multiple consumers maybe this question can help you: https://stackoverflow.com/questions/64603078/what-is-the-cost-of-having-multiple-subscriptions-in-pulsar. The type of subscription depends on your need. If you are OK to not have strong ordering then you can use Shared or KeyShared subscription with multiple consumers. — fhussonnois, Nov 05 '20 at 13:26

Multithreading with pulsar message listeners

1 Answers1