Pulsar Consumer is not consuming fast enough

Question

I have a pulsar client and consumer code as below.

PulsarClient client = PulsarClient.builder()
            .serviceUrl(pulsarServerUrl).enableTlsHostnameVerification(false)
            .listenerThreads(1).ioThreads(1)
            .build();

    BatchReceivePolicy batchReceivePolicy
                    = BatchReceivePolicy.builder().maxNumBytes(10 *
            1024 * 1024).maxNumMessages(100000).timeout(-1, TimeUnit.SECONDS).build();
    consumer = client.newConsumer()
            .topic(topicName)
            .subscriptionType(SubscriptionType.Shared)
            .subscriptionName(subscriptionName)
            .subscriptionInitialPosition(SubscriptionInitialPosition.Latest)
            .batchReceivePolicy(batchReceivePolicy)

            .messageListener((consumer1, msg) -> {
                LOG.info("Message Received from Pulsar : " + new String(msg.getData()));
               
                consumer1.acknowledgeAsync(msg);
            })
            .subscribe();

But with this code, there is a huge backlog in pulsar topic. Its around 12 million. My question is how to make pulsar consumer consume more faster? So that the backlog will be 0. BTW, I have tried batch receive as well, but no luck.

Have you tried to use non persistent topic, and shared subscription as described in documentation ? — Victor Gubin, Aug 10 '22 at 19:14
@VictorGubin Yes, I did.But it's showing same behavior. The messages are piling up in the topic. — Aranya Das, Aug 12 '22 at 13:45

score 0 · Accepted Answer · answered Aug 19 '22 at 05:45

0

Which version are you using?

If you are using Java Client 2.8.3, 2.9.1, or 2.10.0 It might be related to this one https://github.com/apache/pulsar/pull/15162 This is a performance regression and fixed in 2.10.1 and 2.9.3. For the 2.8.x version, 2.8.4 is in the release process, it will be available in 2 ~ 3 weeks.

If not the above problem. It's better to provide more context about how high throughput of the topic.

answered Aug 19 '22 at 05:45

Penghui Li

101
2

We are using 2.10.0 . I will update to 2.10.1 and update you with performance. – Aranya Das Aug 22 '22 at 16:50

Pulsar Consumer is not consuming fast enough

1 Answers1