3

I'm trying to implement a service which consumes a google pubsub subscription at its own pace. By that, I mean I need fine control on when I need to consume messages i.e get a batch of messages, pause for a while, do not get more than X messages...

Using google client libraries I did not find a way to do it as the MessageReceiver is running in its own thread and I don't have any control on what exactly happens.

Basically, being able to consume messages in a synchronous way should solve my issue.

Do you know how I can use the google client libs synchronously ? Or is there another way in the API I missed ?

benjamin.d
  • 2,801
  • 3
  • 23
  • 35

1 Answers1

6

You might try using setFlowControlSettings when you build your subscriber. In particular, you can use setMaxOutstandingElementCount or setMaxOutstandingRequestBytes to limit the messages sent to your MessageReceiver. When you have enough messages outstanding, i.e., messages for which you have not called Ack() or Nack(), to exceed these limits, then your MessageReceiver will not be called until messages have been acked or nacked.

Kamal Aboul-Hosn
  • 15,111
  • 1
  • 34
  • 46
  • I agree with you, and I tried it earlier today, but that doesn't solve all my issues. For instance, How do I pause the retrieval of messages without doing stopAsync? I also need to do stuff like I need at max X messages in a timeframe of Y seconds maximum. – benjamin.d Jun 28 '17 at 22:23
  • If you need that more exacting level of control, then you'll need to use the gRPC or REST libraries directly. The [service APIs overview](https://cloud.google.com/pubsub/docs/reference/service_apis_overview) has some more info. – Kamal Aboul-Hosn Jun 28 '17 at 22:41
  • That's the route I wanted to avoid, but I guess I'm out of options. – benjamin.d Jun 28 '17 at 22:43
  • This is only per client, right? It wouldn't share that limit across subscriptions on multiple clients? Do you know if that's possible? – JohanLejdung Oct 09 '19 at 11:15
  • The limit is not shared across clients, no, they are placed per client. It is not possible to share the limit across multiple subscriptions on multiple clients. – Kamal Aboul-Hosn Oct 09 '19 at 14:17
  • You can look into using Dataflow and use windowing and then eventually send it to pubsub. – pkp9999 Dec 02 '21 at 15:56