0

I'm using Spring Cloud Stream to send and receive messages from Google Pub/Sub topics and have a question regarding publishing. If I have an @Output MessageChanel and I call channel.send(message) and the call returns normally (no Exceptions), am I guaranteed that Pub/Sub now has the message? I'm looking for something akin to "persistent" message guarantees in JMS systems.

I seem to recall that there are batching parameters and/or async settings that can be configured for publishing and I'd also like to know how this all affects the guarentee, if any. These would seem to complicate any guarantees for sending a single message via channel.send.

Thanks.

Dave McCullough
  • 43
  • 2
  • 5

1 Answers1

1

You are correct that Cloud Pub/Sub's Spring integrations by default publish asynchronously. On Spring Integration level, you can control the sync/async switch by using PubSubMessageHandler.setSync(boolean) setter. At that level, it is also possible to set up a publish callback in async mode. Unfortunately, there is no way right now to change this flag from a Spring Cloud Stream integration.

I have a PR out to add the first feature (turning off async) -- https://github.com/spring-cloud/spring-cloud-gcp/pull/2473

Elena Felder
  • 456
  • 3
  • 8
  • Thanks @Elena Felder. – Dave McCullough Jul 22 '20 at 19:48
  • So just to confirm my understanding, message loss is possible when running in aysnc mode, right? Say if a program called channel.send, it returned normally, but then the program died before the async thread got a chance to send the message(s) to pubsub? Two more questions: how much of a performance hit will sync operation impose and any idea when we could see a new build with your PR in it? – Dave McCullough Jul 22 '20 at 20:45
  • Yes, in principle message loss in async mode is possible. To simulate this scenario, start a sample app (e.g. https://github.com/spring-cloud/spring-cloud-gcp/tree/master/spring-cloud-gcp-samples/spring-cloud-gcp-pubsub-binder-sample), turn off internet and test sending a message from the sample UI. The sender code will return successfully, although there will be DNS resolution warnings in the logs. With the new property `spring.cloud.stream.gcp.pubsub.default.producer.sync`, however, the same scenario results in a `java.util.concurrent.TimeoutException` after the default 10 second timeout. – Elena Felder Jul 24 '20 at 02:18
  • `1.2.4.RELEASE` should be coming out in the next two weeks; I am aiming to get the PR in there. – Elena Felder Jul 24 '20 at 02:19
  • So at the risk of being pedantic, if I'm running in sync mode and call channel.send and it returns normally (no Exceptions), then am I guaranteed that PubSub has the message? Also, one more question: when running in async mode, have you seen any silent failures, i.e., where a message fails to be written to PubSub, but no errors/Exceptions show up in the logs? I'm trying to solve some data loss issues in our app and trying to see if it's in the message handling code. – Dave McCullough Aug 22 '20 at 05:30
  • Yes, from the Spring Cloud Stream integration standpoint, if you are explicitly calling channel.send while in synchronous mode and no exception occurs, the message should have reached the Pub/Sub servers. In fact, it's theoretically possible for an exception such as timeout to occur, and for the message to still reach Pub/Sub. See if this example repo helps troubleshoot the scenarios: https://github.com/elefeint/examples/tree/master/pubsub-stream-error-propagation – Elena Felder Aug 25 '20 at 22:20
  • (speculation here) For logging, Pub/Sub client library uses JUL, so if you use logback/slf4j to control levels, you may not be seeing low-level JUL logs. To configure those, use `java.util.logging.config.file` property. You can also use `jul-to-slf4j` library to send JUL to logback. – Elena Felder Aug 25 '20 at 22:24