Are ordering guarantees in Google pubsub enforced upon on acknowledgement or upon arrival?

Question

I have a subscription with ordering enabled and I am setting an ordering key when enqueuing messages on a topic. In practice I see very little evidence of subscriber affinity in the logs. This does not technically violate the specification as subscriber affinity on an ordering key is best effort, but not guaranteed.

What I do however need to guarantee is strict ordering, to avoid lock contention in my database when persisting data read from an ordered subscription. There is a one-to-one correlation between ordering keys and database locks. For a given ordering key I do not want to read the next message until the previous one has been explicitly acknowledged, regardless of which subscriber consumes the next message. This way despite having an ordered subscription shared by many subscribers, and messages with the same ordering key consumed by different subscribers, I will still avoid lock contention in the database, which is currently a very real problem in production.

(I should note that in my case I cannot use message attributes for this, because pubsub does not allow dynamic filtering, and the number of nodes with subscribers is dynamically autoscaled and not deterministic. Therefore mapping a message attribute to a subscriber via modulo division is not an option)

I know that I am not the first person to use ordering keys for this purpose: Subscriber affinity with Ordering Key, but without *in-order* delivery

In practice I found subscriber affinity to be rather loose. I can live with this as long as I know that for a given ordering key a new message will not be read, until the previous message has been acknowledged, regardless of which subscriber reads the next message. Is this guaranteed by pubsub???

If "message1" and "message2" are enqueued in that order and have the same ordering key, on an ordered subscription, "message2" cannot be read until "message1" has been read, but can be "message2" be read before "message1" has been acknowledged???

Answer 1

The ordering guarantee for Cloud Pub/Sub does not guarantee that an acknowledgement has to be received in order for the next message to be delivered. When using the Cloud Pub/Sub client libraries, the guarantee is that the next message for an ordering key will not be delivered to the user-specified callback until the callback completes for the previous message. If you always acknowledge your messages within the callback, then this effectively means that you are guaranteeing the next message will not be delivered until the previous is acked. However, if you acknowledge asynchronously, the next message could be delivered to your callback without the previous one having been acknowledged.

In general, these deliveries will happen to the same subscriber. It should not be possible for a different subscriber to receive a subsequent message if a previous one has not been acknowledged. However, keep in mind that Cloud Pub/Sub has at-least-once delivery semantics and ack deadlines are managed with best effort. Therefore, if you have "message1" and "message2" for the same ordering key and "subscriberA" and "subscriberB" to receive them, it is possible that "subscriberA" could receive "message1" and then "subscriberB" could also receive "message1" before "subscriberA" has acknowledged it. If you need tighter guarantees around ack deadlines, then you should use the exactly once feature in conjunction with message ordering.

The gcloud tool is not going to be a good way to test affinity. Affinity only applies when using streaming pull, which gcloud does not. When using gcloud, any pull request is going to get messages for any ordering key, as there is no identifiable information for each request to distinguish it as coming from a particular subscriber.

Note that the blog post announcing ordering keys states specifically:

Affinity: If there are messages with an ordering key outstanding to a streaming pull subscriber, then additional messages that are delivered are sent to that same subscriber. If no messages are currently outstanding for an ordering key, the service delivers messages to the last subscriber to receive messages for that key on a best-effort basis.

The official Pub/Sub client libraries use streaming pull and should therefore have more observable affinity.