Scaling Google Cloud Pull subscribers

Question

I'm considering Google Cloud Pub/Sub and trying to determine whether I will go for Pull or Push subscriber model.

Notably the pull model is able to handle larger throughput

• Large volume of messages (many more than 1/second)

• Efficiency and throughput of message processing is critical.

However, the push model can sit behind an HTTP load balancer, and is therefore able to auto-scale subscriber nodes during times when the number of queued messages exceeds the capacity of a single subscriber node.

The pull model is also more secure, because it does not require exposing sensitive operations in an HTTP endpoint.

The issue is how can we scale subscriber nodes in the pull model? Is there something in GCP for this kind of situation?

Answer 1

There are several options for auto scaling via a pull subscriber:

• Use GKE and to set up Autoscaling Deployments with Cloud Monitoring Metrics. This allows you to scale the number of instances based on the num_undelivered_messages metric.
• Use a GCE managed instance group and scale based the num_undelivered_messages metric.
• Use Dataflow to process messages from Pub/Sub and set up autoscaling.

Answer 2

When you pull a subscription, you need to be connected to the subscription. For the lowest latency, being connected full time is required. So, you can do this with compute engine.

Your compute pulls your queue and consume the messages. In case of huge amount of message, the compute engine resources (CPU and Memory) will increase. You can put this compute into a manage instance group (MIG) and set scalability threshold, like the quantity of CPU use.

Of course, the message pulling is more efficient in term of network bandwidth and protocol handshake. However, it requires a compute full time up and the scalability velocity is slow.

If you consider the Push subscription, of course, the HTTPS protocol consume much more bandwidth and is not very efficient, but you can push the message to Cloud Run or Cloud Functions. The scalability is very elastic and based on the traffic (number of messages pushed) and not on the CPU usage.

In addition, you can push pubsub message securely to CLoud Functions and Cloud Run by using the correct identification in your pubsub subscription