I have pub/sub topics, a pull subscriptions and a data pipelines that pull messages using python and NodeJS implementation of the GCP pub/sub library.
I have an endless regular streaming workload and ad-hoc workloads. I want to prioritize the ad-hoc workloads immediately when needed.
How do I do that?
Creating an additional pub/sub topic for each pipeline seems like a wrong solution. There must be another way to prioritize ad-hoc workloads.
There is no way to prioritize particular messages. I think the right way is to create a separate topic.
As mentioned in this stackoverflow thread, Pub/Sub does not provide any mechanism to prioritise a message. Moreover this feature would come into picture if the subscribers are not able to keep up with the publishing rate which is resulting in high backlog messages. If subscribers are keeping up and processing and acking messages quickly, then the notion of "priority" messages isn't really necessary.
If a backlog is being built up and some messages needs to be processed with higher priority, then you can consider creating a different "high-priority" topic and subscription. The subscribers subscribe to this subscription as well as the "normal" subscription and prioritize processing messages from the "high-priority" subscription whenever they arrive.
