1

We have streaming pipeline running in Google Dataflow. It pulls Pub/Sub message and saves into BigQuery. For some reason, in last few day we have backlog. System lag shows 9-15 hours. I follow document here, and added following params:

--autoscaling_algorithm=THROUGHPUT_BASED
--max_num_workers=5

However, it does not seems spawning new pipeline. We have pull Pub/Sub.

Vadim Kotov
  • 8,084
  • 8
  • 48
  • 62
Krishna Sunuwar
  • 2,915
  • 16
  • 24

0 Answers0