How to limit througput on an Apache Beam pileline in Go?

Question

I wrote a basic pipeline in Go running on Google Dataflow.

Basically it transforms Pubsub events to elastic documents and then update Elastic document in bulk.

I need to find a way to limit the number of Bulk request per second. Because when my pubsub subscription accumulated a lot of messages and my Dataflow streaming job wants to "catch up", it's literally killing my Elastic cluster.

How would you do it?

I have never user the Go SDK. In Python you would use something like this [RPC batching](https://beam.apache.org/documentation/programming-guide/#batching-rpcs), however, it seems that in Go [timers](https://beam.apache.org/documentation/programming-guide/#timers) have [not been implemented](https://github.com/apache/beam/issues/22737) yet. — CaptainNabla, Apr 25 '23 at 07:44

score 0 · Answer 1 · answered May 01 '23 at 22:20

You can use the Throttle transform. This transform limits the number of elements that pass through it per unit of time. You can specify the number of elements and the duration of the time window. Here is an example:

import "github.com/apache/beam/sdks/go/pkg/beam"

func ThrottlePipeline(s beam.Scope, input beam.PCollection) beam.PCollection 
{
    return beam.ParDo(s, func(_ []byte, emit func(interface{})) {
        emit(1)
    }, input).
        Apply(s, beam.WindowInto(window.NewFixedWindows(time.Second))).
        Apply(s, beam.CombinePerKey(sum.SumInt64)).
        Apply(s, beam.ParDo(func(_ []byte, count int64) {
            time.Sleep(time.Duration(count) * time.Second)
        }, beam.Impulse(s)))
}

You can adjust the duration of the time window to change the throughput rate.

How to limit througput on an Apache Beam pileline in Go?

1 Answers1