1

I have a streaming pipeline in Apache Beam 2.5 that subscribes to a PubSub subscription, parses CSV files received as messages from that subscription, applies some trivial processing to the data, and then stores the results in BigQuery.

Occasionally, the producer of the data sent to the PubSub topic changes the CSV file format (columns are added/removed/renamed) without telling us. When this happens the CSV parsing DoFn (luckily) starts failing and the Pipeline gets stuck retrying processing the element (the pipeline's system lag starts increasing monotonically).

Google's documentation promotes the use of the Drain functionality as the "nice" way to stop a Pipeline, outlining that in this way "in-flight" elements won't risk being lost. Does this mean that all in-flight elements, even the failing ones, will be "committed" when their bundle closes, thus triggering an ACK to PubSub?

In my case, I'd like the failing element NOT to be ACK'ed, so that after the pipeline is fixed, the failing element will be processed when redelivered.

user180940
  • 324
  • 1
  • 18

2 Answers2

1

Another answerer linked a question with a popular (though not yet accepted) answer that states that Dataflow will ACK the message once, for the bundle the message belongs to, "results of the bundle (outputs and state mutations etc) have been durably committed" (At what stage does Dataflow/Apache Beam ack a pub/sub message?).

It's important to note that Dataflow needs to commit state when there are stateful operations in your pipeline. For example, with windowing, Dataflow needs to stash your data somewhere while it waits for the window to pass, at which point it pulls the state back out and sends it off to the next part of your pipeline.

This behavior actually matches what I've observed using Dataflow in production for a few years now. We used to have a stateless pipeline (no windowing, etc) and it NACKed messages when exceptions occurred in any part of the pipeline. When we added windowing, we noticed it ACKing the messages even though the window the message belonged to has not yet passed (and nothing had been output at the end of the pipeline into the sink).

Therefore, the situation you're concerned about, where messages are ACKed even though the message is "bad" will occur, in pipelines that have stateful operations, because the message won't be deemed "bad" by your code until after it has been ACKed so that it can be durably committed. The situation won't occur, and you can safely rely on a NACK for these "bad" messages, if your pipeline has no stateful operations (and all stateless operations finish within the ACK deadline you've configured for your Pub/Sub subscription).

If this is a problem for you, because you have stateful operations in your pipeline, I'd suggest one of two things:

  • Add validation before the Pub/Sub message is published, such that no "bad" messages will enter your pipeline, or
  • Break up your pipeline into two pipelines, one stateless and one stateful, such that messages will only be deemed "bad" in the first pipeline, and can be retried later when the pipeline is updated to no longer deem the message "bad" or the message is discarded through other means if it isn't needed
Matt Welke
  • 1,441
  • 1
  • 15
  • 40
0

According to some related discussions[1], acks only happen when a bundle succeeds. In you case, bundle already fails, which means it won't succeed before drained, I think we don't expect acks.

[1] At what stage does Dataflow/Apache Beam ack a pub/sub message?

Ruoyun Huang
  • 173
  • 10
  • I think the concern is that Dataflow will ACK a message as soon as "results of the bundle (outputs and state mutations etc) have been durably committed" according to that discussion, and note that Dataflow needs to commit state in order to do things like windowing. So the concern would be that Dataflow presumably can't skip ACKing messages (or NACK them) if the operation that fails occurs later in a pipeline than a stateful operation like windowing. – Matt Welke Apr 05 '21 at 22:10