Consuming messages from Google Pubsub and publishing it to Kafka

Question

I am trying to consume Google PubSub messages using synchronous PULL API. This is available in Apache Beam Google PubSub IO connector library. I want to write the consumed messages to Kafka using KafkaIO. I want to use FlinkRunner to execute the job, since we run this application outside GCP.

The problem I am facing is that the consumed messages are not getting ACK'd in GCP PubSub. I have confirmed that the local Kafka instance has the messages consumed from GCP PubSub. The documentation in GCP DataFlow indicates that the data bundle gets finalized when the pipeline is terminated with a data sink, which is Kafka in my case.

But since code is running in Apache Flink and not GCP DataFlow, I think some sort of callback is not getting fired related to ACK'ing the committed message.
What am I doing wrong here?

                   pipeline
                    .apply("Read  GCP PubSub Messages", PubsubIO.readStrings()
                            .fromSubscription(subscription)
                    )
                    .apply(ParseJsons.of(User.class))
                    .setCoder(SerializableCoder.of(User.class))
                    .apply("Filter-1", ParDo.of(new FilterTextFn()))
                    .apply(AsJsons.of(User.class).withMapper(new ObjectMapper()))
                    .apply("Write to Local Kafka",
                            KafkaIO.<Void,String>write()
                                    .withBootstrapServers("127.0.0.1:9092,127.0.0.1:9093,127.0.0.1:9094")
                                    .withTopic("test-topic")
                                    .withValueSerializer((StringSerializer.class))
                                    .values()
                    );

in the google pubsub console, it shows a chart for unacked messages for that subscription. — ChaitanyaBhatt, Nov 09 '20 at 16:28
yes. We use the Google GCP pubsub Java client to consume data using synchronous Pull API which has built-in polling mechanism to consume data in batches of 1000 messages per request and then these messages are processed sequentially through the builder pipeline. — ChaitanyaBhatt, Nov 09 '20 at 20:02
if there is an explicit flag somewhere to differential batch/pull API.. I am unaware of it. — ChaitanyaBhatt, Nov 09 '20 at 20:03

score 0 · Answer 1 · answered Nov 10 '20 at 21:34

0

In the Beam documentation on the PubSub IO class it's mentioned this:

Checkpoints are used both to ACK received messages back to Pubsub (so that they may be retired on the Pubsub end), and to NACK already consumed messages should a checkpoint need to be restored (so that Pubsub will resend those messages promptly).

The ACK are not linked to Dataflow, you should have the same behavior on dataflow. The ack are sent on Checkpoints. Usually the Checkpoints are the windows that you set on your stream flow.

But, you didn't set window! By default, the windows is global, and it closed only at the end, if you stop gracefully your job (and even, I'm not sure about this). Anyway, a better solution is to have fixed windows (for example of 5 minutes) to ack the messages on each of these windows.

answered Nov 10 '20 at 21:34

guillaume blaquiere

66,369
2
47
76

Thank you!! I will try this and update this thread. I am very grateful to you! – ChaitanyaBhatt Nov 11 '20 at 01:43
This solution did not work for me. I did add this to the pipeline `.apply( "FixedWindowsLabel", Window.into(FixedWindows.of(Duration.standardSeconds(5L))))`... but did not help – ChaitanyaBhatt Nov 11 '20 at 23:32
What do you mean? The pipeline works but the messages aren't ack, right? – guillaume blaquiere Nov 12 '20 at 09:02
yes. pipeline works but NO ACKs. Note that ACKs work when I use DirectRunner (the embedded default runner).. somehow, ACKs don't work in FlinkRunner – ChaitanyaBhatt Nov 12 '20 at 17:27
It's strange, maybe a bug. You can try to open an issue in the [Beam JIRA](https://issues.apache.org/jira/projects/BEAM/issues/BEAM-11227?filter=allopenissues) – guillaume blaquiere Nov 12 '20 at 21:03

score 0 · Accepted Answer · answered Nov 13 '20 at 04:35

The way I fixed this solution was by using Guillaume Blaquiere's (https://stackoverflow.com/users/11372593/guillaume-blaquiere) suggestion of looking at Checkpoints. Even after adding the Window.into() function in the pipeline, the source PubSub subscription endpoint did not receive ACKs.
The problem was in the Flink server configuration I had failed to mention checkpoint configuration. Without these parameters, checkpoints are disabled.

state.backend: rocksdb
state.checkpoints.dir: file:///tmp/flink-1.9.3/state/checkpoints/

These configs should go in the flink_home/conf/flink-conf.yaml. After adding these entries and restarting flink. All the backlogged (unack'd messages) went to 0 in the GCP pubsub monitoring chart.

Consuming messages from Google Pubsub and publishing it to Kafka

2 Answers2