0

I'm trying the following example from the google official web site.

import java.io.IOException;
import org.apache.beam.examples.common.WriteOneFilePerWindow;
import org.apache.beam.sdk.Pipeline;
import org.apache.beam.sdk.io.gcp.pubsub.PubsubIO;
import org.apache.beam.sdk.options.Default;
import org.apache.beam.sdk.options.Description;
import org.apache.beam.sdk.options.PipelineOptions;
import org.apache.beam.sdk.options.PipelineOptionsFactory;
import org.apache.beam.sdk.options.StreamingOptions;
import org.apache.beam.sdk.options.Validation.Required;
import org.apache.beam.sdk.transforms.windowing.FixedWindows;
import org.apache.beam.sdk.transforms.windowing.Window;
import org.joda.time.Duration;

public class PubSubToGcs {
  /*
   * Define your own configuration options. Add your own arguments to be processed
   * by the command-line parser, and specify default values for them.
   */
  public interface PubSubToGcsOptions extends PipelineOptions, StreamingOptions {
    @Description("The Cloud Pub/Sub topic to read from.")
    @Required
    String getInputTopic();

    void setInputTopic(String value);

    @Description("Output file's window size in number of minutes.")
    @Default.Integer(1)
    Integer getWindowSize();

    void setWindowSize(Integer value);

    @Description("Path of the output file including its filename prefix.")
    @Required
    String getOutput();

    void setOutput(String value);
  }

  public static void main(String[] args) throws IOException {
    // The maximum number of shards when writing output.
    int numShards = 1;

    PubSubToGcsOptions options =
        PipelineOptionsFactory.fromArgs(args).withValidation().as(PubSubToGcsOptions.class);

    options.setStreaming(true);

    Pipeline pipeline = Pipeline.create(options);

    pipeline
        // 1) Read string messages from a Pub/Sub topic.
        .apply("Read PubSub Messages", PubsubIO.readStrings().fromTopic(options.getInputTopic()))
        // 2) Group the messages into fixed-sized minute intervals.
        .apply(Window.into(FixedWindows.of(Duration.standardMinutes(options.getWindowSize()))))
        // 3) Write one file to GCS for every window of messages.
        .apply("Write Files to GCS", new WriteOneFilePerWindow(options.getOutput(), numShards));

    // Execute the pipeline and wait until it finishes running.
    pipeline.run().waitUntilFinish();
  }
}

This example works in DataFlowRunner works without any issues where as when I tried to run the same example in the local runner it doesn't work. The messages are read from the pubsub queue, but the data is not written the to the gcs file. I'm using apache beam 2.31.0 version.

Balasubramanian Naagarajan
  • 338
  • 3
  • 17

1 Answers1

0

I was able to reproduce this case, without issues. I think you should be careful with these points:

  • For local runs you have to use DirectRunner.
  • Make sure your topic have messages. You have to wait some minutes for your messages to fill your topic and you can see it on the topic.
  • Check the alive time of your messages, I set to 1 day for my test.
  • Check your subscription messages, please be patience. (It takes 3-8 mins)
  • Check the paths you are using on the main run command. I use the following parameters:
mvn compile exec:java \
  -Dexec.mainClass=com.examples.pubsub.streaming.PubSubToGcs \
  -Dexec.cleanupDaemonThreads=false \
  -Dexec.args=" \
    --project=project-id \
    --region=us-central1 \
    --inputTopic=projects/project-id/topics/test-topic \
    --output=gs://project-bucket-id/data \
    --gcpTempLocation=gs://project-bucket-id/tmp \
    --runner=DirectRunner \
    --windowSize=2"

For a fully list of available runners please check this link.

Betjens
  • 1,353
  • 2
  • 4
  • 13