2

I'm looking to use the Pub/Sub emulator with TestPipeline to run end to end and integration tests for dataflow.

Here's my following set up:

  1. Run emulator: gcloud beta emulators pubsub start
  2. In the test, create topics and publish a message to it. host address and port is 127.0.0.1:8085

I manually set my hostPort in the managedChannel:

channel = ManagedChannelBuilder.forTarget("127.0.0.1:8085").usePlaintext().build();
channelProvider = FixedTransportChannelProvider.create(GrpcTransportChannel.create(channel));

This is specified here: https://cloud.google.com/pubsub/docs/emulator#pubsub-emulator-java. However, this isn't the problematic part. I'm able to push and pull to Pub/Sub emulator just fine by following the instructions in the link I juust mentioned. The problem is when I try and use Dataflow's PubSubIO to read messages from the given subscription.

  1. In my streaming pipeline, set the option setPubsubRootUrl to the above address and port (have also tried "http://localhost:8085" and "http://127.0.0.1:8085")
pipeline.options.setPubsubRootUrl("http://127.0.0.1:8085");
pipeline
            .apply(
                PubsubIO.readMessagesWithAttributesAndMessageId().fromSubscription(subscription)
            );

I've confirmed the subscription is correct, and the message is published to the right topic in Pub/Sub emulator.

When I publish my test message to the topic, I get:

[pubsub] INFO: Adding handler(s) to newly registered Channel.
[pubsub] Jul 01, 2020 12:45:38 PM io.gapi.emulators.netty.HttpVersionRoutingHandler channelRead
[pubsub] INFO: Detected HTTP/2 connection

But I get the following repeatedly upon trying to use PubSubIO to read the messages in my dataflow pipeline:

[pubsub] INFO: Adding handler(s) to newly registered Channel.
[pubsub] Jul 01, 2020 12:45:46 PM io.gapi.emulators.netty.HttpVersionRoutingHandler channelRead
[pubsub] INFO: Detected non-HTTP/2 connection.

I've looked online for this, and it seems like the steps I've taken should have solved the problem already (e.g. Local Pubsub Emulator won't work with Dataflow)

Is there a setting that I'm perhaps missing? Can't seem to find any additional information online

user2738206
  • 388
  • 2
  • 4
  • 14

0 Answers0