I'm looking to use the Pub/Sub emulator with TestPipeline to run end to end and integration tests for dataflow.
Here's my following set up:
- Run emulator: gcloud beta emulators pubsub start
- In the test, create topics and publish a message to it. host address and port is 127.0.0.1:8085
I manually set my hostPort in the managedChannel:
channel = ManagedChannelBuilder.forTarget("127.0.0.1:8085").usePlaintext().build();
channelProvider = FixedTransportChannelProvider.create(GrpcTransportChannel.create(channel));
This is specified here: https://cloud.google.com/pubsub/docs/emulator#pubsub-emulator-java. However, this isn't the problematic part. I'm able to push and pull to Pub/Sub emulator just fine by following the instructions in the link I juust mentioned. The problem is when I try and use Dataflow's PubSubIO to read messages from the given subscription.
- In my streaming pipeline, set the option setPubsubRootUrl to the above address and port (have also tried "http://localhost:8085" and "http://127.0.0.1:8085")
pipeline.options.setPubsubRootUrl("http://127.0.0.1:8085");
pipeline
.apply(
PubsubIO.readMessagesWithAttributesAndMessageId().fromSubscription(subscription)
);
I've confirmed the subscription is correct, and the message is published to the right topic in Pub/Sub emulator.
When I publish my test message to the topic, I get:
[pubsub] INFO: Adding handler(s) to newly registered Channel.
[pubsub] Jul 01, 2020 12:45:38 PM io.gapi.emulators.netty.HttpVersionRoutingHandler channelRead
[pubsub] INFO: Detected HTTP/2 connection
But I get the following repeatedly upon trying to use PubSubIO to read the messages in my dataflow pipeline:
[pubsub] INFO: Adding handler(s) to newly registered Channel.
[pubsub] Jul 01, 2020 12:45:46 PM io.gapi.emulators.netty.HttpVersionRoutingHandler channelRead
[pubsub] INFO: Detected non-HTTP/2 connection.
I've looked online for this, and it seems like the steps I've taken should have solved the problem already (e.g. Local Pubsub Emulator won't work with Dataflow)
Is there a setting that I'm perhaps missing? Can't seem to find any additional information online