5

I'm trying to setup my development environment. Instead of using google cloud pubsub in production, I've been using the pubsub emulator for development and testing. To achieve this I set the following environment variable:

export PUBSUB_EMULATOR_HOST=localhost:8586

This worked for the python google pubsub library but when I switched to using java apache beam for google dataflow, the pipeline still points to production google pubsub. Is there a setting, environment variable or method on the pipeline that I need to set so that the pipeline reads for the local pubsub emulator?

Touko
  • 11,359
  • 16
  • 75
  • 105
rezn8
  • 151
  • 1
  • 8

2 Answers2

10

I found the solution in the PubsubOptions interface, and extending it for my own PipelineOptions implementation. Then using setPubsubRootUrl() to set it to localhost:port of the emulator.

rezn8
  • 151
  • 1
  • 8
  • 1
    Ah, It was so simple. stupid me. PubsubOptions options = PipelineOptionsFactory.fromArgs(args).withValidation() .as(PubsubOptions.class); options.setPubsubRootUrl("http://localhost:8085"); – Pari Jul 18 '18 at 10:39
1

You can also specify this option in a configuration file

pubsubRootUrl=http://localhost:8185

or via program argument (haven't tested but should also work)

-Dexec.args="--pubsubRootUrl=http://localhost:8185"

the http:// is required to conform to Java URL format

namp10010
  • 123
  • 1
  • 10