2

Is it possible to use topic created in pub sub emulator as data source with direct runner in apache beam python?

I've set the environment variables(within the script as well):

set PUBSUB_EMULATOR_HOST=localhost:8085
set PUBSUB_PROJECT_ID=<proj_id>

It doesn't seem to be connecting to it because even if I set the emulator host to anything else it gives the same error and if not set, it looks for credentials.

Tried reading messages using topic and subscription id(pull).

Using input topic :

File "lib\site-packages\apache_beam\runners\direct\transform_evaluator.py", line 646, in get_subscription topic_name = sub_client.topic_path(project, short_topic_name) AttributeError: 'SubscriberClient' object has no attribute 'topic_path'

Using subscription id(pull) :

lib\site-packages\apache_beam\runners\direct\transform_evaluator.py", line 694, in _read_from_pubsub sub_client.api.transport.channel.close() AttributeError: 'SubscriberClient' object has no attribute 'transport'

For java, use of emulator has been mentioned in the docs(setPubsubRootUrl(String)) but couldn't find anything for python.

Touko
  • 11,359
  • 16
  • 75
  • 105
Riya Francis
  • 96
  • 9

1 Answers1

0

You have to set the environment variables on the command:

dev_appserver.py app.yaml --env_var PUBSUB_EMULATOR_HOST=${PUBSUB_EMULATOR_HOST}
Nathan Nasser
  • 1,008
  • 7
  • 18
  • i have set the environment variables in command line also tried after adding it in the python script.I'm not facing any issue with using the emulator for creating topics, subscribing or publishing messages directly.I'm facing the issue when using apache beam with the emulator. – Riya Francis Oct 09 '20 at 09:38
  • I recommend to open a issue in the Public Issue Tracker as this needs more investigation – Nathan Nasser Oct 13 '20 at 19:59