1

So, I am trying to run PubSub Kafka connectors. I have a local Kafka Server and local Zookeeper running. Then, I use the command:

.\bin\windows\connect-standalone.bat .\etc\kafka\WorkerConfig.properties 
.\etc\kafka\configSink.properties .\etc\kafka\configSource.properties

but I get the error:

Error while starting connector CPSConnectorSource
org.apache.kafka.connect.errors.ConnectException: Error verifying the subscription TestSubscription for project kohls-sis-sandbox
    at com.google.pubsub.kafka.source.CloudPubSubSourceConnector.verifySubscription(CloudPubSubSourceConnector.java:201)
    at com.google.pubsub.kafka.source.CloudPubSubSourceConnector.start(CloudPubSubSourceConnector.java:116)
    at org.apache.kafka.connect.runtime.WorkerConnector.doStart(WorkerConnector.java:100)
    at org.apache.kafka.connect.runtime.WorkerConnector.start(WorkerConnector.java:125)
    at org.apache.kafka.connect.runtime.WorkerConnector.transitionTo(WorkerConnector.java:182)
    at org.apache.kafka.connect.runtime.Worker.startConnector(Worker.java:178)
    at org.apache.kafka.connect.runtime.standalone.StandaloneHerder.startConnector(StandaloneHerder.java:250)
    at org.apache.kafka.connect.runtime.standalone.StandaloneHerder.putConnectorConfig(StandaloneHerder.java:164)
    at org.apache.kafka.connect.cli.ConnectStandalone.main(ConnectStandalone.java:94)
Caused by: java.io.IOException: Error reading credential file from environment variable GOOGLE_APPLICATION_CREDENTIALS, value 'C:\GooglePlatform\servicekey.json;': File does not exist.
    at com.google.auth.oauth2.DefaultCredentialsProvider.getDefaultCredentialsUnsynchronized(DefaultCredentialsProvider.java:102)
    at com.google.auth.oauth2.DefaultCredentialsProvider.getDefaultCredentials(DefaultCredentialsProvider.java:67)
    at com.google.auth.oauth2.GoogleCredentials.getApplicationDefault(GoogleCredentials.java:54)
    at com.google.auth.oauth2.GoogleCredentials.getApplicationDefault(GoogleCredentials.java:36)
    at com.google.pubsub.kafka.common.ConnectorUtils.getChannel(ConnectorUtils.java:52)
    at com.google.pubsub.kafka.source.CloudPubSubSourceConnector.verifySubscription(CloudPubSubSourceConnector.java:192)
    ... 8 more
Caused by: java.io.IOException: File does not exist.
    at com.google.auth.oauth2.DefaultCredentialsProvider.getDefaultCredentialsUnsynchronized(DefaultCredentialsProvider.java:94)
    ... 13 more

Complete log file here. Property files here

I have a environment variable GOOGLE_APPLICATION_CREDENTIALS set to C:\GooglePlatform\servicekey.json; and the file is located in the specified location. However, the error message states the file does not exist, so is the environment variable set incorrectly?

Cody Ferguson
  • 393
  • 1
  • 4
  • 20
  • Not sure if it helps, but notice the trailing semicolon within the quotes here: `'C:\GooglePlatform\servicekey.json;'`. – 500 - Internal Server Error Aug 29 '17 at 21:48
  • I just followed the same style that I used to add to the PATH variable. I took out the semicolon which might have fixed it. However, I ran into another error which I asked about here https://stackoverflow.com/questions/45966794/pubsub-kafka-connect-node-connection-end-of-file-exception because I felt that it didn't fit under this questions scope. Once I fix that error, I can confirm if this worked! – Cody Ferguson Aug 30 '17 at 17:50
  • This solution work. I also did not have correct permissions set for my service account. I assigned the service account admin and publisher roles. Now it works correctly! – Cody Ferguson Aug 31 '17 at 16:26

1 Answers1

0

So, as "500 - Internal Server Error" stated above, I had to remove the semicolon in the system variable GOOGLE_APPLICATION_CREDENTIALS. I had not set the correct roles for my service account, either. So, make sure that your service account is assigned the Pub/Sub admin role. I also assigned the Pub/Sub publisher role to my service account, but just the Pub/Sub admin role might work.

I should have read the pre-running steps, here, more carefully.

Cody Ferguson
  • 393
  • 1
  • 4
  • 20