0

I am unable to authenticate my Dataflow Beam application when I run it in Intellij Idea. This worked for me at one point recently and now it doesn't.

Auth is failing with 403 forbidden '"Access Denied: Project [myProject]: User does not have bigquery.jobs.create permission in project [myProject].'

  1. I have verified that I DO have this permission in both my gcp user and service account.
  2. I have set GOOGLE_APPLICATION_CREDENTIALS with the path to a service account json in my MacOS Zshell profile.
  3. This same profile configuration works when I run a different client lib Node JS app in VSCode using the same service account token.
  4. This same java Dataflow pipeline authenticates when I run/debug it in Eclipse IDE.
  5. Running mvn package from terminal on the same pipeline is also authenticating and writing the template to my GCS storage bucket.
  6. I have added additional service accounts to my gcloud configuration with 'gcloud auth activate-service-account' and can see them listed with 'gcloud auth list'
  7. I have tried setting the active account to both service accounts.
  8. I have tried setting the --serviceAccount Beam option to a service account I know has correct permissions.
  9. I would like to try to setting the service token path to the BigQueryIO java Dataflow connector as I am able to do with the Node JS client lib, but it doesn't seem possible?
  10. The debugger does work and I can hit a break point.
  11. I have tried installing latest and Version: 2020.1.1 Build: 201.7223.91 29 April 2020
  12. I have tried uninstalling and reinstalling Intellij and creating a new project.

It appears as though this security context is not getting passed to the Dataflow Java Beam library, but the exception output does say 'Inferred default GCP project 'fubotv-prod' from gcloud.' so apparently some args are getting through.

Perhaps there is some cached response build state or something?

I spent all day stuck on this. I am at my wits end. I would really like to debug my Dataflow pipeline again with Intellij. Any solutions, ideas, random words of encouragement are much appreciated!

  • Are you using the same version of the libraries in every try? I mean, are using the exact same configuration and library versions when running the pipeline through Maven, Eclipse and Intellij? – rmesteves May 14 '20 at 15:09
  • It's the same project built from the same main with the same pom. But the way that Intellij builds/rujs the application seems different. I'm really not even sure how it works. Perhaps the key is in the run configuration? I don't fully understand what Intellij does to build and run a java project... –  May 14 '20 at 15:58
  • I should point out related to this - when I run mvn package this does not actually run the application. This simply builds the template and related jars that are later run on the Dataflow platform as a job. But I know this still requires that google credentials are made available. They are different credentials - GCS write vs. BQ job creation - but these are coming from the same service account. My Node JS client lib still needs the same BQ job create creds though. –  May 14 '20 at 16:06
  • Can you take a look in the logs to see if the user who is trying to access BigQuery is the correct one? ( https://cloud.google.com/logging/docs/audit#viewing_audit_logs) – rmesteves May 18 '20 at 16:12
  • Another question: have you tried to set the service account from your code directly? If you dont, have you tried accessing the GOOGLE_APPLICATION_CREDENTIALS variable from your code and printing it to see if its correctly populated? – rmesteves May 18 '20 at 16:15
  • Again, this is not a problem if I package with mvn. In this case the GOOGLE_APPLICATION_CREDENTIALS are picked up. Also, not a problem when use the Node JS client lib and debug the same application with Eclipse - seems to be something specific to Intellij debug? –  May 19 '20 at 18:46
  • I'm not sure how I could further isolate? Is there is a different debug run configuration I could try? –  May 19 '20 at 18:47
  • Thanks for the ideas though! –  May 19 '20 at 19:14
  • My best guess now is something with idea_rt.jar –  May 19 '20 at 19:15
  • Ug. So mystery solved and it's just plain dumb. I have only recently started using multiple service accounts. I have been switching between service tokens mapped to GOOGLE_APPLICATION_CREDENTIALS. When I do so, I have been using source to reload my profile to reflect the change. I would then printenv to see the change reflected in terminal. However, Intellij must have it's own shell context that is separate from the IDE terminal window, a flaw imo, and so that was not being updated. On Mac you have to explicitly restart an application and so the shell context was never getting updated. –  May 19 '20 at 21:48
  • Thats what I meant when I asked you to print the env var from your code. So it seems that in fact IntelliJ has its own shell context – rmesteves May 20 '20 at 09:29
  • If your problem was solved, consider posting it as an answer or let me know if I can do that – rmesteves May 20 '20 at 09:35

1 Answers1

0

As discussed in the comments, a way of debugging what is happening can be printing the var env in your code in IntelliJ in order to see if the environment variables are correct

As concluded in the discussion, IntelliJ has its own SHELL context which makes the OS's variables not accessible if you don't explicitly restart the application.

rmesteves
  • 3,870
  • 7
  • 23
  • Ah, that finally just clicked - to clarify - I did run printenv in the Intellij terminal window and this did not help me identify the issue because it printed the GOOGLE_APPLICATION_CREDENTIALS associated with that runtime context. In contrast, running the command from my code would have shown the value set in the actual debug runtime context. I just misunderstood what you were suggesting. So yes this would have been the answer - looks like I could have done something along the lines of - Process process = Runtime.getRuntime() .exec(String.format("sh -c printenv %s", homeDirectory)); –  May 28 '20 at 14:14