0

I'm running a flask app that will access Bigquery on behalf of users using a service account they upload.

To store those service account credentials, I thought the following might be a good set up:

ENV Var: Stores my credentials for accessing google secrets manager

Secret & secret version: in google secrets manager for each user of the application. This will access the user's own bigquery instance on behalf of the user.

--

I'm still learning about secrets, but this seemed more appropriate than any way of storing credentials in my own database?

--

The google function for accessing secrets is:

def access_secret_version(secret_id, version_id=version_id):
    # Create the Secret Manager client.
     client = secretmanager.SecretManagerServiceClient()

# Build the resource name of the secret version.
     name = f"projects/{project_id}/secrets/{secret_id}/versions/{version_id}"

# Access the secret version.
    response = client.access_secret_version(name=name)

# Return the decoded payload.
    return response.payload.data.decode('UTF-8')

However, this returns JSON as a string. When then using this for big query:

credentials = access_secret_version(secret_id, version_id=version_id)
BigQuery_client = bigquery.Client(credentials=json.dumps(credentials), 
project=project_id)

I get the error:

 File "/Users/Desktop/application_name/venv/lib/python3.8/site- 
 packages/google/cloud/client/__init__.py", line 167, in __init__
    raise ValueError(_GOOGLE_AUTH_CREDENTIALS_HELP)
ValueError: This library only supports credentials from google-auth-library-python. 
See https://google-auth.readthedocs.io/en/latest/ for help on authentication with 
this library.

Locally I'm storing the credentials and accessing them via a env variable. But as I intend for this application to have multiple users, from different organisations I don't think that scales.

I think my question boils down to two pieces:

  1. Is this a sensible method for storing and accessing credentials?
  2. Can you authenticate to Bigquery using a string rather than a .json file indicated here
Rory Buchanan
  • 21
  • 3
  • Are the users giving you a service account key? If they are running on Google Cloud, you shouldn't need to use service account keys at all... Storing exported service account keys in Secret Manager is generally considered an anti-pattern. – sethvargo Jan 29 '22 at 19:22
  • If your app needs to access a customer's BQ install, then you should run your app as a custom service account and the _customer_ grants _your_ service account permission to access _their_ BQ instance. Then you just call `bigquery.Client()`. – sethvargo Jan 29 '22 at 19:24
  • Thanks, that sounds interesting. I felt I was going down an anti-patter root given the friction between secrets_manager / bigquery. Have you got more docs / examples on this solution? I'm unsure how this works in practice – Rory Buchanan Jan 29 '22 at 19:34
  • I don't know enough about your use case, but this is just how IAM works. As I said above, the customer grants your identity permissions on their BQ instance. Are you using GCE, GKE, Cloud Run? What are you actually _doing_? – sethvargo Jan 29 '22 at 19:50
  • Thanks - I've gone away and had a tinker and this certainly comes down to my lack of knowledge on IAM. Will read more :) – Rory Buchanan Jan 29 '22 at 21:06
  • I've managed to get this working. I was using cloud run to run jobs on the users BQ install, and all I needed to do here was grant the right permissions in that instance to the apps service account. I was looking at similar app's documentation on this and overcomplicated it. for example, looker which asks for the full json file. – Rory Buchanan Jan 29 '22 at 21:08
  • https://cloud.google.com/run/docs/securing/service-identity is probably the best documentation on service identities in Cloud Run. – sethvargo Jan 30 '22 at 21:55

0 Answers0