1

I'm currently querying a project (call it data_project) in bigquery through the python SDK. For several reasons I want to restrict the access to this project to a full read-only and switch the queries itself to another project (call it quota_project) without having to specify the project in the query explicitly.

This means I don't want to do SELECT * FROM `project.dataset.table` but rather keep SELECT * FROM `dataset.table` while switching the project it is billed against.

Could be relevant here (but should not imho): quota_project has reservations.

To my understanding, this would be allowed by the option quota_project_id while initializing the credentials.

I tried the following two options which I thought make sense from following the documentation, neither of it worked out (both queries have been billed to the data_project):

import google.cloud.bigquery as bq
from google.oauth2.credentials import Credentials

creds = Credentials.from_authorized_user_file(credential_path, scopes)

# option 1
c = creds.with_quota_project(quota_project)
client = bq.client.Client(credentials=c)
client.query("SELECT 1").result().to_dataframe()

# option 2
client = bq.client.Client(
    credentials=creds, client_options={"quota_project_id": quota_project}
)
client.query("SELECT 1").result().to_dataframe()

Is anybody aware of how to achieve this? Or am I misunderstanding the intention of a "quota project"?

Nico Albers
  • 1,556
  • 1
  • 15
  • 32
  • The method “with_quota_project” is used to add a quota from your specific project and to return a copy of these credentials. You can refer to this [link](https://github.com/googleapis/google-auth-library-python/blob/main/google/oauth2/service_account.py) for more detailed implementation of this method which is used to return these google.auth credentials.Let me know if it's helpful or not? – Prajna Rai T Nov 18 '22 at 09:32
  • Not sure what you mean by `add a quota from project` - I understand this routine as `credentials still using the project id but billing on a different project` (which seems reasonable as the linked credentials class is initiated using project_id and quota_project_id). – Nico Albers Nov 22 '22 at 09:14

1 Answers1

1

A quota project is used by client libraries, etc. for billing purposes. You can set the quota project using the CLI:

gcloud auth application-default set-quota-project my-quota-project

To set Application default credentials, you have to run the command:

gcloud auth application-default login

The existing application default credentials must have the serviceusage.services.use permission on the given project.

You can refer to this regarding to BigQuery quotas and limits and this stack link to know more about query costs in BigQuery.

Prajna Rai T
  • 1,666
  • 3
  • 15
  • Unfortunately this doesn't help. As the code suggests, there are already authenticated credentials which I use (and yes, tried setting the quota project as well via cli, didn't work out either). I'm searching for a reason why adding the quota project manually doesn't lead to its application (or a comment why I'm mistaking the docs, understanding currently that I can have a different GOOGLE_CLOUD_PROJECT_ID - i.e. default project to use for datasets if not is hardcoded in the queries - and quota project). I'll try out adding serviceusage consumer explicit, even if this hadn't raised an error. – Nico Albers Nov 30 '22 at 20:37
  • Update: I checked it, I'm owner on the project I want to use for billing and have Job User permission on the project I want to use as default project - Confused why the code is not working as expected. – Nico Albers Dec 01 '22 at 13:56
  • Are you getting any error messages? When trying to set a quota-project using CLI can you make sure you have authenticated with proper credentials? For example using a service account. – Prajna Rai T Dec 03 '22 at 20:15
  • No I don't and yes, I receive the data, just not through the proper project. I'm looking for an answer whether this is the right approach taken and for a MWE to use different default project and billing project together (if that's possible, otherwise searching for a clear no). – Nico Albers Dec 04 '22 at 11:59
  • `quota_project_id` is project ID used for quota and billing. If you are a billing administrator on only one Cloud Billing account, new projects you create are automatically linked to your existing Cloud Billing account. For more information you can refer to this [document](https://cloud.google.com/billing/docs/how-to/modify-project). To query data from another project this [link](https://stackoverflow.com/questions/71173673/python-bigquery-api-and-accessing-tables-in-another-project/71189292#71189292) can be referred. – Prajna Rai T Dec 04 '22 at 12:56