1

Right now I don't managed to have the Google Cloud Platform Data Loss Prevention (DLP) client library for python working behind a SSL proxy (it works fine with other GCP client lib for example for storage or bigquery): https://cloud.google.com/dlp/docs/libraries#client-libraries-usage-python

So I tried to use request.post to use the API behind a SSL proxy

url = 'https://dlp.googleapis.com/v2/projects/'+os.environ['PROJECT_ID']+'/content:inspect'

headers = {
    'Content-Type': 'application/json',
    'Authorization':  'Bearer {}'.format(subprocess.run('gcloud auth print-access-token', shell=True, check=True, stdout=subprocess.PIPE).stdout.decode().replace('\n', '').replace('\r', ''))
}

}
json_response = requests.post(url=url, json=parsed, headers=headers, proxies=proxies, verify=True)
json.loads(json_response.text)

This is working fine on CloudShell but not on my local machine where SDK is installed. The reason is that on CloudShell:

gcloud auth print-access-token

give me the same token for a period of few minutes while on my local machine (Windows or Mac), every time I execute the command, I got a new token. On my local machine if I replace in the header the gcloud command by the token from CloudShell it works fine. I have the latest version of SDK on both my local machine and on CloudShell.

question 1: it is expected that every time we run gcloud auth print-access-token locally (SDK), we get a new token ? (On CloudShell it is the same token for a period of few minutes)

question 2: what is the easiest/best way to generate a token ? since gcloud auth print-access-token doesn't seems the right way to do it when using local machine and SDK. This is not a productive application. This is just to test the DLP API.

Dr. Fabien Tarrade
  • 1,556
  • 6
  • 23
  • 49

1 Answers1

2

question 1: it is expected that every time we run gcloud auth print-access-token locally (SDK), we get a new token ? (On CloudShell it is the same token for a period of few minutes)

The answer depends on where you run your code. When running from a Google compute service (Cloud Shell is a VM), the token comes from the metadata server. I am now sure if or how long the token is cached. Tokens have an expiration (default 3600 seconds), so it is easy for the metadata server to cache tokens. If your code is running outside of the Google Cloud, the answer depends on the library used.

question 2: what is the easiest/best way to generate a token ? since gcloud auth print-access-token doesn't seems the right way to do it when using local machine and SDK. This is not a productive application. This is just to test the DLP API.

Obtaining tokens from the CLI is just for testing. The normal method is to use the SDK. However, since you are using the REST API, read this article that I wrote on how to create tokens in your own code and use them in REST APIs. My article includes Python source code and an example calling the Compute API to list instances in a project.

Google Cloud – Creating OAuth Access Tokens for REST API Calls

John Hanley
  • 74,467
  • 6
  • 95
  • 159
  • Thanks. For questions 2, it is clear thanks for this great link. Concerning questions 1, I have SDK installed locally and working. How to I see and control the duration of the validity of a token ? I did a deep dive in gcloud options but I couldn't see anything. I tried to run the code also locally where SDK is installed. – Dr. Fabien Tarrade Sep 19 '19 at 11:48
  • 1
    I am not aware of any Google API libraries that support setting expiration. The default and maximum are 3600 seconds. My code allows for changing the expiration. – John Hanley Sep 19 '19 at 13:58
  • To close this thread, the answer from GCP team about why token are regenerating everytime "I understand that you want to know if Cloud Shell and the Windows’ SDK behavior should be the same. This is an expected behavior, the Windows’ SDK is regenerating the token every time the command is executed because it’s a design choice." – Dr. Fabien Tarrade Oct 01 '19 at 16:46
  • @Dr.FabienTarrade - When you say "Windows SDK" do you mean the client libraries or the CLI `gcloud`? The answer from Google has to be prefaced with how the tokens are being created (Access Token, Identity Token, Signed JWT, etc.), by what type (User / Service Account) and by what library. – John Hanley Oct 01 '19 at 17:32
  • sorry, just coy and paste without the context. Yes it is using CLI gcloud on a Windows machine with SDK installed (using service account). – Dr. Fabien Tarrade Oct 01 '19 at 19:41
  • @Dr.FabienTarrade - This means that there is no caching of tokens, so a new one is issued each time. – John Hanley Oct 01 '19 at 20:19
  • Does it HAVE to be a service account or can I not use my own project owner personal account to create DLP jobs via the API? I'm trying to avoid having to create a service account in every single project, this makes automation kind of a pain. – Nathan McKaskle Mar 02 '20 at 21:06
  • @NathanMcKaskle - Create a new question. – John Hanley Mar 02 '20 at 21:08