2

I have been stuck on this problem for the past two days. I am wanting to use Lambda as Cron operation to get data from my database and post it to BigQuery.

I would like to know how to authorize access to BigQuery using a services account file from my Lambda function.

Context

I am using the following:

Attempts

  • Everything has worked fine on my local using sls invoke local --function main . I have set GOOGLE_APPLICATION_CREDENTIALS=/path/to/key.json in my .env. But,obviously /path/to/key.json is a local path.
  • As a test, I tried just putting my service account file into the root directory of my project and set GOOGLE_APPLICATION_CREDENTIALS=./key.json (notice I used a relative path). This does not work locally or in cloud. And yes, I know it is not good practice -- I am just trying to get it working. I believe this may be a WebPack thing, although I am totally clueless on how to use WebPack.
  • I have also thought about perhaps using AWS KMS to encrypt the json and store it as a key-value pair in parameter store (which is the way I eventually want to use it). But, I noticed that BigQuery requires to take in a filepath and not the secret itself See here.

Project Structure

Question

So here is my question:

  1. Is there to connect to BigQuery using the Serverless Framework using a filepath and the .env file?
  2. Why Google does not just allow you to use access keys to connect to BigQuery? Rather I must specify path to my file.
  3. Is there a way to store a file in KMS and decrypt it upon deployment to Lambda?
John Rotenstein
  • 241,921
  • 22
  • 380
  • 470
tintinthong
  • 131
  • 4
  • 1
    Create your service account in GCP -> Give it the appropriate ACLs (BigQuery in this case) -> package it with your Lambda function/zip -> reference it in your code like so: https://github.com/googleapis/nodejs-bigquery/blob/0ec07f994e0e9567025d1c96ad65f9a057a65344/samples/clientJSONCredentials.js#L19 – Graham Polley Jan 07 '20 at 09:13
  • @GrahamPolley The high-level structure is not so much my problem. The problem is mainly (package it with your Lambda function/zip) in the steps you described. – tintinthong Jan 07 '20 at 09:22
  • https://docs.aws.amazon.com/lambda/latest/dg/nodejs-create-deployment-pkg.html – Graham Polley Jan 07 '20 at 13:03

1 Answers1

0

I know this is old but here how I did it:

I created a lambda layer with the cred.json file and attached the lambda layer to the specific function that use gcp big query in my serverless.yml.

I created it outside the .yml because I don't want my cred.json to be part of my team reposity in github (security issue).

With lambda layer, at each lambda start, the file will be available in the local machine at the path: '/opt/cred.json'

add the following into your function block inside your serverless.yml:

layers:
  - arn:aws:lambda:{$region}:{$account):layer:lambda-layer-gcp-bigquery-keyfile:1
lemonpear
  • 58
  • 5