0

Using Google Deployment Manager, has anybody found a way to first create a view in BigQuery, then authorize one or more datasets used by the view, sometimes in different projects, and were not created/managed by deployment manager? Creating a dataset with a view wasn't too challenging. Here is the jinja template named inventoryServices_bigquery_territory_views.jinja:

resources:
- name: territory-{{properties["OU"]}}
  type: gcp-types/bigquery-v2:datasets
  properties:
    datasetReference:
      datasetId: territory_{{properties["OU"]}}
- name: files
  type: gcp-types/bigquery-v2:tables
  properties:
    datasetId: $(ref.territory-{{properties["OU"]}}.datasetReference.datasetId)
    tableReference:
      tableId: files
    view:
      query: >
        SELECT DATE(DAY) DAY, ou, email, name, mimeType
        FROM `{{properties["files_table_id"]}}`
        WHERE LOWER(SPLIT(ou, "/")[SAFE_OFFSET(1)]) = "{{properties["OU"]}}"
      useLegacySql: false

The deployment configuration references the above template like this:

imports:
- path: inventoryServices_bigquery_territory_views.jinja

resources:
- name: inventoryServices_bigquery_territory_views
  type: inventoryServices_bigquery_territory_views.jinja

In the example above files_table_id is the project.dataset.table that needs the newly created view authorized.

I have seen some examples of managing IAM at project/folder/org level, but my need is on the dataset, not project. Looking at the resource representation of a dataset it seems like I can update access.view with the newly created view, but am a bit lost on how I would do that without removing existing access levels, and for datasets in projects different than the one the new view is created in. Any help appreciated.

Edit: I tried adding the dataset which needs the view authorized like so, then deploy in preview mode just to see how it interprets the config:

-name: files-source
  type: gcp-types/bigquery-v2:datasets
  properties:
    datasetReference:
      datasetId: {{properties["files_table_id"]}}
    access:
      view:
        projectId: {{env['project']}}
        datasetId: $(ref.territory-{{properties["OU"]}}.datasetReference.datasetId)
        tableId: $(ref.territory_files.tableReference.tableId)

But when I deploy in preview mode it throws this error:

errors:
- code: MANIFEST_EXPANSION_USER_ERROR
  location: /deployments/inventoryservices-bigquery-territory-views-us/manifests/manifest-1582283242420
  message: |-
    Manifest expansion encountered the following errors: mapping values are not allowed here
      in "<unicode string>", line 26, column 7:
          type: gcp-types/bigquery-v2:datasets
              ^ Resource: config

Strange to me, hard to make much sense of that error since the line/column it points to is formatted exactly the same as the other dataset in the config, except that maybe it doesn't like that the files-source dataset already exists and was created from outside of deployment manager.

Michael
  • 1,428
  • 3
  • 15
  • 34
  • Could you provide more details about your steps and expected results. If possible please provide your deployment to make me able to try it on my test project. – Serhii Rohoza Feb 21 '20 at 15:32
  • I added the deployment configuration file to the post above to try and make it a little more clear. Essentially I've already got an existing dataset which is referenced in the view's SQL with SELECT * FROM `{{properties["files_table_id"]}}`. I want to create a bunch of views to that dataset by passing an ISO country code and the name of the dataset as a parameter to gcloud. However creating the view isn't enough, the view also needs to be authorized. Meaning that the already existing dataset (not created or managed by deployment manager) needs to grant access to this newly created view. – Michael Feb 21 '20 at 16:25
  • To authorize your view you can use separate service account in each project https://cloud.google.com/iam/docs/understanding-service-accounts. To provide necessary permissions to that service account you can use roles https://cloud.google.com/bigquery/docs/access-control#bq-permissions. – Serhii Rohoza Feb 24 '20 at 08:34
  • Thanks Serhii, but I wasn’t running into issues with auth, more with how to design the template. For now I’ve excluded IAM (authorizing the views) from the template, and think I’ll try to do it in Python instead. With that might be able to update the existing list of authorized views without removing the old ones. – Michael Feb 25 '20 at 10:08
  • I thought that your issue related to IAM because of the title and error. If you fixed that issue please change your question if you still interested after decision to switch to Python. – Serhii Rohoza Feb 25 '20 at 10:21
  • I am trying to manage IAM with DM, not having problems where DM has insufficient privileges. Don't think the title needs to be changed. If someone has a solution on how to do this in jinja I think the community would gain value from that, so am going to leave as is. – Michael Feb 25 '20 at 12:45
  • As I understand your problem, to be able to authorize any request you should use service account with necessary permissions, and to be able to do it the different projects that aren't created/managed by deployment manager you should use same service account in the different projects (and it could be cause of security issues). On the other hand, to provide you any recommendation about your deployment some replication at the test projects needed, and to be able to do it, all your deployment files needed as well. – Serhii Rohoza Feb 25 '20 at 13:46

0 Answers0