3

I need to Encrypt the Sensitive fields in the Bq Table but my Loading Is Done through the Dataflow. I thought of 3 Different way to Use it.

  1. Encrypt the whole Table using Customer Managed Key and Make 3 Views on Different Classifications and provide Service account to Users to access the View and Provide that Service account role as Decrypter in KMS and Dataflow Service Account as Encrypter Load the Table. (Problem We do not have View Level Access so that views Required to Maintain in Different Datasets which makes our job more Difficult)

  2. Encrypt the Fields Using The API call in Dataflow While Loading and Make a UDF function to Decrypt that Colum Data at Runtime in Bq Using Service Account.

    Example Id Fields are Encrypted Using API call in Dataflow And we defined a UDF function in Bq to Decrypt it but only those can decrypt that Data who have access in KMS else it will throw an Exception

    In this way we keep a Single Table Open to All Users but Only Authenticate Use can only See the that.

    Problem: (Continuous Call of API at Runtime which makes our quota Exhausted and Cost is Another Matter)

  3. Maintaining Different tables in different datasets which a. Encrypted Tables with Sensitive Field b. Non-Encrypted Table with Non-Sensitive Fields.

    Problem: (Maintenance and Making Data in Sink and Join at Run Time in BQ)

The Above are My Approach and Use case Is Anyone able to help me to see what to Use and Why its better than others.

Tim Dierks
  • 2,168
  • 15
  • 28
BackBenChers
  • 304
  • 2
  • 15
  • Can you clarify why you need to encrypt the sensitive fields? Is there a compliance requirement or do you want certain security properties, or some other reason? This will help me suggest the right solution. – Tim Dierks Jun 16 '18 at 19:47
  • @TimDierks, I want a Certain Security property – BackBenChers Jun 17 '18 at 06:13
  • 1
    @TimDierks one of my Use Case is: My Users can See their Data directly in Bq and can import their result but when they need to See the Sensitive Fields or want their report with Sensitive Data then need to Verify them self and only their Data is Decrypted from that table and Shown Using Looker/Tableau/Datastudio. – BackBenChers Jun 17 '18 at 06:14
  • @BackBenChers, did you manage to get the best practice for your use case yet? If not, I suggest that you can open a new topic at the Google Groups(https://groups.google.com/forum/#!forum/google-cloud-dev) to get suggestions from the related product team. – JL-HaiNan Oct 23 '18 at 22:09
  • @JL-HN thanks for the Suggestion – BackBenChers Oct 24 '18 at 03:43
  • KMS support for Beam IOs is being worked on actively now. We are recently getting answers for your question : ) – Pablo Mar 11 '19 at 23:18

0 Answers0