0

I have dataflow job that is written using apache beam.Here I am loading the data from one table to another table the mode of table is write truncate and the table contains sensitive data so table configured with kms key. I am following this code and trying to fetch the kms key associated with table and setting this key at the time of (BigQueryIO.write) while writing data into table.

https://github.com/apache/beam/blob/49403c00499f7f87c2bf9a4c63dec8ebd68d640d/sdks/java/io/google-cloud-platform/src/test/java/org/apache/beam/sdk/io/gcp/bigquery/BigQueryKmsKeyIT.java#L91-L100

finalOutput.apply("Write success rows to Sensitive BigQuery",
            BigQueryIO.writeTableRows().withoutValidation()
            .to(options.getTargetTable())
            .withKmsKey(NestedValueProvider.of(options.getTargetTable(),new FetchingKMSKey(options.getProject())).toString()).withoutValidation()
            .withCreateDisposition(BigQueryIO.Write.CreateDisposition.CREATE_NEVER)
            .withWriteDisposition(BigQueryIO.Write.WriteDisposition.WRITE_TRUNCATE));

public class FetchingKMSKey implements SerializableFunction<String, String> 

{

private static final BigqueryClient BQ_CLIENT = new BigqueryClient("dataMainPipeline");
    
    public static final Logger LOG = LoggerFactory.getLogger(FetchingKMSKey.class);
     Table table ;
    String tableName;
    String project;
    public FetchingKMSKey(String project) 
    {
        this.project=project;
    }

    public String apply(String tableName) 
    {
          String[] name= tableName.split(Pattern.quote("."));

        try 
        {
            table = BQ_CLIENT.getTableResource(project,name[0],name[1]);
        }
        catch (IOException e)
        {
        
            LOG.error(String.format("exception occured: %s", e.getMessage()));
        } 
        catch (InterruptedException e) 
        {
            
            LOG.error(String.format("exception occured: %s", e.getMessage()));
        }
          String kmsKey =table.getEncryptionConfiguration().getKmsKeyName();    
          return kmsKey;
        
    }

}

After creating a template when I am running the job I am getting error.

"location" : "US",

    "errors" : [ {
      "message" : "The KMS key does not contain a location.",
      "reason" : "invalid"
    } ],

I am completely stuck. anyone could help?

sethvargo
  • 26,739
  • 10
  • 86
  • 156
user_a27
  • 91
  • 1
  • 9
  • Can you share the value of this command `NestedValueProvider.of(options.getTargetTable(),new FetchingKMSKey(options.getProject())).toString()`? – guillaume blaquiere Jul 05 '20 at 12:21
  • Inside this I have implemented serializable interface and create a client using BigQueryClient and call the getTableResources method and with that reference I call the getKmsKey(). i will update the code part as well. – user_a27 Jul 05 '20 at 12:25
  • Ok, now same question: can you put a log on the line `String kmsKey =table.getEncryptionConfiguration().getKmsKeyName()` and paste the result of the KMSkey string content? – guillaume blaquiere Jul 05 '20 at 13:00
  • It's not printing any log I am directly getting an error. The whole code is there if you can give some example that how can I achieve this, it might help. – user_a27 Jul 05 '20 at 13:04
  • Can you try to replace this line `.withKmsKey(NestedValueProvider.of(options.getTargetTable(),new FetchingKMSKey(options.getProject())).toString()).withoutValidation()` by this one `.withKmsKey(new FetchingKMSKey(options.getProject()).apply(options.getTargetTable())).toString()).withoutValidation()`? Just to see if static value (not at runtime evaluation) is better for it!` – guillaume blaquiere Jul 05 '20 at 19:08
  • What do you exactly looking for? – user_a27 Jul 07 '20 at 06:10
  • I would like to not resolve dynamically the kmsKey and to fix it at the program startup. in addition, I don't know the string that generate your FetchingKmsKey class. Just to review the format to be sure that is correct. – guillaume blaquiere Jul 07 '20 at 07:41
  • Here I want to add the point that the key we are fetching from table metadata which is applied to the table only and here I am facing while getting dynamically. The format of the key is fine ,its giving location issue – user_a27 Jul 07 '20 at 07:50
  • I'm not sure that you can set dynamic KMSkey. Open an issue in the [github repo](https://github.com/apache/beam), the dev team can answer you over there! – guillaume blaquiere Jul 07 '20 at 08:30
  • Could you please guide me how to raise an issue. – user_a27 Jul 07 '20 at 09:36
  • As mentioned in the Readme of the repository, you can do this on their [JIRA](https://issues.apache.org/jira/projects/BEAM/issues/BEAM-10406?filter=allopenissues) – guillaume blaquiere Jul 07 '20 at 11:29
  • Let us [continue this discussion in chat](https://chat.stackoverflow.com/rooms/217401/discussion-between-user-a27-and-guillaume-blaquiere). – user_a27 Jul 07 '20 at 17:36

0 Answers0