2

I wanted to encrypt dob field. So I am trying to use the encrypt stage in streamsets. I pass the secret key id, secret key and KMS ARN. When I try to validate the pipeline, I get following error:

2019-05-14 16:52:36,492 xxxx_pipeline_5b58f6b1-d8d8-4349-b963-a513b9f3bf64  WARN    Stage 'EncryptandDecryptFields_01' initialization error: com.amazonaws.encryptionsdk.exception.AwsCryptoException: Can't use non-ARN key identifiers or aliases when no default region is set   Pipeline    *admin  0   preview-pool-1-thread-3
com.amazonaws.encryptionsdk.exception.AwsCryptoException: Can't use non-ARN key identifiers or aliases when no default region is set
    at com.amazonaws.encryptionsdk.kms.KmsMasterKeyProvider$Builder.build(KmsMasterKeyProvider.java:250)
    at com.streamsets.pipeline.stage.processor.crypto.FieldEncryptProcessor.createCryptoMaterialsManager(FieldEncryptProcessor.java:142)
    at com.streamsets.pipeline.stage.processor.crypto.FieldEncryptProcessor.createProvider(FieldEncryptProcessor.java:112)
    at com.streamsets.pipeline.stage.processor.crypto.FieldEncryptProcessor.init(FieldEncryptProcessor.java:90)
    at com.streamsets.pipeline.api.base.BaseStage.init(BaseStage.java:48)
    at com.streamsets.pipeline.api.base.configurablestage.DStage.init(DStage.java:36)
    at com.streamsets.datacollector.runner.StageRuntime.lambda$init$0(StageRuntime.java:211)
    at com.streamsets.datacollector.util.LambdaUtil.withClassLoaderInternal(LambdaUtil.java:148)
    at com.streamsets.datacollector.util.LambdaUtil.withClassLoader(LambdaUtil.java:44)
    at com.streamsets.datacollector.runner.StageRuntime.init(StageRuntime.java:209)
    at com.streamsets.datacollector.runner.StagePipe.init(StagePipe.java:123)
    at com.streamsets.datacollector.runner.StagePipe.init(StagePipe.java:47)
    at com.streamsets.datacollector.runner.Pipeline.initPipe(Pipeline.java:403)
    at com.streamsets.datacollector.runner.Pipeline.lambda$init$0(Pipeline.java:393)
    at com.streamsets.datacollector.runner.PipeRunner.forEach(PipeRunner.java:166)
    at com.streamsets.datacollector.runner.Pipeline.init(Pipeline.java:391)
    at com.streamsets.datacollector.runner.Pipeline.validateConfigs(Pipeline.java:219)
    at com.streamsets.datacollector.runner.preview.PreviewPipeline.validateConfigs(PreviewPipeline.java:60)
    at com.streamsets.datacollector.execution.preview.sync.SyncPreviewer.validateConfigs(SyncPreviewer.java:142)
    at com.streamsets.datacollector.execution.preview.async.AsyncPreviewer$1.call(AsyncPreviewer.java:70)
    at com.streamsets.pipeline.lib.executor.SafeScheduledExecutorService$SafeCallable.lambda$call$0(SafeScheduledExecutorService.java:226)
    at com.streamsets.datacollector.security.GroupsInScope.execute(GroupsInScope.java:33)
    at com.streamsets.pipeline.lib.executor.SafeScheduledExecutorService$SafeCallable.call(SafeScheduledExecutorService.java:222)
    at com.streamsets.pipeline.lib.executor.SafeScheduledExecutorService$SafeCallable.lambda$call$0(SafeScheduledExecutorService.java:226)
    at com.streamsets.datacollector.security.GroupsInScope.execute(GroupsInScope.java:33)
    at com.streamsets.pipeline.lib.executor.SafeScheduledExecutorService$SafeCallable.call(SafeScheduledExecutorService.java:222)
    at java.util.concurrent.FutureTask.run(FutureTask.java:266)
    at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
    at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
    at com.streamsets.datacollector.metrics.MetricSafeScheduledExecutorService$MetricsTask.run(MetricSafeScheduledExecutorService.java:100)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
    at java.lang.Thread.run(Thread.java:748)

In the key provide tab,

I pass the secret key id, secret key and KMS ARN.

Not applicable

This stage should compile without errors

metadaddy
  • 4,234
  • 1
  • 22
  • 46
Mani
  • 41
  • 2
  • Any help is very much appreciated. – Mani May 14 '19 at 16:57
  • Thank you for including the full stacktrace (maybe a bit much, but OK). However, it seems that no source code or configuration info is present in your question, so how would we be able to answer? At least show the ARN if that's the problem, right? – Maarten Bodewes May 15 '19 at 01:27
  • Seems like you are not providing the complete ARN. What is your KMS ARN – metadaddy May 15 '19 at 01:48
  • Metadaddy, Maarten, Thanks for your response. This is the KMS ARN - arn:aws:es:us-east-1:510680944440:domain/axtxnarchsinglefindes – Mani May 15 '19 at 03:20
  • for the Cipher - ALG_AES_256_GCM_IV12_TAG16_HKDF_SHA384_ECDSA_P384 – Mani May 15 '19 at 03:24
  • I got it fixed. The issue was with initialization of the KMS ARN. After initializing, it is working – Mani May 15 '19 at 12:10
  • @Mani could you provide the answer here too, so it's clear to anyone else searching? – metadaddy May 16 '19 at 15:39

0 Answers0