0

I have a dataflow in which data (sample below) is fetched from the azure events hub and sent to other destinations based on destination_type. For E.g s3

Sample Example:

{
    "client_name": "foo",
    "destination_type": "s3",
    "data": {
        "key1": "foo-value1",
        "key2": "foo-value2",
        "key3": "foo-value3"
    }
}

{
    "client_name": "bar",
    "destination_type": "s3",
    "data": {
        "key1": "bar-value1",
        "key2": "bar-value2",
        "key3": "bar-value3"
    }
}

I can fetch this client_name and destination_type using EvaluateJsonPath and make it an attributes

Now based on client_name and destination_type, I have to configure processors property dynamically

I was planning to storage the creds in parameter with sensitive value and of format like

CREDS_<client_name>_S3_ACCESSKEY = <Access Key ID>
CREDS_<client_name>_S3_SECRETACCESSKEY = <Secret Access Key>

Eg. For PutS3Object - Bucket, Access Key ID & Secret Access Key needs to be loaded based on client_name

#{CREDS_${client_name}_S3_ACCESSKEY}
#{CREDS_${client_name}_S3_SECRETACCESSKEY}

But these seem to be not working out, can anyone suggest any alternative way to load sensitive value dynamically

Bhavesh Patel
  • 25
  • 10

2 Answers2

2

Parameters are resolved ahead of time, so you can't use variables to adjust the parameter name during execution.

Additionally, the docs suggest that PutS3Object does not support parameters, only Variable Registry (https://nifi.apache.org/docs/nifi-docs/components/org.apache.nifi/nifi-aws-nar/1.12.1/org.apache.nifi.processors.aws.s3.PutS3Object/index.html).

Parameters are supported.

So what you are trying to do is not possible currently.

Instead, you could create a PutS3Object per client, with their Key in the sensitive field, and then use RouteOnAttribute on the client_name attribute to go to the correct processor. Not ideal, but it would be the simplest flow to build.

Sdairs
  • 1,912
  • 1
  • 13
  • 13
  • 2
    All properties support parameters, it is not like expression language where you have to define which type of variables it can resolve, parameters are built into the framework by default – Bryan Bende Nov 24 '20 at 14:25
  • Yeah, Not sure how to deal with it so far. Thanks for this though. Also, I checked the StreamSets Data Collector, even it doesn't allow such a way. Seems like it goes against the design and principles of the platform. Refer: https://issues.streamsets.com/browse/SDC-7460 – Bhavesh Patel Nov 25 '20 at 13:24
0

Those Attributes in the the PutS3Object support variable registry, so you can use a combination of variable registry update while ou run it.(it can be slow).

Or use Execute streaming command to pass the AccessKey & Secret Key at S3 Put runtime as env variable.

Someting like this:

  ExecuteStreamingCommand Procesor
  Command Path: /usr/bin/env
  Command Arguments: AWS_ACCESS_KEY_ID=${AccessKeyId} AWS_SECRET_ACCESS_KEY=${SecretAccessKey} aws s3 cp ${source_key} $(target_key)

Second one is hack but it works :)

Up_One
  • 5,213
  • 3
  • 33
  • 65