Want to server-side encrypt S3 data node file created by ShellCommandActivity

Question

I created a ShellCommandActivity with stage = "true". Shell command creates a new file and store it in ${OUTPUT1_STAGING_DIR}. I want this new file to be server-side encrypted in S3.

According to document all files created in s3 data node are server-side encrypted by default. But after my pipeline completes an un-encrypted file gets created in s3. I tried setting s3EncryptionType as SERVER_SIDE_ENCRYPTION explicitly in S3 datanode but that doesn't help either. I want to encrypt this new file.

Here is relevant part of pipeline:

{
  "id": "DataNodeId_Fdcnk",
  "schedule": {
    "ref": "DefaultSchedule"
  },
  "directoryPath": "s3://my-bucket/test-pipeline",
  "name": "s3DataNode",
  "s3EncryptionType": "SERVER_SIDE_ENCRYPTION",
  "type": "S3DataNode"
},
{
  "id": "ActivityId_V1NOE",
  "schedule": {
    "ref": "DefaultSchedule"
  },
  "name": "FileGenerate",
  "command": "echo 'This is a test' > ${OUTPUT1_STAGING_DIR}/foo.txt",
  "workerGroup": "my-worker-group",
  "output": {
    "ref": "DataNodeId_Fdcnk"
  },
  "type": "ShellCommandActivity",
  "stage": "true"
}

Answer 1

Short answer: Your pipeline definition looks correct. You need to ensure you're running the latest version of the Task Runner. I will try to reproduce your issue and let you know.

P.S. Let's keep conversation within a single thread here or in AWS Data Pipeline forums to avoid confusion.

Answer on official AWS Data Pipeline Forum page

Answer 2

This issue is resolved when I downloaded new TaskRunner-1.0.jar. I was running older version.