Amazon SNS configuration for Data Pipeline success and failure

Question

I am using the Amazon data pipeline for the automation of some shell activity. Which will run once in a day. So, I was configuring the amazon SNS for letting me know whether the last run of the shell activity was successful or fail. If, failed then email me the reason of failure.

So, I was able to configure the SNS for sending me the mail. But, how should I configure the message part of SNS so that in case of failure, it send me the exact error? Also, in case of success send me the status SUCCESS.

Answer 1

You can try node references to refer to the object on which the action is added http://docs.aws.amazon.com/datapipeline/latest/DeveloperGuide/dp-pipeline-expressions.html shows how to do this.

Answer 2

Ok this is my working Dynamo -> S3 import. https://gist.github.com/osvadimos/2954ce4c0f7fc249594c999822e639f2

Regarding your question. First you have to create Fail/Success Object

public static PipelineObject getSNSFailActivity() {
    String name = "FailureNotify";
    String id = "FailureNotify";

    Field type = new Field()
            .withKey("type")
            .withStringValue("SnsAlarm");

    Field topicArn = new Field()
            .withKey("topicArn")
            .withStringValue("#{myTopicFail}");

    Field role = new Field()
            .withKey("role")
            .withStringValue("DataPipelineDefaultRole");

    Field subject = new Field()
            .withKey("subject")
            .withStringValue("FAIL: #{node.@scheduledStartTime}");

    Field message = new Field()
            .withKey("message")
            .withStringValue("#{myDDBTableName}");


    List<Field> fieldsList = Lists.newArrayList(type,
            role,
            topicArn,
            subject,
            message);

    return new PipelineObject()
            .withName(name)
            .withId(id)
            .withFields(fieldsList);
}

You have to add reference of Fail/Success object to your S3BackupLocation object

public static PipelineObject getS3BackupLocation() {
    String name = "S3BackupLocation";
    String id = "S3BackupLocation";

    Field type = new Field()
            .withKey("type")
            .withStringValue("S3DataNode");
    Field directoryPath = new Field()
            .withKey("directoryPath")
            .withStringValue("#{myOutputS3Location}#{format(@scheduledStartTime, 'YYYY-MM-dd-HH-mm-ss')}");
    Field onFail = new Field()
            .withKey("onFail")
            .withRefValue("FailureNotify");
    Field onSuccess = new Field()
            .withKey("onSuccess")
            .withRefValue("SuccessNotify");

    List<Field> fieldsList = Lists.newArrayList(type,
            directoryPath,
            onFail,
            onSuccess);

    return new PipelineObject()
            .withName(name)
            .withId(id)
            .withFields(fieldsList);
}