0

I have a datapipeline application that I need to respond to. When it finishes, I ssh to an ec-2 instance and execute a script. What is the best way to do ssh to that box after datapipeline finishes? Should I use a lambda function and have it listen for datapline completion and ssh to that box and execute the script? I can't take the script off of that box.

bjamin
  • 131
  • 1
  • 5

2 Answers2

1

There are couple of solutions here that I can think of.

  • At the end of completion of Datapipeline, write a zero byte file on S3 to signal end of it. That S3 put can signal a Lambda function as you noted. But I do not know how to rexec a shell script.

  • Another approach is to install taskrunner on the ec2-box and make that participate in your Data pipeline. That shell script could be the 'activity' in the pipeline. Here is a how that can be done http://docs.aws.amazon.com/datapipeline/latest/DeveloperGuide/dp-how-task-runner-user-managed.html This approach does not need to go outside your current Datapipeline control flow and if the shellscript fails for any reason, the pipeline would fail providing you with integrated view.

user1452132
  • 1,758
  • 11
  • 21
0

Here's what I ended up doing. I wrote a lambda application that reads the SNS message from datapipeline, pulled a PEM from an encrypted S3 bucket, and sshed into the box, and executed the shell script.

bjamin
  • 131
  • 1
  • 5