10

I am trying to understand how to deploy an Amazon Kinesis Client application that was built using the Kinesis client library (KCL).

I found this but it only states

You can follow your own best practices for deploying code to an Amazon EC2 instance when you deploy a Amazon Kinesis application. For example, you can add your Amazon Kinesis application to one of your Amazon EC2 AMIs.

which is not giving a broader picture to me.

These examples use an Ant script to run Java program. Is this the best practice to follow?

Also, I understand even before running the EC2 instances I need to make sure

  1. The developed code JAR/WAR or any other format needs to be on the EC2 instance
  2. The EC2 instance needs to have all the required environment like Ant setup in place already to execute the program.

Could someone please add some more detail on this?

malana
  • 5,045
  • 3
  • 28
  • 41
Sam
  • 1,333
  • 5
  • 23
  • 36
  • I'm having the very same problem, I don't understand how the app can start up on an instance boot. I would've expected it to be installed like some sort of service or daemon. – Kevin Pluck Jul 09 '14 at 08:51
  • 1
    I've posted pretty much the same question on the AWS forums but as yet no reply: https://forums.aws.amazon.com/thread.jspa?threadID=156072&tstart=0 – Kevin Pluck Jul 09 '14 at 08:58
  • I'm going to start a bounty so that we can get step-by-step instructions on deploying a scalable kinesis application. – Kevin Pluck Jul 09 '14 at 08:59

3 Answers3

9

Amazon Kinesis will be responsible for ingesting data, not running your application. You can run your application anywhere, but it is a good idea to run it in EC2, as you are probably going to use other AWS Services, such as S3 or DynamoDB (Kinesis Client Library uses DynamoDB for sharding, for example).

To understand Kinesis better, I'd recommend that you launch the Kinesis Data Visualization Sample. When you launch this app, use the provided CloudFormation template. It will create a stack with the Kinesis stream and an EC2 instance with the application, that uses Kinesis Client Library and is a fully working example to start from.

malana
  • 5,045
  • 3
  • 28
  • 41
Julio Faerman
  • 13,228
  • 9
  • 57
  • 75
  • Thanks Julio. It took me a bit to understand what you were saying but a reply to my question in the AWS forums said the very same thing: https://forums.aws.amazon.com/message.jspa?messageID=556078#556078 basically create an ec2 instance and use a cloudformation template that downloads your jar and create a script that executes it. – Kevin Pluck Jul 13 '14 at 20:52
  • @Sam I recommend this reply as an answer to your question. – Kevin Pluck Jul 13 '14 at 20:53
  • Thanks Julio, Thanks Kevin – Sam Jul 14 '14 at 08:40
1

The best way I have found to host a consumer program is using EMR, but not as a hadoop cluster. Package your program as a jar, and place it in s3. Launch an emr cluster and have it run your jar. Using the data pipeline you can schedule this job flow to run at regular intervals. You can also scale an emr cluster, or use a actual EMR job to process the stream if you choose to get the high tech.

Dan Ciborowski - MSFT
  • 6,807
  • 10
  • 53
  • 88
  • I just have a doubt, if i have a stream with one shard only, How it will be executed on EMR with master and slave nodes – Awadesh Jan 24 '18 at 06:00
  • this answer is very old and probably out of date, I think there are probably better ways to do this now, but I now use Azure instead of AWS – Dan Ciborowski - MSFT Jan 24 '18 at 15:34
1

You can also use Beanstalk. I believe this article is highly useful.

malana
  • 5,045
  • 3
  • 28
  • 41
Ana Todor
  • 781
  • 1
  • 6
  • 15