0

I am new to Flink and EMR cluster deployment. Currently we have a Flink job and we are manually deploying it on AWS EMR cluster via Flink CLI stop/start-job commands.

I wanted to automate this process (Automate updating flink job jar on every deployment happening via pipelines with savepoints) and need some recommendations on possible approaches that could be explored.

OneCricketeer
  • 179,855
  • 19
  • 132
  • 245
priyadhingra19
  • 333
  • 4
  • 15
  • 1
    You could do something like AWS Kinesis Data Analytics already provides (updating a job from new jar on S3): 1. build new jar and push it to S3 2. have a script on EMR to fetch jar from S3 and run it 3. run this script with EMR step (command-runner.jar) – bzu Jun 04 '22 at 08:36

1 Answers1

0

We got an option to automate this process via Flink Rest API support for all flink job operation

https://nightlies.apache.org/flink/flink-docs-master/docs/ops/rest_api/

Sample project which used the same approach : https://github.com/ing-bank/flink-deployer

priyadhingra19
  • 333
  • 4
  • 15