4

I wonder if there is an example of post process for EMR (Elastic MapReduce)? What I am trying to achieve is send an email to group of people right after Amazon's Hadoop finished the job.

James Lim
  • 12,915
  • 4
  • 40
  • 65
Roman Kagan
  • 10,440
  • 26
  • 86
  • 126

2 Answers2

5

You'll want to configure the job end notification URL.

jobEnd.notificationUrl

AWS will hit this url, presumably with query variables that indicate which job has completed (job id).

You could then have this URL on your server process your email notifications, assuming you had already stored a relationship between emails and job ids.

https://issues.apache.org/jira/browse/HADOOP-1111

FlavorScape
  • 13,301
  • 12
  • 75
  • 117
3

An easier way is to use Amazon CloudWatch (monitoring system) and Amazon Simple Notification Service (SNS) to monitor and notify you and others on the status of your EMR jobs.

For example you can set an alarm for your cluster to check when it IsIdle. It will be set to 1 once the job is done (or failed), and you can then get SNS notification as an email (or SMS even). You can set similar alarms on count of JobsFailed and other metrics.

For the complete list of EMR related metrics you can see EMR documentations

You can see more information about it here: http://docs.aws.amazon.com/ElasticMapReduce/latest/DeveloperGuide/UsingEMR_ViewingMetrics.html

Guy
  • 12,388
  • 3
  • 45
  • 67
  • Not sure that I understand your comment. You can define who is going to get the (email/SMS) notifications. They should accept to receive emails from the SNS topic once. Any way you can give some users or roles the ability to work only with SNS using Amazon IAM. – Guy Apr 08 '13 at 21:09
  • SNS allows you to send email to any email address – Guy Apr 09 '13 at 09:02
  • Your solution is good. However, corporate clients usually have their own email templating and localization standards and/or policy about messages on third party servers, making the job notification URL a good candidate for custom implimentations. – FlavorScape Jun 24 '14 at 17:51
  • @crtjer: It looks like there is a way to enable statistics agrigation on your account to make options like this available: http://docs.aws.amazon.com/AmazonCloudWatch/latest/DeveloperGuide/GetSingleMetricAllDimensions.html – Apriori Jun 25 '16 at 03:46