4

When I try to create an EMRcluster resource with those properties:

  • Emr Managed Master Security Group Id
  • Emr Managed Slave Security Group Id

I have this error : Terminated with errors. You must also specify a ServiceAccessSecurityGroup if you use custom security

bbenjii123
  • 41
  • 1
  • 6

1 Answers1

0

Service Access Security Group: Besides the firewall settings mentioned in the 2 security groups mentioned. Internet traffic between AWS EMR Service servers(you dont have any control over this, completely managed by AWS) and your Slave EMR instance, has to be allowed.

     This security group contains 2 entries

HTTPS* (8443)     TCP (6)       8443       ElasticMapReduce-Slave-Private(sg-id)

HTTPS* (8443)     TCP (6)       8443       Default Security Group of VPC

Without this EMR will not work with DataPipeline. Neither Datapipeline specifies a way to list this in pipeline definition. AWS team is aware of this.

So, as a workaround please use the custom template provided by AWS, and clone, edit accordingly to your needs.

Thanks, @blamblam for pointing that out. The previous steps assume, servers have already been created in the private subnets, and you need to allow communication automatically. For launching in private subnet, we will include one more setting, Subnet Id, this will launch your EMR in private subnets. Hope, that helps.

Baibhav Vishal
  • 53
  • 10
  • "use the custom template provided by AWS" What template are you referring to? – Adrian Baker Nov 10 '17 at 03:22
  • "Run job on an Elastic MapReduce cluster" , provided under Source->Build Using a Template. – Baibhav Vishal Nov 21 '17 at 10:36
  • 1
    This is not a valid workaround if you need to run the cluster in a private subnet (as I assume the OP requires, since ServiceAccessSecurityGroup parameter is only needed under these circumstances). The suggested option "Run job on an Elastic MapReduce cluster" option launches the EMR cluster in a public subnet. – blamblam Jan 25 '18 at 10:38
  • @blamblam You can always clone and edit the subnet settings, to launch in private subnet. Dont know if it might be usefull now. – Baibhav Vishal Jun 04 '19 at 07:08