3

I am trying to get my head around this, but cannot figure it out why

  1. I have an EMR cluster to be deployed in AWS Private Subnet.
  2. I checked the documentation here.
  3. From the above I understood the following:

a. For my EMR custom security group, I DO NOT NEED to specify ingress rule for port 9443 i.e. it can be the same as the ElasticMapReduce-Master-Private. The following are my inbound/outbound rules for the EMR Managed Security Group for Master/Slave instances (NOT Service)

Outboundenter image description here

Inboundenter image description here

b. For my custom service security group I can keep it the same as ElasticMapReduce-ServiceAccess, I need to specify ingress rule for port 9443 like the following (Terraform):

resource "aws_security_group_rule" "allow_tcp_from_master_to_service" {
  type                     = "ingress"
  from_port                = 9443
  to_port                  = 9443
  protocol                 = "tcp"
  security_group_id        = join("", aws_security_group.ml.*.id)
  source_security_group_id = join("", aws_security_group.ml_emr.*.id)

  lifecycle {
    create_before_destroy = false
  }
}

But when I deploy this using terraform, I get the AWS error:

Error waiting for EMR Cluster state to be "WAITING" or "RUNNING": TERMINATED_WITH_ERRORS: VALIDATION_ERROR: ServiceAccessSecurityGroup is missing ingress rule from EmrManagedMasterSecurityGroup on port 9443

I am struggling to understand why AWS is saying I need this when I don't? From the documentation it's not clear where this mentioned, so I would appreciate if someone could clarify what's expected here.

Regards,

ha9u63a7
  • 6,233
  • 16
  • 73
  • 108
  • were you able to figure this out? – alex Oct 22 '21 at 21:38
  • @alex not entirely - this is also possibility when your organisation "Over-manages" AWS - I had to destroy everything using TF destroy and then recreate it. – ha9u63a7 Oct 31 '21 at 05:16

2 Answers2

1

Did you have the same SG for master and task instances?

I had different SGs for master and task instances and I solved it by using the same SG for both

José Eduardo Juárez Morales
  • 11
  • 1
0

As per documentation https://docs.aws.amazon.com/emr/latest/ManagementGuide/emr-man-sec-groups.html#emr-sg-elasticmapreduce-sa-private

you have to create different/separate Security group rule for TCP port 9443 which is mandatory from emr 5.30.0, which allows the communication between master instance's security group to the service access security group.

Satya
  • 21
  • 3