Cloudformation Blue/Green Deployment HealthCheckGracePeriodSeconds

Question

I am trying to use the Cloudformation Codedeploy Blue/Green deployment feature so have a task set that looks like the following.

TaskSet:
  Type: AWS::ECS::TaskSet
  Properties:
    Cluster: 
      Fn::ImportValue: !Sub "${ClusterStackName}-ClusterID"
    LaunchType: FARGATE
    NetworkConfiguration:
      AwsVpcConfiguration:
        AssignPublicIp: ENABLED
        SecurityGroups: 
          - !Ref FargateSecurityGroup
        Subnets: 
          - Fn::ImportValue: !Sub "${ClusterStackName}-SUBNET1ID"
          - Fn::ImportValue: !Sub "${ClusterStackName}-SUBNET2ID"
    PlatformVersion: 1.3.0
    Scale:
      Unit: PERCENT
      Value: 1
    Service: !Ref ECSService
    TaskDefinition: !Ref TaskDefinition
    LoadBalancers: 
      - ContainerName: !Ref ServiceName
        ContainerPort: !Ref ContainerPort
        TargetGroupArn: !Ref TargetGroup

My ECS Service definition looks like the following

ECSService:
  Type: AWS::ECS::Service
  DependsOn: HTTPSListener
  Properties: 
    ServiceName: !Sub "${ServiceName}-service"
    Cluster: 
      Fn::ImportValue: !Sub "${ClusterStackName}-ClusterID"
    DeploymentController: 
      Type: EXTERNAL
    DesiredCount: 4
    EnableECSManagedTags: true
    HealthCheckGracePeriodSeconds: 30
    SchedulingStrategy: REPLICA

This causes the following error in Cloudformation

Invalid request provided: CreateService error: Health check grace period is only valid for services configured to use load balancers

However omitting HealthCheckGracePeriodSeconds causes my tasks to fail the healthcheck while starting. I have checked the docs and there appears to be no option to add HealthCheckGracePeriodSeconds to the TaskSet definition.

How can I use Cloudformation blue/green deployment for ECS with a health check grace period?

Full Example

Here is a full template that I would expect to work but does not.

AWSTemplateFormatVersion: 2010-09-09
Parameters: 
  Image: 
    Type: String
  Vpc:
    Type: 'AWS::EC2::VPC::Id'
  Subnet1:
    Type: 'AWS::EC2::Subnet::Id'
  Subnet2:
    Type: 'AWS::EC2::Subnet::Id'

Transform:
  - AWS::CodeDeployBlueGreen

Hooks:
  CodeDeployBlueGreenHook:
    Type: AWS::CodeDeploy::BlueGreen
    Properties:
      TrafficRoutingConfig:
        Type: AllAtOnce
      Applications:
        - Target:
            Type: AWS::ECS::Service
            LogicalID: ECSService
          ECSAttributes:
            TaskDefinitions:
              - TaskDefinition
              - GreenTaskDefinition
            TaskSets:
              - TaskSet
              - GreenTaskSet
            TrafficRouting:
              ProdTrafficRoute:
                Type: AWS::ElasticLoadBalancingV2::Listener
                LogicalID: HTTPListener
              TargetGroups:
                - TargetGroup
                - TargetGroupGreen
    
Resources: 
  ALBSecurityGroup:
    Type: AWS::EC2::SecurityGroup
    Properties:
      GroupDescription: Allow HTTP to load balancer
      VpcId: !Ref Vpc
      SecurityGroupIngress:
      - IpProtocol: tcp
        FromPort: 80
        ToPort: 80
        CidrIp: 0.0.0.0/0
      SecurityGroupEgress:
      - IpProtocol: -1
        CidrIp: 0.0.0.0/0

  FargateSecurityGroup:
    Type: AWS::EC2::SecurityGroup
    Properties:
      GroupDescription: Allow port 80 to service
      VpcId: !Ref Vpc
      SecurityGroupIngress:
      - IpProtocol: tcp
        FromPort: 80
        ToPort: 80
        SourceSecurityGroupId: !Ref ALBSecurityGroup
      SecurityGroupEgress:
      - IpProtocol: -1
        CidrIp: 0.0.0.0/0

  LoadBalancer:
    Type: AWS::ElasticLoadBalancingV2::LoadBalancer
    Properties:
      Name: load-balancer
      Type: application
      IpAddressType: ipv4
      Scheme: internet-facing
      SecurityGroups: [!Ref ALBSecurityGroup]
      Subnets: [!Ref Subnet1, !Ref Subnet2]

  HTTPListener:
    Type: AWS::ElasticLoadBalancingV2::Listener
    Properties:
      DefaultActions:
        - TargetGroupArn: !Ref TargetGroup
          Type: forward
      LoadBalancerArn: !Ref LoadBalancer
      Port: 80
      Protocol: HTTP

  TargetGroup:
    Type: AWS::ElasticLoadBalancingV2::TargetGroup
    Properties:
      HealthCheckEnabled: true
      HealthCheckIntervalSeconds: 30
      HealthCheckPath: "/"
      HealthCheckTimeoutSeconds: 10
      HealthyThresholdCount: 5
      Name: targetgroup-blue
      Port: 80
      Protocol: HTTP
      TargetType: ip
      UnhealthyThresholdCount: 10
      VpcId: !Ref Vpc

  TargetGroupGreen:
    Type: AWS::ElasticLoadBalancingV2::TargetGroup
    Properties:
      HealthCheckEnabled: true
      HealthCheckIntervalSeconds: 30
      HealthCheckPath: "/"
      HealthCheckTimeoutSeconds: 10
      HealthyThresholdCount: 5
      Name: targetgroup-green
      Port: 80
      Protocol: HTTP
      TargetType: ip
      UnhealthyThresholdCount: 10
      VpcId: !Ref Vpc
      
  TaskDefinition:
    Type: AWS::ECS::TaskDefinition
    Properties:
      Family: "task-family-name"
      NetworkMode: awsvpc
      RequiresCompatibilities: [FARGATE]
      Cpu: 512
      Memory: 1024
      ContainerDefinitions:
        - Name: Container
          Essential: true
          Image: !Ref Image
          PortMappings:
            - ContainerPort: 80

  TaskSet:
    Type: AWS::ECS::TaskSet
    Properties:
      Cluster: !Ref ECSCluster
      LaunchType: FARGATE
      NetworkConfiguration:
        AwsVpcConfiguration:
          AssignPublicIp: ENABLED
          SecurityGroups: [!Ref FargateSecurityGroup]
          Subnets: [!Ref Subnet1, !Ref Subnet2]
      Scale:
        Unit: PERCENT
        Value: 100
      Service: !Ref ECSService
      TaskDefinition: !Ref TaskDefinition
      PlatformVersion: LATEST
      LoadBalancers: 
        - ContainerName: Container
          ContainerPort: 80
          TargetGroupArn: !Ref TargetGroup

  ECSService:
    Type: AWS::ECS::Service
    DependsOn: HTTPListener
    Properties: 
      ServiceName: "service"
      Cluster: !Ref ECSCluster
      DeploymentController: 
        Type: EXTERNAL
      DesiredCount: 4
      EnableECSManagedTags: true
      HealthCheckGracePeriodSeconds: 30
      SchedulingStrategy: REPLICA
    
  PrimaryTaskSet:
    Type: 'AWS::ECS::PrimaryTaskSet'
    Properties:
      Cluster: !Ref ECSCluster
      Service: !Ref ECSService
      TaskSetId: !GetAtt 'TaskSet.Id'

  ECSCluster:
    Type: AWS::ECS::Cluster
    Properties:
      ClusterName: Cluster
      CapacityProviders:
        - FARGATE
      DefaultCapacityProviderStrategy:
        - CapacityProvider: FARGATE
          Weight: 1

Do you have load balancer? You can't perform B/G without the balancer. — Marcin, Nov 12 '20 at 22:13
@Marcin I have a load balancer attached to the TaskSet I get an error if I try to attach it to the ECS service — Josh, Nov 13 '20 at 14:11
@Tom How do you use the LB? For service you have to set it up using `LoadBalancers` property. Your current ECSService does not show any LB — Marcin, Dec 10 '20 at 23:54
@Marcin I'm using external deployments, and the connection to the load balancer comes when setting the primary task set on the service as part of the external deployment. The service definition (in CloudFormation) does not have any direct association with a loadbalancer and, as far as I can tell, it cannot if the deployment type is external. — Tom, Dec 14 '20 at 08:41
I never got this working - I ended up increasing the UnhealthyThresholdCount + HealthCheckIntervalSeconds so it would not fail before service has started. A really hacky solution. I will try to post a full template later. — Josh, Dec 14 '20 at 08:50
Is rolling update an option ? It works fine on our project with DeploymentController: Type: "ECS" — sashok_bg, Dec 15 '20 at 14:55
I have edited the question to contain a full example. If I try to move the LoadBalancers key from the TaskSet to the Service I get the error "CreateService error: LoadBalancers must be empty or null" — Josh, Dec 16 '20 at 21:40

score 0 · Answer 1 · answered Dec 16 '20 at 00:54

The error is correct, health checks at the Service level, will need a Load Balancer. However, you can define at the task level, health checks:

  EscTaskDef:
    Type: 'AWS::ECS::TaskDefinition'
    Properties:
...
      ContainerDefinitions:
        - Name: !Ref AWS::StackName
          Image: !Ref Image
          PortMappings:
            - ContainerPort: 80

          HealthCheck:
            Command: [ "CMD-SHELL", "curl -f http://localhost:80/api/healthcheck/ping || exit 1" ]
            StartPeriod: 200

Here is an example of a rollback example using this type of health check with Cloudformation.

Thanks, @Derrops, this might be a good workaround. However, my service does use a load balancer so I believe the service level health check should work. I will post a full minimum template today. — Josh, Dec 16 '20 at 11:58

Cloudformation Blue/Green Deployment HealthCheckGracePeriodSeconds

Full Example

1 Answers1