AWS ECS Fargate Tasks with Spring Boot caught in restart-loop - How to configure LoadBalancer TargetGroup health checks with Pulumi?

Question

I created a simple example Pulumi TypeScript program that should deploy a Spring Boot application into a AWS ECS Fargate Cluster. The Spring Boot app is containerized/Dockerized with the help of Cloud Native Buildpacks/Paketo.io and published to the GitHub Container Registry at ghcr.io/jonashackt/microservice-api-spring-boot (example project here).

I've read through some Pulumi tutorials and started with the usual pulumi new aws-typescript. I now have the following index.ts:

import * as awsx from "@pulumi/awsx";

// Create a load balancer to listen for requests and route them to the container.
let loadbalancer = new awsx.lb.ApplicationListener("alb", { port: 8098, protocol: "HTTP" });

// Define Container image published to the GitHub Container Registry
let service = new awsx.ecs.FargateService("microservice-api-spring-boot", {
    taskDefinitionArgs: {
        containers: {
          microservice_api_spring_boot: {
                image: "ghcr.io/jonashackt/microservice-api-spring-boot:latest",
                memory: 768,
                portMappings: [ loadbalancer ],
            },
        },
    },
    desiredCount: 2,
});

// Export the URL so we can easily access it.
export const apiUrl = loadbalancer.endpoint.hostname;

After selecting the dev stack, a normal pulumi up runs through and provides me with the ApplicationLoadBalancer URL. Here's also a asciicast I prepared to show everything runs smooth:

My problem now is that the Fargate Services are stopped and started constantly. I've looked into CloudWatch logs and I see the Spring Boot apps starting - and beeing stopped after a few seconds again. I already checked the ApplicationLoadBalancer's TargetGroup and I see the Registered Targets becoming unhealthy again and again. How do I fix that?

Answer 1

The default AWS TargetGroup HealthCheckPath is simply / (see the docs). And as a standard Spring Boot application often responds with a HTTP 404 like this:

the ApplicationLoadBalancers health checks Status inside the TargetGroups go to unhealthy, thus triggering a restart of the Fargate Services.

How do we solve this? In Spring Boot you would normally use the spring-boot-actuator. Adding it to your pom.xml the application responds to localhost:yourPort/actuator/health:

    <dependency>
        <groupId>org.springframework.boot</groupId>
        <artifactId>spring-boot-starter-actuator</artifactId>
    </dependency>

So we need to configure the Pulumi created TargetGroup to use the health check path /actuator/health instead of /.

The Pulumi docs tell us how to Manually Configure Target Groups, but how could these exactly be integrated into the TypeScript code? The answer is hidden inside the @pulumi/awsx/lb docs! The example code from the Pulumi tutorial does multiple things from the one line let loadbalancer = new awsx.lb.ApplicationListener("alb", { port: 8098, protocol: "HTTP" });:

1. It creates an ApplicationLoadBalancer
2. It creates a matching TargetGroup
3. It creates a matching ApplicationListener

We simply need to create every component manually, because this way we can configure the healthCheck: path property of the TargetGroup:

import * as awsx from "@pulumi/awsx";

// Spring Boot Apps port
const port = 8098;

// Create a ApplicationLoadBalancer to listen for requests and route them to the container.
const alb = new awsx.lb.ApplicationLoadBalancer("fargateAlb");

// Create TargetGroup & Listener manually (see https://www.pulumi.com/docs/reference/pkg/nodejs/pulumi/awsx/lb/)
// so that we can configure the TargetGroup HealthCheck as described in (https://www.pulumi.com/docs/guides/crosswalk/aws/elb/#manually-configuring-target-groups)
// otherwise our Spring Boot Containers will be restarted every time, since the TargetGroup HealthChecks Status always
// goes to unhealthy
const albTargetGroup = alb.createTargetGroup("fargateAlbTargetGroup", {
    port: port,
    protocol: "HTTP",
    healthCheck: {
        // Use the default spring-boot-actuator health endpoint
        path: "/actuator/health"
    }
});

const albListener = albTargetGroup.createListener("fargateAlbListener", { port: port, protocol: "HTTP" });

// Define Container image published to the GitHub Container Registry
const service = new awsx.ecs.FargateService("microservice-api-spring-boot", {
    taskDefinitionArgs: {
        containers: {
            microservice_api_spring_boot: {
                image: "ghcr.io/jonashackt/microservice-api-spring-boot:latest",
                memory: 768,
                portMappings: [ albListener ]
            },
        },
    },
    desiredCount: 2,
});

// Export the URL so we can easily access it.
export const apiUrl = albListener.endpoint.hostname;

Now with this configuration our Fargate Services should become healty once they're started. And we should be able to see this inside the ALB's TargetGroup in the AWS console: