AWS ALB + ECS + gRPC + NestJS, '14 UNAVAILABLE: Trying to connect an http1.x server'

Question

Here is the thing:

We have NestJS microservices talking through gRPC. We deploy it to AWS ECS on Fargate.

In the current (working) configuration we're using one instance per service and a CloudMap for service discovery.

Now we want to have multiple instances per service. We've tried to configure the ALB.

In the target group, we see targets in a "Healthy" state.
But when we try to call services through the Load Balancer we receive '14 UNAVAILABLE: Trying to connect an http1.x server'

Any ideas?

Here is the part of our CDK code that is responsible for creating the load balancer:

export class ServicePrivateStack extends ServiceFargateStack<PrivateServiceStackProps> {
  constructor(scope: Construct, id: string, props: PrivateServiceStackProps) {
    super(scope, id, props);

    const { discoveryStack, vpc, securityGroup, certificateArn } = props;
    const { stagedName, service } = this;

    // create target group
    const grpcTG = new elbv2.ApplicationTargetGroup(this, 'TG_GRPC', {
      vpc,
      port: 50051,
      targetGroupName: this.makeTGName('GRPC'),
      deregistrationDelay: Duration.seconds(10),
      targetType: elbv2.TargetType.IP,
      protocol: elbv2.ApplicationProtocol.HTTPS,
      protocolVersion: elbv2.ApplicationProtocolVersion.GRPC,
      healthCheck: {
        enabled: true,
        protocol: Protocol.HTTPS,
        port: '50051',
        path: '/grpc.health.v1.Health/Check',    // <— custom, it works
        healthyGrpcCodes: ‘1’,
      },
    });
    grpcTG.addTarget(service);

    // create ALB
    const grpcLB = new elbv2.ApplicationLoadBalancer(this, 'LB_GRPC', {
      vpc,
      securityGroup,
      internetFacing: false,
      http2Enabled: true,
      loadBalancerName: `${stagedName}-LB`,
    });

    // attach grpc listener
    grpcLB.addListener('Listener_GRPC', {
      port: 50051,
      open: true,
      certificates: [{ certificateArn }],
      protocol: elbv2.ApplicationProtocol.HTTPS,
      defaultTargetGroups: [grpcTG],
    });

    // register in cloudmap
    discoveryStack.registerLoadBalancer(NameUtils.serviceDiscoveryName(id), grpcLB);
  }
  // <…>
}

score 0 · Answer 1 · answered Jun 14 '21 at 12:46

You need to set the protocol version to use gRPC explicitly as mentioned in the @aws-cdk/aws-elasticloadbalancingv2 module documentation:

const tg = new elbv2.ApplicationTargetGroup(stack, 'TG', {
  targetType: elbv2.TargetType.IP,
  port: 50051,
  protocol: elbv2.ApplicationProtocol.HTTP,
  protocolVersion: elbv2.ApplicationProtocolVersion.GRPC,
  healthCheck: {
    enabled: true,
    healthyGrpcCodes: '0-99',
  },
  vpc,
});

AWS ALB + ECS + gRPC + NestJS, '14 UNAVAILABLE: Trying to connect an http1.x server'

1 Answers1

Linked