aws fargate and firelens with fluentbit, ES output plugin TCP connection failed randomly

Question

I am running AWS ECS task with Fargate type with Firelens and custom Fluentbit configuration(attached below).

While i could run successfully, when i re-deploy tasks(update ECS Service by increasing number of tasks etc.), very often there are tasks which fail to TCP to Elasticsearch with fluentbit error message below

[error] [io] TCP connection failed: 199.12.31.400:9200 (Connection timed out)

i have no idea even how to troubleshoot this... anyone can share me some ideas? guess there is issue on AWS when switching network w.e ... ?

public IP on ECS Service is enabled.

Elasticsearch IP address i am using is private IP address in same VPC with ECS Service.

Elasticsearch is deployed in EC2, not managed service.

[SERVICE]
    Parsers_File /fluent-bit/parsers/parsers.conf
    Flush 1
    Grace 30
    Log_Level debug

[FILTER]
    Name         parser
    Match        *
    Key_Name     log
    Parser       json
    Reserve_Data True

[FILTER]
    Name modify
    Match *
    Remove container_id
    Remove container_name
    Remove container_name
    Remove ecs_cluster
    Remove ecs_task_arn
    Remove source

[OUTPUT]
    Name   stdout
    Match  *

[OUTPUT]
    Name            es
    Match           *
    Host            199.12.31.400
    Port            9200
    Logstash_Format On
    Logstash_Prefix logstash
    Trace_Output    On
    Type            _doc

ec2 security group allows access to the vpc cidr on port 9200? — karthikeayan, Jun 01 '20 at 09:30
I doubt if firelens container includes elasticsearch plugin, even the firelens example page doesn't have a example for elasticsearch... https://github.com/aws-samples/amazon-ecs-firelens-examples — karthikeayan, Jun 01 '20 at 09:51
fyi, i am using managed elasticsearch, i push my logs to firehose and then from firehose to elasticsearch.. — karthikeayan, Jun 01 '20 at 09:52
i am trying to double check cidr block filtering, maybe someone in the team changed that config..... hmmm ... btw elasticsearch plugin is included by default on fluentbit or at least aws-fluentbit docker image. i have to use existing elasticsearch which our team has used for while : ( — SallGood, Jun 01 '20 at 10:12

score 0 · Answer 1 · answered Jun 08 '20 at 06:02

0

at the end, it was issue of CIDR filtering ... thanks for @kathikeayan for letting me to do double check.

answered Jun 08 '20 at 06:02

SallGood

17
3

aws fargate and firelens with fluentbit, ES output plugin TCP connection failed randomly

1 Answers1