Stack Overflow
Stack Overflow

Questions tagged [firelens]

16 questions
4
votes
0 answers

Not receiving logs when instance is scaled down with AWS firelens

I'm not receiving logs when some autoscaling event happens and in that time period all the logs are lost. Basically logs related to graceful shutdown. I'm using Elastic Beanstalk with ECS and for my logging stack I'm using Grafana's Loki. So I have…
lowkey'd
  • 41
  • 1
4
votes
1 answer

Fargate container_definition field "secretOptions" not passsing datadog API_KEY to logConfiguration

I'm trying to send my ECS Fargate logs to Datadog. To do this I need to pass my Datadog API_KEY as a field in the logConfiguration object. I need to secure my API_KEY so I am using AWS Secrets Manager via the secretOptions key of the…
Sogun
  • 301
  • 4
  • 10
2
votes
1 answer

How to disable JSON format and send only the log message to Cloudwatch with Fluentbit?

I am trying to setup Firelens for my Fargate tasks. I would like to send logs to multiple locations, Cloudwatch and Elasticsearch. But only to Cloudwatch I want to disable JSON format and send only the log message as it is. I have the below…
karthikeayan
  • 4,291
  • 7
  • 37
  • 75
2
votes
1 answer

AWS ECS Fargate, send logs to multiple destinations (CloudWatch Logs and Elasticsearch)

I have some containers deployed in ECS Fargate, that send the logs to Cloudwatch logs. Now, I want to send also the logs to a custom Elasticsearch instance (not Amazon Elasticsearch Service). I have read some info about firelens, but it is not clear…
okelet
  • 716
  • 1
  • 8
  • 23
2
votes
1 answer

How to configure AWS CDK ApplicationLoadBalancedFargateService to log parsed JSON lines with Firelens and Firebit

When I create an ApplicationLoadBalancedFargateService with a Firelens logdriver, and the application writes JSON lines as the log message, such as when using net.logstash.logback.encoder.LoggingEventCompositeJsonEncoder with Logback, the log…
Richard Collette
  • 5,462
  • 4
  • 53
  • 79
1
vote
0 answers

How can I route ECS log to S3 with using AWS Firelens

I am making ECS Service withe Terraform. Now I have to route my ECS Service log to S3 my bucket. I looked it up and they told me to use AWS Firelens. But It dosen't work. Here is my setting. resource "aws_ecs_task_definition"…
Juzzn
  • 41
  • 2
1
vote
2 answers

Awslogs log driver vs AWS Firelens - which one to use and when and cost factor

I am planning to migrate our app to AWS Fargate and so want to set up logging for the same as well and store all the logs in cloudwatch. I could see we have two options in Fargate - either use default awslogs log driver or use AWS Firelens to gather…
Nin
  • 99
  • 1
  • 8
1
vote
0 answers

error io_tls flb_io_tls.c:359 NET - Connection was reset by peer

[2020/12/10 11:44:42] [ warn] net_tcp_fd_connect: getaddrinfo(host='https://abc.amazonaws.com'): Name or service not known [2020/12/10 11:44:42] [error] [io_tls] flb_io_tls.c:359 NET - Connection was reset by peer [2020/12/10 11:44:42] [ warn]…
Paresh Sahoo
  • 127
  • 1
  • 2
  • 6
0
votes
0 answers

Add custom metadata to AWS logs (using firelens for logging and Kinesis Firehose, AWS CDK)

Current setup is this : ECS Task with 2 containers, App Container Logging Container (fluentbit, created by default after using the firelens logging) The logs generated from the App container are sent to a Kinesis Firehose Stream , and then this…
boiken
  • 1
  • 1
0
votes
0 answers

docker build failed to load cache key: pull access denied, repository does not exist

I am trying to build aws/aws-for-fluent-bit and create ECR image. But I download from aws/aws-for-fluent-bit github and build by docker and get docker build error.I don't know how to solve this error. `=> CANCELED [builder 5/33] RUN yum upgrade -y …
Khin Thu Thu Mon
  • 1
  • 1
0
votes
2 answers

How to disable JSON format and send only the log message to Sumologic with Fluentbit?

We are using Fluentbit as as Sidecar container in our ECS fargate Cluster which is running a dotnet application, initially we faced the issue of fluentbit sending the logs in multiline and we solved it using Fluentbit Multilne feature. Now the logs…
Shalmal
  • 494
  • 4
  • 7
0
votes
0 answers

How do certain input plugins know which parser to use?

I am working with AWS FireLense using Fluent Bit to forward logs from ECS. There is a parsers file with a JSON parser, and the inputs are dynamically generated by the AWS FireLense image. The generated inputs are both of type forward. Outside of the…
Calvin
  • 111
  • 1
  • 1
  • 7
0
votes
1 answer

Push ECS Fargate Container logs directly to S3 bucket

I have hosted an ECS Fargate cluster that is using the AWS logs driver and pushing logs to CloudWatch, no issues on the awslog driver. Now what I want to do is push the logs directly from the ECS cluster to an S3 bucket. So I am expecting to use AWS…
sam
  • 11
  • 2
0
votes
0 answers

Firelens/Fluentbit Multiple Output based on log regex

I'm starting to use firelens/fluentbit and I'm trying to send my ECS task logs to different outputs (Opensearch index) based on some tag or string of the log. I saw the example of multiple destinations, and I was able to filter the logs by regex,…
Jocheinfa
  • 25
  • 1
  • 4
0
votes
1 answer

log producer docker container

I am trying to do performance testing with aws firelens. I have a json file with 10 sample log messages. I want to be able to produce docker logs at a set rate. ex: 10,000 log messages/sec from a docker container that will be consumed by aws…
fledgling
  • 991
  • 4
  • 25
  • 48
1
2