6

I am running fluent-bit as a sidecar on my EKS cluster for an application to tail application log files and write events to CloudWatch Logs. Through CloudWatch Logs Insights, I then set up some queries and dashboards to analyze those logs. This all works fine.

I have some older logs over the past week from before I was able to get this setup working. In fluent-bit tail input, it has an option to read new files entirely from the top if it discovers a new file. Using this option, I was able to get the older logs loaded into CloudWatch Logs in the same log group as the up-to-the minute log events.

If I go into the AWS console and into my log group, I can see all of the log streams listed. I can click into each one and see the events and search through them. All looks right. However, when I try to use Insights to query the older streams, no results appear. I have verified that I set a time period for my query that should include the events.

When I run this query, I get no results:

filter @logStream = 'myfile.log'
 | fields @timestamp, @message

Do log events with older timestamps not automatically get pulled into Insights? Is there a long delay before that data becomes available? I don't see anything in the documentation about it.

Mike
  • 1,791
  • 1
  • 17
  • 23

1 Answers1

5

If the event appears in Log groups, but doesn't appear in Log Insights. Did you use the Amazon CloudWatch Logs API PutLogEvents and inject logs with older timestamp ? If yes. You can't view the log Insights events that are previous to the log group creation. Try inject events with timestamp newer than the log group creation time.

Ida Amit
  • 1,411
  • 2
  • 13
  • 27