I am running fluent-bit as a sidecar on my EKS cluster for an application to tail application log files and write events to CloudWatch Logs. Through CloudWatch Logs Insights, I then set up some queries and dashboards to analyze those logs. This all works fine.
I have some older logs over the past week from before I was able to get this setup working. In fluent-bit tail input, it has an option to read new files entirely from the top if it discovers a new file. Using this option, I was able to get the older logs loaded into CloudWatch Logs in the same log group as the up-to-the minute log events.
If I go into the AWS console and into my log group, I can see all of the log streams listed. I can click into each one and see the events and search through them. All looks right. However, when I try to use Insights to query the older streams, no results appear. I have verified that I set a time period for my query that should include the events.
When I run this query, I get no results:
filter @logStream = 'myfile.log'
| fields @timestamp, @message
Do log events with older timestamps not automatically get pulled into Insights? Is there a long delay before that data becomes available? I don't see anything in the documentation about it.