0

I've awslogs setup and it ships log to cloudwatch. It works fine for a few hours and then suddenly it stops.

Here is the log from awslogs.log

2020-07-06 14:58:27,701 - cwlogs.push.reader - WARNING - 23093 - Thread-6 - Fall back to previous event time: {'timestamp': 1594062573000, 'start_position': 85848600L, 'end_position': 85848777L}, previousEventTime: 1594062573000, reason: timestamp could not be parsed from message.
2020-07-06 14:58:27,701 - cwlogs.push.batch - WARNING - 23093 - Thread-6 - Skip event: {'timestamp': 1594062573000, 'start_position': 85848600L, 'end_position': 85848777L}, reason: timestamp is more than 2 hours in future.
2020-07-06 14:58:27,701 - cwlogs.push.reader - WARNING - 23093 - Thread-6 - Fall back to previous event time: {'timestamp': 1594062573000, 'start_position': 85848777L, 'end_position': 85848952L}, previousEventTime: 1594062573000, reason: timestamp could not be parsed from message.
2020-07-06 14:58:27,701 - cwlogs.push.batch - WARNING - 23093 - Thread-6 - Skip event: {'timestamp': 1594062573000, 'start_position': 85848777L, 'end_position': 85848952L}, reason: timestamp is more than 2 hours in future.

Here's my configuration in /var/awslogs/etc/config/api.conf

[/var/log/app.js/api.log]
datetime_format = %Y-%m-%d %H:%M:%S
buffer_duration = 5000
log_stream_name = {hostname}
initial_position = end_of_file
log_group_name = app-js-logs-prod
file = /var/log/app.js/api.log

[/root/.pm2/pm2.log]
datetime_format = %Y-%m-%d %H:%M:%S
buffer_duration = 5000
log_stream_name = {hostname}
initial_position = end_of_file
log_group_name = pm2-logs-prod

I'm not able to find out why the logs are skipped. Any help will be greatly appreciated.

EDIT:

timedatectl output:

root@ip-10-0-5-68:/home/ubuntu# timedatectl
                      Local time: Mon 2020-07-06 15:18:42 UTC
                  Universal time: Mon 2020-07-06 15:18:42 UTC
                        RTC time: Mon 2020-07-06 15:18:43
                       Time zone: Etc/UTC (UTC, +0000)
       System clock synchronized: yes
systemd-timesyncd.service active: yes

My local time should be in IST. Is there something wrong with my system?

Reyansh Kharga
  • 356
  • 5
  • 18

1 Answers1

0

Before requests are sent to CloudWatch a check is being performed against the parsed log in your log(s) files against what the agent believes the time is.

You should first validate that the system time is correct (in its current timezone) and ensure that the logs date timezone matches the local timezone (this is the default).

If these match then give the cloudwatch agent a restart.

If that still does not fix then follow some of the suggestions on this previous post

Chris Williams
  • 32,215
  • 4
  • 30
  • 68
  • I saw the previous post link. That fixes the problem but I can't do that again and again manually. That's my problem. Is there a workaround I can go with. Maybe modifying my configuration file a bit. I am ready for some tradeoff. – Reyansh Kharga Jul 06 '20 at 15:15
  • I would assume that it is likely to be a system issue then, unfortunately cwlogs is actually a legacy product now so support may be minimal. The CloudWatch agent is its replacement: https://docs.aws.amazon.com/systems-manager/latest/userguide/monitoring-cloudwatch-agent.html. – Chris Williams Jul 06 '20 at 15:19
  • I've edited my question with timedatectl output. My local time zone is IST but it shows UTC. – Reyansh Kharga Jul 06 '20 at 15:23
  • I didn't update my timezone. By default it shows in UTC. – Reyansh Kharga Jul 06 '20 at 15:29
  • This is by default, it is generally best practice to run your servers in UTC. However if you want to update to IST you would need to run `timedatectl set-timezone Asia/Kolkata` – Chris Williams Jul 06 '20 at 15:33
  • no luck yet. :(. Isee it always falls back to the same timestamp. why? Fall back to previous event time: {'timestamp': 1594062573000, – Reyansh Kharga Jul 06 '20 at 15:44