1

On my EC2 RHEL instance I have the following awslogs.config in my /var/awslogs/etc directory. I'll cut out the top part and get right to the logging aspect in this code snip.

[/opt/apache-tomcat-8.0.26/logs/PND.log]
datetime_format = %Y-%m-%d %H:%M:%S
file = /opt/apache-tomcat-8.0.26/logs/PND.log
buffer_duration = 5000
log_stream_name = {instance_id}
initial_position = start_of_file
log_group_name = /opt/apache-tomcat-8.0.26/logs/PND.log
[/var/log/secure]
datetime_format = %Y-%m-%d %H:%M:%S
file = /var/log/secure
buffer_duration = 5000
log_stream_name = {instance_id}
initial_position = start_of_file
log_group_name = /var/log/secure
[/var/log/messages]
datetime_format = %b %d %H:%M:%S
file = /var/log/messages
buffer_duration = 5000
log_stream_name = {instance_id}
initial_position = start_of_file
log_group_name = /var/log/messages

Logs from /var/log/messages and /var/log/secure are making it to the AWS log console for CloudWatch but logs from /opt/apache-tomcat-8.0.26/logs/PND.log are not. My REST service is running on Tomcat.

When I ssh to the server I can see log entries streaming into /opt/apache-tomcat-8.0.26/logs/PND.log but nothing is showing up on AWS, however, from that same instance I can see all the log entries from messages and secure.

I checked out awslogs.log file and there are no "No file is found with given path" errors for "/opt/apache-tomcat-8.0.26/logs/PND.log" which makes me think it can find it. If I grep the file I get the following entries.

2015-10-14 16:33:12,585 - cwlogs.push.stream - INFO - 938 - Thread-1 - Starting reader for [xxxxxxxxxxx, /opt/apache-tomcat-8.0.26/logs/PND.log]

So if the file can be read, why aren't I seeing log entries?

I was wondering if perhaps, because the group existed before for another instance, if that somehow blocks the new entries for the new instance, but that doesn't make sense to me. Instances should be able to share groups, which is why we can view streams by instance id.

pjc
  • 240
  • 5
  • 14
  • did you try restarting the cwlogs agent? is the PND.logs a symlink? – Mircea Oct 14 '15 at 21:29
  • the group existing does not matter. – Mircea Oct 14 '15 at 21:30
  • last question: what timezone are the entities in the log in? it may be possible that they're pushed but you're looking at the wrong time window – Mircea Oct 14 '15 at 21:31
  • This can happen if the state file ends up in a bad state. See https://stackoverflow.com/a/41142996/14059 for details on trying to fix it. Otherwise, you can stop the agent, delete the state file, and start the agent. This could cause some data loss or duplication of logs, depending on your configuration. – alanc10n Jan 11 '18 at 17:14

1 Answers1

0

Today I could get this working after a big struggle.

Before this to work, you have to install few basic AWS stuffs in your EC2 instance

1. python
2. sudo pip install awscli
3. sudo pip install -U boto

  1. To configure awslogs I followed the instructions in https://docs.aws.amazon.com/AmazonCloudWatch/latest/logs/QuickStartEC2Instance.html

However log files were not getting posted in Cloud Watch Log.

I did 2 settings with the Credentials.

1. From IAM->users->Security Credentials I generated my 'Access Key Id' and 'Secret Access Key'. Then I set it in /etc/awslogs/awscli.conf along with the region.

2. also It was set in for~/.aws/credentials
In prompt Fire the command 'sudo aws configure'. Then set the 'Access Key Id' and 'Secret Access Key' along with the region.

3. I verified whether read permissions are there for others in the directory where logs are residing.

4. Last step, I specified the log file names with complete path in /var/awslogs/etc/awslogs.conf

Once the service is up and running with sudo service awslogs, verify whether errors are getting logged by fireing sudo tail /var/log/awslogs.log

With the above settings I got rid of all errors.

Then go to 'https://console.aws.amazon.com/cloudwatch/home?region=ap-south-1#logs:' and view the logs there.

Don't forget to fire sudo chkconfig awslogs on, this will make sure this service starts along with system restart.

Thank you

Robin Varghese
  • 1,158
  • 10
  • 22