I have been working on this Cloudwatch Log Agent for long time but never faced the below issue: Actually, I did some changes in the Cloudwatch agent JSON file:
/opt/aws/amazon-cloudwatch-agent/bin/config.json
After doing the changes the log export stopped to the Cloudwatch
{
"agent": {
"metrics_collection_interval": 60,
"run_as_user": "root"
},
"logs": {
"logs_collected": {
"files": {
"collect_list": [{
"file_path": "/var/lib/jenkins/jobs/**/builds/**/logs",
"log_group_name": "log_data",
"log_stream_name": "log_stream",
"retention_in_days": 3
}]
}
}
}
}
After running the command:
/opt/aws/amazon-cloudwatch-agent/bin/amazon-cloudwatch-agent-ctl -a fetch-config -m ec2 -c file:/opt/aws/amazon-cloudwatch-agent/bin/config.json -s
Command Output
***** processing amazon-cloudwatch-agent ******
/opt/aws/amazon-cloudwatch-agent/bin/config-downloader --output-dir /opt/aws/amazon-cloudwatch-agent/etc/amazon-cloudwatch-agent.d --download-source file:/opt/aws/amazon-cloudwatch-agent/bin/config.json --mode ec2 --config /opt/aws/amazon-cloudwatch-agent/etc/common-config.toml --multi-config default
I! Trying to detect region from ec2
D! [EC2] Found active network interface
Successfully fetched the config and saved in /opt/aws/amazon-cloudwatch-agent/etc/amazon-cloudwatch-agent.d/file_config.json.tmp
Start configuration validation...
/opt/aws/amazon-cloudwatch-agent/bin/config-translator --input /opt/aws/amazon-cloudwatch-agent/etc/amazon-cloudwatch-agent.json --input-dir /opt/aws/amazon-cloudwatch-agent/etc/amazon-cloudwatch-agent.d --output /opt/aws/amazon-cloudwatch-agent/etc/amazon-cloudwatch-agent.toml --mode ec2 --config /opt/aws/amazon-cloudwatch-agent/etc/common-config.toml --multi-config default
2022/11/23 06:37:55 Reading json config file path: /opt/aws/amazon-cloudwatch-agent/etc/amazon-cloudwatch-agent.d/file_config.json.tmp ...
2022/11/23 06:37:55 I! Valid Json input schema.
I! Detecting run_as_user...
I! Trying to detect region from ec2
D! [EC2] Found active network interface
No csm configuration found.
Configuration validation first phase succeeded
/opt/aws/amazon-cloudwatch-agent/bin/amazon-cloudwatch-agent -schematest -config /opt/aws/amazon-cloudwatch-agent/etc/amazon-cloudwatch-agent.toml
Configuration validation second phase succeeded
Configuration validation succeeded
Still not able to export the logs
Check for the issue also in the amazon-cloudwatch-agent.log Command:
cat /opt/aws/amazon-cloudwatch-agent/logs/amazon-cloudwatch-agent.log
Output
2022/11/22 19:22:35 Reading json config file path: /opt/aws/amazon-cloudwatch-agent/etc/amazon-cloudwatch-agent.json ...
2022/11/22 19:22:35 Reading json config file path: /opt/aws/amazon-cloudwatch-agent/etc/amazon-cloudwatch-agent.d/file_config.json ...
2022/11/22 19:22:36 I! Valid Json input schema.
2022/11/22 19:22:36 I! Detected runAsUser: root
2022/11/22 19:22:36 I! Changing ownership of [/opt/aws/amazon-cloudwatch-agent/logs /opt/aws/amazon-cloudwatch-agent/etc /opt/aws/amazon-cloudwatch-agent/var] to 0:0
2022-11-22T19:22:36Z I! Starting AmazonCloudWatchAgent 1.247355.0
2022-11-22T19:22:36Z I! AWS SDK log level not set
2022-11-22T19:22:36Z I! Loaded inputs: disk logfile mem
2022-11-22T19:22:36Z I! Loaded aggregators:
2022-11-22T19:22:36Z I! Loaded processors: ec2tagger
2022-11-22T19:22:36Z I! Loaded outputs: cloudwatch cloudwatchlogs
2022-11-22T19:22:36Z I! Tags enabled: host=ip-172-31-1-218
2022-11-22T19:22:36Z I! [agent] Config: Interval:1m0s, Quiet:false, Hostname:"ip-172-31-1-218", Flush Interval:1s
2022-11-22T19:22:36Z I! [processors.ec2tagger] ec2tagger: Check ec2 metadata
2022-11-22T19:22:36Z I! [logagent] starting
2022-11-22T19:22:36Z I! [logagent] found plugin cloudwatchlogs is a log backend
2022-11-22T19:22:36Z I! [logagent] found plugin logfile is a log collection
2022-11-22T19:22:36Z I! [processors.ec2tagger] ec2tagger: EC2 tagger has started initialization.
2022-11-22T19:22:36Z I! [processors.ec2tagger] ec2tagger: Check ec2 metadata
2022-11-22T19:22:36Z I! [processors.ec2tagger] ec2tagger: EC2 tagger has started initialization.
2022-11-22T19:22:36Z I! cloudwatch: get unique roll up list []
2022-11-22T19:22:36Z I! cloudwatch: publish with ForceFlushInterval: 1m0s, Publish Jitter: 1s
2022-11-22T19:22:36Z I! [processors.ec2tagger] ec2tagger: Initial retrieval of tags succeded
2022-11-22T19:22:36Z I! [processors.ec2tagger] ec2tagger: EC2 tagger has started, finished initial retrieval of tags and Volumes
2022-11-22T19:22:36Z I! [processors.ec2tagger] ec2tagger: Initial retrieval of tags succeded
2022-11-22T19:22:36Z I! [processors.ec2tagger] ec2tagger: EC2 tagger has started, finished initial retrieval of tags and Volumes
Attaching the old Logs also where you can see the actual difference from the same file:
022/10/11 06:58:04 Reading json config file path: /opt/aws/amazon-cloudwatch-agent/etc/amazon-cloudwatch-agent.json ...
2022/10/11 06:58:04 Reading json config file path: /opt/aws/amazon-cloudwatch-agent/etc/amazon-cloudwatch-agent.d/file_config.json ...
2022/10/11 06:58:04 I! Valid Json input schema.
2022/10/11 06:58:04 I! Detected runAsUser: root
2022/10/11 06:58:04 I! Changing ownership of [/opt/aws/amazon-cloudwatch-agent/logs /opt/aws/amazon-cloudwatch-agent/etc /opt/aws/amazon-cloudwatch-agent/var
] to 0:0
2022-10-11T06:58:04Z I! Starting AmazonCloudWatchAgent 1.247355.0
2022-10-11T06:58:04Z I! AWS SDK log level not set
2022-10-11T06:58:04Z I! Loaded inputs: disk logfile mem
2022-10-11T06:58:04Z I! Loaded aggregators:
2022-10-11T06:58:04Z I! Loaded processors: ec2tagger
2022-10-11T06:58:04Z I! Loaded outputs: cloudwatch cloudwatchlogs
2022-10-11T06:58:04Z I! Tags enabled: host=ip-172-31-1-218
2022-10-11T06:58:04Z I! [agent] Config: Interval:1m0s, Quiet:false, Hostname:"ip-172-31-1-218", Flush Interval:1s
2022-10-11T06:58:04Z I! [processors.ec2tagger] ec2tagger: Check ec2 metadata
2022-10-11T06:58:04Z I! [logagent] starting
2022-10-11T06:58:04Z I! [logagent] found plugin cloudwatchlogs is a log backend
2022-10-11T06:58:04Z I! [logagent] found plugin logfile is a log collection
2022-10-11T06:58:04Z I! [processors.ec2tagger] ec2tagger: EC2 tagger has started initialization.
2022-10-11T06:58:04Z I! [processors.ec2tagger] ec2tagger: Check ec2 metadata
2022-10-11T06:58:04Z I! [processors.ec2tagger] ec2tagger: EC2 tagger has started initialization.
2022-10-11T06:58:04Z I! cloudwatch: get unique roll up list []
2022-10-11T06:58:04Z I! cloudwatch: publish with ForceFlushInterval: 1m0s, Publish Jitter: 54s
2022-10-11T06:58:04Z I! [processors.ec2tagger] ec2tagger: Initial retrieval of tags succeded
2022-10-11T06:58:04Z I! [processors.ec2tagger] ec2tagger: EC2 tagger has started, finished initial retrieval of tags and Volumes
2022-10-11T06:58:04Z I! [processors.ec2tagger] ec2tagger: Initial retrieval of tags succeded
2022-10-11T06:58:04Z I! [processors.ec2tagger] ec2tagger: EC2 tagger has started, finished initial retrieval of tags and Volumes
2022-10-11T06:58:05Z I! [inputs.logfile] Reading from offset 841815 in /var/log/syslog
2022-10-11T06:58:05Z I! [logagent] piping log from Test/syslog(/var/log/syslog) to cloudwatchlogs with retention 3
2022-10-11T06:58:10Z I! [outputs.cloudwatchlogs] First time sending logs to Test/syslog since startup so sequenceToken is nil, learned new token:(0xc00090410
0): The given sequenceToken is invalid. The next expected sequenceToken is: 49623781346189785515230421563841207337059424902309742034
2022-10-11T06:58:10Z W! [outputs.cloudwatchlogs] Retried 0 time, going to sleep 112.557969ms before retrying.
2022-10-11T07:35:05Z W! [agent] ["outputs.cloudwatch"] did not complete within its flush interval
2022-10-11T07:35:13Z W! [agent] ["outputs.cloudwatch"] did not complete within its flush interval
2022-10-11T07:35:15Z W! [agent] ["outputs.cloudwatch"] did not complete within its flush interval
2022-10-11T07:35:16Z W! [agent] ["outputs.cloudwatchlogs"] did not complete within its flush interval
2022-10-11T07:35:17Z W! [agent] ["outputs.cloudwatchlogs"] did not complete within its flush interval
2022-10-11T07:35:18Z W! [agent] ["outputs.cloudwatch"] did not complete within its flush interval
2022-10-11T07:35:19Z W! [agent] ["outputs.cloudwatch"] did not complete within its flush interval
2022-10-11T07:35:19Z W! [agent] ["outputs.cloudwatchlogs"] did not complete within its flush interval
2022-10-11T07:35:22Z W! [agent] ["outputs.cloudwatch"] did not complete within its flush interval
2022/10/11 07:46:06 I! D! [EC2] Found active network interface
I! Detected the instance is EC2
2022/10/11 07:46:06 Reading json config file path: /opt/aws/amazon-cloudwatch-agent/etc/amazon-cloudwatch-agent.json ...
/opt/aws/amazon-cloudwatch-agent/etc/amazon-cloudwatch-agent.json does not exist or cannot read. Skipping it.
2022/10/11 07:46:06 Reading json config file path: /opt/aws/amazon-cloudwatch-agent/etc/amazon-cloudwatch-agent.d/file_config.json ...
2022/10/11 07:46:06 I! Valid Json input schema.
I! Detecting run_as_user...
I! Trying to detect region from ec2
No csm configuration found.
Configuration validation first phase succeeded
The difference I can spot between the old and the new logs are:
initial retrieval of tags and Volumes
2022-10-11T06:58:05Z I! [inputs.logfile] Reading from offset 841815 in /var/log/syslog
2022-10-11T06:58:05Z I! [logagent] piping log from Test/syslog(/var/log/syslog) to cloudwatchlogs with retention 3
2022-10-11T06:58:10Z I! [outputs.cloudwatchlogs] First time sending logs to Test/syslog since startup so sequenceToken is nil, learned new token:(0xc00090410
0): The given sequenceToken is invalid. The next expected sequenceToken is: 49623781346189785515230421563841207337059424902309742034
2022-10-11T06:58:10Z W! [outputs.cloudwatchlogs] Retried 0 time, going to sleep 112.557969ms before retrying.
2022-10-11T07:35:05Z W! [agent] ["outputs.cloudwatch"] did not complete within its flush interval
2022-10-11T07:35:13Z W! [agent] ["outputs.cloudwatch"] did not complete within its flush interval
2022-10-11T07:35:15Z W! [agent] ["outputs.cloudwatch"] did not complete within its flush interval
2022-10-11T07:35:16Z W! [agent] ["outputs.cloudwatchlogs"] did not complete within it
Please help me out resolving the issue because I am stuck on this for very long time and additionally I have also tried installing the agent again but it didn't helped.
