0

Is it possible to use CloudWatch Logs Agent to push data to another endpoint other CloudWatch Log Group, for example , is it possible to configure CloudWatch Logs Agent to push data to Kafka or RabbitMQ ?

Secondly , which protocol does the CloudWatch Logs Agent uses to send data, HTTP or HTTPS ?

Shivkumar Mallesappa
  • 2,875
  • 7
  • 41
  • 68

2 Answers2

1

Short answer: no, because the logs agent uses code from the AWS CLI, and it's targeted to AWS services.

Longer answer: have you considered the Kinesis Logs Agent?

Or, if you really want Kafka or RabbitMQ, use Logstash with an appropriate plugin.

guest
  • 26
  • 1
0

No You not able to directly send logs from Cloudwatch logs to non AWS Service.

But you can do it using log subscription services. Model is

Using Log Subscription service send the logs to Kinesis Stream.

In Lambda set kinesis stream as trigger. So you could process the records in lambda program and insert into Kafka or RabbitMQ.

Advantages:-

You can Send Multiple AWS Account logs in to Kinesis Stream and Process the records in Single AWS account

Mohan Shanmugam
  • 644
  • 1
  • 6
  • 18
  • Is it possible to push these events to EC2 instance or another Cloudwatch Log Agent ? – Shivkumar Mallesappa Feb 23 '18 at 05:51
  • Two cloudwatch logs are in same account or different account? what about regions? – Mohan Shanmugam Feb 23 '18 at 05:57
  • The cloudwatch logs which you try to send another EC2 instance or Cloudwatch logs originate source is from AWS Service or Non AWS Service. For ex: Application logs or aws logs(Flow logs, s3 access logs etc) – Mohan Shanmugam Feb 23 '18 at 06:00
  • The scenario is I have instances in private subnet they don't have access to internet. So is it possible to configure my instances in private subnet in such a way that they will push data to instances in public subnet or another cloudwatch instance in public subnet and from there I will push data to CloudWatch. I hope I am clear. – Shivkumar Mallesappa Feb 23 '18 at 06:07
  • If you attached Nat Gateway you can send logs from private subnet instance to Cloud watch logs using cloudwatch log agent else use logstash plugin and swrite logs to s3(using vpc endpoint) and process the logs. – Mohan Shanmugam Feb 23 '18 at 06:43
  • Ok. Thanks for your support, will check how to implement this. – Shivkumar Mallesappa Feb 23 '18 at 06:56