-1

I have my ELK server setup on Ubuntu 16.xx (AWS EC2 instance). 1 - I need to monitor my AWS services (SNS, SQS, SES, Lambda, Ec2, S3 ELB CloudWatch .. etc.) 2 - how can I get the real-time logs from my aws service and post to my Kibana dashboard (is there any plugin/beat/integration api). are there community beats can help me to get the logs from AWS to ELK.

https://www.elastic.co/guide/en/beats/libbeat/current/community-beats.html

Thanks

shahid
  • 11
  • 6

4 Answers4

0

https://www.elastic.co/guide/en/logstash/current/plugins-inputs-cloudwatch.html

You probably want to use the Cloudwatch Logstash plugin and config it to pull that data into your elasticsearch, I believe is that is the easiest way to do this.

JamesKn
  • 1,035
  • 9
  • 16
0

You can use the below beats to retrieve cloudwatch logs and metrics.

cloudwatchmetricbeat - A beat for Amazon Web Services' CloudWatch Metrics.

cloudwatchlogsbeat - Reads log events from Amazon Web Services' CloudWatch Logs.

ben5556
  • 2,915
  • 2
  • 11
  • 16
  • Hi Thanks for your reply. The above CloudWatch beats where i have to setup ? 1 -on ELK server ( My ELK server is running on the Docker container) ? how can i setup ? there is an guide or steps to setup the beats and integrate it with AWS. – shahid Nov 05 '18 at 12:08
0

A few more options to stream AWS logs to the ELK cluster are:

1) First push all your AWS logs to a S3 bucket and then use Logstash's "S3 input plugin" to stream logs from S3 to logstash.

2) Use a lambda function to stream all the cloudwatch logs directly to your ELK cluster. refer this: https://github.com/blueimp/aws-lambda/tree/master/cloudwatch-logs-to-elastic-cloud

Shiv Rajawat
  • 898
  • 9
  • 21
0

The Elastic Stack 6.5 has been released on Nov 14th and a new beat called Functionbeat allows you to stream your Cloudwatch logs easily to ES.

Val
  • 207,596
  • 13
  • 358
  • 360