Stream AWS cloudwatch logs to elasticcloud using functionbeat

Question

I can not find a good document which gives details about how to stream data from AWS cloudwatch to elastic cloud. I have set it up as follows for now, but I cannot see data in elastic cloud index I have installed functionbeat locally and updated the config as follows:

functionbeat.provider.aws.endpoint: "s3.amazonaws.com"
functionbeat.provider.aws.deploy_bucket: "filebeat-deploy"
functionbeat.provider.aws.functions:
  - name: cloudwatch
    enabled: true
    type: cloudwatch_logs
    description: "lambda function for cloudwatch logs"
    triggers:
      - log_group_name: my_log_group_name
cloud.id: "<cloud_id>"
cloud.auth: "<username:password>"

I followed this document - https://www.elastic.co/guide/en/beats/functionbeat/current/configuration-functionbeat-options.html

And then I ran ./functionbeat deploy cloudwatch to deploy the function. I have checked I can see the deployment in the bucket filebeat-deploy I can not see the logs from my_log_group_name in elastic cloud

Answer 1

This is possibly because AWS is not able to make a successful connection with your elastic cloud. Usually protocol issue (if u are making output.elasticsearch: host as localhost:9200 because AWS doesn't able to reach to this localhost url unless it is a public one) or permission issue. If you check the functionbeat lambda function cloudwatch logs you can able to see the actual issue. Put logging.level: debug in functionbeat.yml for detailed logs.

Also, you cannot see the logs in kibana right after deploying the functionbeat. Once the subscription filter has added to the log group after the successful deployment you have to invoke the function which you have added subscription filter not the functionbeat lambda function. Because the trigger is added to the functionbeat lambda function.In your case you are added trigger to get logs from this log group "my_log_group_name". So whenever a new item get added into this log group then it will automatically invoke functionbeat lambda function.