I'm currently running logstash in an ec2 instance with the default linux AMI, and attempting to send logs to an AWS ES instance. If I used the standard 'elasticsearch' output, I can send unsigned data to the AWS ES instance, but I'm trying to set up a prod-ready framework, and everything I've read recommends using the AWS labs logstash output plugin here( https://github.com/awslabs/logstash-output-amazon_es ).
I can confirm the plugin is installed, but when I run logstash using the below conf file, I get a message that 'Sending Logstash's logs to /var/log/logstash which is now configured via log4j2.properties', but no data appears in my elasticsearch endpoint with the '/_search?pretty=true' flag when I refresh after making a stdin entry.
input {
stdin {
}
}
output {
amazon_es {
hosts => ["https://search-secretstuff.es.amazonaws.com"]
region => "xxxxx"
aws_access_key_id => 'xxxxxx'
aws_secret_access_key => 'xxxxxx'
index => "prod-logs-%{+YYYY.MM.dd}"
template => "/etc/logstash/mappings/es6-template.json"
}
}In addition to using stdin, I've tried using a file input, ex
input {
file {
path => "/var/log/amazon/ssm/errors.log"
}
}The template I'm using is below, as per the accepted answer for this post (Logstash conf error - amazon_es)
{
"template" : "logstash-*",
"version" : 60001,
"settings" : {
"index.refresh_interval" : "5s"
},
"mappings" : {
"_default_" : {
"dynamic_templates" : [ {
"message_field" : {
"path_match" : "message",
"match_mapping_type" : "string",
"mapping" : {
"type" : "text",
"norms" : false
}
}
}, {
"string_fields" : {
"match" : "*",
"match_mapping_type" : "string",
"mapping" : {
"type" : "text", "norms" : false,
"fields" : {
"keyword" : { "type": "keyword", "ignore_above": 256 }
}
}
}
} ],
"properties" : {
"@timestamp": { "type": "date"},
"@version": { "type": "keyword"},
"geoip" : {
"dynamic": true,
"properties" : {
"ip": { "type": "ip" },
"location" : { "type" : "geo_point" },
"latitude" : { "type" : "half_float" },
"longitude" : { "type" : "half_float" }
}
}
}
}
}
}Does anything in the configuration jump out as a potential pain point? I've tried a number of iterations of both the template file and the logstash.conf file, and now feel like I'm beating my head against the wall to no avail.
