I want to save CSV data in Elasticsearch using Logstash to receive the following result:
"my_field": [{"col1":"AAA", "col2": "BBB"},{"col1":"CCC", "col2": "DDD"}]
So, it's important that CSV data gets saved as the array [...]
in a specific document.
However, I get this result:
"path": "path/to/csv",
"@timestamp": "2017-09-22T11:28:59.143Z",
"@version": "1",
"host": "GT-HYU",
"col2": "DDD",
"message": "CCC,DDD",
"col1": "CCC"
It looks like only the last CSV row gets saved (because of overwriting). I tried to use document_id => "1"
in Logstash, but it obviously provokes the overwriting. How can I save data in the array?
Also, I don't understand how to define that the data gets saved in my_field
.
input {
file {
path => ["path/to/csv"]
sincedb_path => "/dev/null"
start_position => beginning
}
}
filter {
csv {
columns => ["col1","col2"]
separator => ","
}
if [col1] == "col1" {
drop {}
}
}
output {
stdout { codec => rubydebug }
elasticsearch {
action => "update"
hosts => ["127.0.0.1:9200"]
index => "my_index"
document_type => "my_type"
document_id => "1"
workers => 1
}
}