0

I want to save CSV data in Elasticsearch using Logstash to receive the following result:

"my_field": [{"col1":"AAA", "col2": "BBB"},{"col1":"CCC", "col2": "DDD"}]

So, it's important that CSV data gets saved as the array [...] in a specific document.

However, I get this result:

      "path": "path/to/csv",
      "@timestamp": "2017-09-22T11:28:59.143Z",
      "@version": "1",
      "host": "GT-HYU",
      "col2": "DDD",
      "message": "CCC,DDD",
      "col1": "CCC"

It looks like only the last CSV row gets saved (because of overwriting). I tried to use document_id => "1" in Logstash, but it obviously provokes the overwriting. How can I save data in the array? Also, I don't understand how to define that the data gets saved in my_field.

input {
    file {
        path => ["path/to/csv"]
        sincedb_path => "/dev/null"
        start_position => beginning
    }
}

filter {  
    csv {
        columns => ["col1","col2"]
        separator => ","
    }
    if [col1] == "col1" {
    drop {}
    }
}

output {
    stdout { codec => rubydebug }
    elasticsearch {
        action => "update"
        hosts => ["127.0.0.1:9200"]
        index => "my_index"
        document_type => "my_type"
        document_id => "1"
        workers => 1
    }
}
Dinosaurius
  • 8,306
  • 19
  • 64
  • 113

0 Answers0