24

I'm doing "elastic search getting started" tutorial. Unfortunatelly this tutorial doesn't cover first step which is importing csv database into elasticsearch.

I googled to find solution but it doesn't work unfortunatelly. Here is what I want to achieve and what I have:

I have a file with data which I want to import (simplified)

id,title
10,Homer's Night Out
12,Krusty Gets Busted

I would like to import it using logstash. After research over the internet I end up with following config:

input {
    file {
        path => ["simpsons_episodes.csv"]
        start_position => "beginning"
    }
}

filter {
    csv {
        columns => [
            "id",
            "title"
        ]
    }
}

output {
    stdout { codec => rubydebug }
    elasticsearch {
        action => "index"
        hosts => ["127.0.0.1:9200"]
        index => "simpsons"
        document_type => "episode"
        workers => 1
    }
}

I have a trouble with specifying document type so once data is imported and I navigate to http://localhost:9200/simpsons/episode/10 I expect to see result with episode 10.

adelura
  • 556
  • 1
  • 5
  • 12

2 Answers2

16

Good job, you're almost there, you're only missing the document ID. You need to modify your elasticsearch output like this:

elasticsearch {
    action => "index"
    hosts => ["127.0.0.1:9200"]
    index => "simpsons"
    document_type => "episode"
    document_id => "%{id}"             <---- add this line
    workers => 1
}

After this you'll be able to query episode with id 10

GET http://localhost:9200/simpsons/episode/10
Val
  • 207,596
  • 13
  • 358
  • 360
  • That make sense. Sorry for additional question... Command which I execute is `logstash.bat -f '../simpsons_episodes.conf'` result from CMD is as follows: https://pastebin.com/raw/4LTttNDB and it doesn't seems to work yet. Do you think that I have to create index first? – adelura Apr 30 '17 at 10:17
  • It didn't work on windows cmd but worked on git bash. Thank you! – adelura Apr 30 '17 at 14:33
10

I'm the author of moshe/elasticsearch_loader
I wrote ESL for this exact problem.
You can download it with pip:

pip install elasticsearch-loader

And then you will be able to load csv files into elasticsearch by issuing:

elasticsearch_loader --index incidents --type incident csv file1.csv

Additionally, you can use custom id file by adding --id-field=document_id to the command line

MosheZada
  • 2,189
  • 1
  • 14
  • 17
  • I have imported a CSV with this plugin, But I can't see my imported data.. I don't know what's wrong. the data logged on terminal while it was importing but after then I can't find it with `index` – Code Cooker May 29 '20 at 15:52