Not able to map csv file from logstash to kibana in Window

Question

I'm trying to feed data in csv files into elastic search using logstash. My logsatsh config file looks like this:

input {
file {
path => "D:\Log Anlyser\data\cars.csv"
start_position => "beginning"
sincedb_path => "NUL"   
}
}
filter {
csv {
separator => ","
columns => [ "maker", "model", "mileage", "manufacture_year", "engine_displacement", "engine_power", "body_type", "color_slug", "stk_year", "transmission", "door_count", "seat_count", "fuel_type", "date_created", "date_last_seen", "price_eur" ]
}
mutate {convert => ["milage", "integer"] }
mutate {convert => ["price_eur", "float"] }
mutate {convert => ["engine_power", "integer"] }
mutate {convert => ["door_count", "integer"] }
mutate {convert => ["seat_count", "integer"] }  
}
output {
elasticsearch {
hosts => ["localhost:9200"]
index => ["cars-%{+YYYY.MM.dd}"]
}
}

while firing this command for logstash in window : logstash -f cars.conf i am getting this:-

Sending Logstash logs to D:/Log_Anlyser/logstash/logs which is now configured via log4j2.properties
[2019-02-26T12:05:51,690][WARN ][logstash.config.source.multilocal] Ignoring the 'pipelines.yml' file because modules or command line options are specified
[2019-02-26T12:05:51,721][INFO ][logstash.runner          ] Starting Logstash {"logstash.version"=>"6.6.1"}
[2019-02-26T12:05:57,133][INFO ][logstash.pipeline        ] Starting pipeline {:pipeline_id=>"main", "pipeline.workers"=>4, "pipeline.batch.size"=>125, "pipeline.batch.delay"=>50}
[2019-02-26T12:05:57,510][INFO ][logstash.outputs.elasticsearch] Elasticsearch pool URLs updated {:changes=>{:removed=>[], :added=>[http://localhost:9200/]}}
[2019-02-26T12:05:57,664][WARN ][logstash.outputs.elasticsearch] Restored connection to ES instance {:url=>"http://localhost:9200/"}
[2019-02-26T12:05:57,711][INFO ][logstash.outputs.elasticsearch] ES Output version determined {:es_version=>5}
[2019-02-26T12:05:57,742][INFO ][logstash.outputs.elasticsearch] New Elasticsearch output {:class=>"LogStash::Outputs::ElasticSearch", :hosts=>["//localhost:9200"]}
[2019-02-26T12:05:57,758][INFO ][logstash.outputs.elasticsearch] Using mapping template from {:path=>nil}
[2019-02-26T12:05:57,852][INFO ][logstash.outputs.elasticsearch] Attempting to install template {:manage_template=>{"template"=>"logstash-*", "version"=>50001, "settings"=>{"index.refresh_interval"=>"5s"}, "mappings"=>{"_default_"=>{"_all"=>{"enabled"=>true, "norms"=>false}, "dynamic_templates"=>[{"message_field"=>{"path_match"=>"message", "match_mapping_type"=>"string", "mapping"=>{"type"=>"text", "norms"=>false}}}, {"string_fields"=>{"match"=>"*", "match_mapping_type"=>"string", "mapping"=>{"type"=>"text", "norms"=>false, "fields"=>{"keyword"=>{"type"=>"keyword", "ignore_above"=>256}}}}}], "properties"=>{"@timestamp"=>{"type"=>"date", "include_in_all"=>false}, "@version"=>{"type"=>"keyword", "include_in_all"=>false}, "geoip"=>{"dynamic"=>true, "properties"=>{"ip"=>{"type"=>"ip"}, "location"=>{"type"=>"geo_point"}, "latitude"=>{"type"=>"half_float"}, "longitude"=>{"type"=>"half_float"}}}}}}}}
[2019-02-26T12:05:58,179][INFO ][logstash.pipeline        ] Pipeline started successfully {:pipeline_id=>"main", :thread=>"#<Thread:0x274079d5 run>"}
[2019-02-26T12:05:58,226][INFO ][logstash.agent           ] Pipelines running {:count=>1, :running_pipelines=>[:main], :non_running_pipelines=>[]}
[2019-02-26T12:05:58,226][INFO ][filewatch.observingtail  ] START, creating Discoverer, Watch with file and sincedb collections
[2019-02-26T12:05:58,547][INFO ][logstash.agent           ] Successfully started Logstash API endpoint {:port=>9600}

Now While connecting to kibana(localhost:5601) i am not able to map the data. Getting this error:-

Unable to fetch mapping. Do you have indices matching the pattern? Can you please help.

It seems that you got the path wrong. In your config I see `D:\Log Anlyser` while in the log I see `D:/Log_Anlyser`. Could you be missing an underscore there? — Val, Feb 26 '19 at 07:08
You might want to take a look at this question: https://stackoverflow.com/a/53596805/6113627. The issue in your configuration would be that the elasticseach output is missing `manage_template => false` — baudsp, Feb 26 '19 at 10:49

score 3 · Accepted Answer · answered Feb 27 '19 at 07:47

3

I got the problem. Mistake is very silly. Path of CSV file was wrong. Earlier path was path => "D:\Log Anlyser\data\cars.csv". Current Path is`

path => "D:/Log_Anlyser/data/cars.csv"

It will work

answered Feb 27 '19 at 07:47

Sanjiv

980
2
11
29

score 0 · Answer 2 · answered Feb 26 '19 at 08:26

0

There might be few reasons - maybe the data is not reaching ES at all. you can check that by verifying the index exists, by running

GET es-url:9200/_cat/indices/cars*

If an index exists then you should be able to create the index pattern in Kibana.

If the index is missing then either Logstash is not reading the input file, or elasticsearch is not reachable. need to check logstash logs, and make sure data reaches ES.

answered Feb 26 '19 at 08:26

Opster Elasticsearch Expert

1,837
1
15
18

:- In logstash log This is showing;- `[2019-02-26T10:58:39,257][WARN ][logstash.config.source.multilocal] Ignoring the 'pipelines.yml' file because modules or command line options are specified` – Swaraj Feb 26 '19 at 08:37
looks like you are using cars.conf and not pipelines.yml, so that is ok. but I'm not sure your LS config is OK. maybe for troubleshoot replace the ES output with stdout output, in order to verify that data is being read and processed in LS as expected. the ES output looks ok – Opster Elasticsearch Expert Feb 26 '19 at 08:42
Same problem. Is this line ok..? `sincedb_path => "NUL"` – Swaraj Feb 26 '19 at 08:46
I don't think it is ok. if you want LS to start from the begging every time, it should be: sincedb_path => "/dev/null" – Opster Elasticsearch Expert Feb 26 '19 at 09:01
getting this error `D:/Log_Anlyser/logstash/logstash-core/lib/logstash/pipeline.rb:426:in `inputworker' D:/Log_Anlyser/logstash/logstash-core/lib/logstash/pipeline.rb:420:in `block in start_input' [2019-02-26T14:40:49,418][INFO ][filewatch.observingtail ] START, creating Discoverer, Watch with file and sincedb collections [2019-02-26T14:40:49,418][ERROR][logstash.pipeline ] A plugin had an unrecoverable error. Will restart this plugin` – Swaraj Feb 26 '19 at 09:10
sincedb_path just need to be a file path, so you can choose whichever path that leads to a valid file. I haven't notice it is a windows machine, so you might need a different path, but as long as it is a path to a valid file (with write permissions) then you are fine – Opster Elasticsearch Expert Feb 26 '19 at 09:19
:- which path i have to give for window. ?? – Swaraj Feb 26 '19 at 09:35
sorry for the confusion, you can revert it back to sincedb_path => "NUL". I would debug by checking the output looks as expected - by adding/changing the output to stdout and watching the result documents in the screen. if the output looks right you can move forward and check why indices are not created in elasticsearch. https://www.elastic.co/guide/en/logstash/current/plugins-outputs-stdout.html – Opster Elasticsearch Expert Feb 26 '19 at 10:15
:- Okay.. Thanks for help – Swaraj Feb 26 '19 at 11:06
If the stdout output does not display the events, that means the events are not read from the input. There's an issue either in the input file config/location/structure or an issue with the input compatibility with the CSV filter. I would replace them one at a time until a pipeline is working and re-add them one by one in order to find the issue – Opster Elasticsearch Expert Feb 27 '19 at 06:26

Not able to map csv file from logstash to kibana in Window

2 Answers2