It doesn't show any error on console. Here is the command I'm running ( On Windows 10. ) -
logstash --verbose -f logstash-sample.conf
Here is my logstash-sample.conf file -
input {
file {
path => "C:\Users\17739\Documents\IIT\CSP586\tutorial\project\ChicagoSocialHub\backend-build-divvy-status\divvy_stations_status.csv"
start_position => "beginning"
}
}
filter {
csv{
separator => ","
columns => ["altitude", "availableBikes", "availableDocks", "city", "id", "is_renting", "kioskType", "landMark", "lastCommunicationTime", "latitude", "location", "longitude", "postalCode", "stAddress1", "stAddress2", "stationName", "status", "statusKey", "statusValue", "testStation", "totalDocks"]
}
}
output {
elasticsearch {
hosts => ["localhost:9200"]
index => "divvy_stations_status"
document_type => "status"
}
stdout {
codec => rubydebug
}
}
And here is the logstash console output -
C:\Users\17739\Documents\IIT\CSP586\logstash-6.6.2\bin>logstash --verbose -f logstash-sample.conf
Sending Logstash logs to C:/Users/17739/Documents/IIT/CSP586/logstash-6.6.2/logs which is now configured via log4j2.properties
[2019-03-17T12:56:36,728][WARN ][logstash.config.source.multilocal] Ignoring the 'pipelines.yml' file because modules or command line options are specified
[2019-03-17T12:56:36,745][INFO ][logstash.runner ] Starting Logstash {"logstash.version"=>"6.6.2"}
[2019-03-17T12:56:41,603][WARN ][logstash.outputs.elasticsearch] You are using a deprecated config setting "document_type" set in elasticsearch. Deprecated settings will continue to work, but are scheduled for removal from logstash in the future. Document types are being deprecated in Elasticsearch 6.0, and removed entirely in 7.0. You should avoid this feature If you have any questions about this, please visit the #logstash channel on freenode irc. {:name=>"document_type", :plugin=><LogStash::Outputs::ElasticSearch index=>"divvy_stations_status", id=>"f84c43181aab6f7bf9e89c0412ada5b5ead116534f6661194800152751a28e87", hosts=>[//localhost:9200], document_type=>"status", enable_metric=>true, codec=><LogStash::Codecs::Plain id=>"plain_1264be19-323c-4896-8214-929f15a74251", enable_metric=>true, charset=>"UTF-8">, workers=>1, manage_template=>true, template_name=>"logstash", template_overwrite=>false, doc_as_upsert=>false, script_type=>"inline", script_lang=>"painless", script_var_name=>"event", scripted_upsert=>false, retry_initial_interval=>2, retry_max_interval=>64, retry_on_conflict=>1, ilm_enabled=>false, ilm_rollover_alias=>"logstash", ilm_pattern=>"{now/d}-000001", ilm_policy=>"logstash-policy", action=>"index", ssl_certificate_verification=>true, sniffing=>false, sniffing_delay=>5, timeout=>60, pool_max=>1000, pool_max_per_route=>100, resurrect_delay=>5, validate_after_inactivity=>10000, http_compression=>false>}
[2019-03-17T12:56:43,234][INFO ][logstash.pipeline ] Starting pipeline {:pipeline_id=>"main", "pipeline.workers"=>8, "pipeline.batch.size"=>125, "pipeline.batch.delay"=>50}
[2019-03-17T12:56:43,548][INFO ][logstash.outputs.elasticsearch] Elasticsearch pool URLs updated {:changes=>{:removed=>[], :added=>[http://localhost:9200/]}}
[2019-03-17T12:56:43,695][WARN ][logstash.outputs.elasticsearch] Restored connection to ES instance {:url=>"http://localhost:9200/"}
[2019-03-17T12:56:43,735][INFO ][logstash.outputs.elasticsearch] ES Output version determined {:es_version=>6}
[2019-03-17T12:56:43,739][WARN ][logstash.outputs.elasticsearch] Detected a 6.x and above cluster: the `type` event field won't be used to determine the document _type {:es_version=>6}
[2019-03-17T12:56:43,768][INFO ][logstash.outputs.elasticsearch] New Elasticsearch output {:class=>"LogStash::Outputs::ElasticSearch", :hosts=>["//localhost:9200"]}
[2019-03-17T12:56:43,782][INFO ][logstash.outputs.elasticsearch] Using mapping template from {:path=>nil}
[2019-03-17T12:56:43,801][INFO ][logstash.outputs.elasticsearch] Attempting to install template {:manage_template=>{"template"=>"logstash-*", "version"=>60001, "settings"=>{"index.refresh_interval"=>"5s"}, "mappings"=>{"_default_"=>{"dynamic_templates"=>[{"message_field"=>{"path_match"=>"message", "match_mapping_type"=>"string", "mapping"=>{"type"=>"text", "norms"=>false}}}, {"string_fields"=>{"match"=>"*", "match_mapping_type"=>"string", "mapping"=>{"type"=>"text", "norms"=>false, "fields"=>{"keyword"=>{"type"=>"keyword", "ignore_above"=>256}}}}}], "properties"=>{"@timestamp"=>{"type"=>"date"}, "@version"=>{"type"=>"keyword"}, "geoip"=>{"dynamic"=>true, "properties"=>{"ip"=>{"type"=>"ip"}, "location"=>{"type"=>"geo_point"}, "latitude"=>{"type"=>"half_float"}, "longitude"=>{"type"=>"half_float"}}}}}}}}
[2019-03-17T12:56:44,323][INFO ][logstash.inputs.file ] No sincedb_path set, generating one based on the "path" setting {:sincedb_path=>"C:/Users/17739/Documents/IIT/CSP586/logstash-6.6.2/data/plugins/inputs/file/.sincedb_6f34c293ff88e0ad3c31e4a0f32e43d9", :path=>["C:\\Users\\17739\\Documents\\IIT\\CSP586\\tutorial\\project\\ChicagoSocialHub\\backend-build-divvy-status\\divvy_stations_status.csv"]}
[2019-03-17T12:56:44,369][INFO ][logstash.pipeline ] Pipeline started successfully {:pipeline_id=>"main", :thread=>"#<Thread:0x2e991954 run>"}
[2019-03-17T12:56:44,440][INFO ][filewatch.observingtail ] START, creating Discoverer, Watch with file and sincedb collections
[2019-03-17T12:56:44,441][INFO ][logstash.agent ] Pipelines running {:count=>1, :running_pipelines=>[:main], :non_running_pipelines=>[]}
[2019-03-17T12:56:44,786][INFO ][logstash.agent ] Successfully started Logstash API endpoint {:port=>9600}
After running logstash --verbose -f logstash-sample.conf command , My expectation is this new index divvy_stations_status will be visible in elastic search indices(checked with kibana as well , not showing up there as well) . Is that a true expectation ? Here is my
output -
