0

I am new to logstash where I am using file as input and elasticsearch as output. I am injecting a mongo document through file input and when it is inserted to elasicsearch, it is all added under a single field "message". But i want this document to be available under different fields on elasticsearch as a key-value pairs.

Input document:

 {"_id" : "INC-4","name" : "Incident 4","priority" : "HIGH","prioritySort" : 2,"riskScore" : 90,"status" : "NEW","statusSort" : 0,"alertCount" : 1,"pinnedAlertCount" : 0,"containsPinnedAlerts" : false,"averageAlertRiskScore" : 90,"sealed" : false,"totalRemediationTaskCount" : NumberLong(0),"openRemediationTaskCount" : NumberLong(0),"hasRemediationTasks" : false,"created" : ISODate("2021-11-24T13:56:51.322Z"),"lastUpdated" : ISODate("2021-11-24T13:56:51.322Z"), "sources" : [ "Investigate" ],"createdBy" : "admin","breachExportStatus" : "NONE","hasDeletedAlerts" : false,"deletedAlertCount" : 0,"eventCount" : 1,"groupBySourceIp" : [ "" ],"groupByDestinationIp" : [  ""],"sentTo" : false,"_class" : "com.rsa.asoc.commons.domain.Incident", "statushistory" : [  {"date_changed" : ISODate("2021-11-24T13:56:51.328Z"),"status" : "NEW" }]}

logstash config file:

input {
    file {
   path => "C:/Users/poler/Downloads/logstash-oss-7.10.0-windows-x86_64/files/file2.json"
   start_position => "beginning"
   sincedb_path => "NUL"
  }
}
filter {
    }
output {
        elasticsearch {
                action => "index"
                hosts => ["https://<hostIP>:9200"]
                user => "####"
                password => "####"
                ssl_certificate_verification => false
                cacert => "C:\Users\poler\Downloads\logstash-oss-7.10.0-windows-x86_64\elasticsearch-cert.pem"
        }
        stdout{}
}

Can anyone please help me with the logstash config file to get each key-value pair as separate fields on elasticsearch??

Rachana pole
  • 5
  • 2
  • Have you considered using a json filter? – Badger Nov 29 '21 at 16:52
  • On inputting the data in JSON format, I am able to get the input in the right format. Thank you. – Rachana pole Dec 01 '21 at 16:38
  • I also have another question, on using JDBC input plugin, I am querying for data according to a field "lastUpdated" for which i have used jdbc statement as statement => "db.incident.find({lastUpdated: { $gt: :sql_last_value }},{'_id': false})" but my data on Database is like "lastUpdated" : ISODate("2021-12-01T11:45:14.736Z"). Can you please help me how i can read this format of timestamp through JDBC input plugin??? – Rachana pole Dec 01 '21 at 16:39

0 Answers0