Parse log file of json using logstatsh

Question

I have the following json object logs as following in a log file

{"con":"us","sl":[[1,2]],"respstats_1":"t:2,ts:140,m:192.168.7.5,p:|mobfox:1,P,E,0,0.4025:0.0:-:-,0-98;appnexus-marimedia:2,P,L,140,0.038:0.0:-:-,-;","rid":"AKRXRWLYCZIDFM","stats":"t:2,h:2,ts:140,mobfox:0,appnexus-marimedia:140,;m:192.168.7.5;p:","resp_count":0,"client_id":"15397682","err_stats":"mobfox:0-98,"}
{"con":"br","sl":[[1,2,3,4]],"respstats_1":"t:4,ts:285,m:192.168.7.5,p:|smaato:1,P,M,143,0.079:0.0:-:-,-;vserv-specialbuy:2,P,W,285,0.0028:0.0:-:-,-;mobfox:3,P,E,42,0.077:0.0:-:-,0-98;inmobi-pre7:4,P,H,100,0.0796:0.0:-:-,-;","rid":"AKRXRWLYCY4DOU","stats":"t:4,h:4,ts:285,smaato:143,vserv-specialbuy:285,mobfox:42,inmobi-pre7:100,;m:192.168.7.5;p:","resp_count":1,"client_id":"15397682","err_stats":"mobfox:0-98,","ads":[{"pricing":{"price":"0","type":"cpc"},"rank":2,"resp_json":{"img_url":"http://img.vserv.mobi/i/320x50_7/7bfffd967a91e0e38ee06ffcee1a75e5.jpg?108236_283989_c46e3f74","cli_url":"http://c.vserv.mobi/delivery/ck.php?p=2__b=283989__zoneid=108236__OXLCA=1__cb=c46e3f74__dc=1800__cd=usw3_uswest2a-1416567600__c=37742__rs=0a587520_15397682__mser=cdn__dat=3__dacp=12__zt=s__r=http%3A%2F%2Fyeahmobi.go2cloud.org%2Faff_c%3Foffer_id%3D28007%26aff_id%3D10070%26aff_sub%3D108236_283989_c46e3f74","beacons":["http://img.vserv.mobi/b.gif"],"ad_type":"image"},"resp_code":200,"resp_html":"<a href=\"http://c.vserv.mobi/delivery/ck.php?p=2__b=283989__zoneid=108236__OXLCA=1__cb=c46e3f74__dc=1800__cd=usw3_uswest2a-1416567600__c=37742__rs=0a587520_15397682__mser=cdn__dat=3__dacp=12__zt=s__r=http%3A%2F%2Fyeahmobi.go2cloud.org%2Faff_c%3Foffer_id%3D28007%26aff_id%3D10070%26aff_sub%3D108236_283989_c46e3f74\"><img src=\"http://img.vserv.mobi/i/320x50_7/7bfffd967a91e0e38ee06ffcee1a75e5.jpg?108236_283989_c46e3f74\"  alt=\"\" /> <\/a><img src=\"http://img.vserv.mobi/b.gif\"  alt=\"\" />","tid":"vserv-specialbuy","bid":"576111"}]}

How ever I am not able to figure out whether they are multiline or single line but I have used as following configuration

input {
  file {
    codec => multiline {
        pattern => '^{'
        negate => true
        what => previous
    }
    path => ['/home/pp38/fetcher.log']
  }
}
filter {
  json {
    source => message
    remove_field => message
  }
}

output { stdout { codec => rubydebug } }

I am not able to see any kind of output or error when it is started

edited:

I have used the following config which had generated output.

input {
    file {
        codec => "json"
        type => "json"
        path => "/home/pp38/fetcher.log"
        sincedb_path => "/home/pp38/tmp/logstash/sincedb"
    }
}

filter {
  json {
    source => "message"
    target => "message"
  }
}


output {
  stdout { codec => rubydebug }
  elasticsearch {
        hosts => ["localhost:9200"]
    }

}

But i am getting the output where each field is indexed by elasticsearch

how can i append the entire json message to new field as message:jsonContent ?

score 0 · Answer 1 · answered Dec 20 '15 at 06:53

You can handle this with the plain multiline, but for you situation there is a better codec plugin called json_lines.

The json_lines will input a source with multiple jsons(one in each line) and handle each json out of the box.

This codec will decode streamed JSON that is newline delimited. Encoding will emit a single JSON string ending in a \n NOTE: Do not use this codec if your source input is line-oriented JSON, for example, redis or file inputs. Rather, use the json codec. More info: This codec is expecting to receive a stream (string) of newline terminated lines. The file input will produce a line string without a newline. Therefore this codec cannot work with line oriented inputs.

how can i append the entire json message to new field as `message:jsonContent` ? — pavan, Dec 20 '15 at 08:46

Parse log file of json using logstatsh

1 Answers1