logstash issue with json input file

Question

i have the following json in a file-

{
"foo":"bar",
"spam" : "eggs"
},
{
"css":"ddq",
"eeqw": "fewq"
}

and the following conf file-

input { 
file
{ 
   path => "/opt/logstash-1.4.2/bin/sam.json"
   type => "json"
   codec => json_lines
   start_position =>"beginning"
 }
}
output { stdout {  codec => json  } }

but when i run

./logstash -f sample.conf

i don't get any output in stdout.

but when i don't give json as codec and give type => "core2" then it seems to work. Anyone know how i can fix it to work for json type.

The other issue is it gives me the following output when it does give stdout-

{"message":"{","@version":"1","@timestamp":"2015-07-15T02:02:02.653Z","type":"core2","host":"sjagannath","path":"/opt/logstash-1.4.2/bin/sam.json"}{"message":"\"foo\":\"bar\", ","@version":"1","@timestamp":"2015-07-15T02:02:02.654Z","type":"core2","host":"sjagannath","path":"/opt/logstash-1.4.2/bin/sam.json"}{"message":"\"spam\" : \"eggs\" ","@version":"1","@timestamp":"2015-07-15T02:02:02.655Z","type":"core2","host":"sjagannath","path":"/opt/logstash-1.4.2/bin/sam.json"}{"message":"},","@version":"1","@timestamp":"2015-07-15T02:02:02.655Z","type":"core2","host":"sjagannath","path":"/opt/logstash-1.4.2/bin/sam.json"}{"message":"{ ","@version":"1","@timestamp":"2015-07-15T02:02:02.655Z","type":"core2","host":"sjagannath","path":"/opt/logstash-1.4.2/bin/sam.json"}{"message":"\"css\":\"ddq\", ","@version":"1","@timestamp":"2015-07-15T02:02:02.656Z","type":"core2","host":"sjagannath","path":"/opt/logstash-1.4.2/bin/sam.json"}{"message":"\"eeqw\": \"fewq\"","@version":"1","@timestamp":"2015-07-15T02:02:02.656Z","type":"core2","host":"sjagannath","path":"/opt/logstash-1.4.2/bin/sam.json"}{"message":"}","@version":"1","@timestamp":"2015-07-15T02:02:02.656Z","type":"core2","host":"sjagannath","path":"/opt/logstash-1.4.2/bin/sam.json"}{"message":"","@version":"1","@timestamp":"2015-07-15T02:02:02.656Z","type":"core2","host":"sjagannath","path":"/opt/logstash-1.4.2/bin/sam.json"}

I want to know how it can be parsed the right way with the key value pairs in my input file

score 3 · Accepted Answer · edited May 23 '17 at 11:50

3

I found this and edited it to suit your purpose. The following config should do exactly what you want:

input {   
file     {
    codec => multiline
    {
        pattern => "^\}"
        negate => true
        what => previous               
    }
    path => ["/absoute_path/json.json"]
    start_position => "beginning"
    sincedb_path => "/dev/null"
}
}

filter {
mutate    {
    replace => [ "message", "%{message}}" ]
    gsub => [ "message","\n",""]
    gsub => [ "message","},",""]
}
if [message] =~ /^{.*}$/     {
    json { source => message }
}
}

I tried your given json and it results in two events. First with foo = bar and spam = eggs. Second with css = ddq and eeqw = fewq.

edited May 23 '17 at 11:50

Community

1
1

answered Jul 15 '15 at 11:46

hurb

2,177
3
18
32

i works partially. But i can read only the first json data and not the 2nd (with css). Is there something missing? – sharat Jul 17 '15 at 08:56
Might be a missing newline at the end of your json file. Logstash won't read the last line until it is finished with a newline. – hurb Jul 17 '15 at 10:34
ok.Suppose i don't have the comma separator. how would the filter look like? i tried gsub => [ "message","}",""] but it didn't work – sharat Jul 20 '15 at 02:58
Try this as second gsub: `gsub => [ "message","}{","{"]` It should properly read your data without a comma separator. – hurb Jul 20 '15 at 08:10

score 0 · Answer 2 · edited May 23 '17 at 11:59

0

As of my understanding you want to put your complete son document on one line if you want to use the json_lines codec:

{"foo":"bar","spam" : "eggs"}
{"css":"ddq","eeqw": "fewq"}

In your case you have a problem with the structure since you also have a ',' between the son objects. Not the most easy way to handle it. SO if possible change the source to my example. If that is not possible the multiline approach might help you. Check this for reference: input json to logstash - config issues?

edited May 23 '17 at 11:59

Community

1
1

answered Jul 15 '15 at 05:30

Jettro Coenradie

4,735
23
31

In fact, changing the source might be an approach. However, if that is not possible, see my answer. – hurb Jul 15 '15 at 13:17

logstash issue with json input file

2 Answers2