Logstash : Parsing fails while parsing Json data received through web service like twitter / facebook etc

Question

I am receiving the Web service data say from Twitter and logging to file and there after I need to send that data to Logstash so as same can be indexed to Elasticsearch.

I am using below config and that is giving jsonparsefailure with exception as

JSON parse failure. Falling back to plain-text {:error=>#> LogStash::Json::ParserError: Unexpected character (':' (code 58)): expected a >valid value (number, String, array, object, 'true', 'false' or 'null')

My logstash conf files looks like :

input
    {
        file
        {
            path => ["/mnt/volume2/ELK_Prashant/at/events.json"]
            codec => json
            type => json
        start_position => "beginning"
            sincedb_path => "/dev/null"
        }
    }
    output
    {
        stdout { codec => rubydebug }
    }

And data in events.json can be reference from https://dev.twitter.com/rest/reference/get/search/tweets with some sample as below: events.json

[
{ "location": "LA, CA",
        "follow_request_sent": null,
        "profile_link_color": "0084B4",
        "is_translator": false,
        "id_str": "137238150",
        "entities": {
          "url": {
            "urls": [
              {
                "expanded_url": null,
                "url": ""
              }
            ]
          }
        }
}
]

In your `events.json` file, is each JSON event on a single line without any new line character in it? — Val, Aug 02 '16 at 05:16

score 1 · Answer 1 · answered Aug 07 '16 at 19:21

From your sample events.json file, it is clear that you are using a complete json object as input to logstash file plugin but the plugin by default assumes that each event will be single line and because of that only it is able to detect new events coming in and tracking current position.

So your input file should look like this, where each event is separated by new line character

{"location":"LA, CA","follow_request_sent":null,"profile_link_color":"0084B4","is_translator":false,"id_str":"137238150","entities":{"url":{"urls":[{"expanded_url":null,"url":""}]}}}
{"location":"LA, CA","follow_request_sent":null,"profile_link_color":"0084B4","is_translator":false,"id_str":"137238150","entities":{"url":{"urls":[{"expanded_url":null,"url":""}]}}}
{"location":"LA, CA","follow_request_sent":null,"profile_link_color":"0084B4","is_translator":false,"id_str":"137238150","entities":{"url":{"urls":[{"expanded_url":null,"url":""}]}}}

or you have to use multiline codec or filter in input plugin. More info can be found here.

Logstash : Parsing fails while parsing Json data received through web service like twitter / facebook etc

1 Answers1