Transform String into JSON so that it's searchable in Kibana/Elasticsearch

Question

I have Elasticsearch, Filebeat and Kibana running on a Windows machine. Filebeat log has a proper log file and is listening to the path. When I look on the data in Kibana it looks fine.

My issue is that the message field is a String.

Example of one log line:

12:58:09.9608 Trace {"message":"No more Excel rows found","level":"Trace","logType":"User","timeStamp":"2020-08-14T12:58:09.9608349+02:00","fingerprint":"226fdd2-e56a-4af4-a7ff-724a1a0fea24","windowsIdentity":"mine","machineName":"NAME-PC","processName":"name","processVersion":"1.0.0.1","jobId":"957ef018-0a14-49d2-8c95-2754479bb8dd","robotName":"NAME-PC","machineId":6,"organizationUnitId":1,"fileName":"GetTransactionData"}

So what I would like to have now is that String converted to a JSON so that it is possible to search in Kibana for example for the level field.

I already had a look on Filebeat. There I tried to enable LogStash . But then the data does not come anymore to Elasticsearch. And also the log file is not genereated into the LogStash folder.

Then I downloaded LogStash via install guide, but unfortunately I got this message:

C:\Users\name\Desktop\logstash-7.8.1\bin>logstash.bat 
Sending
Logstash logs to C:/Users/mine/Desktop/logstash-7.8.1/logs which
is now configured via log4j2.properties ERROR: Pipelines YAML file is
empty. Location:
C:/Users/mine/Desktop/logstash-7.8.1/config/pipelines.yml usage:  
bin/logstash -f CONFIG_PATH [-t] [-r] [] [-w COUNT] [-l LOG]  
bin/logstash --modules MODULE_NAME [-M
"MODULE_NAME.var.PLUGIN_TYPE.PLUGIN_NAME.VARIABLE_NAME=VALUE"] [-t]
[-w COUNT] [-l LOG]   bin/logstash -e CONFIG_STR [-t] [--log.level
fatal|error|warn|info|debug|trace] [-w COUNT] [-l LOG]   bin/logstash
-i SHELL [--log.level fatal|error|warn|info|debug|trace]   bin/logstash -V [--log.level fatal|error|warn|info|debug|trace]  
bin/logstash --help
[2020-08-14T15:07:51,696][ERROR][org.logstash.Logstash    ]
java.lang.IllegalStateException: Logstash stopped processing because
of an error: (SystemExit) exit

Edit:

I tried to use Filebeat only. Here I set:

processors:
  - add_host_metadata: ~
  - add_cloud_metadata: ~
  - add_docker_metadata: ~
  - add_kubernetes_metadata: ~
  - dissect: 
      tokenizer: '"%{event_time} %{loglevel} %{json_message}"' 
      field: "message" 
      target_prefix: "dissect"
  - decode_json_fields: 
      fields: ["json_message"]

but that gave me:

dissect_parsing_error

The tip with removing the "" at tokenizer helped. Then I got:

I simply refreshed the index and the message was gone. Nice.

But The question is now, how to filter for something in the new field?

Answer 1

The message says, your pipeline config is empty. It seems you did not configured any pipeline yet. Logstash can do the trick (JSON filter plugin), but Filebeat is sufficient here. If you don't want to introduce another Service, this is the better option.

It has the decode_json_fields option to transform specific fields containing JSON in your event to a . Here is the documentation.

For the future case, where your whole event is a JSON, there is the possibility of parsing in filebeat configuring the json.message_key and related json.* option.

EDIT - Added filebeat snippet as an processors example of dissecting the log line into three fields (event_time, loglevel, json_message). Afterwards the recently extracted field json_message, whose value is a JSON object encoded as a string, will be decoded into an JSON structure:

 ... 

filebeat.inputs: 
  - type: log 
    paths: 
      - path to your logfile
  
processors: 
  - dissect: 
      tokenizer: '%{event_time} %{loglevel} %{json_message}' 
      field: "message" 
      target_prefix: "dissect"

  - decode_json_fields: 
      fields: ["dissect.json_message"]
      target: ""

  - drop_fields:
      fields: ["dissect.json_message"]


 ... 

If you want to practice the filebeat processors, try to set the correct event timestamp, taken from the encoded json and written into @timestamp using the timestamp processor.