0

I attempted to import data from a json file into elastic search, but my filebeat had a configuration issue in filebeat.yml or an issue with the input Because my json is regular json, but filebeat only supports ndjson,

Filebeat.yml

  filebeat.inputs:
- type: log
  enabled: true
  paths:
  - Metrics.json
  multiline.pattern: '^{'
  multiline.negate: true
  multiline.match:  after

processors:
- decode_json_fields:
   fields: ["message"]
   process_array: false
   max_depth: 2
   target: Json
   overwrite_keys: true
   add_error_key: false
  

output.elasticsearch:
  hosts: ["localhost:9200"]

Json Input

{
  "Sourcedata": [
    {
      "Language": " C",
      "Code": 106026
    },
    {
      "Language": " C++",
      "Code": 52166
    }
  ]
}
  1. How to convert JSon to ndjson?
  2. What problem with my filebeat

Log shown in beat

2022-01-05T15:02:31.273+0530    INFO    [input.harvester]       log/harvester.go:340    File is inactive. Closing because close_inactive of 5m0s reached.       {"input_id": "34732cd5-11e3-4c86-9bb2-1fca798535a2", "source": "F:\\ELK.7.16.2\\filebeat-7.16.2\\Metrics.json", "state_id": "native::995622912-619633-2230426833", "finished": false, "os_id": "995622912-619633-2230426833", "old_source": "F:\\ELK.7.16.2\\filebeat-7.16.2\\Metrics.json", "old_finished": true, "old_os_id": "995622912-619633-2230426833", "harvester_id": "ef0353b3-2d88-4ba8-a32b-9fd2a79f0c7c"}
2022-01-05T15:02:36.318+0530    INFO    [input.harvester]       log/harvester.go:309    Harvester started for file.
        {"input_id": "34732cd5-11e3-4c86-9bb2-1fca798535a2", "source": "F:\\ELK.7.16.2\\filebeat-7.16.2\\Metrics.json", "state_id": "native::995622912-619633-2230426833", "finished": false, "os_id": "995622912-619633-2230426833", "old_source": "F:\\ELK.7.16.2\\filebeat-7.16.2\\Metrics.json", "old_finished": true, "old_os_id": "995622912-619633-2230426833", "harvester_id": "ad7994a8-93e2-4c87-a0d1-99f85a6849f5"}
2022-01-05T15:02:56.040+0530    INFO    [monitoring]    log/log.go:184  Non-zero metrics in the last 30s        {"monitoring": {"metrics": {"beat":{"cpu":{"system":{"ticks":171,"time":{"ms":15}},"total":{"ticks":405,"time":{"ms":15},"value":405},"user":{"ticks":234}},"handles":{"open":269},"info":{"ephemeral_id":"4dd2fbe2-6e83-49ef-b6d8-0d01c3ed4aca","uptime":{"ms":372182},"version":"7.16.2"},"memstats":{"gc_next":19272448,"memory_alloc":10218592,"memory_total":59019184,"rss":50970624},"runtime":{"goroutines":49}},"filebeat":{"events":{"added":2,"done":2},"harvester":{"closed":1,"open_files":1,"running":1,"started":1}},"libbeat":{"config":{"module":{"running":0}},"output":{"events":{"active":0}},"pipeline":{"clients":1,"events":{"active":0,"filtered":2,"total":2}}},"registrar":{"states":{"current":1,"update":2},"writes":{"success":2,"total":2}}}}}
max
  • 255
  • 2
  • 7
  • Do you have any error message ? what is failing ? – Paulo Jan 05 '22 at 09:06
  • @Paulo no error shown in the log, but harvesting for file has started comments repeatedly coming updated in description – max Jan 05 '22 at 09:40

0 Answers0