I am sending 25000 logs from filebeat to elasticsearch and another http server (spring boot) using filebeat and logstash. To send logs to two places, I use logstash pipeline to pipeline communication mechanism with output isolator pattern. But the problem is filebeat does not send all logs instead, just send 5000 logs and in the debug log it shows as below.
Filebeat debug log
2019-05-29T15:39:46.993+0530 DEBUG [monitoring] memqueue/ackloop.go:160 ackloop: receive ack [295: 0, 1]
2019-05-29T15:39:46.994+0530 DEBUG [monitoring] memqueue/eventloop.go:226 handle ACKs: 1
2019-05-29T15:39:46.994+0530 DEBUG [monitoring] memqueue/eventloop.go:251 try ack index: (idx=0, i=0, seq=296)
2019-05-29T15:39:46.994+0530 DEBUG [monitoring] memqueue/eventloop.go:269 broker ACK events: count=1, start-seq=296, end-seq=296
2019-05-29T15:39:46.994+0530 DEBUG [monitoring] memqueue/eventloop.go:228 handle ACK took: 130.793µs
2019-05-29T15:39:46.994+0530 DEBUG [monitoring] memqueue/ackloop.go:128 ackloop: return ack to broker loop:1
2019-05-29T15:39:46.994+0530 DEBUG [monitoring] memqueue/ackloop.go:131 ackloop: done send ack
2019-05-29T15:39:47.134+0530 DEBUG [input] input/input.go:152 Run input
2019-05-29T15:39:47.134+0530 DEBUG [input] log/input.go:174 Start next scan
2019-05-29T15:39:47.134+0530 DEBUG [input] log/input.go:404 Check file for harvesting: /home/nishan/Documents/input_logs/request-response-logger.log
2019-05-29T15:39:47.134+0530 DEBUG [input] log/input.go:494 Update existing file for harvesting: /home/nishan/Documents/input_logs/request-response-logger.log, offset: 33311469
2019-05-29T15:39:47.134+0530 DEBUG [input] log/input.go:546 Harvester for file is still running: /home/nishan/Documents/input_logs/request-response-logger.log
2019-05-29T15:39:47.134+0530 DEBUG [input] log/input.go:195 input states cleaned up. Before: 1, After: 1, Pending: 0
Below are the configurations in filebeat and logstash.
pipeline.yml
- pipeline.id: intake
queue.type: persisted
config.string: |
input { beats { port => 5043 } }
output { pipeline { send_to => [spring, es_out] } }
- pipeline.id: spring
queue.type: persisted
path.config: "/home/nishan/Documents/logstash-config/spring.conf"
queue.max_bytes: 15mb
- pipeline.id: elastic
queue.type: persisted
path.config: "/home/nishan/Documents/json_push/configurations/elastic.conf"
queue.max_bytes: 500mb
filebeat.yml
filebeat.inputs:
- type: log
enabled: true
paths:
- /home/nishan/Documents/input_logs/*.log
filebeat.config.modules:
path: ${path.config}/modules.d/*.yml
reload.enabled: false
setup.template.settings:
index.number_of_shards: 3
setup.kibana:
output.logstash:
hosts: ["localhost:5043"]
processors:
- add_host_metadata: ~
- add_cloud_metadata: ~
logstash.yml
pipeline.batch.size: 5
pipeline.batch.delay: 500
path.queue: /home/nishan/Documents/data_files
If I set queue.max_bytes: 1024mb in pipeline.yml, it works fine and all data will sent to elasticsearch and spring server. I guess that it could be the problem because of low queue size (15mb) I used. But in logstash reference it said that it will handle back pressure by rejecting connections while queue is filled and it accept connections when queues have space. What could be the reason?
NOTE: I am using logstash and filebeat version 6.7 in ubuntu 18.04.
