Read log file from a remote machine with file input plugin using logstash

Question

Presently I have my logs and logstash running on the same machine, so I read my logs placed on my local machine with this config(using pull model)

input {     
    file {
        path => "/home/Desktop/Logstash-Input/**/*_log"
        start_position => "beginning"
    }
}

Now, we have logstash running on a different machine and want to read the logs remote mechine.

Is there a way to set the ip in file input of config file?

EDIT: I manage to do this with logstash-forwarder which is a push model(log shipper/logstash-forwarder will ship log to logstash index server) but still i am looking for a pull model without shipper, where logstash index server will go and contact directly to remote host.

Fernando · Accepted Answer · 2017-02-22T15:56:47.637

Take a look to FileBeat: https://www.elastic.co/products/beats/filebeat

It´s not a pull model but it seems a better choice than logstash-forwarder.

It monitors log files and forwards them to Logstash or Elasticsearh. It keeps also the state of log files and guarantees that events will be delivered at least one time (depends on log rotation speed). It's really easy to configure:

Input configuration:

input_type: log
paths:
- /opt/app/logs

Output configuration

output.logstash:
  hosts: ["remote_host:5044"]
  index: filebeat_logs

In the logstash side you must install and configure the Beats input plugin:

input {
  beats {
    port => 5044
  }
}

Hello, please add more details to your answer. For more info, please look at http://stackoverflow.com/help/how-to-answer. — Chait, Feb 22 '17 at 14:59

Magnus Bäck · Answer 2 · 2015-07-07T13:31:53.487

3

Logstash doesn't contain any magic to read files from other computer's file systems (and that's probably a good thing). You'll either have to mount the remote file system that contains the logs you're interested in or you have to install a log shipper (like e.g. Logstash) on the remote machine and configure it to send the data to your current Logstash instance (or an intermediate broker like Redis, RabbitMQ, or Kafka).

You could also use the syslog daemon (that's probably already installed on the machine) to ship logs via the syslog protocol, but keep in mind that there's no guarantee of the maximum allowed length of each message.

edited Jul 07 '15 at 13:31

answered Jul 01 '15 at 11:09

Magnus Bäck

11,381
3
47
59

@Maguns I manage to so this with logstash-forwarder which is a push model but still i am looking for a pull model without shipper, where logstash index server will go and contact directly to remote host, yet to explore tcp, do you have any ideas about tcp input plugin – lambodar Jul 07 '15 at 13:11
1

The tcp input plugin is for inbound connections, i.e. it still requires other hosts to connect to Logstash. – Magnus Bäck Jul 07 '15 at 13:29
@LambodarSwain What did you end up with to build pull model for multiple logs? I am also considering shared filesystem to aggregate logs from tomcats and then read them with logstash file plugin. Thanks! – Konstantin Oct 15 '15 at 05:24

score -4 · Answer 3 · answered Jul 07 '15 at 11:34

-4

You can add the remote system IP in the path and access the logs from Remote machine.

input {     
file {
    path => "\\IP address/home/Desktop/Logstash-Input/**/*_log"
    start_position => "beginning"
}}

answered Jul 07 '15 at 11:34

Anilkumar Bathula

278
3
18

1

This only works on Windows systems but the file path in the question indicates that the OP doesn't run Windows. – Magnus Bäck Jul 07 '15 at 11:53
@Magnus Back, K back actually i am working on the Windows. I given my smaple example – Anilkumar Bathula Jul 07 '15 at 12:15
1

Yes, but the OP most likely doesn't run Windows so the answer isn't very useful since accessing network resources via UNC paths isn't generally supported there. – Magnus Bäck Jul 07 '15 at 13:10
no luck @AnilkumarBattula , by the way I am using linux, but thanks for answering :) – lambodar Jul 07 '15 at 13:18

Read log file from a remote machine with file input plugin using logstash

3 Answers3