5

I am a newbie of ELK. I installed first Elasticsearch and Filebeat without Logstash, and I would like to send data from Filebeat to Elasticsearch. After I installed the Filebeat and configured the log files and Elasticsearch host, I started the Filebeat, but then nothing happened even though there are lots of rows in the log files, which Filebeats prospects.

So is it possible to forward log data directly to Elasticsearch host without Logstash at all? I

Rui
  • 3,454
  • 6
  • 37
  • 70
  • Yes, it is definitely possible. Can you update your question with your filebeat configuration, please? Also can state specify which ES version you are using? – Val Apr 25 '16 at 07:52
  • filebeat: prospectors: - paths: - "/opt/logs/*.log" input_type: log output: elasticsearch: hosts: ["front.development.xxx:9200"] – Rui Apr 25 '16 at 07:55
  • It's more legible if you update your question with the properly formatted configuration. – Val Apr 25 '16 at 07:56
  • My configuration is pretty basic as shown above. But after I started the Filebeat and Elasticsearch, nothing happenend at all : – Rui Apr 25 '16 at 07:56
  • What ES version are you using? – Val Apr 25 '16 at 07:57
  • Where is your ES installed? On your local computer or on some remote host? In the latter case, can you update your question with the relevant network configuration you have in your `elasticsearch.yml` file? – Val Apr 25 '16 at 07:58
  • My ES is installed on a different server from the Filebeat, my elasticsearch.yml is empty as I checked just now. Should I configure it? – Rui Apr 25 '16 at 08:01
  • Yes, you should at least have `network.host: front.development.xxx` or a public IP address that your filebeat configuration can use. – Val Apr 25 '16 at 08:06
  • Let us [continue this discussion in chat](http://chat.stackoverflow.com/rooms/110111/discussion-between-user1928863-and-val). – Rui Apr 25 '16 at 08:19

1 Answers1

1

It looks like your ES 2.3.1 is only configured to be reachable from localhost (default since ES 2.0)

You need to modify your elasticsearch.yml file with this and restart ES:

network.host: 168.17.0.100

Then your filebeat output configuration needs to look like this:

output:
   elasticsearch:
     hosts: ["168.17.0.100:9200"]

Then you can check in your ES filebeat-* indices that you're getting the new log data (i.e. the hits.total count should increase over time):

curl -XGET 168.17.0.100:9200/filebeat-*/_search
Val
  • 207,596
  • 13
  • 358
  • 360
  • I reinstalled Elasticsearch and Filebeat. Previously I forgot to run Filebeat with sudo, then encountered error. Then when I rerun Filebeat with sudo, everything went fine based on your suggestion. 100% proved your suggestion now :) Thanks again for your great great help! You mentioned to debug with command - filebeat -e -d "*", this seemed not to work properly for me, but with option "-v" (verbose) I indeed got Filebeat log from syslog as I configured. – Rui Apr 26 '16 at 06:48