0

I am new in ELK configuration.

https://www.digitalocean.com/community/tutorials/how-to-install-elasticsearch-logstash-and-kibana-elastic-stack-on-ubuntu-16-04

I have configure in my local machine and it is work fine. I want to forward my application logs file to elastic search using log-stash of file beats. When I have configure all things working fine for system logs. but I am not able to store my application log to elastic search. Please help me.

This is my log file:

service.log

{"name":"service name", "hostname":"abc", "pid":4474, "userId":"123", "school_id":"123", "role":"student", "username":"mahi123", "serviceName":"loginService", "level":40, "msg":"successFully fetch trail log", "time":"2019-06-01T10:55:46.482Z","v":0}
susenj
  • 342
  • 1
  • 4
  • 12
Hiren Dhokiya
  • 1
  • 2
  • Have you created a elasticsearch .conf file? Provide more info on that. Also, try running `logstash -f ` and see if throws any error. Don't forget to add your service.log file path into the .conf file. – susenj Jun 11 '19 at 12:05
  • Thank you for reply Yes i have created .conf file and the code is as bellow input { file { path => ["path/logs/*.log"] type => "servicelog" } } output { elasticsearch { hosts => ["http://localhost:9200"] index => "servicelog-%{+YYYY.MM.dd}" document_type => "service_logs" } stdout { codec => rubydebug } } and i have run this conf file using this command sudo bin/logstash -f /path/logStash.conf it is working fine but its not forward logs to elastic search – Hiren Dhokiya Jun 11 '19 at 13:00
  • Thank you for your help. it was working fine. Now i want to read logs from multiple path. so can i use path => ["path/logs/*.log", "path/logs2/*.log", "path/logs3/*.log"] – Hiren Dhokiya Jun 13 '19 at 04:50
  • Yes you can. Now i have configure ELK in particular machine. and my logs are in another machine and i want to read logs from one machine to ELK machine so have to configure file-beats on machine where from i have to read logs – Hiren Dhokiya Jun 13 '19 at 06:21
  • i have done this ELK things in my local machine. and i it is working fine. – Hiren Dhokiya Jun 20 '19 at 11:55
  • Now i want to implement with aws Elk service. Means i want to forward from my service base server to aws ELK service machine.Please help, i am new in aws ELK service – Hiren Dhokiya Jun 20 '19 at 11:56

1 Answers1

0

Some troubleshooting steps to take care of when logs do not reach Elastisearch:

  1. Check your log parsing configuration file(usually made with the extension .conf). Make sure it's having the right path to scan logs from, right set of filters etc. To see if this .conf file is actually working, one can try:

logstash -f <elasticsearch.conf file path> If this doesn't throw any error on console, that means you are good at this point and will have to move to next step.

  1. Check if Kibana indices are getting created. Run curl http://<hostipaddress or localhost>:9200/_cat/indices?v.

If yes, go to Kibana Management and create index patterns.

If not, check if your system has enough available memory to serve logstash and elastisearch. free -m would be helpful once you start logstash and elasticsearch services. Many a times, I have seen people trying ELK setup on a machine which has insufficient RAM(4GB sounds good for a standalone setup).

  1. Check your logstash and Elasticsearch services are up and running. If Elasticsearch is getting down or getting restarted during log parsing or indices creation, that's most probably due to lack of system resources.
    -bash-4.2# systemctl status elasticsearch
    �� elasticsearch.service - Elasticsearch
       Loaded: loaded (/usr/lib/systemd/system/elasticsearch.service; enabled; vendor preset: disabled)
       Active: active (running) since Wed 2019-06-05 14:08:26 UTC; 1 weeks 0 days ago
         Docs: http://www.elastic.co
     Main PID: 1396 (java)
       CGroup: /system.slice/elasticsearch.service
               ������1396 /bin/java -Xms1g -Xmx1g -XX:+UseConcMarkSweepGC -XX:CMSInitiatingOccupancyFraction=75 -XX:+UseCMS...

    Jun 05 14:08:26 cue-bldsvr4 systemd[1]: Started Elasticsearch.
    Jun 05 14:08:26 cue-bldsvr4 systemd[1]: Starting Elasticsearch...
    -bash-4.2# systemctl status logstash     
    �� logstash.service - logstash
       Loaded: loaded (/etc/systemd/system/logstash.service; enabled; vendor preset: disabled)
       Active: active (running) since Wed 2019-06-05 14:50:52 UTC; 1 weeks 0 days ago
     Main PID: 4320 (java)
       CGroup: /system.slice/logstash.service
               ������4320 /bin/java -Xms256m -Xmx1g -XX:+UseParNewGC -XX:+UseConcMarkSweepGC -XX:CMSInitiatingOccupancyFrac...

    Jun 05 14:50:52 cue-bldsvr4 systemd[1]: Started logstash.
    Jun 05 14:50:52 cue-bldsvr4 systemd[1]: Starting logstash...
    Jun 05 14:51:08 cue-bldsvr4 logstash[4320]: Sending Logstash's logs to /var/log/logstash which is now configur...rties
    Hint: Some lines were ellipsized, use -l to show in full.
    -bash-4.2#
susenj
  • 342
  • 1
  • 4
  • 12