Elastic ELK stack 8.5 integration with Spring Boots Application using Filebeat

Question

Setting up a pipeline of elastic search, kibana, and logstash in locally and using filebeat to push logs from a spring boot application to the pipeline. U will find the official documentation well-defined, But I created this questions to answer a few points that were not clear. I answered for a single spring boot app scenario, thanks to people who are adding their scenarios as well.

Leons · Answer 1 · 2023-01-11T10:22:36.593

I spend a few days configuring the ELK stack with my spring boot application. Here I won't specify the step-by-step integration, for that, you can refer to the official documentation. This is more focused on what I didn't find in the documentation steps.

Env: This will be focused on setting up the 8.5.3 version in a mac os.

For Elasticsearch and Kibana I didn't have any trouble following the official document word by word.

Elasticsearch: https://www.elastic.co/downloads/elasticsearch

Kibana:https://www.elastic.co/downloads/kibana

Tip: To Check elastic running

curl --cacert config/certs/http_ca.crt -u elastic https://localhost:9200

Enter password when prompted

Enter host password for user 'elastic':

In my project, I needed to extract only a specific log line and process it. U can use the below official document link to download and extract the logstash and filebeat. Then you can use the mentioned configs before you run it.

Logstash: https://www.elastic.co/downloads/logstash

Filebeat: https://www.elastic.co/downloads/beats/filebeat

Filebeat :

First, you need to make permission changes to your filebeat.yml file. Navigate to your filebeat extracted folder and you can use the following config if needed.

filebeat.inputs:

- type: filestream
  id: filebeat-id-name
  enabled: true
  #Path to you log file
  paths:
    - /Users/leons/IdeaProjects/SpringELKDemo/myapplogs.log  

  #I wanted to only process the logs from MainController
  include_lines: ['MainController'] 

output.logstash:
  hosts: ["localhost:5044"]

Then you need to alter the write permission for this file using the below command(mac). Later you can edit the file using sudo nano.

sudo chown root filebeat.yml

Logstash:

Initial a sample file for logstash.conf is available in the config folder inside logstash. you can refer to that, also take a look at mine.

# Sample Logstash configuration for creating a simple
# Beats -> Logstash -> Elasticsearch pipeline.

input {
  beats {
    port => 5044
  }
}
filter {
    #Extracting a portion of log message
    dissect {
        mapping => {
            "message" => "%{}: %{data_message}"
         }
    }
    #converting message json into fields
    json {
        source => "data_message"
    }
    #mapping the message json timestamp with entry timestamp
    date {
        target => "@timestamp"
        match => ["timestamp","yyyy-MM-dd HH:mm:ss.SSS"]
    }
    #removing unneeded fields
    mutate {
        remove_field => ["[event][original]","message","data_message","timestamp"]
    }
}
output {
  elasticsearch {
    hosts => ["https://localhost:9200"]
    index => "myindex"
    user => "elastic"
    password => "*****************"
    ssl_certificate_verification => false
  }
  stdout{
    codec => rubydebug
  }
}

I used the dissect filter to do string manipulation in my logline, that filebeat transferred. Below was my log, and I needed only the exact message which is JSON string

2022-12-15 21:14:56.152  INFO 9278 --- [http-nio-8080-exec-10] c.p.t.springdemo.controller.MainController    : {"name":"leons","id":"123123","msg":"hello world"}

For more on dissect refer official docs

The json filter is used to convert the JSON key: values into fields and values in your elastic document.

Now you should be ready to run logstash and filebeat using official document command. Just for reference use below

Logstash :

bin/logstash -f logstash.conf

Filebeat :

sudo ./filebeat -e -c filebeat.yml

Elastic ELK stack 8.5 integration with Spring Boots Application using Filebeat

1 Answers1