0

Trying to set up elastic search, kibana and logstash to read logs from local folder. It works well on version 7.x.x, but when I try to upgrade to 8 it doesn't.Fx

I am using this YAML file:

version: '3.6'
services:
  Elasticsearch:
    image: elasticsearch:8.4.0
    container_name: elasticsearch
    volumes:
    - elastic_data:/usr/share/elasticsearch/data/
    environment:
    - discovery.type=single-node
    - xpack.license.self_generated.type=basic
    - xpack.security.enabled=false
    ports:
    - '9200:9200'
    - '9300:9300'
    networks:
      - elk

  Logstash:
    image: logstash:8.4.0
    container_name: logstash
    environment:
    - ELASTICSEARCH_HOSTS=http://elasticsearch:9200
    - xpack.monitoring.enabled=true

    volumes:
    - ./logstash/:/logstash
    - D:/test/Logs/:/test/Logs
    command: logstash -f /logstash/logstash.conf 
    depends_on:
    - Elasticsearch
    ports:
    - '9600:9600'
    networks:
    - elk

  Kibana:
    image: kibana:8.4.0
    container_name: kibana      
    ports:
    - '5601:5601'
    environment:
    - ELASTICSEARCH_URL=http://elasticsearch:9200  
    depends_on:
    - Elasticsearch  
    networks:
    - elk
volumes:
  elastic_data: {}

networks:
  elk:

and config for logstash:

input {
    file {
        path => "/test/Logs/test.slog"
        start_position => "beginning"
    }
}

output {
    elasticsearch {
        hosts => ["elasticsearch:9200"]
    }
}

test.slog exist and contain logs.

the logstash docker show the following logs:

[2022-08-27T20:40:32,592][INFO ][logstash.outputs.elasticsearch][main] Installing Elasticsearch template {:name=>"ecs-logstash"}
[2022-08-27T20:40:33,450][INFO ][logstash.javapipeline    ][main] Pipeline Java execution initialization time {"seconds"=>0.95}
[2022-08-27T20:40:33,451][INFO ][logstash.javapipeline    ][.monitoring-logstash] Pipeline Java execution initialization time {"seconds"=>0.94}
[2022-08-27T20:40:33,516][INFO ][logstash.javapipeline    ][.monitoring-logstash] Pipeline started {"pipeline.id"=>".monitoring-logstash"}
[2022-08-27T20:40:33,532][INFO ][logstash.inputs.file     ][main] No sincedb_path set, generating one based on the "path" setting {:sincedb_path=>"/usr/share/logstash/data/plugins/inputs/file/.sincedb_327fd1919fa26d08ec354604c3e1a1ce", :path=>["/test/Logs/test.slog"]}
[2022-08-27T20:40:33,559][INFO ][logstash.javapipeline    ][main] Pipeline started {"pipeline.id"=>"main"}
[2022-08-27T20:40:33,614][INFO ][filewatch.observingtail  ][main][8992bf4e2fad9d8838262d3019319d02ab5ffdcb5b282e821574485618753ce9] START, creating Discoverer, Watch with file and sincedb collections
[2022-08-27T20:40:33,625][INFO ][logstash.agent           ] Pipelines running {:count=>2, :running_pipelines=>[:".monitoring-logstash", :main], :non_running_pipelines=>[]}

But when I go to the Data -> Index Management there is nothing. and also in the Ingest pipeline.

What am I doing wrong?

Linux Dev
  • 133
  • 10

1 Answers1

1

In Elasticsearch 8 the index names created by logstash output follow the pattern .ds-logs-generic-default-%{+yyyy.MM.dd} instead of logstash-%{+yyyy.MM.dd}

This .ds index does not appear under Data -> Index Management but the documents can be queried

You can view the .ds-logs-generic index in Kibana, Management> Dev Tools using

GET _cat/indices

enter image description here

To query the documents you can use the _search API

GET /.ds-logs-generic-default-2022.08.28-000001/_search
{
  "query": {
    "match_all": {}
  }
}

enter image description here

If you want to specify the index name you can add it to the output section of your logstash.conf eg index => "logstash-%{+YYYY.MM.dd}"

output {
    elasticsearch {
        hosts => ["elasticsearch:9200"]
        index => "logstash-%{+YYYY.MM.dd}"
    }
}

The newly created index will show in Kibana under Management > Data > Index Management. You may need to add a few log lines at the end of your logfile to kick the indexing pipeline.

enter image description here

Honky Donkey
  • 611
  • 3
  • 10