Filebeat not creating index in Opensearch

Question

I have installed Filebeat-oss 7.12.0 and opensearch-2.4.0 and opensearchDashboard-2.4.0 on Windows.

Every service is working fine. But index is not getting created in Opensearch dashboard. There is no error.

Logs are:

INFO    log/harvester.go:302    Harvester started for file: D:\data\logs.txt
2022-12-08T18:28:17.584+0530    INFO    [crawler]   beater/crawler.go:141   Starting input (ID: 16780016071726099597)
2022-12-08T18:28:17.585+0530    INFO    [crawler]   beater/crawler.go:108   Loading and starting Inputs completed. Enabled inputs: 2
2022-12-08T18:28:17.585+0530    INFO    cfgfile/reload.go:164   Config reloader started
2022-12-08T18:28:17.584+0530    INFO    [input.filestream]  compat/compat.go:111    Input filestream starting
2022-12-08T18:28:17.585+0530    INFO    cfgfile/reload.go:224   Loading of config files completed.
2022-12-08T18:28:20.428+0530    INFO    [add_cloud_metadata]    add_cloud_metadata/add_cloud_metadata.go:101    add_cloud_metadata: hosting provider type not detected.
2022-12-08T18:28:21.428+0530    INFO    [publisher_pipeline_output] pipeline/output.go:143  Connecting to backoff(elasticsearch(http://localhost:9200))
2022-12-08T18:28:21.428+0530    INFO    [publisher] pipeline/retry.go:219   retryer: send unwait signal to consumer
2022-12-08T18:28:21.428+0530    INFO    [publisher] pipeline/retry.go:223     done
2022-12-08T18:28:21.433+0530    INFO    [esclientleg]   eslegclient/connection.go:314   Attempting to connect to Elasticsearch version 2.4.0
2022-12-08T18:28:21.537+0530    INFO    [esclientleg]   eslegclient/connection.go:314   Attempting to connect to Elasticsearch version 2.4.0
2022-12-08T18:28:21.620+0530    INFO    template/load.go:117    Try loading template filebeat-7.12.0 to Elasticsearch

filebeat.yml is:

filebeat.inputs:

- type: log

  
  paths:
    - D:\data\*
    
- type: filestream
  
  enabled: true

  # Paths that should be crawled and fetched. Glob based paths.
  paths:
    - D:\data\*
    
# ============================== Filebeat modules ==============================

filebeat.config.modules:
  # Glob pattern for configuration loading
  path: ${path.config}/modules.d/*.yml

  # Set to true to enable config reloading
  reload.enabled: false
  
 # ======================= Elasticsearch template setting =======================

setup.template.settings:
  index.number_of_shards: 1
  
#============================== Kibana =====================================
setup.kibana:
  
  host: "localhost:5601"
  
#-------------------------- Elasticsearch output ------------------------------
output.elasticsearch:
  # Array of hosts to connect to.
  hosts: ["localhost:9200"]
  
#================================ Processors =====================================

# Configure processors to enhance or manipulate events generated by the beat.

processors:
  - add_host_metadata:
      when.not.contains.tags: forwarded
  - add_cloud_metadata: ~
  - add_docker_metadata: ~
  - add_kubernetes_metadata: ~

I don't know what the problem is. No index is created in Opensearch dashboard with name filebeat-7.12.0.

score 0 · Accepted Answer · answered Jan 03 '23 at 13:46

0

@Android see my reply on this thread: https://stackoverflow.com/a/74984260/6101900.

You cannot forward events from filebeat to opensearch since its not elasticsearch.

answered Jan 03 '23 at 13:46

Ayush

326
1
5

score 0 · Answer 2 · answered May 15 '23 at 15:03

Try using filebeat version 7.10 ,filebeat version compatible to opensearch till 7.12 ,but some issue with 7.12 filebeat download.

You can download filebeat 7.10 version here:

https://www.elastic.co/downloads/past-releases/filebeat-7-10-0

If you are using filebeat version 7.12, then make sure you make change in opensearch cluster setting as below in devtools.

PUT /_cluster/settings
{
  "persistent" : {
    "compatibility.override_main_response_version" : true
  }
}

Ref link:

https://opensearch.org/docs/1.0/clients/agents-and-ingestion-tools/index/

https://repost.aws/questions/QUSda6PNQgRZWHTejuE6AE6A/aws-opensearch-1-2-4-with-filebeats-oss

Filebeat not creating index in Opensearch

2 Answers2