0

I have installed Filebeat for forwarding and centralizing log data. Filebeat is mainly used with Elasticsearch (directly sends the transactions). I use Opensearch and OpenSearch Dashboards instead of Elasticsearch and Kibana (Opensearch is a forked search project based on old versions of Elasticsearch and Kibana). Both, Filebeat and Opensearch are installed as tarballs on my VirtualBox VDI. I currently use this config file: (comments removed)

###################### Filebeat Configuration #########################


filebeat.inputs:
ma- type: log
  enabled: false
  paths:
    - /var/log/*.log


filebeat.config.modules:
  path: ${path.config}/modules.d/*.yml
  reload.enabled: false

setup.template.settings:
  index.number_of_shards: 1
setup.kibana:

  host: "localhost:5601"

output.elasticsearch:
  # Array of hosts to connect to.
  hosts: ["localhost:9200"]

  username: "admin"
  password: "admin"

processors:
  
  - decode_json_fields:
      fields: ["message"]
      overwrite_keys: true
      target: ""

However, still cannot run ./filebeat setup -e and get error:

2021-09-23T17:55:11.865+0300    ERROR   instance/beat.go:989    Exiting: couldn't connect to any of the configured Elasticsearch hosts. Errors: [error connecting to Elasticsearch at http://localhost:9200: Get "http://localhost:9200": dial tcp 127.0.0.1:9200: connect: connection refused]
Exiting: couldn't connect to any of the configured Elasticsearch hosts. Errors: [error connecting to Elasticsearch at http://localhost:9200: Get "http://localhost:9200": dial tcp 127.0.0.1:9200: connect: connection refused]

I use opensearch-1.0.1 version, and was able to run Opensearch without issues. But can somebody enlight me, how to connect Filebeat with Opensearch, if both are installed as tarballs?

Thank you!

raven19
  • 1
  • 1
  • 1

2 Answers2

1

Filebeat doesn't support OpenSearch since it's not ElasticSearch. You cannot ingest logs like this directly but probably using Opensearch project logstash (and not from Elastic) can be targeted. See:

  1. https://discuss.elastic.co/t/filebeat-with-opensearch/295915
  2. https://opensearch.org/docs/1.2/clients/logstash/index/
Ayush
  • 326
  • 1
  • 5
1

Try using filebeat version 7.10 ,filebeat version compatible to opensearch till 7.12 ,but some issue with 7.12 filebeat download.

You can download filebeat 7.10 version here:

https://www.elastic.co/downloads/past-releases/filebeat-7-10-0

If you are using filebeat version 7.12, then make sure you make change in opensearch cluster setting as below in devtools.

PUT /_cluster/settings
{
  "persistent" : {
    "compatibility.override_main_response_version" : true
  }
}

Ref link:

https://opensearch.org/docs/1.0/clients/agents-and-ingestion-tools/index/

https://repost.aws/questions/QUSda6PNQgRZWHTejuE6AE6A/aws-opensearch-1-2-4-with-filebeats-oss

Divyank
  • 811
  • 2
  • 10
  • 26