13

I want to sync my MongoDB data to ElasticSearch, I read a lot of posts talking about elasticsearch river plugin and mongo connector, but all of them are deprecated for mongo 4 and elasticsearch 7!

As logstash is a proprietary software I would like to use it to sync both... Anyone knows how it's possible to do it?

Nick is tired
  • 6,860
  • 20
  • 39
  • 51
Marco Sousa
  • 185
  • 1
  • 1
  • 9

2 Answers2

4

You may sync MongoDB and Elasticsearch with Logstash; syncing is, in fact, one of the major applications of Logstash. After installing Logstash, all that you need to do is specify a pipeline for your use case: one or more input sources (MongoDB in your case) and one or more output sinks (Elasticsearch in your case), put as a config file (example follows) inside Logstash's pipeline directory; Logstash takes care of the rest.

Logstash officially provides plugins for a lot of commonly used data sources and sinks; those plugins let you read data from and write data to various sources with just a few configuration. You just need to find the right plugin, install it, and configure it for your scenario. Logstash has an official output plugin for Elasticsearch and its configuration is pretty intuitive. Logstash, however, doesn't provide any input plugin for MongoDB. You need to find a third party one; this one seems to be pretty promising.

In the end, your pipeline may look something like the following:

input {
  mongodb {
    uri => 'mongodb://10.0.0.30/my-logs?ssl=true'
    placeholder_db_dir => '/opt/logstash-mongodb/'
    placeholder_db_name => 'logstash_sqlite.db'
    collection => 'events_'
    batch_size => 5000
  }
}
output {
  stdout {
    codec => rubydebug #outputs the same thing as elasticsearch in stdout to facilitate debugging
  }
  elasticsearch {
    hosts => "localhost:9200"
    index => "target_index"
    document_type => "document_type"
    document_id => "%{id}"
  }
}
Farhan Nasim
  • 773
  • 4
  • 13
  • 3
    Note on that input plugin: _"This was designed for parsing logs that were written into mongodb. This means that it may not re-parse db entries that were changed and already parsed."_ So, only useful in a very limited number of use cases. – Madbreaks Dec 19 '19 at 22:36
  • In other words, the [logstash-input-mongodb](https://github.com/phutchins/logstash-input-mongodb) plugin doesn't support tracking "delete" and "update" of documents, and seems to not being maintained – Gabriel Gimenes Dec 23 '22 at 01:12
0

I am developing a simple package that synchronizes the ElasticSearch server against MongoDB.

This package accepts an array of collections for MongoDB, each with a collection name and pipeline.

For example, the following code snippet starts synchronization by specifying MongoDb URI and name, collections array, and ElasticSearch options as they are from the ElasticSearch client package:

const mongoDbOptions: MongoDbOptions = {
    uri: config.mongodbURI,
    dbName: config.dbName,
    collections: [
        {
            name: 'collection-name',
            pipeline: [{ $match: { name: { $regex: 'jan' } } }, { $group: { _id: '$name', totalDocs: { $sum: 1 } } }],
        },
    ],
};
const elasticOptions: ElasticOptions = {
    node: 'http://localhost:9200',
    auth: {
        username: 'elastic',
        password: 'elastic-user-password',
    },
    // tls: {
    //  ca: fs.readFileSync('./http_ca.crt'),
    //  rejectUnauthorized: false,
    // },
};
const syncDbWithElasticsearch = new SyncMongoDbWithElasticSearch(mongoDbOptions, elasticOptions);
syncDbWithElasticsearch.start();

The package is available for download and usage via npm and can be found in this link.

Majid Rafei
  • 59
  • 9