Dump all documents of Elasticsearch

Question

Is there any way to create a dump file that contains all the data of an index among with its settings and mappings?

A Similar way as mongoDB does with mongodump
or as in Solr its data folder is copied to a backup location.

Cheers!

Answer 1

Here's a new tool we've been working on for exactly this purpose https://github.com/taskrabbit/elasticsearch-dump. You can export indices into/out of JSON files, or from one cluster to another.

Answer 2

Elasticsearch supports a snapshot function out of the box:

https://www.elastic.co/guide/en/elasticsearch/reference/current/modules-snapshots.html

Answer 3

We can use elasticdump to take the backup and restore it, We can move data from one server/cluster to another server/cluster.

1. Commands to move one index data from one server/cluster to another using elasticdump.

# Copy an index from production to staging with analyzer and mapping:
elasticdump \
  --input=http://production.es.com:9200/my_index \
  --output=http://staging.es.com:9200/my_index \
  --type=analyzer
elasticdump \
  --input=http://production.es.com:9200/my_index \
  --output=http://staging.es.com:9200/my_index \
  --type=mapping
elasticdump \
  --input=http://production.es.com:9200/my_index \
  --output=http://staging.es.com:9200/my_index \
  --type=data

2. Commands to move all indices data from one server/cluster to another using multielasticdump.

Backup

multielasticdump \
  --direction=dump \
  --match='^.*$' \
  --limit=10000 \
  --input=http://production.es.com:9200 \
  --output=/tmp 

Restore

multielasticdump \
  --direction=load \
  --match='^.*$' \
  --limit=10000 \
  --input=/tmp \
  --output=http://staging.es.com:9200 

Note:

• If the --direction is dump, which is the default, --input MUST be a URL for the base location of an ElasticSearch server (i.e. http://localhost:9200) and --output MUST be a directory. Each index that does match will have a data, mapping, and analyzer file created.

• For loading files that you have dumped from multi-elasticsearch, --direction should be set to load, --input MUST be a directory of a multielasticsearch dump and --output MUST be a Elasticsearch server URL.

• The 2nd command will take a backup of settings, mappings, template and data itself as JSON files.

• The --limit should not be more than 10000 otherwise, it will give an exception.

• Get more details here.

Answer 4

For your case Elasticdump is the perfect answer.
First, you need to download the mapping and then the index

# Install the elasticdump 
npm install elasticdump -g

# Dump the mapping 
elasticdump --input=http://<your_es_server_ip>:9200/index --output=es_mapping.json --type=mapping

# Dump the data
elasticdump --input=http://<your_es_server_ip>:9200/index --output=es_index.json --type=data    

If you want to dump the data on any server I advise you to install esdump through docker. You can get more info from this website Blog Link

Answer 5

ElasticSearch itself provides a way to create data backup and restoration. The simple command to do it is:

CURL -XPUT 'localhost:9200/_snapshot/<backup_folder name>/<backupname>' -d '{
    "indices": "<index_name>",
    "ignore_unavailable": true,
    "include_global_state": false
}'

Now, how to create, this folder, how to include this folder path in ElasticSearch configuration, so that it will be available for ElasticSearch, restoration method, is well explained here. To see its practical demo surf here.

Answer 6

The data itself is one or more lucene indices, since you can have multiple shards. What you also need to backup is the cluster state, which contains all sorts of information regarding the cluster, the available indices, their mappings, the shards they are composed of etc.

It's all within the data directory though, you can just copy it. Its structure is pretty intuitive. Right before copying it's better to disable automatic flush (in order to backup a consistent view of the index and avoiding writes on it while copying files), issue a manual flush, disable allocation as well. Remember to copy the directory from all nodes.

Also, next major version of elasticsearch is going to provide a new snapshot/restore api that will allow you to perform incremental snapshots and restore them too via api. Here is the related github issue: https://github.com/elasticsearch/elasticsearch/issues/3826.

Answer 7

You can also dump elasticsearch data in JSON format by http request: https://www.elastic.co/guide/en/elasticsearch/reference/current/search-request-scroll.html
CURL -XPOST 'https://ES/INDEX/_search?scroll=10m'
CURL -XPOST 'https://ES/_search/scroll' -d '{"scroll": "10m", "scroll_id": "ID"}'

Answer 8

At the time of writing this answer(2021), the official way of backing up an ElasticSearch cluster is to snapshot it. Refer to: https://www.elastic.co/guide/en/elasticsearch/reference/current/snapshot-restore.html

Answer 9

To export all documents from ElasticSearch into JSON, you can use the esbackupexporter tool. It works with index snapshots. It takes the container with snapshots (S3, Azure blob or file directory) as the input and outputs one or several zipped JSON files per index per day. It is quite handy when exporting your historical snapshots. To export your hot index data, you may need to make the snapshot first (see the answers above).

Answer 10

If you want to massage the data on its way out of Elasticsearch, you might want to use Logstash. It has a handy Elasticsearch Input Plugin.

And then you can export to anything, from a CSV file to reindexing the data on another Elasticsearch cluster. Though for the latter you also have the Elasticsearch's own Reindex.