6

I'm using python to access elasticsearch cluster. Now I want to backup my index by using snapshot. The most difficult thing is that: the python-elasticsearch's doc just give me a API description. there is no example to show me how to create snapshot. I tried some parameters, but failed. Can anyone give a snapshot example of elastic-search by using python? The following is my code:

from elasticsearch import Elasticsearch
es = Elasticsearch()
snapshot_body = {
"type": "url",
"settings": {
        "url":  "http://download.elasticsearch.org/definitiveguide/sigterms_demo/"
    }
}
body = {"snapshot": snapshot_body}
es.snapshot.create_repository(repository='test', body=body)
Howardyan
  • 667
  • 1
  • 6
  • 15

2 Answers2

8

Your repository creation is almost correct, you don't need the line body = {"snapshot": snapshot_body}, simply create your repository like this:

es.snapshot.create_repository(repository='test', body=snapshot_body)

Now in order to create a snapshot, all you have to do is this:

es.snapshot.create(repository='test', snapshot='my_snapshot')

If you want to store only a few indices and not all you can also provide a body like this:

index_body = {
  "indices": "index_1,index_2"
}
es.snapshot.create(repository='test', snapshot='my_snapshot', body=index_body)
Val
  • 207,596
  • 13
  • 358
  • 360
  • Hi Val, Thank you for your reply. But I run the command in ipython shell and get the error: NotFoundError: TransportError(404, u'repository_missing_exception', u'[test] missing') my elasticsearch version is 2.3.5, and the python-elasticsearch version is 5.0.1 – Howardyan Dec 19 '16 at 06:30
  • Since you're using ES 2.3.5, you need to use version 2.x of python-elasticsearch according to the [compatibility table](https://github.com/elastic/elasticsearch-py#compatibility) – Val Dec 19 '16 at 06:39
  • HI Val, could you tell me your es version and python version. I can try yours. – Howardyan Dec 19 '16 at 07:17
  • It doesn't matter, all I'm saying is that your python-es version must be compatible with your ES server version, since you have ES 2.3.5 you cannot have python-es 5.0.1, you need python-es 2.3.0 or 2.4.0 – Val Dec 19 '16 at 07:28
0

Save the following sample Python code as a Python file, such as register-repo.py. The client requires the AWS SDK for Python (Boto3), requests and requests-aws4auth packages. The client contains commented-out examples for other snapshot operations.

import boto3
import requests
from requests_aws4auth import AWS4Auth

host = '' # include https:// and trailing /
region = '' # e.g. us-west-1
service = 'es'
credentials = boto3.Session().get_credentials()
awsauth = AWS4Auth(credentials.access_key, credentials.secret_key, region, service, session_token=credentials.token)

# Register repository

path = '_snapshot/my-snapshot-repo-name' # the Elasticsearch API endpoint
url = host + path

payload = {
  "type": "s3",
  "settings": {
    "bucket": "s3-bucket-name",
    # "endpoint": "s3.amazonaws.com", # for us-east-1
    "region": "us-west-1", # for all other regions
    "role_arn": "arn:aws:iam::123456789012:role/TheSnapshotRole"
  }
}

headers = {"Content-Type": "application/json"}

r = requests.put(url, auth=awsauth, json=payload, headers=headers)

print(r.status_code)
print(r.text)

# # Take snapshot
#
# path = '_snapshot/my-snapshot-repo/my-snapshot'
# url = host + path
#
# r = requests.put(url, auth=awsauth)
#
# print(r.text)
#
# # Delete index
#
# path = 'my-index'
# url = host + path
#
# r = requests.delete(url, auth=awsauth)
#
# print(r.text)
#
# # Restore snapshot (all indices except Kibana and fine-grained access control)
#
# path = '_snapshot/my-snapshot-repo/my-snapshot/_restore'
# url = host + path
#
# payload = {
#   "indices": "-.kibana*,-.opendistro_security",
#   "include_global_state": false
# }
#
# headers = {"Content-Type": "application/json"}
#
# r = requests.post(url, auth=awsauth, json=payload, headers=headers)
#
# # Restore snapshot (one index)
#
# path = '_snapshot/my-snapshot-repo/my-snapshot/_restore'
# url = host + path
#
# payload = {"indices": "my-index"}
#
# headers = {"Content-Type": "application/json"}
#
# r = requests.post(url, auth=awsauth, json=payload, headers=headers)
#
# print(r.text)

DONT USE THIS IN US EAST 1 then you have to use this Important If the S3 bucket is in the us-east-1 region, you must use "endpoint": "s3.amazonaws.com" instead of "region": "us-east-1".

To enable server-side encryption with S3-managed keys for the snapshot repository, add "server_side_encryption": true to the "settings" JSON.

https://docs.aws.amazon.com/elasticsearch-service/latest/developerguide/es-managedomains-snapshots.html#es-managedomains-snapshot-registerdirectory

Subrata Fouzdar
  • 724
  • 5
  • 17