AWS Elasticsearch python implementation

Question

I am writing AWS Elasticsearch snapshots backup/restore implementation using python boto. I was able to register AWS S3 as ES snapshots repository. Here is the snippet of my create snapshots function.

import boto 
import urllib
from boto.connection import AWSAuthConnection

class ESConnection(AWSAuthConnection):

def __init__(self, region, **kwargs):
  super(ESConnection, self).__init__(**kwargs)
  self._set_auth_region_name(region)
  self._set_auth_service_name("es")

def _required_auth_capability(self):
  return ['hmac-v4']
def create_snapshots(profile, region_name, es_host,total_snapshots_to_keep):
.
.
.
url = "/_snapshot/es_repository/{}?".format(snapshot_name)
flag = urllib.urlencode({'wait_for_completion':'true'})
resp = client.make_request(method='PUT',
    path=urllib.quote("{}{}".format(url,flag)),
    data='{"indices": "' + audit_index_name + '", "ignore_unavailable": true, "include_global_state": false}')

Main problem seems to be in above snippet where I am trying to construct an url for

{u'status': 400, u'error': {u'root_cause': [{u'reason': u'[es_repository:v4_2017_03_27_snapshot?wait_for_completion=true] Invalid snapshot name [v4_2017_03_27_snapshot?wait_for_completion=true], must not contain the following characters [\, /, *, ?, ", <, >, |, , ,]', u'type': u'invalid_snapshot_name_exception'}], u'type': u'invalid_snapshot_name_exception', u'reason': u'[es_repository:v4_2017_03_27_snapshot?wait_for_completion=true] Invalid snapshot name [v4_2017_03_27_snapshot?wait_for_completion=true], must not contain the following characters [\, /, *, ?, ", <, >, |, , ,]'}}

It is taking my actual snapshot_name and wait_for_completion flag entirely as a snapshot name

Invalid snapshot name [v4_2017_03_27_snapshot?wait_for_completion=true], must not contain the following characters [\, /, *, ?, ", <, >, |, , ,]'}}

Could you please help me pointing out place where I am doing it wrongly in constructing url for elasticsearch? Or is there any better way to accomplish this?

mootmoot · Answer 1 · 2017-04-10T08:09:14.230

The error message already give you the answer. There is some special symbol not supported by s3 object name.

# the ? ending is wrong, perhaps residual of typo from other language? 
url = "/_snapshot/es_repository/{}?".format(snapshot_name)

# correct path 
url = "/_snapshot/es_repository/{}".format(snapshot_name)

(update)

When you use curl , you are using the RESTful API. The "?" mark is notify RESTful API that parameter is follow behind.

However, in your code, you use client.make_request, which is part of boto API. the make_request module only accept typical S3 "path" AKA valid S3 object name.

If you want to simulate the same RESTful path, then I suggest you use python requests with path you just construct. i.e.

import requests
path = urllib.quote("{}{}".format(url,flag))
endpoint_url = "http://.."  # find your snapshot S3 endpoint url and put here   
url = "{}/{}".format(endpoint_url, path) 
data='{"indices": "' + audit_index_name + '", "ignore_unavailable": true, "include_global_state": false}'
rest_resp= requests.post(url, data = data)

Due to boto/boto2 confusing documentation and deprecate issues, you should use boto3

Yes, for that only I need an answer. Basic problem is in constricting of URL itself. I tried placing '?' mark at different places i.e at '/_snapshot/es_repository/{}?' or in snapshot_name or not placing it anywhere. One which I showed in this question is one of the use case And I need help in constructing the url. If I do it manually like curl -XPUT 'localhost:9200/_snapshot/my_backup/snapshot_1?wait_for_completion=true&pretty' And if do using script it fails complaining '?'. — Pankaj Kolhe, Apr 10 '17 at 04:25

AWS Elasticsearch python implementation

1 Answers1