0

I am using Elasticsearch version 6.2.4, its plugins searchguard 6.2.4-12, s3-repository-plugin. SSL is disabled for searchguard in elasticsearch.yml using setting

searchguard.ssl.http.enabled: false

and snapshot restore is enable for any user on serarchguard using setting:

searchguard.enable_snapshot_restore_privilege: true

For testing purpose I created index as:

curl -uUSERNAME:PASSWORD -X PUT "localhost:9200/filebeat-2018.04.11" -H 'Content-Type: application/json' -d'
{
    "settings" : {
        "index" : {
            "number_of_shards" : 3, 
            "number_of_replicas" : 2 
        }
    }
}
'

This successfully created index. Then I create its snapshot using this script:

daysagoyear=$(date --date="30 days ago" +'%Y')
daysagomonth=$(date --date="30 days ago" +'%m')
daysagoday=$(date --date="30 days ago" +'%d')

INDEX_PREFIXES='filebeat-'
es_username="USERNAME"
es_password="PASSWORD"

indices=`curl -u "$es_username":"$es_password" localhost:9200/_cat/indices?v|grep $INDEX_PREFIXES|awk '{print $3}'`

for index in $indices
do
        index_date=`echo "$index"|cut -d "-" -f2`
        index_date=`echo "$index_date"|tr . -`
        index_date_yr=`date -d $index_date "+%Y"`
        index_date_mon=`date -d $index_date "+%m"`
        index_date_day=`date -d $index_date "+%d"`
        delete=0
        SNAPSHOT_NAME=${INDEX_PREFIXES}${index_date}"-snapshot"
        bucket_name="elklogsireland"
        if [ "$daysagoyear" -gt "$index_date_yr" ]
        then
            delete=1
        elif [ "$daysagoyear" -eq "$index_date_yr" -a "$daysagomonth" -gt "$index_date_mon" ]
        then
            delete=1
        elif [ "$daysagoyear" -eq "$index_date_yr" -a "$daysagomonth" -eq "$index_date_mon" -a "$daysagoday" -ge "$index_date_day" ]
        then
            delete=1
        fi
        if [ $delete -eq 1 ]
        then
          echo "Creating snapshot of $index ..."
          # Setting Base Path for S3 Bucket
          #curlsettingstring="-d \'{\"type\": \"s3\", \"settings\": {\"bucket\": \"${bucket_name}\", \"base_path\": \"${index_date}\" }}\'"
          curl -u $es_username:$es_password -XPUT "localhost:9200/_snapshot/$bucket_name" -H 'Content-Type: application/json' -d '{
     "type": "s3",
     "settings": {
       "bucket": "'$bucket_name'",
       "base_path": "'${INDEX_PREFIXES}${index_date}'"
     }
}'
          curl -u $es_username:$es_password -XPUT "http://localhost:9200/_snapshot/$bucket_name/$SNAPSHOT_NAME?wait_for_completion=true" -H 'Content-Type: application/json' -d '{
                "indices": "'${index}'",
                "ignore_unavailable": "true",
                "include_global_state": false
        }'
          if [ $? -eq 0 ];then
             echo "Removing $index ...."
             curl -u $es_username:$es_password -XDELETE "http://localhost:9200/$index"
          else
               echo "$(date +"%Y-%m-%d:%H:%M:%S") ---- Unable to form snapshot $SNAPSHOT_NAME  on s3" >> /var/log/messages
          fi
        fi
done

This script aims at creating snapshot of 30 days old indices and upload them to s3 bucket and then remove them. In process, It upload index snapshot files to folder on s3 that is named after index. It runs successfully and also upload snapshot files to s3 bucket. Now when I restore it, I am running script as:

if [ $# -lt 1 ]
then
  echo "Missing argument. Please provide index name."
  exit 1
fi
es_username="USERNAME"
es_password="PASSWORD"
bucket_name="elklogsireland"
index_name=$1
echo "Index Name: ${index_name}"

curl -u $es_username:$es_password -XPOST "localhost:9200/_snapshot/${bucket_name}/${index_name}-snapshot/_restore" -H 'Content-Type: application/json' -d '{
  "indices": "'$index_name'",
  "ignore_unavailable": "true",
  "include_global_state": false
}'

It takes index name in argument. And as I run it following is returned back:

{"snapshot":{"snapshot":"filebeat-2018-04-10-snapshot","indices":[],"shards":{"total":0,"failed":0,"successful":0}}}

and index not actually formed. Please tell what I am missing and what need to be done further.

srgbnd
  • 5,404
  • 9
  • 44
  • 80
YATIN GUPTA
  • 916
  • 9
  • 17
  • 1
    I can't know for certain without knowing exactly which parameters you're submitting to your shell scripts, but it very much appears that your index is named "filebeat-2018.04.11", but you're trying to restore an index named "filebeat-2018-04-10" from your snapshot file. There are no such indices in that snapshot and so Elasticsearch is successfully restoring all 0 of the matching indices. – Jason Rosendale May 30 '18 at 15:21
  • Thanks, good catch. I was doing mistake in providing indices. – YATIN GUPTA May 31 '18 at 04:00

0 Answers0