I am using Elasticsearch version 6.2.4, its plugins searchguard 6.2.4-12, s3-repository-plugin. SSL is disabled for searchguard in elasticsearch.yml using setting
searchguard.ssl.http.enabled: false
and snapshot restore is enable for any user on serarchguard using setting:
searchguard.enable_snapshot_restore_privilege: true
For testing purpose I created index as:
curl -uUSERNAME:PASSWORD -X PUT "localhost:9200/filebeat-2018.04.11" -H 'Content-Type: application/json' -d'
{
"settings" : {
"index" : {
"number_of_shards" : 3,
"number_of_replicas" : 2
}
}
}
'
This successfully created index. Then I create its snapshot using this script:
daysagoyear=$(date --date="30 days ago" +'%Y')
daysagomonth=$(date --date="30 days ago" +'%m')
daysagoday=$(date --date="30 days ago" +'%d')
INDEX_PREFIXES='filebeat-'
es_username="USERNAME"
es_password="PASSWORD"
indices=`curl -u "$es_username":"$es_password" localhost:9200/_cat/indices?v|grep $INDEX_PREFIXES|awk '{print $3}'`
for index in $indices
do
index_date=`echo "$index"|cut -d "-" -f2`
index_date=`echo "$index_date"|tr . -`
index_date_yr=`date -d $index_date "+%Y"`
index_date_mon=`date -d $index_date "+%m"`
index_date_day=`date -d $index_date "+%d"`
delete=0
SNAPSHOT_NAME=${INDEX_PREFIXES}${index_date}"-snapshot"
bucket_name="elklogsireland"
if [ "$daysagoyear" -gt "$index_date_yr" ]
then
delete=1
elif [ "$daysagoyear" -eq "$index_date_yr" -a "$daysagomonth" -gt "$index_date_mon" ]
then
delete=1
elif [ "$daysagoyear" -eq "$index_date_yr" -a "$daysagomonth" -eq "$index_date_mon" -a "$daysagoday" -ge "$index_date_day" ]
then
delete=1
fi
if [ $delete -eq 1 ]
then
echo "Creating snapshot of $index ..."
# Setting Base Path for S3 Bucket
#curlsettingstring="-d \'{\"type\": \"s3\", \"settings\": {\"bucket\": \"${bucket_name}\", \"base_path\": \"${index_date}\" }}\'"
curl -u $es_username:$es_password -XPUT "localhost:9200/_snapshot/$bucket_name" -H 'Content-Type: application/json' -d '{
"type": "s3",
"settings": {
"bucket": "'$bucket_name'",
"base_path": "'${INDEX_PREFIXES}${index_date}'"
}
}'
curl -u $es_username:$es_password -XPUT "http://localhost:9200/_snapshot/$bucket_name/$SNAPSHOT_NAME?wait_for_completion=true" -H 'Content-Type: application/json' -d '{
"indices": "'${index}'",
"ignore_unavailable": "true",
"include_global_state": false
}'
if [ $? -eq 0 ];then
echo "Removing $index ...."
curl -u $es_username:$es_password -XDELETE "http://localhost:9200/$index"
else
echo "$(date +"%Y-%m-%d:%H:%M:%S") ---- Unable to form snapshot $SNAPSHOT_NAME on s3" >> /var/log/messages
fi
fi
done
This script aims at creating snapshot of 30 days old indices and upload them to s3 bucket and then remove them. In process, It upload index snapshot files to folder on s3 that is named after index. It runs successfully and also upload snapshot files to s3 bucket. Now when I restore it, I am running script as:
if [ $# -lt 1 ]
then
echo "Missing argument. Please provide index name."
exit 1
fi
es_username="USERNAME"
es_password="PASSWORD"
bucket_name="elklogsireland"
index_name=$1
echo "Index Name: ${index_name}"
curl -u $es_username:$es_password -XPOST "localhost:9200/_snapshot/${bucket_name}/${index_name}-snapshot/_restore" -H 'Content-Type: application/json' -d '{
"indices": "'$index_name'",
"ignore_unavailable": "true",
"include_global_state": false
}'
It takes index name in argument. And as I run it following is returned back:
{"snapshot":{"snapshot":"filebeat-2018-04-10-snapshot","indices":[],"shards":{"total":0,"failed":0,"successful":0}}}
and index not actually formed. Please tell what I am missing and what need to be done further.