Elasticsearch: how to make snapshot to minio s3?

Question

when i am trying to make snapshots to my s3 (minio deployed locally), by executing the following command inside my elasticsearch docker container:

curl -X PUT "localhost:9200/_snapshot/s3_repository/snapshot_1?wait_for_completion=true&pretty"

i got the following :

{
  "error" : {
    "root_cause" : [
      {
        "type" : "repository_exception",
        "reason" : "[s3_repository] Could not determine repository generation from root blobs"
      }
    ],
    "type" : "repository_exception",
    "reason" : "[s3_repository] Could not determine repository generation from root blobs",
    "caused_by" : {
      "type" : "i_o_exception",
      "reason" : "Exception when listing blobs by prefix [index-]",
      "caused_by" : {
        "type" : "sdk_client_exception",
        "reason" : "Unable to execute HTTP request: my-first-bucket.s3",
        "caused_by" : {
          "type" : "unknown_host_exception",
          "reason" : "my-first-bucket.s3"
        }
      }
    }
  },
  "status" : 500
}

Any help please ?

Notice, i created s3_repository using the followig :

curl -XPUT 'http://localhost:9200/_snapshot/s3_repository?verify=false&pretty' -H 'Content-Type: application/json' -d'
{
  "type": "s3",
  "settings": {
    "bucket": "my-first-bucket",
    "endpoint": "http://s3:9000"
  }
}'

Notice s3 in http://s3:9000 is the name of the service minio in my docker-compose.yml

Thank you in advance!

Does this host reachable in the scope of the docker-compose? — Netanel Malka, Jan 23 '22 at 19:40
Yes, the s3 container is reachable from elasticsearch container where i used aws cli — Mohamed, Jan 23 '22 at 20:13
Do you have bucket `my-first-bucket` created in minio? It should exist beforehand, ES doesn't create it. — ilvar, Jan 24 '22 at 17:53
Yes, i created `my-first-bucket` at run time, using the environnent variable like so : `services.s3.environment.MINIO_DEFAULT_BUCKETS: "my-first-bucket:public" ` in my docker-compose.yml — Mohamed, Jan 24 '22 at 20:31
`version: '3.9' services: elasticsearch: build: ./repo ports: - 9200:9200 - 9300:9300 volumes: - ./elastic:/usr/share/elasticsearch/data environment: - discovery.type=single-node s3: image: bitnami/minio:2021 ports: - 9000:9000 - 9001:9001 environment: MINIO_ROOT_USER: benighil MINIO_ROOT_PASSWORD: marsmars MINIO_DEFAULT_BUCKETS: my-first-bucket:public,my-second-bucket volumes: - ./minio:/data volumes: elastic: minio:` — Mohamed, Jan 24 '22 at 20:43
`FROM elasticsearch:7.16.3 RUN ./bin/elasticsearch-plugin install repository-s3 --batch RUN echo "benighil" | bin/elasticsearch-keystore add --stdin --force s3.client.default.access_key RUN echo "marsmars" | bin/elasticsearch-keystore add --stdin --force s3.client.default.secret_key` — Mohamed, Jan 24 '22 at 20:47

score 0 · Answer 1 · answered Feb 24 '22 at 19:16

Most likely the problem here is that the bucket is not created at MinIO startup. The environment variable MINIO_DEFAULT_BUCKETS is not officially supported by MinIO at least. After MinIO starts, you can try to confirm if the bucket exists as expected.

As the bucket only needs to be created once, you could do it via a script after MinIO starts.

Please know you can find us at https://slack.min.io/ 24/7/365. If you have commercial questions, please reach out to us on hello@min.io or on our Ask an Expert Chat functionality at https://min.io/pricing?action=talk-to-us.

Elasticsearch: how to make snapshot to minio s3?

1 Answers1