Let AWS EC2 access ElasticSearch behind Cognito Auth

Question

I have an AWS ElasticSearch cluster inside a VPC with a Kibana plugin. I'm trying to achieve 2 things:

1. Have Kibana accessible to the world, behind cognito authentication
2. Let my EC2 inside the same VPC write and read from this ElasticSearch cluster

I'm having problems with the second part.

I configured Cognito authentication to this cluster:

I made a User Pull with a domain name, and an Identity Pool, and I activated Kibana authentication with them.

Also, I changed the access policy to only allow access from the Cognito role:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Principal": {
        "AWS": [
          "arn:aws:iam::000000000000:role/Cognito_KibanaUsersAuth_Role"
        ]
      },
      "Action": "es:*",
      "Resource": "arn:aws:es:us-west-2:000000000000:domain/my-domain/*"
    }
  ]
}

(Like in this guide's step)

Before I did all this, I had EC2 instances in the VPC working freely with the ElasticSearch. I read that after I activate ES Cognito authentication, I need to sign my requests to ElasticSearch, and I did:

region = 'us-west-2'
service = 'es'
credentials = boto3.Session().get_credentials()
awsauth = AWS4Auth(credentials.access_key, credentials.secret_key, region, service)
requests.get(es_url, auth=awsauth)

(Like in this guide)

I'm getting this error: User: anonymous is not authorized to perform: es:ESHttpGet.

I want my EC2 to be authenticated in Cognito, so I can work with ES from there. How do I achieve this?

Answer 1

You have to grant permission to EC2 to access Elastic Search. Create a role allowing access to the Elastic Search domain and then assign that role to your EC2 instance. Update your access policy like this:

    {
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Principal": {
        "AWS": [
          "arn:aws:iam::000000000000:role/Cognito_KibanaUsersAuth_Role"
        ]
      },
      "Action": "es:*",
      "Resource": "arn:aws:es:us-west-2:000000000000:domain/my-domain/*"
    },
{
      "Effect": "Allow",
      "Principal": {
        "AWS": [
          EC2_ROLE_ARN_HERE
        ]
      },
      "Action": "es:*",
      "Resource": "arn:aws:es:us-west-2:000000000000:domain/my-domain/*"
    }
  ]
}

Answer 2

You are attempting to access ES via REST API endpoint as opposed to SDK. You will need to pass signed request to SDK class to access the ES as described in the linked you provided for signing request.

from elasticsearch import Elasticsearch, RequestsHttpConnection from requests_aws4auth import AWS4Auth import boto3

host = '' # For example, my-test-domain.us-east-1.es.amazonaws.com
region = '' # e.g. us-west-1

service = 'es'
credentials = boto3.Session().get_credentials()
awsauth = AWS4Auth(credentials.access_key, credentials.secret_key, region, service)

es = Elasticsearch(
    hosts = [{'host': host, 'port': 443}],
    http_auth = awsauth,
    use_ssl = True,
    verify_certs = True,
    connection_class = RequestsHttpConnection
)