Connecting to AWS Elasticsearch within VPC from Lambda over HTTPS

Question

I have a Lambda written in Python which writes some data to Elasticsearch hosted on AWS. The ES service is within a VPC, so I'm trying to use the internal DNS of the ES to connect to it. This is my code:

        es_client = Elasticsearch(
            hosts=[{'host': es_host, 'port': 443}],
            http_auth=aws_auth,
            use_ssl=True,
            verify_certs=True,
            connection_class=RequestsHttpConnection
        )

However, I get this exception:

ssl.CertificateError: hostname 'x.y.internal' doesn't match '*.us-west-2.es.amazonaws.com

I don't wan't to use the public hostname because it is going to keep changing. How do I connect to the ES service using it's internal DNS?

====== UPDATE =======

I'm able to connect to the ES domain using HTTP with the below code:

es_client = Elasticsearch(
            hosts=[{'host': es_host, 'port': 80}]
        )

But how do I connect over HTTPS?

Have you configured your Lambda to access resources in your VPC? — K Mo, Jun 14 '18 at 06:17

score 0 · Answer 1 · answered Apr 08 '21 at 16:20

Got into similar issue using AWS.HttpClient. This happens when you connect to generated VPC endpoint of ES over https. You have to disable cert verification:

es_client = Elasticsearch(
    hosts=[{'host': es_host, 'port': 443}],
    http_auth=aws_auth,
    use_ssl=True,
    verify_certs=False,
    connection_class=RequestsHttpConnection
)

In case you are using AWS.HttpClient like me you have to disable it like this:

const AWS = require('aws-sdk');
const https = require('https');
AWS.NodeHttpClient.sslAgent = new https.Agent({ rejectUnauthorized: false });

const httpClient = new AWS.HttpClient();

score -1 · Answer 2 · answered Jun 14 '18 at 11:52

-1

You need to use the host ending in .us-west-2.es.amazonaws.com as that is the domain in the SSL certificate that Elasticsearch is sending. If the hostname for the internal DNS is different then that connection will not work as the certificates don't match.

answered Jun 14 '18 at 11:52

Honza Král

2,982
14
11

But that address is going to keep changing. Isn't there a static address I can use to connect? – drunkenfist Jun 14 '18 at 16:13

Connecting to AWS Elasticsearch within VPC from Lambda over HTTPS

2 Answers2