0

I have a lambda function which is trying to read data from elastic search running in an ec2 machine. Both of them are in the same VPC, subnet and have the same security group assigned to it. The Lambda can't seem to access the elastic search instance.

const AWS = require('aws-sdk');
const elasticsearch = require('elasticsearch');

exports.handler = async function (event, context, callback) {

  const client = new elasticsearch.Client({
    host: 'public_dns:9200',
    httpAuth: 'user:password'
  });

  let self = this;

  client.search({
    index: 'index_name',
    scroll: '30s',
    size: 10000,
    body: {
      query: {
        match_all: {}
      }
    }
  })
    .then(response => {
      self.responseString = response.hits.hits;
      console.log(response.hits.hits);
    })
    .catch(error => {
      console.error(error);
    });

  const responseData = {
    statusCode: 200,
    body: JSON.stringify({
      message: self.responseString
    })
  };

  callback(null, responseData);
};

The error i get from lambda is

2023-02-06T23:19:54.890Z    fcd62836-4fe3-4c6a-9871-ee70668ba07c    ERROR   StatusCodeError: Request Timeout after 30000ms
    at /var/task/node_modules/elasticsearch/src/lib/transport.js:397:9
    at Timeout.<anonymous> (/var/task/node_modules/elasticsearch/src/lib/transport.js:429:7)
    at listOnTimeout (node:internal/timers:559:17)
    at processTimers (node:internal/timers:502:7) {
  status: undefined,
  displayName: 'RequestTimeout',
  body: undefined

Any tips on how to debug this would be highly appreciated. Note: I do not wish to make the elastic search endpoint accessible to public.

George Joseph
  • 39
  • 1
  • 8
  • If they are both in the same VPC, why are you using `public_dns`? Does it resolve to a **private IP address** if it is used within the VPC? – John Rotenstein Feb 07 '23 at 01:35

1 Answers1

2

Merely putting two resources "in the same Security Group" does not mean that they are able to communicate with each other. In fact, resources are not 'inside' a Security Group -- rather they are associated with a Security Group.

Security Group rules are applied to each resource individually. This means that if both resources are associated with the same security group, there needs to be a specific rule that allows incoming access from the security group to itself.

Instead of using one Security Group, a preferred configuration would be:

  • A security group on the AWS Lambda function (lambda-SG) with the default "Allow All" outbound rules, and
  • A security group on the Amazon EC2 instance running Elasticsearch (elastic-SG) that permits Inbound access on port 9200 from lambda-SG

That is, elastic-SG specifically references lambda-SG when permitting the inbound access. This means that traffic from the Lambda function will be allowed to communicate with the EC2 instance on that port.

John Rotenstein
  • 241,921
  • 22
  • 380
  • 470
  • Hey John, Thanks for pointing out the mistake in my approach. Adding 2 security groups and following the steps as you have mentioned helped me in resolving the issue. Thank you very much – George Joseph Feb 07 '23 at 02:04
  • Ah, glad it helped! Security Groups are very powerful, but they require a different mindset to traditional networking. – John Rotenstein Feb 07 '23 at 03:01