I am trying to index data being received by an API to an AWS open search domain(using elastic search 7.10), the following is the code for my lambda function
# Dependency imports
from jose import jwt
import json
from pymongo import MongoClient
import requests
from requests_aws4auth import AWS4Auth
import os
import boto3
credentials = boto3.Session().get_credentials()
awsauth = AWS4Auth(credentials.access_key, credentials.secret_key, 'us-east-1',
'opensearchservice'
, session_token=credentials.token)
client=MongoClient('mongodb+srv://{}:{}@cluster0.66zot.mongodb.net/database?
retryWrites=true&w=majority'.format(os.environ['DB_USER'],os.environ['DB_PASS']))
db=client.database
def lambda_handler(event, context):
claims=jwt.get_unverified_claims(event['headers']['Authorization'])
collection=db.messages
messages=json.loads(event['body'])['messages']
for message in messages:
message['user']=claims['sub']
collection.insert_many(messages)
for i in range(len(messages)):
messages[i]['_id']=str(messages[i]['_id'])
headers = {'Content-type': 'application/json', 'Accept': 'text/plain'}
requests.post(os.environ['ELASTIC_HOST']+'/messages/_doc/_bulk?
refresh=true',auth=awsauth,headers=headers,json=messages)
return {
'statusCode':200,
'body':json.dumps("success")
}
for some unknown reason, the API gateway keeps timing out (i.e the elastic search request is not completing).
My API gateway and lambda timeouts are set to 28s just for more information.
I also tried using the official python client for elastic search but had no luck.
Edit: On further investigation, I can see my AWS lambda is not able to connect to the elastic instance itself, so I am guessing it has something to do with access policies but I already whitelisted the CIDR block of the VPC in which the lambda is running.
Edit: I am close but I am getting an error AuthorizationException(403, ' { "message": "The security token included in the request is invalid." })
any pointers would be appreciated
