1

I've been trying to ingest data from an AWS S3 bucket to AWS OpenSearch Serverless using AWS Lambda but I can't find any documentation on how to do it, any ideas? I've seen plenty of AWS OpenSearch Service examples but would they work the same way with Serverless? I'm a bit new to AWS

I've looked at the boto3 client

A A
  • 11
  • 1
  • Hi, I see you're new to SO. If you feel an answer solved the problem, please mark it as 'accepted' by clicking the green check mark. This helps keep the focus on older SO which still don't have answers. – fucalost Jun 25 '23 at 16:51

2 Answers2

0

If the data is already present then you would make calls to s3 using boto3 and read the files and then make calls to Opensearch endpoint using its _bulk endpoint.

If files can arrive later then you can have an event fired by s3 and you can have the same code (slightly modified) in lambda function that can push the data to opensearch.

What kind of data is this?

Prabhat
  • 4,066
  • 4
  • 34
  • 41
0

As Prabhat mentions, boto3 is certainly an option, however, the AWS SDK for Pandas (previously AWS Data Wrangler) is a super simple approach. I've used it extensively for moving data from CSVs, S3, and other locations into OpenSearch with ease.

Using the AWS SDK for Pandas, you might achieve what you're looking for like this...

import awswrangler as wr
from opensearchpy import OpenSearch

items = wr.s3.read_json(path="s3://my-bucket/my-folder/")

# connect + upload to OpenSearch
my_client = OpenSearch(...)
wr.opensearch.index_df(client=my_client, df=items)

The AWS SDK for Pandas can iterate over chunks of S3 items, and there's a tutorial on indexing JSON (and other file types) from S3 to OpenSearch.

Of course, you would need to create a Lambda layer to make the package available in your function. Here is a (pretty much) one-click script to do this.

fucalost
  • 313
  • 1
  • 8