Is it possible to transfer data from Redshift to Elasticsearch?

Question

I'm working on something related to Amazon elasticsearch service.For that,I need to get data from Amazon Redshift.The data to be tranfered is huge i.e. 100 GB.Is there any way to get it directly form Redshift or is it a two step process like Redshift->s3->elasticsearch?

Answer 1

I see, at least in theory, 2 possible approaches for transfering data from Redshift to Elasticsearch:

1. Logstash, using the JDBC input plugin
2. elasticsearch-jdbc

Answer 2

1. Don’t gzip the data unloaded.
2. Use the bulk load on elastic
3. Use a large number of records in the bulk load (>5000) – fewer large bulk loads are better than more smaller ones.
4. When working with AWS elastic search there is a risk of hitting the limits of the bulk queue size.
5. Process a single file in the lambda and then recursively call the lambda function with an event
6. Before recursing wait for a couple of seconds –> setTimeout. When waiting make sure that you aren’t idle for 30 seconds because your lambda will stop.
7. Don’t use s3 object creation to trigger your lambda — you’ll end up with multiple lambda functions being called at the same time.
8. Don’t bother trying to put kinesis in the middle – unloading your data into kinesis is almost certain to hit load limits in kinesis.
9. Monitor your elastic search bulk queue size with something like this:

curl https://%ES-SERVER:PORT%/_nodes/stats/thread_pool |jq
    ‘.nodes |to_entries[].value.thread_pool.bulk’

Answer 3

It looks like there is no direct data transfer pipeline for pushing data into elasticsearch from Redshift. One alternative approach is to first dump the data in S3 and then push into elasticsearch.