0

I am new in these area and I need your support.

  • I have an Elasticsearch environment running in my Docker. First, my data goes here. I can reach the data also by using Kibana on localhost:5601.
  • Afterwards, I need to send these data to AWS DynamoDB and AWS DocumentDB.
  • In that case what do I need to do?
    How can I create a channel between elasticsearch and AWS?

If experienced people can direct me in the right path I can find my way but now I feel a little bit lost.
Could you suggest some resources, tuorials or anything?

Thank you so much!

firefighter
  • 171
  • 1
  • 14
  • 1
    Why people like to vote bad I don't get it. I just asked a question in order to find my way in elasticsearch, I didn't say can you do this task for me or sth. – firefighter Nov 26 '20 at 15:55
  • I never used kibana. But if you run some script (i.e nodejs, python, go, PHP, Java, etc) that facing to elasticsearch, there you go to connect with AWS – Mahdi Ridho Nov 26 '20 at 15:58
  • @firefighter, how you are sending the data to ES??? it would be difficult to send the data from ES to AWS DynamoDB or other DB, you should send the data parallel to these DB, would be great if you can share what is your use-case, – Amit Nov 26 '20 at 16:15
  • 1
    @firefighter don't mention the fact that people downvote without explaining why, it's a [big taboo](https://meta.stackoverflow.com/questions/357436/why-isnt-providing-feedback-mandatory-on-downvotes-and-why-are-ideas-suggestin) around here, and I hate that as much as you do, but unfortunately that's just how it is – Val Nov 26 '20 at 16:16
  • @firefighter, one more thing, in ES data is stored in a different format, inverted index and their tokens are created from JSON doc, so not sure in what format you need data in other databases. Ingesting data from ES is a new use-case for me and would be really interested to see your use-case – Amit Nov 26 '20 at 16:16
  • How is your data getting ingested in elastic-search, if you are using kinesis firehose or logstash then you can use same pipeline to dump data in dyanamo db itself. else if you want to first put data in elastic-search and then in dyanamo then you will need to write a consumer(script) which reads from elastic and dumps to dynamo.(I feel dumping data parallely in both db using same service would be more efficient) – AbhiK Nov 26 '20 at 17:51
  • @ElasticsearchNinja I explained the use case a lot more detaily here: https://stackoverflow.com/questions/65106106/sending-data-from-elasticsearch-to-aws-databases-in-real-time Can you check please ? – firefighter Dec 02 '20 at 12:35
  • @AbhiK I am using lambda function for data ingestion to ES from AWS Greengrass. I am not using kinesis firehose or logstash. In that case, you said that I need to write consumer(script) to dump data to dynamo. What is consumer and how can I do it? Can you suggest a source? Also, I posted the detailed version of the question, can you check please ? Here: https://stackoverflow.com/questions/65106106/sending-data-from-elasticsearch-to-aws-databases-in-real-time – firefighter Dec 02 '20 at 12:40
  • @MahdiRidho I explained the use case a lot more detaily here: https://stackoverflow.com/questions/65106106/sending-data-from-elasticsearch-to-aws-databases-in-real-time Can you check please ? (I don't know what kind of script you mean) – firefighter Dec 02 '20 at 12:42
  • Another approach is to write the data to first to DynamoDB/DocumentDB, and then stream it to Elasticsearch: https://aws.amazon.com/blogs/database/run-full-text-search-queries-on-amazon-documentdb-data-with-amazon-elasticsearch-service/ – Joseph Idziorek Dec 03 '20 at 04:55
  • @JosephIdziorek Untfortunately it's not a case. It has to be first ES then DynamoDB. But I don't know how to do – firefighter Dec 03 '20 at 05:50
  • @firefighter it seems your use case is data from sensor is shipped to lambda, then lambda dumps data to elastic-search. You want to do some processing on data in elastic-search then processed data should be dumped in dynamodb. If you want to setup pipeline explore kinesis streams n kinesis firehose. A lambda function can fetch required data from elastic-search and send it to kinesis, which dumps data to dynamodb. – AbhiK Dec 05 '20 at 13:04
  • But elastic-search can fetch max 10k records, if you want more you need to use scroll api which is slow. Elastic-search is not good option for data processing, its suitable for search n data aggregation queries. – AbhiK Dec 05 '20 at 13:08

0 Answers0