0

I followed step by step this tutorial but when I'm arriving at the "Testing the result" step, I don't see my table name in the indices time as "all products" is showed in the example.

So it means my dynamodb table is not indexed in my Amazon ES domain.

I set all authorizations and configured lambda just as in the example.

Now I just don't know how to do for triggering my dynamodb streams in AWS ES.

ROMANIA_engineer
  • 54,432
  • 29
  • 203
  • 199
John doe
  • 3,680
  • 7
  • 31
  • 65

2 Answers2

0

Check that you have content in your Dynamo DB table. Without this no streaming will occur and as Elasticsearch creates the index(table) the first time something get entered to it, this would result in the index being absent.

If there are content in your Dynamo DB the issue is most likely due to problems with either the integration between Dynamo DB and AWS Lambda or between AWS Lambda and Elasticsearch.

Here are a few things you can do to debug:

  • Check your CloudWatch logs/Monitoring for your lambda to see if it has been invoked
  • Was the invocation successful? If not solve the error and try again
  • Does the lambda execution role have the appropriate permissions to post things to elasticsearch? (Especially check so that the ARN you specified for the roles are pointing to the correct resources in your aws account)
  • Does the lambda execution role have the appropriate permissions to read things from your Dynamo DB stream? (Check the ARN here as well)
  • Validate that the ES_ENDPOINT you specified is pointing to the cluster you created earlier in the tutorial

Hope this helps you solve the issue!

baggero
  • 96
  • 5
  • All these things listed above have been successfully checked. Cloudwatch show the invokation sucess, i set the permissions just as described in the tutorial and then i entered the ARN of my table. The ES_ENDPOINT pointing to my cluster. My issue remains the same. I can't catch the data in Amazon ES Services whereas my cloudwatch show a successfull invocation. – John doe Jul 17 '16 at 19:49
  • Have you tried opening the lambda in the console and manually modify the code to add more logging and debug that way? Also when I had a similar issue before. Turned out to be the region in the ARN being set to _us-east-1_ when it should've been _eu-west-1_ for the elasticsearch cluster. – baggero Jul 20 '16 at 07:38
0

I just replicated what those guys did in the blog as the Lambda function Blueprint mentioned in the blog is not available anymore.

You can use my code here which does the same thing: https://github.com/sportarchive/dynamodb-to-elasticsearch

It uses the ElasticSearch Python library and works perfectly in production. It supports: INSERT, UPDATE and DELETE and all data types in DynamoDB

koxon
  • 818
  • 1
  • 10
  • 12
  • Hello! I am trying to use your code but I can't understand how I should do it. How should I name the function? Should I use the upload zip option or am I just have to copy and paste the code? You write: "The first time you need to create the function in AWS Lambda. make create/DynamoToES DESC="Process DynamoDB stream to ES" Where should I insert the make command? It is the first time I am using Lambda and I just can't find my way to do this... Thank you! – Spyros El. Apr 25 '17 at 09:57
  • Clone the project and run this locally in your terminal at the root of the project you cloned. If you don't know what `make` does, Google it :) OR you can just code your function directly in the console But my project does it all for you, so read the README – koxon Sep 16 '17 at 03:33