Amazon Kinesis: Caught exception while sync'ing Kinesis shards and leases

Question

I am trying to make Snowplow work on AWS. When I am trying to run stream-enrich service on instance, I am getting this exception:

[main] INFO com.amazonaws.services.kinesis.clientlibrary.lib.worker.Worker - Syncing Kinesis shard info
[main] ERROR com.amazonaws.services.kinesis.clientlibrary.lib.worker.ShardSyncTask - Caught exception while sync'ing Kinesis shards and leases
[cw-metrics-publisher] WARN com.amazonaws.services.kinesis.metrics.impl.CWPublisherRunnable - Could not publish 4 datums to CloudWatch

I don't think error is due to Cloud Watch:

Caught exception while sync'ing Kinesis shards and leases

I am seeing the same issue. Other forum posts I have found told me to make sure that the IAM role for the server could access Kinesis, Dynamo and Cloudwatch, but that has not solved my problem. Did you ever find the solution? — one stevy boi, Dec 11 '18 at 21:38
I am facing the same issue & I do have full access. Have you found the solutin ? — Raju Guduri, Aug 21 '19 at 10:54
Actually, the problem was indeed with not able to create / access dynamo db table in which it keeps info of current index of the kinesis packet it has worked successfully on. So, make sure your IAM role can create the table in DynamoDB. For ref: https://docs.aws.amazon.com/streams/latest/dev/kinesis-record-processor-ddb.html — Prakhar Mishra, Sep 19 '19 at 04:07

score 1 · Answer 1 · answered Feb 19 '20 at 19:16

As mentioned in the comments above, this error will crop when you're lacking permissions to AWS resources required by Kinesis Client Library (KCL). This can be the DynamoDB, CloudWatch, or Kinesis. For the Stream Enrich component of Snowplow, you'll need the following permissions:

Read permission to input kinesis stream (collector good)
Write permission to output kinesis streams (enrich good & enrich bad)
List permission to kinesis streams
Read/write/create permission to DynamoDB state table (table name is the “appName” value in your stream enrich application.conf)
PutMetricData to Cloudwatch

A templated version of an IAM policy that meets these needs is as follows:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": [
        "kinesis:DescribeStream",
        "kinesis:GetShardIterator",
        "kinesis:GetRecords",
        "kinesis:ListShards"
      ],
      "Resource": [
        "${collector_stream_out_good}"
      ]
    },
    {
      "Effect": "Allow",
      "Action": [
          "kinesis:ListStreams"
      ],
      "Resource": "*"
    },
    {
      "Effect": "Allow",
      "Action": [
        "kinesis:DescribeStream",
        "kinesis:PutRecord",
        "kinesis:PutRecords"
      ],
      "Resource": [
        "${enricher_stream_out_good}",
        "${enricher_stream_out_bad}"
      ]
    },
    {
      "Effect": "Allow",
      "Action": [
        "dynamodb:CreateTable",
        "dynamodb:DescribeTable",
        "dynamodb:Scan",
        "dynamodb:GetItem",
        "dynamodb:PutItem",
        "dynamodb:UpdateItem",
        "dynamodb:DeleteItem"
      ],
      "Resource": [
        "${enricher_state_table}"
      ]
    },
    {
      "Effect": "Allow",
      "Action": [
        "cloudwatch:PutMetricData"
      ],
      "Resource": "*"
    }
  ]
}

I've written up a blog post that covers required IAM permissions for Stream Enrich and other Snowplow components since documentation on the exact required permissions was sparse/non-existent in the Snowplow documentation.

Hope that helps!

score 0 · Answer 2 · edited Sep 21 '20 at 23:25

So I had this problem when setting up Snowplow. I'm using terraform to automate the infrastructure and got this error after a destroy and re-apply. Here's what I learned.

You give the enricher DynamoDB privilages so it can create a table. If this table is already created before the enricher creates it (but not destroyed by terraform in my case) it is not able to create a table with the same name. It also seemingly won't link to existing tables.

My solution was to delete the existing DynamoDB table via the AWS console, terminate my enricher, and start up a new one. The error no longer appeared and my enricher worked as intended.

score 0 · Answer 3 · answered May 19 '21 at 06:08

I faced this issue today. For me, the issue was that, I changed the kinesis stream names without changing the appName in the enrich configuration.

Once I changed the appName to a new name and deployed an updated to snowplow enrich, I was able to get rid of the error.

Amazon Kinesis: Caught exception while sync'ing Kinesis shards and leases

3 Answers3