0

When using spring-cloud-stream-binder-aws-kinesis V2.0.0.RELEASE with dynamodb streams, I am getting the following error

test-dynamo-stream is the name of my dynamodb table and I have enabled stream for it.

org.springframework.cloud.stream.provisioning.ProvisioningException: The stream [test-dynamo-stream] was not found and auto creation is disabled.; nested exception is com.amazonaws.services.kinesis.model.ResourceNotFoundException: Stream test-dynamo-stream under account xxxxxxxxxxxx not found. (Service: AmazonKinesis; Status Code: 400; Error Code: ResourceNotFoundException)
    at org.springframework.cloud.stream.binder.kinesis.provisioning.KinesisStreamProvisioner.createOrUpdate(KinesisStreamProvisioner.java:145)
    at org.springframework.cloud.stream.binder.kinesis.provisioning.KinesisStreamProvisioner.provisionConsumerDestination(KinesisStreamProvisioner.java:108)
    at org.springframework.cloud.stream.binder.kinesis.provisioning.KinesisStreamProvisioner.provisionConsumerDestination(KinesisStreamProvisioner.java:56)
    at org.springframework.cloud.stream.binder.AbstractMessageChannelBinder.doBindConsumer(AbstractMessageChannelBinder.java:408)
    at org.springframework.cloud.stream.binder.AbstractMessageChannelBinder.doBindConsumer(AbstractMessageChannelBinder.java:90)
    at org.springframework.cloud.stream.binder.AbstractBinder.bindConsumer(AbstractBinder.java:143)
    at org.springframework.cloud.stream.binding.BindingService.doBindConsumer(BindingService.java:169)
    at org.springframework.cloud.stream.binding.BindingService.bindConsumer(BindingService.java:126)
    at org.springframework.cloud.stream.binding.AbstractBindableProxyFactory.createAndBindInputs(AbstractBindableProxyFactory.java:112)
    at org.springframework.cloud.stream.binding.InputBindingLifecycle.doStartWithBindable(InputBindingLifecycle.java:58)
    at java.util.LinkedHashMap$LinkedValues.forEach(LinkedHashMap.java:608)
    at org.springframework.cloud.stream.binding.AbstractBindingLifecycle.start(AbstractBindingLifecycle.java:57)
    at org.springframework.cloud.stream.binding.InputBindingLifecycle.start(InputBindingLifecycle.java:34)
    at org.springframework.context.support.DefaultLifecycleProcessor.doStart(DefaultLifecycleProcessor.java:182)
    at org.springframework.context.support.DefaultLifecycleProcessor.access$200(DefaultLifecycleProcessor.java:53)
    at org.springframework.context.support.DefaultLifecycleProcessor$LifecycleGroup.start(DefaultLifecycleProcessor.java:360)
    at org.springframework.context.support.DefaultLifecycleProcessor.startBeans(DefaultLifecycleProcessor.java:158)
    at org.springframework.context.support.DefaultLifecycleProcessor.onRefresh(DefaultLifecycleProcessor.java:122)
    at org.springframework.context.support.AbstractApplicationContext.finishRefresh(AbstractApplicationContext.java:894)
    at org.springframework.boot.web.servlet.context.ServletWebServerApplicationContext.finishRefresh(ServletWebServerApplicationContext.java:162)
    at org.springframework.context.support.AbstractApplicationContext.refresh(AbstractApplicationContext.java:553)
    at org.springframework.boot.web.servlet.context.ServletWebServerApplicationContext.refresh(ServletWebServerApplicationContext.java:141)
    at org.springframework.boot.SpringApplication.refresh(SpringApplication.java:747)
    at org.springframework.boot.SpringApplication.refreshContext(SpringApplication.java:397)
    at org.springframework.boot.SpringApplication.run(SpringApplication.java:315)
    at org.springframework.boot.SpringApplication.run(SpringApplication.java:1226)
    at org.springframework.boot.SpringApplication.run(SpringApplication.java:1215)

com.amazonaws.services.kinesis.model.ResourceNotFoundException: Stream test-dynamo-stream under account xxxxxxxxxxxx not found. (Service: AmazonKinesis; Status Code: 400; Error Code: ResourceNotFoundException; 
at com.amazonaws.http.AmazonHttpClient$RequestExecutor.handleErrorResponse(AmazonHttpClient.java:1712)
at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeOneRequest(AmazonHttpClient.java:1367)
at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeHelper(AmazonHttpClient.java:1113)
at com.amazonaws.http.AmazonHttpClient$RequestExecutor.doExecute(AmazonHttpClient.java:770)
at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeWithTimer(AmazonHttpClient.java:744)
at com.amazonaws.http.AmazonHttpClient$RequestExecutor.execute(AmazonHttpClient.java:726)
at com.amazonaws.http.AmazonHttpClient$RequestExecutor.access$500(AmazonHttpClient.java:686)
at com.amazonaws.http.AmazonHttpClient$RequestExecutionBuilderImpl.execute(AmazonHttpClient.java:668)
at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:532)
at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:512)
at com.amazonaws.services.kinesis.AmazonKinesisClient.doInvoke(AmazonKinesisClient.java:2809)
at com.amazonaws.services.kinesis.AmazonKinesisClient.invoke(AmazonKinesisClient.java:2776)
at com.amazonaws.services.kinesis.AmazonKinesisClient.invoke(AmazonKinesisClient.java:2765)
at com.amazonaws.services.kinesis.AmazonKinesisClient.executeDescribeStream(AmazonKinesisClient.java:875)
at com.amazonaws.services.kinesis.AmazonKinesisClient.describeStream(AmazonKinesisClient.java:846)
at org.springframework.cloud.stream.binder.kinesis.provisioning.KinesisStreamProvisioner.createOrUpdate(KinesisStreamProvisioner.java:127)
... 14 more

This is my properties file

spring:
    stream:
      bindings:
        input:
          destination: test-dynamo-stream
          group: test-dynamo-stream-group
          content-type: text/plain

      kinesis:
        binder:
          autoCreateStream: false
          kplKclEnabled: true
        bindings:
          input:
            consumer:
              dynamoDbStreams: true
              listenerMode: record
              checkpointMode: record
              workerId: test-worker
              shardIteratorType: TRIM_HORIZON

cloud:
  aws:
    credentials:
      accessKey: <hidden>
      secretKey: <hidden>
    region:
      static: us-east-2
    stack:
      auto: false

Should I give a different name for destination instead of dynamodb table's name? If so, what should be the name and where can I get that from?

  • 1
    You should mark your question with AWS tags as well because location for the stream name from DynamoDB table is out of Spring Cloud Stream scope. But you indeed has to use a stream name on the table, not the table name. Please, read more about DynamoDB Streams in AWS. – Artem Bilan Dec 05 '19 at 21:35
  • @ArtemBilan, I added aws tags now. I have gone through the documentation of dynamodb streams. The stream arn is arn:aws:dynamodb:us-east-2:xxxxxxxxxxxx:table/test-dynamo-stream/stream/ which contains the tablename again. I tried giving the arn and also `test-dynamo-stream/stream` for the destination. But I get a validation error that it should be in the format [a-zA-Z0-9_.-]+ – Vivek Nynaru Dec 06 '19 at 05:28
  • Can you share, please, a stack trace for that validation error? I just wonder where that happens. Please, move this to Kinesis Binder issue: https://github.com/spring-cloud/spring-cloud-stream-binder-aws-kinesis/issues. It turns out that we are missing something in logic to process DynamoDB Streams. Thanks – Artem Bilan Dec 06 '19 at 15:57

0 Answers0