0

I have built a Kinesis Firehose stream to push data into redshift and am trying to push data from an EC2 instance using kinesis agent.

Firehose Stream is able to parse the records but not identify the firehose streams am getting the following java error.

com.amazon.kinesis.streaming.agent.tailing.AsyncPublisher 
[ERROR] AsyncPublisher[kinesis:csvtoredshiftstream:/tmp/redshift.log*]:RecordBuffer
(id=2,records=2,bytes=45) Retriable send error (com.amazonaws.services.kinesis.model.ResourceNotFoundException: 
Stream csvtoredshiftstream under account xyz not found. 
(Service: AmazonKinesis; Status Code: 400; 
Error Code: ResourceNotFoundException; 
Request ID: f4a63623-9a15-b2f8-a597-13b478c81bbc)). Will retry.

Request your pointers to identify and resolve the issue.

Regards, Srivignesh KN

John Rotenstein
  • 241,921
  • 22
  • 380
  • 470
Srivignesh KN
  • 452
  • 8
  • 22
  • is your error on the Firehose-stream or on the Redshift? It seems like you did not create the stream `csvtoredshiftstream` before publishing events. I haven't used firehose, I'm guessing kinesis-agent collects the events into the firehose stream,BUT the problem is the stream does not exist. If so try creating stream first POSTing a `CreateStreamRequest createStreamRequest = new CreateStreamRequest().withStreamName("csvtoredshiftstream")`. – prayagupa Mar 03 '17 at 00:25
  • Thank you @peter, I was able to overcome the error using by specifying the inputs in the agent.json in the following manner for firehose events. { "cloudwatch.emitMetrics": true, "kinesis.endpoint": "", "firehose.endpoint": "firehose.us-west-2.amazonaws.com", "flows": [ { "filePattern": "/tmp/s3streaming.*", "deliveryStream": "S3TestingStream", "partitionKeyOption": "RANDOM" }, { "filePattern": "/tmp/app.log*", "deliveryStream": "yourdeliverystream" } ] } – Srivignesh KN Mar 03 '17 at 06:54

1 Answers1

1

Thank you @peter,

  1. I was able to overcome the error using by specifying the inputs in the agent.json in the following manner for firehose events. { "cloudwatch.emitMetrics": true, "kinesis.endpoint": "", "firehose.endpoint": "firehose.us-west-2.amazonaws.com", "flows": [ { "filePattern": "/tmp/s3streaming.", "deliveryStream": "S3TestingStream", "partitionKeyOption": "RANDOM" }, { "filePattern": "/tmp/app.log", "deliveryStream": "yourdeliverystream" } ] } –

  2. Moreover for Kinesis Streams to work as expected, the S3 bucket also needs to be created in the same region as that of the Streams.

If the Stream is created in West-2 region, the S3 bucket should also be created in the same region.

Thanks & Regards, Srivignesh KN

Srivignesh KN
  • 452
  • 8
  • 22