Stack Overflow
Stack Overflow

Questions tagged [amazon-kinesis-firehose]

Amazon Kinesis Firehose is a fully managed service for delivering real-time streaming data to destinations

Amazon Kinesis Firehose is a fully managed service for delivering real-time streaming data to destinations such as Amazon Simple Storage Service (Amazon S3) and Amazon Redshift.

Firehose is part of the Amazon Kinesis streaming data family, along with Amazon Kinesis Streams. With Firehose, you do not need to write any applications or manage any resources.

You configure your data producers to send data to Firehose and it automatically delivers the data to the destination that you specified.

Links:

  1. Amazon Kinesis - Site

  2. Amazon Kinesis - Documentation

618 questions
10
votes
3 answers

Storing Firehose transfered files in S3 under custom directory names

We primarily do bulk transfer of incoming click stream data through Kinesis Firehose service. Our system is a multi tenant SaaS platform. The incoming click stream data are stored S3 through Firehose. By default, all the files are stored under…
Sriram V
  • 101
  • 1
  • 4
10
votes
2 answers

Is there a way to specify file extension to the file saved to s3 by kinesis firehose

I am setting up a kinesis firehose stream and everything works well with the files getting created on s3 which are delimited. But i was wondering if there is a way to specify an extension to this file since the consumer of this file require it to be…
arjunj
  • 1,436
  • 1
  • 16
  • 28
10
votes
2 answers

How to do Real-time loading into Amazon Redshift?

We are evaluating Amazon Redshift for real time data warehousing. Data will be streamed and processed through a Java service and it should be stored in the database. We process row by row (real time) and we will only insert one row per…
fenix
  • 1,716
  • 2
  • 20
  • 26
10
votes
1 answer

Is there anyway to write data to multiple tables of redshift using a single firehose delivery stream

I am passing some json data to firehose delivery stream which in the end is getting saved into Redshift table. For my use case, I want the data to be stored in different tables. Do I create different delivery stream for different tables? If I create…
user2677679
  • 259
  • 2
  • 12
9
votes
4 answers

AWS CLI V2 "AWS firehose put-record" complaining about Invalid base64:

I have used to be able to send a record to firehose without any problem like this aws firehose put-record --delivery-stream-name my-stream --record='Data="{\"foor\":\"bar\"}"' But since I have updated my cli to version 2 I am getting this…
Am1rr3zA
  • 7,115
  • 18
  • 83
  • 125
9
votes
1 answer

Will PrivateLink allow firehose to access my private Redshift cluster?

I am trying to set up firehose to send data from a kinesis stream to a redshift cluster. Firehose successfully inserts the data to my s3 bucket, but I am receiving the following error when firehose attempts to execute the s3->Redshift copy…
ketcham
  • 922
  • 4
  • 15
9
votes
3 answers

AWS Firehose newline Character

I've read a lot of similar questions around adding newline characters to firehose, but they're all around adding the newline character to the source. The problem is that I don't have access to the source, and a third party is piping data to our…
Matt Westlake
  • 3,499
  • 7
  • 39
  • 80
9
votes
1 answer

Enable cloudwatch logs for kinesis firehose cloudformation

I am trying to catch Cloudwatch logs for my firehose to find any errors when sending data to S3 destination. I created a cloudformation template with logging details "CloudWatchLoggingOptions" : { "Enabled" : "true", "LogGroupName":…
Private
  • 1,661
  • 1
  • 20
  • 51
9
votes
1 answer

what is difference between Kinesis Streams and Kinesis Firehose?

Firehose is fully managed whereas Streams is manually managed. If other people are aware of other major differences, please add them. I'm just learning. Thanks..
Hakan Bayraktar
  • 91
  • 5
9
votes
3 answers

Write to a specific folder in S3 bucket using AWS Kinesis Firehose

I would like to be able to send data sent to kinesis firehose based on the content inside the data. For example if I sent this JSON data: { "name": "John", "id": 345 } I would like to filter the data based on id and send it to a subfolder of…
EMChamp
  • 449
  • 1
  • 7
  • 13
9
votes
2 answers

How to do de-duplication on records from AWS Kinesis Firehose to Redshift?

I read the document of official AWS Kinesis Firehose but it doesn't mention how to handle duplicated events. Does anyone have experience on it? I googled someone use ElasticCache to do filtering, does it mean I need to use AWS Lambda to encapsulate…
Casel Chen
  • 497
  • 2
  • 8
  • 19
8
votes
1 answer

How AWS Athena deals with single line JSONs?

I am currently using Athena along with Kinesis Firehose, Glue Crawler. Kinesis Firehose is saving JSON to single line files as below {"name": "Jone Doe"}{"name": "Jane Doe"}{"name": "Jack Doe"} But I noticed that the athena query select count(*)…
astrohsy
  • 345
  • 3
  • 16
8
votes
4 answers

Lambda.InvokeAccessDenied from Kinesis Firehose

I'm trying to set up Lambda transformations with a Firehose delivery stream. I have an IAM role defined for the Firehose which includes the following policy document: { "Statement": { "Action": [ "lambda:InvokeFunction", …
Nathan Brown
  • 311
  • 1
  • 3
  • 12
8
votes
1 answer

Anyone experienced data lost when using AWS kinesis streams, lambda and firehose?

I'm currently sending a series of xml messages to aws kinesis stream, I've been using this on different projects, so I'm pretty confident that this bit works. Then I've written a lambda to process events from kinesis stream to kinesis…
lorilew
  • 93
  • 6
8
votes
4 answers

Is there any difference in processing times between AWS Kinesis Firehose and Streams?

Reading over the documentation of both offerings (Firehose and Streams), it sounds like Firehose is "near" real-time with a potential of 60 seconds delay between producing a message to emitting it, whereas Streams documentation makes no mentioning…
Joshua Hansen
  • 356
  • 4
  • 7
1
2
3
…
41 42