Stack Overflow
Stack Overflow

Questions tagged [amazon-kinesis-firehose]

Amazon Kinesis Firehose is a fully managed service for delivering real-time streaming data to destinations

Amazon Kinesis Firehose is a fully managed service for delivering real-time streaming data to destinations such as Amazon Simple Storage Service (Amazon S3) and Amazon Redshift.

Firehose is part of the Amazon Kinesis streaming data family, along with Amazon Kinesis Streams. With Firehose, you do not need to write any applications or manage any resources.

You configure your data producers to send data to Firehose and it automatically delivers the data to the destination that you specified.

Links:

  1. Amazon Kinesis - Site

  2. Amazon Kinesis - Documentation

618 questions
0
votes
1 answer

Can I reverse geocode data with Amazon Kinesis Analytics?

When using Amazon Kinesis to process a large volume of incoming lat/long data points, what's the best way to reverse geocode these at scale? Using Kinesis Analytics I can use reference data stored in S3, as documented here:…
rbaker86
  • 1,832
  • 15
  • 22
0
votes
1 answer

How can the RetryOptions of an AWS Kinesis Firehose's RedshiftDestinationConfiguration be managed via CloudFormation

The AWS CLI API defines a RedshiftDestinationConfiguration of a Kinesis Firehose as having a RetryOptions parameter. ref The cloudformation schema also defines a RedshiftDestinationConfiguration of a Kinesis Firehose but has no RetryOptions…
Peter Kneale
  • 1,854
  • 5
  • 19
  • 31
0
votes
2 answers

Kinesis Firehose demo has no delimiters; Redshift can't parse without. Which is wrong?

When I went through first learning steps with Kinesis, Firehose, and Redshift today, I was pleased to discover that Amazon had a "try our demo data producer" setup. I was frustrated to learn that it does not seem to actually work. So, I went…
John Haugeland
  • 9,230
  • 3
  • 37
  • 40
0
votes
1 answer

How do I create real time ticker graph from AWS kinesis data stream?

How can I create a real time graph ticker updating every 2-3 secs just like NASDAQ for data streams coming from my AWS Kinesis Firehose to AWS Kinesis Analytics? How do I get the output from Kinesis Analytics? Trying to achieve something close to…
Rahul Gupta
  • 1,744
  • 2
  • 17
  • 28
0
votes
1 answer

What does it mean to use "Amazon ES for content recommendations"?

Looking at the marketing literature for amazon kinesis analytics, I'm looking through their real-time log analytics flow and they have a 4th step where the data is piped to Amazon ES for content suggestions. What is Amazon ES? Is that their…
MonkeyBonkey
  • 46,433
  • 78
  • 254
  • 460
0
votes
1 answer

Kinesis Agent not transforming records to JSON

I'm using the Amazon Kinesis Agent to tail my nginx web access logs with a view to pushing the access log to Kinesis - ultimately to be used within Kinesis Analytics. The agent is running and is tailing my access log, then pushing records up to…
David Smith
  • 593
  • 2
  • 10
0
votes
0 answers

passing a Firehose name as a parameter to a lambda via EventSourceMapping

I have a CloudFormation Template that creates multiple groups of: Kinesis Stream Firehose Lambda function to copy from Kinesis and an EventSourceMapping to connect all this Now I have a problem, I need to know which Firehose shall the lambda…
oleber
  • 1,089
  • 4
  • 12
  • 25
0
votes
0 answers

Can I use AWS kinesis for data upsert or transformation?

I have 3 different csv files coming every 5–10 secs each and I want to merge those csv files into one based on some update condition for near real time reporting. My current system involves pushing files from s3 to redshift every 10 sec but not able…
Rahul Gupta
  • 1,744
  • 2
  • 17
  • 28
0
votes
2 answers

kinesis agent to lambda, how to get origin file and server

I have a kinesis agent that streams a lot of log files information to kinesis streams and I have a Lambda function that parses the info. On Lambda in addition to the string I need to know the source file name an machine name is it possible?
Avi Zloof
  • 2,923
  • 4
  • 22
  • 28
0
votes
1 answer

how to process static data in AMAZON Kinesis?

How can we process static data in Amazon kinesis? For example I have data in excel sheets and I want to process them in Kinesis. Is it possible? If yes then how to achieve it ?
saumm
  • 1
  • 1
0
votes
1 answer

What AWS IAM role policies do I need to write files to S3?

I'm trying to configure a Kinesis Firehose delivery stream to write files to S3. I've created the Firehose stream to use a role named att1. This is the policy attached to att's configuration. I took the format from this page here…
simplycoding
  • 2,770
  • 9
  • 46
  • 91
0
votes
1 answer

Debug why Firehose is not delivering to Redshift

I setup a Firehose stream that delivers data to my Redshift cluster. It was working for a short period but suddenly seemed to stop delivering to redshift. From my select * from stl_query order by endtime desc limit 10; select * from stl_load_errors…
Niru
  • 1,407
  • 1
  • 28
  • 47
0
votes
1 answer

Round-robining to aws kinesis streams

As an alternative to resharding which causes much latency how can we dynamically create aws kinesis streams and round-robin-ing to streams?
tarun kumar Sharma
  • 897
  • 11
  • 21
0
votes
1 answer

Code in Node.js AWS Lambda Package will not call putRecord() for Adding Data into AWS Kinesis Firehose Stream

I've built an AWS Lambda deployment package (using Node.js) that is executed when an object is PUT into a specified S3 bucket. I've configured the code that gets executed to, for now, add 10 randomly generated records into a Kinesis Firehose stream.…
littleK
  • 19,521
  • 30
  • 128
  • 188
-1
votes
2 answers

How do I enable my AWS Lambda function to write to Firehose?

I have an AWS Lambda function that should write to a Firehose. However, when I make a PutRecordBatchRequest it will time-out and fail to write to the Firehose. The function is in a VPC. I have attached a policy to the lambda's role to allow…
sdgfsdh
  • 33,689
  • 26
  • 132
  • 245
1 2 3
…
41
42