Stack Overflow
Stack Overflow

Questions tagged [amazon-kinesis-firehose]

Amazon Kinesis Firehose is a fully managed service for delivering real-time streaming data to destinations

Amazon Kinesis Firehose is a fully managed service for delivering real-time streaming data to destinations such as Amazon Simple Storage Service (Amazon S3) and Amazon Redshift.

Firehose is part of the Amazon Kinesis streaming data family, along with Amazon Kinesis Streams. With Firehose, you do not need to write any applications or manage any resources.

You configure your data producers to send data to Firehose and it automatically delivers the data to the destination that you specified.

Links:

  1. Amazon Kinesis - Site

  2. Amazon Kinesis - Documentation

618 questions
6
votes
1 answer

Google Cloud Platform: accumulate data from Pub/Sub to files in Cloud Storage without Dataflow

I'm trying to figure out if there is a service on GCP which would allow consuming a stream from Pub/Sub and dump/batch accumulated data to files in Cloud Storage (e.g. every X minutes). I know that this can be implemented with Dataflow, but looking…
Tim
  • 12,318
  • 7
  • 50
  • 72
6
votes
1 answer

Transfer data from Kinesis (or s3) to RDS postgres chron job

I'm pretty new to AWS, and I'm trying to find a way to reliably transfer data from a Kinesis stream to an AWS RDS postgres database table. The records will need to undergo small transformations on the way in, like filter (not all records will be…
bill_e
  • 930
  • 2
  • 12
  • 24
6
votes
2 answers

Is it possible to specify data format conversion in AWS Cloudformation?

The AWS docs makes it look as if you can specify DataFormatConversionConfiguration for a AWS::KinesisFirehose::DeliveryStream in cloudformation, but is not documentation on where the property is supposed to go. Tried adding it under Properties,…
micah
  • 7,596
  • 10
  • 49
  • 90
6
votes
2 answers

What should be service api for aws_api_gateway_integration with firehose in terraform

As per terraform doc, uri for the aws_api_gateway_integration should be resource "aws_api_gateway_integration" "integration" { ... ... uri = "arn:aws:apigateway:{region}:firehose:PutRecord/{service_api}" } I am not able to figure…
Balaji
  • 859
  • 1
  • 16
  • 27
6
votes
2 answers

Partitioning AWS Kinesis Firehose data to s3 by payload

I am using AWS-Kinesis-Firehose to injest data to S3, and consume it afterwards with Athena. I am trying to analyze events from different games, to avoid Athena explore much data I would like to partition the s3 data using an identifier for each…
bracana
  • 1,003
  • 2
  • 11
  • 20
6
votes
3 answers

Sync data from Amazon Aurora to Redshift

I am trying to setup a sync between AWS Aurora and Redshift. What is the best way to achieve this sync? Possible ways to sync can be: - Query table to find changes in a table(since I am only doing inserts, updates don't matter), export these…
Adi
  • 387
  • 3
  • 6
  • 14
6
votes
2 answers

How should records be formatted for AWS Kinesis Firehose to Redshift?

Consider the following: A table in Redshift called 'people' that has fields id, name and age A kinesis firehose stream called 'people' that is configured to write to the 'people' table and the value for 'Redshift table columns' is…
Jamie McCrindle
  • 9,114
  • 6
  • 43
  • 48
6
votes
3 answers

Amazon Kinesis Firehose Buffering to S3

I'm attempting to price out a streaming data / analytic application deployed to AWS and looking at using Kinesis Firehose to dump the data into S3. My question is, when pricing out the S3 costs for this, I need to figure out out how many PUT's I…
Brooks
  • 7,099
  • 6
  • 51
  • 82
6
votes
2 answers

AWS API Gateway Service Proxy to Kinesis Firehose

Can anyone tell me if there is currently an option to bind the Kinesis Firehose delivery stream to an API Gateway Endpoint via Service Proxy. I attempting to do it using the Kinesis service type with the Firehose PutRecord action and the new…
Ryan Fisch
  • 2,614
  • 5
  • 36
  • 57
5
votes
1 answer

Terraform is throwing InvalidArgumentException Duplicate ProcessorParameter passed to ProcessingConfiguration on Kinesis Firehose dynamic partitioning

I'm trying to create a Kinesis Firehose using terraform with dynamic partitioning using two partition queries from the JSON I'm recieving, my processing configuration looks like this processing_configuration { enabled = true processors { …
Zorrero
  • 181
  • 5
5
votes
1 answer

AWS Kinesis Firehose - dynamic partitioning by timestamp other than epoch

My Firehose reads from Eventbridge events that look something like: { "detail": { "key1": "some value", "key2": "some value", "Timestamp": "2022-01-21T19:01:05Z" } } I'd like to perform dynamic partition when saving the events to…
localhost
  • 461
  • 4
  • 19
5
votes
1 answer

Firehose data transformation lambda - produce multiple records from single kinesis record

I have created a Kinesis data stream and I'm feeding records into it that have format like this: { "ufo": 5, "unicorn": 7, "something else": 9, } I'm trying to write a transformation lambda that will transform above record and insert…
Greg0ry
  • 931
  • 8
  • 25
5
votes
1 answer

How to enable Transform source records with AWS Lambda for Firehose with CDK

I'm trying to enable resource transformation (with Lambda) to Kinesis Firehose using CDK. I already know how to do this using the console, but I can't figure out how to implement this with the AWS CDK. This is the Code that I have so far using…
jon
  • 61
  • 4
5
votes
1 answer

Forward Kinesis data --> (different AWS account) Kinesis?

Is there a simple way to forward data from one account's kinesis stream to another account's kinesis? data --> (AWS Account A) Kinesis ---> (AWS Account B) Kinesis --> consumer this part ---> ^^^^ I've seen some options with Lambda…
Chris
  • 171
  • 2
  • 8
5
votes
2 answers

Can Kinesis Firehose receive content uncompressed from CloudWatch Logs subscription?

I'm using Kinesis Firehose to copy application logs from CloudWatch Logs into S3 buckets. Application logs are written to CloudWatch A Kinesis subscription on the log group pulls the log events into a Kinesis stream. A firehose delivery stream uses…
Nic
  • 4,319
  • 5
  • 29
  • 36
1 2 3
…
41 42