Stack Overflow
Stack Overflow

Questions tagged [amazon-kinesis-firehose]

Amazon Kinesis Firehose is a fully managed service for delivering real-time streaming data to destinations

Amazon Kinesis Firehose is a fully managed service for delivering real-time streaming data to destinations such as Amazon Simple Storage Service (Amazon S3) and Amazon Redshift.

Firehose is part of the Amazon Kinesis streaming data family, along with Amazon Kinesis Streams. With Firehose, you do not need to write any applications or manage any resources.

You configure your data producers to send data to Firehose and it automatically delivers the data to the destination that you specified.

Links:

  1. Amazon Kinesis - Site

  2. Amazon Kinesis - Documentation

618 questions
0
votes
1 answer

AWS Kinesis Streams, Create Stream Error

I'm a complete beginner of AWS Kinesis Streams, SO i am following http://docs.aws.amazon.com/streams/latest/dev/fundamental-stream.html tutorial. As shown in the tutorial i install and configure the AWS CLI, then i try to Create a Stream by typing…
GihanDB
  • 591
  • 2
  • 6
  • 23
0
votes
2 answers

Kinesis Firehose Stream Empty

I am triggering an AWS lambda from an EC2 instance multiple times in a loop passing subset of a 350MB dataset to Lambda which manipulates each data set passed to it. The Lambda writes the output to a Kinesis Firehose stream which then writes it to…
Anant Bhandarkar
  • 367
  • 2
  • 11
0
votes
3 answers

Load from S3 into Redshift using Kinesis

I receive my source files in S3, i need to load this data into redshift. The source S3 files are populated via Kinesis Firehose. Source : via Kinesis Webstreams -------------- > S3 Requirement : via Kinesis …
Srivignesh KN
  • 452
  • 8
  • 22
0
votes
1 answer

Kinesis Agent Input - Log file created for each day

New Log file is created for each day with date appended in the name. For example "data_log-2017-05-14" log file is created for 14th May. I need to set up Kinesis Agent to read data from the logs generated. But since each day a new log file is…
shruti
  • 587
  • 4
  • 6
0
votes
1 answer

AWS Cloudwatch logs to ElasticSeacrh via Kinesis Firehose

I am trying to stream AWS cloudwatch logs to Amazon ES using Kinesis Firehose. However, I keep getting the below errors in the ES error log. Any idea? The data could not be decoded as UTF-8 as in…
Bond
  • 855
  • 3
  • 13
  • 24
0
votes
2 answers

Does Amazon Kinesis Firehose support Data Transformations programatically?

I have a use case in which I have to verify that the payloads sent to Kinesis firehose are indeed being sent. In order to do that I came up with the chain Firehose -> Firehose Data transformation(using lambda) -> DDB -> Check for payload in DDB…
Arunav Sanyal
  • 1,708
  • 1
  • 16
  • 36
0
votes
2 answers

API Gateway CORS HTTP 415

Okay, I've been all over these interwebs looking for some insight to my issue; I've probably been through over 80 stack overflow threads RE api gateway and such, but none of them seem to help or speak close enough to my issue. I'm new to API Gateway…
zentechinc
  • 369
  • 1
  • 5
  • 15
0
votes
1 answer

How to get AWS Kinesis Firehose stream details using the Java SDK

Is it possible to get details like buffer size and buffer interval from an AWS Firehose stream using the Java SDK?
Ilyas Patel
  • 400
  • 2
  • 11
  • 27
0
votes
1 answer

boto3 python package installed but aws/config does not exist

I installed AWS CLI and Boto3 but can't find Shared Credentials File to put user access key. I tried to install it on Windows 7 and Ubuntu and I have got the same issue, simply can't find the Shared Credentials File in the default location…
interface
  • 35
  • 4
0
votes
2 answers

Sending data from Kinesis Analytics to Kinesis Firehose not working

I have added Kinesis Firehose Stream as a destination to the Kinesis Analytics Service. Data is properly getting inserted into Kinesis Analytics, but is unable to reach the Kinesis Firehose stream. Have granted all the Kinesis permissions to the IAM…
red-devil
  • 1,064
  • 1
  • 20
  • 34
0
votes
2 answers

Writing data from Kinesis to S3

I am writing data using the AWS SDK from a Java application that posts data to a Kinesis stream. This is done in batches of 10 records at once, using the following code; // Convert to JSON object, and then to bytes... ObjectWriter ow…
Mez
  • 4,666
  • 4
  • 29
  • 57
0
votes
0 answers

How to create firehose stream and check if it's been created?

I need to create an AWS firehose stream, so I made the following code: def create_kinesis_firehose_stream(firehoseStreamName, roleARN, bucketARN, prefix, sizeInMBs, intervalInSeconds): createdFirehose = None try: createdFirehose =…
roeygol
  • 4,908
  • 9
  • 51
  • 88
0
votes
1 answer

How do you filter multi-line JSON data from going into AWS Hive table

I have an AWS IoT rule that sends incoming JSON to a Kinesis Firehose. The JSON data from my IoT publish is all on one line - ex: {"count":4950, "dateTime8601": "2017-03-09T17:15:28.314Z"} The IoT "Test" section in the admin UI allows you to…
rynop
  • 50,086
  • 26
  • 101
  • 112
0
votes
1 answer

Kinesis Agent not sending records to Stream

I have built a Kinesis Firehose stream to push data into redshift and am trying to push data from an EC2 instance using kinesis agent. Firehose Stream is able to parse the records but not identify the firehose streams am getting the following java…
Srivignesh KN
  • 452
  • 8
  • 22
0
votes
1 answer

Kinesis Firehose load csv data into Redshift

I am using Kinesis firehose to process data into redshift and i am trying both Json and Csv formats. The Json format works fine for me and data is getting loaded into redshift table. COPY COMMAND USED FOR JSON : COPY products_json FROM…
Srivignesh KN
  • 452
  • 8
  • 22
1 2 3
…
41 42