Questions tagged [amazon-kinesis]

Amazon Kinesis is a fully managed service for real-time processing of streaming data at massive scale.

Amazon Kinesis is a fully managed service for real-time processing of streaming data at massive scale. Amazon Kinesis can collect and process hundreds of terabytes of data per hour from hundreds of thousands of sources, allowing you to easily write applications that process information in real-time, from sources such as web site click-streams, marketing and financial information, manufacturing instrumentation and social media, and operational logs and metering data.

With Amazon Kinesis applications, you can build real-time dashboards, capture exceptions and generate alerts, drive recommendations, and make other real-time business or operational decisions. You can also easily send data to a variety of other services such as Amazon Simple Storage Service (Amazon S3), Amazon DynamoDB, or Amazon Redshift. In a few clicks and a couple of lines of code, you can start building applications which respond to changes in your data stream in seconds, at any scale, while only paying for the resources you use.

Useful links

1802 questions
20
votes
2 answers

Lambda from API gateway VS kinesis Streams

Background i am studying about AWS kinesis,API gateway. I understand that ,whenever requests hit API gateway,i can forward the data to a stream or i can choose to trigger a lambda(which will do some processing ). Thoughts and Query So,my thought…
Rohan
  • 601
  • 2
  • 9
  • 21
20
votes
3 answers

How to fanout an AWS kinesis stream?

I'd like to fanout/chain/replicate an Input AWS Kinesis stream To N new Kinesis streams, So that each record written to the input Kinesis will appear in each of the N streams. Is there an AWS service or an open source solution? I prefer not to…
Gili Nachum
  • 5,288
  • 4
  • 31
  • 33
20
votes
3 answers

Amazon KCL Checkpoints and Trim Horizon

How are checkpoints and trimming related in AWS KCL library? The documentation page Handling Startup, Shutdown, and Throttling says: By default, the KCL begins reading records from the tip of the stream;, which is the most recently added record.…
Edmondo
  • 19,559
  • 13
  • 62
  • 115
20
votes
2 answers

How to pause / resume a aws lambda function

For example I have lambda functions that consume messages from a KinesisStream. How do stop and resume the function so that I don't incur charges and I don't loose data in the stream. I know that if the events keep failing, Kinesis will keep…
victor m
  • 2,012
  • 2
  • 14
  • 23
19
votes
3 answers

Pushing AWS Lambda data to Kinesis Stream

Is there are way to push data from a Lambda function to a Kinesis stream? I have searched the internet but have not found any examples related to it. Thanks.
Harish
  • 201
  • 1
  • 2
  • 6
19
votes
1 answer

Can I delete data records or shards from amazon Kinesis without deleting stream?

I know data records in Kinesis Stream will be deleted automatically in 24 hrs. But in my application when ever I write some data into stream, for the 2nd time if I want to write some other data, Data inserted first should be deleted. Please anyone…
Kishore Kumar Korada
  • 1,204
  • 6
  • 22
  • 47
18
votes
4 answers

Kinesis Firehose putting JSON objects in S3 without seperator comma

Before sending the data I am using JSON.stringify to the data and it looks like this {"data": [{"key1": value1, "key2": value2}, {"key1": value1, "key2": value2}]} But once it passes through AWS API Gateway and Kinesis Firehose puts it to S3 it…
16
votes
1 answer

What is shards in kinesis data stream

What is shards in kinesis data stream and partition key. I read aws documents but I don't get it. Can someone explain it in simple terms?
Desp
  • 165
  • 1
  • 1
  • 5
15
votes
5 answers

What's the use cases of Streams and Firehose?

I am working on an application that will read and analyze the logs of payment transactions. I know I will use Kinesis Analytics as per my requirements, which takes the input from the Data Streams and Firehose. But I am having trouble deciding which…
14
votes
1 answer

Amazon Kinesis KPL vs AWS SDK pros and cons

The scenario is I would be writing large volumes of data ( terabytes per day) to kinesis stream.I want to know which is a better way to achieve high write throughput. I am considering the below two options for producer clients. Option 1: using…
yin yang
  • 209
  • 5
  • 14
14
votes
4 answers

Concatenate s3 files when using AWS Firehose

I have an AWS Kinesis Firehose stream putting data in s3 with the following config: S3 buffer size (MB)* 2 S3 buffer interval (sec)* 60 Everything works fine. The only problem is that Firehose creates one s3 file for every chunk of data. (In…
14
votes
2 answers

Expected behavior for AWS Kinesis ShardIteratorType TRIM_HORIZON

Context: I'm not necessarily referring to a KCL-based application, just pure Kinesis API calls. Does the using the TRIM_HORIZON shard iterator type immediately give you the earliest published record in the stream (ie earliest available within…
jumand
  • 872
  • 8
  • 17
14
votes
3 answers

Amazon Kinesis + Integration Tests

I'm currently working on a series of web-services which we need to integrate with Kinesis - the implementation has been done, however we have a series of integration tests (our web-services are all using Spring Boot so we use the @WebIntegrationTest…
Ian Jones
  • 173
  • 1
  • 2
  • 10
13
votes
2 answers

Best way to stream/logically replicate RDS Postgres data to kinesis

Our primary datastore is an RDS Postgres database. It would be nice if we could stream all changes to that happen in Postgres to some sink - whether that's kinesis, elasticsearch or any other data store. We use Postgres 9.5 which has support for…
13
votes
1 answer

How can we use AWS Kinesis in a web browser?

How can we use AWS Kinesis in a web browser? I'm interested in AWS Kinesis Stream and wondering if I can use it to send users' activity logs directly from their browsers to AWS. AWS provides a JavaScript SDK which is executable in a web browser but,…
k-kawa
  • 1,289
  • 2
  • 11
  • 18