Questions tagged [amazon-kinesis]

Amazon Kinesis is a fully managed service for real-time processing of streaming data at massive scale.

Amazon Kinesis is a fully managed service for real-time processing of streaming data at massive scale. Amazon Kinesis can collect and process hundreds of terabytes of data per hour from hundreds of thousands of sources, allowing you to easily write applications that process information in real-time, from sources such as web site click-streams, marketing and financial information, manufacturing instrumentation and social media, and operational logs and metering data.

With Amazon Kinesis applications, you can build real-time dashboards, capture exceptions and generate alerts, drive recommendations, and make other real-time business or operational decisions. You can also easily send data to a variety of other services such as Amazon Simple Storage Service (Amazon S3), Amazon DynamoDB, or Amazon Redshift. In a few clicks and a couple of lines of code, you can start building applications which respond to changes in your data stream in seconds, at any scale, while only paying for the resources you use.

Useful links

1802 questions
9
votes
1 answer

Integrating Apache Camel to Amazon Kinesis Streaming Service

Has anyone integrated Apache Camel to Amazon Kinesis Streaming Service? This is related to Using Apache Camel with Amazon AWS which includes a question about other Amazon services being integrated with Apache Camel. I was hoping that Amazon…
8
votes
2 answers

Apache Flink - Unable to use local Kinesis for FlinkKinesisConsumer

So far I have followed the instructions documented for Flink's kinesis connector to use a local Kinesis. Using Non-AWS Kinesis Endpoints for Testing Properties producerConfig = new Properties(); producerConfig.put(AWSConfigConstants.AWS_REGION,…
Jack
  • 1,901
  • 1
  • 19
  • 32
8
votes
2 answers

AWS Lambda and Kinesis Client Library (KCL)

How come I find so little examples of the KCL being used with AWS Lambda. https://docs.aws.amazon.com/streams/latest/dev/developing-consumers-with-kcl.html It does provide a fine implementation for keeping track of your position on the stream…
xtra
  • 1,957
  • 4
  • 22
  • 40
8
votes
1 answer

AWS Kinesis - how to resume consuming from last checkpoint

I'm converting a Kafka consumer to an AWS Kinesis consumer, using the KCL (v2). In Kafka, offsets are used to help a consumer keep track of its most recently-consumed message. If my Kafka app dies, it will use the offset to consume from where it…
Ben Watson
  • 5,357
  • 4
  • 42
  • 65
8
votes
4 answers

Lambda.InvokeAccessDenied from Kinesis Firehose

I'm trying to set up Lambda transformations with a Firehose delivery stream. I have an IAM role defined for the Firehose which includes the following policy document: { "Statement": { "Action": [ "lambda:InvokeFunction", …
8
votes
1 answer

Anyone experienced data lost when using AWS kinesis streams, lambda and firehose?

I'm currently sending a series of xml messages to aws kinesis stream, I've been using this on different projects, so I'm pretty confident that this bit works. Then I've written a lambda to process events from kinesis stream to kinesis…
8
votes
4 answers

Is there any difference in processing times between AWS Kinesis Firehose and Streams?

Reading over the documentation of both offerings (Firehose and Streams), it sounds like Firehose is "near" real-time with a potential of 60 seconds delay between producing a message to emitting it, whereas Streams documentation makes no mentioning…
8
votes
5 answers

Kinesis stream / shard - multiple consumers

I have already read some questions about kinesis shard and multiple consumers but I still don't understand how it works. My use case: I have a kinesis stream with just one shard. I would like to consume this shard using different lambda function,…
p.magalhaes
  • 7,595
  • 10
  • 53
  • 108
8
votes
2 answers

Cross-account lambda trigger by kinesis

I'm trying to trigger a lambda in account 'B' by a Kinesis stream in account 'A'. This is similar to what's described here, except the example uses S3 instead of Kinesis. To do this, I'm trying to set up the right permissions, but running into…
Daniel Kats
  • 5,141
  • 15
  • 65
  • 102
8
votes
0 answers

Kinesis Spark Streaming longevity issues

I'm having issues with the longevity of Spark-Kinesis Streaming application running on spark standalone cluster manager. The program runs for around 50 hours and stops receiving data from kinesis without giving any valid error why it stopped. But if…
sreejith
  • 213
  • 2
  • 8
8
votes
2 answers

Read and write transactions in Amazon Kinesis

I'm new to Kinesis, so this might seem like a very basic question, but I have not been able to find a clear answer to what the actual difference is between a read and write transaction in a Kinesis stream. Relevant parts from Amazon Kinesis…
KennethJ
  • 924
  • 1
  • 7
  • 16
8
votes
2 answers

How to verify that Amazon Kinesis Python client is working

I'm trying to build an Amazon Kinesis Python consumer using the KCL library for Python (https://github.com/awslabs/amazon-kinesis-client-python). I started by checking the sample code. I was able to run both producer and consumer scripts parts of…
Carlos
  • 161
  • 7
7
votes
1 answer

How does kinesis firehose stream data to self managed elasticsearch?

I am hosting Elasticsearch cluster in EKS and I'd like to stream all cloudwatch groups to this Elasticsearch cluster via Kinesis Firehose. But AWS Kinesis firehose doesn't support stream data to Elasticsearch cluster other than AWS hosted ES. What…
Joey Yi Zhao
  • 37,514
  • 71
  • 268
  • 523
7
votes
1 answer

Amazon AWS Kinesis Video Boto GetMedia/PutMedia

Does anybody know of a complete sample as to how to send video to a kinesis video stream, using boto3 sdk? This question was asked initially for both both GetMedia and PutMedia. Now I have got this sample code for the GetMedia part: client =…
minghua
  • 5,981
  • 6
  • 45
  • 71
7
votes
2 answers

WARNING: erroneous pipeline: no element "kvssink"

I'm trying Kinesis Video Stream via https://docs.aws.amazon.com/kinesisvideostreams/latest/dg/gs-send-data.html I have already installled it Success in building the Kinesis Video Streams Producer SDK !!! Now you can set the environment variables…