0

I am facing a very strange problem and I am not sure if it is a bug in aws or my lack of understanding.

Here is the problem. I have a kinesis analytins reoprting every 40 seconds. so lets say it reported the following 40 second ago:

{frist row with some data}, {second row with some data}

So I connect the kinesis analitycs to two destination: 1) lambda: and the result lambda is recieving is as follows:

{
"invocationId": "2db390f4-d5a1-49cf-a792-73827d37ec34",
"applicationArn": "arn:aws:kinesisanalytics:us-east-1:638417958056:application/-analytic-app-trending-stories",
"records": [
    {
        "recordId": "672e17e4-06a9-41cb-a5ba-341cf5f3b879",
        "lambdaDeliveryRecordMetadata": {
            "retryHint": 0
        },
        "data": "eyJzSWQiOjQ0NDQ0LCJjSWQiOjExMjIyLCJjb3VudCI6MjAuMCwicm93VGltZTEiOiIyMDE4LTAxLTE2IDE0OjE2OjQwLjAwMSJ9"
    },
    {
        "recordId": "6fbb409d-ddb0-40b7-b700-45b7ce87abe2",
        "lambdaDeliveryRecordMetadata": {
            "retryHint": 0
        },
        "data": "eyJzSWQiOjQ0NDQ0LCJjSWQiOjIyMiwiY291bnQiOjEzLjAsInJvd1RpbWUxIjoiMjAxOC0wMS0xNiAxNDoxNjo0MC4wMDEifQ=="
    }
]
  }

So as you can see two rows were sent to lambda in one payload and obviously we have to data in array format seond to lambda. After decoding the content of the data in each record I see the same results sent from analytics. So far so good but the problem starts here: I connected analytics to kinesis stream and then I connected the stream to lambda. I expect the same result happened in the first sceario I mean two records in one payload shouuld be recieved in second lambda but suprisingly I recienve just one record in each payload and seems that stream split array and send it in different way for more clarification here is the payload I get:

{
"Records": [
    {
        "kinesis": {
            "kinesisSchemaVersion": "1.0",
            "partitionKey": "hh",
            "sequenceNumber": "49580809756311348244603591366792053449767996205245136898",
            "data": "eyJzSWQiOjQ0NDQ0LCJjSWQiOjExMjIyLCJjb3VudCI6MjAuMCwicm93VGltZTEiOiIyMDE4LTAxLTE2IDE0OjE2OjQwLjAwMSJ9",
            "approximateArrivalTimestamp": 1516112247.8
        },
        "eventSource": "aws:kinesis",
        "eventVersion": "1.0",
        "eventID": "shardId-000000000000:49580809756311348244603591366792053449767996205245136898",
        "eventName": "aws:kinesis:record",
        "invokeIdentityArn": ".....lambda-img-resizer-role",
        "awsRegion": "us-east-1",
        "eventSourceARN": "arn:aws:kinesis:us-east-1:638417958056:stream/bni-tj-sbx22-stream-trending-stories-output"
    }
]
 }

As you see in the payload we have just one data attr. I am totally lost can anyone shed light on this?

John Rotenstein
  • 241,921
  • 22
  • 380
  • 470
Hamed Minaee
  • 2,480
  • 4
  • 35
  • 63
  • 1
    Did you ever figure this out? I am about to embark down a similar path and need to choose between setting my analytic destination as a Lambda directly or setting it to another Kinesis Stream which is consumed by a Lambda. – AJ Venturella Mar 28 '18 at 22:02
  • @AdamVenturella I would recommend not to use as a Lambda directly approach. It seems that the second approach is very buggy and unpredictable – Hamed Minaee Mar 29 '18 at 14:36

0 Answers0