0

I am sending data to kinesis data stream using putRecords request asynchronously (500 records every time). There are 4 shards in the stream in which multiple putRecords requests continuously sends data using a for loop. But when I checked the stream metrics the Incoming data - sum (count) and Incoming data - sum (MB/s) are showing consumption very less comparatively (check screenshot). For example, The incoming data count can hold upto 1,200,000 but showing only 42,183 records being produced. I am expecting the consumption to be at-least half of the limits.

What can be reason for this behaviour?

And

How can I utilize the maximum capacity in the Kinesis Stream (At least half the maximum limit)

Note: There is no delay while sending the records from application

Code:

AmazonKinesisAsync kinesisClient = AmazonKinesisAsyncClientBuilder.standard() // async cleint
                        .withCredentials(DefaultAWSCredentialsProviderChain.getInstance())
                        .withRegion(kinesisServiceConfig.getAwsRegion())
                        .build();

for(long j=0; j<1000000000; j++) {
    PutRecordsRequest putRecordsRequest  = new PutRecordsRequest();
    putRecordsRequest.setStreamName(streamName);
    List <PutRecordsRequestEntry> putRecordsRequestEntryList  = new ArrayList<>(); 
    for (int i = 0; i < 500; i++) { // 500 records kinesis limit for putRecords
        PutRecordsRequestEntry putRecordsRequestEntry  = new PutRecordsRequestEntry();
        putRecordsRequestEntry.setData(ByteBuffer.wrap(String.valueOf(i).getBytes()));
        putRecordsRequestEntry.setPartitionKey(String.format("partitionKey-%d", i));
        putRecordsRequestEntryList.add(putRecordsRequestEntry); 
    }

    putRecordsRequest.setRecords(putRecordsRequestEntryList);
    PutRecordsResult putRecordsResult  = kinesisClient.putRecords(putRecordsRequest);
}
    

enter image description here

enter image description here

Tomerikoo
  • 18,379
  • 16
  • 47
  • 61

0 Answers0