How to shutdown processRecord method of KCL in case of any error?

Question

I am using multiple Scheduler to process record from multiple kinesis stream. Now, my requirement is, If scheduler I failed during processing, so I want to stop it, and don't want to stop scheduler II. But currently, processRecord method is not stopping, it is polling even after I have thrown error. And If I am using Thread.currentThread().stop() to stop particular schedular on exception it is giving error

Caused by: java.lang.IllegalStateException: Client is closed. No more requests can be made with this client.
at software.amazon.awssdk.http.crt.AwsCrtAsyncHttpClient.getOrCreateConnectionPool(AwsCrtAsyncHttpClient.java:208) ~[aws-crt-client-2.17.148-PREVIEW.jar:na]
at software.amazon.awssdk.http.crt.AwsCrtAsyncHttpClient.execute(AwsCrtAsyncHttpClient.java:235) ~[aws-crt-client-2.17.148-PREVIEW.jar:na]
at software.amazon.awssdk.core.internal.http.pipeline.stages.MakeAsyncHttpRequestStage.doExecuteHttpRequest(MakeAsyncHttpRequestStage.java:175) ~[sdk-core-2.17.160.jar:na]
at software.amazon.awssdk.core.internal.http.pipeline.stages.MakeAsyncHttpRequestStage.executeHttpRequest(MakeAsyncHttpRequestStage.java:147) ~[sdk-core-2.17.160.jar:na]
at software.amazon.awssdk.core.internal.http.pipeline.stages.MakeAsyncHttpRequestStage.lambda$execute$1(MakeAsyncHttpRequestStage.java:99) ~[sdk-core-2.17.160.jar:na]
at java.base/java.util.concurrent.CompletableFuture.uniAcceptNow(CompletableFuture.java:753) ~[na:na]
at java.base/java.util.concurrent.CompletableFuture.uniAcceptStage(CompletableFuture.java:731) ~[na:na]
at java.base/java.util.concurrent.CompletableFuture.thenAccept(CompletableFuture.java:2108) ~[na:na]
at software.amazon.awssdk.core.internal.http.pipeline.stages.MakeAsyncHttpRequestStage.execute(MakeAsyncHttpRequestStage.java:95) ~[sdk-core-2.17.160.jar:na]
at software.amazon.awssdk.core.internal.http.pipeline.stages.MakeAsyncHttpRequestStage.execute(MakeAsyncHttpRequestStage.java:60) ~[sdk-core-2.17.160.jar:na]
at software.amazon.awssdk.core.internal.http.pipeline.RequestPipelineBuilder$ComposingRequestPipelineStage.execute(RequestPipelineBuilder.java:206) ~[sdk-core-2.17.160.jar:na]

My processRecord Method code:

@SneakyThrows
@Override
public void processRecords(ProcessRecordsInput processRecordsInput) {

for (KinesisClientRecord s : processRecordsInput.records()) {
  
  try {
   //some opreations on records
   //doing glue validation
   //then at last saving record to dynamodb 

  } catch (GlueSchemaException e) {

  } catch (Exception ex) {
    LOGGER.info("Stopping current thread "+ Thread.currentThread().getName());
    processRecordsInput.checkpointer().checkpoint();
    Thread.currentThread().stop();//you can skip this line, I am trying to stop the further processing
    throw new NullPointerException();
  }
  
  processRecordsInput.checkpointer().checkpoint();
}

I am trying to stop KCL by throwing exception also, but that is also not working, it is still not stopping processing.

score 0 · Answer 1 · answered Jun 12 '23 at 16:30

I am using two scheduler which are reading from different kinesis stream.

//Application.class
 Created HashMap to store the Schedular information.
  public static ConcurrentHashMap<String, Scheduler> schedulerMap = new 
ConcurrentHashMap<>();

Scheduler scheduler = schedulerFactory.getScheduler(schedulerConfigBuilder.buildConfiguration(streamConfig 
    ));
   
schedulerMap.put(streamConfig.getLogicalName(), scheduler);



//KinesisRecordProcessor class
@SneakyThrows
@Override
public void processRecords(ProcessRecordsInput processRecordsInput) {

 for (KinesisClientRecord s : processRecordsInput.records()) {

 try {
  //some opreations on records
  //doing glue validation
 //then at last saving record to dynamodb 

  } catch (GlueSchemaException e) {
  
  } catch (Exception ex) {
 //And used this information to stop schedular as below in case of error
 Application.schedulerMap.get(streamConfig.getLogicalName()).shutdown();
   
 }

 processRecordsInput.checkpointer().checkpoint();
}

How to shutdown processRecord method of KCL in case of any error?

1 Answers1