1

I am using SQL 2.4.1v as consumer and Spring boot as producer for my Kafka topic. While I am trying to insert the records on to the topic, I am getting following error message:

WARN 12044 --- [ad | producer-1] o.a.k.clients.producer.internals.Sender  : [Producer clientId=producer-1] Got error produce response with correlation id 4457 on topic-partition TRANS_INBOUND-20, retrying (0 attempts left). Error: NETWORK_EXCEPTION
WARN 12044 --- [ad | producer-1] o.a.k.clients.producer.internals.Sender  : [Producer clientId=producer-1] Received invalid metadata error in produce request on partition TRANS_INBOUND-20 due to org.apache.kafka.common.errors.NetworkException: The server disconnected before a response was received.. Going to request metadata update now
ERROR 12044 --- [ad | producer-1] o.s.k.support.LoggingProducerListener    : Exception thrown when sending a message with key='1-356194-2018-01-02-STATUS' and payload='com.TransRecord@48323556' to topic TRANS_INBOUND:

org.apache.kafka.common.errors.TimeoutException: Expiring 1 record(s) for TRANS_INBOUND-82: 501 ms has passed since batch creation plus linger time

Following are my producer settings:

    acks: 1
    retries: 1
    batchSize: 100
    lingerMs: 5
    bufferMemory: 33554432 
    requestTimeoutMs: 600
    autoOffsetReset: latest
    enableAutoCommit: false
    reconnectBackoffMaxMs: 1000
    reconnectBackoffMs: 50
    retryBackoffMs: 100

I have already tried with many combinations of batchSize, lingerMs and requestTimeoutMs, nothing is working. I still see the above errors quite often. What might be wrong here, and how can I fix this problem?

halfer
  • 19,824
  • 17
  • 99
  • 186
BdEngineer
  • 2,929
  • 4
  • 49
  • 85
  • 1
    Why have u used the config batch.size ? – Fatema Khuzaima Sagar Feb 03 '20 at 09:37
  • First increase the retries to more than one. But if you're getting timeouts, that's sometimes a network issue, not a Kafka config – OneCricketeer Feb 03 '20 at 14:55
  • 1
    You can use tools like TCPdump or Wireshark for network analytics, or hire a network engineer / talk with your network provider to see if they'll help... Is your stuff running in the cloud? All on localhost? Could you draw any type of network diagram? – OneCricketeer Feb 04 '20 at 13:06
  • Oracle is not in the cloud? Could you run the Spark "producer" in AWS? I would suggest using Kafka Connect rather than Spark, but that's just my thoughts – OneCricketeer Feb 05 '20 at 14:39
  • 1
    That it's micro-batch and not truly CDC or really meant to do database migrations or backups – OneCricketeer Feb 05 '20 at 15:30
  • You might as well use Sqoop in EMR to take a database to S3 – OneCricketeer Feb 05 '20 at 15:31
  • 1
    I would really like to hear the architects decision not to use a proper tool built for the purpose of taking data from Oracle to Kafka (e.g. Debezium, GoldenGate, JDBC Connector, etc)... In any case, you're getting timeout errors because you are running a producer across the internet rather than producing to a local cluter. Run your Producer code in the cloud and pull from the DB remotely. It will probably work better – OneCricketeer Feb 05 '20 at 17:50
  • @OneCricketeer How do you know that he's "running a producer across the internet, rather than a local cluster"? – Marko Bonaci Sep 14 '20 at 11:54
  • @MarkoBonaci Old comments were removed – OneCricketeer Sep 14 '20 at 14:42
  • Possible duplicate of https://stackoverflow.com/q/46750420 – hongsy Jan 04 '21 at 13:24
  • Does this answer your question? [Kafka Producer error Expiring 10 record(s) for TOPIC:XXXXXX: 6686 ms has passed since batch creation plus linger time](https://stackoverflow.com/questions/46750420/kafka-producer-error-expiring-10-records-for-topicxxxxxx-6686-ms-has-passed) – hongsy Jan 04 '21 at 13:24

0 Answers0