0

We are trying to publish data form db2-IIDR (IBM CDC) to Kafka on AWS . Subscription fails due to below error-

 An error occurred during the conversation with Kafka. 
    Error: org.apache.kafka.common.errors.TimeoutException: Batch containing 1 record(s) expired due to timeout while requesting metadata from brokers for cdckafka-subsname-commitstream-0

Are there any pre-requisites to be performed to publish the streams from IIDR to Kafka on aws ?

Some more details -

-Kafka cluster is running on AWS

-IIDR CDC engines --> on premise (both source and target)

-On premise IPs have been white-listed and I can ping/telnet ports from on-prem to aws and vice versa .

Thanks!

Tim Post
  • 33,371
  • 15
  • 110
  • 174
Tony
  • 671
  • 1
  • 9
  • 29
  • You're a bit light on detail here :) What's your deployment look like in terms of network? Everything local to AWS? How have you configured Kafka for the network? etc – Robin Moffatt Jan 25 '19 at 11:13
  • @Robin - I have edited the question . Let me know if you need any more details. – Tony Jan 28 '19 at 09:54

1 Answers1

1

You need to set up your Kafka brokers with listeners that will work with external clients. You can see details here.

Simply pinging from client on-premises to AWS is not enough - you need to validate it with a Kafka client such as kafkacat

Robin Moffatt
  • 30,382
  • 3
  • 65
  • 92