0

We created a new AWS MSK (kafka) cluster

We created a new configuration and assigned to that cluster:

auto.create.topics.enable=true
default.replication.factor=3
min.insync.replicas=2
num.io.threads=8
num.network.threads=5
num.partitions=1
num.replica.fetchers=2
replica.lag.time.max.ms=30000
socket.receive.buffer.bytes=102400
socket.request.max.bytes=104857600
socket.send.buffer.bytes=102400
unclean.leader.election.enable=true
zookeeper.session.timeout.ms=18000

With a python script we want to send a new message:

client = boto3.client('kafka')
response = client.get_bootstrap_brokers(ClusterArn=os.environ.get('MSKARN'))
bootstrap_servers = response['BootstrapBrokerStringTls']

instance = KafkaProducer(
    bootstrap_servers=bootstrap_servers,
    value_serializer=lambda v: json.dumps(v).encode('utf-8'),
    security_protocol='SSL',
    ssl_keyfile='kafka.client.truststore.jsk',
    api_version=(2, 2, 1)
)
    
instance.send("TEST", value=message)

It does not work and says:

kafka.errors.KafkaTimeoutError: KafkaTimeoutError: Failed to update metadata after 60.0 secs.

If we create topic before sending message, it works without any error.

Metalik
  • 873
  • 1
  • 10
  • 28

1 Answers1

2

Found the solution.

The problem was related to this line:

default.replication.factor=3

It must be lte to number of brokers, we had 2 brokers and reset that value.

Metalik
  • 873
  • 1
  • 10
  • 28