0

I'm trying to run Kafka Connect on EKS using the Confluent image. The Kafka Connect cluster refers to an MSK cluster. When the Kubernetes pods are created, I see these logs printed over and over.

[2022-01-07 21:55:12,337] WARN [Producer clientId=producer-3] Got error produce response with correlation id 6695 on topic-partition _connect-configs-0, retrying (2147476960 attempts left). Error: NOT_ENOUGH_REPLICAS (org.apache.kafka.clients.producer.internals.Sender)

Below is my kustomize configuration:

spec:
  selector:
    matchLabels:
      app: $APP_NAME
  replicas: 3
template:
    metadata:
      labels:
        app: $APP_NAME
    spec:
      containers:
        - name: $APP_NAME
          image: $APP_NAME-image
          imagePullPolicy: Always
          ports:
            - containerPort:  8083
          resources:
            requests:
              cpu: "500m"
              memory: 1000Mi
            limits:
              cpu: "1"
              memory: 3000Mi
          env:
            - name: CONNECT_CONFIG_STORAGE_REPLICATION_FACTOR
              value: "3"
            - name: CONNECT_OFFSET_STORAGE_REPLICATION_FACTOR
              value: "3"
            - name: CONNECT_STATUS_STORAGE_REPLICATION_FACTOR
              value: "3"
...

Is there some other configuration to be updated to get rid of these warning messages?

OneCricketeer
  • 179,855
  • 19
  • 132
  • 245
guru
  • 409
  • 4
  • 21
  • 1
    You appear to not have enough in-sync replicas (https://towardsdatascience.com/10-configs-to-make-your-kafka-producer-more-resilient-ec6903c63e3f). I'd work through each of the suggestions in that article. – James McPherson Jan 07 '22 at 22:20
  • I do have 3 partitions for internal Connect topics. – guru Jan 07 '22 at 22:34
  • 2
    Partitions are not the same as replicas. How many broker pods do you have running? – James McPherson Jan 07 '22 at 22:41
  • 1
    Sorry I didn't type it right. I meant to say I had set it to 3 replicas and 2 insync replicas. I had to reassign partitions for it work. Thanks! :) – guru Jan 07 '22 at 22:58
  • One curious point is that when I try to run the docker run command for the same setup, I get the following exception: `Topic '_connect-configs' supplied via the 'config.storage.topic' property is required to have a single partition in order to guarantee consistency of connector configurations, but found 3 partitions.` Any idea why this could be? – guru Jan 07 '22 at 23:22
  • If you didn't manually create the topic, it will automatically be created with 1 partition. Sounds like this is not the case, and thus why you get the warning – OneCricketeer Jan 07 '22 at 23:40
  • 1
    By the way, the Strimzi Kafka Operator has a CRD for KafkaConnect, so you can simplify the need for a Deployment spec (and a LoadBalancer) using the Confluent images – OneCricketeer Jan 07 '22 at 23:41
  • @JamesMcPherson Please consider posting this as answer since it's correct and helped OP to resolve the question. – moonkotte Jan 10 '22 at 13:07
  • @OneCricketeer, does the Strimzi KafkaConnect operator also have mirror maker 2 built in? If not, is there a reference that you could share to setup MirrorMaker2 alongside MSK? – guru Jan 10 '22 at 17:38
  • 1
    MirrorMaker2 is built-into Kafka Connect, so it's just another `connector.class` you can set in the config. But yes, https://strimzi.io/docs/operators/latest/overview.html#configuration-points-topic_str And I believe it should work with any Kafka Cluster, not just one hosted in k8s – OneCricketeer Jan 10 '22 at 19:05
  • @OneCricketeer, I setup KafkaConnect cluster on the destination cluster with config: ```{ "name": "mirror-maker-test-connector", "config": { "connector.class": "org.apache.kafka.connect.mirror.MirrorSourceConnector", "topics": "topic_to_mirror", "source.cluster.alias": "dev-cluster", "source.cluster.bootstrap.servers": "source.server.list", "enabled": "true", "target.cluster.alias": "mirrormaker-test-dev", "name": "mirror-maker-test-connector", "target.cluster.bootstrap.servers": "dest.server.list" } }``` I still see task list empty. – guru Jan 11 '22 at 21:12
  • 1
    @OneCricketeer - I have posted my issue with mirror maker 2 here: https://stackoverflow.com/questions/70673874/mirror-maker2-not-able-to-connect-to-target-cluster-broker – guru Jan 13 '22 at 00:17

0 Answers0