kafka connector cannot find s3 target which is minio url

Question

I've Kafka and Kafka Connector clusters installed by Strimzi on Kubernetes.

This is my s3/minio connector config:

apiVersion: kafka.strimzi.io/v1alpha1
kind: KafkaConnector
metadata:
  name: "minio-connector"
  labels:
    strimzi.io/cluster: mssql-minio-connect-cluster
spec:
  class: io.confluent.connect.s3.S3SinkConnector
  config:
    storage.class: io.confluent.connect.s3.storage.S3Storage  
    format.class: io.confluent.connect.s3.format.avro.AvroFormat
    schema.generator.class: io.confluent.connect.storage.hive.schema.DefaultSchemaGenerator
    partitioner.class: io.confluent.connect.storage.partitioner.DefaultPartitioner
    tasks.max: '1'
    topics: filesql1.dbo.files
    s3.bucket.name: dosyalar
    s3.part.size: '5242880'
    flush.size: '3'
    format: binary
    schema.compatibility: NONE
    max.request.size: "536870912"
    store.url: http://minio.dev-kik.io

and this is my custom Kafka-Connect cluster docker container Dockerfile config:

FROM strimzi/kafka:latest-kafka-2.5.0
USER root:root
COPY ./plugins/ /opt/kafka/plugins/
USER 1001
ENV MINIO_VOLUMES=http://minio.dev-kik.io
ENV AWS_ACCESS_KEY_ID=AKIAIOSFODNN7EXAMPLE
ENV AWS_SECRET_ACCESS_KEY=wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY

There is something wrong in my config. I think my Kafka-Connect pods cannot find or access to my Minio:

    {                                                                                                                                                                                                                  
      "name": "minio-connector",                                                                                                                                                                                       
      "connector": {                                                                                                                                                                                                   
        "state": "RUNNING",                                                                                                                                                                                            
        "worker_id": "10.244.1.183:8083"                                                                                                                                                                               
      },                                                                                                                                                                                                               
      "tasks": [                                                                                                                                                                                                       
        {                                                                                                                                                                                                              
          "id": 0,                                                                                                                                                                                                     
          "state": "FAILED",                                                                                                                                                                                           
          "worker_id": "10.244.1.183:8083",                                                                                                                                                                            
          "trace": "org.apache.kafka.connect.errors.ConnectException: com.amazonaws.SdkClientException: Unable to execute HTTP request: minio.dev-kik.io\n\tat io.confluent.connect.s3.S3SinkTask.start(S3SinkTask.java
    :121)\n\tat org.apache.kafka.connect.runtime.WorkerSinkTask.initializeAndStart(WorkerSinkTask.java:305)\n\tat org.apache.kafka.connect.runtime.WorkerSinkTask.execute(WorkerSinkTask.java:193)\n\tat org.apache.kaf
    ka.connect.runtime.WorkerTask.doRun(WorkerTask.java:184)\n\tat org.apache.kafka.connect.runtime.WorkerTask.run(WorkerTask.java:234)\n\tat java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)\n
    \tat java.util.concurrent.FutureTask.run(FutureTask.java:266)\n\tat java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)\n\tat java.util.concurrent.ThreadPoolExecutor$Worker.run(Thread
    PoolExecutor.java:624)\n\tat java.lang.Thread.run(Thread.java:748)\nCaused by: com.amazonaws.SdkClientException: Unable to execute HTTP request: minio.dev-kik.io\n\tat com.amazonaws.http.AmazonHttpClient$Request
    Executor.handleRetryableException(AmazonHttpClient.java:1201)\n\tat com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeHelper(AmazonHttpClient.java:1147)\n\tat com.amazonaws.http.AmazonHttpClient$Request
    Executor.doExecute(AmazonHttpClient.java:796)\n\tat com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeWithTimer(AmazonHttpClient.java:764)\n\tat com.amazonaws.http.AmazonHttpClient$RequestExecutor.execu
    te(AmazonHttpClient.java:738)\n\tat com.amazonaws.http.AmazonHttpClient$RequestExecutor.access$500(AmazonHttpClient.java:698)\n\tat com.amazonaws.http.AmazonHttpClient$RequestExecutionBuilderImpl.execute(AmazonH
    ttpClient.java:680)\n\tat com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:544)\n\tat com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:524)\n\tat com.amazonaws.services.s3.Amazo
    nS3Client.invoke(AmazonS3Client.java:5052)\n\tat com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:4998)\n\tat com.amazonaws.services.s3.AmazonS3Client.headBucket(AmazonS3Client.java:1413)\n\ta
    t com.amazonaws.services.s3.AmazonS3Client.doesBucketExist(AmazonS3Client.java:1349)\n\tat io.confluent.connect.s3.storage.S3Storage.bucketExists(S3Storage.java:196)\n\tat io.confluent.connect.s3.S3SinkTask.star
    t(S3SinkTask.java:108)\n\t... 9 more\nCaused by: java.net.UnknownHostException: minio.dev-kik.io\n\tat java.net.InetAddress.getAllByName0(InetAddress.java:1281)\n\tat java.net.InetAddress.getAllByName(InetAddres
    s.java:1193)\n\tat java.net.InetAddress.getAllByName(InetAddress.java:1127)\n\tat com.amazonaws.SystemDefaultDnsResolver.resolve(SystemDefaultDnsResolver.java:27)\n\tat com.amazonaws.http.DelegatingDnsResolver.r
    esolve(DelegatingDnsResolver.java:38)\n\tat org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:112)\n\tat org.apache.http.impl.conn.PoolingHttpClientCo
    nnectionManager.connect(PoolingHttpClientConnectionManager.java:374)\n\tat sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)\n\tat sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.j
    ava:62)\n\tat sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)\n\tat java.lang.reflect.Method.invoke(Method.java:498)\n\tat com.amazonaws.http.conn.ClientConnectionManagerFac
    tory$Handler.invoke(ClientConnectionManagerFactory.java:76)\n\tat com.amazonaws.http.conn.$Proxy37.connect(Unknown Source)\n\tat org.apache.http.impl.execchain.MainClientExec.establishRoute(MainClientExec.java:3
    93)\n\tat org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:236)\n\tat org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:186)\n\tat org.apache.http.impl.client.Interna
    lHttpClient.doExecute(InternalHttpClient.java:185)\n\tat org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83)\n\tat org.apache.http.impl.client.CloseableHttpClient.execute(Closeab
    leHttpClient.java:56)\n\tat com.amazonaws.http.apache.client.impl.SdkHttpClient.execute(SdkHttpClient.java:72)\n\tat com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeOneRequest(AmazonHttpClient.java:13
    23)\n\tat com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeHelper(AmazonHttpClient.java:1139)\n\t... 22 more\n"                                                                                          
        }                                                                                                                                                                                                              
      ],                                                                                                                                                                                                               
      "type": "sink"                                                                                                                                                                                                   
    }

Also pls look at that output:

Name:         minio-connector
Namespace:    kafka
Labels:       strimzi.io/cluster=mssql-minio-connect-cluster
Annotations:  API Version:  kafka.strimzi.io/v1alpha1
Kind:         KafkaConnector
Metadata:
  Creation Timestamp:  2020-05-17T07:47:58Z
  Generation:          2
  Managed Fields:
    API Version:  kafka.strimzi.io/v1alpha1
    Fields Type:  FieldsV1
    fieldsV1:
      f:status:
        f:conditions:
        f:connectorStatus:
          f:connector:
            f:worker_id:
          f:tasks:
    Manager:         okhttp
    Operation:       Update
    Time:            2020-05-17T19:22:12Z
  Resource Version:  5349641
  Self Link:         /apis/kafka.strimzi.io/v1alpha1/namespaces/kafka/kafkaconnectors/minio-connector
  UID:               4f4a080d-580e-4243-b187-c8b485461285
Spec:
  Class:  io.confluent.connect.s3.S3SinkConnector
  Config:
    flush.size:              3
    Format:                  binary
    format.class:            io.confluent.connect.s3.format.avro.AvroFormat
    max.request.size:        536870912
    partitioner.class:       io.confluent.connect.storage.partitioner.DefaultPartitioner
    s3.bucket.name:          dosyalar
    s3.part.size:            5242880
    schema.compatibility:    NONE
    schema.generator.class:  io.confluent.connect.storage.hive.schema.DefaultSchemaGenerator
    storage.class:           io.confluent.connect.s3.storage.S3Storage
    store.url:               http://minio.dev-kik.io
    tasks.max:               1
    Topics:                  s3-topic
Status:
  Conditions:
    Last Transition Time:  2020-05-17T19:22:12.526Z
    Status:                True
    Type:                  Ready
  Connector Status:
    Connector:
      State:      RUNNING
      worker_id:  10.244.1.183:8083
    Name:         minio-connector
    Tasks:
      Id:     0
      State:  FAILED
      Trace:  org.apache.kafka.connect.errors.ConnectException: com.amazonaws.SdkClientException: Unable to execute HTTP request: minio.dev-kik.io
              at io.confluent.connect.s3.S3SinkTask.start(S3SinkTask.java:121)
              at org.apache.kafka.connect.runtime.WorkerSinkTask.initializeAndStart(WorkerSinkTask.java:305)
              at org.apache.kafka.connect.runtime.WorkerSinkTask.execute(WorkerSinkTask.java:193)
              at org.apache.kafka.connect.runtime.WorkerTask.doRun(WorkerTask.java:184)
              at org.apache.kafka.connect.runtime.WorkerTask.run(WorkerTask.java:234)
              at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
              at java.util.concurrent.FutureTask.run(FutureTask.java:266)
              at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
              at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
              at java.lang.Thread.run(Thread.java:748)
Caused by: com.amazonaws.SdkClientException: Unable to execute HTTP request: minio.dev-kik.io
  at com.amazonaws.http.AmazonHttpClient$RequestExecutor.handleRetryableException(AmazonHttpClient.java:1201)
  at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeHelper(AmazonHttpClient.java:1147)
  at com.amazonaws.http.AmazonHttpClient$RequestExecutor.doExecute(AmazonHttpClient.java:796)
  at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeWithTimer(AmazonHttpClient.java:764)
  at com.amazonaws.http.AmazonHttpClient$RequestExecutor.execute(AmazonHttpClient.java:738)
  at com.amazonaws.http.AmazonHttpClient$RequestExecutor.access$500(AmazonHttpClient.java:698)
  at com.amazonaws.http.AmazonHttpClient$RequestExecutionBuilderImpl.execute(AmazonHttpClient.java:680)
  at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:544)
  at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:524)
  at com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:5052)
  at com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:4998)
  at com.amazonaws.services.s3.AmazonS3Client.headBucket(AmazonS3Client.java:1413)
  at com.amazonaws.services.s3.AmazonS3Client.doesBucketExist(AmazonS3Client.java:1349)
  at io.confluent.connect.s3.storage.S3Storage.bucketExists(S3Storage.java:196)
  at io.confluent.connect.s3.S3SinkTask.start(S3SinkTask.java:108)
  ... 9 more
Caused by: java.net.UnknownHostException: minio.dev-kik.io
  at java.net.InetAddress.getAllByName0(InetAddress.java:1281)
  at java.net.InetAddress.getAllByName(InetAddress.java:1193)
  at java.net.InetAddress.getAllByName(InetAddress.java:1127)
  at com.amazonaws.SystemDefaultDnsResolver.resolve(SystemDefaultDnsResolver.java:27)
  at com.amazonaws.http.DelegatingDnsResolver.resolve(DelegatingDnsResolver.java:38)
  at org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:112)
  at org.apache.http.impl.conn.PoolingHttpClientConnectionManager.connect(PoolingHttpClientConnectionManager.java:374)
  at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
  at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
  at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
  at java.lang.reflect.Method.invoke(Method.java:498)
  at com.amazonaws.http.conn.ClientConnectionManagerFactory$Handler.invoke(ClientConnectionManagerFactory.java:76)
  at com.amazonaws.http.conn.$Proxy37.connect(Unknown Source)
  at org.apache.http.impl.execchain.MainClientExec.establishRoute(MainClientExec.java:393)
  at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:236)
  at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:186)
  at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185)
  at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83)
  at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:56)
  at com.amazonaws.http.apache.client.impl.SdkHttpClient.execute(SdkHttpClient.java:72)
  at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeOneRequest(AmazonHttpClient.java:1323)
  at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeHelper(AmazonHttpClient.java:1139)
  ... 22 more

      worker_id:        10.244.1.183:8083
    Type:               sink
  Observed Generation:  2
Events:                 <none>

When I grep all logs I see that:

devadmin@vdi-mk2-ubn:~$ kubectl logs mssql-minio-connect-cluster-connect-5bf6d6bbcc-ggh5g -n kafka | grep minio.dev-kik.io
        store.url = http://minio.dev-kik.io
        store.url = http://minio.dev-kik.io
org.apache.kafka.connect.errors.ConnectException: com.amazonaws.SdkClientException: Unable to execute HTTP request: minio.dev-kik.io
Caused by: com.amazonaws.SdkClientException: Unable to execute HTTP request: minio.dev-kik.io
Caused by: java.net.UnknownHostException: minio.dev-kik.io
        store.url = http://minio.dev-kik.io
devadmin@vdi-mk2-ubn:~$ kubectl logs mssql-minio-connect-cluster-connect-5bf6d6bbcc-ggh5g -n kafka | grep http://minio.dev-kik.io
        store.url = http://minio.dev-kik.io
        store.url = http://minio.dev-kik.io
        store.url = http://minio.dev-kik.io
devadmin@vdi-mk2-ubn:~$ kubectl logs mssql-minio-connect-cluster-connect-5bf6d6bbcc-ggh5g -n kafka | grep Unable
org.apache.kafka.connect.errors.ConnectException: com.amazonaws.SdkClientException: Unable to execute HTTP request: minio.dev-kik.io
Caused by: com.amazonaws.SdkClientException: Unable to execute HTTP request: minio.dev-kik.io

It looks like s3 connector looks for Minio/S3 on AWS whereas my minio is not on AWS, it's on Local. When I nslookup Minio url (minio.kik-dev.io), the connect cluster pod can resolve the ip but ping command cannot. I'm not sure it's related with dns resolution.

The problem is driving me crazy for days, could you please advise what I'm missing?

Have you tried to `curl` the URL from inside the pod? Could it be that your cluster has some Egress network policies? Or is it maybe behind some proxy? — Jakub, May 17 '20 at 22:46
yes, curl works but no idea about Egress or proxy stuff. My Kubernetes on-prem simple k8s, there is no any special network config. I use ponly flannel and nginx reverse proxy and ingress. — Tireli Efe, May 18 '20 at 02:02
If `curl` than there should be no issue with network I guess. — Jakub, May 18 '20 at 07:43
I have 3 different topics; 1 of them comes from MS SQL Server table, the other one based on that article s3_topic test topic https://blog.minio.io/journaling-kafka-messages-with-s3-connector-and-minio-83651a51045d and 3'rd one is empty one. Sometimes I change "topic name" in the minio-connector. I just noticed that the issue above occurs when I change/reconfigure the topic in minio connector. Weird... — Tireli Efe, May 18 '20 at 07:52

kafka connector cannot find s3 target which is minio url

0 Answers0