0

I am a newbie with Confluent GCS connector. I followed the documentation outlined in https://docs.confluent.io/4.1.2/connect/kafka-connect-gcs/gcs_connector.html and everything goes according to plan apart from the Kafka topic data not appearing in GCS bucket. The directory structure is there which resembles the kafka topic name but just no actual topic records. I am using the default properties file included with the above documentation for Avro format schema and with a flush size of 3. I downloaded the credentials file and is visible from the VM instance. I also can copy local files from my Google VM instance to the bucket so I know I have correct permissions.

{"name": "gcs",
  "config": {
    "connector.class": "io.confluent.connect.gcs.GcsSinkConnector",
    "tasks.max": "1",
    "topics": "gcs_topic",
    "gcs.bucket.name": "#bucket-name",
    "gcs.part.size": "5242880",
    "flush.size": "3",
    "gcs.credentials.path": "#/path/to/credentials/keys.json",
    "storage.class": "io.confluent.connect.gcs.storage.GcsStorage",
    "format.class": "io.confluent.connect.gcs.format.avro.AvroFormat",
    "partitioner.class": "io.confluent.connect.storage.partitioner.DefaultPartitioner",
    "schema.compatibility": "NONE",
    "confluent.topic.bootstrap.servers": "localhost:9092",
    "confluent.topic.replication.factor": "1",
    "name": "gcs"
  },
  "tasks": [],
  "type": null
}

Connect Properties File is as follows:

bootstrap.servers=localhost:9092
group.id=connect-cluster
key.converter=io.confluent.connect.avro.AvroConverter
key.converter.schema.registry.url=http://localhost:8081
value.converter=io.confluent.connect.avro.AvroConverter
value.converter.schema.registry.url=http://localhost:8081
config.storage.topic=connect-configs
offset.storage.topic=connect-offsets
status.storage.topic=connect-statuses
config.storage.replication.factor=1
offset.storage.replication.factor=1
status.storage.replication.factor=1
internal.key.converter=org.apache.kafka.connect.json.JsonConverter
internal.value.converter=org.apache.kafka.connect.json.JsonConverter
internal.key.converter.schemas.enable=false
internal.value.converter.schemas.enable=false
Plugin.path=~/confluent/share/java,~/confluent/share/confluent-hub-components,~/confluent/etc,~/confluent/etc/
rest.extension.classes=io.confluent.connect.replicator.monitoring.ReplicatorMonitoringExtension
producer.interceptor.classes=io.confluent.monitoring.clients.interceptor.MonitoringProducerInterceptor
consumer.interceptor.classes=io.confluent.monitoring.clients.interceptor.MonitoringConsumerInterceptor
Juancki
  • 1,793
  • 1
  • 14
  • 21
Newbie
  • 1
  • 1
  • Not clear if you're actively sending any data to the topic – OneCricketeer Feb 21 '20 at 13:37
  • Yes sending data as per instructions using an avro producer with some test records. Also consuming it locally to a consumer instance to ensure kafka topic is receiving event records. All okay. – Newbie Feb 21 '20 at 13:51
  • Can you please show your Connect Distributed properties? – OneCricketeer Feb 21 '20 at 14:08
  • connect.stdout recording the following types of messages -Added READ_UNCOMMITTED fetch request for partition connect-offsets-10 at position FetchPosition{offset=0, offsetEpoch=Optional – Newbie Feb 21 '20 at 14:08
  • Also, related - https://stackoverflow.com/q/60334554/2308683 – OneCricketeer Feb 21 '20 at 14:08
  • Based on that message you might have unclean leader election enabled on your brokers – OneCricketeer Feb 21 '20 at 14:09
  • 1
    Stopped and destroyed local kafka cluster. Restarted everything again, sent 80 test events through the producer and it seems to be working. Thank you for your help. – Newbie Feb 21 '20 at 14:58

0 Answers0