I am using confluent connector for kafka and Google Cloud Storage(GCS). I installed confluent using:
(I am doing this on Ubuntu 16.04 and all these commands were executed in the home directory.)
curl https://packages.confluent.io/archive/5.0/confluent-5.0.0-2.11.tar.gz | tar xz
I then installed confluent-hub by:
curl http://client.hub.confluent.io/confluent-hub-client-latest.tar.gz | tar xz
I then installed the kafka-GCS connector using:
confluent-hub install confluentinc/kafka-connect-gcs:5.0.1
I then export the path using:
export PATH=<insert-path-here>/confluent-5.0.0/bin:$PATH
I then start confluent using:
confluent start
it shows:
This CLI is intended for development only, not for production
https://docs.confluent.io/current/cli/index.html
Using CONFLUENT_CURRENT: /tmp/confluent.pc9RaNNQ
Starting zookeeper
zookeeper is [UP]
Starting kafka
kafka is [UP]
Starting schema-registry
schema-registry is [UP]
Starting kafka-rest
kafka-rest is [UP]
Starting connect
connect is [UP]
Starting ksql-server
ksql-server is [UP]
Starting control-center
control-center is [UP]
Checking confluent by:
confluent status
gives me:
This CLI is intended for development only, not for production
https://docs.confluent.io/current/cli/index.html
control-center is [UP]
ksql-server is [UP]
connect is [UP]
kafka-rest is [UP]
schema-registry is [UP]
kafka is [UP]
zookeeper is [UP]
All is fine up until now. Zookeeper and kafka are both up and running. Now when I use the following command to start the kafka-gcs confluent connector:
confluent load gcs -d quickstart-gcs.properties
it gives the following error:
Failed to find any class that implements Connector and which name
matches io.confluent.connect.gcs.GcsSinkConnector
The quickstart-gcs.properties file is in the home directory and looks something like this:
name=gcs-sink
connector.class=io.confluent.connect.gcs.GcsSinkConnector
tasks.max=1
topics=test
gcs.bucket.name=test_bucket
gcs.part.size=5242880
flush.size=3
gcs.credentials.path=/path/to/kafka/key.json
storage.class=io.confluent.connect.gcs.storage.GcsStorage
format.class=io.confluent.connect.gcs.format.avro.AvroFormat
partitioner.class=io.confluent.connect.storage.partitioner
.DefaultPartitioner
schema.compatibility=NONE
confluent.topic.bootstrap.servers=localhost:9092
confluent.topic.replication.factor=1
confluent.license=
The line:
partitioner.class=io.confluent.connect.storage.partitioner
.DefaultPartitioner
actually reads:
partitioner.class=io.confluent.connect.storage.partitioner.DefaultPartitioner
I had to write it like that because of the editor limitations. Sorry for that.
The last line in quickstart-gcs.properties as per the documentations actually reads:
confluent.license=""
but it did not work for me.
Any help here is greatly appreciated.
EDIT 1: As per @cricket_007 comment:
curl http://localhost:8083/connector-plugins
gives:
[{"class":"io.confluent.connect.elasticsearch.ElasticsearchSinkConnector","type":"sink","version":"5.0.0"},{"class":"io.confluent.connect.hdfs.HdfsSinkConnector","type":"sink","version":"5.0.0"},{"class":"io.confluent.connect.hdfs.tools.SchemaSourceConnector","type":"source","version":"2.0.0-cp1"},{"class":"io.confluent.connect.jdbc.JdbcSinkConnector","type":"sink","version":"5.0.0"},{"class":"io.confluent.connect.jdbc.JdbcSourceConnector","type":"source","version":"5.0.0"},{"class":"io.confluent.connect.s3.S3SinkConnector","type":"sink","version":"5.0.0"},{"class":"io.confluent.connect.storage.tools.SchemaSourceConnector","type":"source","version":"2.0.0-cp1"},{"class":"org.apache.kafka.connect.file.FileStreamSinkConnector","type":"sink","version":"2.0.0-cp1"},{"class":"org.apache.kafka.connect.file.FileStreamSourceConnector","type":"source","version":"2.0.0-cp1"}]
