0

i am trying ti run kafka connect on EC2 and send data from MSK to Elastic Search . This is what i am doing .

sudo yum install java-1.8.0
wget https://archive.apache.org/dist/kafka/2.2.1/kafka_2.12-2.2.1.tgz
tar -xzf kafka_2.12-2.2.1.tgz

Kafka connect Installation 

wget /usr/local http://packages.confluent.io/archive/5.2/confluent-5.2.0-2.11.tar.gz -P ~/Downloads/
tar -zxvf ~/Downloads/confluent-5.2.0-2.11.tar.gz -C ~/Downloads/
sudo mv ~/Downloads/confluent-5.2.0 /usr/local/confluent

I have modified two properties file

vim /usr/local/confluent/etc/kafka-connect-elasticsearch/quickstart-elasticsearch.properties

Here is i gave topic name as kafka topic that i have created and connection url for elastic search

second properties file 

vim /usr/local/confluent/etc/kafka/connect-standalone.properties

Here i modified just bootstrap server URL gave all 3 bootstrap url

Finally i am running my connector like this 

/usr/local/confluent/bin/connect-standalone /usr/local/confluent/etc/kafka/connect-standalone.properties /usr/local/confluent/etc/kafka-connect-elasticsearch/quickstart-elasticsearch.properties

And then i get below error 

[2019-12-30 20:35:38,109] INFO Kafka Connect standalone worker initialization took 3890ms (org.apache.kafka.connect.cli.ConnectStandalone:96)
[2019-12-30 20:35:38,109] INFO Kafka Connect starting (org.apache.kafka.connect.runtime.Connect:50)
[2019-12-30 20:35:38,109] INFO Herder starting (org.apache.kafka.connect.runtime.standalone.StandaloneHerder:87)
[2019-12-30 20:35:38,109] INFO Worker starting (org.apache.kafka.connect.runtime.Worker:162)
[2019-12-30 20:35:38,109] INFO Starting FileOffsetBackingStore with file /tmp/connect.offsets (org.apache.kafka.connect.storage.FileOffsetBackingStore:58)
[2019-12-30 20:35:38,111] INFO Worker started (org.apache.kafka.connect.runtime.Worker:167)
[2019-12-30 20:35:38,111] INFO Herder started (org.apache.kafka.connect.runtime.standalone.StandaloneHerder:89)
[2019-12-30 20:35:38,111] INFO Kafka Connect started (org.apache.kafka.connect.runtime.Connect:55)
[2019-12-30 20:35:38,113] ERROR Failed to create job for /usr/local/confluent/etc/kafka-connect-elasticsearch/quickstart-elasticsearch.properties (org.apache.kafka.connect.cli.ConnectStandalone:108)
[2019-12-30 20:35:38,113] ERROR Stopping after connector error (org.apache.kafka.connect.cli.ConnectStandalone:119)
java.util.concurrent.ExecutionException: org.apache.kafka.connect.errors.ConnectException: Failed to find any class that implements Connector and which name matches io.confluent.connect.elasticsearch.ElasticsearchSinkConnector, available connectors are: PluginDesc{klass=class org.apache.kafka.connect.file.FileStreamSinkConnector, name='org.apache.kafka.connect.file.FileStreamSinkConnector', version='2.2.0-cp1', encodedVersion=2.2.0-cp1, type=sink, typeName='sink', location='classpath'}, PluginDesc{klass=class org.apache.kafka.connect.file.FileStreamSourceConnector, name='org.apache.kafka.connect.file.FileStreamSourceConnector', version='2.2.0-cp1', encodedVersion=2.2.0-cp1, type=source, typeName='source', location='classpath'}, PluginDesc{klass=class org.apache.kafka.connect.tools.MockConnector, name='org.apache.kafka.connect.tools.MockConnector', version='2.2.0-cp1', encodedVersion=2.2.0-cp1, type=connector, typeName='connector', location='classpath'}, PluginDesc{klass=class org.apache.kafka.connect.tools.MockSinkConnector, name='org.apache.kafka.connect.tools.MockSinkConnector', version='2.2.0-cp1', encodedVersion=2.2.0-cp1, type=sink, typeName='sink', location='classpath'}, PluginDesc{klass=class org.apache.kafka.connect.tools.MockSourceConnector, name='org.apache.kafka.connect.tools.MockSourceConnector', version='2.2.0-cp1', encodedVersion=2.2.0-cp1, type=source, typeName='source', location='classpath'}, PluginDesc{klass=class org.apache.kafka.connect.tools.SchemaSourceConnector, name='org.apache.kafka.connect.tools.SchemaSourceConnector', version='2.2.0-cp1', encodedVersion=2.2.0-cp1, type=source, typeName='source', location='classpath'}, PluginDesc{klass=class org.apache.kafka.connect.tools.VerifiableSinkConnector, name='org.apache.kafka.connect.tools.VerifiableSinkConnector', version='2.2.0-cp1', encodedVersion=2.2.0-cp1, type=source, typeName='source', location='classpath'}, PluginDesc{klass=class org.apache.kafka.connect.tools.VerifiableSourceConnector, name='org.apache.kafka.connect.tools.VerifiableSourceConnector', version='2.2.0-cp1', encodedVersion=2.2.0-cp1, type=source, typeName='source', location='classpath'}

I am confused about plugin.path here . How do i set it ? Does it comes with kafka-connect or i have to set that ?

OneCricketeer
  • 179,855
  • 19
  • 132
  • 245
Sudarshan kumar
  • 1,503
  • 4
  • 36
  • 83

1 Answers1

1

I'll point out that Logstash also works for connecting Kafka to Elasticsearch

I am confused about plugin.path here . How do i set it ? Does it comes with kafka-connect

Look at the bottom of the connect-standalone.properties and read the comments there, then uncomment the plugin path property

https://github.com/apache/kafka/blob/trunk/config/connect-standalone.properties#L32-L41

As I answered here, it's best not to use tarballs; you've duplicated both Zookeeper and Kafka by downloading both Confluent Platform and Kafka separately.
Please use YUM to install Confluent Platform, which includes Zookeeper, Apache Kafka, and your Elasticsearch connector

Failed to find any class that implements Connector and which name matches io.confluent.connect.elasticsearch.ElasticsearchSinkConnector

OneCricketeer
  • 179,855
  • 19
  • 132
  • 245
  • Thank for clarifying .I have already un commented `plugin.path` but still same error .Next you are suggesting to use `YUM to install Confluent Platform` in that case i dont need my first step where i install java and kafka ? – Sudarshan kumar Dec 31 '19 at 04:51
  • 1
    And what value did you add to the plugin path? Did you put the full path to the Elasticsearch connector lib folder? Regarding YUM, You need Java. Confluent Platform already includes Kafka (therefore Kafka Connect). Alternatively, other tools like Nifi or Streamsets offer far more integrations than just Kafka and Elasticsearch and are both open source Apache Licensed products – OneCricketeer Dec 31 '19 at 06:09
  • So i have downloaded kafka -elastic search connector here `wget https://github.com/confluentinc/kafka-connect-elasticsearch/releases/tag/v5.2.3-cp1-rc1/kafka-connect-elasticsearch-5.2.3-cp1-rc1.tar.gz` .Do i have to build it to get lib ,or i can use config directly ? – Sudarshan kumar Dec 31 '19 at 07:57
  • when i build that i get error like `[ERROR] [ERROR] Some problems were encountered while processing the POMs:` – Sudarshan kumar Dec 31 '19 at 08:14
  • Just one more question ..Will this connector comes with free license ?I am using MSK and do not intend to use any licensed connector to move data from MSK to aws elastic search. – Sudarshan kumar Dec 31 '19 at 09:00
  • You downloaded source code, not a release. You do not need Maven; again, using YUM, you can install Elasticsearch connector as well. Otherwise, you simply needed `plugin.path=/usr/local/confluent/share/java`. Elasticsearch connector is [Confluent Licensed](https://github.com/confluentinc/kafka-connect-elasticsearch/blob/master/LICENSE) which states that you cannot offer any of Confluent's code as a competitive service. – OneCricketeer Dec 31 '19 at 13:03
  • You can also use the Confluent Hub CLI instead with any Kafka Connect installation https://docs.confluent.io/current/connect/managing/confluent-hub/client.html – OneCricketeer Dec 31 '19 at 13:23
  • I did that but still i get some error like `ERROR WorkerSinkTask{id=elasticsearch-sink-0} Task threw an uncaught and unrecoverable exception (org.apache.kafka.connect.runtime.WorkerTask:177) org.apache.kafka.connect.errors.ConnectException: Tolerance exceeded in error handler at org.apache.kafka.connect.runtime.errors.RetryWithToleranceOperator.execAndHandleError(RetryWithToleranceOperator.java:178)` – Sudarshan kumar Dec 31 '19 at 19:09
  • Also i have seen few json messages and then i started getting this error `Unable to start REST server at org.apache.kafka.connect.runtime.rest.RestServer.start(RestServer.java:216) at org.apache.kafka.connect.cli.ConnectStandalone.main(ConnectStandalone.java:87) Caused by: java.io.IOException: Failed to bind to 0.0.0.0/0.0.0.0:8083` and `Caused by: java.net.BindException: Address already in use at sun.nio.ch.Net.bind0(Native Method)` – Sudarshan kumar Dec 31 '19 at 19:36