1

I try to create plugin with "transform" for my data to kafka-connect and use it with different sink connectors. When I install plugin, kafka-connect doesn't see my classes.

I used kafka-connect maven plugin to create my bundle zip. Installation with confluent-hub (from local file) has succeed.

All file are unzip, my worker properties has updated plugin.paths. I run my connect in distributed mode and try to create new connector with transformer from my package.

My plugin structure looks like:

- mwojdowski-my-connect-plugin-0.0.1-SNAPSHOT
|- manifest.json
|- lib
||- my-connect-plugin-0.0.1-SNAPSHOT.jar

and my manifest.json file:

{
  "name" : "my-connect-plugin",
  "version" : "0.0.1-SNAPSHOT",
  "title" : "my-connect-plugin",
  "description" : "A set of transformations for Kafka Connect",
  "owner" : {
    "username" : "mwojdowski",
    "name" : "Marcin Wojdowski<mwojdowski@gmail.com>"
  },
  "tags" : [ "transform", "field", "topic" ],
  "features" : {
    "supported_encodings" : [ "any" ],
    "single_message_transforms" : true,
    "confluent_control_center_integration" : true,
    "kafka_connect_api" : true
  },
  "documentation_url" : "",
  "docker_image" : { },
  "license" : [ {
    "name" : "Confluent Software License",
    "url" : "https://www.confluent.io/software-evaluation-license"
  } ],
  "component_types" : [ "transform" ],
  "release_date" : "2019-08-29"
}

Next, I try to create new connector:

curl -XPOST -H 'Content-type:application/json' 'localhost:8083/connectors' -d '{
    "name" : "custom-file-sink-with-validation",
    "config" : {
    "connector.class" : "FileStreamSink",
        "tasks.max" : "1",
        "topics" : "test_topic",
        "file" : "/tmp/my-plugin-test.txt",
        "key.ignore" : "true",
        "schema.ignore" : "true",
        "drop.invalid.message": "false",
        "behavior.on.malformed.documents": "warn",
        "key.converter":"org.apache.kafka.connect.storage.StringConverter",
        "value.converter":"org.apache.kafka.connect.storage.StringConverter",
        "transforms" : "Validation",
        "transforms.Validation.type" : "org.kafka.connect.my.connector.ValidateId"
    }
}'

After restart of kafka connect, when I try to create new connector, exception is thrown:

{
    "error_code": 400,
    "message": "Connector configuration is invalid and contains the following 2 error(s):\nInvalid value org.kafka.connect.my.connector.ValidateId for configuration transforms.Validation.type: Class org.kafka.connect.my.connector.ValidateId could not be found.\nInvalid value null for configuration transforms.Validation.type: Not a Transformation\nYou can also find the above list of errors at the endpoint `/{connectorType}/config/validate`"
}

I try to install plugin also manually, following doc: https://docs.confluent.io/current/connect/managing/install.html

But it looks, like Connect doesn't load my jars.

When I copy my jar to share/java/kafka it works, but this is not a solution.

I suspect my plugin is skip, because it not contains connectors. In this case should I add my jars to classpath manually? (Opposite to https://docs.confluent.io/current/connect/userguide.html#installing-plugins)

Or should I explicitly point in my connector configuration to try use my plugin?

Regards, M.

Marcin Wojdowski
  • 21
  • 2
  • Where's your plugin_path set to? e.g.: within connect-distributed.properties plugin.path=/ – bob dylan Aug 29 '19 at 09:54
  • I install it into: /usr/share/kafka-connect/plugins. Same path is set in plugin.path. – Marcin Wojdowski Aug 29 '19 at 10:03
  • Also I tried to install in suggested by confluent-hub: <>/share/java – Marcin Wojdowski Aug 29 '19 at 10:06
  • I forget about version. I'm using Confluent 5.1.0. – Marcin Wojdowski Aug 29 '19 at 10:07
  • sorry i missed question: It's set in connect-distributed.properties. – Marcin Wojdowski Aug 29 '19 at 10:15
  • @MarcinWojdowski, Plugin can contain only Transform or even Converter. Could you add properties files?. You should start with checking log file. Kafka Connect print in logs plugins, that are loaded: It is something like that: [2019-08-29 13:16:49,772] INFO Loading plugin from: /Users/bwardzinski/programs/tests/plugins/rest-kafka-connector-assembly-0.1.jar (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader:220) [2019-08-29 13:16:51,071] INFO Registered loader: PluginClassLoader{pluginLocation=file:/Users/bwardzinski/programs/tests/plugins/rest-kafka-connector-assembly-0.1.jar} – Bartosz Wardziński Aug 29 '19 at 11:20
  • Hi @wardziniak, I've checked and plugin is loaded at the beginning. ``` ... INFO Loading plugin from: /user/mwojdowski/connect-plugins/mwojdowski-my-connect-plugin-0.0.1-SNAPSHOT (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader:220) INFO Registered loader: PluginClassLoader{pluginLocation=file:/user/mwojdowski/connect-plugins/mwojdowski-my-connect-plugin-0.0.1-SNAPSHOT/} (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader:243) INFO Added plugin 'org.kafka.connect.my.connector.ValidateId' (org.apache.kafka.connect.runtime.isolation.DelegatingClas ... ``` – Marcin Wojdowski Sep 03 '19 at 14:14
  • @MarcinWojdowski, is it only one node cluster of Kafka Connect? – Bartosz Wardziński Sep 03 '19 at 16:49

1 Answers1

1

Sorry, problem was really trivial. During refactoring one of the packages get "'s" at the end and I miss to update it in config.

"transforms.Validation.type" : "org.kafka.connect.my.connectors.ValidateId"

instead of 

"transforms.Validation.type" : "org.kafka.connect.my.connector.ValidateId"

I refactor it moment before switch from standalone to distributed. One more time sorry for worrying You and thank You for your support.

Regards, Marcin

Marcin Wojdowski
  • 21
  • 2