7

I have the following json on a topic that the JDBC connector publishes to

{"APP_SETTING_ID":9,"APP_SETTING_NAME":"my_name","SETTING_KEY":"my_setting_key"}

Here's my connector file

name=data.app_setting
connector.class=io.confluent.connect.jdbc.JdbcSourceConnector
poll.interval.ms=500
tasks.max=4
mode=timestamp
query=SELECT APP_SETTING_ID, APP_SETTING_NAME, SETTING_KEY,FROM MY_TABLE with (nolock)
timestamp.column.name=LAST_MOD_DATE
topic.prefix=data.app_setting

key.converter=org.apache.kafka.connect.json.JsonConverter
key.converter.schemas.enable=false
value.converter=org.apache.kafka.connect.json.JsonConverter
value.converter.schemas.enable=false

transforms=InsertKey
transforms.InsertKey.type=org.apache.kafka.connect.transforms.ValueToKey
transforms.InsertKey.fields=APP_SETTING_ID

This does add a key but its also a json format like 

{"APP_SETTING_ID":9}

While I just want 9 to be the key instead of the map. In the database it's stored as a Long value.

Fizi
  • 1,749
  • 4
  • 29
  • 55

1 Answers1

6

ValueToKey takes a list of fields within the value, and returns a mapping of those fields to their values.

I think you need a second transform to extract only a single one of those fields. 

transforms=ReplaceKey,ExtractKey

# Replaces the key with fields in the value. Creates a map for all listed fields
transforms.ReplaceKey.type=org.apache.kafka.connect.transforms.ValueToKey
transforms.ReplaceKey.fields=APP_SETTING_ID

# Extracts a specfic field from the key, assuming it's a map/struct type
transforms.ExtractKey.type=org.apache.kafka.connect.transforms.ExtractField$Key
transforms.ExtractKey.field=APP_SETTING_ID
OneCricketeer
  • 179,855
  • 19
  • 132
  • 245
  • if possible, could you please also explain what's happening here. Specifically in the ExtractField$Key part – Fizi Oct 11 '18 at 14:48
  • 1
    The dollar sign is just Java class syntax for accessing an inner class – OneCricketeer Oct 11 '18 at 14:49
  • So are you extractin the key? By intuition it should be the value 9 no? Also, what should the key.converter be in this case. Currently its org.apache.kafka.connect.json.JsonConverter even though its not JSON now. It still works.Just not sure what the converter is doing – Fizi Oct 11 '18 at 14:52
  • 1
    Yes, it gets 9 for your example. If your data isn't JSON, then the converter would fail in serialization. A plain string or integer is still valid JSON, though – OneCricketeer Oct 11 '18 at 14:54
  • One last question :) - I tried org.apache.kafka.connect.converters.LongConverter but my kafka doesnt recognize it. I am using kafka 1.0 and Confluent 3.0 Is it something that needs to be installed or does it come with a different version number – Fizi Oct 11 '18 at 15:04
  • That is only available starting with Kafka 2.0 https://issues.apache.org/jira/browse/KAFKA-6913 . Other than that, It would be recommended to use Confluent 4.x, which includes Kafka 1.x libraries – OneCricketeer Oct 11 '18 at 18:43