2

I am using "confluentinc/kafka-connect-jdbc:10.2.6" as my JDBC connector to transfer Kafka topics into my questDB.

When I provide explicit topic names it is working as expected. But when I use topic names based on regex then it's not working, the tables are not being created in my questDB database.

What am I missing in my JDBC settings?

Thanks!

explicit version (working):

{
  "name": "jdbc_sink_ftx",
  "config": {
    "connector.class": "io.confluent.connect.jdbc.JdbcSinkConnector",
    "topics": "trades-FTX-ETH-USD-PERP, trades-FTX-FTM-USD-PERP",
    "table.name.format": "${topic}",
    "connection.url": "jdbc:postgresql://questdb:8812/qdb?useSSL=false",
    "connection.user": "admin",
    "connection.password": "quest",
    "auto.create": "true",
    "insert.mode": "insert",
    "dialect.name": "PostgreSqlDatabaseDialect"
  }
}

regex-version (not working):

{
  "name": "jdbc_sink_ftx_regex",
  "config": {
    "connector.class": "io.confluent.connect.jdbc.JdbcSinkConnector",
    "topics.regex": "FTX",
    "table.name.format": "${topic}",
    "connection.url": "jdbc:postgresql://questdb:8812/qdb?useSSL=false",
    "connection.user": "admin",
    "connection.password": "quest",
    "auto.create": "true",
    "insert.mode": "insert",
    "dialect.name": "PostgreSqlDatabaseDialect"
  }
}
CarloP
  • 99
  • 1
  • 12

2 Answers2

0

It would be best to check QuestDB logs. Grepping for ' E ' should show errors that usually include description and OS error code in this case. Topic names in Kafka Connector are used as table names. QuestDB uses table names as directories on disk and validates them for security purposes. '.', '/' and '' are not allowed characters in table name. Specific OS can also apply additional restrictions on file names.

It would be best to check QuestDB logs though.

Vlad Ilyushchenko
  • 244
  • 2
  • 4
  • I have checked the questDB logs. When I start up the working JDBC connector with explicit "topics" then the logs contain the corresponding outputs (like "ReaderPool open ..." etc). But when I start the JDBC connector with the "topics.regex" then the questDB logs don't show any activity, as if nothing arrived at questDB at all. I know that Kafka topic names are used as table names in questDB, and that's not a problem when I use explicit "topics" in the JDBC connector. And I don't use those forbidden characters that you mentioned in the topic names. – CarloP Jan 12 '22 at 20:56
0

The regex needs to match with the complete topic names like so:

"topics.regex": "trades-FTX-.*"
CarloP
  • 99
  • 1
  • 12