0

We've setup our postgres source database so that we have all tables that we are interested in, in two schemas. Essentially we have schema1 and schema2, and the tables in these schemas are identical. When we make a deployment/migration we get rid of schema1 and create schema3, always having two active schemas, one of them being a 'backup' in case we need to revert.

This is typically fine because we've defined a search path so that when running unqualified queries (without schema name, just select * from table_name) it always picks the correct schema. For all (most?) clients this works, as we just need to write queries without the schema and everything is set.

For the JDBC source connector this isn't the case though, and it cannot handle such a case, where a single whitelisted table lives in two schemas. I've looked at various workarounds but not found anything that works, I also don't have much control over the postgres database so making drastic changes to how we operate there isn't feasible.

One workaround which I identified is just using the jdbc connector in the query mode, but using this implementation we lose much of the convenience that the connector gives. It also makes the implementation unnecessarily complex.

Other than that I can't find a way to make this connector work with our setup.

Anton
  • 581
  • 1
  • 5
  • 23
  • How do you specify the search path when connecting through JDBC? –  Mar 01 '23 at 13:31
  • Would Debezium be viable workaround? – OneCricketeer Mar 01 '23 at 15:00
  • @a_horse_with_no_name The search path is defined on the user in the postgres database, not in the JDBC connector. – Anton Mar 01 '23 at 18:31
  • @OneCricketeer Unfortunately not, since I'm working with views – Anton Mar 01 '23 at 18:31
  • Well, if the search path is set on user level, then Kafka must override it explicitly. Check if that connector has a specific property to disable this behaviour –  Mar 01 '23 at 18:52
  • @a_horse_with_no_name I went through all configs and wasn't able to identify one, unfortunately. – Anton Mar 02 '23 at 09:54
  • I've also looked into limiting the access of the JDBC connector to just a single schema, but even this doesn't work as its accessing tables and schemas through the system catalog – Anton Mar 02 '23 at 09:57
  • Essentially, the only solution I'm seeing atm is having the backup schema in another database – Anton Mar 02 '23 at 10:10

0 Answers0