1

Probably some confusion from my part on how ksqlDB works. I am trying to understand how ksqlDB recreates a table in the event of a failure or repartition with the underlying streams tasks. The documentation suggests that this is from a changelog topic but I cannot see a changelog created when I create a table (unless I do create table ... as select).

If I create a table and the topic through ksqlDB:

CREATE TABLE TEST_TABLE_CREATION (
    ID VARCHAR PRIMARY KEY,
    COL1 VARCHAR,
    COL2 BOOLEAN,
    COL3 INT
) WITH (
    KAFKA_TOPIC = 'test_table_creation_tab',
    VALUE_FORMAT = 'JSON',
    PARTITIONS = 2
);

Then I can see it creates a topic "test_table_creation_tab" and sets cleanup.policy to compact. Thus, it could recreate the table from the topic if needed. Still no change_log topic though. Should I be expecting one?

Secondly, if I create the topic first (and set cleanup.policy to delete) and then create the table from it then it does not change the topic to have log compaction. Therefore, how would it recreate the state of the table in case of failure?

Hoping someone can help.

Trying all this through Confluent Cloud.

Thanks

user2294382
  • 871
  • 3
  • 11
  • 25

1 Answers1

0

What you see with CREATE TABLE TEST_TABLE_CREATION... is it isn't materialized yet - not until you do something involving the table downstream will it be materialized then creating a changelog at that point. Doing a CREATE TABLE AS SELECT... materializes the table immediately, so you see the changelog immediately.

If I create the topic first (and set cleanup.policy to delete) and then create the table from it then it does not change the topic to have log compaction.

Recovering from a topic with a cleanup.policy=delete would still work as expected, it would just not be as efficient since it would have multiple values for a given key, so it will replay every value vs. having a single value per key in a compacted topic.

bbejeck
  • 1,310
  • 8
  • 7