2

Using KSQL (Confluent: Version: 5.0.1) I'm able to drop a table / stream normally (using DROP [TABLE|STREAM]) <NAME> when the linked topic exist and when it is registered (Registered=true).

However, if the topic is dropped first (Registered=false) then the associated stream or table can't be dropped with KSQL pointing out that "No topic with name Foo was registered".

The problem is that the stream / table still shows in the list and no new stream / table can be added using the same name.

Is there any way to delete them after the topic was dropped?

Brian Tompsett - 汤莱恩
  • 5,753
  • 72
  • 57
  • 129
Matthias
  • 178
  • 2
  • 6

4 Answers4

6

From the official documentation, I would do it like this:

SHOW QUERIES;

Write down the QueryID;

TERMINATE <QueryID>;

DROP TABLE <tablename>;

[See documentation...]

ordago
  • 377
  • 3
  • 20
Uwe Ranft
  • 91
  • 1
  • 4
  • You can run the DESCRIBE YOUR_STREAM_NAME command to find query id of specific STREAM. Query id exist at sourceDescription->writeQueries->id in the query result JSON. – vlyalcin Mar 09 '22 at 12:18
1

I think I found an answer, or at least in the right direction.

You need to re-register the topic with the stream. After a bit of hacking around, I managed to reattach my topics to streams, the drop the streams.

REGISTER TOPIC <ksql_topic_name> WITH (KAFKA_TOPIC='<kafka_topic_name>', VALUE_FORMAT='<format>');

Then,

DROP STREAM <stream_name>;

I don't know if this works in all cases, but it got me back to where I want to be. If anyone else can iterate on this and find a more robust solution I would like to hear about it.

simchuck
  • 71
  • 1
  • 6
0

I've been wrestling with these type issues for a while. Here's where I'm at, from my notes:

Having trouble deleting streams/tables/topics? Delete them in zookeeper:

  • Shut down any producers or consumers associated with topics you'll be deleting
  • Attempt to drop your tables and streams from KSQL CLI
  • Shut down KSQL server
  • Delete topics with 'kafka-topic' CLI

  • Delete in zookeeper like this:

    zookeeper-shell localhost:2181

    ls /brokers/topics

    _confluent-controlcenter-5-0-0-1-TriggerEventsStore-changelog, _confluent-controlcenter-5-0-0-1-error-topic, ...

    rmr /brokers/topics/yourTopicName

Now do a " kafka-topics --list --zookeeper localhost:2181' and they should be gone for good.

OR, for the ABSOLUTE NUCLEAR OPTION:

  • Attempt to drop your tables and streams from KSQL CLI
  • Shut down KSQL server
  • Shut down your brokers and zookeepers
  • Delete eveything in the kafka logs dir on each broker node (location of dir is defined in your properties file).
  • Start up your zookeepers again and delete the /brokers/topics from zookeeper CLI

Start everything back up and it should all be gone. But be careful, you'll lose everything, all your data.

medloh
  • 941
  • 12
  • 33
  • Topics can be deleted using kafka-topics CLI - Thats not the problem. It's the tables and the streams that are still registered and that can't be deleted once a topic has been dropped (Registered = false). NUCLEAR OPTION - There must be a better way to remove some dead streams / tables – Matthias Dec 12 '18 at 20:15
  • So when you delete the backing topic from zookeeper they still exist as streams and tables in KSQL for you? – medloh Dec 13 '18 at 16:38
  • Thanks medloh for your comments. - when I delete topics with 'kafka-topic' CLI then the topic is removed from kafka and from KSQL - when I do this via zookeeper then the topic is removed from kafka but not from KSQL. - associated streams and tables in KSQL still exist – Matthias Dec 14 '18 at 18:38
  • I feel your pain--topic, streams, tables, they all seem to hard to get rid of permanently. Even with the nuclear option I describe above they sometimes come back later. So I'm guessing there is a KSQL metadata store somewhere that persists that info. Let me know if you make any progress on this, hopefully the next version will behave better. – medloh Dec 14 '18 at 21:01
0

You would have to terminate any running queries first.

SHOW QUERIES;  --> that will show you the active queries (COPY QUERY ID)

Then:

TERMINATE [QUERY ID];

Then: DROP [TABLE OR STREAM];

Moutasem Akkad
  • 11
  • 2