6

I'm not sure if it's already answered. As I didn't get proper explanation, posting my question here.

Why kafka streams state.dir is stored under /tmp/kafka-streams?

I know I can change the path by providing the state dir config in the stream code like below

StreamsConfig.STATE_DIR_CONFIG,"/var/abc-Streams"

But will there be any impact of changing the directory?

or

Can I configure the state DB in an application directory and not in /tmp.

As per the confluent documentation, for : Stateful operations :

automatically creates and manages such state stores when you are calling stateful operators such as count() or aggregate(), or when you are windowing a stream

but didn't specify where exactly it's being stored.

ANy thoughts?

OneCricketeer
  • 179,855
  • 19
  • 132
  • 245
kuti
  • 161
  • 1
  • 3
  • 13
  • @cricket_007: Any thoughts on the consequences of changing the state store directory? In my case, just today I changed it and weirdly all messages flew downstream! E.g. Given from topic A -> aggregate(..) -> to topic B, all events were again produced to topic B, but not re-consumed from A. – Vassilis Apr 29 '19 at 20:16

1 Answers1

9

Why kafka streams state.dir is stored under /tmp/kafka-streams?

There are several reasons.

  1. Usually /tmp directory has a default write permission. So you don't have to struggle with write permissions as a beginner.
  2. /tmp directory is short lived directory. On each system reboot, it is cleaned, hence you don't experience the over flooded disk storage in case you forgot to delete the state.dir. Downside is, you lose the states from previous run hence you need to rebuild the states from scratch.

If you want to reuse the states stored in state.dir, you should store it somewhere except /tmp.

All the state-stores are stored in the location specified in state.dir. If not specified, it is /tmp/kafka-streams/<app-id> directory.

Nishu Tayal
  • 20,106
  • 8
  • 49
  • 101