0

I currently have a Kafka Streams application that has a GlobalKTable materialized as a state store and persisted locally on disk. When I deploy this application to OpenShift/Kubernetes it only has access those resources within that specific pod.

What's the best way to create PVCs and then update the deployment or deployment configs dynamically to reference those stored rocksdb logs? As it is possible for those individual pod instances to go down or to have a new build triggered through a CI/CD pipeline I don't want to edit the individual pod yamls. By default I believe the state stores are persisted locally in /tmp/kafka-streams/<application-name>.

animusdx
  • 380
  • 3
  • 16
  • To me it's not clear what you mean, please give us more details. What is your architecture on OpenShift? How many pods, brokers and topic partitions do you have? Are you using Strimzi? Why would you want pods to go down? This sounds so wrong. What do you mean with that broken sentence: "it only has access those resources within that specific pod"? – Tony Stark Nov 25 '20 at 15:04
  • Yes sorry, I'm currently using the Strimzi operator. The Kafka architecture isn't really what I'm trying to configure here. I'm trying to configure an instance of my Kafka Streams application that connects to the Strimzi Kafka clusters on OpenShift. Of course I don't want the pods to go down but I meant in the case of a new triggered Ci/CD pipeline build that pod would go down and a new pod would replace it with the new ImageStream and what not. And sorry by that sentence I meant something more like the local storage of the state store is currently limited by the memory on that pod since no PV – animusdx Nov 25 '20 at 15:30
  • I guess your problem is that your data vanishes from storage after pod restarts? You'll need persistent storage. However what is available for persistent storage depends on OpenShift's architecture in your company. If for example OpenShift runs on OpenStack, then you could use Cinder storage. Just do not use NFS. – Tony Stark Nov 25 '20 at 16:04

0 Answers0