4

I have a situation where i'm trying to create a Kubernetes CronJob which does some data processing. While the processing is being performed, I need to use external storage to temporarily store the data (as it's too big for nodes themselves), so i create a PVC using a StorageClass that i have on the cluster. This in turn dynamically provisions an EBS volume on AWS, and PV on my cluster, my Job runs, uses the PVC/PV/EBS Volume and everything looks good.

The problem is that I don't want to keep this EBS Volume after the job is finished, but I need a new one for the next execution (Let's say that this CronJob is executed once a month and lasts for about 30 minutes). If I delete the PVC itself after job is finished, the PV/EBS Volume will be deleted as expected, but then the next CronJob will fail as the volume doesn't exist anymore.

One possible solution that I see is to have a CronJob run a Pod which:

  • dynamically creates a PVC to be used for this month's execution
  • dynamically creates a separate job which uses this PVC to run my original data processing
  • waits for the dynamic job to be over and deletes the PVC

While i figure the above would solve my issue, I'm interested if there could be a more elegant solution to this problem that I'm missing?

hpdobrica
  • 134
  • 7

1 Answers1

0

You can consider using EFS; where you can move away from create/delete/re-create cycle, each pod can have its own dynamic access point. And other benefit like pre-populate static data on EFS for re-use, multi-AZ.

gohm'c
  • 13,492
  • 1
  • 9
  • 16