0

I have a Spring Batch that partitions into "Slave Steps" and run in a thread pool, here is the configuration: Spring Batch - FlatFileItemWriter Error 14416: Stream is already closed

I'd like to run this Spring Batch Job in Kubernetes. I checked this post: https://spring.io/blog/2021/01/27/spring-batch-on-kubernetes-efficient-batch-processing-at-scale by @MAHMOUD BEN HASSINE.

From the post, on Paragraph:

  1. Choosing the Right Kubernetes Job Concurrency Policy As I pointed out earlier, Spring Batch prevents concurrent job executions of the same job instance. So, if you follow the “Kubernetes job per Spring Batch job instance” deployment pattern, setting the job’s spec.parallelism to a value higher than 1 does not make sense, as this starts two pods in parallel and one of them will certainly fail with a JobExecutionAlreadyRunningException. However, setting a spec.parallelism to a value higher than 1 makes perfect sense for a partitioned job. In this case, partitions can be executed in parallel pods. Correctly choosing the concurrency policy is tightly related to which job pattern is chosen (As explained in point 3).

Looking into my Batch Job, if I start 2 or more pods, it sounds like one/more pods will fail because it will try to start the same job. But on the other hand, it sounds like more pods will run in parallel because I am using partitioned job.

My Spring Batch seems to be a similar to https://kubernetes.io/docs/tasks/job/fine-parallel-processing-work-queue/

This said, what is the right approach to it? How many pods should I set on my deployment? Do the partition/threads will run on separate/different pods, or the threads will run in just one pod? Where do I define that, in the parallelism? And the parallelism, should it be the same as the number of threads?

Thank you! Markus.

msuzuki
  • 105
  • 2
  • 15

1 Answers1

1

A thread runs in a JVM which runs inside container that in turn is run in a Pod. So it does not make sense to talk about having different threads running on different Pods.

The partitioning technique in Spring Batch can be either local (multiple threads within the same JVM where each thread processes a different partition) or remote (multiple JVMs processing different partitions). Local partitioning requires a single JVM, hence you only need one Pod for that. Remote partitioning requires multiple JVMs, so you need multiple Pods.

I have a Spring Batch that partitions into "Slave Steps" and run in a thread pool

Since you implemented local partitioning with a pool of worker threads, you only need one Pod to run your partitioned Job.

Mahmoud Ben Hassine
  • 28,519
  • 3
  • 32
  • 50
  • in a scenario where instead of running multiple threads, run the several jobs in a different pods, I would use remote Partitioning (something like this: https://dataflow.spring.io/docs/feature-guides/batch/partitioning/) where the grid_size would be the concurrency number, and sequentially the number of pods/hubernets Job parallellism? – msuzuki Jun 17 '21 at 09:47
  • ok in a remote partitioning setup, you would have a Job template for the manager (with parallelism of 1) and a Job template for workers (with parallelism of 1 or more, which could be equal to the grid-size, but this is not necessary as you can have more workers than partitions or vice versa). The idea to have a Job per partition is that if a partition fails (ie the first Pod created by kubernetes fails), kubernetes will automatically create another Pod for that partition. So you get automatic restart of failed partitions, until all of them are successfully executed, without manual intervention – Mahmoud Ben Hassine Jun 17 '21 at 09:58
  • then in this case, I would need a queue where the Manager puts the jobs, and the workers pulls from the queue. Correct? – msuzuki Jun 17 '21 at 10:18
  • yes, remote partitioning requires a queue, please check the docs here: https://docs.spring.io/spring-batch/docs/4.3.x/reference/html/spring-batch-integration.html#remote-partitioning. – Mahmoud Ben Hassine Jun 17 '21 at 13:22