Limit the total number of tasks running at same time in spring cloud data flow?

Question

Every time we launch a spring cloud task, it starts a new jvm (java.exe), so if 25 tasks are launched , then it will start 25 jvm.

I was wondering to see How to limit the total number of all tasks (running for all deployed jars) at the same time ?

Let’s say if I have to limit the total number of all tasks running at a time to 25. Is there any setting in SCDF we can do this ?

Please let me know

We have [spring-cloud-task#81](https://github.com/spring-cloud/spring-cloud-task/issues/81) in PR to prevent the simultaneous launch of the same task. There was some discussion about limiting the number of task launches regardless of whether it is the same task or not. In any case, more details about what and how you're orchestrating the tasks would be useful. — Sabby Anandan, Dec 11 '17 at 23:12
We are creating multiple tasks in SCDF (Ex : Task 1, Task 2, Task 3) and each of these tasks do different functions. So we have three jars each for the above tasks. The users can launch these tasks from UI , so eventually, we might end with numerous concurrent tasks running at the same time which might up with resource and memory issues. So we want to limit the total number of concurrent tasks running at same time. I have seen lot of discussion around this , it will be great if configurable is added field in the dataflow using which we cans set this. Please let me know — Raj_Twister, Dec 13 '17 at 17:30
Also is there anyway to queue the task request so that tasks are run when the total number of concurrent tasks running becomes less than the max number we set — Raj_Twister, Dec 13 '17 at 22:41

score 1 · Answer 1 · answered Feb 13 '20 at 10:09

1

In the newer version of SCDF there is a deployer property that limits the number of concurrent task executions.

deployer.<appName>.kubernetes.maximumConcurrentTasks

By default, the value is 20.

answered Feb 13 '20 at 10:09

Siddhant Sorann

313
3
15

pvrforpranavvr · Answer 2 · 2021-04-13T12:31:21.850

By default maximumTaskExecutions is 20. You can check the runningExecutionCount and maximumTaskExecutionsvalue using below url

http://localhost:9393/tasks/executions/current

Local Machine

If you are running spring-cloud dataflow server in local machine using jar file then you need to pass an argument like below.

java -jar spring-cloud-dataflow-server-<VERSION>.jar --spring.cloud.dataflow.task.platform.<PLATFORM-TYPE>.accounts.<ACCOUNT-NAME>.maximum-concurrent-tasks=<TASK-COUNT>

eg:

java -jar spring-cloud-dataflow-server-2.7.1.jar --spring.cloud.dataflow.task.platform.local.accounts.default.maximum-concurrent-tasks=1

KUBERNETES

If you are running spring cloud server in kubernetes platform then you need to change the configuration of Kubernetes task platforms

Go to src/kubernetes/server/server-config.yaml , add the property in exact place

spring:
  cloud:
    dataflow:
      task:
        platform:
          <PLATFORM-TYPE>:
            accounts:
              <ACCOUNT-NAME>:
                 maximum-concurrent-tasks: <TASK-COUNT>

eg:

spring:
  cloud:
    dataflow:
      task:
        platform:
          kubernetes:
            accounts:
              dev:
                maximum-concurrent-tasks: 10
              qa:
                maximum-concurrent-tasks: 30

Then apply the config kubectl apply -f src/kubernetes/server/server-config.yaml
Then restart spring-cloud-dataflow server.

Note:

The PLATFORM-TYPE refers to one of the currently supported deployers: local, cloudfoundry, or kubernetes
ACCOUNT-NAME is the name of a configured platform account (default if no accounts are explicitly configured).

Reference

Limit the total number of tasks running at same time in spring cloud data flow?

2 Answers2