Set SCDF CTR to execute next task immediately when step is complete

Question

I'm using SCDF 2.9.6, which is mariadb 10.3.x(used in my company, cannot update version) compatible with flyway. Compiling SCDF 2.10.x project with lower flyway dependency resulted whole compile process failure... I guess this is the highest version I could use.

Now question arises, when I have to set Composed Task Runner property that 2.9.x version does not support - setting app.composed-task-runner.interval-time-between-checks property in 2.9.x seems not working, so I have to find walkaround. I know this Runner is Spring Batch project, so added Spring Batch tag.

There is a simple Composed Task asdf1, which runs a,b,c,d,e task consequently.

Each of child tasks are not that heavy - they complete their job in 5 seconds. However, CTR executes next task(step) after 10 seconds.

It seems, in some property file, there is a "minimum runtime" for each of task(step) in Composed Task Runner.

Is there any way to set this minimum runtime to 0, so the task(step) is always executed when the predecessor finishes, in lower SCDF version?

Thank for any Information provided for this question.

About the Flyway/Maria version, the version of Flyway required by Dataflow (Spring Boot) has a minimum supported version of MariaDB 10.4 and you will likely be unable to get it working w/ lower versions. You can disable the flyway w/ [spring.flyway.enabled=false](https://docs.spring.io/spring-cloud-dataflow/docs/2.10.0/reference/htmlsingle/#configuration-local-rdbms-schema). — onobc, Jan 20 '23 at 14:06

score 1 · Accepted Answer · answered Jan 20 '23 at 12:25

The property intervalTimeBetweenChecks is present in 2.9.x as well.

The documentation show how to set properties for Composed Task Runner.

app.composed-task-runner.interval-time-between-checks=1000 will set it to 1 second.

I did a quick test and use kail -ns=default | tee pods.log to combine all the logs. I use 5s and the logs show that there was 7s between the last log entry from a and the first log entry from b. This will include the time for launching the container and the application being active.

default/ctr-pg4n5m85q4[ctr-wrvzxj6x05]: 2023-01-20 12:17:31.755  INFO 1 --- [           main] o.s.b.c.l.support.SimpleJobLauncher      : Job: [FlowJob: [name=ctr]] launched with the following parameters: [{run.id=3}]
default/ctr-pg4n5m85q4[ctr-wrvzxj6x05]: 2023-01-20 12:17:31.762  INFO 1 --- [           main] o.s.c.t.b.l.TaskBatchExecutionListener   : The job execution id 11 was run within the task execution 22
default/ctr-pg4n5m85q4[ctr-wrvzxj6x05]: 2023-01-20 12:17:31.773  INFO 1 --- [           main] o.s.batch.core.job.SimpleStepHandler     : Executing step: [ctr-a_0]
default/ctr-a-q30xnqr0r4[ctr-a-mgy3nlpym5]: 2023-01-20 12:17:34.893  INFO 1 --- [           main] io.spring.timestamp.TaskApplication      : Starting TaskApplication v2.0.2 using Java 1.8.0_352 on ctr-a-q30xnqr0r4 with PID 1 (/workspace/BOOT-INF/classes started by cnb in /workspace)
default/ctr-a-q30xnqr0r4[ctr-a-mgy3nlpym5]: 2023-01-20 12:17:34.895  INFO 1 --- [           main] io.spring.timestamp.TaskApplication      : No active profile set, falling back to 1 default profile: "default"
default/ctr-a-q30xnqr0r4[ctr-a-mgy3nlpym5]: 2023-01-20 12:17:35.363  INFO 1 --- [           main] com.zaxxer.hikari.HikariDataSource       : HikariPool-1 - Starting...
default/ctr-a-q30xnqr0r4[ctr-a-mgy3nlpym5]: 2023-01-20 12:17:35.533  INFO 1 --- [           main] com.zaxxer.hikari.HikariDataSource       : HikariPool-1 - Start completed.
default/ctr-a-q30xnqr0r4[ctr-a-mgy3nlpym5]: 2023-01-20 12:17:35.548 DEBUG 1 --- [           main] o.s.c.t.c.SimpleTaskAutoConfiguration    : Using org.springframework.cloud.task.configuration.DefaultTaskConfigurer TaskConfigurer
default/ctr-a-q30xnqr0r4[ctr-a-mgy3nlpym5]: 2023-01-20 12:17:35.548 DEBUG 1 --- [           main] o.s.c.t.c.DefaultTaskConfigurer          : No EntityManager was found, using DataSourceTransactionManager
default/ctr-a-q30xnqr0r4[ctr-a-mgy3nlpym5]: 2023-01-20 12:17:35.641 DEBUG 1 --- [           main] o.s.c.t.r.s.TaskRepositoryInitializer    : Initializing task schema for postgresql database
default/ctr-a-q30xnqr0r4[ctr-a-mgy3nlpym5]: 2023-01-20 12:17:35.714 DEBUG 1 --- [           main] o.s.c.t.r.support.SimpleTaskRepository   : Starting: TaskExecution{executionId=23, parentExecutionId=22, exitCode=null, taskName='ctr-a', startTime=Fri Jan 20 12:17:35 UTC 2023, endTime=null, exitMessage='null', externalExecutionId='null', errorMessage='null', arguments=[--management.metrics.tags.service=task-application, --spring.datasource.username=root, --spring.datasource.url=jdbc:postgresql://postgresql:5432/dataflow, --spring.datasource.driverClassName=org.postgresql.Driver, --management.metrics.tags.application=${spring.cloud.task.name:unknown}-${spring.cloud.task.executionid:unknown}, --spring.cloud.task.name=ctr-a, --spring.datasource.password=postgres-root, --spring.cloud.task.parent-execution-id=22, --spring.cloud.task.executionid=23]}
default/ctr-a-q30xnqr0r4[ctr-a-mgy3nlpym5]: 2023-01-20 12:17:35.752  INFO 1 --- [           main] io.spring.timestamp.TaskApplication      : Started TaskApplication in 1.068 seconds (JVM running for 1.231)
default/ctr-a-q30xnqr0r4[ctr-a-mgy3nlpym5]: 2023-01-20 12:17:35.755  INFO 1 --- [           main] io.spring.timestamp.TaskApplication      : 2023-01-20 12:17:35.754
default/ctr-a-q30xnqr0r4[ctr-a-mgy3nlpym5]: 2023-01-20 12:17:35.764 DEBUG 1 --- [           main] o.s.c.t.r.support.SimpleTaskRepository   : Updating: TaskExecution with executionId=23 with the following {exitCode=0, endTime=Fri Jan 20 12:17:35 UTC 2023, exitMessage='null', errorMessage='null'}
default/ctr-a-q30xnqr0r4[ctr-a-mgy3nlpym5]: 2023-01-20 12:17:35.772  INFO 1 --- [ionShutdownHook] com.zaxxer.hikari.HikariDataSource       : HikariPool-1 - Shutdown initiated...
default/ctr-a-q30xnqr0r4[ctr-a-mgy3nlpym5]: 2023-01-20 12:17:35.783  INFO 1 --- [ionShutdownHook] com.zaxxer.hikari.HikariDataSource       : HikariPool-1 - Shutdown completed.
default/ctr-pg4n5m85q4[ctr-wrvzxj6x05]: 2023-01-20 12:17:38.648  INFO 1 --- [           main] .c.d.c.ComposedTaskStepExecutionListener : AfterStep processing for stepExecution ctr-a_0
default/ctr-pg4n5m85q4[ctr-wrvzxj6x05]: 2023-01-20 12:17:38.650  INFO 1 --- [           main] .c.d.c.ComposedTaskStepExecutionListener : AfterStep processing complete for stepExecution ctr-a_0 with taskExecution 23
default/ctr-pg4n5m85q4[ctr-wrvzxj6x05]: 2023-01-20 12:17:38.654  INFO 1 --- [           main] o.s.batch.core.step.AbstractStep         : Step: [ctr-a_0] executed in 6s881ms
default/ctr-pg4n5m85q4[ctr-wrvzxj6x05]: 2023-01-20 12:17:38.664  INFO 1 --- [           main] o.s.batch.core.job.SimpleStepHandler     : Executing step: [ctr-b_0]
default/ctr-b-kg57k4plg0[ctr-b-76ro49d5jm]: 2023-01-20 12:17:42.755  INFO 1 --- [           main] io.spring.timestamp.TaskApplication      : Starting TaskApplication v2.0.2 using Java 1.8.0_352 on ctr-b-kg57k4plg0 with PID 1 (/workspace/BOOT-INF/classes started by cnb in /workspace)
default/ctr-b-kg57k4plg0[ctr-b-76ro49d5jm]: 2023-01-20 12:17:42.757  INFO 1 --- [           main] io.spring.timestamp.TaskApplication      : No active profile set, falling back to 1 default profile: "default"
default/ctr-b-kg57k4plg0[ctr-b-76ro49d5jm]: 2023-01-20 12:17:43.274  INFO 1 --- [           main] com.zaxxer.hikari.HikariDataSource       : HikariPool-1 - Starting...
default/ctr-b-kg57k4plg0[ctr-b-76ro49d5jm]: 2023-01-20 12:17:43.443  INFO 1 --- [           main] com.zaxxer.hikari.HikariDataSource       : HikariPool-1 - Start completed.
default/ctr-b-kg57k4plg0[ctr-b-76ro49d5jm]: 2023-01-20 12:17:43.455 DEBUG 1 --- [           main] o.s.c.t.c.SimpleTaskAutoConfiguration    : Using org.springframework.cloud.task.configuration.DefaultTaskConfigurer TaskConfigurer
default/ctr-b-kg57k4plg0[ctr-b-76ro49d5jm]: 2023-01-20 12:17:43.456 DEBUG 1 --- [           main] o.s.c.t.c.DefaultTaskConfigurer          : No EntityManager was found, using DataSourceTransactionManager
default/ctr-b-kg57k4plg0[ctr-b-76ro49d5jm]: 2023-01-20 12:17:43.558 DEBUG 1 --- [           main] o.s.c.t.r.s.TaskRepositoryInitializer    : Initializing task schema for postgresql database
default/ctr-b-kg57k4plg0[ctr-b-76ro49d5jm]: 2023-01-20 12:17:43.630 DEBUG 1 --- [           main] o.s.c.t.r.support.SimpleTaskRepository   : Starting: TaskExecution{executionId=24, parentExecutionId=22, exitCode=null, taskName='ctr-b', startTime=Fri Jan 20 12:17:43 UTC 2023, endTime=null, exitMessage='null', externalExecutionId='null', errorMessage='null', arguments=[--management.metrics.tags.service=task-application, --spring.datasource.username=root, --spring.datasource.url=jdbc:postgresql://postgresql:5432/dataflow, --spring.datasource.driverClassName=org.postgresql.Driver, --management.metrics.tags.application=${spring.cloud.task.name:unknown}-${spring.cloud.task.executionid:unknown}, --spring.cloud.task.name=ctr-b, --spring.datasource.password=postgres-root, --spring.cloud.task.parent-execution-id=22, --spring.cloud.task.executionid=24]}
default/ctr-b-kg57k4plg0[ctr-b-76ro49d5jm]: 2023-01-20 12:17:43.649  INFO 1 --- [           main] io.spring.timestamp.TaskApplication      : Started TaskApplication in 1.114 seconds (JVM running for 1.311)
default/ctr-b-kg57k4plg0[ctr-b-76ro49d5jm]: 2023-01-20 12:17:43.651  INFO 1 --- [           main] io.spring.timestamp.TaskApplication      : 2023-01-20 12:17:43.650
default/ctr-b-kg57k4plg0[ctr-b-76ro49d5jm]: 2023-01-20 12:17:43.657 DEBUG 1 --- [           main] o.s.c.t.r.support.SimpleTaskRepository   : Updating: TaskExecution with executionId=24 with the following {exitCode=0, endTime=Fri Jan 20 12:17:43 UTC 2023, exitMessage='null', errorMessage='null'}
default/ctr-b-kg57k4plg0[ctr-b-76ro49d5jm]: 2023-01-20 12:17:43.664  INFO 1 --- [ionShutdownHook] com.zaxxer.hikari.HikariDataSource       : HikariPool-1 - Shutdown initiated...
default/ctr-b-kg57k4plg0[ctr-b-76ro49d5jm]: 2023-01-20 12:17:43.677  INFO 1 --- [ionShutdownHook] com.zaxxer.hikari.HikariDataSource       : HikariPool-1 - Shutdown completed.
default/ctr-pg4n5m85q4[ctr-wrvzxj6x05]: 2023-01-20 12:17:46.421  INFO 1 --- [           main] .c.d.c.ComposedTaskStepExecutionListener : AfterStep processing for stepExecution ctr-b_0
default/ctr-pg4n5m85q4[ctr-wrvzxj6x05]: 2023-01-20 12:17:46.422  INFO 1 --- [           main] .c.d.c.ComposedTaskStepExecutionListener : AfterStep processing complete for stepExecution ctr-b_0 with taskExecution 24
default/ctr-pg4n5m85q4[ctr-wrvzxj6x05]: 2023-01-20 12:17:46.425  INFO 1 --- [           main] o.s.batch.core.step.AbstractStep         : Step: [ctr-b_0] executed in 7s761ms
default/ctr-pg4n5m85q4[ctr-wrvzxj6x05]: 2023-01-20 12:17:46.433  INFO 1 --- [           main] o.s.b.c.l.support.SimpleJobLauncher      : Job: [FlowJob: [name=ctr]] completed with the following parameters: [{run.id=3}] and the following status: [COMPLETED] in 14s675ms
default/ctr-pg4n5m85q4[ctr-wrvzxj6x05]: 2023-01-20 12:17:46.441  INFO 1 --- [ionShutdownHook] j.LocalContainerEntityManagerFactoryBean : Closing JPA EntityManagerFactory for persistence unit 'default'
default/ctr-pg4n5m85q4[ctr-wrvzxj6x05]: 2023-01-20 12:17:46.443  INFO 1 --- [ionShutdownHook] com.zaxxer.hikari.HikariDataSource       : HikariPool-1 - Shutdown initiated...
default/ctr-pg4n5m85q4[ctr-wrvzxj6x05]: 2023-01-20 12:17:46.445  INFO 1 --- [ionShutdownHook] com.zaxxer.hikari.HikariDataSource       : HikariPool-1 - Shutdown completed.
default/ctr-pg4n5m85q4[ctr-wrvzxj6x05]: 2023-01-20 12:17:46.421  INFO 1 --- [           main] .c.d.c.ComposedTaskStepExecutionListener : AfterStep processing for stepExecution ctr-b_0
default/ctr-pg4n5m85q4[ctr-wrvzxj6x05]: 2023-01-20 12:17:46.422  INFO 1 --- [           main] .c.d.c.ComposedTaskStepExecutionListener : AfterStep processing complete for stepExecution ctr-b_0 with taskExecution 24
default/ctr-pg4n5m85q4[ctr-wrvzxj6x05]: 2023-01-20 12:17:46.425  INFO 1 --- [           main] o.s.batch.core.step.AbstractStep         : Step: [ctr-b_0] executed in 7s761ms
default/ctr-pg4n5m85q4[ctr-wrvzxj6x05]: 2023-01-20 12:17:46.433  INFO 1 --- [           main] o.s.b.c.l.support.SimpleJobLauncher      : Job: [FlowJob: [name=ctr]] completed with the following parameters: [{run.id=3}] and the following status: [COMPLETED] in 14s675ms
default/ctr-pg4n5m85q4[ctr-wrvzxj6x05]: 2023-01-20 12:17:46.441  INFO 1 --- [ionShutdownHook] j.LocalContainerEntityManagerFactoryBean : Closing JPA EntityManagerFactory for persistence unit 'default'

Set SCDF CTR to execute next task immediately when step is complete

1 Answers1