1

We have created a Spring Batch job to be executed in Spring Cloud Data Flow through Spring Cloud Task (a simple task, it only executes the job). The execution has been checked both with the UI and the REST API, and everything is OK in an ideal case. The problem comes when we try to stop and restart a job. Following the REST API Guide:

  1. Launch the task: https://docs.spring.io/spring-cloud-dataflow/docs/current/reference/htmlsingle/#api-guide-resources-task-executions-launching
  2. Get information of the task execution using the task execution id returned in the previous point: https://docs.spring.io/spring-cloud-dataflow/docs/current/reference/htmlsingle/#api-guide-resources-task-executions-detail
  3. Here comes the first issue: if we stop the task with the task execution id (https://docs.spring.io/spring-cloud-dataflow/docs/current/reference/htmlsingle/#api-guide-resources-task-executions-stopping) the job associated to the task stops its execution (in the logs) but in de database the job stays as "STARTED". And then it is impossible to restart the job execution as it never reachs the "STOPPED" status --> For the SCDF gurus... isn't that a bug or there is a technical reason?
  4. If we want to stop the job, it is necessary to get the jobExecutionIds first param from the response of the point 2, and make the request with the job execution id (https://docs.spring.io/spring-cloud-dataflow/docs/current/reference/htmlsingle/#api-guide-resources-job-executions-stop)
  5. Waiting the appropiate time to complete the running step execution, the job reaches the "STOPPED" status. If now we restart the job (https://docs.spring.io/spring-cloud-dataflow/docs/current/reference/htmlsingle/#api-guide-resources-job-executions-restart) a new task execution is created, linked with the new job execution, and it completes the processing.

And now the million dollar question: if we want to integrate SCDF with a planifier (like Control-M) how can we make polling requests to check the after-restart execution's status if the restart request does not return any execution id? If there is just a normal execution, we can send polling requests with the job execution id (step 4) associated to the task execution id (step 2), but after the restart we are "blind"

RLS
  • 65
  • 2
  • 10

2 Answers2

0

Here comes the first issue: if we stop the task with the task execution id (https://docs.spring.io/spring-cloud-dataflow/docs/current/reference/htmlsingle/#api-guide-resources-task-executions-stopping) the job associated to the task stops its execution (in the logs) but in de database the job stays as "STARTED". And then it is impossible to restart the job execution as it never reachs the "STOPPED" status

This looks like a bug and feel free to report it at https://github.com/spring-cloud/spring-cloud-dataflow/issues. When reporting, please add any additional information such as the SCDF server log etc., to investigate it better.

if we want to integrate SCDF with a planifier (like Control-M) how can we make polling requests to check the after-restart execution's status if the restart request does not return any execution id? If there is just a normal execution, we can send polling requests with the job execution id (step 4) associated to the task execution id (step 2), but after the restart we are "blind"

SCDF server exposes REST endpoints to check the status of Job Executions. You can either hit the server endpoints or use the REST client to check. For more information on the REST client, you can check [here].1

Ilayaperumal Gopinathan
  • 4,099
  • 1
  • 13
  • 12
  • Hello Ilayaperumal Gopinathan, thanks for your answer. I will report the issue in GitHub. About the status of Job Executions, I know that there are endpoints to get the execution info, the problem is that we can't know the execution ID in advance. The starting request returns the ID that can be used to recover the info, but the restarting request doesn't return anything. – RLS Mar 23 '20 at 08:17
  • The restarting request doesn't return the execution ID because it is expected to use the same job execution ID it is trying to restart. Not sure if I misunderstand your requirement. – Ilayaperumal Gopinathan Mar 24 '20 at 06:31
  • In Spring Batch the job execution ID can't be reused, each execution (either if it's a start or a restart) has its own ID. Other thing is the "job instance id" that is common between start and the restarts. This parameter is the "jobId" field returned in the Job Execution Detail request [link](https://docs.spring.io/spring-cloud-dataflow/docs/current/reference/htmlsingle/#api-guide-resources-job-executions-detail). – RLS Mar 25 '20 at 09:24
  • Thinking about this, I have realized that there is another request in the SCDF API for instances: [link](https://docs.spring.io/spring-cloud-dataflow/docs/current/reference/htmlsingle/#api-guide-resources-job-instances-detail). And with it, it's possible to get the data (the cons are that there are needed 4 requests) – RLS Mar 25 '20 at 09:25
0

Well, I don't know if there is a directer way to do this, but this is a possible solution for the integration with an external system:

  1. Launch the task: https://docs.spring.io/spring-cloud-dataflow/docs/current/reference/htmlsingle/#api-guide-resources-task-executions-launching this returns the task execution id
  2. Recover the job execution id associated to the previous task execution id: https://docs.spring.io/spring-cloud-dataflow/docs/current/reference/htmlsingle/#api-guide-resources-task-executions-detail being job execution id the field "executionId"
  3. Use the job execution id to stop the execution (optionally, it could have been finished by an error): https://docs.spring.io/spring-cloud-dataflow/docs/current/reference/htmlsingle/#api-guide-resources-job-executions-stop
  4. Use the job execution id to restart/resume the execution: https://docs.spring.io/spring-cloud-dataflow/docs/current/reference/htmlsingle/#api-guide-resources-job-executions-restart
  5. Use the "old" job execution id to get the job instance id: https://docs.spring.io/spring-cloud-dataflow/docs/current/reference/htmlsingle/#api-guide-resources-job-executions-detail the job instance id is the field "jobId"
  6. Use the job instance id to get the "new" job execution id: https://docs.spring.io/spring-cloud-dataflow/docs/current/reference/htmlsingle/#api-guide-resources-job-instances-detail where the field "jobExecutions" is an array of execution where the first position is the last execution, so we can get the new job execution id from the field "executionId"
RLS
  • 65
  • 2
  • 10