0

I am trying to execute multiple spring cloud task jobs within spring cloud data flow container on PCF. These jobs reads the raw file from a http source and then parses it and writes that to mysql db.These jobs are written in plain java and not with spring batch. I have binded mysql db with the scdf container on PCF . I believe spring cloud task will use mysql db to store the execution status of these status .I want the actual output records also to go in mysql.

My question is how the output records for each of these jobs will get stored in mysql db ? Will it use different schema for each of these parser jobs ? If not then how can I configure it to do so ?

Please share your thoughts if you have encountered this scenario.

Thanks!

Nilanjan

1 Answers1

0

To orchestrate Tasks in SCDF, you'd have to supply an RDBMS and it looks like you've already done that. The task repository is primarily used to persist Task executions as a historical representation, so you can drill into the entire history of executions via the GUI/Shell.

You'd configure task-repository at the server level - see this CF-server's manifest.yml sample (under services: section) for reference.

My question is how the output records for each of these jobs will get stored in mysql db ?

If you'd like to also use the same datastore for all the tasks, it can be configured via SPRING_CLOUD_DEPLOYER_CLOUDFOUNDRY_TASK_SERVICES env-var. Anything supplied via this token would be automatically propagated to all the Task applications.

However, it is your responsibility to make sure the right database driver is in the classpath of your Task application. In your case, you'd need to have one of the mysql drivers.

Will it use different schema for each of these parser jobs ?

It is up to your business requirements. Whether it is a different schema or different set of tables, you'd have to determine what's needed for your requirements and make sure it exist/setup before binding the Task application via SPRING_CLOUD_DEPLOYER_CLOUDFOUNDRY_TASK_SERVICES.

If not then how can I configure it to do so ?

If you have to use a different datasource, you can supply a different mysql binding for the Task application that includes your requirement specific schema/table changes. Review this section to learn how autoconfiguration kicks in on PCF.

As an alternative option, you can selectively supply different mysql binding at each application, too - here's some docs on that.

Sabby Anandan
  • 5,636
  • 2
  • 12
  • 21
  • Hi Sabby, Thanks a lot for your detailed explanation. I believe the property app.jdbc.spring.cloud.deployer.cloudfoundry.services=mysqlService will also work with task launch command also. As i understand I have to create a mysql service on PCF and while launching the task I have to pass this property which binds my task to the mysql instance. Can I pass this property when i launch the task from DataFlowRestTemplate API ? Thanks ! – Nilanjan Roy Mar 26 '17 at 12:59
  • Just now I got it from your another post :curl http://localhost:9393/tasks/deployments/foo\?arguments\=\&properties\= -d "". If i am using DataFlowRestTemplate how to set that ? – Nilanjan Roy Mar 26 '17 at 13:04