0

I have an use-case where I need to move data from an SQL based warehouse to mongo every day. We have chosen SpringXD as a tool for this ETL job. I however need to pass in few wild carded parameters to my query which change every day. I was wondering if anyone knew or could provide me with a sample DSL for passing in wild carded parameters to an SQL select query when using Spring XD.

Dipayan
  • 203
  • 4
  • 12

1 Answers1

1

Hey so this would be a great use of the SpringXD Rest API. http://docs.spring.io/spring-xd/docs/1.3.1.RELEASE/reference/html/#REST-API

What you can do is have a small script or application that programmatically deploys your streams with the job parameters you want. Let's say you have a job "myBatchJob" and two parameters that will change. All you have to do is encode the String parameters into the response, and then you can change the parameters as often as you want! This is an example in Python that posts a job deployment to the XD Rest API. Getting the encoded parameters right can be a bit tricky, but is worth working through to get to flexibility you are looking for

       import urllib2
       req = urllib2.Request('http://localhost:9393/jobs/executions?jobParameters=%7B%22-myParameter1(string)%22:%22foo%22,%22-myParameter2(string)%22:%22bar%22%7D&jobname=myBatchJob')
       response = urllib2.urlopen(req)
       print response.read()
mross1080
  • 144
  • 1
  • 7