How can we parameterise Azure Synapse Spark Jobs?

Question

The Spark Jobs UI in Azure Synapse has the option to pass command line arguments to the underlying code, but there doesn't appear to be any option to parameterise these arguments.

Similarly the Spark Job Definition activity in Azure Pipelines doesn't offer any parameterisation options.

Is there any way to pass parameters to a Azure Synapse job?

score 2 · Accepted Answer · answered May 26 '21 at 12:21

2

It's there in Azure Synapse Pipelines. Released May 2021.

answered May 26 '21 at 12:21

Piotr Gwiazda

12,080
13
60
91

Looks totally different than this. Also how do we parameterize a .net job? – David Beavon Aug 26 '22 at 23:39

score 0 · Answer 2 · answered Dec 29 '20 at 07:24

Currently, the product team working on the public document/tutorial on How can we parameterise Spark jobs.

For now, you can use the job definition JSON file to parameterize the Spark job. Attached one sample file:

{
  "targetBigDataPool": {
    "referenceName": "yifso-1019",
    "type": "SparkComputeReference"
  },
  "requiredSparkVersion": "2.4",
  "jobProperties": {
    "name": "job definition sample",
    "file": "wasbs://ContainerName@StorageName.blob.core.windows.net/SparkSubmission/artifact/default_artifact.jar",
    "className": "sample.LogQuery",
    "args": [],
    "jars": [],
    "pyFiles": [],
    "archives": [],
    "files": [],
    "conf": {
      "spark.hadoop.fs.azure.account.key.StorageName.blob.core.windows.net": "StorageAccessKey"
    },
    "numExecutors": 2,
    "executorCores": 4,
    "executorMemory": "14g",
    "driverCores": 4,
    "driverMemory": "14g"
  }
}

The job definition JSON can be modified, imported, and run directly.

which field in this sample file is the parameter that gets it value passed in from synapse? — Tdawg90, Oct 24 '22 at 19:00

How can we parameterise Azure Synapse Spark Jobs?

2 Answers2