Pass parameters from Azure Data Factory to Delta Live Table pipeline

Question

I'm trying to pass some variables from Azure Data Factory to Delta live tables by passing them in the body of the web activity i'm using to call the DLT API.

{"fullRefresh":true,"configuration":{"ingestionsourceName":"XXX"}}

The same code provided in the JSON format of the DLT setting works, but not through ADF. Please let me know how I would able to achieve it.

{"fullRefresh":true,"configuration":{"ingestionsourceName":"XXX"}} provided in the DLT settings works but not passed from ADF.

Answer 1

I've tried your Json data to do update using dlt Api.

Update api used - 2.0/pipelines/{pipeline_id}/updates

with content

{"fullRefresh":true,"configuration":{"ingestionsourceName":"XXX"}}

Using below api i got status of the update

2.0/pipelines/{pipeline_id}/requests/{request_id}

and it shows terminated but without these configurations as shown in the below image.

and

To make full_refresh true you need to use property as full_refresh.

{"full_refresh":"true"}

According to this documentation you can only give following json content for update api. full_refresh,refresh_selection and full_refresh_selection.

So, after using full_refresh i got results.

But for configuration you need to set it while creating new dlt pipeline using 2.0/pipelines api.

Below is the content used in adf body.

{
    "name": "newtst2",
    "storage": "dbfs:/dlt_metastore",
     "configuration": {
    "ingestionsourceName": "adf"
    },
      "clusters": [
{
  "label": "default",
  "autoscale": {
    "min_workers": 1,
    "max_workers": 2,
    "mode": "ENHANCED"
  }
}
],
 "libraries": [
{
  "notebook": {
    "path": "/Users/user/dlt"
  }
}
],
 "continuous": false
 }

After running the activity.

Know you can get the details using this id.

Next you can just make update request giving full_refresh.