Getting Azure Synapse pipeline run ID when running noteboot

Question

How can I get Azure Synapse Pipeline run ID using python SDK in a synapse notebook? I have a ML pipeline that do a batch prediction everyday. What I want to do is get the pipeline run ID to save inside a DataFrame that is created in one of the notebook that I run in synapse.

I'm running my pipeline using the following major activities:

notebook: I'm using a synapse notebook to read some parquet files from a blob preprocess using pandas and save again in another blob.

I saw that you can put a system variable in the file name with the pipeline run id. But I want to now if there is a way to get the current pipeline run id and my notebook is being executed using the azure python sdk

Please provide the information that what you have tried so far and how you are running ML pipeline from synapse? — Rakesh Govindula, Nov 29 '22 at 18:56

score 1 · Accepted Answer · answered Nov 30 '22 at 06:10

You can do it using the Toggle parameter cell option in Synapse notebook cell.

enter image description here

Use Toggle parameter cell option in synapse notebook cell and give any parameter name and assign any value to it. (Here I have given empty string).

In Notebook activity of pipeline, use Base parameters and give same name and same data type for it. In dynamic content of the parameter give @pipeline().RunId like below.

enter image description here

Execute this activity and Go to Monitor-> Pipeline runs -> your pipeline-> Notebook activity snapshot and you can see the output of the Notebook.

enter image description here

You can use this parameter in the Notebook as per your requirement.

You're indeed a god! Thank you my friend, this helped me A LOT — Gabriel Padilha, Dec 01 '22 at 16:35

Getting Azure Synapse pipeline run ID when running noteboot

1 Answers1