ETL / ELT pipelines - Metainformation about the pipeline

Question

how do you add metainformation about the used ETL / ELT code (and version of this ELT code) to the produced sink files / tables?

Do u consider it as required to have information like "PipelineID" or "DataProductionTime" in the targetfolder?

To answer your first question we would need to know which ETL solution you are using. Your second question is asking for an opinion and therefore out of scope for this forum (plus the fact you should only ask one question per post) — NickW, Nov 27 '22 at 11:51

score 0 · Accepted Answer · answered Nov 28 '22 at 07:14

how do you add metainformation about the used ETL / ELT code (and version of this ELT code) to the produced sink files / tables?

You can do it using the pipeline dynamic content and additional column in copy activity.

This is my source file:

enter image description here

In source of copy activity use additional column.

enter image description here

In ADF you can find meta information about the pipeline in System variables of dynamic content.

enter image description here

Target file in sink folder:

enter image description here

Do u consider it as required to have information like "PipelineID" or "DataProductionTime" in the targetfolder?

All this depends on your requirement like if you want to know by which pipeline you are getting data you can use this.

Nice, runId is good as info here. Further i'll need the branch name -> https://stackoverflow.com/questions/74598386/azure-datafactory-get-current-branch-as-parameter — R. Maier, Nov 28 '22 at 09:19

1 Answers1