0

I am building a new data pipeline for our team. This data pipeline would collect data from multiple sources and ingest them into a single table. I am looking into a couple of options within Azure to achieve this (Synapse being the main option). I was only able to create a pipeline using the Synapse Studio. In the future I might need to add in other source tables without any manual configurations through the UI (1000s of source tables). I was wondering if there is a way to automatically build pipes using some sort of template. Is there any other equivalent to achieve this if not? Thanks!

Perseus
  • 1
  • You have to get out of the mindset of pipeline per table. You want few pipelines and lots of config. The Copy activity for example supports schemaless import and auto-create tables so you can have one Copy pipeline that could import many tables of all different formats. Have a look at the metadata driven pipeline tool included in the gallery for more info. – wBob Jun 01 '22 at 19:37

1 Answers1

0

The short answer is yes, you can template out pipeline patterns and even automate the creation of pipelines.

The longer answer of how to accomplish this is going to depend on the details of what you're trying to accomplish. On the simpler end, you can create a classic parent-child pipeline setup where a parent pipeline is fetching the configuration to pass into parameterized child pipelines. This works well in a lot of situations - especially where there is a clear common pattern to your data load. You can look at this documentation for more information on using parameters with your pipelines: How to use parameters, expressions and functions in Azure Data Factory

From a more complicated perspective, the pipelines are nothing but json documents that define the flow of the pipeline. This json can be generated and purpose built for you data load needs. It can also be managed outside of the ADF/Synapse environment which may have some advantages as well.

If you can provide some more details of your exact scenario I can likely give more details on what a possible solution would look like.

MrVitaminP
  • 96
  • 4