Stack Overflow
Stack Overflow

Questions tagged [azure-data-factory]

Microsoft Azure Data Factory is a cloud service used to invoke (orchestrate) other Azure services in a controlled way using the concept of time slices. Data factories are predominately developed using hand crafted JSON, this provides the tool with instructions on what activities to perform. While still in preview, the introduction of Azure Data Factory V2 allows for an easier data integration experience using the api and not only JSON files.

Azure Data Factory is a tool with many levels of complexity and often understanding the technology doesn't come that naturally to data professionals already working with tools like SQL Server Integration Services and the SQL Server Agent. Azure Data Factory is neither of these things and sits separately in the Microsoft Data Platform stack as a service orchestration tool. It is not an ETL tool.

https://azure.microsoft.com/en-gb/services/data-factory/

This tag is to support our community of developers by sharing knowledge and through questions gain an understanding of Azure Data Factory's concepts. For example; Pipelines, Activities and Time Slices. These terms will help us design data factory flows that support our wider data solution architecture.

Controlling your Azure Data Factory will also become very important as we learn what JSON attributes are required, and where, for more complex time bound time data movements. Configuring those time slices isn't easy when validation hits data sets and activity properties at deployment time.

Finally, consider using PowerShell when working with Azure Data Factory. A rich set of cmdlets are available offering functionality not yet possible via the Azure Portal UI. See link below.

https://learn.microsoft.com/en-us/powershell/resourcemanager/azurerm.datafactories/v2.5.0/azurerm.datafactories

Getting these PowerShell scripts right will also feature in this tag once we operationalise our factory data flows.

Slowly but surely, Microsoft is developing the version 2 of Azure Data Factory which offers more possibilities for a better data integration experience and performance. The official documentation for this version can be seen here: https://learn.microsoft.com/en-us/azure/data-factory/

8447 questions
78
votes
9 answers

The subscription is not registered to use namespace 'Microsoft.DataFactory' error

Going through this tutorial "Create a pipeline with Copy Activity using Visual Studio" and receiving this error when I hit publish. Creating datafactory-Name:VSTutorialFactory,Tags:,Subscription:Pay-As-You-Go,ResourceGroup:MyAppGroup,Location:North…
user964787
  • 1,127
  • 2
  • 11
  • 20
67
votes
11 answers

The client with object id does not have authorization to perform action 'Microsoft.DataFactory/datafactories/datapipelines/read' over scope

I was trying to invoke data factory pipeline from azure function programmatically. Its throwing following error. link: http://eatcodelive.com/2016/02/24/starting-an-azure-data-factory-pipeline-from-c-net/ AuthorizationFailed: The client…
ravibhat
  • 811
  • 1
  • 7
  • 19
26
votes
1 answer

How to escape a single quote in a dynamic expression with functions?

Need to insert a parameter into a string using the dynamic data function from the pipeline parameters. Have tried backslash, double backslash, double single quote,@, @@, and other such nonsense. Just need a simple way to escape a single quote in a…
jamesbascle
  • 854
  • 1
  • 10
  • 17
25
votes
2 answers

Azure Data Factory project with Visual Studio 2017

I am not sure whether Azure Data Factory project is supported on Visual Studio 2017 at the moment. I have just installed VS 2017 but cannot open our solution as there is one azure data factory project. Does Azure Data Factory support on Visual…
cuongle
  • 74,024
  • 28
  • 151
  • 206
20
votes
4 answers

How to run SQL Script in Azure Data Factory v2?

There is NO Sql Script activity in Azure Data Factory V2. So how can I create a stored proc, a schema in a database? What are my options?
AlexB
  • 4,167
  • 4
  • 45
  • 117
19
votes
3 answers

Variables and Parameters in Azure Data Factory?

Im just learning ADF, can you explain to me the difference between Variables and Parameters? I cant find a good explanation about it.
Ruben Maldonado
  • 227
  • 1
  • 2
  • 8
19
votes
7 answers

Disable activity in Azure Data factory pipeline without removing it

So I am testing each of the activities of the pipeline and I want to disable some of the activities in it. Essentially there is an activity of sending emails which I want to disable as I wanted to see the output of prior activities. Offcourse I dont…
Gagan
  • 1,775
  • 5
  • 31
  • 59
16
votes
1 answer

What is the difference between ADF Pipeline and ADF Data flow?

What is the difference between ADF Pipeline and ADF Data flow? Why are sinks/sources supported in Pipeline and Data flow different? Is it possible to create a pipeline that reads data from source, filter, use joins and store data to a sink without a…
user989988
  • 3,006
  • 7
  • 44
  • 91
15
votes
2 answers

Azure Data Factory v2: Activity execute pipeline output

Is there a way to reference the output of an executed pipeline in the activity "Execute pipeline"? I.e.: master pipeline executes 2 pipelines in sequence. The first pipeline generates an own created run_id that needs to be forwarded as a parameter…
Johannes Vink
  • 163
  • 1
  • 1
  • 8
14
votes
2 answers

How to get custom output from an executed pipeline?

I would like to be able to get custom output from an "Execute Pipeline Activity". During the execution of the invoked pipeline, I capture some information in a variable using the "Set Variable" activity. I would like to be able to use that value in…
Heather Sawatsky
  • 352
  • 3
  • 14
13
votes
3 answers

How to transform xml data using datafactory pipeline

How do we save data inside of an XML payload to blob storage? input yes he is sdfsdfjijOIJOISJDFQPWORPJkjsdlfkjlksdf== desired result yes he…
Alex Gordon
  • 57,446
  • 287
  • 670
  • 1,062
13
votes
3 answers

Azure Data Factory specify custom output filename when copying to Blob Storage

I'm currently using ADF to copy files from an SFTP server to Blob Storage on a scheduled basis. The filename structure is AAAAAA_BBBBBB_CCCCCC.txt. Is it possible to rename the file before copying to Blob Storage so that I end up with a folder-like…
Florin D. Preda
  • 1,358
  • 1
  • 11
  • 25
12
votes
2 answers

Moving a DocumentDB Collection to Azure Data Lake Storage

I was wondering what's the best practice moving a documentDB to the Azure Data Lake Storage. Should I create a file for each document in a collection or move the entire documentDB? Also I didn't find much information on how I can access the…
reachify
  • 3,657
  • 2
  • 19
  • 22
11
votes
6 answers

How to get the Azure Data Factory parameters into the ARM template parameters file (ARMTemplateParametersForFactory.json) after publishing

I am trying to create my Azure DevOps release pipeline for Azure Data Factory. I have followed the rather cryptic guide from Microsoft (https://learn.microsoft.com/en-us/azure/data-factory/continuous-integration-deployment ) regarding adding…
NER1808
  • 1,829
  • 2
  • 33
  • 45
10
votes
4 answers

Import python module to python script in databricks

I am working on a project in Azure DataFactory, and I have a pipeline that runs a Databricks python script. This particular script, which is located in the Databricks file system and is run by the ADF pipeline, imports a module from another python…
Cristian Ispan
  • 571
  • 2
  • 5
  • 23
1
2 3
…
99 100