Stack Overflow
Stack Overflow

Questions tagged [azure-adf]

The Azure Data Factory (ADF) is a service designed to allow developers to integrate disparate data sources

35 questions
0
votes
0 answers

Is it possible to iterate multiple Azure data factory pipeline run ids and activity runs metadata onto Azure dashboard using fewer variables

Is it possible to iterate multiple Azure data factory pipeline run ids and activity runs metadata onto the Azure dashboard using fewer variables. for example, variable 'bytesread' iterate using some loop to capture all of 'bytesread' values from…
Pankaj Kushwaha
  • 21
  • 3
0
votes
1 answer

ADF - Failed To Send Request to Databricks Cluster

I have noticed that my orchestrated pipelines (in ADF) sometimes fail due to this error: ErrorCode=FailedToReadFromAzureDatabricksDeltaLake,Failed to read from azure databricks delta lake. Error message : Failed to send request to Azure Databricks…
Oscar Dyremyhr
  • 117
  • 12
0
votes
1 answer

What is the "Resource" when setting up a call to REST API from ADF with Managed Indentity

I have created an Azure Function App and I can call it from browser with a URL similar to this: https://myfunctionblahblh.appserviceenvironment.net/api/names/Peter/TGIF?code=my_function_app_key_Goes_Here Now I want to call it from my ADF pipeline:…
UnskilledCoder
  • 192
  • 1
  • 11
0
votes
1 answer

ADF dynamic content using concat - need to embed commas inside of string for long list of columns

The use case seems pretty simple.... Produce a sql statement as part of a copy activity that includes a hard coded column listing and also concatenated to a parameter-provided database and table name (since the database and table names can change…
Gary
  • 69
  • 6
0
votes
1 answer

ADF - replace switch statement for dataflows with a parameterized solution

I have a switch statement that looks at a variable value and based on that determines which data flow to execute. The problem with this is, I need to update the switch statement every time I add a new ID/dataflow. Is there an alternative design to…
Mehdi Ibrahim
  • 2,434
  • 1
  • 13
  • 13
0
votes
1 answer

ADF Pipeline to Split an a JSON File with array of objects into seperate items in cosmos db

I am creating a pipeline for the first time. The requirement is to fetch a JSON file with an array of objects. Split the array and save each object as a separate item in the cosmos DB. In pipeline I am using CopyData Activity. And providing the…
Sahaduna
  • 33
  • 4
0
votes
1 answer

ADF-SQL server untrusted domain

I am trying to do a lookup from a sql server table. I am able to connect to the sql server instance via MSSQL management studio, but in ADF I get the below error. Cannot connect to SQL Database: '', Database: '', User: '****'. Check the linked…
dragonachu
  • 451
  • 1
  • 7
  • 20
0
votes
1 answer

How to parse each row of an excel using Azure Data Factory

here is my requirement: I have an excel with few columns in it and few rows with data I have uploaded this excel in Azure blob storage Using ADF I need to read this excel and parse the records in it one by one and perform an action of creating…
sparsh agarwal
  • 15
  • 1
  • 5
0
votes
1 answer

ADF Copy only when a new CSV file is placed in the source and copy to the Container

I want to copy the file from Source to target container but only when the Source file is new (latest file is placed in source). I am not sure how to proceed this and not sure about the syntax to check the source file greater than target. Should i…
user2703679
  • 35
  • 2
  • 10
0
votes
1 answer

OData in Datafactory

I have a task toget some data from an external supplier. They have a Rest OData API. I have to connect using a subscription-key(APIKey). When creating the OData LService, I add an Auth Header: "subscription-key" and in the Value field, I enter my…
Tom Brox
  • 1
  • 2
0
votes
3 answers

Check if all files are available in storage - Azure ADF

In Azure data factory, how to check if an array of string(filenames) contains value? I'm getting file names from get metadata activity and I need to check if all 4 filenames that I have are available in storage account before proceeding. I'm…
ADFDev
  • 3
  • 2
0
votes
1 answer

Azure ADF Get Pipeline Run - What is the message attribute?

Microsoft ADF has a set of apis to get the status of a pipeline run. Ref: https://learn.microsoft.com/en-us/rest/api/datafactory/pipeline-runs/get In the response, there is an attribute called message. message | string | The message from a pipeline…
Halley
  • 521
  • 10
  • 35
0
votes
1 answer

how to exclude files with certain prefix in azure data factory pipeline while copying the data

I want to exclude the .mp4 file while copying it from Azure Storage (blob) to Sftp. Following is my source settings
SarangRN
  • 91
  • 13
0
votes
1 answer

adf function to read only certain portion of filename based on pattern

I have file names as SMP_ACC_STG_20210987654.txt and another filename SMP_ACC_STG_BS_20210987654.txt. I can use @substring(item().name,0,11) and i get SMP_ACC_STG for first file which is correct but for second file I need to get filename as…
Nanda
  • 159
  • 1
  • 13
0
votes
1 answer

ADF Json File Column count and Flattening all columns in file without data flow

How to count the number of all column in JSON file using Azure Data Factory activity ? How to Flatten complete objects(All columns) in JSON file using Azure Data Factory activity?
krishna bhovi
  • 1
1
2
3