Using Azure Data Factory to get data from a REST API

Question

Is it possible to use Azure Data Factory to get data from a REST API and insert it to a Azure database table?

score 10 · Accepted Answer · edited Nov 12 '19 at 09:43

Data factory offers a generic HTTP connector and a specific REST connector, allowing you to do retrieve data from HTTP endpoints by using GET or POST methods.

Example: HTTP Linked Service

{
    "name": "HttpLinkedService",
    "properties":
    {
        "type": "Http",
        "typeProperties":
        {
            "authenticationType": "Anonymous",
            "url" : "https://en.wikipedia.org/wiki/"
        }
    }
}

score 3 · Answer 2 · answered Feb 29 '16 at 13:53

I have done this using Custom .Net Activities. I had a need to pull data from Salesforce API. I have a write up on how to do this here: http://eatcodelive.com/2016/02/06/accessing-azure-data-lake-store-from-an-azure-data-factory-custom-net-activity/ it is utilizing the data lake store but you can store the data anywhere.

Also have a look here for an example from microsoft: https://azure.microsoft.com/en-us/documentation/articles/data-factory-use-custom-activities/

Hope that helps.

score 2 · Answer 3 · answered Mar 09 '16 at 00:48

This can be achieved with Data Factory. Data Factory is useful if you want to run batches on a schedule and have a single place for monitoring and management. There is sample code in our GitHub repo for an http loader to blob here https://github.com/Azure/Azure-DataFactory. Then the blob can be consumed for other processing, like putting it into your table. To do this, you would create a pipeline that contains your custom loader, then an activity with blob as the input dataset and table as the output dataset.

score 1 · Answer 4 · answered Dec 16 '16 at 06:43

1

https://github.com/Azure/Azure-DataFactory/tree/master/Samples/HttpDataDownloaderSample

Hi this is a custom data activity that dowloads a file by http request. You can achieve what you desire making small changes. Hope it helps.

answered Dec 16 '16 at 06:43

ing.alfano

406
1
9
18

The link is dead, use this instead: https://github.com/Azure/Azure-DataFactory/tree/master/SamplesV1/HttpDataDownloaderSample – J Marlow Feb 28 '20 at 10:32

score -1 · Answer 5 · answered Jan 04 '19 at 03:17

-1

There is now support for REST as a data source in Azure Data Factory, including pagination

answered Jan 04 '19 at 03:17

Ivan Wilson

403
3
12

score -2 · Answer 6 · answered Feb 24 '16 at 07:38

-2

Short answer: Yes :)

Long answer (Tutorial basically): https://azure.microsoft.com/en-us/documentation/articles/data-factory-get-started/

answered Feb 24 '16 at 07:38

Pedro G. Dias

3,162
1
18
30

This tutorial uses a azure storage blob as the input data. I want to retrieve data from another source using http calls. – dagrun Feb 24 '16 at 11:25
Indeed, but the principle is exactly the same, just replace blob with REST client and you're green :) – Pedro G. Dias Feb 24 '16 at 11:29

Using Azure Data Factory to get data from a REST API

6 Answers6

Linked