0

How do I create azure datafactory for incremental load using python? Where should I mention file load option(Incremental Load:LastModifiedOn) while creating activity or pipeline??

We can do that using UI by selecting File Load Option. But how to do the same pragmatically using python?

Does python api for datafactory support this or not?

bad_coder
  • 11,289
  • 20
  • 44
  • 72
Radhi
  • 6,289
  • 15
  • 47
  • 68

2 Answers2

0

According to this blog, you could only get the Incremental Load feature from ADF SDK by setting modifiedDatetimeStart and modifiedDatetimeEnd properties.

You could set the above properties in the additional_properties of azure_blob_dataset class.

enter image description here

Jay Gong
  • 23,163
  • 2
  • 27
  • 32
0

My investigations suggest that the Python SDK has not yet implemented this feature. I used the SDK to connect to my existing instance and fetched two example datasets. I did not find anything that looked like the 'last modified date'. I tried dataset.serialize() , dataset.__dict__ , dataset.properties.__dict__ . I also tried .__slots__ .

Trying serialize() is significant because there ought to be parity between the JSON generated in the GUI and the JSON generated by the Python. The lack of parity suggests the SDK version lags behind the GUI version.

UPDATE: The SDK's are being updated.

MartinJaffer-MSFT
  • 683
  • 4
  • 6