-1

you can see a different response from API

Explanation: In the above-shared screenshot, I compared two different files. On the left side is the data of web page no 1 while on the right side, I get the data of web page no 1080. You can see that on web page no 1 I am getting a shipment in array [ ] form which is empty, while on web page 1080 I am getting a shipment in object {} format. Furthermore, inside this object, I am getting an array [ ] of pd-option

This will help you to understand the array and object of shipment

So my question is it necessary to have the same format of the file in ADF BECAUSE I want to transform 1700 files to flatten or not then how we can give different formats in Azure ADF to flatten the files?

junaidbilal
  • 7
  • 3
  • To flatten the JSON in ADF, it must be in array format. also if you are doing the flatten for 1700 files it becomes difficult to flatten the files if all of your files don't have same schema. – Rakesh Govindula Jun 13 '23 at 12:12
  • Please provide enough code so others can better understand or reproduce the problem. – Community Jun 14 '23 at 05:15
  • I have one attribute name shipment in the JSON file there are 100 records total in one file, in some records shipment is in this format "shipment":[ ] while in some records shipment is in this format "shipment": { "id": 171700, "order_print_process_id": 170541 } so I need to flat the file in ADF I hope you can understand this in a better way. – junaidbilal Jun 14 '23 at 05:53

1 Answers1

0

  • To flatten a file, you need to have the file in the same format. You can use azure dataflows to flatten the data or any data related operations.

  • If the data is not in the same format, you can use the transformations to bring them to same format as well.

  • The following is an example of the same. Lets say you have the following data in file1.json:

{
    "id":"A1",
    "shipment":[]
}
  • And file2.json has the following data:
{
    "id":"A1",
    "shipment":{ "id": 171700, "order_print_process_id": 170541 }
}
  • You cannot use the same logic to flatten the file here because they are of different type. When you read the file1.json, you can see the projection of that file:

enter image description here

  • The projection for file2.json would be different from the file1.json because of the difference in shipment property value.

enter image description here

  • You can flatten the file1.json, but you need to use a different logic in order to extract the shipment from file2.json. Use select transformation as shown below to extract the properties.

enter image description here

  • So, you need to process these files based on the shipment value. Using the same logic on both the files would not work.
Saideep Arikontham
  • 5,558
  • 2
  • 3
  • 11