0

I have a JSON with the following schema:

"invoices-report": {
  "locations": {
    "location": {
      "@code": "TX",
      "@name": "Texas",
      "invoices": {
        "invoice": [
          {
            "@InvoiceNumber": "111111",
            "@AccountName": "Name",
            "@InvoiceDate": "2021-11-01",
            "items": {
              "item": [
                {
                  "@QTY": "1.00",
                  "@Rate": "5",
                  "@Description": "Services"
                },
                {
                  "@QTY": "2.00",
                  "@Rate": "5",
                  "@Description": "Services1"
                },
                 …
                {
                  "@QTY": "2.00",
                  "@Rate": "5",
                  "@Description": "ServicesN"
                },                   
                }
              ]
            }
          },
          {
            "@InvoiceNumber": "222222",
            "@AccountName": "Name2",
            "@InvoiceDate": "2021-11-01",
            "items": { …..}

Basically I have an array of invoice numbers with subarray of Invoice Details. I would like to split it to 50 parallel branches to speed up processing. My flow looks like this:

Parse Json -> Create Variable ->

length(array(body('Parse_JSON')?['invoices-report']?['locations']?['location']?['invoices']?['invoice']))

For example, the variable returns 56,900 invoices. I would like to create 50 branches based on array order No. - >56,900/50=1,138 - > if invoice order numbers 0 - 1,138 - process these invoices in the first branch and so on. Each branch would also contain "Apply to each" function with degree of Parallelism of 50. Could you please explain how do I divide the array on 50 branches? Thank you in advance!

General Grievance
  • 4,555
  • 31
  • 31
  • 45
Nataliia
  • 23
  • 6
  • When you say “branch” what do you mean? Are you going to call a child flow with each packet of data in parallel? – Skin Jan 11 '22 at 20:04
  • When I add a new step to the flow, there are 2 options: 1) Add an action 2) Add a parallel branch. I am referring to the second option. The maximum amount of parallel actions can be 50, which I would like to utilize to speed up the precessing. First branch would work with first 1138 invoices, 2nd branch with the next 1138 and so on. – Nataliia Jan 11 '22 at 20:12
  • So you want to create 50 parallel branches that contain all of the same actions?!? If so, that’s suicide! That’s potentially a maintenance nightmare. You should look at using child flows. – Skin Jan 11 '22 at 20:22

1 Answers1

1

To do what you want, you need to use the SKIP and TAKE functions over the array to split it up so you can then call your child flow.

Here is an example of the approach you need to take.

FYI, the packet size san be anything, it doesn't need to be a perfect division of the array length.

Flow

Your Do Until loop should process until the position of the slicer reaches the end of the array.

This is the expression in the right hand side of the Do Until condition ... length(variables('Array'))

This is the Skip/Take expression ... take(skip(variables('Array'), variables('Slicer Position')), variables('Packet Size'))

https://learn.microsoft.com/en-us/azure/logic-apps/workflow-definition-language-functions-reference#skip https://learn.microsoft.com/en-us/azure/logic-apps/workflow-definition-language-functions-reference#take

Skip Take

So when it runs, you can pass in the Sub Array variable to your child flow and it will process that section of the data in parallel.

Packets

You should be able to apply that concept to your own data set.

Skin
  • 9,085
  • 2
  • 13
  • 29