0

I have a pipeline with about 20 copy activities which are independent to one another. This copy activity is basically copy data from on premises data source to azure sql DB. I have two questions. Can I run all 20 activities at the same time parallel? At the moment it is just running 4-6 activities and rest are going in queue and getting started only after the earlier finished. How can I run 20 same time. PS : I have increased the max concurrency within the Self hosted IR to 24 but still it's same.

Question 2: is there any option within the pipelines to include any parameter such that it could execute any number of parallel activities to execute?

Thanks for your help

Learning always
  • 1
  • 1
  • 1

1 Answers1

0

There is no option to increase the number of parallels. You can only scale up by increasing the number of concurrent jobs that a node can run as you did. If processor usage is high and available memory is low, although you change limit concurrent jobs larger, it can't achieve your requirement. You can refer to this doc.

Steve Johnson
  • 8,057
  • 1
  • 6
  • 17