0

I want to write a program that calculates the time it takes to read in a folder of .py files and calculate the cyclomatic complexity of each of the files. I have Radon installed to calculate the complexity, but I also want to be able to implement a distributed system that creates a set of n workers, where each worker is given a seperate file in the folder which is then calculated using radon.

I'm using dask for the distributed system and was wondering is it possible to achieve what I'm asking above. I.e. if I have a folder of 10 .py files, I can create 1 worker which will read in all the files and calculate complexity, then my program will log the time it took to do that. Or i could specify 10 worker nodes who will look for work (i.e. the files to calculate) and each will take a file and run concurrently, then the program will log the time it took to do that.

I have the basic program set up using dask, which calls a function, but am unsure if you can give a list of items which is distributed over a set workers which then call the function and return the results.

Is this possible using dask?

J.Doe
  • 21
  • 3
  • Short answer: yes, this is totally possible, and very much a typical pattern for dask. Perhaps if you could describe your trouble, we could help you more? – mdurant Nov 29 '17 at 01:33
  • I am trying to develop a REST service which is provided a folder (containing a bunch of python .py files), I want the program to calculate the complexity of each file, my program then logs the time it takes to do this, essentially the function, in this case calculate complexity, is not of the concern, I just want to be able to get a set of n workers to take each file that has not yet been analyzed and analyze it. Essentially I want this program to illustrate that with the addition of workers the time to perform this task on a folder of files decreases – J.Doe Nov 29 '17 at 10:37

1 Answers1

0

Yes, this is possible. I recommend reading documentation about either dask.delayed or dask futures:

MRocklin
  • 55,641
  • 23
  • 163
  • 235