0

I have an embarrassingly parallel problem I want to scale across multiple computers using .Net.

I’m currently using the Task Parallel Library to scale across multiple cores on a single computer but I want to scale further.

How can I do this?

Sarah Kessington
  • 127
  • 5
  • 3
    It's hard to say since you haven't given any details about the problem, but message queuing is an easy way to scale across multiple computers. – Daniel Mann Mar 17 '14 at 20:31

2 Answers2

3

I’ve created an open source library you can use to scale embarrassingly parallel solutions across multiple computer using .Net.

http://simplescale.github.io/

Keith Harrison
  • 1,567
  • 14
  • 22
1

Check this out: http://research.microsoft.com/en-us/projects/Dryad/

Dryad is an infrastructure which allows a programmer to use the resources of a computer cluster or a data center for running data-parallel programs. A Dryad programmer can use thousands of machines, each of them with multiple processors or cores, without knowing anything about concurrent programming.

UPDATED: If you need data intensive parallel processing, check out HDInsight. If the work is more compute intensive, check out the Microsoft HPC Pack (currently v2012 SP1).

The essence of HPC is executing code on many machines at the same time. For HPC Pack 2012 SP1, this means spreading the application’s logic across multiple compute nodes, where each node is a computer that runs Windows

http://research.microsoft.com/en-us/projects/azure/high-perf-computing-on-windows-azure.pdf

Noel
  • 3,288
  • 1
  • 23
  • 42
  • "Microsoft discontinued active development on Dryad, shifting focus to the Apache Hadoop framework" http://en.wikipedia.org/wiki/Dryad_%28programming%29 – Sarah Kessington Mar 17 '14 at 20:33