2

We are running some heavy computations that can take a lot of time (even days). We have an architecture similar to that of a compiler (compiler + linker):

  • many blocks of data that need to be computed (they are computed by a solver.exe executable) - similar to a compiler
  • a gatherer application that combines all the results into the final form - similar to a linker

What I saw is that distcc does the distribution over the network quite nicely. Unfortunately distcc works under unix environment.

Is there a platform that can provide something similar to what distcc does, but under Windows (7+) platform?

LATER EDIT: Added C++ tag because if such framework exists we would benefit from our C++ coding experience.

INS
  • 10,594
  • 7
  • 58
  • 89
  • You can use distcc on Windows boxes under Cygwin. – Retired Ninja May 15 '13 at 06:43
  • Are you using Visual Studio? – Bob Jun 24 '16 at 19:19
  • @Adrian I can use it, yes. – INS Jul 13 '16 at 21:13
  • But to be clear, you're not looking for an exclusive "C++ compiler/linker" solution just for a generic "job distributor"? – Bob Jul 13 '16 at 21:15
  • I don't work at either Incredibuild or Electric-Cloud. If you have no complex cross-job dependencies OR if you can auto-generate the dependencies i.e. `job1: job2 job3` etc, then I would go with Incredibuild. I've been using Electric-Cloud and the only advantage - and for my C++ project with badly written makefiles, it's necessary and a big one - is that EC can dynamically resolve implicit dependencies between jobs. – Bob Jul 13 '16 at 21:29
  • @INS because of this feature, EC charges much more. Incredibuild's recurring fee is low and constant i.e. doesn't depend on how much time you save. Plus, it has a Visual Studio plugin so you can leverage that for your code . . . as long as VS generates "parallel-correct" makefiles which I think it does. From your description, it sounds like you could define your tests as `target`s in a makefile to be done using your `solver.exe` and then you'd make your `final` target depend on all your test results that need to be linked `final: test1 test2 test3` – Bob Jul 13 '16 at 21:35
  • @INS FYI, I really am using EC for my task cause I need its awesome dynamic dependency resolver. My company inherited a makefile mess that cannot be executed using `--jobs=16` bc the dependencies of all targets are not fully defined . . . in case you think I have something against that company. – Bob Jul 13 '16 at 21:36

2 Answers2

4

I don't know of any free/open source solutions for Windows, but there are two commercial offerings which may help:

ElectricAccelerator from Electric Cloud. ElectricAccelerator was originally designed to accelerate software builds, but can distribute arbitrary processes to a network of computers for execution in parallel. Based on what you've said, it should be straightforward to use Accelerator for your needs. It is available on both Linux and Windows.

Xoreax Incredibuild was also originally designed to accelerate software builds, but like Accelerator, Incredibuild can distribute arbitrary processes. It is available only on Windows.

Disclaimer: I am the architect of ElectricAccelerator

Eric Melski
  • 16,432
  • 3
  • 38
  • 52
  • @INS if you're looking for a simple "job distributor" then Incredibuild is good enough. ElectricAccelerator is way more expensive and unnecessary if all you need is a job distributor. – Bob Jun 23 '16 at 18:38
  • @Adrian ElectricAccelerator has a freemium offering now called [ElectricAccelerator Huddle](http://electric-cloud.com/products/electricaccelerator/huddle/), which uses a pay-as-you-go or "utility-based" pricing model -- you only pay for the time you save. We've gotten nothing but good feedback about the affordability and "fairness" of this billing model. – Eric Melski Jun 24 '16 at 16:49
  • @EricMelski yes Huddle is awesome and I'm evaluating it but one only needs it if (a) you need to execute makefiles in parallel AND (b) the makefiles have implicit dependencies i.e. not everything is listed in the prereq for a target. Otherwise, Incredibuild does the same thing and for far less cost. – Bob Jun 24 '16 at 19:05
  • @INS I priced out Huddle vs Incredibuild and for my project, it'd be thousands of dollars per year. But for what Huddle does - dynamic dependency discovery in makefiles - that's a fair price; our inherited makefiles cannot be used with `make`'s `--jobs=N` and therefore cannot be used with Incredibuild bc it requires your makefile be "parallel-ready". But if you don't NEED that feature because (1) your makefiles were written properly or automatically generated OR (2) your "jobs" have no inter-dependencies . . . then Huddle is way over-kill. – Bob Jun 24 '16 at 19:15
3

FastBuild is free and opensource. It is available for Windows/Linux/OSX. It has its own language for build config files. You can setup workers on local network and a client that connects to workers and delivers work to them. FastBuild works for C++ and should probably work for any language, as you can setup any compiler executable and dependencies (compiler is distributed to workers, so that you don't need to setup compiler on every machine). The only drawback that I found is that in order to connect workers with clients you need to setup a Windows network share with writable folder, I think that's very inconvenient way to setup a build cluster, a better way would be to write a config with list of servers, maybe it will be fixed in the future. Also FastBuild cluster can be run only on local network, would be great to be possible to setup many computers though internet, it shouldn't be difficult to implement, also internet connection should be encrypted to secure secret sources.

Arty
  • 14,883
  • 6
  • 36
  • 69