-2

I have a C++ code that need to be run independently 100,000 times, each with a different set of arguments. One single run takes around 20 minutes on a small laptop. I would like to parallelize the execution using a cloud infrastructure like GCP. What is the best way I could accomplish this task in a reasonable amount of time (1-2 days).

Botje
  • 26,269
  • 3
  • 31
  • 41
anupgp
  • 1
  • 2
  • 1
    Please edit your question so that it can be answered with facts and not result in opinion based answers. Do not ask others to design something for you. Do your own research, write the code, test and measure the results. Read this link: https://stackoverflow.com/help/how-to-ask – John Hanley Aug 28 '20 at 02:13
  • 1
    Please post your code, or we can't help you – Jérôme Teisseire Aug 28 '20 at 05:23
  • Assuming GCP cores are only as fast as your "small laptop", you would need 1400 cores for two days. A quick calculation puts this experiment at $1600 for a single shot. Are you willing to spend that much? – Botje Aug 28 '20 at 07:38

1 Answers1

1

My first approach would be to simply spin up a bunch of VMs and use GNU parallel to process the different tasks. You can keep track of which runs fail and re-run them later. See the --joblog and --resume or --resume-failed options.

A slightly more advanced approach would use an explicit queue of tasks, with records of which task succeeded and failed. This approach would use something like Sidekiq, Celery or Minion.

If you're not willing or able to do much of the heavy lifting yourself, take a look at Google Cloud Tasks. You can package your C++ code with a small Python or NodeJS wrapper that invokes it, then set up your worker pool as you like and then submit the different configurations of your problem as tasks.

Botje
  • 26,269
  • 3
  • 31
  • 41