0

thanks in advance for your patience - I'm not a programmer, but a researcher. I have a model worked up in a short python script that is computationally intensive. A time horizon input of 3 takes about 2 minutes on my 2020 MacBook Pro, but I need it to run 6 steps' worth, and it takes time exponential in input size. My 2020 MacBook Pro has not returned an answer with the input of 6 in almost 24 hours now.

So I finally worked out how to use free trial time on the Google Cloud Platform, winging it all the way. I set up a virtual instance, using an "enhanced computing" VM with 8 cores. I managed to get my script up there, install the python modules needed, and so on (this all took me quite some time). I finally got it working - but for an input of 3, which requires 2m8s on my laptop, the same script / environment on the VM took 4m36s!

So I thought I was basically renting a supercomputer ... but apparently I'm totally not? Any advice appreciated! Can I use a faster machine on Google Cloud? Or is there a better service I can try? I don't want to use my own money for this project if I can help it, but I might.

One other thing: monitoring the Google VM, it's only using 12.5% of the CPU. Presumably this is because there are 8 cores and it's only using one (so I should have set up fewer cores). But running the script on my home desktop (Ubuntu 18.04), htop tells me there are 12 python threads and all CPUs on full throttle, so I assumed the Google VM (Debian 10) could do the same?

Steve Petersen
  • 13
  • 2
  • 1) a common misconception is that cloud virtual machines are big powerful computers. Cloud VMs are slices of a big processor. One cloud VM core is roughly equivalent to 1/2 an Intel processor core (a hyper-thread). My desktop is as large as 64 cloud VMs. I run dozens of virtual machines at the same time. – John Hanley Dec 09 '21 at 22:37
  • 2) If your code is written for multiple-processors / multi-threading, then yes you can use all available cores. Since your application running in the cloud is only using one core, you might need to install/update/replace the threading library. Which one, no idea as your question does not include that information. – John Hanley Dec 09 '21 at 22:37
  • Thank you! I did know many cloud services are not especially powerful computers - but I went looking for something that I thought *was* powerful, and the best I could find was Google's "compute engine" (promising name at least!) with a VM designed for enhanced computing. Where *can* I find actually powerful computer time to rent? Or is it just out of the reach of individuals' budgets? – Steve Petersen Dec 10 '21 at 00:36
  • Google has Compute-optimized instance types. However, I think you have a threading issue that needs to be resolved first before you can take full advantage of large instance sizes. – John Hanley Dec 10 '21 at 03:22
  • Thanks again! I can't work out why my desktop is able to use all 12 cores, so I can't replicate it on the VM. I didn't write any fancy multiprocessing script or anything ... it's just got 12 identical-looking processes running somehow. – Steve Petersen Dec 10 '21 at 03:43
  • One item on desktop systems. The OS can move threads between cores. That does not mean you are using all cores, just one core at a time. You might create a question showing your code, the libraries, etc. Someone might provide help on performance. – John Hanley Dec 10 '21 at 06:08
  • Another important difference is the CPU speed. Your local CPU can reach 4+Ghz, in the cloud it's most of the time 2.4Ghz. So, the difference that you observe (2m8s VS 4m36s) makes perfectly sense if you compare the raw CPU performances. The power of the cloud is the parallelization. Write a distributed program is the most efficient, most scalable in the cloud. – guillaume blaquiere Dec 10 '21 at 08:30

0 Answers0