0

Using LSF, I have submitted a Python job using -n N where N>1. This means it will use multiple cores, which may or may not be on the same node. I have not written any explicit code for inter-process communication, but I do use libraries that can make use of multiple cores, such as numpy, scipy, and numexpr.

I'm confused because LSF tells me my script is distributed over multiple nodes, i.e. different physical machines, but my code does not take this into account. What is LSF actually doing in this case, and what is the practical implication if it uses multiple cores on different machines, rather than all on the same node?

gerrit
  • 24,025
  • 17
  • 97
  • 170
  • A "python task" will only start one interpreter on one machine unless the "task" (script) itself starts more interpreters. If the script needs to start interpreters on other nodes, it'll need addresses and instructions for how to access and execute code on those machines. Whether a "task" communicates with other processes (local and non-local) depends entirely on how the task is coded. The performance (and even correctness) of distributing work onto many nodes depends on the code. – Bi Rico Aug 18 '16 at 18:21
  • @BiRico As stated in the question, I'm doing none of that. The question is, what does LSF do/mean in this case? I've edited the question, hopefully it's more clear; I believe the question is not broad. – gerrit Aug 18 '16 at 19:03
  • I'm not the close vote, that's someone else. LSF is making recourses available to your script, which the script is ignoring. (Numpy can sometimes use multiple cores on a single machine if it's configured properly). – Bi Rico Aug 18 '16 at 19:08
  • See http://stackoverflow.com/q/32923464/252489. Nothing specific about python, LSF does the same for every job. – Gowtham Oct 31 '16 at 19:04

0 Answers0