Calling numpy on parallel processors in IJulia notebook

Question

I want to run a simple code in the IJulia notebook which uses the python library numpy. I call numpy with PyCall:

using PyCall

@pyimport numpy as np

This works fine. I then want to split this up to several processors. I add the processors:

addprocs(4)

Then, I run N/proc iterations for a function f, where proc is my number of processors. I split the load evenly between the four processors on my computer:

n=round(Int,N/proc);

proc_sum = @parallel (+) for i=1:proc

        f(n)

end

return proc_sum / proc

Without numpy, this works fine. However, when I try to split the code with numpy to different processors, I get the error

ERROR (unhandled task failure): On worker 3:

UndefVarError: np not defined

Is there any way to have numpy work on the other processors? Note that I have Julia 0.5.2, and I have Canopy. I know there have been issues reported before with PyCall and Canopy, but I would greatly prefer keeping Canopy on my machine.

Do `add_procs` before all other things, and do `@everywhere(@pyimport numpy as np)` — Frames Catherine White, May 23 '17 at 03:14

Pietro Vertechi · Accepted Answer · 2017-05-23T17:30:11.250

2

To further expand on what has been said already, everything you need should be loaded in all the processes. E. g. :

addprocs(4) @everywhere using PyCall @everywhere @pyimport numpy as np

What you wrote errored because all processes tried to use @pyimport but only the main process had PyCall loaded. If you require many packages to do your computations, maybe the easier is to do all the loading in one script, i.e. load_modules.jl and then simply run

addprocs(4) @everywhere include("load_modules.jl")

EDIT: It seems that using is not very robust with @everywhere (fixed on Julia 0.6, see here). What seems to work better is:

addprocs(4) import PyCall @everywhere using PyCall @everywhere @pyimport numpy as np

edited May 23 '17 at 17:30

answered May 23 '17 at 16:07

Pietro Vertechi

91
2

I did use @everywhere when "using PyCall", but then I get the error "WARNING: replacing module PyCall" repeated four times. – Joshuah Heath May 23 '17 at 16:10
Are you on Julia 0.6 ? It's probably this issue: https://github.com/JuliaLang/julia/issues/19187 . It's marked as solved, so it'll stop happening in the next release. – Pietro Vertechi May 23 '17 at 17:19
No, I am on Julia 0.5.2 – Joshuah Heath May 23 '17 at 17:22
I see. It's solved here: https://github.com/JuliaLang/julia/pull/21718. You can simply type in their solution and it should work, I'll update the original answer. – Pietro Vertechi May 23 '17 at 17:28

Tasos Papastylianou · Answer 2 · 2017-05-24T21:53:44.060

0

I'm expanding on Lyndon's comment to provide a more comprehensive answer.

As per the documentation, processes are independent, and thus rely on their own, independent workspace. Therefore, any functions, modules or variables that will be needed by a process need to be made available to that process first.

If you want to make something available to all existing processes, you can use the @everywhere macro; clearly, to make something available to "all existing processes", these processes need to have been created first.

So:

addprocs(4); # create 4 extra processes (i.e. workers); this is in addition
             # to the main process that handles the REPL

@everywhere import Pycall
@everywhere PyCall.@pyimport numpy as np # load module on _all_ 5 active processes

edited May 24 '17 at 21:53

answered May 23 '17 at 07:07

Tasos Papastylianou

21,371
2
28
57

I still have an error: @everywhere(@pyimport numpy as np) tells me that "@pyimport not defined" – Joshuah Heath May 23 '17 at 14:53
well, it sounds like @pyimport is not defined then, doesn't it! :) Import it in the process before using it. – Tasos Papastylianou May 24 '17 at 08:27

Calling numpy on parallel processors in IJulia notebook

2 Answers2

Linked