5

I have a python script I hope to do roughly this:

  1. calls some particle positions into an array

  2. runs algorithm over all 512^3 positions to distribute them to an NxNxN matrix

  3. feed that matrix back to python

  4. use plotting in python to visualise matrix (i.e. mayavi)

First I have to write it in serial but ideally I want to parrallelize step 2 to speed up computation. What tools/strategy might get me started. I know Python and Fortran well but not much about how to connect the two for my particular problem. At the moment I am doing everything in Fortran then loading my python program - I want to do it all at once.I've heard of py2f but I want to get experienced people's opinions before I go down one particular rabbit hole. Thanks

Edit: The thing I want to make parallel is 'embarrassingly parallel' in that is is just a loop of N particles and I want to get through that loop as quickly as possible.

Griff
  • 2,064
  • 5
  • 31
  • 47

3 Answers3

5

You have two basic options for binding. First to use f2py, the other to use interoperability with C in your Fortran and bind using Cython. Tutorial for f2py is here. It is not at all difficult, there are some directives for f2py to place to your Fortran code, but often they are not needed.

For parallelization, the first approach to use is probably OpenMP, if parallelization on a single machine is enough for you. It uses threads and is easy to use for loops with embarrassing parallelism. Just make sure you do not write to any global variables in the threads and if yes, use synchronization directives for that.

Vladimir F Героям слава
  • 57,977
  • 4
  • 76
  • 119
  • I personally love `f2py` -- as far as parallelizing on a single machine goes, it doesn't get a whole lot easier than python's `multiprocessing`. You don't need to worry about thread-safety as everything is run as a separate process. – mgilson Dec 13 '12 at 19:38
  • That's what I like on coarrays (and MPI, but that is not so straightforward). I actually use Python only for postprocessing, I did not find myself to be any more productive than in Fortran for computational code. Partly because needs to often switch between NumPy and Fortran array indexing conventions. In this regards I consider using Python as a step back in comparison with Chapel. – Vladimir F Героям слава Dec 13 '12 at 20:09
  • If I have numpy - I have f2py? I am trying there examples but I get compiling errors. – Griff Dec 13 '12 at 21:14
  • Well, I don't know which platform and Python distribution you use. On Linux most probably yes, just check your package manager. On Windows, honestly, I don't know. – Vladimir F Героям слава Dec 13 '12 at 22:23
  • @VladimirF do you know of a very simple tutorial for the Cython option? I've soured on f2py... – bdforbes Dec 13 '12 at 23:16
2

An alternative approach to VladimirF's suggestion, could be to set up the two parts as a client server construct, where your Python part could talk to the Fortran part using sockets. Though this comes with the burden to implement some protocol for the interaction, it has the advantage, that you get a clean separation and can even go on running them on different machines with an interaction over the network.

In fact, with this approach you even could do the embarrassing parallel part, by spawning as many instances of the Fortran application as needed and feed them all with different data.

haraldkl
  • 3,809
  • 26
  • 44
1

More an answer than a comment in the same direction as @haraldkl - I am currently using sockets to send data from Fortran to Ptyhon and viceversa. A minimal working example can be found in this repo. Sockets in Fortran are a bit tricky since there is no built-in library supporting them and you need to use a wrapper around the C library. Still, works great.

My idea is to incorporate this strategy in a bigger parallel Fortran code I have even though I have not yet figured out that part. Probably threading the sockets in Python and using multiple ports could work.

b-fg
  • 3,959
  • 2
  • 28
  • 44