Octave parallel package: User-defined function call issues

Question

I have trouble calling a user-defined function with pararrayfun (likewise for parcellfun). When I execute the following code:

pkg load parallel

function retval = mul(x,y)
  retval = x*y;
endfunction

vector_x = 1:2^3;
vector_y = 1:2^3;

vector_z = pararrayfun(nproc, @(x,y) mul(x,y), vector_x, vector_y)
vector_z = pararrayfun(nproc, @(x,y) x*y, vector_x, vector_y)

I get the following output:

vector_z =

  -1  -1  -1  -1  -1  -1  -1  -1

vector_z =

    1    4    9   16   25   36   49   64

That is, the call to the user-defined function does not seem to work, whereas the same as an anonymous function is working.

The machine is x86_64 with Debian bullseye and 5.10.0-1-amd64 kernel. Octave's version is 6.1.1~hg.2020.12.27-1. The pkg list command gives me:

Package Name  | Version | Installation directory
--------------+---------+-----------------------
   dataframe  |   1.2.0 | /usr/share/octave/packages/dataframe-1.2.0
    parallel *|   4.0.0 | /usr/share/octave/packages/parallel-4.0.0
      struct *|  1.0.16 | /usr/share/octave/packages/struct-1.0.16

Funny thing is that the same code works flawless on armv7l with Debian buster and 4.14.150-odroidxu4 kernel. That is the call to the user-defined function and the anonymous function produce the output:

parcellfun: 8/8 jobs done
vector_z =

    1    4    9   16   25   36   49   64

parcellfun: 8/8 jobs done
vector_z =

    1    4    9   16   25   36   49   64

On that machine Octave's version is 4.4.1 and pkg list gives:

Package Name  | Version | Installation directory
--------------+---------+-----------------------
   dataframe  |   1.2.0 | /usr/share/octave/packages/dataframe-1.2.0
    parallel *|   3.1.3 | /usr/share/octave/packages/parallel-3.1.3
      struct *|  1.0.15 | /usr/share/octave/packages/struct-1.0.15

What is wrong and how can I fix this behavior?

Answer 1

This is probably a bug, but do note that the new version of parallel has introduced a few limitations as per its documentation (also see the latest release news) which may relate to what's happening here.

Having said that, I want to clarify this sentence:

the call to the user-defined funtion does not seem to work, whereas the same as an anonymous function is working.

That's not what's happening. You're passing an anonymous function in both cases. It's just that the first calls mul inside, and the second calls mtimes.

As for your error (bug?) this may have something to do with mul being a 'command-line' function. It's not clear from the documentation if command-line functions are a limitation and this is simply an oversight in the docs, or if ill-treatment of command-line functions is a genuine bug. I think if you put it in its own file it should work fine. (and equally, if you do, it's worth passing it as a handle directly, rather than wrapping it inside another anonymous function).

Having said that, I think the -1's you see are basically "error returns" from inside pararrayfun's guts. The reason for this is the following: if instead of creating mul as a command-line function, you make it an anonymous function:

mul = @(x,y) x * y

Observe what the three calls below return:

x = pararrayfun( nproc, @(x,y) mul(x,y), vector_x, vector_y )   # now it works as expected.
x = pararrayfun( nproc, mul, vector_x, vector_y )   # same: mul is a valid handle expecting two inputs
x = pararrayfun( nproc, @mul, vector_x, vector_y )  # x=-1,-1,-1,-1,-1,-1,-1,-1

If you had tried the last command using normal array fun, you would have seen an error relating to the fact that you accidentally passed @mul instead of mul, when mul is a proper handle. In pararrayfun, it just does the calculation, and presumably -1 was the return value from an internal error.

I don't know exactly why a command-line function fails, but presumably it has something to do with the fact that pararrayfun creates separate octave instances under the hood, which need access to all function definitions, and perhaps command-line functions cannot be transfered / compiled in the new instance as easily as in the parent instance, because of the way they are created / compiled in the current session.

In any case, I think you'll solve your problem if instead of a command-line function definition, you create an external function or (if dealing with simple enough functions) a handle to an anonymous function.

However, I would still submit a bug to the octave bug tracker to help the project :)