What is causing the `external pointer is not valid` error in `parallel::parSapply`?

Question

I am trying to pass an object (i.e., of R6 class; this particular one) to a number of workers created using parallel::makePSOCKcluster() and I get:

Error in checkForRemoteErrors(val) : 
  one node produced an error: external pointer is not valid

Based on this post by Henrik Bengtsson:

[...] there is a set of object types that cannot be passed on to another R process and be expected to work there.

I want to understand whether the object I am trying to pass falls in this category and, if so, what my options are.

Here is a MRE:

Scenario 1: (working) Creating the model object inside each worker.

(function() {
    # Create cluster.
    cluster <- parallel::makePSOCKcluster(parallel::detectCores() - 1)

    # Stop cluster.
    on.exit(parallel::stopCluster(cluster))

    # Bare minimum data.
    x <- matrix(rnorm(100), 10, 10)
    y <- runif(10)

    # Run operation.
    result <- parallel::parSapply(cluster, c(1), function(i) {
        # The 'osqp' object.
        model <- osqp::osqp(P = crossprod(x), q = -crossprod(x, y), pars = list(verbose = FALSE))

        # Calling the solver.
        return(model$Solve()$x)
    })

    # Inspect result.
    print(result)
})()

Scenario 2: (not working) Creating the model object in the main and passing it to the workers.

(function() {
    # Create cluster.
    cluster <- parallel::makePSOCKcluster(parallel::detectCores() - 1)

    # Stop cluster.
    on.exit(parallel::stopCluster(cluster))

    # Bare minimum data.
    x <- matrix(rnorm(100), 10, 10)
    y <- runif(10)

    # The 'osqp' object.
    model <- osqp::osqp(P = crossprod(x), q = -crossprod(x, y), pars = list(verbose = FALSE))

    # Run operation.
    result <- parallel::parSapply(cluster, c(1), function(i) {
        # Calling the solver.
        return(model$Solve()$x)
    })

    # Inspect result.
    print(result)
})()

Scenario 1 works so it seems I can use osqp inside the workers. But, when instead I create that object outside and pass it to the workers (i.e., Scenario 2), it fails.

To provide a bit more context, I have no control over the model creation. I am receiving an instance created elsewhere and I am only allowed to call a few methods on that instance (e.g., $Update()).

Update 1

It does not seem to be related to the fact that R6 instances are environments. The following still works as intended.

# Create mock model class.
ModelMock <- R6::R6Class("ModelMock",
    public = list(
        Solve = function() {
            return(list(x = "Mocked model output."))
        }
    )
)

(function() {
    # Create cluster.
    cluster <- parallel::makePSOCKcluster(parallel::detectCores() - 1)

    # Stop cluster.
    on.exit(parallel::stopCluster(cluster))

    # The mocked 'osqp' object.
    model <- ModelMock$new()

    # Run operation.
    result <- parallel::parSapply(cluster, c(1), function(i) {
        # Calling the solver.
        return(model$Solve()$x)
    })

    # Inspect result.
    print(result)
})()

I don't see you passing the `model` object to the workers. Make it a parameter of the function or copy it to the workers explicitly. — Roland, Jun 23 '21 at 07:13
@Roland I thought when called within a function `parSapply` serializes the variables in the local environment implicitly (as discussed here: https://stackoverflow.com/a/35857490/5252007). I just tried what you said (i.e., both via `clusterExport` and passing it as an argument to the worker function. Same result. — Mihai, Jun 23 '21 at 07:19
I'm not sure regarding the data structure here. If you copy `model`, do you really copy everything? Or does it need stuff that is in some other environment (that you didn't copy). — Roland, Jun 23 '21 at 07:29
@Roland I think `R6` objects are pointers to environments. I can try to deep clone the object and see if that helps. But, in my experience, I was able to pass `R6` objects to the workers. Let me try this and will report back. — Mihai, Jun 23 '21 at 07:37
@Roland I updated my question (please see above). It doesn't seem to be related to the fact that `model` is an `R6` instance. — Mihai, Jun 23 '21 at 07:46
Look at `environment(model$Solve)`. It contains an environment `private` that contains a pointer `.work`. That external pointer is what the error message refers to. I'm not sure this can be fixed in pure R since the pointer is probably created and managed by compiled code. — Roland, Jun 23 '21 at 09:09
I think you are right, it seems that `.work` is indeed created by an `Rcpp` export (i.e., [this one](https://github.com/osqp/osqp-r/blob/c38b1de60b8fbc2f9648a47e26bfb3223dbe3722/src/RcppExports.cpp#L9-L23)). And, `model$.__enclos_env__$private$.work` is of type `externalptr`. So I can create and use this pointer just fine within a worker, but cannot access it from within if it was created in the main process. This is probably because each `R` process created for the workers comes with its own memory space. This is a bummer. Indeed, not sure what can be done... — Mihai, Jun 23 '21 at 09:23
Some kind of deep copy method would need to be created for the class. — Roland, Jun 23 '21 at 09:34
Thanks @Roland. I updated the question with a summary based on these comments, in case others run into the same issue. — Mihai, Jun 23 '21 at 09:41

Mihai · Answer 1 · 2021-06-23T10:33:15.220

Roland pointed out that environment(model$Solve) contains a private environment that contains an externalptr object .work:

typeof(model$.__enclos_env__$private$.work)  
# "externalptr"

This pointer .work is created using compiled code, i.e., via an Rcpp export (see this export).

It seems that this pointer is managed by compiled code and, as such, I can not use it within the workers. It is fine to call the compiled code and create this pointer from within the workers. What is not fine is to create this pointer in another process (i.e., the main process) and then pass it to the worker processes. This is probably because each worker is created as a separate R process, with its own memory space.

Not ideal, but what might work, as Roland pointed out, is to somehow copy the data at that pointer and ensure it is passed to the workers. But this likely requires an Rcpp implementation.

For those interested in this particular package, you may also follow this issue on GitHub.

What is causing the `external pointer is not valid` error in `parallel::parSapply`?

1 Answers1