SCP failure running pyCOMPSs application

Question

I have an error running an application implemented with pyCOMPSs. The application was working well but I have done some changes in the application and it has stopped to work. This is the stack I have got from the application:

--- START OF NESTED EXCEPTION STACK TRACE ---
*** stack trace of GATInvocationException
    --- START OF NESTED EXCEPTION STACK TRACE ---
    *** stack trace of GATInvocationException
        --- START OF NESTED EXCEPTION STACK TRACE ---
        *** stack trace of IOException
        java.io.IOException: Error during SCP transfer.
        at com.trilead.ssh2.SCPClient.put(SCPClient.java:592)
        at com.trilead.ssh2.SCPClient.put(SCPClient.java:456)
        at org.gridlab.gat.io.cpi.sshtrilead.SshTrileadFileAdaptor.put(SshTrileadFileAdaptor.java:418)
        ...
Caused by: java.io.IOException: Remote scp terminated with error (scp: /gpfs/scratch/wdir_pycompss: No such file or directory).
        at com.trilead.ssh2.SCPClient.readResponse(SCPClient.java:59)
        at com.trilead.ssh2.SCPClient.sendFiles(SCPClient.java:166)
        at com.trilead.ssh2.SCPClient.put(SCPClient.java:588)
        ... 15 more

        --- END OF NESTED EXCEPTION STACK TRACE ---
    *** stack trace of MethodNotApplicableException
LocalFile: cannot copy to remote destination
    --- END OF NESTED EXCEPTION STACK TRACE ---
--- END OF NESTED EXCEPTION STACK TRACE ---

It says that a folder is not found but it is defined as workingDir in the project.xml and the application is creating this folder. Am I doing something wrong?

Does this folder exists before starting the execution? Is it created by a task? — COMP Superscalar, Mar 21 '16 at 15:01

score 4 · Accepted Answer · answered Mar 21 '16 at 15:17

The working dir defined in the project.xml must exist before the execution of the defined tasks because it is the place where the remote tasks will be executed. Depending on the communication adaptor used by the (py)COMPSs runtime, it could be created if it doesn't exists. From the stack you sent I can infer you are running with the GAT adaptor and it is not the case.

The wrong thing you are doing is that a task is creating the working dir, and it is not allowed.The working dir folder should be created at deployment time. In this case, the runtime is trying to run the task inside the folder that should be created during the execution. Inside the code of a tasks can create a folder as a sandbox with a relative path and it will be created inside the working dir, but remember that it will not persist from one task to another even if they are executed in the same worker node.

SCP failure running pyCOMPSs application

1 Answers1