Nextflow is running executable inside docker container volume, but can't find it

Question

I'm working on a nextflow pipeline, that uses a different docker container in each process.

It looks something like this:

params.exeA = "$projectDir/bin/exeA" //is inside docker volume of containerA
params.exeB = "$projectDir/bin/1.0/exeB" //is docker volume of containerB
params.inA = "$projectDir/TestData/A/*"
params.outA = "$projectDir/TestData/A_Out/"
params.outB = "$projectDir/TestData/B_Out/*"

process A{

  container 'link.to.container.registryA'
    
  input:
     path t_path
     path out_path_A
     //some more parameters


  output:
     path "${out_path_A}"

  script:
  """
  ${params.exeA} --input "${t_path}" --output "${out_path_A}" //some more parameters

  """
 
}
process B{

  container 'link.to.container.registryB'
    
  input:
     path resultOfA

  output:
     path ...

  script:
  """
  ${params.exeB} --someVal "${params.someVal}" --pathA  "${resultOfA}"

  """
 
}
workflow {

files_ch = Channel.fromPath(params.input_path, type: 'dir')
a_ch = A(files_ch, params.outA)
b_ch = B(a_ch, params.outB)
}

When I try to run it, it throws the error:

Error executing process > 'A (1)'

Caused by:
  Process `A (1)` terminated with an error exit status (127)

Command executed:

  /home/projects/myProject/bin/exeA ...(some parameters)

Command exit status:
  127
Command output:
  (empty)

Command error:
  .command.sh: line 2: /home/projects/myProject/bin/exeA: No such file or directory

Confusingly, it still created the output-File of exeA I was expecting inside a workingDir. But the workflow stops at that point and doesn't continue with process B. I don't know how this is possible... Nextflow can't find the executable, but was still able to call it???

I already tested both processes individually and had the same error.

Steve · Answer 1 · 2023-04-14T00:31:32.200

The problem is that exeA is not being localized inside your process working directory (the same is true for exeB). To fix this, you would need to pass in the executable by declaring it in your input block (using the path qualifier). But you almost never need or want to do this. A better approach would be to add the executable to your PATH environment variable inside the Docker container, which would allow you to call it without having to specify an absolute path to it. This would also help ensure your workflow is portable. Note that you can also use the publishDir directive to publish the process output files to a 'results' folder. I think all you need is something like:

params.input_path = "${projectDir}/TestData/A/*"
params.someVal = 'my_string'

params.outdir = './results'

process A {

    container 'link.to.container.registryA'

    publishDir "${params.outdir}/A", mode: 'copy'

    input:
    path t_path
    path out_path_A

    output:
    path "A_Out"

    """
    exeA \\
        --input "${t_path}" \\
        --output A_Out
    """
}

process B {

    container 'link.to.container.registryB'

    publishDir "${params.outdir}/B", mode: 'copy'

    input:
    path resultOfA

    output:
    path "B_Out"

    """
    exeB \\
        --someVal "${params.someVal}" \\
        --pathA "${resultOfA}" \\
        --output B_Out
    """
}

workflow {

    files_ch = Channel.fromPath( params.input_path, type: 'dir' )

    a_ch = A( files_ch )
    b_ch = B( a_ch )
}

Nextflow is running executable inside docker container volume, but can't find it

1 Answers1