Overriding Nextflow Parameters with Commandline Arguments

Question

Given the following nextflow.config:

google {
  project = "cool-project"
  region = "europe-west4"
            
  lifeSciences {
    bootDiskSize = "200 GB"
    debug = true
    preemptible = true
  }
}

Is it possible to override one or more of those settings using command line arguments. For example, if I wanted to specify that no preemptible machines should be used, can I do the following:

nextflow run main.nf -c nextflow.config --google.lifeSciences.preemptible false

?

score 5 · Accepted Answer · answered Feb 25 '21 at 01:59

Overriding pipeline parameters can be done using Nextflow's command line interface by prefixing the parameter name with a double dash. For example, put the following in a file called 'test.nf':

#!/usr/bin/env nextflow

params.greeting = 'Hello'

names = Channel.of( "foo", "bar", "baz" )

process greet {

    input:
    val name from names

    output:
    stdout result

    """
    echo "${params.greeting} ${name}"
    """
}

result.view { it.trim() }

And run it using:

nextflow run -ansi-log false test.nf --greeting 'Bonjour'

Results:

N E X T F L O W  ~  version 20.10.0
Launching `test.nf` [backstabbing_cajal] - revision: 431ef92cef
[46/22b4f0] Submitted process > greet (1)
[ca/32992c] Submitted process > greet (3)
[6e/5880b0] Submitted process > greet (2)
Bonjour bar
Bonjour foo
Bonjour baz

This works fine for pipeline params, but AFAIK there's no way to directly override executor config like you describe on the command line. You can however, just parameterize these values and set them on the command line like described above. For example, in your nextflow.config:

params {

  gc_region = false
  gc_preemptible = true

  ...
}

profiles {

  'test' {
    includeConfig 'conf/test.config'
  }

  'google' {
    includeConfig 'conf/google.config'
  }

  ...
}

And in a file called 'conf/google.config':

google {
  project = "cool-project"
  region = params.gc_region
            
  lifeSciences {
    bootDiskSize = "200 GB"
    debug = true
    preemptible = params.gc_preemptible
  }
}

Then you should be able to override these in the usual way:

nextflow run main.nf -profile google --gc_region "europe-west4" --gc_preemptible false

Note that you can also specify multiple configuration profiles by separating the profile names with a comma:

nextflow run main.nf -profile google,test ...

So just to be clear: if no config file is involved, say e.g. to override the **default** singularity.enabled from false to true, this is not possible? — mths, Feb 16 '23 at 10:47
@mths But whenever you run a pipeline, Nextflow will, by default, look for configuration files in the current directory, the project directory and in your $NXF_HOME or $HOME folder. AFAIK, setting `singularity.enabled = true` can only be done inside a configuration file (using `-with-singularity ` I think also enables Singularity, but that's another story). To enable Singularity from the command line, the usual way would be to include a 'singularity' profile with `singularity.enabled = true` in the profile's scope. — Steve, Feb 16 '23 at 11:28
worth noting that you can also override many params values at once if you use a separate JSON file and the command line arg `-params-file myparams.json`. Since this allows you to use JSON notation, you are even able to use more complex datatypes such as native `map` and `array` objects for your params values, allowing you to forgo something like a samplesheet.csv, if you want — user5359531, May 19 '23 at 19:01

Overriding Nextflow Parameters with Commandline Arguments

1 Answers1