2

I want to pass on all the parameters in one variable. To make this happen, I am passing a dictionary in my Python client for Papermill using:

params = {"aviral":"srivastava"}
pm.execute_notebook( 'path/to/input.ipynb', 'path/to/output.ipynb', parameters = params )

Now, I want to do run the corresponding command from bash. In the document it states: 

papermill local/input.ipynb s3://bkt/output.ipynb -p alpha 0.6 -p l1_ratio 0.1

But nowhere is given how to pass on a dictionary.

I tried many permutations and combinations like:

papermill test_params.ipynb params_output.ipynb -p params {"aviral":"srivastava"}

papermill test_params.ipynb params_output.ipynb -p params '{"aviral":"srivastava"}'

I also read to pass a dict in the format: '{"aviral":"srivastava"}' and then json.loads would do the work. But I am not having the liberty for converting the input. It has to be used right away in the script.

Just a minimal script to actually check the type of the variable passed:

params = {} # I tag this with `parameter`
print(type(params))
print(params)

Yes, I see the params variable got changed when running with papermill but the value passed is a complete string.

Aviral Srivastava
  • 4,058
  • 8
  • 29
  • 81
  • 2
    In general you can only pass text strings on the command-line, that's an architectural feature, nothing to do with the language or the program you are calling. Of course the program is free to *interpret* those text strings any way it wants. What you could do is look at the source code of the python module and see how it does it. – cdarke May 27 '19 at 09:48

2 Answers2

1

Since, in your case, you can not modify the template notebook directly to parse the string input, how about having a python script that you call which then executes papermill? That allows you to parse the string into a python dictionary and then pass that to the papermill execution. You could also pass through the notebook names in the same way.

cwalvoort
  • 1,851
  • 1
  • 18
  • 19
1

Have you tried using a YAML string or file? 

$papermill local/input.ipynb s3://bkt/output.ipynb -f parameters.yaml

From the Docs:

"When using YAML to pass arguments, through -y, -b or -f, parameter values can be arrays or dictionaries"

https://papermill.readthedocs.io/en/latest/usage-execute.html

PKL
  • 95
  • 7
  • Also note that a dict can be converted to a (temporary) yaml file and consequently to base64-formatted string using `PARAMS=$(cat params.yaml| base64)`. You'd then call `papermill ... -b $PARAMS` (quoting from the docs). – Vojta F Feb 07 '23 at 11:17