Stack Overflow
Stack Overflow

Questions tagged [papermill]

Papermill is a tool for parameterizing and executing Jupyter Notebooks.

Help / Support / How do I?

For help and support or technical, please submit your questions to Stack Overflow: https://stackoverflow.com/questions/tagged/papermill.

Also check out the papermill FAQ before asking a question.

When looking for info, especially build info, one needs to check in the GitHub site.

Report Issues

Please report bugs, feature requests and installation / compatibility issues on the papermill issues tracker on GitHub. If you need help with using papermill, please do not use the issue tracker for that. Instead, direct your questions to Stack Overflow.

Documentation

Take a look at the Papermill documentation on ReadTheDocs.

72 questions
3
votes
5 answers

ETL in Airflow aided by Jupyter Notebooks and Papermill

So my issue is that I build ETL pipelines in Airflow, but really develop and test the Extract, Transform and Load functions in Jupyter notebooks first. So I end up copy-pasting back and forth all the time, between my Airflow Python operator code and…
rimkashox
  • 978
  • 9
  • 14
3
votes
0 answers

Using papermill CLI how to finish earlier a jupyter notebook if certain conditions are not met?

Papermill throws exception if I need to finish the code earlier. For example with quit(): nbclient.exceptions.DeadKernelError: Kernel died I would like to know if there is a way to instruct the code exit without papermill throwing the…
Luis R. Gonzalez
  • 358
  • 3
  • 16
2
votes
1 answer

Papermill Run Errors in Github Actions with AttributeError

I am running my Python Notebook as part of my GitHub Action CI and it worked for quite some time. But today it stopped working complaining about this error: Input Notebook: 03_Pcap.ipynb Output Notebook: /tmp/ipynb/03_Pcap.ipynb Traceback (most…
Daniel Habenicht
  • 2,133
  • 1
  • 15
  • 36
2
votes
1 answer

Add papermill parameters to notebook manually

Is is possible to add papermill parameters to a jupyter notebook manually, e.g. in an editor? Is it possible to add papermill parameters to a .py file and have them persist when converted to a .pynb file? Context: I am working on running jupyter…
Bryce
  • 197
  • 1
  • 1
  • 13
2
votes
1 answer

Airflow Papermill operator: task externally skipped after 60 minutes

I am using Airflow in a Docker container. I run a DAG with multiple Jupyter notebooks. I have the following error everytime after 60 minutes: [2021-08-22 09:15:15,650] {local_task_job.py:198} WARNING - State of this instance has been externally set…
delalma
  • 838
  • 3
  • 12
  • 24
2
votes
0 answers

What does "ERROR - Can't compile non template nodes" in Airflow with the PapermillOperator mean?

I'm running my first Papermill task, which is to run a Jupyter notebook. Here is the operator: t5 = PapermillOperator( task_id="run_example_notebook", input_nb=path2, output_nb=path2, parameters={"msgs": "Ran from Airflow at {{…
IamWarmduscher
  • 875
  • 2
  • 10
  • 27
2
votes
2 answers

Execute a Jupyter notebook with papermill and output a unique filename

I'd like to use papermill as part of a data science workflow to record experiments. The key idea is that the output notebook should be stored as a unique artifact -- an immutable record of the experiment. As such, I want the output filename to be a…
Jonathan
  • 1,287
  • 14
  • 17
2
votes
2 answers

How to parameterise a Python dictionary in papermill?

I want to pass on all the parameters in one variable. To make this happen, I am passing a dictionary in my Python client for Papermill using: params = {"aviral":"srivastava"} pm.execute_notebook( 'path/to/input.ipynb', 'path/to/output.ipynb',…
Aviral Srivastava
  • 4,058
  • 8
  • 29
  • 81
2
votes
0 answers

How to run papermill on a remote server?

I want to run a papermill Python script to execute a .ipynb file. As of now, it runs on local but I want to run it on another server, say another Docker container. I tried reading the…
Aviral Srivastava
  • 4,058
  • 8
  • 29
  • 81
2
votes
2 answers

Papermill + Scrapbook : is it possible to reglue text in papermill notebook?

I use a Papermill notebook ("orchester_notebook") to launch multiple times a "training_notebook" with different sets of parameters. For each set of parameters, a new "result_notebook" is saved with printed results (text) and figures. I know how to…
Brigitte Maillère
  • 847
  • 1
  • 9
  • 27
2
votes
1 answer

Bash script with multiple papermill commands does not fail on notebook errors

I have a refresh_data.sh file which contains multiple papermill commands, for example: papermill notebook_1.ipynb output_1.ipynb -p start "2017-12-01" -p date "2017-12-31" papermill notebook_2.ipynb output_2.ipynb -p start "2018-01-01" -p date…
Giacomo
  • 1,796
  • 1
  • 24
  • 35
1
vote
1 answer

Getting+setting higher context/scope variable python

Is it possible to get+set a higher context/scope's variables by passing the context itself? I'd like to: # main_jupyter.ipynb from my_module import MyModule # class var_to_access = 'foo' my_module.set_var_to_bar(ipynb_context) #…
nitsujri
  • 1,448
  • 2
  • 16
  • 29
1
vote
1 answer

aws auto-stop-idle does not detect papermill

I am using papermill to parametrize jupyter notebook deployed on AWS Sagemaker. I also used this lifestyle configuration that will auto shutdown if there are no running/idle notebooks. Unfortunately, it does not detect the Papermill process and…
Led
  • 662
  • 1
  • 19
  • 41
1
vote
1 answer

Papermill With Cronjob: Error: papermill is a directory and can't find filepath

I am trying to execute a jupyter notebook with crontab and papermill and cannot find the correct filepath to give the crontab to run papermill. Here is what I have currently #on one line: 0 * * * * /path-to-papermill/papermill…
megansorel
  • 21
  • 3
1
vote
1 answer

Trying to run a jupyter notebook from AWS Lambda - getting: "errorMessage": "No such kernel named python3"

I am trying to run a Jupyter notebook from Lambda using the following code which uses papermill: import os import boto3 import subprocess # to add paths import sys # pip install custom package to /tmp/ and add to path subprocess.call('pip install…
RHD
  • 11
  • 2
1
2
3 4 5