Apache Airflow, Papermill and custom kernels

Question

I currently am attempting to execute jupyter notebooks with apache airflow and papermill.

In order to use my custom enviorments and custom user kernels I am using the BashOperator and running a command in this format

runuser {user} -c 'papermill {input_notebook_path} {output_notebook_path} -k {kernel_name}'

This is what the Dag log looks like after attempting execution: Dag logs

I have been unable to use the python or papermill operator because the kernels/envs that the notebooks use are not installed globally.

Any ideas?

Answer 1

The work around for this is to edit the airflow "Bash Operator" to have a directory you create instead of creating its own for each dag.

Also the folder the output notebook is going to must be 777