2

I've been using Jupyter Notebooks for my data analysis and loved it. Now I'm in the process of cleaning up the scripts and documenting everything. The analysis is a "once-through" with static input and static output so no continuous running scripts. I've been using nbconvert to create .py scripts based on my notebooks. There is only one challenge: What to do with jupyter magic commands? (e.g. !mkdir -p /foo/bar)

I used command line commands such as aws cli and some ETL using standard linux commands. I know I can use solutions such as python's subprocess, split python and bash or use native python libraries such as boto. Is there a better solution that does not require rewriting code?

The goal is to run the converted notebook (to .py script) from a command line or in other scripts.

Bhargav Rao
  • 50,140
  • 28
  • 121
  • 140
Rutger Hofste
  • 4,073
  • 3
  • 33
  • 44
  • I assume 'for production' here would mean speed is imperative. However, for the benefit of others that may find this post and just are concerned with running the code without Jupyter: an option is you can leave in lines like `!mkdir -p /foo/bar` and change the extension to `.ipy` & use `ipython` to run it on the command line. It can though be slow as essentially those lines with exclamation points get sent out to separate shell each time. For pure Python (*faster!*) You do have to take time to convert. The use of `os.system()` and the sh module can often make things easier without subprocess. – Wayne Jan 25 '22 at 18:05

0 Answers0