5

I'm currently working with pandas and ipython. Since pandas dataframes are copied when you perform operations with it, my memory usage increases by 500 mb with every cell. I believe it's because the data gets stored in the Out variable, since this doesn't happen with the default python interpreter.

How do I disable the Out variable?

Thomas K
  • 39,200
  • 7
  • 84
  • 86
parchment
  • 4,063
  • 1
  • 19
  • 30
  • Can you provide an example of your code? I thought operations could be either `inplace=True` or not... – IanS Jun 14 '16 at 10:38
  • @IanS I'm using operators like `+, /` etc. Even if I'm not, I'd rather not use `infile` for everything. – parchment Jun 14 '16 at 11:15

1 Answers1

8

The first option you have is to avoid producing output. If you don't really need to see the intermediate results just avoid them and put all the computations in a single cell.

If you need to actually display that data you can use InteractiveShell.cache_size option to set a maximum size for the cache. Setting this value to 0 disables caching.

To do so you have to create a file called ipython_config.py (or ipython_notebook_config.py) under your ~/.ipython/profile_default directory with the contents:

c = get_config()

c.InteractiveShell.cache_size = 0

After that you'll see:

In [1]: 1
Out[1]: 1

In [2]: Out[1]
---------------------------------------------------------------------------
KeyError                                  Traceback (most recent call last)
<ipython-input-2-d74cffe9cfe3> in <module>()
----> 1 Out[1]

KeyError: 1

You can also create different profiles for ipython using the command ipython profile create <name>. This will create a new profile under ~/.ipython/profile_<name> with a default configuration file. You can then launch ipython using the --profile <name> option to load that profile.

Alternatively you can use the %reset out magic to reset the output cache or use the %xdel magic to delete a specific object:

In [1]: 1
Out[1]: 1

In [2]: 2
Out[2]: 2

In [3]: %reset out

Once deleted, variables cannot be recovered. Proceed (y/[n])? y
Flushing output cache (2 entries)

In [4]: Out[1]
---------------------------------------------------------------------------
KeyError                                  Traceback (most recent call last)
<ipython-input-4-d74cffe9cfe3> in <module>()
----> 1 Out[1]

KeyError: 1

In [5]: 1
Out[5]: 1

In [6]: 2
Out[6]: 2

In [7]: v = Out[5]

In [8]: %xdel v    # requires a variable name, so you cannot write %xdel Out[5]

In [9]: Out[5]     # xdel removes the value of v from Out and other caches
---------------------------------------------------------------------------
KeyError                                  Traceback (most recent call last)
<ipython-input-9-573c4eba9654> in <module>()
----> 1 Out[5]

KeyError: 5
Bakuriu
  • 98,325
  • 22
  • 197
  • 231
  • You say you're having trouble using the cache_size setting - where are you trying to set it? That is an IPython config option, not a Jupyter config option, so it still goes under `~/.ipython/profile_default/`. – Thomas K Jun 14 '16 at 17:18
  • @ThomasK I tried using the `%config` magic and passing the option `--InteractiveShell.cache_size=0` from the command line. I know that you can create profiles, but I believe there must be at least a way to do this without having to create one or change the default profile. Anyway if that's the only way I will edit it in the answer. – Bakuriu Jun 14 '16 at 17:22
  • You can set it temporarily with the `%config` magic or the command line option. Setting it permanently requires doing so in a profile. – Thomas K Jun 15 '16 at 11:31
  • @ThomasK Yeah, the problem is that using `%config` and the command line option doesn't seem to work at all. – Bakuriu Jun 15 '16 at 11:43
  • Oh, right. It might be something that only has an effect when the application is starting. – Thomas K Jun 15 '16 at 11:50
  • You can use `%reset -f in out`. It looks like `_` is classified under the "in" category, even if it's an output, and you could want to remove it. – creanion Oct 29 '21 at 18:13