Saving multiple matplotlib figures with multiprocessing using jupyter notebook

Question

I try to use multiprocess to plot and save many figures. Using code in Saving multiple matplotlib figures with multiprocessing, I get an error:

NameError: name 'plt' is not defined

It works if I use the following code with library import inside the function and with the multiprocess fork of multiprocessing:

# taken from https://stackoverflow.com/questions/24866070/saving-multiple-matplotlib-figures-with-multiprocessing

from multiprocess import Pool

def do_plot(number):
    import numpy.random as random
    import matplotlib.pyplot as plt
    
    fig = plt.figure(number)

    a = random.sample(1000)
    b = random.sample(1000)

    # generate random data
    plt.scatter(a, b)

    plt.savefig("%03d.jpg" % (number,))
    plt.close()

    print("Done ", number)

pool = Pool(4)
pool.map(do_plot, range(4))

However, my real function depends on a big dataframe defined outside of it:

import pandas as pd 
df = pd.DataFrame({'A':[1,2,3],'B':[1,2,3]})

def do_plot(number):
    import numpy.random as random
    import matplotlib.pyplot as plt
    
    fig = plt.figure(number)

    a = df['A']
    b = df['B']

    # generate random data
    plt.scatter(a, b)

    plt.savefig("%03d.jpg" % (number,))
    plt.close()

    print("Done ", number)

pool = Pool(4)
pool.map(do_plot, range(4))

which returns:

NameError: name 'df' is not defined

What should I do to use a dataframe and other objects defined outside the do_plot function ?

Please clarify your specific problem or provide additional details to highlight exactly what you need. As it's currently written, it's hard to tell exactly what you're asking. — Community, Oct 17 '22 at 09:03

score 0 · Answer 1 · answered Oct 17 '22 at 10:10

In matplotlib saving figures in multiprocessing doesn't work instead saving the full figure works. Here is a solution below:

import matplotlib.pyplot as plt
from multiprocessing import Pool
import random

def do_plot(ax):
    a = random.sample(range(1000),100)
    b = random.sample(range(1000),100)

    ax.scatter(a, b)

plots = 4
fig, ax = plt.subplots(plots)
pool = Pool(plots)
pool.map(do_plot, ax)
fig.savefig('full_figure.png')

Output:

Saving multiple matplotlib figures with multiprocessing using jupyter notebook

1 Answers1