0

I'm trying to plot a 2-D image with matplotlib, which expects data points in nested list format. I've got as far as a very neat, idiomatic way to generate this:

zs = [[cost_at(x, y) for x in x_range] for y in y_range]
plt.contourf(x_range, y_range, zs, 1000)

And it works - for small data. However, I now need to do exactly the same thing except for X and Y range too big for the full nested list to fit in memory. It seems to me it should be possible to call the API with lazy lists that hopefully would be adequate substitutes, assuming the library accesses them by iterators.

What's the way to do the above except with lazy lists?

rwallace
  • 31,405
  • 40
  • 123
  • 242
  • What does it happen to the nested list? Does it go very slowly? So you basically would like to optimise the execution time? –  Dec 19 '17 at 20:44
  • @J.C.Rocamonde **_X and Y range too big for the full nested list to fit in memory_** – Bahrom Dec 19 '17 at 20:46
  • "It seems to me it should be possible to call the API with lazy lists" -- what does the documentation for `plt.contourf()` say? – John Coleman Dec 19 '17 at 20:46
  • A lazy version of a list-comprehension would be an equivalent generator expression! – juanpa.arrivillaga Dec 19 '17 at 21:02
  • 1
    In order for a contour plot to be generated, all the data must be known at once. While technically one could of course write a contour algorithm that works with part of the data sourrounding the currently calculated region, this is not how matplotlib works internally. Also in such an algorithm you would not work with rows and columns of data. That being said, the way to create a contour plot is to provide a numpy array. In case the numpy array would be too big for memory, intelligent stitching of the array into smaller pieces may be a way to go. – ImportanceOfBeingErnest Dec 19 '17 at 21:27
  • @ImportanceOfBeingErnest I tried a numpy array. Good news, it works. Bad news, it doesn't reduce memory consumption. Actually it looks like that's because almost all the memory is being used internally in matplotlib. – rwallace Dec 19 '17 at 22:27
  • No, it will not reduce memory, there are the same numbers in that array as in the list of lists. Is the question how to reduce memory consumption when plotting a contour plot? In that case one would start thinking about how much data is actually needed. If the data is that big that it fills the complete memory of a modern computer, one would surely find a way to obtain the same visual result but with much less data. – ImportanceOfBeingErnest Dec 19 '17 at 22:35

1 Answers1

2

In Python, a "lazy list" can be created with a generator expression:

zs = ((cost_at(x, y) for x in x_range) for y in y_range)

By using parentheses instead of square brackets, you create a nested generator expression which provides cost_at(x, y) one at a time instead of calculating them all at once.

Tai
  • 7,684
  • 3
  • 29
  • 49
Hai Vu
  • 37,849
  • 11
  • 66
  • 93
  • Neat solution! For the record, plt.contourf sadly turns out not to accept generators, but this is still a neat solution for other contexts. Thanks! – rwallace Dec 19 '17 at 21:17
  • 1
    Isn't it very confusing to accept an answer which ultimately does not work? That way future readers will think they can use generators to plot contour plots with matplotlib - which isn't the case. – ImportanceOfBeingErnest Dec 19 '17 at 21:28
  • @ImportanceOfBeingErnest That's why I added that comment. It's not like I'm expecting an alternative answer that does work with matplotlib. – rwallace Dec 19 '17 at 22:05