Iterating over arrays on disk similar to ImageDataGenerator

Question

I have 70'000 2D numpy arrays on which I would like to train a CNN network using Keras. Holding them in memory would be an option but would consume a lot of memory. Thus, I would like to save the matrices on disk and load them on runtime. One option would be to use ImageDataGenerator. The problem is that it only can read images.

I would like to store the arrays not as images because when I would save them as (grayscale) images then the values of arrays are changed (normalized etc.). But in the end I would like to feed the original matrices into the network and not changed values due to saving as image.

Is it possible to somehow store the arrays on disk and iterate over them in a similar way as ImageDataGenerator does?

Or else can I save the arrays as images without changing the values of the arrays?

I'm not very sure about this, but could `numpy.memmap` work? — Alex, Jul 16 '19 at 01:27

score 1 · Accepted Answer · answered Jul 16 '19 at 04:10

1

Instead of using ImageDataGenerator, you can define your own custom data generator class, by overriding few simple methods for the class.

You can follow this medium post for more reference on this.

https://medium.com/@ensembledme/writing-custom-keras-generators-fe815d992c5a

answered Jul 16 '19 at 04:10

venkata krishnan

1,961
1
13
20

By the way, do I have to normalize the values between 0 and 1 for CNN? The actual values I have in the matrix are between 0 and 20. – machinery Jul 16 '19 at 08:53
Yes. I generally do min max normalization to range the values between 0 and 1 – venkata krishnan Jul 19 '19 at 01:36

Iterating over arrays on disk similar to ImageDataGenerator

1 Answers1