Keras' `model.fit_generator()` behaves different than `model.fit()`

Question

I have a huge dataset that I need to provide to Keras in the form of a generator because it does not fit into memory. However, using fit_generator, I cannot replicate the results I get during usual training with model.fit. Also each epoch lasts considerably longer.

I implemented a minimal example. Maybe someone can show me where the problem is.

import random
import numpy

from keras.layers import Dense
from keras.models import Sequential

random.seed(23465298)
numpy.random.seed(23465298)

no_features = 5
no_examples = 1000


def get_model():
    network = Sequential()
    network.add(Dense(8, input_dim=no_features, activation='relu'))
    network.add(Dense(1, activation='sigmoid'))
    network.compile(loss='binary_crossentropy', optimizer='adam')
    return network


def get_data():
    example_input = [[float(f_i == e_i % no_features) for f_i in range(no_features)] for e_i in range(no_examples)]
    example_target = [[float(t_i % 2)] for t_i in range(no_examples)]
    return example_input, example_target


def data_gen(all_inputs, all_targets, batch_size=10):
    input_batch = numpy.zeros((batch_size, no_features))
    target_batch = numpy.zeros((batch_size, 1))
    while True:
        for example_index, each_example in enumerate(zip(all_inputs, all_targets)):
            each_input, each_target = each_example
            wrapped = example_index % batch_size
            input_batch[wrapped] = each_input
            target_batch[wrapped] = each_target
            if wrapped == batch_size - 1:
                yield input_batch, target_batch


if __name__ == "__main__":
    input_data, target_data = get_data()
    g = data_gen(input_data, target_data, batch_size=10)
    model = get_model()
    model.fit(input_data, target_data, epochs=15, batch_size=10)  # 15 * (1000 / 10) * 10
    # model.fit_generator(g, no_examples // 10, epochs=15)        # 15 * (1000 / 10) * 10

On my computer, model.fit always finishes the 10th epoch with a loss of 0.6939 and after ca. 2-3 seconds.

The method model.fit_generator, however, runs considerably longer and finishes the last epoch with a different loss (0.6931).

I don't understand in general why the results in both approaches differ. This might not appear like much of a difference but I need to be sure that the same data with the same net produce the same result, independent from conventional training or using the generator.

Update: @Alex R. provided an answer for part of the original problem (some of the performance issue as well as changing results with each run). As the core problem remains, however, I merely adjusted the question and title accordingly.

Answer 1

I don't understand how the loss can be unstable with larger batch size, as there should be less fluctuations with larger batches. However, looking at Keras documentation, the fit() routine looks like:

fit(self, x, y, batch_size=32, epochs=10, verbose=1, callbacks=None, validation_split=0.0, 
    validation_data=None, shuffle=True, class_weight=None, sample_weight=None, 
    initial_epoch=0)

which has a default batch_size=32 and epochs=10. Wheras the fit_generator() looks like:

fit_generator(self, generator, steps_per_epoch, epochs=1, verbose=1,
              callbacks=None, validation_data=None, validation_steps=None, 
              class_weight=None, max_queue_size=10, workers=1,
              use_multiprocessing=False, initial_epoch=0)

Specifically the "step_per_epoch" are defined by:

steps_per_epoch: Total number of steps (batches of samples) to yield from generator before declaring one epoch finished and starting the next epoch. It should typically be equal to the number of unique samples of your dataset divided by the batch size.

So for starters, it sounds like your fit_generator is taking in massively more numbers of samples, compared to your fit() routine. See here for more details.

Answer 2

Batch sizes

• In fit, you're using the standard batch size = 32.
• In fit_generator, you're using a batch size = 10.

Keras probably runs the weight updates after each batch, so, if you're using batches of different size, there is a chance of getting different gradients between the two methods. And once there a different weight update, both models will never meet again.

Try to use fit with batch_size=10, or use a generator with batch_size=32.

---

Seed problem?

Are you creating a new model with get_model() for each case?

If so, the weights in both models are different, and naturally you will have different results for both models. (Ok, you've set a seed, but if you're using tensorflow, maybe you're facing this issue)

On the long run they will sort of converge, though. The difference between both doesn't seem that much.

---

Checking data

If you are not sure that your generator yields the same data as you expect, do a simple loop on it and print/compare/check the data it yields:

for i in range(numberOfBatches):
    x,y = g.next() #or next(g)
    #print or compare x,y here. 

---

Answer 3

Make sure to shuffle your batches within your generator.

This discussion suggests you turn on shuffle in your iterator: https://github.com/keras-team/keras/issues/2389. I had the same problem and this resolved it.

Answer 4

As for the loss, that is possibly due to the batch size difference that has already been discussed.

As for the difference in training time, model.fit_generator() allows you to specify the number of "workers". This parameter refers to how many instances of your model are being trained across different areas in your dataset at the same time. If your computer architecture is optimized correctly, you should be able to change the workers parameter to 4 or 8 and see large reductions in training time.

Answer 5

Hope I am not late to the party. The most important thing I would add:

In Keras, using fit() is fine for smaller datasets which can be loaded into memory. For most practical-use cases, almost all datasets are large and cannot be loaded into memory at once.

For larger datasets we have to use fit_generator().

Answer 6

Make sure that your generator actually returns different batches each time. I ran into this issue with my generator. When you're initializing your batch numpy placeholders before the while loop then it is possible that even though you might change those variables inside the for loop then the initialized variables might only change once during the first for loop. My issue was exactly that. I had a similarly structured generator but I was returning the batches after the for loop: Why is this python generator returning the same value everytime?

You can check whether your generator works by using this snippet that checks if all the generated batches are indeed different:

g = data_gen(input_data, target_data, batch_size=10)
input_list = []
target_list = []
for _ in range(100):
    input, target = next(g)
    input_list.append(input)
    target_list.append(target)
inputs = np.concatenate(input_list, axis=0)   
targets = np.concatenate(target_list, axis=0)

all_different = True
for i in range(1, inputs.shape[0]):
    if np.array_equal(inputs[0], inputs[i]):
        all_different = False
print('All batches different') if all_different else print('Generator broken. Initialize your numpy arrays inside the while loop or yield input.copy, target.copy()')