Fitting a keras model when layers aren't trainable gives inconsistent results

Question

I'm trying to determine the accuracy of my model without training and updating the weights so I've set all of my layers to trainable = False.

When I run fit_generator on a generator with shuffle = False, I get consistent results each time.

When I run fit_generator on a generator with shuffle = True, the results jump around a bit. Given that the input data is the same, and the model isn't training, I would expect the internal state of the model not to change and the accuracy to be the same on the same dataset regardless of ordering.

However this ordering dependency implies that some sort of state in the model is changing despite trainable = False. What's happening inside the model that's causing this?

Is this reproducible even after setting seed using `numpy.random.seed()`? — abhiieor, Jul 10 '17 at 16:24
Haven't tried setting the seed, wouldn't that just ensure the same ordering of the input data? My understanding is that ordering shouldn't matter here but I'm clearly missing something — jvans, Jul 10 '17 at 16:27
My suggestion is mostly to test your hypothesis. Neural nets by design are noisy so a reproducible case is always to present. — abhiieor, Jul 10 '17 at 16:29

score 0 · Answer 1 · answered Jul 10 '17 at 18:26

0

This is a really interesting phenomenon. It probably arises due to the fact that most of neural networks packages use float32 precision - which gives you an accuracy up to 5-7 decimal point. Here you can read a detailed explaination.

answered Jul 10 '17 at 18:26

Marcin Możejko

39,542
10
109
120

Fitting a keras model when layers aren't trainable gives inconsistent results

1 Answers1