Do you need to standardize inputs if you are using Batch Normalization?

Question

I've been playing around with batch normalization in Keras. I was wondering if batch normalization also normalizes the inputs to the neural network. Does that mean I do not need to standardize my inputs to my network and rely on BN to do it?

Answer 1

While you can certainly use it for that, batch normalization is not designed to do that and you will most likely introduce sampling error in your normalization due to the limited sample size (sample size is your batch size).

Another factor for why I would not recommend using batch normalization for normalizing your inputs is that it introduces the correction terms gamma and beta (trained parameters) which will skew your training data if not disabled.

For normalization of your test data I would recommend using z-score normalization on the complete training set (e.g., via sklearn's StandardScaler) or some appropriate alternative, but not batch normalization.