Stack Overflow
Stack Overflow

Questions tagged [batch-normalization]

Batch Normalization is a technique to improve learning in neural networks by normalizing the distribution of each input feature in each layer across each minibatch to N(0, 1).

Batch Normalization allows to use much higher learning rates and be less careful about initialization. It also acts as a regularizer, in some cases eliminating the need for Dropout.

  1. Ioffe, Sergey; Christian Szegedy (2015). Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift.
397 questions
209
votes
8 answers

Where do I call the BatchNormalization function in Keras?

If I want to use the BatchNormalization function in Keras, then do I need to call it once only at the beginning? I read this documentation for it: http://keras.io/layers/normalization/ I don't see where I'm supposed to call it. Below is my code…
pr338
  • 8,730
  • 19
  • 52
  • 71
90
votes
4 answers

Batch Normalization in Convolutional Neural Network

I am newbie in convolutional neural networks and just have idea about feature maps and how convolution is done on images to extract features. I would be glad to know some details on applying batch normalisation in CNN. I read this paper…
akshata bhat
  • 1,103
  • 2
  • 9
  • 9
82
votes
4 answers

Instance Normalisation vs Batch normalisation

I understand that Batch Normalisation helps in faster training by turning the activation towards unit Gaussian distribution and thus tackling vanishing gradients problem. Batch norm acts is applied differently at training(use mean/var from each…
Ruppesh Nalwaya
  • 1,409
  • 2
  • 14
  • 22
41
votes
2 answers

Batch normalization instead of input normalization

Can I use batch normalization layer right after input layer and not normalize my data? May I expect to get similar effect/performance? In keras functional it would be something like this: x = Input (...) x = Batchnorm(...)(x) ...
user2146414
  • 840
  • 11
  • 29
33
votes
2 answers

How the number of parameters associated with BatchNormalization layer is 2048?

I have the following code. x = keras.layers.Input(batch_shape = (None, 4096)) hidden = keras.layers.Dense(512, activation = 'relu')(x) hidden = keras.layers.BatchNormalization()(hidden) hidden = keras.layers.Dropout(0.5)(hidden) predictions =…
Wasi Ahmad
  • 35,739
  • 32
  • 114
  • 161
23
votes
2 answers

What is right batch normalization function in Tensorflow?

In tensorflow 1.4, I found two functions that do batch normalization and they look same: tf.layers.batch_normalization (link) tf.contrib.layers.batch_norm (link) Which function should I use? Which one is more stable?
KimHee
  • 728
  • 2
  • 12
  • 22
21
votes
1 answer

tf.layers.batch_normalization large test error

I'm trying to use batch normalization. I tried to use tf.layers.batch_normalization on a simple conv net for mnist. I get high accuracy for train step (>98%) but very low test accuracy (<50%). I tried to change momentum values (I tried…
MrG
  • 213
  • 1
  • 2
  • 4
17
votes
3 answers

Where to apply batch normalization on standard CNNs

I have the following architecture: Conv1 Relu1 Pooling1 Conv2 Relu2 Pooling3 FullyConnect1 FullyConnect2 My question is, where do I apply batch normalization? And what would be the best function to do this in TensorFlow?
whoisraibolt
  • 2,082
  • 3
  • 22
  • 45
17
votes
1 answer

Batch normalization with 3D convolutions in TensorFlow

I'm implementing a model relying on 3D convolutions (for a task that is similar to action recognition) and I want to use batch normalization (see [Ioffe & Szegedy 2015]). I could not find any tutorial focusing on 3D convs, hence I'm making a short…
Alessio B
  • 353
  • 4
  • 9
15
votes
3 answers

significance of "trainable" and "training" flag in tf.layers.batch_normalization

What is the significance of "trainable" and "training" flag in tf.layers.batch_normalization? How are these two different during training and prediction?
Amit Gupta
  • 2,698
  • 4
  • 24
  • 37
15
votes
2 answers

How should "BatchNorm" layer be used in caffe?

I am a little confused about how should I use/insert "BatchNorm" layer in my models. I see several different approaches, for instance: ResNets: "BatchNorm"+"Scale" (no parameter sharing) "BatchNorm" layer is followed immediately with "Scale" layer:…
Shai
  • 111,146
  • 38
  • 238
  • 371
14
votes
2 answers

Batch normalization during testing

For batch normalization during testing, how does one calculate the mean and variance of each activation input (in each layer and input dimension)? Does one record the means and variances from training, calculate the means and variances of the entire…
user8384788
  • 331
  • 1
  • 2
  • 10
13
votes
2 answers

How to correctly use batch normalization with U-net in keras?

I am trying to use batch normalization layers whith U-net for the segmentation task. Same layers works fine for res-net, vgg, xception etc., and I'm curious if it is an architecture dependent problem? During the training everything is fine, metrics…
Primakov
  • 286
  • 3
  • 10
13
votes
3 answers

Why batch normalization over channels only in CNN

I am wondering, if in Convolutional Neural Networks batch normalization should be applied with respect to every pixel separately, or should I take the mean of pixels with respect to each channel? I saw that in the description of Tensorflow's…
Kuba
  • 291
  • 2
  • 5
  • 14
12
votes
2 answers

How to do fully connected batch norm in PyTorch?

torch.nn has classes BatchNorm1d, BatchNorm2d, BatchNorm3d, but it doesn't have a fully connected BatchNorm class? What is the standard way of doing normal Batch Norm in PyTorch?
patapouf_ai
  • 17,605
  • 13
  • 92
  • 132
1
2 3
…
26 27