Count Neurons in Keras (with different layers), is my approach correct?

Question

I'm trying to determine the number of 'neurons / nodes' within my Keras network not the parameter. I'm using an already implemented variant, so I didn't develop anything by myself.

That I can get an overview of the network and the number of parameters with summary, I know. The problem here is, I don't want to know how many parameters I have but how many 'neurons'. Background is, for an 8 to 8 fully connected layer, i get 64 parameters. But I want to get to the 16. That the whole story with a Conv2D layer is not so easy to make, I also know.

My first approach was to multiply all values of the output_shape variable and add them afterwards. Can I do that or is it wrong?

thats is the list form model summary:

Layer (type)                            Output Shape            
================================================================
input_image (InputLayer)                (None, None, None, 1)   
zero_padding2d_1 (ZeroPadding2D)        (None, None, None, 1)   
conv1 (Conv2D)                          (None, None, None, 64)  
bn_conv1 (BatchNorm)                    (None, None, None, 64)  
activation_1 (Activation)               (None, None, None, 64)  
max_pooling2d_1 (MaxPooling2D)          (None, None, None, 64)  
res2a_branch2a (Conv2D)                 (None, None, None, 64)  
bn2a_branch2a (BatchNorm)               (None, None, None, 64)  
activation_2 (Activation)               (None, None, None, 64)  
res2a_branch2b (Conv2D)                 (None, None, None, 64)  
bn2a_branch2b (BatchNorm)               (None, None, None, 64)  
activation_3 (Activation)               (None, None, None, 64)  
res2a_branch2c (Conv2D)                 (None, None, None, 256) 
res2a_branch1 (Conv2D)                  (None, None, None, 256) 
bn2a_branch2c (BatchNorm)               (None, None, None, 256) 
bn2a_branch1 (BatchNorm)                (None, None, None, 256) 
add_1 (Add)                             (None, None, None, 256) 
res2a_out (Activation)                  (None, None, None, 256) 
res2b_branch2a (Conv2D)                 (None, None, None, 64)  
bn2b_branch2a (BatchNorm)               (None, None, None, 64)  
activation_4 (Activation)               (None, None, None, 64)  
res2b_branch2b (Conv2D)                 (None, None, None, 64)  
bn2b_branch2b (BatchNorm)               (None, None, None, 64)  
activation_5 (Activation)               (None, None, None, 64)  
res2b_branch2c (Conv2D)                 (None, None, None, 256) 
bn2b_branch2c (BatchNorm)               (None, None, None, 256) 
add_2 (Add)                             (None, None, None, 256) 
res2b_out (Activation)                  (None, None, None, 256) 
res2c_branch2a (Conv2D)                 (None, None, None, 64)  
bn2c_branch2a (BatchNorm)               (None, None, None, 64)  
activation_6 (Activation)               (None, None, None, 64)  
res2c_branch2b (Conv2D)                 (None, None, None, 64)  
bn2c_branch2b (BatchNorm)               (None, None, None, 64)  
activation_7 (Activation)               (None, None, None, 64)  
res2c_branch2c (Conv2D)                 (None, None, None, 256) 
bn2c_branch2c (BatchNorm)               (None, None, None, 256) 
add_3 (Add)                             (None, None, None, 256) 
res2c_out (Activation)                  (None, None, None, 256) 
res3a_branch2a (Conv2D)                 (None, None, None, 128) 
bn3a_branch2a (BatchNorm)               (None, None, None, 128) 
activation_8 (Activation)               (None, None, None, 128) 
res3a_branch2b (Conv2D)                 (None, None, None, 128) 
bn3a_branch2b (BatchNorm)               (None, None, None, 128) 
activation_9 (Activation)               (None, None, None, 128) 
res3a_branch2c (Conv2D)                 (None, None, None, 512) 
res3a_branch1 (Conv2D)                  (None, None, None, 512) 
bn3a_branch2c (BatchNorm)               (None, None, None, 512) 
bn3a_branch1 (BatchNorm)                (None, None, None, 512) 
add_4 (Add)                             (None, None, None, 512) 
res3a_out (Activation)                  (None, None, None, 512) 
res3b_branch2a (Conv2D)                 (None, None, None, 128) 
bn3b_branch2a (BatchNorm)               (None, None, None, 128) 
activation_10 (Activation)              (None, None, None, 128) 
res3b_branch2b (Conv2D)                 (None, None, None, 128) 
bn3b_branch2b (BatchNorm)               (None, None, None, 128) 
activation_11 (Activation)              (None, None, None, 128) 
res3b_branch2c (Conv2D)                 (None, None, None, 512) 
bn3b_branch2c (BatchNorm)               (None, None, None, 512) 
add_5 (Add)                             (None, None, None, 512) 
res3b_out (Activation)                  (None, None, None, 512) 
res3c_branch2a (Conv2D)                 (None, None, None, 128) 
bn3c_branch2a (BatchNorm)               (None, None, None, 128) 
activation_12 (Activation)              (None, None, None, 128) 
res3c_branch2b (Conv2D)                 (None, None, None, 128) 
bn3c_branch2b (BatchNorm)               (None, None, None, 128) 
activation_13 (Activation)              (None, None, None, 128) 
res3c_branch2c (Conv2D)                 (None, None, None, 512) 
bn3c_branch2c (BatchNorm)               (None, None, None, 512) 
add_6 (Add)                             (None, None, None, 512) 
res3c_out (Activation)                  (None, None, None, 512) 
res3d_branch2a (Conv2D)                 (None, None, None, 128) 
bn3d_branch2a (BatchNorm)               (None, None, None, 128) 
activation_14 (Activation)              (None, None, None, 128) 
res3d_branch2b (Conv2D)                 (None, None, None, 128) 
bn3d_branch2b (BatchNorm)               (None, None, None, 128) 
activation_15 (Activation)              (None, None, None, 128) 
res3d_branch2c (Conv2D)                 (None, None, None, 512) 
bn3d_branch2c (BatchNorm)               (None, None, None, 512) 
add_7 (Add)                             (None, None, None, 512) 
res3d_out (Activation)                  (None, None, None, 512) 
res4a_branch2a (Conv2D)                 (None, None, None, 256) 
bn4a_branch2a (BatchNorm)               (None, None, None, 256) 
activation_16 (Activation)              (None, None, None, 256) 
res4a_branch2b (Conv2D)                 (None, None, None, 256) 
bn4a_branch2b (BatchNorm)               (None, None, None, 256) 
activation_17 (Activation)              (None, None, None, 256) 
res4a_branch2c (Conv2D)                 (None, None, None, 1024)
res4a_branch1 (Conv2D)                  (None, None, None, 1024)
bn4a_branch2c (BatchNorm)               (None, None, None, 1024)
bn4a_branch1 (BatchNorm)                (None, None, None, 1024)
add_8 (Add)                             (None, None, None, 1024)
res4a_out (Activation)                  (None, None, None, 1024)
res4b_branch2a (Conv2D)                 (None, None, None, 256) 
bn4b_branch2a (BatchNorm)               (None, None, None, 256) 
activation_18 (Activation)              (None, None, None, 256) 
res4b_branch2b (Conv2D)                 (None, None, None, 256) 
bn4b_branch2b (BatchNorm)               (None, None, None, 256) 
activation_19 (Activation)              (None, None, None, 256) 
res4b_branch2c (Conv2D)                 (None, None, None, 1024)
bn4b_branch2c (BatchNorm)               (None, None, None, 1024)
add_9 (Add)                             (None, None, None, 1024)
res4b_out (Activation)                  (None, None, None, 1024)
res4c_branch2a (Conv2D)                 (None, None, None, 256) 
bn4c_branch2a (BatchNorm)               (None, None, None, 256) 
activation_20 (Activation)              (None, None, None, 256) 
res4c_branch2b (Conv2D)                 (None, None, None, 256) 
bn4c_branch2b (BatchNorm)               (None, None, None, 256) 
activation_21 (Activation)              (None, None, None, 256) 
res4c_branch2c (Conv2D)                 (None, None, None, 1024)
bn4c_branch2c (BatchNorm)               (None, None, None, 1024)
add_10 (Add)                            (None, None, None, 1024)
res4c_out (Activation)                  (None, None, None, 1024)
res4d_branch2a (Conv2D)                 (None, None, None, 256) 
bn4d_branch2a (BatchNorm)               (None, None, None, 256) 
activation_22 (Activation)              (None, None, None, 256) 
res4d_branch2b (Conv2D)                 (None, None, None, 256) 
bn4d_branch2b (BatchNorm)               (None, None, None, 256) 
activation_23 (Activation)              (None, None, None, 256) 
res4d_branch2c (Conv2D)                 (None, None, None, 1024)
bn4d_branch2c (BatchNorm)               (None, None, None, 1024)
add_11 (Add)                            (None, None, None, 1024)
res4d_out (Activation)                  (None, None, None, 1024)
res4e_branch2a (Conv2D)                 (None, None, None, 256) 
bn4e_branch2a (BatchNorm)               (None, None, None, 256) 
activation_24 (Activation)              (None, None, None, 256) 
res4e_branch2b (Conv2D)                 (None, None, None, 256) 
bn4e_branch2b (BatchNorm)               (None, None, None, 256) 
activation_25 (Activation)              (None, None, None, 256) 
res4e_branch2c (Conv2D)                 (None, None, None, 1024)
bn4e_branch2c (BatchNorm)               (None, None, None, 1024)
add_12 (Add)                            (None, None, None, 1024)
res4e_out (Activation)                  (None, None, None, 1024)
res4f_branch2a (Conv2D)                 (None, None, None, 256) 
bn4f_branch2a (BatchNorm)               (None, None, None, 256) 
activation_26 (Activation)              (None, None, None, 256) 
res4f_branch2b (Conv2D)                 (None, None, None, 256) 
bn4f_branch2b (BatchNorm)               (None, None, None, 256) 
activation_27 (Activation)              (None, None, None, 256) 
res4f_branch2c (Conv2D)                 (None, None, None, 1024)
bn4f_branch2c (BatchNorm)               (None, None, None, 1024)
add_13 (Add)                            (None, None, None, 1024)
res4f_out (Activation)                  (None, None, None, 1024)
res5a_branch2a (Conv2D)                 (None, None, None, 512) 
bn5a_branch2a (BatchNorm)               (None, None, None, 512) 
activation_28 (Activation)              (None, None, None, 512) 
res5a_branch2b (Conv2D)                 (None, None, None, 512) 
bn5a_branch2b (BatchNorm)               (None, None, None, 512) 
activation_29 (Activation)              (None, None, None, 512) 
res5a_branch2c (Conv2D)                 (None, None, None, 2048)
res5a_branch1 (Conv2D)                  (None, None, None, 2048)
bn5a_branch2c (BatchNorm)               (None, None, None, 2048)
bn5a_branch1 (BatchNorm)                (None, None, None, 2048)
add_14 (Add)                            (None, None, None, 2048)
res5a_out (Activation)                  (None, None, None, 2048)
res5b_branch2a (Conv2D)                 (None, None, None, 512) 
bn5b_branch2a (BatchNorm)               (None, None, None, 512) 
activation_30 (Activation)              (None, None, None, 512) 
res5b_branch2b (Conv2D)                 (None, None, None, 512) 
bn5b_branch2b (BatchNorm)               (None, None, None, 512) 
activation_31 (Activation)              (None, None, None, 512) 
res5b_branch2c (Conv2D)                 (None, None, None, 2048)
bn5b_branch2c (BatchNorm)               (None, None, None, 2048)
add_15 (Add)                            (None, None, None, 2048)
res5b_out (Activation)                  (None, None, None, 2048)
res5c_branch2a (Conv2D)                 (None, None, None, 512) 
bn5c_branch2a (BatchNorm)               (None, None, None, 512) 
activation_32 (Activation)              (None, None, None, 512) 
res5c_branch2b (Conv2D)                 (None, None, None, 512) 
bn5c_branch2b (BatchNorm)               (None, None, None, 512) 
activation_33 (Activation)              (None, None, None, 512) 
res5c_branch2c (Conv2D)                 (None, None, None, 2048)
bn5c_branch2c (BatchNorm)               (None, None, None, 2048)
add_16 (Add)                            (None, None, None, 2048)
res5c_out (Activation)                  (None, None, None, 2048)
fpn_c5p5 (Conv2D)                       (None, None, None, 256) 
fpn_p5upsampled (UpSampling2D)          (None, None, None, 256) 
fpn_c4p4 (Conv2D)                       (None, None, None, 256) 
fpn_p4add (Add)                         (None, None, None, 256) 
fpn_p4upsampled (UpSampling2D)          (None, None, None, 256) 
fpn_c3p3 (Conv2D)                       (None, None, None, 256) 
fpn_p3add (Add)                         (None, None, None, 256) 
fpn_p3upsampled (UpSampling2D)          (None, None, None, 256) 
fpn_c2p2 (Conv2D)                       (None, None, None, 256) 
fpn_p2add (Add)                         (None, None, None, 256) 
fpn_p5 (Conv2D)                         (None, None, None, 256) 
fpn_p2 (Conv2D)                         (None, None, None, 256) 
fpn_p3 (Conv2D)                         (None, None, None, 256) 
fpn_p4 (Conv2D)                         (None, None, None, 256) 
fpn_p6 (MaxPooling2D)                   (None, None, None, 256) 
rpn_model (Model)                       [(None, None, 2), 
                                             (None, None, 2), 
                                             (None, None, 4)]       
rpn_class (Concatenate)                 (None, None, 2)         
rpn_bbox (Concatenate)                  (None, None, 4)         
input_anchors (InputLayer)              (None, None, 4)         
ROI (ProposalLayer)                     (None, 1000, 4)         
input_image_meta (InputLayer)           (None, 18)              
roi_align_classifier (PyramidROIAlign)  (None, 1000, 7, 7, 256) 
mrcnn_class_conv1 (TimeDistributed)     (None, 1000, 1, 1, 1024)
mrcnn_class_bn1 (TimeDistributed)       (None, 1000, 1, 1, 1024)
activation_34 (Activation)              (None, 1000, 1, 1, 1024)
mrcnn_class_conv2 (TimeDistributed)     (None, 1000, 1, 1, 1024)
mrcnn_class_bn2 (TimeDistributed)       (None, 1000, 1, 1, 1024)
activation_35 (Activation)              (None, 1000, 1, 1, 1024)
pool_squeeze (Lambda)                   (None, 1000, 1024)      
mrcnn_class_logits (TimeDistributed)    (None, 1000, 6)         
mrcnn_bbox_fc (TimeDistributed)         (None, 1000, 24)        
mrcnn_class (TimeDistributed)           (None, 1000, 6)         
mrcnn_bbox (Reshape)                    (None, 1000, 6, 4)      
mrcnn_detection (DetectionLayer)        (None, 100, 6)          
lambda_3 (Lambda)                       (None, 100, 4)          
roi_align_mask (PyramidROIAlign)        (None, 100, 14, 14, 256)
mrcnn_mask_conv1 (TimeDistributed)      (None, 100, 14, 14, 256)
mrcnn_mask_bn1 (TimeDistributed)        (None, 100, 14, 14, 256)
activation_37 (Activation)              (None, 100, 14, 14, 256)
mrcnn_mask_conv2 (TimeDistributed)      (None, 100, 14, 14, 256)
mrcnn_mask_bn2 (TimeDistributed)        (None, 100, 14, 14, 256)
activation_38 (Activation)              (None, 100, 14, 14, 256)
mrcnn_mask_conv3 (TimeDistributed)      (None, 100, 14, 14, 256)
mrcnn_mask_bn3 (TimeDistributed)        (None, 100, 14, 14, 256)
activation_39 (Activation)              (None, 100, 14, 14, 256)
mrcnn_mask_conv4 (TimeDistributed)      (None, 100, 14, 14, 256)
mrcnn_mask_bn4 (TimeDistributed)        (None, 100, 14, 14, 256)
activation_40 (Activation)              (None, 100, 14, 14, 256)
mrcnn_mask_deconv (TimeDistributed)     (None, 100, 28, 28, 256)
mrcnn_mask (TimeDistributed)            (None, 100, 28, 28, 6)  
================================================================
Total params: 44,678,198
Trainable params: 44,618,934
Non-trainable params: 59,264

And my counted Neurons 105,641,486. That looks wrong because there are a lot more than the weights (parameters). I'm not sure if I can really add all layers?

And if anyone's wondering why I want to do this. I want to compare it to a biological neural network, and i have only the neuron count of the brain and not all connections between there. I know they are not comparable but good enough for what I want to do.

thanks for hints and help

You don’t want to include layers like batchnorm or activation layers. One hack is to set input shape to (None, 1). Then, the number of trainable weights should be pretty close to the number of neurons. — Tim, Jan 28 '19 at 21:13
Hi, I tried this way. I set the image size of my network to 1x1 pixel. But this had no influence, I get the same number of parameters. Probably it is due to the functionality of Mask R-CNN with the combination of an RPN + Resnet. — Pascal Z., Feb 05 '19 at 19:32

score 1 · Accepted Answer · answered Jan 30 '19 at 16:38

1

A few things:

In convolutional layers, neurons == filters
If you count other layers like activation, padding and pooling/sampling, you will be counting additional neurons that don't exist (these layers don't have neurons)
BatchNormalization layers do have parameters, but I'm not sure you want to consider them as having neurons. Nevertheless, they have learnable parameters for scaling and bias, besides the non-trainable parameters for mean and variance. (A good reason for always using use_bias=False in any layer directly before the batch norm)

So, just count the number of filters in each Conv layer. Add the BatchNorm channels if you want to.

answered Jan 30 '19 at 16:38

Daniel Möller

84,878
18
192
214

Hi, thank you very much for your answer, unfortunately I had many exams last week and therefore could not write an answers earlier. Do you have maybe a paper or something like that where it is described? I would have thought now that pooling layers can also be seen as neurons because they make a decision. But since I am not an expert in this field (i hope i will be like one in the future) this is a pure assumption. – Pascal Z. Feb 05 '19 at 19:15
I think that's a decision for you to make. Not sure if there is really a clear definition for "neuron". Mine is: "something that has respective trainable weights". Filters have their respective trainable weights, dense layer units have their respective trainable weights. The `BatchNormalization` is that strange case I mentioned. They have weights for scaling and biasing, but not really for performing "intelligent" operations. All the other layers use plain functions without trainable weights. – Daniel Möller Feb 08 '19 at 12:01
Depending of what is important for you to count, add the filters of the pooling layers and activations as well.... Although activations could be considered part of the previous neurons, since they're are filterwise. – Daniel Möller Feb 08 '19 at 12:03

Count Neurons in Keras (with different layers), is my approach correct?

1 Answers1