Currently I’m struggling with improving the results on semantic segmentation problem using deeplabV3+ trained on my own dataset. I’ve trained deeplabV3+ successfully a few times using different pretrained models from the model zoo, all based on xception_65, but my results keep staying in the same miou range, somewhere around this interval [10, 11]. I have only one GPU at my disposal with 11GB GPU memory. My dataset has 8 classes with various object sizes, from little to big, and is quite unbalanced. Here are the label weights: [1, 4, 4, 17, 42, 36, 19, 20]. In my dataset I have 757 instances for training and 100 validation.
When training the general tendency is: the first 10k iterations my loss decreases, but then it only oscillates.
I’ve tried:
to adjust parameters like: the learning rate, last_layer_gradient_multiplier, weight decay
training on various image sizes 321, 513, 769
some kind of weighting using the above weights in this formula
weights = tf.to_float(tf.equal(scaled_labels, 0)) * 1 + tf.to_float(tf.equal(scaled_labels, 1)) * 4 + tf.to_float(tf.equal(scaled_labels, 2)) * 4 + tf.to_float(tf.equal(scaled_labels, 3)) * 17 + tf.to_float(tf.equal(scaled_labels, 4)) * 42 + tf.to_float(tf.equal(scaled_labels, 5)) * 36 + tf.to_float(tf.equal(scaled_labels, 6)) * 19 + tf.to_float(tf.equal(scaled_labels, 7)) * 20 + tf.to_float(tf.equal(scaled_labels, ignore_label)) * 0.0
I’ve trained without fine tuning the batch normalization parameters (fine_tune_batch_norm = False). Although I also tried training those parameters (fine_tune_batch_norm = True) with a 321 crop size in order to be able to fit a batch size of 12 in my GPU.
The point being I need some tips to figure out what I can do to improve those results. What do you guys think? Do I need more data in order to increase my miou or hardware?
