I've been trying to run Mask R-CNN on a small custom dataset (images + .json polygon data, converted into binary masks) on Google Colab (the script is a mix between different tutorials using the Mask R-CNN/ Matterport repo on github). Everything works fine up until training the heads- the model runs up until the first epoch and then it throws "UnboundLocalError: local variable 'image_id' referenced before assignment" (error attached)/ data generator - zeroDivisionError
Where it occurs in the code: training_heads_first
Error code:
Starting at epoch 0. LR=0.001
Checkpoint Path: logs/pipes20200225T2257/mask_rcnn_pipes_{epoch:04d}.h5
Selecting layers to train
fpn_c5p5 (Conv2D)
fpn_c4p4 (Conv2D)
fpn_c3p3 (Conv2D)
fpn_c2p2 (Conv2D)
fpn_p5 (Conv2D)
fpn_p2 (Conv2D)
fpn_p3 (Conv2D)
fpn_p4 (Conv2D)
In model: rpn_model
rpn_conv_shared (Conv2D)
rpn_class_raw (Conv2D)
rpn_bbox_pred (Conv2D)
mrcnn_mask_conv1 (TimeDistributed)
mrcnn_mask_bn1 (TimeDistributed)
mrcnn_mask_conv2 (TimeDistributed)
mrcnn_mask_bn2 (TimeDistributed)
mrcnn_class_conv1 (TimeDistributed)
mrcnn_class_bn1 (TimeDistributed)
mrcnn_mask_conv3 (TimeDistributed)
mrcnn_mask_bn3 (TimeDistributed)
mrcnn_class_conv2 (TimeDistributed)
mrcnn_class_bn2 (TimeDistributed)
mrcnn_mask_conv4 (TimeDistributed)
mrcnn_mask_bn4 (TimeDistributed)
mrcnn_bbox_fc (TimeDistributed)
mrcnn_mask_deconv (TimeDistributed)
mrcnn_class_logits (TimeDistributed)
mrcnn_mask (TimeDistributed)
/usr/local/lib/python3.6/dist-packages/tensorflow/python/ops/gradients_impl.py:110: UserWarning: Converting sparse IndexedSlices to a dense Tensor of unknown shape. This may consume a large amount of memory.
"Converting sparse IndexedSlices to a dense Tensor of unknown shape. "
/usr/local/lib/python3.6/dist-packages/keras/engine/training_generator.py:49: UserWarning: Using a generator with `use_multiprocessing=True` and multiple workers may duplicate your data. Please consider using the `keras.utils.Sequence class.
UserWarning('Using a generator with `use_multiprocessing=True`'
Epoch 1/10
---------------------------------------------------------------------------
RemoteTraceback Traceback (most recent call last)
RemoteTraceback:
"""
Traceback (most recent call last):
File "/usr/local/lib/python3.6/dist-packages/mrcnn/model.py", line 1695, in data_generator
image_index = (image_index + 1) % len(image_ids)
ZeroDivisionError: integer division or modulo by zero
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/usr/lib/python3.6/multiprocessing/pool.py", line 119, in worker
result = (True, func(*args, **kwds))
File "/usr/local/lib/python3.6/dist-packages/keras/utils/data_utils.py", line 641, in next_sample
return six.next(_SHARED_SEQUENCES[uid])
File "/usr/local/lib/python3.6/dist-packages/mrcnn/model.py", line 1805, in data_generator
dataset.image_info[image_id]))
UnboundLocalError: local variable 'image_id' referenced before assignment
"""
The above exception was the direct cause of the following exception:
UnboundLocalError Traceback (most recent call last)
<ipython-input-21-0da65da78eb9> in <module>()
----> 1 model.train(dataset_train, dataset_val, epochs=10,layers="heads", learning_rate=config.LEARNING_RATE)
10 frames
/usr/local/lib/python3.6/dist-packages/mrcnn/model.py in data_generator()
1803 # Log it and skip the image
1804 logging.exception("Error processing image {}".format(
-> 1805 dataset.image_info[image_id]))
1806 error_count += 1
1807 if error_count > 5:
UnboundLocalError: local variable 'image_id' referenced before assignment
I have checked the .json file, it isn't empty, data seems to be loaded correctly.
I'm not quite sure whether this has something to do with the set-up in the code generally, or whether it's something to do with the mrcnn library. Any advice on how to tackle this would be really helpful!
I am sharing the Colab notebook here for reference: https://colab.research.google.com/drive/1vWoBwbcie68J8UYcIEFd4ApYbdsHb1ot
