The model cannot be trained well by MASK RCNN

Question

I am using MRCNN in python to train 20 images (with annotated images info saved as json file) for object detection. The problem is that at the best case the loss is around 4 which it shows that the model has not learned well (the loss fluctuates a lot during the learning process for each epochs). Obviously, when using the trained model for detection the result is wrong, it means that it cannot detect the object and randomly selects some pixels as the object. Can someone kindly help me how I can improve the performance and also some hints about initial weights if the object is not one of the objects in COCO database. Thanks in advance.