MaskRCNN should find exactly one element

Question

I trained a maskrcnn-model with matterport with one class to detect. It worked.

Now I want to predict some unseen images. I know that the object is present on each image and that it appears only once per image. How do I use my model to do so? An possibility that came to my mind was:

num_results = 0
while num_results = 0:
    model = mrcnn.model.MaskRCNN(mode='inference', config=pred_config)
    model.load_weights('weight/path')
    results = model.detect([img], verbose=1)
    num_results = compute_num_of(results)
    # lower DETECTION_MIN_CONFIDENCE from pred_config

But I think this is very time consuming because I load that model and its weights at every step. What would be best practice here?

Thanks