INFO:tensorflow:Waiting for new checkpoint at models/faster_rcnn

Question

I used the transfer learning approach to develop a detection model using the faster_rcnn algorithm.

To evaluate my model, I used the following commands-

!python model_main_tf2.py --model_dir=models/faster_rcnn_inception_resnet_v2 --pipeline_config_path=models/faster_rcnn_inception_resnet_v2/pipeline.config --checkpoint_dir=models/faster_rcnn_inception_resnet_v2

However, I have been getting the following error/info message: -

INFO:tensorflow:Waiting for new checkpoint at models/faster_rcnn_inception_resnet_v2
I0331 23:23:11.699681 140426971481984 checkpoint_utils.py:139] Waiting for new checkpoint at models/faster_rcnn_inception_resnet_v2

I checked the path to the checkpoint_dir is correct. What could be the problem and how can I resolve it?

Thanks in advance.

Answer 1

You need to run another script for training to generate new checkpoint. model_main_tf2.py does not do both at once, i.e., it won't train model and evaluate the model at the end of each epoch.

One way to get what you want modifying checkpoint_max_to_keep in https://github.com/tensorflow/models/blob/13ec3c1460b928301d208115aed0c94fb47538b7/research/object_detection/model_lib_v2.py#L445 to keep all checkpoints, then evaluate separately. This does not work exactly same as you want, but it generates the curves.

Answer 2

A similar situation happened to me. I don't know if this is the solution or just a workaround but it did work for me. I simply exported my model and provided the path to that checkpoint folder.

fintune_checkpoint_model_directory
   |
   \---checkpoint(folder)
       |
       \---checkpoint(file with no extension)
       \---ckpt-1.data0000of0001
       \---ckpt-1.index

and then simply run the model_main_tf.py file for evaluation.

Answer 3

if you trained your model with a few number of steps it can be a problem maybe a few checkpoints can affect that TensorFlow can't generate evaluation so try to increase the number of steps