0

I was following google's Object Detection API retraining on my own dataset but have encountered a series of problems.

One of them is the following:

"Traceback (most recent call last):
  File "/usr/lib/python2.7/runpy.py", line 162, in _run_module_as_main
    "__main__", fname, loader, pkg_name)
  File "/usr/lib/python2.7/runpy.py", line 72, in _run_code
    exec code in run_globals
  File "/root/.local/lib/python2.7/site-packages/object_detection/train.py", line 198, in <module>
    tf.app.run()
  File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/platform/app.py", line 44, in run
    _sys.exit(main(_sys.argv[:1] + flags_passthrough))
  File "/root/.local/lib/python2.7/site-packages/object_detection/train.py", line 194, in main
    worker_job_name, is_chief, FLAGS.train_dir)
  File "/root/.local/lib/python2.7/site-packages/object_detection/trainer.py", line 290, in train
    saver=saver)
  File "/usr/local/lib/python2.7/dist-packages/tensorflow/contrib/slim/python/slim/learning.py", line 776, in train
    master, start_standard_services=False, config=session_config) as sess:
  File "/usr/lib/python2.7/contextlib.py", line 17, in __enter__
    return self.gen.next()
  File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/training/supervisor.py", line 960, in managed_session
    self.stop(close_summary_writer=close_summary_writer)
  File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/training/supervisor.py", line 788, in stop
    stop_grace_period_secs=self._stop_grace_secs)
  File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/training/coordinator.py", line 386, in join
    six.reraise(*self._exc_info_to_raise)
  File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/training/supervisor.py", line 949, in managed_session
    start_standard_services=start_standard_services)
  File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/training/supervisor.py", line 713, in prepare_or_wait_for_session
    max_wait_secs=max_wait_secs)
  File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/training/session_manager.py", line 387, in wait_for_session
    is_ready, not_ready_msg = self._model_ready(sess)
  File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/training/session_manager.py", line 435, in _model_ready
    return _ready(self._ready_op, sess, "Model not ready")
  File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/training/session_manager.py", line 492, in _ready
    ready_value = sess.run(op)
  File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/client/session.py", line 767, in run
    run_metadata_ptr)
  File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/client/session.py", line 965, in _run
    feed_dict_string, options, run_metadata)
  File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/client/session.py", line 1015, in _do_run
    target_list, options, run_metadata)
  File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/client/session.py", line 1035, in _do_call
    raise type(e)(node_def, op, message)
UnavailableError: {"created":"@1502405189.800982817","description":"EOF","file":"external/grpc/src/core/lib/iomgr/tcp_posix.c","file_line":235,"grpc_status":14}
"    
  pathname:  "/var/sitecustomize/sitecustomize.py"    
 }

I am not too sure on what a grpc is - so am quite at a standstill with this error. Anyone that could help it would be great!! Thanks!!

Antonia Calvi
  • 127
  • 1
  • 3
  • I am running a Mac OS X Yosemite 10.10.5 if that is of any help – Antonia Calvi Aug 14 '17 at 21:57
  • This is likely an OOM error [1]. Are you using GPUs? [1]: https://stackoverflow.com/questions/45600567/connection-reset-by-peer-on-adapted-standard-ml-engine-object-detection-trainin – rhaertel80 Aug 15 '17 at 00:19

1 Answers1

1

This is likely an out-of-memory error (see this question).

You can try using a larger machine type, particularly for the master, e.g. large_model, complex_model_l, or complex_model_l_gpu. You do this by passing a file to the --config argument of gcloud with contents similar to the following:

trainingInput:
  runtimeVersion: "1.0"
  scaleTier: CUSTOM
  masterType: complex_model_l_gpu
  workerCount: 9
  workerType: standard_gpu
  parameterServerCount: 3
  parameterServerType: standard
rhaertel80
  • 8,254
  • 1
  • 31
  • 47
  • I am using the following --config file: `trainingInput: runtimeVersion: "1.0" scaleTier: CUSTOM masterType: standard_gpu workerCount: 5 workerType: standard_gpu parameterServerCount: 3 parameterServerType: standard` Also, my tensorflow is not tensorflow GPU - maybe that's an issue as well? – Antonia Calvi Aug 16 '17 at 15:27
  • The problem is your masterType: standard has too little RAM. When you submit a job with GPU workers, then it runs on a machine with GPU-enabled TensorFlow. This should speed up training using the object detection API. However, you are also welcome to try without GPUs, but I don't think that's the problem. – rhaertel80 Aug 17 '17 at 04:38