I'm trying to train a object detection model on tensorflow/object-detection and I'm following this github page. As I perform training, it seemes like training job is successfully performed but after looking at the log, I found it is stopped due to this error:
{
insertId: "1vzaw47f4on11p"
logName: "projects/one-staging-e2903/logs/ml.googleapis.com%2Fobject_detection_09_27_2018_18_39_10"
receiveTimestamp: "2018-09-27T09:43:32.946033224Z"
resource: {…}
severity: "ERROR"
textPayload: "The replica ps 0 exited with a non-zero status of 1. Termination reason: Error.
Traceback (most recent call last):
File "/usr/lib/python2.7/runpy.py", line 174, in _run_module_as_main
"__main__", fname, loader, pkg_name)
File "/usr/lib/python2.7/runpy.py", line 72, in _run_code
exec code in run_globals
File "/root/.local/lib/python2.7/site-packages/object_detection/model_main.py", line 29, in <module>
from object_detection import model_lib
File "/root/.local/lib/python2.7/site-packages/object_detection/model_lib.py", line 27, in <module>
from object_detection import eval_util
File "/root/.local/lib/python2.7/site-packages/object_detection/eval_util.py", line 27, in <module>
from object_detection.metrics import coco_evaluation
File "/root/.local/lib/python2.7/site-packages/object_detection/metrics/coco_evaluation.py", line 20, in <module>
from object_detection.metrics import coco_tools
File "/root/.local/lib/python2.7/site-packages/object_detection/metrics/coco_tools.py", line 47, in <module>
from pycocotools import coco
File "/root/.local/lib/python2.7/site-packages/pycocotools/coco.py", line 49
import matplotlibnmatplotlib.use('Agg')nimport matplotlib.pyplot as plt
^
SyntaxError: invalid syntax
The replica ps 1 exited with a non-zero status of 1. Termination reason: Error.
Traceback (most recent call last):
File "/usr/lib/python2.7/runpy.py", line 174, in _run_module_as_main
"__main__", fname, loader, pkg_name)
File "/usr/lib/python2.7/runpy.py", line 72, in _run_code
exec code in run_globals
File "/root/.local/lib/python2.7/site-packages/object_detection/model_main.py", line 29, in <module>
from object_detection import model_lib
File "/root/.local/lib/python2.7/site-packages/object_detection/model_lib.py", line 27, in <module>
from object_detection import eval_util
File "/root/.local/lib/python2.7/site-packages/object_detection/eval_util.py", line 27, in <module>
from object_detection.metrics import coco_evaluation
File "/root/.local/lib/python2.7/site-packages/object_detection/metrics/coco_evaluation.py", line 20, in <module>
from object_detection.metrics import coco_tools
File "/root/.local/lib/python2.7/site-packages/object_detection/metrics/coco_tools.py", line 47, in <module>
from pycocotools import coco
File "/root/.local/lib/python2.7/site-packages/pycocotools/coco.py", line 49
import matplotlibnmatplotlib.use('Agg')nimport matplotlib.pyplot as plt
^
SyntaxError: invalid syntax
The replica ps 2 exited with a non-zero status of 1. Termination reason: Error.
Traceback (most recent call last):
File "/usr/lib/python2.7/runpy.py", line 174, in _run_module_as_main
"__main__", fname, loader, pkg_name)
File "/usr/lib/python2.7/runpy.py", line 72, in _run_code
exec code in run_globals
File "/root/.local/lib/python2.7/site-packages/object_detection/model_main.py", line 29, in <module>
from object_detection import model_lib
File "/root/.local/lib/python2.7/site-packages/object_detection/model_lib.py", line 27, in <module>
from object_detection import eval_util
File "/root/.local/lib/python2.7/site-packages/object_detection/eval_util.py", line 27, in <module>
from object_detection.metrics import coco_evaluation
File "/root/.local/lib/python2.7/site-packages/object_detection/metrics/coco_evaluation.py", line 20, in <module>
from object_detection.metrics import coco_tools
File "/root/.local/lib/python2.7/site-packages/object_detection/metrics/coco_tools.py", line 47, in <module>
from pycocotools import coco
File "/root/.local/lib/python2.7/site-packages/pycocotools/coco.py", line 49
import matplotlibnmatplotlib.use('Agg')nimport matplotlib.pyplot as plt
^
SyntaxError: invalid syntax
To find out more about why your job exited please check the logs: https://console.cloud.google.com/logs/viewer?project=321761520370&resource=ml_job%2Fjob_id%2Fobject_detection_09_27_2018_18_39_10&advancedFilter=resource.type%3D%22ml_job%22%0Aresource.labels.job_id%3D%22object_detection_09_27_2018_18_39_10%22"
timestamp: "2018-09-27T09:43:32.946033224Z"
}
I've searched about this and got this page and changed models/research/object_detection/dataset_tools/create_pycocotools_package.sh
file as described in the answer, but I still get the same error.
Could anyone please solve this? Thank you very much in advance.