gcloud problems with local prediction

Question

I am using gcloud local prediction to test my exported model. The model is a TensorFlow object detection model which has been trained on custom dataset. I am using the following gcloud command:

gcloud ml-engine local predict --model-dir=/path/to/saved_model/ --json-instances=input.json --signature-name="serving_default" --verbosity debug

When I am not using verbose the command does not output anything. With verbose set as debug I get the following traceback:

DEBUG: [Errno 32] Broken pipe
Traceback (most recent call last):
  File "/google-cloud-sdk/lib/googlecloudsdk/calliope/cli.py", line 984, in Execute
    resources = calliope_command.Run(cli=self, args=args)
  File "/google-cloud-sdk/lib/googlecloudsdk/calliope/backend.py", line 784, in Run
    resources = command_instance.Run(args)
  File "/google-cloud-sdk/lib/surface/ai_platform/local/predict.py", line 83, in Run
    signature_name=args.signature_name)
  File "/google-cloud-sdk/lib/googlecloudsdk/command_lib/ml_engine/local_utils.py", line 103, in RunPredict
    proc.stdin.write((json.dumps(instance) + '\n').encode('utf-8'))
IOError: [Errno 32] Broken pipe

Details on my exported model :

MetaGraphDef with tag-set: 'serve' contains the following SignatureDefs:

signature_def['serving_default']:
  The given SavedModel SignatureDef contains the following input(s):
    inputs['inputs'] tensor_info:
        dtype: DT_STRING
        shape: (-1)
        name: encoded_image_string_tensor:0
  The given SavedModel SignatureDef contains the following output(s):
    outputs['detection_boxes'] tensor_info:
        dtype: DT_FLOAT
        shape: (-1, 300, 4)
        name: detection_boxes:0
    outputs['detection_classes'] tensor_info:
        dtype: DT_FLOAT
        shape: (-1, 300)
        name: detection_classes:0
    outputs['detection_features'] tensor_info:
        dtype: DT_FLOAT
        shape: (-1, -1, -1, -1, -1)
        name: detection_features:0
    outputs['detection_multiclass_scores'] tensor_info:
        dtype: DT_FLOAT
        shape: (-1, 300, 2)
        name: detection_multiclass_scores:0
    outputs['detection_scores'] tensor_info:
        dtype: DT_FLOAT
        shape: (-1, 300)
        name: detection_scores:0
    outputs['num_detections'] tensor_info:
        dtype: DT_FLOAT
        shape: (-1)
        name: num_detections:0
    outputs['raw_detection_boxes'] tensor_info:
        dtype: DT_FLOAT
        shape: (-1, 300, 4)
        name: raw_detection_boxes:0
    outputs['raw_detection_scores'] tensor_info:
        dtype: DT_FLOAT
        shape: (-1, 300, 2)
        name: raw_detection_scores:0
  Method name is: tensorflow/serving/predict

I used the following code to generate my input.json for prediction :

with open('input.json', 'wb') as f:
    img = Image.open("image.jpg")
    img = img.resize((width, height), Image.ANTIALIAS)
    output_str = io.BytesIO()
    img.save(output_str, "JPEG")
    image_byte_array = output_str.getvalue()
    image_base64 = base64.b64encode(image_byte_array)
    json_entry = {"b64": image_base64.decode()}
    #instances.append(json_entry
    request = json.dumps({'inputs': json_entry})
    f.write(request.encode('utf-8'))
f.close()

{"inputs": {"b64": "/9j/4AAQSkZJRgABAQAAAQABAAD/......}}

I am testing the prediction with one image.

how large is your input.json file? and what version is python? — Travis Webb, Oct 28 '19 at 02:09
@TravisWebb Python version 3.6.5, size of input.json 137KB (I am sending one one image in prediction request), Google Cloud SDK 268.0.0, beta 2019.05.17 bq 2.0.49, core 2019.10.18, gsutil 4.45 — CyberPunk, Oct 28 '19 at 07:56
@gogasca I was not able to fix this. It was failing on gcp ml engine as well. Here is the thread that was related to prediction on gcp ml engine. https://groups.google.com/forum/#!topic/ml-engine/mo8o54pQCak — CyberPunk, Dec 12 '19 at 08:13
latest suggestion was to reduce number of instances from 300, did that work? — gogasca, Dec 12 '19 at 09:08
I see we have a limit of 100 can you update if you are able to predict with local predict and API ? — gogasca, Dec 12 '19 at 09:33
can you point me to somewhere in the docs where it is written ? — CyberPunk, Dec 12 '19 at 11:32
I dont see it in the docs, already reached out to docs team to update/point me to right document....but I tested it, if you follow my reply on thread you should be able to test it yourself. In the SDK code which you can see under google-cloud-sdk/lib/googlecloudsdk/command_lib/ml_engine/predict_utilities.py limit is set to 100 if limit and line_num >= limit: raise InvalidInstancesFileError( 'Online prediction can process no more than ' + six.text_type(limit) + ' instances per file. Please use batch prediction instead.') — gogasca, Dec 13 '19 at 00:30
We updated docs: A single online prediction request must contain no more than 1.5 MB of data. Requests created using the gcloud can handle no more than 100 instances per file. To get predictions for more instances at the same time, use batch prediction. — gogasca, Dec 13 '19 at 06:17

score 1 · Answer 1 · answered May 14 '20 at 09:06

I've met the same problem and found out that ml_engine/local_utils.py uses python to run ml_engine/local_predict.pyc that is built for python2.7. My python is python3, so when ml_engine/local_utils.py tries to run ml_engine/local_predict.pyc using python (actually python3), it fails with error:

RuntimeError: Bad magic number in .pyc file

Solution 1:

You can just make python2 as default one in system.

Solution 2:

I changed ml_engine/local_utils.py with such patch:

83c83
<   python_executables = files.SearchForExecutableOnPath("python")
---
>   python_executables = files.SearchForExecutableOnPath("python2")
114a115
>   log.debug(args)
124,126c125,130
<   for instance in instances:
<     proc.stdin.write((json.dumps(instance) + "\n").encode("utf-8"))
<   proc.stdin.flush()
---
>   try:
>     for instance in instances:
>       proc.stdin.write((json.dumps(instance) + "\n").encode("utf-8"))
>     proc.stdin.flush()
>   except:
>     pass

try-catch needed to make script able to read and print an error ocurred while running ml_engine/local_predict.pyc.

score 1 · Answer 2 · answered Nov 06 '20 at 21:53

Unlike @Roman Kovtuh, I was able to run using python3. However, his technique for creating an exception handler allowed me to determine that tensorflow was not installed in the environment that was visible to the process. Once done, the process worked.

My changes to googlecloudsdk/command_lib/ml_engine/local_utils.py:

106,109c106
<     try:
<       proc.stdin.write((json.dumps(instance) + '\n').encode('utf-8'))
<     except Exception as e:
<       print(f'Error displaying errors with instance {str(instance)[:100]}.  Exception {e}')
---
>     proc.stdin.write((json.dumps(instance) + '\n').encode('utf-8'))

I upvoted @Roman Kovtuh because this really helped.

score 0 · Answer 3 · answered Oct 28 '19 at 13:16

0

According to this page, the binary input must be suffixed by _bytes

In your TensorFlow model code, you must name the aliases for your binary input and output tensors so that they end with '_bytes'.

Try to suffix your input with _bytes, or rebuild your model with a compliant input_serving function.

answered Oct 28 '19 at 13:16

guillaume blaquiere

66,369
2
47
76

it means in my exported model instead of "inputs" it should be "inputs_bytes" and likewise in json file ? – CyberPunk Oct 28 '19 at 13:33
Tried it by re exporting the model with signature def "input_bytes" but still the same error. also renamed input json with "input_bytes" still no luck – CyberPunk Oct 28 '19 at 13:47

score 0 · Answer 4 · answered Dec 12 '19 at 08:01

When running the command, the local SDK file /usr/lib/google-cloud-sdk/lib/googlecloudsdk/command_lib/ml_engine/local_utils.py, failure seems to be happening when reading file contents:

  for instance in instances:
    proc.stdin.write((json.dumps(instance) + '\n').encode('utf-8'))
  proc.stdin.flush()

In your case, I expect to see JSON correctly formatted, otherwise we normally get:

ERROR: (gcloud.ai-platform.local.predict) Input instances are not in JSON format. See "gcloud ml-engine predict --help" for details.

This is a snippet of the code I normally use to generate my b64 encoded image with resize.

import base64
from PIL import Image

INPUT_FILE = 'image.jpg'
OUTPUT_FILE = 'image_b64.json'


def convert_to_base64_resize(image_file):
  """Open image, resize, base64 encode it and create a JSON request"""
  img = Image.open(image_file).resize((240, 240))
  img.save(image_file)  
  with open(image_file, 'rb') as f:
    jpeg_bytes = base64.b64encode(f.read()).decode('utf-8')   
    predict_request = '{"image_bytes": {"b64": "%s"}}' % jpeg_bytes 
    # Write JSON to file
    with open(OUTPUT_FILE, 'w') as f:
      f.write(predict_request)
    return predict_request

convert_to_base64_resize(INPUT_FILE)

Will be great to see a copy of your JSON file or image and compare contents.

For normal troubleshooting, I also use tensorflow serving, specially in order to validate that my model works locally. (TensorFlow serving supports pointing to the GCS location) Remember that local prediction with json instances expect this format:

{"image_bytes": {"b64": body }}

I assume your model looks like this after the changes suggested above:

...
signature_def['serving_default']:
  The given SavedModel SignatureDef contains the following input(s):
    inputs['image_bytes'] tensor_info:
        dtype: DT_STRING
        shape: (-1)
        name: input_tensor:0
...

gcloud problems with local prediction

4 Answers4

Solution 1:

Solution 2: