How to prepare warmup request file for tensorflow serving?

Question

Current version of tensorflow-serving try to load warmup request from assets.extra/tf_serving_warmup_requests file.

2018-08-16 16:05:28.513085: I tensorflow_serving/servables/tensorflow/saved_model_warmup.cc:83] No warmup data file found at /tmp/faster_rcnn_inception_v2_coco_2018_01_28_string_input_version-export/1/assets.extra/tf_serving_warmup_requests

I wonder if tensorflow provides common api to export request to the location or not? Or should we write request to the location manually?

Christis · Accepted Answer · 2018-08-23T23:22:43.087

8

At this point there is no common API for exporting the warmup data into the assets.extra. It's relatively simple to write a script (similar to below):

import tensorflow as tf
from tensorflow_serving.apis import model_pb2
from tensorflow_serving.apis import predict_pb2
from tensorflow_serving.apis import prediction_log_pb2

def main():
    with tf.python_io.TFRecordWriter("tf_serving_warmup_requests") as writer:
        request = predict_pb2.PredictRequest(
            model_spec=model_pb2.ModelSpec(name="<add here>"),
            inputs={"examples": tf.make_tensor_proto([<add here>])}
        )
    log = prediction_log_pb2.PredictionLog(
        predict_log=prediction_log_pb2.PredictLog(request=request))
    writer.write(log.SerializeToString())

if __name__ == "__main__":
    main()

edited Aug 23 '18 at 23:22

answered Aug 23 '18 at 22:12

Christis

96
2

Thanks! I pretty much did the same. Just wonder if there's standarder way. – tianyapiaozi Aug 24 '18 at 01:39
Finished reading warmup data for model at /home/serving/models/1/assets.extra/tf_serving_warmup_requests. Number of warmup records read: 150. --------------------------------- After adding the warmup tfrecords, where do I put the script or do I have to call the script frequently or by just passing the tfrecord, tfserving will itself take care of the frequent calls to improve latency. – The Gr8 Adakron Apr 06 '19 at 05:48

score 2 · Answer 2 · edited Mar 25 '22 at 16:09

2

We refered to the official doc

Specially, we used Classification instead of Prediction, so we altered that code to be

log = prediction_log_pb2.PredictionLog(
            classify_log=prediction_log_pb2.ClassifyLog(request=<request>))

edited Mar 25 '22 at 16:09

rodrigo-silveira

12,607
11
69
123

answered Oct 26 '18 at 09:47

Xin Liang

21
3

Official doc now at https://www.tensorflow.org/tfx/serving/saved_model_warmup – Stefan Jul 03 '21 at 10:38

gogasca · Answer 3 · 2020-01-24T07:02:09.710

This is a complete example of an object detection system using a ResNet model. The prediction consist of an image.

import tensorflow as tf
import requests
import base64

from tensorflow.python.framework import tensor_util
from tensorflow_serving.apis import predict_pb2
from tensorflow_serving.apis import prediction_log_pb2


IMAGE_URL = 'https://tensorflow.org/images/blogs/serving/cat.jpg'
NUM_RECORDS = 100


def get_image_bytes():
    image_content = requests.get(IMAGE_URL, stream=True)
    image_content.raise_for_status()
    return image_content.content


def main():
    """Generate TFRecords for warming up."""

    with tf.io.TFRecordWriter("tf_serving_warmup_requests") as writer:
        image_bytes = get_image_bytes()
        predict_request = predict_pb2.PredictRequest()
        predict_request.model_spec.name = 'resnet'
        predict_request.model_spec.signature_name = 'serving_default'
        predict_request.inputs['image_bytes'].CopyFrom(
            tensor_util.make_tensor_proto([image_bytes], tf.string))        
        log = prediction_log_pb2.PredictionLog(
            predict_log=prediction_log_pb2.PredictLog(request=predict_request))
        for r in range(NUM_RECORDS):
            writer.write(log.SerializeToString())    

if __name__ == "__main__":
    main()

This script will create a file called “tf_serving_warmup_requests”

I moved this file to /your_model_location/resnet/1538687457/assets.extra/ and then restart my docker image to pickup the new changes.

How to prepare warmup request file for tensorflow serving?

3 Answers3