sagemaker invoke_endpoint signature_def feature prep

Question

I have a sagemaker tensorflow model using a custom estimator, similar to the abalone.py sagemaker tensorflow example, using build_raw_serving_input_receiver_fn in the serving_input_fn:

def serving_input_fn(params):
    tensor = tf.placeholder(tf.float32, shape=[1, NUM_FEATURES])
    return build_raw_serving_input_receiver_fn({INPUT_TENSOR_NAME: tensor})()

Predictions are being request from java-script using json:

  response = @client.invoke_endpoint(
    endpoint_name: @name,
    content_type: "application/json",
    accept: "application/json",
    body: values.to_json
    )

Everything fine so far. Now I want to add some feature engineering (scaling transformations on the features using a scaler derived from the training data). Following the pattern of the answer for Data Normalization with tensorflow tf-transform I've now got serving_input_fn like this:

def serving_input_fn(params):
    feature_placeholders = {
        'f1': tf.placeholder(tf.float32, [None]),
        'f2': tf.placeholder(tf.float32, [None]),
        'f3': tf.placeholder(tf.float32, [None]),
    }
    features = {
        key: tf.expand_dims(tensor, -1)
        for key, tensor in feature_placeholders.items()
    }
    return tf.estimator.export.ServingInputReceiver(add_engineering(features), feature_placeholders)

From saved_model_cli show --dir . --all I can see the input signature has changed:

signature_def['serving_default']:
The given SavedModel SignatureDef contains the following input(s):
inputs['f1'] tensor_info:
    dtype: DT_FLOAT
    shape: (-1)
    name: Placeholder_1:0
inputs['f2'] tensor_info:
    dtype: DT_FLOAT
    shape: (-1)
    name: Placeholder_2:0
inputs['f3'] tensor_info:
    dtype: DT_FLOAT
    shape: (-1)
    name: Placeholder:0

How do I prepare features for prediction from this new model? In python I've been unsuccessfully trying things like

requests = [{'f1':[0.1], 'f2':[0.1], 'f3':[0.2]}]
predictor.predict(requests)

also need to send prediction requests from java-script.

Raman · Answer 1 · 2018-03-26T19:55:24.520

You can define an

def input_fn(data=None, content_type=None):

This would be called directly when a call is made to SageMaker. You can do your feature preparation in this function. model_fn would be called after this function. Make sure you return a dict of string and TensorProto. dict{"input tensor name", TensorProto} from the input_fn method.

You can find more details where

https://docs.aws.amazon.com/sagemaker/latest/dg/tf-training-inference-code-template.html

A sample input_fn would look something like below

def input_fn(data=None, content_type=None):
    """
    Args:
        data: An Amazon SageMaker InvokeEndpoint request body
        content_type: An Amazon SageMaker InvokeEndpoint ContentType value for data.
    Returns:
        object: A deserialized object that will be used by TensorFlow serving as input.
    """

    # `inputs` is based on the parameters defined in the model spec's signature_def
    return {"inputs": tf.make_tensor_proto(data, shape=(1,))}

thanks, will try. I had thought of this but it felt like a workaround...also would like to have found sagemaker's default input_fn in code to take as a template but couldn't find it. — JonL, Mar 26 '18 at 18:34

score 0 · Answer 2 · answered Mar 27 '18 at 19:35

Have managed to make the feature values available on their way into prediction via a sagemaker input_fn definition, as suggested by Raman. It means going back to the build_raw_serving_input_receiver_fn serving_input_fn I started with (top of post). The input_fn looks like this:

def input_fn(data=None, content_type=None):
    if content_type == 'application/json':
        values = np.asarray(json.loads(data))
        return {"inputs": tf.make_tensor_proto(values=values, shape=values.shape, dtype=tf.float32)}
    else:
        return {"inputs": data}

Although I can't pass e.g. a scaler from training into this procedure, it will probably work to embed it in the model.py file that sagemaker requires (which contains this input_fn defn). What I have is responding correctly to by addressed from python either by

data = [[0.1, 0.2, 0.3]]
payload = json.dumps(data)

response = client.invoke_endpoint(
    EndpointName=endpoint_name,
    Body=payload,
    ContentType='application/json'
)
result = json.loads(response['Body'].read().decode())

or

values = np.asarray([[0.1, 0.2, 0.3]])
prediction = predictor.predict(values)

This is all new to me... please recommend improvements/alert me to potential problems if you know of any.

sagemaker invoke_endpoint signature_def feature prep

2 Answers2