I want to add explanation to my model running in Vertex AI using the Vertex AI SDK.I get a silent error when running the batch prediction using ModelBatchPredictOp, where the ModelBatchPredictOp node runs infinitely.Here is my ModelBatchPredictOp definition;
ModelBatchPredictOp(
project=project_id,
job_display_name="tensorflow-ex-batch-prediction-job",
location=project_location,
model=champion_model.outputs["model"],
instances_format="csv",
predictions_format="jsonl",
gcs_source_uris=gcs_source_uris,
gcs_destination_output_uri_prefix=gcs_destination_output_uri_prefix,
machine_type=batch_prediction_machine_type,
starting_replica_count=batch_prediction_min_replicas,
max_replica_count=batch_prediction_max_replicas,
generate_explanation=True,
)
I have narrowed down the issue to inputTensorName key in the INPUTMETADATA spec.The 'inputTensorName' key in the INPUTMETADATA spec takes in a string for it's value(INPUTMETADATA spec). In my case I have a tensorflow model defined using the functional API meaning it has multiple inputs, as shown below;
# numeric/categorical features in Chicago trips dataset to be preprocessed
NUM_COLS = ["dayofweek", "hourofday", "trip_distance", "trip_miles", "trip_seconds"]
ORD_COLS = ["company"]
OHE_COLS = ["payment_type"]
def build_and_compile_model(dataset: Dataset, model_params: dict) -> Model:
"""Build and compile model.
Args:
dataset (Dataset): training dataset
model_params (dict): model parameters
Returns:
model (Model): built and compiled model
"""
# create inputs (scalars with shape `()`)
num_ins = {
name: Input(shape=(), name=name, dtype=tf.float32) for name in NUM_COLS
}
ord_ins = {
name: Input(shape=(), name=name, dtype=tf.string) for name in ORD_COLS
}
cat_ins = {
name: Input(shape=(), name=name, dtype=tf.string) for name in OHE_COLS
}
# join all inputs and expand by 1 dimension. NOTE: this is useful when passing
# in scalar inputs to a model in Vertex AI batch predictions or endpoints e.g.
# `{"instances": {"input1": 1.0, "input2": "str"}}` instead of
# `{"instances": {"input1": [1.0], "input2": ["str"]}`
all_ins = {**num_ins, **ord_ins, **cat_ins}
exp_ins = {n: tf.expand_dims(i, axis=-1) for n, i in all_ins.items()}
# preprocess expanded inputs
num_encoded = [normalization(n, dataset)(exp_ins[n]) for n in NUM_COLS]
ord_encoded = [str_lookup(n, dataset, "int")(exp_ins[n]) for n in ORD_COLS]
ohe_encoded = [str_lookup(n, dataset, "one_hot")(exp_ins[n]) for n in OHE_COLS]
# ensure ordinal encoded layers is of type float32 (like the other layers)
ord_encoded = [tf.cast(x, tf.float32) for x in ord_encoded]
# concat encoded inputs and add dense layers including output layer
x = num_encoded + ord_encoded + ohe_encoded
x = Concatenate()(x)
for units, activation in model_params["hidden_units"]:
x = Dense(units, activation=activation)(x)
x = Dense(1, name="output", activation="linear")(x)
model = Model(inputs=all_ins, outputs=x, name="nn_model")
model.summary()
logging.info(f"Use optimizer {model_params['optimizer']}")
optimizer = optimizers.get(model_params["optimizer"])
optimizer.learning_rate = model_params["learning_rate"]
model.compile(
loss=model_params["loss_fn"],
optimizer=optimizer,
metrics=model_params["metrics"],
)
return model
As a consequence when getting the input layer, using;
serving_input = list(
loaded_model.signatures["serving_default"].structured_input_signature[1].keys()
)[0]
INPUT_METADATA = {
"input_tensor_name": serving_input,
"encoding": "BAG_OF_FEATURES",
"modality": "numeric",
"index_feature_mapping": cols,
}
I only get one Input layer corresponding to one of the input tensors (cols) in either NUM_COLS, ORD_COLS or OHE_COLS. This causes an infinite run when running the ModelBatchPredictOp in the prediction pipeline as only the name to one input tensor is passed as the value to the inputTensorName. Running
list(model.signatures["serving_default"].structured_input_signature[1].keys())
returns the a list of all the input layers corresponding to the input tensor names (cols) defined in NUM_COLS, ORD_COLS and OHE_COLS.
How do I specify the value of inputTensorName in order to capture all the input layers? Or is there a work around to assign multiple input tensor names to inputTensorName?
