gcloud ml-engine local predict --text-instances fails with "Could not parse" error

Question

I'm trying to make the tensorflow boston sample (https://github.com/tensorflow/tensorflow/tree/master/tensorflow/examples/tutorials/input_fn) work on google cloudml and I seem to be successfull with the training, but I struggle with the subsequent predictions.

I've tweaked the code to fit with tf.contrib.learn.Experiment and learn_runner.run(). It runs both locally and in the cloud with "gcloud ml-engine local train ..."/"gcloud ml-engine jobs submit training ...".
I can with the trained model run estimator.predict(input_fn=predict_input_fn)) and get meaningful predictions with the given boston_predict.csv set.
I can create and version the model in the cloud with "gcloud ml-engine models create ..." and "gcloud ml-engine versions create ..."

But

Local predictions over "gcloud ml-engine local predict --model-dir=/export/Servo/XXX --text-instances boston_predict.csv" fails with a "InvalidArgumentError (see above for traceback): Could not parse example input <..> (Error code: 2). See below for transcript. It fails similarly with a headerless boston_predict.csv.

I've looked up the expected format with "$ gcloud ml-engine local predict --help ", read the https://cloud.google.com/ml-engine/docs/how-tos/troubleshooting but in general failed to find via google or stackexhange reports of my specific error.

I'm a noob, so I'm probably erring in some basic way, but I cannot spot it.

All and any help is appreciated,

:-)

yarc68000.

-------environment----------

(env1) $ gcloud --version
Google Cloud SDK 170.0.0
alpha 2017.03.24
beta 2017.03.24
bq 2.0.25
core 2017.09.01
datalab 20170818
gcloud 
gsutil 4.27

(env1) $ python --version
Python 2.7.13 :: Anaconda 4.3.1 (64-bit)

(env1) $ conda list | grep tensorflow
tensorflow                1.3.0                     <pip>
tensorflow-tensorboard    0.1.6                     <pip>

------------execution and error : boston_predict.csv ----------

$ gcloud ml-engine local predict --model-dir=<..>/export/Servo/1504780684 --text-instances 1709boston/boston_predict.csv
<..>
ERROR:root:Exception during running the graph: Could not parse example input, value: 'CRIM,ZN,INDUS,NOX,RM,AGE,DIS,TAX,PTRATIO'
[[Node: ParseExample/ParseExample = ParseExample[Ndense=9, Nsparse=0, Tdense=[DT_FLOAT, DT_FLOAT, DT_FLOAT, DT_FLOAT, DT_FLOAT, DT_FLOAT, DT_FLOAT, DT_FLOAT, DT_FLOAT], dense_shapes=[[1], [1], [1], [1], [1], [1], [1], [1], [1]], sparse_types=[], _device="/job:localhost/replica:0/task:0/cpu:0"](_arg_Placeholder_0_0, ParseExample/ParseExample/names, ParseExample/ParseExample/dense_keys_0, ParseExample/ParseExample/dense_keys_1, ParseExample/ParseExample/dense_keys_2, ParseExample/ParseExample/dense_keys_3, ParseExample/ParseExample/dense_keys_4, ParseExample/ParseExample/dense_keys_5, ParseExample/ParseExample/dense_keys_6, ParseExample/ParseExample/dense_keys_7, ParseExample/ParseExample/dense_keys_8, ParseExample/Const, ParseExample/Const_1, ParseExample/Const_2, ParseExample/Const_3, ParseExample/Const_4, ParseExample/Const_5, ParseExample/Const_6, ParseExample/Const_7, ParseExample/Const_8)]]
<..>

------- execution and error headerless boston_predict.csv ------

(here I try with a boston_predict.csv with the first line omitted)

$ gcloud ml-engine local predict --model-dir=<..>/export/Servo/1504780684 --text-instances 1709boston/boston_predict_headerless.csv
<..>
ERROR:root:Exception during running the graph: Could not parse example input, value: '0.03359,75.0,2.95,0.428,7.024,15.8,5.4011,252,18.3'
[[Node: ParseExample/ParseExample = ParseExample[Ndense=9, Nsparse=0, Tdense=[DT_FLOAT, DT_FLOAT, DT_FLOAT, DT_FLOAT, DT_FLOAT, DT_FLOAT, DT_FLOAT, DT_FLOAT, DT_FLOAT], dense_shapes=[[1], [1], [1], [1], [1], [1], [1], [1], [1]], sparse_types=[], _device="/job:localhost/replica:0/task:0/cpu:0"](_arg_Placeholder_0_0, ParseExample/ParseExample/names, ParseExample/ParseExample/dense_keys_0, ParseExample/ParseExample/dense_keys_1, ParseExample/ParseExample/dense_keys_2, ParseExample/ParseExample/dense_keys_3, ParseExample/ParseExample/dense_keys_4, ParseExample/ParseExample/dense_keys_5, ParseExample/ParseExample/dense_keys_6, ParseExample/ParseExample/dense_keys_7, ParseExample/ParseExample/dense_keys_8, ParseExample/Const, ParseExample/Const_1, ParseExample/Const_2, ParseExample/Const_3, ParseExample/Const_4, ParseExample/Const_5, ParseExample/Const_6, ParseExample/Const_7, ParseExample/Const_8)]]
<..>

Follow-up question: do you intend to use the model for online prediction? If so, I'd recommend using JSON as input. — rhaertel80, Sep 07 '17 at 14:36

score 0 · Accepted Answer · answered Sep 07 '17 at 14:51

There are likely two problems.

First, it looks as though the graph that you are exporting is expecting tf.Example protos as input, i.e. has a parse_example(...) op in it. The Boston sample does not appear to be adding that op, so I suspect that is part of your modifications.

Before showing the code you want for the input_fn, we need to talk about the second problem: versioning. Estimators existed in previous versions of TensorFlow under tensorflow.contrib. However, various parts have migrated into tensorflow.estimator with successive TensorFlow versions and the APIs have changed as they've moved.

CloudML Engine currently (as of 07 Sep 2017) only supports TF 1.0 and 1.2, so I'll provide a solution that works with 1.2. This is based on the census sample. This is the input_fn you need in order to use CSV data, although I generally recommend exporting models that are independent of input format:

# Provides the data types for the various columns.
FEATURE_DEFAULTS=[[0.0], [0.0], [0.0], [0.0], [0.0], [0.0], [0.0], [0], [0.0]]

def predict_input_fn(rows_string_tensor):
  # Takes a rank-1 tensor and converts it into rank-2 tensor
  # Example if the data is ['csv,line,1', 'csv,line,2', ..] to
  # [['csv,line,1'], ['csv,line,2']] which after parsing will result in a
  # tuple of tensors: [['csv'], ['csv']], [['line'], ['line']], [[1], [2]]
  row_columns = tf.expand_dims(rows_string_tensor, -1)
  columns = tf.decode_csv(row_columns, record_defaults=FEATURE_DEFAULTS)
  features = dict(zip(FEATURES, columns))

  return tf.contrib.learn.InputFnOps(features, None, {'csv_row': csv_row})

And you'll need an export strategy like this:

saved_model_export_utils.make_export_strategy(
    predict_input_fn,
    exports_to_keep=1,
    default_output_alternative_key=None,
)

which you'll pass as a list of size 1 to the constructor of tf.contrib.learn.Experiment.

Great. That worked. I'm still digesting why though. For training I use a tf.estimator.inputs.pandas_input_fn() (from the boston sample) that works for input. More abstractly is the problem I had that this call creates placeholders that are tied to this particular pandas dataframe, and that for prediction, I need a different input placeholder that is independent of the training input? If that at all made sense.. If you have a url that teaches me this part better I'm grateful. — yarc68000, Sep 08 '17 at 12:39
Unfortunately, I don't know of a good place to get that info. I've made a note to ensure the info gets covered in docs, although it may take a bit for that to happen. The long and short of it is that TF's architecture requires you to build two totally separate graphs for training and prediction. tf.estimator hides most of those details (if you write a custom estimator, you'll notice you need to deal with a `mode` argument when building your graph) except for one thing: the `input_fn`. During training, input usually comes from files, passes through queues, etc. During prediction, it's just fed. — rhaertel80, Sep 08 '17 at 14:28

gcloud ml-engine local predict --text-instances fails with "Could not parse" error

1 Answers1