0

I'm currently doing exploration on deploying models on Google ML Engine. At first, I developed a model using TensorFlow 1.1.0 as it's the latest version exist (by the time this question is asked). However, it turned out that the highest supported version of TensorFlow on GCP is 1.0.1.

The problem is, previously when I was using TensorFlow 1.1.0, SavedModelBuilder would correctly save the model as SavedModel and its variables under variables/ directory. However, when I switch to TensorFlow 1.0.1, it didn't work similarly: The SavedModel file was created, but no files was created under variables/ and hence no model can be built using only the SavedModel file (missing files under variables/).

Is it a known bug? Or should I do something in order to make the SavedModelBuilder on TensorFlow 1.0.1 works as what TensorFlow 1.1.0 do?

Thank you.

EDIT, more detail:

Actually, there is no explicit tf.Variables exist in my model. However, there exist several tf.contrib.lookup.MutableDenseHashTables and they're exported correctly in TensorFlow 1.1.0, but not in TensorFlow 1.0.1 (as no variable was exported at all in 1.0.1).

sokokaleb
  • 115
  • 7
  • To clarify, you're saying that the `tf.contrib.lookup.MutableDenseHashTable` is exported as a variable in 1.1.0, but not 1.0.1? – jwayne May 05 '17 at 10:19
  • @jwayne I'm not sure if it's exported as a variable. However, saving and loading the model (including the `MutableDenseHashTable` using `SavedModelBuilder` in 1.1.0 works fine, but not in 1.0.1. – sokokaleb May 05 '17 at 11:52

1 Answers1

0

It looks like the ability to save and load models in TensorFlow without variables was introduced in this commit which is only available in 1.1.0.

As a workaround, you could create a dummy (unused) variable in your model.

Edit: Based on OP update, it sounds like there is a MutableDenseHashTable that isn't being saved out.

You can run TensorFlow 1.1 on CloudML Engine, but it requires manually adding it as an additional package.

First, download the TensorFlow 1.1 wheel. Then specify it as an additional package to your training job, e.g.,

gcloud ml-engine jobs submit training my_job \
    --module-name trainer.task \
    --staging-bucket gs://my-bucket \
    --package-path /my/code/path/trainer \
    --packages tensorflow-1.1.0-cp27-cp27mu-manylinux1_x86_64.whl
rhaertel80
  • 8,254
  • 1
  • 31
  • 47
  • Actually, as the SavedModel will be used in serving (as a serving model), what I need to save is the variable(s) and its values so that both the graph and its variables+content can directly be loaded once restored. – sokokaleb May 04 '17 at 17:39