2

Why does keras.Model.save() produce different binaries with every run, when, AFAIU, I have taken all the necessary steps for complete reproducibility of the results and even binaries?

You can verify this by simply executing the following script within a docker container. The saved_model.pb and contents of the variables directory always change. Create a container with:
docker run --rm --volume=/host/dir/with/script:/workspace -it tensorflow/tensorflow:2.4.1 bash

# Setup environmental variables.
import os
os.environ["TF_DETERMINISTIC_OPS"] = "1"
os.environ["TF_CPP_MIN_LOG_LEVEL"] = "2"

import numpy as np
import random
import tensorflow as tf
import time


def initialise_seeds(seed):
    os.environ["PYTHONHASHSEED"] = str(seed)
    random.seed(seed)
    np.random.seed(seed)
    tf.random.set_seed(seed)


SEED = 42
initialise_seeds(SEED)

if __name__ == '__main__':
    mnist = tf.keras.datasets.mnist

    (x_train, y_train), (x_test, y_test) = mnist.load_data()
    x_train, x_test = x_train / 255.0, x_test / 255.0

    model = tf.keras.models.Sequential([
        tf.keras.layers.Flatten(input_shape=(28, 28)),
        tf.keras.layers.Dense(128, activation='relu'),
        tf.keras.layers.Dropout(0.2),
        tf.keras.layers.Dense(10),
        tf.keras.layers.Softmax()
    ], name="best_ever")

    model.compile(
        optimizer='adam',
        loss=tf.keras.losses.SparseCategoricalCrossentropy(),
        metrics=['accuracy']
    )
    model.fit(x_train, y_train, epochs=2)
    model.evaluate(x_test,  y_test, verbose=2)

    # Save the model weights.
    ckpt = "/cp-0.ckpt"
    model.save_weights(model.name + ckpt)
    # Also save keras model.
    model.save(model.name)

    # Convert the model.
    converter = tf.lite.TFLiteConverter.from_keras_model(model)
    # Include necessary functions.
    converter.target_spec.supported_ops = [tf.lite.OpsSet.TFLITE_BUILTINS,
                                           tf.lite.OpsSet.SELECT_TF_OPS]
    tflite_model = converter.convert()

    # Save the model.
    with open('model.tflite', 'wb') as f:
        f.write(tflite_model)

    # Load the TFLite model from disk and allocate tensors.
    interpreter = tf.lite.Interpreter(model_path="model.tflite")
    interpreter.allocate_tensors()

    # Get input and output tensors.
    input_details = interpreter.get_input_details()
    output_details = interpreter.get_output_details()

    # Test model ouput on the same data to ensure correct function. 
    input_shape = input_details[0]['shape']
    input_data = np.ones((input_shape), dtype=np.float32) * 0.5
    interpreter.set_tensor(input_details[0]['index'], input_data)

    interpreter.invoke()

    # The function `get_tensor()` returns a copy of the tensor data.
    # Use `tensor()` in order to get a pointer to the tensor.
    output_data = interpreter.get_tensor(output_details[0]['index'])
    print(output_data.shape)

Thanks.

  • Its not keras, it is more likely that it is the HDF5 library (h5py), since datasets inside the HDF5 file also have metadata like timestamps. It is also possible that the TF formats also have such information that changes between runs. – Dr. Snoopy Apr 22 '21 at 15:09
  • @Dr.Snoopy, my example is exporting in .pb format. AFAIK, this doesn't use HDF5? I also assumed it might be some timestamp that is causing this, but I was hoping I can get some pointers before I go deep in the tensorflow code. – thomasarvanitidis Apr 23 '21 at 10:13
  • I have observed something very similar, follow https://github.com/tensorflow/tensorflow/issues/54067 if interested. – bers Jan 25 '22 at 12:58

0 Answers0