write persisted keras model to s3

Question

I ran an lstm model in keras and persisted it to a json object. I want to write this json object directly to an s3 bucket. Boto3 tutorials really don't cover this, they only show how to write files to s3. I really need to be able to store keras models in json directly from python (in memory) to s3. Any ideas?

bigmac · Answer 1 · 2020-08-10T12:50:57.303

As there is only one other answer to this question, please note that model.to_json() saves only the model architecture and the initialized weights but NOT the trained weights. Unfortunately the keras API documentation is not clear on this, but if you load a model using model_from_json it will run, but using the initial weights. Therefore the output of your model will likely be nonseniscal.

There does not currently appear to be a simple way to write model files directly from memory (although have a look at the example here).

In order to save the model with trained weights use the method to save model objects proposed in the keras documentation.

from keras.models import load_model
import boto3

model.save('my_model.h5')  # creates a HDF5 file 'my_model.h5'
client = boto3.client('s3')
client.upload_file(Filename='my_model.h5',
                  Bucket=BUCKET_NAME,
                  Key='my_model.h5')

del model  # deletes the existing model

To download the model and load it:

client.download_file(BUCKET_NAME,
                     'my_model.h5',
                     'my_model.h5')
# returns a compiled model
# identical to the previous one
model = load_model('my_model.h5')

You can then delete the file again if you do not want to save it locally.

score 1 · Answer 2 · answered Dec 22 '17 at 00:00

You can use Client.put_object() to write to an S3 object directly without saving to and uploading a file. Here's a concrete example:

import boto3

# Convert your existing model to JSON
saved_model = model.to_json()

# Write JSON object to S3 as "keras-model.json"
client = boto3.client('s3')
client.put_object(Body=saved_model,
                  Bucket='BUCKET_NAME',
                  Key='keras-model.json')

You can then download the keras-model.json and then load it back into a Keras model like this using model_from_json.

from keras.models import model_from_json

# Read the downloaded JSON file
with open('keras-model.json', 'r') as model_file:
   loaded_model = model_file.read()

# Convert back to Keras model
model = model_from_json(loaded_model)

# Confirmation
model.summary()

This answer is misleading, as "model_from_json" does not do what is suggested. It loads initial (random) but not the trained weights. Please clarify or delete this answer. — bigmac, Dec 14 '20 at 18:17

score 0 · Answer 3 · answered Jan 07 '21 at 22:10

There is a way to do this even for the h5 type through the h5py library.

For trained models, the h5 type is necessary as it includes the weights. The json version does not, as explained above.

import io
import h5py
import boto3

# Create bytes io as target.
with io.BytesIO() as model_io:
    # Instanciate h5 file using the io.
    with h5py.File(model_io, 'w') as model_h5:
        # Save the Keras model to h5 object (and indirectly to bytesio).
        model.save(model_h5)
        # Make sure the data is written entirely to the bytesio object.
        model_h5.flush()
    # Upload to S3.
    client = boto3.client('s3')
    client.put_object(
        Bucket='BUCKET_NAME',
        Key='keras-model.h5',
        Body=model_io,
    )

I did not test this exact snippet, but I am have used something along these lines before and it worked.

write persisted keras model to s3

3 Answers3