1

I have trained and saved a model using Amazon SageMaker which saves the model in the format of model.tar.gz which when untarred, has a file model_algo-1 which is a serialized Apache MXNet object. To load the model in memory I need to deserialize the model. I tried doing so as follows:

import mxnet as mx print(mx.ndarray.load('model_algo-1'))

Reference taken from https://docs.aws.amazon.com/sagemaker/latest/dg/cdf-training.html

However, doing this yields me the following error:

Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "/usr/local/lib/python3.4/site-packages/mxnet/ndarray/utils.py", line 
175, in load
ctypes.byref(names)))
File "/usr/local/lib/python3.4/site-packages/mxnet/base.py", line 146, in 
check_call
raise MXNetError(py_str(_LIB.MXGetLastError()))
mxnet.base.MXNetError: [19:06:25] src/ndarray/ndarray.cc:1112: Check failed: 
header == kMXAPINDArrayListMagic Invalid NDArray file format

Stack trace returned 10 entries:
[bt] (0) /usr/local/lib/python3.4/site-packages/mxnet/libmxnet.so(+0x192112) 
[0x7fe432bfa112]
[bt] (1) /usr/local/lib/python3.4/site-packages/mxnet/libmxnet.so(+0x192738) 
[0x7fe432bfa738]
[bt] (2) /usr/local/lib/python3.4/site-
packages/mxnet/libmxnet.so(+0x24a5c44) [0x7fe434f0dc44]
[bt] (3) /usr/local/lib/python3.4/site-
packages/mxnet/libmxnet.so(MXNDArrayLoad+0x248) [0x7fe434d19ad8]
[bt] (4) /usr/lib64/libffi.so.6(ffi_call_unix64+0x4c) [0x7fe48c5bbcec]
[bt] (5) /usr/lib64/libffi.so.6(ffi_call+0x1f5) [0x7fe48c5bb615]
[bt] (6) /usr/lib64/python3.4/lib-dynload/_ctypes.cpython-
34m.so(_ctypes_callproc+0x2fb) [0x7fe48c7ce18b]
[bt] (7) /usr/lib64/python3.4/lib-dynload/_ctypes.cpython-34m.so(+0xa4cf) 
[0x7fe48c7c84cf]
[bt] (8) /usr/lib64/libpython3.4m.so.1.0(PyObject_Call+0x8c) 
[0x7fe4942fcb5c]
[bt] (9) /usr/lib64/libpython3.4m.so.1.0(PyEval_EvalFrameEx+0x36c5) 
[0x7fe4943ac915]

Could someone suggest how this can be resolved?

Vasanti
  • 1,207
  • 2
  • 12
  • 24
  • How did you create model.tar.gz? Could you please share the code that created this file? – Indhu Bharathi Apr 08 '18 at 17:23
  • The link you provided says k-means model can be loaded using the code you used. Note that not all models can be saved/loaded as a NDArray. What model are you using? – Indhu Bharathi Apr 09 '18 at 06:43
  • @IndhuBharathi I am using a Linear Learner model. Could you suggest how can this model be loaded? Also, it would be great if you could point out some documentation which suggests what is the format for loading the models for each of the algorithms. – Vasanti Apr 09 '18 at 18:19

2 Answers2

1

If your model is serialized into archive properly, then there should be at least 2 files:

  1. model_name.json - it contains the architecture of your model
  2. model_name.params - it contains parameters of your model

So, to load the model back, you need to:

  1. Restore the model itself by loading json file.
  2. Restore model parameters (you don't use mxnet nd.array for that, but full model).

Here is the code example how to do it:

# sym_json - content of .json file    
net = gluon.nn.SymbolBlock(
        outputs=mx.sym.load_json(sym_json),
        inputs=mx.sym.var('data'))

# params_filename - full path to parameters file
net.load_params(params_filename)

If you also want to check serialization of your model as well, take a look to this example. This example shows how to serialize a trained model manually before uploading to SageMaker.

More details on serializing and deserializing model manually can be found here.

Sergei
  • 1,617
  • 15
  • 31
1

I trained a stock Linear Learner algorithm through AWS Sagemaker. It makes a model object called model.tar.gz in the output folder. As Vasanti noted, there is some notation that these objects are mxnet object in an article

I knew I had to unpack the tar, what I didn't realize was how many times. I started with this code:

import subprocess  
cmdline = ['tar','-xzvf','model.tar.gz'] 
subprocess.call(cmdline)

which yields the file called 'model_algo-1' which led me to this page. However, it's still a packed file. So run:

cmdline = ['tar','-xzvf','model_algo-1'] 
subprocess.call(cmdline)

This yields:

  • additional-params.json
  • manifest.json
  • mx-mod-0000.params
  • mx-mod-symbol.json

From there, you can utilize Sergei's post:

# load the json file
import json
sym_json = json.load(open('mx-mod-symbol.json'))
sym_json_string = json.dumps(sym_json)

# open model
import mxnet as mx
from mxnet import gluon
net = gluon.nn.SymbolBlock(
        outputs=mx.sym.load_json(sym_json_string),
        inputs=mx.sym.var('data'))

# params file
net.load_parameters('mx-mod-0000.params', allow_missing=True)

Now, if only I knew what to do with this mxnet / gluon object to get what I really want which is a feature importance rank order and weight for some model explainability.

Doubledown
  • 458
  • 1
  • 6
  • 13
  • Using sub-process to deserialise the model did not work for me. what worked is: `import os os.system('unzip model_algo-1')`, which i got from here: https://stackoverflow.com/questions/51790953/logistic-regression-in-sagemaker – anything_everything Aug 10 '22 at 01:16