Efficient way to write a python dictionary with Numpy Nd-Array values into a Json File

Question

How do I efficiently write a Python dictionary, where the values are Numpy Nd-Arrays to a Json File? I obtain an error saying that the Numpy Nd-Array is not Json-Serializable. Is there any way to overcome this?

Did you try converting the Numpy arrays to (possibly nested) lists? — PM 2Ring, Jun 01 '17 at 08:34
yes, it is possible to convert the Numpy arrays to lists and it then works. My question goes rather in other direction: is there any quicker way to achieve this purpose, without all the converting and copying to lists? — AlexGuevara, Jun 01 '17 at 08:46

score 3 · Accepted Answer · answered Jun 01 '17 at 09:18

JSON only supports a limited number of datatypes. If you want to store other types of data as JSON then you need to convert it to something that JSON accepts. The obvious choice for Numpy arrays is to store them as (possibly nested) lists. Fortunately, Numpy arrays have a .tolist method which performs the conversion efficiently.

import numpy as np
import json

a = np.array(range(25), dtype=np.uint8).reshape(5, 5) 
print(a)
print(json.dumps(a.tolist()))

output

[[ 0  1  2  3  4]
 [ 5  6  7  8  9]
 [10 11 12 13 14]
 [15 16 17 18 19]
 [20 21 22 23 24]]
[[0, 1, 2, 3, 4], [5, 6, 7, 8, 9], [10, 11, 12, 13, 14], [15, 16, 17, 18, 19], [20, 21, 22, 23, 24]]

.tolist will convert the array elements to native Python types (int or float) if it can do so losslessly. If you use other datatypes I suggest you convert them to something portable before calling .tolist.

score 1 · Answer 2 · answered Sep 13 '21 at 08:56

Here is a full working example of an Encoder/Decoder that can deal with NumPy arrays:

import numpy
from json import JSONEncoder,JSONDecoder
import json

# ********************************** #

class NumpyArrayEncoder(JSONEncoder):
    def default(self, obj):
        if isinstance(obj, numpy.ndarray):
            return obj.tolist()
        return JSONEncoder.default(self, obj)

class NumpyArrayDecoder(JSONDecoder):
    def default(self, obj):
        if isinstance(obj, list):
            return numpy.asarray(obj)
        return JSONEncoder.default(self, obj)

# ********************************** #

if __name__ == "__main__":

    # TO TEST

    numpyArrayOne = numpy.array([[11 ,22, 33], [44, 55, 66], [77, 88, 99]])
    numpyArrayTwo = numpy.array([[51, 61, 91], [121 ,118, 127]])
    
    # Serialization
    numpyData = {"arrayOne": numpyArrayOne, "arrayTwo": numpyArrayTwo}
    print("Original Data: \n")
    print(numpyData)
    print("\nSerialize NumPy array into JSON and write into a file")
    with open("numpyData.json", "w") as write_file:
        json.dump(numpyData, write_file, cls=NumpyArrayEncoder)
    print("Done writing serialized NumPy array into file")
    
    # Deserialization
    print("Started Reading JSON file")
    with open("numpyData.json", "r") as read_file:
        print("Converting JSON encoded data into Numpy array")
        decodedArray = json.load(read_file, cls=NumpyArrayDecoder)
        
    print("Re-Imported Data: \n")
    print(decodedArray)

Efficient way to write a python dictionary with Numpy Nd-Array values into a Json File

2 Answers2