Sending openCV image via Flask_socketIO to client causes TypeError: Object of type ndarray is not JSON serializable

Question

I have a base64 image being sent from the browser to my flask server. When the server gets the base64 string, it converts it to a format for OpenCV:

image = '{}'.format(image64).split(',')[1]

im_bytes = base64.b64decode(image)
im_arr = np.frombuffer(im_bytes, dtype=np.uint8)  # im_arr is one-dim Numpy array
frame = cv2.imdecode(im_arr, flags=cv2.IMREAD_COLOR)

Some processing happens on the frame and then, using flask_signalio, is sent back to the browser using:

encode_param = [int(cv2.IMWRITE_JPEG_QUALITY), 90]
result, encimg = cv2.imencode('.jpg', updated_frame, encode_param)
socketio.emit('response_back', {'image_data': encimg}, namespace='/test')

In the browser, I have some JS to display the image that it receives:

socket.on('response_back', function(image){
    ....
});

I have two issues.

1. Calling socketio.emit('response_back', {'image_data': encimg}, namespace='/test') results in an exception on the server: TypeError: Object of type ndarray is not JSON serializable.

How can I fix this? What is wrong here? It's strange because I am sure I am sending ('emitting') the data correctly, as shown here and here. I've also tried setting up SocketIO to use binary data: socketio = SocketIO(app, binary=True). Nothing works. Always the same error.

2. How do I deal with the image data back on the client/browser? (i.e. in the socket.on('response_back', function(image) code)?

Answer 1

After nearly an hour, I've found the solution.

First, flask_socketio can send byte data, but NOT ndarray! The solution is to encode to bytes, with .tobytes(). For example, on the server:

encode_param = [int(cv2.IMWRITE_JPEG_QUALITY), 90]
result, encimg = cv2.imencode('.jpg', updated_frame, encode_param)
socketio.emit('response_back', {'frame': encimg.tobytes()}, namespace='/test')

For the second problem, we need to convert the ArrayBuffer data:

var arrayBufferView = new Uint8Array( result.frame );
var blob = new Blob( [ arrayBufferView ], { type: "image/jpeg" } );
var urlCreator = window.URL || window.webkitURL;
var imageUrl = urlCreator.createObjectURL( blob );
var image_id = document.getElementById('imagebox');
image_id.src = imageUrl;