How can I improve my python openCV video-stream?

Question

I've been working on a project where I use a raspberry pi to send a live video feed to my server. This kinda works but not how I'd like it to. The problem mainly is the speed. Right now I can send a 640x480 video stream with a speed of around 3.5 FPS and a 1920x1080 with around 0.5 FPS, which is terrible. Since I am not a professional I thought there should be a way of improving my code.

The sender (Raspberry pi):

def send_stream():
    connection = True
    while connection:
        ret,frame = cap.read()
        if ret:
            # You might want to enable this while testing.
            # cv2.imshow('camera', frame)
            b_frame = pickle.dumps(frame)
            b_size = len(b_frame)
            try:
                s.sendall(struct.pack("<L", b_size) + b_frame)
            except socket.error:
                print("Socket Error!")
                connection = False

        else:
            print("Received no frame from camera, exiting.")
            exit()

The Receiver (Server):

    def recv_stream(self):
        payload_size = struct.calcsize("<L")
        data = b''
        while True:
            try:
                start_time = datetime.datetime.now()
                # keep receiving data until it gets the size of the msg.
                while len(data) < payload_size:
                    data += self.connection.recv(4096)
                # Get the frame size and remove it from the data.
                frame_size = struct.unpack("<L", data[:payload_size])[0]
                data = data[payload_size:]
                # Keep receiving data until the frame size is reached.
                while len(data) < frame_size:
                    data += self.connection.recv(32768)
                # Cut the frame to the beginning of the next frame.
                frame_data = data[:frame_size]
                data = data[frame_size:]

                frame = pickle.loads(frame_data)
                frame = cv2.cvtColor(frame,cv2.COLOR_BGR2RGB)

                end_time = datetime.datetime.now()
                fps = 1/(end_time-start_time).total_seconds()
                print("Fps: ",round(fps,2))

                self.detect_motion(frame,fps)

                self.current_frame = frame

            except (socket.error,socket.timeout) as e:
                # The timeout got reached or the client disconnected. Clean up the mess.
                print("Cleaning up: ",e)
                try:
                    self.connection.close()
                except socket.error:
                    pass
                self.is_connected = False
                break

Answer 1

One potential reason could because of I/O latency when reading frames. Since cv2.VideoCapture().read() is a blocking operation, the main program is stalled until a frame is read from the camera device and returned. A method to improve performance would be to spawn another thread to handle grabbing frames in parallel instead of relying on a single thread to grab frames in sequential order. We can improve performance by creating a new thread that only polls for new frames while the main thread handles processing/graphing the most recent frame.

Your current approach (Sequential):

Thread 1: Grab frame -> Process frame -> Plot

Proposed approach (Parallel):

Thread 1: Grab frame

from threading import Thread
import time

def get_frames():
    while True:
        ret, frame = cap.read()
        time.sleep(.01)

thread_frames = Thread(target=self.get_frames, args=())
thread_frames.daemon = True
thread_frames.start()

Thread 2: Process frame -> Plot

def process_frames():
    while True:
        # Grab most recent frame
        # Process/plot frame
        ...

By having separate threads, your program will be in parallel since there will always be a frame ready to be processed instead of having to wait for a frame to be read in before processing can be done.

Note: This method will give you a performance boost based on I/O latency reduction. This isn't a true increase of FPS as it is a dramatic reduction in latency (a frame is always available for processing; we don't need to poll the camera device and wait for the I/O to complete).

Answer 2

After searching the internet for ages, I found a quick solution which doubled the fps (This is still way too low: 1.1 fps @1080p). What I did was I stopped using pickle and used base64 instead. apparently pickling the image just takes a while. Anyway this is my new code:

The sender (Raspberry pi):

def send_stream():
global connected
connection = True
while connection:
    if last_frame is not None:

        # You might want to uncomment these lines while testing.
        # cv2.imshow('camera', frame)
        # cv2.waitKey(1)
        frame = last_frame

        # The old pickling method.
        #b_frame = pickle.dumps(frame)

        encoded, buffer = cv2.imencode('.jpg', frame)
        b_frame = base64.b64encode(buffer)

        b_size = len(b_frame)
        print("Frame size = ",b_size)
        try:
            s.sendall(struct.pack("<L", b_size) + b_frame)
        except socket.error:
            print("Socket Error!")
            connection = False
            connected = False
            s.close()
            return "Socket Error"
    else:
        return "Received no frame from camera"

The Receiver (Server):

    def recv_stream(self):
    payload_size = struct.calcsize("<L")
    data = b''
    while True:
        try:
            start_time = datetime.datetime.now()
            # keep receiving data until it gets the size of the msg.
            while len(data) < payload_size:
                data += self.connection.recv(4096)
            # Get the frame size and remove it from the data.
            frame_size = struct.unpack("<L", data[:payload_size])[0]
            data = data[payload_size:]
            # Keep receiving data until the frame size is reached.
            while len(data) < frame_size:
                data += self.connection.recv(131072)
            # Cut the frame to the beginning of the next frame.
            frame_data = data[:frame_size]
            data = data[frame_size:]

            # using the old pickling method.
            # frame = pickle.loads(frame_data)

            # Converting the image to be sent.
            img = base64.b64decode(frame_data)
            npimg = np.fromstring(img, dtype=np.uint8)
            frame = cv2.imdecode(npimg, 1)

            frame = cv2.cvtColor(frame,cv2.COLOR_BGR2RGB)

            end_time = datetime.datetime.now()
            fps = 1/(end_time-start_time).total_seconds()
            print("Fps: ",round(fps,2))
            self.detect_motion(frame,fps)

            self.current_frame = frame

        except (socket.error,socket.timeout) as e:
            # The timeout got reached or the client disconnected. Clean up the mess.
            print("Cleaning up: ",e)
            try:
                self.connection.close()
            except socket.error:
                pass
            self.is_connected = False
            break

I also increased the packet size which increased the fps when sending from my local machine to my local machine while testing, but this didn't change anything whatsoever when using the raspberry pi.

You can see the full code on my github: https://github.com/Ruud14/SecurityCamera