How to integrate deep learning algorithms and tcp transmission video services

Question

How to transfer real-time video processed by artificial intelligence to other platforms through tcp protocol. I will use the computer as a server to accept the video, and the Raspberry Pi as a client to transmit the video.

I used OpenCV to implement real-time video transmission in the tcp protocol, and there is a complete and feasible object identification code (from https://github.com/PINTO0309/MobileNet-SSD-RealSense, thanks to the engineer PINTO0309).I receive video on the computer side (Windows system), object recognition and video transmission in the Raspberry Pi.

server


import socket,time,cv2,numpy

def ReceiveVideo():
    address = ('193.169.4.155', 50000)
    s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
    s.bind(address)

    s.listen(1)

    def recvall(sock, count):
        buf = b''
        while count:
            newbuf = sock.recv(count)
            if not newbuf: return None
            buf += newbuf
            count -= len(newbuf)
        return buf

    conn, addr = s.accept()
    print('connect from:'+str(addr))
    while 1:
        start = time.time()
        length = recvall(conn,16)
        stringData = recvall(conn, int(length))
        data = numpy.frombuffer(stringData, numpy.uint8)
        decimg=cv2.imdecode(data,cv2.IMREAD_COLOR)
        cv2.imshow('SERVER',decimg)

        end = time.time()
        seconds = end - start
        fps  = 1/seconds;
        conn.send(bytes(str(int(fps)),encoding='utf-8'))
        k = cv2.waitKey(10)&0xff
        if k == 27:
            break
    s.close()
    cv2.destroyAllWindows()

if __name__ == '__main__':
    ReceiveVideo()

client



import socket,cv2,numpy,time,sys

def SendVideo():
    address = ('193.169.4.155', 50000)
    sock = socket.socket(socket.AF_INET,socket.SOCK_STREAM)
    sock.connect(address)

    global capture

    capture = cv2.VideoCapture(0)
    ret, frame = capture.read()
    encode_param=[int(cv2.IMWRITE_JPEG_QUALITY),15]

    while ret:

        time.sleep(0.01)

        result, imgencode = cv2.imencode('.jpg', frame , encode_param)
        data = numpy.array(imgencode)
        stringData = data.tostring()

        sock.send(str.encode(str(len(stringData)).ljust(16)));

        sock.send(stringData);

        receive = sock.recv(1024)
        if len(receive):print(str(receive,encoding='utf-8'))

        ret, frame = capture.read()
        if cv2.waitKey(10) == 27:
            break
    sock.close()
if __name__ == '__main__':
    SendVideo()

Object identification code come from https://github.com/PINTO0309/MobileNet-SSD-RealSense/blob/master/MultiStickSSDwithPiCamera_OpenVINO_NCS2.py

I have encountered the following problem after modifying the code multiple times.

The video card after transmission is in the first frame.
Unable to transfer, the Raspberry Pi display error is [Errno:32]Broken pipe, and the pc side displays an error.

OpenCV(3.4.2) C:\projects\opencv-python\opencv\modules\highgui\src\window.cpp:356: error: (-215:Assertion failed) size.width>0 && size.height>0 in function 'cv::imshow'

I think second error means not getting the picture.

I finally want to realize real-time transmission of the recognized video to the computer.

score 0 · Answer 1 · answered Dec 05 '19 at 23:15

Your logic for capturing frame from video is wrong: i add some comment here

# you take one frame here
ret, frame = capture.read()
encode_param=[int(cv2.IMWRITE_JPEG_QUALITY),15]

# this will loop without taking any new frame
while ret:

    time.sleep(0.01)
    # as result: at the first frame will be encoded successfully
    # but from second times, frame will be double/tripple encoded and became un-usable
    result, imgencode = cv2.imencode('.jpg', frame , encode_param)
    data = numpy.array(imgencode)
    stringData = data.tostring()

How to fix: change your code into something like this

encode_param=[int(cv2.IMWRITE_JPEG_QUALITY),15]

while(True):
    ret,frame = cap.read()
    result, imgencode = cv2.imencode('.jpg', frame , encode_param)
    data = numpy.array(imgencode)
    stringData = data.tostring()

How to integrate deep learning algorithms and tcp transmission video services

1 Answers1