How do I stream python OpenCV output to HTML canvas?

Question

I'm trying to make an web interface for my python OpenCV code using Flask,with the intent of using a canvas HTML element to draw a "crop area" to extract details from the frame stream.Following some examples I've found here on Stackoverflow I was able to stream the output of my OpenCV code to an img element, but not to the canvas nor the video element. Python Code:(Minimal)

import cv2

from flask import Flask, render_template,Response

app = Flask(__name__)

video_capture = cv2.VideoCapture(0)

def gen():    

    while True:
        ret, image = video_capture.read()
        cv2.imwrite('t.jpg', image)
        yield (b'--frame\r\n'
           b'Content-Type: image/jpeg\r\n\r\n' + open('t.jpg', 'rb').read() + b'\r\n')
    video_capture.release()


@app.route('/')
def index():
    """Video streaming"""
    return render_template('index.html')

@app.route('/video_feed')
def video_feed():
    """Video streaming route. Put this in the src attribute of an img tag."""
    return Response(gen(),
                mimetype='multipart/x-mixed-replace; boundary=frame')


if __name__ == '__main__':
    app.run()

HTML Code:

<html>
<head>
    <title>Video Streaming </title>
</head>
<body>
    <div>
        <h1>Live Video Streaming </h1>
        <img id="img" src = "{{ url_for('video_feed') }}">
    </div>
</body>
</html>

you will have to learn JavaScript for this. – furas Mar 04 '20 at 08:15 — furas, Mar 04 '20 at 08:15

score 5 · Answer 1 · answered Mar 04 '20 at 15:15

For future reference to anyone that bumps into this question,although the answer provided by @furas works as intended, I've found that for my usecase it is also possible to assign the output frame to the canvas background using CSS,allowing to display the video only,and draw freely on top of that.

HTML Code

<html>
<head>
    <title>Video Streaming </title>
</head>
<style>
    .myCanvas{
       background-image: url("{{ url_for('video_feed') }}"); 
    }
</style>
<body>
    <div>
        <h1>Live Video Streaming </h1>
        <img id="img" src = "{{ url_for('video_feed') }}">
        <canvas class="myCanvas" height="600" width="480">
    </div>
</body>
</html>

worked perfectoooooo – techshot nextgen Dec 13 '21 at 15:40 — techshot nextgen, Dec 13 '21 at 15:40

score 3 · Accepted Answer · answered Mar 04 '20 at 09:29

This HTML with JavaScript display stream on Canvas and in <img> at the same time.

For single static image it needs to run onload(). For stream MJPEG it needs to use setInterval() to redraw image periodically.

It works for me on Chrome but my Firefox displays only first frame on Canvas. I think I had this problem before. Probably it needed something more to work correctly but I don't remeber what was it.

<html>
<head>
    <title>Video Streaming </title>
</head>
<body>

    <div width="640px" height="480px" style="display:inline-block">
        <h1>Image</h1>
        <img id="img" src="{{ url_for('video_feed') }}">
    </div>

    <div width="640px" height="480px" style="display:inline-block">
        <h1>Canvas</h1>
        <canvas id="canvas" width="640px" height="480px"></canvas>
    </div>

<script >

    var ctx = document.getElementById("canvas").getContext('2d');
    var img = new Image();
    img.src = "{{ url_for('video_feed') }}";

    // need only for static image
    //img.onload = function(){   
    //    ctx.drawImage(img, 0, 0);
    //};

    // need only for animated image
    function refreshCanvas(){
        ctx.drawImage(img, 0, 0);
    };
    window.setInterval("refreshCanvas()", 50);

</script>

</body>
</html>

I found old information mjpeg HTML5 video doesn't stream with and it seems <video> may not support mjpeg format so it may not works with this stream.

Full working code.

I use render_template_string instead of render_template so everyone can put all code in one file and test it.

import cv2
from flask import Flask, render_template, render_template_string, Response

app = Flask(__name__)
video_capture = cv2.VideoCapture(0)

def gen():    
    while True:
        ret, image = video_capture.read()
        cv2.imwrite('t.jpg', image)
        yield (b'--frame\r\n'
           b'Content-Type: image/jpeg\r\n\r\n' + open('t.jpg', 'rb').read() + b'\r\n')
    video_capture.release()


@app.route('/')
def index():
    """Video streaming"""
    #return render_template('index.html')
    return render_template_string('''<html>
<head>
    <title>Video Streaming </title>
</head>
<body>
    <div>
        <h1>Image</h1>
        <img id="img" src="{{ url_for('video_feed') }}">
    </div>
    <div>
        <h1>Canvas</h1>
        <canvas id="canvas" width="640px" height="480px"></canvas>
    </div>

<script >
    var ctx = document.getElementById("canvas").getContext('2d');
    var img = new Image();
    img.src = "{{ url_for('video_feed') }}";

    // need only for static image
    //img.onload = function(){   
    //    ctx.drawImage(img, 0, 0);
    //};

    // need only for animated image
    function refreshCanvas(){
        ctx.drawImage(img, 0, 0);
    };
    window.setInterval("refreshCanvas()", 50);

</script>

</body>
</html>''')

@app.route('/video_feed')
def video_feed():
    """Video streaming route. Put this in the src attribute of an img tag."""
    return Response(gen(),
                mimetype='multipart/x-mixed-replace; boundary=frame')


if __name__ == '__main__':
    app.run()

Thank you for your response, simple and concise! – Augusto Afonso Mar 04 '20 at 15:08 — Augusto Afonso, Mar 04 '20 at 15:08

How do I stream python OpenCV output to HTML canvas?

2 Answers2

HTML Code