How to Parse an MPEG Video Stream using .NET Core

Question

For a personal project, I am attempting to read from a IP Camera's MPEG stream and perform some computer-vision tasks on the individual frames I receive (Using .NET Core 2.2).

I perform a GET request to the Camera's MPEG endpoint and I'm returned a multipart/x-mixed-replace response that continually streams what appear to be individual frames as JPEG images.

What I'm having trouble understanding is the correct way to parse out the frames from the multi-part response - how the frames are retrieved is unimportant to me, if there's a better way to parse this kind of stream, I'm certainly open to change - video retrieval/processing is a whole new world for me :)

For reference:

MPEG Endpoint:

GET http://192.168.0.14/mjpeg.cgi

Example Response Header:

Content-Type: multipart/x-mixed-replace;boundary=--video boundary--

Example Response Body:

Content-length: 41142
Date: 02-02-2019 12:43:19 AM
Content-type: image/jpeg

[payload]

--video boundary--
Content-length: 41220
Date: 02-02-2019 12:43:19 AM
Content-type: image/jpeg

[payload]

--video boundary--

What I have so far:

var client = new HttpClient() {
    BaseAddress = new Uri(streamUrl)
};

var stream = await client.GetStreamAsync(resource);

using (var streamReader = new StreamReader(stream)) {                
    while(true) {
        await GetFrameStart(streamReader);
        var frame = await ReadFrame(streamReader);

        // Get byte[] from returned frame above and construct image
    };
}

// ---

// Fast forward to the start of the Frame's bytes
static async Task GetFrameStart(StreamReader reader) {
    string buffer = null;
    while (buffer != string.Empty) {
        buffer = await reader.ReadLineAsync();
    }
}

// Read entire byte array until the video boundary is met
static async Task<String> ReadFrame(StreamReader reader) {
    string result = string.Empty;
    string line = null;

    while(!isBoundary(line)) {
        line = await reader.ReadLineAsync();
        if (!isBoundary(line)) result += line;
    };

    return result;
}

The above seems to be giving me back the individual [payload] of each frame, however if I when I convert the string to bytes and write to disk, the image is not a valid jpeg:

var bytes = Encoding.UTF8.GetBytes(frame);

using (var fs = new FileStream("test.jpeg", FileMode.Create, FileAccess.Write)) {
    fs.Write(bytes, 0, bytes.Length);
}

[obviously reading bytes array as string(ReadLine) as UTF8 and converting back to bytes arra doesn't make sens at all](https://dotnetfiddle.net/ZqM23s) (run it few times if you get lucky and bytes arrays are equal) — Selvin, Feb 02 '19 at 19:44
Ah okay, I have made some wrong assumptions along the way. Am I right in thinking BinaryReader is the tool I need for this job - read the raw bytes in to a buffer and adjust the boundary-checking parts to drop any non-image related data? — Stevie, Feb 02 '19 at 21:37
isBoundary is a simple check to see if the current line contains the boundary delimiter, i.e. `--video boundary--` — Stevie, Feb 07 '19 at 19:27

timpur · Answer 1 · 2019-10-17T09:04:22.297

1

You could use Microsoft.AspNetCore.WebUtilities.MultipartReader

 var boundary = GetBoundary(context.Request.ContentType);
 var reader = new MultipartReader(boundary, context.Request.Body);
 var section = await reader.ReadNextSectionAsync();

See this Answer for more info:

Hope this helps.

edited Oct 17 '19 at 09:04

answered Oct 16 '19 at 23:45

timpur

11
3

Links to external resources are encouraged, but please add context around the link so your fellow users will have some idea what it is and why it’s there. Always quote the most relevant part of an important link, in case the target site is unreachable or goes permanently offline. See: [How to anwser](https://stackoverflow.com/help/how-to-answer). – Eduardo Baitello Oct 17 '19 at 00:24

score 0 · Answer 2 · answered Feb 04 '19 at 09:08

You could use ffmpeg with a single line:

ffmpeg -i http://192.168.0.14/mjpeg.cgi -vcodec copy frame%d.jpg

This extracts all the frames with numbered names.

If you install ffmpeg, you can call it from your application with System.Diagnostics.Process like this:

var process = new Process()
{
    StartInfo = new ProcessStartInfo()
    {
        FileName = "ffmpeg",
        Arguments = "-i http://192.168.0.14/mjpeg.cgi -vcodec copy frame%d.jpg",
        UseShellExecute = false,
        CreateNoWindow = true,
        RedirectStandardOutput = true,
        RedirectStandardError = true
    },
    EnableRaisingEvents = true
};

process.ErrorDataReceived += (sender, data) => Console.WriteLine(data.Data);

process.Start();
process.BeginErrorReadLine();

Thanks for the reply! This looks like it would probably work, in that it would write images to disk - however I would prefer as much as possible to read frames directly in to memory so that I can manipulate them before they're written. — Stevie, Feb 07 '19 at 19:29
How do u convert it to http? I'm trying to convert rstp to http with ffmpeg that's why i'm asking — Q.Rey, Jan 08 '20 at 09:29

How to Parse an MPEG Video Stream using .NET Core

2 Answers2

Linked