I am working on a Silverlight application that implements a custom MediaStreamSource that is feeding a MediaElement with elementary H.264 NAL units. My stream starts off with a key frame, but I noticed that on average it takes about 20 frames to render one image. I have my GOP size set to 8 on my H.264 encoder. This video is coming from a security camera and is being viewed in a live stream application.
The main issue is that this induces a decent amount of latency between when events happen in real life and when the image is actually rendered. A small latency is expected, but it turns out to be about 3 seconds from receiving the first frame until the first image is rendered. Shouldn't the first key frame theoretically contain enough information to decode an image? For testing I have a sample Silverlight application that loads a captured H.264 stream from a file and internally buffers the entire file, this way once the MediaStreamSource opens, it immediately consumes ~20 frames and renders the image with virtually no latency.
Any Microsoft / Silverlight / H.264 experts care to elaborate on why this might possibly be happening?
