What I'm trying to do is load an MPD file with two AdaptationSets, a video listing and an audio listing (both belonging to the same, original mp4 and stripped with MP4Box) – ultimately to play them both simultaneously in an attempt to obtain the same, cohesive playback that the original file had.
The examples below work, but I can only stream one AdaptationSet or the other at a time, once the manifest is parsed out. I feel like, using the Media Source Extensions API, I should just be able to add one SourceBuffer for audio and one for video to the mediaSource, but that doesn't appear to work (not outlined below). Is that the right idea, though? I suppose the other option is creating a video DOM element and an audio DOM element and trying to hash it out from there, but even the thought of that sounds wrong to me… Any reference material you guys have in regard to MSE, MPEG-DASH, etc. that has any info on handling two simultaneous streams?
…
This is roughly what my JavaScript boils down to, upon retrieving the MPD manifest:
...
xhr.onload = function(_) {
var video = document.getElementById("video");
var info = parseManifest(xhr.response);
if typeSupportedEtc(info.mimeType) {
var mediaSource = new MediaSource();
video.src = URL.createObjectURL(mediaSource)
mediaSource.addEventListener("sourceopen", function(_) {
var sourceBuffer = this.addSourceBuffer(info.mimeType);
// Initialize this nonsense...
// Fetch next segment, append to buffer...
// Then loop over segments til you can't loop no mo'...
// Indicate end of stream.
});
}
};
...
The MPD itself is nothing special, so just posting the relevant section for brevity:
...
<Period duration="...">
<AdaptationSet ...>
...
<SegmentList timescale="24000" duration="96000">
<Initialization sourceURL="file-video-init.mp4"/>
<SegmentURL media="file-video-01.m4s"/>
...
</SegmentList>
...
</AdaptationSet>
<AdaptationSet ...>
...
<SegmentList timescale="48000" duration="192000">
<Initialization sourceURL="file-audio-init.mp4"/>
<SegmentURL media="file-audio-01.m4s"/>
...
</SegmentList>
...
</AdaptationSet>
</Period>
...
(I've tried a number of methods of synthesizing the manifest, but this way seems to have yielded the least complexity, for the sake of this problem.)
