-1

I have a working MediaPlayer based decoder that consumes livestreamed H264 video from incoming frames, using MediaStreamSource and responding to requests for samples from MediaStreamSource.SampleRequested. This is integrated into Unity in order to display these frames on a IDirect3DSurface.

I am happily doing 30fps on my desktop, but when running on a Hololens 2 I am only visually seeing about maybe 5fps.

That being said, I have numerous metrics inside the MediaPlayer code that show the sample requests, sample resolutions/deferrals and VideoFrameAvailable callbacks are all still happening at 30fps as they do on Desktop. I am not losing frames, it is more like only every X frame actually gets displayed, and intermediate ones are not. Finally, if I use a test MediaSource created from a URI and run through the same pipeline, I do see 30fps on the Hololens 2, so that hopefully rules out issues with my surface / display logic.

I do see a fairly large number of _com_errors raised inside d3d11.dll CDecodeContext::BeginFrame when the debugger is attached, but these never make it up to my code and perhaps are harmless? ref: https://social.microsoft.com/Forums/azure/fr-FR/f76a80db-3bf0-49b1-8c4f-4d3b90c03f94/how-to-track-down-comerrors?forum=winappswithnativecode

Are there any known issues with this kind of streaming that are specific to the Hololens 2?

I can post my relevent code if needed, but the C++ MediaPlayer code is fairly verbose so I will hold off unless specific pieces need examined. The fact that it works so well on Desktop suggests that there can't be anything TOO fundamentally wrong...hopefully...

EDIT: Out of curiosity, I just rebuilt my project for the Hololens 1 and tested it - things are displaying well! This more and more points to a specific issue in the Hololens 2 MediaPlayer implementation?

Rich Logan
  • 39
  • 6
  • 1
    Do you have provided a DirectX buffer to manage Direct3DSurface? I found this thread pointing this out: https://stackoverflow.com/questions/60061565/poor-performance-on-h264-decoding. If it still cannot work after this attempt, please feel free to feedback. – Hernando - MSFT Jul 09 '20 at 09:12
  • @Hernando-MSFT thanks for commenting. I pass my `IDirect3DSurface` to the `CopyFrameToVideoSurface` (https://learn.microsoft.com/en-us/uwp/api/windows.media.playback.mediaplayer.copyframetovideosurface?view=winrt-19041) function during the `VideoFrameAvailable` callback. It looks like the that buffer functionality is used when operating at the lower MediaFoundation level - so I'm not sure how I would apply that here. My understanding is that `CopyFrameToVideoSurface` will do a GPU copy into the same surface that will be rendered/displayed? – Rich Logan Jul 09 '20 at 12:29

1 Answers1

0

After some debugging and further research, it appears that there are indeed performance issues with the YUV to BGRA conversion on ARM64. The solution for me was to pass the output IDirect3DSurface with a format of DXGI_FORMAT_NV12 and map two shader resource views (one single channel 8 bit (DXGI_FORMAT_R8_UNORM) and one dual channel 8+8 bit (DXGI_FORMAT_R8G8_UNORM)) in order to access the Y and UV planes. This allows NV12 output without any performance issues.

In my case, I could then do the YUV to RGB conversion myself on the GPU in shader code.

The reference that put me on this path was in the ID3D12 documentation, but is equally applicable to Direct3D 11: https://learn.microsoft.com/en-us/windows/win32/api/d3d12/nf-d3d12-id3d12device-createshaderresourceview

Rich Logan
  • 39
  • 6