I Found a way around this issue.
My goal was to have a small image unstreched in the center of a control and the background to be filled with a streched blurry version of itself to fill the void.
With the following XAML I had the same issue with the corners of the image turning white like this:
<Image ClipToBounds="True" Source="{Binding VideoDecoder.LiveImage}" Stretch="Fill">
<Image.Effect >
<BlurEffect Radius="100" RenderingBias="Performance" />
</Image.Effect>
</Image>

I replaced the image with a grid and used a visual brush for the background which gives a little more flexibility than the image. Please note that setting the viebox of the visual brush like this removes the white fading but also cuts of part of the image, in my application however this was accetable:
<Grid>
<Grid.Background>
<VisualBrush Viewbox="0.06,0.1,0.85,0.81">
<VisualBrush.Visual>
<Image Source="{Binding VideoDecoder.LiveImage}">
<Image.BitmapEffect>
<BlurBitmapEffect KernelType="Gaussian" RenderOptions.BitmapScalingMode="LowQuality" RenderOptions.EdgeMode="Unspecified" Radius="100"/>
</Image.BitmapEffect>
</Image>
</VisualBrush.Visual>
</VisualBrush>
</Grid.Background>
</Grid>
