Tensorize Image in 0-1 pixel range in Windows ML (UWP)

Question

I'd like to use a pre-trained MobileNet which expects images in the pixel range 0-1 with Windows ML in an UWP app.

The problem is, that ImageFeatureValue only supports a range of 0-255.

So I need a replacement of ImageFeatureValue which can also resize the images but is capable to use the pixel range 0-1.

I found this approach on GitHub:

{
    SoftwareBitmap bitmapBuffer = new SoftwareBitmap(BitmapPixelFormat.Bgra8, 224, 224, BitmapAlphaMode.Ignore))
    VideoFrame buffer = VideoFrame.CreateWithSoftwareBitmap(bitmapBuffer))
    await inputFrame.CopyToAsync(buffer);
    SoftwareBitmap resizedBitmap = buffer.SoftwareBitmap;
    WriteableBitmap innerBitmap = new WriteableBitmap(resizedBitmap.PixelWidth, resizedBitmap.PixelHeight);
    resizedBitmap.CopyToBuffer(innerBitmap.PixelBuffer);
    int[] pixels = innerBitmap.GetBitmapContext().Pixels;
    float[] array = NormalizeImage(pixels);
}

private float[] NormalizeImage(int[] src)
{
    var normalized = new float[src.Length * 3];
    for (int i = 0; i < src.Length; i++)
    {
        var val = src[i];
        normalized[i * 3 + 0] = (float)(val & 0xFF) / (float)255;
        normalized[i * 3 + 1] = (float)((val >> 8) & 0xFF) / (float)255;
        normalized[i * 3 + 2] = (float)((val >> 16) & 0xFF) / (float)255;
    }
    return normalized;
}

The problem is that WritableBitmap does not offer GetBitmapContext().Pixels in UWP Apps (Only in .NET).

I haven't found another way to convert a SoftwareBitmap / WritableBitmap into a pixel array.

How could I accomplish this?

I think, the TensorFoat can be created from the array like this:

long[] shape = {1, 3, 224, 224};
TensorFloat tf = TensorFloat.CreateFromArray(shape, array);

Is this correct?

Thank you!

Please check `PixelBuffer` [document](https://learn.microsoft.com/en-us/uwp/api/windows.ui.xaml.media.imaging.writeablebitmap.pixelbuffer?view=winrt-18362). For getting pixel array you could call `ToArray` method `ScenarioWriteableBitmap.PixelBuffer.ToArray();` — Nico Zhu, Apr 24 '20 at 09:06

score 1 · Answer 1 · answered Jan 21 '21 at 22:10

Check the PixelBuffer document as Nico Answered. Following the example to copy the image contents to the WriteableBitmap's pixel buffer.

// An array containing the decoded image data, which could be modified before being displayed 
byte[] sourcePixels = pixelData.DetachPixelData(); 

// Open a stream to copy the image contents to the WriteableBitmap's pixel buffer 
using (Stream stream = Scenario4WriteableBitmap.PixelBuffer.AsStream()) 
{ 
    await stream.WriteAsync(sourcePixels, 0, sourcePixels.Length); 
}

Then Normailze the byte array

for (UINT32 i = 0; i < size; i += 4)
{
    UINT32 pixelInd = i / 4;
    pFloatTensor[pixelInd] = (float)sourcePixels[i];
    pFloatTensor[(height * width) + pixelInd] = (float)sourcePixels[i + 1];
    pFloatTensor[(height * width * 2) + pixelInd] = (float)sourcePixels[i + 2];
}

At the last, you would then create a TensorFloat variable from that float array and bind it before evaluation (check TensorFloat.CreateFromArray())

Tensorize Image in 0-1 pixel range in Windows ML (UWP)

1 Answers1