transform DenseTensor in Microsoft.ML.OnnxRuntime

Question

I am currently trying to port the code from https://github.com/AsuharietYgvar/AppleNeuralHash2ONNX/blob/master/nnhash.py to C# using Micrisoft.ML.OnnxRuntime. For Image loading I use SixLabors.ImageSharp.

So far loading the onnx session file works but I think I have trouble preprocessing the input image in a correct way. The original code does like so:

image = Image.open(sys.argv[3]).convert('RGB')
image = image.resize([360, 360])
arr = np.array(image).astype(np.float32) / 255.0
arr = arr * 2.0 - 1.0
arr = arr.transpose(2, 0, 1).reshape([1, 3, 360, 360])

And now I have trouble finding out how to transpose the image data array. My code so far is like this:

img.Mutate(x =>
{
     x.Resize(new ResizeOptions
     {
         Size = new SixLabors.ImageSharp.Size(360, 360),
         Mode = SixLabors.ImageSharp.Processing.ResizeMode.Stretch
     });
});

Tensor<float> arr = new DenseTensor<float>(new int[] { 1, 3, 360, 360 });

for (int y = 0; y < img.Height; y++)
{
    Span<Rgb24> pixelSpan = img.GetPixelRowSpan(y);
    for (int x = 0; x < img.Width; x++)
    {
        arr[0, 0, y, x] = (pixelSpan[x].R / 255f * 2) - 1;
        arr[0, 1, y, x] = (pixelSpan[x].G / 255f * 2) - 1;
        arr[0, 2, y, x] = (pixelSpan[x].B / 255f * 2) - 1;
    }
}

but it seems like setting the order/layout of the Tensor is wrong. The problem is, I cant find any transform function in the Microsoft.ML.OnnxRuntime library to resemble the last line (arr.transpose(2, 0, 1).reshape([1, 3, 360, 360])) in the original code.

So how could I correctly build the input image Tensor for Microsoft.ML.OnnxRuntime?

score 0 · Answer 1 · answered Feb 18 '22 at 06:09

The transpose is not required as you are already creating the dense tensor in the correct shape. In the python code, we are taking an array that is [360, 360, 3] and transposing that into a array of [1, 3, 360, 360]. While your dotnet code is taking the RGB values from the row, and placing the values directly into the tensor.

This is the code that we have been using to do this is below:

public virtual Tensor<float> ConvertImageToTensor(ref List<Image<Rgb48>> images, int[] inputDimension)
{
    inputDimension[0] = images.Count;

    Tensor<float> input = new DenseTensor<float>(inputDimension);
    for (var i = 0; i < images.Count; i++)
    {
        var image = images[i];

        for (var y = 0; y < image.Height; y++)
        {
            var pixelSpan = image.GetPixelRowSpan(y);
            for (var x = 0; x < image.Width; x++)
            {
                input[i, 0, y, x] = pixelSpan[x].R;
                input[i, 1, y, x] = pixelSpan[x].G;
                input[i, 2, y, x] = pixelSpan[x].B;
            }
        }
    }
    return input;
}

I tried that in the beginning but the results are not correct. I don't know if its because I use Microsoft.ML but I tried two images that should generate the same hash (tested with original python implementation) but they don't. — C0dR, Feb 24 '22 at 08:12
What specifically is not correct? Does it not run? Or are the results different? One place to potentially look is the pixel format: https://docs.sixlabors.com/api/ImageSharp/SixLabors.ImageSharp.PixelFormats.Rgb24.html — SkywalkerIsNull, Feb 24 '22 at 08:22
it does run but the resulting hashes are different/not correct. I ran it on two images with almost the same content which do produce the same output on the python script but does not in C#. — C0dR, Feb 25 '22 at 13:56

transform DenseTensor in Microsoft.ML.OnnxRuntime

1 Answers1