0

Context:

We are working on a Machine Learning project using a pre-trained model which was further trained by a contractor using a Python codebase. There are two models for this Machine Learning process: one for identification and another for classification, and each has its own specific requirements for frame size thanks to them both being pre-trained using Keras-Retinanet.

The goal here is to take a grayscale image (stored as a byte array), apply a color palette, then resize it. The initial way in which the model was trained was using Python's OpenCV library with their default interpolation with their cv2.resize() function. As such we need to replicate that behavior in order to get consistent results (as far as I am aware).

Thus far we have been completely unable to load any format of an array into a Mat object, and the error that comes back is obfuscated, so it's very much unclear what the problem is.

Before this is just flagged as duplicate, I have already tried applying the following solutions:

Convert RGB array to Mat (OpenCv)

https://reposhub.com/dotnet/graphics/SciSharp-SharpCV.html

At the moment we are trying to use the OpenCVSharp4 library. As far as I have been able to find, in order to replicate the resize behavior then we need to store the data in a Mat object, or rebuild their interpolation on our own, which seems like an even bigger waste of time.

        public static float[,] ConvertToBGR(byte[] grayscale, Color[] palette)
        {
            // initilize 3 layer array
            float[,] bgrImage = new float[grayscale.Length, 3];

            // loop through the grayscale image and get the color for each height value in the grayscale image
            for (int i = 0; i < grayscale.Length; i++)
            {
                bgrImage[i, 0] = palette[grayscale[i]].B;
                bgrImage[i, 1] = palette[grayscale[i]].G;
                bgrImage[i, 2] = palette[grayscale[i]].R;
            }

            return bgrImage;
        }


Mat matrix = new Mat(totalRows, totalColumns, MatType.CV_32FC3, ConvertToBGR(rawImageData, colorPallet));

This is an example of one of the many different attempts to get this to load into a matrix object. We have also tried a number of other array formats:

/*
float[,,]: [3colorLayers, height, width]
float[,]: [3colorLayers, greyscaleLength]
{
    {B, G, R},
    {B, G, R},
    ...
}
float[]: [3x greyscaleLength]
{B, G, R, B, G, R, B, G, R,...}

Even went so far as to just try applying the flat grayscale to a matrix which also failed (rawImageData is a byte[]):
Mat matrix = new Mat(totalRows, totalColumns, MatType.CV_8UC1, rawImageData);
*/

How can I convert an array to a matrix? Is there a way to use NumSharp? Is there a better solution?

user2521951
  • 1
  • 2
  • What is the object 'Mat'? I would use the OpenCV solution and treat the return value as a object. Then pass the object to OpenCVSharp4. c# doesn't need to know what the object looks like. – jdweng Oct 07 '20 at 15:30
  • The Mat object is a Matrix, it inherits from CvArr which is required by the resize function. The documentation we've found for OpenCVSharp specifies that the constructor for a `Mat` object asks for a `System.Array` object, which to my understanding, is literally any array. Casting the returned value as an object gives a syntax error, and casting it as an `Array` object produces the same error we've been getting: _The type initializer for 'OpenCvSharp.NativeMethods' threw an exception._ – user2521951 Oct 07 '20 at 18:23
  • why do you need to do any casting? Leave it as a object and just pass between OpenCV to OpenCVSharp. – jdweng Oct 07 '20 at 20:59
  • I feel like the opencv tag is throwing this conversation off. I am working entirely in C# and only have access to OpenCVSharp. I was required to use that tag because OpenCVSharp is a depreciated tag. I need to convert a float array to an OpenCVSharp4 `Mat` object or exactly replicate the `cv2.resize()` function from OpenCV. – user2521951 Oct 07 '20 at 21:16
  • So it turns out...I'm just stupid and don't read things. I did not add the OpenCvSharp4.runtime.win package to the project, so none of the actual native behavior was actually available. This whole thing works just fine if you properly include both packages, which is literally spelled out in the description for OpenCvSharp4 "OpenCV wrapper for .NET. Since this package includes only core managed libraries, another package of native bindings for your OS is required (OpenCvSharp4.runtime.*)." – user2521951 Oct 07 '20 at 21:39

1 Answers1

0

I did not add the OpenCvSharp4.runtime.win package to the project, so none of the native behavior was actually available. This whole thing works just fine if you properly include both packages, which is literally spelled out in the description for OpenCvSharp4 "OpenCV wrapper for .NET. Since this package includes only core managed libraries, another package of native bindings for your OS is required (OpenCvSharp4.runtime.*)."

user2521951
  • 1
  • 2