How to get the image pixel at real locations in opencv?

Question

I want to retrieve the rgb of a pixel in the image. But the location is not integer location but real values (x,y). I want a bilinear interpolated value. How could I do it opencv?

Thanks a lot

Answer 1

There is no simple function for subpixel access but I can suggest you few options:

1. Use getRectSubPix and extract 1 pixel region:

   cv::Vec3b getColorSubpix(const cv::Mat& img, cv::Point2f pt)
   {
       cv::Mat patch;
       cv::getRectSubPix(img, cv::Size(1,1), pt, patch);
       return patch.at<cv::Vec3b>(0,0);
   }
   

2. Use more flexible but less precise remap with one-pixel map:

   cv::Vec3b getColorSubpix(const cv::Mat& img, cv::Point2f pt)
   {
       cv::Mat patch;
       cv::remap(img, patch, cv::Mat(1, 1, CV_32FC2, &pt), cv::noArray(),
           cv::INTER_LINEAR, cv::BORDER_REFLECT_101);
       return patch.at<cv::Vec3b>(0,0);
   }
   

3. Implement bilinear interpolation yourself, as it is not a rocket science:

   cv::Vec3b getColorSubpix(const cv::Mat& img, cv::Point2f pt)
   {
       assert(!img.empty());
       assert(img.channels() == 3);
   
       int x = (int)pt.x;
       int y = (int)pt.y;
   
       int x0 = cv::borderInterpolate(x,   img.cols, cv::BORDER_REFLECT_101);
       int x1 = cv::borderInterpolate(x+1, img.cols, cv::BORDER_REFLECT_101);
       int y0 = cv::borderInterpolate(y,   img.rows, cv::BORDER_REFLECT_101);
       int y1 = cv::borderInterpolate(y+1, img.rows, cv::BORDER_REFLECT_101);
   
       float a = pt.x - (float)x;
       float c = pt.y - (float)y;
   
       uchar b = (uchar)cvRound((img.at<cv::Vec3b>(y0, x0)[0] * (1.f - a) + img.at<cv::Vec3b>(y0, x1)[0] * a) * (1.f - c)
                              + (img.at<cv::Vec3b>(y1, x0)[0] * (1.f - a) + img.at<cv::Vec3b>(y1, x1)[0] * a) * c);
       uchar g = (uchar)cvRound((img.at<cv::Vec3b>(y0, x0)[1] * (1.f - a) + img.at<cv::Vec3b>(y0, x1)[1] * a) * (1.f - c)
                              + (img.at<cv::Vec3b>(y1, x0)[1] * (1.f - a) + img.at<cv::Vec3b>(y1, x1)[1] * a) * c);
       uchar r = (uchar)cvRound((img.at<cv::Vec3b>(y0, x0)[2] * (1.f - a) + img.at<cv::Vec3b>(y0, x1)[2] * a) * (1.f - c)
                              + (img.at<cv::Vec3b>(y1, x0)[2] * (1.f - a) + img.at<cv::Vec3b>(y1, x1)[2] * a) * c);
   
       return cv::Vec3b(b, g, r);
   }
   

Answer 2

Unfortunately I don't have enough points to post this as a comment on the accepted answer... I adjusted the code to suit my own problem which requires interpolation on a single channel matrix of floats.

I thought I'd like some intuition of which of the approaches are the fastest.

I implemented the 3 methods from Andrey Kamaev's answer as well as a simple nearest neighbour (basically just rounding off the co-ordinates).

I ran an experiment with a matrix A(100x100) which I just filled with garbage. I then made a matrix B(400x400) which is filled with values interpolated from a such that: B(i,j) = A(i/4, j/4).

Each run was done 1000 times and here are the average times:

• Nearest Neighbour: 2.173 ms
• getRectSubPix: 26.506 ms
• remap: 114.265 ms
• manual: 5.086 ms
• manual without borderInterpolate: 3.842 ms

So nearest neighbour for super speed if you don't really care about the actual interpolation too much and just need a value - particularly if your data varies very smoothly. For anything else just I'd go with the manual bilinear interpolation as it seems consistently faster than the other methods. (OpenCV 2.4.9 - Ubuntu 15.10 Repo - Feb 2016).

If you know all 4 your contributing pixels are within the bounds of your matrix, then your can make it basically equivalent in time to Nearest Neighbour - although the difference is pretty negligible anyway.

Answer 3

bilinear interpolation just means weighting the value based on the 4 nearest pixels to the one you are examining. The weights can be calculated as follows.

cv::Point2f current_pos; //assuming current_pos is where you are in the image

//bilinear interpolation
float dx = current_pos.x-(int)current_pos.x;
float dy = current_pos.y-(int)current_pos.y;

float weight_tl = (1.0 - dx) * (1.0 - dy);
float weight_tr = (dx)       * (1.0 - dy);
float weight_bl = (1.0 - dx) * (dy);
float weight_br = (dx)       * (dy);

Your final value is calculated as the sum of the products of each pixel with its respective weight

Answer 4

Using mapping can be more efficient if you want to do this repeatedly or consistently. ANother advantage is choosing an interpolation method and how to handle border conditions. Finally some of the interpolation functions are also implemented on GPU. remap

Answer 5

Here's an example of how to sample a subpixel value with floating point accuracy in Python. According to the docs it uses bilinear interpolation behind the scenes:

import cv2 as cv

def get_subpixel(img: cv.Mat, x: float, y: float):
    """Get interpolated pixel value at (@x, @y) with float precision"""
    patch = cv.getRectSubPix(img, (1,1), (x, y), np.zeros((1,1)), cv.CV_32F)
    if patch is not None:
        return patch[0][0]
    return None

Note that it uses getRectSubPix suggested by @andrey-kamaev but forces opencv to use floating point precision in the returned patch.