Detecting similar objects in a picture

Question

I'm trying to detect similar objects in a picture. The purpose of the code is to detect the gold and click on it.

I have tried scanning pixel by pixel but it wasn't efficient and the results weren't satisfying. I'll add that the game is running on windows mode and classes like robot are working fine. Also the gold might be in different places every time.

depending on the size you can skip like 5 pixels every step. if you want to write a bot, java might not be the best solution — XtremeBaumer, May 10 '17 at 14:13
Too broad. Your question boils down to: how to detect objects in pictures. And thing is: whole books are written about that subject. — GhostCat, May 10 '17 at 14:14
This is a task for OpenCV which can be used with java as mentioned here: http://stackoverflow.com/questions/29182521/opencv-with-javacv-or-only-opencv-to-use-java-api You could create a binary image, detect the spots, dismiss what is too big (i.e. your robot) and be left with the golden spots — pandaadb, May 10 '17 at 14:16
Particularly you can check this: https://opencv-java-tutorials.readthedocs.io/en/latest/07-image-segmentation.html — pandaadb, May 10 '17 at 14:32

score 2 · Accepted Answer · edited May 23 '17 at 11:47

As a very quick example I wrote up this using your image:

public class OpenCVTest {

    public static void main(String[] args) {
        OpenCV.loadLibrary();

        Mat m = Highgui.imread("/home/artur/Pictures/test.png", Highgui.CV_LOAD_IMAGE_GRAYSCALE);
        LoadImage( m);
        Mat res = Mat.zeros(m.size(), m.type());
        Imgproc.adaptiveThreshold(m, res, 255, Imgproc.ADAPTIVE_THRESH_MEAN_C, Imgproc.THRESH_BINARY, 15, 20);
        LoadImage(res);
        Mat cannyRes = Mat.zeros(m.size(), m.type());
        Imgproc.Canny(res, cannyRes, 55, 5.2);
        LoadImage(cannyRes);

        Imgproc.dilate(cannyRes, cannyRes, new Mat(), new Point(-1, -1), 2);
        Imgproc.erode(cannyRes, cannyRes, new Mat(), new Point(-1, -1), 2);

        LoadImage(cannyRes);

        List<MatOfPoint> contours = new ArrayList<>();
        Imgproc.findContours(cannyRes, contours, new Mat(), Imgproc.RETR_LIST,Imgproc.CHAIN_APPROX_SIMPLE);

        System.err.println(contours.size());

        contours = contours.stream().filter(s -> s.size().area() > 50 && s.size().area() <= 100).collect(Collectors.toList());

        for(MatOfPoint p : contours) {
            Size size = p.size();
            System.err.println("-- -- --");
            System.err.println(size.area());
        }

        Imgproc.drawContours(cannyRes, contours, 20, new Scalar(233, 223,212));
        LoadImage(cannyRes);
    }

    public static void LoadImage( Mat m) {
        Path path = Paths.get("/tmp/", UUID.randomUUID().toString() + ".png");
        Highgui.imwrite(path.toString(), m);
        JFrame frame = new JFrame("My GUI");
        frame.setDefaultCloseOperation(JFrame.EXIT_ON_CLOSE);

        frame.setResizable(true);
        frame.setLocationRelativeTo(null);

        // Inserts the image icon
        ImageIcon image = new ImageIcon(path.toString());
        frame.setSize(image.getIconWidth() + 10, image.getIconHeight() + 35);
        // Draw the Image data into the BufferedImage
        JLabel label1 = new JLabel(" ", image, JLabel.CENTER);
        frame.getContentPane().add(label1);

        frame.validate();
        frame.setVisible(true);
    }

}

I read the image
I use adaptive threshhold to create a binary image
I use canny to detect the edges in my image
I use dilate/erode to remove background noise
I use the contour finder to find objects in my image
I dismiss any contour that has a arbitrary size

The resulting contours are roughly your yellow spots. This is not very accurate as I didn't invest time playing with the different parameters, but you can fine tune that.

Hope that helps,

Have fun playing. You can see how to set up OpenCV here: Java OpenCV from Maven

Thank you, but do you think that creating an image and analyzing it every second is efficient? because each time you click on the object(the gold) you will have to scan again the screen/generate an image because the player will be in a different location. — Imri Persiado, May 10 '17 at 17:35
Aren't you the one who put the objects on the canvas? you shouldn't have to detect them as you should already know where the objects are. Also, this would not be once every second but at least 30 times a second (which will give you a relatively fluent picture) or maybe even 60 frames every second. However, in CV, once you have the objects, and you know that they move with the picture, you can use different techniques to track them instead of rediscovering them each time. But again, if the game is in your control, you ought to know where the objects are placed — pandaadb, May 10 '17 at 18:38

Detecting similar objects in a picture

1 Answers1