Get dominating color of a specific area in an image: Color Query for Web Page Elements

Question

Problem Description

I am writing a Java application that lets programmers query for page elements on a web page by specifying visible attributes. One of the most important and difficult is Color.

To be specific, i need a way to get the user-visible color of web page elements using Selenium 2 and Webdriver. I want to be able to query for color values (#ff0000) or names (red).

One parameter should control the percentage of similar colors needed to be "dominating" enough. If set to 100% the element is not allowed to have any other color. If set to 50% the element needs to be halfway filled with the color.

There should be another parameter to control the "tolerance" of these colors. With a higher tolerance, red could also match the orange "Ask Question" button here on Stackoverflow.

Example

Given the well-known Stackoverflow web page, i highlighted the page element to check: With a higher color tolerance and a not too high domination percentage, the following queries should return the specified result:

color('#FFEFC6') // exact match: true
color('yellow') // match in tolerance range: true
color('orange') // true
color('blue') // false
color('green') // false

My first approach

Best bet would be using CSS attributes like color and background-color. But these do not take images into account, which are needed for good color queries. Also, they could produce difficulties because of css selector inheritance and the handling of transparency. In addition, absolutely positioned elements with a higher z-index above the current element could produce unexpected results.

Given is the web page element to check. It is represented either as JavaScript DOM element (or JQuery object) or as RemoteWebElement in the Java bindings of Webdriver.

It is possible to take automated Screenshots of the current state of the web page (i am using Firefox), see here: Take a screenshot with Selenium WebDriver

The coordinates of the page element to check are known. Therefore, the screenshot image could cropped to that size and area and be analyzed somehow to check if the query returns true or false.

Implementation

I am not limited to Java in this case. JavaScript would be very nice because i am doing the other queries with the help of JQuery too. Performance matters. I am counting on you, i fear this is a very difficult task. Therefore i need your input.

UPDATE

I solved this issue by taking screenshots and analyzing the pixel data of the relevant part. That way i can deal with all kinds of background images and transparency. It's part of the Abmash framework, which is open source and free to anybody to use: Abmash on Github

Answer 1

Easiest way:

• Get screenshot (save in memory)
• Crop screenshot to the element top = el.offsetTop, left = el.offsetLeft, width = el.offsetWidth, height = el.offsetHeight
• Get the pixel data for the cropped image
• Loop through the pixels getting the total sums of the R, G, B elements then divide the total sums by the pixel count to get the average. Test the average color against your constraints.

If you really want to use JavaScript

• You could send the pixel data to JavaScript for processing if you're intent on doing the final check in JavaScript.
• Or you could send JavaScript the IMAGE URI for the cropped image. Then draw that IMG to a CANVAS then loop through the pixel with ctx2d.getImageData(...)

Only do the above if the element is an IMG or a has a background-image CSS. Just use color and background-color CSS checks otherwise.