Automated visual layout testing

Question

Is it possible to unit test that something is visually rendered as wanted (for some definition of wanted)? Specifically for HTML and CSS renders your page as "expected"

I would personally imagine such tools allow you to define a test in a declarative manner of how something is supposed to look and then run a complex heuristic on the actual file your testing that determines how "close" it is to your declarative definition.

Would there be any other ways to do machine testing on, say a HTML file for whether the content is visually consistent.

I'm not asking to almost impossible question of whether something "looks good", but rather whether it looks consistent or whether it matches a given pattern.

The final question is, if these tools don't exist, are they worth writing? Would tools of such a nature actually give you meaningful results that are not bound by a very large margin of error. I.e. could they accurately predict whether some change to code has broken your visual layout without giving a large amount of false negatives or missing things that are obviously broken.

Highly related: http://stackoverflow.com/questions/4043706/automated-testing-of-css-and-html-front-end-coding — thirtydot, Mar 29 '11 at 13:46
You said: `"test in a declarative manner of how something is supposed to look"` - the declaration of "how something is supposed to look" is called HTML/CSS. Any other way of declaring how something is supposed to look would either miss nuances of HTML/CSS, or otherwise be as complicated as the two combined. This type of tool (as you state it) is not feasible to make. There are also the different rendering engines to consider. Gecko and WebKit are open source, so you have something to work with there, but what about Trident (Internet Explorer)? — thirtydot, Mar 29 '11 at 14:18
@thirtydot Partition for Microsoft to make IE open source? I see what you mean though. I was more suggestion having some declarative definition of what the page looks like as an image and then converting the HTML to an image and comparing that. — Raynos, Mar 29 '11 at 15:07

score 6 · Answer 1 · edited Jun 05 '17 at 20:03

I would personally imagine such tools allow you to define a test in a declarative manner of how something is supposed to look and then run a complex heuristic on the actual file your testing that determines how "close" it is to your declarative definition.

There is a way to test exactly what you want using Galen Framework, which is a tool I have helped to develop. It is a special tool with its own language which allows to express how the website should look like. It is based on Selenium and can be run in Selenium Grid if you need to check it in different browsers. At the moment what the tool is doing - it is getting the location of specified elements on page and checks them relatively to each other. So for example if you want to check that menu panel is below header and stretches to the width of a browser and has 50 pixel height, you do it like this:

menu
    below: header 5 px
    width: 100% of screen/width
    height: 50px

In case you don't want to assert for exact values you can do it like this

menu
    below: header 0 to 5 px
    width: 90 to 100% of screen/width
    height: 45 to 55px

The tool is usefull for testing responsive design as you can use custom tags to match exact specifications for your needs. For instance if we would like to express that a "side-panel" on desktop should be on the right side of a screen with static width 300 pixels, but on mobile it should drop below the "content" and stretch to complete width of a browser, you do it like this:

@ desktop
---------------------
content
    inside: screen 0px left

side-panel
    inside: screen 0px right
    near: content 0px right
    width: 300px

@ mobile
--------------------
content, side-panel
    width: 100% of screen/width

side-panel
    below: content 0px

There are not a lot of information about the tool since it is a new one. At the moment there is only one article explaining how to test the responsive web page using Galen Framework http://mindengine.net/post/2013-11-16-tdd-for-responsive-design-or-how-to-automate-testing-of-website-layout-for-different-devices/.

You can also find complete documentation on official website http://galenframework.com

score 1 · Answer 2 · answered Nov 05 '14 at 03:52

1

You may try some tools like this: Screenster,mogotest. Both using visual matching for layout testing.

answered Nov 05 '14 at 03:52

S.Roshanth

1,499
3
25
36

score 0 · Answer 3 · answered May 23 '14 at 17:19

Well, if you are interested in CSS and HTML rendering on several browsers you can automate tests that compare screenshots.

I've been involved in a similar test, first all our CSS are compiled from less and then tested. Basically the process is to generate base screenshots an then when there are changes it compare new screenshots.

Here is a small CSS test project in NodeJS that i created as baseline: https://github.com/adcarabajal/generator-css-testing

Cheers

score 0 · Answer 4 · answered Jul 26 '17 at 08:32

0

If you are interested in CSS HTML visual testing, differencify does that for you. It is a nodejs library built on top of chrome headless

answered Jul 26 '17 at 08:32

Nima Soroush

12,242
4
52
53

score 0 · Answer 5 · answered Dec 09 '22 at 05:04

If you use Java + Selenium test automation, you can try this plug-in: https://github.com/webrelab/layout_testing

This component allows you to detect changes in the visual display of the page based on a comparison of the current state with the reference one.

This component does not use screenshots. The work is based on the calculation of the values of the attributes of styles, sizes and arrangement of elements.

The component automatically finds all visually significant elements of the screen in the specified area and performs all necessary measurements.

score 0 · Answer 6 · answered Mar 01 '12 at 08:59

I don't think you need complex image analysis. As was mentioned there will be small differences between browsers anyway. Also something you might dismiss as too small might be very important, such as a small icon changing.

If you want cross browser testing it would be extremely difficult to automate. But you could create a tool to check for unexpected visual changes. Saving a desired DOM state and then comparing it with the changed version could help highlight issues. If you change a CSS class expecting it to only effect on page, running your tests would reveal undesired knockon affects on other pages.

score -9 · Accepted Answer · answered Mar 29 '11 at 13:49

-9

The final question is, if these tools don't exist, are they worth writing?

I don't think so, since it takes seconds to look at the page and see if all the elements are rendered correctly, while you probably will spend at least several weeks writing and testing such tool from scratch.

answered Mar 29 '11 at 13:49

Nikita Barsukov

2,957
3
32
39

1

The initial cost should marked against the seconds saved my the X amount of people that are going to use the open-source tool. It's a question of accuracy and marketing whether it's worth doing. – Raynos Mar 29 '11 at 14:03
6

I cannot restrain myself from commenting on this one. It sounds like typical argument of old school programmers about unit tests. Yes it takes seconds to check one page, but what if you have complex web site with lots of screens or a dynamic page: you might find out that for certain combination of parameters something overflows and completely breaks the layout. – Steves Aug 27 '12 at 16:27
3

Steves, I agree completely. It takes seconds to look at one page once. But we usually need to look at 20+ pages 50+ times. – Dennis Golomazov Aug 01 '13 at 19:17
1

Testing for different browsers on different OSs with different screen resolutions and different device types does not take seconds. It is a laborious process - which is why it is often simply not done. Depending on the application being tested, investing time and effort on creating a good automated testing setup may definitely be worth investing in. – Mauritz Hansen Dec 08 '14 at 12:08
I think Nikita is right if you have a small site that is not a company's main source of income. However, if you have a large site with lots of money riding on it, you may want to consider automating layout testing. The weeks you spend building a tool or money spent buying an appropriate tool will save you a lot more in the long term on some QA cost and bugs. – Phil Oct 29 '15 at 15:54

Automated visual layout testing

7 Answers7