Using perceptual diffs to spot mistakes during web development

posted by sacah on testing, programming,

So I started this a month ago when I saw a video from Google about pdiffs, then got promptly distracted on other projects and games...

Then today saw a post from BBC about this sort of thing and figured I'd post with how far I've gotten, and see if it spurs me to do more testing on how best this can be displayed.

Basically, you want an easy way to check for style/layout changes. Automating this is difficul, ttime consuming, and eye balling it can all look the same after a few iterations.
So you use perceptual diffs to spot the difference between 2 images, one a control image of your website that's correct. The second image is a current screenshot, and if they differ, you'll have a diff image highlighting the differences.

The problem I had with this diff image was it's hard to match the differences when you're looking at a desktop site, having 3 images side by side, each 960px wide, my eyes couldn't pick smaller differences even with the help of the diff image. Say you add a 1px margin to the bottom of a repeating news element, it get's pretty hard to tell that with 3 separate images (Control, Latest change, Diff).

So my idea is around slicing the Control and Latest change screenshot into 100px wide columns, then alternating them next to each other. Having this 100px side by side really helps pick out the differences that are highlighted by the diff image.

Next I tried using the diff image to darken the sliced images where no differences were. This resulted in a final image like this:


Here you can see the brighter areas are where a difference was detected. So if you look at the image of Obama, in the Control slices you'll notice a white line below, in the Latest change slices you'll notice the white line above. This highlights that the image moved down in the Latest changes from the Control.
You'll also notice that when looking at the text.

 Here is a resized full size you a bit of idea.



So to figure out what worked I've just been using ImageMagick. I think the next step might be writing this in something with a bit more custom control to do more highlighting of changes vs the rest of the image. Below is the commands I've been using.
news.com.au-before-tweaks.png is the Control
news.com.au-after-tweaks.png is the Latest changes

perceptualdiff -output pdiff.png news.com.au-before-tweaks.png news.com.au-after-tweaks.png
..\ImageMagick-6.8.6-0\convert pdiff.png -threshold 0%% pdiff-des.png
..\ImageMagick-6.8.6-0\composite -compose CopyOpacity pdiff-des.png news.com.au-before-tweaks.png news.com.au-before-tweaks-alpha.png
..\ImageMagick-6.8.6-0\composite -compose CopyOpacity pdiff-des.png news.com.au-after-tweaks.png news.com.au-after-tweaks-alpha.png

..\ImageMagick-6.8.6-0\convert news.com.au-before-tweaks.png -level -85%% news.com.au-before-tweaks-bg.png
..\ImageMagick-6.8.6-0\convert news.com.au-before-tweaks-bg.png +gravity -crop 100x6280 bg_tiles_%%03d_a.png
..\ImageMagick-6.8.6-0\convert bg_tiles_*_a.png -matte -bordercolor none -border 1x0 bg_tiles_%%03d_a.png
..\ImageMagick-6.8.6-0\convert news.com.au-before-tweaks-bg.png +gravity -crop 100x6280 bg_tiles_%%03d_b.png
..\ImageMagick-6.8.6-0\convert bg_tiles_*_b.png -matte -bordercolor none -border 1x0 bg_tiles_%%03d_b.png
..\ImageMagick-6.8.6-0\convert bg_tiles_*.png +append news.com.au-before-tweaks-bg.png

..\ImageMagick-6.8.6-0\convert news.com.au-before-tweaks-alpha.png +gravity -crop 100x6280 tiles_%%03d_a.png
..\ImageMagick-6.8.6-0\convert tiles_*_a.png -matte -bordercolor none -border 1x0 tiles_%%03d_a.png
..\ImageMagick-6.8.6-0\convert news.com.au-after-tweaks-alpha.png +gravity -crop 100x6280 tiles_%%03d_b.png
..\ImageMagick-6.8.6-0\convert tiles_*_b.png -matte -bordercolor none -border 1x0 tiles_%%03d_b.png
..\ImageMagick-6.8.6-0\convert tiles_*.png +append tiled_news.png

..\ImageMagick-6.8.6-0\composite tiled_news.png news.com.au-before-tweaks-bg.png news.com.au-final.png
del tile*.png
del bg_tile*.png

Let me know if you go further with this and get a better result, or even a different approach that works for you.