not a feature, but the future of an app is under testing.

Follow by Email

Total Pageviews

Saturday, December 13, 2014

Tools for Screenshot Comparison Testing: Huxley, Wraith, Selenium-Wraith

Testing the user interface (UI) testing may be the most difficult part of test automation. When we think from the perspective of users there are not hundreds but there may be thousands of different users' environment but it is clear that simulating and testing UI related cases for all possible users' environment is impossible! However we can cover easily the %80 of them by using statistical data taken from analytics tools as Pareto Analysis says 20% of reasons cause the %80 of all problem. If you are doing some kind of internet business, there are two important factors for user environments which are operating system and browsers for normal internet users.  Therefore analytic tools should give us feedback about operating system and versions of operating system; browsers and versions of browsers. To get a general trend we can look at the statistic but sure it is more convenient to take your own statistic depends on your user profile.

In general approach  testing Chrome, Internet Explorer and Firefox covers more than %83 but don't forget may be Safari users may have more loyalty for you.

As said, testing UI related tests are not easy even you write perfect unit testing for front-end you can not assure that if the UI not are corrupted unless you check them on the real browsers by eyes. Therefore for this kind of problem, you should use screenshot comparison tools that can automatically check two images taken from related pages with desired browsers and screen size after specific scenarios are executed. This check should be performed pixel by pixel comparison. For this purposes, I have two open source tools to explain how we can cover majority of the UI issues with screenshots comparison tools in this post. 


Wraith is initiated by BBC and open source screenshots comparison tool. Basic idea behind the Wraith is to take screenshot with desired screen widths from two different url and compare them. Comparison is performed pixel-by-pixel so at the end it gives a result with percentage of differences. For this kind of purposes it works very fine. 

Wraith can be run under PhantomJS, SlimmerJS or CasperJS but it is not possible to run it without headless mode. If your application has not problem with this web browser engine, as I have some problem with AngularJS with PhamtomJS, you can use Wraith. You can read this post from Thoughtworkers for more information. For comparing images, Wraith uses Image Magic so you need to install it too.

gem install wraith

Setup for Wraith project:
wraith setup 
wraith setup casper


Running Wraith:
wraith capture configs/config.yaml

Summary of Wraith Experience:

  • it is very easy to setup and use
  • very successful if you compare two different urls but cannot compare ready images to screenshot
  • You don't need to write scripts
  • Taking full screenshot so that you can compare whole page
  • Has some problem for flash objects
  • It only works under headless mode (PhantomJS or eq.)
  • PhantomJS has an option for adding cookies so that you can test login required page
  • You can not write test cases, it just compare two different screenshots
  • You can not compare same url over different browsers
  • Selenium-Wraith

    Selenium-Wraith is written to add Selenium feature to Wraith. It can also used with Cucumber to BBD style test cases. It seems to me not mature as much as Wraith itself but good idea and it covers the some missing parts of Wraith.


    Huxley is another screenshots comparison tool developed by Facebook development teams. It is a open source  tool and written in Python. It uses Selenium standalone server so you must download and run it before recording and executing your cases. Good thing for the Huxley is that you can record a case with the parameter --record so you can execute some steps and then take screenshots for comparing. It also let you take screenshots where you need so you are free to compare not only two urls but also with many different cases.

    pip install huxley
    run Selenium Server:
    java -jar selenium-server-standalone-2.44.0.jar
    huxley --record

    First you need to create a Huxleyfile and add your testing urls like following format: 
    After you run huxley by --record parameter, it lets you to create your case and by pressing enter key it takes a screenshot. Cases is recorded inside of the home.huxley and about.huxley file as record.json as json format like the following: 

    Summary for Huxley Experience:

  • Easy to use
  • You can record case and run it
  • You can use Chrome, Firefox, Internet Explorer and Safari  by modifying the browser in main function in module.
  • You can set the browser size if you want to test responsive layout by modifying the screensize in main function in module:
  • Supporting Continuous Integration