Tools for Screenshot Comparison Testing: Huxley, Wraith, Selenium-Wraith
Testing the user interface (UI) testing may be the most difficult part of test automation. When we think from the perspective of users there are not hundreds but there may be thousands of different users' environment but it is clear that simulating and testing UI related cases for all possible users' environment is impossible! However we can cover easily the %80 of them by using statistical data taken from analytics tools as Pareto Analysis says 20% of reasons cause the %80 of all problem. If you are doing some kind of internet business, there are two important factors for user environments which are operating system and browsers for normal internet users. Therefore analytic tools should give us feedback about operating system and versions of operating system; browsers and versions of browsers. To get a general trend we can look at the w3shcool.com statistic but sure it is more convenient to take your own statistic depends on your user profile.
In general approach testing Chrome, Internet Explorer and Firefox covers more than %83 but don't forget may be Safari users may have more loyalty for you.
As said, testing UI related tests are not easy even you write perfect unit testing for front-end you can not assure that if the UI not are corrupted unless you check them on the real browsers by eyes. Therefore for this kind of problem, you should use screenshot comparison tools that can automatically check two images taken from related pages with desired browsers and screen size after specific scenarios are executed. This check should be performed pixel by pixel comparison. For this purposes, I have two open source tools to explain how we can cover majority of the UI issues with screenshots comparison tools in this post.
Wraith is initiated by BBC and open source screenshots comparison tool. Basic idea behind the Wraith is to take screenshot with desired screen widths from two different url and compare them. Comparison is performed pixel-by-pixel so at the end it gives a result with percentage of differences. For this kind of purposes it works very fine.
Wraith can be run under PhantomJS, SlimmerJS or CasperJS but it is not possible to run it without headless mode. If your application has not problem with this web browser engine, as I have some problem with AngularJS with PhamtomJS, you can use Wraith. You can read this post from Thoughtworkers for more information. For comparing images, Wraith uses Image Magic so you need to install it too.
gem install wraith
Setup for Wraith project:
wraith setup wraith setup casper
wraith capture configs/config.yaml
Summary of Wraith Experience:
Selenium-Wraith is written to add Selenium feature to Wraith. It can also used with Cucumber to BBD style test cases. It seems to me not mature as much as Wraith itself but good idea and it covers the some missing parts of Wraith.
Huxley is another screenshots comparison tool developed by Facebook development teams. It is a open source tool and written in Python. It uses Selenium standalone server so you must download and run it before recording and executing your cases. Good thing for the Huxley is that you can record a case with the parameter --record so you can execute some steps and then take screenshots for comparing. It also let you take screenshots where you need so you are free to compare not only two urls but also with many different cases.
pip install huxley
java -jar selenium-server-standalone-2.44.0.jar
First you need to create a Huxleyfile and add your testing urls like following format:
[home] url=http://www.maidan.dev [about] url=http://www.maidan.dev/#/aboutAfter you run huxley by --record parameter, it lets you to create your case and by pressing enter key it takes a screenshot. Cases is recorded inside of the home.huxley and about.huxley file as record.json as json format like the following:
Summary for Huxley Experience: