Visual Regression Testing Tools

Trends in web design change almost daily, so companies try to keep up with them. While web developers update pages layout according to the new design, QA team should make sure that not only pages with the new design look good, but also the rest of the website layout is ok( + usually cross-browser compatibility). A small change in one component can lead to obscure changes in other elements. Regression testing allows finding such changes.

Briefly, regression testing is the process of testing changes to make sure that the already existing functionality still works with the new changes. 

Such tests can be done manually (it often happens), but depending on the size of a project it can take a lot of efforts and be time-consuming. That’s why it is recommended to perform the visual regression testing by using special automated programs to optimize the testing process.

There are numerous tools for visual regression testing. In my opinion, BackstopJS is the most simple and flexible tool.

BackstopJS

Pros: 

• Simple to start with
• Integrated Docker rendering — to eliminate cross-platform rendering shenanigans
• CLI reports
• Render tests with Chrome Headless, Phantom and Slimer
• Simulate user interactions with Puppeteer, ChromyJS and CasperJS scripts
• Ability to test certain page blocks and ignore the others
• Plays nice with CI and source control

Install BackstopJS

BackstopJS can either be installed globally (recommended) or locally in your project. To install globally, run this command:

npm install -g backstopjs

Configure BackstopJS

If your project doesn’t have configured BackstopJS yet, you can create a default configuration file in your current working directory. Run this command in your project’s directory:

backstop init

BackstopJS will create a default configuration file and project scaffolding in your current working directory. Please note: this will overwrite any existing files!

By default, BackstopJS places backstop.json in the root of your project. And also by default, BackstopJS looks for this file when invoked.

{
 "id": "backstop_default",
 "viewports": [
   {
     "label": "phone",
     "width": 320,
     "height": 480
   },
   {
     "label": "tablet",
     "width": 1024,
     "height": 768
   }
 ],
 "onBeforeScript": "puppet/onBefore.js",
 "onReadyScript": "puppet/onReady.js",
 "scenarios": [
   {
     "label": "BackstopJS Homepage",
     "cookiePath": "backstop_data/engine_scripts/cookies.json",
     "url": "https://garris.github.io/BackstopJS/",
     "referenceUrl": "",
     "readyEvent": "",
     "readySelector": "",
     "delay": 0,
     "hideSelectors": [],
     "removeSelectors": [],
     "hoverSelector": "",
     "clickSelector": "",
     "postInteractionWait": 0,
     "selectors": [],
     "selectorExpansion": true,
     "expect": 0,
     "misMatchThreshold" : 0.1,
     "requireSameDimensions": true
   }
 ],
 "paths": {
   "bitmaps_reference": "backstop_data/bitmaps_reference",
   "bitmaps_test": "backstop_data/bitmaps_test",
   "engine_scripts": "backstop_data/engine_scripts",
   "html_report": "backstop_data/html_report",
   "ci_report": "backstop_data/ci_report"
 },
 "report": ["browser"],
 "engine": "puppeteer",
 "engineOptions": {
   "args": ["--no-sandbox"]
 },
 "asyncCaptureLimit": 5,
 "asyncCompareLimit": 50,
 "debug": false,
 "debugWindow": false
}

Let’s look at the main parts of the file.

• id – used for screenshots naming.  Set this property when sharing reference files with teammates — otherwise, omit and BackstopJS will auto-generate one for you to avoid naming collisions with BackstopJS resources.
• viewports – an array of screen size objects with the help of which your site will be tested. You can add as many objects to the viewports as possible but at least there must be one
• scenarios – this is where you set up specific data for your test:
  • scenarios[n].label – required. Used for screenshot naming
  • scenarios[n].url – required. That’s exactly what we want to test. It can be an absolute URL or an URL local to your current working directory

Other scenario properties are not required and could just be added as necessary.

Those are the main parameters but there are still many additional parameters that can help in the testing process.

• onBeforeScript – used to set up browser state before the test
• onReadyScript – script to modify the UI state prior to screenshots e.g. hovers, clicks, etc
• readyEvent – the test can’t be run until this string has been logged to the console
• readySelector – the test won’t start until this selector appears
• hideSelectors – an array of selectors set to visibility: hidden
• removeSelectors – an array of selectors set to display: none
• delay – wait (in milliseconds)

You can emulate user behavior:

• hoverSelector – move the mouse cursor over the specified DOM element prior to the screenshot creation
• clickSelector – click the specified DOM element prior to a screenshot

Comparing two environments:

• referenceUrl – instead of using reference screenshots it is possible to compare pages across two different environments

And that’s not all that BackstopJS can do. See a full list in the official documentation.

BackstopJS supports using Chrome-Headless, PhantomJS or SlimerJS for web app rendering. Chrome-headless (via Puppeteer) is currently the default value and will be installed by default.

Run tests with BackstopJS

First we need to generate test bitmaps. Run this command:

backstop reference

This command will create reference files without previewing in
 bitmaps_reference folder. By default, this command calls the URL property specified in your config. It is useful to compare staging and production, for example: 

"scenarios": [
  {
    "label": "cat meme feed sanity check",
    "url": "http://www.example.com",
    "referenceUrl": "http://staging.example.com:81",
    // ...
  }
]

Now we can tun the tests and compare reference files with new ones:

backstop test

It will generate new images in the bitmaps_test/<timestamp>/ folder and a report comparing the most recent reference screenshots and the current ones will be displayed.

If all test cases have passed, you can update the reference screenshots. Run the following command:

backstop approve

When running this command, all images (with changes) from your most recent test batch will be promoted to your reference collection. Subsequent tests will be compared against your updated reference files.

Different features

• Testing click and hover interactions

BackstopJS can click or hover on element with selector that you define

• Setting cookies

BackstopJS ships with an onBefore script that makes it easy to import cookie files from cookiePath.

• Targeting elements

BackstopJS makes it super easy to capture screenshots of your entire layout or just parts of your layout. This is defined in the your scenario.selectors array. Elements are defined with standard CSS notation. 

• Testing Progressive apps, SPAs and AJAX content

The problem testing these scenarios is knowing when to take the screenshot. BackstopJS solves this problem with two config properties: readySelector, readyEvent and delay.

• Dealing with dynamic content

For obvious reasons, this screenshot approach is not optimal for testing live dynamic content. The best way to test a dynamic app would be to use a known static content data stub – or ideally many content stubs of varying lengths which, regardless of input length, should produce certain specific bitmap output.

• Changing test sensitivity

"misMatchThreshold" (percentage 0.00%-100.00%) will change the amount of difference BackstopJS will tolerate before marking a test screenshot as “failed”. The default setting is 0.1, this may need to be adjusted based on the kinds of testing you’re doing.

• Running custom scripts

Simulate user actions (click, scroll, hover, wait, etc.) or states (cookie values) by running your own Casper.js script on ready. For each scenario, the custom .js file you specify is imported and run when the BackstopJS ready events are fulfilled.

Running tests in Docker container

Screenshots in bitmaps_reference folder should be uploaded to your VCS so everyone in the team should have the same results. However, there could be some difference between the test results on Mac and Linux (or Windows) because of the render difference in these environments. So to resolve this issue you can run tests in a BackstopJS Docker container.

If you don’t have docker, first, you need to install it on your machine, more about it in Docker Documentation. Then make sure Docker is running on your machine and run the commands adding a --docker  flag onto your commands, for example: 

backstop test --docker

The above flag will cause BackstopJS to hit your Docker local client, spin up the BackstopJS container at https://hub.docker.com/r/backstopjs/backstopjs/ and execute your test.

More about BackstopJS Docker container https://github.com/garris/BackstopJS/tree/master/docker