Automated Visual Regression Tests with JLineup

JLineup can be used as a simple command line tool or it can run as a web server which takes your screenshot jobs as REST calls. The only additional requirement is a Chrome or a Firefox web browser. Once you have downloaded JLineup, literally all you have to do is call it before a deployment, deploy, and then run it again. We integrated this into our build pipeline, which looks like this (a little simplified):

Whatever your integration would look like, it’s good to not let JLineup fail your build if it detects changes, because if changes are intended, you want the possibility to go on with your deployment. We solve this through using gates or transitions, which can be opened or closed. Whenever JLineup detects differences, the transition to the Live deployment is closed. Of course we have integrated it into our monitoring systems so that when a Live deployment is closed the team checks the JLineup HTML report which was generated:

If we have a false positive, which means the detected changes are OK, we open the transition manually and the build goes on. Otherwise, the bug has to be fixed and a new build is started thereafter. Depending on your CI system, there may be other ways to integrate JLineup into the flow. If you work with feature toggles, the pipeline should rarely stop because of visual differences.

Quick Tutorial

So, you would like to give it a try? It’s easy and it’s free! JLineup is Open Source, licensed under the Apache-2.0 license. This tutorial is running on Linux, which is the platform we support. It should also work under MacOS or Windows, but we (currently) don’t test this.

Installation

You can simply download the CLI version by getting the jlineup-cli.jar from Maven Central:

Now you have a jlineup.jar in your current directory.

For best results, you need a „real“ web browser installed on your system. JLineup supports Google Chrome (Chromium also works) or Firefox at the moment. As JLineup is written in Java, you need a Java Runtime Environment version 8 or higher. OpenJDK is fine! JLineup can use PhantomJS internally, if no other browser is available. This is not recommended, as PhantomJS is not maintained any more. More and more modern web features don’t work in PhantomJS, so it’s only an emergency fallback for very simple websites. If these prerequisites are fulfilled, you can make a test run:

If you see an error message now, something is wrong. You should see a usage explanation with all possible command line arguments for JLineup. JLineup doesn’t have any other dependencies, just make sure you have the jlineup.jar in your current directory or somewhere in $PATH to use it.

Basic usage

For a first test, you don’t need any configuration:

Notice the step declaration ‚before‘. JLineup is printing the generated config to standard out and it starts to browse to time.gov and to make screenshots of it’s current state. Wait some seconds and make the ‚after‘ run:

Below your current directory, there should be a ‚report‘ folder now:

Open the report.html file with a browser and you can see your first JLineup comparison report! For programmatic analysis of the JLineup job outcome, you can also find the report.json file with detailed insights about the job’s outcome. If the ‚before‘ and the ‚after‘ run of a job show no differences, the ‚after‘ run of JLineup exits with return code 0. If there are differences, it exits with 1. For ‚real‘ usage, you shouldn’t use the –url parameter, but you should create a configuration file.

Here’s a small example config: