Test Case Documentation

Motivation

From my point of view the test case documentation of automatic tests is important. It helps to maintain the tests in the future and the documentation is useful for co-workers if they would like to understand the tests or if they would like to gain more information about a test. It is also important that the documentation is easy available, e.g. from the test report.

We distinguish between log outputs that are written during test execution and the documentation of the test case itself. This post will outline how we cover both of these topics.

Environment

We implement our tests for our internet portal with Java, TestNG and Selenium WebDriver. We execute our automatic tests with the continuous integration system Jenkins (with testng-plugin).

Test case documentation

To document the test cases we are using Javadoc. The Javadoc for each test method contains a brief description what the test is doing and which steps are executed. The Javadoc class summary section outlines which area of the software is covered with the tests of the class.

One of the steps during building and executing the tests is to generate the Javadoc, which could be for example done with a Gradle task (or Ant, Maven).

Test execution documentation

I would recommend adding log output statements in the source code of the test. The log statements are written during the test run in a log file. This helps in analyzing why a test failed. The log statements can be accessed from the Jenkins TestNG Results view. This includes:

  • Log statements,
  • Output of the current URL when a test fails and
  • Generate a screenshot of the current page when a test fails

Listed below you will find some code snippets showing how we implemented this. The Java class containing all the snippets is attached to this post.

Log statements

The following example shows how to write log statements, this could be encapsulated in a utility method.

Reporter.log(new Date() + ">> Your log statement", true);

Output current URL and screenshot

In case of a test failure the current URL is written to the log file and also a screenshot is taken. In the example below you have to adapt the path of this image file to your environment. The link to the screenshot is also written to the log file, so that the screenshot can easily be accessed from the test results view. We implemented a method in our base test class with the TestNG annotation AfterMethod which is called after every test.

...
private static final String HREF = "Last URL: <a href="%s">%s</a>";
private static final String HREF_IMG = "&lt;a href="%s"&gt;&lt;img src="%s" alt="" width="100" height="100" /&gt;&lt;/a&gt;";
...
@AfterMethod (alwaysRun = true)
public void logResult(final ITestContext context, final ITestResult result)
 throws IOException {
  Reporter.setCurrentTestResult(result);
  if (!result.isSuccess()) {
    Reporter.log(String.format(HREF, driver.getCurrentUrl(),
      driver.getCurrentUrl()));
    File scrFile =
     (TakesScreenshot)driver).getScreenshotAs(OutputType.FILE);
    String imagePath = result.getName() + ".jpg";
    File file = new File(imagePath);
    FileUtils.copyFile(scrFile, file);
    Reporter.log(String.format(
      HREF_IMG, file.getName(), file.getName()));
  }
}

Linking test report with Javadoc

We include in our test logging also a hyperlink to the Javadoc at the beginning of the log output of each test. We find this helpful because it saves time to look up the Javadoc. In order to get this link for every test case we implement a method with the TestNG annotation BeforeMethod in our base test class. This method is called before the test is executed.

import java.lang.reflect.Method;// Adapt the javadoc path to your environment
private static final String JAVADOC_LINK = "<a href="build/docs/javadoc/%s/%s.html#%s">%s</a>";
 
@BeforeMethod (alwaysRun = true)
public void logTestStart(final ITestResult result, final Method method) {
  Reporter.setCurrentTestResult(result);
  Class&lt;?&gt; c = this.getClass();
  String javadoc = String.format(JAVADOC_LINK,
    c.getPackage().getName().replace(".", "/"),
    c.getSimpleName(), method.getName(), 
    c.getSimpleName() + ":" + method.getName());
  Reporter.log("&gt;&gt; Start test method: " + javadoc);
}

Compare Screenshots with Selenium WebDriver

Use case and motivation

Sometimes it‘s necessary to test the design or the correct position of one or more elements on a website. In this case, it could be useful to compare screenshots of the tested website with a reference screenshot.

And it’s a good idea, to do this kind of test automatically.

Below we will show how to compare a taken screenshot of any website with a reference screenshot and which tools are recommendable to do this smart.

We will point some pitfalls and defiance with this. E.g. how the test reacts with changing elements like advertising media.

Compare screenshots implementation

The example below shows how an image comparison with ImageMagick® and im4java can be implemented. The source code is not following coding standards, it serves only to illustrate the image comparison.

ImageMagick can be used to create, edit, compose, or convert bitmap images. The functionality of ImageMagick is typically utilized from the command line. Im4java is a pure-java interface to the ImageMagick command line.

The first source code snippet shows the method to compare images which we are using later. The different metrics which can be used to compare images are explained on the ImageMagic website. When the images are not equal the compare command will throw an exception.

import org.im4java.core.CompareCmd;
import org.im4java.process.StandardStream;
import org.im4java.core.IMOperation;
...
boolean compareImages (String exp, String cur, String diff) {
  // This instance wraps the compare command
  CompareCmd compare = new CompareCmd();
 
  // For metric-output
  compare.setErrorConsumer(StandardStream.STDERR);
  IMOperation cmpOp = new IMOperation();
  // Set the compare metric
  cmpOp.metric("mae");
 
  // Add the expected image
  cmpOp.addImage(exp);
 
  // Add the current image
  cmpOp.addImage(cur);
 
  // This stores the difference
  cmpOp.addImage(diff);
 
  try {
    // Do the compare
    compare.run(cmpOp);
    return true;
  }
  catch (Exception ex) {
    return false;
  }
}

The next source code snippet is using the just introduced method to compare images. At first a webpage is opened and a screenshot is taken.

import org.openqa.selenium.WebDriver;
import org.openqa.selenium.firefox.FirefoxDriver;
...
// Get the driver and open the page
WebDriver driver = new FirefoxDriver();
driver.get("http://testandwin.net");
 
// Take Screenshot
File scrFile = ((TakesScreenshot)driver).
getScreenshotAs(OutputType.FILE);
 
String current = "c:/temp/image.png";
FileUtils.copyFile(scrFile, new File(current));
 
// Compare the images
boolean compareSuccess =
  compareImages("c:/temp/expected.png", current, "c:/temp/difference.png");
 
// Close the driver
driver.close();

It is also possible to take a screenshot only from a certain web element. If you would like to do this you can include the following code snippet before the FileUtils.copyFile(…) call.

import java.awt.image.BufferedImage;
import javax.imageio.ImageIO;
import org.openqa.selenium.Point;
...
WebElement webElement = ...;
BufferedImage image = ImageIO.read(scrFile);
Point point = webElement.getLocation();
BufferedImage elementImage = image.getSubimage(
      point.getX(), point.getY(), 
      webElement[0].getSize().getWidth(), webElement[0].getSize().getHeight());
ImageIO.write(elementImage, "png", scrFile);

How to deal withing changing parts

Most websites contain dynamic elements like advertising media, version numbers, dates, etc. These elements make it almost impossible to compare screenshots. The solution we are using is to hide those web elements with the method listed below.

hideElement(WebElement e, WebDriver d) {
  ((JavascriptExecutor)d).executeScript("arguments[0].style.visibility='hidden'", e);
}

Compare Image Size

When compare screenshots it could be useful to compare first the image size because when the image size is different the comparison will fail. For example image sizes can differ when taking screenshots on different machines.

import java.awt.image.BufferedImage;
...
BufferedImage image = ImageIO.read(new File(path));
String size = image.getWidth() + "x" + image.getHeight();

Prerequisites

Before using  these examples, you have to install ImageMagick on the machine which is running the tests. The installation is described on ImageMagick website.

Additionally you have to include the im4java jar in your classpath.

Example images

Below you will find one example of the screenshot comparison. The first one shows the current image, the second one the expected image and the third one is the difference image.
Compare Screenshots - Current Image

Compare Screenshots - Source Image

Compare Screenshots - Difference Image