Author: mira

Testing websites on mobile devices with Selenium

Use case and motivation

Nowadays many people use mobile devices, for that reason it is important to optimize your websites for it. More devices imply more bugs and a bigger test effort. To address this topic effectively you should execute your automated tests also on mobile devices. To keep the effort low it is a good idea to use the framework Selendroid for Android devices and ios-driver for iOS devices. Selendroid is a test automation framework which drives off the UI of Android mobile web. As the Selendroid tests are written using Selenium, it is possible to use the same tests for desktop and mobile devices web tests. Ios-driver works with Selenium in the same way as Selendroid.

In this post we explain how to work with Selendroid. Ios-driver follows later.

Testing websites on Android devices

Prepare your system for Selendroid

To prepare your system for testing websites on mobile devices, you have to install and configure some programs. First you have to install Java SDK and to set the JAVA_HOME variable. After that you have to get the latest version of the Android SDK standalone. Please follow the steps on the Android website (http://developer.android.com/sdk/index.html) to install it and set the ANDROID_HOME variable. Now it is time to get the actual selendroid-standalone.jar from the Selendroid website (http://selendroid.io/). If you want to run the tests on a Selenium grid you have to prepare the grid node in the same way.

Connect devices

Selendroid can be used on emulators and on real devices. If you want to test on a real device you have to connect your devices with your local system or with the Selenium grid node system. For Samsung devices you should install Samsung Kies because it contains a special driver for your Samsung device (http://www.samsung.com/de/support/usefulsoftware/KIES/). First activate the developer tools on your Android device and enable USB debugging. Connect your device with your computer via USB (connect devices without a cable did not work for us with Selenium). After that change the connection from your devices from mtp to ptp. Now you are ready to test on your device.

Start Selendroid on local system

Open your command line or shell on your computer. To start Selendroid enter java -jar selendroid-standalone-0.14.0-with-dependencies.jar (you can change the port with parameter -port <portnummer>).

Start Selendroid on Selenium Grid

On Selenium grid you have to start the grid hub and the node. First put the selendroid-standalone-0.14.0-with-dependencies.jar and selendroid-grid-plugin-0.14.0.jar (http://selendroid.io/scale.html) on grid hub. Copy both jars in the same folder where the selenium-server-standalone.jar is stored. Open your command line or shell and start Selenium grid hub with Selendroid:

java -cp "libs/selendroid-grid-plugin-0.14.0.jar:libs/selendroid-standalone-0.14.0-with-dependencies.jar:libs/selenium-server-standalone-2.43.1.jar" org.openqa.grid.selenium.GridLauncher -capabilityMatcher io.selendroid.grid.SelendroidCapabilityMatcher -role hub > $LOG_DIR/$APP_NAME-console.log.

After starting Selendroid you have to configure and to start the grid node. Put both jars on grid node in the same folder selendroid-standalone-0.14.0-with-dependencies.jar and selendroid-grid-plugin-0.14.0.jar. Create a json file in the same folder with following node configuration:

{
"capabilities":
[{
"browserName": "android",
"maxInstances": 1,
"seleniumProtocol": "WebDriver"
}],
"configuration":{
"proxy": "io.selendroid.grid.SelendroidSessionProxy",
"maxSession": 1,
"register": true,
"hubPort": <hupPortnumber>,
"remoteHost": "http//:mynode:<portnumber>",
"hubHost": grid hup ip adress
}

After that register the node with following commands:
java -jar selendroid-standalone-0.14.0-with-dependencies.jar -port <portnumber>
curl -H "Content-Type: application/json" -X POST --data @selendroid-nodes-config.json http://mygridhubip/grid/register

Selendroid implementation

After you have prepared your system, you must configure your test to run your tests on Android devices. Selendroid can be integrated as a node into the Selenium Grid, so you need two different configuration for local and grid.  First you have to add the actual selendroid.jar to your project. You can download it from Maven central (http://search.maven.org/#search|ga|1|selendroid). Than you have to set capabilities and driver for Android.

Code snippet local configuration:

WebDriver driver = new SelendroidDriver(
    new URL("http://localhost:4444/wd/hub"), 
    SelendroidCapabilities.android());

Code snippet for grid configuration:

DesiredCapabilities capabilities = SelendroidCapabilities.android();
capabilities.setBrowserName("android");
WebDriver driver = new SelendroidDriver(
    new URL("http://myseleniumgridhub:port/wd/hub"), capabilities);

The following code snippet shows a Selendroid test example. You can use the same test for desktop browser if you configure the capabilities accordingly for a desktop browser.

@Test
public void myFirstSelendroidTest() {
  driver.get("http://testandwin.net");
  WebElement element = driver.findElement(
    By.xpath(".//*[@id='search-3']/form/label/input"));
  element.sendKeys("Test Automation");
}

Issues

In case Umlauts could not be set correctly when calling sendKeys, please have a look to at http://selendroid.io/advanced.html#syntheticEvents. You can use the following codesnipet which works great with special characters:

Configuration configurable = (Configuration) driver;
configurable.setConfiguration(DriverCommand.SEND_KEYS_TO_ELEMENT, "nativeEvents", false);

If you change your selendroid version to a new selendroid version it could happen that you get this error: android.util.AndroidException: INSTRUMENTATION_FAILED: io.selendroid.io.selendroid.androiddriver/io.selendroid.server.ServerInstrumentation. This error occur if there is a previous version installed on your device conflicting with new version. To fix this issue open command line step into the platform-tools folder of android-sdk and try uninstalling the package from your device with the following command:
adb shell pm uninstall io.selendroid.io.selendroid.androiddriver

Compare Screenshots with Selenium WebDriver

Use case and motivation

Sometimes it‘s necessary to test the design or the correct position of one or more elements on a website. In this case, it could be useful to compare screenshots of the tested website with a reference screenshot.

And it’s a good idea, to do this kind of test automatically.

Below we will show how to compare a taken screenshot of any website with a reference screenshot and which tools are recommendable to do this smart.

We will point some pitfalls and defiance with this. E.g. how the test reacts with changing elements like advertising media.

Compare screenshots implementation

The example below shows how an image comparison with ImageMagick® and im4java can be implemented. The source code is not following coding standards, it serves only to illustrate the image comparison.

ImageMagick can be used to create, edit, compose, or convert bitmap images. The functionality of ImageMagick is typically utilized from the command line. Im4java is a pure-java interface to the ImageMagick command line.

The first source code snippet shows the method to compare images which we are using later. The different metrics which can be used to compare images are explained on the ImageMagic website. When the images are not equal the compare command will throw an exception.

import org.im4java.core.CompareCmd;
import org.im4java.process.StandardStream;
import org.im4java.core.IMOperation;
...
boolean compareImages (String exp, String cur, String diff) {
  // This instance wraps the compare command
  CompareCmd compare = new CompareCmd();
 
  // For metric-output
  compare.setErrorConsumer(StandardStream.STDERR);
  IMOperation cmpOp = new IMOperation();
  // Set the compare metric
  cmpOp.metric("mae");
 
  // Add the expected image
  cmpOp.addImage(exp);
 
  // Add the current image
  cmpOp.addImage(cur);
 
  // This stores the difference
  cmpOp.addImage(diff);
 
  try {
    // Do the compare
    compare.run(cmpOp);
    return true;
  }
  catch (Exception ex) {
    return false;
  }
}

The next source code snippet is using the just introduced method to compare images. At first a webpage is opened and a screenshot is taken.

import org.openqa.selenium.WebDriver;
import org.openqa.selenium.firefox.FirefoxDriver;
...
// Get the driver and open the page
WebDriver driver = new FirefoxDriver();
driver.get("http://testandwin.net");
 
// Take Screenshot
File scrFile = ((TakesScreenshot)driver).
getScreenshotAs(OutputType.FILE);
 
String current = "c:/temp/image.png";
FileUtils.copyFile(scrFile, new File(current));
 
// Compare the images
boolean compareSuccess =
  compareImages("c:/temp/expected.png", current, "c:/temp/difference.png");
 
// Close the driver
driver.close();

It is also possible to take a screenshot only from a certain web element. If you would like to do this you can include the following code snippet before the FileUtils.copyFile(…) call.

import java.awt.image.BufferedImage;
import javax.imageio.ImageIO;
import org.openqa.selenium.Point;
...
WebElement webElement = ...;
BufferedImage image = ImageIO.read(scrFile);
Point point = webElement.getLocation();
BufferedImage elementImage = image.getSubimage(
      point.getX(), point.getY(), 
      webElement[0].getSize().getWidth(), webElement[0].getSize().getHeight());
ImageIO.write(elementImage, "png", scrFile);

How to deal withing changing parts

Most websites contain dynamic elements like advertising media, version numbers, dates, etc. These elements make it almost impossible to compare screenshots. The solution we are using is to hide those web elements with the method listed below.

hideElement(WebElement e, WebDriver d) {
  ((JavascriptExecutor)d).executeScript("arguments[0].style.visibility='hidden'", e);
}

Compare Image Size

When compare screenshots it could be useful to compare first the image size because when the image size is different the comparison will fail. For example image sizes can differ when taking screenshots on different machines.

import java.awt.image.BufferedImage;
...
BufferedImage image = ImageIO.read(new File(path));
String size = image.getWidth() + "x" + image.getHeight();

Prerequisites

Before using  these examples, you have to install ImageMagick on the machine which is running the tests. The installation is described on ImageMagick website.

Additionally you have to include the im4java jar in your classpath.

Example images

Below you will find one example of the screenshot comparison. The first one shows the current image, the second one the expected image and the third one is the difference image.
Compare Screenshots - Current Image

Compare Screenshots - Source Image

Compare Screenshots - Difference Image