Mobile test automation with AWS Device Farm

Utilizing the Java SDK to integrate AWS Device Farm efficiently into a CI pipeline.

Somewhere along our mobile test automation journey most of us reach a point where we consider moving to a cloud platform. The advantages sound promising: a large variety of devices, no maintenance effort, and flexible scaling. The reality is slightly more complicated, we must always understand our exact requirements before purchasing a plan with any of these vendors. While in certain cases building a device lab on-premise can be an effective solution, cloud providers are an exciting and useful tool to take any mobile test automation to the next level.

In this article, we describe how to utilize AWS Device Farm’s public cloud in a way that we can easily integrate into a CI pipeline. For this example, we use Java, Cucumber, and Appium but the concepts apply to other technologies as well.

Server-side execution model

When executing tests, AWS (at least in their public cloud) uses an approach which I call server-side execution. This means that the test code runs entirely on their infrastructure. The alternative to this is what providers like Browserstack or Saucelabs use where the test code runs on the user’s machine, called client-side execution, and interacts with the cloud service via a web service. In general, client-side execution is easier to implement; if you already know how to execute tests locally, the configuration changes you need to make are minimal.

AWS’s approach is a bit trickier as it requires a few extra steps, namely packaging and uploading everything necessary to run your tests to their servers. This comes with some challenges and pitfalls which I’ll cover in this article.

Executing tests via the web console

The easiest way to start running tests on AWS Device Farm is through their web console. It is a tedious process if you need to do this frequently (we will talk about using their API in just a few moments), but going through the web console is an excellent way to familiarise ourselves with the different steps in the process.

First, let’s log into AWS and open the Device Farm dashboard:

Then we create a new project:

And create our first run:

As we are testing a native Android application (StackOverflow), our first step is to upload its APK file:

Next, we need to select the type of test. We use Appium Java JUnit:

Using Maven, we can create a test package as a zip file by following the steps in the documentation. Don’t get thrown off by the statement that only Java 8 is supported. We will deal with this later (and of course use Java 12 instead).

In this demo, we use the JustTestLah! test framework using Java, Cucumber, and Appium (Disclaimer: I’m the author).

You can either upload the test package from this link or build the demo yourself using Maven (the zip file will be generated in justtestlah-demos/target):

Once you upload the zip to AWS, you will be asked whether to use the standard environment or create a custom environment, we opt for the latter. The test spec is a YAML file listing a set of shell commands to execute during various stages of the test execution. This gives us some control over the way tests are executed on AWS.

For now, replace the default spec with the following:

version: 0.1

      # AWS still only supports Java 8, so we install Java 12 ourselves
      - wget -q
      - tar -xzf openjdk-12_linux-x64_bin.tar.gz
      - export JAVA_HOME=$(pwd)/jdk-12
      - export PATH=$(pwd)/jdk-12/bin:$PATH
      - which java
      - java -version

      # Required for image recognition but AWS doesn't support it yet (
      #- npm i -g opencv4nodejs

      - export APPIUM_VERSION=1.9.1
      - avm $APPIUM_VERSION

      - echo "whoami `whoami`"
      - aws --version

      # Starting Appium server
      - echo "Starting appium server"
      - >-
        appium --log-timestamp --chromedriver-executable $DEVICEFARM_CHROMEDRIVER_EXECUTABLE  >> $DEVICEFARM_LOG_DIR/appiumlog.txt 2>&1 &

      - >-
        while [ true ];
            if [ $start_appium_timeout -gt 30 ];
                echo "appium server never started in 30 seconds. Exiting";
                exit 1;
            grep -i "Appium REST http interface listener started on" $DEVICEFARM_LOG_DIR/appiumlog.txt >> /dev/null 2>&1;
            if [ $? -eq 0 ];
                echo "Appium REST http interface listener started on";
                echo "Waiting for appium server to start. Sleeping for 1 second";
                sleep 1;

      # Prepare the test code
      - echo "Extracting JAR files"
      - mkdir classes
      - cd classes
      # Unpack everything into `classes` (hacky!)
      - unzip -o ../\*.jar
      - cd ..
      # Copy the feature files to `src/test/resources` (so we don't need to change the `` parameter)
      - mkdir -p src/test/resources/features
      - cp -r classes/features src/test/resources
      # Remove `` from the classpath to avoid conflicts during property resolution
      - rm classes/

      - echo "Writing"
      # We base64 encode the run configuration (``) into
      # this template (see
      # io.github.martinschneider.justtestlah.awsdevicefarm.TestSpecFactory).
      # During the execution we decode it and write it back to a file which is
      # then passed to the test execution. The alternative would be to put it
      # into the test package but we don't want to re-upload this for every
      # change of configuration.
      - echo "I2p1c3R0ZXN0bGFoIHByb3BlcnRpZXMKI1N1biBNYXkgMDUgMDk6MTY6MTggU0dUIDIwMTkKZmVhdHVyZXMuZGlyZWN0b3J5PXNyYy90ZXN0L3Jlc291cmNlcy9mZWF0dXJlcy9zdGFja292ZXJmbG93CnBsYXRmb3JtPWFuZHJvaWQKb3BlbmN2Lm1vZGU9c2VydmVyCmN1Y3VtYmVyLnJlcG9ydC5kaXJlY3Rvcnk9dGFyZ2V0L3JlcG9ydC9jdWN1bWJlcgpwYWdlcy5wYWNrYWdlPWlvLmdpdGh1Yi5tYXJ0aW5zY2huZWlkZXIuanVzdHRlc3RsYWguZXhhbXBsZXMuc3RhY2tvdmVyZmxvdy5wYWdlcwp0YWdzPW5vdCBAb3BlbmN2CnN0ZXBzLnBhY2thZ2U9aW8uZ2l0aHViLm1hcnRpbnNjaG5laWRlci5qdXN0dGVzdGxhaC5leGFtcGxlcy5zdGFja292ZXJmbG93LnN0ZXBzCmFuZHJvaWQuYXBwQWN0aXZpdHk9Y29tLnN0YWNrZXhjaGFuZ2Uuc3RhY2tvdmVyZmxvdy5NYWluQWN0aXZpdHkKYW5kcm9pZC5hcHBQYWNrYWdlPWNvbS5zdGFja2V4Y2hhbmdlLnN0YWNrb3ZlcmZsb3cKbW9iaWxlLmFwcGl1bVVybD1odHRwXDovLzEyNy4wLjAuMVw6NDcyMy93ZC9odWIK" | base64 --decode >

      # Set the path to the app package (we don't care about the platform, we
      # just set both and ignore the other).
      - echo "android.appPath=$DEVICEFARM_APP_PATH" >>
      - echo "ios.appPath=$DEVICEFARM_APP_PATH" >>

      # When executing the code on AWS `cloudprovider=local`
      # Only when we trigger the tests on our machine we set
      # `cloudprovider=aws`.
      - echo "cloudprovider=local" >>
      - cat

      # Finally...
      - echo "Executing test scenarios"
      - java -Dappium.screenshots.dir=$DEVICEFARM_SCREENSHOT_PATH$(pwd)/ -Dlogback.configurationFile=classes/logback-aws.xml -cp classes:dependency-jars/* org.junit.runner.JUnitCore TestRunner



I will explain the details of this configuration later in the article, for now, let’s save it and move on to the device selection:

AWS uses a concept called device pools. Each test execution will run all your tests on all devices in the pool. We just need one device to start with so we create a new device pool with a single Google Pixel:

Next, there is a step called Specify device state which we skip for now (keep the defaults). On the last page, we can review the run. Let’s reduce the timeout so we don’t use up precious device minutes in case something goes wrong:

Finally, we can trigger the test execution:

You can see the run on the dashboard. Click on it and select our test device (Google Pixel) to see live logs and a live-stream of the device (it will take a few minutes to set-up):

This is the scenario we are testing:

Feature: Search and tags 
@web @android @ios
Scenario Outline: Use the search function 
	Given I am on the homepage 
	When I search for "<tag>"
	And I select the first question 
	Then the question is tagged with "<tag>"
		| tag |
		| selenium |
		| appium |

If everything goes well you will receive a success message on the test results page:

Side note: Take note of the duration of the test execution. There is quite a significant overhead for setting up and tearing down the device (and some more from executing the test spec and initializing the WebDriver) that you need to take into account when planning your test executions. Due to this overhead, AWS is generally much better suited to run larger test suites in a single execution than smaller ones in multiple runs. To some extent, this is true for all cloud solutions but this kind of overhead is different for different vendors and thus an important selection criterion.

What next?

Congratulations! You successfully executed your first Appium tests on AWS Device Farm. Let’s look at some of the challenges and see how using a custom test spec can make our lives easier. Then, we will showcase how to use their API to automate the process of scheduling tests.

  1. Re-upload the test package with every change

You can re-use test packages previously uploaded to AWS Device Farm. You only need to package and upload them again if there are changes. This sounds ok in theory, however, since these packages include everything necessary to run the tests, including not only changes in the framework or its dependencies but also changes in your tests, your test data or your test configuration. Changing a single parameter (for example, specifying which tests to run) always requires a new test package.

Let’s say your test suite contains 100 tests and you want to distribute them across 10 devices to reduce the overall execution time. AWS and most other cloud providers don’t support this out of the box.

The naive approach to solving this on Device Farm is to create 10 test packages for each subset of tests, with each configured to execute 10 different tests. However, this requires packaging and uploading multiple large, almost identical zip files. Now assume that next time you want to distribute your tests across 20 devices or that you want to execute a slightly different set of tests. You always need to create and upload new test packages all over again.

This is how we first used Device Farm before they introduced custom environments and it was rather painful. Continually building and uploading 100+ MB zip files makes a CI pipeline a lot more troublesome than necessary.

Custom environments for the rescue

If you take another look at the custom environment specification above, you can see how we can utilize this feature to remove this restriction. The base64 encoded information includes our test configuration. This allows us to customize test runs by uploading a relatively small YAML file (the test spec) rather than re-creating the much larger test package. We encode the configuration during the creation of the test spec. When it’s executed we decode it back into a file which is then passed to the execution as a parameter.

- echo "I2p1c3R0ZXN0bGFoIHByb3BlcnRpZXMKI1N1biBNYXkgMDUgMDk6MTY6MTggU0dUIDIwMTkKZmVhdHVyZXMuZGlyZWN0b3J5PXNyYy90ZXN0L3Jlc291cmNlcy9mZWF0dXJlcy9zdGFja292ZXJmbG93CnBsYXRmb3JtPWFuZHJvaWQKb3BlbmN2Lm1vZGU9c2VydmVyCmN1Y3VtYmVyLnJlcG9ydC5kaXJlY3Rvcnk9dGFyZ2V0L3JlcG9ydC9jdWN1bWJlcgpwYWdlcy5wYWNrYWdlPWlvLmdpdGh1Yi5tYXJ0aW5zY2huZWlkZXIuanVzdHRlc3RsYWguZXhhbXBsZXMuc3RhY2tvdmVyZmxvdy5wYWdlcwp0YWdzPW5vdCBAb3BlbmN2CnN0ZXBzLnBhY2thZ2U9aW8uZ2l0aHViLm1hcnRpbnNjaG5laWRlci5qdXN0dGVzdGxhaC5leGFtcGxlcy5zdGFja292ZXJmbG93LnN0ZXBzCmFuZHJvaWQuYXBwQWN0aXZpdHk9Y29tLnN0YWNrZXhjaGFuZ2Uuc3RhY2tvdmVyZmxvdy5NYWluQWN0aXZpdHkKYW5kcm9pZC5hcHBQYWNrYWdlPWNvbS5zdGFja2V4Y2hhbmdlLnN0YWNrb3ZlcmZsb3cKbW9iaWxlLmFwcGl1bVVybD1odHRwXDovLzEyNy4wLjAuMVw6NDcyMy93ZC9odWIK" | base64 --decode >
- java$(pwd)/ -Dlogback.configurationFile=classes/logback-aws.xml -cp classes:dependency-jars/* org.junit.runner.JUnitCore TestRunner

For a handful of arguments, this may be overkill, but for a larger set of configuration values, a base64 encoded string proves to be quite a robust and elegant way (it also spares us the headache of escaping special characters in YAML).

JustTestLah! uses Cucumber, so, for example, we can run different scenarios by passing separate Cucumber tags:

tags=@regression and @stable and not @skip


tags=@regression and @unstable and @skip

The test package can be the same for each of the runs, only the test spec changes.

Further improvements

We can take this approach even further and remove the need for a test package altogether. Instead, we configure the custom environment to fetch the necessary code and configuration from an external source and build it entirely on the AWS instance.

This allows us to separate our test code from our framework code. The latter should be reasonably stable, only the tests themselves change more frequently (we fetch them during the test spec execution). Without smart usage of the test spec, such scenarios would be a headache to implement.

2. Control the execution environment

If you execute tests on your infrastructure, you are in full control of the tools you use and their versions. With AWS Device Farm, it’s not that simple.

The default environment still uses Java 8 (which is four major releases behind the current JDK 12). Other libraries may have similar issues, but for Java, there’s an easy fix it can be downloaded and installed by adding a handful of lines to the environment configuration:

      # AWS still only supports Java 8, so we install Java 12 ourselves
      - wget -q
      - tar -xzf openjdk-12_linux-x64_bin.tar.gz
      - export JAVA_HOME=$(pwd)/jdk-12
      - export PATH=$(pwd)/jdk-12/bin:$PATH
      - which java
      - java -version

However, this is a workaround and not a solution. You pay for the device minutes from the time the execution starts and any additional, time-consuming steps in the test spec can add up in the long run.

Executing tests via API

Now that we’ve gone through the steps of manually creating a test execution let’s see how we can use the excellent AWS Java SDK (there are SDKs for other programming languages too) to automate the process.

For this, you need to create an AWS user with access to Device Farm and generate an access key and a secret key. I recommend setting up AWS CLI as well and configure it by calling:

aws configure

This will ask you for the access and secret keys generated before. As a region, set us-west-2. Your keys will be stored in your user home (under ~/.aws/credentials) and the default credentials provider used in the SDK will pick them up from there. Be careful to keep them as far away from your source code as possible.

Once this is done, let’s have a look at some JAVA code. The full source can be found on GitHub. For this demo, we call AWS from inside a JUnit runner class (the interesting bits are inside the run method). Note, that instead of a device pool we use a concept called device filters to determine the test device to use.

package io.github.martinschneider.justtestlah.awsdevicefarm;

import com.amazonaws.auth.AWSStaticCredentialsProvider;
import com.amazonaws.auth.BasicAWSCredentials;
import io.github.martinschneider.justtestlah.utils.FileEntity;
import java.util.Map;
import java.util.UUID;
import org.apache.http.HttpResponse;
import org.apache.http.client.methods.HttpPut;
import org.apache.http.impl.client.CloseableHttpClient;
import org.apache.http.impl.client.HttpClients;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;

/** Service class to interact with AWS, basically a wrapper around {@link AWSDeviceFarm}. */
public class AWSService {

  private static final Logger LOG = LoggerFactory.getLogger(AWSService.class);

  private AWSDeviceFarm aws;

  /** Constructor */
  public AWSService() {
    aws = buildClient();

   * Constructor
   * @param awsConfiguration key-value map holding AWS configuration
  public AWSService(Map<String, String> awsConfiguration) {

    if (awsConfiguration == null) {
      // use the default credentials provider
      aws = buildClient();
    } else {
      String awsAccessKey = awsConfiguration.get("accessKey");
      String awsSecretKey = awsConfiguration.get("secretKey");
      String awsRegion = awsConfiguration.get("awsRegion");
      if (awsAccessKey == null || awsSecretKey == null || awsRegion == null) {
        // if any of the values is not set we use the default credentials provider
        aws = buildClient();
      } else {
        aws = buildClient(awsAccessKey, awsSecretKey, awsRegion);

   * Constructor
   * @param awsAccessKey the AWS access key
   * @param awsSecretKey the AWS secret key
   * @param awsRegion the AWS region
  public AWSService(String awsAccessKey, String awsSecretKey, String awsRegion) {
    aws = buildClient(awsAccessKey, awsSecretKey, awsRegion);

  /** @return {@link AWSDeviceFarm} */
  public AWSDeviceFarm getAws() {
    return aws;

  private AWSDeviceFarm buildClient() {
    LOG.debug("Building AWS Device Farm client using default credentials provider");
    return AWSDeviceFarmClientBuilder.standard().build();

  private AWSDeviceFarm buildClient(String awsAccessKey, String awsSecretKey, String awsRegion) {
    LOG.debug("Building AWS Device Farm client");
    LOG.debug("awsAccessKey={}", awsAccessKey);
    LOG.debug("awsSecretKey={}", awsSecretKey);
    LOG.debug("awsRegion={}", awsRegion);
    return AWSDeviceFarmClient.builder()
            new AWSStaticCredentialsProvider(new BasicAWSCredentials(awsAccessKey, awsSecretKey)))

   * Upload a file to AWS Device Farm (modified from
   * @param file the file to upload
   * @param projectArn the ARN of the Device Farm project
   * @param uploadType the {@link UploadType}
   * @param synchronous true, if the execution should wait for the download to succeed
   * @return {@link Upload}
   * @throws InterruptedException {@link InterruptedException}
   * @throws IOException {@link IOException}
   * @throws AWSDeviceFarmException {@link AWSDeviceFarmException}
  public Upload upload(File file, String projectArn, UploadType uploadType, Boolean synchronous)
      throws InterruptedException, IOException, AWSDeviceFarmException {
    CreateUploadRequest appUploadRequest =
        new CreateUploadRequest()
            .withName(UUID.randomUUID() + "_" + file.getName())
    Upload upload = aws.createUpload(appUploadRequest).getUpload();

    CloseableHttpClient httpClient = HttpClients.createSystem();
    HttpPut httpPut = new HttpPut(upload.getUrl());
    httpPut.setHeader("Content-Type", upload.getContentType());

    FileEntity entity = new FileEntity(file);

    LOG.debug("AWS S3 upload URL: {}", upload.getUrl());

    Thread thread =
        new Thread() {
          public void run() {
            HttpResponse response = null;
            try {
              response = httpClient.execute(httpPut);
            } catch (IOException exception) {
              throw new AWSDeviceFarmException(
                  String.format("Error uploading file to AWS: %s", exception.getMessage()));
            if (response.getStatusLine().getStatusCode() != 200) {
              throw new AWSDeviceFarmException(
                      "Upload returned non-200 responses: %d",
    int progress = 0;
    while (thread.isAlive()) {
      int newProgress = entity.getProgress();
      if (newProgress > progress) {"{}% completed {}", progress, file.getAbsolutePath());
        progress = newProgress;

    if (synchronous) {
      while (true) {
        GetUploadRequest describeUploadRequest = new GetUploadRequest().withArn(upload.getArn());
        GetUploadResult describeUploadResult = aws.getUpload(describeUploadRequest);
        String status = describeUploadResult.getUpload().getStatus();

        if ("SUCCEEDED".equalsIgnoreCase(status)) {
"Uploading {} succeeded: {}", file.getName(), describeUploadRequest.getArn());
        } else if ("FAILED".equalsIgnoreCase(status)) {

              "Error message from device farm: '{}'",
          throw new AWSDeviceFarmException(String.format("Upload %s failed!", upload.getName()));
        } else {
          try {
                "Waiting for upload {} to be ready (current status: {})", file.getName(), status);
          } catch (InterruptedException exception) {
  "Thread interrupted while waiting for the upload to complete");
            throw exception;
    return upload;

Now, even if the tests run on AWS’ infrastructure, we can wrap them in a local JUnit execution (with some limitations) and execute them just the same way as local Appium tests or using a cloud provider with client-side execution.

If you pull JustTestLah! from Github and install it into your local Maven repository by calling mvn install -DskipTests=true you can trigger the same tests we scheduled through the web console before by calling:

mvn -pl justtestlah-demos test -Dtest=TestRunner

The only value you need to set in is the ARN (Amazon Resource Name) of your Device Farm project. You can easily find this using AWS CLI:

aws devicefarm list-projects

Please note, that the path to must be absolute (feel free to contribute to JustTestLah! to improve this). You can find the demo configs under justtestlah-demos/demos. Using a symlink ln -s justtestlah-demos/demos /demos all commands work as shown. Alternatively, modify the path as needed.

You can easily skip uploading a new app package or test package by specifying the ARN of an existing one in the properties file:


If you wanted to run the same tests on Browserstack, you’d just pass a different configuration (set your Browserstack email and access key in

mvn -pl justtestlah-demos test -Dtest=TestRunner

For local execution (make sure Appium is running, and you have at least one connected Android phone or running emulator) you would call:

mvn -pl justtestlah-demos test -Dtest=TestRunner

As you can see, using this technique the cloud provider for executing the tests is nothing more than a configuration value. You can seamlessly switch between local executions, AWS Device Farm and Browserstack. The next level of encapsulation would be to maintain different devices across multiple platforms and let the framework handle everything else, i.e. pick the cloud provider for each test which can offer the requested device (and can do it fastest).

If you have any experience with this kind of setup or want to experiment with it please drop me a line.


Besides the device filters, there is some additional configuration you can define for AWS test executions (those were the settings we skipped during the Specify device state on the web console):

# Device filters

# Device configuration
# set this to true if you use device slots 

# Additional AWS Devicefarm configuration

Maven integration

For a slightly different way of integrating AWS Device Farm (but using the same Java SDK), you can have a look at my Maven plugin. It provides goals for each step (uploading app package, uploading test package, scheduling a run etc.) which allows you to separate them. For example, you can upload a new app package every time there is a new build and upload a new test package every time there is a change in your test code. The test execution itself can then be independent of both of these and just re-use the latest (or any specific) app and test packages.

Jenkins plugin

There is also a Jenkins plugin for AWS Device Farm but it is not as flexible as the solutions described in this article.


In this write-up, we showcased a PoC-style demo of how to run tests on AWS Device Farm in just about the same way as locally or on client-side execution platforms like Browserstack. Getting your tests running on AWS Device Farm and especially integrating them into a build process in an efficient way is not a straightforward process (for all but the most basic requirements), and I hope the examples can help others who are facing similar challenges.

Once things are running, AWS is a reliable cloud provider for mobile automation, so it’s worth sticking out and customizing things to your needs.

If you have any questions or comments feel free to contact me.