A few months ago Amazon introduced a cloud service named Device Farm for iOS and Android developers to test their apps on physical smartphones and tablets. At the moment they have 200 unique Android devices available to test on with the following testing frameworks supported: Appium Java JUnit, Appium Java TestNG, Calabash, Instrumentation, UI Automator, XCTest.
Integrating Device Farm with Jenkins CI
Amazon provides a way to manually upload APKs and run a Fuzz test to exercise a test app. That works well, but we wanted a way to automate this testing of each new build. Luckily for us, someone has already written a Jenkins plugin for Device Farm.
We already had integration tests running on on a Jenkins slave machine. On
git push, Jenkins rebuilds our test projects and runs tests on a real
Android device via the
gradle connectedCheck command. Since it is just a
single device, we could not be completely sure that different Android devices with
different OS versions will behave similarly. Enter the AWS Device Farm.
By default Amazon offers a pool with five popular Android devices:
- Samsung Galaxy S5 (T-Mobile)
- Samsung Galaxy Tab 4 10.1 Nook (Wifi)
- Amazon Kindle Fire HDX 7 (WiFi)
- LG G Pad 7.0 (AT&T)
- Samsung Galaxy S6 (Verizon)
However, you can configure your own pool with only the devices you need. It doesn't comes free: at the time of this writing, the cost is 17 cents per device/minute with 250 free minutes available. In order to optimize costs, it makes sense to test on a larger pool of devices when shipping release builds and use smaller pools for your regular CI testing.
Here is step-by-step guide on how to configure your AWS account with Jenkins.
After installing the Jenkins plugin for AWS Device Farm, go to the
Jenkins -> Configure System page and enter your AWS access key ID and secret
In your project configuration you now should able to see AWS Device Farm section in the "Post-build Actions" section. Here you select your AWS Device Farm project, which should appear from a dropdown box, along with the device pool and path to your APK built from your Jenkins machine.
We currently don't have UI tests, but you can add them here. For instrumented tests, AWS requires a
separate APK, which we build with
That's it! You should be good to go.
After testing has been completed, you will see nice charts on your Jenkins project page, along with links to web reports on AWS:
In case of a failure, you can download
logcat and instrumentation outputs. Also you can select the "Performance"
tab, where you can see CPU/Memory load charts:
We hope this guide will help you to move faster and to ship better quality apps.