The definition of “mobile” as we once knew it is changing – going beyond smartphones and tablets, apps are now being incorporated into cars, wearable devices and home appliances. As mobile apps become increasingly integrated with a variety of connected devices, the process of testing these applications is becoming more complex.
Not only must each mobile application be tested on a wide variety of smartphones and tablets, but the addition of connected devices creates another level of complexity. However, it is possible to create a mobile application that delivers a consistent experience across all devices and platforms, and it doesn’t have to take months of testing to get there. This quality can be achieved through mobile testing automation. Leveraging automation tools such as Appium, Calabash and Selendroid, you can bring your mobile application to market faster, immersing your users in addictive mobile experiences across all of their connected devices.
The quality assurance team (QA) cannot guarantee that if a tested application works well on a given device, it will work 100% on another device even if it’s from the same product family – the screen resolution, CPU, Memory, OS optimization and hardware could be different. For an application that is intended to perform on a variety of connected devices outside of smartphones (for example, an Apple Watch), the process of evaluating the quality of the application becomes exponentially more complex.
The process of acquiring these devices, or a reliable emulator, and manually testing across all potential devices and operating systems could take months even with a team of top-notch quality assurance engineers. Even if you took the time to manually test across all devices, the extended period of time it would take to release the application could jeopardize your ROI should a competitor launch a similar application more quickly.
Automating the testing process reduces the time it takes to launch a quality product by using specific tools, such as Xamarin Calabash Cloud Test lab (Open Source), where the application can be tested on a plethora of real devices to ensure full functionality across all devices. Below, we’ll explain some of the automation options that are available and provide examples of how they may be used depending on the application.
Mobile Testing Frameworks
Image Source : Perfecto Mobile
Here are some tools you can use for automation testing and the situations in which each tool is ideal:
- An IDE ( Integrated Development Environment), so that scripting is reduced
- A keyword-driven approach, so that manual testers can effectively use it
- Integration with a mobile cloud to help overcome device challenges by identifying objects based on both property and image
1. Appium (Android and iOS)
Appium is an open-source tool for automating native, mobile web, and hybrid applications on iOS and Android platforms. Native apps are those written using the iOS or Android SDKs. Mobile web apps are web apps accessed using a mobile browser (Appium supports Safari on iOS and Chrome or the built-in browser app on Android). Hybrid apps have a wrapper around a “webview” — a native control that enables interaction with web content. It has no dependency on mobile device OS because APPIUM has a framework that translates Selenium Webdriver commands into UIAutomation (iOS) or UIAutomator (Android) commands depending on the device type (not OS type). It supports all languages that have Selenium client libraries, such as Java, Objective-C, JavaScript with node.js, PHP, Ruby, Python, C# etc.
2. Calabash (Android and iOS) / Xamarin Cloud Mobile Test Lab
Calabash is a free, open source project that helps automated acceptance testing frameworks for mobile apps to run tests on iOS, Android and simulators. Calabash could be compared to Selenium WebDriver, however, it is important to realize that interacting with a web app from a desktop computer is vastly different than interacting with a native app using a touch screen.
Calabash provides APIs that are specialized to native apps running on touch screen devices. Testers will create a sampling of mobile devices representative of the devices from which users will access the application. This is required in order to ensure they have not simply tested the app on an emulator or simulator, but on an actual real device. This is because emulators are not always reliable – the variability of OS, device type, fragmentation, screen, memory and other factors can impact the way applications function on a mobile device.
Calabash consists of two libraries – Calabash Android and Calabash iOS. This helps in writing tests in domain-specific languages. They enable test code to interact with the apps, wherein each of these interactions consists of a number of end-user actions like gestures, assertions or screenshots. You can even test the HTML5 part of the application using the Cucumber interface.
3. Robotium
Robotium supports both native and hybrid apps through its Android test automation framework. It provides test case developers an easier way to write functional, system and acceptance test scenarios with minimal knowledge of the application. The framework can handle multiple android activities automatically and can integrate smoothly with Maven or Ant for continuous integration. With the support of Robotium, test case developers can write function, system and acceptance test scenarios, spanning multiple Android activities. Robotium officially supports Android 1.6 and up. Robotium has full support for Activities, Dialogs, Toasts, Menus and Context Menus. To develop stable and reliable tests, Robotium offers many methods that react to different graphical elements within an Android app, such as:
clickOnText(“Secure Login”); clickOnButton(“Save”); searchText(“Logout”); goBack(); getButton(); isRadioButtonChecked();.
With these simple methods, robust automated tests can be implemented really quickly. By combining them with JUnit, you then have additional ways of checking values or user interactions on the device for the correct response, which in turn makes these tests even more powerful. The test case execution is generally faster and more robust.
4. MonkeyTalk (Android and iOS)
MonkeyTalk automates real, functional interactive tests for iOS and Android apps – everything from simple “smoke tests” to sophisticated data-driven test suites for native, mobile, hybrid apps, real devices or simulators. It is a powerful testing tool. The same script can be used on both Android and iOS of an app, i.e. record on iOS, playback on Android and vice versa. MonkeyTalk is simple and easy to learn, and its scripts can be understood by the layman. Both phones and tablets are supported, ei
ther tethered or over the network. MonkeyTalk scripts can be easily extended, even by non-programmers, with custom, user-defined commands.
5. Selendroid (Android)
Selendroid is a test automation framework which drives on the UI of Android native/hybrid applications and the mobile web. Tests are written using the Selenium 2 client API. You can test the application under test using Selendroid without any modification of the app. You just need the binary file (APK) installed on the computer. In order to install the binary file on the device, the test app and mobile app must be signed with same sign key. Selendroid test app can interact with multiple devices or simulators simultaneously, which is a great advantage of Selendroid. So you can test your app with various android devices to check compatibility. Selendroid can simulate human user actions on an app, such as touch, swipe, drag and drop. You can change the hardware devices (plug and unplug) during testing without restarting or stopping the test because Selendroid recognizes the new devices automatically.
6. UI Automator (Android ) & UI Automator (IOS)
The Android UIAutomator testing framework lets you test your UI (user interface) efficiently by creating automated functional UI test cases that can be run against your app on one or more devices. The UIAutomator API is bundled in the UIAutomator.jar file under /platforms/directory. The API includes these key classes, interfaces, and exceptions that allow you to capture and manipulate UI components on the target app class for interacting with the device’s UI by simulation user actions and introspection of the screen content. It relies on the platform accessibility APIs to introspect the screen and to perform some actions on the remote view tree. It also allows injecting of arbitrary raw input events simulating user interaction with keyboards and touch devices. One can think of a UiAutomation as a special type of accessibility service which does not provide hooks for the service lifecycle and exposes other APIs that are useful for UI testing automation.
For IOS UI Automation, use the Automation Instrument to automate user interface tests for your iOS app through written test scripts. These scripts simulate user actions by calling UI Automation, a JavaScript programming interface that specifies actions to be performed in your app as it runs. During the tests, the system returns log information to you. UI Automation testing is an important feature for mobile application developers that was introduced in iOS4. It is supported by a new instrument object called “Automation.” It’s quite suitable for UI testing of productivity style applications and is both a probe for Instruments, as well as a JavaScript library provided by Apple, to exercise and validate a running application. It’s a powerful tool you can use to collect data about the performance and behavior of one or more processes in the system. Each instrument collects and displays a different type of information, such as file access, memory use, and so forth.
Automation Instrument works from scripts (written in JavaScript). It simulates/fires required events on the target application. The test script must be a valid, executable JavaScript file accessible to the instrument on the host computer. You can launch Instruments by opening Xcode and select:
Xcode > Open Developer Tool > Instruments.
7. Xamarin With (VSMC ) Visual Studio Mobile Center – Continuous Integration (CI)
Continuous Integration (CI) is a development practice that requires developers to integrate code into a shared repository several times a day. Each check-in is then verified by an automated build, allowing teams to detect problems early.This practice can be applied with a great tool called Visual Studio Mobile Center. This is platform offered by Microsoft to enable developers to build, test and release their applications with everything needed to apply Continuous Integration. Visual Studio Mobile Center’s tight integration with “GitHub” and GIT commits can be set up to automatically trigger iOS and Android builds, and automatic upload of those builds, to Xamarin Test Cloud to run pre-defined, automated UI tests. This is very helpful while committing new code changes to a Git Repository – it can trigger the entire process in the cloud of compiling and building the .ipa and .apk files, transferring them to Test Cloud and running the pre-written tests. It provides a single platform to manage all aspects of continuous integration and continuous deployment, including build, test, distribution, crash reporting, and analytics.
This is what Visual Studio Mobile Center offers:
- Continuous integration – the developer is able to connect to the “GitHub” repository, select branch and configure build
- App distribution – release application to selected users who are members of specific Mobile Center groups
- Crashes – With Mobile Center SDK, testers have full control over applications – they can monitor crashes and react fast
- Analytics – Mobile Center SDK enables testers to automatically track session, device properties and pages
Mobile platforms: Objective-C, Swift, Java, Xamarin, React Native
Lifecycle features: Build, Test, Crash, Beta Distribution, Analytics
Cloud features: Authentication, Easy tables, Offline sync
Automated Testing in Action: Mobile Health Apps
To demonstrate the benefits of automated testing, let’s look at how it works with a mHealth application. This particular app requires a login/Signup, user authentication and some other functionality, such as showing health data for daily activities – exercise, counting steps and distance traveled, how often the user stands, daily calorie burn, BMI and nutrition information.
First, testers will look for common functionalities in the application. For example, the login, logout and some fitness activity tracking functionalities may be similar. We can create test submodules for login, logout & other repetitive functionalities using Appium using hand-written test scripts or any record & playback automation tool based on our need and environment. Testers will identify these common functions and create test cases for them. Once test cases have been created, automated tests can be run repeatedly. Additionally, these can be completed more rapidly than traditional regression testing which uses manual testing techniques. Automated software testing can reduce the time required to run repetitive tests from weeks to hours, reducing tim
e investment and associated costs by more than half.
Work flow path scripts: the above diagram shows how to perform a certain activity using an automation tool.
Recommendations
As you can see, there are many amazing mobile testing automation tools available that can minimize the costs associated with manual mobile testing. You may be asking yourself the question: “how can I automate the testing of my next mobile application, and what tools should I use?”
It is not a matter of choosing one tool or technique because there are simply too many compromises that must be made to go completely one way or the other. Most likely, you will need to use a combination of testing tools and techniques to meet your quality requirements. Generally, you can narrow your choices down based on the following criteria:
- Take advantage of a quality device emulator. Emulated devices provide a cost-effective means of performing the bulk of your testing in a well-instrumented test environment, quickly and efficiently. You will want to use your device emulator with various options, such as bypassing the network, using the live network and a good scripting language, so you should look for these features during your selection process. When you look at device emulators for testing, make sure they have the instrumentation and network options to provide the flexibility you will need. Also, find out if the emulator has the necessary diagnostics to isolate problems, as well as flexibility in network stacks needed to test different network options. Determine if your emulated device solution contains a high-level scripting solution to allow you to replay your test cases over and over again. Finally, look for an emulated device which will allow you to change device profiles quickly.
- Invest in a real device cloud and automate wherever possible. You might be in a situation where you have to test on a remote live network and not have the necessary device. Having an account with a vendor that lets you access remote real devices at any time is very useful, a great solution to have in your bag of tricks. For example, Xamarin Calabash Cloud Test Lab.
Most businesses will continue to have a mix of manual and automation testing. Manual testing will never go away and, in some cases, it is even a product requirement. This is why it’s vital to have a competent technology partner to advise on the right mix of automated and manual testing. At Productive Edge, we’ve found that as you optimize your overall testing strategy, investing in techniques like BDD (Behavior-Driven Development) can make things much easier for everyone involved with both manual and automation testing throughout the product lifecycle. Achieving release agility with high quality is fully dependent, today more than ever, on continuous testing. This is gained through proper testing automation. Automation is key in today’s digital world, and doing it wisely can shorten time to market, reduce the number of resources required and cut regression testing time.
For a complimentary consultation, connect with us today.
