Recap | What is mobile app testing?
Mobile app testing is what it sounds like – testing to ensure that your mobile app meets the quality standard to fit your business goals. For the most part, mobile app testing can work in the same frameworks as any generic software testing and you can use all of the material in our blog or on our site.
This article will mainly focus on the challenges which are unique to mobile app testing, for which there are several. If you want a consultation about your own testing approach, you can apply to work with Global App Testing below.
Why is it important to test your mobile apps?
It is important to test all software, and mobile apps are no exception. Bugs in production are much more expensive to fix than bugs in prototype or pre-release stages, and the #shiftleft movement reflects this: earlier and more aggressive QA cycles tend to result in faster software releases. As a testing company, we are biased.
You should test more if:
-
Software bugs affect your business goals. They could impact general metrics including NPS, growth, churn, or impact flow-specific metrics like checkout completion or signup flows. Remember that some goals (such as “shares” by customers) tend to be difficult to diagnose, but do require a non-buggy experience for users as a prerequisite. Calculate the value of high-quality software to your users.
-
Your developers spend time fixing software bugs which they should spend building. When this is the case, it’s a false economy to focus on releasing over testing, because the releases will need to be fixed. Calculate the value of high-quality software to your operations.
In both cases, businesses (particularly small businesses) are inclined to cut corners and low-ball the volume of testing required.
Most common types of mobile app testing
Our process of recommending a testing suite for a software business is highly specific to that business; apps in high regulatory environments, for example, put a practical importance on ensuring that their app complies with regulation.
For all businesses, the major functional test categories – test cases which reflect regression suites new feature suites, and compatibility suites, apply. We’d also recommend exploratory testing, which refers to when testers use a product without a script, to find edge case issues. Relying on the expertise of QA testers rather than stipulating test cases exactly can require less work and surface more bugs.
Mobile apps tend to require greater compatibility testing than web applications. This is because:
-
There are more versions of live mobile operating systems than web browser, so the device environment is more fragmented.
-
There are more different styles and shape of physical devices, so the hardware-oriented errors tend to creep in.
-
For all kinds of reasons, it’s more difficult to automate mobile app tests than tests for web software.
We go into more detail on all three of these challenges below.
Main challenges specific to mobile app testing
The challenges which are specific to mobile app testing arise because of the complexer compatibility needs associated with mobile;
-
How to prioritize different devices. You should run the tests across physical devices, emulators, and remote outsourcing in a mobile app QA process. Since your users are likely to use devices of all kinds, this is one of the greatest challenges for mobile testing.
-
How do you handle version distribution? It can be difficult to test on the right version of your mobile app, considering the large number of builds which are stored and the variety of versions which will be in use by customers. App store rules can also come into play here.
-
Automation on mobile apps is less straightforward. The high burden of test cases required for versions and test cases; the fragmentation of devices; and the barriers to automated scripts which are put in place by security within device; together make automating mobile app testing more difficult.
-
Debugging and data collection. How do you collect information about what went wrong with your application? How do you access logs? How do you create good quality data?
Challenge 1/ When to perform manual versus automated app tests
It’s difficult to automate mobile app tests for the following reasons:
-
Device Fragmentation: Mobile devices come in various sizes, screen resolutions, and operating systems. Creating and maintaining automated tests that can run seamlessly on all these devices is a challenge.
-
Platform Diversity: Android and iOS have different app development frameworks, which make it challenging to create a single script that works for both platforms. This requires more resources and time to create and maintain separate test scripts for each platform.
-
Frequent OS Updates: Mobile operating systems are frequently updated, which may affect app behavior and compatibility. Test automation scripts need to be constantly updated to accommodate these changes, which can be resource-intensive.
-
Dynamic UI Elements: Mobile apps often have more dynamic UI elements and gestures such as swipes, scrolls, and pinches. Automating these interactions can be challenging and may require additional tools.
-
Network Connectivity: Mobile apps are often affected by network conditions, including connectivity, speed, and latency. To accurately test an app, automation scripts should be able to simulate different network conditions, which can be complex to implement.
The above causes lead to three outcomes:
-
A high time investment to automate tests relative to other software; that’s more test cases .
-
Test cases which sometimes have to guide a physical device through automation scripts like non-trivial and browserstack.
-
A shorter test half-life; more flaky tests which occur earlier in the lifetime of an automated test.
For you, this will raise the skill level required to undertake the testing and raise the level of work required to maintain those tests – altering the balance of where testing ROI falls.
Challenge 2/ Getting the appropriate device mix in mobile app testing
In order to get the right device mix for your business, you’re likely to use a mixture of emulators, real devices in-house, and an outsourced solution for mobile app testing.
Emulators
An emulator is a software tool that mimics the functionality of a specific hardware, software, or operating system. It enables developers and testers to run software designed for that specific hardware or OS on their computers.
Official emulators include Android’s Studio Emulator for Android apps or iOS simulator for iOS apps, which we’d generally recommend over third party emulators.
Emulators are the primary tool of choice for much mobile app testing because they’re easier to set up and use than large numbers of physical devices and Android and Apple have made them easy to research and use. But they are limited in accuracy against real-world devices, so keep the following limitations in mind:
-
Hardware interactions: emulators have limitations in testing real-life hardware scenarios such as sensors, battery consumption, camera functionality.
-
Less accurate gesture and interaction testing. Since a human is not interacting with the interface of the device, some of the same issues which crop up in automated tests recur here. Does the device work when you swipe up?
-
Performance testing: emulators may not accurately reflect the performance of your devices, when it comes to processing speed, memory usage, and device responsiveness.
-
Limited coverage: in general, it’s difficult for emulators to cover the huge range of devices, OS systems, Geographies, and OS version fragmentation to cover real user experience in the real world.
The approach of many of Global App Testing’s clients is to use emulators to an extent, but to bookend the process with edge cases and diverse geographies with real-world devices; in-house or via Global App Testing.
Physical device farms in-house
Physical devices can solve some of the above problems. To do this, you should purchase a diverse set of devices, ensuring that the most important devices among your users are well-represented among the device suite. You can attempt to automate the tests on these devices (see below) and you should document and share results within a test management platform like TestRail.
This will solve hardware interaction and accurate gesture testing, but there are clear downsides to doing this in-house.
-
It is expensive to do this in-house.
-
It is difficult to maintain, store, create adequate security, manage these devices
-
It does not solve the issues around geography: you are still testing from one place and one physical environment
-
Device obsolescence and the continued need to buy new mobiles would drive the costs of device management higher still
-
Although you can test on a sample of e.g. phone battery conditions, it is difficult to maintain perfect control over them as devices degrade at unpredictable speeds
Our overall view on in-house physical device farms is that it is certainly useful to have a small set of physical devices for quick smoke tests and other tests a developer or QA may wish to execute locally; but that as a comprehensive testing solution, physical device farms in-house are just too complex an undertaking to be managed in-house, especially for smaller companies.
Outsourced testing for mobile app testing
Global App Testing is a mobile app testing company which is able to test applications using a distributed “crowd” of over 70,000 testers. This gives us a range of software versions, devices, environments, and geographies, and allows us to test your product in different conditions on your behalf. You can stipulate test conditions; otherwise we’d be happy to consult on the preferred device distribution. This is one great area where the scale of a crowd creates an advantage.
Challenge 3/ Version distribution in mobile app testing
Why is version distribution an issue in mobile app testing?
Unlike web software, users download a specific version of your mobile app and may never update it. That makes version distribution and version control a huge challenge. It can make it difficult to test every version, including some dev versions which haven’t been numbered effectively. You also have to work around app store rules in some instances.
Version control in mobile app testing
Recap | What is version control?
Version control, also known as source control or revision control, is a system that manages and tracks changes in files, particularly source code files, during software development. It allows you to keep a history of changes, compare different versions of the files, and if needed, revert to a previous version. Version control systems also facilitate collaboration among team members by allowing multiple developers to work on the same project without overwriting each other's changes.
Best practices for mobile app testing?
-
For mobile app testing, it's essential to use a version control system that supports the platform (Android, iOS) and language (Java, Kotlin, Swift, Objective-C) used in the project. Git is a popular choice due to its widespread adoption, ease of use, and support for branching and merging.
-
Branching strategy: Implement a branching strategy that accommodates the development and testing of new features, bug fixes, and hotfixes. For mobile applications, a common approach is to use a Gitflow-like branching model, which includes:
-
-
A master branch for stable releases
-
-
-
A develop branch for integrating new features and bug fixes
-
-
-
Feature branches for working on individual features
-
-
-
Hotfix branches for critical bug fixes that need to be released immediately
-
-
Tagging your app releases with version numbers (major, minor, and patch levels) helps keep track of the versions being tested and deployed. This is crucial in mobile app testing, as different versions may be available on various app stores or devices.
-
Binary management: Mobile app testing involves managing numerous binary files, such as APKs and IPAs. Use a binary repository, like JFrog Artifactory or Nexus Repository, to store and manage these binary files alongside your source code. This enables testers to access the required version of the app easily.
-
Platform-specific testing: Mobile app testing may involve different version control requirements for different platforms (Android, iOS). It's crucial to maintain separate branches for each platform, as they may progress at different rates due to device-specific bug fixes or platform-centric features.
-
Continuous Integration (CI): Integrate your version control system with a CI tool, like Jenkins or CircleCI, to automate the build and testing processes for your mobile app. This ensures that the app is always in a testable state and allows you to catch issues early in the development process.
Mobile software testing tools
Here are some well-known tools you can use
1. Global App Testing
The Global App Testing dashboard enables you to send and receive tests in 190 countries to 70,000 testers around the world. We’re the only manual testing execution solution on this list, making us perfect.
2. TestRail
TestRail is a test management tool that helps manage test cases, track test results, and report on testing progress for web and mobile applications.
It integrates with various test automation tools and has a user-friendly interface for creating and organizing test cases. It also offers real-time insights into test results and progress. We’re a partner of TestRail, so we’re huge fans of the system.
3. Appium
Appium is an open-source automation tool for testing native, mobile web, and hybrid applications on iOS and Android platforms. It supports various programming languages, allows testing on real devices, and offers cross-platform testing capabilities.
Appium requires a good amount of setup and configuration, and can be challenging for beginners.
4. Smartbear Zephyr
SmartBear Zephyr is a test management tool that supports test planning, execution, and reporting for web and mobile applications. It offers real-time test management and reporting, integrates with various test automation tools, and allows for flexible test planning and execution.
We’re a partner of Smartbear, so we’re huge fans of the system.
5. Calabash
Calabash is an open-source mobile app testing framework that supports automated acceptance testing of Android and iOS applications. It supports cross-platform testing, allows integration with popular CI/CD tools, and has a large community for support.
6. Robotium
Robotium is an open-source Android testing framework that supports black-box UI testing of native and hybrid applications. It is easy to use, has a simple API for writing test scripts, and supports various test scenarios.
Common mistakes to avoid when testing mobile apps
Failure to define quality standards clearly.
Teams can make the mistake of not establishing clear quality standards at the outset of a project. This can lead to ambiguity and confusion about what is expected, and can result in a lack of focus on quality goals. Spelling out the risk categories and the business imperative to avoid each one will be a worthwhile exercise to ensure you understand what quality target you’re shooting for.
Fumbling process; testing in a wasteful way
The benefit of setting up a test suite or schema is to avoid unnecessary duplication and error. For example, smoke tests are useful at the start of a regression test suite after a certain threshold to ensure that you don’t continue with a test which is doomed to fail. Within regression tests, there’s a decision to make about when to cancel an ongoing test which has already failed; and when to let the entire test suite run through to identify all the bugs and issues in this version.
Getting automated testing wrong
We’ve produced a blog identifying when it’s suitable to automate versus manually conduct your test. Automation is extremely important in order to achieve the speed and quality goal.
Failing to look at quality in your culture
A culture of quality is something that you should be investing in so that you don’t find that you pay over-the-odds to clean up. We’ve written a book about how to encourage quality in your culture.
Not getting set up with Global App testing
If you’ve enjoyed the above article, consider training with Global App Testing to understand how you could get started with Global App Testing today.
Outcomes to calculate in a generic testing process
Talking broadly about testing, we tend to think about a few key outcomes in your testing process.
1/ Fast testing – that could either mean tests which fit with an agile or devops sprint cycle, or it could mean tests with a low operational burden along in terms of time investment or other costs.
2/ Scalable testing – clients want to build a quality framework which will survive rapid growth or peaks and troughs in demand for testing
3/ Broad testing – businesses are looking to ensure their test process covers every device and OS combination
4/ Rigorous or high-quality testing – ensuring that your product is the standard which meets you business goals.
