Testing in production, or 'production testing,' means evaluating software in its live environment. This practice was once met with skepticism but increasingly recognized for its value. As the software industry evolves rapidly, perceptions about testing in production are changing. What was once frowned upon is now becoming an endorsed practice in many scenarios. Despite this shift, the lingering doubts about its effectiveness and the fear of potential backlash remain for many. If you are skeptical about the merits of testing in production, this post will help you address your concerns and shed some light on it.
Testing in production refers to the practice within software development where new code changes are evaluated directly with live user traffic instead of being tested in a controlled staging environment. This approach is intended not to replace other functional and non-functional testing types but to enhance and complement them. Adopting testing in production often hinges on risk management considerations for the product's launch. While testing in production can offer valuable insights from real-world usage and contribute to the application's improvement, it necessitates careful execution to mitigate potential adverse effects on users.
The adoption of testing in production comes with its set of challenges:
At Global App Testing, we use a crowd of professional testers from around the world to provide real-time feedback and insights, facilitating continuous testing and improvement of the product before, during, and after release. This continuous evaluation provides immediate insights into the risks associated with new code submissions, enhancing the ability to deliver timely and error-free software. Automated production testing, covering web, mobile, desktop, and APIs, can significantly increase both efficiency and effectiveness.
Why would you consider testing in production? In certain situations, you might find yourself without alternative options. Creating a staged testing environment might be too complex or costly, or you may need to capture genuine usage data, necessitating a direct approach to the live environment. Certain tests, particularly those assessing an application's scalability, are more accurately conducted within the production environment. Load testing, for instance, is most effective when performed on the actual system used by your users.
Ultimately, the intricacies of software development play a pivotal role in this decision. Despite having a robust QA strategy, employing the latest best practices, and utilizing advanced tools, some issues are bound to slip into the live environment. Thus, testing or monitoring your application in production serves as a crucial safeguard, acting as a final barrier to identifying and addressing these unforeseen bugs.
How to effectively conduct testing in production? Here are the key strategies that harness the benefits of production testing:
In the production environment, choosing the right testing tools is crucial, covering a broad spectrum to meet various testing needs. Essential tools include Application Performance Monitoring (APM) tools like New Relic and Datadog for real-time performance insights and user analytics platforms like Google Analytics to understand user interactions and identify potential UX issues.
Feature flagging services like LaunchDarkly enable the safe testing of new features by allowing features to be toggled without a code deployment. Automated testing tools, such as Selenium for the web, Appium for mobile applications, and JMeter and LoadRunner for load testing, are critical for efficiently identifying and resolving issues. Platforms like JIRA and TestRail support manual testing, while Real User Monitoring (RUM) tools like Raygun and LogRocket provide direct insights into user experiences and errors. When used effectively, this toolkit significantly improves application quality and reliability, blending automated and manual testing methods for a seamless user experience.
Global App Testing further enhances this approach, supporting both automated and manual testing with the flexibility to meet project-specific requirements, from exploratory tests to large-scale test execution, ensuring high-quality product delivery.
So, should you test in a production environment? Don't just trust us – we consulted three testing experts actively utilizing production testing, gathering insights on its advantages and potential pitfalls.
So, what are the benefits? Well, it turns out that there are a number of them!
There is no margin for error in terms of production environments. You are testing in the real-life environment of the user, so you can effectively test their user experience.
Testers are skilled at evaluating software for issues. Still, with their unpredictable usage patterns, customers are likely to encounter or cause unforeseen problems in your app. It's wise to anticipate these scenarios and be ready to address them.
There are definitely some elements of testing that benefit from the production environment. Ajeet Dhaliwal highlighted the importance of testing in production when looking at payment functions. You cannot precisely match the payment process, so conducting the test in the production environment can be beneficial.
However, testing in production does not always work. It can have some real drawbacks if it isn't used correctly or if it is used unnecessarily. As explained by software engineer Cindy Sridharan in her extremely thorough article on the subject:
"When done poorly or haphazardly, "testing in production" does, in fact, very much live up to this reputation. Testing in production is by no means a substitute to pre-production testing nor is it, by any stretch, easy."
What Cindy means is that testing in production is very difficult to get right. Assuming this is an easy fix would not be correct. Here are the drawbacks our experts highlighted:
Performance testing in production can negatively affect all users, including customers, due to inherent limitations and legal constraints on using real data. In fact, some QA experts and engineers hear the term and run a mile. That's because there are a lot of scenarios where testing in production is a big 'no.'
Directly testing in production risks exposing users to bugs, potentially leading to dissatisfaction or losing potential customers. Your customer may encounter issues, churn, or simply write off your software as buggy. That's why companies who test in production need to have the ability to take risks and not suffer hugely if they go wrong.
Poorly executed production testing can introduce severe issues, highlighting the need for a skilled and cautious approach. As Cindy Sridharan says, testing in production is a cautious art. You have to have a highly skilled team willing to take risks. Otherwise, you could end up with a real mess you did not want on your hands.
Testing in production is a tricky subject matter. While it has benefits, the QA community takes a cautious approach - tread carefully, they say! But that doesn't mean you should run a mile from the prospect of testing in production. In fact, it can be a handy tool in your testing toolkit. Take a risk assessment of how your software could deal with the potential drawbacks, and see how it levels up regarding the benefits. Testing in production could be right for you. And, if you want to learn more about how Global App Testing can help you during any phase of your product development, sign up and schedule a call today!
Automated testing vs Manual testing - What's the difference?
5 Best practices for testing web applications
Quality Assurance vs Quality Control