On March 22, Android users around the globe suddenly saw notifications pop up on their devices saying that apps had stopped running. Critical apps such as Gmail, Google Pay, Amazon, Yahoo and certain banking apps couldn't be opened, creating widespread consumer concerns. Later, Google revealed the cause was a bug residing in the Android System WebView. Some users were able to remediate this issue by manually uninstalling the latest update and waiting for Google to release a fix. While the issue was resolved by relying on affected consumers to manually update, major crashes and painful manual workarounds can leave a lasting negative impression for users and the brand's reputation.
Software bugs are inevitable in code, so engineering teams don't realistically need to aim for 100% error-free software. However, they should have pre-production quality assurance measures in place that act as a safety net for situations like this. These tools provide comprehensive error diagnostics and actionable insights that allow software engineers to prioritize the bugs creating the most damaging user experience. Even giants like Google and Facebook still experience lapses in this process, but it is a critical step in delivering consistent, quality software.
Post-Mortem Evaluation: Breaking Down App Stability Data from the Crash
At the start of the Android app outage, Bugsnag data illustrating app stability showed four times the volume of regular Android errors registered within one day, indicating significant impact across the Android user base. The Webview bug caused approximately 75% of the crashes in the leading Android projects monitored. These projects saw around 40 times more crashes compared to the same period in the previous week. On top of that, the worst-affected projects saw 200 times the number of crashes compared to the same period in the previous week.
Additionally, an estimated 2 million users were impacted across all apps that were monitored. There was also a detected drop in overall application stability by at least 2% in Android applications, with the worst-affected projects seeing a 10% decrease in app stability scores, meaning 1 in 10 Android customers were experiencing a crash.
It's also worth noting that this Android WebView error was caused by a Native Development Kit error (NDK), which can only be detected if your crash reporting supports NDK crash detection, and if it is enabled. App stability monitoring is critical in situations like this, because certain systems don't make you opt-in for NDK monitoring like you do with others. Make sure NDK error detection is available by default.
Best Practices To Protect Your Apps from Similar Outages
Given that it was an operating system component at fault in this scenario, there is not a lot development teams could have done to prevent applications from crashing in this situation. However, there are many other types of serious app outages that can be prevented by implementing best practices and defensive programming. Below are some proactive steps engineering teams can take to protect their applications from similar problems that may impact application stability:
1. Monitor for Stability Issues in Production
This is critical for engineering teams to gain immediate visibility into crashes and spikes in errors. Not only can engineering react quickly to fix issues, but it supports impact analysis which can be used to provide clear guidance to support and customer success teams to handle customer communications with confidence. Configure team notifications and incident management integrations to quickly align the team and deal with business-critical issues.
2. Track Application Freezes
This will give the team visibility into if certain features are the root cause of any ANRs (Application Not Responding) being captured. You can track application freezes by using the stack trace to see if the line of code that was running when the application froze and set off the ANR. Stack trace information identifies where in the program the error occurs so that it can be fixed.
3. A/B Test New Features
This will help teams understand how certain features are impacting application stability before releasing them to production. You should also always phase the rollouts and test features with a small group of users before releasing to your entire user base.
The Key Takeaway
Because consumers rely heavily on mobile apps to navigate day-to-day life, application stability is absolutely critical, especially in today's relentlessly competitive environment. Difficult-to-prevent system errors like the Android Systems Webview crash highlight the importance of minimizing preventable errors with defensive programming and better handling of malformed data.
The silver lining of outages like this is that it draws attention to the dire need for good software design and process. It surfaces where software engineering teams need to introduce new best practices or where to to fine-tune existing ones.
The Latest
The mobile app industry continues to grow in size, complexity, and competition. Also not slowing down? Consumer expectations are rising exponentially along with the use of mobile apps. To meet these expectations, mobile teams need to take a comprehensive, holistic approach to their app experience ...
Users have become digital hoarders, saving everything they handle, including outdated reports, duplicate files and irrelevant documents that make it difficult to find critical information, slowing down systems and productivity. In digital terms, they have simply shoved the mess off their desks and into the virtual storage bins ...
Today we could be witnessing the dawn of a new age in software development, transformed by Artificial Intelligence (AI). But is AI a gateway or a precipice? Is AI in software development transformative, just the latest helpful tool, or a bunch of hype? To help with this assessment, DEVOPSdigest invited experts across the industry to comment on how AI can support the SDLC. In this epic multi-part series to be posted over the next several weeks, DEVOPSdigest will explore the advantages and disadvantages; the current state of maturity and adoption; and how AI will impact the processes, the developers, and the future of software development ...
Half of all employees are using Shadow AI (i.e. non-company issued AI tools), according to a new report by Software AG ...
On their digital transformation journey, companies are migrating more workloads to the cloud, which can incur higher costs during the process due to the higher volume of cloud resources needed ... Here are four critical components of a cloud governance framework that can help keep cloud costs under control ...
Operational resilience is an organization's ability to predict, respond to, and prevent unplanned work to drive reliable customer experiences and protect revenue. This doesn't just apply to downtime; it also covers service degradation due to latency or other factors. But make no mistake — when things go sideways, the bottom line and the customer are impacted ...
Organizations continue to struggle to generate business value with AI. Despite increased investments in AI, only 34% of AI professionals feel fully equipped with the tools necessary to meet their organization's AI goals, according to The Unmet AI Needs Surveywas conducted by DataRobot ...
High-business-impact outages are costly, and a fast MTTx (mean-time-to-detect (MTTD) and mean-time-to-resolve (MTTR)) is crucial, with 62% of businesses reporting a loss of at least $1 million per hour of downtime ...
Organizations recognize the benefits of generative AI (GenAI) yet need help to implement the infrastructure necessary to deploy it, according to The Future of AI in IT Operations: Benefits and Challenges, a new report commissioned by ScienceLogic ...
Splunk's latest research reveals that companies embracing observability aren't just keeping up, they're pulling ahead. Whether it's unlocking advantages across their digital infrastructure, achieving deeper understanding of their IT environments or uncovering faster insights, organizations are slashing through resolution times like never before ...