The Amazon S3 Outage - When the Internet's Hard Drive Fails …
March 06, 2017

Denis Goodwin
SmartBear

Share this

Last week, SmartBear observed a sudden and protracted 5X increase in web page timeout errors associated with the failure of Amazon's S3 cloud-based storage service. Looking a bit more closely at our data, we dug up a few more interesting angles on the impact of the failure.

Event Timeline

The issue hit suddenly – we saw an immediate spike in errors at 12:35 p.m. EST, and by 12:45 p.m. EST, error rates were 5X normal. For some specific types of timeout errors, the spike was more than 10X normal. At 3:30 p.m. EST, the error rate began dropping and by 3:50 p.m. EST, rates had returned to normal.


Web vs. API

The issue hit web pages hard, while API monitors were not noticeably impacted by this outage. Web pages and web apps often utilize content storage hosted by cloud services such as Amazon S3.

Common failure scenarios on Tuesday included page elements failing to load, which could cause either the whole web page to time out or specific content on a page might not render. Depending on the design of a given page, this partial content failure could be relatively minor or it could render a critical web journey non-functional. File uploads and downloads that rely on S3 storage endpoints were particularly hard hit.

In order to get a complete picture of application health, it's necessary to monitor your real user's journey through the application. The monitored user journeys that depended heavily on content hosted in S3 failed. Those that didn't have that dependency continued functioning. I personally experienced this with Slack – I was able to use the app, however files could not be uploaded presumably because these files are stored by Slack using S3 as the storage mechanism.

While far less pronounced than the spike in errors, some response time degradation was observed in API monitors that continued running successfully. Given that the issue affected Amazon's storage services rather than their hosting services for applications, this makes sense.

Geographic Impact

The issue was more acutely felt in the United States, but we observed impacts all over the globe. The spike in page errors was seen on websites dependent on Amazon S3, many of which are U.S.-hosted websites that are likely monitored from U.S. locations. Unsurprisingly, error counts spiked by as much as 25X in some U.S. monitoring locations. While not as significant as the U.S. locations, timeout and page error increases were also observed from Canada, Europe and Asia.

Takeaways

Much of the web is built on the backs of cloud providers. Most of the time, these cloud services provide a great user experience. Amazon will learn from the root cause of this issue and likely emerge from this outage more resilient than ever. It's impossible to control all aspects of these shared services – but here are three steps to take that are in your control.

1. Identify your business critical applications

2. Proactively monitor user journeys on these applications

3. Don't rely on your third party provider to tell you when it is down

It is key to utilize independent monitoring services to ensure your applications are up, functioning correctly and fast. Furthermore, missing content can be catastrophic or merely inconvenient to a critical user journey – it's important that your monitoring tool can be configured to know the difference.

Denis Goodwin is Director of Product Management, AlertSite, SmartBear Software.

Share this

The Latest

July 26, 2017

The retail industry is highly competitive, and as retailers move online and into apps, tech factors play a deciding role in brand differentiation. According to a recent QualiTest survey, a lack of proper software testing — meaning glitches and bugs during the shopping experience — is one of the most critical factors in affecting consumer behavior and long-term business ...

July 25, 2017

Consumers aren't patient, and they are only one back-button click from Google search results and competitors' websites. A one-second delay can bump the bounce rate by almost 50 percent on mobile, and a two-second delay more than doubles it ...

July 24, 2017

Optimizing online web performance is critical to keep and convert customers and achieve success for the holidays and the entire retail year. Recent research from Akamai indicates that website slowdowns as small as 100 milliseconds can significantly impact revenues ...

July 21, 2017

Public sector organizations undergoing digital transformation are losing confidence in IT Operations' ability to manage the influx of new technologies and evolving expectations, according to the 2017 Splunk Public Sector IT Operations Survey ...

July 20, 2017

It's no surprise that web application quality is incredibly important for businesses; 99 percent of those surveyed by Sencha are in agreement. But despite technological advances in testing, including automation, problems with web application quality remain an issue for most businesses ...

July 19, 2017

Market hype and growing interest in artificial intelligence (AI) are pushing established software vendors to introduce AI into their product strategy, creating considerable confusion in the process, according to Gartner. Analysts predict that by 2020, AI technologies will be virtually pervasive in almost every new software product and service ...

July 18, 2017

Organizations are encountering user, revenue or customer-impacting digital performance problems once every five days, according a new study by Dynatrace. Furthermore, the study reveals that individuals are losing a quarter of their working lives battling to address these problems ...

July 17, 2017
Mobile devices account for more than 60 percent of all digital minutes in all 9 markets profiled in comScore's report: Mobile’s Hierarchy of Needs ...
July 14, 2017

Cloud adoption is still the most vexing factor in increased network complexity, ahead of the internet of things (IoT), software-defined networking (SDN), and network functions virtualization (NFV), according to a new survey conducted by Kentik ...

July 13, 2017

Gigabit speeds and new technologies are driving new capabilities and even more opportunities to innovate and differentiate. Faster compute, new applications and more storage are all working together to enable greater efficiency and greater power. Yet with opportunity comes complexity ...