Incident Playbook: How to Plan for and Respond to Outages
January 10, 2017

Scott Klein
Atlassian

Share this

There's nothing like a major web outage to remind us how much our applications rely on other web services and technologies to function. In late October of last year, a Distributed Denial of Service (DDoS) attack on Dyn, one of the largest Domain Name Service (DNS) providers on the internet, disrupted service for consumer and business applications across the web. This attack shed light on the delicate interdependent nature of the web as productivity and uptime across the world was effected, even for organizations that didn't directly use Dyn as a DNS provider.

For an IT team, it can be a rude awakening when one major service provider's downtime triggers outages of technologies across the web, eventually hitting their application in its destructive path. As we saw with this attack, the disruption of one service can have a domino effect on many others. From an outsiders' perspective, it can make the internet seem like a wobbly house of cards. While you and your IT team might not see it that way, consider how an outage of this magnitude is perceived by your business colleagues and customers.

To alleviate these concerns, it's worth developing an incident response plan so that when another outage inevitably occurs, you are able to show confidence and transparency while managing the incident. Incident response plans can be developed in three steps to ensure you and your team are prepared to not only respond to an incident, but also maintain clear communications with stakeholders to quell panic.

1. Identify and quantify incidents

There doesn't have to be a major DDoS attack for a response plan to come in handy (think bugs in production, broken builds, login issues, etc.). Your team needs to be able to quickly identify and quantify the incident in order to appropriately respond.

If your team can answer these questions to define what constitutes an incident, they will be better equipped to respond:

■ How many customers are impacted?

■ How long does the incident need to last?

■ How degraded is the service and how do you determine severity level?

2. Create an incident communication plan

A well-built incident response plan is largely about effective communication, from who should be included in remediation conversations, to what is communicated to end users.

Consider the following when creating an incident response plan:

■ Identify the incident commander. This person is already at work or on call and available to provide direction during an incident.

■ Establish internal channels for alerting and communicating with teams reacting to the incident. We recommend creating a "war room," or dedicated chat room, to use for the duration of the incident and cut out unnecessary noise.

■ Know how you will communicate with end users. Whichever solution you use, be sure it's reliable and available when you need it. This usually means something that isn't hosted outside your core tech stack, because you don't want your communication forum having downtime when you need it most.

■ Build templates and preset language for common incidents. The last thing you need to worry about during an incident is how to phrase an alert during an already high-tension situation. While every incident is unique, most of them we find fall into a few broad categories: total outage, regional outage, delayed response times, to name a few. By deciding on some stock language for these templates and saving it ahead of time, you'll have more time to dedicate to the incident when it actually happens. This also accelerates the time from detection to communication.
Plan to send periodic updates during the incident to keep the team and stakeholders informed.

3. Communicate with transparency

During an incident, streamlined, transparent communications can free up your team to focus on fixing the problem while ensuring customers have the most up-to-date, accurate information. Even if your outage is the result of a third-party service provider, it's your responsibility to make sure your customers have the right information. While you don't want to play a blame game during an incident, there's nothing wrong with letting your users know that you're waiting on a fix from another provider. This helps your users understand that there isn't a lot you can do.

If the service provider having the outage is communicating with users somewhere, it's smart to direct your users there. During the Dyn outage, we saw a lot of Dyn's customers direct their users to Dyn's service page. This helped clear up the confusion around the interdependencies of these structures and helped users get the right information in the quickest possible way.

Postmortem and evolve

The lifecycle of an incident doesn't end when your application is up and running again, and neither should the actions that your team takes. Include these steps into your response plan to ensure you're covering all of your bases - both now and in the future.

■ After the dust has settled, draft an incident postmortem to explain what happened and what precautions are being implemented to prevent a repeat incident. It's always good to write a postmortem, especially for major outages. This is your opportunity to tell your customers what went wrong and what you're doing to avoid the incident in the future. A good postmortem includes an apology for any inconvenience your service outage may have caused and acceptance of responsibility. Finger pointing (even if it genuinely was another party at fault) is never a good look. Finish off your postmortem by explaining your remediation plan.

■ Conduct a post-incident review with your team to evaluate the processes and strategy enacted during the incident and how those can be improved to better handle future incidents.

There's no telling when the next incident will occur, what the scale will be or what systems will be affected, but if your team is prepared with a incident response plan, they can act swiftly to respond to the issue without the frenzy and panic that typically ensues. And when the next major domino drops and the rest of the world is furiously clicking "reload" and overloading customer service lines, your customers and stakeholders will receive clear, transparent communications the entire time.

Scott Klein is StatusPage Product Manager at Atlassian.

Share this

The Latest

January 20, 2017

Traditionally, Application Performance Management (APM) is usually associated with solutions that instrument application code. There are two fundamental limitations with such associations. If instrumenting the code is what APM is all about, then APM is applicable only to homegrown applications for which access to code is available ...

January 19, 2017

The correlation between mobile app crashes and increasing churn rates (or declining user retention) has long been suspected. In the report, titled Crash and Churn, Apteligent set out to understand the impact of per user crash rate on churn ...

January 18, 2017

In Fall 2016, Paessler AG surveyed 650 system administrators from 49 countries to get a "state of the SysAdmin" and find out how their jobs are changing, how they spend their time, and what their priorities are. The survey responses led to some interesting findings – namely, that when it comes to today's SysAdmins, things are not as they seem. Here are some of the key findings that illustrate the gap between perception and reality ...

January 17, 2017

Choosing an application performance monitoring (APM) solution can be a daunting task. A quick Google search will show popular products, but there's also a long list of less-well-known open source products available, too. So how do you choose the right solution? ...

January 13, 2017

Digital transformation is a key initiative for enterprises that want to reach new customers and offer greater value via technology. Changing user expectations, new modes of engagement and the need to improve responsiveness are the main factors driving companies to update outdated processes and develop new applications as part of a digital transformation strategy. But in order to deliver on the promise of digital transformation, organizations must also modernize their IT infrastructure to support speed, scale and change ...

January 12, 2017

Digital transformation is evolving the enterprise to one in which high performance applications are now the norm as organizations use video, graphics and other information intensive multimedia to populate these new channels of engagement. Digital technologies, and high performance applications, create further pressure on IT staffs which are grappling with PCs that are past their optimum performance. As a result, IT is looking at alternatives to swapping out PCs and investing in more costly equipment that will inevitably have an expiration date. One solution is to build on virtualization solutions that incorporate high-performance thin clients ...

January 11, 2017

If your business depends on mission-critical web or legacy applications, then monitoring how your end users interact with your applications is critical. Most monitoring solutions try to infer the end-user experience based on resource utilization. However, resource utilization cannot provide meaningful results on how the end-user is experiencing an interaction with an application. The true measurement of end-user experience is availability and response time of the application, end-to-end and hop-by-hop ...

January 10, 2017

There's nothing like a major web outage to remind us how much our applications rely on other web services and technologies to function. In late October of last year, a Distributed Denial of Service (DDoS) attack on Dyn, one of the largest Domain Name Service (DNS) providers on the internet, disrupted service for consumer and business applications across the web. This attack shed light on the delicate interdependent nature of the web as productivity and uptime across the world was effected ...

January 09, 2017

As an IT professional, I'm used to words that mean different things to different people. For example, "log monitoring" could mean anything from simple text files to logfile aggregation systems. "Uptime" is also notoriously hard to nail down. Heck, even the word "monitoring" itself can be obscure. This is why I'm not surprised that application performance monitoring (APM) can mean so many different things depending on the context ...

January 06, 2017

Big data continues to be the fastest-growing segment of the information management software market. New findings released by Ovum estimate that the big data market will grow from $1.7bn in 2016 to $9.4bn by 2020, comprising 10% of the overall market for information management tooling ...