The Ad Hoc
Incident Response Playbook

Laying the foundation for a flexible, blameless incident response practice.

Illustration of three layers of paper with icons of a computer, crane, clipboard, and trash can emanating from the layers.
Written by:

Eric Isbell, Jason Williams, John French, Deirdre Holub, Bill Mill, Elissa Frankle Olinsky, and Craig Butler

Don’t Panic.

Software breaks all the time. Sometimes it breaks and is still usable; other times it breaks and systems grind to a halt. Since it is a certainty that software will fail, it’s essential your team is prepared and has a plan in place to respond.

This playbook is for government agencies and other civic organizations looking to develop and mature their practical understanding of incident response from a reactionary mindset to a more flexible, cross-functional, and blameless approach.

Follow our plays to better equip your organization and teams with practices that will lead to calmer, more effective incident response and ultimately more secure digital services.

A flexible approach

At Ad Hoc, we believe that there is no “one size fits all” approach to incident response. A procedure that works for one organization may be wrong for another.

It is not our goal in this playbook to prescribe certain tactical tools or solutions, but rather to give you the strategies you can use to figure out what will be best for your situation. Focus on determining guiding principles based on your specific organizational needs, structure, and team resources. Guiding principles can then lead to flexible and enduring answers.

A prepared mindset

For many teams, responding to security incidents feels frantic, chaotic, and stressful. Some teams might even let days or weeks go by before realizing there is a problem, which only complicates the response.

If we think of incident response like healthcare, preventative care and testing are equally as important as the emergency room. If we never invest time, effort, or energy before a medical emergency, then we run the risk of only ever reacting to emergencies instead of having the chance of preventing them.

At Ad Hoc, we believe in prevention and emergency response. Software incidents can be incredibly costly, damaging, and harmful to organizations and their users. The more your team can understand its systems, plan for various types of incidents, and practice responding to them, the easier it will be to respond to real-world issues. And the less impact issues are likely to have.

A cross-functional responsibility

At Ad Hoc, we take a cross-functional approach to building software, products, and services. That means that our entire team works together to determine how best to approach problems. We approach security (and therefore incident response) in the same way we approach accessibility or quality — as the responsibility of the entire team.

Every team member has a valuable and unique perspective that can contribute to creating resilient systems and remediating issues as they arise. Designers and researchers might have key insights into user behavior that can contribute to discovering the root cause of an incident. Product managers and writers may be able to coordinate communications with end users and stakeholders. And engineers can help update code and isolate activity.

Making incident response a cross-functional effort maximizes a team’s resources. It allows teams to prevent or respond to incidents more efficiently while also improving stakeholder and user-facing communication during an incident. Cross-functional responses improve outcomes without placing all the burden on software engineers.

Survey the landscape of your service

“He who defends everything defends nothing.” - Frederick the Great

In this play, you will assemble key information about the landscape of your product or service, its security needs, and what is important to your organization so that you can make good decisions about procedures in later steps.

In many ways, this play is about understanding potential risks so that they can be prevented in the future. It is similar to understanding flood zones and historical weather data when buying or building a home. There may very well be risks or tradeoffs, but the goal of this play is to document and understand them so that you can start to make more informed decisions.

Start by observing and documenting the digital and physical aspects of your service or product.

Every meaningful incident response process must begin with building the team’s shared understanding of the details of the system, product, or application they’re supporting. That knowledge must also include what the biggest vulnerabilities are likely to be. If this process feels overwhelming, it might help to start by asking the engineers to make a simple diagram of how the system works.

Work with your team and organization to:

  • Document your service, product, or system
  • Define what would constitute an incident in this context
  • Determine what types of incidents you are likely to encounter and what data might be sensitive or targeted by bad actors
    • For example, availability incidents, security incidents, data leaks, etc.

Try to define expectations for the expected operation of your system. This often takes the form of a Service Level Agreement or some other documented policy. These policies help you know when system performance dips below expected levels and might trigger an availability incident or what expected behavior should look like to users.

Then, define what is important to your organization in terms of an incident response process.

A service that lets users browse product documentation will need a very different incident response policy than a service that lets users schedule physicians appointments. A small team or organization will have fewer resources available than a larger one but may also face fewer numbers of risks. So, rather than starting with someone else’s answers, define what is important to your organization, so that you can later find the best possible procedure for your situation.

Again, Ad Hoc recommends beginning with guiding principles. What is most important to your organization? Once you understand these principles, you can more easily prioritize and write procedures. In the case of incident response, a useful way to determine your organization’s values is in terms of risks and potential damages in the case of an incident. In the following example, notice how the questions about data and storage lead naturally to questions about risk.

  • What type(s) of data does your application or team utilize?
  • How/where is it stored?
  • What risk does this data present?
  • What data or information is most likely to be targeted by outside actors?
  • What data would be most damaging if it was accidentally exposed?
  • What kind of breaches or exposures would put you or your stakeholders at risk of having to testify before Congress?

Prioritizing the most vulnerable systems or data can help your team develop processes that achieve the highest value for the least effort.

Determine what incident response processes already exist, if any.

In many cases, your team may not be starting completely from scratch, so it might be helpful to take some time to understand what processes already exist.

  • Do an audit of any existing incident response documentation, the outcomes of past incidents, or any relevant process documentation.
  • Identify the biggest gaps in those processes based on what your team decided was the most valuable or important.

If you’re starting from scratch developing an incident response process, don’t try to do everything all at once. Identify what systems or data are most important to your team and prioritize from there.

Your team should consider running a threat modeling exercise where you try to anticipate what type of threats your system might attract.

Build a clear list of all relevant stakeholders and users.

A key part of understanding your system and preparing an incident response plan is documenting who might be impacted by incidents. This includes users who might rely on the functionality of the software to do things like make a doctor’s appointment or file an insurance claim. This list should also include stakeholders who are responsible for the successful operation of that software for users.

Making this list ahead of time will help in the later stages of developing an incident communication plan.

  • Document key stakeholders and try to include contact information in your documentation.
    • Consider checking with stakeholders during this process to verify their preferred method of communication during an incident.
  • Document different users — both internal and external — who might be impacted during different types of incidents.
    • Check to see if you already have this information documented by a UX designer or researcher.
Return to playlist ⇧

Download the full playbook

Now that you have a deeper understanding of why you should have an incident reponse plan and how surveying the landscape of your service will start you on the right track, continue reading about the other plays that’ll help you prepare for any system emergency.

  • Play 2: Be prepared, have a plan
  • Play 3: Define the team roles
  • Play 4: Use it or lose it
  • Play 5: Applying your process to a real incident
  • Play 6: No matter what, take care of each other

By providing your contact information, you’ll gain access to Ad Hoc’s full playbook of strategies that your team should implement before, during, and after an incident. You’ll also be able to check out a recording of our webinar with several authors of the playbook and hear their thoughts on the importance of an incident response plan and how to best prepare. With these resources as your guide, you can develop a comprehensive plan that you can use to respond faster and more effectively when an incident hits.

We look forward to talking with you about playbook and would be happy to answer any questions you might have. Reach us at hello@adhoc.team.

* = required


All fields are required. Ad Hoc respects your right to privacy. By submitting this form, you consent to receive emails from a representative of Ad Hoc, which may include the use of automated technology.