A different take on security incident response, pitfalls of IR as a profession, and a path for advancing the field forward
Looking at the unexplored aspects of incident response, including the human side, the impact IR has on the responders doing it, and some of the ways we as an industry can evolve to handle incidents
There are plenty of discussions about incident response (IR) in cybersecurity. That is not surprising: since every organization will eventually experience a security event, we need to make sure we are ready to respond. Sharing knowledge about best practices in IR, publishing post-mortems about past incidents, reviewing tools of the trade, and rehearsing the actions during tabletop exercises are some of the many ways to achieve that readiness.
In today’s deep dive, I am collaborating with my friend and experienced incident responder Nathan Case to discuss the unexplored aspects of incident response, including the human side, the impact IR has on the responders doing it, and some of the ways we as an industry can evolve to handle incidents.
Welcome to Venture in Security! Before we begin, do me a favor and make sure you hit the “Subscribe” button. Subscriptions let me know that you care and keep me motivated to write more. Thanks folks!
Lastly, over 3,300 copies of my best selling book “Cyber for Builders: The Essential Guide to Building a Cybersecurity Startup” have been distributed to the readers so far.
Four steps of incident response
At the fundamental level, there are four steps for handling any incident:
Avoiding an incident. This involves doing the work to reduce the probability of the incident occurring - strengthening the organization’s controls, reducing the attack surface, training employees about being vigilant when doing their work and running a security program that considers the business's needs.
Preparing for the incident. Although the idea that every company will eventually suffer a security incident may sound pessimistic, deterministic, and even nihilistic, it is the reality of our complex and interconnected world. The question isn’t whether or not we will get breached, but how will we react when it inevitably happens. Preparing for an incident can involve putting in controls so that when an incident occurs, sensitive data doesn’t get leaked, customers don’t get impacted, and business operations don’t get shut down for hours, days, or even weeks.
Dealing with the incident. This step is triggered when an organization learns it has suffered a security event. When it happens, the IR plan kicks in, and security professionals work to remove bad actors from their organization’s environment.
Recovering from the incident is crucial for restoring normal operations and returning the business to its regular mode. Surprisingly, this phase of incident response is often the least practiced. It's perplexing that many organizations neglect this step, likely due to the perceived effort or cost involved. However, consistent practice is essential. If this process proves too challenging or resource-intensive, we must explore new, more efficient recovery methods. In any incident, this aspect of Incident Response will always be required, and it is often found lacking.
This process is a great oversimplification, but it does get one point across that IR isn’t about the actions taken during the incident; it is about what happens long before and immediately after the security event. NIST has four similar topics, but this lacks the reality of “Avoiding the Incident” and includes those actions in “Preparation”. Breaking this out here points out the differences between these two actions, avoidance, and acceptance, and helps to visualize the reality of these topics for later.
Each of the four steps is complex. However, we argue that of all four, it’s the second, namely preparing for the incident, and the fourth, recovering from the incident, are the topics that we have all been struggling the most with.
As a species, we are notoriously bad at preparing for things, good or bad. IR readiness and business continuity are so hard because, on the psychological level, we don’t want to admit that we need to be ready to respond. We are unwilling to accept that we are inevitably going to fail. You don’t have to look at security to see the evidence that this is true. Take end-of-life planning as an example: we know that each of us is going to die, and yet the overwhelming majority of us don’t think about end-of-life planning as if somehow we can escape death. The number of people who die intestate (i.e., without leaving a will), who don’t carry life insurance despite being able to afford it easily, and who don’t have advanced care directives is staggering. Instead of preparing for the inevitable, we hide our heads like ostriches, thinking that somehow we will be the first to avoid it.
Another area where we often need to catch up is recovering from security breaches. There are two main reasons for this. First, businesses are typically over-eager to resume revenue generation as quickly as possible, so once operations are back on track, they consider the incident resolved. Consequently, everything that follows—such as post-mortems, investigations, follow-ups, and reporting—is often viewed as mere formalities, disconnected from the core incident response. Secondly, creating a comprehensive recovery plan requires time, testing, and resources. Due to the complexity and the investment required, this task gets pushed to the backlog surprisingly quickly. In reality, however, many other aspects require attention after an incident, including the well-being and effectiveness of the responders themselves.
The human side of incident response
Every profession is multifaceted and complex, and incident response is not an exception. As humans, we tend to do a great job summarizing and sharing knowledge about tactical aspects of this complexity, such as how to get into cybersecurity IR, what skills people need to possess to be successful, what processes need to be in place for the response to be efficient, and what tools, be they vendor products or open source solutions are the best for the job. Since cybersecurity evolved from communities of hacking and open source, there is a lot of sharing and exchange of learnings - on online forums and Discord channels, at industry events, and in communities of peers.
Not all aspects of incident response get an equal amount of discussion. In particular, there is one that seems to be swept under the rug, almost intentionally forgotten, ignored, hidden away. We are talking about the impact IR has on people who work through these incidents. Akin to a kid hiding under the sheets because there is a monster under his bed, we don’t want to talk about the human side of IR. It is a scary topic to discuss. And yet, we must because this impact goes beyond being “overworked and overstretched.”
Impact of incident response on the psyche of full-time IR professionals
Not everyone suffers equally
Before we discuss the impact of incident response on the psyche of IR professionals, let us clarify that we aren’t talking about people on security teams who are expected to handle incident response once or twice per year. Though many of these problems will likely still impact their lives, they won’t be as severe as for those who do IR for a living.
God Syndrome
People in charge of large-scale events often suffer from the so-called God syndrome - the feeling that they are almighty, invincible, and capable of fully predicting what others will think, do, or say.
This isn’t unique to cybersecurity incident response. Anyone who handles emergency incidents, from doctors to rescue divers, soldiers, and firefighters, everyone who gets to feel in control of other people’s lives for such a long time, eventually becomes a victim of this syndrome. Simply put, most humans aren’t set up emotionally to constantly deal with large-scale incidents without becoming desensitized and damaging their ability to empathize, collaborate, and see others as equals.
Cybersecurity gives security practitioners a sense of superiority because they see things nobody else does, know who does what, and can control other people’s computers. They take pride in “owning” things. With time, they get used to this perception of control, and many develop a feeling of superiority. Being an incident responder builds into the pride that eats people alive. The fact that hundreds of people are listening to every word the person running IR says for many weeks is intoxicating. Suddenly, security practitioners become more than before - they feel different, better, and above everyone else.
The causes and effects of God syndrome have been studied by hundreds of researchers in different areas of emergency response. There are articles and comprehensive books about this phenomenon in medicine, air traffic control, and firefighting to name some. Unfortunately, there is no research about the problem in security and cybersecurity incident response. This gap in research and the fact that these problems aren’t being discussed has a real impact on human lives. How do you do IR and remain humble? How do you get a sense of achievement when all of your successes are under NDAs, and you’re not allowed to talk about anything you feel proud of? How do you go home after an incident and not feel like you’re a million feet tall? How do you raise a kid that cares about their peers? How do you not accidentally squish people all over without realizing that you’re doing it unintentionally? All these are real questions, and we are unlikely to answer them without putting effort into first researching the problems.
One of the simplest yet most crucial pieces of advice for managing the symptoms associated with a God complex—especially during periods of high stress or mania—is to ensure you get enough sleep. Sleep deprivation can exacerbate feelings of invincibility, grandiosity, and impaired judgment, which are often linked to both a manic state and a God complex.
Why Sleep Matters
Lack of sleep impairs cognitive function, emotional regulation, and decision-making. For someone prone to grandiose thoughts or a God complex, these impairments can heighten the risk of making poor decisions, overestimating capabilities, or disregarding the input of others. Adequate sleep helps maintain a more balanced perspective and better self-awareness, which are key to mitigating the effects of a God complex.
Practical Implementation
Establishing a structured on-call schedule for 6 to 8 hours of rest is a practical step. While this might be challenging in environments with small teams or during high-intensity incidents, it’s essential to prioritize breaks between shifts. These breaks are not just about physical rest—they provide mental and emotional decompression, allowing team members to regain clarity and perspective.
Example Approach
Rotating Shifts: Implement rotating shifts in which each team member is on call for a set period (e.g., 6-8 hours), followed by a mandatory rest period.
Scheduled Breaks: Scheduled breaks should be enforced even within a shift to allow team members to rest and recharge. This can help prevent burnout and maintain high performance levels.
Buddy System: Pair up team members to monitor each other's need for breaks and encourage rest when signs of fatigue or stress appear.
By integrating structured rest into the workflow, not only do you combat the physical exhaustion that can lead to a God complex, but you also foster a healthier, more sustainable work environment.
Post-traumatic stress disorder (PTSD)
Cybersecurity incident response can lead to post-traumatic stress disorder (PTSD), as the intense and high-stakes nature of the work can have profound psychological effects. It's common for those who have experienced traumatic incidents to mentally replay them, attempting to follow a path that no longer exists, which can exacerbate their stress and anxiety.
One way people might try to avoid PTSD is by becoming dispassionate about their work, treating incident response as just another job, and distancing themselves from the consequences of their actions. However, this approach is often at odds with the very nature of IR. Incident response is a field driven by a passion for making a difference, protecting organizations, and keeping the world safe. This passion draws many professionals to cybersecurity in the first place, but it also makes them more vulnerable to the emotional toll of the job.
For many IR professionals, the stakes are incredibly high—any mistake or delay could lead to catastrophic outcomes for the organization. This immense pressure creates a pervasive sense of fear and anxiety, particularly after dealing with severe incidents. The adrenaline rush that comes with responding to an incident can mask the immediate emotional impact, but the long-term effects often linger. Feelings of guilt, hyper-vigilance, and even nightmares are not uncommon among those who have been through intense IR situations.
Unfortunately, many organizations fail to acknowledge or address these mental health challenges. IR teams are often expected to be on call 24/7, ready to respond at a moment's notice, which can prevent them from fully recovering from the psychological strain of previous incidents. The focus on technical recovery frequently overshadows the need for emotional and psychological support, leaving responders without the care they need.
Organizations must recognize that effective incident response includes caring for the mental well-being of their IR teams. Providing access to mental health resources, fostering open discussions about stress and trauma, and ensuring that there is time for recovery between incidents can help reduce the risk of PTSD. A comprehensive IR plan should encompass the technical and operational aspects of recovery as well as the human ones. A resilient IR team is technically proficient and mentally and emotionally healthy.
The zero-sum mindset
Fighting adversaries is a zero-sum game: when one side wins, another must lose. People who are used to that game naturally adopt the same mindset in other areas of life. They need help to develop win-win solutions to problems because no matter the topic (getting more budget for security, getting more people aware about security, etc.), everything feels like a battle they must win. This zero-sum mindset often hurts the ability of incident responders and security practitioners to develop relationships, build careers, and become successful in business.
It helps to see many things as a continuous process. James Carse wrote a book, “Finite and Infinite Games”, that explores the difference between approaching things as a game with an end, or a game that goes on forever. While individual battles are finite, security as a process is infinite.
“For starters, what do you do after you win a finite game? You have to sign yourself up for another one, and you must find a way to showcase your past winnings. Finite players have to parade around their wealth and status. They need to display the markers of winning they have accumulated so that other players know whom they are dealing with. Carse argues that these players spend their time in the past because that’s where their winning is.
Infinite players, in contrast, look to the future. Because their goal is to keep the game going, they focus less on what happened and put more effort into figuring out what’s possible. By playing a single, non-repeatable game, they are unconcerned with the maintenance and display of past status. They are more concerned with positioning themselves to deal effectively with whatever challenges come up.” - Source: Finite and Infinite Games: Two Ways to Play the Game of Life
Finding a path for advancing security and incident response
Incidents won’t disappear and that cannot be the goal
We will never get the number of security incidents to zero, and that shouldn’t be the goal. Instead, we need to continuously raise the bar for attackers, reduce the damage from security breaches when they inevitably happen, and make it easier and faster to recover.
When there is a bad actor who is working hard to achieve their goals, inevitably, they will succeed. That’s just how statistics and probability work. Take bank robberies as an example. They existed for as long as there were banks, and we’ve worked tirelessly to stop them for centuries. Some of the brightest people on the planet, funded by insurance companies, have been designing safes that cannot be cracked. Banks and their insurers employ pen-testing firms that test how robust banks’ security measures are. We’ve employed security guards, installed alarms and emergency communication systems, and equipped those transporting cash with armored vehicles and security. Despite all these investments and centuries of data about what works and what doesn’t, the number of breaches is not zero. In the US alone, there were 1,612 bank robberies in 2022 according to FBI data which is fewer than before, but still quite a lot. That said, we’ve been quite successful at using a combination of deterrence, prevention, detection, and response to:
Deter bad actors from attempting to commit a crime.
Making it much harder to penetrate banks’ defenses.
Ensure that when robbers are successful, the amount of money they get to take with them will be insignificant and won’t have a material impact on the bank's balance sheet.
Make it easier to capture evidence of the crime, recreate the scene, investigate how it happened, and subsequently capture perpetrators.
Cybersecurity is no different. We know that incidents will continue to happen, so our goal should be to ensure that an employee clicking on the fishing link won’t bring down the whole company, that an attacker cannot operate for months undetected, and that treasure troves of data won’t get exposed during a single incident.
Steps we should be taking to move the state of incident response forward
Generating a pipeline of incident responders
The number of security incidents has been and will continue to go up. To address the problem, we need to ensure we have many strong incident responders. Unfortunately, it does not appear like we do. The difference between having some experience responding to a security breach and being an IR professional is like that between a person who can do CPR or First Aid and an Emergency Medical Technician (EMT). We need both, but we have neither. If we continue building on this analogy, the problem seems to be two-fold:
We don’t force CPR and First Aid training on every person who works in dangerous conditions, mostly because people are all different. Not every security practitioner is ready to respond to an incident even though it will inevitably happen. Good IR training is hard to come by.
We don’t have enough “EMTs” or people who run at the crater, and there is no good way to produce them at scale. People who have been doing IR for companies like Google, Microsoft, or Amazon, as well as many military veterans become really good at response, and some of them have started their own IR firms. While we can see what incident response looks at scale across hundreds of companies, there are very few of these people, and we need substantially more of them.
If we are to secure a world in which incidents happen as frequently as they do today, we need to generate a pipeline of incident responders. There must be another answer to this question.
Researching the human side of incident response
In addition, we need to invest in developing a deeper understanding of the human side of incident response. We need to have solid academic research into problems such as the God syndrome, PTSD, and zero-sum mindset affecting those who choose IR as their profession. There is plenty of knowledge about the psyche of all kinds of people who have to remain calm under pressure, who have a lot of access, and security incident response needs to catch up, not because of how unique it is, but because of how similar it is. There is a great book about rescue diving titled “Under Pressure: Diving Deeper with Human Factors”. If you substitute the words “rescue diving” for “incident response”, it is incredible how similar the two are in terms of their impact on human psychology.
The outcomes of research into the psychology of IR should inform how organizations approach people in charge of incident response work, and the IR process/operations themselves. In particular,
We need to make IR people work less and spend less time on incidents. They need time to recover, mandatory vacations, and time to retro the incidents that they are part of.
Companies must understand that when an incident happens, the entire organization has to step up and take part.
Incident responders need to delegate power to people more, in the proportion they need it to do their part. This means that IR professionals should focus more on orchestrating and coordinating the response, and less on being heroes and doing all the heavy lifting themselves.
Aggregating the data about incidents and incident response
Today, we need more data about incident response in general. Neither CISA, nor FBI or CIA, are currently able to aggregate data about the incidents across the nation.
Reports such as Verizon DBIR and CrowdStrike Global Threat Report are as good as we can get, but they need to be more comprehensive. It doesn’t appear that anyone can authoritatively say how much money companies have paid in ransomware in 2022, 2023, or any other year. That is a problem. We know that the reason we have been fairly good at identifying and responding to early signs of medical epidemics is the fact that all states aggregate and share data about infections and diseases daily. The government can then quickly identify anomalies and spot early warning signs of deadly diseases before it becomes too late to react. We need a similar approach to data aggregation in cybersecurity that goes beyond what we get with SEC disclosures of “material incidents”.
We believe aggregating the data about incidents and incident response should be the government's responsibility. The good news is that the infrastructure is already there, and our success with ISACS shows that it can be done well. What is needed now is the political will to take the last steps to formalize it. A government agency such as CISA is well suited to be given a broad mandate to become a central aggregator of cybersecurity data in the United States. Adding data about mental health and recovery can actually give us a solid understanding of the impact that long-term IR has on a company or person. It can also tell us where we are on the resilience bell curve. We get to know the current ability to receive incidents and respond to them with effective focus from our teams and country.
We know that companies are not incentivized to share sensitive data and anything that has to do with IR is buried under non-disclosure agreements and legal red tape to reduce liability. Infrastructure and security providers are also working hard to keep their incidents under wraps because they want to be seen as secure. That said, we also know that insurance companies have visibility into what’s happening on the ground, beyond just publicly-traded corporations. Every insurance company knows exactly when their policyholders are breached and what was the cause of the breach, whether or not they engaged a third party for ransomware negotiations, as well as if and how much they paid in ransom. Insurance providers can be mandated to aggregate this data and report it to CISA in an anonymized way so that it can further analyze it and share it with the rest of the industry.
Insurance companies are in the business of statistics and data science. They’ve done a fantastic job at aggregating data about other types of risks, from car crashes to fires, deaths from natural causes, medical malfunction, and all kinds of diseases. Without all this data they would not be able to build actuarial tables and underwrite insurance policies. Getting insurance companies to collect and share some data about security incidents sounds like a natural next step for the government to gain better visibility into the state of cyber.
Reduce the number of incidents on a global scale
Nation-state attackers want to take stuff from security teams - they want information, money, or control that push their needs forward. This means we don’t have a security issue; we have a power issue, which boils down to a money issue.
What happens if we take a hard left turn and throw the problem on its head? Take an ostracized small country or region. Isolated from the global supply chain, culture, and economy, the region has resorted to becoming a global crime syndicate for survival. But what if we approached this differently? What if we explored ways to reintegrate this region into the global community instead of focusing solely on sanctions and isolation? By creating opportunities for its people to contribute to the global culture and economy—perhaps through legal avenues like coding or offering services—we might reduce the incentive for cybercrime. Investing in small towns abroad that are currently hotbeds of cybercrime and allowing them to generate legitimate revenue could create a more stable economic base. It could be a win-win for all parties involved if we can make it less appealing for these actors to target banks. Honestly, this way, everyone gets to take a nap now and again, and we all get to play the game a little longer.
Additionally, there have been discussions around establishing NATO-like cybersecurity alliances. Such collaboration could significantly benefit global cybersecurity efforts, whether this is a military alliance or an organization akin to the UN. By fostering intergovernmental cooperation and establishing clear rules for cybersecurity, we could create a more secure and stable digital landscape. This would enhance defense mechanisms and provide a framework for holding nation-state attackers accountable on a global scale.
If you like my blog, please subscribe & share it with your friends. I do this in my free time, so seeing the readership grow helps me to stay motivated and write more. I don’t send anything except my writing and don’t sell your data to anyone as I have better stuff to do.
If you are a builder - current or aspiring startup founder, security practitioner, marketing or sales leader, product manager, investor, software developer, industry analyst, or someone else who is building the future of cybersecurity, check out my best selling book, Cyber for Builders.
If you are a security practitioner, check out & spread the word about the VIS Angel Syndicate.
If your company is interested in sponsoring Venture in Security, check out Sponsorships. Thank you!
 | A guest post by
|