I’ve been involved in incident response for a long time; whether as a client, consultant or working for Symantec, and have seen something that has been pretty consistent over these many years: the time between a customer experiencing an incident versus when they finally call for assistance. When I talk to our Global Partners and our Symantec Incident Response Services team, they find the same thing; they usually get a call on Friday, late afternoon, local time zone, and the client has been working the case for anywhere between a week or months. When I ask my peers about this, the thing that bothers me is that we all chuckle about it: “Oh yeah, Friday, at 3PM, that phone is going to ring.” I hear it from all of them. But it’s not funny.
When I went from owning my own IT consulting business to leading the IT department for a consulting company I was no different from my current peers. I would spend hours, days, and weeks, working an incident with my team, and, whether it was Code Red or a switch/router problem, we always waited until we exhausted every ounce of energy we had, and every thought, until we finally called someone. Don’t get me wrong; when you do get it figured out and solve the problem internally, it feels great, but doing it this way may not be the best way.
When I moved on to a role leading an Information Security Team for a large financial firm, it didn’t take long before I realized that waiting cost more money than calling in reinforcements. I would constantly remind my team that calling support was good. We had already paid for it, so why not use it when we had security incidents and calling for assistance from a professional incident responder was the right thing to do? Even then, the team always wanted to try to do it themselves first and we always ended up waiting until Friday afternoon to engage someone--partly to prove they were worth their salt (we already knew they were, but that’s human nature, I guess), partly for the challenge, and partly because they weren’t certain the nature of the incident and were worried to ask for money to pay an Incident responder to then find out it was a false positive.
However, the problem wasn’t that they ruined people’s weekends or that they failed to get the job done; the real loss was in opportunities. When you wait, you may lose the opportunity to gather evidence, intelligence, and data that could have contained or prevented different aspects of the attack or in some cases captured or identified your attack or the attacker’s motives. This becomes a significant problem for the industry as a whole. We continue to lag behind the attackers because we don’t immediately act to shut them down and prevent them from being successful. They face an industry that works in silos, islands of companies and organizations, that are all slow to react and that don’t share what they know. We need to come up with a simple way to explain to all security professionals exactly how long before you call someone is long enough and in a subsequent blog, when and what to share.
I have postulated the following equation: TID – TCA = ∆T = LO
Time of Incident Detected – Time to Call for Assistance = Delta Time = Lost Opportunity.
Let’s start with Time of Incident Detected. For this equation this is when the event has been determined to be an incident that requires a response. There will be some time spent before this point investigating and researching the event but the need for a response is undetermined. There is also the time when the incursion took place that may precede the discovered event but you may never be able to figure out when that took place. The important factor is the time when you finally realize or determine that you have an active incursion that requires countermeasures. Prior to this point in time you couldn’t have taking swifter action, there are too many false positives and explainable events to act on every one. It’s only now that the clock starts ticking. How long from this point do you wait to call in experts that are trained and practiced in battling intruders and can minimize the damage to your company and its customers?
That leads us to Time to Call for Assistance. From the moment you realize that this is an incident, the clock becomes your enemy as much as the attacker. What is the first thing you do? Do you reach for your Incident Response Plan? Many companies have them but they rarely use them. If you do grab that plan, what is the first thing listed? NIST recommends several components such as a Mission, Strategies and goals, senior management approval, etc. all good things if you are a company that has a dedicated Incident Response team. If you do have a team, this equation still works; it’s just the time before you call your own team. If you don’t have expert practiced incident responders, then you are scrambling people in your Information Security, Infrastructure, Network teams as well as other smart people that are respected. The problem that companies encounter is either they think the problem is solved, but the attack is actually just hiding; the symptom was treated and the disease is left uncured. The other problem is that they spend a considerable amount of time battling only to finally determine they need expert help and make a call. At that moment we can calculate the time from the Incident to the Time of Call for Assistance. It’s this period of time that is the most damaging for many reasons.
One reason this is damaging is the amount of potential evidence lost or mishandle. If your team is made up of many cross functional teams from varying groups that aren’t trained in handling evidence then you might lose or mishandle a key piece of information that leads to lost data, lost time, public exposure, etc. Other reasons are the loss of productivity to your company, the hit to employee morale, greater exposure to a larger attacker community, board actions, legal actions, etc.
In my next post we’ll discuss Delta Time and X and start to bring this all together.