We’ve written about network disaster recovery planning before. It’s one of those things you won’t appreciate until you need it. Then, it’s priceless. The Applied Technologies team prides itself on our ability to partner with and utilize the latest and best technology to protect our clients’ networks and data from disaster. But protection is only half the battle. Whether we’re talking about natural disasters like tornados or malicious attacks, no company is 100% protected from disaster. And with the marked increase in recent years of both cyber-security attacks and the frequency of natural disasters, we advise clients to think about disaster recovery as a “when” and not “if” business priority.
This is the story of how Applied Technologies helped one client who recently experienced a network breach recover their data, resecure their network, and minimize business impact by following a thorough and documented disaster recovery plan.
The Situation
It was a Monday morning. Our client noticed they were having issues accessing the internet and business operation tools. While this was strange, there wasn’t an immediate cause for alarm. They contacted their Applied Technologies Account Manager about the issue, who began investigating the root cause of the issue. But within the next hour, we collectively recognized this was more than just an infrastructure issue. The client’s network had been breached in a ransomware attack. An email from the attacker confirmed our suspicions that someone had gained access to their network, and their business and data were at risk. We pulled the client’s entire physical connection to the internet immediately. But the real work was just getting started.
Our Disaster Recovery Solution
There are a handful of immediate priorities to consider when your company experiences a cyber attack. We advise clients to make the following four contacts in this order:
- Reach out to your Applied Technologies Account Manager. Our Network team is uniquely qualified to take the lead and guide you through this process.
- Work with our team to contact in-house emergency contacts, as well as emergency contacts at the companies whose technology is at work in your network. In the case of our customers, that’s often companies like VMware, Veeam, Unitrends, NetApp, and Cisco.
- In some cases (like this one), our next call is to a forensics and incident response team to determine how the attacked gained access, and exactly what actions they took.
- From there, we begin the data recovery and restoral process. The goal is to get a snapshot of your network at the moment it was compromised, so we can download or restore backups and get your business operational again.
- Finally, we recommend you consult with your attorney. Every state has different requirements for what and when you need to disclose, depending on the scope of the breach, the data that was compromised, and the laws in the state.
Following the assembly of key contacts, we worked together to proceed through our client’s Disaster Recovery Plan. The plan identified a hierarchy of systems the company needed to restore operations. Having protected the network from further attack, we wiped all data from the onsite hardware. It was transferred to external discs an shipped off to our forensics team for analysis. We then worked through the priority list in the plan to ensure we minimized the downtime and restored minimal business continuity systems first.
We had decisions to make. We needed to restore the onsite environment no matter what. But how? In some cases, we might spin up a remote site or us a DRaaS solution and work from there. In this case, we worked with our forensics team to identify a point in time from which we could recover lost data from our backups. Because this breach was discovered quickly, we were able to recover 100% of lost data from recent backups. Some things can’t be restored from backups. For instance, the configurations for the client’s virtual server environment had to be rebuilt from the ground up.
Because the client’s infrastructure was thoroughly documented in their disaster recovery plan, we were able to begin the restoration process quickly and get them back up and running in a matter of days.
The Results
Our client experienced just a week of complete system downtime as a result of the attack. Downtime is never ideal, but if they hadn’t partnered with Applied Technologies to create, document, practice, and update their disaster recovery plan, this could have been catastrophic. There are three critical reasons why the disaster recovery process was so successful for this client:
- The client had an up-to-date disaster recovery plan that accounted for a variety of scenarios, including the one they encountered.
- Their disaster recovery plan was updated regularly to ensure it accounted for new hardware and software that had been added to the system along the way. We recommend our clients work with us to update their plan as frequently as twice a year.
- The team had practiced its disaster recovery plan. Everyone who had a role knew what it was, and how it contributed to the overall process. They knew which servers were the most critical and the order in which to restore them so business could continue as soon as possible.
In this story, we explained the network disaster recovery process from a ransomware attack. But it’s not the only risk to your data. Human error, malicious employee behavior, hardware failure, natural disasters, power failure, and other cybercrime put your company and your data at risk. If you’re operating without a disaster recovery plan, or the one you have is out of date, your business is operating without a critical safety net.
Applied Technologies can help. Reach out to our qualified team today to discuss the disaster recovery planning process and align yourself with a partner that has real life experience helping clients like you bounce back from disaster.