Disaster Recovery (D.R.) comes in many different shapes and sizes. There is no perfect model that will save you or your company indefinitely. Disaster Recovery has to be maintained, monitored, and regularly evaluated. D.R. is a set of tools that safeguards your company against a set of unprecedented events. Although there are well-thought-out plans for many companies, there is one critical point often left and forgotten about- and it can make your company very fragile in the event of a tragedy.
It is typical for companies to assign the role of building a Disaster Recovery plan to the Information Technology (I.T.) department. Perhaps the I.T. Director or Chief Information Security Officer. The I.T. Director (or full-time equivalent F.T.E.) will design a redundant system to keep systems alive. So to paint a picture, I will explain the demand for an “example company” and a secure D.R. plan.
For "example company", they have 2,000 employees with multiple locations spanning several cities. They started off with a single server that hosted several virtual machines (V.M.) that serviced them with authentication, files, printing, sharing, email, and the other typical services you would expect. However, this was their setup when they had less than 100 employees and a single I.T. Technician. Since they have expanded, they have upgraded their equipment and have added several servers. The I.T. Director during the present time has designed the following fail-safes in the case of a disaster:
· Each location has its own on-prem servers that sync with the corporate office.
· Each location has two uplink internet connections to different internet service providers (I.S.P.).
· Each location has an LTE cellular data connection that provides the I.T. Department a connection to the network equipment in case of an outage of both internet providers.
· The locations can function for nearly "as long as needed" without their WAN connections with their on-premises servers. Once the connection is restored, the servers will re-sync.
· Each switch and server utilize channel bonding (link aggregation).
· There is an extra server, switch, and router firewall at the corporate office that can be quickly restored to any location.
· All servers and network equipment have service contracts.
· All servers and network equipment reside on uninterruptable power supplies which utilize an "online" battery connection, so there is minimal downtime during short-term power loss.
· There are gas generators at each location, and they are tested every month by the maintenance department.
· The I.T. Department has utilized load balancers, replication, and high availability where necessary.
· There are daily backups being performed on-site and off-site.
· Last but not least, the company has purchased a hot site where they can offload production to a location that is ready to serve their immediate needs.
This is a very expensive plan to have in place for a business of this size. Hundreds of thousands of dollars would be involved to make this plan work. There would have to be constant monitoring, and a plan of this length would almost need a separate job title to keep up with. Any lack of monitoring or checkups could put the plan at risk. However still, this plan of action is very impenetrable and secure. Many events could happen that would not shake this plan. In a lot of cases, your company would not even budge when hit with a tragedy.
The point of this blog was not to fascinate you with how redundant you can make a system, but it was merely to point out one fact. The I.T. Department is ready, but the company is not. From experience, it is commonly left up to the I.T. Director to set up the disaster recovery plan. A good I.T. Director informs management, staff, and clients of their plan. Those who listen nod their head and they applaud the plan, few will remember it. Now, let's throw something out there. The I.T. Department is ready, the people are not. Some disasters can widely impact departments within your company- not even from a technological standpoint. Some disasters can wipe out your I.T. department's entire disaster recovery plan within minutes- no matter how redundant, leaving your business defenseless. This is their critical point. I have found this single fault in so many businesses. But be grateful; there is a solution.
The solution is that while your I.T. department is ready, your other departments must do the same. Your I.T. department cannot be your only point of failure, and you must learn to survive- with or without them. That sounds a bit dramatic, but seriously. A single infection of malware could devastate your entire D.R. plan, and a department of four I.T. guys would not be able to get the business back online without encountering any revenue or business loss. But, imagine the business has preparations among its departments. These are items to be considered for a hospital environment:
· How do you continue to chart on your patients?
· What medical I.T. devices can be substituted with mechanical devices?
· How do you check insurance without a computer?
· How do you get bills out without a computer?
· How do you perform payroll and accounts payables without a standing network?
Merely every department should have a policy that outlines the procedures they should take when the I.T. department takes a hit. Hopefully, nothing so severe would strike your business. These policies could help with business continuity even when your I.T. department has short-term downtime. So if nothing else, they are still a reinforcement to your business operations.
A disaster recovery plan is a certain necessity in every business environment. No matter how big or small. There is always something that can be done to lessen the effects of any tragedy that can occur. A company's biggest mistake is relying on a fraction of their resources to bear the weight of all disasters. However, assigning the responsibility to every resource within your company makes you much better prepared for such an event.
