Disaster Recovered

We had an interesting corporate experience the other week - our Melbourne office basement was flooded with 7 feet of water by a burst main on Sunday night. The building was closed down until it could be pumped out and re-certified for safe occupancy. Now this is not a 2-up-2-down humpy in the suburbs, this is a major sky scraper in the Melbourne CBD - a significant event. Press coverage here for more details. I am based in Sydney and was not directly affected but my colleague Tim Wragg was peering through the door at the chaos and brings some personal experience to the event.

What I thought would be worth sharing is the basic learnings for us from what could have been a lot more catastrophic than it was. There were probably three main areas of recovery - people, systems and real estate - to get our business back to normal - and it was of primary importance that our clients experienced as little disruption as possible.

The problem with the real estate was that in the affected building all the electricity and communications terminals were in the basement and therefore completely debilitated by the flooding. So regardless of any plans for dual carriers etc, everything was wiped out. Also because the building occupancy certification was cancelled by the council and would not be reissued until after the water was pumped out we were not allowed to have people at the desks anyway.

Many of our corporate systems are delivered from a central Sydney infrastructure so the main operational functions that needed to be switched over were the telephone help desks. That was done in short order without any problem and only seconds of interruption to service, probably unnoticed by anyone. We also flew a few of those team members to Sydney for the week to continue business as usual in that area. We already run a full VPN based framework to allow people to work from home, on site at clients, whilst travelling overseas etc so the physical location of the teams affected was not a concern. We didn't need to rush out and get temporary desk space - the combination of mobile phones and Internet access allowed most people to work as normal, just from a different physical place.

People - this was the trickiest area - how do you notify large volumes of people that the office is not open and they should stay at home or arrange to work from a client site (for those in consultancy roles). Well bulk SMS was the way we did it and it worked reasonably well - for the employees with company issued mobile phones that is. Within 2 hours of notification by the building management we had passed on the initial "stay at home, await further instructions" alert. Of course there are lots of people in the business that do not have a corporate mobile phone and the lesson learned was that we did not have accurate up-to-date mobile numbers for many of these people so there was a fair amount of time spent dialling land lines. Even still a few were missed and had to be repelled at the door.

Also, the SMS system that we used required data entry of the target phone numbers for the text message - when time is of the essence this is not ideal - so we have changed the bulk SMS system we use to one that allows us to keep an online list "ready to go". We will also be updating our corporate processes to ensure we capture and update these details for employees in future.

Some people responded that they had left their laptop pcs in the office over the weekend and we were lucky enough to be able to access the building under supervision to recover these and a few desktop pcs for key staff. If this hadn't been possible then some individuals would have been left in the lurch whilst we cobbled together spare machines for them.

As it was, although the building was out of commission for a full week we were remarkably well protected from the affects of a lock-out. The real question is what shape would we have been in if it had been the Sydney building that was uninhabitable and we are assessing that in light of the Melbourne event.

Walk out the front door of your office, lock it and shut off the lights and think about what you would do to get back up and running. Document it - that's the start of your disaster recovery/business continuity plan.