Disaster Recovery 101 - Back to the Fundamentals
Are your backups collecting virtual dust rather than readying you for the next unavoidable disaster? Get back to basics on crafting an ironclad disaster recovery plan. We outline the essential infrastructure, applications, staffing, and execution steps often glossed over by the check-the-box enterprise crowd.
Going beyond mere data recovery, we detail considerations around standing up replacement infrastructure, understanding system interdependencies, and restoring functionality faster with increased automation. Cloud's scaling and affordability make DR exercises less daunting these days if configured properly on the front-end.
With major outages increasingly likely, the principles detailed could dictate whether your business emerges unscathed or shutters for good. We share tips for pragmatic preparation reflecting our scar tissue from failures past when lackadaisical DR rigor proved painful. Ever try troubleshooting recovery steps in the midst of a raging hurricane...or gotten that dreaded 2AM offline alert while welcoming your newborn? We have!
Major takeaways:
- Start recovery documentation with manual steps, increase automation later
- Test often - the cloud enables cost-effective exercising at scale
- Availability outshines recovery - build resilient systems and staff cross-training
- Know precise RTOs and RPOs aligned to business priority and appetite
Stay tuned as we separate the mavens from the mayhem when adverse events strike. Get your data DR ducks in order now before things migrate south!
News articles from this episode:
Speaker:
ATR2500x-USB Microphone & Logitech BRIO: This week, we're talking about disasters
Speaker:
and I don't just mean the monster snow storm hitting the Northeast right now.
Speaker:
We're talking disasters to take out your entire infrastructure.
Speaker:
We're getting back to basics on perhaps the most important, most
Speaker:
often neglected aspect of backup preparedness your disaster recovery plan.
Speaker:
We'll walk through the essential elements.
Speaker:
Every business needs to address before catastrophe strikes.
Speaker:
A box of tapes or backups in the cloud.
Speaker:
Is not a Dr.
Speaker:
Plan.
Speaker:
I'll even share a crazy story from the early days of my it career.
Speaker:
That taught me the hard way.
Speaker:
Why a documented recovery plan matter.
Speaker:
I honestly still can't believe what my employers did that day.
Speaker:
I'm w Curtis Preston, AKA Mr.
Speaker:
Backup.
Speaker:
And I've sat where you're sitting.
Speaker:
I want to turn you and everyone like you and to a cyber recovery hero.
Speaker:
This is the backup wrap-up.
Speaker:
W. Curtis Preston: Welcome to the backup wrap up.
Speaker:
I'm your host, w Curtis Preston, and I have with me a guy who as awaiting the
Speaker:
results of my battery test as much as I am Prasanna Malaiyandi how's it going?
Speaker:
Prasanna
Prasanna Malaiyandi:
I am good, Curtis.
Prasanna Malaiyandi:
Yes.
Prasanna Malaiyandi:
I am dying to understand like, do you have any degradation in
Prasanna Malaiyandi:
your carb because you sort of ran into an issue last week, right?
Prasanna Malaiyandi:
W. Curtis Preston: maybe, maybe.
Prasanna Malaiyandi:
Um, yeah, I killed my battery in my Tesla and I'm currently running a, a battery
Prasanna Malaiyandi:
test and I will, I will just say this.
Prasanna Malaiyandi:
It's, it's not, it's clearly not designed for like the, for me, uh, it's, it's
Prasanna Malaiyandi:
actually something inside service mode of the Tesla and you have to like, go into
Prasanna Malaiyandi:
super secret mode or whatever to do it.
Prasanna Malaiyandi:
And, um, you know, it's okay that I'm, that I do it, but I will just
Prasanna Malaiyandi:
say that the UI is really crappy.
Prasanna Malaiyandi:
Yeah, it's not intended for you.
Prasanna Malaiyandi:
It's intended for a service technician.
Prasanna Malaiyandi:
W. Curtis Preston: Yeah, and so like.
Prasanna Malaiyandi:
It finished, you know, it discharged, uh, to 13% last night, which I
Prasanna Malaiyandi:
thought it was gonna go like closer to zero and then it charged overnight
Prasanna Malaiyandi:
and now it's doing something.
Prasanna Malaiyandi:
Last time I checked, like you said, it's probably measuring voltages and
Prasanna Malaiyandi:
whatnot, but like there's nothing on the screen that says doing something.
Prasanna Malaiyandi:
The only way you can do is you can say, Hey, show me the report of the battery
Prasanna Malaiyandi:
test, and it could go, you haven't done a battery test for 500 miles.
Prasanna Malaiyandi:
And, um, and so it's like, oh, I guess it's still running.
Prasanna Malaiyandi:
Like, it's just a really weird user interface, but, um, so we'll see.
Prasanna Malaiyandi:
But I'm
Prasanna Malaiyandi:
Crossing my fingers.
Prasanna Malaiyandi:
At least you'll get some data.
Prasanna Malaiyandi:
W. Curtis Preston: yeah, yeah, absolutely.
Prasanna Malaiyandi:
Data, data data's king.
Prasanna Malaiyandi:
Mm-Hmm.
Prasanna Malaiyandi:
W. Curtis Preston: So, uh, speaking of data, let's talk
Prasanna Malaiyandi:
about the news of the week.
Prasanna Malaiyandi:
I'm gonna say the big story that jumped up.
Prasanna Malaiyandi:
You know, whenever we do this, I, I, I, I just google backup news
Prasanna Malaiyandi:
and I see if anything pops up.
Prasanna Malaiyandi:
And the thing, that thing that popped up was kind of a big deal,
Prasanna Malaiyandi:
and that is that there are 50,000 WordPress sites that are affected by
Prasanna Malaiyandi:
a critical bug in a backup plugin.
Prasanna Malaiyandi:
Called backup migration.
Prasanna Malaiyandi:
That helps admins automate site backups to local storage
Prasanna Malaiyandi:
or to a Google Drive account.
Prasanna Malaiyandi:
And, um, it's a scary, it's a scary little, uh, uh, bug, wouldn't you say?
Prasanna Malaiyandi:
Oh yeah, especially because most of these backup tools,
Prasanna Malaiyandi:
of course, need admin level access.
Prasanna Malaiyandi:
So if you have a flaw in that, of course someone who exploits it
Prasanna Malaiyandi:
now has full access to everything.
Prasanna Malaiyandi:
W. Curtis Preston: Yeah, and this was scary because it says, it, it it, you
Prasanna Malaiyandi:
can do it without, uh, user interaction.
Prasanna Malaiyandi:
You can, uh, and, and once you do it, you gain.
Prasanna Malaiyandi:
Uh, basically full control of the site.
Prasanna Malaiyandi:
Um, and, uh, it says, um, basically they just have to pass the right
Prasanna Malaiyandi:
values to the right thing and poof, they end up controlling your site.
Prasanna Malaiyandi:
The fir you know what, the first thing I did was right.
Prasanna Malaiyandi:
I.
Prasanna Malaiyandi:
Check.
Prasanna Malaiyandi:
Check to make sure you're not running it.
Prasanna Malaiyandi:
W. Curtis Preston: Yeah, that's exactly, 'cause I knew I had a migration plugin
Prasanna Malaiyandi:
'cause I might, I migrated backup Central from, um, one, well, it's
Prasanna Malaiyandi:
actually within the same company, but I migrated it from, essentially from one.
Prasanna Malaiyandi:
Server to another server not that long ago.
Prasanna Malaiyandi:
And I used a migration plugin to do it, and I saw that word backup migration.
Prasanna Malaiyandi:
I was like, wait, was it, what is this the one I used?
Prasanna Malaiyandi:
Right?
Prasanna Malaiyandi:
And, uh, and it wasn't so, um, I, but I, but I, then I, I took the moment I
Prasanna Malaiyandi:
realized that the migration plugin that I used was still installed and activated.
Prasanna Malaiyandi:
And I'm like, you know what?
Prasanna Malaiyandi:
I'm just gonna go ahead and install
Prasanna Malaiyandi:
Yeah.
Prasanna Malaiyandi:
Now, uh, the one good thing is once the developer was aware of this issue, they
Prasanna Malaiyandi:
pa they had a patch available within a couple hours, which is good, right?
Prasanna Malaiyandi:
They were very responsive.
Prasanna Malaiyandi:
However, now they just need to make sure that all the sites get updated
Prasanna Malaiyandi:
who are using the plugin, and that's, I think the bigger challenge
Prasanna Malaiyandi:
W. Curtis Preston: Yeah.
Prasanna Malaiyandi:
It really is the bigger challenge, especially given that, uh, the,
Prasanna Malaiyandi:
the developer doesn't mention anything about this on the.
Prasanna Malaiyandi:
wordpress.org plugin website.
Prasanna Malaiyandi:
So I'm a little disappointed in that, but you know.
Prasanna Malaiyandi:
Um, I dunno what to say about that, but yeah, so I, I'll just say that.
Prasanna Malaiyandi:
I'll just put it as a general note.
Prasanna Malaiyandi:
If you do have a WordPress site or you know, whatever, if you
Prasanna Malaiyandi:
have a website, look at any, you know, any plugins that you have.
Prasanna Malaiyandi:
Like in this case, I really should have deactivated this migration.
Prasanna Malaiyandi:
Plug it a long time ago.
Prasanna Malaiyandi:
Make sure that any plugins that you have that are running, you actually
Prasanna Malaiyandi:
need them, deactivate them, uh, and remove them, uh, if you're not.
Prasanna Malaiyandi:
So, um, what's our next, uh, news item?
Prasanna Malaiyandi:
So the next news item is Veeam just came out with
Prasanna Malaiyandi:
their big launch and they've basically doubled down on a lot of cybersecurity
Prasanna Malaiyandi:
related features, and I thought maybe we could talk about some of those.
Prasanna Malaiyandi:
W. Curtis Preston: Yeah, there are a lot.
Prasanna Malaiyandi:
Um, right.
Prasanna Malaiyandi:
So, uh, you know, some of them you might expect, right?
Prasanna Malaiyandi:
The idea of like inline malware detection, right?
Prasanna Malaiyandi:
Detecting the malware as it's coming in, in the backup, looking for a file system
Prasanna Malaiyandi:
activity that looks a little weird, right?
Prasanna Malaiyandi:
That, that would include, or that would include things like, hey,
Prasanna Malaiyandi:
there, somebody's, uh, appears to be.
Prasanna Malaiyandi:
Changing a lot of files, changing the file names, different file types.
Prasanna Malaiyandi:
You never had this file type.
Prasanna Malaiyandi:
And where you see that is if you see an encryption attack.
Prasanna Malaiyandi:
They tend to create a file type as the encrypted files have their own file
Prasanna Malaiyandi:
type and they're also integrating with, EDR and XDR tools and SEIM/SOAR tools.
Prasanna Malaiyandi:
Which that sounds pretty good.
Prasanna Malaiyandi:
So they're also integrating with ServiceNow to, it's
Prasanna Malaiyandi:
sort of looking at, Hey, here's another part of your environment that we could
Prasanna Malaiyandi:
potentially detect bad activity, so let's make sure we're plugging into your
Prasanna Malaiyandi:
existing infrastructure tools, rather than you having a report somewhere else
Prasanna Malaiyandi:
that you rarely ever look at, right?
Prasanna Malaiyandi:
That then you're like, oh, I saw that my backup system noticed
Prasanna Malaiyandi:
this like 30 days ago, and I'm just starting to see that now.
Prasanna Malaiyandi:
So I'm glad they're integrating into the rest of the environment.
Prasanna Malaiyandi:
The reporting environment, which makes sense.
Prasanna Malaiyandi:
Reporting and monitoring.
Prasanna Malaiyandi:
one thing from that list that I did like, and I know Curtis, you talk about
Prasanna Malaiyandi:
this a lot, is, I don't know if you saw, there's mention of four Eyes, right?
Prasanna Malaiyandi:
So now in case you need to make certain modifications to backup settings and
Prasanna Malaiyandi:
other things like that, you basically need two admins or four eyes in
Prasanna Malaiyandi:
order to do those sort of operations.
Prasanna Malaiyandi:
So.
Prasanna Malaiyandi:
This helps in a lot of the ransomware cases where a ransomware actor gets in
Prasanna Malaiyandi:
and they basically change the backup policy, change retention down to one
Prasanna Malaiyandi:
day, and then all your backups are gone.
Prasanna Malaiyandi:
Right?
Prasanna Malaiyandi:
So these sort of things could potentially now require having four uh, two
Prasanna Malaiyandi:
people in order to be able to do that.
Prasanna Malaiyandi:
W. Curtis Preston: Yeah, I like that.
Prasanna Malaiyandi:
I also like the integration, uh, with Yara.
Prasanna Malaiyandi:
Uh, so basically once you've identified some malware that you have, they can
Prasanna Malaiyandi:
find that malware and get rid of it in your backups, which that sounds great.
Prasanna Malaiyandi:
Uh, they got KMS integration.
Prasanna Malaiyandi:
They got a lot.
Prasanna Malaiyandi:
I mean, it's just like there's really too much for us to cover in,
Prasanna Malaiyandi:
in this level of, in this level.
Prasanna Malaiyandi:
Uh.
Prasanna Malaiyandi:
Of report,
Prasanna Malaiyandi:
um,
Prasanna Malaiyandi:
link, include some links to the articles, to
Prasanna Malaiyandi:
the websites and places you could go look to see more details.
Prasanna Malaiyandi:
But I'm glad that there, there's so much focus though on cyber security
Prasanna Malaiyandi:
just given what we see in backup environments with ransomware and
Prasanna Malaiyandi:
other threat actors going purposefully after those backup infrastructure.
Prasanna Malaiyandi:
W. Curtis Preston: Yeah.
Prasanna Malaiyandi:
Uh, there's a great blog from Jorge DeLaCruz.
Prasanna Malaiyandi:
Uh, we'll, we'll link to that, I think probably be the, it lists all of them
Prasanna Malaiyandi:
and sort of explains why, you know, why the different features are important.
Prasanna Malaiyandi:
So that is the news of the week.
Prasanna Malaiyandi:
I hope you enjoyed the news, or maybe you didn't enjoy it because you, uh, maybe
Prasanna Malaiyandi:
there was bad news in there for you.
Prasanna Malaiyandi:
I don't know.
Prasanna Malaiyandi:
In our continued series of, uh, backup to basics.
Prasanna Malaiyandi:
We're talking today about DR.
Prasanna Malaiyandi:
Plans, and today we're gonna talk about what goes into a DR
Prasanna Malaiyandi:
plan and, um, you know, because.
Prasanna Malaiyandi:
One of the things you, you may have heard me say, uh, you know, is, is
Prasanna Malaiyandi:
although this, this phrase is now pretty dated, a box of tapes is not a DR plan.
Prasanna Malaiyandi:
Uh, a a bunch of a bunch of backups in the cloud is also not a DR plan, right?
Prasanna Malaiyandi:
You're like, oh, I got, I got all my stuff up in S3 with object lock, turn on.
Prasanna Malaiyandi:
Good,
Prasanna Malaiyandi:
Now what?
Prasanna Malaiyandi:
W. Curtis Preston: But, um, yeah, now what?
Prasanna Malaiyandi:
Right?
Prasanna Malaiyandi:
So, um, the first thing here.
Prasanna Malaiyandi:
That I've got, uh, you know, you need to assume that you're starting from scratch
Prasanna Malaiyandi:
if you're doing a disaster recovery plan.
Prasanna Malaiyandi:
I will say that the most likely reason today that you're going to
Prasanna Malaiyandi:
be actually using your DR plan is that you had a ransomware attack,
Prasanna Malaiyandi:
but that you cannot assume that and.
Prasanna Malaiyandi:
Also, you cannot assume, depending on decisions that hopefully you made well in
Prasanna Malaiyandi:
advance, you cannot assume the first thing that should be in a DR plan, and that is
Prasanna Malaiyandi:
that you know where you're competing and infrastructure is gonna come from, right?
Prasanna Malaiyandi:
Uh, why?
Prasanna Malaiyandi:
Why would that not be the case?
Prasanna Malaiyandi:
Right?
Prasanna Malaiyandi:
You got a ransomware attack.
Prasanna Malaiyandi:
They didn't blow up your servers, they didn't set 'em on fire.
Prasanna Malaiyandi:
Why might not you have servers to
Prasanna Malaiyandi:
Because those servers you might need to
Prasanna Malaiyandi:
preserve for forensic purposes to really figure out what's going on.
Prasanna Malaiyandi:
You don't know if you can cleanly delete everything on them, start from scratch.
Prasanna Malaiyandi:
It's probably gonna take you more time to do that.
Prasanna Malaiyandi:
Although one can argue given sort of supply chain issues right now,
Prasanna Malaiyandi:
that maybe it'll be less time.
Prasanna Malaiyandi:
To do that than to actually buy new servers, have 'em
Prasanna Malaiyandi:
drop ship to your data center.
Prasanna Malaiyandi:
But, uh, not saying that's the only option for you.
Prasanna Malaiyandi:
W. Curtis Preston: I'm personally in the Yes, you need to
Prasanna Malaiyandi:
take a forensic copy, right?
Prasanna Malaiyandi:
So that's the, that's one of the first things you, you, you want
Prasanna Malaiyandi:
do is take a forensic copy of any servers that it appears have
Prasanna Malaiyandi:
been affected or, or infected.
Prasanna Malaiyandi:
The, I'm personally a wipe and restore person.
Prasanna Malaiyandi:
Right.
Prasanna Malaiyandi:
Um, you know, everybody there, there, other people have different opinions
Prasanna Malaiyandi:
on that, but I, I think you, you've struck on it a pretty good one.
Prasanna Malaiyandi:
Supply chain, which is another reason why perhaps where your
Prasanna Malaiyandi:
competing infrastructure come, will come from in a DR is the cloud.
Prasanna Malaiyandi:
Right.
Prasanna Malaiyandi:
But just the make,
Prasanna Malaiyandi:
It.
Prasanna Malaiyandi:
It could.
Prasanna Malaiyandi:
Right?
Prasanna Malaiyandi:
And I think for a lot of cases that makes sense.
Prasanna Malaiyandi:
But I know for instance, if it's like a mission critical application
Prasanna Malaiyandi:
for your business that you cannot lose, I'm pretty sure even before you
Prasanna Malaiyandi:
get hit by a ransomware, you should already have a DR plan in place.
Prasanna Malaiyandi:
Maybe that implies sort of replicating to another co-location facility that you
Prasanna Malaiyandi:
have equipment already set up ready to go.
Prasanna Malaiyandi:
So you don't necessarily have to worry about the procurement process, but
Prasanna Malaiyandi:
W. Curtis Preston: Yeah, I think, I think that'll be a running story here, right?
Prasanna Malaiyandi:
Basically the whole point of the DR plan is the second word, right?
Prasanna Malaiyandi:
Is the plan part, right?
Prasanna Malaiyandi:
Hopefully you're doing this, um, you know, the, um, you wanna take the next
Prasanna Malaiyandi:
Yeah.
Prasanna Malaiyandi:
So the next one is, uh.
Prasanna Malaiyandi:
Thinking about once you now have your disaster recovery site up and
Prasanna Malaiyandi:
running, how do you protect it?
Prasanna Malaiyandi:
Because you've failed over, you've recovered to that environment.
Prasanna Malaiyandi:
Your production applications are running there.
Prasanna Malaiyandi:
You need to make sure that you have a way to protect it in case you need to worry
Prasanna Malaiyandi:
about how do I restore from a backup or something else happens while I'm running
Prasanna Malaiyandi:
in that disaster recovery site, because that's literally your only copy of data.
Prasanna Malaiyandi:
W. Curtis Preston: Yeah.
Prasanna Malaiyandi:
Um, and you know, and some people say, oh, gee, Chris, you know, like, you
Prasanna Malaiyandi:
know, why are you talking about backups?
Prasanna Malaiyandi:
We haven't even got to the Restore yet because make sure that you
Prasanna Malaiyandi:
have put this as part of your plan.
Prasanna Malaiyandi:
Make sure that once you have that.
Prasanna Malaiyandi:
You are now immediately going to start backing up.
Prasanna Malaiyandi:
Don't be the person that got the environment up and running and then
Prasanna Malaiyandi:
something else bad happened, right?
Prasanna Malaiyandi:
Because there could be, for example, a second part, there could be a
Prasanna Malaiyandi:
second wave of a ransomware attack.
Prasanna Malaiyandi:
Um, so yeah, make sure that you've got that, that replacement,
Prasanna Malaiyandi:
the, the backup, um, of the
Prasanna Malaiyandi:
and, and Curtis, I guess the question for you is
Prasanna Malaiyandi:
when you talked about failing over, like potentially using cloud as your
Prasanna Malaiyandi:
infrastructure, make sure if that requires using a different backup
Prasanna Malaiyandi:
solution, that you are trained on it, you've procured it, right.
Prasanna Malaiyandi:
You know how to use it
Prasanna Malaiyandi:
W. Curtis Preston: Mm-Hmm.
Prasanna Malaiyandi:
you need to start protecting if you
Prasanna Malaiyandi:
fail over into the cloud.
Prasanna Malaiyandi:
W. Curtis Preston: Yeah.
Prasanna Malaiyandi:
Great, great.
Prasanna Malaiyandi:
So if you're, if you're currently have an on-prem infrastructure and
Prasanna Malaiyandi:
now you're using cloud as your, uh, recovery mechanism, then make sure you
Prasanna Malaiyandi:
already know how to protect the cloud.
Prasanna Malaiyandi:
Right.
Prasanna Malaiyandi:
Uh, because it, the cloud is not magic, right?
Prasanna Malaiyandi:
So it's just somebody else's computer.
Prasanna Malaiyandi:
Um, so the next is about recovery requirements, right?
Prasanna Malaiyandi:
If we are making a plan, then you need to know what your requirements are.
Prasanna Malaiyandi:
What are where, what is your RTO and RPO, your recovery time
Prasanna Malaiyandi:
objective and your recovery point
Prasanna Malaiyandi:
Isn't it zero?
Prasanna Malaiyandi:
Zero for
Prasanna Malaiyandi:
everything?
Prasanna Malaiyandi:
W. Curtis Preston: yeah, zero.
Prasanna Malaiyandi:
Well, you, yeah.
Prasanna Malaiyandi:
Thi this is, you know, I, I think we've done an entire episode on how
Prasanna Malaiyandi:
to come up with the RTO and RPO.
Prasanna Malaiyandi:
The, um, the challenge there, you know, when, when you go to somebody, when
Prasanna Malaiyandi:
you, when you go to a business owner, because as we often say, this is not,
Prasanna Malaiyandi:
these are not numbers that should come from you, they should come from the
Prasanna Malaiyandi:
businesses, the business units, the, uh, or whatever it is you call in, in
Prasanna Malaiyandi:
a, in a governmental world, um, you know, the, the business units that are
Prasanna Malaiyandi:
going to be affected by the disaster.
Prasanna Malaiyandi:
They should be the ones who come up and say, well, uh, you know,
Prasanna Malaiyandi:
these are our requirements.
Prasanna Malaiyandi:
And yes, every single one of them, when you ask them, they will say zero and zero.
Prasanna Malaiyandi:
And then you come back to 'em and you say, okay, well we can do zero and zero.
Prasanna Malaiyandi:
It's going to cost $1 billion.
Prasanna Malaiyandi:
And, and then they go, oh, well we can't afford zero and zero.
Prasanna Malaiyandi:
So what can you afford?
Prasanna Malaiyandi:
Right?
Prasanna Malaiyandi:
And so you, you work, you know, but you, you, you have
Prasanna Malaiyandi:
to know what your RTO and RPO.
Prasanna Malaiyandi:
Are going into, uh, doing a disaster.
Prasanna Malaiyandi:
and you also set the expectations that way too, right?
Prasanna Malaiyandi:
Because now everyone signed off before the disaster happens that Yes, in order
Prasanna Malaiyandi:
to recover this application, because I'm willing to spend X amount of dollars, I
Prasanna Malaiyandi:
expect that it will take 12 hours or 24 hours or whatever it is, and that's what
Prasanna Malaiyandi:
we've all agreed to and signed off on.
Prasanna Malaiyandi:
W. Curtis Preston: Yeah, exactly.
Prasanna Malaiyandi:
Uh, the other thing to just remember about RTO, especially RTO, is that
Prasanna Malaiyandi:
that's fully functioning, restore, you know, of the environment, right?
Prasanna Malaiyandi:
So that means that the application was working here.
Prasanna Malaiyandi:
And now it's fully the application that I need and the other
Prasanna Malaiyandi:
applications that it needs.
Prasanna Malaiyandi:
Those are all running and functioning.
Prasanna Malaiyandi:
It doesn't just mean the restore.
Prasanna Malaiyandi:
A lot of people focus on just the restore and just remember that that's
Prasanna Malaiyandi:
literally only one part of the, of
Prasanna Malaiyandi:
And, and when you say restore, restore of the data, right?
Prasanna Malaiyandi:
W. Curtis Preston: Restore of the data, right?
Prasanna Malaiyandi:
Yeah.
Prasanna Malaiyandi:
Well even restore of the, you know, the server and getting
Prasanna Malaiyandi:
the server up and running.
Prasanna Malaiyandi:
You're like, okay, the server's up and running.
Prasanna Malaiyandi:
Right.
Prasanna Malaiyandi:
Is the application up and running?
Prasanna Malaiyandi:
Right.
Prasanna Malaiyandi:
Uh, which may, which may res require, um,
Prasanna Malaiyandi:
Prasanna Malaiyandi: Coordination with other,
Prasanna Malaiyandi:
W. Curtis Preston: other applications, right?
Prasanna Malaiyandi:
Yeah.
Prasanna Malaiyandi:
Which is a great way to lead into the next one, which is
Prasanna Malaiyandi:
what are your recovery priorities and prerequisites, because like you mentioned,
Prasanna Malaiyandi:
Curtis, in order to get this application functioning, I might need five other
Prasanna Malaiyandi:
applications to also be up and running.
Prasanna Malaiyandi:
So I need to make sure that those five are up before I can ensure that
Prasanna Malaiyandi:
this application is up and running.
Prasanna Malaiyandi:
So.
Prasanna Malaiyandi:
Understanding the dependencies and also the priorities.
Prasanna Malaiyandi:
Like, is Curtis' laptop the most important thing in the
Prasanna Malaiyandi:
business to bring up and running?
Prasanna Malaiyandi:
Or is it really our billing system, right?
Prasanna Malaiyandi:
Or our e-commerce site, right?
Prasanna Malaiyandi:
And so understanding what is critical to get the business running is important
Prasanna Malaiyandi:
because you're not gonna have time to bring everything up at the same time.
Prasanna Malaiyandi:
W. Curtis Preston: Well, if you were following Curtis's usual
Prasanna Malaiyandi:
suggestions, you weren't using active directory for your backup server.
Prasanna Malaiyandi:
And so the only thing that you need to get into the backup server is the password
Prasanna Malaiyandi:
manager that's running on Curtis's laptop.
Prasanna Malaiyandi:
I'm gonna say yes, Curtis' laptop Being restored is
Prasanna Malaiyandi:
What happens if Curtis' laptop is infected
Prasanna Malaiyandi:
with ransomware encrypted?
Prasanna Malaiyandi:
W. Curtis Preston: Well, then you're screwed.
Prasanna Malaiyandi:
Right?
Prasanna Malaiyandi:
That's just, that's what, that's what we, we had a, we had a good episode on that.
Prasanna Malaiyandi:
Right?
Prasanna Malaiyandi:
The, you know, restoring from, from nothing.
Prasanna Malaiyandi:
Right.
Prasanna Malaiyandi:
How do you, how do you go from nothing?
Prasanna Malaiyandi:
Um, the, um, the next one, uh, is about people.
Prasanna Malaiyandi:
This is an interesting one because, um.
Prasanna Malaiyandi:
You could look at this, uh, you know, basically how, how
Prasanna Malaiyandi:
will you get the people, right?
Prasanna Malaiyandi:
Um, you need to prepare for a number of eventualities.
Prasanna Malaiyandi:
One of them are that the people are either not alive,
Prasanna Malaiyandi:
Prasanna Malaiyandi: available, I think is a.
Prasanna Malaiyandi:
W. Curtis Preston: They're not, well, they're not, they're
Prasanna Malaiyandi:
not alive, or they're not.
Prasanna Malaiyandi:
They can't access your environment, right?
Prasanna Malaiyandi:
So depending on what we're talking about, especially in this remote
Prasanna Malaiyandi:
world that we live in now, right?
Prasanna Malaiyandi:
Where a lot of people are working remote, the first thing you need are,
Prasanna Malaiyandi:
you know, uh, people that can, you know, the, the people at the keyboard.
Prasanna Malaiyandi:
So, um, how will they get to that keyboard?
Prasanna Malaiyandi:
How will they, um, you know, well, how will they get to
Prasanna Malaiyandi:
Or, or a virtual
Prasanna Malaiyandi:
W. Curtis Preston: you're talking about.
Prasanna Malaiyandi:
Or virtual keyboard.
Prasanna Malaiyandi:
Right.
Prasanna Malaiyandi:
Um, the, um, how, how, how will that happen?
Prasanna Malaiyandi:
Right.
Prasanna Malaiyandi:
And you need to include all the things that could happen if you live in a, you
Prasanna Malaiyandi:
know, if we're talking about natural disasters that you're protecting from, you
Prasanna Malaiyandi:
need to say, oh, well, the roads are out.
Prasanna Malaiyandi:
The roads in between A and B are out, the roads are where they're snowed over.
Prasanna Malaiyandi:
Uh, we had a flood and you can't get from A to B.
Prasanna Malaiyandi:
So these are things that you need to take into account.
Prasanna Malaiyandi:
I think there's a great episode there where we have the, we interviewed
Prasanna Malaiyandi:
the, the guy that was, you know, the, the, the guy on the ground for that
Prasanna Malaiyandi:
disaster at the Caribbean island.
Prasanna Malaiyandi:
And, uh, he talked about all the, the critical infrastructure
Prasanna Malaiyandi:
that was not available to him.
Prasanna Malaiyandi:
Um, and, um.
Prasanna Malaiyandi:
It's like all those
Prasanna Malaiyandi:
W. Curtis Preston: Yeah.
Prasanna Malaiyandi:
Yeah.
Prasanna Malaiyandi:
You,
Prasanna Malaiyandi:
Yeah.
Prasanna Malaiyandi:
W. Curtis Preston: yeah.
Prasanna Malaiyandi:
And, and also speaking of the people, you, you do need to figure that out,
Prasanna Malaiyandi:
thanks for bringing that up, is, uh, you, once you get the people in,
Prasanna Malaiyandi:
they're probably not gonna leave.
Prasanna Malaiyandi:
So, so figure out where they're gonna, um, sleep and also figure out how they're
Prasanna Malaiyandi:
gonna eat and, you know, use the bathroom showers, uh, hopefully at some point.
Prasanna Malaiyandi:
Um,
Prasanna Malaiyandi:
one of the things, I don't know if where it falls under, maybe
Prasanna Malaiyandi:
it's like part of the infrastructure ones, maybe it's part of the people one, do
Prasanna Malaiyandi:
you remember the episode about the dire show that hit and the land hurricane?
Prasanna Malaiyandi:
Right.
Prasanna Malaiyandi:
And one of the problems, and I can't, unfortunately, I can't
Prasanna Malaiyandi:
remember the guest's name, but one of the things you mentioned was
Prasanna Malaiyandi:
they had no internet connectivity.
Prasanna Malaiyandi:
So he literally sent people to the next town over so they can sit in a
Prasanna Malaiyandi:
Starbucks, connect in and manage their infrastructure and systems because no one
Prasanna Malaiyandi:
in that area had access internet access.
Prasanna Malaiyandi:
W. Curtis Preston: Yeah.
Prasanna Malaiyandi:
Um, and we'll just go to a Starbucks is probably not a good,
Prasanna Malaiyandi:
I don't know if they actually went to a Starbucks, but
Prasanna Malaiyandi:
W. Curtis Preston: So, no, I'm just saying that's what he did, uh,
Prasanna Malaiyandi:
because that was his only choice.
Prasanna Malaiyandi:
Uh, just make sure you, you, you figure that aspect, that figure, that aspect out.
Prasanna Malaiyandi:
Um, and then we have a.
Prasanna Malaiyandi:
just another quick thing is also realize that
Prasanna Malaiyandi:
the people you're relying on, they might have other priorities as well.
Prasanna Malaiyandi:
For instance, if it's an earthquake in the local region, maybe they're
Prasanna Malaiyandi:
taking care of family, maybe they're dealing with other things.
Prasanna Malaiyandi:
I know we had the episode with the person talking about the female
Prasanna Malaiyandi:
response right to, and how to prepare yourself for disasters.
Prasanna Malaiyandi:
And that's something that we don't always think about, but you should
Prasanna Malaiyandi:
also take into consideration.
Prasanna Malaiyandi:
W. Curtis Preston: Yeah, absolutely.
Prasanna Malaiyandi:
You mean people don't prioritize company above all us?
Prasanna Malaiyandi:
We would like to think that, but yes, real life happens.
Prasanna Malaiyandi:
W. Curtis Preston: Yeah.
Prasanna Malaiyandi:
All right.
Prasanna Malaiyandi:
You get the next one.
Prasanna Malaiyandi:
All right, so the next, and I like this one, Curtis,
Prasanna Malaiyandi:
because I always have a, I always remember your story, you tell, and it's really
Prasanna Malaiyandi:
how good is your documentation, right?
Prasanna Malaiyandi:
And.
Prasanna Malaiyandi:
Only as, because like we talked about people, if people aren't there,
Prasanna Malaiyandi:
right, who can now operate and make sure and recover your environment?
Prasanna Malaiyandi:
Or if people are there, who created it, who's the one who's coordinating
Prasanna Malaiyandi:
and making sure, okay, are all the steps in order to bring up this
Prasanna Malaiyandi:
application, I need to go talk.
Prasanna Malaiyandi:
I.
Prasanna Malaiyandi:
And make sure that these five other applications are up and running, that
Prasanna Malaiyandi:
I have the networking there that I've already pre-configured the compute, right?
Prasanna Malaiyandi:
There's a whole bunch of steps and so making sure everything's documented.
Prasanna Malaiyandi:
'cause the last thing you wanna do is have a disaster and be like,
Prasanna Malaiyandi:
oh, I forgot steps four and six.
Prasanna Malaiyandi:
And those are critical for bringing up this application and sitting
Prasanna Malaiyandi:
there and trying to troubleshoot in the midst of a crisis.
Prasanna Malaiyandi:
W. Curtis Preston: Yeah.
Prasanna Malaiyandi:
Um, this was something that was basically pounded into me very early in my career.
Prasanna Malaiyandi:
The bank that I worked with, um, had, um, I.
Prasanna Malaiyandi:
You had to have, because it was a bank, the the SEC would, you know,
Prasanna Malaiyandi:
required us to, to do these things.
Prasanna Malaiyandi:
Right.
Prasanna Malaiyandi:
And, uh, we had documentation, and I remember a fight over what
Prasanna Malaiyandi:
the documentation was going to be created in the, the, the company's
Prasanna Malaiyandi:
standard was word perfect.
Prasanna Malaiyandi:
And we had this one guy that really liked Microsoft Word.
Prasanna Malaiyandi:
Uh, we all know that, how that worked out.
Prasanna Malaiyandi:
I mean word, word, uh, took over.
Prasanna Malaiyandi:
But, and that was like, he's like, we're perfect stakes.
Prasanna Malaiyandi:
And you know, Microsoft Word is so much better.
Prasanna Malaiyandi:
And we're like, it doesn't matter.
Prasanna Malaiyandi:
You gotta do the documentation in the format that we, that we use.
Prasanna Malaiyandi:
Right.
Prasanna Malaiyandi:
And um, and part of that was, well, a big part of that was that we were a.
Prasanna Malaiyandi:
Uh, like not everybody had a, like a, nobody had a laptop, but not everybody
Prasanna Malaiyandi:
necessarily had a, a desktop that ran Windows where you could run these things.
Prasanna Malaiyandi:
We actually used word perfect because it ran on Unix-based infrastructure.
Prasanna Malaiyandi:
Right.
Prasanna Malaiyandi:
Uh, and we could do shared documentation.
Prasanna Malaiyandi:
So.
Prasanna Malaiyandi:
Yeah, the important thing is to write this documentation and test it, test it, test
Prasanna Malaiyandi:
it, test it, and make sure that you te that you have other people that test the
Prasanna Malaiyandi:
documentation that you wrote because they will find the things that you left out.
Prasanna Malaiyandi:
Um, also make sure to think about I.
Prasanna Malaiyandi:
How and where you keep this documentation do not out rule the
Prasanna Malaiyandi:
idea of making multiple printed copies of that, of that documentation.
Prasanna Malaiyandi:
The downside of that is, you know, you gotta kill a couple of
Prasanna Malaiyandi:
trees every time you update it.
Prasanna Malaiyandi:
But, uh, maybe use loose leaf notebooks so that you can, uh, easily update a
Prasanna Malaiyandi:
section of the plan if you change it.
Prasanna Malaiyandi:
But, uh, and have it in multiple formats like PD, F, and, you know, have it
Prasanna Malaiyandi:
available in places where you can easily access it, uh, paper, online, maybe
Prasanna Malaiyandi:
even, you know, and, and a, a and a truly online, like not anything in your, in your
Prasanna Malaiyandi:
infrastructure, something in the cloud somewhere where you can possibly get it.
Prasanna Malaiyandi:
Just make sure all of that stuff that's, you know, authenticated and secure
Prasanna Malaiyandi:
and don't put passwords in that documentation please.
Prasanna Malaiyandi:
W. Curtis Preston: Don't put passwords on that documentation.
Prasanna Malaiyandi:
Exactly.
Prasanna Malaiyandi:
Um, another thing, uh, here that I've got is, you know how much of this Dr.
Prasanna Malaiyandi:
Runbook that you have, how much of it is docu is, um, is automated?
Prasanna Malaiyandi:
Because the more you can automate.
Prasanna Malaiyandi:
The more that a disaster recovery, uh, actual execution is going to work.
Prasanna Malaiyandi:
And, um, you know, so the question is in that DR plan, just make sure you
Prasanna Malaiyandi:
understand that level of automation that you have or don't have, uh, and
Prasanna Malaiyandi:
continually strive for more automation.
Prasanna Malaiyandi:
You know, because a two things.
Prasanna Malaiyandi:
One is your.
Prasanna Malaiyandi:
You're more than likely to have a successful recovery
Prasanna Malaiyandi:
if you have automation.
Prasanna Malaiyandi:
And second, it makes testing the recovery a, a whole lot easier.
Prasanna Malaiyandi:
Again, I'll go back to when I used to have to do this in the field and
Prasanna Malaiyandi:
what we did was we, we threw, you know, we blew away a weekend, right?
Prasanna Malaiyandi:
We'd have to come in at early Saturday morning and we would start the disaster
Prasanna Malaiyandi:
recovery test and basically it took the entire weekend and it was hell.
Prasanna Malaiyandi:
Right.
Prasanna Malaiyandi:
I was just like, and, and basically the worst part was that there were
Prasanna Malaiyandi:
two things that really stunk about it.
Prasanna Malaiyandi:
One was if you were the one at the keyboard, super high stress, right?
Prasanna Malaiyandi:
Or if you were me, it wasn't me at the keyboard, but I was back there.
Prasanna Malaiyandi:
I was the one that had written the documentation that there
Prasanna Malaiyandi:
following that hopefully works.
Prasanna Malaiyandi:
Uh, and then you got the boss and they're stressed out whether or
Prasanna Malaiyandi:
not this is gonna look, you know, whether or not this is gonna work
Prasanna Malaiyandi:
because then they'll look good or bad.
Prasanna Malaiyandi:
Um, and then meanwhile you got a whole bunch of other people that are just.
Prasanna Malaiyandi:
This is so boring.
Prasanna Malaiyandi:
They're just standing around just in case they're needed, hoping that
Prasanna Malaiyandi:
they're not needed, but they're
Prasanna Malaiyandi:
Just in case.
Prasanna Malaiyandi:
W. Curtis Preston: standing around.
Prasanna Malaiyandi:
And we didn't have, we didn't have YouTube either.
Prasanna Malaiyandi:
So, uh, you know, you needed to, you needed to bring a
Prasanna Malaiyandi:
book, uh, to have something to
Prasanna Malaiyandi:
So here's a question, Curtis.
Prasanna Malaiyandi:
So from your documentation, from this experience at the bank, and I'm guessing
Prasanna Malaiyandi:
that a lot of it was probably more manual than automated, just given the timing.
Prasanna Malaiyandi:
W. Curtis Preston: Yeah, there was no
Prasanna Malaiyandi:
Um, how often were you successful at really
Prasanna Malaiyandi:
running through your entire Dr.
Prasanna Malaiyandi:
Playbook?
Prasanna Malaiyandi:
I don't mean to put you on the spot,
Prasanna Malaiyandi:
W. Curtis Preston: So,
Prasanna Malaiyandi:
I think it's
Prasanna Malaiyandi:
W. Curtis Preston: yeah.
Prasanna Malaiyandi:
So if you define success as.
Prasanna Malaiyandi:
We had the documentation.
Prasanna Malaiyandi:
Someone other than Curtis followed the documentation, and we got to
Prasanna Malaiyandi:
the end of that recovery without having to consult Curtis on what
Prasanna Malaiyandi:
he really meant by this sentence.
Prasanna Malaiyandi:
The answer is never.
Prasanna Malaiyandi:
Yeah.
Prasanna Malaiyandi:
W. Curtis Preston: Right.
Prasanna Malaiyandi:
So the goal of every DR test was to, um, you know, was to help essentially
Prasanna Malaiyandi:
make the, the documentation better.
Prasanna Malaiyandi:
And, um, the only way really to do that is to do it, you know, to
Prasanna Malaiyandi:
fire it in anger, as we say, right?
Prasanna Malaiyandi:
Um, and, and so we just, we just set that, that was our, um, we,
Prasanna Malaiyandi:
we considered that a success if we found something that we could fix,
Prasanna Malaiyandi:
Better to do it while testing rather than in when you need it.
Prasanna Malaiyandi:
W. Curtis Preston: Yeah, it was, it was always successful in that we
Prasanna Malaiyandi:
always restored all the data that we were supposed to and the applications
Prasanna Malaiyandi:
that we were supposed to restore, uh, because my backups worked.
Prasanna Malaiyandi:
Dang it.
Prasanna Malaiyandi:
Uh, but, uh, it was not successful by that other metric.
Prasanna Malaiyandi:
I think this goes to the last point in the plan, right, which
Prasanna Malaiyandi:
is make sure that it has been tested.
Prasanna Malaiyandi:
Because only because you guys were doing this periodic testing, did you
Prasanna Malaiyandi:
realize, hey, there was some issues with the documentation, or in other
Prasanna Malaiyandi:
cases there might have been issues with.
Prasanna Malaiyandi:
Uh, I know when we had the episode with the disaster on the island and
Prasanna Malaiyandi:
the tropics, right, um, that one of the issues they had was they had satellite
Prasanna Malaiyandi:
connectivity back to the mainland, to the us and that's where their active
Prasanna Malaiyandi:
directory infrastructure was running.
Prasanna Malaiyandi:
And so when they got hit by this hurricane, they lost
Prasanna Malaiyandi:
connectivity, which means that their systems weren't functioning.
Prasanna Malaiyandi:
And so if they had done that sort of testing, they would've rec realized
Prasanna Malaiyandi:
it earlier on and been able to have a solution rather than putting a
Prasanna Malaiyandi:
person on a plane with some backups of active directory to fly out to
Prasanna Malaiyandi:
the island to then do the restore so they can get up and running.
Prasanna Malaiyandi:
W. Curtis Preston: Yeah.
Prasanna Malaiyandi:
Testing, testing, testing.
Prasanna Malaiyandi:
Right.
Prasanna Malaiyandi:
And this, and again, this is why I like automation, is you can test.
Prasanna Malaiyandi:
You can, the more you automate, the more often you can test and
Prasanna Malaiyandi:
the more you know, you can know that things are actually working.
Prasanna Malaiyandi:
Um, yeah, there, there, there's just no substitute for
Prasanna Malaiyandi:
but I think one of the things to realize is you don't need to
Prasanna Malaiyandi:
strive for a hundred percent automation.
Prasanna Malaiyandi:
It's okay in your first versions to have a bunch of things manual and
Prasanna Malaiyandi:
improve on it over time because you don't know if the audit like what to
Prasanna Malaiyandi:
automate until you've actually done the manual process and laid it all out.
Prasanna Malaiyandi:
So start somewhere, have something there and then figure out, okay, what
Prasanna Malaiyandi:
are the pieces I can start automating?
Prasanna Malaiyandi:
How should I automate it?
Prasanna Malaiyandi:
And all the rest.
Prasanna Malaiyandi:
W. Curtis Preston: Yeah, absolutely.
Prasanna Malaiyandi:
Absolutely.
Prasanna Malaiyandi:
Do you have a recommendation on how to test?
Prasanna Malaiyandi:
Because I know sometimes testing is expensive from an
Prasanna Malaiyandi:
infrastructure perspective.
Prasanna Malaiyandi:
I know you went in on a Saturday when the bank was closed, right?
Prasanna Malaiyandi:
Or they, it wasn't as busy, and so you were able to do the testing.
Prasanna Malaiyandi:
A lot of people don't necessarily have the infrastructure in order to be able
Prasanna Malaiyandi:
to do these periodic testing because you probably at least want to test your DR.
Prasanna Malaiyandi:
Environment.
Prasanna Malaiyandi:
At least once every couple of months just to make sure you're not
Prasanna Malaiyandi:
missing anything and all the rest.
Prasanna Malaiyandi:
So is there a better way people can do DR testing?
Prasanna Malaiyandi:
I.
Prasanna Malaiyandi:
W. Curtis Preston: Well, I think this is a great time to discuss like
Prasanna Malaiyandi:
my opinions about the use of the cloud, if at all possible, right?
Prasanna Malaiyandi:
The more you can utilize the cloud, the more it can basically set aside
Prasanna Malaiyandi:
some of these, uh, budget constraints.
Prasanna Malaiyandi:
You're still gonna need to take time and there's, there's.
Prasanna Malaiyandi:
Um, there's nothing I can tell you there other than that, you know, a backup in Dr.
Prasanna Malaiyandi:
That hasn't been tested isn't really a backup in Dr.
Prasanna Malaiyandi:
And, uh, you, you, you just won't know.
Prasanna Malaiyandi:
If it's any good until you really, really, really need it.
Prasanna Malaiyandi:
And, uh, please don't do what we had the one guest do where he wiped
Prasanna Malaiyandi:
out his entire data center and then, and then did the restore.
Prasanna Malaiyandi:
Oh my lord, that story just gave me like a heart attack.
Prasanna Malaiyandi:
Um, yeah, I would say use the cloud as much as possible.
Prasanna Malaiyandi:
I realize it's maybe not possible for everything, but, um, you know, um.
Prasanna Malaiyandi:
Yeah.
Prasanna Malaiyandi:
Another thing that you can do is you can use new infrastructure to test the
Prasanna Malaiyandi:
recovery of current infrastructure, right?
Prasanna Malaiyandi:
Like so before you put a particular set of like if you're buying physical servers
Prasanna Malaiyandi:
before you buy, before you actually put those servers into production, consider
Prasanna Malaiyandi:
using those servers to conduct DR tests.
Prasanna Malaiyandi:
That's a thought.
Prasanna Malaiyandi:
I don't know, do you have any other, other ideas
Prasanna Malaiyandi:
I think, yeah, that's pretty much it.
Prasanna Malaiyandi:
I know it gets expensive and I think that's one of the things that people
Prasanna Malaiyandi:
sort of are like, oh, it's so costly from a time and money perspective that they
Prasanna Malaiyandi:
don't necessarily know how to approach it.
Prasanna Malaiyandi:
But I do like what you talked about.
Prasanna Malaiyandi:
Try to leverage infrastructure.
Prasanna Malaiyandi:
You could spin up and spin down quickly that doesn't require you to
Prasanna Malaiyandi:
own it and manage it the entire time.
Prasanna Malaiyandi:
Makes sense.
Prasanna Malaiyandi:
W. Curtis Preston: Right.
Prasanna Malaiyandi:
Um, so we'll, we'll finish with, uh, telling the story that you alluded
Prasanna Malaiyandi:
to earlier, and that is, uh, go back.
Prasanna Malaiyandi:
Um.
Prasanna Malaiyandi:
About, uh, 29 years, and I was, um, I was in the, you know,
Prasanna Malaiyandi:
I was working at the bank.
Prasanna Malaiyandi:
I had all the documentation I had for restorers.
Prasanna Malaiyandi:
It wasn't a Dr per se, but it was a recovery of a critical app.
Prasanna Malaiyandi:
And, uh, my daughter, uh, had just been born and I was in the hospital.
Prasanna Malaiyandi:
Um, you know, with my wife, you know, you know the whole classic
Prasanna Malaiyandi:
thing of like, father, do you know, son, daughter, brand new baby there.
Prasanna Malaiyandi:
And you're, you know, you're having that wonderful moment.
Prasanna Malaiyandi:
And the phone in my wife's hotel room rang.
Prasanna Malaiyandi:
Um, again, for those of you that are, are born in this world, this was a world
Prasanna Malaiyandi:
where we didn't have cell phones, right?
Prasanna Malaiyandi:
You meant the Hoss,
Prasanna Malaiyandi:
W. Curtis Preston: I had left
Prasanna Malaiyandi:
Prasanna Malaiyandi: meant the hospital room,
Prasanna Malaiyandi:
W. Curtis Preston: What's that?
Prasanna Malaiyandi:
Hotel
Prasanna Malaiyandi:
W. Curtis Preston: say?
Prasanna Malaiyandi:
Did I say a hotel room?
Prasanna Malaiyandi:
Yeah.
Prasanna Malaiyandi:
W. Curtis Preston: Um, so we were.
Prasanna Malaiyandi:
Uh, yeah.
Prasanna Malaiyandi:
So no cell phone.
Prasanna Malaiyandi:
And I had even left my beeper behind, right?
Prasanna Malaiyandi:
That was the way that you got in touch.
Prasanna Malaiyandi:
I'm like, I'm going to see my daughter and my wife.
Prasanna Malaiyandi:
I, you know, I, I don't need the beeper.
Prasanna Malaiyandi:
Uh, and what had happened while I was gone is that there had been this like really
Prasanna Malaiyandi:
big restore that had gone down and they.
Prasanna Malaiyandi:
Called me in my wife's hospital room.
Prasanna Malaiyandi:
Right.
Prasanna Malaiyandi:
They called the hospital 'cause they knew the hospital.
Prasanna Malaiyandi:
'cause the hospital was literally within line of sight to the, to the bank where
Prasanna Malaiyandi:
I worked, uh, where I worked, right.
Prasanna Malaiyandi:
This was Christian Hospital in Newark, Delaware.
Prasanna Malaiyandi:
Uh, 'cause I was at, what at that time was MBNA and it was just
Prasanna Malaiyandi:
literally right down the road.
Prasanna Malaiyandi:
The, the phone rang and, and I answered it, and it was somebody from
Prasanna Malaiyandi:
the bank, and they wanted to know if I could help them with this restore
Prasanna Malaiyandi:
that, you know, they were doing.
Prasanna Malaiyandi:
And I said, well, did you, did you like look at the document?
Prasanna Malaiyandi:
Because I made, I, I made such a thing about documentation.
Prasanna Malaiyandi:
Right.
Prasanna Malaiyandi:
Did you look at the documentation and was there a problem with the documentation?
Prasanna Malaiyandi:
Did you, you know, is there something, a question that I, a, you know,
Prasanna Malaiyandi:
that I, that I need to answer?
Prasanna Malaiyandi:
They basically started, um, they started saying, well, no, I hadn't
Prasanna Malaiyandi:
really looked at the documentation.
Prasanna Malaiyandi:
I just called you first.
Prasanna Malaiyandi:
And I just, I just hung up.
Prasanna Malaiyandi:
I just, I just, just like, I can't believe that you just called
Prasanna Malaiyandi:
me in my wife's hospital room.
Prasanna Malaiyandi:
Just after she'd given birth to our first child because of a stupid restore.
Prasanna Malaiyandi:
No matter how, you know, a a, you know, without even looking at the documentation.
Prasanna Malaiyandi:
This is why you need to have documentation.
Prasanna Malaiyandi:
This is why you need to make sure it's followed.
Prasanna Malaiyandi:
But of course, if you've got stupid people that don't wanna read
Prasanna Malaiyandi:
it, there's not much you can do.
Prasanna Malaiyandi:
But, um, all right.
Prasanna Malaiyandi:
Well, uh, as always, thanks a lot persona
Prasanna Malaiyandi:
thanks Curtis.
Prasanna Malaiyandi:
It's always fun.
Prasanna Malaiyandi:
W. Curtis Preston: And, uh, thank you to the listeners.
Prasanna Malaiyandi:
We'd be nothing without you.
Prasanna Malaiyandi:
That is a wrap.
Listen On
