Crafting the Perfect Disaster Recovery Runbook
In this information-packed episode, backup guru W. Curtis Preston and expert guest Prasanna Malaiyandi explore the elements of crafting an effective disaster recovery (DR) runbook. They discuss how a properly structured runbook eliminates confusion during crises by clearly outlining responsibilities and contacts. From making runbooks accessible and absorbed to keeping them actionable with constant updates and tests, they share the 8 critical factors for DR success. Tune in to transform your backup admins into confident cyber recovery heroes with battle-tested runbooks.
Speaker:
W. Curtis Preston: If your organization was hit with a disaster or cyber attack
Speaker:
tomorrow, would each person know exactly what their responsibilities were?
Speaker:
If your answer is not an unequivocal yes, then you need a Dr.
Speaker:
Runbook.
Speaker:
What is the Dr.
Speaker:
Runbook?
Speaker:
Well, you're in the right place.
Speaker:
It's the culmination of all the planning and designing that you did
Speaker:
when putting together your DR system.
Speaker:
When a disaster strikes, you should simply be able to pull it out and follow it.
Speaker:
Every person in the org should know what their role is and who
Speaker:
they should be communicating with.
Speaker:
Our goal here is to turn unappreciated backup admins into cyber recovery heroes.
Speaker:
So let's talk about how to make a runbook and what it should look like.
Speaker:
This is the backup Wrap up.
Speaker:
Welcome to the show.
Speaker:
I'm your host, w Curtis Preston.
Speaker:
And I have with me my celebration consultant Prasanna.
Speaker:
Malaiyandi.
Speaker:
How's it going?
Speaker:
Prasanna,
Prasanna Malaiyandi:
I'm doing well.
Prasanna Malaiyandi:
What are we celebrating?
Prasanna Malaiyandi:
W. Curtis Preston: think, you know, we're celebrating my, my big
Prasanna Malaiyandi:
new contract that I landed today
Prasanna Malaiyandi:
Woo.
Prasanna Malaiyandi:
W. Curtis Preston: Very excited about that.
Prasanna Malaiyandi:
Um, we.
Prasanna Malaiyandi:
Thanks.
Prasanna Malaiyandi:
Uh, I'm, I'm gonna be working a lot with a new company and helping them to spread
Prasanna Malaiyandi:
their word about what they do and, we'll talk more about that in later days,
Prasanna Malaiyandi:
but today is the day that I actually heard that I got the contract, so I'm
Prasanna Malaiyandi:
Woo hoo.
Prasanna Malaiyandi:
W. Curtis Preston: What do you, what do you think I should do?
Prasanna Malaiyandi:
Uh, I think you should go for Korean barbecue.
Prasanna Malaiyandi:
W. Curtis Preston: By myself without the rest of the family.
Prasanna Malaiyandi:
Yeah, yeah.
Prasanna Malaiyandi:
No, you're right.
Prasanna Malaiyandi:
That's okay.
Prasanna Malaiyandi:
Fine.
Prasanna Malaiyandi:
Then how about going and watching a movie?
Prasanna Malaiyandi:
W. Curtis Preston: Oh yeah, maybe I'll go, maybe I'll go do just that.
Prasanna Malaiyandi:
I'll do that.
Prasanna Malaiyandi:
Um, yeah.
Prasanna Malaiyandi:
Well 'cause, 'cause you know, you know, my wife's down in San
Prasanna Malaiyandi:
Diego training to be a trainer.
Prasanna Malaiyandi:
I.
Prasanna Malaiyandi:
To train poll workers.
Prasanna Malaiyandi:
So she's down in San Diego with her mom, uh, and my, the rest of
Prasanna Malaiyandi:
my family is spread to the wind.
Prasanna Malaiyandi:
So yeah, if I went to Korean barbecue, I would be going there all by myself
Prasanna Malaiyandi:
and they would never speak to me again.
Prasanna Malaiyandi:
Given, given
Prasanna Malaiyandi:
Just don't tell them.
Prasanna Malaiyandi:
W. Curtis Preston: thing to Do What's that?
Prasanna Malaiyandi:
Just don't tell them.
Prasanna Malaiyandi:
Do you think they listen to the podcast?
Prasanna Malaiyandi:
Do you think they know
Prasanna Malaiyandi:
W. Curtis Preston: I just don't tell.
Prasanna Malaiyandi:
I would feel guilty, uh, but
Prasanna Malaiyandi:
I wouldn't feel guilty going, uh, to go see a movie because I do have the Regal
Prasanna Malaiyandi:
Unlimited Pass so I can go see a movie.
Prasanna Malaiyandi:
Do it.
Prasanna Malaiyandi:
W. Curtis Preston: So, uh, we've been talking about DR for a while now,
Prasanna Malaiyandi:
and, you know, once you've decided, I mean, we talked so much about it.
Prasanna Malaiyandi:
I, I think this is our.
Prasanna Malaiyandi:
Sixth episode about, you know, basically putting together a DR plan.
Prasanna Malaiyandi:
You've got to decide what's in it.
Prasanna Malaiyandi:
You've got to decide what you know, uh, what your RTO and
Prasanna Malaiyandi:
RPO are for each application.
Prasanna Malaiyandi:
You need to decide whether or not you're going to do this yourself
Prasanna Malaiyandi:
or you're gonna use a service.
Prasanna Malaiyandi:
Where are you gonna do a cold site, hot site, warm site?
Prasanna Malaiyandi:
You can use the cloud, you can use on-prem stuff.
Prasanna Malaiyandi:
You gotta make all those decisions and hopefully you put a lot of
Prasanna Malaiyandi:
planning into this and hopefully you got a lot of input from the business
Prasanna Malaiyandi:
that this was not an IT driven.
Prasanna Malaiyandi:
I mean, it could be an IT driven, I suppose, but just really, you know,
Prasanna Malaiyandi:
I hope that you got input from all of the stakeholders, all of the.
Prasanna Malaiyandi:
People that will be impacted by this Dr.
Prasanna Malaiyandi:
Plan.
Prasanna Malaiyandi:
And you put it all together to make all of them as happy as possible, right?
Prasanna Malaiyandi:
you're gonna be the knight and shining armor right?
Prasanna Malaiyandi:
When something happens.
Prasanna Malaiyandi:
W. Curtis Preston: knight and shining armor.
Prasanna Malaiyandi:
The uh, and this will date me, but here I come to save the day.
Prasanna Malaiyandi:
No one's gonna get that reference.
Prasanna Malaiyandi:
Anyway.
Prasanna Malaiyandi:
Did you get that reference?
Prasanna Malaiyandi:
Mighty Mouse.
Prasanna Malaiyandi:
W. Curtis Preston: Mighty Mouse.
Prasanna Malaiyandi:
Yeah.
Prasanna Malaiyandi:
Good job.
Prasanna Malaiyandi:
Good job.
Prasanna Malaiyandi:
All right.
Prasanna Malaiyandi:
Um, so I, so, so when we've done all those decisions, when, when
Prasanna Malaiyandi:
you have an actual disaster, you should be able to pull out what.
Prasanna Malaiyandi:
I was just gonna say, you don't run around like
Prasanna Malaiyandi:
a chicken one with its head cut off scrambling and everyone just being
Prasanna Malaiyandi:
like, panic, panic, panic, panic.
Prasanna Malaiyandi:
W. Curtis Preston: Yeah, that is not what you're supposed to do.
Prasanna Malaiyandi:
Um, what are you gonna pull out?
Prasanna Malaiyandi:
Prasanna.
Prasanna Malaiyandi:
Well, what you're supposed to do is pull out your Dr.
Prasanna Malaiyandi:
Runbook, which has been created ahead of time.
Prasanna Malaiyandi:
Everyone has read, everyone knows about it, and you just
Prasanna Malaiyandi:
follow what it says in the Dr.
Prasanna Malaiyandi:
Runbook and you're good to go.
Prasanna Malaiyandi:
W. Curtis Preston: Yeah.
Prasanna Malaiyandi:
And, and I will say that, you know, we're talking mainly
Prasanna Malaiyandi:
here about disaster recovery.
Prasanna Malaiyandi:
A Dr.
Prasanna Malaiyandi:
Runbook fits within a cyber recovery runbook.
Prasanna Malaiyandi:
This is part of it.
Prasanna Malaiyandi:
This is the actual.
Prasanna Malaiyandi:
Putting everything back together.
Prasanna Malaiyandi:
Part if what we're talking about is recovering from a cyber attack,
Prasanna Malaiyandi:
you do have to address that first.
Prasanna Malaiyandi:
And, uh, we should talk about that.
Prasanna Malaiyandi:
We, we will talk about that in other episodes, but at this point, basically you
Prasanna Malaiyandi:
have decided, you know, you're going to restore your organization and everybody
Prasanna Malaiyandi:
should basically know just exactly.
Prasanna Malaiyandi:
Where they're going, what they're doing, what they're not doing, who they're
Prasanna Malaiyandi:
telling, who they're not telling.
Prasanna Malaiyandi:
Um, because you know, Dr.
Prasanna Malaiyandi:
You know, an actual disaster recovery does not happen in a vacuum.
Prasanna Malaiyandi:
What do you think about that?
Prasanna Malaiyandi:
Yeah, no, it, yeah, it doesn't happen in a vacuum.
Prasanna Malaiyandi:
There's usually other causes, and then it's also a very,
Prasanna Malaiyandi:
very, very high stress event.
Prasanna Malaiyandi:
Right, and so you wanna make sure that you're not thinking on the fly,
Prasanna Malaiyandi:
trying to figure out what to do, that everything has sort of been thought
Prasanna Malaiyandi:
ahead of time, so you just can execute.
Prasanna Malaiyandi:
It's like a doctor, right?
Prasanna Malaiyandi:
And yes, I know there will be cases that you need to worry
Prasanna Malaiyandi:
about, but a doctor, right?
Prasanna Malaiyandi:
They go to school, they do the procedures, they know what to do, and so they are
Prasanna Malaiyandi:
able to say, okay, I'm doing this, this, and this every single time.
Prasanna Malaiyandi:
W. Curtis Preston: Yeah, actually I, I may have a better
Prasanna Malaiyandi:
example to that and that was.
Prasanna Malaiyandi:
You know, I, I know I've told you the story, but I don't think I've told
Prasanna Malaiyandi:
the listeners, and that is the time that we lost an engine on a plane
Prasanna Malaiyandi:
when I was in flight on an MD 80.
Prasanna Malaiyandi:
And for those of you that don't know, an MD 80 only has two engines.
Prasanna Malaiyandi:
Right?
Prasanna Malaiyandi:
So we were on initial ascent and the um.
Prasanna Malaiyandi:
Uh, we, we lost an engine.
Prasanna Malaiyandi:
And for those of you that have never experienced that, which is probably
Prasanna Malaiyandi:
most of you, if not all of you, uh, if you've ever been in a car where
Prasanna Malaiyandi:
you've accidentally and you're, and you're sort of accelerating and
Prasanna Malaiyandi:
then you slipped out of gear, right?
Prasanna Malaiyandi:
Like, you, you, your leg bumped the, you know, the, uh, the stick shift or
Prasanna Malaiyandi:
the, uh, what, what do you call that?
Prasanna Malaiyandi:
The, the, the gear shift?
Prasanna Malaiyandi:
I don't know.
Prasanna Malaiyandi:
The,
Prasanna Malaiyandi:
you know, in an automatic.
Prasanna Malaiyandi:
Whatever.
Prasanna Malaiyandi:
Basically, suddenly you find yourself in neutral.
Prasanna Malaiyandi:
There's that sort of lunge that happens, right?
Prasanna Malaiyandi:
And, um, uh, if you've never felt that, then I, I don't
Prasanna Malaiyandi:
know what to, what to tell you.
Prasanna Malaiyandi:
But, but imagine that, but up in the air.
Prasanna Malaiyandi:
So suddenly you're luing, you're lunging forward, and you're diving, you're,
Prasanna Malaiyandi:
you're, you know, you're, you're rapidly.
Prasanna Malaiyandi:
Um, uh, losing altitude and, uh, flight attendants are running back and forth.
Prasanna Malaiyandi:
They're doing their thing.
Prasanna Malaiyandi:
We were on initial ascent, so everybody was still strapped in and everything.
Prasanna Malaiyandi:
And, um, the, and then the, the.
Prasanna Malaiyandi:
The plane did this massive like banking maneuver
Prasanna Malaiyandi:
to take us back and it was definitely one of those, you know,
Prasanna Malaiyandi:
Yep.
Prasanna Malaiyandi:
W. Curtis Preston: and they, they, you know, 'cause they got us back.
Prasanna Malaiyandi:
They, they basically, as far as I know, they turned that plane around as
Prasanna Malaiyandi:
quickly, whatever the tightest turning radius that you could possibly do.
Prasanna Malaiyandi:
I'm sure that there were massive alerts that went off in the, in the tower, right?
Prasanna Malaiyandi:
The, the air traffic controller tower.
Prasanna Malaiyandi:
They cleared the runways, we went back and we landed, and then, you know, and
Prasanna Malaiyandi:
then the firetrucks came out and uh, and that's when everybody freaked out, right?
Prasanna Malaiyandi:
Once we were down and then the firetrucks came out.
Prasanna Malaiyandi:
Prior to that, everybody just sat there and white knuckled it and, um, you know,
Prasanna Malaiyandi:
and just, you know, just tried not to cry.
Prasanna Malaiyandi:
My my point of all of this was that pilot had done that hundreds of times
Prasanna Malaiyandi:
already, because he goes to a simulator.
Prasanna Malaiyandi:
And, uh, and trains for that.
Prasanna Malaiyandi:
And they had trained for all of this.
Prasanna Malaiyandi:
The tower knew what to do.
Prasanna Malaiyandi:
Why?
Prasanna Malaiyandi:
Because they had a runbook.
Prasanna Malaiyandi:
They
Prasanna Malaiyandi:
followed to the letter because they've got lives in their hands.
Prasanna Malaiyandi:
They knew it.
Prasanna Malaiyandi:
They, they, they had a, you know, a checklist that they go down when you
Prasanna Malaiyandi:
have to make an emergency landing with a plane that's lost an engine.
Prasanna Malaiyandi:
That's just a, you know, they, everybody just went and did
Prasanna Malaiyandi:
everything that they needed to do.
Prasanna Malaiyandi:
Which I will say apparently didn't involve letting us know
Prasanna Malaiyandi:
what the hell was happening.
Prasanna Malaiyandi:
That was,
Prasanna Malaiyandi:
that was not their, that was not their concern.
Prasanna Malaiyandi:
Their concern
Prasanna Malaiyandi:
you had, yeah, because there was nothing
Prasanna Malaiyandi:
you as passengers could do.
Prasanna Malaiyandi:
W. Curtis Preston: There was one thing we could do.
Prasanna Malaiyandi:
You know what it was?
Prasanna Malaiyandi:
Ah,
Prasanna Malaiyandi:
Yeah.
Prasanna Malaiyandi:
W. Curtis Preston: this is basically, um, all right, so, yeah, and, and so
Prasanna Malaiyandi:
that's why you want to have a runbook.
Prasanna Malaiyandi:
You wanna have a runbook, you want to train, you want to practice, and you want
Prasanna Malaiyandi:
this stuff to be basically muscle memory.
Prasanna Malaiyandi:
And that involves who you notify, how you notify, whether or not, you
Prasanna Malaiyandi:
know, depending on who you are as an organization, whether or not you have
Prasanna Malaiyandi:
to notify the media, whether or not you have to notify stockholders, you know,
Prasanna Malaiyandi:
all of the, all of these things, right?
Prasanna Malaiyandi:
Um, uh, law enforcement depending on why you're, why you're,
Prasanna Malaiyandi:
you know, uh, doing this.
Prasanna Malaiyandi:
Yep.
Prasanna Malaiyandi:
Or
Prasanna Malaiyandi:
your insurance
Prasanna Malaiyandi:
W. Curtis Preston: a list or your, yeah.
Prasanna Malaiyandi:
Your insurance company.
Prasanna Malaiyandi:
Your cyber insurance
Prasanna Malaiyandi:
Yeah.
Prasanna Malaiyandi:
W. Curtis Preston: Exactly.
Prasanna Malaiyandi:
Um, so I have a list I made, uh, I like, uh, alliterations.
Prasanna Malaiyandi:
And I made an eight item alliteration for things that would go, that
Prasanna Malaiyandi:
sort of make a good runbook.
Prasanna Malaiyandi:
And the first one, uh, I'll, we'll just go back and forth with
Prasanna Malaiyandi:
them.
Prasanna Malaiyandi:
All right.
Prasanna Malaiyandi:
Yep.
Prasanna Malaiyandi:
W. Curtis Preston: The first one is that it needs to be authoritative,
Prasanna Malaiyandi:
and by that I mean there needs to be one runbook right there.
Prasanna Malaiyandi:
There can be versioning over time, but when the disaster, uh, strikes,
Prasanna Malaiyandi:
there's one place where everyone knows to go to get the runbook.
Prasanna Malaiyandi:
Now, the runbook may be.
Prasanna Malaiyandi:
Electronic.
Prasanna Malaiyandi:
It may be printed, it may be on an iPad.
Prasanna Malaiyandi:
It may be, it may be a, a bunch of different things, but everyone should
Prasanna Malaiyandi:
know exactly where and when that is.
Prasanna Malaiyandi:
And there, and it may take different forms.
Prasanna Malaiyandi:
There may be a paper version and an electronic version, but
Prasanna Malaiyandi:
those two versions should match.
Prasanna Malaiyandi:
There should be one authoritative, uh, version of the runbook.
Prasanna Malaiyandi:
So how in organizations, and I'm sure this
Prasanna Malaiyandi:
can be solved, how do you make sure everyone is going off the same runbook
Prasanna Malaiyandi:
W. Curtis Preston: Well, I, I, I think that's just a matter of.
Prasanna Malaiyandi:
process?
Prasanna Malaiyandi:
W. Curtis Preston: it all goes to training, right?
Prasanna Malaiyandi:
Everyone just has continually explained this is where the
Prasanna Malaiyandi:
runbook is when it's time, right?
Prasanna Malaiyandi:
And then whenever there are, um, the, the, the biggest.
Prasanna Malaiyandi:
Problem slash concern is if you are keeping a paper version of the runbook,
Prasanna Malaiyandi:
right?
Prasanna Malaiyandi:
You have to make sure that when you come up with a new runbook, that
Prasanna Malaiyandi:
you destroy the old paper version.
Prasanna Malaiyandi:
Just realize that if that's the case, this, this is going to be a
Prasanna Malaiyandi:
risk to the first thing, which is that it should be authoritative.
Prasanna Malaiyandi:
Um, the electronic version, assuming we're all accessing the same electronic version,
Prasanna Malaiyandi:
uh, that should be much easier 'cause it's one file and you can control it.
Prasanna Malaiyandi:
One file that's maybe replicated in
Prasanna Malaiyandi:
multiple places, but you know, one
Prasanna Malaiyandi:
but just make sure that you are aware, like if one
Prasanna Malaiyandi:
of the disasters you're worried about is lack of electricity or flooding or
Prasanna Malaiyandi:
other aspects like that, that you wanna make sure it's on some medium that
Prasanna Malaiyandi:
it, that can handle whatever disaster you're looking to support in that area.
Prasanna Malaiyandi:
W. Curtis Preston: Which is why some people like the paper
Prasanna Malaiyandi:
copy, right?
Prasanna Malaiyandi:
Um, I can't argue against it.
Prasanna Malaiyandi:
It's just, it's just hard to
Prasanna Malaiyandi:
keep up to date.
Prasanna Malaiyandi:
Right.
Prasanna Malaiyandi:
And, um, you know, and you're gonna kill a bunch of trees.
Prasanna Malaiyandi:
so the next one is accurate, which basically
Prasanna Malaiyandi:
means that the runbook contains all the correct information.
Prasanna Malaiyandi:
So I.
Prasanna Malaiyandi:
You know exactly what servers, what applications, all the
Prasanna Malaiyandi:
people, whoever is required.
Prasanna Malaiyandi:
So if people leave right that, that it is constantly updated and
Prasanna Malaiyandi:
it is accurate with what you need.
Prasanna Malaiyandi:
W. Curtis Preston: Yeah,
Prasanna Malaiyandi:
that it matches the current situation, right.
Prasanna Malaiyandi:
Yeah.
Prasanna Malaiyandi:
So here's a question for you, Curtis.
Prasanna Malaiyandi:
Do you think it's worse to have a no runbook or to have a outdated runbook?
Prasanna Malaiyandi:
W. Curtis Preston: I still think no runbooks worse.
Prasanna Malaiyandi:
Right?
Prasanna Malaiyandi:
Right.
Prasanna Malaiyandi:
If, if you, you pull out the runbook and it says, you know, please
Prasanna Malaiyandi:
turn to Curtis and tell him to
Prasanna Malaiyandi:
do the empty squat.
Prasanna Malaiyandi:
And you're like, uh, Curtis quit three weeks ago.
Prasanna Malaiyandi:
You still got a plan, right?
Prasanna Malaiyandi:
Uh, no.
Prasanna Malaiyandi:
A bad plan is still better than no plan, I
Prasanna Malaiyandi:
think.
Prasanna Malaiyandi:
Okay.
Prasanna Malaiyandi:
W. Curtis Preston: But, um, yeah.
Prasanna Malaiyandi:
Yeah.
Prasanna Malaiyandi:
Um, so the next one that I have is accessible.
Prasanna Malaiyandi:
It needs to be accessible.
Prasanna Malaiyandi:
We've talked about this a little bit already, and that is just make sure that
Prasanna Malaiyandi:
no matter what happens, you know, you alluded to that earlier, no matter what
Prasanna Malaiyandi:
happens, make sure that it's accessible.
Prasanna Malaiyandi:
To me, that would mean that I would have an automatic system.
Prasanna Malaiyandi:
Where there is the authoritative copy where we always do edits,
Prasanna Malaiyandi:
we always only edit here.
Prasanna Malaiyandi:
Right?
Prasanna Malaiyandi:
And then that is then replicated via an automated system to all of the other
Prasanna Malaiyandi:
places where that are read-only copies.
Prasanna Malaiyandi:
You might replicate it to an application.
Prasanna Malaiyandi:
Maybe your company has its own application that runs on iPads.
Prasanna Malaiyandi:
That application is, is automatically updated every time
Prasanna Malaiyandi:
you update the documentation.
Prasanna Malaiyandi:
Maybe you do print out a new one, maybe you figure out a way where you do loose
Prasanna Malaiyandi:
leaf papers and you, you, and you only reprint the pages that have changed.
Prasanna Malaiyandi:
And the new thing, you know, you could do that so you're not killing
Prasanna Malaiyandi:
an entire, you know, forest every time you, you update the DR book.
Prasanna Malaiyandi:
But, uh, but I, I do think that you need to, um.
Prasanna Malaiyandi:
Created in such a way that it, it's available no matter what happens.
Prasanna Malaiyandi:
And I think that last bit is key because remember, if this is a
Prasanna Malaiyandi:
corporate environment, say you don't have access to your two-factor authentication
Prasanna Malaiyandi:
or your active directory, right?
Prasanna Malaiyandi:
How is someone gonna get access to the runbook in order to be
Prasanna Malaiyandi:
able to restore those services?
Prasanna Malaiyandi:
I.
Prasanna Malaiyandi:
W. Curtis Preston: exactly.
Prasanna Malaiyandi:
What was the company, it was a big company a few months ago
Prasanna Malaiyandi:
that remember there was a big.
Prasanna Malaiyandi:
Was an attack or a disaster?
Prasanna Malaiyandi:
Some.
Prasanna Malaiyandi:
And they were locked out of the building.
Prasanna Malaiyandi:
Do you remember this?
Prasanna Malaiyandi:
Yeah, this was,
Prasanna Malaiyandi:
W. Curtis Preston: It was a big company.
Prasanna Malaiyandi:
yeah, because there are key cards.
Prasanna Malaiyandi:
Was it meta?
Prasanna Malaiyandi:
W. Curtis Preston: I think it was Meta, right?
Prasanna Malaiyandi:
They were, they had an outage and then they were locked out of the building.
Prasanna Malaiyandi:
They couldn't get into the building.
Prasanna Malaiyandi:
because the authentication system was,
Prasanna Malaiyandi:
Yeah.
Prasanna Malaiyandi:
don't, don't be that person.
Prasanna Malaiyandi:
Um,
Prasanna Malaiyandi:
So the next one is absorbed, which is basically is
Prasanna Malaiyandi:
your runbook understood by everyone in it, which I think a lot of people
Prasanna Malaiyandi:
kind of take that for granted.
Prasanna Malaiyandi:
They're like, oh yeah, I just wrote this.
Prasanna Malaiyandi:
Anyone in it should be able to read it, but can they really read
Prasanna Malaiyandi:
it and understand what's in there?
Prasanna Malaiyandi:
W. Curtis Preston: Yeah, this is, I, I think this is kind
Prasanna Malaiyandi:
of about marketing really.
Prasanna Malaiyandi:
Right?
Prasanna Malaiyandi:
You know, um, in the previous episode we talked about how good of a recovery
Prasanna Malaiyandi:
mindset does your organization have, and this is part of that, right?
Prasanna Malaiyandi:
So how well does your, you know, does, does the word runbook, if you said Dr.
Prasanna Malaiyandi:
Runbook, does everybody in your organization know
Prasanna Malaiyandi:
what we're talking about?
Prasanna Malaiyandi:
Because just like.
Prasanna Malaiyandi:
We want to make sure it's available in all of the different scenarios.
Prasanna Malaiyandi:
We wanna make sure that if only Steve is available right
Prasanna Malaiyandi:
now, Steve knows where the Dr.
Prasanna Malaiyandi:
Runbook is.
Prasanna Malaiyandi:
D knows he, he knows what the Dr.
Prasanna Malaiyandi:
Runbook is.
Prasanna Malaiyandi:
He knows where to get it
Prasanna Malaiyandi:
right, and he knows how to not get an old version of it,
Prasanna Malaiyandi:
right?
Prasanna Malaiyandi:
So this is, this is about making sure everybody's all on the same, uh,
Prasanna Malaiyandi:
all on the same
Prasanna Malaiyandi:
And it's, would you say though, Curtis, I know we
Prasanna Malaiyandi:
mentioned specifically around it, but is it really just limited to it or is
Prasanna Malaiyandi:
it pretty much all functional heads should be aware of this because it
Prasanna Malaiyandi:
could be touching them as well, right?
Prasanna Malaiyandi:
W. Curtis Preston: Well, I think anybody that needs to actually use the runbook.
Prasanna Malaiyandi:
Yeah.
Prasanna Malaiyandi:
W. Curtis Preston: Right.
Prasanna Malaiyandi:
I mean, there, there are people in the runbook that you're
Prasanna Malaiyandi:
going to want to contact so that they're aware of what's happening.
Prasanna Malaiyandi:
They don't necessarily need to know too much about the runbook.
Prasanna Malaiyandi:
I mean, would not hurt that they would know about it.
Prasanna Malaiyandi:
So the next one that I have is that it needs to be active and, and by
Prasanna Malaiyandi:
this, just that, um, it's very similar to the concept of accurate, and that
Prasanna Malaiyandi:
is that it needs to be part of your.
Prasanna Malaiyandi:
It change and control process.
Prasanna Malaiyandi:
Whenever you add a server you have a change control process.
Prasanna Malaiyandi:
This needs to be part of the change control process so that it's an
Prasanna Malaiyandi:
active part of your organization, that you're continually updating the
Prasanna Malaiyandi:
runbook every time you make a change.
Prasanna Malaiyandi:
Um, that shouldn't be too hard.
Prasanna Malaiyandi:
Again, assuming you have a solid recovery mindset.
Prasanna Malaiyandi:
Yeah, and I think for that one too, it's is that part
Prasanna Malaiyandi:
of your DNA of the company, right?
Prasanna Malaiyandi:
That has to be part of the culture of it that says, Hey, whenever we
Prasanna Malaiyandi:
are doing a change, we have to make sure that yes, this, this, and this.
Prasanna Malaiyandi:
Just like we always talk about backup, go update your backup systems to make sure
Prasanna Malaiyandi:
you can back up that new application.
Prasanna Malaiyandi:
Dr.
Prasanna Malaiyandi:
Runbook should also be updated as well.
Prasanna Malaiyandi:
Uh, the next one is, and I know I kind of touched on this
Prasanna Malaiyandi:
earlier, right, is adaptable.
Prasanna Malaiyandi:
And this really goes back to.
Prasanna Malaiyandi:
Everything in the runbook may not be perfect.
Prasanna Malaiyandi:
Things may change outside the environment.
Prasanna Malaiyandi:
The situation may change, and so you need to make sure that you have backup plans,
Prasanna Malaiyandi:
not backup, like data backup, but backup like alternate plans to make sure that
Prasanna Malaiyandi:
you are still able to execute and recover your systems, even if not a hundred
Prasanna Malaiyandi:
percent of either the infrastructure is available or other aspects like that.
Prasanna Malaiyandi:
W. Curtis Preston: Yeah, it's sort of like, I dunno if you've ever read any
Prasanna Malaiyandi:
contracts, but there's like a common phrase in a contract that like any, if
Prasanna Malaiyandi:
any part of this contract is invalid, that doesn't invalid, that only invalidates
Prasanna Malaiyandi:
that part, not the entire contract.
Prasanna Malaiyandi:
Sort of like here, like if you hit a single bump in the road
Prasanna Malaiyandi:
and then your entire run buck is off kilter, uh, that's not good.
Prasanna Malaiyandi:
You need
Prasanna Malaiyandi:
to this, I think this is probably more about.
Prasanna Malaiyandi:
The practicing it and doing it right.
Prasanna Malaiyandi:
Um, that, uh, you get, and you, you, you, you use the,
Prasanna Malaiyandi:
you know, like the chaos monkey
Prasanna Malaiyandi:
concept to throw wrenches into the works, uh, to, to see how well people handle,
Prasanna Malaiyandi:
uh, wrenches along the way.
Prasanna Malaiyandi:
Yeah, it's like your example you gave Curtis of your flight
Prasanna Malaiyandi:
experience and losing an engine, right?
Prasanna Malaiyandi:
Maybe the pilot hadn't trained for engine failure win on ascent, right?
Prasanna Malaiyandi:
But maybe he was able to adapt and figure out, okay, yeah,
Prasanna Malaiyandi:
this is where I'm going to do.
Prasanna Malaiyandi:
W. Curtis Preston: Yeah, exactly.
Prasanna Malaiyandi:
Uh, so the next one that I have is auditable.
Prasanna Malaiyandi:
And you know, this is.
Prasanna Malaiyandi:
Um, I think the best way, you know, you've heard me say it a bunch of times, the best
Prasanna Malaiyandi:
way to have your Dr, uh, system tested is to have someone who isn't the person
Prasanna Malaiyandi:
who put it together, do the one that's actually doing the, the thing, right?
Prasanna Malaiyandi:
So, um, if you can bring in an outside IT person and they follow your Dr.
Prasanna Malaiyandi:
Runbook and they recover your organization, ominous
Prasanna Malaiyandi:
dominus, man, you, you're
Prasanna Malaiyandi:
the best.
Prasanna Malaiyandi:
Do you wanna talk about your experiences with Dr.
Prasanna Malaiyandi:
Runbooks at the big bank you used to work at?
Prasanna Malaiyandi:
I know we've talked
Prasanna Malaiyandi:
W. Curtis Preston: Yeah.
Prasanna Malaiyandi:
Yeah, yeah.
Prasanna Malaiyandi:
I know, I know.
Prasanna Malaiyandi:
Yeah.
Prasanna Malaiyandi:
Not everybody's listen to every episode.
Prasanna Malaiyandi:
So, you know, we used to do a, we used to do a full DR test every six months.
Prasanna Malaiyandi:
And, uh, it was a really big thing.
Prasanna Malaiyandi:
Like we, you know, we shuttered everybody up for the weekend and, and, uh, we
Prasanna Malaiyandi:
spent lots of money doing these DR tests and we tried to do it like this where,
Prasanna Malaiyandi:
you know, I had a lot of documentation.
Prasanna Malaiyandi:
I had my runbook out there and they, they would, they would go to follow it and.
Prasanna Malaiyandi:
Um, we defined a successful recovery as a recovery where the person running,
Prasanna Malaiyandi:
the DR didn't have to turn around and go, so, Curtis, what did you mean by.
Prasanna Malaiyandi:
This.
Prasanna Malaiyandi:
Right.
Prasanna Malaiyandi:
And it by that standard, we never had a successful recovery during
Prasanna Malaiyandi:
the three years that I worked there.
Prasanna Malaiyandi:
Uh, we always recovered, but there was always some, some turnaround.
Prasanna Malaiyandi:
And so that goes back to that, that adaptable thing that
Prasanna Malaiyandi:
I, you know, that I said.
Prasanna Malaiyandi:
Right.
Prasanna Malaiyandi:
Um, but it also goes to.
Prasanna Malaiyandi:
Testing more often every six months isn't good enough, right?
Prasanna Malaiyandi:
Every six months was the only way we could do it back then.
Prasanna Malaiyandi:
But nowadays with the cloud, you know how big of a fan I'm with the cloud.
Prasanna Malaiyandi:
But nowadays with, with infrastructure as code and all these other things that
Prasanna Malaiyandi:
we're able to do today for many, if not most organizations, they could do this
Prasanna Malaiyandi:
in the cloud and they can automate it.
Prasanna Malaiyandi:
And that a lot of this stuff that, that we have to figure out in our heads
Prasanna Malaiyandi:
could actually be completely automated.
Prasanna Malaiyandi:
Um,
Prasanna Malaiyandi:
And.
Prasanna Malaiyandi:
And I pretty much alluded to the next,
Prasanna Malaiyandi:
Yeah.
Prasanna Malaiyandi:
The next one, which is basically Affirm, so test regularly testing is
Prasanna Malaiyandi:
a best way to make sure that your Dr.
Prasanna Malaiyandi:
Runbook is accurate and that it works.
Prasanna Malaiyandi:
And I think just like you gave the example Curtis, in the three years, you
Prasanna Malaiyandi:
didn't really get to that level that you guys set, and that's perfectly fine.
Prasanna Malaiyandi:
Don't expect perfection in your Dr.
Prasanna Malaiyandi:
Runbook.
Prasanna Malaiyandi:
It will require tweaks.
Prasanna Malaiyandi:
It will require changes over time, and that's okay.
Prasanna Malaiyandi:
It's a starting point.
Prasanna Malaiyandi:
It's better than having nothing like you had alluded to earlier.
Prasanna Malaiyandi:
W. Curtis Preston: Yeah.
Prasanna Malaiyandi:
Agreed.
Prasanna Malaiyandi:
Um, yeah, so that, that's sort of the elements, if you
Prasanna Malaiyandi:
will, of what makes a good Dr.
Prasanna Malaiyandi:
Runbook.
Prasanna Malaiyandi:
Let's talk about what sort of the ingredients of a Dr.
Prasanna Malaiyandi:
Runbook, right.
Prasanna Malaiyandi:
It, it's
Prasanna Malaiyandi:
just,
Prasanna Malaiyandi:
That's the way.
Prasanna Malaiyandi:
It's not just, we set up the infrastructure and we
Prasanna Malaiyandi:
just randomly run a Dr.
Prasanna Malaiyandi:
Runbook.
Prasanna Malaiyandi:
That's not at all.
Prasanna Malaiyandi:
W. Curtis Preston: Yeah.
Prasanna Malaiyandi:
Yeah.
Prasanna Malaiyandi:
Um, so the, the first thing here, and this is one of those where the, the first
Prasanna Malaiyandi:
thing is the last thing that you do, and that is you have, uh, an overview, you
Prasanna Malaiyandi:
have an executive summary, um, and, um.
Prasanna Malaiyandi:
And, and like a technical overview to just give, give everybody,
Prasanna Malaiyandi:
uh, you know, in that scenario of like we lost all our IT people.
Prasanna Malaiyandi:
Uh, now you've got some IT people that are not your IT people, but they're
Prasanna Malaiyandi:
different IT people and they're following, you know, so you give them an overview,
Prasanna Malaiyandi:
uh, as well as, uh, executive overview.
Prasanna Malaiyandi:
You, you, you need to write this at the end because you don't know what
Prasanna Malaiyandi:
you're, you know, what you're doing.
Prasanna Malaiyandi:
An overview of right.
Prasanna Malaiyandi:
Yeah, and I would say that a lot of technical folks
Prasanna Malaiyandi:
might think, Hey, isn't a runbook, just here are the procedures that I do.
Prasanna Malaiyandi:
And I think these first couple that we're gonna talk about of what goes
Prasanna Malaiyandi:
in a runbook are critical, which will help you either in marketing or just
Prasanna Malaiyandi:
to make it more complete for people who may not be doing this day to day.
Prasanna Malaiyandi:
Right.
Prasanna Malaiyandi:
So the next one is really.
Prasanna Malaiyandi:
An inventory.
Prasanna Malaiyandi:
So all your products and vendors and the contacts for those.
Prasanna Malaiyandi:
So who is your, what is your SaaS provider for email, right?
Prasanna Malaiyandi:
Is it Microsoft?
Prasanna Malaiyandi:
Is it Google?
Prasanna Malaiyandi:
What is your account there?
Prasanna Malaiyandi:
Who is the contact that you reach out to in the vendor in case, like
Prasanna Malaiyandi:
do you have a technical account manager, an account rep, et cetera.
Prasanna Malaiyandi:
So these are all critical things to document because you
Prasanna Malaiyandi:
might need their help, right?
Prasanna Malaiyandi:
It's not just, Hey, I need to recover my own systems, but you might need help with.
Prasanna Malaiyandi:
Recovering some things that are in their infrastructure as well,
Prasanna Malaiyandi:
so just things to be aware of.
Prasanna Malaiyandi:
W. Curtis Preston: And there can be an escalation there, right?
Prasanna Malaiyandi:
So you know, who is the person that you normally talk to at Microsoft or Dell
Prasanna Malaiyandi:
or HP or whatever it is that you know, where you're buying your infrastructure.
Prasanna Malaiyandi:
Uh, M Center, you know, who is your normal point of contact.
Prasanna Malaiyandi:
And then you should also, and by the way, you should have.
Prasanna Malaiyandi:
Multiple
Prasanna Malaiyandi:
ways to contact them, email, whatever, whatever you have.
Prasanna Malaiyandi:
If you have, you know, phone, right?
Prasanna Malaiyandi:
If you have, you know, their Twitter account, whatever.
Prasanna Malaiyandi:
Um, and then also like management, right?
Prasanna Malaiyandi:
You know, if, if, if, if, if they, if you go to reach out to Steve
Prasanna Malaiyandi:
and Steve, I use that name a lot.
Prasanna Malaiyandi:
You go to reach out to Steve and Steve isn't there, um, then you
Prasanna Malaiyandi:
know who then, then what do you do?
Prasanna Malaiyandi:
Because in the middle of a disaster, you may need some additional stuff.
Prasanna Malaiyandi:
You may need some additional hardware, you know, storage, whatever.
Prasanna Malaiyandi:
Um, and so you, you know, or you may need some additional expertise.
Prasanna Malaiyandi:
I push the button and the button didn't go.
Prasanna Malaiyandi:
Who do I, you know, where's my, where's my 9 1 1?
Prasanna Malaiyandi:
Um, can you think what else would be there
Prasanna Malaiyandi:
Yeah, and it doesn't have to necessarily be even your vendor.
Prasanna Malaiyandi:
It could be an MSP, right?
Prasanna Malaiyandi:
Your consulting partners, other folks as well who are important to keep track of
Prasanna Malaiyandi:
because they could be there to help you and assist you when you're recovering.
Prasanna Malaiyandi:
W. Curtis Preston: Right,
Prasanna Malaiyandi:
Or it could be your cyber insurance company information too.
Prasanna Malaiyandi:
W. Curtis Preston: Yeah.
Prasanna Malaiyandi:
Yeah.
Prasanna Malaiyandi:
We, we've mentioned them a couple of times in this, in this episode.
Prasanna Malaiyandi:
Uh, the next one sounds very similar, and that is the contacts for the
Prasanna Malaiyandi:
people needed, the DR Recovery.
Prasanna Malaiyandi:
What I'm thinking here is, this is your staff,
Prasanna Malaiyandi:
right?
Prasanna Malaiyandi:
Um, this is everybody.
Prasanna Malaiyandi:
And again, you know, we, we joked about it earlier.
Prasanna Malaiyandi:
Um, and that is that.
Prasanna Malaiyandi:
The, the successful one is one where we didn't have to turn
Prasanna Malaiyandi:
around and ask for Curtis.
Prasanna Malaiyandi:
Well, if it's a real disaster, it's okay to call Curtis,
Prasanna Malaiyandi:
right?
Prasanna Malaiyandi:
Well, not me, but, 'cause I didn't do your Dr.
Prasanna Malaiyandi:
Runbook.
Prasanna Malaiyandi:
But you, you've got a Curtis in your organization.
Prasanna Malaiyandi:
Everyone should have a Curtis in their organization.
Prasanna Malaiyandi:
Don't you think?
Prasanna Malaiyandi:
Prasanna.
Prasanna Malaiyandi:
How about everyone should have a
Prasanna Malaiyandi:
Prasanna in their organization?
Prasanna Malaiyandi:
W. Curtis Preston: Everybody should have a, a person who knows way too
Prasanna Malaiyandi:
much about many random subjects.
Prasanna Malaiyandi:
Yes, definitely, definitely.
Prasanna Malaiyandi:
Someone should have a, you should have a Curtis and a
Prasanna Malaiyandi:
Prasanna in every environment.
Prasanna Malaiyandi:
Uh, so that's step one in your Dr.
Prasanna Malaiyandi:
Road because identify the Curtis in the Prasannas in your environment.
Prasanna Malaiyandi:
Um, and, um, the, um, uh, but yeah, this is contacts and again, escalation.
Prasanna Malaiyandi:
Right.
Prasanna Malaiyandi:
I reached out to Curtis.
Prasanna Malaiyandi:
Curtis didn't, you know, didn't answer his phone.
Prasanna Malaiyandi:
Who do I call now?
Prasanna Malaiyandi:
Right?
Prasanna Malaiyandi:
Who?
Prasanna Malaiyandi:
You know, you call, you know Curtis's boss and Curtis's boss's boss.
Prasanna Malaiyandi:
Right.
Prasanna Malaiyandi:
Um, and, and also, um, contacts within the organization from a
Prasanna Malaiyandi:
here's who we need to keep, uh,
Prasanna Malaiyandi:
you know, abreast of everything that we're doing.
Prasanna Malaiyandi:
Right.
Prasanna Malaiyandi:
Management.
Prasanna Malaiyandi:
Um, and hopefully you have like.
Prasanna Malaiyandi:
A designated person in your senior management that you, you just go
Prasanna Malaiyandi:
to them and let them communicate everything to everybody else because
Prasanna Malaiyandi:
you're kind of busy doing the thing.
Prasanna Malaiyandi:
Yeah.
Prasanna Malaiyandi:
W. Curtis Preston: Right.
Prasanna Malaiyandi:
Um, and, uh, this is what back in the day nowadays, we would, um.
Prasanna Malaiyandi:
We would do it differently, but back in the day, this is what we used pagers for.
Prasanna Malaiyandi:
We had an automated paging system and we would just send a page out to everybody.
Prasanna Malaiyandi:
Everybody that was involved in the process got a page at every
Prasanna Malaiyandi:
step of the process so that everybody knew what was go, what was going on,
Prasanna Malaiyandi:
which means you need to maintain a list of all those pager numbers and
Prasanna Malaiyandi:
all that kind of stuff, which of course, no one.
Prasanna Malaiyandi:
In my mind as you're talking about sort of
Prasanna Malaiyandi:
the escalations, I was thinking about your story about the Bobs,
Prasanna Malaiyandi:
W. Curtis Preston: Yeah.
Prasanna Malaiyandi:
Toms.
Prasanna Malaiyandi:
Toms
Prasanna Malaiyandi:
oh, Toms.
Prasanna Malaiyandi:
Toms.
Prasanna Malaiyandi:
W. Curtis Preston: The two Toms.
Prasanna Malaiyandi:
Yeah.
Prasanna Malaiyandi:
Uh, the two Toms standing over the left and right shoulder.
Prasanna Malaiyandi:
Yeah.
Prasanna Malaiyandi:
That was a, that was a pretty funny moment where we were in the middle of
Prasanna Malaiyandi:
an actual disaster and we were, um.
Prasanna Malaiyandi:
You know, one of our guys was in the knock and he was on speaker phone with what?
Prasanna Malaiyandi:
With the guy that was actually in the data center swapping tapes.
Prasanna Malaiyandi:
And he said, he said, oh, are, are you in the knock?
Prasanna Malaiyandi:
And he is like, yeah.
Prasanna Malaiyandi:
And he goes, so let me guess you got Tom and Tom over your left and
Prasanna Malaiyandi:
right shoulder, except he didn't realize he was on speaker phone.
Prasanna Malaiyandi:
And, and Tom and Tom, which was their bosses.
Prasanna Malaiyandi:
Let's see their boss's boss's boss and their boss's boss's boss's boss.
Prasanna Malaiyandi:
Wow.
Prasanna Malaiyandi:
W. Curtis Preston: And, and, and they did indeed have, 'cause this was a big bank,
Prasanna Malaiyandi:
right?
Prasanna Malaiyandi:
So there were many layers of, so this was basically the CEO's second in command
Prasanna Malaiyandi:
and then the one after that, right?
Prasanna Malaiyandi:
And they were standing over his left and right shoulder.
Prasanna Malaiyandi:
And when they, when they said this, they, they both just took one step back.
Prasanna Malaiyandi:
Oh, good times.
Prasanna Malaiyandi:
Good times.
Prasanna Malaiyandi:
Yeah.
Prasanna Malaiyandi:
So the next thing is really what a lot of people think about when they
Prasanna Malaiyandi:
think about a runbook, which is really the procedures which are those
Prasanna Malaiyandi:
step-by-step instructions on how to actually recover from that disaster.
Prasanna Malaiyandi:
So this might spell out sort of, okay, what exactly from an
Prasanna Malaiyandi:
infrastructure I need to bring up, what applications and what order, right?
Prasanna Malaiyandi:
It's probably multiple sections because it's not like one big bang and you
Prasanna Malaiyandi:
bring everything up in one go, right?
Prasanna Malaiyandi:
So depending on what the disaster is, it might have multiple.
Prasanna Malaiyandi:
Different sections with different procedures for each one, but it should
Prasanna Malaiyandi:
be labeled numbered and easy to find.
Prasanna Malaiyandi:
W. Curtis Preston: Yeah, so you've defined, this is
Prasanna Malaiyandi:
why I said this is the ult.
Prasanna Malaiyandi:
This is the culmination of all of your efforts.
Prasanna Malaiyandi:
You've defined the RTO and RPO of everything in your environment, and
Prasanna Malaiyandi:
then you defined a recovery priority of everything in your environment.
Prasanna Malaiyandi:
And so this is gonna say, okay, you know, recovery, you know, let's say you've got.
Prasanna Malaiyandi:
I don't know, three recovery groups, right?
Prasanna Malaiyandi:
The stuff you're gonna do very, you know, the stuff that's gotta come
Prasanna Malaiyandi:
up before anything else comes up.
Prasanna Malaiyandi:
And then the next set, which is like the stuff that comes up right
Prasanna Malaiyandi:
after all that stuff comes up.
Prasanna Malaiyandi:
And then everything else.
Prasanna Malaiyandi:
And the first group is probably really small.
Prasanna Malaiyandi:
Second group is probably decent sized.
Prasanna Malaiyandi:
And then the third group is probably the biggest
Prasanna Malaiyandi:
because not everything has to be up and running, you know, immediately to
Prasanna Malaiyandi:
be able to turn the lights back on.
Prasanna Malaiyandi:
So all of that should be spelled out in the, you know, you, you may also, because
Prasanna Malaiyandi:
we've talked about this previously, you may also use completely different
Prasanna Malaiyandi:
recovery methods based on the RPO
Prasanna Malaiyandi:
and the RTO of different.
Prasanna Malaiyandi:
You know,
Prasanna Malaiyandi:
uh, parts of your organization, you might use a DR as a service product
Prasanna Malaiyandi:
that does replication based, uh, DR for your tier one applications.
Prasanna Malaiyandi:
And you might use backup replications for your tier two, and you might use,
Prasanna Malaiyandi:
you know, tape, uh, or, or maybe a replicated copy of your backup into
Prasanna Malaiyandi:
S3, uh, and hope you don't ever have to
Prasanna Malaiyandi:
use it.
Prasanna Malaiyandi:
For your, uh, for your tier, you know, your tier three
Prasanna Malaiyandi:
things right?
Prasanna Malaiyandi:
And all of that should be spelled out on all those procedures.
Prasanna Malaiyandi:
Um, what, what else do you think?
Prasanna Malaiyandi:
Any other thoughts on the procedures?
Prasanna Malaiyandi:
No, I think that's, but this is where it goes
Prasanna Malaiyandi:
back to that, uh, what was active that we had talked about earlier, right?
Prasanna Malaiyandi:
When you are making changes to your infrastructure, that's why
Prasanna Malaiyandi:
it's critical that you go and you update these procedures.
Prasanna Malaiyandi:
Because when you need to execute, you wanna make sure that it is accurate,
Prasanna Malaiyandi:
so you can have the best version of.
Prasanna Malaiyandi:
Or best, best chance of recovery
Prasanna Malaiyandi:
W. Curtis Preston: Yeah.
Prasanna Malaiyandi:
Yeah, exactly.
Prasanna Malaiyandi:
Uh, and then, um, our final step is escalations.
Prasanna Malaiyandi:
And we've already talked about this, but nothing's gonna go right.
Prasanna Malaiyandi:
Prasanna Malaiyandi: really, Curtis, why not?
Prasanna Malaiyandi:
I thought that's why we, and plan and
Prasanna Malaiyandi:
document, and.
Prasanna Malaiyandi:
W. Curtis Preston: As much as, I'm gonna say like a disaster re an actual disaster.
Prasanna Malaiyandi:
Recovery is much more like a rugby match.
Prasanna Malaiyandi:
Then it is like a baseball game, right?
Prasanna Malaiyandi:
Baseball game.
Prasanna Malaiyandi:
It's like you got clear lines, right?
Prasanna Malaiyandi:
I'm gonna go this way, then I'm gonna go that way, then I'm gonna go that
Prasanna Malaiyandi:
way, then I'm gonna go that way.
Prasanna Malaiyandi:
Or for the rest of the world, a cricket game, right?
Prasanna Malaiyandi:
Or you know, rounders, uh, you know what, you know, basically, you know, rugby.
Prasanna Malaiyandi:
Uh, you know, is, is is like, it just seems to me comparatively speaking like a
Prasanna Malaiyandi:
giant mess of, there's an ultimate goal.
Prasanna Malaiyandi:
We all have the same goal, but, uh, you know, it's just, it, it you, there's
Prasanna Malaiyandi:
this constant adjustment around, and so you you need to have those lines
Prasanna Malaiyandi:
of escalation so that people know.
Prasanna Malaiyandi:
And also there may be a point in your recovery where.
Prasanna Malaiyandi:
You need Tom to call the, to call the equivalent of the TOM over
Prasanna Malaiyandi:
in the other organization, uh, where you're not getting any love,
Prasanna Malaiyandi:
right?
Prasanna Malaiyandi:
It's, it's a weekend and your favorite IT vendor is not answering their phone.
Prasanna Malaiyandi:
Uh, or they're, or they're not giving you the amount of love, you know, they've,
Prasanna Malaiyandi:
you, you've contracted for a four hour.
Prasanna Malaiyandi:
Uh, delivery time of new hardware and, and they haven't even called you
Prasanna Malaiyandi:
back and you're at like three hours.
Prasanna Malaiyandi:
This is when, this is when you escalate on your side so that you can get
Prasanna Malaiyandi:
them to escalate on the other side.
Prasanna Malaiyandi:
Uh, and you've just gotta have all those, um, you know, all those numbers in there.
Prasanna Malaiyandi:
Um, and then I'll tell you what not to do.
Prasanna Malaiyandi:
Here's, here's another story.
Prasanna Malaiyandi:
Um, today's about the stories.
Prasanna Malaiyandi:
So we had, we had a, an employee that and she got handed this pile of tapes
Prasanna Malaiyandi:
and we were told her, she's like, Hey, you got this pile of tapes at some
Prasanna Malaiyandi:
point, bank of America's gonna call you.
Prasanna Malaiyandi:
They'll tell you what to do with these tapes.
Prasanna Malaiyandi:
And she got handed the escalation list that came with the tapes.
Prasanna Malaiyandi:
And after a few hours working, she decided she couldn't handle the
Prasanna Malaiyandi:
anticipation anymore, so she decided to.
Prasanna Malaiyandi:
Go down the escalation list to see if she could get an answer as to what she
Prasanna Malaiyandi:
was supposed to do with these tapes.
Prasanna Malaiyandi:
So, but I guess she wasn't familiar with how to use an escalation list.
Prasanna Malaiyandi:
She
Prasanna Malaiyandi:
just looked at the
Prasanna Malaiyandi:
oh no.
Prasanna Malaiyandi:
W. Curtis Preston: and so at three o'clock in the morning, she called
Prasanna Malaiyandi:
the CEO of Bank of America at home
Prasanna Malaiyandi:
and said, I got these pile of tapes, got these pile of tapes here.
Prasanna Malaiyandi:
I don't know what I'm supposed to do with 'em.
Prasanna Malaiyandi:
No.
Prasanna Malaiyandi:
W. Curtis Preston: That's not how you use an escalation list.
Prasanna Malaiyandi:
Yeah.
Prasanna Malaiyandi:
W. Curtis Preston: That was, that was my point of bringing that story up.
Prasanna Malaiyandi:
So basically that's the Dr.
Prasanna Malaiyandi:
Runbook, I mean.
Prasanna Malaiyandi:
It's, it's putting down all of the plans and procedures and the what to do when
Prasanna Malaiyandi:
the plans and procedures don't work into a form where you can easily index and
Prasanna Malaiyandi:
get through it and whatnot and follow it.
Prasanna Malaiyandi:
I do think it that there should be an authoritative one version
Prasanna Malaiyandi:
that you're constantly updating.
Prasanna Malaiyandi:
Um, and then just make sure it gets, you know, automatically
Prasanna Malaiyandi:
disseminated to the other things.
Prasanna Malaiyandi:
And, um.
Prasanna Malaiyandi:
Uh, and you should be doing this frequently.
Prasanna Malaiyandi:
You should be testing it frequently so that an actual disaster.
Prasanna Malaiyandi:
So when you're doing an actual disaster, you're not testing just the DR plan.
Prasanna Malaiyandi:
You're also testing your runbook, right?
Prasanna Malaiyandi:
You're testing your team's ability to follow that runbook.
Prasanna Malaiyandi:
I a hundred percent agree and I think this is
Prasanna Malaiyandi:
where it becomes critical though.
Prasanna Malaiyandi:
In technology, we typically do a poor job of documentation.
Prasanna Malaiyandi:
Everyone wants to go build the cool stuff, put things together, and then
Prasanna Malaiyandi:
they hate writing documentation.
Prasanna Malaiyandi:
This is one of those things that'll pay tremendous dividends when you actually
Prasanna Malaiyandi:
encounter a disaster by actually having a documented runbook ahead of
Prasanna Malaiyandi:
time and following these procedures and doing the testing, like you said.
Prasanna Malaiyandi:
So instead of something taking.
Prasanna Malaiyandi:
Say a month trying to recover as everyone's scrambling because
Prasanna Malaiyandi:
no one knows what to do 'cause they've never documented or tested.
Prasanna Malaiyandi:
It's all documented.
Prasanna Malaiyandi:
At least you'll reduce the stress level of everyone in the organization
Prasanna Malaiyandi:
as you're going through this.
Prasanna Malaiyandi:
W. Curtis Preston: And there you go.
Prasanna Malaiyandi:
This, welcome to those of you that listen to this entire series.
Prasanna Malaiyandi:
Welcome to the end of the Disaster Recovery.
Prasanna Malaiyandi:
Series six episodes.
Prasanna Malaiyandi:
If you didn't listen to the other five, go back, go back.
Prasanna Malaiyandi:
This was some good stuff.
Prasanna Malaiyandi:
And, uh, Prasanna, thanks for sticking with me.
Prasanna Malaiyandi:
Uh, and by the way, you know, for those of you that didn't know, we're basically just
Prasanna Malaiyandi:
reading from my book, which is Modern Data Protection, uh, which is from O'Reilly.
Prasanna Malaiyandi:
Um, you know, for those of you that didn't know that's who I was, I'm,
Prasanna Malaiyandi:
you know, I'm w Curtis Preston, the author of the book, and, um, you know.
Prasanna Malaiyandi:
Uh, Mr.
Prasanna Malaiyandi:
Backup, whatever.
Prasanna Malaiyandi:
So, um, thanks, Prasanna.
Prasanna Malaiyandi:
Great comments as always.
Prasanna Malaiyandi:
Uh, likewise Curtis.
Prasanna Malaiyandi:
W. Curtis Preston: And thanks for those of you that have, uh, stuck with
Prasanna Malaiyandi:
us, uh, through these six episodes.
Prasanna Malaiyandi:
And for the rest of you go back and listen to the other episodes of
Prasanna Malaiyandi:
this is the only one you listen to.
Prasanna Malaiyandi:
Um, that is a wrap.
Listen On
