Disaster Recovery Testing: Start Small Before Going Big

You found the backup wrap up your go-to podcast for all things

backup recovery and cyber recovery.

In this episode, we jump into disaster recovery testing, and trust me, you don't

wanna learn these lessons the hard way.

I've got some wild stories about DR.

Tests gone wrong.

Including one from my early days at a bank that'll make you cringe.

My co-host persona, and I break down exactly how to approach DR.

Testing the right way, starting with the basics and working your way up.

We'll tell you why non-destructive testing is absolutely critical.

Seriously, you don't wanna blow up your production environment just to test Dr.

And how to set realistic success criteria that won't make you cry Another episode

from The Lessons From the Trenches.

I hope you like it.

By the way, if you don't know who I am, you're a first time listener.

I'm w Curtis Preston, AKA, Mr.

Backup, and I've been passionate about backup and recovery for over 30 years.

Ever since.

I had to tell my boss that we had no backups of the production

database that we had just lost.

I don't want that to happen to you.

That's why I do this podcast.

On this podcast, we turn unappreciated backup admins into Cyber Recovery Heroes.

This is the backup wrap up.

Welcome to the show.

Hi, I am your host, w Curtis Preston, AKA, Mr.

Backup, and if you could take just a quick moment to subscribe

or follow so that you'll get our great content, that would be great.

I am sitting here with none other than the guy who's very concerned.

About my obsession with serial killers lately, or at least one serial killer.

How's it going?

How's it going?

Persona?

I am good, Curtis.

Yeah,

about me?

so I think after this you should watch Hannibal,

the TV show, not the movie.

They both are good, but

Right?

Right?

And then we could discuss about whether people taste differently

depending on what they eat.

that's always something you talk about.

Uh,

your latest obsession is Dexter.

Yeah.

Which, which, for the record, I'm rewatching, right?

I, I enjoyed Dexter when it was on and when I had to wait a week, right?

going back to those old days

oh Lord,

I, you know, I'm, I'm kind.

I think you're, aren't you?

Like, don't you watch you, you do not watch shows when they're on.

You wait until they're done and then you binge them,

right?

Yeah, for

the most part, although these days I don't actually have like broadcast tv,

Right.

it's whatever's available on Netflix or Amazon or Take your pick.

We, we, we actually have YouTube tv, so we have broadcasts and there's

some shows that we watch on there.

Um, but there shows where, like, you know, this episode, that episode,

they're, they're all the same, you know?

And

Yeah.

you don't, it's not like, uh, it's not like

Dexter where there's an ongoing

so here's the funny thing, right?

So it was, it started off first on.

tv.

Right.

And like you said, you had to wait a

Showtime.

Yeah.

And then what I used to do, right, so I didn't have Showtime,

Mm-Hmm.

so I had to wait for it to come out on Netflix DVDs,

Oh,

right.

I would request the DVDs.

And of course I was cheap, frugal, however you want to say it, right?

And so I had like the two DVD plan, so I'd request like two DVDs, right?

So you'd get like six episodes.

Yeah.

Five or six episodes, you'd binge watch those and then you'd send them back.

And then you'd have to wait a week to then get the next set of DVDs

so funny.

in

order to watch it.

And that's I think, how I ended up watching Dexter.

And I think Breaking Bad might've been the same way too.

Yeah, you're just, you're cheaper than me.

Dexter is currently on Netflix,

so, um, yeah.

Anyway, so, and you could do, you can watch all these things while you do.

Disaster recovery testing.

'cause there's a lot of time, there's a lot of time when you do DR testing,

there's a lot of downtime, right?

You sit there and you stare at the screen.

Um, and, um, I'm gonna, I'm

gonna, I'm gonna start out a story.

What's that?

but can you really?

What,

I understand like if you're watching Dexter or pick one of these

very, the shows that pull you in,

uh huh.

Would you actually be focused or would your DR testing basically balloon like

10 times the normal amount of time?

Because you're

like, oh yeah,

I forgot to get back to that.

well, it de, you know what, it's gonna depend on the type of DR test

you do because some DR tests, there's a lot of waiting, there's a lot of,

I'm gonna start the restore part, and then I sit there for many, many hours.

And if you got one of those, then.

Doesn't really matter whether you're focused or not, as long

as somebody's keeping an eye on the, uh, percentage done.

And I'm gonna start this with a story from back in the day.

The first backup, uh, the first restore test I ever did.

The first, like, well, at least the first one I can really remember.

And this was when we had, um, you know, I was at the bank.

And which MBNA, which at the time was the second largest credit card corporation,

and I was in charge of backups.

And we had, I had talked the boss into moving to what would become

the first of many commercial backup products that I had, uh, used.

And that product was a product called SMarch, which as I've mentioned

before, should have been called SM Back because it was not an archive

product, it was a backup product.

Well, they were out, they were out of, uh, Minnesota area and,

and we had converted to them, but I like a good.

Good little backup guy.

I had done like a parallel implementation and so I was still

running my old dump tapes and I was running this new, uh, fancy tool.

And one of the things that this tool had was built-in, uh, compression.

Um, I wasn't using the compression on the tape drives.

I was using compression in the

software,

uh, to compress, uh, to go to the tape drives.

So we had our first major.

Failure, uh, file server, HP FS oh one.

I still remember the name of the server

just like get burned into your

Yeah.

Yeah.

And, and this is related to another story that we, to a friend of mine that,

that I've referred to before where she was a consultant and she accidentally

basically re, this was a self-inflicted disaster where a consultant was trying

to clean up home directories and she just did a really, really good job

of cleaning up all the home directors.

So I, I put my, uh, SMR tapes.

Uh, you know, in my front pocket and I put my backup tapes in my back, my

back pocket, and I went down there like Mighty Mouse here comes said the day.

And um, some of you will get

Not available on iTunes, I

Not available at iTunes.

And we went in there and I put, these were DDS, uh, tapes, right?

And these, 'cause this, we were all hp, HP loved DDS.

And so I popped into DDS tape, I pulled up my SMR software

and I started the restore test.

And, well, it wasn't a restore test.

It was, it was, I was testing, I was testing in anger, uh, you know, we

were testing like this was for real.

And, and I had not done any actual testing.

I.

And so I kicked off the Restore and um, it.

I'm, I'm watching and I, I, I, you know, like a good little Unix boy.

I had a, for a while loop

running and it was doing a, a DF on the, on the, um, on the, you know, to

display the size of the file system.

And I'm watching, and like a long time was going by and there

was no change in the size of

the file system.

And I was like, this is weird.

And then I went over and I looked, I just.

Just outta curiosity, I looked at the

tape drive, right.

You know, it's kind like, it's kinda like your car dies, you know, you

open up the hood like, I have any idea what's going on in inside there.

You know, look in there

and I see the, I see the light on the tape drive and, and it

goes, B blink, B blink, blink,

one 1002, 1003, 1004,

blink, blink.

Right?

And there were these giant pauses in between.

The blinks.

And so I'm like, that's strange.

It always blinks when it's, you know, when it's reading or writing data, right?

So I called up, I called up the guys and I said, Hey,

The

vendor.

Yeah,

vendor.

Yeah.

Called it the vendor.

What's going on here?

They said, well, by any chance did you use the compression feature?

And I said yes.

Yes I did.

They go, yeah.

So let us explain to you how the compression feature works.

So when we're backing up files, um, basically, um, we do the equivalent

of compress minus CI think it was, to, to send the result to standard out

and it send it straight to the tape.

But we don't know how to do that on the way back in.

And so what we do is.

We, uh, we, we read the tape, we read the entire file that we're gonna restore

into, and we restore it to temp, and then we run uncompress in place in

temp, and then we move the res, the

uncompressed restored file.

To where it's gonna go.

And we do that one file at a

Oh,

because we're concerned that we're gonna fill up temp.

And I'm like, oh.

So like if temp is, if I have a single file that's bigger than

temp, it's just not gonna work.

They're like, yeah.

And they're like, if this is a concern, we suggest perhaps

you don't use this feature.

it would've been helpful to know before I needed

Right, right.

And so thank God I had the, I had the backup tape in my backup

pocket and I pulled 'em out.

I was like, okay, we're just using, you know,

uh, dump here.

And, um, and we restored and everything was basic, right?

Um, and luckily I had backups from the previous night.

And really the moral of this story is.

Don't do that.

Right.

Don't test your backups before

you need them.

Right.

Do a DR test and that's what we're talking about today.

Do a DR test before you actually need to do DR and, and thi this May, we

will see I the, you know, sometimes when we have these episodes, persona,

you know, you know, we have these conversations beforehand and I'm

like, I don't know if we can fill up an entire episode over this topic.

We said that I think over the last one.

Yeah,

And it ended up not being a problem at all.

This is not one of those episodes.

This is one of those episodes where I think we might go along and we'll end

up turning this into two episodes, um, because there's a lot to talk about here

because before we do any testing, right, um, what do you think we need to do?

Well, you, well, you need to understand,

there.

By the way, there's probably no wrong answer here.

Uh, but

well, well, I think before you could

un unless your answer is, unless your answer is nothing,

um,

no, no.

So I was, I was going through my head.

I was thinking even before you get to testing,

Yeah.

you need to understand what are the business requirements for how quickly

you need to bring up that site, which in which actually you don't

necessarily think about as testing, but that's actually part of your

backup system design before you even get to the testing part.

Yeah.

So we, we have to agree on what success would be, right?

And we have to agree on obviously what we're gonna test, but we have to

agree on the parameters of that test.

And so, um.

What?

What do you think are going what?

Go ahead.

You and I was just going to ask about like your parameters, right?

What are

those parameters in my mind?

Some of those are how quickly do I need to be able to bring up what the scope is,

right.

Am I failing over and

trying to recover a file, an application, a data center, right?

What am I looking to actually test?

So it depends what you're trying to test.

And what the extent is that you want to test.

And also, I think the other thing is how close do you want to get to testing

an actual disaster as part of that?

Right.

Um, and I, I would say that there are different kinds of DR tests and, um.

The, there's, and, and if you haven't done any, I would say let's start small, right?

If we've never, if, if we've never done a DR test of any kind, I would

probably start with, what are you

I was thinking about the test you don't wanna try is who is the guy in Alaska?

Oh yeah,

yeah.

Okay.

We definitely have to put a link to that episode.

That was the most amazing episode ever.

In fact, you know what?

We'll pro we'll probably replay it over, uh, over the holiday break, Uh,

by the way, we love him and it, and, and it had a happy ending, but oh my God.

What, what a

nightmare,

think he was rebuilding a raid array, if I recall.

He was swapping the disks around

it was self-inflicted in that he said, I want to move the discs around.

Because they were like.

different sizes I

Yeah.

Yeah.

Like, and he's like, the only way I can do this is to just, to just

wipe everything and start over.

And he is like, yeah, okay.

So that's what I'll do.

I'll just wipe everything and then I'll restore everything.

My s my backup system works.

And this is how he tested his backup system.

Please do.

He definitely learned some things along the way, but, um, yeah, so don't do

Don't do that.

Yeah,

Don't do that.

Uh, but

learn from his example.

Listen to that episode and, and,

and have

a heart attack as you're listening.

because the thing that ran through my head was someone who's never done DR

testing would've been like, yeah, I'm just gonna shoot in the head my production

site with all the applications, and I make sure that it, or just test it out

and see does it fail over properly?

Yes.

The key here is non-destructive DR testing.

Right?

Um, and so, yeah, so, so to go back to like setting the scope, I, if this

is your first time doing DR testing, I would set the scope as small as possible.

In fact, if you've never done DR testing, I would not even be doing DR testing.

I would just be doing restore testing

and

What's the difference, Curtis?

What's that?

What's the

the question is, the question is are we going to declare a disaster

and say that this site is down in some way, um, and then we're

gonna fail over to another site?

Or are we simply just going to bring another server online?

Right.

De depending on how you define, uh, some, and, you know, depending on the

day of the week and the day of the year in which year it is, I have called.

Every time you have to restore, uh, a server of any kind, a disaster.

It's just a disaster of different levels, right?

So if the server, if a server caught on fire, that's a disaster,

right?

Um, and, or, or if a server just died, right?

Or if, like the, the story in the beginning, I think that was a disaster

because it took out a production workload,

right?

So I would say perhaps start with a single.

Production workload and restore it.

First off, just restore it, you know, like in place, not, not in place, not

in place, I meant like within the data

center or within whatever computing environment you're using.

And then start talking about VPNs and

you know, that sort of stuff.

So, uh, that would be defining the scope as small as you can

for the first test that you can.

so maybe like, uh, like maybe it might be a directory within a file server.

If you've never done

any restore testing, yes.

A simple directory within a file server, does our backup system work at all?

Right.

Um, and then the next, the next is going to be some type of

recovery of an entire server.

Hopefully you have that server virtualized because doing that is going to be,

uh, obviously much easier assuming you're treating that VM as a vm.

Uh, if it's not virtualized, then we we're gonna start going down the

level of, um, bare metal recovery,

right?

BMR By the way, we should probably do an episode on that.

I

didn't think about that.

We should do an episode on that.

I know you keep mentioning server as you're talking about this.

Would you also qualify that that could be server or an application?

thanks for bringing that up.

That's why you keep me around, you know?

So it depends on what you mean by application, right?

Um.

You could, these are like the different

level.

You talk about restoring a directory.

I would also look at restoring a single database within a server, right?

If that's what you mean by application, then I would say yes.

Sometimes when we say application, actually a lot of times when we

say application, what we mean is

application

Multiple

on multiple servers and multiple things, all interconnect.

And if that's what you're talking about, I'm gonna say no

yeah.

yeah, I

was thinking like a, my,

time out.

yeah, I was thinking more like restore your MySQL database.

Exactly

right.

Um,

table within your MySQL database.

Right.

yeah.

Um, and again, non-destructively,

Yeah,

right.

Um, I.

Uh, just, just a, just an entity and you can try all of these things, right?

You can try restoring a database.

You can try restoring an entire, all of the, all of the databases on a server.

You can try restoring a server with the databases, um, and or any other

applications that might like a web server.

You can try all of these things individually and make sure that

you've got those pieces down.

That this is about defining the scope.

Try the all of the different pieces first and make sure you've got the,

the recovery path for each of those parts of your infrastructure down

before you decide, okay, we're gonna pretend we're gonna blow up the data

center.

and the other reason to bring that up, it's important to test these different

types of components because there are gonna be different nuances like how

you deal with databases and recovering.

That is definitely gonna be different than file servers, which is probably

also gonna be different than servers.

And so it's important to understand the nuances of what is possible and

the steps for each one of these.

And also the different backup systems.

So if you've got.

Physical, you know, you've got physical servers that are not virtualized.

You've got servers running in Hyper V or VMware or you know, any, any, any sort

of on-premises virtualization set.

What was the third one

Broadcom,

brought?

VMware,

It's, it will always be

VMware to me.

It will always be world, um, the, um, if you've got.

VMs running in, uh, AWS, GCP Azure.

If you've got basically all of the different places that you

have infrastructure, probably have different backup and recovery

methodologies for each of them.

And so you should also be looking at testing all of those.

And you should be looking at testing each of them individually as a,

an overall process that you're working towards developing, uh,

declaring a much bigger disaster.

Yeah.

I, I think it's important also to be familiar because sometimes you might have

a real disaster that doesn't require you

to recover every single component within that higher level application, right?

So being familiar with the individual components is also important

depending on what the disaster is.

Yeah.

Agreed.

Um, and you know, there, there, there's an application that we haven't

talked about in terms of including or not including it in your Dr.

Scope and that is, what about SaaS applications like Microsoft 365?

Um, there are a couple of different scenarios there.

One is your.

Um, account is damaged in some sort of logical way, meaning logical

corruption, meaning a ransomware attack.

You deleted it.

Right?

We, we've covered that

on,

provider damaged it.

yeah.

The provider.

Well, that's, that, that was, I'm gonna list that as like a

third.

Well, if they damaged your account.

And there is an example of that.

Uh, for example, the sales for

Yep.

That's what I was thinking.

story where they went and blew up everybody's permissions and everybody

had to restore that themselves.

Um, that man, I hate that story.

I really do, because Salesforce, in my opinion, did not own up to.

Uh, they, they, didn't step up to the plate

Yeah.

at, at the time.

I remember writing a blog post for, uh, Druva.

I was working for Druva at the time, and I remember writing for Blo, a blog

post that said something like, proof that, that Salesforce should not be

trusted with your backup infrastructure.

Um, 'cause they clearly don't know what they're doing.

The but there's, but, so there's your account being damaged in some way.

And then there's OVH Cloud.

And what happened there Where the entire infrastructure goes?

Poof.

Right.

So, uh, I think you should have that as, as you, you should come up with that

as a scenario that you need to test.

It's going to be challenging in most cases because just let's just

talk about the different scenarios.

Let's say it's AWS the way most people back up AWS If AWS goes down

and takes your backups with it.

You're screwed.

You're screwed.

Um, the way that most people back up most cloud infrastructure.

And then there's the fact that most people trust their SaaS provider for data

protection, which they should not, right?

I talk about that all the time.

They should not, But, but but even if you're backing up your, your

data, um, to a third party and, and it's not, they should not.

Right?

I talk about that all the time.

They should not.

This has been

a day in eight.

Problem though,

This is an age old problem, but we're talking about testing today.

So by the way, this is definitely gonna be two episodes.

We haven't even gotten to the testing yet.

All we're talking about is setting up the requirements.

So,

so this is definitely gonna be a second

episode.

Um, so we talk about, we talk about setting the scope and we

talk about starting small first.

Each of these individual components in your infrastructure and um,

were you about to say something?

that this is not destructive and

not disruptive.

Yeah.

Nice, nice.

I like that.

Non-destructive, non-disruptive DR testing.

I

like it.

because you don't wanna take down your production.

You don't wanna affect, say, ongoing Dr.

Resiliency that's available on the secondary site because you're

about to do some of this testing.

There are cases where you do want to impact that, but when you're doing sort

of these individual component levels, you may not want to impact your overall DR.

Posture while you're doing this

Yeah, that that last one is probably the hardest and it

and may actually be impossible.

What are we talking about there?

We're saying that if you have a DR system, I hope you have a DR system

if you have one, you know, see if there's a way

to test your DR without messing up your dr.

Um, I'm not sure if that's possible

in, in many scenarios, but.

but one of the things I think about right is a lot of data protection

vendors, they do test and dev, right?

You can spin

up a copy off of your storage that is a writeable copy that you can then use

for testing out your recovery systems

without impacting the actual recovery instance.

right.

And so that's what I'm thinking about is just like those sort of scenarios or maybe

you're able to wheel in an extra server or beg, borrow steel, an extra server to use

for your recovery testing or DR testing,

Yeah.

You know, it's, it's funny that you say that because I, I, you know, when

you, I like this idea wheeling in a, a

server.

I mean, I, I, I think many of our listeners, you know, and, and I, and every

time I, by default, I'm always talking about data center, even though I, you

know, and then in my brain goes, Hey,

nobody has a data center anymore.

Um, I don't think many people are wheeling in a server.

I think that they're, they're, you know, they're looking at.

Cloud

infrastructure as a way.

I, I know that not everybody can do that.

And that's obviously another thing that you have to decide upfront is

how are we going to do this recovery?

Are we going to do it in the cloud?

Are we gonna do it?

The alternate infras, alternate infrastructure, um, these are all

things that you have to decide upfront.

We have to decide what it is we're gonna restore.

We have to decide, um, where we're going to restore and, and how.

Right and.

The deciding the where.

Again, this is all about planning.

This is all stuff that needs to be

decided upfront.

This is also something that's going to be part of your backup and Dr.

Design, you will have decided up upfront, you know how we're going to do Dr.

Uh, well, disaster recovery, how we're going to do it.

And that place is most likely the place that you're going

to, uh, be doing DR testing.

And, you know, there are a lot of choices here.

Cloud infrastructure, I think, is the best choice for most environments.

Another choice is aging infrastructure, right?

So

you move, you know, you move your older stuff out and that

becomes your DR environment.

Um, another choice, another very common choice is basically, um,

there, there's two ways to basically.

Rent infrastructure for the purposes of disaster recovery.

One way is to contract with, um, you know, a company that will provide

what you need if and when you need it.

Uh, and that costs, you know, let's say this much.

And then there's a company that will provide everything you need,

always available to you all the time.

Even when you don't need it.

And that will cost this much.

Yeah, exactly.

I can't do the fingers, I gotta do the hands.

Right.

It's significantly more.

Uh, and, and this is why I push everybody to the cloud as much as you can.

'cause that's the beautiful thing about the cloud, is that you can just literally

snap your fingers and, um, you know, and use, and you can also use infrastructure

as code so that you can just make all of the hardware that you need

magically appear.

What's that?

and work right.

And work.

Yeah.

Magically appear when you need it.

And then the moment you no longer need it because your test is over, you can

also snap your fingers and it all goes away and you just pay for it only when,

uh, it's, you know, up and running.

Um, so the, the next thing to talk about is, so we decided

what it's we're gonna test.

We decided where we're gonna test it, how we're gonna test it,

what about our success criteria?

Yeah.

I think this is important to note upfront what it means to be

successful, but I think it's also important to be realistic, right.

With your success criteria, especially if this is sort of your

first time doing this, because I remember the story you tell Curtis

about your, your runbooks that you

Yeah.

Yeah.

when you used to work at the bank.

And how a lot of times they were not able to get through and

actually do the recovery because they, a step would be skipped or

something wouldn't work properly.

And so I think it's sort of a learning process.

So don't be too hard on yourself.

If the first 10, 20, a hundred times you try doing your recovery testing, it's

not like a hundred percent successful.

Yeah.

may not be your fault.

Things change, environments change, hard

work may change, right?

There's so many other factors,

but.

this is, this is very closely related to the discussions we've had over, um,

doing, uh, um, cyber recovery testing.

Right?

And your disaster recovery very likely will be part of an

overall cyber recovery process.

And I agree with you that I.

Um, it's interesting.

I was go mentally, I was going somewhere completely different.

But you, you, once again, this is why we make such a good team.

Um, you were more like, Hey, make the re make the, um, um,

you know, the requirements.

Uh, be nice to yourself, especially if it's the first time out.

You know, requirement number one, no one dies, right?

Yep.

No fires are created.

No one quits.

Um.

And, um, the, you know, set your expectations low and nobody can

take you, take 'em away from you.

I'm borrowing heavily from, uh, Michael p Connolly, the comedian who,

who says that's the, the happiness to life is to lower your expectations.

And he's like, my goal for today is to go to the bathroom outside my pants.

Um,

so yeah.

Set the, uh, you know, the success criteria low in the beginning.

Um, where I was going was of course, REO and RPO, which we talked about before.

The, that, that is definitely going to determine and the overall success,

once you click the stopwatch and you begin the recovery process.

And then you click it, that everything is back up and running and, and you've

tested that, that, that whatever it is that you destroyed or you pretended you

destroyed, is now up and running and fully functional Again, as we've mentioned

in the RTO and RPO, that doesn't just mean the time of the restore, it's,

it's, you know, it's.

ah, here's a question.

Is it RTO and RPO or is it RTA and RPA?

the, it's, that is the objective,

right?

The RTO and RPO are, are the objective that we are shooting for.

And so the goal, um, in a recovery test of any kind is that the RTA, that's recovery

time, actual, uh, and recovery point actual, are less than, uh, or equal to

the RTO and RPO.

go back and listen to our episode.

We covered it, I think a couple episodes ago, right?

No, it was, well, well, I don't know.

It depends on when this one gets published.

We just, we just did it.

Yeah.

That's why I'm saying we just did it not.

Oh, was that really the last episode?

Well, it just published.

Um, uh, but again, I don't know, you know,

we'll see how these things, you know, but, um, yeah, if you're, if, if RTO and RPO

don't just roll off your tongue and you don't know what they are and, and, and,

you know, all of that stuff, it, they literally should be in every conversation

having anything to do with backup and Dr.

Design.

But that to me is the ultimate success criteria, right?

Another success criteria is the degree, and this, this is gonna be

a percentage, um, a per a percentage achieved, and that is the degree to which

you were able to follow your runbook

and just do what's in the runbook.

This is

what you, this is what you were alluding to before, right?

Well, and hopefully you have a runbook.

Yeah.

Like you said, that, that, that assumes you have a run book.

Yeah.

Um.

And, and the way we always did this at the bank, as we mentioned

before, is that we would have someone other than the person who runs the

backup system do the DR testing.

Right?

Here's the runbook, please follow it because we're gonna pretend

that Curtis got hit by a bus.

It was never anything nice.

It was never

that I won the lottery and then just flew the coop.

It was always Curtis got hit by a bus or got swallowed up in the, in the sink.

The great sinkhole of 2024.

Um,

long do you think it's gonna be until people don't even know what a bus is?

I think, I think we're good.

I think we're

good.

You know, I, um, it's funny, definitely an aside, when I was working the

election, uh, the last two weeks we're at a school and one of the hardest

time we're, we're, we're at a, like a multifunction building next to a school.

And one of the hardest times was during pickup and drop off because

there are all these parents that are picking up and dropping off their kids.

And I was like.

You know, if only there was like a large vehicle that we could put all the kids

in and like we could like paint it yellow so that people could see it and then like

have a sign that comes out and make sure, oh, traffic stops in both directions.

If only we could do all that.

And then all these parents wouldn't have to like, uh, take their kids

to school and waste all of that gas and all of those cars sitting there.

And I'm sure they're, the gas is running while they're sitting

there for a half hour waiting for their kid to come outta school.

Anyway, I digress.

I don't know.

Hmm.

But, um, so that, I, I think that, we'll, we'll stop there because

basically it, the number one thing, the number one success criteria that

I'm gonna say is that you know what your success criteria are before.

What are we gonna restore?

How are we gonna restore it?

Where are we going to restore it, and how long is it, you know, what, what

timeframe are we trying to fit in?

And also what other, what other, like the thing we talked about with the.

With the, uh, uh, the other criteria being that, that we can

do it without Curtis' help, right?

Or what, whatever, you know, whatever your, your Curtis is, right?

Um, decide on what all of those are upfront and you have a much

better chance of being successful when you actually do the recovery.

What do you think?

No, that makes sense.

Okay.

And with that, I will thank you once again for being a great co-host persona.

I try, I try.

This was a fun, I I like talking about disaster recovery testing.

Yeah, absolutely.

And we will, uh, hope you guys enjoyed this as well.

Uh, that is a wrap.

The backup wrap up is written, recorded, and produced by me w Curtis Preston.

If you need backup or Dr.

Consulting content generation or expert witness work,

check out backup central.com.

You can also find links from my O'Reilly Books on the same website.

Remember, this is an independent podcast and any opinions that

you hear are those of the speaker and not necessarily an employer.

Thanks for listening.