Disaster Recovery Lessons from 9/11 and Beyond

You found the backup wrap up your go-to podcast for all things

backup recovery and cyber recovery.

In this episode, we talk about some hard earned disaster recovery lessons

from major events like nine 11.

We talk about what we learned about DR that day, and we talk about

those critical human elements of DR.

That people often forget, like where your recovery team is going to

sleep when the hotels are all gone.

Disasters happen and they're never convenient.

Whether it's terrorists, hurricanes, or ransomware, you

need to think through what you do.

If you're completely isolated from the world, the time to

learn these lessons are now.

So I hope you enjoy this episode.

By the way, if you don't know who I am, I'm w Curtis Preston, AKA, Mr. Backup,

and I've been passionate about backup and disaster recovery for over 30 years.

Ever since.

I had to tell my boss that there were no backups of the

big database that we just lost.

I don't want that to happen to you, and that's why I do this podcast.

On this podcast, we turn unappreciated backup admins into Cyber Recovery Heroes.

This is the backup wrap up.

Welcome to the show.

Hi, I am w Curtis Preston, AKA, Mr. Backup, and I have with me a guy who

is as lazy as I am when it comes to how to move rugs around the house.

Prasanna Malaiyandi, how's it going?

Prasanna.

I am doing well, Curtis, by the way, isn't it amazing how quiet a room gets

when you put something on the floor?

Yeah, I, I, I wonder if anybody, uh, if anybody notices the difference

in sound because it, you know, and it's been, they probably just got

so used to the other sound, right?

Because I. I went with LVP, you know, upstairs and um, and then in

my office and it just got so echoy and I put the stuff on this wall.

And this over here is a acoustic panels.

Um, you know, behind me there are acoustic panels up on the ceiling

and it still wasn't the same.

I. And then I found, you know, I, I, I, this, this rug is from actually my

living room because we bought a nicer rug and, uh, a nicer, bigger rug.

And so we put that one down there and then I moved here.

But when I moved it in, I, I, I did what I thought was like

a really weird way to do it.

I didn't actually move the furniture.

I was like lifting it up and trying to do it without having to move everything out.

And you said, well, that's exactly how I did it.

exactly.

Yeah.

And I, I, I thought it was just me, but apparently.

It wasn't.

because at least with the rug, well, it depends on how your desk is too.

But like you can at least unfurl part of it and just kind of like shove it down.

Like you just lift

up the front part and

then you lift up the back part.

So,

yeah.

I, I think different people might go, no, no, we're gonna,

we're gonna move everything out.

We're gonna put the rug down and we'll move everything back in.

And they might think of that as easier, but I was like, I

don't wanna move everything.

well, it's more than moving.

It's setting everything back up again.

Yeah, I, I looked into it.

I could have potentially moved the desk without disassembling everything.

It was just, you know, but, um, yeah, so I, it was just nice to hear

when I was talking to you for once.

You weren't like, 'cause so many times I tell you about something that I'm

doing and then you're like, that's the dumbest thing I've ever heard.

Why would you do it that way?

And you're like, oh no, that's, uh, that's how I do it.

Yeah.

Especially if you're space limited.

Like for

What's that?

especially if you are space limited

Yes, space limited is true.

I was just looking, so I have my desk right here and then

like that way, that way I

have the door, but there's no way I could fit my desk outside of

the door with it fully assembled.

Oh yeah.

I'd have to at least take the top off, which is just a giant chore.

So.

Oh yeah.

Well, in my case, I don't have that excuse 'cause I have two French doors, so I

wouldn't have had that excuse, but yeah.

But it still, it still would've been, it still would've been annoying.

And so, but we figured, we figured it out and now I have better sound.

It's a beautiful thing.

And, uh, so, uh, today we're gonna talk about DR and disaster recovery and

lessons learned, and especially, um, some lessons learned from at least one

major event, uh, one major disaster.

And, you know, I, I don't want to, um, you know, the, these, these events,

especially this event, disasters are, are always difficult on, on the people.

Uh, and, and I don't want in any way make light of those disasters

or I, I don't know, whatever, whatever I'm trying to say.

Right.

So proper respect to the people.

'cause some of the, well, especially the one, the main one that we're

gonna talk about, people died in these disasters and, um, you know,

with respect to those people.

Having said that.

We can learn from the things that happened, uh, at that time.

Right?

And so let's talk about, and and the disaster.

The, the main disaster I'm talking about at this point would be what

we all called nine 11, right?

So September 11th, 2001.

Uh, I lived through it.

You lived through it.

Uh, so how old would you have been in 2001?

I would rather not say

I was very young.

You were very young.

I, I was actually in college.

okay.

I was here, um, and um, I was in this house and my kids were young.

My oldest would've been, uh, seven, and my younger one would've been in four.

And what I remember was she might've been in, she might've been five,

she might've been in preschool or might've been in kindergarten.

What I remember, and for those that don't know, I live in North County San Diego,

which means I live just south of Camp Pendleton, which is, uh, you know, one

of the biggest, uh, Marine Corps bases.

The, the idea was that we were under attack because there were multiple events

that were happening simultaneously.

Right.

Multiple planes hit the, the trade center, there was the plane that hit the Pentagon.

There was the plane that went down in Pennsylvania.

There was this feeling that, like we were being attacked

as a country, which we were, and living near a military base.

I don't know what I thought I was going to accomplish by keeping my kids home

from school, but that's what I did.

Yeah.

I was just like, I'm gonna, I'm gonna, I'm gonna, I'm gonna, I'm

gonna keep my kid, like, it, it, like, I, I, I don't even know what

I was thinking, but like, you know, thinking back on it now I'm thinking

that probably I was thinking that, um.

I, I, I, if the world's gonna end, I want my kids near me.

I

mean, it was like, it was, it was, it was, it's kind of morbid.

But that's,

I, I just remember that I didn't, you know, that, that I kept my kids near me.

Um, and I remember that I knew multiple people that worked in

the Real World Trade Center.

None of them were hurt that day

Mm.

and for various reasons.

One, uh, and I know one person that was in the World Trade Center that day.

I knew other people that worked in the World Trade Center, but

for one reason or another, chose not to go into work that day.

Hmm.

I know another person that was supposed to be on Flight 11, which

was from Boston to New York, which was the flight that ended up going.

I don't know if that's the one that went into the Pentagon or

if that's the one that crashed in Pennsylvania, but I knew a person that

was supposed to be on that flight.

That person did not get on that flight.

And, uh, and then the other person that, that I knew was the person in

the, in the trade center, it was, uh, Michael Hingis that was, uh, the, the

blind person that made it, that made it down thanks to his seeing eye dog.

Um, and he, you know, he.

Somewhat famous.

As a result.

He actually went on to become a motivational speaker.

And, um, and, uh, just to, just to lighten things up, I, I'll talk about,

uh, how I first met Michael, and that was, and I, it seems like I've told

this story relatively recently, but I was at my very first trade show.

This would've been in the early nineties, and it was at, it was in

New York, um, at the Javit Center.

In Manhattan, and it was Unix Expo, and I saw, uh, these hot swappable

dish drives, you know, where, where you

could push a button and pull the drives out.

And I just thought that was the coolest thing I'd ever seen.

'cause at the, at the time that, that was unlike anything I'd ever

seen it, it, that was really, really new and really, really cool.

Now we just take it for granted.

But back then it was really cool and he was, he was the one that was.

The se standing in the booth demonstrating these, these, uh, disc drives and

literally his, his shtick was.

It's so easy a blind man can do it.

Right?

And um, and we were like, this is amazing.

We're like, you guys are the only ones with it.

And he goes, yes, we are the only ones with this product.

And then we walked through the trade center or the trade show and we saw

several other vendors with the product.

And one of us said to the other.

Well, in his defense, he can't see the other vendors.

So true.

Um, do you, do you remember, do you have memories of that day?

Of course You do.

Oh yeah, so I remember, so I went to school, I was in Pittsburgh actually.

Mm-hmm.

And so we had a lot of friends, or I had a lot of friends whose families were

living in the city in New York City.

Right.

And so I remember everyone just was very, very concerned.

Um, and like you had mentioned, there was sort of the plane flying towards.

Uh, over Pennsylvania.

And so everyone in Pittsburgh was keeping a close eye being like, Hey,

where is that plane actually going?

Because

it was supposed to fly over Pittsburgh,

right.

right?

So I remember everyone sort of being worried, concerned because they had family

and relatives and they weren't able, like all the phone lines were shut down, right?

So.

No one was able to figure out like what was going on.

I remember going with a bunch of other folks down to the common area

and just kind of like just being like shellshocked as we're watching the news.

Yeah, I, yeah, absolutely.

Um, there's a, there's a great line in the beginning of a movie that I like.

It's become problematic now.

Um, maybe it was always problematic, but there's a movie called Love, actually,

and in the beginning, which is, uh, voiceover from, um, Hugh, Hugh Grant.

And, and, and he was saying on nine 11.

There were a lot of phone calls that were made from the plains,

and he said, to my knowledge, none of them were messages of hate.

They were

messages of love.

Hmm.

I just got a little of a clump there.

Anyway, so, okay.

So, um, when we think about that event, there is, in my world, we

immediately started talking about.

The, we saw the things that happened to companies, and we're gonna talk

about, I'm gonna talk about sort of two, two different kinds of companies.

One of them.

So first off, let's talk about, uh, the, the difference between

a cold site and a hot site.

Do you, so when we talk about disaster recovery, there was this

idea that we, that we're gonna have another site, like ready to go

Yep.

and we talk about cold site and the hot site.

Do you want to talk about what that means?

Yeah, so a hot site basically is you have a site, right?

A disaster recovery site that is fully operational, has all the

equipment, has everything replicated to it, and basically once you push

the BRI big red button, right?

Everything sort of fails over.

It's available, operational, all ready to go, sort of ready

to serve traffic and take over, usually minutes to hours within a

failure.

right.

And then a cold site is very much the opposite of that, right?

It's a site that's sort of ready to start a restore.

Uh, like I suppose there'd be a, there'd be a no site,

Yeah.

right?

That's where, uh, a bad thing happened.

And we're gonna go find some hardware to restore

to, uh, cold site, the, the.

Implication there is that you have some hardware ready to go, but

you haven't restored anything.

A warm site is somewhere in between those two things,

I've have, have you heard of this term that some people call pilot?

A pilot light

Yeah.

talk to.

Talk to me

So it's basically not quite cold, but not quite warm or hot, right?

So

it's a little bit better than cold, but not quite to the extent.

And a lot of the trade off comes from not necessarily having to

eat all of the costs upfront.

So

as an example, if your pilot site is, say, in the cloud, you might have your

data available, but not necessarily your compute and everything else ready to go.

Yeah, I think that I, I think I would still call that a warm sight, but

I mean, there is this concept, it comes from the concept of a pilot

light.

I. Right.

Which for those of you that don't know, when you have a

gas old school that

nowadays we have electronic ignition, but there used to be this in inside,

if you had a gas water heater or a gas furnace, there would be this

little flame that would burn all the

time.

And that would, that's called your pilot light.

And if the pilot light goes out, then you're gonna have to relight it

because otherwise your, your heat won't work.

Um, modern days we use.

electronic electronic ignition,

but, um, because a pilot like just wastes a lot of,

yep.

It's always running and you're always consuming gas.

Yeah.

Um, so there, so the, the best from a DR perspective, right?

The, the, the, the Cadillac, if you will, is, um.

The, I don't know if that's the right term anymore because nobody buys Cadillac, the

Ferrari, the Rolls Royce.

Does anybody buy, do they still make Rolls

Oh yeah.

Okay.

Um, is the hot site

that it's, it's, it's ready to go.

Number one.

It's ready to go when you need it, and, and two, it's kept and

it's ready to go within a, a few.

Minutes or seconds, right?

It's kept as up to date as possible, as much as technology

would allow you to do so.

And one of the challenges with a hot site is, is latency.

And so you might want to put the hot site

as close

as you can to, uh, the, the site that you're preparing

and what happened on nine 11.

So there were many companies to, for their hot site, uh, or had

their main data center in one tower

and had their hot site in another data in the other tower.

which makes perfect sense.

As long as nothing would take out both towers

Yep.

and.

So unfortunately, as we know, you know, both towers, I mean, that,

that's what I woke up to by the way.

I, I

woke up to my wife saying both of the, 'cause I'm on the west coast, so both

of the towers had already collapsed

as you know, as I was waking up.

And so people lost or companies lost their.

their.

primary site and their hot site in, in the same moment.

And so one of the things we learned, nine 11 is to make sure that you are,

if you're doing some sort of hot site or warm site, is to put that site, I'm

gonna say nowhere near, um, that the, the site that you're being protected.

Now let's talk about that.

Um, there were actually some attempts, um, I lived through

this because I was working.

For companies at the time, and that is there were attempts at regulation to say,

If you're a bank or whatever you need to put, if you're financial training,

you need to put a copy of your data.

Uh, that's, that's hot over 200 miles away.

Yep.

was an attempt at regulation.

What's the problem with that?

Um, 200 miles away if you're, depending on the type of disaster isn't far enough.

And so that's

not the problem.

But the second is latency.

yeah, that's the problem, right?

It's just simply not feasible because the, the, the round trip time, the 200 mile

round trip time, uh, was just far too long that couldn't keep the data up to date.

depending on the

Based on the technology at the time,

Yeah.

Well, and I think it depends.

So I do so.

Many years after nine 11, I was working at a storage company and one of the

things that they did was they also like talking to financial customers, right?

Is many of 'em had what they would call dark fiber,

Yep.

right?

Where they would basically run fiber optics, two fiber network between

their two sites, and it would.

You're right.

It wouldn't be completely eliminate the latency, but it would definitely

help versus say routing it over a public network of any type.

Yeah, we definitely, you would.

I, I think that there was an assumption of dark fiber, uh, at that point, but even

200 miles on a straight piece of glass, speed of light has a speed of, speed of

light is not, it's not instantaneous.

It's whatever it is.

187,000 miles a second or whatever,

six hundred or something like

something like that.

Right.

Um, the, the round trip time is gonna be measured in milliseconds.

It's not gonna be, it's not, it's, it's, it's going to significantly increase

latency, especially if we start talking about synchronous transfer of data.

Right.

Now let's talk about synchronous versus sacred.

Synchronous versus asynchronous transfer of data.

You want to give that a shot?

Yeah, so synchronous is.

Basically a right comes into your production site.

It gets forwarded over to your DR site.

The right gets, now there are different flavors, but typically the right gets

committed on the DR site acknowledged back to the primary site, and then the

primary site acknowledges the client during which, so you have to add up

basically the latency of going over the writes on both sides, coming back before

the client acknowledges, in which case you're guaranteed that that right has hit

both sites and so the client can move on.

Which is great as long as it doesn't take too long.

Yes.

So that's synchronous and asynchronous is something, is, is uh, different than that.

And we, we've had some different, between you and I, we've had some

different understandings of different kinds of asynchronous, but go ahead.

Yeah, so for me, asynchronous is, well, what I would call semi synchronous, but

that's a different case is where, uh, you accept the right on the production,

you forward it over to the secondary or DR site, but, but while it's in

process of being committed on the other side, you can acknowledge the client.

So there is a lag.

Um, you could decide how long that lag is, depending on technology and the vendor.

Some allow you to specify at an IO level so you can say, I want

10 transactions outstanding.

Others allow you to do it in terms of seconds.

So I'm allowing up to 10 seconds or 30 seconds, um,

Before you start, before you start kicking back a performance issue to the client.

Before

you start Yeah.

Putting back pressure.

Now

interestingly, there is a mode, which I'm not sure if you're aware of,

uh, that some financial co companies requested, which is called Domino mode.

Talk to me.

So it's a form of synchronous replication where it, because in

synchronous replication, typically you write to the client, it sends it over.

If the right fails on the secondary, it'll still accept it on the primary.

And acknowledge back to the client, right?

So you're not guaranteed that it'll stop writes if it can't write to

both sides At the same time, there's a mode called domino mode where if

it can't write to both sides, it will not acknowledge the client.

So that's why it's called a Domino.

One takes out the other.

I, I would think that,

this is one of those, this is one of those things where, you know, uh, this

is reality versus the idea, right?

To me.

domino mode that you described, that's synchronous.

Anything other than that is not synchronous, right?

And anything other than the domino mode that you described,

I would call asynchronous, right?

So these either synchronous, it's sort of like immutable and not immutable, right?

It's either synchronous or it's not synchronous.

And if it's synchronous, then it shouldn't acknowledge the right to the

client until both writes have been done.

And if one of them fails, then they're not done.

yeah.

So at least from most of the vendors I've seen,

they've never implemented it that way.

Yeah.

That's interesting.

Well, they're wrong.

So, um, uh, yeah, so that was, so that was a lesson we thought we

learned at the time, but we need to make sure we put it far enough away.

But then they were like, it's gotta be synchronous and

it's gotta be 200 miles away.

They're like, eh, it's not gonna work.

Right.

Um, nowadays, you know, you hinted at it earlier, nowadays we would do this with

the cloud and we can put it actually, because, you know, you, you did say that

your first problem was that it wasn't far enough, and that's probably true, right?

Because especially when we start talking about certain areas

like Southern Florida, right?

200 miles isn't far enough.

Um, and um, so with the cloud, you can put it.

Pretty much anywhere.

Now, if we're going to do that, if, especially if we're gonna use

public networks, we're pretty much going to have to use asynchronous

of some sort, right?

Uh, so we're gonna send the data and put it another place we're going to,

you know, like you said, you can have a buffer, you can have a certain amount

of time that it's allowed to get behind before it, uh, like you said, put, what

do you mean when you say back pressure?

So this is where you start to, um, elongate the time

before acknowledging a right.

So

to the client, because typically your client will sort of throttle itself

because at some point your latencies are gonna get into the seconds

and they'll be like, no, no, no.

Something's going on.

I'll slow down.

right.

Because otherwise you're just gonna start dropping the writes.

And.

And, and this is a, a configuration choice on the part of the customer

where they can say, I don't want to ever put back pressure.

I wanna, uh, you know, that, that the data protection is less important

than actually getting the job done.

And then other clients would say, if I don't back it up, I don't

wanna write it in the first place.

Right.

Um, I, I, obviously, I tend to be more towards the latter than the former.

Yeah.

Uh, from, uh, most of the companies I've seen are vendors.

Yeah.

They're not in line with you.

Meaning Meaning that they would just go ahead and do it anyway?

Yeah, because most customers Right.

Unless you

have very, very strict regulations.

Right.

They're like, best effort

Yeah.

I, I think it's

because they will

that I care about data protection more than the average

person,

Because the thing is, at some point it will catch back up.

Hopefully,

Hopefully yes.

Depending on what the problem was.

right.

Um, and, and again, as long as we're okay with the potential,

right, um, uh, I, I would think that there should still be some number.

number might be measured in hours if we're hours behind updating

our other copy, something.

Might be drastically wrong that we need to look at.

Yep.

And the other thing to also mention is with a lot of this.

High end.

I normally refer to it as tier one storage systems

Yeah,

because these are tier one applications with very strict requirements.

Um, usually also they provide the ability to do automatic failover.

So it's kind of, think of it like high availability plus clustering.

right.

So if you take a look at a lot of the tier one storage, right, they might have two

storage systems in both locations with the drives that are all interconnected.

So in case one unit fails, the other unit can take over the diss of the other side.

Um, the clients are also connected to both sides, so they don't

have to worry about failing over.

Um, I can't remember what it was called.

It's like the optimized and non-optimized connectivity for fiber channel,

which allows it to have a preferred path and a non-preferred path.

So you still

have connectivity, so your clients will automatically fail over, so you

don't have to do anything, and so your writes can still continue to happen.

Yeah.

So that's a big thing, is like, you know, we, we, we learned that

we should have it farther away.

We learned that maybe we shouldn't have it too far away, but, but,

but now with the, with the.

With the cloud, we can potentially have it pretty much anywhere, but we

definitely have to rely on some sort of asynchronous, uh, uh, communication.

When, I think about our, our friends that came on the show.

Talk about their experiences with disaster.

I think there's some, that was a major, that was a hurricane

that took out an island

and they had multiple data centers on that island.

One was more in the high ground than the other.

So one was flooded, the other was not.

And they were gonna recover from one data center to the other.

And that's, and they had everything they needed there.

They did.

They needed personnel.

They had to fly people to the island.

But other things happened that we can also learn from.

Do you, what do you remember from those?

So from, so there were two things I remember.

One was sort of.

The people process stuff, which I think we rarely think about.

And then the other was the technology piece.

So from a people process perspective, it was, do you have the right people in

country who have the expertise and the

skills to

recover?

Where are they going to sleep?

Where are they going to get food?

Right?

All the things that you kind of take for granted, right?

They had none of that.

Like how do they communicate

Yeah.

He, as I recall, he, he turned a, a, um, a conference room into a

hotel room

Right.

And slept on a cot and ate rice and beans for two

weeks.

And he was lucky to

get rice and beans right?

Because there were a lot of people who didn't even get that.

Yeah.

Um.

Yeah, that's, that is definitely a, a lesson learned is that, you

know, make sure to take the human element into your DR design.

Right.

Um, and the one that, the one that stands out to me was

the reliance on the mainland.

Yeah.

Right.

That when you have a true disaster, whether it's on an island or just.

You know, wherever you might not be able to get connectivity to the rest

of your computing infrastructure.

And in this case, their authentication and authorization, their IAM system

relied on active directory, which

all of which was in, was in the mainland.

And um, so this is just, you know, the lesson there is to just make sure that.

To just take that into consideration, right.

Just the, the realize that in a real disaster, you may be very

isolated from the rest, rest of the world, and you need to take

that into consideration in your DR.

Design.

Yeah.

And, but I think this becomes so difficult, right, Curtis, because

like how many scenarios are you gonna play in your head and how much time

are you gonna focus on some of these?

Now granted, a hurricane on a tropical island is probably

a high likelihood event,

Yes.

Right.

Well, I,

of that, but.

well, I, I, I think it's, I think it's a totally, I think that there's two

things that you cannot count on, right.

Um, I. and and I, I would just say one thing that covers the number of

things, and that is utilities, right?

You cannot count on the internet.

You cannot count on power.

You cannot count on, um, you know, uh, water,

right?

You can't count on those three things.

And so I, I'm just saying, I don't, I don't think you, you don't need

to think of all of the reasons that one or more of those might not.

Yeah.

available.

You just need to plan for them not being available.

If you don't have internet, if you don't have whatever it is you,

however you communicate between your sites, you are going to be isolated.

If you don't have power, you're gonna need to supply your own power,

right?

Um, these are just things that you can think, you can

think through these things, and these are all things that you can say either.

I'm just saying, have that discussion.

And say, you know what, if we don't have power, we're not gonna do anything.

We're not

gonna spend $15 billion on power generators

Yeah,

in case we, you know, we might, or, or you might be subject

to regulations, or you might

have, uh, financial reasons why downtime is enough.

That, or the cost of downtime is enough that you're gonna pay for generators.

Um, do you remember how they got internet over there?

It was satellite.

Yeah.

satellite.

I like this idea of making sure that you think about what would

you do if the utilities that you're normally counting on are not

available?

If you get completely isolated, what would you do?

There are a number of reasons why that might end up being the case.

or the, or the people that you normally depend on are not available.

Yes.

This is why we had this little thing called documentation.

Yeah.

Um,

And,

and

go ahead.

and hopefully I know when we've had Mike, Dr. Mike on the podcast, right?

Um, he's talked about sort of doing tabletop exercises, right?

So have you done a tabletop exercise, which is like, Hey, what happens if the

main IT person, Susie, is unavailable?

Right, right.

And, and that's what you do.

You work through those various scenarios.

You, you hire somebody to come in as an outsider is the best way to do that.

When we start talking about ransomware, um, Dr. Mike Sailor's company would,

would, would be a great resource for that.

It's good to use a very negative person.

Think the most negative person in your environment, the most

pess, pessimistic person.

And, uh, no, no.

Pessimist thinks they're negative.

They, they think they're, they're realists,

will realize

Um, yeah.

Sort of the final thought that I'm thinking is, and, and I, I remember.

Doing a disaster recovery of my own, and that is, I did it where.

I had not yet tested the throughput of the

backup system that we were doing and the throughput of the backup system we

were doing, it turned out to be crap.

Right?

And as a result, uh, that was a really hard day.

And in Curtis land, right?

This is

very early in my career, I learned it.

What's that?

Yes, it was the compression thing.

Yeah.

So make sure that when, when you make configuration changes to your

backup system or your DR system, make sure that you test that.

Just realize that in general, restore speed is slower than backup speed.

It just is.

It's the way it, you know it, you know, back in the day with tape, it

was because we were doing multiplexing.

Nowadays it would would with dedupe.

It's because of dedupe.

Um, and so just sort of plan for that.

But don't, don't just assume how much slower it is, uh, test it and see how much

slower it is and make sure you figure that into the, to the disaster recovery plan.

But, um, so, um, with that, I think that's enough for now in terms of lessons learned

from just, you know, the pains of others.

Um.

Just think of the scenarios that might be, you know, that you might be subject to.

Um, make sure you've got at least one copy of your data

that's far away from everything.

Um, and the way to probably do that today is cloud taped can still play a role.

In fact, our previous episode we talk about why tape is

still not dead in backup.

Uh, it certainly is on life support, but there

is, you know.

There is, there is a use for tape and backup and it's disaster recovery.

So, um, um, especially when we start talking about disaster

recovery from a ransomware attack.

So, well, thanks for chatting again, my friend.

No, it was good.

Uh, one thing I'm surprised you, I thought you were gonna mention, but you did not,

The 3, 2, 1 rule.

1 rule.

You were just

You know what's funny is we were so close.

We were so close.

Yeah.

Yeah.

The whole 3, 2, 1 rule with, you know, three copies of your data on two

different media, one of which is offsite.

That, that, that's a core design concept for anything.

Uh, backup in dr. And, and that's, yeah, you're right.

We never, we, we didn't mention, but now we have, so you

have corrected our oversight.

Thank you very much.

And thank you to our listeners.

We'd be nothing without you.

That is a wrap.

The backup wrap up is written, recorded, and produced by me w Curtis Preston.

If you need backup or Dr. Consulting content generation or expert witness

work, check out backup central.com.

You can also find links from my O'Reilly Books on the same website.

Remember, this is an independent podcast and any opinions that

you hear are those of the speaker and not necessarily an employer.

Thanks for listening.