No one cares about your backup window!

You found the backup wrapup, your go-to podcast for all things

backup recovery and cyber recovery.

You know what, no one cares about your backup speed.

That's right.

I said it.

Sure.

When designing your backup system, backup speed does matter, but

you know what really counts?

Recovery speed.

That's what keeps you employed.

I've got some wild stories about how I learned this particular lesson

the hard way, including that time at Motorola when I discovered that my tape

drives could write but couldn't read.

That was fun.

Uh, joined persona and I as we dig into why recovery testing matters

more than bragging about your backup speeds and why you should

probably start doing some of it.

Before you actually need to trust me on this one.

You don't wanna learn this lesson the way I did.

By the way, if you don't know who I am, I'm w Curtis Preston, AKA, Mr. Backup,

and I've been passionate about this topic for over 30 years, ever since.

I had to tell my boss that we had no backups of the production

database that we had just lost.

I don't want that to happen to you, and that's why I do this.

On this podcast, we turn unappreciated backup admins into Cyber Recovery Heroes.

This is the backup wrap up.

Welcome to the show.

Hi, I am w Curtis Preston, AKA, Mr. Backup, and I have with me

a guy who wishes he had my new coffee machine Prasann Malaiyandi.

How's it going?

Prasanna?

I am GI Act.

I'm good Curtis.

So I actually don't wish for your coffee machine

Mm-hmm.

reasons.

Okay.

One, I don't have space to put a coffee maker.

Uh,

Without tossing something off the kitchen counter.

mm-hmm.

I really need to cut back on the caffeine.

Like at work we get coffee

Yeah.

I try not to have more than one a day.

And on Wednesdays we have someone come who makes coffee and uh, I get one

single shot latte to start with, and before they're done for the day, I get.

An iced oat milk with like a half a shot and they laugh at me.

They're like, how much is a half a shot this

What is a, what a, that is so funny.

Because I

Yeah.

so

No.

I st I still think you wish you could have it.

You just, you, you, um, you like the taste, but you don't Yeah.

You know, it'd be bad for you.

Yeah.

are you enjoying your new coffee?

Fancy coffee machine?

So I've, I've made like three espressos with it so far.

Uh, I have a ne espresso, by the way.

I've, I've kind of always thought about getting one and I, you

know, and I finally bought it.

Um, but with, you know how, like, you know how like your.

Your feed is tailored to stuff that you do.

Yeah.

This morning the first thing that greeted me when I woke up was is Nespresso

is changing their prices, you know, increasing the cause of their pods.

Um, and also,

in it.

yeah, and also there was a thing on Reddit about, um.

It was a picture of what A-A-U-P-S place looks like from the

Nespresso pod recycling program.

Let's just say it's a hot mess.

yeah,

Um, they are doing it, it's just that it, it's logistically a, a challenge.

Um,

and it's also like, I think.

Sometimes they're like, oh, separate out your bottles and

plastic, or your plastics and trash.

And it's

yeah.

to actually recycle properly is so high.

Yeah.

if they're just like, yep, we'll take it and we're just gonna burn it all.

I, I think they're doing it.

But, uh, there is this thing about like, um, that there is the belief that

they, they give you these bags, they give you a free bag that if you just

put your stuff in there, you can drop it off at UPS and they'll just take it.

Right?

And they're saying you're supposed to fill it up to this line.

Um, but not beyond the line.

And the belief is that if you don't fill it up to that line, the cost to ship

that bag is less than the, than what they're able to recycle out of the bag.

Um, but anyway, but

up to at least the line.

yeah.

Well, but if you go too far over the line, like it, there's issues, but,

um, but I, there was an incident.

I, I, I want to hear about, let me guess.

let me guess what the incident is.

Yeah,

a pod in when a pod was already in there and everything exploded.

no, that it, it takes care of all that, you know, all you have to do is open it

and it automatically removes the old pod.

Um, but the, the way that, the way the one that I bought, it comes with a tower.

Tower O water, right?

So, and it's detachable.

So like, by the way, the, the, the kitchen counter problem, um, I solved

that by having this in my office.

So this is the tower of water for those of you watching it on YouTube.

And it literally just, you know, I'm holding it in my hand and then

I can put it over here and, you know, there's this little slot,

Yeah.

you see that?

And so that, that tells where to go, but literally you just pop it on there.

Um, so I might.

Have just knocked it right over like,

And

just after.

like spilled everywhere.

Yeah.

And that's a good what, 32

Yeah.

ounces at least, maybe even more.

And I just dumped the entire thing on the floor.

Um, literally.

Five minutes after putting it in, because I have it, I, I, I have a

better place for it long term, but where it's at right now is a bit precarious.

And, um, I literally, five minutes after having it here, I, I dumped

that entire thing on the floor and had to mop up like crazy.

And, I mean, there's power cords down

I, I was actually worried about your power strips and power cores and

Yeah.

Yeah.

Yeah, there, there's some bricks down there, some, you know, some

power bricks and things like that.

Those are

Yeah, so it wasn't, it was, it was just a big water watery mess.

But anyway.

So, um, let's chat a bit.

I, I, I have, do you think, do you think the title is clickbait?

That's the first thing I told you before we started this recording.

I was like, ah, it's a little bit click bait, isn't it?

Yeah.

No one cares about your backup Speed.

Right.

Um, and, and let's just talk about.

similar to what we've talked about before, which is

Yeah.

your backups.

I.

Yeah.

No one cares about your backup.

You know, you, you, one of the things that you've heard me say on a, on

a repeated basis is that no one wants to be the backup admin, right?

No one grows up to be the backup admin, um, and no one is in it going.

I, you know, I hate my job as a network admin.

I'd really much rather be the backup person, right?

No one, no one has ever said that in the history of time.

Um, many people have said, God, I hate my job in backup and I can't

wait to become a real cis admin.

That's, and I mean, I mean, I mean that I love all of you, my backup people.

You are me.

I am you.

Uh, I'm just saying that it is a tough, tough job and one of the reasons.

Is it No one cares about your backup.

Yeah.

They only care about the restore, right?

No one cares about the millions of backups that you got, right?

They only care about the one restore that you got wrong, and, uh, they, yeah.

What's that?

so going along what you're saying though, think people don't grow up

to be backup admins, but I think there's a way to make backup sexy.

Like I'm just imagining in my head

Mm-hmm.

like it's Superman, right?

It's Clark Kent is backup.

Uh huh

And Superman is restore saving the day,

Here I come to save the day.

right?

If you think about it, you are the hero.

You are the one who saves the company and brings things back up.

So, so for the YouTube audience, um, I am backup guy, right?

Yes, Mr. Backup.

Restore man,

restore man,

And,

guy.

and for our listeners who are listening to this via podcast,

Curtis, between those two sentences, just took off his glasses then put

And I'm like a totally different guy.

I mean, it worked for Superman, right?

That's that.

That is just the funniest I've, I've always, I've always

been am abused by that, but

but here's what I'm thinking.

Sorry.

I know this is

yeah.

a bit, but I think we backup people need to start thinking differently, right?

I think if you think I'm a backup person,

Mm-hmm.

right?

I don't think you're giving yourself enough credit for how important

you are in the organization.

Backup

Absolutely.

an end.

You are really the recovery person.

Yeah.

And that's why, you know, the tagline of the show.

We say that we turn, uh, we turn unappreciated backup admins into

cyber recovery heroes, right?

Absolutely.

I am on board for you or I am on board with you.

Having said that, let's go back to the title of the episode, which is

no one cares about your Backup Speed.

Okay?

And, and, and, and I say that be.

backup speed all the

Yeah, when do we talk about it?

We talk about it when we're designing the backup system, when

we're working with the vendor

Speeds and

design the backup system and yeah, speeds of feeds.

And I'd say this is it, it, we don't focus on it as much today

as we did back in the day, because back in the day you had a number.

You're like, I have, I have 18 terabytes and I gotta get 18 terabytes

from here to there full once.

A week.

Um, and, um, and, and therefore there's some math that has to happen.

I gotta transfer.

I'm gonna do, I'm gonna, I'm gonna spread my, I got 18 terabytes.

I'm gonna pick a better number.

21 terabytes.

So I'm gonna do three Tera.

Tag I, yeah.

Right.

Okay.

So I'm gonna do three terabytes a night, and that means I need a network, I

need, I need server, and I need, uh, you know, tape or D drives that are capable

of receiving three terabytes a night, plus, you know, let's say 10% of the

remaining terabytes, which would be, um.

Another roughly two terabytes.

So I need, I need a system that's capable of receiving five terabytes a night.

Backup speed mattered so very much when that was the way you did backup.

Um, now I think mainly the concern is that obviously we need enough time

in order to get the backup done, but, um, and I will say this, so I'll,

I'll, I'll, I'll disagree with the.

With the title first before we get into the, the reason why I actually strongly

believe in this statement, um, if you, if your backup is not finishing within

a reasonable amount of time within what we refer to as the backup window, what,

what, what do we call the backup window?

Your backup window is the time in which you have to finish a backup.

Typically,

Right.

resources are idle, or as an example, you might say, I have eight hours

during normal business days, Monday through Friday to get a backup done.

Maybe I have all weekend because people aren't actively using the system.

It's

Yep.

Yeah, exactly.

And historically, the reason why the backup window, and this is why, another

reason why we focused so much on backup speed back in the day, which was I.

Backup was really hard on the servers.

It was a, there was a significant performance impact on the servers

when you were running a backup, and so therefore, really the only time we

you could do it was when no one else was using the servers, and so you

were given a window back in the day.

I remember that my window started at like 10 o'clock at night.

And I needed to be done by six in the morning, which meant I had eight hours.

That was my backup window, and that meant that I had to do those.

Um, we came up with five terabytes.

I had to do those five terabytes within eight hours.

Which, what?

Well, three plus the two.

Remember the three, the three terabyte of full plus two terabytes from the 10%

of the, don't you remember that part?

Yeah.

Yeah.

So I need to do those five terabytes within eight hours, which need,

which means I need to do almost a terabyte an hour, which back in the

day was a real challenge, right?

I remember the first time I saw a, a headline that, that, that

somebody had, was able to achieve a terabyte, an hour of backup back then.

Like, if you couldn't do that, you, you, you, well, you just couldn't

get your backup done within the window and therefore, um, you

weren't gonna get your backup done.

but, but I think.

Mm-hmm.

arguing or against the title

Uhhuh.

is I think because backup speeds have improved technology has made

it more efficient, which I know we haven't talked about yet, but has also

allowed us to reduce the backup window from say, once a day, eight hours.

Yep.

Or your backup frequency, I should say,

Yep,

once a day down to say, every four hours with less impact on

yep.

and everything else, and

I, I would.

to take it more frequently and give yourself better protection.

I would actually say that the first thing that it did was it actually blew out the

backup window because, because we can do backups and, and they don't significantly

impact the performance of the server.

Hopefully, depending on your backup, you may actually, depending on the server

and depending on the backup, you may still, if you're still, for example,

if you're doing database backups.

And you're still required to do full and incremental backups.

Those could still have a significant performance impact on the server.

But everything else where we're doing.

Either some type of block, a level incremental or some type of source site

deduplication where you're the, the performance impact of an individual backup

on a server is relatively nil, which means that you could do it pretty much any time

of day, and that means that as long as you do it within 24 hours, then that's fine.

And then I would.

I completely agree with you that in some cases we've then said, Hey,

because the backup is so quick, we can actually do it more often

than the quintessential once a day

Yep.

we could do it.

Um, you know, every four hours, every hour, every five minutes, there are

backups at some, you know, different ones.

Take every five minutes, um, and we can therefore have a much tighter RPO.

What is that?

Recovery point objective.

Right, exactly.

So if back in the day your backup frequency was once a day, then your RPA,

your recovery point, actual, uh, you know, a lot of people use the term RPO to mean.

R-P-A-R-P-O is the objective.

It's the, it's the, uh, SLA.

Yeah.

Yeah.

The RPA is the actual capabilities of your system.

So the, your, if your backup is once every 24 hours, then the best RPA

you're going to get is likely 24 hours.

You could actually get 48 hours if a backup fails, for example.

Right?

Um, but if we could do a backup in 30 seconds.

With, with minimal impact to a server, you could do it four times

a day, like you said earlier, and we could get a much tighter RPA

I think the one other thing to mention is, and you brought it up

too, is with databases you still have to do fulls and incrementals,

Most databases.

Yeah.

And I think that's where backup speed still plays a part,

Yep.

as databases are growing in size, you

Mm-hmm.

be able to back that up within the window.

So.

Having backups that can go really fast is still a good thing.

Yeah.

a new application comes online.

Yeah.

But, but even then, like the DBA, he just, he or she just assumes that.

You are, you are going to get the backup done.

No one's gonna be impressed.

Hey, I figured out how to back up your, your data.

You know my backup's really fast and now it runs in only four hours.

No one cares,

Yeah.

right?

Going back to the title, no one cares.

You.

Oh, you got the backup done really fast.

No one cares, right?

Yep.

Um, now I will say this, the.

no one cares as long as they're able to meet those windows.

But yeah.

No one cares that you can do it.

They will care very much if you cannot do it right.

Um, and and that's what where I was about to, to head.

And that is the one concern because as you've heard me say

before, I've spent my entire career fighting the laws of physics.

Right?

And so, um, the, even though we have.

Reduced down to the smallest possible increment.

I think in terms of the size of each individual, incremental backup.

When you, when you do block level incremental, or you do source side

deduplication you, you're, you're basically transferring the smallest amount

of data from the server to the whatever it is that you're backing up, that that

could possibly be transferred even.

The, even though we have done that, you add up all the servers, that's a

certain number of gigabytes or terabytes or petabytes that must be transferred

within a particular period of time.

And so still the what matters very much is bandwidth.

Yes.

Um, and you, and unlike, again, back in the day, if we ran the numbers and

the problem was the network, we could just build another network, right?

And, and, and that's what I used to do when I, I, I remember the first

time I did this and these were, do you remember Sun E four fifties?

Do you remember those servers?

Were those A?

Those were the, weren't the sun rays, were they?

No, they, they were like the size and shape of a, of like

a, like a dorm refrigerator.

Right.

And the thing about the E four fifties is, that was amazing.

Was that the, it was, they had the, the great PCI slots and, and they had

multiple, like full speed PCI slots.

And so what you could do is you could put multiple.

Uh, you know, ethernet cards in there and, um, and each of

them could run at full speed.

And back then the, the dream was to get a gigabit card in there, right?

Yep.

And so it was super cheap and easy to build a, a super fast

separate network that that.

Wouldn't be impact.

You know, the backups wouldn't be impacted by production loads and production

loads wouldn't be impacted by backups.

The difference now is in many cases, many of our backups are no

longer going over the corporate LAN.

They're going over the wan, they're going over the internet,

Yep.

hopefully, and I really mean this.

Hopefully you're encrypting that data

Yep.

you're sending it.

Right?

But since you're sending out, out over public networks, but.

But still backup speed does matter in that you've got to make sure

that you have enough bandwidth.

You've got to make sure that you've, you've done these calculations and

you've figured out the, the databases that we talked about that are gonna

need full and incremental backup, those speeds will very much matter.

Um, this is more true with sort of traditional databases, right?

Rdbm S'S relational database management systems, um, like

Oracle Cybase, SQL Server.

Yeah,

Uh, Informix.

I haven't heard Informix in in a minute, but

when, when you're talking about bandwidth, right?

The

Uhhuh.

that came to mind is, and your son, E four 50 example, the other thing that

came to mind is there's also a big push these days to make servers dense.

Mm-hmm.

Right, virtualization, right?

You want to take

Yep.

rather than leave them sitting.

And so you may end up with bottlenecks on those servers that prevent you

from actually sending data out.

So instead of having 10 servers with 10 network interfaces, now

you maybe only have two, right?

And they can only support a certain amount of bandwidth leaving the system

Yep.

10 independent systems before.

Yeah, agreed.

Yeah.

So that, that my bandwidth comment starts at that server and continues

on to your LAN continues on to the WAN

Yep.

and continues on to the vendor.

Right?

It basically, there could be a bottleneck anywhere along that

and, and a, and a bottleneck there.

Um.

Would mean that your backups would not get done within 24 hours,

Yep.

which would then impact your, your r you know, your ability to meet your RPO.

Yeah.

Um, so if they don't care about your backup speed Prasanna,

what do they care about?

How quickly does it take to restore your data?

Exactly.

How quickly can I get my data back?

Right.

Um, and I. I, you know, early on in my career, I was as guilty of this as anyone.

And that is, it's so easy to basically spend all your time designing the backup

system, testing the backup system.

Does it have the capability to do what I needed to do and not

really do any restore testing?

Because the, so first off, let me put out a, what do you call it?

A um.

An axiom would, would that be the right word?

Um, just because your system can back up at a terabyte an hour does not mean

it can restore at a terabyte an hour.

Why might that be?

Oh, a thousand percent.

Uh.

A lot of it, it, having worked at storage vendors in the past that built

de-duplicated appliances, a lot of it is.

The way it writes,

Mm-hmm.

streams, it's very unique, and so you're able to optimize.

But when you're reading data out, you have to read every single piece of data

in order to restore that application.

It's not like, oh, 99% is duplicates.

I can just drop it and just send the

Yep.

I actually have to read all the data out and how it's laid out on disk

may not be optimized, especially with deduplication and other technologies.

And we, I, I have in the past, referred to this as the dedupe tax.

Yep.

difference between if we, if we lay the data down on disc, like without

dedupe, and then we lay it down on disc with dedupe, the difference in

restore speed between the two is often referred to as, you know, the dedupe tax.

And I've seen it be really, really, really bad.

me too.

Yeah,

Yep.

I, I can think of one vendor and I'll let them go nameless.

'cause they're still around as a vendor.

They still sell, uh, dedupe appliances.

I'm pretty sure they've addressed this issue, but when

it first came out, it had a 90%.

dedupe tax in that, I remember that the, the, the sy the, you know, version one

of the system had the ability to, to, uh, write data at 400 megabytes per second.

Its ability to read data was 40 megabytes per second.

The aggregate throughput of the entire appliance was

only 40 megabytes per second.

And, um, it.

Was not good.

And, and if you didn't test that, then you know, you, you know,

Do you

you'd find it out at the wrong time.

you wanna talk about your example?

Which one?

The one that you had the test with, the compression it had

Oh

it and you know, you know that one.

yeah.

Yeah.

Um.

So, you know, back in the day, um, back before, like when I used my

first commercial backup product, they had a software compression feature.

Um, that the idea was that it was, this was before I was using tape drive

compression and they had this ability and, um, it was, it was allowing me to

do things that I wouldn't be able to do.

And so I chose this as a, as a setup option during, during my initial setup.

And I didn't do restore, I didn't do restore testing.

And then, um, there was a moment when, uh, very bad things happened

and I needed to pull my tapes out.

I will just say, uh, so I remember running over there and being super

excited that I was gonna, now, you know, I was essentially going into battle

with an untested weapon, but whatever.

I was super excited and, um, I remember.

Standing there and, and firing off the restore.

And I remember watching the tape drive going like blink, blink, and then

really long period of time then Blink.

Blink.

And what I found out was that, um, when they do compression,

the way they do it is literally.

For the Unix people in the crowd, they were doing a compressed minus C to, they

would com, they would write it to temp and then they would write that compressed

tape to that compressed file to tape.

And then when they were restoring it, they would restore that compressed file

to temp. Then they would run uncompress in place, not uncompressed minus C, like

they weren't doing it to standard out.

They would uncompress it.

And then they would move that file to, uh, where it needed to go.

Um, and so that's when I learned, uh, from the vendor, like, yeah,

we don't recommend this feature if speed is is important to you.

You're like,

okay.

anywhere

we, we,

Yeah.

where was that?

In the manual?

Right.

Um, and.

So there are things that you're, you're, you're not going to know

until you do, uh, restore testing.

So you, you have to do restore testing, right?

I mean, for one thing, it, uh, it, how are you gonna know?

The restore even works, even works at all.

Um, I'll, I'll, I'll tell, I'll tell you another story.

Curtis is

was, yeah, I was, what's that?

Are you sure it's not the espresso?

This is an espresso talking.

I was working, I was actually, um, at the headquarters of Motorola

Hmm.

and um, working with the computer simulation modeling and research division,

which was a cutting edge division at the time that was using computers to simulate.

Damage to phones and to improve the design as a result.

And I remember that this was the group that basically came

up with the Star Tech, right?

Which was the, the phone that put Motorola on the map.

Right.

And um.

I was using their tape drives and I, I had their backup system and I had it going

and everything was, you know, was going and I was, I was a full on admin, but, but

backups was part of my responsibilities.

And, um, I had been, I'd been there for a few months and I'd been doing backups

and I just sort of assumed that the backup system that I inherited worked.

Yeah.

And I, um, I just remember the first time I went to go do a restore.

And what I found is, uh, so these were IBM, like 34 80 something like

that, old school IBM cartridges.

They were really good at writing.

They were completely incapable of reading.

I, I really don't remember what the solution to this was,

you

but I remember that I was like, I, I would kick off the restore and, and

the restore would always fail because.

Um, the, the tape drives couldn't read.

And I'm like, you know, like it's, it's like you have two jobs.

It's not just one job.

You have two jobs.

You have write and you have read.

You have to do one to do the other, but the other is really, really important.

You can't do just the one and not do the other.

Um, it reminds me of, uh, the Seinfeld episode with the, the car rental

where he is like, where they, where he's like, they're like, we don't

have, we don't have a car for you.

And he is like, but I had a reservation.

He goes, yes, we understand you have a reservation, but you don't have a car.

And he's like, anybody can, anybody can take the reservations to, you

know, take the reservations, you know?

But the, the important part is the holding of the reservation.

And that's, that's the way it is to back up.

You know, it, it's important to back up.

It's really important to restore.

Um, I, I literally have, I have no memory of what the solution to that

was, but I definitely remember the day that I went to go do a restore

and absolutely nothing happened.

at three in the morning with the popped in your head.

I, you know, it was probably a really traumatic event and I, you know,

I, I, I wiped it outta my memory.

Um.

And, and so go ahead.

But I guess one question is, so I know we talked about restore

speeds, newer technologies, newer storage vendors, right?

They are addressing this,

Mm-hmm.

The speeds have improved for restorers because new use cases in

addition to just restoring data such as instant access and other things

where performance does matter.

And if you look at some vendors like Vast, right?

And other things they are leveraging, um.

flash more heavily in

Right.

to help improve performance.

And so I think some parts of that are being addressed, but you still

also have like the reading the data out and everything else, and

just like to get a full copy of the data out is just very expensive.

Yeah, I, I would say when we talk about dedupe, and I'm a fan of dedupe.

It's really important how you write the data and it's really important what it's

like to get that data back en mass, right?

It's really easy to just write the bits that you need for a backup.

How you write it very much determines how you're going to be

able to read it and um, and so.

As a result, um, again, the only way you're going to know that

is to is to do restore testing.

Yep.

The other thing is that when you do restore testing, you have to

look at all of the different.

Ways in ways in which, and things that you need to restore, right?

You need to look at all of the different applications.

You need to look at all of the different types of servers and

VMs and, and serverless, um, applications that you might have and,

dependencies.

and dependencies, right?

Well, then we start talking about disaster recovery, right?

When we start talking about disaster recovery, we introduce

the concept of a recovery group.

What is that?

It's everything that is needed in order to bring up your environment or your

application or whatever that object is

you are looking to recover.

right.

So let's say you want to restore, um, I, you know, even though no

one has exchange anymore, you wanna restore your exchange server, right?

Everybody, everybody's moved to 365.

Someone's

gonna get, I'm gonna get like a bunch of emails from people say,

Hey man, I still use exchange.

Listen.

It's time to, it's time to move on people anyway, so you wanna restore

your exchange server, um, you know, you have to restore active directory,

for example, before you do that.

And I. Um, I did, does exchange use, it doesn't use SQL underneath, right?

No,

no.

but there may, but, but there may be an application where you need an

application on top of a database.

Maybe there's multiple databases, maybe there's, uh, authentication,

authentication and authorization, and I think is probably the

biggest one that that is a dependency for most environments.

Can you think of anything else like that?

sometimes even, well, depending on if anyone's still doing

bare metal recoveries.

Mm-hmm.

Right.

Bringing up your host first and your server before you can actually start

doing your application, which might be a completely separate recovery process.

Yeah.

If, if you're doing bare metal recovery, uh, which is a term that we used, I

mean, literally it means you have a piece of, you have a piece of metal,

you have a box, and it's got nothing on it, and you're gonna recover from there.

Most people, again, no one does this anymore.

And the reason why they don't do it is because they're doing VMware

Yep.

um, you know, other, or Hyper V or, you know, uh, any of

the other different methods.

I don't think anybody's gonna be doing VMware anymore.

Um, everybody, everybody I talk to that has anything to do with VMware, it's like,

yeah, we're trying to figure out how to move off of VMware thanks to Broadcom.

And, uh, and Broadcom will probably still make money.

'cause basically they're like, we're gonna, we're gonna.

We're gonna do a 10 x, you know, um, we're only gonna have five customers less,

and we're gonna, and we're gonna charge them $10 million each and we're fine.

Um, the, um, totally lost my train of thought.

Oh, so, so no one does bare metal recovery because it's so much easier

to do a bare metal recovery, or the equivalent of a bare metal recovery.

When you have a vm, you can back up the VM whole.

You just restore that as a file and boom, you're off, you're off to the races.

but remember, even there, you might need to bring back your vCenter instance,

Yes.

And everything

That's a perfect ex example of a, of a, um, I. Of a dependency, right?

Uh, if you have, if, if you, you're going to have to install vCenter,

which means hopefully have a backup of the vCenter environment, right?

Um, and there's probably an authentication authorization system to do that.

It, you know, it, it, it, you, you have to do all of these dependencies.

This is why you do recovery testing and you.

You just create these different scenarios, and I will say that they are going to

very much care about your recovery speeds.

Yep.

They don't care about your backup speed, but they care very much

about your recovery speeds.

And they, they, they really don't care about these dependencies.

They're like, look, all I care about is this application.

Why is it taking me so long?

Well, because I have to store 37 other servers before I get, before

I get to restore your application.

Um, the.

Did you have a, were you about to say something?

The one other thing to think about from uh, restore speed perspective

Mm-hmm.

make sure it's realistic what you're looking to test.

Uhhuh,

an example, you might just test, say gigabyte of files

right?

gigabytes of files or one application, right?

And that may not actually reflect the system actually behaves when you're

trying to do these massive restores.

Yep.

Yeah, I, I completely agree and disagree at the same time.

You gotta walk before you can run.

If you've done no recovery testing, I would highly recommend

restoring 10 gigabytes of files.

You know, you know who would agree with me right now?

Jeff.

No.

Stewart.

Oh, sorry, Stuart.

Yes.

Yeah,

Yep,

so, so Stewart very much would've liked to have done a restore of, of,

you know, 10 or 20 gigabytes of his data back before he needed to do a

restore of, of 800 gigabytes of data.

go back and listen to that episode.

I think we'll put

Yeah.

show notes.

Yeah.

It's a great episode about the a, a true, a true life restore of a, of a

basically Prasannal data, but it's, it was 800 gigabytes of Prasannal data,

so it was a significant, you know.

and this is by an experienced backup admin,

Yeah, right.

A person who'd spent an entire career being a backup admin, but like a

lot of people, like he's using cloud backup for his Prasannal data and

you know, he's, he, he has zero control over how this thing works.

He just pushes the button and magic happens.

Right?

And let's just say it wasn't so magic.

So, yeah, so you've got to do your restore speed testing.

They very much, and, and I would say start small, work your way up.

But I completely agree with you that you want to test, um, you know as much

as you can, as real as, and that's the beauty of the cloud, is that you can

basically push a button, recreate your environment, test the crap out of it,

uh, and then delete it, and you just pay for it, uh, during the duration.

What I don't think you should do.

Is what our friend in Alaska did, and that is delete your entire data center

and then test your, uh, recovery speed.

We, we do.

We love you, Paul, but, uh, you know, you, you, you know, you know you did wrong.

You, you made it out.

You made it out to the other side.

But, uh, you know, um, and we're gonna have you on again soon, um, uh, to tell

us, you know, what else you've been up to.

'cause, uh, that, that sounds interesting.

So no one cares about your backup speed Prasanna.

Yeah.

They only care about your restores and how

Exactly.

for them to get their data back.

I will also say that investigate the idea of restoring in advance backup products.

Support the idea, you know, you, you mentioned one, this idea of instant

recovery where you can just literally basically push a button and, and

there's a copy of your data ready to roll that you can just mount that.

That's a great idea.

Um, there are also companies that essentially you create a recovery

group and they essentially pretor it, um, you know, on an ongoing basis

so that when you go to do an actual recovery, you can just literally

push a button and go, um, networks.

Um, that's, that's the, that's the, the only way to have a really fast

restore is to have already restored it before you actually need it.

Yep.

And then the other way is snapshots.

Yep.

Right.

If, if you're able to just mount a snapshot and roll, uh, that's awesome.

Mm-hmm.

And that's why so many people are fans of doing backups and recoveries that way.

Yep.

All right.

Well thanks for another great chat Prasanna.

Anytime Curtis and remember, only three espressos a day or Americanos?

Only three.

I've, I've

Only

had.

I've had two so far, so I mean, I could, I can go get one.

You you're

You know, you know what, you know what's funny is like when you buy

a thing, they send you, like this, this, um, the Yeah, they send you.

So this is what comes with the Nespresso.

These are all the pods and um,

that's

this pod right here.

By the way,

Yeah.

that, when you buy a sleeve, it doesn't come like that.

Yeah.

Um, so this pod right here, you know what this one is?

What?

This is a, what's the point?

Pod.

You know what it is?

It's decaf

Oh yeah.

like No, no.

Thank you.

Um, tio decaf.

Yeah.

No, thank you.

I'm good.

I'm like, what is the point of that?

Anyway, well, thanks for listening folks.

Uh, you're why we do this and, um, good luck out there.

And that is a wrap.

The backup wrap up is written, recorded, and produced by me w Curtis Preston.

If you need backup or Dr. Consulting content generation or expert witness

work, check out backup central.com.

You can also find links from my O'Reilly Books on the same website.

Remember, this is an independent podcast and any opinions that

you hear are those of the speaker and not necessarily an employer.

Thanks for listening.