Check out our companion blog!
Nov. 4, 2024

RTO vs RPO: The Foundations of Backup Design

RTO vs RPO: The Foundations of Backup Design

In this essential episode of The Backup Wrap-up, we dive deep into RTO vs RPO – the foundational concepts that drive backup and recovery system design. Curtis and Prasanna break down why these aren't just technical metrics, but crucial business decisions that should come from your stakeholders.

Learn why different applications need different RTOs and RPOs, how these metrics influence your backup frequency and system design, and why getting them wrong can cost your company millions. We'll show you how to have productive conversations with stakeholders about recovery objectives, and why the common answer of "zero downtime" isn't always the right one. Whether you're new to backup or a seasoned pro, this episode will reshape how you think about recovery objectives.

Transcript
Speaker:

You found the backup wrap up your go-to podcast for all things



Speaker:

backup recovery and cyber recovery.



Speaker:

In this episode, we tackle something that a lot of backup



Speaker:

folks get wrong, RTO versus RPO.



Speaker:

You might think this is a simple topic, but I guarantee there's



Speaker:

more here than meets the eye.



Speaker:

We're going to break down why these aren't just backup metrics.



Speaker:

They're business decisions that should come from your stakeholders, not from it.



Speaker:

And if you're thinking, Curtis, how can you spend an entire



Speaker:

episode on these two terms?



Speaker:

Well stick around.



Speaker:

We'll show you why these two concepts drive everything from backup frequency



Speaker:

to the overall system design and why getting them wrong can cost your company.



Speaker:

Tons of money.



Speaker:

This is one you won't wanna miss.



Speaker:

By the way, if you don't know who I am, I'm w Curtis Preston, AKA, Mr.



Speaker:

Backup, and I've been passionate about backup and recovery for over 30 years.



Speaker:

Ever since.



Speaker:

I had to tell my boss that we had no backups of the



Speaker:

production database we just lost.



Speaker:

I don't want that to happen to you, and that's why I do this.



Speaker:

On this podcast, we turn unappreciated backup admins into Cyber Recovery Heroes.



Speaker:

This is the backup wrap up.



Speaker:

Welcome to the show.



Speaker:

If I could ask you to take just a quick second to subscribe or



Speaker:

follow wherever you're watching us.



Speaker:

Remember, you can watch us on YouTube if you wanna see



Speaker:

our bright and smiling faces.



Speaker:

Or if you think I have a face built for radio, then uh, you



Speaker:

can use the podcast format.



Speaker:

But either way, if you follow or or subscribe to us and you'll make



Speaker:

sure you get our great content.



Speaker:

Uh, I am w Curtis Preston, also known as Mr.



Speaker:

Backup, and I have with me a guy who, for some reason today is just a little bit too



Speaker:

bossy Prasanna Malaiyandi, how's it going?



Speaker:

Prasanna.



Speaker:

I am good.



Speaker:

I think it's also a voice made for letters.



Speaker:

Boy, I've never heard that one.



Speaker:

I've heard you got a face for radio buddy.



Speaker:

Um, but, uh, yeah, I've never heard voice made for letters.



Speaker:

You know, there, there's a guy that I know that I spent some time with actually



Speaker:

just last week, and he has a voice for.



Speaker:

Like, um, you know,



Speaker:

basically people are always like, you've got an amazing like, radio voice.



Speaker:

And he's actually, um, he's around my age, maybe even a little bit older, and



Speaker:

he's actually trying to become a, um, like someone who, uh, does audio books,



Speaker:

um, uh, amazingly, you know what his name is.



Speaker:

What.



Speaker:

His name is Will Shakespeare.



Speaker:

This is actually his name.



Speaker:

His name is Will Shakespeare.



Speaker:

I introduced him to another friend of mine, uh, Robert Lewis Stevenson.



Speaker:

And, um,



Speaker:

think you're totally making this up.



Speaker:

No, these are two people.



Speaker:

They're the, I, you, I can show 'em to you on Facebook.



Speaker:

I'm friends with Facebook.



Speaker:

One of 'em I know via the IT world.



Speaker:

The other one I know via the, um, election worker world.



Speaker:

So, uh, we're gonna talk about, um, for some people, you know, when



Speaker:

you saw the title, you're like, how can you talk about this topic for,



Speaker:

you know, a, a podcast episode?



Speaker:

And you know what, this may be one of those times where we're gonna have a 10



Speaker:

minute podcast, but I don't think so.



Speaker:

Yeah, I don't think, I don't, I don't think that has ever happened.



Speaker:

I don't think that will, unless like one of us, like falls ill



Speaker:

in the middle of a podcast.



Speaker:

Yeah, yeah.



Speaker:

We do have the ability to, to Jer Actually, I'd say more me, I



Speaker:

have the ability to Jer Java and you have the ability to humor me.



Speaker:

Um,



Speaker:

Yeah,



Speaker:

so we're talking about



Speaker:

They're actually earmuffs like hearing protectors, not headphones.



Speaker:

I'm just really good at reading lips.



Speaker:

I'm gonna say that this topic, there are few foundational topics



Speaker:

that are more important than the topic that we're talking about



Speaker:

Ooh, I, I,



Speaker:

What do you think?



Speaker:

I, once you introduce the topic, I'm gonna ask about the other one



Speaker:

that may or may not tie this one.



Speaker:

Okay.



Speaker:

I mean, there are things that are very, you know, some, some of things



Speaker:

like, you know, they're like all equal.



Speaker:

But I'll just say this, if you are not factoring in this concept, when



Speaker:

you are designing your backup system and when you're maintaining your



Speaker:

backup system, you just not doing your



Speaker:

job



Speaker:

Well, and, and I think that some people might kind of implicitly be doing some



Speaker:

of this without realizing it's like, you know, sometimes you do things



Speaker:

but you may not use the exact word.



Speaker:

Well, you might not use the word.



Speaker:

And so, so today we're gonna make sure that you know the word and, and, and it's



Speaker:

actually, there's actually four words.



Speaker:

Uh, there, there's two that we're gonna start with, and then we're



Speaker:

gonna add two more on top of that



Speaker:

Can this be like Sesame Street?



Speaker:

This letter is introduced by.



Speaker:

this episode is brought to you by the letters RT.



Speaker:

O and RPO.



Speaker:

That's what we're talking about this week, is recovery time objective



Speaker:

and recovery point objective.



Speaker:

And a lot of people, it's a little bit like, um, the effect and affect.



Speaker:

Uh, people always constantly miss the two.



Speaker:

And uh, I'm one of those people that constantly, when I.



Speaker:

I, I have to think twice.



Speaker:

Yeah.



Speaker:

I have to think like, is it affect?



Speaker:

This is the, the effect.



Speaker:

Um, yeah, like affect is the verb, effect is the noun.



Speaker:

Um, and, um, yeah.



Speaker:

Anyway, welcome to English, but what, why do you think, I think



Speaker:

RTO and RPO are so important?



Speaker:

Uh,



Speaker:

you know, when designing a backup



Speaker:

Because it drives everything, right?



Speaker:

It because it's not only understanding the requirements that are coming



Speaker:

from your stakeholders who hopefully you are talking to, right?



Speaker:

It also, uh, it helps define like what your backup system looks like,



Speaker:

what technologies you use, right?



Speaker:

All the rest of that, all because of these two key phrases.



Speaker:

Now I'm gonna, I'm gonna put something out there, and I'm gonna confess



Speaker:

that in my first several years.



Speaker:

Of working in the backup world.



Speaker:

I never once thought about RTO or RPO and, and when I say that,



Speaker:

I don't mean just the term.



Speaker:

I



Speaker:

mean it's like we, we really didn't, I'd say we focused a little



Speaker:

bit on recovery time, but very little time on the recovery point.



Speaker:

Right.



Speaker:

Um,



Speaker:

you go any further, can we first define what RPO and RTO



Speaker:

are just for our listeners?



Speaker:

Yeah.



Speaker:

Just because you're throwing out these terms and so.



Speaker:

And, and, and, and for those who, for those that struggle with the two.



Speaker:

Right.



Speaker:

And give you a just my, my quick, um.



Speaker:

Thing.



Speaker:

Um, and that is so recovery time objective is the goal that we've set that how



Speaker:

long a recovery time should take, right.



Speaker:

You know, um, from beginning to end, from bad thing happening, which could be



Speaker:

anything from, I deleted a file generally.



Speaker:

Generally, we're talking more about disaster recovery here than



Speaker:

an individual file, but it could be an individual, uh, database.



Speaker:

You delete one file, you take a database down.



Speaker:

That database is behind a really big application and poof, you know,



Speaker:

we're talking about RTO right?



Speaker:

But it's the time that we agree that a system, that it should take the, the



Speaker:

backup and DR system to restore the.



Speaker:

Thing, the entity, whatever it is that has been taken down back to



Speaker:

full, um, fully functional status.



Speaker:

How's that?



Speaker:

I agree with everything except that last sentence you said, and I wanna be very



Speaker:

careful here because a lot of people like this is called a recovery time



Speaker:

objective, not a restore time objective.



Speaker:

A lot of people in backup think it's just getting the data back,



Speaker:

that, well, that's why, that's why I put that



Speaker:

thing at the end.



Speaker:

Yeah.



Speaker:

Right, but it's not just the restore, right?



Speaker:

It's not just getting the data back, right?



Speaker:

It is



Speaker:

yeah.



Speaker:

Yeah,



Speaker:

that entity, which I'm glad you put the entity word right, because



Speaker:

it's more than just the data.



Speaker:

And I think that's an important concept that backup folks should understand is



Speaker:

you're just part of the process, right?



Speaker:

What you do is you're restoring the data, but there still has to



Speaker:

be that recovery of bringing the application back online, bringing all



Speaker:

the other components back up, right?



Speaker:

Doing, um.



Speaker:

Making sure that everything is good to go before you actually



Speaker:

bring up the application.



Speaker:

And all of that is encompassed in the RTO.



Speaker:

Yeah.



Speaker:

It, it, it's literally from the moment that the, the thing is damaged



Speaker:

to the thing is fully functional.



Speaker:

Yeah.



Speaker:

Whatever the thing was,



Speaker:

right?



Speaker:

We're talking.



Speaker:

We could be a data center, could be, um, a server, it could be a database, it



Speaker:

could be a really important file system behind, uh, you know, a database, right?



Speaker:

yeah.



Speaker:

And, but this, but, sorry, I don't mean to be pedantic, but this is



Speaker:

No, it's



Speaker:

just you two.



Speaker:

You mean to be pedantic, but go ahead.



Speaker:

So let me be, so it's really important though, because I think sometimes,



Speaker:

like going back to my initial point, right, that you should talk



Speaker:

to your stakeholders to understand the business objectives, right?



Speaker:

As a backup team, you are only responsible for a piece of the entire recovery.



Speaker:

So make sure when the business says, I have four hours to bring this application,



Speaker:

to recover this application, whatever it is, you're just a part of that, right?



Speaker:

You can't say, oh, I'm gonna take three hours and 59 minutes and I'm good,



Speaker:

Right.



Speaker:

right?



Speaker:

Yeah.



Speaker:

Because that doesn't meet the business objectives.



Speaker:

That's not the case.



Speaker:

Agreed.



Speaker:

So that's the recovery time objective.



Speaker:

And the key there is recovery time.



Speaker:

Right?



Speaker:

Just, you know, when you hear that recovery time, we're talking about the



Speaker:

amount of time of the recovery, right?



Speaker:

Um, and then the next one.



Speaker:

Is a little bit less obvious, I think.



Speaker:

Um, and that is the recovery point objective, which is the amount of data



Speaker:

that we've agreed we can lose as measured.



Speaker:

By time, right?



Speaker:

Um, so we agree that in a recovery we can lose one hour's worth of data, four hours



Speaker:

worth of data, two weeks worth of data, whatever the recovery point objective is.



Speaker:

That is, it's the amount of data that we have as a business.



Speaker:

Go into those stakeholders.



Speaker:

As a business, we've agreed to the amount of data that, the acceptable



Speaker:

amount of data loss as measured by time.



Speaker:

I do agree with your definition, right?



Speaker:

And I've always thought about it as how much data can I lose?



Speaker:

Right?



Speaker:

And it might be okay, I'm good losing 24 hours.



Speaker:

It's not a high I.



Speaker:

Change rate application or it's not a mission critical application versus



Speaker:

my financial database that keeps track of all my transactions that I'm



Speaker:

probably not good with the one day loss.



Speaker:

Right.



Speaker:

That is probably minutes.



Speaker:

Um, it, it's really about.



Speaker:

The RPO is driven by the kind of business you're in, the kind of application



Speaker:

that we're talking about, and it's driven by the amount of revenue that



Speaker:

you would lose when you lose data.



Speaker:

Right.



Speaker:

Um, and there, and, and just the, just the one key thing is that as measured by



Speaker:

time, it's just that normally when we talk about the amount of something in data,



Speaker:

we're talking about number of gigabytes or



Speaker:

terabytes.



Speaker:

In this case we're saying as measured by



Speaker:

time.



Speaker:

Right?



Speaker:

So three hours worth of data,



Speaker:

not



Speaker:

be a small amount or a large amount, and I think depending on the technology



Speaker:

you choose as well, I know you can also get down to fry fine grain, like



Speaker:

outstanding iOS that you are willing to lose in some of the technologies



Speaker:

that you could possibly use.



Speaker:

right.



Speaker:

So that's a definition of RTO and RPO.



Speaker:

And we, we say that this is, um, there's a lot of questions that come up here, right?



Speaker:

But with without those two values, we can't properly design a backup system,



Speaker:

right?



Speaker:

So here's the question.



Speaker:

I'm the backup guy at TI Squad company, and I need to start



Speaker:

designing my backup system.



Speaker:

What RTO and RPO should I be using?



Speaker:

You don't decide that



Speaker:

What



Speaker:

I am the backup guy.



Speaker:

I decide all things backup.



Speaker:

no.



Speaker:

Curtis, no, no, no.



Speaker:

Because no matter what you pick, if you don't communicate and get the



Speaker:

requirements from your stakeholders, I.



Speaker:

They'll be wrong and they will always yell at you, and it's,



Speaker:

they, well they'll, they'll still probably



Speaker:

yeah.



Speaker:

Well,



Speaker:

but, well.



Speaker:

so here's, so here's, here's my version of that answer, and that is, I.



Speaker:

Your RTO and RPO is not a backup decision,



Speaker:

right?



Speaker:

It is a business decision



Speaker:

yep,



Speaker:

and you, you,



Speaker:

you might not even have input on these numbers.



Speaker:

The, the only input you can have is, uh, I can't do that.



Speaker:

Well, well there, I think you have two inputs.



Speaker:

Mm-Hmm



Speaker:

One is I can't do that because it's technically impossible.



Speaker:

The second is I can do that and it's going to cost you a boatload of money,



Speaker:

Yeah, yeah,



Speaker:

yeah.



Speaker:

I think the cost is the other thing that you can come back with,



Speaker:

which might also sway the business.



Speaker:

yeah, so the first thing is that, that the RTO and RPO need to come from on high



Speaker:

and from, you know, side to side, right?



Speaker:

They need to come from you.



Speaker:

You've, you've brought up the word stakeholders many times.



Speaker:

Why don't you define what we mean when we say stakeholders?



Speaker:

Yeah, so stakeholders are people within your company who are either running the



Speaker:

application or who are in the business side who understand the risks with this



Speaker:

data and the importance of this data, and can help quantify how much is acceptable



Speaker:

as an overall company to lose, or the amount of time it takes to recover.



Speaker:

Yeah, and I would say that a lot of times, like, so you talked about people



Speaker:

that are running the application, sometimes the people running the



Speaker:

application don't have any more, I.



Speaker:

Yeah.



Speaker:

Knowledge than you



Speaker:

do.



Speaker:

It's really like the cu the customer of that application.



Speaker:

And by customer I mean the internal customer,



Speaker:

right?



Speaker:

The person who, who, whoever's making these recommendations needs to be



Speaker:

somebody with purse strings, right?



Speaker:

Um, that knows.



Speaker:

The, that knows the business impact of that application knows the, the, the,



Speaker:

the ins and outs of the business and how you make money, how you lose money.



Speaker:

And, um, because they need to be able to say, look, when this application is down,



Speaker:

we're losing a thousand dollars an hour,



Speaker:

whatever it is, right?



Speaker:

We're losing, you know, I, I, I've worked with companies that say that, that they



Speaker:

will lose a million dollars a minute.



Speaker:

Of downtime, right?



Speaker:

That number.



Speaker:

So, so the, how much money will I lose while the system is down?



Speaker:

That's going to drive your RTO.



Speaker:

Right.



Speaker:

If, if we're, if we're losing a, a million dollars a minute that the system is down,



Speaker:

that means I can spend $10 million for a system that's gonna get me back up in



Speaker:

five minutes because, you know, there's a



Speaker:

50% RTO there, or I'm sorry, RROI,



Speaker:

there's a 50, you know, 50% ROI there.



Speaker:

Um,



Speaker:

But on the flip side,



Speaker:

gotta go ahead.



Speaker:

but on the flip side, if you're like, Hey, I'm gonna spend $5 million on



Speaker:

something that's just gonna cost a thousand dollars a day, if it goes



Speaker:

down, uh, you're probably not gonna



Speaker:

You're not exactly, you're not gonna spend, you know, $5 million to get,



Speaker:

you know, to, to get you that system that can get you a five minute RTO.



Speaker:

If, uh, you know, you're, you're only gonna lose a thousand dollars a day,



Speaker:

right?



Speaker:

You're, you're just not gonna do that, right?



Speaker:

So, RTO and RPO, um, are agreeing on the RTO and RPO are gonna



Speaker:

drive, um, the cost of the system.



Speaker:

So, so if, if the amount of money that we lose while the system is down



Speaker:

drives the RTO, what drives the RPO?



Speaker:

How much data you can end up losing,



Speaker:

Yeah, I, I think I, I was asking myself the question in my



Speaker:

head as I was saying it.



Speaker:

It's, it's a difficult question because they're very similar.



Speaker:

Yeah.



Speaker:

So if, if



Speaker:

you're losing a a thousand dollars a minute, I.



Speaker:

Right.



Speaker:

The, if basically your company is, is, um, generating revenue



Speaker:

at a thousand dollars a minute.



Speaker:

If you lost, um, an hour's worth of data, you are going to lose



Speaker:

$60,000.



Speaker:

Yep.



Speaker:

Did I do that math?



Speaker:

Right?



Speaker:

Right.



Speaker:

If you're losing a thousand dollars a minute, your company's gonna lose



Speaker:

$60,000 of business that you had.



Speaker:

Yep.



Speaker:

Um, if you, if you lose an hour's worth of data.



Speaker:

But there's also,



Speaker:

in addition.



Speaker:

RTO also applies with



Speaker:

Yeah.



Speaker:

In addition to the amount of business that your system will or that your business



Speaker:

will lose, uh, either bus business that it had or business that it could have



Speaker:

had, also, someone needs to be able to



Speaker:

come up with the intangible things like damage to brand identity,



Speaker:

right?



Speaker:

Um, also.



Speaker:

Potential future lost data due to this is, this is primarily, well,



Speaker:

it's gonna drive both of them.



Speaker:

Because if you, if you, if you took an order from somebody and



Speaker:

then an order just disappears



Speaker:

and you don't even know who they are,



Speaker:

then um, you know, not only are you gonna not get that business,



Speaker:

you're gonna anger a customer.



Speaker:

Also lawsuits potentially.



Speaker:

And potential lawsuits.



Speaker:

Right?



Speaker:

Exactly.



Speaker:

Right.



Speaker:

That's a good point.



Speaker:

So you have to think about all of these different things.



Speaker:

This is, but again, this is what a business



Speaker:

does.



Speaker:

This is not your job as a backup and Dr person.



Speaker:

This is not your job.



Speaker:

If you're thinking, you're listening to this and you're like, gee, this



Speaker:

all sounds really complicated, and the big responsibility, yes.



Speaker:

It's why it's not yours.



Speaker:

Um, it's, it's the CEO, the CFO, uh, you know, a lot of



Speaker:

people with Cs in their names



Speaker:

management



Speaker:

and then the people that they appoint, uh, under them to, to make these decisions.



Speaker:

Yeah.



Speaker:

were saying, what would



Speaker:

Oh, the risk management team.



Speaker:

Risk management team.



Speaker:

Exactly right.



Speaker:

So if RTO and RPO drive backup design, what parts of backup



Speaker:

design do RTO drive and RPO drive?



Speaker:

So let's, let's do RTO first.



Speaker:

What, What, element of backup design does RTO drive?



Speaker:

It is how you do recoveries.



Speaker:

Def define how, what do, what do you mean?



Speaker:

All right.



Speaker:

Mechanisms that you use for recovering your data.



Speaker:

For example, do you have a standby DR.



Speaker:

Site available and ready?



Speaker:

Do you have data in a replica database?



Speaker:

Are you having to retrieve data from tape?



Speaker:

Yeah, all of these are, I was looking for something maybe a



Speaker:

little higher level than that.



Speaker:

Those are all correct answers.



Speaker:

Um, what I was thinking was, it's gonna deri, it's, it's going to



Speaker:

drive the, the, the design that determines the speed of recovery.



Speaker:

Yeah.



Speaker:

Right,



Speaker:

which everything you just said



Speaker:

determine the speed of the recovery, right?



Speaker:

If you have an aging system with an aging backup server and an aging, an



Speaker:

aging storage, and you're having to do a full recovery, uh, from older systems,



Speaker:

that's going to take a really long time.



Speaker:

But if you've got a cutting edge backup system with failover, Dr.



Speaker:

All of these things, um, then you should be able to do a very quick recovery.



Speaker:

Uh, you, well, you should be able to do a very quick restore,



Speaker:

which will enable to do, enable you to do a very quick recovery.



Speaker:

Um, so RTO primarily DR.



Speaker:

Drives, I'm gonna say the, the power of the backup system, the oomph, the



Speaker:

speed, uh, its ability to restore data quickly, which sometimes the



Speaker:

quickest way to restore data quickly is to restore it before you need it.



Speaker:

Right, and, and to, and to not restore it at all.



Speaker:

There are mechanisms to either restore the data before you need



Speaker:

it or to not restore it at all.



Speaker:

What would be a way to not restore it at all



Speaker:

Do you basically have a copy sitting there that is



Speaker:

or,



Speaker:

available



Speaker:

or



Speaker:

or a



Speaker:

snapshot?



Speaker:

Oh



Speaker:

I know how much you love when I bring that up.



Speaker:

yes.



Speaker:

So if,



Speaker:

Which is like copy sitting there.



Speaker:

well, it's a virtual copy,



Speaker:

Okay.



Speaker:

right?



Speaker:

Um, it's very important to distinguish



Speaker:

that, uh, at least what I would call a real snapshot is



Speaker:

a virtual copy.



Speaker:

Yes.



Speaker:

Um, yeah.



Speaker:

Um, if you're able to, if, if it's, if it's a copy just ready



Speaker:

to go and you're able to just.



Speaker:

Restore as I make quotes in the air by simply going back to an earlier



Speaker:

point in time, um, you are going to get a very quick RTO right?



Speaker:

Yes.



Speaker:

I just wondering also is re,



Speaker:

I'm wondering though if like even fast, like what's even faster than a snapshot?



Speaker:

I don't know what's faster than the



Speaker:

Because with the snapshot you still have to have someone



Speaker:

go and restore the snapshot.



Speaker:

Well,



Speaker:

it's, you're not really doing a restore, right?



Speaker:

I mean, a, a



Speaker:

good snapshot system.



Speaker:

You're just doing pointers, right?



Speaker:

So what do, I'm not sure what



Speaker:

Oh, so I'm thinking more like, um, as an example, clustered high



Speaker:

availability with the DR Replicated copy and automatic failover.



Speaker:

Yeah, yeah, that's all fine, right?



Speaker:

That that's all I, I don't know if that's faster than a snapshot, but



Speaker:

it's very similar.



Speaker:

Basically, you have stuff that's, it's immediately, immediately,



Speaker:

available and ready to go, right?



Speaker:

So there are a number of ways that you can, number of ways



Speaker:

from, from days to seconds.



Speaker:

To, to meet different levels of RTO based on, um, based on what you're



Speaker:

willing to spend and how you're willing to design your backup system.



Speaker:

Right?



Speaker:

Can I bring up a story?



Speaker:

sure.



Speaker:

So I was listening to a podcast on my walk, and it's of this automotive company.



Speaker:

They sell bolts and nuts,



Speaker:

Okay.



Speaker:

right?



Speaker:

Um, and they were talking about how they were, they were a small shop



Speaker:

and they have like 30 people now.



Speaker:

And the guy, the co-founder or the CEO founder, he basically ran it.



Speaker:

And they had a bunch of systems and things like that, and he



Speaker:

was kind of managing everything.



Speaker:

And then they got hit with, uh, ransomware



Speaker:

Yeah.



Speaker:

and they lost all their data.



Speaker:

And then he realized he had a backup, but it was from three months ago.



Speaker:

Right?



Speaker:

So his RPO was three months,



Speaker:

Nice.



Speaker:

Well.



Speaker:

RPO.



Speaker:

No, his RPO never changed his RPA.



Speaker:

We haven't got to that



Speaker:

yet.



Speaker:

Yet.



Speaker:

Well that's what he had.



Speaker:

Yes.



Speaker:

And he basically said though, so they had a bunch of invoices and customers and



Speaker:

luckily they had everything hard copy.



Speaker:

He said it took them six months to manually enter, to manually enter



Speaker:

all the transactions that they were missing before those three months.



Speaker:

At least he was able to get, uh, to put that data back,



Speaker:

right.



Speaker:

Yeah.



Speaker:

Uh, by the way, this, this is just as good a time as any to bring up the



Speaker:

terms RPA and our, our RTA and RPA.



Speaker:

So recovery time actual and recovery point actual.



Speaker:

So a lot of people say, you know, and I do it, I do it occasionally,



Speaker:

so it's not like I'm, I'm



Speaker:

calling outy or Prasanna.



Speaker:

Um, we say, oh, we, you know, we recovered it in two hours.



Speaker:

So we had a two hour RTO.



Speaker:

The So recovery, it's recovery point objective, recovery time objective.



Speaker:

This is the goal.



Speaker:

This is the service level agreement, right?



Speaker:

The



Speaker:

SLA,



Speaker:

Oh.



Speaker:

So I guess in my, in that example case, they had no agreement because they did



Speaker:

yeah, that.



Speaker:

That's actually the most



Speaker:

common.



Speaker:

That's the most common, right?



Speaker:

No, no.



Speaker:

SLA, no, R-G-O-R-P-O.



Speaker:

We just do our best case and, um.



Speaker:

But how, how fast the system is actually able to do something is your RTA and RPA.



Speaker:

Right.



Speaker:

Your recovery to recovery time.



Speaker:

Actual and recovery point actual, uh, I've also in, in various times in my



Speaker:

career, referred to that as the RTR and R-T-R-P-R recovery time reality.



Speaker:

Yeah.



Speaker:

Um,



Speaker:

but



Speaker:

that?



Speaker:

How do you actually



Speaker:

figure out



Speaker:

the only way to calculate is to actually do it right.



Speaker:

So hopefully everyone is doing restore testing



Speaker:

or recovery



Speaker:

story testing is the only way to know for re for sure.



Speaker:

All right, so let's talk about, you talked about how, what, uh, aspects of



Speaker:

backup design are determined by RTO.



Speaker:

What this answer is much easier.



Speaker:

What aspect of backup design is determined or is Yes, is determined by RPO?



Speaker:

How frequently can you back up your data?



Speaker:

Yeah.



Speaker:

So if you're backing up your data more often, you are going



Speaker:

to have a better RPA, right?



Speaker:

Um, so if you have, if, what's the most common, uh, backup



Speaker:

frequency, would you say?



Speaker:

Probably once a day.



Speaker:

Once a day.



Speaker:

And so the most common RPA is at minimum 24 hours.



Speaker:

Well, the, the, you know, there are situations where



Speaker:

you might get better than that



Speaker:

if you, like you said earlier, if you had a failure right after



Speaker:

you just finished a backup,



Speaker:

but worst



Speaker:

um, you might, that's best case



Speaker:

scenario, worst case scenario.



Speaker:

Is more like 36 hours because you're, you're, you know, at the end of a



Speaker:

day before the, you know, and, um.



Speaker:

The, and, and that, that's assuming that last night's backup worked,



Speaker:

and I've done enough backups to know that that isn't always the case.



Speaker:

And I would also say I would apply Murphy's Law of backups, which is the



Speaker:

the degree to which, you know, a ba the, the, the possibility that a backup is



Speaker:

successful is inversely proportional to, um, how, how much you need that backup.



Speaker:

Yeah.



Speaker:

Is there also a Schrodinger's cat for backups?



Speaker:

Yeah.



Speaker:

Yeah,



Speaker:

exist, or you don't know if a backup exists or not until you actually,



Speaker:

Until you actually look at it.



Speaker:

yeah,



Speaker:

Um,



Speaker:

I think we need that on a T-shirt.



Speaker:

By the way, if you would buy a T-shirt from



Speaker:

the backup wrap up, right, with these phrases on it.



Speaker:

Let Curtis know on X or LinkedIn or somewhere, or leave a



Speaker:

Yeah,



Speaker:

on YouTube.



Speaker:

We've been getting a lot of YouTube comments lately.



Speaker:

Um, yeah, let me know on YouTube, by the way, did I tell you, you know



Speaker:

how it's how back when I was not feeling well or I hurt myself and



Speaker:

I said, no one cares.



Speaker:

Then I said, if anyone cares, please say something.



Speaker:

We got comments.



Speaker:

One of,



Speaker:

you know what?



Speaker:

One of them said,



Speaker:

What?



Speaker:

India cares about Curtis.



Speaker:

Aw.



Speaker:

That's what I got.



Speaker:

Um, so.



Speaker:

Yeah, so RPO basically determines your backup frequency.



Speaker:

If, if you've got an RPO measured in hours, you can't have a backup frequency



Speaker:

measured in, in, you know, many hours.



Speaker:

Right?



Speaker:

Um, and the, um, I, I will say that,



Speaker:

just talk about the different ways that people do backups, sort of normal.



Speaker:

Regular, what I would call full file incremental backups.



Speaker:

Pretty much the best you're gonna be able to do is daily.



Speaker:

Right.



Speaker:

If you're able to do some type of block level incremental forever,



Speaker:

where each backup, you're only backing up the blocks that have changed,



Speaker:

or even better, the deduplicated blocks that have changed, right?



Speaker:

Because that's gonna be an even smaller, uh, set of data.



Speaker:

Then you could actually do the ba It does two things.



Speaker:

One is it makes the, it makes the backup.



Speaker:

Quicker, which is good.



Speaker:

Um, which generally means that you can do it more often, uh, and then it, it



Speaker:

does allow you, it also reduces the, the impact of the backup, meaning that while



Speaker:

the backup is running, it's not having too much of a, of an impact on production.



Speaker:

And so that allows you to do it more often.



Speaker:

To do it, you know, once an hour is very common.



Speaker:

Uh, if you've got a, a modern backup system, you could do it once an



Speaker:

hour or even more often than that.



Speaker:

I.



Speaker:

Yeah.



Speaker:

While you're mentioning this about the incrementals and chain block tracking,



Speaker:

one thing I realized we didn't cover is depending on what technologies you use,



Speaker:

it will have an impact on your RTO because if you have say, multiple, uh, recovery



Speaker:

points, you have to recover from first before you can get to that final point.



Speaker:

Like if you have to recover a full plus multiple incrementals.



Speaker:

That is going to take you more time, which will impact your RTO as well.



Speaker:

Absolutely.



Speaker:

Um, I do want to talk, so what we'll round out this discussion with is that



Speaker:

idea that you brought up earlier, which



Speaker:

is the,



Speaker:

that, um, every stakeholder is gonna want an RTO and an RPO of zero.



Speaker:

Because if you ask people, if you ask it like this, how much data.



Speaker:

You know, how, how long are we allowed to be down and how much



Speaker:

data are we allowed to lose?



Speaker:

And the answer is zero and zero.



Speaker:

Yep.



Speaker:

How do we get from there to something that's more reasonable and possible?



Speaker:

$1 billion.



Speaker:

Yeah.



Speaker:

What, so yeah, so you go back, you're like, okay, all right.



Speaker:

We've taken the requirements, we've gone and we've met with the powers



Speaker:

that be, and we've decided that if we go with a fully redundant system.



Speaker:

Uh, with, you know, with, with, uh, synchronous replication and a hot site



Speaker:

that's, you know, this far away we can do, we can give you the RTO and an RPO of



Speaker:

zero and it's going to cost $1 billion.



Speaker:

Right.



Speaker:

And you do that with a straight face, right?



Speaker:

You just go, okay, okay, we got, you know, we got a thing, we got a thing, we got the



Speaker:

vendor, we got this vendor standing by to



Speaker:

take your check.



Speaker:

And uh, we can do it for $1 billion.



Speaker:

And then, um, um.



Speaker:

What I generally find is at that point, obviously I have a Plan B and a plan C,



Speaker:

and uh, I've also consulted with this vendor.



Speaker:

And this vendor says that they can do an RTO and an RPO of



Speaker:

four hours for only $1 million.



Speaker:

$1 million.



Speaker:

Sounds a lot.



Speaker:

Sounds like a lot,



Speaker:

but it's a better than a billion.



Speaker:

it to $1



Speaker:

So, so here's a question.



Speaker:

So as a backup admin, when you're going to these discussions with the stakeholders,



Speaker:

Mm-Hmm.



Speaker:

is it.



Speaker:

Worthwhile to go with a couple options, like you mentioned where you're like,



Speaker:

and what would those common options be?



Speaker:

Right.



Speaker:

Do you think it's zero?



Speaker:

Zero, right?



Speaker:

It's a four hour RPO.



Speaker:

Maybe it's a one day RPO.



Speaker:

Like



Speaker:

Yeah.



Speaker:

Um,



Speaker:

what?



Speaker:

What would make sense?



Speaker:

Yeah.



Speaker:

I think it's going to be driven.



Speaker:

Most backup designs.



Speaker:

In fact, I'm gonna say all backup designs are driven by cost and reality,



Speaker:

right?



Speaker:

So, you know, they want zero and zero.



Speaker:

Or let's say, let's say they want 15 minutes.



Speaker:

We're like, look, we can do 15 minutes, 15 minutes costs this much.



Speaker:

This is the way to do it.



Speaker:

Um, if they, if they say.



Speaker:

We wanna, we wanna, you know, a very tight RTO, but we're okay with



Speaker:

an hour's loss, data loss, right?



Speaker:

Snapshots and replication.



Speaker:

Great way to do that.



Speaker:

If they want very, you know, like as close to zero as possible, you've really



Speaker:

got to do CDP or something like it.



Speaker:

You gotta do synchronous replication.



Speaker:

And by the way, it's important to mention here that not everything has



Speaker:

to have the same RTO and RPO in fact.



Speaker:

In your environment?



Speaker:

Everything should in your environment, everything should definitely not have the



Speaker:

same RTO and RPO.



Speaker:

If that's the case, it means you've sort of settled you.



Speaker:

You've definitely got an application that needs more than that.



Speaker:

Uh, and you settled for simplicity, which is very common for people to do, but



Speaker:

that's not really the right thing to do.



Speaker:

You've got one or two applications within your environment that would really lose



Speaker:

a lot of money, and it's totally okay to have a different backup design for them.



Speaker:

Right.



Speaker:

You have the gold, silver, bronze, right?



Speaker:

Um, I, I definitely like, go ahead.



Speaker:

W, and I know you mentioned backup technology, like a different backup



Speaker:

technology that might also drive having different backup vendors as



Speaker:

well to satisfy these different needs.



Speaker:

Yes, it definitely will be.



Speaker:

I mean, my, as a



Speaker:

backup person, my preference is always one vendor if possible.



Speaker:

Um, and, um, you know, one design with options, uh,



Speaker:

because again, simplicity, um,



Speaker:

means recovery, I think.



Speaker:

Um, but.



Speaker:

but.



Speaker:

yeah, it, it is really important to understand that it's very common



Speaker:

for you to have multiple RTOs and RPOs within an environment.



Speaker:

I will also state, and this is somewhat controversial and not



Speaker:

everybody agrees with me here, it's also okay to have different RTOs



Speaker:

and RPOs based on scenario, right?



Speaker:

Like if we've got, um, if a server died or a server caught on fire or a server



Speaker:

got whatever, you know, server goes, poof.



Speaker:

seems to me that that, that, that RTO and RPO, um, would be



Speaker:

much, much shorter than if an asteroid hit Oceanside, right?



Speaker:

Um,



Speaker:

it's just there are different levels of disasters and there's different amounts



Speaker:

of understanding that you can have.



Speaker:

So



Speaker:

I think it's totally okay to have different kinds of RTOs



Speaker:

and RPOs for different kinds of



Speaker:

and, and I think this is where we kind of like from talking to Mike



Speaker:

about our ransomware recovery and other things like that, right?



Speaker:

I think this is where sort of the incident response plans come in, right?



Speaker:

Where different scenarios you may have different RTOs and RPOs.



Speaker:

Because they are different and there are things that are outside of your



Speaker:

control, like recovering from ransomware is very different because of the initial



Speaker:

steps you have to do to isolate and figure out what's going on, rather



Speaker:

than, oh, this server, the disc died.



Speaker:

Now I need to recover.



Speaker:

Or someone accidentally deleted a file.



Speaker:

Yeah.



Speaker:

And by the way, you know, speaking of recovering from ransomware, you know



Speaker:

when, when we've talked about that, when you get to the actual recovery, it's



Speaker:

very common for a person who knows what they're talking about in that world.



Speaker:

Say, listen, I want you to, you might be recovering the same data set 50 different



Speaker:

times from 50 different data points.



Speaker:

That's something you really gotta figure out in your backup design.



Speaker:

Right.



Speaker:

Because there are backup designs where that is not gonna happen anytime soon.



Speaker:

Right.



Speaker:

Um, but, uh, well, hey, once again, uh, we did not do a 10 minute episode.



Speaker:

Yay.



Speaker:

And we're all alive.



Speaker:

But hold on.



Speaker:

So going back to my initial question I had, or which is, I know you said RTO



Speaker:

and RPO are probably the most important things, foundational things in backup



Speaker:

yeah,



Speaker:

system design.



Speaker:

What did you think?



Speaker:

Do you think that second from that would be the 3, 2, 1 rule?



Speaker:

Well, the 3, 2, 1 rule is, um,



Speaker:

it is also foundational, right?



Speaker:

Um,



Speaker:

and.



Speaker:

I, again, is it more important?



Speaker:

Is it if you don't, if you don't follow the 3, 2, 1 rule and all your backups



Speaker:

are all in the same place as the production, why are they're not even



Speaker:

Back up.



Speaker:

Yep.



Speaker:

Yeah.



Speaker:

Agreed.



Speaker:

Maybe next episode, should we, we should dive down into 3, 2, 1 rule.



Speaker:

We haven't done that in a while.



Speaker:

Yep.



Speaker:

All right.



Speaker:

Prasanna, bossy mc.



Speaker:

Bossy



Speaker:

man.



Speaker:

Um.



Speaker:

my bossy pants,



Speaker:

I.



Speaker:

I hope you folks have enjoyed our episode this week on RTO versus RPO.



Speaker:

That is a wrap.



Speaker:

The backup wrap up is written, recorded and produced by me w Curtis Preston.



Speaker:

If you need backup or Dr.



Speaker:

Consulting content generation or expert witness work,



Speaker:

check out backup central.com.



Speaker:

You can also find links from my O'Reilly Books on the same website.



Speaker:

Remember, this is an independent podcast and any opinions that you



Speaker:

hear are those of the speaker.



Speaker:

And not necessarily an employer.



Speaker:

Thanks for listening.