How Forever Incremental Backup Changed the World

You found the backup wrap up your go-to podcast for all things

backup recovery and cyber recovery.

In this episode, we're covering forever incremental backup technology and

how that completely revolutionized the entire backup industry.

I. Remember the old days of doing weekly fos that brought

your systems to their knees?

Well, those days are thankfully gone for most people.

We're gonna walk through the evolution of, from traditional approaches with

fools and incrementals to synthetic fos, and finally to true forever incremental.

If you're still doing things the old way, well, you're missing

out on some serious benefits.

Better performance, lower storage costs, and way faster restores.

Perhaps this episode will cause you to rethink things a little bit.

By the way, if you don't know who I am, I'm w Curtis Preston, AKA, Mr. Backup,

and I've been passionate about backup and recovery for over 30 years, ever since.

I had to tell my boss that we had no backups of the production

database that we had just lost.

I don't want that to happen to you, and that's why I do this.

On this podcast, we turn unappreciated backup admins into Cyber Recovery Heroes.

This is the backup wrap up.

Welcome to the show.

Hi, I am w Curtis Preston, AKA, Mr. Backup, and I have with me a guy that

I bet is pretty dang happy right now.

Prasanna Malaiyandi.

Why are you looking like that?

very confused about what I'm happy about.

Really?

Do I need to explain recent events that happened at your house

Oh

that, that.

For those of you by the way, um, who are listening to this podcast via

audio, we also provide video on YouTube.

So if you wanted to see our expressions just then and really see what Curtis

is laughing about, uh, go check us

Yeah,

Backup wrap up.

so what events happened in the last 48 hours?

my wife was in India

Yeah.

Visiting family and she just got back.

Yes.

it was good.

Suddenly, you're so much harder to get on the phone between that and your new job.

I don't even know you anymore.

He's like, why would I be happy?

I, I was really, I was, I thought it was like, I was thinking, I was like,

was it something YouTube related?

Was it something like I, we had watched or we had talked about and.

No, the fact that your wife is back

although she's also slightly jet lagged, so,

so well, it's what, what, what's it is 12 hours difference.

right now it is 13 and a half.

Yeah.

Yeah.

So

does

of course she's,

jet lagged than I do.

So

yeah.

and come back, it takes me like three weeks.

And,

Yeah.

have a dog, which I think some people have seen on the podcast.

Um, it's funny because he gets jet lagged when I'm jet lagged.

So he, his hours will shift to match with mine.

So

That's really adorable.

and he would be like, Hey, I'm ready.

Let's go, let's go.

And then when I'm passed out, he's asleep.

So he's

Interesting.

Yes.

Your dog that's named after an Indian dessert.

cream.

Yep.

Ice cream.

Oh yeah.

Ice cream.

Yeah.

Kulfi, which I have now had, um.

your, did you know they have various flavors of Kulfi.

They're not all just like one.

Oh, I thought it was all mango.

So it's met like mango Kulfi and then what?

Coconut Kulfi.

What are what?

What are the other typical

and pistachio.

Pistachio and, and.

I think there are other ones too.

Yeah.

I don't, yeah, I don't.

I've used cardamom as a, an ingredient, you know, when I made the, the

mango pie thing, but I don't think I know what it tastes like by itself.

So I'll have to, I'll have to sprinkle some and, uh, see what that tastes like.

Or maybe just go and

Yeah.

some

next time you're at the Indian store.

Actually, sometimes even the Costcos carry like a sampler pack

Oh, yeah, so like when I go to the cash and carry down in Mira Mesa.

the cash and carry will definitely have it.

If you go check out the freezer section.

Just

Interesting.

all before you get home.

Yeah.

I, oh man.

I'm, you're, you're literally like, like my mouth is Yeah, yeah.

Thinking about, thinking about it sounds yummy.

Hmm.

Uh, so we're gonna talk about something that's just as yummy.

That is if you're into, into the backup thing, uh, we're gonna talk

about forever incremental and sort of.

Uh, just how big of a deal that is or was.

Um, it, at, at this point?

It's in, in most modern backup products that have come out in

the last 10 years, I'd say, uh, ones that are actually new, right?

So, you know, when we look at the products like Rubrik, Cohesity, or

modern SaaS applications like Druva, Hyku, keep it, Alcion, uh, right.

You know, these are all of these products.

Have forever.

Incremental in common, right?

Nobody wants to do full backups anymore.

Uh, or, or anything like full backups, right?

So, um, so let's talk about that.

Let's talk about what, you know, where were we, what was it like back in the

day, and, um, and then sort of how we got the, from there to here and why it's

such a big deal and why you should care.

Right.

And why this perhaps may give you a reason to finally, um, you

know, what, what is it, shuck?

The mortal coil.

What, what, what, what is that?

What,

I'm just

thinking are there company, and maybe as we're talking we can figure this out, like

are there still vendors that exist that don't support forever Incrementals?

That's actually a really good question and I'm gonna say, uh, absolutely

right, because my concept of what a true forever incremental will be based on.

Like a, a complete redesign of the, of the underneath.

Right.

Um, so the phrase I was looking for is shuttle off sh shuffle off the

mortal coil of your old backup product,

by the way, wait, quick story.

So I'm on, I'm on a plane one time and this, this, um.

This pilot, he was talking about the fact that we were about to take off and

he referred to taking off as shuffling off the mortal coil of this earth.

And I'm like, dude, that's a phrase for dying.

What?

Why the hell are you, why would you do that?

Right.

, it's a Shakespeare, it's from the to be or not to be soliloquy

I,

Hamlet.

in all fairness though, I would say 97% of people probably had

no idea what that phrase meant.

They, they never even heard.

Yeah.

Yeah.

And, and, and, and, and just to, just to, uh, like, you know, since I'm over

here quoting Shakespeare, let me just put this, I'm not a huge Shakespeare person.

In fact, it, it wasn't until, I dunno, several years ago, and we were in England

where I, I, I got to actually see my first, um, uh, Shakespeare play, which

was Henry V. Some people say Henry the VI don't, I think it's actually

called Henry five, which is the, um.

The one with the St. Crispin stay speech.

Uh, once more into the breach, close the wall up with, with

her English dead and all that.

But anyway, sorry, I just bored everybody.

Sorry.

Sorry, you're asleep.

Wake back up.

Wake back up.

We're, we're gonna talk about backup, which is so much more

exciting than Shakespeare.

Um, so, all right, so first let's talk about, do you want to define, uh, so what?

let's talk about the old days first.

that's what I, that's what I'm saying.

That's what I'm saying.

Let's, do you wanna define, uh, what a full and, and

Yeah.

an incremental backup is?

Yeah.

So.

Normally when you think about backups, right, you talk,

typically people talk about fulls.

Incrementals, right?

A full is basically your entire application, dataset, whatever

it is, everything is backed up.

Mm-hmm.

then between doing every fulls, you would do incrementals, which are just the things

that have changed since that last full.

Right, right.

you do these periodically, and normally you do incrementals

because they will be smaller.

So you could say do a full once a week and then incrementals during the week

such that you can finish in time or during your backup window, and then the next

weekend you would do like another full,

Right, right now, slightly, I'll slightly change your

definition or a pen or whatever.

Um, so you specifically described what I would call a

cumulative incremental, right?

So a, uh, what is referred to as a sort of a regular, incremental

or a differential incremental.

Yep,

it is one where it's the changes just since the last incremental or since

the last backup of any kind, right?

And then you

Um,

levels and all the

yeah.

Yeah, yeah,

But yeah, at a high level, yeah, there are those three types, if you will.

yeah.

And, and, and then what really, um, what really what it really

came down to is how things behaved when it came time for a restore.

Right.

So if you were going to restore something, then you needed the full,

the latest cumulative, incremental, if you were doing such a thing, and

then any incrementals since that day.

And so, um, those of us, uh.

I, I remember at some point we, we used to do weekly fulls.

I remember my preferred backup design switching to monthly fulls,

weekly cumulative incrementals, and then daily incrementals.

So if you were, let's say three weeks into the month, you would need the full

from the first of the month, the latest, uh, weekly cumulative, incremental,

and then three or four days worth of.

Incrementals

Yeah,

and, uh, in order to do a restore.

. So what it meant was that you would often, the way the Restore worked is you

would restore everything that was in the full, then you would restore everything

that was in the cumulative incremental, and then you would restore everything

that was in each of the incrementals.

What that meant was there were many files that you were actually restoring

multiple times and you were overwriting them, which was just inefficient.

Um, and what that also meant was.

That you had, uh, uh, duplicate data, the old way of doing backups created

a ton of duplicate data on the tapes.

You could say duplicate, or you could also say it's independent,

depending on the glass.

Half full glass, half empty.

Well, so no, I, I understand what you're saying.

What I'm talking about is the fulls,

Yeah,

the fulls over time created a ton of duplicate data.

that's true.

Right.

I know what you're thinking.

You're thinking, you're talking about the, within the, within the, the incrementals.

I'm mainly talking about

yeah,

how many different copies of the exact same file were all

over, you know, everywhere.

and especially because most times, right, you look at sort of like a

2% daily change rate, which maybe

Yeah.

a seven or 8% change rate per week.

And

Yeah.

those weekly cumulatives or even the monthly fulls, right, you're

basically backing up a bunch of data, which you don't necessarily

need that haven't changed.

Right, right.

You, you, the reason we did it the way we did, it wasn't so much about independence.

It was about.

Minimize.

So if you did a monthly full, followed by nothing but regular

incrementals, you would have to load 30 tapes at the end of the day, right?

At the end of, and even if it wasn't tapes, you would have

to, you have to do 30 restores.

Yeah,

So, so the, the, the, the differential or the cumulative incremental, um,

was about minimizing the level of effort during a restore, right?

And then came along this thing called dedupe.

Uh, which at least helped minimize, I think the, all of that duplicate data.

because I think with deduplication it was nice because you didn't

have to change what you were doing from like a policy perspective

Right,

perspective.

But you got, like you said, it was able to.

Dedupe, of course,

right.

in the backend and save you the storage costs.

And that's why when, if you, if you were doing a, a typical weekly, by

the way, nobody did the, the monthly full thing that I recommended, most

people did weekly fulls, right?

And so when you had weekly fulls, um.

This is why, and you, and you had like 90 day retention.

This is why just with even crappy dedupe, you got 20 to one,

Yeah.

uh, data reduction, right?

And this is what helped, um, pay for, you know, the what to help

created the, the market for all that, all that dedupe work, right?

And then we could argue over, um, you know, well we're, we're

20 to one, we're 30 to one, we're, you know, whatever, right?

And, um, we, we saved this much more money.

Um.

You know, once you get rid of all those fulls, everything else

was, was really kind of incre

Yeah.

incremental.

Um, but the real, I think the real challenge was just the core backup design

of when we go to do a restore, we have to restore, uh, data multiple times.

Right.

It just simply wasn't efficient.

Well, and I think even before we get there, I'd like to cover

Mm-hmm.

uh,

Okay.

if you had the existing backup schedule, right, where I'm still doing

weekly fulls and daily incrementals,

Mm-hmm.

still transferring all that data over the network.

Right?

And I think this is where client side deduplication would probably

come next, in my opinion, right?

Yep.

it would help you reduce the amount of data you're actually

sending over the network.

Right.

And that'll help save you cost as well as time.

Yeah, that's actually a really good point.

Thanks for bringing that up.

So the, one of the other like, like basically what you're saying that one

of the other, um, downsides to doing it the old way was both the transfer

across the network as well as the impact, uh, to the clients, right?

If you're doing a full, on a regular basis, um, you're, um.

There's a, there's a cost to that

Yep.

in both, you know, you brought up bandwidth, it's

bandwidth storage and compute.

Right?

Um, it's like all the things.

There was also a, um, sort of a, like a perfect storm brewing.

Yeah.

So we're doing fulls on a regular basis.

We're doing cumulative incrementals on a regular basis, and this is

how we were doing things, and then along came virtualization.

Why was that a problem?

Because a lot of the data was

I,

the same, right?

When you look across all of your VMs, they're probably spun up

from the same golden copy, of your

Well, the,

and

yeah, well, the.

amongst them.

Well, that, that's a, this is another example of, that's a great answer.

It's not the question I was asking.

It's

Um,

resource on the backend because

that's what I was going after.

Yeah,

and the compute, right?

You don't have your

yeah,

instance

yeah.

That's, that's what I was going after.

Right?

So, so that was a real problem when VMware got really popular and we just

kept doing the same old thing because.

Remember, for a long time we backed up VMware.

We backed up the physical, the virtual machines, just like

they were physical machines.

Yes.

Guest ba. We just basically installed an A, an agent and every guest,

and that's how we did the backups.

And, you know, and what happened was you'd do just randomly, you'd end up

doing a bunch of, at the same time.

Which worked fine when we were 20 individual machines, but now we're 20

VMs all on the same physical machine, and that just broke, like virtualization

just broke back up overnight.

Right.

Um, the latest thing to break back up overnight is, um, containers.

Yep.

That's a whole other thing.

We're not talking about that right now, but, um, the

so

ahead.

though, this is what sort of VMware, Plus the backup vendors to figure out

is there something better we can do

Yeah.

order to help satisfy backup needs?

Because it's not going away, right?

You have to

Right,

VMs.

right, right.

So the the real key I think, I think the real goal, I. Like, like if we could

go back in time and I, I don't know.

I don't know.

This is one of the things where like, I don't know if we, I, I wasn't in the

rooms right when they were designing this stuff, but I'd like to think that

somebody said, you know, if, because my memory is, we had issues with tape.

The, the, there was the fundamental mismatch of speed tape.

You know, people thought it was slow, it was actually fast.

That was a problem.

And we need, and so we started doing disc staging.

We started doing disc backups because of that, right?

And then DDU came along and DDU said, Hey, we can make disc better.

We can make disc, you know, more affordable.

And, and so, so that was, and then it just sort of happened.

But even if.

We if, if that wasn't the case, I think the design of backups ultimately ending

up being, uh, what we're talking about today, which is forever incremental,

it, it accomplishes multiple purposes.

One is it, it, it solves the issues that we have with tape.

Because in order for forever incremental work, you've gotta be using disc, right?

Uh, for, for true forever, incremental work to work, you've gotta be using disc.

Um, and then the other thing is that it, it solves the, the cost issues.

It solves the performance issues, restore, you know, it does a lot of things right.

Um,

you first define what forever incremental is?

'cause I don't

I don't want to,

that.

Before, before we get to forever incremental, we gotta talk about

something that was, that was a stop gap technology bef to get from we'd,

we wanted to get to a point where we weren't doing fullsS anymore.

Everybody agreed that full stunk, except especially when we're

talking about virtualization, right?

What was the stop gap technology that backup vendors came out with before

we went to the whole For incremental?

Uh, they created what they called synthetic fulls,

Mm-hmm.

which is basically you are.

Creating a full on the backend, but you're sending data and stitching it together and

piecing it together to make it look like what the full would look like, because you

Yeah.

backups were on that

Right.

So it's like we already know, we, we already know what the files are in

the full, because, you know, we did it on we, we did another incremental.

So we know what's deleted.

We know what's new.

We know what a full would look like.

If we went and made one, why don't we just make one over here

without having to do it over there.

Um, and there were a couple of different ways to do that.

Uh, there was the backup vendor way and there was the, um, the, well, the

backup software vendor way, and then there was the backup hardware vendor way.

Yeah.

I'll talk about the backup software vendor way first.

It's like, we know what all the files should be on a full,

we know where all of them are.

We're just gonna copy them all together.

You know, in one image, and then we're gonna copy that over to a new

set of tapes or a new set of disc

Mm.

so that we essentially create an actual full, um, without actually having to

transfer any data around the network.

It might be transferring it, uh, well it is transferring

it within the backup system.

Right.

And then there was the, the, and, and that took a certain amount of time.

Right.

Um, and then.

A, a vendor that you might be familiar with came out with

a different way to do it.

What was that?

And basically, um, they basically said, just tell us how you wanna

stitch everything together,

Yeah.

it all together.

Yeah.

So it was like, it was like an API, right.

You know where all the stuff is.

We have all the stuff.

Just tell us which stuff you want, where, and we'll go.

Yep.

And so they could make essentially a new full.

When in reality it's just a bunch of pointers to what was already there.

Right.

this

And it was really quick.

Right,

it was really quick because the key is, especially on a de-duplicated system,

you're not actually copying the data.

It's all pointer manipulation.

right, right.

moving any data, you're just changing metadata, which is very quick, very fast.

That was a great singing group from the nineties,

What

I

metadata.

manipulation.

But I'm, anyway, so yeah, so that was sort of a stop gap solution.

It allowed people to stop doing real fulls, solve the issue of the, the,

the load on the client, solve the issue of the load on the network.

But we still, we were still creating fulls.

So the concept of forever incremental.

And, and true forever incremental is that we first have to design a backup product

that just no longer needs repeated fulls.

Right?

And, and by the way, this generally does only apply to, um.

Uh, file system backups for the most part.

Um, not database and application backups.

Most of them still require repeated fullss.

Um, you know, we'll, we'll, some of, some of them have forever incremental, but.

I would also say, or would you also say for virtualized

workloads, this also applies 'cause

Um, oh, you mean, uh, the, the backup of the di of the system itself?

Yes.

Yes.

So, well, so I'll, so if I, if, if I was to use a term that would apply to,

to both of them, I'd say volume-based.

Yeah.

Backups.

Right?

So either a file system or the, or a,

Disc.

file or a VDK file, right?

Um, and the idea is that we're just gonna do one full, and then

we're generally going to be doing block level incremental backups.

Um, not just incremental, but block level incremental.

What's the difference between those?

Well, a block level incremental allows you to look at changes

within, a specific object.

As an example, say you had, uh, 10 gigabyte PST.

File representing your email and instead of doing a normal incremental, which

would transfer the entire 10 gig file over when you back it up using traditional

incrementals, block level incremental would just take the changes in within that

10 gig PST file and transfer that over.

Exactly, and so it, it, it's another data reduction technology like deduplication.

And it, it, it just, it, I I think it's a, it was an order of

magnitude reduction in the number of bytes that has to be transferred.

And, and the way this worked in, in, uh, file system backups was that

you had to have like an API into the actual blocks, the, the discs, right.

Uh, which.

Was somewhat problematic.

The, the, the easier way to get this was in the virtualization

world where you had an API, right, like the change block tracking API

Mm-hmm.

you can just, the backup app can just say to the, the virtualization

vendor, Hey, I'm here to back up again.

Just give me a pointer to all of the blocks that have changed

since the last time I was here.

Yep.

Um, and then we can transfer just those change blocks, but.

But.

importantly, for a true increment, for a true forever incremental, um, to, to, to,

for me to think that it's a truly forever incremental is that you're going to change

how the data is stored on the backend.

So there really isn't a concept of full, right?

Um, the way that you're storing each new backup is that each new backup from a

restore perspective behaves like a full.

Yeah.

Right.

You

Yeah.

want to like put that in?

Go ahead.

so I was just gonna put it in my own words, which

Yeah.

basically you're able to send only the changes and end up with

the full on the backend, such

Right.

each one is a full, they can be independently restored without

any dependencies on others.

Yeah.

So you don't have this idea of restoring the full and restoring

the incrementals individually.

Right.

You just restore the late, you know, it's sort of, it's, it's sort of like take what

we used to do, take the synthetic full.

Every time we do an incremental, we're essentially creating a new synthetic full.

Yep,

How's that?

yep.

Yeah.

Yeah, because if you think about it, it's just kind of changing

where the processing happened,

Right.

Because with Synthetic fulls, the client was telling the storage, okay, these

are the pieces I want you to stitch together, and it might have been pieces

across multiple different backups,

Mm-hmm.

right?

In the case of an incremental forever, it's like, Hey, you know

your last backup go apply these changes on top of that last backup

Yeah.

Yeah.

offsets.

Right.

Um, and go ahead.

it needs to be a system that allows you to keep those copies as independent,

What do you, what do you mean by that?

um, uh, backup technologies which say, which claim to be incremental forever.

Mm-hmm.

They only keep the latest copy,

Right.

for

I.

you have a full, you apply the changes, you now get a new full.

That old full is gone from the system.

Hmm.

Yeah.

have the most recent copy, right?

And normally for backup, because you want to be able to go back further in

time, you need to make sure you are preserving those previous copies as well.

And so it

Yeah.

require a technology like deduplication, which allows you to efficiently store

multiple copies, snapshots, whatever technology is available on the backend

in order to make that possible.

Yeah, so snapshots and replication.

A great example of forever incremental technology, right?

Each new, um.

Each time you do a snapshot and a replication, you get a new, something

that looks like a full, uh, when really all you only transferred was the, the,

you know, the bites that have changed.

And then, um, I can, I can immediately think of when I was,

when you were talking, I was like, what is he talking about?

And then I realized what you were talking about.

So here's an example of forever incremental that isn't what we're

talking about, and that's replication.

It

Yep.

Right.

Re just regular replication without anything

Yep,

beside it.

You end up with each new, every time you do a a, an if, if you want to call

it that, in an incremental backup, all you're left with is the latest

version and you can't go back in time.

You know, you know what I was actually thinking of when I made those comments?

Oh, what were you,

Oracle incremental merge

oh, okay.

because Oracle incremental merge applies to changes on top of a full, but it

does not preserve the previous versions.

Right.

Yeah, that's a good point.

Um, which, which wouldn't it, that's really just a, a type

of, synthetic full, right?

Yeah, yeah, I didn't even realize that, by the way.

Um, so how do you preserve the history?

That's why

Do you just.

do you need a application that, or a storage system that

allows you to take snapshots or

okay.

copy of the file before it

gotcha.

Why do

where was it?

Curtis?

Why do I still remember these things?

It's why you're here?

So we've defined what a

Mm-hmm.

a little bit about the benefits,

Yeah.

Being able to quickly restore because you have the full copy, uh, being independent,

um, not having to transfer all the data.

Right?

Um, can

And also also cost savings.

Right, right.

Yeah.

you think of any cons?

Well, the, the only thing you know, one of the things I said

in the beginning was that.

In order to do forever, incremental, successfully, you've

got to store the data on disc.

And, and again, not, not to be, you know, pro tape and all that stuff,

but we did an episode relatively recently where we talked about how you

have to, uh, protect the backup disc.

The only con that I can think of is that your entire backup is just sitting

there ready to be attacked, right.

Yeah,

Um.

I have, I have a

Go ahead.

You have, you have a con, okay?

So having worked with vendors, and I'm not saying this has happened, but

it is a potential thing you should think about is when you have forever

incrementals, You're always re.

Copying over the existing backup

Mm-hmm.

bug that somehow makes, that corrupts a part of your backup, right?

You've now impacted how many versions back because you're never getting a

full copy of that data again, right?

It's always

Right,

And so

Yeah.

be one of the cons a

Yeah.

approach.

That was, that was a, um, a critique of ddu, uh, because that

has the same issue, a critique of dedupe when Dedupe first came out.

And all you can say is that you just do your best to ensure

that that's not happening.

And that's why we do fingerprinting and, and, you know, integrity

checks and all that kind of stuff.

your backup.

Yes.

And also test your backups.

Right.

Test your backup, test your backup, test your backups.

Um, but go ahead.

I do have another one too.

If you, unless you

Oh, you do?

Okay.

Go ahead.

The other one I also wanted to mention is, right if Forever incremental backups

are so great, why isn't everyone using it?

Mm-hmm.

one of the challenges is it requires vendor support.

Yeah.

application to support it.

So VMware is supported for virtualization.

companies support it for their own applications, but

not everyone supports it.

Not everyone makes it available.

Yeah.

so you sort of are limited in terms of what applications, what

APIs are available, and can you get the changes that you need in

order to be able to create your forever incremental in the backend.

Exactly, and, and know, I can think of modern database applications

that do support it, right?

Um, I can think, for example, uh, when you back up Salesforce, right?

You're backing up at the object level and it's very easy to.

Uh, and, and let me rephrase that, because Salesforce uses

the term object to mean something very different than what we mean.

Um, because why should we all use the same terms?

of course.

A Salesforce object is like the user's table, right?

Um, so I'm talking about like, I made an update to the, to a, you know, a

field in a record in a table, and, um.

Uh, you can back up that change as right, and, and, and you can do it incrementally

or you can do it independently.

And so it would be very easy to store all of those, those

objects in a, in a way that.

It, you know, would, would foster this idea of a forever incremental.

Um, whereas like Oracle for example, there really isn't a way to do it.

The, at best what we can get out of Oracle is the concept of a, um, synthetic full,

uh, and the synthetic full that they have.

Has the issue that you, that you mentioned before, whereas once you

do the synthetic full, you don't have the previous incrementals.

Um, so yeah, it, that is, that is definitely a, um, it, it

requires your backup vendors to support it, and it requires your

application vendors to support it.

Yeah.

Absolutely.

I can't remember where I was going before you brought up that other, um, I, I,

I will say this, that, you know, the title of this is, you know, how Forever

Incremental Backup changed the world.

It, it, it really has, right?

The idea, when you look at these modern backup products, the

idea of doing a, of a, of a full backup is just, it's just gone.

Right?

Um, all of the downsides to doing a regular full backup.

Are, are gone.

Uh, the, the, the, the, the compute considerations, the

network considerations, the storage considerations, they're all gone without

any negative like impact to the restore.

In fact, it's the opposite.

The restore is now, since we know what a full looks like when we go to do the

restore, that is to going back to Harken, back to the original way we did restores.

We're no longer restoring the same file multiple times during a single restore.

Why would we do that?

We know where all the files are, we know where all the blocks are, whatever

it is that we're, we're restoring.

We know what the latest version of that block here file is, and

we're just gonna restore that one.

I have a story.

Um, how you do forever, incremental Matters.

The original, the OG, forever incremental product is TSM.

mm-hmm.

And when, and the way it was implemented back then was to tape.

And they, tried to bring this concept of forever incremental to the world without

changing the underlying storage technology because back then it was, it was, uh.

tape.

It was, uh, if you, if you brought disc into it, it was, it was crazy talk, right?

So the, so the, so, so they can get a lot of credit for trying to

accomplish this, this, um, this idea.

But I, I still think when we look back to TSM of old, I, I think it was, I

think it was a failed attempt because my earlier statement of the only proper way

to do forever incremental is to do it on disc when they were doing it on tape.

What it meant was that when you, um, when you went to go to do a restore, the, the,

the full was on like 5,700 tapes, right?

sounds worse than just doing your traditional fulls and incrementals.

Yeah, well, it was, and I remember, I, I remember doing a, there was a one

particular restore that I remember doing using TSM in a production environment.

And I, I don't remember the actual numbers, but I'll, I'll just use like,

uh, I'll just use fake numbers just to, to, to to, to describe what it was like.

It was like we were doing like a 10 gigabyte restore and it took like.

Two weeks because of how many hundreds of tapes that it was.

Thousands of tapes.

Um, because of the number of files, the number of tapes that they were

on, and how long it takes to load and unload an individual tape.

Right?

Um, and they had technologies, uh, called reclamation and they, they,

they had a number of technologies that tried to mini minimize this issue.

Um, but

It still existed.

well, yeah, it still is existed.

And also some people, they would run out of time in the day to do reclamation.

Reclamation is where you take a bunch of tapes that, um, like they were, that

most of the files on those, on those tapes were no longer needed because

they, they'd expired and then you take, yeah, I've got a bunch of files that are

10% and then you copy them to one tape

It's

to that's now a hundred percent.

It's like garbage collection, but on tape.

And it was a very, um, labor intensive process and, um, IO intensive process.

And as a result, it took a long time.

And so some people would, would turn it off.

Um, and, uh, I think that's what happened in, in, in this case.

And so there, there were literally just, I don't rem there were so many

tapes that we had to restore and it, you know, it sounds like I'm bagging

on T-S-M-T-S-M is still around.

It's now called, uh, spectrum Protect.

Yeah, but the addition of bringing in disc, I think makes this, this whole

issue kind of, uh, not a big deal,

Yeah.

but, um, but yeah, I still think that you look at all of the modern backup

products that have come out in the last 10 plus years, and they're all like this.

They're all one full, followed by a whole bunch of incrementals, and

the world is a better place for it.

And, um, you young whippersnappers have never had to deal with

tape

incremental backups.

You just don't know

what

what you uh, yeah.

You don't know the pain that you, that Yeah.

That you had.

You know, it's funny, this morning I was hanging out with some people and I

was talking to 'em, and they're, it's this lady who shares my birthday, but

she's about 15 years older than me,

Hmm.

we started talking about all of the technology that we

remember when it came out.

Right.

And the first, the first.

Piece of technology I remember being introduced in my life was the, the VCR

Hmm.

and it, it came, they, I remember them wheeling one into the classroom

Hmm.

in my school.

And it was gigantic.

Right.

It was like, and it was the kind where you put the tape in and push it down, right?

Yeah.

Um, anyway, this is, this is like that, this is like the

way, the way we did backups.

And by the way, you know.

I'm, I'm glad I, I'm glad I remembered this because you asked a question earlier.

Why is it that not everybody does things in this new way?

Yeah.

Because backup is the stickiest application in the data center.

Yep.

So many people are still using the same backup application

that they were 15 years ago.

Some people change their applications like they change

their underwear, but other people.

They just, it, it's a pain.

It's costly, it's risk, uh, it's fraught with risk and, and, and

difficulty and a giant learning curve.

And during that learning curve, your, your data's at risk.

And so most people just, they'd rather stick with the ugly, old backup product.

it.

Because it works.

Yeah.

And, and, and so it, it, it gave birth to incrementally helpful

technologies like deduplication.

Don't change anything.

Just send your backups over here instead of sending your backups over there.

And we'll get rid of the duplicates and we'll make it sort of look like a,

an incremental forever, but just keep doing it the dumb old way because we

don't want you to have to change your backup product in order to get to,

and, and I guess this is, you know.

If, if you're one of those people that's still using the backup, the

same backup product that you had 15 years ago and you're still using it

the same way you were using it 15 years ago, and you're not using incremental

forever as a, a true technology in your world, maybe time to think about it.

but I, I would probably say.

Newer workloads, people are probably using that, right?

Because once again, environments are a mishmash of technologies, right?

Yeah.

people probably do have those old databases, old applications,

which they are using an old infrastructure backup infrastructure

for, and for their virtualization or other newer workloads, they're

probably using something different.

Yeah.

Uh, remember I did that, I did that survey a few, uh, months ago and

I asked them how many backup apps they had, and 30% had 30, I think

more than 30% had three or more backup apps in their environment.

Yep.

And it's because of what you're talking about.

They have, they start using.

Containers and their current backup app doesn't know what to do.

And so they use that for the new technology and yeah.

Yeah, that's very common.

And I guess what I'm saying is maybe it's time to check out the water of

the, of, of truly incremental backups and, and experience all the beauty.

Yeah.

All right.

Well thanks for the chat again.

Anytime Curtis.

Anytime.

All right.

And we wanna thank our listeners.

Yo, you're why we're here, you know, otherwise, just two guys sitting here

talking over a couple of microphones.

Uh, that is a wrap.