Understanding Backup Levels (Backup to Basics Series)
Ok, so maybe not the most interesting topic. ;) But we promise you, this episode has a great story that involves Mr. Backup being kidnapped by a client, basically because he had a backup level issue. Learn about full backups, incrementals, cumulative incrementals, differentials, numbered levels, tower of hanoi backups, and why all this matters. It turns out it matters a lot more these days for structured backups than filesystem backups, which have typically gone to an incremental forever setup.
Mentioned in this episode:
Interview ad
Hi, and welcome to Backup Central's Restored All podcast.
W. Curtis Preston:I'm your host, Debbie Curtis Preston, a k a, Mr.
W. Curtis Preston:Backup, and I with me, a guy that I'm hoping knows a little
W. Curtis Preston:something about plumbing.
W. Curtis Preston:Prasanna.
W. Curtis Preston:Malaiyandi, how's it going?
Prasanna Malaiyandi:I am good, Curtis.
W. Curtis Preston:how's that?
W. Curtis Preston:YouTube?
W. Curtis Preston:How's that?
W. Curtis Preston:YouTube, the, the YouTube knowledge that you have, does it cover plumbing
Prasanna Malaiyandi:kind of, sort of.
Prasanna Malaiyandi:Unfortunately it's not something I'm as familiar with, but depends
Prasanna Malaiyandi:on what you're looking to do.
W. Curtis Preston:Yeah.
W. Curtis Preston:I gotta replace tonight.
W. Curtis Preston:I gotta replace a couple of valves.
W. Curtis Preston:Um, Shut off valves.
W. Curtis Preston:Um, one of them, I, one of them is on the, the, it's the one that goes
W. Curtis Preston:behind my fridge and in order to like, it's like I, I can't easily,
W. Curtis Preston:it's a, it's a, what do you call it?
W. Curtis Preston:It's the, the threaded, no, it's not a compression fit.
W. Curtis Preston:It's the, it's a threaded pipe.
W. Curtis Preston:Like, uh, I really gotta get in there and get a hold of the, the upper part
W. Curtis Preston:of the pipe, uh, in order to hold that pipe still while I'm undoing it.
W. Curtis Preston:Cuz it, my first attempts at it did not, I don't want to be
W. Curtis Preston:cracking off a pipe up in the wall.
Prasanna Malaiyandi:Yeah, I was just gonna say that's, yeah,
Prasanna Malaiyandi:the last thing you want to do.
Prasanna Malaiyandi:I was thinking for the shutoff though, I'm assuming you took the handle off
W. Curtis Preston:The handle
Prasanna Malaiyandi:
Speaker:to give you more space
W. Curtis Preston:it?
W. Curtis Preston:No, there's no handle.
W. Curtis Preston:It's just, I mean, the handle's like integrated.
W. Curtis Preston:It's really small.
W. Curtis Preston:It's not a space problem.
W. Curtis Preston:It's just, uh, it's um, you know,
Prasanna Malaiyandi:
Speaker:Or they also say heat.
W. Curtis Preston:Yeah, give it
Prasanna Malaiyandi:So when I saw my, oh, so when we had our water heater
Prasanna Malaiyandi:replaced, I was watching cuz I'm always curious to see what people do.
Prasanna Malaiyandi:And our pipes were really, really bad at old.
Prasanna Malaiyandi:because the last time our water heater was replaced was like 15 years ago.
Prasanna Malaiyandi:And so the guy's like, you just heat it up really, really hot and then
Prasanna Malaiyandi:you tighten it first to break the stuff off and then you loosen it.
W. Curtis Preston:Oh, interesting.
W. Curtis Preston:Yeah.
W. Curtis Preston:Maybe I'll see.
W. Curtis Preston:I'll take a look at that.
Prasanna Malaiyandi:Yeah.
W. Curtis Preston:Uh,
Prasanna Malaiyandi:took like in a settling torch or a butane torch
Prasanna Malaiyandi:and then heated up nice and hot, and then he tightened it just a little.
W. Curtis Preston:what, I'm not gonna be able to do that.
W. Curtis Preston:You know why?
W. Curtis Preston:There's this
Prasanna Malaiyandi:Oh, the drywall.
Prasanna Malaiyandi:Oh,
W. Curtis Preston:
Speaker:Plastic, worse than dry.
Prasanna Malaiyandi:yeah.
W. Curtis Preston:Um, so hey, we're gonna talk about, uh, we're, you
W. Curtis Preston:know, in our continuing, um, backup to basic series, you know, we talked
W. Curtis Preston:about, like recently we talked about a difference between backup and archive.
W. Curtis Preston:, in a previous episode we talked about protecting your backup data.
W. Curtis Preston:This one, um, you know, this one is, um, it's about backup levels, right?
W. Curtis Preston:And, you know, it's, honestly, I'll just say this isn't the most
W. Curtis Preston:exciting topic, Um, and maybe, maybe, we'll, maybe we'll make it a short
W. Curtis Preston:episode if it's not interesting.
W. Curtis Preston:You know, we're not gonna make, we're not gonna make a long episode cuz
W. Curtis Preston:this isn't gonna be one that's gonna be, uh, littered with stuff about
W. Curtis Preston:cyber attacks and things like that.
W. Curtis Preston:Backup levels
Prasanna Malaiyandi:
Speaker:really straightforward.
W. Curtis Preston:really straightforward.
W. Curtis Preston:Yeah.
W. Curtis Preston:Uh, I do want to help for people that sometimes struggle with
W. Curtis Preston:the different types of backups.
W. Curtis Preston:Um, I want them to help them understand.
W. Curtis Preston:I, I'd say that, you know, if you just go and I'm, uh, throughout
W. Curtis Preston:our usual disclaimer Prasanna, and I work for different companies.
W. Curtis Preston:I work for Druva, he works for Zoom, and the, this is an independent podcast.
W. Curtis Preston:It's not theirs.
W. Curtis Preston:And it's, uh, the opinions that you hear are ours.
W. Curtis Preston:Uh, be sure to rate us, uh, go to your favorite, um, pod catcher and
W. Curtis Preston:uh, scroll down to wherever they got the stars and give us all the stars.
W. Curtis Preston:And the best thing is if you give us a comment, those are always nice,
W. Curtis Preston:especially on Apple Podcasts, which is.
W. Curtis Preston:the primary way that people listen to us over here.
W. Curtis Preston:And, um, also if you wanna join the conversation, I am w Curtis Preston gmail
W. Curtis Preston:and also WC Preston for now on Twitter.
W. Curtis Preston:I dunno what's
Prasanna Malaiyandi:do you have a master on?
W. Curtis Preston:Twitter man, I don't have a Macedon account.
W. Curtis Preston:Uh, I am interested in that.
W. Curtis Preston:We'll see if, I don't know if the average person can, can
Prasanna Malaiyandi:I don't think so.
Prasanna Malaiyandi:From what everything I've heard.
Prasanna Malaiyandi:Yeah.
W. Curtis Preston:It took me a minute, right?
W. Curtis Preston:Twitter, you know, you sign up for account, you're done right here.
W. Curtis Preston:It's like, well, what community do you want it attach to?
W. Curtis Preston:I'm like, oh, it's so complicated.
W. Curtis Preston:I don't wanna know what community, you know?
W. Curtis Preston:Um, we'll see.
W. Curtis Preston:But, uh, anyway, at WC Preston on Twitter, maybe I should sign it
W. Curtis Preston:for my, at WC Preston on Mastodon.
W. Curtis Preston:Make sure I get that.
Prasanna Malaiyandi:I think you were throwing out the disclaimer though,
Prasanna Malaiyandi:because for this chapter, if people want to actually read the chapter and
Prasanna Malaiyandi:understand all these terms, right.
Prasanna Malaiyandi:Druva is offering a free ebook version.
W. Curtis Preston:I, you know, that wasn't why I was saying
W. Curtis Preston:it, but, we'll, we'll do that.
W. Curtis Preston:So we are discussing my book, modern Data Protection.
W. Curtis Preston:There's a picture of it with the little armadillo for those of you watching on
W. Curtis Preston:the video version on backup central.com.
W. Curtis Preston:And uh, you can get a free ebook copy of it by going to druva.com/ebook.
W. Curtis Preston:That's d r uva.com/ebook.
W. Curtis Preston:And.
W. Curtis Preston:while supplies last, or while the contract lasts.
W. Curtis Preston:So, you know, this, this, this episode will live for a long time.
W. Curtis Preston:If, if you go there and it's not there, it's not my fault.
W. Curtis Preston:So, uh, first off, you know, again, this is pretty basic, but.
W. Curtis Preston:You know, you should know what a full backup is.
W. Curtis Preston:Obviously.
W. Curtis Preston:That's it.
W. Curtis Preston:It backs up everything, right?
W. Curtis Preston:Typically.
W. Curtis Preston:Well, not typically.
W. Curtis Preston:You always have to do a full backup before you can do any,
W. Curtis Preston:any kind of incremental backup.
W. Curtis Preston:That's the, that's the next type.
W. Curtis Preston:The, there are amazingly, a handful of different types of
W. Curtis Preston:the traditional incremental
Prasanna Malaiyandi:but before you get to incremental, so.
W. Curtis Preston:Yeah.
Prasanna Malaiyandi:Even for fulls, are there different types of fulls?
W. Curtis Preston:Well, there's not different levels for fulls, but there
W. Curtis Preston:are different ways that fulls get done.
W. Curtis Preston:Right.
W. Curtis Preston:Like you, like we could be doing a file system backup or we could be doing a block
W. Curtis Preston:level backup, but either way you're still backing up all the things, all the bites.
W. Curtis Preston:Yeah.
W. Curtis Preston:Um, the, uh, and it's, it's the full is the thing that puts the most amount of,
W. Curtis Preston:of stress on the thing being backed up.
W. Curtis Preston:Right.
W. Curtis Preston:The, this is why we try to limit them whenever.
W. Curtis Preston:, and this is why, this is why backups broke when VMware took off because everybody was
W. Curtis Preston:just doing full and incremental backups and they weren't coordinating them.
W. Curtis Preston:And you know, it was just you,
Prasanna Malaiyandi:lot of resources being used.
W. Curtis Preston:Yeah, exactly right.
W. Curtis Preston:And um, if you do, if you do a nightly full backup, or let's say the weekend
W. Curtis Preston:full backup, and you do 'em all on Friday, And you've got 20 different
W. Curtis Preston:physical servers, no big deal.
W. Curtis Preston:But if you do 20 full backups on 20 VMs that are all on one
W. Curtis Preston:physical server, this is a problem.
W. Curtis Preston:Right.
W. Curtis Preston:Um, so I've got a couple different types here.
W. Curtis Preston:There's the, the typical incremental backup, which is, it basically,
W. Curtis Preston:it backs up everything since the last backup of whatever kind.
W. Curtis Preston:And typically what what you do, and this is with backups that have
W. Curtis Preston:levels, you do a full backup, then you do a series of incremental
W. Curtis Preston:backups, and then you do another full.
W. Curtis Preston:right?
W. Curtis Preston:That's the, that's the sort of the typical way.
W. Curtis Preston:There is something called a cumulative incremental backup.
W. Curtis Preston:Um, I prefer that to the term differential backup, and I can
W. Curtis Preston:talk about that in a minute.
W. Curtis Preston:But a cumulative incremental backup is essentially just that it backs up
W. Curtis Preston:everything that has changed since the last full different, um, different product.
W. Curtis Preston:Uh, do different things.
W. Curtis Preston:Uh, the differential backup is a term that you see a lot in windows and, um,
W. Curtis Preston:I, I, and I, I don't use it because I find the term, um, the, the, because
W. Curtis Preston:depending on where you ask the term differential means different things.
W. Curtis Preston:And so I, I don't like it.
W. Curtis Preston:Right?
W. Curtis Preston:So I like the term cumulative incremental.
W. Curtis Preston:Even if your backup product doesn't use it, just learn
W. Curtis Preston:what, whatever they call it.
W. Curtis Preston:What?
Prasanna Malaiyandi:wait.
Prasanna Malaiyandi:So as far as I understand it, , you have the thing that's like full then.
Prasanna Malaiyandi:So say I do a full on the weekend, right on Monday, I do, uh, incremental,
W. Curtis Preston:Uhhuh.
Prasanna Malaiyandi:right?
Prasanna Malaiyandi:Which will be the differences for whatever happened between Sunday and Monday,
W. Curtis Preston:Yeah.
Prasanna Malaiyandi:right?
Prasanna Malaiyandi:Tuesday I do another incremental
W. Curtis Preston:Yeah,
Prasanna Malaiyandi:right?
Prasanna Malaiyandi:That'll be the differences between Monday and Tuesday.
W. Curtis Preston:yeah,
Prasanna Malaiyandi:And then Wednesday I do another incremental.
Prasanna Malaiyandi:That's the difference between Tuesday and Wednesday.
Prasanna Malaiyandi:Now, for different, uh, for cumulative incrementals, it's, I
Prasanna Malaiyandi:did my still, I'll do my Sunday, full Monday I'll do my cumulative.
Prasanna Malaiyandi:Incremental, or yeah, incremental, which will just be
Prasanna Malaiyandi:Sunday to Monday, differences.
Prasanna Malaiyandi:Tuesday, when I do the cumulative incremental, it'll
Prasanna Malaiyandi:be from Sunday to Tuesday
W. Curtis Preston:Correct.
Prasanna Malaiyandi:and Wednesday when I do a cumulative incremental, it'll
Prasanna Malaiyandi:be the differences between Sunday and.
W. Curtis Preston:Correct.
W. Curtis Preston:And so what, what we used to do back in the day was I would do a monthly
W. Curtis Preston:full weekly cumulative incrementals and daily incrementals, right?
W. Curtis Preston:So that way, even if I'm 28 days into the month, I would only need
W. Curtis Preston:three backups to get the job done.
W. Curtis Preston:. Right.
W. Curtis Preston:Um, I, I don't use the term differential.
W. Curtis Preston:Differential is what a lot of places call accumulative a criminal backup.
W. Curtis Preston:But again, different products use it to mean different things, so I don't use it.
W. Curtis Preston:Um, but interestingly enough, what we haven't talked about is backup
W. Curtis Preston:levels, which is the term that, again, you don't see too much these days.
W. Curtis Preston:Have you, have you, are you seeing it?
Prasanna Malaiyandi:Very, very rarely do Does anyone talk.
Prasanna Malaiyandi:The only place I've seen levels really is when you're talking
Prasanna Malaiyandi:like database backups, but most.
W. Curtis Preston:what?
W. Curtis Preston:That is true.
W. Curtis Preston:Right?
Prasanna Malaiyandi:Yeah.
Prasanna Malaiyandi:But most backups software itself no longer talks about the levels.
Prasanna Malaiyandi:Right.
Prasanna Malaiyandi:It's either fulls or incrementals.
W. Curtis Preston:I think it was too complicated.
W. Curtis Preston:Yeah.
W. Curtis Preston:The basically if to, to do.
W. Curtis Preston:To do a full on a weekend and then daily regular incrementals, you would
W. Curtis Preston:do a 0, 1, 2, 3, 4, 5, 6, 7, right?
W. Curtis Preston:Well, seven.
W. Curtis Preston:And then maybe, um, so then, and again, this is, this is impossible to do without
W. Curtis Preston:a whiteboard, but if you did, um, a zero followed by all the way up to the five, if
W. Curtis Preston:you did a one again following that five.
W. Curtis Preston:you would get all the changes since to zero.
Prasanna Malaiyandi:Hmm.
W. Curtis Preston:So it was accumulative.
W. Curtis Preston:Incremental, right.
W. Curtis Preston:But what some people would do is they would do this, um, this,
W. Curtis Preston:there, there's, there was this thing called towers of Hanoi, which,
W. Curtis Preston:um, it, it comes from the game.
Prasanna Malaiyandi:like
W. Curtis Preston:Yes.
W. Curtis Preston:Yes.
W. Curtis Preston:It's like the thing with the game.
W. Curtis Preston:Yes,
Prasanna Malaiyandi:three.
Prasanna Malaiyandi:Yeah.
W. Curtis Preston:Comes from that game.
W. Curtis Preston:Um, and so the, and again, this is really, don't try to understand this, I'm just
W. Curtis Preston:putting it out there just for, this is for the true backup anoracks, so who's gonna
Prasanna Malaiyandi:Daniel Rosen Hill.
Prasanna Malaiyandi:We're looking.
Prasanna Malaiyandi:Daniel Rose Hill.
Prasanna Malaiyandi:We're looking at you.
W. Curtis Preston:Exactly.
W. Curtis Preston:So, um, let's see.
W. Curtis Preston:So what I've got here is, , um, a towers of Illinois Schedule 0 3 2 5 4 7 6.
W. Curtis Preston:And what it does is it, is it like every file that's changed
W. Curtis Preston:ends up on two different, um,
Prasanna Malaiyandi:Mm,
W. Curtis Preston:two different, um,
Prasanna Malaiyandi:days.
W. Curtis Preston:backups.
Prasanna Malaiyandi:Yeah,
W. Curtis Preston:Um, and um,
Prasanna Malaiyandi:you don't run the risk of if a backup is bad,
Prasanna Malaiyandi:you've lost the file, which could happen if you just had a full and you
Prasanna Malaiyandi:never did another incremental again.
W. Curtis Preston:Yeah.
Prasanna Malaiyandi:Or sorry you never did another full again, I, I
Prasanna Malaiyandi:think one thing we forgot to mention Curtis though, is the benefits of doing
Prasanna Malaiyandi:a a, we might have talked about it implicitly, but the benefits of doing a
Prasanna Malaiyandi:cumulative incremental versus a normal incremental when it comes to restores.
W. Curtis Preston:Well, you know, here's the thing.
W. Curtis Preston:The benefits were much bigger when we were using tape.
Prasanna Malaiyandi:Hmm.
W. Curtis Preston:I, I think that as, and, and what we're gonna discover is
W. Curtis Preston:that this is why a lot of this has, I think, gone by the wayside is because,
W. Curtis Preston:because when we were using tape, you'd gr you'd grab the full tape, you would grab
W. Curtis Preston:the cumulative incremental tape, and you would grab any incremental since then.
W. Curtis Preston:So you've got maximum like eight tapes if you did a full,
W. Curtis Preston:followed by 30 incremental.
W. Curtis Preston:, you would, um, you would need 28 tapes, you know, at the end of the month.
W. Curtis Preston:So this way you only need like a handful of tapes.
W. Curtis Preston:And what it allowed you to do was it allowed you to do
W. Curtis Preston:less frequent full backups.
W. Curtis Preston:Because it was a, it was a, an extra cost.
W. Curtis Preston:You're like, well, by doing a weekly cumulative incremental,
W. Curtis Preston:I could do a monthly full backup instead of a weekly full backup.
W. Curtis Preston:Thus really reducing how big the backup system needed to be,
W. Curtis Preston:how many tapes I needed to use.
W. Curtis Preston:Um, And, and by doing that cumulative incremental, it made
W. Curtis Preston:the restore much, much faster.
W. Curtis Preston:I would argue if you have a decent backup system today and you've got a monthly
W. Curtis Preston:full and 30 days of incremental, or even 60 days of incremental, the difference
W. Curtis Preston:between restore times, um, depending on how you store the backup data, shouldn't
W. Curtis Preston:be as significant as they were back in the day when we were loading tapes, cuz
W. Curtis Preston:every tape loaded was like two minutes
Prasanna Malaiyandi:Yeah.
Prasanna Malaiyandi:But,
W. Curtis Preston:to the first.
Prasanna Malaiyandi:but I do wonder though if when you are doing
Prasanna Malaiyandi:those restores, depending on what you're restoring, like I could
Prasanna Malaiyandi:see a file system maybe not being a big deal for doing the 60 days.
Prasanna Malaiyandi:I'm just wondering like database backups, right.
Prasanna Malaiyandi:To do those incremental restores.
W. Curtis Preston:Yeah.
W. Curtis Preston:Um, I think that's a val, I think that's a valid.
W. Curtis Preston:Uh, concern, right?
W. Curtis Preston:This is why to this day we still do more frequent
Prasanna Malaiyandi:Fulls that need it.
W. Curtis Preston:on, on database backups.
W. Curtis Preston:Right.
W. Curtis Preston:The, that's, that's a good point.
W. Curtis Preston:Um, and since a significant portion of the world is on, is,
W. Curtis Preston:is structured data then, uh, yeah, that's a really good point, right?
W. Curtis Preston:So, um, the.
W. Curtis Preston:So now we've got a just some different types of, so we're done with the levels.
W. Curtis Preston:Now we're talking about different types of backups.
W. Curtis Preston:What about a block level incremental backup?
W. Curtis Preston:What is that?
Prasanna Malaiyandi:
Speaker:Block level, incremental.
Prasanna Malaiyandi:
Speaker:Are you referring to like database slash v.
Prasanna Malaiyandi:
Speaker:Where you're just copying the differences in whatever the application
Prasanna Malaiyandi:
Speaker:block size is, rather than block backing up the entire image or
Prasanna Malaiyandi:
Speaker:the entire file potentially again.
W. Curtis Preston:Yeah, so basically this is the difference between, and
W. Curtis Preston:again, this really only applies if you're looking at things at the block level,
W. Curtis Preston:which you tend to do at the database, but there are some file systems that
W. Curtis Preston:are so dense that backing it up at the block level would be faster than
W. Curtis Preston:backing it up at the, at the file level.
Prasanna Malaiyandi:Yeah.
W. Curtis Preston:And so you can do a block level incremental, basically.
W. Curtis Preston:The point is, when I'm doing an incremental.
W. Curtis Preston:A file system in increment, a file system incremental.
W. Curtis Preston:I'm backing up the files that have changed since the full, if I'm doing
W. Curtis Preston:a block level incremental, I'm backing up the blocks that have changed since
W. Curtis Preston:the full or since the previous bite.
W. Curtis Preston:Right.
W. Curtis Preston:And then here's something that hopefully you'll know something about, and it's
W. Curtis Preston:called Source side deduplication.
W. Curtis Preston:This is another way to do incremental.
W. Curtis Preston:What can you tell us about.
Prasanna Malaiyandi:Yeah.
Prasanna Malaiyandi:Source side de-duplication.
Prasanna Malaiyandi:So when de-duplication appliances came out, um, what you ended up happening
Prasanna Malaiyandi:is whenever you would do a full right, you were talking right Curtis, about
Prasanna Malaiyandi:doing weekly fulls, daily incrementals.
Prasanna Malaiyandi:Every time you did that full, probably 98% of the data was the same.
Prasanna Malaiyandi:So on the source you would read it, you would send it over the network, it would
Prasanna Malaiyandi:go to your de-duplicated appliance and it would throw away 98% of the data.
Prasanna Malaiyandi:And so source side de-duplication is, Instead of sending all of
Prasanna Malaiyandi:that data to be thrown away.
Prasanna Malaiyandi:On the appliance, you basically split the deduplication algorithm
Prasanna Malaiyandi:such that you're running some of the processing on the source.
Prasanna Malaiyandi:So you figure out what is duplicate, usually at a smaller block level
Prasanna Malaiyandi:or some other, uh, granular size and only send the unique blocks
Prasanna Malaiyandi:over to the target appliance.
Prasanna Malaiyandi:And the rest of the data is sort of reconstructed, if you will.
Prasanna Malaiyandi:There are different methods you could look at.
Prasanna Malaiyandi:It's sort of rebuilt because you still need to account for those pieces
Prasanna Malaiyandi:of data, even though you haven't technically sent the data over the wire.
W. Curtis Preston:Yeah, so if we think about it, like the only reason
W. Curtis Preston:ever to do an incremental backup is to save something, save bandwidth,
W. Curtis Preston:save CPU time, save storage.
W. Curtis Preston:And what, you know, what you're talking about is even when you had
W. Curtis Preston:target site dation with like data domain and the like, you, you.
W. Curtis Preston:, you save storage.
W. Curtis Preston:You, you can do full backup every day and you save storage.
W. Curtis Preston:But you were still using up all that CPU in that network by doing
W. Curtis Preston:Source I duplication and making that decision before you send it.
W. Curtis Preston:You save CPU time, you save bandwidth, you save storage, you save all the things.
W. Curtis Preston:Right?
W. Curtis Preston:Um, source ID is not as prevalent as targets.
W. Curtis Preston:I dedup.
W. Curtis Preston:There's, there's only a handful of companies that,
W. Curtis Preston:that do it, but uh, they are,
Prasanna Malaiyandi:Yep.
W. Curtis Preston:Um,
Prasanna Malaiyandi:it does require more integration into the backup product on the
Prasanna Malaiyandi:source in order to take advantage of it.
W. Curtis Preston:exactly right.
W. Curtis Preston:Uh, for the record, Druva is a source site duplication product.
W. Curtis Preston:So the next thing we talk about here is called synthetic full backups,
W. Curtis Preston:uh, which is and interesting.
W. Curtis Preston:I remember when this concept first came out, again, we're trying to
W. Curtis Preston:minimize the impact of the backup.
W. Curtis Preston:on the system, we're backing up.
W. Curtis Preston:So the idea was we already have all of the bites necessary to create a full backup.
W. Curtis Preston:Why do we need to go get, you know, we're doing a, we're doing an increment.
W. Curtis Preston:We're doing the next backup and 99% of what's on the system we
W. Curtis Preston:already have on disc somewhere.
W. Curtis Preston:This is really a disc thing.
W. Curtis Preston:Uh, we did it on tape too, but it was a lot more work.
Prasanna Malaiyandi:Yeah.
W. Curtis Preston:So we'll go get the 1% that has changed.
W. Curtis Preston:and then we'll create a full backup by copying the stuff we already
W. Curtis Preston:have and the stuff that we just got.
W. Curtis Preston:So we create a synthetic full.
W. Curtis Preston:Um, have you, have you run into those out in the wild?
Prasanna Malaiyandi:no, it's very common, uh, especially when
Prasanna Malaiyandi:you think about VMware images.
W. Curtis Preston:Mm-hmm.
Prasanna Malaiyandi:depending on how it's being backed up, a lot of folks
Prasanna Malaiyandi:would back up a VM as incrementals and then, because like you said, most of
Prasanna Malaiyandi:the data is the same, they would sort of synthesize a full image on the target
Prasanna Malaiyandi:side in order to have that full copy.
Prasanna Malaiyandi:Therefore, when you need to restore, you have that.
Prasanna Malaiyandi:What image you could pull from or would a lot of people start doing is you take
Prasanna Malaiyandi:that one image and you can now mount.
Prasanna Malaiyandi:, right.
Prasanna Malaiyandi:And access it directly.
Prasanna Malaiyandi:Instant access is what simple some people call it.
Prasanna Malaiyandi:So you can spin up the VM from your target system and then
Prasanna Malaiyandi:vMotion it over or whatever else.
Prasanna Malaiyandi:So you have shorter RTOs than trying to first restore the
Prasanna Malaiyandi:data before you spin it up.
W. Curtis Preston:and there, and there's a couple different ways that
W. Curtis Preston:you can create a synthetic full.
W. Curtis Preston:You can either do it by copying, which is what I was talking about before.
W. Curtis Preston:There's also this concept of virtual synthetic full,
W. Curtis Preston:where you can just create one.
W. Curtis Preston:, you know, magically, right?
W. Curtis Preston:Like the, I know that again, data domain I know supported
W. Curtis Preston:that on the back end, right?
W. Curtis Preston:That if the backup product could say, you have all the bites you need to make a
W. Curtis Preston:full, why don't you just put 'em together and, you know, everything's beautiful,
Prasanna Malaiyandi:You know what else does it too,
W. Curtis Preston:what's that?
Prasanna Malaiyandi:Oracle?
Prasanna Malaiyandi:When you do incremental merch,
W. Curtis Preston:That's right.
W. Curtis Preston:Incremental merge creates.
W. Curtis Preston:Exactly, exactly.
W. Curtis Preston:Um, but here's the thing.
W. Curtis Preston:Wherever we can get rid of full backups, I think we should get rid
W. Curtis Preston:of full backups, Right, because they're just the dumbest thing.
W. Curtis Preston:Right?
W. Curtis Preston:They're just, they're just, they're, we did 'em because that's the way we did 'em.
W. Curtis Preston:And we still do 'em because that's the way we used to do 'em.
W. Curtis Preston:Uh, so I'm a fan of, go ahead.
Prasanna Malaiyandi:I, so I, I wanna be careful, right?
Prasanna Malaiyandi:I think we should.
Prasanna Malaiyandi:Get rid of the concept of backing up full copies of data every
Prasanna Malaiyandi:single time or on some increment.
Prasanna Malaiyandi:I a hundred percent agree with that statement.
W. Curtis Preston:Isn't that what I just said?
Prasanna Malaiyandi:No, no, no, no, no.
Prasanna Malaiyandi:But, but I don't agree with, but there's still a necessity to have the
Prasanna Malaiyandi:equivalent of periodic fulls, right?
Prasanna Malaiyandi:Something that represents a full backup.
Prasanna Malaiyandi:Now it could be stored on de-duplicated systems to save storage space, cuz
Prasanna Malaiyandi:most of it's the same, but I'm just saying you don't wanna go the route of,
Prasanna Malaiyandi:it's just a full once a year and just incrementals physically stored on disk.
Prasanna Malaiyandi:So using virtual synthetics or some other mechanism to end up with full copies
Prasanna Malaiyandi:periodically, I think is beneficial.
W. Curtis Preston:So I would argue that that's what Incremental Forever
W. Curtis Preston:is a true incremental forever system.
W. Curtis Preston:Right.
W. Curtis Preston:That basically you, which is what, what I was about to talk
Prasanna Malaiyandi:Okay.
Prasanna Malaiyandi:Okay.
W. Curtis Preston:Um, and that, you know, the idea is store each backup
W. Curtis Preston:in such a way so that a, any restore from any backup, you just, the, it's
W. Curtis Preston:stored in such a way that it looks like a full right and a restore from it
W. Curtis Preston:behaves like a full, I, I don't think we need to keep making fulls or make or
W. Curtis Preston:synthetically creating an occasional full.
W. Curtis Preston:I think that if you design it from the beginning, so that you do
W. Curtis Preston:incremental forever, and then you store that data in such a way that
W. Curtis Preston:every backup is essentially a full
Prasanna Malaiyandi:Yep.
W. Curtis Preston:from, from a way it behaved the way it behaves
W. Curtis Preston:during a restore, then I think that's as good as it's gonna get.
W. Curtis Preston:Right.
Prasanna Malaiyandi:I, I think that works in the cases of file
Prasanna Malaiyandi:systems and virtual machines.
Prasanna Malaiyandi:I do think in the case of databases, it is a little database is always that oddball.
W. Curtis Preston:I had the caveat back earlier where I was
W. Curtis Preston:like, wherever we can, right.
W. Curtis Preston:Um,
Prasanna Malaiyandi:Databases have so many nuances with fulls
Prasanna Malaiyandi:and incrementals and log archived, redo logs, and all the rest that I
Prasanna Malaiyandi:think, yeah, for everything else, a hundred percent agree with you, Curtis
W. Curtis Preston:Yeah.
W. Curtis Preston:Um, and there are, there are a handful of products that back up this way
W. Curtis Preston:that basically, said full backups is so from the tape land, right?
W. Curtis Preston:Except for database backups.
W. Curtis Preston:Right.
W. Curtis Preston:Um, generally speaking, there're gonna be products that have come
W. Curtis Preston:out in the last 10 ish years.
W. Curtis Preston:Right.
W. Curtis Preston:Um, products that have been around for 20 years.
W. Curtis Preston:They're gonna have spent time in the tape land and they're gonna have
W. Curtis Preston:parts of their architecture that are left over from the tape plan.
Prasanna Malaiyandi:I also do wonder if some of that is just the
Prasanna Malaiyandi:people who are banishing the systems.
Prasanna Malaiyandi:It's something that they are used to, and so change is hard, you know?
W. Curtis Preston:is hard.
W. Curtis Preston:Inertia is something.
W. Curtis Preston:Yeah, exactly.
W. Curtis Preston:And I would say like if you, if you're doing source side dup with incremental
W. Curtis Preston:forever, I think from an efficiency perspective, assuming that the way you
W. Curtis Preston:store it, everything behaves like a full, an efficiency perspective, I don't see it.
W. Curtis Preston:I don't know how it could get any more efficient in that.
W. Curtis Preston:Right.
W. Curtis Preston:You could maybe argue the block level incremental forever where you're.
W. Curtis Preston:Right.
W. Curtis Preston:So there, there are like c d P and near CDP systems that's continuous data
W. Curtis Preston:protection where they're doing block level changes and they're replicating that.
W. Curtis Preston:That's also quite efficient.
W. Curtis Preston:It's not source I, it's still, it's still, so I would say,
W. Curtis Preston:so I'll restate my statement.
W. Curtis Preston:Either source iddu or block level of replication and then incremental
W. Curtis Preston:forever stored in such a way so that everything behaves like a full.
Prasanna Malaiyandi:Yeah.
W. Curtis Preston:you know, that includes a handful of
W. Curtis Preston:products , one of which is Druva.
W. Curtis Preston:But, um, you know, there, there are other products that behave like that.
W. Curtis Preston:Like, you know, your old employer, NetApp, right?
W. Curtis Preston:That was a block level incremental replication.
W. Curtis Preston:Um, yeah.
W. Curtis Preston:Um, so I'm gonna throw out one, um, this is gonna be one of our shorter episodes.
W. Curtis Preston:I think.
W. Curtis Preston:We'll see.
W. Curtis Preston:We're almost done.
W. Curtis Preston:I'm gonna throw out one, like, uh, what's the cranky old man?
W. Curtis Preston:Comment
Prasanna Malaiyandi:get off my lawn.
W. Curtis Preston:this is the archive bit and Windows.
W. Curtis Preston:I hate the archive bit and windows.
W. Curtis Preston:Um, so what is the archive bit?
W. Curtis Preston:It's a flag on every file that when your backup software backs up
W. Curtis Preston:that file, it can unset that flag.
W. Curtis Preston:If the file is new or it changes, the archive bit is set.
W. Curtis Preston:And then when the flag, uh, when the back, the, uh, backup software backs
W. Curtis Preston:it up, the archive flag is unset.
Prasanna Malaiyandi:Why.
W. Curtis Preston:This is the old way and it's still the current way.
W. Curtis Preston:Uh, not looking.
W. Curtis Preston:This is if you're doing a file system level backup in Windows.
W. Curtis Preston:Um, I think it's so dumb for multiple reasons.
W. Curtis Preston:The first of which it's name,
Prasanna Malaiyandi:Archive.
W. Curtis Preston:it should be the backup bit.
Prasanna Malaiyandi:Yeah.
W. Curtis Preston:They called it the archive bit anyway, but that's not
W. Curtis Preston:really my problem is you can't, if anyone comes along with like, any, cuz there
W. Curtis Preston:are a number of like third party tools that can back up your Windows system.
W. Curtis Preston:If you run a third party tool to create your own backup, like let's
W. Curtis Preston:say you, you know, a lot, some people don't trust the backup system, right?
W. Curtis Preston:So if you, if you being an admin go and decide to make your own backup
W. Curtis Preston:and you use something that uses the archive bit, it will clear the
W. Curtis Preston:archive bit on all the files that have changed since the last full.
W. Curtis Preston:and then when you go, when the real backup software comes in, it actually
Prasanna Malaiyandi:not gonna find,
W. Curtis Preston:until the next full backup.
Prasanna Malaiyandi:
Speaker:and I bet people don't
W. Curtis Preston:problem with it.
Prasanna Malaiyandi:Yeah.
W. Curtis Preston:Yeah.
W. Curtis Preston:And of course people don't notice it.
W. Curtis Preston:Um, and so I'm gonna go back to the story, um, that I talked about.
W. Curtis Preston:I don't know what the, the one about me being kidnapped, was
W. Curtis Preston:that in this episode or was
W. Curtis Preston:That
Prasanna Malaiyandi:was a previous.
W. Curtis Preston:Okay, well I hinted at this story from um, this other,
W. Curtis Preston:from the other episode, but here's the full story cuz we got time And you know
W. Curtis Preston:what, if you don't want another old backup story then you know, thanks for
W. Curtis Preston:joining and uh, see you next episode.
W. Curtis Preston:But this is a good story and if figures in a good buddy of mine, Rob Worman.
W. Curtis Preston:I know Rob listens to the podcast.
W. Curtis Preston:, he figures into this story.
W. Curtis Preston:So I was at this large, you know, national entertainment company and I was helping
W. Curtis Preston:them to redesign their backup system.
W. Curtis Preston:We were doing a massive redesign.
W. Curtis Preston:And one of the, and one of the things, you know, like we pushed them from, you
W. Curtis Preston:know, weekly folds to monthly folds.
W. Curtis Preston:We, we changed all their multiplexing setting.
W. Curtis Preston:We changed what, uh, we changed how.
W. Curtis Preston:, the schedules were, I mean, we changed everything and we were doing this
W. Curtis Preston:to, to do massive efficiency change.
W. Curtis Preston:They were running like, it was like 18 tape drives simultaneously,
W. Curtis Preston:and I told 'em that that was their pro, they were gonna buy, uh, two
Prasanna Malaiyandi:Oh, this is a sh when you're, you were, you're talking about the
Prasanna Malaiyandi:shoe shining problem and they were trying to tape goes too fast for what they need.
W. Curtis Preston:Yeah.
W. Curtis Preston:And so they were gonna buy two new tape drives and that was gonna fix the problem.
W. Curtis Preston:And I'm like, no, it's not.
W. Curtis Preston:It's actually gonna make it worse.
W. Curtis Preston:And I explained to 'em and I'm like, listen, give me a few weeks and I'll
W. Curtis Preston:redesign your backups to submit.
W. Curtis Preston:It'll cost you less than what those tape drives were gonna cost.
W. Curtis Preston:Cuz they were expensive.
W. Curtis Preston:These were the storage tech tape drives, like the 90, what?
W. Curtis Preston:I can't remember the tape name.
W. Curtis Preston:Tape names.
W. Curtis Preston:But anyway, um, so we did this and.
W. Curtis Preston:, um, and things went really well, but so I knew about the archive bit.
W. Curtis Preston:This is all about the archive bit.
W. Curtis Preston:I knew about the archive bit and I knew that if I had, I basically
W. Curtis Preston:left their parallel backup system.
W. Curtis Preston:I left their parallel policy.
Prasanna Malaiyandi:Up and
W. Curtis Preston:I left their existing policies running in parallel.
W. Curtis Preston:Well, I was gonna do the regular backups, uh, during the day,
W. Curtis Preston:um, and something like that.
W. Curtis Preston:It's been a while.
W. Curtis Preston:And, but I knew that I would screw them up if I had the archive bid on both systems.
W. Curtis Preston:So I used, this was net backup.
W. Curtis Preston:I used a feature of net backup that said, okay, don't use the archive bit on.
W. Curtis Preston:On the sink and what I didn't notice or what happened over time, over about
W. Curtis Preston:two weeks is the arc, the incremental backups got bigger and bigger and
W. Curtis Preston:bigger and bigger, and they got huge
Prasanna Malaiyandi:on your new
W. Curtis Preston:and they on the new system and they got so
W. Curtis Preston:big that they weren't finishing.
W. Curtis Preston:By the way, when I got there, they were backing up 28 hours a day.
W. Curtis Preston:So my, my opinion was no matter what I was doing, it still wasn't gonna be that bad.
W. Curtis Preston:, but backup, but the backups weren't finishing and I didn't understand, right.
W. Curtis Preston:So, so we shut off the old backups.
W. Curtis Preston:We're like, okay, we we're, we're beyond the point of no return.
W. Curtis Preston:We shut off the old backups, backups still weren't finishing.
W. Curtis Preston:And uh, you know, I was working through diff, you know,
W. Curtis Preston:multiple levels of support.
W. Curtis Preston:And I actually at, at one point ended up with my buddy Rob,
Prasanna Malaiyandi:Hmm.
W. Curtis Preston:I'm like, dude, I don't understand.
W. Curtis Preston:And we, you know, we figured out that when that.
W. Curtis Preston:Doesn't use the archive bit.
W. Curtis Preston:What it does is a, again, this is, this is a hundred years ago.
W. Curtis Preston:I don't know what it does today.
W. Curtis Preston:If it doesn't use the archive bit, it would, it would traverse the directory
W. Curtis Preston:tree and then it would, as soon as it encountered a directory that whose
W. Curtis Preston:modification time had changed, it would back up everything in that directory tree,
Prasanna Malaiyandi:Okay.
Prasanna Malaiyandi:Including data that does, yeah,
W. Curtis Preston:including data that hadn't changed, right?
W. Curtis Preston:It, it was really inefficient.
W. Curtis Preston:And so it tried to add that level of efficiency by working up the,
W. Curtis Preston:you know, so long story short, the back, the incremental backups were,
W. Curtis Preston:were getting close to full backups.
W. Curtis Preston:And so once we figured that out, we're like, okay, okay,
W. Curtis Preston:okay, okay, we figured it out.
W. Curtis Preston:We're gonna tell it to use the archive bit now.
W. Curtis Preston:And, um, and we turned on the, we turned on the archive bit.
W. Curtis Preston:and it was the same the next day.
W. Curtis Preston:Right?
Prasanna Malaiyandi:Okay,
W. Curtis Preston:Now I look, now I look, and meanwhile this, this
W. Curtis Preston:manager, he is looking at me, you know, looking at me over the, you know, he's
W. Curtis Preston:beginning to think I'm a complete moron.
W. Curtis Preston:And, um, he, um, I said, look, I, you know, we figured it out.
W. Curtis Preston:This is what, you know, he's like, but it, but it didn't fix the problem.
W. Curtis Preston:And that's when I was really talking to Rob.
W. Curtis Preston:I'm like, I don't understand.
W. Curtis Preston:And again, I was on the edge here.
W. Curtis Preston:I would, no one had done what I was doing, you know, and so, so what we
W. Curtis Preston:figured out was when you tell net backup not to use the archive bit,
W. Curtis Preston:it doesn't use the archive bit.
W. Curtis Preston:Now, what that means is that when it backs up a file, it doesn't
W. Curtis Preston:clear the archive bit, which was the whole point of me using the thing.
W. Curtis Preston:And so that meant that
Prasanna Malaiyandi:The
W. Curtis Preston:we,
Prasanna Malaiyandi:when you turn it on
W. Curtis Preston:to run the, we had to run the one backup.
W. Curtis Preston:that would use the archive bit to clear everything.
W. Curtis Preston:But because we had run one in a long time, it was also gonna be really long.
W. Curtis Preston:And so then it's like, okay, tomorrow night is gonna be all better.
W. Curtis Preston:And that's when the boss said, okay, smarty pants, no one's going anywhere.
W. Curtis Preston:He was, he was really upset at the level of instability.
W. Curtis Preston:And I can understand him being.
W. Curtis Preston:But he was really upset at the level of instability in his mind that
W. Curtis Preston:I had introduced into his system.
W. Curtis Preston:Nevermind that the new backup system was backing up 50% more
W. Curtis Preston:data than the old system because I changed the, the inclusion factor.
W. Curtis Preston:And they had, they had, they had left out a lot of really important data.
W. Curtis Preston:So it went from 20 terabytes to 30 terabytes and.
W. Curtis Preston:Nevermind the fact that we were now creating two copies, whereas before
W. Curtis Preston:he wasn't even getting one copy done.
W. Curtis Preston:So I was feeling pretty good.
W. Curtis Preston:But all he sees is his
Prasanna Malaiyandi:Stability.
Prasanna Malaiyandi:Yep.
W. Curtis Preston:Yeah.
W. Curtis Preston:And so he's like, well, nobody's going home until, and literally
W. Curtis Preston:he just, he stayed with us.
W. Curtis Preston:I mean, to his credit,
Prasanna Malaiyandi:Okay.
W. Curtis Preston:stayed with us and like he ordered pizza and stuff, uh,
W. Curtis Preston:but he literally wouldn't let us leave.
W. Curtis Preston:I couldn't even say like, well, I kicked off the back house.
W. Curtis Preston:We're gonna go get dinner.
W. Curtis Preston:I ain't nobody going anywhere.
Prasanna Malaiyandi:He's like bathroom.
Prasanna Malaiyandi:You already had your bathroom break.
W. Curtis Preston:but we're gonna sit here and Yeah.
W. Curtis Preston:And so we were there until pretty late.
W. Curtis Preston:Um, I don't really remember how late, but I remember like, it was like a whole
W. Curtis Preston:second day, you know, like it was, it was a long, long, long, long night.
W. Curtis Preston:And that my friends, . That's what happens when you don't understand backup levels.
W. Curtis Preston:Anyway, All right.
W. Curtis Preston:Well I think this is enough torturing people with backup level information.
Prasanna Malaiyandi:
Speaker:I, I a hundred percent
W. Curtis Preston:talk about, go ahead.
W. Curtis Preston:Yeah.
W. Curtis Preston:Next week we're gonna talk about metrics.
Prasanna Malaiyandi:Ooh,
W. Curtis Preston:yeah.
W. Curtis Preston:I think it's a little more exciting.
Prasanna Malaiyandi:yep.
W. Curtis Preston:It's backups.
W. Curtis Preston:Like how exciting could it be
Prasanna Malaiyandi:Come on, Curtis.
Prasanna Malaiyandi:What are you talking?
Prasanna Malaiyandi:Telling our audience, our listeners,
W. Curtis Preston:Woo.
W. Curtis Preston:I dunno what to tell you.
W. Curtis Preston:You know what, maybe you're excited as you're as excited
W. Curtis Preston:about backups as I am, maybe, to which I say, welcome to the party.
W. Curtis Preston:All right, well, uh, thanks for listening folks, and remember to
Listen On
