Seven reasons why your restore may slower than your backup
So many people are surprised when their restore is slower than their backup. You shouldn't be, as it's quite common. The good news is there are things you can do to make it faster – if you know them in advance. W. Curtis Preston (Mr. Backup) and Prasanna Malaiyandi tackle the seven reasons why your restore may be slower than your backup. Topics covered include RAID penalties, tape issues, database concerns, and others. You'll walk away knowing what to do in order to find out how slow your restores are – and how to fix them. This podcast is packed with good info! (And the death of a USB hub.)
Mentioned in this episode:
Interview ad
Are we going to have a funeral for your USB hub?
W. Curtis Preston:I don't think we'll have a funeral.
Prasanna Malaiyandi:Here lies a USB hub.
Prasanna Malaiyandi:It served me well.
W. Curtis Preston:Hi, and welcome to Backup Central's Restore it All podcast.
W. Curtis Preston:I'm your host, W.
W. Curtis Preston:Curtis Preston.
W. Curtis Preston:AKA Mr.
W. Curtis Preston:Backup and I have with me, my computer peripheral
W. Curtis Preston:consultant, Prasanna Malaiyandi.
W. Curtis Preston:How's it going?
Prasanna Malaiyandi:Good.
Prasanna Malaiyandi:I do not specialize in recommending mice or keyboards or other accessories
W. Curtis Preston:Uh, but you know, as usual you came in very handy during
W. Curtis Preston:my peripheral crisis, the Preston, the Preston peripheral put, I need
W. Curtis Preston:something with a P Preston peripheral.
W. Curtis Preston:Come on, give me a word, a problem.
Prasanna Malaiyandi:It's going to be one of those.
W. Curtis Preston:the Preston peripheral problem of 2022, uh, I
W. Curtis Preston:got to buy two new USB hubs, man, and no turning it off and on again.
W. Curtis Preston:Didn't fix it.
Prasanna Malaiyandi:I'm telling
W. Curtis Preston:out thing.
Prasanna Malaiyandi:surge protectors.
W. Curtis Preston:Yeah.
W. Curtis Preston:I don't know why I don't have a surge protector.
W. Curtis Preston:You think, you know, as a computer guy, I would know.
Prasanna Malaiyandi:Well, and it's funny that you bring this up because literally
Prasanna Malaiyandi:two days ago I just went and replaced most of the surge protectors in my house.
Prasanna Malaiyandi:Did you know, surge protectors only have a finite amount of life.
Prasanna Malaiyandi:And then after that they do nothing.
W. Curtis Preston:no,
Prasanna Malaiyandi:Yeah,
W. Curtis Preston:you know what I do on all my outlets that's not
W. Curtis Preston:a surge protector, what the thing is, this is what I'm curious.
W. Curtis Preston:I got to look into and see if it is a surge protector.
W. Curtis Preston:Although if it did it, didn't do its job.
W. Curtis Preston:Um, I have the.
Prasanna Malaiyandi:whole house.
W. Curtis Preston:No, no.
W. Curtis Preston:The, the, the power monitoring thingy, um, I have one on every outlet that matters,
W. Curtis Preston:uh, so that I can figure out, you know, where all the power's going to my house.
Prasanna Malaiyandi:Gotcha.
Prasanna Malaiyandi:But those don't do surge protection.
W. Curtis Preston:You don't think so.
Prasanna Malaiyandi:Nope.
Prasanna Malaiyandi:And
W. Curtis Preston:You'd for all that money that, you
W. Curtis Preston:know, cause it's one outlet.
Prasanna Malaiyandi:yeah.
Prasanna Malaiyandi:Well, so here's the thing with surge protectors.
Prasanna Malaiyandi:Uh, there's also a notion, so I did a lot of research
Prasanna Malaiyandi:because that's just the type of
W. Curtis Preston:Of course you did.
W. Curtis Preston:How many YouTube videos did you watch?
Prasanna Malaiyandi:None.
Prasanna Malaiyandi:I just read a lot of articles, but, but there is something right.
Prasanna Malaiyandi:Some surge protectors have what they call let in voltage or a pass through
Prasanna Malaiyandi:voltage, which is how much it actually allows in before it like clamps down
Prasanna Malaiyandi:on the surge, because that's what a surge protector is supposed to do.
Prasanna Malaiyandi:Right.
Prasanna Malaiyandi:You get a spike and it's supposed to clamp down to prevent it.
Prasanna Malaiyandi:And so some of them have.
Prasanna Malaiyandi:Normally you want it to be like 400 volts or less, which is still a lot of voltage
Prasanna Malaiyandi:which could fry your device, but it's much better than letting it all pass through.
Prasanna Malaiyandi:And so the lower, the number, the better it is.
Prasanna Malaiyandi:And the challenge is a lot of surge protectors after their life has gone,
Prasanna Malaiyandi:they don't automatically shut down.
Prasanna Malaiyandi:So they're just kind of letting everything pass through and
Prasanna Malaiyandi:they're not protecting you at all,
Prasanna Malaiyandi:but there are some brands.
Prasanna Malaiyandi:Yeah.
Prasanna Malaiyandi:There are some brands.
Prasanna Malaiyandi:So.
Prasanna Malaiyandi:The protection is gone.
Prasanna Malaiyandi:It actually shuts off the outlet.
W. Curtis Preston:Huh,
Prasanna Malaiyandi:So, you know, you have to replace it because
Prasanna Malaiyandi:how many people go and look at the green little protected light that's
Prasanna Malaiyandi:on their surge protector, right?
Prasanna Malaiyandi:That's hidden in a corner behind, like underneath your desk.
Prasanna Malaiyandi:Like no one ever does that.
Prasanna Malaiyandi:And so.
W. Curtis Preston:you know, this falls under the, I could go and spend
W. Curtis Preston:a whole lot of money every few years, and I don't feel like I should have to.
W. Curtis Preston:Like, it it's bad enough that I got to buy it in the first place, but
W. Curtis Preston:then if I got to I, and so I didn't know this, I didn't know, to even
W. Curtis Preston:look at the little green light.
W. Curtis Preston:I didn't know that was a thing.
W. Curtis Preston:Am I losing my, my tech cred?
Prasanna Malaiyandi:Some of them don't but it's one of the things
Prasanna Malaiyandi:you should just take a look and yeah, they say two to five years.
Prasanna Malaiyandi:It also depends on like the power in your area and how clean it is.
Prasanna Malaiyandi:If you get a lot of spikes, things like that, or you can live in like
Prasanna Malaiyandi:an area with lots of lightning.
Prasanna Malaiyandi:I think California, it's pretty good for the most part.
W. Curtis Preston:We would need to have some rain,
Prasanna Malaiyandi:Yeah.
Prasanna Malaiyandi:So not as big of a concern, but it's just one of those things yet
Prasanna Malaiyandi:periodically you might want to change or if you have like a ups.
Prasanna Malaiyandi:Yeah.
Prasanna Malaiyandi:Or if you have a
W. Curtis Preston:on the other hand,
Prasanna Malaiyandi:yeah.
Prasanna Malaiyandi:Florida on the other hand, lots of lightning and everything else, but yeah.
Prasanna Malaiyandi:Or if you have a ups, typically that already has surge
Prasanna Malaiyandi:protection built in as well.
W. Curtis Preston:Right, right.
Prasanna Malaiyandi:So things to think about.
W. Curtis Preston:Once Again, see, this is why you're my
W. Curtis Preston:computer peripheral consultant.
W. Curtis Preston:Doesn't that, that counts as a peripheral.
W. Curtis Preston:Doesn't it.
W. Curtis Preston:But a surge protector
Prasanna Malaiyandi:I think so.
W. Curtis Preston:into the computer.
Prasanna Malaiyandi:Yeah.
W. Curtis Preston:It's a accessory.
W. Curtis Preston:It's it's on the periphery.
Prasanna Malaiyandi:Yeah.
Prasanna Malaiyandi:Curtis, Curtis.
W. Curtis Preston:Uh, it's been an interesting week.
W. Curtis Preston:We did have a minor and I mean really minor, just random surge a while back.
W. Curtis Preston:And, um, and my, and my USB hub just stopped delivering data.
W. Curtis Preston:Like it
Prasanna Malaiyandi:But that's weird that it only stops delivering data.
W. Curtis Preston:yeah.
W. Curtis Preston:Yeah.
W. Curtis Preston:And it was, it was immediately that moment because I was actually using
W. Curtis Preston:my camera, which is a USB device.
W. Curtis Preston:And it just, you know, the second it happened, it was like, Oop, no
Prasanna Malaiyandi:but everything else works.
Prasanna Malaiyandi:Right.
Prasanna Malaiyandi:All the other devices plugged in.
Prasanna Malaiyandi:So you should probably be glad that that $30 hub took the hit rather than you
Prasanna Malaiyandi:having to go replace like five devices.
W. Curtis Preston:Yeah.
W. Curtis Preston:That's good.
W. Curtis Preston:Yeah.
W. Curtis Preston:It's just like, I'm trying to figure out, well, I can't, I can't
W. Curtis Preston:figure out exactly what happened.
W. Curtis Preston:You know what I mean?
W. Curtis Preston:From an electrical electrical perspective,
Prasanna Malaiyandi:No.
W. Curtis Preston:it just, it fried the brains, but it didn't fry.
W. Curtis Preston:You know what it is?
W. Curtis Preston:Is it fried the chip?
Prasanna Malaiyandi:Yeah, but not the power circuit.
W. Curtis Preston:power circuit.
W. Curtis Preston:Yeah.
W. Curtis Preston:The power circuit is probably pretty, pretty basic.
W. Curtis Preston:And then, yeah, good times.
W. Curtis Preston:May you live in interesting times?
Prasanna Malaiyandi:Are we going to have a funeral for your uSB hub
W. Curtis Preston:I don't think we'll have a funeral.
Prasanna Malaiyandi:Here lies a USB hub.
Prasanna Malaiyandi:It served me well.
W. Curtis Preston:It's served me.
W. Curtis Preston:Well, Curtis should have brought a surge protector.
W. Curtis Preston:I want to move on.
Prasanna Malaiyandi:What do you want to talk about?
W. Curtis Preston:I want to talk about slow restores.
W. Curtis Preston:This is seven ways to have a slow restore.
W. Curtis Preston:Should be really popular as a podcast, seven ways to have a slow restore.
W. Curtis Preston:And, uh, it's based on this article that I found on network world, this guy seems
W. Curtis Preston:to really knows what he's talking about.
W. Curtis Preston:What do you think?
Prasanna Malaiyandi:Who was it?
Prasanna Malaiyandi:The
W. Curtis Preston:Uh, it's a W.
W. Curtis Preston:Curtis Preston, that guy.
Prasanna Malaiyandi:Is he a relative of yours?
W. Curtis Preston:He is related He is related.
W. Curtis Preston:Uh, I see him on a pretty regular basis.
W. Curtis Preston:Uh, although sometimes when I'm looking at him, uh, I
Prasanna Malaiyandi:He's dashing.
W. Curtis Preston:Oh, he's, he's gorgeous.
W. Curtis Preston:Uh, and um, I like the picture that they have on the article.
W. Curtis Preston:Just some random dude looking into some sort of computer innards, like
W. Curtis Preston:he's going to figure out anything, but.
W. Curtis Preston:You know what I mean?
W. Curtis Preston:Like you look at that picture.
W. Curtis Preston:What's that guy going to figure out.
Prasanna Malaiyandi:
Speaker:everything in the world,
W. Curtis Preston:Yeah.
W. Curtis Preston:So I I'd like to start this podcast with a story.
Prasanna Malaiyandi:not our disclaimer.
W. Curtis Preston:oh yeah, sure.
W. Curtis Preston:We'll do the disclaimer Prasanna and I work for different companies.
W. Curtis Preston:He works for zoom.
W. Curtis Preston:I worked for Druva this is not a, uh, this is not a podcast of either company.
W. Curtis Preston:The opinions that you hear are all Prasanna's and, uh, be sure to rate our
W. Curtis Preston:podcast ratethispodcast.com/restore.
W. Curtis Preston:And if you care about this topic and any of the related topics, security, you
W. Curtis Preston:know, cybersecurity, ransomware, backup, recovery, disaster recovery, uh, you know,
W. Curtis Preston:I don't know, did I forget a category?
W. Curtis Preston:So it's barbecue and what's that all privacy.
W. Curtis Preston:Yeah, absolutely.
W. Curtis Preston:If you're, you know, anything that we can, that's in the
W. Curtis Preston:periphery, it's a big word today.
W. Curtis Preston:that word's going to pop up at least one
Prasanna Malaiyandi:the word of the day.
W. Curtis Preston:Uh, then just reach out to me @wcpreston on Twitter, you
W. Curtis Preston:can DM me or, uh, wcurtispreston@gmail.
W. Curtis Preston:And, uh, so yeah, so I wanna, I want to tell a little story and I'm I'm, if
W. Curtis Preston:you're a, if you're a longtime listener of the podcast, you may have heard this
W. Curtis Preston:story before, but you know, based on the listenership, I don't think anybody out
W. Curtis Preston:there has listened to all the podcasts except for maybe Daniel Rose Hill.
W. Curtis Preston:Hi Daniel.
W. Curtis Preston:Um, he's our backup anorak.
Prasanna Malaiyandi:Hi Daniel, hope the M-disc is working out.
W. Curtis Preston:He's he's been a guest on the podcast and, uh, you
W. Curtis Preston:know, big fan of the podcast anyway.
W. Curtis Preston:So back in the day when I got my first.
W. Curtis Preston:Uh, commercial backup and recovery tool.
W. Curtis Preston:And I, and I can actually say what it was because this it's a, it's a
W. Curtis Preston:company that's gone by the wayside.
W. Curtis Preston:A company's name was software moguls.
W. Curtis Preston:They were headquartered in, um, uh, Minnesota.
W. Curtis Preston:They were a suburb of Minneapolis and the, the name of the
W. Curtis Preston:product was SM-arch SM-arch
W. Curtis Preston:arc
Prasanna Malaiyandi:I think you
Prasanna Malaiyandi:brought this up a couple
Prasanna Malaiyandi:times.
W. Curtis Preston:Yeah.
W. Curtis Preston:Yeah.
W. Curtis Preston:W which is funny because you know, it should be a SM dash back,
W. Curtis Preston:but that's a whole other thing.
W. Curtis Preston:That's a different podcast.
W. Curtis Preston:Um, cause archive is not backup, right?
W. Curtis Preston:Yeah.
W. Curtis Preston:Okay.
W. Curtis Preston:And in case that wasn't obvious to everybody, and if you don't understand the
W. Curtis Preston:difference then look at our podcasts, we definitely have talked about that topic.
Prasanna Malaiyandi:Or purchase curtis's book.
W. Curtis Preston:Well, actually, you know what?
W. Curtis Preston:You don't even have to purchase it.
W. Curtis Preston:Now you can get a, uh, you know, if you, if you do it in time, uh,
W. Curtis Preston:for a limited time only, you can get a free ebook version of my
W. Curtis Preston:book by going to druva.com/podcast.
W. Curtis Preston:And you can get a free copy.
W. Curtis Preston:So the, um, and there is a whole chapter that basically
W. Curtis Preston:says archive is not backup.
W. Curtis Preston:And wow.
W. Curtis Preston:I really got off the topic.
W. Curtis Preston:All right.
W. Curtis Preston:So they had a feature.
W. Curtis Preston:This is way before deduplication . This is way before multiplexing, but they had
W. Curtis Preston:a feature that was inline compression.
W. Curtis Preston:And this was again, before all tape drives had compression.
W. Curtis Preston:And so they were going to really make my tapes like so much bigger.
W. Curtis Preston:Right.
W. Curtis Preston:And.
W. Curtis Preston:So I turned on this feature and, you know, I had been running this new
W. Curtis Preston:commercial backup product for a couple of months, but being the paranoid
W. Curtis Preston:backup person that I was, I still had the old system running in parallel.
Prasanna Malaiyandi:just in case.
W. Curtis Preston:Yeah.
W. Curtis Preston:And then we had our first major restore.
W. Curtis Preston:And I remember, um, I remember exactly where I was.
W. Curtis Preston:I remember exactly, you know, where the server was.
W. Curtis Preston:And I remember that I had to hop in my car and drive down.
W. Curtis Preston:Um, if there's any listeners in Delaware, I was in, I was on Christiana road.
W. Curtis Preston:Uh, Newark Delaware.
W. Curtis Preston:And I drove down the street and I, I remember going in there and what I
W. Curtis Preston:did is I put the old backup tapes in my back pocket, but I brought the new
W. Curtis Preston:fancy backup tapes in my front pocket.
W. Curtis Preston:And I put the tape in the drive and I went to go, um, I kicked off the first restore
W. Curtis Preston:and me being who I was, I created a while loop that, you know, you know, while
W. Curtis Preston:true; do df -k /directory; sleep 60; done.
W. Curtis Preston:Right.
W. Curtis Preston:And I'm watching this thing and I'm watching it.
W. Curtis Preston:I'm watching, I'm watching it.
W. Curtis Preston:It's not changing.
W. Curtis Preston:The restore is just running and like, after What I felt was
W. Curtis Preston:a really long period of time.
W. Curtis Preston:It finally changed to 1%.
W. Curtis Preston:And I'm like, based on, based on the current rate, this restore is going to
W. Curtis Preston:take yeah, it was going to take forever.
W. Curtis Preston:And by the way, it was probably two gigabytes.
W. Curtis Preston:This, you know what I
W. Curtis Preston:mean?
W. Curtis Preston:Yeah.
W. Curtis Preston:Back in the day it was, it was, it was probably less than two gigabytes.
W. Curtis Preston:Cause I remember our biggest server was Zeus and Zeus was, uh, six gigabytes.
W. Curtis Preston:Um, and that was, that was the entire server.
W. Curtis Preston:So this is one file system.
W. Curtis Preston:So it could have been, you know, it could have been one gigabyte.
W. Curtis Preston:And so I was like, what is going on?
W. Curtis Preston:And I went over to the tape drive and I'm looking at the tape drive and I'm looking
W. Curtis Preston:at the little Blinky light that indicates that that data is being read or written.
W. Curtis Preston:And I see blink, blink.
Prasanna Malaiyandi:pause.
W. Curtis Preston:Long pause, long pause, blink, blink.
W. Curtis Preston:And this went on.
W. Curtis Preston:I'd made it like a 911 call to software moguls.
W. Curtis Preston:I'm like, Hey man, whiskey tango foxtrot (WTF)?
W. Curtis Preston:I'm restoring this primary server.
W. Curtis Preston:This is the first time I am using my new fancy backup system
W. Curtis Preston:that we paid all this money for.
W. Curtis Preston:Uh, I remember that it was $16,000.
W. Curtis Preston:I remember that, that, you know, the
Prasanna Malaiyandi:That was.
W. Curtis Preston:caboodle,
W. Curtis Preston:it was a lot of money.
W. Curtis Preston:It was a lot of servers, but it was a lot of money.
W. Curtis Preston:Right.
W. Curtis Preston:And, um, this was 19, 19 93, right.
W. Curtis Preston:93 94.
W. Curtis Preston:And.
W. Curtis Preston:Um, they're like, well, did you, by chance turn on the compression feature.
W. Curtis Preston:Yeah.
Prasanna Malaiyandi:You're like, of course you're
Prasanna Malaiyandi:saying that I can save space.
Prasanna Malaiyandi:Of course, I'm going to turn it on.
W. Curtis Preston:Yeah.
W. Curtis Preston:So they're like, so here's how the compression feature works
W. Curtis Preston:during backup during backup.
W. Curtis Preston:It runs a compress minus C.
W. Curtis Preston:This is all Unix stuff, compress minus C and S and redirects the compression, which
W. Curtis Preston:sends the compression to standard out.
W. Curtis Preston:And then, and then it redirects it to a temporary file in /tmp right.
W. Curtis Preston:A filename.Z.
W. Curtis Preston:And then, then we copied filename.Z to, uh, to the tape.
W. Curtis Preston:During restore.
W. Curtis Preston:We, um, we restore filename.Z to /tmp, and then we run uncompress in place.
W. Curtis Preston:And then once it's done, then we copy it from temp to the file system.
W. Curtis Preston:And so, yeah, basically working as designed dude, like you
W. Curtis Preston:did test the restore, right?
W. Curtis Preston:When you, you should have known.
W. Curtis Preston:Right.
W. Curtis Preston:And so basically, you know, I was, luckily the story had a happy ending.
W. Curtis Preston:I had, I had the
Prasanna Malaiyandi:Yeah, you have to think in the back
Prasanna Malaiyandi:pocket.
Prasanna Malaiyandi:And,
W. Curtis Preston:I restored it and
W. Curtis Preston:everything was beautiful, you know, and
Prasanna Malaiyandi:and did you get rid of that
Prasanna Malaiyandi:software
Prasanna Malaiyandi:or that solution?
W. Curtis Preston:not, I did not get rid of the software.
W. Curtis Preston:Uh, in fact, uh, we'll, we'll bring that full circle.
W. Curtis Preston:So I did not, I did not get rid of the software.
W. Curtis Preston:I turned off the feature.
W. Curtis Preston:Right.
W. Curtis Preston:Um, and, uh, continued to use the software for the next couple of years.
W. Curtis Preston:And then when I left.
W. Curtis Preston:Uh, MBNA which at that time was the second largest credit card company.
W. Curtis Preston:I left MBNA to go into consulting and they put me in the headquarters of
W. Curtis Preston:Amoco was my first account, which was the American oil company in Chicago.
W. Curtis Preston:And, uh, they didn't have any decent backups.
W. Curtis Preston:Right.
W. Curtis Preston:And so I was like, man, they need, they need like commercial backup software.
W. Curtis Preston:And they said, well, we, we had some commercial backup software, but
W. Curtis Preston:we kinda dumped it because nobody could figure out how to use it.
W. Curtis Preston:I'm like, what did you have?
W. Curtis Preston:And they go SM-arch.
W. Curtis Preston:Seriously, like the one commercial product out of 50 that I know.
W. Curtis Preston:And you, this is literally complete coincidence.
W. Curtis Preston:So like, when I say that, like, I ended up being Mr.
W. Curtis Preston:Backup because due to a series of events beyond my control,
W. Curtis Preston:this is an example of that.
W. Curtis Preston:My first ever client was using the one and only commercial piece of product
W. Curtis Preston:that I, that I, you know, that I knew.
W. Curtis Preston:And I was able to call up to SM-arch to the company, software moguls.
W. Curtis Preston:And I said, listen, I'm at Amoco and I'm going to save your ass.
W. Curtis Preston:Right.
W. Curtis Preston:So if you could just rework the license so that it'll work in
W. Curtis Preston:the current environment, because whatever they bought, it doesn't
W. Curtis Preston:match what they, what they have now.
W. Curtis Preston:And they agreed to do it for me.
W. Curtis Preston:And so we got a license and we got the, we got everything backed up
W. Curtis Preston:and that's when that's when, uh, everything started falling apart.
W. Curtis Preston:And didn't, we have a podcast on why I used to be called a crash.
Prasanna Malaiyandi:Yes, we did
Prasanna Malaiyandi:have an episode.
W. Curtis Preston:Yeah.
W. Curtis Preston:So that episode kicks in after this episode, because basically what happened
W. Curtis Preston:is that the moment I got a decent backup of the entire data center, the
W. Curtis Preston:data center just started falling apart.
W. Curtis Preston:Like they, you know, and we ended up restoring like crazy.
W. Curtis Preston:So I got really, really good at restoring servers.
Prasanna Malaiyandi:But if you think about it, most people
Prasanna Malaiyandi:probably don't get that experience.
Prasanna Malaiyandi:Right.
Prasanna Malaiyandi:It's kind of like you learn trial by fire.
Prasanna Malaiyandi:Right?
Prasanna Malaiyandi:What you learned in the matter of many months is probably more than most
Prasanna Malaiyandi:people learn over like five years.
W. Curtis Preston:I have also fought a giant fire as well, an actual fire.
W. Curtis Preston:Remember, that's a whole other story.
Prasanna Malaiyandi:That
Prasanna Malaiyandi:is another story as
W. Curtis Preston:I have lots of lots of lessons.
W. Curtis Preston:That's what, that's the one advantage of being oaf.
W. Curtis Preston:So, uh, yeah, so, okay.
W. Curtis Preston:So what we're talking about here is reasons, and this is not one of
W. Curtis Preston:them, but this is just, uh, just to sort of give you an example of.
W. Curtis Preston:That the restore speed of a given backup will almost always be slower
W. Curtis Preston:than the backup speed of that backup.
W. Curtis Preston:Okay.
W. Curtis Preston:And there are a lot of reasons for that.
W. Curtis Preston:And I don't think that this is especially if, like you said, if you haven't.
W. Curtis Preston:Um, done this, you know, I used the phrase fired in anger, right?
W. Curtis Preston:If you've never fired your backups in anger, then you don't
W. Curtis Preston:know what I'm talking about.
W. Curtis Preston:Trust me, this is the case for many, many systems.
W. Curtis Preston:Not all, right?
W. Curtis Preston:It's not a universal truth, but it, but, but there are many reasons
W. Curtis Preston:that it can often be the case.
W. Curtis Preston:So, uh, Y you are looking at the article just as I am, right.
W. Curtis Preston:The first TA talk about the first problem that we have here.
Prasanna Malaiyandi:Yeah.
Prasanna Malaiyandi:So the first one is just sort of around if you're using a disk based subsystem, more
Prasanna Malaiyandi:than likely you're going to have raid.
Prasanna Malaiyandi:Right.
Prasanna Malaiyandi:Which
W. Curtis Preston:Right?
Prasanna Malaiyandi:simple explanation.
Prasanna Malaiyandi:It's a bunch of disk.
Prasanna Malaiyandi:Brought together to make it look like one disc with some level of redundancy
Prasanna Malaiyandi:within the, uh, within those disks.
Prasanna Malaiyandi:Right.
Prasanna Malaiyandi:And there are different types of encoding.
Prasanna Malaiyandi:Of course, erasure coding versus your normal parity based raid.
Prasanna Malaiyandi:But what happens is when you're writing to a raid disk or set of
Prasanna Malaiyandi:disks, Every time you do a write.
Prasanna Malaiyandi:There is some amount of additional writes that need to happen
Prasanna Malaiyandi:because you are keeping additional information with the data.
Prasanna Malaiyandi:So in case one disk fails, it can always be recalculated and
Prasanna Malaiyandi:you can get back your data.
Prasanna Malaiyandi:And this is normally known as parity information,
W. Curtis Preston:Right, right.
Prasanna Malaiyandi:Calculating parity isn't free, right?
Prasanna Malaiyandi:It's a bunch of checksum operations or other mechanisms in order
Prasanna Malaiyandi:to be able to calculate that.
Prasanna Malaiyandi:And then you have to end up writing that data across all of those discs.
Prasanna Malaiyandi:And so when you're doing the write.
Prasanna Malaiyandi:The performance could have some penalty, right?
Prasanna Malaiyandi:Because you have to do all the calculations and send all the
Prasanna Malaiyandi:writes to the appropriate places
W. Curtis Preston:Right.
W. Curtis Preston:so so just generally speaking, and by the way, this, this generally only applies.
W. Curtis Preston:To like RAIDs two through six, no one uses RAID two.
W. Curtis Preston:So really three through six, no one uses three or four or five anymore.
W. Curtis Preston:So really what we're talking it's RAID Uh, it doesn't uh, and what,
W. Curtis Preston:and why don't they use raid?
W. Curtis Preston:Well, I mean, some people still do, but they really shouldn't and why not?
W. Curtis Preston:But like anything lower than
Prasanna Malaiyandi:Yeah.
Prasanna Malaiyandi:So in most cases it basically only handles a single disk failure.
Prasanna Malaiyandi:So you lose one disc and you can handle that case.
Prasanna Malaiyandi:But if you
W. Curtis Preston:Like, how are, how often do you list?
W. Curtis Preston:Do you lose multiple disks though,
Prasanna Malaiyandi:At the worst possible time, right?
Prasanna Malaiyandi:Because, because normally what I've seen happen, right?
Prasanna Malaiyandi:Having worked at past storage vendors is you are, you have a disk fail.
Prasanna Malaiyandi:Now, if you have a spare, it's going to start rebuilding and
Prasanna Malaiyandi:repopulating that new disc that just got added and now the problem is.
Prasanna Malaiyandi:When you're in the process of rebuilding that disc, you now have to do reads
Prasanna Malaiyandi:across all the other discs and put additional load on your system, which
Prasanna Malaiyandi:could potentially lead to another disk failure, especially if your
Prasanna Malaiyandi:discs have been bought around the same time or have similar age, right.
Prasanna Malaiyandi:Or come from similar batches.
Prasanna Malaiyandi:Right.
Prasanna Malaiyandi:All these sorts of issues.
Prasanna Malaiyandi:And so.
Prasanna Malaiyandi:If you have one disk failed, it's highly likely that another disk may fail.
Prasanna Malaiyandi:And you're hoping that you can finish a rebuild before the next disc fails.
Prasanna Malaiyandi:But if you start thinking about like eight, 12 terabyte drives, that
Prasanna Malaiyandi:might take some time.
Prasanna Malaiyandi:And so
W. Curtis Preston:
Speaker:that's the real problem.
W. Curtis Preston:
Speaker:Yeah, that's the real modern problem is that you've got these giant disk drives
W. Curtis Preston:
Speaker:that take a really long time to rebuild.
W. Curtis Preston:
Speaker:And so that, that risky time.
W. Curtis Preston:
Speaker:In between you've had a disc failure and you you've rebuilt that failed disc.
W. Curtis Preston:
Speaker:That can be a really long time.
W. Curtis Preston:
Speaker:And during that time you could suffer another disc failure and
W. Curtis Preston:
Speaker:then you'd be, then you'd be Sol.
W. Curtis Preston:
Speaker:And that's why everybody uses RAID six, or at least they should be.
W. Curtis Preston:
Speaker:And if you're not, you should really look into that.
W. Curtis Preston:
Speaker:But I just want to make a point, this isn't a problem with raid 10, right.
W. Curtis Preston:
Speaker:Or raid one, which isn't really raid, right?
W. Curtis Preston:
Speaker:Uh, well, no RAID.
W. Curtis Preston:
Speaker:Right.
W. Curtis Preston:
Speaker:One is fine.
W. Curtis Preston:
Speaker:Zero.
W. Curtis Preston:
Speaker:Yeah.
W. Curtis Preston:
Speaker:A raid one is mirroring, uh, but re generally most people
W. Curtis Preston:
Speaker:use RAID 10, uh, which is, um, it's mirroring plus striping,
Prasanna Malaiyandi:Yep.
Prasanna Malaiyandi:And, and there are optimizations that some vendors do, trying to minimize the
Prasanna Malaiyandi:amount of data that gets written out.
Prasanna Malaiyandi:Like you write in full stripes rather than partial writes.
Prasanna Malaiyandi:You try to aggregate as much data as possible.
Prasanna Malaiyandi:Right.
Prasanna Malaiyandi:There are optimizations people try to do, right.
Prasanna Malaiyandi:But in the end, there's only so much you could do for having to recompute parity.
Prasanna Malaiyandi:Right.
Prasanna Malaiyandi:And the checksums and then send it to.
W. Curtis Preston:exactly.
W. Curtis Preston:And, but just the general rule is.
W. Curtis Preston:That it is slower to write to a RAID array than it is to read from a raid
W. Curtis Preston:array, a parity based RAID array.
W. Curtis Preston:And I would say the same is also true of an eraser coding based array.
W. Curtis Preston:Um, so, and so that is the first reason why you might have, um, um, a, you know, a
W. Curtis Preston:penalty when writing and then next we have is this little thing called copy-on-write.
W. Curtis Preston:Snapshots.
W. Curtis Preston:So I'm a huge fan of snapshots.
W. Curtis Preston:Um, I am right.
W. Curtis Preston:Um, I mean, you know, I ha I have caveats that, you know, they need to
W. Curtis Preston:be copied in order to be a backup.
Prasanna Malaiyandi:They're there for good purpose.
W. Curtis Preston:they are, they have a great purpose.
W. Curtis Preston:Having said that I am less of a fan of the copy-on-write.
W. Curtis Preston:Style of snapshots.
W. Curtis Preston:Right.
W. Curtis Preston:And, uh, I need to explain what that means.
W. Curtis Preston:So once you create a snapshot, uh, it creates a moment in time
W. Curtis Preston:that the snapshot, you, you didn't really create anything.
W. Curtis Preston:You just create a, like a, it's like a view into your storage, right?
W. Curtis Preston:You didn't copy anything.
W. Curtis Preston:Then when you go to overwrite a block of data with new data, they, the snapshot
W. Curtis Preston:system needs to preserve the old block that you had when you created snapshot.
W. Curtis Preston:So it copies that block out into a snapshot area.
W. Curtis Preston:And so when you go to read that snapshot, it gets most of the data
W. Curtis Preston:from the main drive, and then it gets.
W. Curtis Preston:Any before images from that other thing.
W. Curtis Preston:So that's why it's called copy on, write?
W. Curtis Preston:Because when you write, you're going to copy the data.
W. Curtis Preston:The longer you hold on to a snapshot, the more blocks that have to get copied out.
W. Curtis Preston:And the more blocks you have to read when you go to, um, to do that, and
W. Curtis Preston:which is why, um, and by the way, this is different than redirect on write.
W. Curtis Preston:Which is, um, where.
W. Curtis Preston:You simply write the new block in a new place.
W. Curtis Preston:Right.
W. Curtis Preston:And it's a series of pointers.
W. Curtis Preston:It's a lot more complicated and redirect on write.
W. Curtis Preston:Is close to, but not the same as what NetApp does really close
W. Curtis Preston:NetApps says it's different.
W. Curtis Preston:And you know, I'm sure it is, but it's close enough to that, but this is
W. Curtis Preston:why NetApp and products like NetApp.
W. Curtis Preston:They can have tons of snapshots without it impacting their
W. Curtis Preston:write performance, but it is.
W. Curtis Preston:Absolute certainty that if you have copy-on-write snapshots and you keep a
W. Curtis Preston:lot, you keep them around for a long time.
W. Curtis Preston:Um, or you just created a copy-on-write snapshot.
W. Curtis Preston:And now you go to do a large restore.
W. Curtis Preston:It's going to do a copy of every single block that you're trying to overwrite
W. Curtis Preston:before it can overwrite it, which means there's just going to be a big
W. Curtis Preston:penalty when you're going to do that.
W. Curtis Preston:Um, and you know, th th this isn't me, you know, I remember I never
W. Curtis Preston:worked for NetApp, but I remember when I was explaining this.
W. Curtis Preston:To somebody and they're like, oh, or you're just a NetApp lover.
W. Curtis Preston:And you're just, I'm like, okay, it's just a fact, right?
W. Curtis Preston:Like I'm I remember being at a large, not Amoco, but an a really
W. Curtis Preston:large oil and gas company and a certain other large storage vendor.
Prasanna Malaiyandi:Yup.
W. Curtis Preston:One that you might be very familiar with, um, came in and we
W. Curtis Preston:just asked them a point blank question.
W. Curtis Preston:The customer wants to keep six months of user browsable snapshots, what
W. Curtis Preston:would happen to their performance?
W. Curtis Preston:And they were like, no one does that.
W. Curtis Preston:That was a response.
W. Curtis Preston:No one does that.
W. Curtis Preston:Well, we're doing it here with the NetApp systems.
W. Curtis Preston:We already have what would happen if they, and they literally had to guess
W. Curtis Preston:they were, they guessed it like a 50%.
W. Curtis Preston:Performance hit was, was the best guess.
W. Curtis Preston:So anyway, so if you have a copy on write snapshot based storage
W. Curtis Preston:array, you've created a snapshot and then you go to do a large restore.
W. Curtis Preston:You're going to have a huge write penalty when you, um, overwrite that.
W. Curtis Preston:Um, so, and then the next, uh, what about this file system bit?
Prasanna Malaiyandi:Yeah.
Prasanna Malaiyandi:So the challenge with file systems, right?
Prasanna Malaiyandi:And this is when it comes to writing into a file system, right?
Prasanna Malaiyandi:It's no longer the small, like your laptop file systems we're talking about,
Prasanna Malaiyandi:but these very, very dense file systems.
Prasanna Malaiyandi:If you look at some of the scale-out file systems out there with millions
Prasanna Malaiyandi:and millions of files on it, right when you're restoring the file.
Prasanna Malaiyandi:First, it creates a file that it wants to restore the data too.
Prasanna Malaiyandi:And then separately, it has to pull the data for the actual data contained
Prasanna Malaiyandi:within the file system itself.
Prasanna Malaiyandi:Right.
Prasanna Malaiyandi:And so, because there are these two steps and depending on how many files
Prasanna Malaiyandi:or files are in the file system, because in the end, all of that
Prasanna Malaiyandi:needs to be tracked in that system.
Prasanna Malaiyandi:And so creating these files can actually take a tough take quite a while.
Prasanna Malaiyandi:And so if you're pulling all this data and say you have millions and millions
Prasanna Malaiyandi:of files that could take you much longer than say having one large file versus a
Prasanna Malaiyandi:million small files.
W. Curtis Preston:Yeah, exactly.
W. Curtis Preston:That, that it could actually, I could actually take more time
W. Curtis Preston:to create the files than it does to actually transfer the data.
Prasanna Malaiyandi:Most people think it's just, oh, I'm just creating the file.
Prasanna Malaiyandi:Isn't that simple.
Prasanna Malaiyandi:But you have to remember when you're restoring file.
Prasanna Malaiyandi:It's not only creating the file.
Prasanna Malaiyandi:Right.
Prasanna Malaiyandi:It's also setting appropriate permissions on the file or anything
Prasanna Malaiyandi:else that needs to be done to the file in addition to moving the data.
W. Curtis Preston:And it is a really small amount of time, but when you have
W. Curtis Preston:millions of files, it's a small amount of time divided or multiplied times millions.
W. Curtis Preston:And I, and this is why, by the way, there, there, there are products that
W. Curtis Preston:are specifically designed to do it.
W. Curtis Preston:The only one that's coming to my mind and I hope I get the product name.
W. Curtis Preston:Cause it's been awhile, but a net backup had maybe still has
W. Curtis Preston:a product called flashbackup.
W. Curtis Preston:And what it does is when you have a scenario like this, where you
W. Curtis Preston:have a very dense file system, they can back it up at the block level.
W. Curtis Preston:And then when you need to restore the entire file system, they
W. Curtis Preston:restored it to block level.
W. Curtis Preston:When you restore it at the block level, you don't have this problem.
W. Curtis Preston:Right.
W. Curtis Preston:We don't have this problem when we restore an entire VM.
Prasanna Malaiyandi:Yep.
Prasanna Malaiyandi:I will say I'll just want to say yeah.
Prasanna Malaiyandi:Image based or app aware image-based
Prasanna Malaiyandi:backups is I think
Prasanna Malaiyandi:the
Prasanna Malaiyandi:industry
W. Curtis Preston:exactly.
W. Curtis Preston:So, but if you are, if you are restoring at the file system level and you have
W. Curtis Preston:a very dense file system is going to take awhile, it's going to take
W. Curtis Preston:longer than, than, than otherwise.
W. Curtis Preston:Um, the next one it's, it's a bit odd, but it is what it is.
W. Curtis Preston:It's a bit, it's a bit like it's a bit more out there.
W. Curtis Preston:But if you, you know, and I, I mentioned overburdened transaction logs.
W. Curtis Preston:So in a transaction log, in a database, it's got to keep track of all the
W. Curtis Preston:transactions depending on how you.
W. Curtis Preston:Do your restore.
W. Curtis Preston:So if you've got a re the way you do a database restore is you generally have
W. Curtis Preston:a backup of the database from, let's say 24 hours ago, maybe even longer
W. Curtis Preston:than that could be 12 or whatever.
W. Curtis Preston:It depends on a number of factors.
W. Curtis Preston:And then you have a number of transaction logs that you use to
W. Curtis Preston:move that database forward in time from when the backup was taken up
W. Curtis Preston:to the point in time of the outage.
W. Curtis Preston:And if the transaction logs are, um, you know, if the storage, if the
W. Curtis Preston:performance of that is not up to snuff, it can really slow down the playing
W. Curtis Preston:replaying of all those transaction logs.
W. Curtis Preston:And this is something you might not notice during normal operations,
W. Curtis Preston:but the replaying of the transaction logs, it's like you're taking.
W. Curtis Preston:What could be 24 hours of transactions and you're playing them all within 20 minutes.
W. Curtis Preston:Right.
W. Curtis Preston:And so it could really bog down your, your transaction logs and if your transaction
W. Curtis Preston:logs, uh, if the storage is not up to snuff, uh, so what, what does this say?
W. Curtis Preston:It put your transaction logs on flash and that's all I got to say
Prasanna Malaiyandi:The other thing I would also say is.
Prasanna Malaiyandi:Make sure you understand how long it'll take to replay those lines.
Prasanna Malaiyandi:Right.
Prasanna Malaiyandi:So for instance, if you were only doing, you had in your example, 24
Prasanna Malaiyandi:hours to do your normal backups, but say the customer decides, oh,
Prasanna Malaiyandi:I'm only going to do it once a week.
Prasanna Malaiyandi:Now you have seven days worth of transaction logs to play back,
Prasanna Malaiyandi:maybe in the case of a, not so heavily used database, that's fine.
Prasanna Malaiyandi:But if this is like amazon.com, right, and you're trying to play back a week worth of
Prasanna Malaiyandi:transactions, that's a lot of records to
Prasanna Malaiyandi:replay against the database.
W. Curtis Preston:Yeah.
W. Curtis Preston:And we're going to cover this again at the end, but basically I would, I would
W. Curtis Preston:do this, do a test restore and see how long, you know, if you're, if you're
W. Curtis Preston:doing it once a day and then you play a typical days worth of transaction logs and
W. Curtis Preston:you're like, oh, well that takes one hour.
W. Curtis Preston:Are we okay with one, you know, a one hour RTO.
W. Curtis Preston:It actually, it's going to be more than an hour RTO.
W. Curtis Preston:Cause it's going to be the time to restore the database and then the
W. Curtis Preston:time to restore the transaction logs.
W. Curtis Preston:So then you can adjust perhaps your backup frequency,
Prasanna Malaiyandi:And do it all against, or do it for a
Prasanna Malaiyandi:production backup that you did.
Prasanna Malaiyandi:Don't do it for like a test instance that you're just trying out,right do it for
Prasanna Malaiyandi:an actual production instance that you can actually test and see, see what in
Prasanna Malaiyandi:real life those transactions look like.
Prasanna Malaiyandi:And you'll get an understanding of your RTO
W. Curtis Preston:would you recommend doing like our friend in Alaska did?
Prasanna Malaiyandi:Yeah.
Prasanna Malaiyandi:Don't do it in your production environment while you're tearing down.
Prasanna Malaiyandi:Right.
Prasanna Malaiyandi:Paul, we love you.
Prasanna Malaiyandi:But man, that was crazy story.
W. Curtis Preston:Oh God, that was a crazy story.
W. Curtis Preston:Um, uh, what was it, what was that episode?
W. Curtis Preston:Um,
Prasanna Malaiyandi:It was with Paul van Dyke.
Prasanna Malaiyandi:Episode 1 35, it admin deletes entire data center.
Prasanna Malaiyandi:Then tests his backups.
W. Curtis Preston:Yeah, that would be the one.
W. Curtis Preston:Uh, so when you say testing it with production data, you don't mean
W. Curtis Preston:testing your restored by restoring your production database on top of your desk.
W. Curtis Preston:Do you mean using your production backups and restoring it to it to a test area?
Prasanna Malaiyandi:That is
W. Curtis Preston:Key key differentiator there.
W. Curtis Preston:So the, the next thing, and this is, I think, I think this is less
W. Curtis Preston:of a problem for most people, but for those of you still backing up
W. Curtis Preston:to tape, this is a real problem.
W. Curtis Preston:So multiplexing, and again, if you go back to the backup is evil episode
W. Curtis Preston:from four or five episodes ago.
W. Curtis Preston:Uh, we talked about this.
W. Curtis Preston:We talked about that.
W. Curtis Preston:That multiplexing is evil.
W. Curtis Preston:It was, and still is a necessary evil.
W. Curtis Preston:If you're backing up to a modern tape drive, the reason is that
W. Curtis Preston:the tape drive wants to go a lot faster than the backup can go.
W. Curtis Preston:And so you take and you, interleave a bunch of different backups together
Prasanna Malaiyandi:Which sounds amazing.
W. Curtis Preston:Which sounds amazing.
W. Curtis Preston:And it makes a tape drive happy and it eliminates shoe shining
W. Curtis Preston:or at least reduces shoe shining.
W. Curtis Preston:And everything's great.
W. Curtis Preston:But the problem is when you go to restore, you have to read all of those
W. Curtis Preston:backups and throw away all, but the one that you need and modern multiplexing
W. Curtis Preston:settings they're as high as like 32.
W. Curtis Preston:So you're, you're throwing away, you know, I dunno what a 32 divided by a hundred.
W. Curtis Preston:Uh, no.
W. Curtis Preston:Is it, would that be 97?
W. Curtis Preston:What
Prasanna Malaiyandi:you're throwing away.
Prasanna Malaiyandi:97% of the data.
W. Curtis Preston:is that really?
W. Curtis Preston:97% of the data.
W. Curtis Preston:Is, are you, are you just really good in your head or did you divide
Prasanna Malaiyandi:no.
Prasanna Malaiyandi:Can in my
Prasanna Malaiyandi:head,
W. Curtis Preston:Okay.
W. Curtis Preston:Uh, so yeah, so you're throwing away 97% of the data, which means that
W. Curtis Preston:your restore speed is gonna suck!
Prasanna Malaiyandi:It's actually 96.37, sorry.
Prasanna Malaiyandi:8
W. Curtis Preston:good.
W. Curtis Preston:That was pretty good.
W. Curtis Preston:pretty good.
W. Curtis Preston:in your head, um, thing there.
W. Curtis Preston:So, um, And that's why this is why we stopped using tape.
W. Curtis Preston:This is why I stopped recommending the use of tape as a primary protection mechanism.
W. Curtis Preston:I'm not even that big of a fan of it as a secondary production mechanism.
W. Curtis Preston:Um, you know, this is, you know, we, we talked about this when an
W. Curtis Preston:episode with Brian Greenberg, uh, and his, uh, a colleague where we,
W. Curtis Preston:you know, th there, there is a.
W. Curtis Preston:Th there is a group of people that are bigger fans of tape now because of
W. Curtis Preston:ransomware, but you got to address this issue and you got to make sure that you
W. Curtis Preston:understand that when you restore from tape, if you used multiplexing, then
W. Curtis Preston:you're basically, if you did not use multiplexing, you don't have this problem.
W. Curtis Preston:If you did use Mo I'm sorry.
W. Curtis Preston:If you did not have multiplexing, then you don't have this problem,
W. Curtis Preston:but you have a different problem.
W. Curtis Preston:You you'll just you'll.
W. Curtis Preston:Well, you want, you have full backups and you have one stream you'll more than
W. Curtis Preston:likely be suffering shoe shining from your restore instead of shoe shining
W. Curtis Preston:during your backups, because you'll get, you'll get the, uh, the raid penalty
W. Curtis Preston:and the, the write, the write speed.
W. Curtis Preston:Even if you don't get the right, the raid penalty you're discouraged probably
W. Curtis Preston:has a limit at which it can write.
W. Curtis Preston:And it's probably different than the speed at which the tape.
W. Curtis Preston:Tape drive can go.
W. Curtis Preston:A lot of people don't realize that tape drives are typically way faster
W. Curtis Preston:than most, uh, in terms of throughput, not random access, but throughput.
Prasanna Malaiyandi:Yup.
W. Curtis Preston:So you'll, so your choices, your choices, both suck.
W. Curtis Preston:That's why I don't like using tape anymore for backups.
W. Curtis Preston:I like using them for archive.
W. Curtis Preston:Because they're much better than disk at holding onto
W. Curtis Preston:data for long periods of time.
W. Curtis Preston:Um, so, so really what we're talking about here, and that's the end of the
W. Curtis Preston:reasons, and some of those you can address, you could potentially say, well,
W. Curtis Preston:because of the restore speed problem.
W. Curtis Preston:We're going to stop using RAID six, or we're going to go to RAID 10, right?
W. Curtis Preston:That's a huge cost because that is a significant difference in the
W. Curtis Preston:number of disks that you will need.
W. Curtis Preston:Although the jump, the jump from raid six to 10 is not
Prasanna Malaiyandi:How bad as yeah.
W. Curtis Preston:five to 10.
W. Curtis Preston:Right.
W. Curtis Preston:Um, and by the way, it should be RAID 10, not raid zero plus one.
W. Curtis Preston:There is a difference between read 10 and RAID zero plus one.
W. Curtis Preston:Uh, there is a difference in the num the number of drives that you can lose.
Prasanna Malaiyandi:Or, Or, do you think some of this goes away also, if you're
Prasanna Malaiyandi:considering like SSD for primary storage.
W. Curtis Preston:You know, that's a good question.
W. Curtis Preston:Uh, and the answer is I have no idea.
W. Curtis Preston:Um, SSD is really good at random, you know, it's fast at writing, but if the
W. Curtis Preston:problem is the calculation, then maybe it
W. Curtis Preston:doesn't
Prasanna Malaiyandi:yeah, I don't know.
Prasanna Malaiyandi:Maybe if you get a wide RAID group plus
Prasanna Malaiyandi:yeah.
Prasanna Malaiyandi:Interesting.
Prasanna Malaiyandi:Yeah.
W. Curtis Preston:I don't know.
W. Curtis Preston:Um, I mean, we're all going to be moving to SSDs.
Prasanna Malaiyandi:Yeah.
W. Curtis Preston:I honestly think that we're going to get to a point where almost
W. Curtis Preston:everything is either on SSD or tape,
Prasanna Malaiyandi:Yeah.
Prasanna Malaiyandi:You do the two ends of the spectrum?
Prasanna Malaiyandi:You decide where your workload runs and you're
Prasanna Malaiyandi:good to go.
W. Curtis Preston:Yeah.
W. Curtis Preston:Yeah.
W. Curtis Preston:So all I'm saying here is just be aware of these things now.
W. Curtis Preston:Don't don't be like me.
W. Curtis Preston:Don't be like what happened?
W. Curtis Preston:Right.
W. Curtis Preston:And find out that your, your raid penalty, when you go to do a large
W. Curtis Preston:restore and everyone is looking at you,
Prasanna Malaiyandi:Yeah.
W. Curtis Preston:figure this out.
W. Curtis Preston:Now think about the worst case scenario that you have, and then go test
Prasanna Malaiyandi:Yeah.
Prasanna Malaiyandi:Or,
W. Curtis Preston:the biggest server.
Prasanna Malaiyandi:I was thinking of when doing your file server
Prasanna Malaiyandi:restores, don't just pick a single directory with like a hundred files in.
Prasanna Malaiyandi:Right.
Prasanna Malaiyandi:Pick something more substantial to restore.
Prasanna Malaiyandi:So you can understand what the real world performance looks like rather
Prasanna Malaiyandi:than you having to do it in urgent need.
W. Curtis Preston:You need to do test restores and you need to
W. Curtis Preston:do representative test restores, similar sizes, similar hardware.
W. Curtis Preston:Um, you know, generally you're going to get slower hardware to test on.
W. Curtis Preston:I do think that VMware and virtual I'll just say virtualization in
W. Curtis Preston:general makes this a lot easier.
W. Curtis Preston:It's a whole lot easier to restore an entire VM than "back in the day" when we
W. Curtis Preston:did a bare metal recovery of a physical server, that was a giant pain in the butt.
W. Curtis Preston:You'll notice for those of you that get the, the new book, modern data protection.
W. Curtis Preston:Barely mentioned BMR because you just shouldn't be doing that at this point.
W. Curtis Preston:Right.
W. Curtis Preston:Um, you just, everything should be virtualized in this point.
W. Curtis Preston:It should either be a VM in the cloud or a VM in one of your, you know, pick your
W. Curtis Preston:favorite hypervisor, the advantages from a backup and recovery perspective alone.
W. Curtis Preston:Um, you know, figure that out or work it out.
W. Curtis Preston:So this is all I'm saying is, is.
W. Curtis Preston:Is test it now and then set expectations because it's just like, it's just
W. Curtis Preston:like, you know, fights in a marriage.
W. Curtis Preston:So many times you get over, you get over a fight over something so stupid.
W. Curtis Preston:And it's because one of you just had a different set of
W. Curtis Preston:expectations than the other.
W. Curtis Preston:Just make sure that you go in, like you have a meeting before the bad
W. Curtis Preston:thing happens and say, listen, I've been doing some tests restores.
W. Curtis Preston:And it turns out that the raid five penalty of our umpty-squat array.
W. Curtis Preston:It means that our restore is going to take roughly 50% more
W. Curtis Preston:amount of time than the backups.
W. Curtis Preston:Let's talk about that now.
W. Curtis Preston:And we can either accept that and then don't yell at me when this
W. Curtis Preston:happens during production or, um, let's make a change to the design.
Prasanna Malaiyandi:I think another important point is it's not just a
Prasanna Malaiyandi:one time at the start of a project and you're done because data
Prasanna Malaiyandi:sets change requirements change.
Prasanna Malaiyandi:This is an ongoing basis.
Prasanna Malaiyandi:You should be doing realistic restores going back, communicating
Prasanna Malaiyandi:with your stakeholders, right?
Prasanna Malaiyandi:Keeping them up to date on what's going on because what might have
Prasanna Malaiyandi:been agreed upon on day one, right?
Prasanna Malaiyandi:Three months for now, when the requirements have changed, right?
Prasanna Malaiyandi:If you don't go back and communicate them, then.
Prasanna Malaiyandi:And set expectations and things may still
Prasanna Malaiyandi:blow up.
W. Curtis Preston:And I'd like to suggest, and maybe we should
W. Curtis Preston:do a whole podcast on this of just ways to affect test restores.
W. Curtis Preston:But one thing that I tried was.
W. Curtis Preston:When we procure when we procured a new server.
W. Curtis Preston:We, the backup team was given access to that server for a little while
W. Curtis Preston:before it got production access.
W. Curtis Preston:And what we would do is use that to test full server restores.
W. Curtis Preston:And you can do that with a new box that you bought in to brought
W. Curtis Preston:in, to be a VM-ware server or hyper V or AHV or whatever.
W. Curtis Preston:Um, and then just test the crap out of a test, different VMs, you know, make
W. Curtis Preston:sure that it's in some kind of bubble.
Prasanna Malaiyandi:Yeah,
W. Curtis Preston:So that it doesn't start sending out exchange email,
Prasanna Malaiyandi:cool.
W. Curtis Preston:speaking of exchange, what's my opinion
W. Curtis Preston:on on-prem exchange Prasanna.
Prasanna Malaiyandi:Don't do it.
W. Curtis Preston:That's right.
W. Curtis Preston:Who the hell is still doing on prem Exchange?
W. Curtis Preston:You know what, if you're out there and you're listening to this and you're
W. Curtis Preston:doing on-prem exchange and you're like, why does he keep yelling at me?
W. Curtis Preston:I want to know what is your deal?
W. Curtis Preston:What is it that you like?
W. Curtis Preston:About on-prem exchange that you, you know, that you don't get.
W. Curtis Preston:Sure.
Prasanna Malaiyandi:I think one of them could be data residency related.
W. Curtis Preston:Do you really think that's a thing?
W. Curtis Preston:Like people wanting the, the copy of the, just their data just in their data center,
Prasanna Malaiyandi:Not in their data center, but if there are regulations.
W. Curtis Preston:what do you, you can, regulations should keep.
W. Curtis Preston:That, you know, that's a good question.
W. Curtis Preston:I don't know.
W. Curtis Preston:I, you know, there, there are so many industries and so many
W. Curtis Preston:regulations, there could be something, but I am not aware of any,
Prasanna Malaiyandi:I'm not particularly aware, but yeah, that's
Prasanna Malaiyandi:the only thing that comes to mind is.
Prasanna Malaiyandi:And most like Microsoft Azure.
Prasanna Malaiyandi:They are pretty good in terms of where they're located these days.
Prasanna Malaiyandi:So I could see that being less of an issue versus like five years ago.
Prasanna Malaiyandi:But I'm just wondering if there was still some of those customers.
W. Curtis Preston:I could see there being like the touchy feely problem.
W. Curtis Preston:I'm like, I just want to touch it with my hot little hands.
W. Curtis Preston:I get that.
W. Curtis Preston:Although I disagree with that, uh, you know, the, the value of
W. Curtis Preston:physically touching your server.
W. Curtis Preston:Vastly overrated.
W. Curtis Preston:Um, and I think that you, you, you gain a vast amount of security and
W. Curtis Preston:whatnot by using SaaS services and by using, um, you know, IaaS services
W. Curtis Preston:where you can just point and click and say, I need this firewall and this
W. Curtis Preston:set of rules and this set of thing.
W. Curtis Preston:And you just get all that stuff with a point click button, rather than
W. Curtis Preston:having to piece it all together.
W. Curtis Preston:Um, you know, I'll take, I'll take the security of an average
W. Curtis Preston:cloud vendor over the average data center any day of the week.
W. Curtis Preston:And that isn't just because I work for Druva I've I've always said that.
W. Curtis Preston:Um, so anyway, well it's been, uh, you know, it's been one of those sad episodes.
Prasanna Malaiyandi:It's not
W. Curtis Preston:we delivered
W. Curtis Preston:nothing but bad
Prasanna Malaiyandi:It's not sad.
Prasanna Malaiyandi:It's just things that we think people should be aware.
Prasanna Malaiyandi:Right.
Prasanna Malaiyandi:And because otherwise, well, it's not sad because it would be sad if we didn't
Prasanna Malaiyandi:tell them this information and then things blew up and escalated right.
Prasanna Malaiyandi:Way.
Prasanna Malaiyandi:Cause like you, right.
Prasanna Malaiyandi:If they have an issue where they need to do a restore,
Prasanna Malaiyandi:they had never tested it out.
Prasanna Malaiyandi:And now they're like what happened?
W. Curtis Preston:Yeah.
W. Curtis Preston:Yeah.
W. Curtis Preston:Don't be don't.
W. Curtis Preston:don't.
W. Curtis Preston:be like
W. Curtis Preston:earn it.
W. Curtis Preston:Yeah.
W. Curtis Preston:We need the little stick figure.
W. Curtis Preston:Here it is.
W. Curtis Preston:Didn't test his restores.
W. Curtis Preston:Curtis had to use the old back up.
W. Curtis Preston:Don't be like Curtis.
Prasanna Malaiyandi:least Curtis
Prasanna Malaiyandi:had an old
Prasanna Malaiyandi:backup.
W. Curtis Preston:Yeah.
W. Curtis Preston:You know, actually there is in the book, there is a stick figure.
W. Curtis Preston:There is one of those stick figures that, that talks about Curtis.
W. Curtis Preston:I forgot exactly.
W. Curtis Preston:I forgot which one it was, but we did one of those little stick figure
W. Curtis Preston:memes of don't be like Curtis.
W. Curtis Preston:Um, so anyway, well, uh, you know, thanks for discussing this article
W. Curtis Preston:written by this brilliant guy.
Prasanna Malaiyandi:Anytime Curtis.
Prasanna Malaiyandi:Yeah, that
Prasanna Malaiyandi:author was really
Prasanna Malaiyandi:good.
Prasanna Malaiyandi:Maybe we should have him on the podcast.
W. Curtis Preston:
Speaker:Great minds think alike.
W. Curtis Preston:
Speaker:I like that.
W. Curtis Preston:
Speaker:And, uh, thanks to the listeners.
W. Curtis Preston:
Speaker:Um, you know, we'd be nothing without you and remember to subscribe
Listen On
