Backup is evil (or at least how many people do it is)
This week we are joined by John "Ricky" Martin, Director of Strategy at NetApp (and former owner of a tape recovery business), to talk about his paper that declares that backup is fundamentally evil and done in an unintelligent way. Mr. Backup wasn't sure how this one was going to go, and there were at least one or two arguments along the way. No blows were thrown, though. We definitely talk about what a tape recovery business is, and what it was like to do that. We also talk about tape backup, full backups, multiplexing, tape handling, and other elements of how backup is still done today by many people. It's a fun episode where you should learn a lot.
Mentioned in this episode:
Interview ad
Nobody wants to talk about backup.
Ricky Martin:Nobody wants it.
Ricky Martin:It's evil, but it's a necessary evil, but nonetheless, nobody
Ricky Martin:really wants to talk about it.
Ricky Martin:It's like talking about plumbing, right?
Ricky Martin:Nobody cares.
Ricky Martin:Nobody cares about plumbing until you get backed up.
Ricky Martin:*rimshot*
Ricky Martin:oh,
W. Curtis Preston:Yeah.
W. Curtis Preston:Hi and welcome to Backup Central's Restore it All podcast.
W. Curtis Preston:I'm your host, W.
W. Curtis Preston:Curtis Preston, AKA Mr.
W. Curtis Preston:Backup, and I have with me my clarified butter consultant, Prasanna Malaiyandi.
W. Curtis Preston:how's it going Prasanna?
Prasanna Malaiyandi:I'm good, Curtis.
Prasanna Malaiyandi:I have to say my wife was quite surprised when we were talking about
Prasanna Malaiyandi:Ghee, which is Indian clarified butter.
Prasanna Malaiyandi:And she's like, what are you talking to Curtis about?
Prasanna Malaiyandi:Because I think , first,, we started off with Herbal teas . Because you're
Prasanna Malaiyandi:like, yeah, there's not enough flavor.
Prasanna Malaiyandi:Should I try using like loose leaf?
Prasanna Malaiyandi:And we were talking about that.
Prasanna Malaiyandi:And then I switched over to ghee, and she's like, who are you talking to?
Prasanna Malaiyandi:And why does Curtis care about this?
W. Curtis Preston:And then suddenly I got to talk to her and she, and I was
W. Curtis Preston:like, I'm thinking about trying Ghee like, is there a brand that I should try?
W. Curtis Preston:And your wife's like, well, I make my own.
W. Curtis Preston:And I'm like, of course you do.
W. Curtis Preston:and so,
Prasanna Malaiyandi:It's a staple in like Indian cooking, right?
Prasanna Malaiyandi:Or
W. Curtis Preston:I didn't even know ghee existed until a couple of years ago.
W. Curtis Preston:And then it just keeps coming up.
W. Curtis Preston:there was this meme on Facebook, where it was like, it's two people
W. Curtis Preston:and he's like, this butter is amazing.
W. Curtis Preston:And he says, actually, it's ghee.
W. Curtis Preston:And she says, thanks for clarifying,
Prasanna Malaiyandi:I did see that one on Reddit.
W. Curtis Preston:it's yeah, it's been coming up a lot.
W. Curtis Preston:And so I see it for me as a way to solve a long running argument in this
W. Curtis Preston:house, because the thing about ghee for those that don't know is that it's
W. Curtis Preston:shelf stable that By doing what you do.
W. Curtis Preston:You can just leave it on the counter and it lasts much longer
W. Curtis Preston:than butter would on the counter
Prasanna Malaiyandi:least a couple of months or until you finish it.
W. Curtis Preston:Yeah.
W. Curtis Preston:Yeah, exactly.
W. Curtis Preston:I don't think this jar that I bought by the way I have received my first jar of
W. Curtis Preston:ghee today and I have already eaten some So I like to do toast with liberal butter
W. Curtis Preston:, and I want it to be easy to spread.
W. Curtis Preston:So I want the butter on the counter.
W. Curtis Preston:My wife is concerned about the butter going bad.
W. Curtis Preston:So she's constantly putting my soft butter in the refrigerator and I'm like, dammit.
W. Curtis Preston:And then I have to slice it.
W. Curtis Preston:And then I gotta nuke it just so I can spread it and it ticks me off.
W. Curtis Preston:And so then I had this moment, I was like, ghee!
W. Curtis Preston:Ghee can solve this problem.
W. Curtis Preston:and so I was like, I bet Prasanna knows about ghee.
W. Curtis Preston:And so now I've, I ordered.
Prasanna Malaiyandi:And since you got it, how was it?
W. Curtis Preston:You know what it wasn't life-changing, but it did taste good.
W. Curtis Preston:It tastes like butter, obviously.
W. Curtis Preston:Super spreadable.
W. Curtis Preston:it was like butter.
W. Curtis Preston:it was like butta.
W. Curtis Preston:But, there's going to be some moments I'm going to make some toast
W. Curtis Preston:and then I'm going to spread it.
W. Curtis Preston:and, yeah, I'm going to be, I'm going to enjoy some ghee.
W. Curtis Preston:That's all I'm saying.
Prasanna Malaiyandi:Sometimes my wife likes to spread ghee on sourdough bread
Prasanna Malaiyandi:and then just toss it on a skillet and let it like crisp up on both sides.
W. Curtis Preston:Oh yeah.
W. Curtis Preston:Yeah.
W. Curtis Preston:Yeah.
W. Curtis Preston:So by, by the way, is ghee an Indian word.
Prasanna Malaiyandi:I believe it is.
W. Curtis Preston:I've been informed that ghee is from Sanskrit.
W. Curtis Preston:That means sprinkled.
Prasanna Malaiyandi:Interesting.
W. Curtis Preston:Yeah.
Prasanna Malaiyandi:I learn something all the time.
W. Curtis Preston:Yeah, you never know what you're going to learn
W. Curtis Preston:here on the restore it all podcast.
W. Curtis Preston:This is going to prove to be, I think, an interesting discussion.
W. Curtis Preston:There could be arguments.
W. Curtis Preston:We, we might not agree.
W. Curtis Preston:I might not agree with our, guest here, but he certainly has me when
W. Curtis Preston:it comes to experience, he may be, I think he is the guest that we've
W. Curtis Preston:had that has the longest time in IT.
W. Curtis Preston:and, interestingly enough, he's about the same age as me, but he's been
W. Curtis Preston:in IT for 10 years longer than me.
Prasanna Malaiyandi:
Speaker:'cause he wasn't slacking.
Prasanna Malaiyandi:
Speaker:Like you.
W. Curtis Preston:Yeah.
W. Curtis Preston:Yeah.
W. Curtis Preston:He was, I was slacking in high school and college.
W. Curtis Preston:He's been in IT for 40 years.
W. Curtis Preston:You've been at NetApp for 16 years.
W. Curtis Preston:Currently the director of market strategies, welcome to
W. Curtis Preston:the podcast, John/Ricky Martin.
Ricky Martin:Thank you for inviting me.
Ricky Martin:It's a pleasure to be here and talk to you face to face, so to speak.
Ricky Martin:for the first time.
W. Curtis Preston:So to speak, in what stands for face-to-face in the
W. Curtis Preston:COVID world, are you in the bay area?
W. Curtis Preston:I assume,
Ricky Martin:no.
Ricky Martin:I'm a bay area.
Ricky Martin:so I'm in Sydney.
Ricky Martin:So I actually look out over a thing called Shipwrights Bay, which is next
Ricky Martin:to Botany Bay, which is a bay area.
Ricky Martin:so let's go with that.
Ricky Martin:yeah.
W. Curtis Preston:I actually, I have a little story from my visit to Sydney.
W. Curtis Preston:I visited the rock, There was like a plaque, as there usually is.
W. Curtis Preston:I was reading a plaque.
W. Curtis Preston:and I remember it said something like the founding date or the first
W. Curtis Preston:arrival date was like 1770 something.
W. Curtis Preston:I don't remember if it was just before 1776 or just after, but I just
W. Curtis Preston:said, oh, that's really interesting.
W. Curtis Preston:That's close to, the founding date of the U S and I wonder if there's any
W. Curtis Preston:relation to that and this Aussie looked at me and they were like, you're kidding.
W. Curtis Preston:Right?
W. Curtis Preston:No, not kidding.
W. Curtis Preston:You do know that you were a prison colony first and that then you had the big battle
W. Curtis Preston:and you told the Brits to, shove off.
W. Curtis Preston:And so then they sent them here, Oh no, I've taken plenty of American history.
W. Curtis Preston:Literally had no idea that we were a penal colony before Australia
W. Curtis Preston:was .So yay, school system.
W. Curtis Preston:But it's interesting.
W. Curtis Preston:it's sort of a running joke about Australia, being,
W. Curtis Preston:originally a penal colony.
W. Curtis Preston:But that, for some reason, I don't know, we just had better marketing or something.
W. Curtis Preston:I don't know.
Prasanna Malaiyandi:no one wants to acknowledge that.
Prasanna Malaiyandi:I think in the U S.
Ricky Martin:Yeah.
Ricky Martin:You see kind of more up on the, the religious persecution
Ricky Martin:and freedom from that sort of
W. Curtis Preston:Yeah.
W. Curtis Preston:Yeah, we just we'd glossed over that other stuff.
Ricky Martin:History is written by the victors.
W. Curtis Preston:So you and I, interacted somewhere out there in the
W. Curtis Preston:Twitter verse, and this paper that you wrote back in 2018 with a nice long
W. Curtis Preston:name, improving economics and business workloads by using a self-protecting data
W. Curtis Preston:infrastructure, Short title, I think, is backup is dead and NetApp is awesome.
W. Curtis Preston:Maybe I think that's the, that's my version of my reading of this paper.
W. Curtis Preston:How I, how do I do
Ricky Martin:Pretty good.
Ricky Martin:Look at it.
Ricky Martin:It's the whole thinking for that came out of something else I did
Ricky Martin:many years ago, even when I was still working at Legato, when I was
Ricky Martin:basically the APEC guy for all Legato.
W. Curtis Preston:Some listeners don't know.
W. Curtis Preston:So Legato.
W. Curtis Preston:They had Networker, the backup product, which currently is owned
W. Curtis Preston:by Dell, because Legato got bought by EMC, EMC got bought by Dell.
W. Curtis Preston:so Dell Networker used to be
W. Curtis Preston:yeah.
W. Curtis Preston:I, I spent plenty of time, making, Networker backups back in the day.
W. Curtis Preston:So I have my time around Networker.
W. Curtis Preston:So you were actually at Legato, a backup company.
W. Curtis Preston:And you had a presentation.
W. Curtis Preston:I understand.
W. Curtis Preston:That was not very backup friendly.
Ricky Martin:It was called Backup is Evil, have you ever tried to get people
Ricky Martin:to like you when you had those like trade fairs and you've got, everybody's
Ricky Martin:like presenting their wares and things like that, and everybody kind of
Ricky Martin:walks past your stand cause you're talking about backup and let's face
Ricky Martin:it backup is just boring as bat...?
W. Curtis Preston:Yeah.
W. Curtis Preston:Yeah.
Ricky Martin:just, nobody wants to talk about backup.
Ricky Martin:Nobody wants it.
Ricky Martin:It's evil, but it's a necessary evil, but nonetheless, nobody
Ricky Martin:really wants to talk about it.
Ricky Martin:It's like talking about plumbing, right?
Ricky Martin:Nobody cares.
Ricky Martin:Nobody cares about plumbing until you get backed up.
Ricky Martin:*rimshot*
Ricky Martin:oh,
W. Curtis Preston:Yeah.
Ricky Martin:sorry.
Ricky Martin:so in order to get people to come and talk to me, I was taught,
Ricky Martin:saying that backup is evil, right?
Ricky Martin:Because back then even 15, almost 20 years ago, the whole idea about doing
Ricky Martin:full backups, which pulls all of your data across from all of your subsystems,
Ricky Martin:pushes it across your networks.
Ricky Martin:It doesn't just touch every piece of your infrastructure.
Ricky Martin:It treads all over it in great big hobnail boots.
Ricky Martin:If anything's going to break, it's going to break during backup.
Ricky Martin:Okay.
W. Curtis Preston:You're going to, you're going to have.
Prasanna Malaiyandi:true.
W. Curtis Preston:You're going to have to hobnail boots.
W. Curtis Preston:Is that a British phrase?
Ricky Martin:Hobnail boots, yeah.
Ricky Martin:It's a British thing.
Ricky Martin:It's like boots that are so thick that they've got like these nails
Ricky Martin:at the bottom to provide tread,
W. Curtis Preston:oh, okay.
W. Curtis Preston:All right.
W. Curtis Preston:All right.
W. Curtis Preston:I'm with you.
W. Curtis Preston:we, I'm bilingual, by the way, I speak both, both English
W. Curtis Preston:and American, but go ahead.
Ricky Martin:Yeah.
Ricky Martin:So basically, having spent a long time in data centers at three o'clock in
Ricky Martin:the morning, troubleshooting why backup has broken something or something isn't
Ricky Martin:working or why it goes really fast in one direction of the network because
Ricky Martin:the duplexing settings are wrong.
Ricky Martin:But restores go at 76 kilobytes per second.
Ricky Martin:We've all been here.
W. Curtis Preston:Yeah, multiplexing is definitely evil.
Ricky Martin:Yeah.
Ricky Martin:So all of these things, which we do multiplexing and a whole bunch of stuff
Ricky Martin:is done because we are trying to do something which is fundamentally a stupid
Ricky Martin:idea in the first place, which is to take all of the data that we have and copy
Ricky Martin:it across a network to some other device and hope that works on a regular basis.
Ricky Martin:And more to the point, hope that at some stage, when we need to restore all of
Ricky Martin:that stuff, it will, it will somehow work . And actually expect that to work.
Ricky Martin:And what works in IT without testing?
Ricky Martin:Nothing.
Ricky Martin:When was the last time you actually tested, recovering the majority of
Ricky Martin:your infrastructure from your backups?
Ricky Martin:And I would get lots of people going and you might get a bank going, oh,
Ricky Martin:we have to do that twice a year.
Ricky Martin:And I'm going, I bet you didn't restore it from tape.
Ricky Martin:Yeah, that they mumbled and they shuffled and they would look
Ricky Martin:uncomfortable and I would go.
Ricky Martin:So all it is this an insurance policy, right?
Ricky Martin:Backup is just there as a way of protecting you against
Ricky Martin:some form of disaster.
Ricky Martin:Now it's not a cheap insurance policy, the amount of money you spend on backup is.
Ricky Martin:There's a lot of money in data protection.
Ricky Martin:So would you pay for insurance?
Ricky Martin:So you just sit there and you'd go.
Ricky Martin:If you were to try and run that full recovery right in there, how much
Ricky Martin:do you think you would get back?
Ricky Martin:Just pull a figure out of the air and people would say 40% maybe
Ricky Martin:might come back successfully.
Ricky Martin:That's a reasonable number.
Ricky Martin:Okay.
Ricky Martin:I guess what would you do if your wife said if our house burns down.
Ricky Martin:Our insurance policy will cover the house.
Ricky Martin:You get most of the house, she'd go and get a new insurance policy.
W. Curtis Preston:Yeah.
Ricky Martin:And so yet in IT, we all wander around in the back of our heads.
Ricky Martin:We know that the chances of actually successfully recovering is pretty low.
Ricky Martin:And again, I will say this isn't an abstract thing.
Ricky Martin:This is the reality of almost every single person who pays for ransomware.
Ricky Martin:And that's a butt-ton of.
Ricky Martin:Right.
Ricky Martin:It fundamentally doesn't address the problem that we want it to.
Ricky Martin:And that's not to say that backup per se is evil, but the way that most people
Ricky Martin:approach backup is and it just stems from this moving all of your data from one spot
Ricky Martin:across a network to another spot, putting it into something which is meant to be an
Ricky Martin:offline medium, taking that offline medium and putting it into a fireproof safe.
Ricky Martin:Now here's the other thing, all of this tape handling, right?
Ricky Martin:I've seen situations where, what could be the very last copy of a company's
Ricky Martin:data being handed to a guy who gets paid less than the guy at McDonald's to be put
Ricky Martin:into the back of a rusty van, driven over some of the worst roads in the Southern
Ricky Martin:hemisphere, To be put into what you hope is the right environmental conditions.
Ricky Martin:So you sit there and you go, this stuff is good for seven or 20
Ricky Martin:years or whatever the case may be.
Ricky Martin:None of the things which are on side that little label.
Ricky Martin:Yeah.
Ricky Martin:Please keep between this, this humidity and this temperature and this free
Ricky Martin:from vibration of blah, blah, blah, blah, blah, blah, blah, blah, blah.
Ricky Martin:Tape isn't bad, but people don't treat it the way that it needs
Ricky Martin:to be treated in order for it to have the kinds of recoverability
Ricky Martin:that people expect it to have.
Ricky Martin:So I ran a tape recovery business for a while.
Ricky Martin:And about 25% of the tapes that were sent to me to be recovered - and a
Ricky Martin:lot of this was for legal reasons and stuff like that - failed, There were
Ricky Martin:just media errors or things like that.
Ricky Martin:Now, to be fair, there's probably some selection bias there.
Ricky Martin:the reasons why they said probably it's because they
Ricky Martin:couldn't recover it, but still.
W. Curtis Preston:Let me ask you, and by the way, this is the
W. Curtis Preston:conversation that we started with.
W. Curtis Preston:That's what I, now I remember you actually said you had this company, this
W. Curtis Preston:is how you and I first started talking.
W. Curtis Preston:and that's how, and that's how you're here.
W. Curtis Preston:so what does that mean?
W. Curtis Preston:A tape recovery business.
Ricky Martin:So there's a lot of people out there who have tapes that they need
Ricky Martin:to recover data from for other legal discovery or something else that, and
Ricky Martin:it's usually not an operational recovery.
Ricky Martin:It's usually a recovery from say a tape that they no longer
Ricky Martin:have the tape drive for.
Ricky Martin:So my tape recovery business included things like chain of custody,
Ricky Martin:where I would get the tape and I would find the old tape drives.
Ricky Martin:And I would find the old copies of the operating systems than the
Ricky Martin:copies of the backup software.
Ricky Martin:And I would then recover that data onto a piece of removable media and
Ricky Martin:send that back to the data owner.
W. Curtis Preston:and you're saying that a significant portion of the
W. Curtis Preston:time the tape was just worthless?
Ricky Martin:They needed significant amounts of extra handling.
Ricky Martin:So it was media errors, the unrecoverable media error, .Most backup software won't
Ricky Martin:read a tape past that unrecoverable media portion, You actually have to do
Ricky Martin:unnatural things to try and move past there and recover the rest of the data.
Ricky Martin:So I would recover as much as I could from those tapes.
Ricky Martin:And as I said, about 25% of the stuff that I was sent was.
Ricky Martin:Just not there.
Ricky Martin:So this whole keeping tape as an archive medium for 20 years, I just don't.
Ricky Martin:I personally, based upon my experience would never trust that.
Ricky Martin:Because you can't test it!
W. Curtis Preston:Yeah.
W. Curtis Preston:Yeah
Prasanna Malaiyandi:No, this is interesting because we've had some tape
Prasanna Malaiyandi:specialists, like guys who understand all the physical characteristics
Prasanna Malaiyandi:of tape on the podcast as well.
Prasanna Malaiyandi:Joe Jurneke , Mark Lance, talking about like tape and the
Prasanna Malaiyandi:physical properties of tape.
Prasanna Malaiyandi:And at least from what I can gather as a complete tape newbie, it seemed
Prasanna Malaiyandi:yes, there are issues, but there was a lot of resiliency built into tape.
Prasanna Malaiyandi:To handle some of these issues.
Prasanna Malaiyandi:Now, I don't know.
Prasanna Malaiyandi:Maybe if it's a software thing or like you said, maybe some of
Prasanna Malaiyandi:these old tape drives, maybe they weren't handled in the right way.
Prasanna Malaiyandi:And that's why you're seeing some of these issues.
Prasanna Malaiyandi:Maybe it's just the fact you live in the land down under.
Prasanna Malaiyandi:And so that's why there are issues with tape.
W. Curtis Preston:We don't have any of these problems in the Northern hemisphere.
W. Curtis Preston:I'm just saying.
Prasanna Malaiyandi:But my sample size is also very small.
Prasanna Malaiyandi:I don't hang out with people like unlike Curtis.
Prasanna Malaiyandi:I know Curtis you've run into issues with doing restores from tapes, Where
Prasanna Malaiyandi:you've had issues like some tape drives.
Prasanna Malaiyandi:It just doesn't work.
W. Curtis Preston:Yeah.
W. Curtis Preston:My experience was that just tape or no tape, but my experience.
W. Curtis Preston:Over 30 years of working with people with backups.
W. Curtis Preston:Is that 99% of the problem was not the medium.
W. Curtis Preston:It was the person behind the medium, right?
W. Curtis Preston:It was.
W. Curtis Preston:Yeah.
W. Curtis Preston:and I will agree that tape is a problematic medium, right?
W. Curtis Preston:There are some things about it that I don't think are problematic,
W. Curtis Preston:but I do think it's a problematic medium for a long list of reasons.
W. Curtis Preston:You've mentioned some of them like multiplexing, but you talked about
W. Curtis Preston:the environmental control, that 25%.
W. Curtis Preston:you said just not there.
W. Curtis Preston:And again, I do agree with that.
W. Curtis Preston:That's probably some sample bias, right?
W. Curtis Preston:Because you, because you were given all the worst stuff.
W. Curtis Preston:and I also, I agree with Prasanna, like when was this?
W. Curtis Preston:When did you have this tape recovery business?
Ricky Martin:Oh, this was, not long before I joined, NetApp.
Ricky Martin:So that would have been 16 years ago.
Ricky Martin:Right.
Ricky Martin:So, you know, a a lot has happened in sixteen
W. Curtis Preston:So it's not ancient history, but it's not recent either.
Ricky Martin:no,
W. Curtis Preston:that would've been the early LTO days.
Ricky Martin:Yeah, early LTO.
Ricky Martin:I think we were up to LTO-2 around about that point in time.
Ricky Martin:DLT, something or other, I've say tape is only as good as its handlers.
W. Curtis Preston:Hm.
Ricky Martin:And when you think about it, who gets given the
Ricky Martin:job of looking after backup?
Ricky Martin:It is usually the most junior or the person who is like
Ricky Martin:at the end of their career.
Ricky Martin:Generally speaking the care factors of both of those groups of
Ricky Martin:people are usually not the same.
Ricky Martin:They're not your gun SREs that sit there and proactively think about how do I
Ricky Martin:eliminate every possible cause of failure.
W. Curtis Preston:yeah, by the way I fight that issue all the time.
W. Curtis Preston:You know?
W. Curtis Preston:because nobody else wanted to do that job.
W. Curtis Preston:That's how I got my job and I just never got out of it.
W. Curtis Preston:I just, I guess maybe I wasn't smart enough.
W. Curtis Preston:I never got out of backup, but the, and then actually I see
W. Curtis Preston:that towards the end of people's career, I often see them in DR.
W. Curtis Preston:I see them start in backup and end in DR.
W. Curtis Preston:I heard you talk about you talking about the full aspect and we can both
W. Curtis Preston:completely agree that the concept of occasional full backups is stupid.
W. Curtis Preston:we did it back in the day when, when we had to do it, because if you didn't do an
W. Curtis Preston:occasional full, what, how are you going to do a restore with a tape from, you're
W. Curtis Preston:going to do one full and then you're going to do incrementals for seven years.
W. Curtis Preston:You're not going to do that.
W. Curtis Preston:You're going to do, you're going to do an occasional full.
Ricky Martin:Incremental, forever restore.
Ricky Martin:We used to talk about that with Tivoli storage mangler.,
W. Curtis Preston:Yeah.
Ricky Martin:But the thing is that's not to say that doing it occasionally
Ricky Martin:for a really good reason is a bad idea, but using it as your first line of
Ricky Martin:defense is just living in fantasy land because tape has this really wonderful.
Ricky Martin:Is that a tape sitting in a fireproof safe offsite?
Ricky Martin:Right.
Ricky Martin:Is air gapped in genuinely air gapped right?
Ricky Martin:It is safe from hackers, from network, from software failures.
Ricky Martin:It's safe from every known form of data loss, short of a nuclear bomb.
Ricky Martin:And some of them are probably safe against nuclear bombs too ,if they're
Ricky Martin:in the right kind of location.
Ricky Martin:It is this wonderful catch-all thing that protects against
Ricky Martin:all known forms of data loss.
Ricky Martin:The trouble is that tape in that fireproof safe is not suitable
Ricky Martin:for operational recoveries or even really disaster recoveries.
Ricky Martin:it's the last line of defense
W. Curtis Preston:Yeah.
Ricky Martin:Use it for what it's good for.
Prasanna Malaiyandi:You should not be trying to do your user deleted a file.
Prasanna Malaiyandi:Let me try to restore it.
Prasanna Malaiyandi:Oh, wait, I got to go call, recall a tape off premises and do the restore, right?
Prasanna Malaiyandi:It's a last line of defense, like you said, but it should not be your first.
Ricky Martin:No, it shouldn't be.
Ricky Martin:and that's kinda what I put inside these like massive tables, which
Ricky Martin:sort of talk about, what protects against various failure domains
Ricky Martin:and, tape is green across the board.
W. Curtis Preston:Let me ask you, when you look at this paper that it wasn't
W. Curtis Preston:that long ago, when you wrote it, it's when I'm listening to you talk about it.
W. Curtis Preston:I'm hearing a lot of complaints about tape and about things
W. Curtis Preston:that we did because of tape.
W. Curtis Preston:What about.
W. Curtis Preston:disk based systems that aren't doing the things that you talked about?
W. Curtis Preston:So not doing repeated fulls, not doing multiplexing.
W. Curtis Preston:and you they're less susceptible to the environmental stuff that you talked about.
W. Curtis Preston:Generally the same companies that you were talking about before, but just
W. Curtis Preston:not using tape as a primary mechanism.
W. Curtis Preston:How much does that address your concerns?
Ricky Martin:With the right combination of operational procedures and the right
Ricky Martin:combination of technologies, it completely addresses them in a really elegant way.
Ricky Martin:And I'm gonna bring up the whole, a snapshot is a backup, right?
Ricky Martin:Even at NetApp.
Ricky Martin:Now you will find people go, no snapshot's not a backup.
Ricky Martin:We can't say that anymore.
Ricky Martin:A snapshot is a form of backup that protects against
Ricky Martin:certain kinds of failures.
Ricky Martin:So I sit there and say, what are your causes of failure as well?
Ricky Martin:There's user failures.
Ricky Martin:protects against that pretty well.
Ricky Martin:Application failures?
Ricky Martin:Yep.
Ricky Martin:Protects against that pretty well.
Ricky Martin:Array failures?
Ricky Martin:Bow does not, right?
Ricky Martin:Site failures, but bow does not.
Ricky Martin:Metro failures, not really.
Ricky Martin:In order to protect against those things.
Ricky Martin:You then need to combine that with replication to a separate physical device
Ricky Martin:and preferably a separate location.
Ricky Martin:So suddenly applications, Yep..
Ricky Martin:Users.
Ricky Martin:Yep.
Ricky Martin:Arrays.
Ricky Martin:Yup.
Ricky Martin:Sites.
Ricky Martin:Yep.
Ricky Martin:Metro.
Ricky Martin:Yes.
Ricky Martin:A malicious actor with privileged, local access.
Ricky Martin:A hacker that's got your, yeah.
Ricky Martin:Sorry.
Ricky Martin:It deletes a snapshot and goes to your remote site and
Ricky Martin:deletes the snapshot, but then.
Ricky Martin:Bow, it's gone.
Ricky Martin:So you then have to layer on what sometimes referred to as operational air
Ricky Martin:gapping, which is things like WORM, right?
Ricky Martin:Two factor authentication lots of people having to turn the key at the same time.
Ricky Martin:And while it's not a real air gap, that's good enough to stop people
Ricky Martin:using nuclear bombs in the wrong way.
Ricky Martin:Okay.
Ricky Martin:It's good enough to protect your backup.
Ricky Martin:So when you combine those things together and I've got this thing: a geo distributed
Ricky Martin:object store with replication to a separate administrative domain, protects
Ricky Martin:against users, user failures, application, array, site, Metro, malicious actors,
Ricky Martin:and malicious actors with site access.
Ricky Martin:All of these things, that you now have to get to the point where
Ricky Martin:you've got commandos going into both data centers with axes, finding
Ricky Martin:their way to the array and chopping them both apart at the same time.
W. Curtis Preston:But you went from one extreme to the other.
W. Curtis Preston:You went from legato to NetApp.
W. Curtis Preston:I'm talking about all the companies in the middle.
W. Curtis Preston:You went immediately, because I have my issues and by the way,
W. Curtis Preston:I'm a fan of NetApp, right?
W. Curtis Preston:I'm a fan of snapshots.
W. Curtis Preston:I do not call them backup by themselves.
W. Curtis Preston:I call them like a convenience copy.
W. Curtis Preston:I just, it hurts my heart to call them backup Okay.
W. Curtis Preston:Without replication.
W. Curtis Preston:All right.
W. Curtis Preston:we'll just have to agree to disagree on that, but, when we started talking
W. Curtis Preston:about a pure play system, like NetApp, the concern that I have there is if you
W. Curtis Preston:don't change forms at some point, it's that concern of the rolling code problem.
W. Curtis Preston:If something goes wrong with data OnTap, could that take out everything?
W. Curtis Preston:Both the primary and the secondaries, you know, Hacker, but just something
W. Curtis Preston:goes wrong with data OnTap and then poof, all my stuff goes.
W. Curtis Preston:That's why even back when I was like at my height of my NetApp love, I still
W. Curtis Preston:wanted, I wanted, and if Stephen listens to this one, it's gonna, I wanted an
W. Curtis Preston:NDMP backup of the, you know, back in the day, so so we have two extremes,
W. Curtis Preston:I guess my question is, what about the other people that aren't NetApp that
W. Curtis Preston:are doing that are there, they're not doing it because what I heard again, I
W. Curtis Preston:heard you talking about the full copy.
W. Curtis Preston:I heard you talk about multiplexing?
W. Curtis Preston:and I heard you talk about, the environmental concerns.
W. Curtis Preston:If you look at many modern data protection systems, they're not
W. Curtis Preston:doing the repeated fulls are doing block level incremental forever.
W. Curtis Preston:They're not using tape.
W. Curtis Preston:I haven't recommended tape as part of a backup system in.
W. Curtis Preston:I don't know, 15 years more than that.
W. Curtis Preston:the only people left that I know that there are some, uh, Brian, I'm
W. Curtis Preston:talking to you there what's that?
Prasanna Malaiyandi:was going to say Matt over at Spectra.
W. Curtis Preston:yeah.
W. Curtis Preston:So Spectralogic uses tape for backup.
W. Curtis Preston:Of course they do.
W. Curtis Preston:When they got hit by ransomware, attack, they recovered from that ransomware
W. Curtis Preston:attack using their tape backup.
Prasanna Malaiyandi:but wait, I want to just comment on that.
Prasanna Malaiyandi:Some part was recovered from tape.
Prasanna Malaiyandi:A lot of it though, they said they were able to pull back from
Prasanna Malaiyandi:snapshots, which they had on their.
W. Curtis Preston:which, and again, I got no issue with snapshots, I do still
W. Curtis Preston:feel that there is a role of a different system, because earlier you were talking
W. Curtis Preston:about this craziness, you said of the idea of copying it to a different,
W. Curtis Preston:that's the same thing NetApp is doing.
W. Curtis Preston:I'm just saying, do it in a different form.
W. Curtis Preston:Not use NetApp, use something else.
W. Curtis Preston:Just don't do the dumb things that you said, that repeated fulls.
W. Curtis Preston:That's been dumb for, a long time.
W. Curtis Preston:Definitely multiplexing is multi, you talked about it, it was evil.
W. Curtis Preston:It was a necessary evil.
W. Curtis Preston:We had no choice back in the day.
W. Curtis Preston:Multiplexing was the only way we could get the backups to, we could get enough
W. Curtis Preston:data to make the tape drives go fast enough and be at least semi happy.
W. Curtis Preston:but then we all knew that if we were going to do a restore, that
W. Curtis Preston:was not going to be a good day.
W. Curtis Preston:so what do you think about that?
Ricky Martin:As long as you're using some form of minimal replication, like
Ricky Martin:block level incremental, even to just file level incremental and replication.
Ricky Martin:I'm a happy man, Plus WORM.
Ricky Martin:It's a good idea, right?
Ricky Martin:The trouble is I still see customers who have this as an option, still
Ricky Martin:electing to do streaming backups, because that's what they're comfortable with.
W. Curtis Preston:Well that I can't fix.
Ricky Martin:No.
Ricky Martin:So certainly it's that.
Ricky Martin:The other thing I'll also say is that, and I didn't want to turn
Ricky Martin:this into a, an advert for NetApp, but I'm going to work for NetApp
Ricky Martin:and I'm still very keen about the
W. Curtis Preston:we're going to mention Druva at some point.
Ricky Martin:Yeah.
Ricky Martin:Yeah.
Ricky Martin:The, like you talk about pushing this into a different format.
Ricky Martin:That's exactly why the latest versions of what it's basically called snap mirror.
Ricky Martin:It's a very different way of doing things.
Ricky Martin:It's basically taking the snapshots and pushing this off and
Ricky Martin:putting it into an object store.
W. Curtis Preston:I want to hear about that.
W. Curtis Preston:but I just realized that I haven't yet done our standard,
W. Curtis Preston:disclaimer, that Prasanna and I do work for different companies.
W. Curtis Preston:I work for Druva.
W. Curtis Preston:He works for Zoom.
W. Curtis Preston:we're not speaking for either company.
W. Curtis Preston:The opinions here are ours.
W. Curtis Preston:And, please rate this podcast at ratethispodcast.com/restore.
W. Curtis Preston:And if you want to come argue with me, Then have at it, baby.
Prasanna Malaiyandi:Pick a time Curtis will be there.
W. Curtis Preston:Or, if you want to come on and you want to go,
W. Curtis Preston:Curtis, I think backups are amazing.
W. Curtis Preston:I think that John Martin guy was a moron and we want to talk about just
W. Curtis Preston:how much we love backups, whatever man, in this space, cybersecurity,
W. Curtis Preston:ransomware, beer and backups.
W. Curtis Preston:I keep threatening, we gotta do a, do another beer and backups
W. Curtis Preston:episode
Prasanna Malaiyandi:We do have.
W. Curtis Preston:I actually guested on a show that was called beer and bytes.
W. Curtis Preston:That's literally the name of their podcast, beer and
W. Curtis Preston:bytes, and I was required.
W. Curtis Preston:Oh, it was so horrible.
W. Curtis Preston:I had to bring beer to, to
Prasanna Malaiyandi:Oh, poor baby.
W. Curtis Preston:buy by the end of the, by the end they
W. Curtis Preston:were drinking the whole episode.
W. Curtis Preston:By the end of the episode, I was a little loopy.
W. Curtis Preston:yeah.
W. Curtis Preston:So tell me about this, the SnapMirror to object
Ricky Martin:It's been called a few things within NetApp, sometimes snap
Ricky Martin:mirror to cloud, It's basically using, the same block level replication technology,
Ricky Martin:you would expect to find it inside of, a NetApp array to go from it's called
Ricky Martin:snap mirror, To go from array to array where we're keeping like an ONTAP
Ricky Martin:file system with figuring out what's what the change has been made there.
Ricky Martin:And we ship across like a blob of blocks and we apply that transactionally to the
Ricky Martin:other ONTAP file system and away we go.
Ricky Martin:Okay.
Ricky Martin:And so they're in, you have your problem, but what happens if that blob of stuff
Ricky Martin:includes a level of corruption that screws up the file system at the other end.
Ricky Martin:Now I can say I have never heard of that happening, but just cause
Ricky Martin:it's never happened doesn't mean it couldn't theoretically happen.
Ricky Martin:Again, what this does is this replicates rather than to another
Ricky Martin:NetApp array, it replicates directly through to an S3 object store.
Ricky Martin:So we're able to apply these blobs of changes directly into an S3 object, and
Ricky Martin:then we can run through and grab that stuff and re-present that as a filesystem.
Ricky Martin:In fact, we can even do incremental restores from that into some other areas.
Ricky Martin:So we've actually satisfied that I'll call it an objection, fair, you know why.
Ricky Martin:Yeah.
Ricky Martin:If you're not changing the nature of the format in which you're storing the
Ricky Martin:data, you have this potential problem.
Ricky Martin:What neatly addresses that?
Ricky Martin:The other thing that preserves a lot of the goodness of array to array mirroring,
Ricky Martin:the other thing it does, and this is the thing where the NetApp approach either.
Ricky Martin:Better than using,
W. Curtis Preston:third party.
Ricky Martin:a third party because all of those things drop these
Ricky Martin:things into backup formats, which are meant primarily for recoverability.
Ricky Martin:Remember how I talked about what works in IT that you can't test.
Ricky Martin:Can you easily test the fact that stuff is there and is working?
Ricky Martin:Can you sit there and say, is there 90%, a hundred percent, how quickly can I
Ricky Martin:go through and start using that data?
Ricky Martin:Again, all of those things typically require some form of recovery process.
Ricky Martin:And if we're talking about.
Ricky Martin:80 90% of your estate, which has just been crypto locked.
Ricky Martin:We're now talking about 10 to 20 days worth recovering this stuff
Ricky Martin:back onto something which is primary.
Ricky Martin:If that copy is on something, which is primary, right?
Ricky Martin:This allows us to use this replica, not just for data protection, but we
Ricky Martin:can use it for, compliance, checking.
Ricky Martin:It's a usable copy of the data and more to the point it's a testable copy of that.
Prasanna Malaiyandi:full disclosure.
Prasanna Malaiyandi:I used to work for NetApp on replication products.
Prasanna Malaiyandi:I love the technology, But I'm going to start poking a little, a few holes.
Prasanna Malaiyandi:Ricky.
Prasanna Malaiyandi:one of the things though, right?
Prasanna Malaiyandi:And this sounds very familiar.
Prasanna Malaiyandi:I'm sure you're aware of it way back when there was snap mirror to tape where
Prasanna Malaiyandi:people want it to be able to dump to tape and ship it off places and restore it.
Prasanna Malaiyandi:Except it looks like this is a much, much better designed, right?
Prasanna Malaiyandi:With the incremental forever being able to connect and being able
Prasanna Malaiyandi:to instantly access your files.
Prasanna Malaiyandi:It sounds great.
Prasanna Malaiyandi:But just going back to Curtis.
Prasanna Malaiyandi:One of the challenges is when you have a single vendor, Even though you are
Prasanna Malaiyandi:changing the format and writing it to object store, I would claim underlying
Prasanna Malaiyandi:it's still the NetApp file system.
Prasanna Malaiyandi:Correct.
Ricky Martin:No,
Prasanna Malaiyandi:You're not writing out what waffle
Prasanna Malaiyandi:or anything else like that.
Ricky Martin:there is no waffle on the other end.
Prasanna Malaiyandi:Okay.
Prasanna Malaiyandi:So then I back
W. Curtis Preston:waffle on this answer,
Prasanna Malaiyandi:Okay.
Ricky Martin:no, not waffling on this answer.
Prasanna Malaiyandi:Okay.
Ricky Martin:The data structure, which has by necessity has to be different
Ricky Martin:because the semantics for being able to access this stuff, You can't write over.
Ricky Martin:Yeah.
Ricky Martin:Just think about it.
Ricky Martin:You've got get put in a bunch of other things.
Ricky Martin:I would be happy to arrange.
Ricky Martin:For anybody, who's interested to talk to a real expert on this, but if you
Ricky Martin:think about it coming back to snap mirror to tape, which was a similar kind of
Ricky Martin:thing, it was still basically a stream of the waffle file system on tape.
Ricky Martin:Then there was another advance which was called advanced tape, which is
Ricky Martin:still there right now, which basically allows you to do image-based backups
Ricky Martin:of things because let's face it.
Ricky Martin:Just doing file-based backups.
Ricky Martin:I've got to say like in the old days doing a high density file
Ricky Martin:system backup, which is oh no, God, no, please don't make me do this.
Ricky Martin:and then the answer was image-based backups and I'd go
Ricky Martin:back to the legato, which is when I started my backup was evil.
Ricky Martin:The thing for that was something called, God.
Ricky Martin:Do you remember bud tool and they're there I'm Celestra?
Ricky Martin:And that was an image based dump that was used by 30 that's where this started.
Ricky Martin:That's what recently our backup was able to start it because like it
Ricky Martin:was back up, kill your location, did nasty things to your server
Ricky Martin:and blah, blah, blah, blah, blah.
Ricky Martin:this snap mirror to tape was that.
Ricky Martin:And then we then with advanced tape which allows us to do these.
Ricky Martin:This then takes that and logically improves it in a number of areas.
Ricky Martin:So while it's really new technology, it's only been available in ONTAP for,
Ricky Martin:I think the last two or three versions.
Ricky Martin:it's the foundation for so much of NetApp's ongoing data protection.
Ricky Martin:cloud backup, or you may not know cloud backup, which is that's based
Ricky Martin:upon this same technology, right?
Ricky Martin:Because it's going through and pushing stuff directly onto S3.
Prasanna Malaiyandi:Gotcha.
Prasanna Malaiyandi:And I do like the functionality that you mentioned, where you can instantly
Prasanna Malaiyandi:Mount that copy from the cloud.
Prasanna Malaiyandi:And start verifying, do I have my data there?
Prasanna Malaiyandi:Because Curtis and I have chatted with some folks in the past with
Prasanna Malaiyandi:ransomware recovery half the time is just figuring out like where your data
Prasanna Malaiyandi:is doing the actual recovery process.
Prasanna Malaiyandi:Being able to quickly access it like you mentioned, and not have
Prasanna Malaiyandi:to wait for a tape to come back.
Prasanna Malaiyandi:It's critical in being able to get your environment back up and running, identify
Prasanna Malaiyandi:which applications you need to bring back first and where that data exists.
Prasanna Malaiyandi:So
Ricky Martin:Yeah.
Ricky Martin:So if
Ricky Martin:you can,
Prasanna Malaiyandi:that instant access functionality.
Ricky Martin:absolutely.
Ricky Martin:If you can Mount that, that recovery data store.
Ricky Martin:Yeah.
Ricky Martin:And access it, just using standard tools, not the backup tool, Because that's
Ricky Martin:the other problem you have is when you put this into a relative proprietary
Ricky Martin:format, how do you access that easily?
Ricky Martin:If this is just mountable via NFS or SMB share, especially for like
Ricky Martin:high-density file system backups, you can start running statistical
Ricky Martin:things where you just like, run like a chaos monkey of can I see that file?
Ricky Martin:Can I see this file?
Ricky Martin:I should be able to see this thing here.
Ricky Martin:Okay.
Ricky Martin:Statistically be happy that the stuff which I should be
Ricky Martin:able to recover, I can recover.
Ricky Martin:It takes away that testability problem that I talked about before.
Ricky Martin:And so you can always sit there and have a really high degree of confidence
Ricky Martin:that if there is a disaster, I can recover this stuff and you can run
Ricky Martin:tests on it, but you can sit there and say, how long would it take me?
Ricky Martin:And I can tell you with.
Ricky Martin:well-designed NetApp infrastructure, right?
Ricky Martin:That recovering four petabytes worth of data can literally take you 10
Ricky Martin:minutes, Going back to that previous recovery point, and you can test it
W. Curtis Preston:Oh, you mean because you're not actually recovering the
W. Curtis Preston:entire four petabytes you're bringing back the bits that are different.
Ricky Martin:All you have to do is change the metadata.
Ricky Martin:And suddenly you got a view, the dead of what it looked like an hour ago, two
Ricky Martin:hours ago, five days ago, 10 days ago, whatever the case may be, and you don't
Ricky Martin:have to run through a recovery process.
Ricky Martin:So in the same way that remember we only back up so that we can recover,
Ricky Martin:but recovery is what it's all about.
Ricky Martin:So if I can recover, like in 10 minutes, Yeah, that, that might
Ricky Martin:be look, let's say it's an hour.
Ricky Martin:Once you throw on all the other appropriate operational procedures,
Ricky Martin:compare that to trying to recover a petabytes worth of stuff from any
Ricky Martin:third party backup tool before you can start using it, There's a week long.
W. Curtis Preston:Yeah, for, for what it's worth, at least, our position at
W. Curtis Preston:Druva, when you look at especially a high density file system or anything like that,
W. Curtis Preston:your primary protection mechanisms should be something like what you described,
W. Curtis Preston:because there's just no beating that.
W. Curtis Preston:I don't care how I don't care how fast you are bringing four petabytes
W. Curtis Preston:back from anywhere is going to take.
W. Curtis Preston:and, I, I do think, you talked a lot about testing and I couldn't agree more.
W. Curtis Preston:I, and I remember back, the first company that really did this to
W. Curtis Preston:put this on the map was Veeam.
W. Curtis Preston:They had the sure backup and still have this sure backup, functionality
W. Curtis Preston:that you automatically could specify.
W. Curtis Preston:some systems that they would recover, they don't recover it.
W. Curtis Preston:They just bring it up.
W. Curtis Preston:That's the key, just the same thing that you talked about.
W. Curtis Preston:And I know that with Druva we have the ability to do that on certain workloads.
W. Curtis Preston:I know that we can do that Dr.
W. Curtis Preston:and I know that there are other, some other systems, and by the
W. Curtis Preston:way, we can all agree that.
W. Curtis Preston:the more testing you can do the better number one and number two,
W. Curtis Preston:the easier it is to do testing, the easier, the more you're going to do it.
W. Curtis Preston:I remember, when I worked at the.
W. Curtis Preston:Um, we had to do, I think you, you said about the every six months, right?
W. Curtis Preston:The re the recovery thing, and no, we did not do the whole data center.
W. Curtis Preston:We picked a couple of critical workloads and we would do it.
W. Curtis Preston:And it was a huge pain in the butt that involved 50 people
W. Curtis Preston:that all had to be there, whether they needed to be there or not.
W. Curtis Preston:And it was horrible.
W. Curtis Preston:It was a horrible process.
W. Curtis Preston:It was like, that was like traumatic.
W. Curtis Preston:and, I know I've said it on the podcast before, but one of the things that.
W. Curtis Preston:I would write the documentation, but somebody else had to read the
W. Curtis Preston:documentation and execute the recovery.
W. Curtis Preston:And, and I wasn't supposed to help write it.
W. Curtis Preston:Like the idea was that I got blown up in the whatever, and so they have to do it.
W. Curtis Preston:And I think that's a great way to do recovery, but we need to do
W. Curtis Preston:more automated recovery testing.
W. Curtis Preston:we pushed this at Druva where, we're like, if something is a critical
W. Curtis Preston:workload, by the way, I have a similar thought to you, just obviously
W. Curtis Preston:with a deal, with a different tool.
W. Curtis Preston:So if it's a workload that's critter.
W. Curtis Preston:That needs to come back.
W. Curtis Preston:If it's something that if you get a ransomware attack and it needs to
W. Curtis Preston:be up now, then it should be covered by you should pre-configure it with
W. Curtis Preston:our DR service, And the only way to do DR, in my opinion, today for most
W. Curtis Preston:people is to do it in the cloud.
W. Curtis Preston:Number one, and number two, to do it in advance, meaning that if it's
W. Curtis Preston:really important, it needs to be just like you talked about with your
W. Curtis Preston:image that you can just Mount it.
W. Curtis Preston:Cause that's what Veeam did..
W. Curtis Preston:Right that's why Veeam can do that automated testing is because they could
W. Curtis Preston:just Mount the image and we can do the same thing with some of our workloads.
W. Curtis Preston:but they, and then basically if you could pre restore the data so that, because
W. Curtis Preston:if you are, I'll bring up, go back to the days of when EMC was EMC, right?
W. Curtis Preston:One of their slogans was if you're reaching for a tape, you're dead.
W. Curtis Preston:That was one of their things that you th the recovery needs
W. Curtis Preston:to have been done already.
W. Curtis Preston:They were pushing like SRDF and whatnot.
W. Curtis Preston:But you and I agree on that, that you, if you're going to have to pull down
W. Curtis Preston:some giant blob of data, whether it's from a bunch of tapes from a big, dedupe
W. Curtis Preston:disk array whether it's from a bunch of object storage in the cloud, If in
W. Curtis Preston:order to get back up and running, you have to bring a bunch of stuff back
W. Curtis Preston:and you haven't already restored it.
W. Curtis Preston:You're going to be in a world of hurt, no matter what technology you're in.
Ricky Martin:Yeah, absolutely.
Ricky Martin:The biggest downfall from our perspective NetApp's approach is that
Ricky Martin:it requires to have your data sitting.
Ricky Martin:On tap on waffle as your primary data source and market share says that's about
Ricky Martin:15% of the enterprise data out there.
Ricky Martin:There's as I would love it to be a hundred percent, but realistically
Ricky Martin:it will never be a hundred percent,
W. Curtis Preston:I bet your stock price would be a lot better.
Ricky Martin:Oh shit.
Ricky Martin:Yeah.
Ricky Martin:But, and so you have these other tools and heterogeneous backup is
Ricky Martin:still an incredibly important thing.
Ricky Martin:My real beef is about people.
Ricky Martin:My problem is not the competitors, my problem is people's practices
Ricky Martin:and the way in which they define their requirements based upon
Ricky Martin:like tape based backup thinking.
W. Curtis Preston:It's yeah,
Prasanna Malaiyandi:Yeah, they haven't modernized.
W. Curtis Preston:yeah.
W. Curtis Preston:That, so a good friend of mine, Reed, who also listens to the podcast and,
W. Curtis Preston:he talked, he tells a story about the, the grandmother that, that.
W. Curtis Preston:Always cut.
W. Curtis Preston:What was, it was something about cutting the ends off the meat, low for something.
W. Curtis Preston:And, and she'd done it for years.
W. Curtis Preston:And when you find out it's why did, why do we cut the ends off to the meatloaf?
W. Curtis Preston:And it's because back in the day, it wouldn't fit in the pan.
W. Curtis Preston:If he didn't do that.
W. Curtis Preston:But she's still doing it, but she's still doing it right.
W. Curtis Preston:30 years later.
W. Curtis Preston:And that is absolutely true that there are so like the full backup is the best
W. Curtis Preston:example of what you're talking about.
W. Curtis Preston:People that no longer use tape.
W. Curtis Preston:So even if you're not using a tool like Druva, Veeam, Rubrik, Cohesity.
W. Curtis Preston:These are all incremental forever technologies that they
W. Curtis Preston:were built to be that way.
W. Curtis Preston:A lot of the other products, what they've done, they weren't built to be that way.
W. Curtis Preston:They, but they've invented the concept of the, what's it called?
W. Curtis Preston:Prasanna.
W. Curtis Preston:Synthetic full.
W. Curtis Preston:Thank you.
W. Curtis Preston:This synthetic full.
W. Curtis Preston:So at least migrate to a synthetic full.
W. Curtis Preston:if your product needs an occasional full, if you can do
W. Curtis Preston:it synthetically, please do that.
Prasanna Malaiyandi:don't send all the bits over the
W. Curtis Preston:if you're still, I think if you're still using a product
W. Curtis Preston:that requires an occasional full.
W. Curtis Preston:. Sorry, and I, I know it's gonna make me
W. Curtis Preston:but if you're still using a backup product that requires an occasional
W. Curtis Preston:full in today's world, I agree with you.
W. Curtis Preston:but that's, I've always said, you were, you with your
W. Curtis Preston:presentation back with legato.
W. Curtis Preston:I had a similar presentation where I was like full backups are stupid.
W. Curtis Preston:I remember having a guy had a presentation of It was like 10 things
W. Curtis Preston:wrong with most people's backups.
W. Curtis Preston:And one of them was that they were doing full backup.
W. Curtis Preston:This was back, they were still doing weekly fulls when they could go onto
W. Curtis Preston:monthly or quarterly foals and use.
W. Curtis Preston:Since, it's stopped doing lake folds.
W. Curtis Preston:You did that with tape because you had no other choice.
W. Curtis Preston:and they,
Prasanna Malaiyandi:it makes me feel good, Curtis
W. Curtis Preston:what's
Prasanna Malaiyandi:it when they do it.
Prasanna Malaiyandi:That's probably what they're thinking.
Prasanna Malaiyandi:It's the way
Prasanna Malaiyandi:that things.
W. Curtis Preston:what's really hilarious is they do that weekly full.
W. Curtis Preston:Despite the impact that has on the production environment.
W. Curtis Preston:And then they store it on a D duplication disk array.
Ricky Martin:And know what drives me nuts is even when people have
Ricky Martin:a really easy option to go from fulls to using something modern.
Ricky Martin:And I can't give out the answers, but I have seen how many NetApp
Ricky Martin:customers still use NDMP dump.
Ricky Martin:And
Prasanna Malaiyandi:Yeah.
Ricky Martin:please stop on me.
Ricky Martin:We have
Ricky Martin:this really easy.
W. Curtis Preston:I remember being a fan of NDMP at one point, It
W. Curtis Preston:had its purpose at one point that purpose I think is, has, it's past
Ricky Martin:And yet, And so I think might my whole thing is that backup.
Ricky Martin:Don't think about doing this backup system, build a recovery system.
Ricky Martin:Think about what your most likely failure scenarios are.
Ricky Martin:Think about failure domains, users, arrays, sites.
Ricky Martin:In fact, a site is more likely to fail than an array is these days,
Ricky Martin:I've seen more arrays and data centers taken out from plumbing
Ricky Martin:accidents than I've arrays go down.
W. Curtis Preston:you have a good, viewpoint to, to have that data.
W. Curtis Preston:That's interesting that's a, that's an interesting statement.
Ricky Martin:Yeah, because you sit there and go, what
Ricky Martin:happens if the array goes down?
Ricky Martin:Arrays.
Ricky Martin:this isn't just NetApp arrays.
Ricky Martin:NetApp arrays are really awesome, but arrays these
Ricky Martin:days are pretty damn reliable.
Ricky Martin:The state of the art is that they stay up for five years, unless
Ricky Martin:you need to reboot them for.
Ricky Martin:Unless the, I'm not here to be mean about the competition.
Ricky Martin:There are some that were really awful, not NetApp arrays, for the most part,
Ricky Martin:when an array goes down, it's because there's been a messy power center failure.
Ricky Martin:And then something comes up in certain arrays.
Ricky Martin:Don't come back up online, all the NetApps ones do.
Ricky Martin:Oh, there's been a plumbing accident where it's just seriously,
Ricky Martin:I see a toilet breaks upstairs.
Ricky Martin:the data center floor is underneath that.
Ricky Martin:And you've got water dripping down the racks and it's oh shit.
Prasanna Malaiyandi:Or someone forgot to change the battery on
Prasanna Malaiyandi:a generator fail over switch.
Ricky Martin:yep.
Ricky Martin:Or somebody tested the generator fail over and didn't realize that
Ricky Martin:all the PDUs were wired up wrong.
Ricky Martin:I.
W. Curtis Preston:that is why we test things, but it, it's cause we, our
W. Curtis Preston:last guest w w did we have a guest?
W. Curtis Preston:It was darn it.
W. Curtis Preston:I totally, we had our last guest
Prasanna Malaiyandi:Yeah, He He basically tried to do an exchange recovery
W. Curtis Preston:That's right.
W. Curtis Preston:It was exchange recovery.
Prasanna Malaiyandi:and it didn't work.
Prasanna Malaiyandi:And he basically had to go through and luckily it was a test environment.
Prasanna Malaiyandi:Not all the pressure of people yelling at you, but he's yeah, we have to figure
Prasanna Malaiyandi:out how to do like test our restores.
Ricky Martin:Yeah, keep in mind, Chernobyl was because
Ricky Martin:of infrastructure testing
W. Curtis Preston:I did not know that.
Ricky Martin:wrong.
W. Curtis Preston:No, you know what?
W. Curtis Preston:I did know that I never, I guess I never really thought about.
W. Curtis Preston:I never put it in that context.
W. Curtis Preston:Cause I, according to the movie I saw the movie, that's literally the
W. Curtis Preston:extent that I know of Chernobyl, that, and a bunch of people are currently
W. Curtis Preston:blockading it from Russian soldiers.
W. Curtis Preston:but, which based on what I saw in the movie sounds like a good idea,
Prasanna Malaiyandi:Yeah.
W. Curtis Preston:but stay away, don't touch the building.
W. Curtis Preston:Don't touch the glowing building.
Ricky Martin:I It's not a laughing matter, but yeah, it's
W. Curtis Preston:Yeah.
Ricky Martin:testing is doing, making this infrastructure testable I'm
Ricky Martin:really cause I used to be a software developer and test driven development.
Ricky Martin:Is how you do things.
Ricky Martin:And if we think about all the things about creating agile data infrastructures or
Ricky Martin:becoming into the cloud and all the rest of the stuff, this is about how do we make
Ricky Martin:what we do testable and incremental and small changes and continuous development,
Ricky Martin:continuous operations, traditional backup approaches, which depend upon I
Ricky Martin:have to make sure my weekly full is done before I begin my change management.
Ricky Martin:And I have to allow for at least nine hours.
Ricky Martin:to recover my environment in case it goes wrong.
Ricky Martin:Means I can only do changes on the weekend.
Ricky Martin:That means you get 50 change windows a year.
Ricky Martin:And I ask people how many change windows is 50 change windows enough for you to
Ricky Martin:get all the stuff you need to get done.
Ricky Martin:And they go nuts.
Ricky Martin:I said, how many do you need?
Ricky Martin:They go about 300.
Ricky Martin:I go.
Ricky Martin:So that means you have to be able to not just back up but recover
Ricky Martin:more or less instantaneously.
Prasanna Malaiyandi:Yeah.
Ricky Martin:And you need to be able to sit there and rehearse all of this stuff.
Ricky Martin:Yeah.
Ricky Martin:During the day when everybody is there, not like how was it Chernobyl.
Ricky Martin:Everybody had gone home and all the good people were sleeping.
Ricky Martin:You need to test all that stuff, make sure that it works And then
Ricky Martin:you rehearse that you script it, you test it, you make sure it works.
Ricky Martin:And then you execute that because I've been in situations where people did
Ricky Martin:an SAP upgrade the wrong slow way.
Ricky Martin:And the SAP upgrade was going to take 17 hours, but they couldn't stop.
Ricky Martin:Because that it just starts it.
Prasanna Malaiyandi:Uh,
Ricky Martin:mean?
Ricky Martin:And the thing is, if you sit there and you rehearse that on a test dev copy,
Ricky Martin:which you spin up instantaneously, and you can do this in the cloud or wherever
Ricky Martin:you want to do it, then something going to, oh, no, don't do that.
Ricky Martin:Let's do it this other way.
Ricky Martin:And that's my entire thing about that long white paper is I'm trying to make the
case:think about failure scenarios, think about operational efficiencies, right?
case:How do you support the business in doing what it is they want to do?
case:Because the reason why the backup is given to the most junior guy, why
case:it never really gets enough funding to do a really proper job, or if it
case:does, it happens once every three years and becomes broken within four
case:months, Is because it is not aligned.
case:In fact, it's antithetical to most business level objectives.
case:It's a boat anchor on change management.
case:It can be so much more.
case:In fact, the backup can provide the test dev copies to
case:accelerate change That's what.
case:Really long paper is all about it's please stop thinking.
case:Don't use tape or traditional backups or any other form of backup for anything that
case:is not really designed to be good for yes.
case:Use tape to make off-site copies and put them in your fireproof safe
case:because you know what it's still is proof against nuclear bombs
case:and all the rest of that stuff.
case:that's a good thing to have at least one copy on tape.
case:Don't build your entire thinking around it.
case:Build your thinking around business requirements.
case:Sorry.
case:There's my rant.
W. Curtis Preston:no, I get it.
W. Curtis Preston:Yeah.
W. Curtis Preston:I do still think that your primary beef is with a tape based system and a system
W. Curtis Preston:that works like a tape based system.
W. Curtis Preston:Rather than backup itself.
W. Curtis Preston:and to me, backup is a broad tent, that
Prasanna Malaiyandi:
Speaker:call it data protection.
Ricky Martin:Yes.
W. Curtis Preston:I'm okay with that.
W. Curtis Preston:Yeah.
W. Curtis Preston:And it includes the things that you do.
W. Curtis Preston:It includes DDP, right?
W. Curtis Preston:It includes, a bunch of things and many of which I'm not a fan of, but
W. Curtis Preston:they still meet the basic definition.
W. Curtis Preston:and, this hasn't been nearly as painful as I thought it might've been.
Ricky Martin:Backup is evil is just clickbait.
W. Curtis Preston:Yeah.
W. Curtis Preston:Yeah.
W. Curtis Preston:that's, what's going to have to be the title of this episode.
W. Curtis Preston:It's going
Prasanna Malaiyandi:there you
W. Curtis Preston:be so sa Sosa is Ricky Martin.
W. Curtis Preston:All right.
W. Curtis Preston:thanks for coming on, for coming on the podcast.
Ricky Martin:You're welcome.
Ricky Martin:It's been a pleasure.
Ricky Martin:It really has.
Ricky Martin:No, I don't really get to talk to many people about.
W. Curtis Preston:And they don't let you out much over there.
W. Curtis Preston:and persona, I know you didn't get a word, too many words in edgewise
Prasanna Malaiyandi:That's okay.
W. Curtis Preston:between me and Ricky.
Prasanna Malaiyandi:I'm glad I did not have to tell you
Prasanna Malaiyandi:built to go to your corners.
Prasanna Malaiyandi:So that was a plus.
W. Curtis Preston:That's why you were going to be the, the, what do you call it?
W. Curtis Preston:the moderator.
Prasanna Malaiyandi:Yep,
W. Curtis Preston:This was, this wasn't so bad.
W. Curtis Preston:I will say in the beginning when he was just really railing against backup, my
Prasanna Malaiyandi:I could feel your blood boiling from here.
W. Curtis Preston:It hurt a little bit.
W. Curtis Preston:I was like, oh, but I was like, I've said a lot of the same things.
W. Curtis Preston:It's just,
Prasanna Malaiyandi:Yep.
W. Curtis Preston:yeah.
W. Curtis Preston:Anyway, so thanks.
W. Curtis Preston:thanks for not saying much
Prasanna Malaiyandi:Yeah.
Prasanna Malaiyandi:It was entertaining.
Prasanna Malaiyandi:It's all good.
W. Curtis Preston:And, thanks to the listeners.
W. Curtis Preston:we only do this for you.
W. Curtis Preston:we do it so Ricky can have somebody to talk to and, we, make sure
W. Curtis Preston:to subscribe to the podcast, so that you can restore it all.
Listen On
