Check out our companion blog!
May 6, 2024

Cloud Disaster Recovery: Lessons from Failures

Capping our series on cloud disasters is this one on cloud disaster recovery. In this episode, we review the lessons we learned from discussing 10 cloud disasters. We talk about the critical strategies and techniques to ensure your data is protected and recoverable in the event of a disaster in the cloud. From the basic 3-2-1 rule to the pitfalls of solely trusting your cloud provider for proper disaster recovery, we dive into real-world examples and expert insights to help you build a robust cloud disaster recovery plan for your cloud data.

Discover the importance of regular testing, the role of third-party backup solutions, and the key considerations for choosing a reliable cloud provider. Whether you're a small business or a large enterprise, this episode provides actionable advice to enhance your cloud disaster recovery posture and maintain business continuity in the face of unexpected disruptions. Tune in now and learn how to safeguard your valuable data in the cloud era.

Transcript

Speaker:

W. Curtis Preston: Welcome to the exciting final episode in

 

 


Speaker:

our series on cloud disasters.

 

 


Speaker:

During which we conclusively answered the question.

 

 


Speaker:

Do I have to back up my data in the cloud?

 

 


Speaker:

The answer is an unequivocal.

 

 


Speaker:

Yes.

 

 


Speaker:

Not having a good backup and disaster recovery plan for your data in

 

 


Speaker:

the cloud is nothing short of an existential threat to your company.

 

 


Speaker:

Two out of the 10 companies we covered in this series immediately went out

 

 


Speaker:

of business after what happened.

 

 


Speaker:

And another killed off an entire line of business.

 

 


Speaker:

The cloud is an amazing place and I consider myself an

 

 


Speaker:

advocate, but it is not magic.

 

 


Speaker:

It is just someone else's computer.

 

 


Speaker:

In this episode, we will summarize the lessons that we learned and the major

 

 


Speaker:

arguments that we settled from our coverage of no less than 10 disasters.

 

 


Speaker:

If you're just joining us, hopefully this episode will wet your appetite

 

 


Speaker:

enough to have you go back and listen to, or watch those other episodes.

 

 


Speaker:

We had a good time making them.

 

 


Speaker:

And we also learned a lot.

 

 


Speaker:

By the way, if you don't know who I am, I'm w Curtis press an AKA Mr.

 

 


Speaker:

Backup.

 

 


Speaker:

And I've been passionate about backup and recovery for over 30 years.

 

 


Speaker:

Ever since I had to tell my boss that we had no backups of the

 

 


Speaker:

production database, we just lost.

 

 


Speaker:

I don't want that to happen to you.

 

 


Speaker:

And that's why I do this on this podcast.

 

 


Speaker:

We turn unappreciated backup admins into cyber recovery heroes.

 

 


Speaker:

This is the backup wrap up.

 

 


Speaker:

W. Curtis Preston: Welcome to the show.

 

 


Speaker:

I'm w Curtis Preston, AKA, Mr.

 

 


Speaker:

Backup, and I have with me my trauma sympathizer Prasanna

 

 


Speaker:

Malaiyandi, how's it going?

 

 


Speaker:

Persona.

 

 


Prasanna Malaiyandi:

I am good, Curtis.

 

 


Prasanna Malaiyandi:

Yeah.

 

 


Prasanna Malaiyandi:

Um, it's interesting of all the things you can complain about or that are

 

 


Prasanna Malaiyandi:

wrong in this world for you, the biggest issue is different operating

 

 


Prasanna Malaiyandi:

systems using back slashes differently.

 

 


Prasanna Malaiyandi:

W. Curtis Preston: Y.

 

 


Prasanna Malaiyandi:

Yeah, so.

 

 


Prasanna Malaiyandi:

It's because I'm being forced to operate a Windows based net backup server.

 

 


Prasanna Malaiyandi:

I.

 

 


Prasanna Malaiyandi:

And I only know how to work in like Unix land, like when I'm doing automation

 

 


Prasanna Malaiyandi:

and scripting and stuff, which is the only way I know how to do anything.

 

 


Prasanna Malaiyandi:

Right.

 

 


Prasanna Malaiyandi:

I, I, I don't understand.

 

 


Prasanna Malaiyandi:

I, I guess people, if they grew up in Windows Land, they learn PowerShell.

 

 


Prasanna Malaiyandi:

They learn, you know, batch programming and what I didn't learn all that stuff.

 

 


Prasanna Malaiyandi:

I learned shell scripting.

 

 


Prasanna Malaiyandi:

And so when I want to administer a Windows-based system, the first

 

 


Prasanna Malaiyandi:

thing I do is install siggu so that I have a Unix-based environment.

 

 


Prasanna Malaiyandi:

But when I'm interacting with, um.

 

 


Prasanna Malaiyandi:

Files.

 

 


Prasanna Malaiyandi:

Right.

 

 


Prasanna Malaiyandi:

Um, so sig one is nice in that it translates all the file directory names

 

 


Prasanna Malaiyandi:

and everything into slashes, right?

 

 


Prasanna Malaiyandi:

So it's, I just have to, um, sort of figure that out.

 

 


Prasanna Malaiyandi:

And basically, like Deco backslash becomes slash sig drive slash d, so

 

 


Prasanna Malaiyandi:

you just sort of, that's not too bad.

 

 


Prasanna Malaiyandi:

But in the case of net backup, I'm doing things like.

 

 


Prasanna Malaiyandi:

I am running commands that output file names that sometimes are in Windows

 

 


Prasanna Malaiyandi:

format and sometimes are in Unix format.

 

 


Prasanna Malaiyandi:

And, um, the, what I've recently discovered is that, and it took me,

 

 


Prasanna Malaiyandi:

it took me literally just banging my head into the wa into the wall

 

 


Prasanna Malaiyandi:

a few times to figure this out.

 

 


Prasanna Malaiyandi:

Is that, um.

 

 


Prasanna Malaiyandi:

Back slashes, behave differently.

 

 


Prasanna Malaiyandi:

Um, you know, like I wanna massage them and the first thing I wanna do

 

 


Prasanna Malaiyandi:

is change 'em and four slashes, right?

 

 


Prasanna Malaiyandi:

So that I can, so that they're not gonna be escaping everything in my unit.

 

 


Prasanna Malaiyandi:

She script, right?

 

 


Prasanna Malaiyandi:

Um, they behave differently on like in standard in then they do in a file.

 

 


Prasanna Malaiyandi:

So if I take the output of a command, I.

 

 


Prasanna Malaiyandi:

Such as BPPL include, which is, you know, show me the files that are in this policy.

 

 


Prasanna Malaiyandi:

Uh, and I pipe that directly into a, uh, you know, into a series of

 

 


Prasanna Malaiyandi:

pipes to massage the, the output, uh, that backslash behaves differently.

 

 


Prasanna Malaiyandi:

I have to escape it differently if I'm just doing a, a straight.

 

 


Prasanna Malaiyandi:

Yeah,

 

 


Prasanna Malaiyandi:

W. Curtis Preston: if I out, if I take the, the command and actually

 

 


Prasanna Malaiyandi:

output it to a file and then

 

 


Prasanna Malaiyandi:

and then look at it.

 

 


Prasanna Malaiyandi:

Um, and then basically I have to es like if I'm doing it on the command

 

 


Prasanna Malaiyandi:

line, I have to escape the escape twice.

 

 


Prasanna Malaiyandi:

If I'm doing, if I'm doing it in a filing, I only have to escape it once.

 

 


Prasanna Malaiyandi:

And that's been driving me just absolutely bonkers.

 

 


Prasanna Malaiyandi:

So I think what you do is you just add a whole

 

 


Prasanna Malaiyandi:

bunch of escapes in there, you know,

 

 


Prasanna Malaiyandi:

W. Curtis Preston: I

 

 


Prasanna Malaiyandi:

slashes

 

 


Prasanna Malaiyandi:

W. Curtis Preston: it works.

 

 


Prasanna Malaiyandi:

I'm just like, does two will two work?

 

 


Prasanna Malaiyandi:

No.

 

 


Prasanna Malaiyandi:

Two, no, no.

 

 


Prasanna Malaiyandi:

3, 3, 3.

 

 


Prasanna Malaiyandi:

No.

 

 


Prasanna Malaiyandi:

And then three, like three, uh, tells me that I, that I have too

 

 


Prasanna Malaiyandi:

many and I have like, uh, it, it, it, it escapes like something

 

 


Prasanna Malaiyandi:

later and I'm like, no, no, no, no.

 

 


Prasanna Malaiyandi:

Okay, so I won't take four.

 

 


Prasanna Malaiyandi:

You know, and I just, I just do a bunch of them until.

 

 


Prasanna Malaiyandi:

I get what I

 

 


Prasanna Malaiyandi:

It works.

 

 


Prasanna Malaiyandi:

W. Curtis Preston: Yeah.

 

 


Prasanna Malaiyandi:

It's like, it's like, uh, it's like writing code, right?

 

 


Prasanna Malaiyandi:

You compile it, it fails.

 

 


Prasanna Malaiyandi:

You're like, oh, I'll fix this, and then it fails again.

 

 


Prasanna Malaiyandi:

It's like, oh, I'll fix this, and then you just keep going until

 

 


Prasanna Malaiyandi:

eventually you get zero errors.

 

 


Prasanna Malaiyandi:

W. Curtis Preston: And, and, and then, so I, I find that as annoying as this is, I

 

 


Prasanna Malaiyandi:

find that it's actually easier to take a net backup command, output it to a file,

 

 


Prasanna Malaiyandi:

and then hash through that file, rather than just doing it as a series of pipes.

 

 


Prasanna Malaiyandi:

And as annoying as that is, it's more, it seems more consistent, um, albeit slower.

 

 


Prasanna Malaiyandi:

And it's also slower because the first thing I have to do on that

 

 


Prasanna Malaiyandi:

file, you know what I have to do?

 

 


Prasanna Malaiyandi:

I have to run the dos2unix command on it

 

 


Prasanna Malaiyandi:

because of the, the new line.

 

 


Prasanna Malaiyandi:

Because the new line is also different.

 

 


Prasanna Malaiyandi:

Yeah,

 

 


Prasanna Malaiyandi:

W. Curtis Preston: It, it's been fun is what I would say.

 

 


Prasanna Malaiyandi:

Oh, Curtis

 

 


Prasanna Malaiyandi:

W. Curtis Preston: Ah, so, uh, so speaking of fun persona, we

 

 


Prasanna Malaiyandi:

have now terminated our cloud

 

 


Prasanna Malaiyandi:

concluded.

 

 


Prasanna Malaiyandi:

W. Curtis Preston: series.

 

 


Prasanna Malaiyandi:

Concluded,

 

 


Prasanna Malaiyandi:

W. Curtis Preston: What's that?

 

 


Prasanna Malaiyandi:

concluded.

 

 


Prasanna Malaiyandi:

W. Curtis Preston: Why did I say terminated?

 

 


Prasanna Malaiyandi:

Yeah.

 

 


Prasanna Malaiyandi:

Terminated just seems.

 

 


Prasanna Malaiyandi:

W. Curtis Preston: said I'm done.

 

 


Prasanna Malaiyandi:

I'm done with all these people losing data.

 

 


Prasanna Malaiyandi:

Yeah.

 

 


Prasanna Malaiyandi:

We have concluded our cloud disaster series.

 

 


Prasanna Malaiyandi:

What did you think?

 

 


Prasanna Malaiyandi:

I liked it actually.

 

 


Prasanna Malaiyandi:

It's one of those things, like a disaster happens.

 

 


Prasanna Malaiyandi:

You read it on the news and then you forget about it, right?

 

 


Prasanna Malaiyandi:

It says to the next day.

 

 


Prasanna Malaiyandi:

And or you, there are many of these that like I don't think

 

 


Prasanna Malaiyandi:

any of us had heard about.

 

 


Prasanna Malaiyandi:

Right.

 

 


Prasanna Malaiyandi:

And just being able to go through and just understand and kind of take a look right

 

 


Prasanna Malaiyandi:

and getting your perspective right and seeing, okay, what are these disasters?

 

 


Prasanna Malaiyandi:

And really sort of doing the research ahead of time.

 

 


Prasanna Malaiyandi:

It was kind of fun.

 

 


Prasanna Malaiyandi:

I liked them.

 

 


Prasanna Malaiyandi:

W. Curtis Preston: Yeah, it was fun.

 

 


Prasanna Malaiyandi:

It was, um, what I liked about it was that enough time had passed with most

 

 


Prasanna Malaiyandi:

of these disasters that we often had the other half of the story, right?

 

 


Prasanna Malaiyandi:

When the, when, when they happen, uh, you know, at the mo you know,

 

 


Prasanna Malaiyandi:

at that moment you check it and you hear like in the case of, um.

 

 


Prasanna Malaiyandi:

The, uh, the Musi outage, the first thing you hear is that, you know, musi has sued

 

 


Prasanna Malaiyandi:

Google because, you know, they lost their data and, and that makes the headlines

 

 


Prasanna Malaiyandi:

and everybody's, oh, you know, there's a lawsuit, you know, but you know, this

 

 


Prasanna Malaiyandi:

is America, so anybody can sue anything.

 

 


Prasanna Malaiyandi:

I mean, you may remember that there was a woman who sued.

 

 


Prasanna Malaiyandi:

Google Maps because she drove her car off a pier.

 

 


Prasanna Malaiyandi:

Right.

 

 


Prasanna Malaiyandi:

Because Google Maps told her to go that way.

 

 


Prasanna Malaiyandi:

Right.

 

 


Prasanna Malaiyandi:

You know, you could sue for anything, but that doesn't mean

 

 


Prasanna Malaiyandi:

you're gonna be successful.

 

 


Prasanna Malaiyandi:

And it was nice to go back and look at the, um, you know what happened

 

 


Prasanna Malaiyandi:

after, in some of the cases, we got some additional insight from people

 

 


Prasanna Malaiyandi:

that were, you know, in and around.

 

 


Prasanna Malaiyandi:

I, I think of the.

 

 


Prasanna Malaiyandi:

OOVH incident.

 

 


Prasanna Malaiyandi:

Right?

 

 


Prasanna Malaiyandi:

Uh, that was nice.

 

 


Prasanna Malaiyandi:

We got some insight into that and how that OVH really was known for the, you

 

 


Prasanna Malaiyandi:

know, being a low cost provider and so that nobody was pro, nobody was surprised

 

 


Prasanna Malaiyandi:

when they, uh, didn't have, didn't have.

 

 


Prasanna Malaiyandi:

Yeah.

 

 


Prasanna Malaiyandi:

Yeah.

 

 


Prasanna Malaiyandi:

Well, the other thing also that I came to realize from the series is,

 

 


Prasanna Malaiyandi:

like you said, time had elapsed.

 

 


Prasanna Malaiyandi:

It's also interesting to see how the company, some companies try

 

 


Prasanna Malaiyandi:

to sweep things under the rug,

 

 


Prasanna Malaiyandi:

W. Curtis Preston: Right,

 

 


Prasanna Malaiyandi:

right?

 

 


Prasanna Malaiyandi:

And thanks to the wonders of the internet where nothing is ever gone, right?

 

 


Prasanna Malaiyandi:

Being able to uncover, okay, really, what was it that was initially

 

 


Prasanna Malaiyandi:

said versus what do they say today?

 

 


Prasanna Malaiyandi:

W. Curtis Preston: Yeah.

 

 


Prasanna Malaiyandi:

Uh, thank you very much.

 

 


Prasanna Malaiyandi:

Um, internet archive, right?

 

 


Prasanna Malaiyandi:

That was, that was, that was very helpful.

 

 


Prasanna Malaiyandi:

In some cases we weren't able to get stuff, but, and there were a lot of cases

 

 


Prasanna Malaiyandi:

where we were able to get basically the first version of the story versus the,

 

 


Prasanna Malaiyandi:

the later, the, the latter version.

 

 


Prasanna Malaiyandi:

Yeah.

 

 


Prasanna Malaiyandi:

Uh, but, but I would say in most of these cases, right, the initial

 

 


Prasanna Malaiyandi:

information that people saw was what was allowed to be released, right?

 

 


Prasanna Malaiyandi:

Whatever the company had at that time, which sometimes they're working off

 

 


Prasanna Malaiyandi:

incomplete information, but they still wanted to provide some information

 

 


Prasanna Malaiyandi:

to their users, such as like the OVH incident or Rackspace, right?

 

 


Prasanna Malaiyandi:

Where it's just, okay, what information is available?

 

 


Prasanna Malaiyandi:

Let me at least push something out there.

 

 


Prasanna Malaiyandi:

W. Curtis Preston: Right.

 

 


Prasanna Malaiyandi:

And that, and that's what we encourage, uh, companies to do, right, is to

 

 


Prasanna Malaiyandi:

be upfront about what's happening.

 

 


Prasanna Malaiyandi:

And it's okay to say you don't know yet, but acknowledge what's happening.

 

 


Prasanna Malaiyandi:

You know, acknowledge what you know when you know it.

 

 


Prasanna Malaiyandi:

Don't make stuff up.

 

 


Prasanna Malaiyandi:

Right.

 

 


Prasanna Malaiyandi:

Um.

 

 


Prasanna Malaiyandi:

And the, and then, and then the, the other thing that I would say is, um.

 

 


Prasanna Malaiyandi:

Is we like it when they don't try to pass blame.

 

 


Prasanna Malaiyandi:

That happened a couple of times.

 

 


Prasanna Malaiyandi:

Uh, when the blame is, you know, clearly on you and your design,

 

 


Prasanna Malaiyandi:

uh, don't try to pass blame.

 

 


Prasanna Malaiyandi:

So we've got, I came up with a few lessons that I wrote down, uh, and

 

 


Prasanna Malaiyandi:

if, and if, and if a few others pop up in our head while we're recording

 

 


Prasanna Malaiyandi:

this episode, just come out with 'em.

 

 


Prasanna Malaiyandi:

Um, and the first one is actually, uh, you put that on there.

 

 


Prasanna Malaiyandi:

So you wanna, you wanna go for it?

 

 


Prasanna Malaiyandi:

Yeah.

 

 


Prasanna Malaiyandi:

Yeah.

 

 


Prasanna Malaiyandi:

So I would say that the most important thing for a lot of

 

 


Prasanna Malaiyandi:

these is if it's important to you, the data, of course, back it up,

 

 


Prasanna Malaiyandi:

W. Curtis Preston: Yeah.

 

 


Prasanna Malaiyandi:

Which is, which is basically backup lesson number one.

 

 


Prasanna Malaiyandi:

Is

 

 


Prasanna Malaiyandi:

Or zero, I would say.

 

 


Prasanna Malaiyandi:

W. Curtis Preston: Yeah.

 

 


Prasanna Malaiyandi:

Backup.

 

 


Prasanna Malaiyandi:

It's Rule zero.

 

 


Prasanna Malaiyandi:

Uh, it's one that I absolutely cannot argue with, right?

 

 


Prasanna Malaiyandi:

It's something that I say to you, right?

 

 


Prasanna Malaiyandi:

Yep.

 

 


Prasanna Malaiyandi:

W. Curtis Preston: I, I say this all the time, that if, if this data

 

 


Prasanna Malaiyandi:

really is important to you, then it should be, you know, backed up, right?

 

 


Prasanna Malaiyandi:

And we can talk in a little bit as to what constitutes a backup and,

 

 


Prasanna Malaiyandi:

uh, that'll be some of the lessons that we learned along the way.

 

 


Prasanna Malaiyandi:

But, um.

 

 


Prasanna Malaiyandi:

I, I'm gonna say in almost all instances, what constitutes a

 

 


Prasanna Malaiyandi:

backup is you making a backup.

 

 


Prasanna Malaiyandi:

Not, not saying, oh, I got it because I got it, because it's the cloud,

 

 


Prasanna Malaiyandi:

or because it's SaaS, or because it's, you know, it's not my problem.

 

 


Prasanna Malaiyandi:

No, it's always your problem.

 

 


Prasanna Malaiyandi:

Yeah, and there were companies, right, like Musi went out

 

 


Prasanna Malaiyandi:

of business because they lost all their data and they couldn't recover it, and

 

 


Prasanna Malaiyandi:

that was the end of the company, right?

 

 


Prasanna Malaiyandi:

So,

 

 


Prasanna Malaiyandi:

W. Curtis Preston: Yeah.

 

 


Prasanna Malaiyandi:

And, and I'd say that that's, that's the first lesson I'm gonna say that

 

 


Prasanna Malaiyandi:

wasn't actually on the list, is that if you don't get this right.

 

 


Prasanna Malaiyandi:

Your entire company may just go away, right?

 

 


Prasanna Malaiyandi:

That happened, uh, in at least two instances in the, the list

 

 


Prasanna Malaiyandi:

of the companies that we found.

 

 


Prasanna Malaiyandi:

Either your company or a significant portion of your company may go away.

 

 


Prasanna Malaiyandi:

I think of Rackspace, right?

 

 


Prasanna Malaiyandi:

Rackspace is now completely out of the hosted exchange business because.

 

 


Prasanna Malaiyandi:

Uh, of what happened with the rec based outage, right?

 

 


Prasanna Malaiyandi:

So either your entire company or a significant portion of your company

 

 


Prasanna Malaiyandi:

can literally just go, poof, overnight.

 

 


Prasanna Malaiyandi:

Yeah.

 

 


Prasanna Malaiyandi:

So what's the first one, Curtis, in your opinion, like what's the

 

 


Prasanna Malaiyandi:

most important lesson from all of the ones that we've looked at?

 

 


Prasanna Malaiyandi:

W. Curtis Preston: 3, 2, 1.

 

 


Prasanna Malaiyandi:

Rule still rules.

 

 


Prasanna Malaiyandi:

And, and I know that there are other versions of the 3, 2, 1 rule.

 

 


Prasanna Malaiyandi:

I'm mu in this case, what I'm saying is if what you are doing for backup, as I make

 

 


Prasanna Malaiyandi:

quotes in the air, doesn't at least comply to the 3, 2, 1 rule, it is not a backup.

 

 


Prasanna Malaiyandi:

Right?

 

 


Prasanna Malaiyandi:

There are, there are many situations when we look at the stories that we, that we

 

 


Prasanna Malaiyandi:

talked about here, where people had what they might have considered a backup.

 

 


Prasanna Malaiyandi:

Right.

 

 


Prasanna Malaiyandi:

Um, you know, musi is probably the best example of that, right?

 

 


Prasanna Malaiyandi:

Um, where they had their backups with them, but basically

 

 


Prasanna Malaiyandi:

they deleted their account.

 

 


Prasanna Malaiyandi:

Right.

 

 


Prasanna Malaiyandi:

Um, the, the other one would be Code Spaces.

 

 


Prasanna Malaiyandi:

That's probably the most infamous, I think, where Code Spaces was making

 

 


Prasanna Malaiyandi:

backups within the, within AWS, but they didn't follow the 3, 2, 1 rule.

 

 


Prasanna Malaiyandi:

They didn't separate those backups from production.

 

 


Prasanna Malaiyandi:

And so, when.

 

 


Prasanna Malaiyandi:

Um, you know, when the, the bad actor went in there, they deleted the entire account.

 

 


Prasanna Malaiyandi:

And again, poof, they went there, went their primary as well as their backup.

 

 


Prasanna Malaiyandi:

And the same would be true of the SaaS.

 

 


Prasanna Malaiyandi:

Um, people that, you know, when I think of specifically Microsoft 365, but this is

 

 


Prasanna Malaiyandi:

true in so many other cases, is that, um.

 

 


Prasanna Malaiyandi:

It that if you are not separating that data, if you're not making

 

 


Prasanna Malaiyandi:

another copy, they don't have, uh, multiple copies of your data, right?

 

 


Prasanna Malaiyandi:

That, that are accessible to you.

 

 


Prasanna Malaiyandi:

They may have, may have a DR copy, but as we saw in some of the cases, the DR.

 

 


Prasanna Malaiyandi:

Copy does doesn't always work.

 

 


Prasanna Malaiyandi:

Um, that Dr.

 

 


Prasanna Malaiyandi:

Copy

 

 


Prasanna Malaiyandi:

might not be to you, and it might not, it might not work, but.

 

 


Prasanna Malaiyandi:

But the point is that if you're not, if you don't have a whatever they

 

 


Prasanna Malaiyandi:

have, it's all in the same place.

 

 


Prasanna Malaiyandi:

And when very bad things happen, uh, your SOL

 

 


Prasanna Malaiyandi:

And specifically around the Microsoft case, right?

 

 


Prasanna Malaiyandi:

They may have a DR copy, but like you said, that's for their purposes

 

 


Prasanna Malaiyandi:

not to allow you to do your restores.

 

 


Prasanna Malaiyandi:

That's to make sure if something blows up somewhere from an infrastructure

 

 


Prasanna Malaiyandi:

perspective, they can break things back up and they'll try their

 

 


Prasanna Malaiyandi:

best to give you back all your data, but there's no guarantees.

 

 


Prasanna Malaiyandi:

W. Curtis Preston: Yeah, exactly.

 

 


Prasanna Malaiyandi:

Actually we're gonna get more into detail into that in, in a latter step.

 

 


Prasanna Malaiyandi:

But yeah, so short version is, I, I have no problem with app

 

 


Prasanna Malaiyandi:

pending to the 3, 2, 1 rule.

 

 


Prasanna Malaiyandi:

If you wanna say 3, 2, 1, 1, 0, 7, 9, 5, I don't care.

 

 


Prasanna Malaiyandi:

Right.

 

 


Prasanna Malaiyandi:

Um, and I, and I don't disagree with any of the versions that I've heard.

 

 


Prasanna Malaiyandi:

Right.

 

 


Prasanna Malaiyandi:

Things like I.

 

 


Prasanna Malaiyandi:

Making sure that, you know, one of the copies, the biggest one is

 

 


Prasanna Malaiyandi:

making sure that one of those copies is an immutable copy, which is a

 

 


Prasanna Malaiyandi:

relatively new requirement, and that has to do with, uh, cyber attacks.

 

 


Prasanna Malaiyandi:

I don't have a problem with that.

 

 


Prasanna Malaiyandi:

But if what you're doing doesn't comply with the 3, 2, 1 rule, uh, three

 

 


Prasanna Malaiyandi:

copies of your data on two different types of media, one of which is

 

 


Prasanna Malaiyandi:

physically separated from the other, um, then it, you don't have a backup.

 

 


Prasanna Malaiyandi:

I do wonder, and maybe it doesn't have to be for

 

 


Prasanna Malaiyandi:

this episode, if we should really figure out does the three or have a

 

 


Prasanna Malaiyandi:

discussion on an episode of does the 3, 2 1 rule really need to be changed?

 

 


Prasanna Malaiyandi:

W. Curtis Preston: Um, I,

 

 


Prasanna Malaiyandi:

I think,

 

 


Prasanna Malaiyandi:

I think just because I know that there's like different,

 

 


Prasanna Malaiyandi:

like you said, the 3, 2, 1, 1 0.

 

 


Prasanna Malaiyandi:

Zero.

 

 


Prasanna Malaiyandi:

Zero.

 

 


Prasanna Malaiyandi:

It may not be a zero, but you know what I mean.

 

 


Prasanna Malaiyandi:

Right.

 

 


Prasanna Malaiyandi:

W. Curtis Preston: Yeah, well there, there's various versions

 

 


Prasanna Malaiyandi:

out there and I'm fine with those.

 

 


Prasanna Malaiyandi:

I do myself, like, you know, a, you know, I add to it.

 

 


Prasanna Malaiyandi:

One of these copies needs to be immutable.

 

 


Prasanna Malaiyandi:

I am, I fully agree with that.

 

 


Prasanna Malaiyandi:

Does it need to be part of the 3, 2, 1 rule?

 

 


Prasanna Malaiyandi:

I don't know.

 

 


Prasanna Malaiyandi:

I, you know.

 

 


Prasanna Malaiyandi:

I like the 3, 2, 1 for, for what it does, because what it does is it proves what

 

 


Prasanna Malaiyandi:

we've been talking about here is that if you have a cloud service and all of

 

 


Prasanna Malaiyandi:

the copies of your data are all in the same place, you don't have a backup.

 

 


Prasanna Malaiyandi:

Right?

 

 


Prasanna Malaiyandi:

Yep.

 

 


Prasanna Malaiyandi:

Speaking of which, uh, I would say that the next lesson learned is really around.

 

 


Prasanna Malaiyandi:

Right.

 

 


Prasanna Malaiyandi:

Like you mentioned, the cloud is not backing up your data.

 

 


Prasanna Malaiyandi:

You may think it is backing up your data.

 

 


Prasanna Malaiyandi:

They may be doing DR copies for their own purposes, but it's not a backup

 

 


Prasanna Malaiyandi:

that you as a customer, a user can go restore from, can go back to any point

 

 


Prasanna Malaiyandi:

in time, recover it all the rest of that.

 

 


Prasanna Malaiyandi:

Which I think a lot of people mistake, right?

 

 


Prasanna Malaiyandi:

They're like, oh, I have a Salesforce, or Take your pick and it's a SaaS service.

 

 


Prasanna Malaiyandi:

They host my data.

 

 


Prasanna Malaiyandi:

They should be doing my backups.

 

 


Prasanna Malaiyandi:

Or Microsoft 365, they should be doing my backups.

 

 


Prasanna Malaiyandi:

And I know Curtis way back in the day when we were first working together, right?

 

 


Prasanna Malaiyandi:

You were like, have you ever taken a look at Microsoft's

 

 


Prasanna Malaiyandi:

contracts, terms of service?

 

 


Prasanna Malaiyandi:

Do you ever see backup in there anywhere?

 

 


Prasanna Malaiyandi:

W. Curtis Preston: Right.

 

 


Prasanna Malaiyandi:

Yeah.

 

 


Prasanna Malaiyandi:

It's just, it is just not there, right?

 

 


Prasanna Malaiyandi:

That and that and that.

 

 


Prasanna Malaiyandi:

That's the part it is.

 

 


Prasanna Malaiyandi:

If you think that your SaaS service is backing up your data, please go

 

 


Prasanna Malaiyandi:

look for it in your contract, right?

 

 


Prasanna Malaiyandi:

Because you're not gonna find it.

 

 


Prasanna Malaiyandi:

There may be, there are some exceptions.

 

 


Prasanna Malaiyandi:

Interestingly enough, one of the exceptions was covered on

 

 


Prasanna Malaiyandi:

our, on our stories, right?

 

 


Prasanna Malaiyandi:

Where we had a vendor that was backing up, we're gonna talk, we're gonna,

 

 


Prasanna Malaiyandi:

we had a vendor who was advertising that part of their service was backup,

 

 


Prasanna Malaiyandi:

which almost never happens, right?

 

 


Prasanna Malaiyandi:

You don't see that.

 

 


Prasanna Malaiyandi:

Go look at what, what Microsoft advertises for 365.

 

 


Prasanna Malaiyandi:

Look at Salesforce.

 

 


Prasanna Malaiyandi:

Backup is not part of the offering either in terms of what they

 

 


Prasanna Malaiyandi:

advertise or what's in your contract.

 

 


Prasanna Malaiyandi:

And if it's not in your contract, it doesn't exist.

 

 


Prasanna Malaiyandi:

Right.

 

 


Prasanna Malaiyandi:

So the next one, and this is we, we've covered this a little bit already, and

 

 


Prasanna Malaiyandi:

that is if they are backing up their data center, that backup is not for you.

 

 


Prasanna Malaiyandi:

Right.

 

 


Prasanna Malaiyandi:

Um, and, and I see this a lot.

 

 


Prasanna Malaiyandi:

Um, there's again, this, this one guy that I interact with a lot online that,

 

 


Prasanna Malaiyandi:

that really believes very strongly in the Microsoft way of doing things

 

 


Prasanna Malaiyandi:

and in his idea that Microsoft has, for example, multiple delayed copies

 

 


Prasanna Malaiyandi:

of your data, like replicated, uh, delayed, replicated copies of your data.

 

 


Prasanna Malaiyandi:

That it's, it's gonna use that in the case of disaster to be

 

 


Prasanna Malaiyandi:

able to recover your environment.

 

 


Prasanna Malaiyandi:

And that appears to be true.

 

 


Prasanna Malaiyandi:

I, I've never found it anywhere documented.

 

 


Prasanna Malaiyandi:

Right?

 

 


Prasanna Malaiyandi:

Which, and if it's not documented, if it's not in my contract that I don't care.

 

 


Prasanna Malaiyandi:

But it appears to be true, however.

 

 


Prasanna Malaiyandi:

The story of KPMG.

 

 


Prasanna Malaiyandi:

If that story with the size of KPMG 145,000 employees, if what happened

 

 


Prasanna Malaiyandi:

to KPMG doesn't prove to you that you don't get to use Microsoft's backup to

 

 


Prasanna Malaiyandi:

save your butt, I don't know what will.

 

 


Prasanna Malaiyandi:

This is a, this is a giant company paying tons of money to Microsoft every month.

 

 


Prasanna Malaiyandi:

Even they weren't able to recover from this giant blunder because it

 

 


Prasanna Malaiyandi:

was their fault, and that was not enough to have Microsoft use their

 

 


Prasanna Malaiyandi:

Dr copy, uh, to, to save them.

 

 


Prasanna Malaiyandi:

Yep.

 

 


Prasanna Malaiyandi:

Or, or if we go back and think about like

 

 


Prasanna Malaiyandi:

the Salesforce example, right?

 

 


Prasanna Malaiyandi:

Where they changed on permissions and everyone had

 

 


Prasanna Malaiyandi:

access to every record, right?

 

 


Prasanna Malaiyandi:

And it was like, how do we undo that?

 

 


Prasanna Malaiyandi:

And so they tried their best and hopefully if you had a sandbox copy, they were

 

 


Prasanna Malaiyandi:

able to help you out a little bit.

 

 


Prasanna Malaiyandi:

But they were basically like, sorry, uh, I can't help you.

 

 


Prasanna Malaiyandi:

Good luck.

 

 


Prasanna Malaiyandi:

W. Curtis Preston: Yeah, that's interesting because the Salesforce

 

 


Prasanna Malaiyandi:

story, it's, it was their blunder,

 

 


Prasanna Malaiyandi:

Yep.

 

 


Prasanna Malaiyandi:

W. Curtis Preston: right?

 

 


Prasanna Malaiyandi:

It was their blunder that, that, um, I.

 

 


Prasanna Malaiyandi:

Where they messed up everybody's account.

 

 


Prasanna Malaiyandi:

And even then, when we know that Salesforce has a, a backup, as I make

 

 


Prasanna Malaiyandi:

quotes in the air, um, that you can pay for like $10,000 an instance,

 

 


Prasanna Malaiyandi:

and it takes like weeks to get it.

 

 


Prasanna Malaiyandi:

But they were like, no, we're not gonna crack open Pandora's box, even though it

 

 


Prasanna Malaiyandi:

was us that messed up all these accounts.

 

 


Prasanna Malaiyandi:

So, yeah.

 

 


Prasanna Malaiyandi:

So th those two incidents, if, if those don't prove to you that these DR copies

 

 


Prasanna Malaiyandi:

that these vendors are making aren't for your purposes, I, I don't know what will.

 

 


Prasanna Malaiyandi:

Yeah, and these are large companies, right?

 

 


Prasanna Malaiyandi:

It's not like a mom and pop cloud vendor, right?

 

 


Prasanna Malaiyandi:

Doing something or a SaaS application.

 

 


Prasanna Malaiyandi:

W. Curtis Preston: Yeah.

 

 


Prasanna Malaiyandi:

Going alongside that, so we're saying don't trust

 

 


Prasanna Malaiyandi:

the cloud vendor to do your backup.

 

 


Prasanna Malaiyandi:

Right.

 

 


Prasanna Malaiyandi:

And instead you really should be using a third party backup service.

 

 


Prasanna Malaiyandi:

If we looked at some of the cases like Deduce where academics, people

 

 


Prasanna Malaiyandi:

in academics and research, they lost.

 

 


Prasanna Malaiyandi:

Studies, right?

 

 


Prasanna Malaiyandi:

Because, uh, DDU went down or they lost some data, and if they had simply done

 

 


Prasanna Malaiyandi:

a backup or even simply exported their data, right, they would've been fine.

 

 


Prasanna Malaiyandi:

They wouldn't have had an issue, but they didn't because there wasn't part

 

 


Prasanna Malaiyandi:

of the service, there wasn't anything that could easily back that up.

 

 


Prasanna Malaiyandi:

And so people lost data.

 

 


Prasanna Malaiyandi:

W. Curtis Preston: Yeah, exactly.

 

 


Prasanna Malaiyandi:

I mean the, the Deduce story, uh, the Rackspace story, where it would've been

 

 


Prasanna Malaiyandi:

probably much easier for companies when they, when Rackspace went down and then

 

 


Prasanna Malaiyandi:

they immediately migrated everybody over to, to 365 would've been much easier

 

 


Prasanna Malaiyandi:

for them to recover from that situation, uh, if they had their own backup.

 

 


Prasanna Malaiyandi:

Um, and, and of course the OVH disaster where, uh, you know, the, they, so both

 

 


Prasanna Malaiyandi:

in the case of OVH and um, deduce, we know that backup was something that

 

 


Prasanna Malaiyandi:

they advertised as part of the service.

 

 


Prasanna Malaiyandi:

By the way, specifically in Deduce, you looked, if you look at their

 

 


Prasanna Malaiyandi:

website from back then, it says they're doing nightly backups.

 

 


Prasanna Malaiyandi:

And, and we now know that we don't know exactly how or why, but the closest

 

 


Prasanna Malaiyandi:

available backup was o was a month out and the one that was, and, and there

 

 


Prasanna Malaiyandi:

was something a little funky with it.

 

 


Prasanna Malaiyandi:

So they actually went two months out.

 

 


Prasanna Malaiyandi:

Um, what, how, how is that a nightly backup?

 

 


Prasanna Malaiyandi:

Right, right.

 

 


Prasanna Malaiyandi:

So I, I guess what you're saying is, or what I'm saying is that when.

 

 


Prasanna Malaiyandi:

Bad things happen and bad things happen.

 

 


Prasanna Malaiyandi:

Admins do wrong things.

 

 


Prasanna Malaiyandi:

Software engineers do wrong things.

 

 


Prasanna Malaiyandi:

When you're trusting a single vendor to do both your IT and your backups

 

 


Prasanna Malaiyandi:

of your it, if they turn out to be incompetent, they're probably also

 

 


Prasanna Malaiyandi:

incompetent in your backup, right?

 

 


Prasanna Malaiyandi:

And so this isn't, because both of us used to work for a cloud backup vendor, right?

 

 


Prasanna Malaiyandi:

This isn't us trying to, you know, get money for our firm employer.

 

 


Prasanna Malaiyandi:

Uh, this is just common sense man.

 

 


Prasanna Malaiyandi:

Yeah, and I would also say for those companies where

 

 


Prasanna Malaiyandi:

they are focused on both your primary IT application as well as the backup,

 

 


Prasanna Malaiyandi:

typically they're not going to put all the money, all the effort in the backup side

 

 


Prasanna Malaiyandi:

because their main goal is to sell and add new features to the primary application.

 

 


Prasanna Malaiyandi:

And so usually you end up with something that's either half baked, not up

 

 


Prasanna Malaiyandi:

to date, doesn't support all your various scenarios, that someone who is

 

 


Prasanna Malaiyandi:

specialized in backup is thinking about.

 

 


Prasanna Malaiyandi:

W. Curtis Preston: Yeah, all those things there.

 

 


Prasanna Malaiyandi:

Yeah.

 

 


Prasanna Malaiyandi:

There are gonna be lots of important features, both from a security standpoint,

 

 


Prasanna Malaiyandi:

a recovery standpoint, all of that, that are going to be useful to you if

 

 


Prasanna Malaiyandi:

you ever actually need to do a restore.

 

 


Prasanna Malaiyandi:

In the case of any of the events that happened to the people

 

 


Prasanna Malaiyandi:

in our, in the stories that we covered in the last, what was it?

 

 


Prasanna Malaiyandi:

Like eight weeks, something like that.

 

 


Prasanna Malaiyandi:

Uh, what we got next.

 

 


Prasanna Malaiyandi:

So, I think you did touch on this a bit earlier,

 

 


Prasanna Malaiyandi:

but be careful when someone sells you a feature and tells you it's

 

 


Prasanna Malaiyandi:

backup, but it's not backup.

 

 


Prasanna Malaiyandi:

W. Curtis Preston: Yeah.

 

 


Prasanna Malaiyandi:

Yeah, so this one's interesting because, and I, it sounds like

 

 


Prasanna Malaiyandi:

we pick on Microsoft a lot.

 

 


Prasanna Malaiyandi:

I, it's not Microsoft.

 

 


Prasanna Malaiyandi:

My issue is not Microsoft.

 

 


Prasanna Malaiyandi:

It's the people that seem to, that seem to, like.

 

 


Prasanna Malaiyandi:

In the case of Microsoft 365, you will not see the documentation referring

 

 


Prasanna Malaiyandi:

to retention policies as backup.

 

 


Prasanna Malaiyandi:

You won't see it.

 

 


Prasanna Malaiyandi:

Retention policies are an e-discovery feature.

 

 


Prasanna Malaiyandi:

E discovery and archive have nothing to do with backup, right?

 

 


Prasanna Malaiyandi:

They behave completely differently in terms of the way they save

 

 


Prasanna Malaiyandi:

data, the way they pull out data.

 

 


Prasanna Malaiyandi:

Um, and if you, if you don't understand what I'm saying, you

 

 


Prasanna Malaiyandi:

don't believe what I'm saying, go and try to restore a mailbox.

 

 


Prasanna Malaiyandi:

Using 365 retention policies, please go try that.

 

 


Prasanna Malaiyandi:

Um, the, um,

 

 


Prasanna Malaiyandi:

tell us if it could turn out to be a plausible point

 

 


Prasanna Malaiyandi:

in time that that mailbox looked like

 

 


Prasanna Malaiyandi:

W. Curtis Preston: doesn't do it.

 

 


Prasanna Malaiyandi:

Yeah.

 

 


Prasanna Malaiyandi:

W. Curtis Preston: It.

 

 


Prasanna Malaiyandi:

It's all about retention policies and the associated E-discovery feature

 

 


Prasanna Malaiyandi:

is all about how to get all email that ever went into or came out

 

 


Prasanna Malaiyandi:

of that box over a period of time.

 

 


Prasanna Malaiyandi:

It is not designed to restore your mailbox to the way it looked yesterday.

 

 


Prasanna Malaiyandi:

Right.

 

 


Prasanna Malaiyandi:

Doesn't restore folders.

 

 


Prasanna Malaiyandi:

It doesn't restore all kinds of stuff, and it doesn't know how.

 

 


Prasanna Malaiyandi:

It, again, just you, you gotta, maybe you have to experience it to truly

 

 


Prasanna Malaiyandi:

understand this, and that is that it doesn't know how to make your mailbox

 

 


Prasanna Malaiyandi:

look the way it looked yesterday.

 

 


Prasanna Malaiyandi:

It doesn't understand that concept.

 

 


Prasanna Malaiyandi:

That is a backup and restore concept.

 

 


Prasanna Malaiyandi:

Right?

 

 


Prasanna Malaiyandi:

And so that's, that's

 

 


Prasanna Malaiyandi:

Not an archive and retrieval.

 

 


Prasanna Malaiyandi:

W. Curtis Preston: Yeah, it's about arch archive and retrieval.

 

 


Prasanna Malaiyandi:

Right?

 

 


Prasanna Malaiyandi:

Which, if you don't know what we're talking about, there's a whole

 

 


Prasanna Malaiyandi:

episode that we talk about why arch archive and backup are different.

 

 


Prasanna Malaiyandi:

Go, go check that out.

 

 


Prasanna Malaiyandi:

Uh, put a link in the show notes.

 

 


Prasanna Malaiyandi:

But the, but the other reason with 365 is when we look at that story of KPMG if

 

 


Prasanna Malaiyandi:

you, if you have a backup software, um.

 

 


Prasanna Malaiyandi:

Let's say, you know, pick your favorite backup software.

 

 


Prasanna Malaiyandi:

If you configure it really, really, really, really wrong, like just the

 

 


Prasanna Malaiyandi:

worst backup configuration ever.

 

 


Prasanna Malaiyandi:

You know what it doesn't do,

 

 


Prasanna Malaiyandi:

Blow away your production.

 

 


Prasanna Malaiyandi:

W. Curtis Preston: it doesn't blow away your production.

 

 


Prasanna Malaiyandi:

Which is another key difference between Microsoft retention

 

 


Prasanna Malaiyandi:

policies and backup, because that's what happened in the KPMG story.

 

 


Prasanna Malaiyandi:

you almost want a separation of duties, right?

 

 


Prasanna Malaiyandi:

That's the roles and responsibilities.

 

 


Prasanna Malaiyandi:

It's like no different than a cloud provider and the customers who are

 

 


Prasanna Malaiyandi:

on top of the cloud provider, right?

 

 


Prasanna Malaiyandi:

Each person has their own responsibilities and their roles and what they do

 

 


Prasanna Malaiyandi:

and what is provided, and it's all clearly articulated in contracts.

 

 


Prasanna Malaiyandi:

W. Curtis Preston: Yeah, retention policies.

 

 


Prasanna Malaiyandi:

When this person went to go do what they wanted to do, which was just delete one

 

 


Prasanna Malaiyandi:

person's chat, it's like, if I get this right, I will delete one person's chat

 

 


Prasanna Malaiyandi:

and I will have a really, uh, poor.

 

 


Prasanna Malaiyandi:

Facsimile of backup that is really lousy at Restore.

 

 


Prasanna Malaiyandi:

That's the best case scenario.

 

 


Prasanna Malaiyandi:

If I get it wrong, I'm gonna delete the personal chat

 

 


Prasanna Malaiyandi:

history of 145,000 employees.

 

 


Prasanna Malaiyandi:

It could, by the way, it could have been worse.

 

 


Prasanna Malaiyandi:

It could have been worse because.

 

 


Prasanna Malaiyandi:

Retention policies when you're in them, when you, when you first open them.

 

 


Prasanna Malaiyandi:

The, by the way, retention policy, the manual is 25 pages, right?

 

 


Prasanna Malaiyandi:

But you can go in there and you can go for all services, right?

 

 


Prasanna Malaiyandi:

Meaning

 

 


Prasanna Malaiyandi:

Prasanna Malaiyandi: It's not just for chat.

 

 


Prasanna Malaiyandi:

Yeah.

 

 


Prasanna Malaiyandi:

W. Curtis Preston: for all services.

 

 


Prasanna Malaiyandi:

Please set retention to one copy.

 

 


Prasanna Malaiyandi:

So,

 

 


Prasanna Malaiyandi:

W. Curtis Preston: copy.

 

 


Prasanna Malaiyandi:

Literally in a, in a few mouse clicks, you can blow away your entire 365 environment.

 

 


Prasanna Malaiyandi:

could you imagine if that had happened?

 

 


Prasanna Malaiyandi:

And KPMG, which is a huge company, loses everything.

 

 


Prasanna Malaiyandi:

W. Curtis Preston: Yeah.

 

 


Prasanna Malaiyandi:

I mean, it's bad enough that what they lost.

 

 


Prasanna Malaiyandi:

People are like, oh, it's just chat.

 

 


Prasanna Malaiyandi:

Well, chat is often quite valuable.

 

 


Prasanna Malaiyandi:

Yeah,

 

 


Prasanna Malaiyandi:

Clearly you've never lived in a company that used chat.

 

 


Prasanna Malaiyandi:

Uh,

 

 


Prasanna Malaiyandi:

Prasanna Malaiyandi: don't use smoke signals.

 

 


Prasanna Malaiyandi:

W. Curtis Preston: what's that?

 

 


Prasanna Malaiyandi:

They don't use smoke signals

 

 


Prasanna Malaiyandi:

W. Curtis Preston: Yeah, it reminds me, it reminds me there was a company.

 

 


Prasanna Malaiyandi:

It was, I'm pretty sure this was a Xerox commercial years ago.

 

 


Prasanna Malaiyandi:

And it was like, does Xero, is your, is this, describe your company, like

 

 


Prasanna Malaiyandi:

if your copier goes down, do people say it's okay, we'll use carbon paper.

 

 


Prasanna Malaiyandi:

Do you remember that?

 

 


Prasanna Malaiyandi:

That was

 

 


Prasanna Malaiyandi:

a, that was a, that was a long time ago.

 

 


Prasanna Malaiyandi:

Um, so what's our next lesson?

 

 


Prasanna Malaiyandi:

So going alongside what you talked about with Microsoft

 

 


Prasanna Malaiyandi:

365 and retention policies is, that's like a specific case for Microsoft.

 

 


Prasanna Malaiyandi:

But also cloud providers in general.

 

 


Prasanna Malaiyandi:

How about their designs are just awful.

 

 


Prasanna Malaiyandi:

Like, I think that's actually being too kind to say awful,

 

 


Prasanna Malaiyandi:

W. Curtis Preston: Yeah.

 

 


Prasanna Malaiyandi:

right?

 

 


Prasanna Malaiyandi:

I think it's just downright pathetic.

 

 


Prasanna Malaiyandi:

Uh, one of the ones that comes to mind is OVH.

 

 


Prasanna Malaiyandi:

W. Curtis Preston: Right,

 

 


Prasanna Malaiyandi:

know I hate picking on OVH 'cause we always

 

 


Prasanna Malaiyandi:

pick on OVH, but I'm picking on OVH.

 

 


Prasanna Malaiyandi:

Right?

 

 


Prasanna Malaiyandi:

And it's really around the fact that they offered a backup service and

 

 


Prasanna Malaiyandi:

they basically had the backup server in the same data center, like just a

 

 


Prasanna Malaiyandi:

couple rows down in a separate rack.

 

 


Prasanna Malaiyandi:

And so when they had a fire that damaged both production and their backup data.

 

 


Prasanna Malaiyandi:

W. Curtis Preston: Yeah, I mean, I mean, that's bad.

 

 


Prasanna Malaiyandi:

Uh, I, I don't knows some equally bad, similarly bad.

 

 


Prasanna Malaiyandi:

Um, when I think about the fact, you know, if we go back in time, there was a time

 

 


Prasanna Malaiyandi:

when Carbonite was everywhere, right?

 

 


Prasanna Malaiyandi:

There were advertise.

 

 


Prasanna Malaiyandi:

You would think that Carbonite was the single biggest

 

 


Prasanna Malaiyandi:

backup vendor on the planet.

 

 


Prasanna Malaiyandi:

And they were storing customer backup data on raid five arrays.

 

 


Prasanna Malaiyandi:

Right.

 

 


Prasanna Malaiyandi:

I'm gonna say pro prosumer quality raid five arrays with, with no

 

 


Prasanna Malaiyandi:

dual parity, you know, um, nothing.

 

 


Prasanna Malaiyandi:

And then they, you know, and it was really just a matter of time before

 

 


Prasanna Malaiyandi:

what happened to them happened.

 

 


Prasanna Malaiyandi:

Right.

 

 


Prasanna Malaiyandi:

Um, you know, they had a double disc failure and Right.

 

 


Prasanna Malaiyandi:

You know, there was that one.

 

 


Prasanna Malaiyandi:

And then, you know, we have the other one.

 

 


Prasanna Malaiyandi:

Of, um, you know, code spaces where they stored their backup in the same

 

 


Prasanna Malaiyandi:

place that they stored production.

 

 


Prasanna Malaiyandi:

We already talked about that with the 3, 2, 1 rule.

 

 


Prasanna Malaiyandi:

Yeah, and I think the other one there to also talk about is.

 

 


Prasanna Malaiyandi:

And I don't know if it's really poor design or just not understanding

 

 


Prasanna Malaiyandi:

backup and backup requirements is, I would actually say Rackspace,

 

 


Prasanna Malaiyandi:

right?

 

 


Prasanna Malaiyandi:

They had a hosted exchange offering, which is great, but they didn't

 

 


Prasanna Malaiyandi:

really have a backup solution.

 

 


Prasanna Malaiyandi:

They had never tested it out.

 

 


Prasanna Malaiyandi:

They were sort of trying things on the fly after they ran into issues.

 

 


Prasanna Malaiyandi:

W. Curtis Preston: Yeah, the part of that story that was the worst was

 

 


Prasanna Malaiyandi:

it's like, okay, we've been working on this for a couple weeks now.

 

 


Prasanna Malaiyandi:

We think we have a good plan for what, you know, for how

 

 


Prasanna Malaiyandi:

we're gonna get your data back.

 

 


Prasanna Malaiyandi:

Uh, you know, and then like a little bit more time passes and it's like,

 

 


Prasanna Malaiyandi:

oh, we've now tested the plan.

 

 


Prasanna Malaiyandi:

It works.

 

 


Prasanna Malaiyandi:

Now we need to go do the plan.

 

 


Prasanna Malaiyandi:

Like weeks are going by, so you're like, okay, so what you're telling us

 

 


Prasanna Malaiyandi:

is you're just making this up as you go along, that you never tested this before.

 

 


Prasanna Malaiyandi:

Uh, yeah, not, not good.

 

 


Prasanna Malaiyandi:

Um, the, um, uh, this next one here is that if your cloud vendor is

 

 


Prasanna Malaiyandi:

the low cost leader, it will show.

 

 


Prasanna Malaiyandi:

This was the case of OVH.

 

 


Prasanna Malaiyandi:

They were well acknowledged to be the low cost leader in that space, and it means

 

 


Prasanna Malaiyandi:

that they cut corners in a lot of areas.

 

 


Prasanna Malaiyandi:

They cut corners in terms of firefighting.

 

 


Prasanna Malaiyandi:

They cut corners in terms of not segregating the backup

 

 


Prasanna Malaiyandi:

data, all of that stuff,

 

 


Prasanna Malaiyandi:

Yeah.

 

 


Prasanna Malaiyandi:

I don't think there's anything wrong with being a low cost offering, as

 

 


Prasanna Malaiyandi:

long as people understand that and the expectations are there, right?

 

 


Prasanna Malaiyandi:

You get what you paid for, right?

 

 


Prasanna Malaiyandi:

Not everyone can afford the Ferrari they may buy, right?

 

 


Prasanna Malaiyandi:

People go buy the Geo Metro, right?

 

 


Prasanna Malaiyandi:

Way back in the day, so.

 

 


Prasanna Malaiyandi:

There are offerings for different people, and that totally makes sense.

 

 


Prasanna Malaiyandi:

But I think what becomes a challenge is when you're not transparent

 

 


Prasanna Malaiyandi:

about what people are getting or you set expectations incorrectly and

 

 


Prasanna Malaiyandi:

misrepresent what they are getting.

 

 


Prasanna Malaiyandi:

W. Curtis Preston: Yeah, agreed.

 

 


Prasanna Malaiyandi:

Um, and there are, and well, and, and again, if you're using a low cost

 

 


Prasanna Malaiyandi:

provider, we go back to rule number one.

 

 


Prasanna Malaiyandi:

Make sure that you're, make sure that you're backing up the data somewhere else.

 

 


Prasanna Malaiyandi:

I mean, you should do it like no matter what your provider is, but definitely if

 

 


Prasanna Malaiyandi:

you're doing a low cost provider, right.

 

 


Prasanna Malaiyandi:

Um, make sure that you, you know, because they're gonna, they're

 

 


Prasanna Malaiyandi:

gonna, they're gonna cut corners.

 

 


Prasanna Malaiyandi:

So the final one, uh, in our lessons learned is.

 

 


Prasanna Malaiyandi:

SSDs aren't perfect.

 

 


Prasanna Malaiyandi:

And I know Curtis, we've talked, I think we've had multiple episodes about this.

 

 


Prasanna Malaiyandi:

In fact, at home when, uh, I was looking for a new sort of, not really backup

 

 


Prasanna Malaiyandi:

drive, but just extra storage and debating between SSD or just getting spinning

 

 


Prasanna Malaiyandi:

disc, and I know chatting with you, I was like, yeah, we should probably get SS or.

 

 


Prasanna Malaiyandi:

Probably you should get a disc drive because we're not using the data often

 

 


Prasanna Malaiyandi:

and just the issues that we've seen around SSDs, if you're not using it all the time,

 

 


Prasanna Malaiyandi:

the data isn't guaranteed to be there.

 

 


Prasanna Malaiyandi:

And, uh.

 

 


Prasanna Malaiyandi:

This is one of those things where, uh, one of my favorite YouTube channels that

 

 


Prasanna Malaiyandi:

I watch all the time, life Uncontained, they lost a bunch of data on SSDs

 

 


Prasanna Malaiyandi:

and people think they're bulletproof.

 

 


Prasanna Malaiyandi:

Yes, there's less mechanical spinning parts versus traditional hard drives,

 

 


Prasanna Malaiyandi:

but they could still lose data.

 

 


Prasanna Malaiyandi:

W. Curtis Preston: Yeah, I, I think we could say that they, they probably

 

 


Prasanna Malaiyandi:

are more reliable than hard drives, especially if you're moving them around.

 

 


Prasanna Malaiyandi:

Right.

 

 


Prasanna Malaiyandi:

Um, you know, which is the case in your, obviously in your mobile

 

 


Prasanna Malaiyandi:

device or any sort of camera in the field, but they are not infallible

 

 


Prasanna Malaiyandi:

and there are some specific.

 

 


Prasanna Malaiyandi:

Um, ways that they behave that are unique to them.

 

 


Prasanna Malaiyandi:

Some u you know, some unique problems to SSDs.

 

 


Prasanna Malaiyandi:

One of them, you know, you alluded to there, is that it does require, um, you

 

 


Prasanna Malaiyandi:

know, a, a power to ensure that all of the voltages are there and over time.

 

 


Prasanna Malaiyandi:

If you're not using a given, um, SSD drive, those voltages can drop

 

 


Prasanna Malaiyandi:

and you can just lose data there.

 

 


Prasanna Malaiyandi:

There, um, I, I think we're, I think based on, we got a really good

 

 


Prasanna Malaiyandi:

message from one of our listeners, and I want to thank them for that.

 

 


Prasanna Malaiyandi:

Uh, I, I wanna do some more research and talk to somebody

 

 


Prasanna Malaiyandi:

who is a specialist in this area.

 

 


Prasanna Malaiyandi:

It's not my area of expertise.

 

 


Prasanna Malaiyandi:

I don't think it's yours either.

 

 


Prasanna Malaiyandi:

Uh, but, but the short version is.

 

 


Prasanna Malaiyandi:

That while SSDs and, and various other, you know, not just SSDs, but,

 

 


Prasanna Malaiyandi:

but, but solid state devices, uh, of every kind that, while they, I

 

 


Prasanna Malaiyandi:

think they are more reliable than the, uh, a standard hard drive,

 

 


Prasanna Malaiyandi:

especially if you're moving it around.

 

 


Prasanna Malaiyandi:

So they're great for laptops.

 

 


Prasanna Malaiyandi:

Uh, they are not infallible and they still need to be backed up.

 

 


Prasanna Malaiyandi:

Which brings us back to Rule zero.

 

 


Prasanna Malaiyandi:

And what's that

 

 


Prasanna Malaiyandi:

if it's important.

 

 


Prasanna Malaiyandi:

Back it up.

 

 


Prasanna Malaiyandi:

W. Curtis Preston: exactly?

 

 


Prasanna Malaiyandi:

So the cloud isn't perfect.

 

 


Prasanna Malaiyandi:

SSDs aren't perfect.

 

 


Prasanna Malaiyandi:

Uh, software engineers aren't perfect.

 

 


Prasanna Malaiyandi:

Admins aren't perfect.

 

 


Prasanna Malaiyandi:

All of these are why we back up and, um,

 

 


Prasanna Malaiyandi:

Don't forget malicious users are out there.

 

 


Prasanna Malaiyandi:

W. Curtis Preston: Of course malicious users, malicious

 

 


Prasanna Malaiyandi:

and, bad actors in general.

 

 


Prasanna Malaiyandi:

Right.

 

 


Prasanna Malaiyandi:

So, you know, like, like, uh, cyber attacks.

 

 


Prasanna Malaiyandi:

Only a few of the incidents that we covered were, were cyber attacks.

 

 


Prasanna Malaiyandi:

The rest were, uh, mistakes.

 

 


Prasanna Malaiyandi:

Um, there was a fire, trying to think what else, but generally it was just mistakes.

 

 


Prasanna Malaiyandi:

This is why we make backups.

 

 


Prasanna Malaiyandi:

Yeah.

 

 


Prasanna Malaiyandi:

and and I think it's also, and I think the other important thing.

 

 


Prasanna Malaiyandi:

Like these are just cases we found, right?

 

 


Prasanna Malaiyandi:

There are probably hundreds out there that aren't publicized, right?

 

 


Prasanna Malaiyandi:

The company's either gone out of business or no one noticed, right, that

 

 


Prasanna Malaiyandi:

their data was missing or other things.

 

 


Prasanna Malaiyandi:

These are just things that we have been able to find and at least some information

 

 


Prasanna Malaiyandi:

to be able to report out to our listeners on, Hey, here's what we see going on.

 

 


Prasanna Malaiyandi:

W. Curtis Preston: Exactly.

 

 


Prasanna Malaiyandi:

Well, this has been fun.

 

 


Prasanna Malaiyandi:

I hope that our, um, you know, listeners have enjoyed it as much as you and

 

 


Prasanna Malaiyandi:

I have enjoyed making the series.

 

 


Prasanna Malaiyandi:

Yeah.

 

 


Prasanna Malaiyandi:

Yeah.

 

 


Prasanna Malaiyandi:

No, this has been great, Curtis.

 

 


Prasanna Malaiyandi:

Thank you for suggesting this.

 

 


Prasanna Malaiyandi:

W. Curtis Preston: All right, well, uh, thanks to our listeners

 

 


Prasanna Malaiyandi:

and, uh, be sure to subscribe so that you don't miss an episode.

 

 


Prasanna Malaiyandi:

That is a wrap