Shocking RackSpace Hosted Exchange Ransomware Attack
Tomorrow marks two weeks from when the RackSpace outage started on Dec 2, 2022. They confirmed it was via a ransomware attack and it is not. going. well. We're going to do a deeper dive into this once it is all over, but this is a first-blush look at what is happening and RackSpace's reaction to it. When we recorded this episode, their reaction was not looking good. I'm sad to say it's gotten even worse. Check it out!
Mentioned in this episode:
Interview ad
Hi, and welcome to Backup Centrals Restore All podcast.
Prasanna Malaiyandi:I'm your host, W.
Prasanna Malaiyandi:Curtis Preston, a k a, Mr.
Prasanna Malaiyandi:Backup, and I have with me my medical non consultant, uh,
Prasanna Malaiyandi:Prasanna Malaiyandi How's it going?.
W. Curtis Preston:Good.
W. Curtis Preston:Curtis, how are you?
W. Curtis Preston:, how are you feeling?
Prasanna Malaiyandi:a as you know, it's been a rough week or two.
Prasanna Malaiyandi:Um, you know, and, and you know, and, and I'm, I'm now down to three,
Prasanna Malaiyandi:technically down to two medications.
W. Curtis Preston:Okay.
W. Curtis Preston:That's an
Prasanna Malaiyandi:one I can take, one I can take on demand.
Prasanna Malaiyandi:Uh, the other, um, or so, so two that I'm supposed to continue
Prasanna Malaiyandi:taking until they're gone.
Prasanna Malaiyandi:Um, it's been, it's been.
Prasanna Malaiyandi:Oh, you know, I don't
W. Curtis Preston:You, you, I do have to say though, you sound a lot better than
W. Curtis Preston:like three days ago when we talked so.
Prasanna Malaiyandi:Yeah.
Prasanna Malaiyandi:Yeah.
Prasanna Malaiyandi:Well in the worry, and we'll see if it happens on the podcast is if,
Prasanna Malaiyandi:is if I get to actually, um, you know, as you know, I like to laugh.
W. Curtis Preston:Yeah.
Prasanna Malaiyandi:But if I, if I start to get an actual deep laugh, I will cough.
W. Curtis Preston:Mm.
Prasanna Malaiyandi:There's this cough that, and apparently I did a little
Prasanna Malaiyandi:research that it is a common side effect of a leftover viral infection.
Prasanna Malaiyandi:Yeah.
Prasanna Malaiyandi:So, um, I had a, I had a, um, I got a really bad sinus infection from a tooth
Prasanna Malaiyandi:extraction, and then I got the flu.
Prasanna Malaiyandi:Um, and um, apparently we, you know, we went to reinvent, right?
Prasanna Malaiyandi:Um,
W. Curtis Preston:Couple of weeks ago.
W. Curtis Preston:No, last
Prasanna Malaiyandi:yeah.
Prasanna Malaiyandi:And it was last week.
Prasanna Malaiyandi:Yeah.
Prasanna Malaiyandi:And apparently almost the whole crew got sick when they got home.
Prasanna Malaiyandi:Um, either flu or covid.
Prasanna Malaiyandi:Um, so, you know, I don't know.
Prasanna Malaiyandi:It, it may cause us to, I wouldn't be surprised as if, if it causes us
Prasanna Malaiyandi:to have some sort of new procedure or policy or something, you know,
Prasanna Malaiyandi:because I was supposed to go to another trade show right after,
W. Curtis Preston:yeah, and you.
Prasanna Malaiyandi:that didn't, that
W. Curtis Preston:Well, I know in the news recently, and once again, this
W. Curtis Preston:is not medical advice, but just kinda keeping people up to date, right.
W. Curtis Preston:In my, the county that I live in, they just actually were flagging it and
W. Curtis Preston:saying, yeah, the number of cases are currently on the rise, and especially
W. Curtis Preston:with the holidays and all the rest, they're like, be safe, be vigilant.
W. Curtis Preston:Mask up if you can stay home if you're sick.
Prasanna Malaiyandi:And what sucks is, you know, like I'm, you
Prasanna Malaiyandi:know, I, I, I respect those who want to continue wearing a mask.
Prasanna Malaiyandi:I have no issue if you wanna wear a mask.
Prasanna Malaiyandi:I Prasannally, and I wore them when it was, you know, when I
Prasanna Malaiyandi:thought, When it a, when it was required, and I didn't complain.
Prasanna Malaiyandi:I'm currently in the face of like, I am so done with the mask
Prasanna Malaiyandi:And so I wear a mask when I, you know, when I kind of have to, uh, but like, I
Prasanna Malaiyandi:didn't want to be the guy walking or I didn't want to be, like, out of all of
Prasanna Malaiyandi:the attendees of the 60,000 attendees
Prasanna Malaiyandi:right there there were like 20 of them were wearing masks and I
Prasanna Malaiyandi:didn't want to be the 21st, you
Prasanna Malaiyandi:know?
W. Curtis Preston:And I think that's the hard part, right?
W. Curtis Preston:It's like peer pressure, societal pressure, right?
W. Curtis Preston:It's like, Hey, I shouldn't be any different than everyone else, and.
W. Curtis Preston:. I think it's one of those
Prasanna Malaiyandi:And also
W. Curtis Preston:has to sort of judge and figure out their own risk
W. Curtis Preston:and figure out what they want to do.
Prasanna Malaiyandi:yeah.
Prasanna Malaiyandi:Um, the, um, the mask thing came up on, um, Sebastian Maniscalco,
Prasanna Malaiyandi:which is a comedian that I love, and his most recent thing.
Prasanna Malaiyandi:Um, He was talking about.
Prasanna Malaiyandi:He said that he got the vaccine, not, he didn't get it for him.
Prasanna Malaiyandi:He's like, I didn't get it for you.
Prasanna Malaiyandi:He says, I got it for the same reason.
Prasanna Malaiyandi:Uh uh.
Prasanna Malaiyandi:All the Italians.
Prasanna Malaiyandi:No, the Italians.
Prasanna Malaiyandi:He said the Italians did it because he said we found out that we
Prasanna Malaiyandi:couldn't taste food if we got Covid.
Prasanna Malaiyandi:So he is like, so that's why he got the vaccine?
Prasanna Malaiyandi:Uh,
W. Curtis Preston:But I'm glad that you're feeling better.
Prasanna Malaiyandi:I should throw out our disclaimer that, uh, this is not a
Prasanna Malaiyandi:Druva podcast, not a Zoom podcast either.
Prasanna Malaiyandi:Where, where, uh, Prasanna happens to work.
Prasanna Malaiyandi:Uh, these opinions are ours, and, uh, none of this is medical or legal advice.
Prasanna Malaiyandi:God forbid, or, you know, red, I'll tell you what, we, we give official device
Prasanna Malaiyandi:advice, but, uh, you know, gadget, gadget,
W. Curtis Preston:and put the holidays around the corner.
W. Curtis Preston:By the way, FYI, for those, I don't know when this podcast is
W. Curtis Preston:going out, but Best Buy is running a sale on those Ember Smart Mugs.
W. Curtis Preston:If you are looking for one, now is the time to buy one.
Prasanna Malaiyandi:Yeah.
Prasanna Malaiyandi:I think it's gonna be too late by the time this podcast goes out,
Prasanna Malaiyandi:but maybe it'll be in time for you to buy, buy me a birthday, a gift.
Prasanna Malaiyandi:It'll be coming up on my, my 57th birthday.
W. Curtis Preston:Dang.
W. Curtis Preston:Curtis.
Prasanna Malaiyandi:Getting, getting old up there.
Prasanna Malaiyandi:Not as old as Stewart though.
Prasanna Malaiyandi:Hey, Stewart
W. Curtis Preston:uh, Stewart, I think we're gonna need to have
W. Curtis Preston:Stewart come back on the podcast to have a conversation with you.
Prasanna Malaiyandi:Yeah, yeah.
Prasanna Malaiyandi:Um, so I, I wanted to talk about this.
Prasanna Malaiyandi:You know, the story that we're gonna talk about today and, um, actually
Prasanna Malaiyandi:two stories, but the, the main story, it's one of these where you're like,
Prasanna Malaiyandi:it falls into the, are you kidding me?
Prasanna Malaiyandi:Category, right?
Prasanna Malaiyandi:And it also falls into the, um, the story helps prove a
Prasanna Malaiyandi:couple of points of mine, right?
Prasanna Malaiyandi:And you, you will see me.
Prasanna Malaiyandi:You will hear me talking about those points here on this podcast today.
Prasanna Malaiyandi:So we're talking of course.
Prasanna Malaiyandi:What are we talking about?
Prasanna Malaiyandi:Prasanna?
W. Curtis Preston:Uh, what are we talking about?
W. Curtis Preston:Oh, yeah, so, uh, the recent, yeah, the recent outage that happened at Rackspace,
W. Curtis Preston:I want to say it was December 2nd,
Prasanna Malaiyandi:Yeah, that was the beginning of it.
W. Curtis Preston:Yeah.
W. Curtis Preston:and it's now December 8th and people are still not, or the
W. Curtis Preston:service is still not up and running.
Prasanna Malaiyandi:and Rackspace as of now, their official line is,
Prasanna Malaiyandi:we don't know when this is gonna end.
Prasanna Malaiyandi:We don't know when or if we're gonna be able to restore data.
W. Curtis Preston:So, so, so maybe just a quick background on Rackspace for
W. Curtis Preston:the listeners who may not be familiar.
W. Curtis Preston:Right.
W. Curtis Preston:So Rackspace is a, what would you call 'em?
W. Curtis Preston:They're kind of.
Prasanna Malaiyandi:where they have a bunch of racks,
W. Curtis Preston:Well, it's kind of like Amazon before, or AWS before aws, right.
W. Curtis Preston:In a sense, they were kind
Prasanna Malaiyandi:Yeah.
Prasanna Malaiyandi:They're, they're, yeah.
Prasanna Malaiyandi:Yeah.
Prasanna Malaiyandi:They're, they're, they're closer to a colo than a, than a cloud facility, I would
W. Curtis Preston:Yeah.
W. Curtis Preston:But they do offer managed services, including what got hit,
W. Curtis Preston:which is their hosted exchange.
W. Curtis Preston:It.
W. Curtis Preston:Environment, so it's not Microsoft 365 that they're just sort of proxying
W. Curtis Preston:through and buying Microsoft 365 licenses for this is, they're running exchange
W. Curtis Preston:servers in their environment, right?
W. Curtis Preston:And basically giving you customers a similar sort of experience that you
W. Curtis Preston:would get with a SaaS service, right?
W. Curtis Preston:So they're managing all of the infrastructure, the email servers,
W. Curtis Preston:provisioning accounts, everything else.
W. Curtis Preston:And you as a customer, you're just.
W. Curtis Preston:email service provided by Rackspace.
Prasanna Malaiyandi:Let me ask you a question.
W. Curtis Preston:Mm-hmm.
Prasanna Malaiyandi:I understand why somebody would, I mean, I disagree, but
Prasanna Malaiyandi:I understand , why somebody would use on-premises exchange over Microsoft 365.
Prasanna Malaiyandi:What does a company gain by using hosted, uh, Microsoft Exchange?
Prasanna Malaiyandi:That I don't, I don't
W. Curtis Preston:So I, I think it comes back to a couple things, right?
W. Curtis Preston:One is, are they using Rackspace for other services already?
W. Curtis Preston:And this is just yet another thing that they're just using Rackspace for, right?
W. Curtis Preston:That could be one.
W. Curtis Preston:The second is maybe they have certain compliance regulations or other things
W. Curtis Preston:which they feel cannot be fulfilled by.
W. Curtis Preston:I, I, I know it's, but it's a very
Prasanna Malaiyandi:that
W. Curtis Preston:it's a very niche.
Prasanna Malaiyandi:no, no, but that's why I ask that question because if
Prasanna Malaiyandi:you, if you're not compliant sitting on Microsoft Services, why are you compliant
Prasanna Malaiyandi:sitting on somebody else's services?
W. Curtis Preston:be, well, it might be that Rackspace has
W. Curtis Preston:found a differentiator, right?
W. Curtis Preston:Or provide the value add, right?
Prasanna Malaiyandi:They must, they must.
Prasanna Malaiyandi:Maybe it's uptime
W. Curtis Preston:Yeah.
W. Curtis Preston:Maybe, well, maybe it's uptime, maybe it's replication.
W. Curtis Preston:Maybe they're offering backup.
W. Curtis Preston:We'll get to that later, right?
W. Curtis Preston:But there are all these other things potentially you could
W. Curtis Preston:be adding as a value add.
W. Curtis Preston:right?
W. Curtis Preston:In addition to just what Microsoft provides, right?
W. Curtis Preston:Or maybe it's like an e-discovery compliance style thing that
W. Curtis Preston:they're also providing in.
W. Curtis Preston:It's hard to tell cuz I don't know what they offer fully for the managed services.
W. Curtis Preston:Or it could be maybe there are certain data residency requirements
W. Curtis Preston:that aren't met by Microsoft today because of how they operate.
Prasanna Malaiyandi:Speaking of which I'd like to off, I'd
Prasanna Malaiyandi:like to announce another sale.
W. Curtis Preston:Okay.
W. Curtis Preston:What's your next.
Prasanna Malaiyandi:Uh, right now Rackspace stock is on sale.
Prasanna Malaiyandi:Um,
W. Curtis Preston:Oh, Curtis.
Prasanna Malaiyandi:it went from, of a high of five this
Prasanna Malaiyandi:week down to three something.
Prasanna Malaiyandi:Um, yeah, it's been, it's been taking a hit.
Prasanna Malaiyandi:And, and, and also, um, There there's been the announcement of at least one class
Prasanna Malaiyandi:action lawsuit, um, on business wire.
Prasanna Malaiyandi:Um, Cole and Van note announces filing of Rackspace ransomware
Prasanna Malaiyandi:data breach class action.
Prasanna Malaiyandi:So let's talk about what you know, cuz there's like a half a dozen,
Prasanna Malaiyandi:I don't know, plus stories or you know, various stories out there.
Prasanna Malaiyandi:What do we know about the outage?
Prasanna Malaiyandi:So,
W. Curtis Preston:So, so far, we know that on December 2nd
W. Curtis Preston:they brought down their services.
W. Curtis Preston:They said they noticed a security incident that mainly,
Prasanna Malaiyandi:it took, it took them a day, as far as I recall.
Prasanna Malaiyandi:It took them a day to say it was a security incident.
W. Curtis Preston:
Speaker:Well, they noticed some.
W. Curtis Preston:
Speaker:Issue and they brought down their environment.
Prasanna Malaiyandi:right,
W. Curtis Preston:Um, I believe they only brought down their exchange hosted
W. Curtis Preston:environment, not all the other services.
Prasanna Malaiyandi:In fact, I think.
Prasanna Malaiyandi:Right.
Prasanna Malaiyandi:Yes.
Prasanna Malaiyandi:Yes.
W. Curtis Preston:it was limited to just that.
W. Curtis Preston:And so they brought everything down and they kept everything down.
W. Curtis Preston:Like you said, they investigated it.
W. Curtis Preston:They then published to people saying we had a security incident,
W. Curtis Preston:and I think that was December 3rd.
W. Curtis Preston:and then they've been doing periodic updates, I would say, of where they're at.
W. Curtis Preston:But it's just more of the, we're still investigating.
W. Curtis Preston:We're still investigating.
W. Curtis Preston:We don't have a time yet, but yeah.
Prasanna Malaiyandi:the comment in the lawsuit, it referred to him
Prasanna Malaiyandi:as a numerous, uh, very opaque, um, you know, uh, announcements, right?
Prasanna Malaiyandi:That, that, that did, they didn't offer.
Prasanna Malaiyandi:We're looking into it, you know, that was basically what they offered.
W. Curtis Preston:and for companies, right?
W. Curtis Preston:This is their email, right?
W. Curtis Preston:. This is, uh, really critical for a lot of companies in order to do business and
W. Curtis Preston:being down for six, seven days with no e t A on when they're going to be fixed.
W. Curtis Preston:Or what the process is cuz they haven't even talked about what the
W. Curtis Preston:recovery mechanisms are either.
W. Curtis Preston:And I know we'll talk a lit bit later, Curtis, about sort of what they're
W. Curtis Preston:offering for a Band-Aid solution I guess, right now or, but yeah.
W. Curtis Preston:So as of right now though, all those customers are a little hosed.
Prasanna Malaiyandi:Yeah, I mean it's been, we're now on December 8th.
Prasanna Malaiyandi:This thing is still ongoing.
Prasanna Malaiyandi:I'm hoping that by the time this episode publishes the,
W. Curtis Preston:They'll be back up and run.
Prasanna Malaiyandi:one way or the other.
W. Curtis Preston:Yeah, but could you just imagine as a, like when
W. Curtis Preston:you had your company, Curtis, if your email went down for a week,
W. Curtis Preston:what would the impact be for you?
Prasanna Malaiyandi:I mean, it just, it is just ridiculous, right?
Prasanna Malaiyandi:I, I, I suppose, well, we'd be, you know, really unable to communicate
Prasanna Malaiyandi:with outsiders, which is kind of the, the point of a company, right?
Prasanna Malaiyandi:Uh, maybe you'd be able to talk to a few people via chat and phone
Prasanna Malaiyandi:calls and whatnot, but email.
Prasanna Malaiyandi:Is such a critical part of, of a typical company that the idea of email
Prasanna Malaiyandi:going down for at this point almost a week or more, uh, is just unthinkable.
Prasanna Malaiyandi:I, I, I can't imagine the, the, the cost that they have, uh, that their,
Prasanna Malaiyandi:that their clients have incurred.
Prasanna Malaiyandi:They're going to incur costs, they're gonna inc.
Prasanna Malaiyandi:Reputation costs are gonna incur in financial costs.
Prasanna Malaiyandi:Uh, I think when this all comes out, this is gonna be, I think this is gonna
Prasanna Malaiyandi:be very bad for the likes of Rackspace.
Prasanna Malaiyandi:Now let's talk
W. Curtis Preston:oh, before you get, uh, it's also going to be interesting.
W. Curtis Preston:So, two things I wanna throw out there.
W. Curtis Preston:One is, it'll be interesting once this is all done, if they
W. Curtis Preston:continue to stay in the hosting.
W. Curtis Preston:email hosting business.
W. Curtis Preston:Right.
W. Curtis Preston:That'll be one interesting thing.
W. Curtis Preston:I did see a number that said that it's supposed to be a 30
W. Curtis Preston:million a year business for
Prasanna Malaiyandi:Yeah.
W. Curtis Preston:Right.
W. Curtis Preston:Which isn't peanuts.
W. Curtis Preston:But at the same time, if you don't have customers who are coming, right,
W. Curtis Preston:I'm sure there are a lot of customers who are like, I don't think I'm gonna
W. Curtis Preston:stick around with Rackspace for my.
Prasanna Malaiyandi:yeah, exactly.
Prasanna Malaiyandi:Um, so there was this, uh, or is this guy, his name's Kevin Beaumont.
Prasanna Malaiyandi:So he is a security researcher and he was affected by the hosted exchange
W. Curtis Preston:Hmm.
Prasanna Malaiyandi:and, um, or he noticed it or something.
Prasanna Malaiyandi:I don't know.
Prasanna Malaiyandi:Um, I don't know if he, um,
W. Curtis Preston:If he's a customer,
Prasanna Malaiyandi:I don't know if he was effective or, yeah.
Prasanna Malaiyandi:But basically he, he just started poking around and he was looking at their.
Prasanna Malaiyandi:Their email services and, um, he noticed the version that the, the, the version of
Prasanna Malaiyandi:Microsoft Exchange that they were running is apparently very old and is PR is, and
Prasanna Malaiyandi:it's before there's, there's something called, uh, proxy Nutshell and that they
Prasanna Malaiyandi:were the patches that were available.
Prasanna Malaiyandi:Um, according to this, the version that they're running is from August.
W. Curtis Preston:and it was patched in September.
Prasanna Malaiyandi:
Speaker:yeah, which is passed in
W. Curtis Preston:I, I think I read something that said one server was
W. Curtis Preston:unpatched, but I believe many of their other servers were patched.
Prasanna Malaiyandi:Well, what I, what I, what he said later on in the article
Prasanna Malaiyandi:was, the way this works is all it takes is
W. Curtis Preston:One.
W. Curtis Preston:Yep.
Prasanna Malaiyandi:In fact, all it takes is one account,
Prasanna Malaiyandi:one compromised account on one.
Prasanna Malaiyandi:Compromised system, uh, and then you're in.
Prasanna Malaiyandi:Right.
Prasanna Malaiyandi:Um,
W. Curtis Preston:which is I think how they moved horizontally across
W. Curtis Preston:the entire environment, right?
Prasanna Malaiyandi:exactly right.
Prasanna Malaiyandi:Uh, and, and he, and he goes on to, to, to basically point out to people who are,
Prasanna Malaiyandi:who actually have managed, um, Exchange to, or you know, whether they're managing
Prasanna Malaiyandi:it themselves or somebody else is managing it to basically say, listen, you, you
Prasanna Malaiyandi:need to be running this past version.
Prasanna Malaiyandi:Right.
Prasanna Malaiyandi:Um, and in fact there have been two, there have been two versions of
Prasanna Malaiyandi:the software since that August 9th version that they have not patched,
W. Curtis Preston:and it's kind of scary, like you think about patch
W. Curtis Preston:management, especially for these critical vulnerabilities, right?
W. Curtis Preston:It's how do you schedule downtime to be able to apply the patches in
W. Curtis Preston:the right order at the time, right?
W. Curtis Preston:Maybe they applied it for all the other systems.
W. Curtis Preston:This one maybe fell through the cracks, or maybe they had a problem trying to update
W. Curtis Preston:this one, so they're like, ah, we'll just get to it on the next patch cycle.
W. Curtis Preston:You.
Prasanna Malaiyandi:Yeah,
W. Curtis Preston:I, I don't know what actually happened, by the way,
W. Curtis Preston:this is all hypothetical, right?
W. Curtis Preston:Or
Prasanna Malaiyandi:Yeah, it's all hypothetical.
Prasanna Malaiyandi:Should, yeah, that, that this is all, this is what appears to happen based
Prasanna Malaiyandi:on the information we have available.
Prasanna Malaiyandi:And we, we also don't know.
Prasanna Malaiyandi:There's no evidence.
Prasanna Malaiyandi:And again, Rackspace isn't helping with its o o opacity.
Prasanna Malaiyandi:Would that be the, would that be the right word?
Prasanna Malaiyandi:Opaqueness.
Prasanna Malaiyandi:O I think opacity seems like the right word anyway.
Prasanna Malaiyandi:Uh, they're not really saying much.
Prasanna Malaiyandi:Um, but we don't know.
Prasanna Malaiyandi:Also, we don't know that,
Prasanna Malaiyandi:like, It it appears that they had a hu that, that they had the, this
Prasanna Malaiyandi:unpatched or yeah, this unpatched server.
Prasanna Malaiyandi:But we don't know that that's what caused the outage.
Prasanna Malaiyandi:But we do know well.
Prasanna Malaiyandi:Based on the information we have, we do know that they weren't up on
Prasanna Malaiyandi:their patches, which is, you know, this is one of the reasons why you
Prasanna Malaiyandi:go with a hosted provider, right?
Prasanna Malaiyandi:Is that they're, is that they're gonna handle all of these problems
Prasanna Malaiyandi:that you don't, you know, you don't want to handle yourself.
W. Curtis Preston:Yeah, you're like, I wanna focus on my business and not set
W. Curtis Preston:it up and managing email infrastructure.
Prasanna Malaiyandi:It's very to, to, to put it mildly.
Prasanna Malaiyandi:It's very disappointing on the part of Rackspace.
Prasanna Malaiyandi:Um, I, you know, I actually did some, um, some consulting work for
Prasanna Malaiyandi:Rackspace back a hundred years ago.
Prasanna Malaiyandi:Uh, and they seemed like a, you know, a smart group of people.
Prasanna Malaiyandi:I was in there to put in a, what would now be a competitor, a Druva.
Prasanna Malaiyandi:Um, they were, They were, it was, it was a rocky install.
Prasanna Malaiyandi:That's what I remember, . It was a rocky install.
Prasanna Malaiyandi:Um, but it's interesting.
Prasanna Malaiyandi:So, so, so we have this thing with the patching that we don't have it, it appears
Prasanna Malaiyandi:that they were behind on their patches.
Prasanna Malaiyandi:Why, why did they patch most of them, but, but not one of them.
Prasanna Malaiyandi:Why?
Prasanna Malaiyandi:I don't know.
Prasanna Malaiyandi:It means that they've, that would, that would suggest that they don't
Prasanna Malaiyandi:have a, a strong patch management.
Prasanna Malaiyandi:Process.
Prasanna Malaiyandi:Right.
Prasanna Malaiyandi:Um, and then the other question is, why is it a week in and they haven't been able to
Prasanna Malaiyandi:restore their services and there's really only one valid answer to that question.
Prasanna Malaiyandi:And by valid,
W. Curtis Preston:I think there are two.
W. Curtis Preston:Oh,
Prasanna Malaiyandi:okay.
W. Curtis Preston:No, actually you're right.
W. Curtis Preston:No, there are no valid answers.
W. Curtis Preston:Really.
Prasanna Malaiyandi:We, there are no valid answers, but there
Prasanna Malaiyandi:are, there are phrases that can answer the question that I posed.
Prasanna Malaiyandi:They, I don't think they're valid.
Prasanna Malaiyandi:I, I, I would say one, you know, they, they don't want to pay their ransom.
Prasanna Malaiyandi:Right.
Prasanna Malaiyandi:Um, but the, and I hope for their sake from a company, History standpoint,
Prasanna Malaiyandi:I hope for their sake that this isn't an extortion ransomware situation.
Prasanna Malaiyandi:Right?
W. Curtis Preston:Yep.
W. Curtis Preston:Where someone has exfiltrated some data and now they're like, Hey, pay up.
W. Curtis Preston:Or
Prasanna Malaiyandi:cuz if you're hosted exchange, you've got dozens,
Prasanna Malaiyandi:hundreds of companies in there.
W. Curtis Preston:Yeah.
Prasanna Malaiyandi:By the way, they're saying that this is, um, a portion of.
Prasanna Malaiyandi:Hosted exchange environment, by the way, but apparently
Prasanna Malaiyandi:the outage is affecting all of
W. Curtis Preston:Yeah.
W. Curtis Preston:I think another,
Prasanna Malaiyandi:yeah.
Prasanna Malaiyandi:Go ahead.
W. Curtis Preston:I think another valid reason that it could be right is, do
W. Curtis Preston:you remember when we had Tony Mendoza from Spectralogic on when he was talking
W. Curtis Preston:about how they recovered from ransomware?
W. Curtis Preston:, right.
W. Curtis Preston:I think a lot of it is, do you need to bring in those experts?
W. Curtis Preston:Hopefully they had cyber insurance.
Prasanna Malaiyandi:They, they according well, yeah.
Prasanna Malaiyandi:What we do know is that they have a quote leading cybersecurity
Prasanna Malaiyandi:firm, , and they're helping them out.
Prasanna Malaiyandi:Yeah.
W. Curtis Preston:So hopefully they're able, I thought it
W. Curtis Preston:was Mandiant or one of those.
W. Curtis Preston:I may have stumbled across that in some article
Prasanna Malaiyandi:Yeah.
Prasanna Malaiyandi:I just saw the phrase, leading, leading company.
W. Curtis Preston:So I think.
W. Curtis Preston:one is it takes time to bring them up to do the investigation, to
W. Curtis Preston:figure out what they need to recover.
W. Curtis Preston:So I'm thinking that doing that and then also making sure you have, because
W. Curtis Preston:that's the one thing that stood out for me from talking to Tony Mendoza, was you
W. Curtis Preston:needed to have sort of clean machines that you could start using for restores.
W. Curtis Preston:Otherwise it's just gonna continue propagating, right?
W. Curtis Preston:And so finding the hardware, right, because this is a managed company, right?
W. Curtis Preston:So they probably have servers in rack, so procuring the
Prasanna Malaiyandi:You would think if they had, if, if, if a company would have.
Prasanna Malaiyandi:, you know, I, I'm, I'm guessing they have an entire space just for racks.
Prasanna Malaiyandi:Uh, uh, uh,
W. Curtis Preston:so, but, but finding the equipment right and then identifying
W. Curtis Preston:the points in time, which are valid.
W. Curtis Preston:Right.
W. Curtis Preston:Standing up the infrastructure, the networking, right, making sure that
W. Curtis Preston:there are no further security issues.
W. Curtis Preston:Granted, seven days seems kind of long, but I thought
W. Curtis Preston:that's kind of what Tony said.
W. Curtis Preston:It took 'em before they started getting up
Prasanna Malaiyandi:two weeks.
W. Curtis Preston:two weeks before, well before they were fully up and
W. Curtis Preston:running, but I think it was about a week before they could finally
W. Curtis Preston:start doing restores and bringing up
Prasanna Malaiyandi:I, I, I don't remember by, by the way,
Prasanna Malaiyandi:that was a really good episode.
Prasanna Malaiyandi:Uh, it was ransomware victim tells their story.
Prasanna Malaiyandi:Um, the, um, I, it's funny, you, you, you, you actually gave
Prasanna Malaiyandi:them a, that's an actual valid
W. Curtis Preston:that that is a valid, that's why I said yeah,
Prasanna Malaiyandi:Yeah.
Prasanna Malaiyandi:Yeah.
Prasanna Malaiyandi:That's pro.
Prasanna Malaiyandi:Possibly the only valid answer is, hey, This is hard.
W. Curtis Preston:Yeah.
W. Curtis Preston:But they should be more transparent, you know?
Prasanna Malaiyandi:they should be more transparent.
Prasanna Malaiyandi:Why don't companies understand that?
Prasanna Malaiyandi:I don't get that.
Prasanna Malaiyandi:Um, but the worry is because due to the lack of transparency, is that,
Prasanna Malaiyandi:is that they don't have backups.
Prasanna Malaiyandi:They're so, so what, what have they done in the meantime for
Prasanna Malaiyandi:customers who just can't do.
W. Curtis Preston:Yeah.
W. Curtis Preston:So this is the thing that I saw in, that they were actually recommending is they're
W. Curtis Preston:like, Hey, we went and procured Microsoft 365 licenses for you, so you can stand
W. Curtis Preston:up your email and continue operating.
W. Curtis Preston:And they're gonna help them set up, uh, what is it, forwarding rules.
W. Curtis Preston:So
Prasanna Malaiyandi:Yeah, so that incoming email will go to
W. Curtis Preston:And they have like the Microsoft, uh, team on
W. Curtis Preston:standby to help customers, right.
Prasanna Malaiyandi:Meanwhile, there's, there's all these, there's all these
Prasanna Malaiyandi:email servers out there that have been trying to send mail to cus to, to, uh,
Prasanna Malaiyandi:Microsoft, uh, to Rackspace customers.
Prasanna Malaiyandi:And they're we're, we're trying to send the mail.
Prasanna Malaiyandi:We're trying to send the mail.
Prasanna Malaiyandi:Nobody's taking it.
W. Curtis Preston:yeah.
W. Curtis Preston:Well, and the thing is, even with this solution, right, it's only going forward.
W. Curtis Preston:You're receiving new emails, right?
W. Curtis Preston:All your old stuff is, oh, who knows what happened to that, right?
Prasanna Malaiyandi:they did say that they provided for some
Prasanna Malaiyandi:customers an archive of, of emails.
W. Curtis Preston:I thought they said they were trying to
W. Curtis Preston:get to that, but they don't yet.
Prasanna Malaiyandi:well, I thought that they gave it for some, but not all.
Prasanna Malaiyandi:Um, by the way, I was, I was, that, that's a clue for me, the fact
Prasanna Malaiyandi:that they said they had archives of the email, but not backups,
W. Curtis Preston:I was looking at the article or from
Prasanna Malaiyandi:Uhhuh,
W. Curtis Preston:we are working to provide customers with archive of
W. Curtis Preston:inboxes where available to eventually import over to Microsoft 365.
Prasanna Malaiyandi:Right.
W. Curtis Preston:So it may
Prasanna Malaiyandi:the word archive is a bit,
W. Curtis Preston:concerning.
Prasanna Malaiyandi:you know?
Prasanna Malaiyandi:Yeah.
Prasanna Malaiyandi:Uh, archive is different than backup.
Prasanna Malaiyandi:You know, we have an episode on that.
Prasanna Malaiyandi:Please go listen to that if you have it.
Prasanna Malaiyandi:Um, the, um, Yeah, I don't know.
Prasanna Malaiyandi:I'm just not, I'm not, I don't have a good feeling here on,
W. Curtis Preston:a little queasy in your tummy and it's not from being sick.
Prasanna Malaiyandi:Yeah.
Prasanna Malaiyandi:Yeah.
Prasanna Malaiyandi:Um, I, I don't, I don't know what they're doing over there.
Prasanna Malaiyandi:Um, like I said, it, it all, it all starts with why didn't they have the
Prasanna Malaiyandi:service patch in the first place?
Prasanna Malaiyandi:That's, that's the first concerning thing.
Prasanna Malaiyandi:Um, the fact that we're a weekend and they're not saying that, you know,
Prasanna Malaiyandi:if they had said, l listen, we're we are, we have to do a server by server
Prasanna Malaiyandi:scrub to, to verify that the, you know, we've identified the malware.
Prasanna Malaiyandi:We now have to do a server by server scrub to make sure the malware isn't anywhere
Prasanna Malaiyandi:else and we need to wipe the servers.
Prasanna Malaiyandi:You know, but they're not saying anything.
Prasanna Malaiyandi:They're just saying We're, we're working on
W. Curtis Preston:yeah.
W. Curtis Preston:The other thing I wanna know,
Prasanna Malaiyandi:any, yeah.
Prasanna Malaiyandi:Go.
W. Curtis Preston:the other thing I'm wondering though is I know
W. Curtis Preston:you're talking about backup, right?
W. Curtis Preston:And restoring data.
W. Curtis Preston:Do they not have a DR environment?
W. Curtis Preston:You know, I know we, you know, we talk about this all day.
W. Curtis Preston:Or maybe their DR.
W. Curtis Preston:Environment was compromised as well.
W. Curtis Preston:Right, and so we always talk about, right, you need a DR environment for
W. Curtis Preston:situations like this where you can quickly spin up and continue operating
W. Curtis Preston:rather than trying to go back and restore your data, keep those backups
W. Curtis Preston:just in case so you can restore them.
W. Curtis Preston:But you really should have a DR environment so you're not
W. Curtis Preston:spending six plus days trying to bring up your environment.
Prasanna Malaiyandi:But if you have a DR environment and you're doing
Prasanna Malaiyandi:host exchange, you're gonna have to be doing some sort of real time
Prasanna Malaiyandi:replication in order to, to have that DR environment be, uh, you know, effective.
W. Curtis Preston:Yep.
W. Curtis Preston:And maybe they're doing
Prasanna Malaiyandi:not, I'm not saying I, I can't.
W. Curtis Preston:Yeah.
W. Curtis Preston:And, but even that replication, right?
W. Curtis Preston:If it, so assuming the ransomware did not go horizontally into that DR
W. Curtis Preston:environment, which is a big assumption.
W. Curtis Preston:If you were doing exchange level replication at the application
W. Curtis Preston:level, hopefully your DR site shouldn't have been compromised
Prasanna Malaiyandi:It has a delayed, there's a delayed replication
W. Curtis Preston:And you're also doing it at the application level, right?
W. Curtis Preston:Rather
Prasanna Malaiyandi:Right, right,
W. Curtis Preston:at the database object level.
W. Curtis Preston:Right.
W. Curtis Preston:So
Prasanna Malaiyandi:Yeah.
Prasanna Malaiyandi:Yeah, that makes sense.
Prasanna Malaiyandi:It's interesting, you know, it's like, well, did it, did it attack
Prasanna Malaiyandi:exchange, or did it attack windows?
Prasanna Malaiyandi:We don't know.
Prasanna Malaiyandi:We don't know anything.
Prasanna Malaiyandi:We don't know anything.
Prasanna Malaiyandi:Prasanna, why don't we know anything?
W. Curtis Preston:That's how these things go.
W. Curtis Preston:Well, hopefully they publish more information.
W. Curtis Preston:I'm not holding my breath for that though, but I think it could be a good
W. Curtis Preston:learning opportunity because I wanna say that the US government, right, had
W. Curtis Preston:a big push for patch exchange servers because of these specific issues.
W. Curtis Preston:Like a couple months ago, I think there was like a cisa.
Prasanna Malaiyandi:well that would've been helpful a couple of months
W. Curtis Preston:I thought so.
W. Curtis Preston:I could be wrong, but I thought there was something
Prasanna Malaiyandi:So, uh, I want to tack on.
Prasanna Malaiyandi:, um, basically put the, put, you know, on, on a related note to this, cause
Prasanna Malaiyandi:I'm tired of talking about Rackspace.
Prasanna Malaiyandi:It's too depressing.
Prasanna Malaiyandi:Uh, I wanna talk about a company I'd never heard of before.
Prasanna Malaiyandi:Um, they're referred to as, uh, south Korea's, um, Google, and that
Prasanna Malaiyandi:is the name of the, I don't know if, I don't know if I'm pronouncing
Prasanna Malaiyandi:it right, but it looks like Ka.
Prasanna Malaiyandi:um, like, it's like, it's like the way cocoa is spelled properly, but except
Prasanna Malaiyandi:with Case That's the, so I'm sort of cacao, that's how I'm pronouncing it.
Prasanna Malaiyandi:Like, like, you know, like the, the Bean for, for chocolate.
Prasanna Malaiyandi:Um, so they,
W. Curtis Preston:This article we found on the register,
Prasanna Malaiyandi:yeah.
Prasanna Malaiyandi:On the register.
Prasanna Malaiyandi:Um, so, oh, it's from si Simon.
Prasanna Malaiyandi:I know Simon.
Prasanna Malaiyandi:Hi Simon.
Prasanna Malaiyandi:I don't know if he
W. Curtis Preston:what was it, title of the article just for
Prasanna Malaiyandi:Uh, well there's two articles.
Prasanna Malaiyandi:There's Data Center Fire Takes Out South Korea's Top Two Web
Prasanna Malaiyandi:Giants, and then a follow up article that was back in October.
Prasanna Malaiyandi:A follow up article is it's 2022 and a Korean web Giant only now decided to
Prasanna Malaiyandi:write a Dr Plan So, uh, the first part is reminiscent of, um, the O V H fire,
Prasanna Malaiyandi:and that is that these guys apparently, What I'm, again, what I'm deducing there,
Prasanna Malaiyandi:there were two web giants, Neve and Cacao.
Prasanna Malaiyandi:They both experienced service interruptions after a data center that
Prasanna Malaiyandi:hosts much of their infrastructure was shut down by a Sunday fire.
Prasanna Malaiyandi:So they, they are Google-like, but they're, they're, uh, in that they
Prasanna Malaiyandi:have like Facebook and messaging and a lot of different stuff like that.
Prasanna Malaiyandi:I don't think they're search engine type stuff, but they.
Prasanna Malaiyandi:They and this other web giant, um, were taken down by this data center fire,
Prasanna Malaiyandi:which as we, again, we, we can only infer from things they, they're, they're
Prasanna Malaiyandi:hosted in somebody else's data center.
Prasanna Malaiyandi:It's not their data center.
Prasanna Malaiyandi:And, and it's not a cloud data center
W. Curtis Preston:it's like a cola.
Prasanna Malaiyandi:a data center.
Prasanna Malaiyandi:Yeah, it's like a colo and then the, the, the.
Prasanna Malaiyandi:Sort of outcome from that is that this company has decided to build
Prasanna Malaiyandi:their own data center and they think now that maybe they should get a DR.
Prasanna Malaiyandi:Plan.
W. Curtis Preston:it's, it's surprising how often sort of backup
W. Curtis Preston:and, and Chris, I know we've talked about it so many times, right?
W. Curtis Preston:And, and we've had folks on talking about, okay, this is why it's important to have
W. Curtis Preston:a plan and to have stakeholders aligned.
W. Curtis Preston:But no one ever thinks about DR and backup until it's.
Prasanna Malaiyandi:Yeah.
Prasanna Malaiyandi:Uh, so I guess what happened here is that the Korean government, south
Prasanna Malaiyandi:Korean government sternly criticized cacao and its c e o resigned.
Prasanna Malaiyandi:So they, um, they've unveiled a strategy to create tech teams, develop
Prasanna Malaiyandi:a business continuity plan, and built disaster recovery facilities.
Prasanna Malaiyandi:Um, you know, and the, the, the line in here, it's a little odd in this day and
Prasanna Malaiyandi:age that a company of their size doesn't have these things in place already.
Prasanna Malaiyandi:But this idea that the blaze, you know, the, the, so the impact of
Prasanna Malaiyandi:this on everyday Koreans is that they all rely on this service.
Prasanna Malaiyandi:This is what, this is why I saw it as very much this related.
Prasanna Malaiyandi:They all rely on these services of like, , uh, communi for communications, right?
Prasanna Malaiyandi:Um, it says they've assumed the status of de facto telecommunications
Prasanna Malaiyandi:infrastructure, right?
W. Curtis Preston:And so when you go down, that affects so
W. Curtis Preston:many people day in and day out,
Prasanna Malaiyandi:yeah.
W. Curtis Preston:yep.
Prasanna Malaiyandi:Yeah.
Prasanna Malaiyandi:So, um, so you have two companies where they were, they were hosting
Prasanna Malaiyandi:things, they were providing services.
Prasanna Malaiyandi:Many other people.
Prasanna Malaiyandi:In one case it's thousands of companies.
Prasanna Malaiyandi:In another case it's millions of, of everyday citizens that used them.
Prasanna Malaiyandi:And then both of them lost data due to some sort of thing.
Prasanna Malaiyandi:Uh, one was an attack, one was a fire, and neither of them had
Prasanna Malaiyandi:a plan to, to work around that.
W. Curtis Preston:And, and, and
Prasanna Malaiyandi:What am I doing here?
W. Curtis Preston:they should listen to the podcast.
W. Curtis Preston:That's what they should be doing.
W. Curtis Preston:But,
Prasanna Malaiyandi:what it is.
W. Curtis Preston:but I think.
W. Curtis Preston:I want to give maybe the benefit of the doubt that maybe they had
W. Curtis Preston:planned for some types of outages, but they may not have planned for
W. Curtis Preston:these types, these specific ones.
W. Curtis Preston:Although in the case of
Prasanna Malaiyandi:mean like the loss of an entire data center,
W. Curtis Preston:Right.
W. Curtis Preston:May,
Prasanna Malaiyandi:what a Dr.
Prasanna Malaiyandi:Plan is,
W. Curtis Preston:well, maybe they were relying on the other, like the host.
W. Curtis Preston:Data center infrastructure deal with that.
W. Curtis Preston:Great.
W. Curtis Preston:Hey, should have asked a question.
W. Curtis Preston:Yep.
Prasanna Malaiyandi:Right?
Prasanna Malaiyandi:Like, you know, we talk about this all the time.
Prasanna Malaiyandi:Right?
Prasanna Malaiyandi:And, and, and by the way, you know, so I, I mentioned in the ear in
Prasanna Malaiyandi:the early part of the podcast that this was gonna touch on a couple,
Prasanna Malaiyandi:couple of hobby horses for me.
Prasanna Malaiyandi:One is that, You should not trust your supplier of services to be
Prasanna Malaiyandi:the backer of services, right?
W. Curtis Preston:Unless it's written in contract or even then
Prasanna Malaiyandi:What?
Prasanna Malaiyandi:No, no, no.
Prasanna Malaiyandi:I don't think so.
Prasanna Malaiyandi:I, I think that, I think there's a, um,
W. Curtis Preston:separation
Prasanna Malaiyandi:there's an, yeah.
Prasanna Malaiyandi:Yeah.
Prasanna Malaiyandi:That, that backup.
Prasanna Malaiyandi:Backup should be a backup, right?
Prasanna Malaiyandi:Um, that it should be done by somebody else.
Prasanna Malaiyandi:This is a, this is.
Prasanna Malaiyandi:Sort of a separation of powers, um, separation
W. Curtis Preston:Or shared responsibility model.
Prasanna Malaiyandi:Shared responsibility, right?
Prasanna Malaiyandi:I, I think it's just best practice to have your primary stuff by one vendor and
Prasanna Malaiyandi:your backup stuff done by another vendor.
Prasanna Malaiyandi:That's the way it's always been done.
Prasanna Malaiyandi:, and then suddenly we start throwing that up in the air as to maybe
Prasanna Malaiyandi:that's, you know, maybe we can save money if we use the services, right?
Prasanna Malaiyandi:I think back to O V H and what happened there where we, where they lost the data
Prasanna Malaiyandi:center, it turns out their backup stunk because they were just in the corner.
Prasanna Malaiyandi:Right?
W. Curtis Preston:But I, I wanna challenge you that, on that challenge.
W. Curtis Preston:You on that though, Curtis, right?
W. Curtis Preston:I
Prasanna Malaiyandi:right.
W. Curtis Preston:I think that, yes, 90% of the time don't trust the vendor, right?
W. Curtis Preston:The single vendor to do the right thing.
Prasanna Malaiyandi:Yeah.
W. Curtis Preston:I feel if you can ask the right question.
W. Curtis Preston:, do the tests, prove it out, get it in contract,
Prasanna Malaiyandi:Yeah.
W. Curtis Preston:the full testing, which is effort, right?
W. Curtis Preston:It might just be easier to go do it yourself, right?
W. Curtis Preston:But I think if you can get it in terms and verify it and prove it out,
W. Curtis Preston:I Prasannally think it's okay to go with, like to trust that single person
Prasanna Malaiyandi:Do you have an example of that?
Prasanna Malaiyandi:Because I don't,
W. Curtis Preston:AWS backup.
Prasanna Malaiyandi:um, yeah, so, uh, Well, but, but that's not the same though.
Prasanna Malaiyandi:That's AWS Backup is a feature that you use,
W. Curtis Preston:Yes.
Prasanna Malaiyandi:There's no, there's no AWS backup service that
Prasanna Malaiyandi:you rely on, and by service, I mean, you, you understand what I'm saying?
Prasanna Malaiyandi:It's not like
W. Curtis Preston:it's not a managed service.
Prasanna Malaiyandi:it is, it is a feature that you
Prasanna Malaiyandi:use and you, you are going
Prasanna Malaiyandi:to use.
Prasanna Malaiyandi:The snapshots and you're responsible for managing it.
Prasanna Malaiyandi:What I'm talking about is relying, you know, putting all
Prasanna Malaiyandi:of the burden on the vendor.
Prasanna Malaiyandi:I don't know of a service where,
W. Curtis Preston:But don't they like, like if you, there are companies
W. Curtis Preston:that do manage services, right?
W. Curtis Preston:Where they take over your infrastructure and they operate it
Prasanna Malaiyandi:yep.
W. Curtis Preston:case.
W. Curtis Preston:So, but they'll bring in multiple vendors.
W. Curtis Preston:Right?
W. Curtis Preston:Sometimes.
Prasanna Malaiyandi:That that's a different, that's a different thing.
Prasanna Malaiyandi:I, I, let's just say, you know, um, I'm, I'm just, I'm just saying if,
Prasanna Malaiyandi:if you're asking me my druthers,
W. Curtis Preston:Yeah.
W. Curtis Preston:No, no, it's no.
Prasanna Malaiyandi:phrase, I would rather you have one vendor,
Prasanna Malaiyandi:dut and another vendor do B,
W. Curtis Preston:A hundred percent agree a hundred percent.
Prasanna Malaiyandi:Yeah.
Prasanna Malaiyandi:Um, I, I can't think of a situation.
Prasanna Malaiyandi:So for example, I was going to perhaps, Think of, so it's one reason, like, I
Prasanna Malaiyandi:don't like whatever Microsoft is currently supplying regarding Microsoft 365.
Prasanna Malaiyandi:First off, they don't have a backup service for Microsoft 365.
Prasanna Malaiyandi:They do have an archive service.
Prasanna Malaiyandi:They have an e-discovery service, but they don't have a backup service, so,
W. Curtis Preston:but if I took, uh, who is, who is, uh, Eric?
W. Curtis Preston:Firstly, what was the company he worked for?
W. Curtis Preston:Right?
W. Curtis Preston:Would you say that they are a service, a managed service provider?
Prasanna Malaiyandi:Yeah, they are.
Prasanna Malaiyandi:Uh, well they were, yeah, they were.
Prasanna Malaiyandi:Yes.
Prasanna Malaiyandi:They're, cuz they do, they do private cloud, essentially infrastructure.
Prasanna Malaiyandi:Right?
W. Curtis Preston:but then they're bringing in other vendors and they're
W. Curtis Preston:offering the entire package, right?
Prasanna Malaiyandi:yeah.
Prasanna Malaiyandi:Good.
Prasanna Malaiyandi:Uh, good, better, best.
Prasanna Malaiyandi:Right.
Prasanna Malaiyandi:Again, if you're asking me if.
Prasanna Malaiyandi:Again, my choice would be to have a different vendor do backup and recovery.
Prasanna Malaiyandi:Right?
Prasanna Malaiyandi:Because if you find out your vendor is an idiot,
W. Curtis Preston:You're protected.
Prasanna Malaiyandi:
Speaker:It's, you're protected.
Prasanna Malaiyandi:
Speaker:Well, the chances of you having two idiot vendors is less, right?
Prasanna Malaiyandi:
Speaker:Uh, so that's, that's one hobby horse.
Prasanna Malaiyandi:
Speaker:And then the other hobby horse is, I can't believe, um, you
Prasanna Malaiyandi:
Speaker:know, that just this idea that.
Prasanna Malaiyandi:
Speaker:People relied entirely on their, people just relied entirely under their backup.
Prasanna Malaiyandi:
Speaker:Clearly, nobody pushed them on it, right?
Prasanna Malaiyandi:
Speaker:Nobody pushed them on, you know, what's your DR plan?
Prasanna Malaiyandi:
Speaker:All these people paying 'em all this money, and they're not pushing
Prasanna Malaiyandi:
Speaker:them on, what's your DR plan?
W. Curtis Preston:Yep.
Prasanna Malaiyandi:You know?
Prasanna Malaiyandi:You know, I don't, I don't know.
Prasanna Malaiyandi:I don't know what to say about.
W. Curtis Preston:Dr.
W. Curtis Preston:Is Asmar.
Prasanna Malaiyandi:I don't wanna be victim blaming at the same time.
Prasanna Malaiyandi:You know, this is clearly Rackspace and ca cow's responsibility.
Prasanna Malaiyandi:Uh, but if you are sitting there and you are using vendors and you are , you're
Prasanna Malaiyandi:using vendors for stuff, you should be inquiring as to their DR plans and
Prasanna Malaiyandi:their ransomware readiness plans and
W. Curtis Preston:Was this covered in your book when we talked
W. Curtis Preston:about talking about stakeholders?
Prasanna Malaiyandi:I can't, I can't remember.
Prasanna Malaiyandi:I can't remember if it wasn't or not, or if it was or not.
Prasanna Malaiyandi:Um, mainly the book was about how to make sure you're ready.
Prasanna Malaiyandi:Right?
Prasanna Malaiyandi:But I don't know.
Prasanna Malaiyandi:So been, it's a depressing, it's a depressing episode.
Prasanna Malaiyandi:We should talk about puppies for.
W. Curtis Preston:Well, it's not a depressing episode.
W. Curtis Preston:I think this is one of those things, right?
W. Curtis Preston:The silver lining is, this is lessons that other companies can look at, and
W. Curtis Preston:hopefully they can be like, Hey, we really should be thinking about what we're doing,
W. Curtis Preston:and do we have Dr, do we have backup?
W. Curtis Preston:Do we have a
Prasanna Malaiyandi:
Speaker:Take this opportunity.
W. Curtis Preston:Yeah.
W. Curtis Preston:Or the other thing is if you are using a SaaS service or who, or a managed service
W. Curtis Preston:provider, ask them, what are your plans?
W. Curtis Preston:Take this opportunity because everyone's gonna be asking, Hey, what is going on?
Prasanna Malaiyandi:Yeah.
Prasanna Malaiyandi:If, if, yeah, if you're one of these people that I don't understand,
Prasanna Malaiyandi:by the way, that's using hosted exchange in somebody else's data
Prasanna Malaiyandi:center instead of Microsoft 365, uh, by the way, feel free to contact us.
Prasanna Malaiyandi:Uh, you know, I didn't mention, you know, that, that, you know, you can reach out
Prasanna Malaiyandi:to us at WC presson on Twitter or w Curtis Preston at gmail as long as Twitter stays
Prasanna Malaiyandi:up and, um, You know, we'd, I'd love to, if, if you've got an answer as to why you
Prasanna Malaiyandi:would use hosted exchange over Microsoft 365, uh, I would love to know it.
Prasanna Malaiyandi:But if you have that, yeah.
Prasanna Malaiyandi:It's time to reach out to that vendor or hosted anything,
W. Curtis Preston:anything.
W. Curtis Preston:Exactly.
W. Curtis Preston:It's not just email.
Prasanna Malaiyandi:Yeah, I mean, I, you know, this, this podcast is
Prasanna Malaiyandi:hosted, backup Central is hosted, but um, I know what the backup setup is.
Prasanna Malaiyandi:Hello?
Prasanna Malaiyandi:Kopi
Prasanna Malaiyandi:Coffee always wants to get on the podcast.
Prasanna Malaiyandi:Um, yeah.
Prasanna Malaiyandi:So I know what the backups are on that because I, I make them happen.
Prasanna Malaiyandi:Right.
Prasanna Malaiyandi:But, um,
W. Curtis Preston:And then for your Backup Central's website too, right?
W. Curtis Preston:You have a hosted WordPress site, but you're doing the backups yourself.
Prasanna Malaiyandi:Yeah.
Prasanna Malaiyandi:Well, it's part of the Sea Sea Panel Sea panel.
Prasanna Malaiyandi:It's, it's run by C panel.
Prasanna Malaiyandi:And then C panel allows me to configure a backup, which that backup, daily backup
Prasanna Malaiyandi:is then copied to, uh, S3 actually, right?
Prasanna Malaiyandi:And then, and it's automatically deleted after a certain number of.
Prasanna Malaiyandi:Days.
Prasanna Malaiyandi:Right.
Prasanna Malaiyandi:Um, so it's not, it's not, the backup is not stored.
Prasanna Malaiyandi:So that's another example.
Prasanna Malaiyandi:Kind of like what you were talking about, like aw s backup.
Prasanna Malaiyandi:I don't rely on C panel or my hosting provider to do the backup, by the
Prasanna Malaiyandi:way, they do offer a backup service.
Prasanna Malaiyandi:See, I do eat my
W. Curtis Preston:That.
W. Curtis Preston:That's why I was telling, that's why I brought that up, that example, right.
Prasanna Malaiyandi:They do offer a hosted backup service.
Prasanna Malaiyandi:LiquidWeb is the name of my hosting provider, by the way, love.
Prasanna Malaiyandi:Been with them now for a really long time and I, you know, they are a actual managed
Prasanna Malaiyandi:server service provider for hosting and, um, the, they do offer a backup
Prasanna Malaiyandi:service and instead I use the features of C Panel to create a backup, which
Prasanna Malaiyandi:is then, um, you know, sent over to s3.
Prasanna Malaiyandi:And, uh, I think I keep 90 days even though I think that's ridiculous.
Prasanna Malaiyandi:And my Amazon S3.
Prasanna Malaiyandi:Is, uh, like $2 and 83 cents, something like that.
W. Curtis Preston:Maybe you should be looking at S three ia.
W. Curtis Preston:Your cost might drop
Prasanna Malaiyandi:Yeah.
Prasanna Malaiyandi:Yeah.
Prasanna Malaiyandi:Cuz I, you know, I've, I've used my backups two or three times, but,
Prasanna Malaiyandi:um, yeah, I don't know if it's, I don't know if it's possible, but
W. Curtis Preston:the saving of 35 cents.
Prasanna Malaiyandi:Yeah.
Prasanna Malaiyandi:But, uh, yeah.
Prasanna Malaiyandi:Anyway, just l just investigate your vendors.
Prasanna Malaiyandi:Will you check to see if they're doing, you know, I, I
Prasanna Malaiyandi:wish the best for Rackspace.
Prasanna Malaiyandi:I wish the best for their customers.
Prasanna Malaiyandi:I hope that by the time you hear this, this has all been sorted out.
Prasanna Malaiyandi:If it hasn't, dear Lord.
Prasanna Malaiyandi:Anyway, well, alright, uh, hope you enjoyed this episode.
Prasanna Malaiyandi:Remember to subscribe so that you can restore it all.
Listen On
