Rackspace Ransomware Attack: Lessons Learned
In this episode, we examine the Rackspace ransomware attack that crippled the company's hosted exchange environment, affecting thousands of customers. We discuss the timeline of events, the importance of timely patching, and the challenges Rackspace faced in restoring customer data. Learn about the value of comprehensive disaster recovery plans and third-party backups in protecting your organization from similar attacks. Don't miss this opportunity to gain valuable insights from one of the most significant ransomware incidents in recent years.
Mentioned in this episode:
Blank Midroll
Speaker:
A ransomware attack on Rackspace in 2023 left thousands of customers without
Speaker:
access to their critical email data for months and led Rackspace to completely
Speaker:
abandon the hosted exchange business line.
Speaker:
On this week's episode of the backup wrap-up we discuss a detailed timeline
Speaker:
of this event and most important, the lessons that we can learn from it.
Speaker:
The incident in this episode is one of the many stories that are behind
Speaker:
the recommendations that you may have heard from me throughout the years.
Speaker:
I'm w your Curtis Preston, AKA Mr.
Speaker:
Backup.
Speaker:
And there's a reason I'm so passionate about this subject.
Speaker:
It's because in my first job as a backup admin, my company lost an important
Speaker:
database and I couldn't restore it.
Speaker:
Since that moment, I've dedicated my career to making sure that
Speaker:
would never again, happen to me.
Speaker:
Or anyone who bothers to listen to me?
Speaker:
We take unappreciated backup admins and turn them into cyber recovery heroes.
Speaker:
This is the backup wrap up.
Speaker:
W. Curtis Preston: Welcome to the show.
Speaker:
I'm your host, w Curtis Preston, AKA, Mr.
Speaker:
Backup, and with me, I have my consultant that will help reduce
Speaker:
my level of starstruck today.
Speaker:
I'm hoping
Prasanna Malaiyandi:
I don't think that's possible.
Prasanna Malaiyandi:
W. Curtis Preston: I'm gonna be,
Prasanna Malaiyandi:
That sound.
Prasanna Malaiyandi:
So yes, I think you should tell people who may not have caught that.
Prasanna Malaiyandi:
Who are you gonna go see today?
Prasanna Malaiyandi:
W. Curtis Preston: I'm gonna meet William Shatner today.
Prasanna Malaiyandi:
I am
Prasanna Malaiyandi:
Are you
Prasanna Malaiyandi:
W. Curtis Preston: super psyched.
Prasanna Malaiyandi:
Yeah, I, I actually bought it.
Prasanna Malaiyandi:
There's an event.
Prasanna Malaiyandi:
There's a, there's a premiere of this new documentary that's about William Shatner.
Prasanna Malaiyandi:
Um, and it's in la It's, and it's, uh, they're gonna do the screening.
Prasanna Malaiyandi:
They're gonna do q and a, and then there is a.
Prasanna Malaiyandi:
Uh, birthday party for him, his 93rd birthday party for him, uh, afterwards.
Prasanna Malaiyandi:
And it's being held in the original, um, in the studio where
Prasanna Malaiyandi:
they originally filmed the pilot.
Prasanna Malaiyandi:
Um, and so it, I'll also be meeting, uh, Kevin Smith and, um, so.
Prasanna Malaiyandi:
Hopefully I will.
Prasanna Malaiyandi:
My dream, if I can, if I can get a selfie with William Shatner,
Prasanna Malaiyandi:
that'll be, you know, um,
Prasanna Malaiyandi:
Prasanna Malaiyandi: You'll be over the moon.
Prasanna Malaiyandi:
W. Curtis Preston: that'll be, I'll be over the moon.
Prasanna Malaiyandi:
Yeah.
Prasanna Malaiyandi:
I've already met.
Prasanna Malaiyandi:
I met, um, deforest Kelly.
Prasanna Malaiyandi:
I met, uh, Michelle Nichols.
Prasanna Malaiyandi:
I met, uh, George Decay and this will be, um, there is one
Prasanna Malaiyandi:
remaining, uh, star, original Star Trek member that's still alive.
Prasanna Malaiyandi:
Walter Koenig.
Prasanna Malaiyandi:
Um, that would be the, the one person who's still possible
Prasanna Malaiyandi:
to meet that I haven't met.
Prasanna Malaiyandi:
But, uh, yeah, William, I'm super excited about that.
Prasanna Malaiyandi:
So.
Prasanna Malaiyandi:
Does it count as meeting if you go visit
Prasanna Malaiyandi:
the grave site of the person?
Prasanna Malaiyandi:
W. Curtis Preston: Oh, that's just wrong.
Prasanna Malaiyandi:
That's
Prasanna Malaiyandi:
I'm just
Prasanna Malaiyandi:
W. Curtis Preston: That's just wrong.
Prasanna Malaiyandi:
Uh, yeah.
Prasanna Malaiyandi:
So, uh, just help me, help me keep my, my heart pitter Pat.
Prasanna Malaiyandi:
I'm definitely a, definitely a fan and meeting him, uh, will be very, very cool.
Prasanna Malaiyandi:
Uh, this week.
Prasanna Malaiyandi:
so wait, what is your, if you got a chance to ask
Prasanna Malaiyandi:
him a question, what would it be?
Prasanna Malaiyandi:
W. Curtis Preston: Oh, it's definitely not gonna be one of
Prasanna Malaiyandi:
those, like an episode 57, you know?
Prasanna Malaiyandi:
Um, wow.
Prasanna Malaiyandi:
I'm not prepared for that question.
Prasanna Malaiyandi:
I'll have to think about that.
Prasanna Malaiyandi:
Wow.
Prasanna Malaiyandi:
Did I
Prasanna Malaiyandi:
stump Curtis?
Prasanna Malaiyandi:
W. Curtis Preston: You did, you stumped me.
Prasanna Malaiyandi:
Yeah.
Prasanna Malaiyandi:
I'll have to, I'll, yeah, I'll definitely, you know, I'm gonna be, I'm gonna be so
Prasanna Malaiyandi:
nerded out, like I'm, I'm gonna be, yeah.
Prasanna Malaiyandi:
Um, I just, I, if I get to say two words to him, I'll be,
Prasanna Malaiyandi:
you know, I'll be like, hi.
Prasanna Malaiyandi:
Um, you know, I, yeah.
Prasanna Malaiyandi:
I, I just hope I don't do, like, I've met a lot of famous people and
Prasanna Malaiyandi:
so many times I've been like, chill.
Prasanna Malaiyandi:
But I remember there was this one person that I just randomly ran into in an
Prasanna Malaiyandi:
airport and I literally screamed their name like a, like a 10-year-old girl.
Prasanna Malaiyandi:
And, um, that was very embarrassing.
Prasanna Malaiyandi:
I just hope I don't go, William, that would
Prasanna Malaiyandi:
okay.
Prasanna Malaiyandi:
I'm sure he is used to it, you know?
Prasanna Malaiyandi:
W. Curtis Preston: Yeah, I'm sure.
Prasanna Malaiyandi:
Yeah.
Prasanna Malaiyandi:
Um, so this week we're continuing our series about cloud disasters
Prasanna Malaiyandi:
and this one is pretty bad.
Prasanna Malaiyandi:
Um, you know, and again, this is yet another story that's gonna
Prasanna Malaiyandi:
prove the point back your stuff up.
Prasanna Malaiyandi:
Right.
Prasanna Malaiyandi:
You know, even, even if this is actually, this is a really good.
Prasanna Malaiyandi:
Story that basically proves that even if the vendor is backing it up
Prasanna Malaiyandi:
for you and the backups are included as part of the package, something
Prasanna Malaiyandi:
so catastrophic might happen that those backups don't come in handy.
Prasanna Malaiyandi:
Does that sound about right, Pana?
Prasanna Malaiyandi:
It does, but I have.
Prasanna Malaiyandi:
Two comments about that.
Prasanna Malaiyandi:
W. Curtis Preston: Yeah.
Prasanna Malaiyandi:
Okay.
Prasanna Malaiyandi:
the first is, this reminds me a lot about the
Prasanna Malaiyandi:
OVH story that we did a while ago.
Prasanna Malaiyandi:
So if you haven't let heard that episode, go back, give it a listen, because it
Prasanna Malaiyandi:
was also the case with OVH that they said they were doing backups, but
Prasanna Malaiyandi:
people were not able to restore their backup because they were sitting in
Prasanna Malaiyandi:
the same data center as a production and there was a fire, so not so good.
Prasanna Malaiyandi:
W. Curtis Preston: Yeah.
Prasanna Malaiyandi:
And, and I'm gonna say, so this, this story is about Rackspace, which I'm gonna
Prasanna Malaiyandi:
say I have no ill will against Rackspace.
Prasanna Malaiyandi:
I, I feel for the people that had to go through this, uh, the thing I struggle
Prasanna Malaiyandi:
with is the ways in which Rackspace tried to deflect, blame Rackspace.
Prasanna Malaiyandi:
The company tried to deflect blame.
Prasanna Malaiyandi:
Uh, and so based on that, we've got a pretty solid timeline of the
Prasanna Malaiyandi:
events Now, just, just for color.
Prasanna Malaiyandi:
And I didn't know this, this part I'm about to say.
Prasanna Malaiyandi:
I didn't know this until, until I was researching for the story.
Prasanna Malaiyandi:
Prior to this event happening, Rackspace had already suffered, uh, a sharp.
Prasanna Malaiyandi:
Decline in value.
Prasanna Malaiyandi:
At the height of their value, April, 2021, they were a, an $8 billion company.
Prasanna Malaiyandi:
And by the time this event happened, they had dropped,
Prasanna Malaiyandi:
over over 90% of their value.
Prasanna Malaiyandi:
They were then an $800 million company.
Prasanna Malaiyandi:
And, as of today's recording, they
Prasanna Malaiyandi:
are valued at $340 million, which is 5% of where they were with at their high.
Prasanna Malaiyandi:
W. Curtis Preston: Right?
Prasanna Malaiyandi:
Yeah.
Prasanna Malaiyandi:
Um, so, so they were already in sort of trouble and I, I think that may
Prasanna Malaiyandi:
be why they tried to deflect blame.
Prasanna Malaiyandi:
So, um, let's start with sort of the, before the story, right?
Prasanna Malaiyandi:
So before the story, there was something called the proxy, not she
Prasanna Malaiyandi:
exploit, uh, in September of 2022.
Prasanna Malaiyandi:
It was publicly announced and it basically, it allowed someone to
Prasanna Malaiyandi:
gain control of an exchange server.
Prasanna Malaiyandi:
It was announced September 30th, 2022, November 8th, Microsoft
Prasanna Malaiyandi:
released a security update but there was a minor issue with the patch.
Prasanna Malaiyandi:
And Rackspace claimed this was why they didn't install it at that time,
Prasanna Malaiyandi:
but by November 17th, Microsoft had fixed that, , that issue.
Prasanna Malaiyandi:
You know, we talk about three things, right?
Prasanna Malaiyandi:
Password management, patch management, and MFA.
Prasanna Malaiyandi:
And then if everybody just did this, then it would've stopped.
Prasanna Malaiyandi:
Uh, you know, it would stop so much.
Prasanna Malaiyandi:
And this story is so much I.
Prasanna Malaiyandi:
Evidence of that, uh, because November 17th that minor issue
Prasanna Malaiyandi:
with the patch was fixed.
Prasanna Malaiyandi:
So they could have, and in my opinion, should have immediately put on this
Prasanna Malaiyandi:
security patch because it was such a huge exploit there was a CVA attached to it.
Prasanna Malaiyandi:
And, uh, well-known within the industry, they should have immediately patched
Prasanna Malaiyandi:
all of their, uh, exchange servers.
Prasanna Malaiyandi:
By the way, I should mention what we're talking about is that
Prasanna Malaiyandi:
Rackspace had a hosted exchange service, not Microsoft 365.
Prasanna Malaiyandi:
They ran hosted exchange on their own servers in their own, uh, data center.
Prasanna Malaiyandi:
Before you continue on, I think it's important to state that
Prasanna Malaiyandi:
for that September 30th, right, there was a workaround that was deployed, right?
Prasanna Malaiyandi:
That
Prasanna Malaiyandi:
pretty much Microsoft was like, Hey, we haven't quite figured out the patch
Prasanna Malaiyandi:
yet, which will come out November 8th.
Prasanna Malaiyandi:
But in the meantime, here's a workaround to make sure you don't get impacted,
Prasanna Malaiyandi:
which Rackspace did apply, apply.
Prasanna Malaiyandi:
Right.
Prasanna Malaiyandi:
W. Curtis Preston: they, they did or they did not.
Prasanna Malaiyandi:
They did apply the workaround.
Prasanna Malaiyandi:
W. Curtis Preston: Yeah.
Prasanna Malaiyandi:
Okay.
Prasanna Malaiyandi:
Yeah.
Prasanna Malaiyandi:
Yeah.
Prasanna Malaiyandi:
So it's not a permanent fix, but at least
Prasanna Malaiyandi:
sort of protects you for now.
Prasanna Malaiyandi:
So then the other thing is, um.
Prasanna Malaiyandi:
around this time there were actually two exploits, Right.
Prasanna Malaiyandi:
So there was a proxy, not shell exploit, and then there was
Prasanna Malaiyandi:
another one, um, O-W-A-S-S-R-F.
Prasanna Malaiyandi:
I don't know what that stands for, but that's what they
Prasanna Malaiyandi:
called it, right?
Prasanna Malaiyandi:
And these two are kind of related.
Prasanna Malaiyandi:
And so the patch though, that came out in November would have fixed both.
Prasanna Malaiyandi:
W. Curtis Preston: Right.
Prasanna Malaiyandi:
they were applied, but the workaround that was
Prasanna Malaiyandi:
applied in the end of September only addressed the proxy nutshell issue.
Prasanna Malaiyandi:
It
Prasanna Malaiyandi:
did not expl address the second exploit.
Prasanna Malaiyandi:
W. Curtis Preston: Yeah.
Prasanna Malaiyandi:
By the way, the OWA most certainly stands for Outlook.
Prasanna Malaiyandi:
Web access would be my guess.
Prasanna Malaiyandi:
I don't know what SSRF but stands for, but yeah, it is kind
Prasanna Malaiyandi:
of complicated that basically.
Prasanna Malaiyandi:
That there was a patch that the, the PA had, they applied the patch, they would've
Prasanna Malaiyandi:
fixed at a, at that time, unknown problem.
Prasanna Malaiyandi:
Um, but the, but they didn't apply the patch.
Prasanna Malaiyandi:
And then two weeks goes by and then what happened?
Prasanna Malaiyandi:
And then on November 29th, Rackspace says that they
Prasanna Malaiyandi:
were attacked by a group called Play, which gained access to their exchange
Prasanna Malaiyandi:
environment using stolen credentials, and that they had access to some of Rackspace
Prasanna Malaiyandi:
exchange environments, which, if I
Prasanna Malaiyandi:
was a customer on hosted exchange, I would be kind of freaked out.
Prasanna Malaiyandi:
W. Curtis Preston: Yeah, exactly right.
Prasanna Malaiyandi:
So they, they gain privileged access of their exchange servers.
Prasanna Malaiyandi:
We're, not sure if they knew in November, but because they first
Prasanna Malaiyandi:
notified people December 2nd.
Prasanna Malaiyandi:
Literally at two o'clock in the morning.
Prasanna Malaiyandi:
Right.
Prasanna Malaiyandi:
Based on the, the, the stuff that we have, they, they may have at that point
Prasanna Malaiyandi:
realized what happened and they were able to trace it back to November 29th.
Prasanna Malaiyandi:
And at 2:00 AM right?
Prasanna Malaiyandi:
This is when they were like, yeah, here's what happened.
Prasanna Malaiyandi:
We noticed something, and then they just brought everything down, right?
Prasanna Malaiyandi:
They were
Prasanna Malaiyandi:
like, yep, we're not gonna allow any more access
Prasanna Malaiyandi:
W. Curtis Preston: response, right?
Prasanna Malaiyandi:
The, the next part of the story is the one that really sets it apart.
Prasanna Malaiyandi:
I don't know any other story like this.
Prasanna Malaiyandi:
The res, their response was, and, and again, if you think about now that if
Prasanna Malaiyandi:
you think about where they were as a company, this part maybe makes more sense.
Prasanna Malaiyandi:
But what they decided to do was they said, you know what?
Prasanna Malaiyandi:
This is gonna take us a while.
Prasanna Malaiyandi:
This the, I'm making up words here.
Prasanna Malaiyandi:
We've been thinking about shooting this thing in the head anyway, and so let's
Prasanna Malaiyandi:
just move everybody over to Microsoft 365.
Prasanna Malaiyandi:
So December 2nd at 2:00 AM is when they first started telling people that
Prasanna Malaiyandi:
they had this problem, and by 8:00 PM that evening, they had made the
Prasanna Malaiyandi:
decision to move everybody over to 365.
Prasanna Malaiyandi:
Yeah, that I could, I would have loved to have
Prasanna Malaiyandi:
been a fly on the wall in those meetings, right when they were
Prasanna Malaiyandi:
trying to
Prasanna Malaiyandi:
W. Curtis Preston: not have wanted to be in the meeting.
Prasanna Malaiyandi:
Yeah.
Prasanna Malaiyandi:
Yeah, I, that's why I said I wanted to be a fly on the wall.
Prasanna Malaiyandi:
Right.
Prasanna Malaiyandi:
W. Curtis Preston: Yeah.
Prasanna Malaiyandi:
and hearing these conversations because you
Prasanna Malaiyandi:
know, it must have been a difficult decision to come to, right?
Prasanna Malaiyandi:
W. Curtis Preston: Yeah, because that would've been a competing service, right?
Prasanna Malaiyandi:
So if, if it's not obvious, like if you used hosted exchange, you were
Prasanna Malaiyandi:
very consciously using hosted exchange, not Microsoft 365, and you had reasons
Prasanna Malaiyandi:
for doing that, and they're like, guys, this is gonna take us a while.
Prasanna Malaiyandi:
We're gonna move everybody over to 365.
Prasanna Malaiyandi:
But what did they not move?
Prasanna Malaiyandi:
Uh, so there were two things.
Prasanna Malaiyandi:
They did not move, right?
Prasanna Malaiyandi:
The, probably the most important thing was their emails,
Prasanna Malaiyandi:
right?
Prasanna Malaiyandi:
What people cared about with the hosted exchange service, right?
Prasanna Malaiyandi:
Because they basically said, we will recreate things for you, make it easy.
Prasanna Malaiyandi:
So you have all your stuff up and running at Microsoft 365, but we can't
Prasanna Malaiyandi:
get you back all your emails yet.
Prasanna Malaiyandi:
W. Curtis Preston: Right.
Prasanna Malaiyandi:
Right, so that was one thing.
Prasanna Malaiyandi:
I think the second thing, and I don't know if they were forthcoming with
Prasanna Malaiyandi:
this, but the fact that with their hosted exchange implementation, they
Prasanna Malaiyandi:
offered backup as part of the service,
Prasanna Malaiyandi:
W. Curtis Preston: Right.
Prasanna Malaiyandi:
right?
Prasanna Malaiyandi:
When they told customers, Hey, Microsoft 365 is where you should
Prasanna Malaiyandi:
be looking at, I do wonder if they told people, by the way, you need to
Prasanna Malaiyandi:
figure out your own backup solution.
Prasanna Malaiyandi:
W. Curtis Preston: Yeah, none of the communications that we found
Prasanna Malaiyandi:
between them and customers, uh, showed that they'd said anything.
Prasanna Malaiyandi:
Uh, but the idea that they would, in the middle of the outage basically
Prasanna Malaiyandi:
abandoned an entire business line.
Prasanna Malaiyandi:
Move everybody over to 365.
Prasanna Malaiyandi:
What they did do was they were able, apparently they, they did
Prasanna Malaiyandi:
automate the process of creating the accounts for them over on 365.
Prasanna Malaiyandi:
So you, you were able to, um, you know, essentially you, you were able
Prasanna Malaiyandi:
to start sending and receiving email.
Prasanna Malaiyandi:
Relatively quickly considering how long the rest of this took within
Prasanna Malaiyandi:
a day, it looked like uh, or so you were able to send and receive
Prasanna Malaiyandi:
email using your old email address.
Prasanna Malaiyandi:
If you were an exchange hosted exchange customer, and now you're on 365.
Prasanna Malaiyandi:
You just didn't have access to any of the email you had received up to that point.
Prasanna Malaiyandi:
Yeah, which I like.
Prasanna Malaiyandi:
I go back and forth on that.
Prasanna Malaiyandi:
It's like, great.
Prasanna Malaiyandi:
I could send and I could see what people are sending me, but I have a lot
Prasanna Malaiyandi:
of old stuff and I would be freaking out if I lost all of my old emails.
Prasanna Malaiyandi:
Or a lot of times if these are businesses and organizations, maybe
Prasanna Malaiyandi:
they have contracts which are being sent back and forth via emails.
Prasanna Malaiyandi:
W. Curtis Preston: Yeah, like literally stuff that you got today, right?
Prasanna Malaiyandi:
Stuff that you got yesterday, stuff you're actively working on.
Prasanna Malaiyandi:
And a lot of people use their email system as sort of a somewhat,
Prasanna Malaiyandi:
sometimes temporary, sometimes permanent storage system, right?
Prasanna Malaiyandi:
And, and they're like, I know where that contract is.
Prasanna Malaiyandi:
It's in the email that I got, you know, three days
Prasanna Malaiyandi:
Prasanna Malaiyandi: and I can search for it.
Prasanna Malaiyandi:
I can find it.
Prasanna Malaiyandi:
Yep.
Prasanna Malaiyandi:
W. Curtis Preston: You're not able.
Prasanna Malaiyandi:
Yeah.
Prasanna Malaiyandi:
So on one on one hand, like, like you said, I would've wanted to be a fly
Prasanna Malaiyandi:
on the wall because they're saying we need to do what's, we need to get
Prasanna Malaiyandi:
our customers up and running again.
Prasanna Malaiyandi:
We need them, we need to get them, be able to send and receive email.
Prasanna Malaiyandi:
And I think that was a good decision.
Prasanna Malaiyandi:
Prasanna Malaiyandi: That's probably the most
Prasanna Malaiyandi:
important thing to do first.
Prasanna Malaiyandi:
W. Curtis Preston: yeah.
Prasanna Malaiyandi:
The, the, the bad decisions happened way before this.
Prasanna Malaiyandi:
In my opinion, this was a good decision, um, because as we're going
Prasanna Malaiyandi:
to find out in the story, they didn't exactly have a good, uh, backup.
Prasanna Malaiyandi:
I.
Prasanna Malaiyandi:
System, um, at least not one that, that I would recognize, right?
Prasanna Malaiyandi:
So they actually advertise backup as part of the service.
Prasanna Malaiyandi:
And again, I I, I'm gonna put this out as this is why, when, when I say,
Prasanna Malaiyandi:
even if the SaaS vendor advertise backup as part of the service, you
Prasanna Malaiyandi:
might want to consider a third party.
Prasanna Malaiyandi:
And I'll put an asterisk, especially if they charge extra for it.
Prasanna Malaiyandi:
Um, I think in this case it was just included as part of the, the, well,
Prasanna Malaiyandi:
I'm gonna put it, I'm gonna put in a, I'm gonna put it, especially,
Prasanna Malaiyandi:
I'm gonna put two asterisk, especially if they charge for it.
Prasanna Malaiyandi:
'cause then that gives you an incentive to pay somebody else instead.
Prasanna Malaiyandi:
But then I'm gonna say, especially if they don't charge for it, which means they're
Prasanna Malaiyandi:
probably not spending any money on it.
Prasanna Malaiyandi:
Prasanna Malaiyandi: Enough money to do that.
Prasanna Malaiyandi:
Yeah.
Prasanna Malaiyandi:
I wanna focus on that for a second
Prasanna Malaiyandi:
because I did remember earlier on in the episode, I said I had two
Prasanna Malaiyandi:
comments and we only covered one.
Prasanna Malaiyandi:
So here's a second
Prasanna Malaiyandi:
W. Curtis Preston: Oh.
Prasanna Malaiyandi:
Right is, does this apply If you are using a SaaS
Prasanna Malaiyandi:
or whatever backup product in and of itself, that you should have a second
Prasanna Malaiyandi:
vendor because who knows what that backup vendor will do, and maybe you
Prasanna Malaiyandi:
can never get your data back out.
Prasanna Malaiyandi:
I understand Rackspace, they offered backup and it, the backups didn't work.
Prasanna Malaiyandi:
Right now, there are a whole slew of SaaS data protection companies, or you could
Prasanna Malaiyandi:
roll your own.
Prasanna Malaiyandi:
Right.
Prasanna Malaiyandi:
In those cases though, do you have the same recommendation that if I do decide
Prasanna Malaiyandi:
to use a SaaS data protection company, I should probably use two SaaS data
Prasanna Malaiyandi:
protection companies because I don't
Prasanna Malaiyandi:
know if one can get me my data back?
Prasanna Malaiyandi:
W. Curtis Preston: Yeah, no, that, that's not what I'm saying.
Prasanna Malaiyandi:
I'm saying it because I'm talking about a SaaS service and then hiring
Prasanna Malaiyandi:
a SaaS data protection company that puts your data in two different places.
Prasanna Malaiyandi:
I would, I would not argue.
Prasanna Malaiyandi:
Having another one, but it's gonna be really, it's a, it's already
Prasanna Malaiyandi:
going to be a big enough cost.
Prasanna Malaiyandi:
I think the idea of putting the data in two different, completely different
Prasanna Malaiyandi:
protection zones, risk factors, you know, earthquake and flood zones, all of
Prasanna Malaiyandi:
it already, diversifies it.
Prasanna Malaiyandi:
yeah.
Prasanna Malaiyandi:
W. Curtis Preston: Yeah.
Prasanna Malaiyandi:
I think that already diversifies it.
Prasanna Malaiyandi:
so now my second question.
Prasanna Malaiyandi:
W. Curtis Preston: I'm just saying that if you have a backup, if you
Prasanna Malaiyandi:
have a SaaS service, like 365, so 365 is about to start offering backup.
Prasanna Malaiyandi:
Um, and and I'm just saying I still like the idea of a third party copy.
Prasanna Malaiyandi:
Yeah.
Prasanna Malaiyandi:
Okay.
Prasanna Malaiyandi:
So now my second question follow up to that is there's a lot of cloud public,
Prasanna Malaiyandi:
cloud providers, right, that people hook into that leverage things like snapshots.
Prasanna Malaiyandi:
W. Curtis Preston: Yeah.
Prasanna Malaiyandi:
So they're just an orchestration layer on top.
Prasanna Malaiyandi:
They're doing all the data movement.
Prasanna Malaiyandi:
They're moving your data around, they're taking the copies.
Prasanna Malaiyandi:
Do you have the same concerns with those as well?
Prasanna Malaiyandi:
W. Curtis Preston: I do.
Prasanna Malaiyandi:
Um, basically the, that's why again, I li I like the orchestration
Prasanna Malaiyandi:
companies and the ones I like best are the ones that ultimately take
Prasanna Malaiyandi:
a copy of the data outside, right?
Prasanna Malaiyandi:
We, we use snapshots to orchestrate and to create the backup, and then we use
Prasanna Malaiyandi:
something else to get the data out of.
Prasanna Malaiyandi:
You know, your favorite cloud vendor, again, getting it out,
Prasanna Malaiyandi:
storing it in another place.
Prasanna Malaiyandi:
Second best to that would be storing it in another region, in, in another account.
Prasanna Malaiyandi:
But you know, this is just the, basically I see, like, I think
Prasanna Malaiyandi:
of like, let's say AWS as NetApp.
Prasanna Malaiyandi:
Right.
Prasanna Malaiyandi:
So I need another copy, a final copy of the data that isn't on AWS because
Prasanna Malaiyandi:
of rolling code concerns, right?
Prasanna Malaiyandi:
So you get a bug and it, and it rolls, uh, you know, and takes
Prasanna Malaiyandi:
out both primary and the backup.
Prasanna Malaiyandi:
then are you also concerned though, because a lot of
Prasanna Malaiyandi:
these SaaS data protection companies are built on top of the big clouds.
Prasanna Malaiyandi:
W. Curtis Preston: So I do, I do want to, um, then make sure that they're
Prasanna Malaiyandi:
stored in different regions and whatnot.
Prasanna Malaiyandi:
I, you know, I can only, I can only,
Prasanna Malaiyandi:
Prasanna Malaiyandi: you can only go so far.
Prasanna Malaiyandi:
W. Curtis Preston: that, yeah, you could only go so far, right?
Prasanna Malaiyandi:
It'd be like the same.
Prasanna Malaiyandi:
The same would be true if we weren't talking to cloud and we were saying
Prasanna Malaiyandi:
they also, your backup service also uses Soliris, right At, at some point there's
Prasanna Malaiyandi:
a risk that you just can't get away from.
Prasanna Malaiyandi:
Sorry, I, I know we never talked about that before, so
Prasanna Malaiyandi:
I was very curious about your take on
Prasanna Malaiyandi:
W. Curtis Preston: Yeah.
Prasanna Malaiyandi:
Yeah.
Prasanna Malaiyandi:
Okay.
Prasanna Malaiyandi:
So, uh, so we were talking about December 2nd is when, um, they, you
Prasanna Malaiyandi:
know, when they made this move, right?
Prasanna Malaiyandi:
Um, the, the, they first mentioned that they believe it was a
Prasanna Malaiyandi:
security incident on December 3rd.
Prasanna Malaiyandi:
Uh, and then, uh, December 2nd or December 6th, they say that
Prasanna Malaiyandi:
it was a ransomware incident.
Prasanna Malaiyandi:
And then finally, 14th, they revealed the attack was from a, their, their
Prasanna Malaiyandi:
words financially motivated threat actor.
Prasanna Malaiyandi:
Um, so we don't know the details, uh, of, you know, the extortion, but
Prasanna Malaiyandi:
there was some kind of extortion.
Prasanna Malaiyandi:
We also don't know whether or not they ultimately, I.
Prasanna Malaiyandi:
Paid that money.
Prasanna Malaiyandi:
Um, you know,
Prasanna Malaiyandi:
and, and this.
Prasanna Malaiyandi:
W. Curtis Preston: I, I, I advise as much as possible not
Prasanna Malaiyandi:
to pay the money, but go ahead.
Prasanna Malaiyandi:
and this goes just the earlier thing about talking
Prasanna Malaiyandi:
about a security incident and then changing it to ransomware incident.
Prasanna Malaiyandi:
I get it.
Prasanna Malaiyandi:
But I know you and I, we've talked in the past about vendors or companies
Prasanna Malaiyandi:
should be more transparent to a certain extent about what's going on in order to.
Prasanna Malaiyandi:
Build confidence in the public in terms of they have things handled,
Prasanna Malaiyandi:
they're figuring things out.
Prasanna Malaiyandi:
It's okay.
Prasanna Malaiyandi:
And so I wanted to get your take on that messaging that came from Rackspace.
Prasanna Malaiyandi:
Do you think that caused, like, do you think that would've caused a
Prasanna Malaiyandi:
lot of concern for customers or the public in terms of their ability
Prasanna Malaiyandi:
to handle things first, calling it a security incident and then a
Prasanna Malaiyandi:
ransomware, and then sort of this back
Prasanna Malaiyandi:
and forth on data?
Prasanna Malaiyandi:
I.
Prasanna Malaiyandi:
W. Curtis Preston: I think it's possible that they just,
Prasanna Malaiyandi:
they revealed what they knew.
Prasanna Malaiyandi:
Like you do have to be careful saying only what you know for sure.
Prasanna Malaiyandi:
Yeah.
Prasanna Malaiyandi:
W. Curtis Preston: The thing that needs to be understood at this point as they move
Prasanna Malaiyandi:
through the story is that be like, on one hand, I'm glad they did what they did.
Prasanna Malaiyandi:
They moved everybody over to 365, got everybody working, but
Prasanna Malaiyandi:
everybody's gonna be clamoring for their emails right from the previous.
Prasanna Malaiyandi:
Uh, but because they did it the way they did it, so it, they, if they
Prasanna Malaiyandi:
had migrated exchange into 365, they could have brought the data with them,
Prasanna Malaiyandi:
but it would've taken longer because they still, they had dead servers.
Prasanna Malaiyandi:
Right.
Prasanna Malaiyandi:
Yeah.
Prasanna Malaiyandi:
They don't have the data yet.
Prasanna Malaiyandi:
W. Curtis Preston: A migration takes a while too.
Prasanna Malaiyandi:
Right.
Prasanna Malaiyandi:
But because it did it the way they did it, the only option at this
Prasanna Malaiyandi:
point is to create PSTs of individual users and then import those PSTs
Prasanna Malaiyandi:
to those users on the other side.
Prasanna Malaiyandi:
Um, and so they, they said on December 18th that they had, um, they
Prasanna Malaiyandi:
had created in, um, that I, I, I.
Prasanna Malaiyandi:
I'm a little confused where like on one hand they said that they had, I think
Prasanna Malaiyandi:
they had figured out a way to start restoring the affected exchange servers,
Prasanna Malaiyandi:
but they hadn't yet restored all of them.
Prasanna Malaiyandi:
Right.
Prasanna Malaiyandi:
And this is when they, when they first announce, they're like, so
Prasanna Malaiyandi:
here's how this is gonna work.
Prasanna Malaiyandi:
We're gonna restore an exchange server.
Prasanna Malaiyandi:
If you're on that exchange server, you will then be able to export,
Prasanna Malaiyandi:
uh, A PST file, and then you'll then be able to download a PST file for
Prasanna Malaiyandi:
each user, and then you will then be able to upload that into 365.
Prasanna Malaiyandi:
And they first announced this on December 18th, and then December
Prasanna Malaiyandi:
already two weeks after they shut down the service.
Prasanna Malaiyandi:
W. Curtis Preston: Thank you very much.
Prasanna Malaiyandi:
That is two weeks after they shut down the service.
Prasanna Malaiyandi:
Right.
Prasanna Malaiyandi:
Um, and uh, it also, it's important to understand that by exporting
Prasanna Malaiyandi:
it as a PSD and then importing it, it's not gonna be a perfect restore.
Prasanna Malaiyandi:
'cause , I'm pretty sure that you're gonna lose folders and
Prasanna Malaiyandi:
all of that in this process.
Prasanna Malaiyandi:
Uh, not to mention the fact that it's just you're gonna lose metadata too.
Prasanna Malaiyandi:
Right.
Prasanna Malaiyandi:
So, you're so in my mind I'm thinking, oh, this is great.
Prasanna Malaiyandi:
It's a way for people to do things.
Prasanna Malaiyandi:
Right.
Prasanna Malaiyandi:
And then
Prasanna Malaiyandi:
on December 20th, they said that they had just begun testing
Prasanna Malaiyandi:
the above recovery procedure.
Prasanna Malaiyandi:
W. Curtis Preston: Yeah.
Prasanna Malaiyandi:
Yeah.
Prasanna Malaiyandi:
So they hadn't actually tried it out or done anything.
Prasanna Malaiyandi:
They're kind of shooting from the hip, and I get it.
Prasanna Malaiyandi:
They're urgently trying to figure out how to get the data back for the customers.
Prasanna Malaiyandi:
But at the same time, it doesn't inspire confidence.
Prasanna Malaiyandi:
W. Curtis Preston: Yeah, and the thing is that if they had prepared for this
Prasanna Malaiyandi:
event eventuality back in the day, they could have created a different procedure.
Prasanna Malaiyandi:
Because there are backup software products that allow you to,
Prasanna Malaiyandi:
um, directly extract PST data.
Prasanna Malaiyandi:
From a backup, right?
Prasanna Malaiyandi:
It it does.
Prasanna Malaiyandi:
They, they do exist.
Prasanna Malaiyandi:
It does happen.
Prasanna Malaiyandi:
So if they had tested this beforehand, they wouldn't have
Prasanna Malaiyandi:
done it the way they did it.
Prasanna Malaiyandi:
They would've figured out a way to directly, instead of restoring
Prasanna Malaiyandi:
exchange servers and then saying, Hey, customers, go get your PSDs.
Prasanna Malaiyandi:
They would've been able to just directly create the PSDs.
Prasanna Malaiyandi:
They would've figured that out beforehand.
Prasanna Malaiyandi:
Um,
Prasanna Malaiyandi:
and then it's just a matter of
Prasanna Malaiyandi:
W. Curtis Preston: think that would've been a much quicker method.
Prasanna Malaiyandi:
Yeah.
Prasanna Malaiyandi:
And they would've just had to execute on it rather than trying
Prasanna Malaiyandi:
to figure all this out, because I don't know if a lot of folks.
Prasanna Malaiyandi:
Understand when you're trying to recover from a ransomware attack, right?
Prasanna Malaiyandi:
You're not only trying to figure out everything that went wrong,
Prasanna Malaiyandi:
but also all the extreme pressure you're under lack of sleep, right?
Prasanna Malaiyandi:
People yelling at you possibly, right?
Prasanna Malaiyandi:
Worrying about, am I gonna have a job after this?
Prasanna Malaiyandi:
Yeah.
Prasanna Malaiyandi:
W. Curtis Preston: I can't imagine the number of people that were, you
Prasanna Malaiyandi:
know, companies that were yelling about how much money they were spending
Prasanna Malaiyandi:
and blah, blah, blah, blah, blah.
Prasanna Malaiyandi:
Right?
Prasanna Malaiyandi:
Um, yeah.
Prasanna Malaiyandi:
So December 22nd is the first day that they notified some customers that they
Prasanna Malaiyandi:
could start retrieving some of their mail.
Prasanna Malaiyandi:
This is three weeks since the incident, right?
Prasanna Malaiyandi:
Um, and it wasn't until January 5th, which is another, what, two weeks I.
Prasanna Malaiyandi:
They said they had 50% done.
Prasanna Malaiyandi:
I mean, this would, this took a while.
Prasanna Malaiyandi:
Yeah.
Prasanna Malaiyandi:
So here's my question, Curtis.
Prasanna Malaiyandi:
I know we talked earlier about, okay, they should have had
Prasanna Malaiyandi:
these procedures documented.
Prasanna Malaiyandi:
They should have thought about them, right, and tested it
Prasanna Malaiyandi:
out.
Prasanna Malaiyandi:
So then they had a process, Do you honestly think people think about
Prasanna Malaiyandi:
these scenarios and walk through?
Prasanna Malaiyandi:
Because normally when you're thinking disaster recovery or
Prasanna Malaiyandi:
backup and recovery, right?
Prasanna Malaiyandi:
It's like, oh, I lost an email, or I lost a part of something, or someone
Prasanna Malaiyandi:
accidentally deleted a user, right?
Prasanna Malaiyandi:
I think that the mind needs or people's mind, people need to change.
Prasanna Malaiyandi:
And start to start thinking about some of these cases.
Prasanna Malaiyandi:
But when they designed the system, do you think that like they were like, Hey,
Prasanna Malaiyandi:
I wonder what'll happen if the exchange servers all get hit by ransomware and
Prasanna Malaiyandi:
we have to rebuild and all our customers are gonna leave us for Microsoft 365?
Prasanna Malaiyandi:
W. Curtis Preston: So I'll answer that in two ways.
Prasanna Malaiyandi:
One is probably not right, and two, they probably should have.
Prasanna Malaiyandi:
Right.
Prasanna Malaiyandi:
One of the things I talk a lot about in, you know, when I'm talking about
Prasanna Malaiyandi:
how to design your systems and stuff is that go get the most scary person
Prasanna Malaiyandi:
in your environment and have them.
Prasanna Malaiyandi:
Come up with scenarios, right?
Prasanna Malaiyandi:
Come up with recovery scenarios.
Prasanna Malaiyandi:
This is the whole point of tabletop exercises, right?
Prasanna Malaiyandi:
You get those super negative people that that interject.
Prasanna Malaiyandi:
Well, what if, well, what if, you know, we're down for weeks?
Prasanna Malaiyandi:
Make
Prasanna Malaiyandi:
that's like me,
Prasanna Malaiyandi:
W. Curtis Preston: about this a lot.
Prasanna Malaiyandi:
Make the decision upfront that if we get hit by ransomware.
Prasanna Malaiyandi:
We're gonna immediately move everybody over to 365, and if
Prasanna Malaiyandi:
we do that, we're going to need to do it to recover this way.
Prasanna Malaiyandi:
They, they should have been able to foresee this decision.
Prasanna Malaiyandi:
Right, because by doing it the way they did it, again, and I don't disagree with
Prasanna Malaiyandi:
the way they did it, but by doing the way they did it, they necessitated this weird
Prasanna Malaiyandi:
double, you know, double or two step, super painful restore method that put
Prasanna Malaiyandi:
a lot of the work onto their customers.
Prasanna Malaiyandi:
Uh, and it's, it took them months.
Prasanna Malaiyandi:
Right.
Prasanna Malaiyandi:
Um, before, you know, before everybody was at, at least able, we don't even know,
Prasanna Malaiyandi:
um, exactly how many customers actually, were able to successfully recover.
Prasanna Malaiyandi:
Yeah, I think they said it was something like 3000
Prasanna Malaiyandi:
customers were impacted in their
Prasanna Malaiyandi:
hosted exchange environment.
Prasanna Malaiyandi:
Right.
Prasanna Malaiyandi:
W. Curtis Preston: right, right.
Prasanna Malaiyandi:
They said this was interesting is that they also said that play, or remember
Prasanna Malaiyandi:
play is the name of the ransomware group.
Prasanna Malaiyandi:
They accessed to use their words, the email of 27 customers and said there
Prasanna Malaiyandi:
is no evidence that they read it.
Prasanna Malaiyandi:
Um, know, the, the, what does that mean?
Prasanna Malaiyandi:
Right?
Prasanna Malaiyandi:
So again, this is where you, you have to look at like legalese.
Prasanna Malaiyandi:
They have.
Prasanna Malaiyandi:
The fact that they have no evidence that they read it doesn't mean they
Prasanna Malaiyandi:
didn't read it, it, they don't have evidence that they didn't read it right.
Prasanna Malaiyandi:
You can't prove a negative, right?
Prasanna Malaiyandi:
They downloaded the email of 27 customers.
Prasanna Malaiyandi:
And so it's likely that there could be, you know, there could
Prasanna Malaiyandi:
have been secondary attacks where play goes after the customers.
Prasanna Malaiyandi:
We didn't get any news of that, so maybe it didn't happen.
Prasanna Malaiyandi:
But, um, you know, just when you see messages like that, we have no evidence
Prasanna Malaiyandi:
that that doesn't mean it didn't happen.
Prasanna Malaiyandi:
It just means that they, they can't prove it.
Prasanna Malaiyandi:
It happened right?
Prasanna Malaiyandi:
Do you think, since you're just mentioning about
Prasanna Malaiyandi:
this, do you think there are possible ways they could have leveraged
Prasanna Malaiyandi:
security offerings, encryption, other things to protect these emails?
Prasanna Malaiyandi:
So even if play did attack their exchange server, I guess it depends
Prasanna Malaiyandi:
on what level they actually were able to exploit the exchange server.
Prasanna Malaiyandi:
W. Curtis Preston: I, I mean, they had admin access to exchange.
Prasanna Malaiyandi:
Prasanna Malaiyandi: that all bets are off?
Prasanna Malaiyandi:
W. Curtis Preston: far as all bets are off.
Prasanna Malaiyandi:
So even if there was encryption, I don't know if, if exchange has the
Prasanna Malaiyandi:
ability to encrypt it or if they could store the data on encrypted drives.
Prasanna Malaiyandi:
It doesn't matter once you're inside the application.
Prasanna Malaiyandi:
Right.
Prasanna Malaiyandi:
Because the, the, the data, it doesn't have record level encryption or anything
Prasanna Malaiyandi:
like that, that I'm aware of, but um.
Prasanna Malaiyandi:
Yeah.
Prasanna Malaiyandi:
Um, and, and the, the, the other part, and this is by the way, this is the
Prasanna Malaiyandi:
part of the story where Rackspace tries to shift blame by saying,
Prasanna Malaiyandi:
well, hey, this exploit that, that we got hit by was a zero day exploit.
Prasanna Malaiyandi:
Which is true in that it was an unknown exploit at the time that they got hit.
Prasanna Malaiyandi:
But if they had put the patch on that they should have put
Prasanna Malaiyandi:
at a minimum two weeks prior.
Prasanna Malaiyandi:
They wouldn't have been because that patch fixed the unknown problem at the time.
Prasanna Malaiyandi:
Yeah.
Prasanna Malaiyandi:
It's, I think, I know we always talk about it on the podcast.
Prasanna Malaiyandi:
It's patch, patch, patch, right?
Prasanna Malaiyandi:
When something comes, especially something with this level of severity, I.
Prasanna Malaiyandi:
Right.
Prasanna Malaiyandi:
It's like the log four J stuff that came out, what, 2021 December
Prasanna Malaiyandi:
and what a mess that was as well.
Prasanna Malaiyandi:
But everyone realized how severe of a security issue it was and
Prasanna Malaiyandi:
started patching their systems.
Prasanna Malaiyandi:
I think something similar needed to happen here as well.
Prasanna Malaiyandi:
it.
Prasanna Malaiyandi:
the fact that I get it, large customers, you have hosts, you need
Prasanna Malaiyandi:
to schedule downtime, but a high severity security issue like this
Prasanna Malaiyandi:
that was being exploited actively, like there's no reason you should have
Prasanna Malaiyandi:
waited two weeks to apply a patch.
Prasanna Malaiyandi:
W. Curtis Preston: Yeah.
Prasanna Malaiyandi:
And, and this, this is kind of what I wanted to go at here.
Prasanna Malaiyandi:
Uh, you know, we, there's a lot of lessons that could be learned from this incident.
Prasanna Malaiyandi:
One is third party backup.
Prasanna Malaiyandi:
If customers had it's had third party backup, which was totally
Prasanna Malaiyandi:
possible, they would be done.
Prasanna Malaiyandi:
I.
Prasanna Malaiyandi:
They would've, they would not have suffered this outage, um,
Prasanna Malaiyandi:
just like the Salesforce outage, um, in a previous episode, right?
Prasanna Malaiyandi:
If they had third party backup, they could have fixed it in minutes.
Prasanna Malaiyandi:
Um, the, and the other is this really hammers home when
Prasanna Malaiyandi:
there's a really severe exploit.
Prasanna Malaiyandi:
That is well known and there's a patch introduced for that exploit.
Prasanna Malaiyandi:
You need to put it in now.
Prasanna Malaiyandi:
Not.
Prasanna Malaiyandi:
Oh, we've got our patch management system.
Prasanna Malaiyandi:
We do this once every two weeks or whatever the, you know, whatever it is.
Prasanna Malaiyandi:
Look at this company basically, you know, a month from the, the announcement of the
Prasanna Malaiyandi:
patch, two months from the announcement of, uh, the actual exploit they got hit.
Prasanna Malaiyandi:
Um, and if they had simply just put the patch in.
Prasanna Malaiyandi:
When it was available, and, and I, I realized that it was available
Prasanna Malaiyandi:
sooner and there was a minor pa, but, but that, that issue was fixed.
Prasanna Malaiyandi:
There was two weeks between the time the, the patch was fully
Prasanna Malaiyandi:
fixed and fully available.
Prasanna Malaiyandi:
And the, when this actually happened, and it, it, it was yet another, this
Prasanna Malaiyandi:
was already a suffering company.
Prasanna Malaiyandi:
And if you look at the, the stock value of.
Prasanna Malaiyandi:
Um, rack Rackspace.
Prasanna Malaiyandi:
It had another sharp decline.
Prasanna Malaiyandi:
What, what did you say?
Prasanna Malaiyandi:
Wasn't it 15%
Prasanna Malaiyandi:
Yeah.
Prasanna Malaiyandi:
In
Prasanna Malaiyandi:
W. Curtis Preston: on December 2nd, which is the day they announced
Prasanna Malaiyandi:
that this had happened, right?
Prasanna Malaiyandi:
And so, you know, they were already suffering and this created
Prasanna Malaiyandi:
yet another, uh, decline in the and which did not recover.
Prasanna Malaiyandi:
It did not recover from that sharp decline.
Prasanna Malaiyandi:
Just think about this, think about you.
Prasanna Malaiyandi:
You really have to prioritize that patch management system, right?
Prasanna Malaiyandi:
You really have to make sure that you put in patches as soon as they come out.
Prasanna Malaiyandi:
Uh, you know, high level, high exploit.
Prasanna Malaiyandi:
Patches need to go in.
Prasanna Malaiyandi:
And again, I'm gonna once again say that I think the backup system needs
Prasanna Malaiyandi:
to be at the front of the line, right?
Prasanna Malaiyandi:
Patch those first, because that's the last line of defense.
Prasanna Malaiyandi:
And then make sure you, you know, in this case, the patch only applied to exchange,
Prasanna Malaiyandi:
but this is the point that we make.
Prasanna Malaiyandi:
If they had just put the patches in, this event would've never happened.
Prasanna Malaiyandi:
Yeah.
Prasanna Malaiyandi:
And you would still have 3000 customers using Rackspace hosted exchange, right?
Prasanna Malaiyandi:
I.
Prasanna Malaiyandi:
W. Curtis Preston: Yeah.
Prasanna Malaiyandi:
Yeah.
Prasanna Malaiyandi:
The other comment I was gonna make, Curtis, is
Prasanna Malaiyandi:
Given that they probably had the managed exchange solution for quite a while,
Prasanna Malaiyandi:
I wonder if they ever had a process.
Prasanna Malaiyandi:
I know we've talked about on the podcast of going back and looking
Prasanna Malaiyandi:
at their disaster recovery plans or their backup and recovery procedures.
Prasanna Malaiyandi:
Or if it was just sort of, Hey, we created this once, it should be fine.
Prasanna Malaiyandi:
We never have to really use it, so we'll never go back and make sure
Prasanna Malaiyandi:
it's up to date and all the rest.
Prasanna Malaiyandi:
W. Curtis Preston: Well, I mean, clearly they didn't, they didn't.
Prasanna Malaiyandi:
Account for ransomware, right?
Prasanna Malaiyandi:
A typical DR recovery scenario would've worked fine here, right?
Prasanna Malaiyandi:
If they, if the building caught fire, they knew how to re, they had backups.
Prasanna Malaiyandi:
They knew how to restore their exchange servers.
Prasanna Malaiyandi:
I think, I still think it took them longer than it should have, but
Prasanna Malaiyandi:
they had, they had a plan for that.
Prasanna Malaiyandi:
But that plan ransomware breaks a lot,
Prasanna Malaiyandi:
Why would ransomware be different than
Prasanna Malaiyandi:
the building going up in smoke?
Prasanna Malaiyandi:
W. Curtis Preston: Because, great question.
Prasanna Malaiyandi:
Because they likely were still fighting the ransomware itself, right?
Prasanna Malaiyandi:
When the building goes up in smoke, they could just literally restore everything.
Prasanna Malaiyandi:
It's gonna take a couple days, but at this point they're, they have an unknown time.
Prasanna Malaiyandi:
Yeah.
Prasanna Malaiyandi:
W. Curtis Preston: On the moment of December 2nd, they're like, we have
Prasanna Malaiyandi:
been taken down and we have no long, no idea how long we're going to be down.
Prasanna Malaiyandi:
Yeah.
Prasanna Malaiyandi:
Might be a.
Prasanna Malaiyandi:
day, might be a week, might be a.
Prasanna Malaiyandi:
W. Curtis Preston: Exactly right.
Prasanna Malaiyandi:
They, they're looking at these, at these other businesses that
Prasanna Malaiyandi:
are down for months at a time
Prasanna Malaiyandi:
and they're saying, we don't want to be that.
Prasanna Malaiyandi:
We don't wanna stop people's email for that amount of time.
Prasanna Malaiyandi:
Boom, let's do over here.
Prasanna Malaiyandi:
Had they, had it been adjusted regular, a fire of flood or whatever, they
Prasanna Malaiyandi:
would, they should have been able to say, this is gonna take us three days.
Prasanna Malaiyandi:
Three days is gonna stink, but.
Prasanna Malaiyandi:
It's not enough for us to abandon our entire business
Prasanna Malaiyandi:
model and move over to 365.
Prasanna Malaiyandi:
In this case, it was an unknown scenario, unknown amount of
Prasanna Malaiyandi:
time that they're gonna be down.
Prasanna Malaiyandi:
And so they decide to do this method that ended up making
Prasanna Malaiyandi:
everything take much longer.
Prasanna Malaiyandi:
Right?
Prasanna Malaiyandi:
Which, and they didn't test for this method.
Prasanna Malaiyandi:
so yes, I agree with everything, but I still wanna
Prasanna Malaiyandi:
go back to another clarifying point.
Prasanna Malaiyandi:
Couldn't they have treated ransomware?
Prasanna Malaiyandi:
Like I wanna, I wonder if they actually had disaster recovery plans in place.
Prasanna Malaiyandi:
Because
Prasanna Malaiyandi:
if I yeah, and this.
Prasanna Malaiyandi:
W. Curtis Preston: giving them the benefit of the doubt.
Prasanna Malaiyandi:
Yeah, because because in my mind, right, if I
Prasanna Malaiyandi:
had a ransomware happen, right as a company, there are two options, right?
Prasanna Malaiyandi:
One is I could try to figure out what all went wrong, rebuild my servers
Prasanna Malaiyandi:
in the same data center, procure hardware, all the rest, or I could just
Prasanna Malaiyandi:
treat it like a fire, shoot it in the head, connect completely, disconnect
Prasanna Malaiyandi:
everything, validate my DR site is still good, that there's no ransomware
Prasanna Malaiyandi:
there, and then bring everything up or restore on clean hardware, et cetera.
Prasanna Malaiyandi:
Like
Prasanna Malaiyandi:
W. Curtis Preston: if there's not ransomware or if there is ransomware.
Prasanna Malaiyandi:
even if there was ransomware,
Prasanna Malaiyandi:
W. Curtis Preston: But the problem is if you have a solid backup system, you have a
Prasanna Malaiyandi:
recent backup, and then you go to restore it, you know you're restoring the The
Prasanna Malaiyandi:
bad stuff.
Prasanna Malaiyandi:
Yeah, So, but then that should have just been a matter of figuring
Prasanna Malaiyandi:
out what is good and what is bad.
Prasanna Malaiyandi:
W. Curtis Preston: Right, which is going to take an unknown amount of time.
Prasanna Malaiyandi:
That was the problem.
Prasanna Malaiyandi:
Right.
Prasanna Malaiyandi:
And, and, um, yeah, I, I don't know if they had a DR plan.
Prasanna Malaiyandi:
I do, I do ask myself why it took them, the amount of time it took them
Prasanna Malaiyandi:
to restore all the exchange servers.
Prasanna Malaiyandi:
I, I just, I just go off the, I just go off the, you know, the message that
Prasanna Malaiyandi:
I, or the information that I have.
Prasanna Malaiyandi:
yeah.
Prasanna Malaiyandi:
W. Curtis Preston: Yeah.
Prasanna Malaiyandi:
And so the end of the story is this, uh, the company continued
Prasanna Malaiyandi:
to suffer, uh, additional losses in the value of their company.
Prasanna Malaiyandi:
They, they, they went from 800 million, then down to 350 million
Prasanna Malaiyandi:
now, uh, and they, they are close as of the taping of this episode.
Prasanna Malaiyandi:
They're close to a restructuring deal.
Prasanna Malaiyandi:
I don't know if this event had ever happened, if how different
Prasanna Malaiyandi:
the world would be right now, but it certainly didn't help.
Prasanna Malaiyandi:
Um, so please, folks, all I can say is, you know, put those
Prasanna Malaiyandi:
patches in when you, you know, and test your recovery procedures.
Prasanna Malaiyandi:
All of the recovery scenarios, right?
Prasanna Malaiyandi:
And if ransomware isn't one of the recovery scenarios, then
Prasanna Malaiyandi:
you need to rewrite your recovery plans.
Prasanna Malaiyandi:
W. Curtis Preston: You need to reconsider your recovery scenarios.
Prasanna Malaiyandi:
Absolutely.
Prasanna Malaiyandi:
Uh, all right, well, uh, I think this was a good episode.
Prasanna Malaiyandi:
Yeah.
Prasanna Malaiyandi:
Don't be like Rackspace
Prasanna Malaiyandi:
W. Curtis Preston: was, you know, ouch.
Prasanna Malaiyandi:
All right.
Prasanna Malaiyandi:
Uh, thanks.
Prasanna Malaiyandi:
Thanks for the chat.
Prasanna Malaiyandi:
like, uh.
Prasanna Malaiyandi:
It was fun, Curtis, and I am sure I will hear all about your,
Prasanna Malaiyandi:
uh, your, uh, event tonight.
Prasanna Malaiyandi:
W. Curtis Preston: You, you will be hearing all about it.
Prasanna Malaiyandi:
You'll probably be getting some live, some live chat or some
Prasanna Malaiyandi:
live, uh, texts during the event.
Prasanna Malaiyandi:
Um, and, uh, I, uh, be safe out there folks.
Prasanna Malaiyandi:
That is a wrap.
Listen On
