Cloud catastrophes: Codespaces.com deleted out of existence
In 2014, software-as-a-service company Code Spaces disappeared overnight after a devastating cyber attack. Thousands of coders lost access to their work when insufficient cloud backups failed under pressure. The company was forced to go out of business.
Learn the tragic tale of how Code Spaces ignored standard data protection rules, putting their business and clients at risk. We’ll unpack what went wrong with their cloud architecture and backup systems, allowing a single hacker to destroy their SaaS company.
Understand why you still need backup - even native cloud redundancy isn't enough. Our hosts explore the hard lessons from this cloud catastrophe and equip you with actionable advice around security, access controls, preparation, and backup policies. Safeguard your slice of the cloud and avoid the mistakes that ultimately shuttered Code Spaces.
Articles covering this story:
- https://www.esecurityplanet.com/networks/code-spaces-destroyed-by-cyber-attack/
- https://www.itgovernance.co.uk/blog/the-attack-that-forced-code-spaces-out-of-business-what-went-wrong
- https://www.breaches.cloud/incidents/codespaces/
- https://threatpost.com/hacker-puts-hosting-service-code-spaces-out-of-business/106761/
- https://thehackernews.com/2014/06/cyber-attack-on-code-spaces-puts.html
- https://www.csoonline.com/article/547518/disaster-recovery-code-spaces-forced-to-close-its-doors-after-security-incident.html
- https://blogs.manageengine.com/it-security/passwordmanagerpro/2014/08/20/code-spaces-aws-security-breach-a-sad-reminder-of-the-importance-of-cloud-environment-password-management.html
Speaker:
W. Curtis Preston: Spoiler alert, the company in this episode ceased to exist
Speaker:
due to what happened in this story.
Speaker:
Disasters happen, and since the cloud is just someone else's data
Speaker:
center, they sometimes happen in the cloud, and sometimes they
Speaker:
take companies along with them.
Speaker:
This episode is the first in a series called Cloud Disasters.
Speaker:
Each episode tells the real story.
Speaker:
Of a company who failed to back up their cloud data and suffered as a result.
Speaker:
And these aren't podunk cloud vendors either.
Speaker:
Every vendor covered in this story is a major vendor.
Speaker:
I'm not kidding around when I say you should back up your cloud data.
Speaker:
I don't care that there are those who think I'm just pedaling fud.
Speaker:
Newsflash, the only reason we've ever backed up anything is because
Speaker:
of the fear of losing it and the uncertainty and doubt we have in
Speaker:
the system that we're backing up.
Speaker:
Fans of the podcast know, the whole reason that I became a backup specialist is that
Speaker:
in 1993, I was unable to recover a very important Oracle database for my employer.
Speaker:
I didn't want that to happen to anyone else.
Speaker:
So I've dedicated myself to helping others protect themselves
Speaker:
from feeling that awful feeling.
Speaker:
This show is aimed at you, the unappreciated backup admin, and we want
Speaker:
to turn you into a cyber recovery hero.
Speaker:
This is the backup wrap up.
Speaker:
Welcome to the show.
Speaker:
I'm your host, w Curtis Preston, and I have with me my continued and necessary
Speaker:
Tesla consultant Prasanna Malaiyandi.
Speaker:
How's it going Prasanna
Speaker:
persona
Prasanna Malaiyandi:
Pana.
Prasanna Malaiyandi:
I am doing well Curtis, and how have you been?
Prasanna Malaiyandi:
W. Curtis Preston: I, well, as you know, I have been.
Prasanna Malaiyandi:
Fiddling.
Prasanna Malaiyandi:
Uh, why am I not surprised?
Prasanna Malaiyandi:
W. Curtis Preston: I have been, fiddling with the automations of my Tesla now.
Prasanna Malaiyandi:
Tesla's come in, I'm gonna say two battery flavors, right?
Prasanna Malaiyandi:
There's NMC and LFP.
Prasanna Malaiyandi:
Nickel Manganese Cobalt and Lithium Iron Phosphate.
Prasanna Malaiyandi:
And I have the latter.
Prasanna Malaiyandi:
And the latter is not supposed to be subject to the same don't
Prasanna Malaiyandi:
charge it to a hundred percent unless you need it right away.
Prasanna Malaiyandi:
Issue.
Prasanna Malaiyandi:
Uh,
Prasanna Malaiyandi:
In fact tell
Prasanna Malaiyandi:
you to charge it to a hundred
Prasanna Malaiyandi:
W. Curtis Preston: I do tell you to charge it to a hundred
Prasanna Malaiyandi:
percent at least once a week.
Prasanna Malaiyandi:
Um, but I was charging it to a hundred percent every day.
Prasanna Malaiyandi:
And I was thinking that I, that even though it doesn't have necessarily
Prasanna Malaiyandi:
the same issues, I could still get better battery life by not charging to
Prasanna Malaiyandi:
a hundred percent unless I needed to.
Prasanna Malaiyandi:
Uh, which, uh, for me is not all the time.
Prasanna Malaiyandi:
And so, but I'm also very absent-minded, and so I couldn't like lower the
Prasanna Malaiyandi:
percentage and then remember to, to, to, to raise the percentage later.
Prasanna Malaiyandi:
And so, uh, I found this wonderful app called Tessie, and I've been obsessing
Prasanna Malaiyandi:
over it for about a week or two.
Prasanna Malaiyandi:
A week.
Prasanna Malaiyandi:
It's been nine days.
Prasanna Malaiyandi:
W. Curtis Preston: Yeah.
Prasanna Malaiyandi:
Yeah.
Prasanna Malaiyandi:
I will say if there's any listeners that are a Tesla owner, I.
Prasanna Malaiyandi:
Uh, you should just get Tessie, uh, T-E-S-S-I-E in the app store.
Prasanna Malaiyandi:
I bought it for the automation.
Prasanna Malaiyandi:
What I also got was this immense amount of analytics and reporting
Prasanna Malaiyandi:
and, um, all this great, great stuff.
Prasanna Malaiyandi:
And also reminders.
Prasanna Malaiyandi:
That's another, speaking of being absent-minded, it will tell me, hey.
Prasanna Malaiyandi:
Idiot, you're home and your car's not plugged in.
Prasanna Malaiyandi:
Yeah, because how many times has that
Prasanna Malaiyandi:
happened to you, Curtis, that you've come home and you're like,
Prasanna Malaiyandi:
W. Curtis Preston: you know, a couple, uh, enough that it was annoying.
Prasanna Malaiyandi:
Uh, it, I will say it, nothing is more annoying than, you know, basically
Prasanna Malaiyandi:
driving your car down to, you know, the electrical equivalent of fumes.
Prasanna Malaiyandi:
And then, um, and then getting up in the morning and going,
Prasanna Malaiyandi:
okay, I'm ready to drive today.
Prasanna Malaiyandi:
And you're like, oh, crap, I gotta go to the.
Prasanna Malaiyandi:
I gotta go the supercharger for 20 minutes before I could do anything else.
Prasanna Malaiyandi:
Um, yeah, so it's got, you know, it's got that the notifications, it
Prasanna Malaiyandi:
reminds me to, to rotate my tires.
Prasanna Malaiyandi:
And also gives me analytics about my driving and my efficiency and, yeah.
Prasanna Malaiyandi:
So, yeah.
Prasanna Malaiyandi:
So happy, happy, happy, happy, happy.
Prasanna Malaiyandi:
Um, but you know, who's not happy?
Prasanna Malaiyandi:
The people that are gonna be in this new series, they're not happy.
Prasanna Malaiyandi:
And this is a new series.
Prasanna Malaiyandi:
One of the things that you've heard us say is what Bana.
Prasanna Malaiyandi:
Back up the cloud.
Prasanna Malaiyandi:
W. Curtis Preston: Yes, backup up the cloud.
Prasanna Malaiyandi:
The cloud is not magic.
Prasanna Malaiyandi:
There is no such thing as the cloud.
Prasanna Malaiyandi:
It's just somebody else's computer.
Prasanna Malaiyandi:
Uh uh, the cloud doesn't magically back up itself.
Prasanna Malaiyandi:
Despite what you may have been told the cloud there are parts of the cloud where.
Prasanna Malaiyandi:
People generally agree with me.
Prasanna Malaiyandi:
I, I think probably the best example would be something like AWS, you know, like EC2.
Prasanna Malaiyandi:
They're like, yes, we know EC2 needs to be backed up.
Prasanna Malaiyandi:
Um, and, and then we just argue a little bit over how that's going to be done.
Prasanna Malaiyandi:
But when we get to the extreme it, the other end of that, we get the
Prasanna Malaiyandi:
SaaS world, we get the Microsoft 365 lovers who say, oh, this, that,
Prasanna Malaiyandi:
this doesn't need to be backed up.
Prasanna Malaiyandi:
you know, or G
Prasanna Malaiyandi:
Suite and, we have gone back over the last, uh, several years and selected, um,
Prasanna Malaiyandi:
a number of, you know, poor victims of.
Prasanna Malaiyandi:
This belief and, um, I, I can think of no better, uh, story to start this
Prasanna Malaiyandi:
out than code spaces because it, I, I think it was kind of the first,
Prasanna Malaiyandi:
it was probably the first that was sort of well
Prasanna Malaiyandi:
publicized as well as having very dire consequences for not backing up the cloud.
Prasanna Malaiyandi:
I.
Prasanna Malaiyandi:
W. Curtis Preston: Right.
Prasanna Malaiyandi:
And it, it's interesting from a timing perspective, it happened in 2014.
Prasanna Malaiyandi:
Which to me is the year of the beginning of the massive level of ransomware.
Prasanna Malaiyandi:
Now, I know ransomware actually goes way back longer.
Prasanna Malaiyandi:
You know, much longer be before that.
Prasanna Malaiyandi:
But to me, 2014 is really when I started seeing ransomware kind of everywhere.
Prasanna Malaiyandi:
And this was technically a ransomware attack, not in the traditional sense
Prasanna Malaiyandi:
that we think of today, but, but it was, so Code spaces.com was a site, and this
Prasanna Malaiyandi:
is the, the irony of all ri ironies.
Prasanna Malaiyandi:
And, and I will say that unlike Alanis Morissette, I actually
Prasanna Malaiyandi:
know what the word ironic means.
Prasanna Malaiyandi:
Um, I love the song.
Prasanna Malaiyandi:
It's just, there's so many things to that song that are not in any way ironic.
Prasanna Malaiyandi:
They just suck.
Prasanna Malaiyandi:
Um, code spaces.com was a site to store your code.
Prasanna Malaiyandi:
It was like, um.
Prasanna Malaiyandi:
GitHub.
Prasanna Malaiyandi:
W. Curtis Preston: It was like, it was like a GitHub and they had
Prasanna Malaiyandi:
many, many customers and it was a safe space to store your code.
Prasanna Malaiyandi:
Hence the name code spaces.com.
Prasanna Malaiyandi:
Yeah, so basically like you mentioned, it was a place
Prasanna Malaiyandi:
companies could store their code and this was way back in the day.
Prasanna Malaiyandi:
And so it's like, hey, if you just have your own code sitting locally
Prasanna Malaiyandi:
on your system, because not everyone was comfortable with the cloud, they
Prasanna Malaiyandi:
offered a service that allows you to store your code there and keep it safe.
Prasanna Malaiyandi:
And now according to a cash version of their website, because of course
Prasanna Malaiyandi:
you can't find anything anymore about them, uh, they had over 200 customers.
Prasanna Malaiyandi:
A week or 200 companies a week using their service, which isn't
Prasanna Malaiyandi:
like small beans, right?
Prasanna Malaiyandi:
And yes, it's not like the thousands of millions, but there are 200
Prasanna Malaiyandi:
customers who now no longer have access to their code anymore because
Prasanna Malaiyandi:
of what happened in code spaces.
Prasanna Malaiyandi:
The other interesting thing is according to their websites, and I will quote
Prasanna Malaiyandi:
backing up, data is one thing, but is meaningless without a recovery plan.
Prasanna Malaiyandi:
Not only that, a recovery plan and one that is well practiced and
Prasanna Malaiyandi:
proven to work time and time again.
Prasanna Malaiyandi:
Code Spaces has a full recovery plan that has been proven to
Prasanna Malaiyandi:
work and is in fact practiced.
Prasanna Malaiyandi:
W. Curtis Preston: That
Prasanna Malaiyandi:
do you think about that, Curtis?
Prasanna Malaiyandi:
W. Curtis Preston: That sounds really good.
Prasanna Malaiyandi:
Um, you know, based on what we know happened, clearly they didn't test for
Prasanna Malaiyandi:
all scenarios.
Prasanna Malaiyandi:
W. Curtis Preston: Right.
Prasanna Malaiyandi:
And they specifically didn't test for cyber attack.
Prasanna Malaiyandi:
Um, but.
Prasanna Malaiyandi:
the time could you fault them really?
Prasanna Malaiyandi:
W. Curtis Preston: I don't know.
Prasanna Malaiyandi:
Well, they had poor backup design.
Prasanna Malaiyandi:
Just, you know what, what we're
Prasanna Malaiyandi:
gonna get to, what we're gonna find out.
Prasanna Malaiyandi:
They had full backup.
Prasanna Malaiyandi:
They had poor backup design.
Prasanna Malaiyandi:
They failed to follow what persona?
Prasanna Malaiyandi:
The 3, 2, 1 rule.
Prasanna Malaiyandi:
You know, we haven't talked about this in a long, long time.
Prasanna Malaiyandi:
Do you wanna quickly mention it to some of our
Prasanna Malaiyandi:
W. Curtis Preston: Yeah.
Prasanna Malaiyandi:
So the 3, 2, 1 role, and by the way, many companies have said
Prasanna Malaiyandi:
it's gotta be more than 3, 2, 1.
Prasanna Malaiyandi:
Yes, I agree.
Prasanna Malaiyandi:
Right?
Prasanna Malaiyandi:
These days it has to be more than 3, 2, 1, but if it's not 3, 2, 1, there's no point
Prasanna Malaiyandi:
in talking about the other ones, right?
Prasanna Malaiyandi:
Having at least three versions on two different media, and the, the,
Prasanna Malaiyandi:
the idea here is on, on things that are subject to different.
Prasanna Malaiyandi:
Risk profiles, right?
Prasanna Malaiyandi:
Maybe it's disco and tape, maybe it's, uh, on-Prem and off-Prem.
Prasanna Malaiyandi:
Maybe it's, you know, on-prem and Cloud.
Prasanna Malaiyandi:
Maybe it's, um, you know, a different region, et cetera.
Prasanna Malaiyandi:
And then one three, the one is make sure that there's something off site.
Prasanna Malaiyandi:
Um, they had neither the two nor the one,
Prasanna Malaiyandi:
but we're gonna get to that in a minute.
Prasanna Malaiyandi:
and, and for those listeners who wanna know more,
Prasanna Malaiyandi:
we actually had an episode with the person who coined the term 3, 2, 1, who
Prasanna Malaiyandi:
comes from digital photography, in fact.
Prasanna Malaiyandi:
And so you should go take a listen to that episode.
Prasanna Malaiyandi:
We'll
Prasanna Malaiyandi:
W. Curtis Preston: Yeah, we'll put.
Prasanna Malaiyandi:
Yeah, we'll put a show notes.
Prasanna Malaiyandi:
Peter Krogh.
Prasanna Malaiyandi:
Um, great guy.
Prasanna Malaiyandi:
So what happened?
Prasanna Malaiyandi:
Well, we have that a hacker gained privilege credentials that we still
Prasanna Malaiyandi:
don't know how that happened, right?
Prasanna Malaiyandi:
They're saying probably through phishing or possibly through, uh,
Prasanna Malaiyandi:
stored EC2 access keys in a public code
Prasanna Malaiyandi:
repository.
Prasanna Malaiyandi:
Which happened a lot back then and still does today.
Prasanna Malaiyandi:
W. Curtis Preston: Yeah.
Prasanna Malaiyandi:
Yeah.
Prasanna Malaiyandi:
The, um, there was a, uh, security researcher at Tripwire that said, that
Prasanna Malaiyandi:
this is a problem for people how to, how to, manage authentication codes like this.
Prasanna Malaiyandi:
And, uh, they said they, they had seen thousands of EC2 accounts
Prasanna Malaiyandi:
abused after storing their EC2 keys in public code repositories.
Prasanna Malaiyandi:
Ouch.
Prasanna Malaiyandi:
Right.
Prasanna Malaiyandi:
Um, but we don't, so we don't know.
Prasanna Malaiyandi:
Someone
Prasanna Malaiyandi:
got access.
Prasanna Malaiyandi:
Yeah.
Prasanna Malaiyandi:
Would it be ironic if Code Spaces was using code spaces
Prasanna Malaiyandi:
to store their code and they left a public repository with their EC2 key?
Prasanna Malaiyandi:
W. Curtis Preston: That wouldn't indeed be ironic.
Prasanna Malaiyandi:
Um, but yeah, so we don't know exactly how this hacker, uh, this,
Prasanna Malaiyandi:
you know, bad actor got access to the, to the environment, but they did.
Prasanna Malaiyandi:
And the first thing that they did was they started a DDoS attack.
Prasanna Malaiyandi:
Uh, what is a
Prasanna Malaiyandi:
DDoS attack persona?
Prasanna Malaiyandi:
This is a DI distributed denial of service, so
Prasanna Malaiyandi:
you have a bunch of servers outside hammering various servers at a company.
Prasanna Malaiyandi:
Causing it, flooding it with a lot of traffic, which then
Prasanna Malaiyandi:
causes it to stop responding.
Prasanna Malaiyandi:
So if you're code spaces, someone did a DDoS attack on you now, you wouldn't
Prasanna Malaiyandi:
be able to serve and function as a service to those 200 paying customers.
Prasanna Malaiyandi:
W. Curtis Preston: Yeah, the really interesting thing about this is that the
Prasanna Malaiyandi:
DDoS attack was apparently subterfuge.
Prasanna Malaiyandi:
Right.
Prasanna Malaiyandi:
Um, you know, look over here
Prasanna Malaiyandi:
while I, nothing up but
Prasanna Malaiyandi:
which I think is still common today, right?
Prasanna Malaiyandi:
There are still a lot of companies who they're trying to hide their tracks
Prasanna Malaiyandi:
and they're like, Hey, everyone's gonna fight, and DDoS were common.
Prasanna Malaiyandi:
And so they have a plan in place and everyone's scrambling there where you're
Prasanna Malaiyandi:
like, Hey, look at my left hand while I'm doing something with my right.
Prasanna Malaiyandi:
W. Curtis Preston: And then I, I think the, I don't know if interesting
Prasanna Malaiyandi:
is right, but the hacker left contact details for themselves in
Prasanna Malaiyandi:
the customer's Amazon dashboard.
Prasanna Malaiyandi:
Right.
Prasanna Malaiyandi:
Uh, this was, this was, I, I think, I think maybe we haven't mentioned it.
Prasanna Malaiyandi:
This was an AWS customer.
Prasanna Malaiyandi:
And this is why we call it ransomware, right?
Prasanna Malaiyandi:
Even though it's not your traditional ransomware like we
Prasanna Malaiyandi:
think today, it more or less is.
Prasanna Malaiyandi:
W. Curtis Preston: Right.
Prasanna Malaiyandi:
Yeah.
Prasanna Malaiyandi:
So then, uh, I'll quote from there.
Prasanna Malaiyandi:
There was a page, by the way, there was a page.
Prasanna Malaiyandi:
Uh, basically code spaces.com died after this, and it became a, they,
Prasanna Malaiyandi:
they replaced it with just one page that says, here's what happened.
Prasanna Malaiyandi:
And we're, I'm gonna read quotes from that page that are no longer
Prasanna Malaiyandi:
available because they sold the domain.
Prasanna Malaiyandi:
Um, upon realizing that somebody had access to our control panel, we started
Prasanna Malaiyandi:
to investigate how access had been gained and what access that person
Prasanna Malaiyandi:
had, uh, to the data in our systems.
Prasanna Malaiyandi:
It became clear that so far no machine had access had been achieved due to the
Prasanna Malaiyandi:
intruder not having our private keys.
Prasanna Malaiyandi:
Um, that's what they thought, right?
Prasanna Malaiyandi:
Um, but things
Prasanna Malaiyandi:
turned, uh, yeah, turned ugly.
Prasanna Malaiyandi:
Uh, why don't you talk, talk about
Prasanna Malaiyandi:
the next one there.
Prasanna Malaiyandi:
So what was the response?
Prasanna Malaiyandi:
So Code Spaces did a smart thing and one of their first response.
Prasanna Malaiyandi:
Actions was to change all of its EC2 passwords, but quickly, code
Prasanna Malaiyandi:
spaces discovered that the attacker had created backup logins, which any
Prasanna Malaiyandi:
sensible person's going to do, right?
Prasanna Malaiyandi:
You're never gonna say, Hey, I'm only gonna have one admin in my environment.
Prasanna Malaiyandi:
And so the attacker was able to create all these backup logins, and so now they
Prasanna Malaiyandi:
were able to just go back into the system.
Prasanna Malaiyandi:
And continue doing the attacks.
Prasanna Malaiyandi:
And once they realized that Code Spaces was trying to actually recover and take
Prasanna Malaiyandi:
control from the attacker, the attacker then started to go and just delete things
Prasanna Malaiyandi:
from the control panel because they had super user access at that point.
Prasanna Malaiyandi:
Right.
Prasanna Malaiyandi:
So they could do whatever they wanted.
Prasanna Malaiyandi:
W. Curtis Preston: Yeah, somewhere in here, in the version of the story
Prasanna Malaiyandi:
that I have, I remember there was a d, there was a, there was an attempt
Prasanna Malaiyandi:
at a ransom, basically give us this amount of money, or, or, or we're
Prasanna Malaiyandi:
gonna, you know, do bad things.
Prasanna Malaiyandi:
Um, but the, but you know, they, they obviously didn't want to pay the ransom.
Prasanna Malaiyandi:
And then, uh, then what
Prasanna Malaiyandi:
happened?
Prasanna Malaiyandi:
And so finally code spaces got their control
Prasanna Malaiyandi:
panel access back, but not before.
Prasanna Malaiyandi:
The attacker had caused quite a bit of damage, so the attacker had gone
Prasanna Malaiyandi:
removed all the EBS syn snapshots, S3 buckets, all the amis, which are the
Prasanna Malaiyandi:
Amazon machine instances, some EBS instances and several machine instances.
Prasanna Malaiyandi:
And this is a quote from the same webpage that Curtis was talking about.
Prasanna Malaiyandi:
In summary, most of our data backups, machine configurations
Prasanna Malaiyandi:
and offsite backups were either partially or completely deleted.
Prasanna Malaiyandi:
W. Curtis Preston: What I, I, you know, I did.
Prasanna Malaiyandi:
I never noticed that phrase before.
Prasanna Malaiyandi:
And they said And offsite backups.
Prasanna Malaiyandi:
What?
Prasanna Malaiyandi:
Offsite backups.
Prasanna Malaiyandi:
What I mean, what be, because the hacker only had access to
Prasanna Malaiyandi:
the one account.
Prasanna Malaiyandi:
I am guessing that they replicated
Prasanna Malaiyandi:
to another AWS region within the
Prasanna Malaiyandi:
W. Curtis Preston: Okay.
Prasanna Malaiyandi:
Okay.
Prasanna Malaiyandi:
Another zone within the
Prasanna Malaiyandi:
same account.
Prasanna Malaiyandi:
All right.
Prasanna Malaiyandi:
Or or another region.
Prasanna Malaiyandi:
Yeah,
Prasanna Malaiyandi:
W. Curtis Preston: Yeah.
Prasanna Malaiyandi:
yeah, But still within the
Prasanna Malaiyandi:
same account
Prasanna Malaiyandi:
Possibly within the same account.
Prasanna Malaiyandi:
W. Curtis Preston: Yeah.
Prasanna Malaiyandi:
Um, yeah, yeah, possibly you're right.
Prasanna Malaiyandi:
We don't know for sure, but if it wasn't the same account, then the hacker had
Prasanna Malaiyandi:
to gain access to multiple accounts.
Prasanna Malaiyandi:
Right.
Prasanna Malaiyandi:
Yeah.
Prasanna Malaiyandi:
Yep.
Prasanna Malaiyandi:
W. Curtis Preston: Um, yeah.
Prasanna Malaiyandi:
So basically this is the equivalent of blowing up somebody's data center, right?
Prasanna Malaiyandi:
Um, because basically they just.
Prasanna Malaiyandi:
In a matter of a few keystrokes, they just deleted essentially everything, right?
Prasanna Malaiyandi:
Everything that mattered, or enough things that mattered that they, um,
Prasanna Malaiyandi:
you know, uh, took out the company.
Prasanna Malaiyandi:
One of the things that I don't know if you found any
Prasanna Malaiyandi:
information about Curtis is did they ever reach out to law enforcement
Prasanna Malaiyandi:
or even AWS's security operations to be like, Hey, I have this issue.
Prasanna Malaiyandi:
Can you help me?
Prasanna Malaiyandi:
W. Curtis Preston: Yeah, we don't, I, I, I'm going to.
Prasanna Malaiyandi:
Assume that once you know, the feces hit the rotary oscillator,
Prasanna Malaiyandi:
I'm sure they called AWS.
Prasanna Malaiyandi:
I mean, of course they called AWS, but what we know is that
Prasanna Malaiyandi:
right, if you know AWS isn't magic, and if you didn't follow the architecture
Prasanna Malaiyandi:
and do the things that you were supposed to do, A-W-A-W-S can't
Prasanna Malaiyandi:
undo that.
Prasanna Malaiyandi:
Yeah, well, I was just wondering before it changed, its EC2
Prasanna Malaiyandi:
passwords, if they had reached out to AWS, if they could have helped them in some
Prasanna Malaiyandi:
way or been like, Hey, here are the best practices for locking down your account,
Prasanna Malaiyandi:
W. Curtis Preston: Yeah, I'm, I'm gonna guess based on how they responded
Prasanna Malaiyandi:
that they did not do that right.
Prasanna Malaiyandi:
Um, I don't think their response was the best thing they
Prasanna Malaiyandi:
could have done at the time.
Prasanna Malaiyandi:
Um, so then we have, um.
Prasanna Malaiyandi:
So here's again, I'm reading from their co their, from their quote
Prasanna Malaiyandi:
here, code spaces will not be able to operate beyond this point.
Prasanna Malaiyandi:
The cost of resolving this issue to date and the expected cost of refunding.
Prasanna Malaiyandi:
Customers who have been left without the service they paid for, we'll
Prasanna Malaiyandi:
put code spaces in an irreversible position, both financially and
Prasanna Malaiyandi:
in terms of ongoing credibility.
Prasanna Malaiyandi:
No kidding.
Prasanna Malaiyandi:
Um, as such, at this point.
Prasanna Malaiyandi:
We have no alternative but to cease trading and concentrate on supporting
Prasanna Malaiyandi:
our affected customers and exporting any remaining data they have left with us.
Prasanna Malaiyandi:
Ouch.
Prasanna Malaiyandi:
W. Curtis Preston: That is, um, that's a tough one.
Prasanna Malaiyandi:
So basically, you know, hacker gets in Hacker, uh, you know.
Prasanna Malaiyandi:
Offers are ransom.
Prasanna Malaiyandi:
They try to, instead of paying the ransom, they try to lock
Prasanna Malaiyandi:
the hacker out unsuccessfully,
Prasanna Malaiyandi:
Yep.
Prasanna Malaiyandi:
W. Curtis Preston: and then the hacker deletes the company, the uh, right.
Prasanna Malaiyandi:
I mean, they deleted basically everything, you know, as much as they could get
Prasanna Malaiyandi:
access to in that, you know, in that
Prasanna Malaiyandi:
account.
Prasanna Malaiyandi:
That's crazy.
Prasanna Malaiyandi:
W. Curtis Preston: Yeah.
Prasanna Malaiyandi:
Yeah.
Prasanna Malaiyandi:
This is one of those where it's like, I, I always used this story to
Prasanna Malaiyandi:
recommend backup design for the cloud.
Prasanna Malaiyandi:
Even though I don't think I've ever encountered someone who says,
Prasanna Malaiyandi:
well, I don't need to back up EC2, I don't think I've ever heard that.
Prasanna Malaiyandi:
Yeah.
Prasanna Malaiyandi:
W. Curtis Preston: I have very commonly found people whose backup design was very
Prasanna Malaiyandi:
similar to this company's backup design.
Prasanna Malaiyandi:
And, um, so, well let me ask you this.
Prasanna Malaiyandi:
What do you think, um, what could they have done differently to stop this?
Prasanna Malaiyandi:
To prevent this from happening.
Prasanna Malaiyandi:
I would say the first could have been take your
Prasanna Malaiyandi:
backups to a different account.
Prasanna Malaiyandi:
In a different region, or even in the same region, that would've
Prasanna Malaiyandi:
at least kept your data safe.
Prasanna Malaiyandi:
W. Curtis Preston: Yeah, I, I, yeah, I, I completely agree with you.
Prasanna Malaiyandi:
I would probably just say as long as you, as long as you're, I, I guess what.
Prasanna Malaiyandi:
I'm, I'm trying to, I'm factor, I'm thinking in my head, like,
Prasanna Malaiyandi:
from a cost perspective, does it cost extra to send to another
Prasanna Malaiyandi:
region?
Prasanna Malaiyandi:
Is that,
Prasanna Malaiyandi:
Prasanna Malaiyandi: Yeah, usually it does.
Prasanna Malaiyandi:
Yep.
Prasanna Malaiyandi:
W. Curtis Preston: Okay.
Prasanna Malaiyandi:
So
Prasanna Malaiyandi:
But they already have an offsite backups,
Prasanna Malaiyandi:
right?
Prasanna Malaiyandi:
So.
Prasanna Malaiyandi:
W. Curtis Preston: right?
Prasanna Malaiyandi:
Yeah.
Prasanna Malaiyandi:
So.
Prasanna Malaiyandi:
Um, I, I think that your backup should be copied to another
Prasanna Malaiyandi:
region and another account.
Prasanna Malaiyandi:
I, I actually, and, and, and, and I'll say that, you know, my, my
Prasanna Malaiyandi:
opinion is somewhat peppered by having worked for a company that does this.
Prasanna Malaiyandi:
But there are companies that will backup your cloud data and then get it out
Prasanna Malaiyandi:
of the cloud into their cloud, and I think that's as secure as it can be.
Prasanna Malaiyandi:
Right.
Prasanna Malaiyandi:
And I think that it should be then stored in some type of immutable type offering.
Prasanna Malaiyandi:
Um, basically get it, get, get it out of the region for security against bad things
Prasanna Malaiyandi:
that might happen that aren't hackers.
Prasanna Malaiyandi:
And then get it out of the account to secure it against hackers and the.
Prasanna Malaiyandi:
If that costs you money, figure out a way to do that that costs
Prasanna Malaiyandi:
you as little as possible.
Prasanna Malaiyandi:
And, and, and I do think the companies that can back up, take the, the
Prasanna Malaiyandi:
incremental data and then maybe de-dupe it before they pull it out.
Prasanna Malaiyandi:
If they can do that, you can minimize the egress cost of moving it out.
Prasanna Malaiyandi:
Um, the, the other thing, um.
Prasanna Malaiyandi:
Yeah, so, so they didn't have it in a different account.
Prasanna Malaiyandi:
They didn't have object lock turned on.
Prasanna Malaiyandi:
Um,
Prasanna Malaiyandi:
Object lock did not exist back
Prasanna Malaiyandi:
then.
Prasanna Malaiyandi:
By the way.
Prasanna Malaiyandi:
W. Curtis Preston: what's that?
Prasanna Malaiyandi:
Object lock didn't exist back then, so.
Prasanna Malaiyandi:
W. Curtis Preston: Okay, so we won't, we won't blame them for that.
Prasanna Malaiyandi:
But, but, but since object lock exists now, we'll say this is what you should do.
Prasanna Malaiyandi:
Uh, but you know what did exist back then that they did not use multifactor
Prasanna Malaiyandi:
authentication?
Prasanna Malaiyandi:
Oh, for their access into their admin account or into their
Prasanna Malaiyandi:
W. Curtis Preston: Yeah.
Prasanna Malaiyandi:
Yeah.
Prasanna Malaiyandi:
So if somebody gains access to your admin keys and they're able to log in.
Prasanna Malaiyandi:
If you don't have MFA you, you have zero protection against someone
Prasanna Malaiyandi:
either stealing or accidentally, you know, inadvertently getting access
Prasanna Malaiyandi:
to administrative level keys and, uh.
Prasanna Malaiyandi:
I mean M-F-A-M-F-A-M-F-A mfa, I mean, how many, how many times
Prasanna Malaiyandi:
do we have to say it right?
Prasanna Malaiyandi:
Um, good password management, MFA and, uh, patch management.
Prasanna Malaiyandi:
We, we say this all the time.
Prasanna Malaiyandi:
If you just did those three things, you'd stop roughly 90% of attacks.
Prasanna Malaiyandi:
And in this case, if they had had MFA, this, uh, bad actor
Prasanna Malaiyandi:
would not have been able to
Prasanna Malaiyandi:
gain access.
Prasanna Malaiyandi:
Yeah, and and I think it's important to say
Prasanna Malaiyandi:
just because you use MFA and patch management and everything else.
Prasanna Malaiyandi:
Doesn't mean that you don't need backup, you still need backup because
Prasanna Malaiyandi:
that is how you are gonna recover from this, plus other issues as well.
Prasanna Malaiyandi:
The security side of things just sort of helps to protect you from
Prasanna Malaiyandi:
letting the hackers in to some extent.
Prasanna Malaiyandi:
Not gonna be a hundred percent foolproof, but hopefully, like
Prasanna Malaiyandi:
Curtis said, protects you.
Prasanna Malaiyandi:
And 80, 90% of the cases.
Prasanna Malaiyandi:
W. Curtis Preston: Yeah, to borrow from and totally abuse
Prasanna Malaiyandi:
a quote from Shakespeare.
Prasanna Malaiyandi:
There is more on heaven and earth than that is dreamt of.
Prasanna Malaiyandi:
In your philosophy, there are many, many ways that your
Prasanna Malaiyandi:
data can be attacked, deleted.
Prasanna Malaiyandi:
Set on fire exploded.
Prasanna Malaiyandi:
Sucked into a sinkhole.
Prasanna Malaiyandi:
There's so many different things that can happen to your data.
Prasanna Malaiyandi:
That's why you have backup.
Prasanna Malaiyandi:
And backup protects against all of them.
Prasanna Malaiyandi:
And, and we're saying backup and Dr and all of those things that come with it.
Prasanna Malaiyandi:
Um, but uh, the other thing that they also didn't do was this idea
Prasanna Malaiyandi:
of, um, you know, least privileged.
Prasanna Malaiyandi:
Do you
Prasanna Malaiyandi:
want to talk about that?
Prasanna Malaiyandi:
Yeah, so normally you do not want, in a company, you don't
Prasanna Malaiyandi:
want the intern to have the same level of access as your CEO or your IT admin.
Prasanna Malaiyandi:
And so you wanna be able to say, Hey, whatever access a person needs
Prasanna Malaiyandi:
to something, that's all they should have access to and nothing else.
Prasanna Malaiyandi:
And so you wanna have, make sure that you are focused on that and
Prasanna Malaiyandi:
don't just say, Hey everyone, you guys have admin credentials so you
Prasanna Malaiyandi:
can do anything and everything.
Prasanna Malaiyandi:
Because if one person who inadvertently gets compromised,
Prasanna Malaiyandi:
now everything is exposed.
Prasanna Malaiyandi:
So you wanna scope down their access to only what they need and that's it.
Prasanna Malaiyandi:
W. Curtis Preston: Exactly.
Prasanna Malaiyandi:
Now, when, when reading the, uh, articles about this, one of
Prasanna Malaiyandi:
the things that other people I.
Prasanna Malaiyandi:
Uh, dinged this company for, was that they didn't have an established
Prasanna Malaiyandi:
procedure for locking down the account.
Prasanna Malaiyandi:
And when I thought about that, I just found myself wondering.
Prasanna Malaiyandi:
I.
Prasanna Malaiyandi:
Huh?
Prasanna Malaiyandi:
How exactly would that happen?
Prasanna Malaiyandi:
And the best that I could find, you know, how, how would you do that?
Prasanna Malaiyandi:
And the best that I could find is that you would have a secondary account
Prasanna Malaiyandi:
that has access to this account that you then have a procedure to do things
Prasanna Malaiyandi:
like, um, disable, I think about.
Prasanna Malaiyandi:
You know, what, what is, what is the cloud equivalent to blocking somebody out?
Prasanna Malaiyandi:
And I, and I think the, the, the quickest way would be to
Prasanna Malaiyandi:
disable particular I am profiles.
Prasanna Malaiyandi:
Um, there, there was some, I, you know, and I'm not, I'm not
Prasanna Malaiyandi:
an expert on this, uh, I don't think you're an expert on this.
Prasanna Malaiyandi:
I would say talk to your cloud company.
Prasanna Malaiyandi:
Talk to your cloud provider.
Prasanna Malaiyandi:
Ask them, Hey, I am worried that one day my cloud entire environment
Prasanna Malaiyandi:
might become compromised.
Prasanna Malaiyandi:
How can I automate, basically locking everything out?
Prasanna Malaiyandi:
Right?
Prasanna Malaiyandi:
Once we determine that a hacker is in our environment, I would really like
Prasanna Malaiyandi:
a button that I can press from another account that shuts everything and down.
Prasanna Malaiyandi:
And everything like this can be automated and yes, that is a, uh,
Prasanna Malaiyandi:
you know what, what, what's the term?
Prasanna Malaiyandi:
The, you know, yeah.
Prasanna Malaiyandi:
The nuclear option.
Prasanna Malaiyandi:
That is the nuclear option.
Prasanna Malaiyandi:
But once you have an a hacker in your account to me that that
Prasanna Malaiyandi:
would be the proper option.
Prasanna Malaiyandi:
Shut everything down.
Prasanna Malaiyandi:
Uh, except for like, I would think create a new IAM profile that you can
Prasanna Malaiyandi:
use after you've done this, and then nuke everything that isn't that, um.
Prasanna Malaiyandi:
And, and that should be automated and that, and I don't think this
Prasanna Malaiyandi:
is something we normally talk
Prasanna Malaiyandi:
about.
Prasanna Malaiyandi:
I don't think it's something we talk about, but
Prasanna Malaiyandi:
it has to be out there somewhere.
Prasanna Malaiyandi:
I'm sure that AWS or Google or pick your favorite cloud provider,
Prasanna Malaiyandi:
they probably have a procedure.
Prasanna Malaiyandi:
I think the danger is you don't want it too automated because there's also the
Prasanna Malaiyandi:
risk that a hacker or someone else could trigger that and shut down your company.
Prasanna Malaiyandi:
So it's sort of one of those nuclear options.
Prasanna Malaiyandi:
So you don't wanna make it too easy.
Prasanna Malaiyandi:
But I'm sure that
Prasanna Malaiyandi:
W. Curtis Preston: Yeah.
Prasanna Malaiyandi:
a.
Prasanna Malaiyandi:
Yeah.
Prasanna Malaiyandi:
W. Curtis Preston: use the nuclear option to.
Prasanna Malaiyandi:
This is one of those where this, this is the nuclear button, and so you just,
Prasanna Malaiyandi:
maybe you have an account that just does this and that account is like
Prasanna Malaiyandi:
completely separate from everything else.
Prasanna Malaiyandi:
Like we, we talk about having an account.
Prasanna Malaiyandi:
That is the backup account, right?
Prasanna Malaiyandi:
That is used for backups and no one ever logs into this account.
Prasanna Malaiyandi:
And you create it in such a way that if anyone ever does log in,
Prasanna Malaiyandi:
does log into the account, it sets off alerts everywhere and Right.
Prasanna Malaiyandi:
It, it can be like a honeypot account, but this account, um, yeah,
Prasanna Malaiyandi:
I'm not sure how to do, again, I'm not an expert in this, but I would,
Prasanna Malaiyandi:
I would create a separate account.
Prasanna Malaiyandi:
I would make that account as secure as humanly possible.
Prasanna Malaiyandi:
Again, ask your cloud provider how to do that, uh, the, the best way to do that.
Prasanna Malaiyandi:
But I just noticed that everybody said that almost everybody, they're like, they
Prasanna Malaiyandi:
should have had procedures for what to do in this situation, and, and they didn't
Prasanna Malaiyandi:
have them.
Prasanna Malaiyandi:
I think
Prasanna Malaiyandi:
perhaps,
Prasanna Malaiyandi:
right?
Prasanna Malaiyandi:
W. Curtis Preston: what's that?
Prasanna Malaiyandi:
At least manual procedures, right
Prasanna Malaiyandi:
W. Curtis Preston: At least manual procedures, right?
Prasanna Malaiyandi:
Because I don't think that changing the passwords on IM profiles was
Prasanna Malaiyandi:
the, the quickest way to do that.
Prasanna Malaiyandi:
Right?
Prasanna Malaiyandi:
I think the thing they should have focused on perhaps was kicking out,
Prasanna Malaiyandi:
currently logged in sessions, uh.
Prasanna Malaiyandi:
Yeah, I don't know.
Prasanna Malaiyandi:
And the problem is, it's like, how, how do you, you, you have to,
Prasanna Malaiyandi:
you have to build your incident response around your environment.
Prasanna Malaiyandi:
And one of the things that they could have done is maintain an inventory of,
Prasanna Malaiyandi:
um, basically privileged, super privileged accounts and look and see if there
Prasanna Malaiyandi:
were any new ones.
Prasanna Malaiyandi:
Well, that's the thing I was going to mention is they
Prasanna Malaiyandi:
should at least have had monitoring.
Prasanna Malaiyandi:
When someone adds a super privileged user, they should have been flagged
Prasanna Malaiyandi:
about that immediately, right?
Prasanna Malaiyandi:
Because that's not a common occurrence.
Prasanna Malaiyandi:
W. Curtis Preston: Right.
Prasanna Malaiyandi:
And so monitoring, alerting is also looks like something
Prasanna Malaiyandi:
that people should be doing to catch these sort of issues as well.
Prasanna Malaiyandi:
W. Curtis Preston: Yeah, and, and, and I'd love, by the way, I'd love
Prasanna Malaiyandi:
other suggestions from listeners.
Prasanna Malaiyandi:
I would love to hear from you.
Prasanna Malaiyandi:
If you go to backup wrap up.com, there's actually a button on there
Prasanna Malaiyandi:
that you can leave voicemails.
Prasanna Malaiyandi:
You can send us notes and, uh, you know, you know, if you'd like, we can actually
Prasanna Malaiyandi:
even play your response on the air.
Prasanna Malaiyandi:
I would love to hear better suggestions than we have from a security perspective
Prasanna Malaiyandi:
because I'm, you know, I'm Mr.
Prasanna Malaiyandi:
Back.
Prasanna Malaiyandi:
I'm, I'm not Mr.
Prasanna Malaiyandi:
Security.
Prasanna Malaiyandi:
I, I play, I play a security on tv.
Prasanna Malaiyandi:
Um, the, uh, but, but, but, uh, the summary statement.
Prasanna Malaiyandi:
From a backup perspective, if they had simply followed the 3, 2, 1 rule, if they
Prasanna Malaiyandi:
had made another copy of their backups in another account, in another location,
Prasanna Malaiyandi:
if they had used a cloud provider to do this for them so that then a copy of all
Prasanna Malaiyandi:
their data was stored in, in a completely different company, if they had done any
Prasanna Malaiyandi:
of those things, they would've at least had a copy so that once they got on the
Prasanna Malaiyandi:
other side of the attack, they could have.
Prasanna Malaiyandi:
Then recovered all the data because that's the, the true disaster here.
Prasanna Malaiyandi:
As that once they've been attacked and once the attacker gained access
Prasanna Malaiyandi:
to their account, they were able to delete all their data, both
Prasanna Malaiyandi:
their primary and their backups
Prasanna Malaiyandi:
So I know we've been focused solely on code space as the
Prasanna Malaiyandi:
company, but I think there's also blame to be placed on those 200 companies who
Prasanna Malaiyandi:
were using Code Spaces for not also having a backup of their data and relying solely
Prasanna Malaiyandi:
on code spaces as their service provider.
Prasanna Malaiyandi:
W. Curtis Preston: You know, that is an interesting, we, you know, we
Prasanna Malaiyandi:
tend to focus here on the fact that it was a provider, but what this
Prasanna Malaiyandi:
really was, was a SaaS provider.
Prasanna Malaiyandi:
Yep.
Prasanna Malaiyandi:
W. Curtis Preston: Right.
Prasanna Malaiyandi:
Yep.
Prasanna Malaiyandi:
W. Curtis Preston: Yeah.
Prasanna Malaiyandi:
So we don't know what happened to those other companies.
Prasanna Malaiyandi:
And so this is, this is a double lesson, right?
Prasanna Malaiyandi:
If you're, if you're running in the cloud, make sure you've got
Prasanna Malaiyandi:
a backup of that, of that data.
Prasanna Malaiyandi:
If you're using a SaaS provider, make sure you have another copy of
Prasanna Malaiyandi:
the data that you're putting in that SaaS provider, because that would be
Prasanna Malaiyandi:
another way for at least the, the, um, the, the, the thing that's difficult.
Prasanna Malaiyandi:
Here, again, I agree with you, the thing that's.
Prasanna Malaiyandi:
The, the difference here is that unlike many of the SaaS providers, this company
Prasanna Malaiyandi:
specifically said, Hey, we got it.
Prasanna Malaiyandi:
We got your backups, this data, and it's tested and it's all this stuff, right?
Prasanna Malaiyandi:
Um, the, um, you know, I just had a thought.
Prasanna Malaiyandi:
Um, if we go to LinkedIn and we search for code spaces.
Prasanna Malaiyandi:
Find people that used to work in code spaces.
Prasanna Malaiyandi:
Oh, that would be, I wish, I wish we could talk to somebody
Prasanna Malaiyandi:
that was involved in this, but
Prasanna Malaiyandi:
uh, I think we can
Prasanna Malaiyandi:
I'm sure there are probably NDAs.
Prasanna Malaiyandi:
W. Curtis Preston: What's that?
Prasanna Malaiyandi:
Prasanna Malaiyandi: There's probably NDAs.
Prasanna Malaiyandi:
W. Curtis Preston: Oh, they're probably NDA.
Prasanna Malaiyandi:
Yeah.
Prasanna Malaiyandi:
All right.
Prasanna Malaiyandi:
Well back up the cloud, I told you so.
Prasanna Malaiyandi:
Any final thoughts for you, persona?
Prasanna Malaiyandi:
No, I totally agree with that, and I like this because I
Prasanna Malaiyandi:
know we bring up code spaces a lot, so I think that hopefully our listeners now
Prasanna Malaiyandi:
understand why we talk about it and what they should not be doing, and why we
Prasanna Malaiyandi:
harp so much on things like the 3, 2, 1 rule on MFA, because you don't want your
Prasanna Malaiyandi:
company to have to shut its doors because they were unable to recover their data.
Prasanna Malaiyandi:
W. Curtis Preston: Yeah.
Prasanna Malaiyandi:
And sadly, this will not be the last company that.
Prasanna Malaiyandi:
You know, basically ceased to exist, uh, because they didn't
Prasanna Malaiyandi:
properly back up their data.
Prasanna Malaiyandi:
Yeah,
Prasanna Malaiyandi:
W. Curtis Preston: All right.
Prasanna Malaiyandi:
Well, uh, thanks for, uh, joining me persona, as always.
Prasanna Malaiyandi:
anytime and looking forward to see your analytics on your car.
Prasanna Malaiyandi:
W. Curtis Preston: I will see what I can do and I will also thank our
Prasanna Malaiyandi:
listeners, we be nothing without you.
Prasanna Malaiyandi:
Thanks for listening.
Prasanna Malaiyandi:
And be sure to subscribe so you don't miss an episode.
Prasanna Malaiyandi:
That is a wrap.
Listen On
