Get Ready Before You Get Got: Ransomware Response Planning

ATR2500x-USB Microphone & Logitech BRIO-2: If you're like the majority of people

I've talked to, you don't really have an incident response plan for ransomware.

That means you'll be flying by the seat of your pants when you get hit.

The good news is I have just the episode for you.

It's an extremely popular episode from earlier this year where we talk with

Melissa Palmer about what to put in your response plan and how to build it.

It was so popular that it makes the perfect episode for our winter break.

Even if you've heard it before, it's worth a second.

Listen.

If this is your first time listening to us.

Hi, I'm W Curtis Preston AKA Mr.

Backup.

And I've dedicated my career of over three decades to helping those of

you that have the job that I had when I first started the backup person.

This podcast is just for you.

We turned backup admins into cyber recovery heroes.

This is the backup wrap up.

ATR2500x-USB Microphone-1: Welcome to the show.

W. Curtis Preston: I'm your host, W Curtis Preston, aka a Mr.

Backup, and I have with me my super expensive vacation planner coordinator.

How's it going?

Prasanna

I'm doing well, Curtis, how are things going?

Are you excited?

We are.

I we're having technical difficulties, as you could tell.

We're trying to keep this real, but yes, doing this for the fifth ta,

fifth time, it's a little hard, but

W. Curtis Preston: I am excited, um, uh, and my wife is starting to get excited.

I started showing her some pictures a while ago and she's

been like downplaying it.

Like she doesn't want to get excited.

She wants to be sort of, Excited, but I needed her to prep for the vacation

because this is, so this is, we're going to the Maldives, uh, which for

those that don't know, is a series of islands off the southern coast of India.

And, um, and, and I'm on one of those islands and, and it's a tiny island that

literally we could walk from one end to the other in probably about 10 minutes.

Um, and.

We're staying in one of those things over the water,

Prasanna Malaiyandi: Oh, the Villas over the.

W. Curtis Preston: villas over the water with our, we have our own

pool, and then right on the other side of the pool is the ocean.

Um, I mean, it's

really, really cool.

Can I stow away in your luggage

W. Curtis Preston: Yeah, I mean, it looks really cool.

uh, we're very excited.

We're having our, a repeat guest and, um, we, we had her on, uh, a few

weeks ago and we got talking about ransomware, one of our favorite topics.

And we, we, we got into this phase where it was like, you know what?

That, that is a great conversation, but there's no way we could, we could

do it justice on that recording.

So it was, Hey, we're gonna have her come back.

And, uh, she is, uh, she's been in the industry for quite a while and she's been

specializing in, uh, she's done VMware.

Uh, she did.

Now she's, she's working, uh, Starting to specialize in security and ransomware.

So we're, uh, and she's the author of the vmiss.net blog, and we are

excited to have her on the podcast.

Again, Melissa Palmer, aka @vmiss.

How's it going?

you for

having me back.

It's going good.

I was surprised that you were like, Ooh, I'll

come back on the podcast after

yeah, that was, of course, when I come back

Well, thank you for

scare.

It takes a lot more.

You said it.

I've been in around this industry for a while.

It takes a lot more than that to scare me away after all these years.

And Curtis, I think, uh, now might be a good time

to put out our normal disclaimer.

W. Curtis Preston: Yeah, prasanna and I work for different companies.

Uh, he works for Zoom.

I work for Druva.

This is not a podcast of either company and the opinions that you hear are ours.

Also, be sure to rate us at, uh, Uh, rate this podcast.com/restore

and, um, if you wanna join the conversation, reach out to me.

By the way, I, I gotta give a bunch of ways cuz I, I got some

complaints and people say, well, I don't use Twitter anymore.

So how you give your Twitter address.

So my LinkedIn is, you know, linkedin.com/ally/mr.

Backup.

Uh, you can find me there.

Uh, you can find me on Facebook.

I'm on Facebook, Facebook Messenger, but my email is, uh, w Curtis Preston.

Uh, my Facebook is w Curtis Preston.

I'm pretty easy to find if you're looking for me.

Um, and reach out to me and we'll get you in on the, on the conversation.

Yeah.

Um, the, um, this, this thing of responding to a ransomware attack,

this, this is something I've been spending a lot of time on lately, uh,

because I've been, I'm, I'm working on writing my next book, which will be

about responding to ransomware attacks.

You know, one of the things that you said in the pre-call was that if, if

the first time you're thinking about responding to a ransomware attack is

after you got a ransomware attack,

Um,

W. Curtis Preston: it's not so good.

Right.

, there's a lot of, yeah.

In fact, when I was looking at the, sort of the outline that I've been

working on for the book, most of the outline is the first half , right?

Everything that you need to do before, right.

Um,

that's, it's like you can't just talk about ransomware

recovery, Right, Like, it, it, it's a hard topic to talk about because

you're like, there's all this other stuff that if you haven't done it, guess what?

You are not gonna be able to recover.

So we can't just talk about recovering.

It doesn't work that way.

W. Curtis Preston: Right.

It's sort of like I, I've made the joke, uh, a few times probably on

the pod where I've said, listen, you know, I've been in the backup

industry, you know, a long time.

I, I've decided to give up backups and I'm just gonna skip straight to restores.

Right?

You can't really , you can't really do that.

Just like I've also said that if I'd have known how great grandkids were,

I would've just gone straight to them.

Um, but not, not really

Prasanna Malaiyandi: It's not how it works.

Yeah.

W. Curtis Preston: Yeah.

It is a really

good analogy though.

It really

W. Curtis Preston: Yeah, it is, it is.

By the way, you want a little, little sad thing.

So my granddaughter and her mother and, and her husband,

uh, are, this is their last day

Oh, I was gonna ask you about

W. Curtis Preston: been living here for a while, and they're moving out tomorrow.

So,

Hmm.

W. Curtis Preston: little sad moment.

Little sad moment.

No.

W. Curtis Preston: Um, but, uh, anyway, so, you know, sorry to bring that down.

So let's talk about what, what do you think, Melissa?

Let, let's sort of go through those things that we really needed to have done before.

Uh, well, lemme, lemme try to set the stage a little bit.

Like, does everybody remember like, the disaster recovery tests, like

back in the day, you go to the colo, you got the checkbook, the.

the

Clipboard you make, the checkbox isn't like, I don't know, you play

doom for a while and eat some food.

Someone restores a server and it's like,

well, it kind of worked and we're good.

Yeah,

that's how old I am.

Um, so and then you're like, oh, it kind of worked.

So we passed our d r test, but we can't actually recover.

Right?

So what you need to do is actually do a ransomware recovery test where

you actually recover everything.

There's a novel concept, and when you do that, you're gonna figure out all the.

but you didn't do cuz it's not gonna work or something's not gonna

whatever.

But it, it's, you know, talking from the backup lens cuz I was

at Veeam for quite some time.

Um, something I talked a lot about with Veeam customers was, you know, trying to

understand the whole recovery process.

Cuz if I'm the backup admin and we get ransomware, I don't just

go start restoring stuff all over.

Like that's not what happens.

It's not like, oh no, right somewhere tech, let me start restoring servers.

We'll

be back online in 20

minutes.

Like it doesn't work that way.

, you have to figure out what happened.

Before you can start restoring, you have to figure out what happened.

You have to figure out if the threat actors are still around.

You have to understand what was impacted.

I have heard a lot of people say, um, oh, well, we treat ransomware

different and we just recover in place.

So we're good to go.

And I'll go

back to the little VMware.

Yeah, I'll go back to the VMware ransomware thing.

Well, if your VMware environment is ransomware, guess what?

You're not recovering in place cuz there's nowhere to recover to.

Uh, so it's understanding all those different things.

You need to have some kind of understanding of what happened

before you can recover.

And that is generally driven by the incident response process, which is

gonna be driven by the security team.

So again, if you haven't talked to the security team before,

ransomware has attacked you.

You're gonna have a bad time.

Or vice versa, if the security team hasn't talked to you about

how backup integrates into that process.

that's really scary.

That's really, That's really, that's really, disturbing.

Those are actually

really even, I think that's

scarier.

W. Curtis Preston: I think it's, it's a, it's a combination, right?

Well, you know, uh, yesterday, I think that was yesterday, we recorded

a, a great podcast, uh, by the way, with Tom from Gestalt, um, that,

that, uh, net, uh @networkingnerd.

Yeah.

and he, uh, we were talking a lot about the networking side of the, the

response, right?

Shutting down things.

Um, and, and using a combination of technologies, many of which are easier

to use if you, if you set them up front.

Right.

And, uh, talking about things like VLANs and, uh, you know, like one of

the things we talked about was having a VLAN for all of your desktops and

laptops, so that if you want to stop everybody from doing anything, you

just shut off those VLANs and boom.

Um, there, you know, instead of having to notify 5,000 users, hey, stop doing

anything, you just shut off their network.

So they can't, they can't do anything.

And then if stuff is still happening, , um, well, it's not the users,

right?

It's, it's malware,

right?

back to segmentation.

W. Curtis Preston: know, yeah, the, the network segmentation and the, the

security part, I think, um, What, what, what role do you think the, I'll ask you

what you think before I say what I think

So what role do you think cyber insurance companies and then the, the companies

that they can put you in touch with?

The, the

Cyber insurance is becoming more and more interesting

cuz it gets to the point where they hand you the list of things you

need to do before they'll issue your policy and guess what you're

gonna probably be able to cover anyway.

Um, but a big part of, I've seen in a lot of policies lately is

having, um, basically an instant response from on retainer ready to go

as part of your policy.

And

I think that is invaluable.

I.

, everybody should have some kinda relationship with an IR firm

if you can't do it in house.

And uh, even if you can, right?

Sometimes you do still need that outside perspective.

I know a lot of larger orgs are like, no, no, we do our own ir, well, you do

your own ir, but you're not dealing with ransomware every day and these people are

so you might want a little bit of help.

W. Curtis Preston: Yeah.

Yeah.

Um, you know, um, I hate to do it, but a another, another movie reference.

I just saw the , the movie plane, and you know, the plane goes down in the

middle of nowhere and they brought in the guy, they brought in the incident

response guy basically once he showed up.

Right.

See, there's a movie reference for everything,

I haven't, I

can't tell you the last movie I've watched.

I really can't.

I don't

W. Curtis Preston: I can, I can, I can pull up my app, uh,

cuz I have the Regal Unlimited.

tell you the last thing I watched.

I can't tell you the last movie I watched, cuz I don't remember.

W. Curtis Preston: I, I, yeah, I, I saw like three this week.

So

in, in the theaters

so back to the cyber insurance from movies.

Uh,

I, yes.

Yeah.

No, but, but, but I think, well, this is one of the points that I remember

because remember when Tony came on from SPECT Logic, Curtis, and he was like,

oh my God, they got hit with ransomware.

And he's like, just the previous month they had signed up for cyber insurance.

They had an IR firm come in, give them sort of the list of, Hey, here's

everything you need to do to help.

Right.

And he was like, that was probably the most valuable thing of that sort of

cyber insurance policy was having the experts who could walk you through.

W. Curtis Preston: And it, and it wasn't even like he, he was just

lucky enough to have already, you know, contracted with them.

Right.

But the best I think would be to

, well, not that you would know this, but to do it not a month in advance, but

obviously way in

right.

W. Curtis Preston: to get, and to give you some time to work with the incident

response team and to make sure that you are doing the things that they want

but that's like that's like the problem, right?

Like it's not, if it's when, and you don't know when.

It could be tomorrow, it could be next week, it could be next month.

It could be next year.

Like you don't

W. Curtis Preston: It could have been three weeks ago.

and you just haven't realized it yet, right?

W. Curtis Preston: Yeah.

Do it today.

Yeah.

my favorite.

W. Curtis Preston: Yeah.

Uh, so, which is why it doesn't matter when you invent a time machine.

You know, I have bad news to you.

W. Curtis Preston: What

I haven't invented a time machine

because there are certain

points I've always promised to myself.

If I invented the time machine, I would go back to this point and tell

myself I invented the time machine.

And if that hasn't happened, I haven't invented it because

time is not linear, right?

So I haven't invented a time machine.

I'm very upset about that.

W. Curtis Preston: Yeah.

Me neither.

Um, but, um, well, it's been a weird, it's been, we've been jumping in and out

of the topic here on this podcast, but,

Incident response.

W. Curtis Preston: yeah.

So we, we, we get the cyber insurance folks because I

think in the, in the initial.

Ransomware phase, what people thought of cyber insurance was just a

company to pay their ransom for you, and that they're definitely saying

they're not interested in it anymore.

Yeah.

And there's

more costs beyond the ransom, right?

So

you paid the ransom, but what about everything else?

Um, that's the thing.

And policies have changed over time, like, back in the day a couple years ago, right?

Like before the pandemic, uh, it was like easy to get cyber insurance.

Like, oh yeah, I'll take a cyber insurance policy for 5 million, please, whatever.

And

now it's hard.

And if you do actually use your, I've seen a lot of cases where if you

actually use the insurance policy,

guess what?

They don't necessarily drop you, but guess what Your deductible co becomes.

What they paid for your last ransomware attack, right?

So if I had to pay 2.5 million, guess what?

I now have a 2.5 million deductible for my next attack because

let's face it.

We get IR in, right?

We figured out what happened, we have to recovered, and then there's a whole

stage where we have to do a postmortem, figure out how they got in, if they're

still in and close up the gaps.

That doesn't always happen cuz people are so, like, ohms are back, we're good to go.

Happy day, happy

day.

And they get hit again

because they never fixed the way they got in in the first place.

W. Curtis Preston: What, what do you think about the idea of.

And again, this would be driven by management.

And you know, a lot of times, like you said, management isn't necessarily

at that moment thinking about the the best way to do something.

They just wanna do the fastest way to do something.

right?

So another thing I've been looking into is the idea of wouldn't the best

practice to be to figure out how they got in before you do the recovery,

before you turn everything back on.

Yeah.

And that, that's where the IR firms come in, because.

they'll kind of get in and they'll be able to do that.

They'll be able to say like, you guys are so messed up.

You didn't have any logging unabled anywhere.

Like we, we can't tell right now.

Right?

It really depends on what happens in that first phase.

Um,

W. Curtis Preston: Yeah.

and it comes back to kind of getting ready for the

attack and what kind of security practice you have in some places.

Yeah.

We could see, people can figure out, uh, throw in a tool and say, yeah, guess what?

They came in here.

We know we're good to go.

Other times they might not find it just

because there was never.

they came in.

They went out before you even knew

or nothing was

W. Curtis Preston: under

or we didn't, you know, we didn't have logging

on or whatever.

Or they turned something off or,

W. Curtis Preston: Logging is a beautiful thing and, and also

a system to get those logs off

yeah, that's what

people like

forget about, like who cares about the logs, like whatever their logs.

No, you're, you're going to care about the

logs someday, I promise you.

W. Curtis Preston: Yeah, I mean, even if it's something as simple of making

sure that the logs are represented as text somewhere, that is then

backed up by the backup system so that you can restore all of them.

That's basic, but there are systems that you can buy that

will just automatically, uh,

exfiltrate all of those logs for you.

Yeah.

Yeah.

I wanna go back to a point you made earlier, Melissa, about

sort of, okay, how do you make sure that you fix the things that broke so everyone

isn't like, Hey, my VMs are back up.

I don't need to worry about these things anymore.

Have you heard any cases where, I know sometimes executives have

sort of financial liability, right?

I've heard of that trend, right?

Like your guess what your bonus is tied to if you get ransomware or not, and how you.

And stuff like that, that's starting to happen in some places.

Um, but a lot of it comes down to maybe the processes were

never clearly defined upfront.

Right.

And that's where a lot of the cyber insurance stuff can

actually come in and help.

Well, they'll be like, you need to show us your response process.

And they'll be like, here you go.

And they'll be like, okay, so where's the rest of it?

Or something like that, right?

Like, what, what

happened?

Like, this is it.

Like here's

a page.

Like it's not gonna work.

Um, and again, it comes back to.

the old school DR test.

Like there needs to be ransomware recovery tests and postmortems of

that ransomware recovery test, right?

Like y'all need to get in room, figure out what worked, what didn't

work.

W. Curtis Preston: Having done the old school DR test, I'm curious as to how

they do a ransomware recovery test.

Because one of the hardest parts of a ransomware recovery is that the

attacker is there is still attacking, like with a dr, you just say,

okay, those six systems are dead.

So, yeah.

So

here's where it

gets complicated.

You need to test multiple types of recoveries, right?

So maybe I'm recovering, please.

I, I can't.

, I will vomit in my mouth if I say maybe I'm recovering in place.

I can't even like say that.

So we're not gonna say that, but like maybe I'm going to my second site.

Maybe I'm going to a warm site.

Maybe I'm going to a hot site.

Maybe I'm going to a public cloud.

Maybe I'm going to a VMware cloud.

You gotta test all those, right?

Because

you don't know where you're going until that incident response

phase starts, especially when law enforcement gets involved, right?

So let's say stuff's really bad, the FBI comes, and guess what?

We are quarantining your whole data center while we investigate.

Then what do you do?

Yeah.

You're down for business, otherwise,

do?

No, you go to public cloud, you go to um, a service provider, you go someplace else.

So you have to have all that ironed out ahead of time.

You have to know that there's different considerations for recovery from

ransomware attack than a traditional

disaster.

So I guess, you know, from a traditional disaster, like what if

the zombies eat both data centers,

right?

Then you would still need to go to the

but people probably aren't thinking about that though, right?

The fact that, hey, maybe the F B I will come quarantine, right?

Do you have your backups offsite?

Do you have it in someplace that you can bring it up?

And like you mentioned earlier, Melissa, it's like things you should plan for ahead

of time before you get to the point where you are trying to recover from ransomware.

Exactly.

And again, unless an organization, so I have a couple of examples

of, I don't wanna say Dr.

Done wrong, but uh, I worked for an uh, company when I was

an intern on Wall Street and everything was in New York City.

and nine 11 happened and they were a block from the World Trade Center.

That's what they couldn't, they couldn't do anything like they were done.

Right.

Like they were just done.

So they like rebuilt their systems in a hotel room someplace.

Right.

And that kicked off a huge project to say, we actually need a second data

center and it needs to be not around here.

Right.

Um, I'm also on the east coast, right?

So New York, hurricane Sandy, we had this hurricane roll through.

And again, like the data centers are like 20 miles from each other.

Guess.

, they both tanked.

Um, so things like that.

So until an organization actually has something happen to them, it's really,

and here's the issue, the, the, the difference between disaster recovery

and ransomware recovery, when we talk about it, traditional disaster

recovery stuff, until it happens, it's easy to accept the risk, right?

Well, you know what?

It's cheaper for us to just like recover from this disaster and be down for

two weeks than it is to actually put everything into place where we build a

second site, yada, yada, yada, yada, et.

that's because the risk is so low, right?

And there's all kinds of equations for

this in, you know, cybersecurity and stuff like that.

But when you change it to ransomware, the risk is going to, it's going to

happen like a probability of one.

It

will happen.

Um, and that's what people don't understand.

Like this is going to happen.

It's not like you can say like, well, you know, we haven't had a hundred

years storm ever, so we'll be fine.

Um, it's different like that.

And a lot of people, I've actually seen a huge uptick in people getting.

I don't think a lot of people are where they need to be.

Um, but I think as people get ready and it gets harder and harder to attack

people because they've put like some semblance of security in it, right?

You're gonna go for the low-hanging fruit, you're gonna see the people

who aren't ready get hit harder and you're just gonna see more and more

attacks and the threat actors are gonna have to get more creative.

So here's a question for you.

Normally when we think about backup and recovery, right, it's always

about restoring your data or your application because there might be

a hardware failure, an application fault, user error, et cetera.

Sometimes people talk about ransomware in the same context as

disaster recovery and sort of those

Ransomware is a disaster.

I

but, but here's the question though, Melissa

is, Like you had just mentioned, it's not the same as a flood or a

hurricane or something like that.

And so are we kind of pushing ourselves and kind of giving people

the false impression that it is similar to those other disasters

and things that they shouldn't worry about versus we should be treating

it similar to like an application failure or user failure and treating it

similar.

It's like more towards that side of the spectrum than this side.

and you know, that all falls under DR

anyway, like hardware failure

and all that kind of stuff.

Um, and again, in a lot of those cases, it's easy to say, well, you know what?

I don't really want a second site.

It's

just cheaper to deal with the hardware.

It'll take we'll rush order.

I was in a situation at a company, we'll just rush order at a new array from

E M C that will solve our problems.

Like that was the plan and that happened.

Um, so crazy stuff like that.

But the problem, why I like to make the analogy so much is the problem

is when you tell someone that you have to get ready to recover from

ransomware, they're just like, I don't.

what to do.

You have to put it in some context that kind of makes sense.

I mean, disaster recovery is definitely like not sexy, even though

I've done it most in my career.

Um, but it's something that everybody has an inkling about at least, right?

Everybody kind of knows that there is usually a DR test once or twice or year a

minimum.

Um, so it's a way, it's a starting

point, right?

It's not your final destination, but it's a starting

point.

It's a.

place to start context.

Maybe you have some playbook, some processes that we can leverage to go build

on top of that and say, okay, so how do we make sure that we can recover now under

any

W. Curtis Preston: I like to, I like to say that it's a subset, right?

A DR is a subset of a ransomware recovery, but there's so much else, right?

And the big thing, the but, and I think you said it already, Prasanna, but the

big thing to me, the difference between a DR and a ransomware attack, um, is

that the, the disaster isn't, Right.

You're, you're still right

that the disaster never

W. Curtis Preston: a flood is gone, you're like, okay, all

these servers got wiped out.

So those are the

because the threat is still there.

Just because you

recovered from the ransomware attacked doesn't mean they're not

gonna hit you again, or someone else

isn't gonna hit

W. Curtis Preston: Right.

Well, and, and how do you even know,

um, You know, like when you, when when a hurricane wipes out a data

center, you're like, okay, those are the servers we need to restore.

But how do, when you walk into your data center and there's a

ransomware attack going on, how do you even know which servers have

been affected or not affected?

Right.

That's, that is a big part of it.

Yeah, and I guess the other thing is even like you

might see the active infection, like things are being encrypted, et cetera,

but it might just be lying silently.

Right.

We've talked about dwell time in the past, right.

Where it's

chill.

They just chill in there for a while.

Like, who knows?

Um, I, I can't remember off the top of my head, but I remember reading like a big

name breach or something like that, or a big name attack, and they said they were

in the network for like six months or

I think Solar Winds was like

was it?

I don't remember.

But I remember reading a couple of them where they've been in

there a significant period of time and who knows what they're doing

there, right?

Like who knows

unless you catch them.

So it's about

W. Curtis Preston: yeah.

The meantime is something like 60 days actually is what I, what I read.

Um,

be the worst ransomware person.

I'd be like, let's go, let's go.

It's like, no, you're not supposed to do that.

You gotta

take your time and traverse

through the network and get ad.

I'd be like, let's go encrypt VMware.

Let's go.

I'd be caught so fast.

Or maybe I wouldn't, maybe I.

You're only caught if someone's monitoring and watching.

Right Melissa?

Right.

And you need

to be looking for the right things.

W. Curtis Preston: Yeah.

As soon as you encrypt a, a vm, uh, you're gonna set off alarm or two.

Um, but I, I think you encrypt, I think you encrypt a lot of

files that no one's looking at.

Right.

But the moment you start

Once you hit the the thing,

the only thing is you'll hit.

You'll hopefully you'll be caught as soon as you start encrypting the VMs.

You do them all at once, so it doesn't matter.

W. Curtis Preston: Yeah.

Right.

Cuz it's,

I got all of 'em.

It doesn't matter that you caught me doing the first one, I did them all.

Um, but yeah, so generally they're in their wreaking havoc, steal maybe

exfiltrating data, doing some stuff before they go encryption habit.

Or maybe like, I've heard cases recently where they don't even

bother, like encrypting stuff.

They're just stealing data at this point and

be like, by the way, look what we have.

Is that easier by the way, to steal data?

Because it seems that you can sort of fly under the radar if you just steal

data because people will probably, maybe they notice, maybe they don't,

but it's not as obvious as, say,

It

is definitely not as obvious as encrypting stuff, I'm like

this weird monitoring nerd too.

I had like this monitoring fetish at Veeam.

It was very strange.

Um, so like, I would like really hone in on like what to look

for to catch that too, right?

But not everybody is crazy like me.

Um,

W. Curtis Preston: I think, I think,

yeah, I do.

To answer your question, Prasanna, I do think that exfiltration as an overall

process is easier in that if you can get any data out that there's a, there's a

much higher chance that they will respond.

That they will pay the ransom.

Right?

Because backups aren't gonna help.

I'm looking at my black hat over there.

I'm wondering if I should like, put it on for this discussion or something.

Um,

like you would probably like see like, all right, like if I'm a bad person,

I'm not a bad person, I'm a good person.

Um, like they start small,

right?

They grab a file here and there and they see if they

if anyone notices.

this, grab that, right?

Like, you don't go and just be like, oh look, here's the final.

25 million gigabytes of MP3s.

I'm gonna take it all at once.

No, they're like picky and choosy.

They try to find the sensitive data.

They take a little bit here and there.

Maybe they only need to grab a couple spreadsheets.

Right?

It's not like,

I think there's this misnomer that like they get in there and I'm just gonna

start downloading massive chunks of

data.

W. Curtis Preston: well,

that's the whole point of

so you could exfiltrate a vm, just like

download the vmd K and be like,

W. Curtis Preston: yeah, exactly.

ad.

Have a

nice life

W. Curtis Preston: that's that whole phase of the, um, the initial phase of an attack

is trying to expand out, seeing what you can find out, seeing if you can find

a spreadsheet called customer database

You know?

Right.

W. Curtis Preston: xls , right.

Um,

or like.

you might not bother encrypting everything, but if you

can't find much, you say, all right, I'll steal some stuff and tell 'em I

have some files, but I won't tell them what I'll hope that'll make them pay.

And I'll just

go, you know, encrypt some stuff while.

Which is more illegal?

Is one more legal than the other?

I think they both are pretty bad,

is one more illegal than the other?

W. Curtis Preston: Well, they're both extortion.

Yeah,

W. Curtis Preston: the act,

The act

but if you're actually exfiltrating, you're stealing it.

W. Curtis Preston: yeah.

That's gonna depend on where this happens.

Uh, whether or not exfiltrating the data is a different crime.

And damaging the data.

Um, but, uh, but in the, the extortion happens on both sides, right?

And that's

definitely illegal in

that

W. Curtis Preston: pretty much

every jurisdiction

legal kids.

Yeah, so we talked about, so we talked

about incident response.

You've now been hit by a ransomware attack.

in, then let's just take VMware environments, right?

So what do you see people doing like, or what are things that they

should be doing that they're not?

Like, how do they even approach

Yeah, so he,

VMware environment gets encrypted Now, what

Um, to me it's trash.

I would throw it away and start over, like, I'm not even joking.

Throw it

W. Curtis Preston: No, not

and, and, and, and how much?

And and how much would you, when you say throw it away, are you talking about

throwing away the virtual machines, throwing away the ESXi servers, the.

the host, wipe the storage array, wipe it all and start over.

Um, and, and here's the thing, right?

So like, you know, I, I like it.

I have this weird side of me that also does like weird blogging stuff, right?

And like, I like SEO and stuff like that.

And even my career at Veeam people are like, how do I back up my VMware host?

you don't, they're like, what do you mean?

I'm like, you don't, um, you automate the build process

and the configuration, right?

You don't actually back up your host and restore it.

It's, you

You just rebuild

thing.

It's a clean install and you configure it.

Um, so that's what people need to be testing to is how I would

actually recover is almost misnomer.

Cuz Prasannally I would trash it.

Um, how do I re rapidly rebuild a VMware environment?

And that's something.

People don't do every day, right?

Like that stuff runs like you might have not even reinstalled.

You could have just been

upgrading for the last like 10 years and like, whatever, probably not 10, probably

four or five years, you'll get a new host.

I don't know.

It depends.

Um, so that's something that people don't practice and don't do.

Um, and you can actually do that all.

for the most part, um, in a nested virtualization environment.

Get all your processes down stuff.

So it's a pretty low co I mean, you should test on your physical hardware

at some point for any drivers and stuff, but it's actually a relatively low

cost and effort thing to figure out.

It's not rocket science.

But when you do this testing, wouldn't you also want to

involve, say like your networking team,

Yes, you would wanna, any of

this testing, you wanna involve anybody?

Everybody, right?

Everybody should be involved in this.

everybody.

And that's I think, one of the biggest problems we see that they're not,

W. Curtis Preston: So when you say,

They're like, I don't have time to do this.

W. Curtis Preston: when you say rebuild the VMware environment,

um, obviously you're talking about vm, you know, wiping the hosts and,

and the storage and all of that.

When we get to the phase of actually bringing back VMs,

Mm-hmm.

W. Curtis Preston: what way would you do that?

Um, so most backup software these days have something

built in where it'll actually scan for ransomware as you are restoring, right?

And find the ransomware if it's there.

Cause at that point, you know what you're infected with,

so you know what to look for.

Um, so I would be either scanning it or, you know, if you have really good.

and then you can decide how you're gonna fix it, or you're just gonna go

back to an earlier point or whatever.

Um, you know, some people are really good with the IR stuff and say, we know the

ransomware came in this date, this time we are absolutely a million percent certain

because we have all these logs go back to the last known good restore point, right?

Um, so it really depends.

But the backup people gonna be a big part of that, right?

Because it's gonna be

W. Curtis Preston: Y Yeah, I,

do they have built in?

W. Curtis Preston: this is something I put a lot of thought into lately

of if the meantime of a, of a.

Infection is 60 days, and some of them are twice that,

um, the, the idea of of saying, oh, well we got, we got infected December 1st,

so we're gonna restore to December 1st.

That's a

That doesn't, it doesn't always work.

In some cases it might, in some cases it won't.

And then you're going

back to scanning,

W. Curtis Preston: So you've got, you've got to, I think in most

cases, if many, if not most cases, you're gonna do a restoring.

Yeah.

I've seen kind of almost like two stage recoveries too.

Like get the bare minimum of stuff something up and run something

online up and running, right.

To restore services and then do the full recovery later.

So you're not, you might be like, all right, so you know what?

We can roll these servers back to December 29th.

We can use the newest copy of the database.

We can mash it together and make it work and serve our customers

while we're actually restoring everything the right way.

Rackspace,

So it did that.

W. Curtis Preston: Prasanna.

Yeah.

you okay?

You were eating another sip of tea there.

W. Curtis Preston: It's what I thought of when you, when you, as soon as

she said that, I, yeah, I know.

Yeah.

Just make sure.

Unlike Rackspace, just make sure that you thought of this beforehand.

Right.

The only way that this is gonna work is if you identify what are the three

services that need to be up right away so that we can function as a company and

what are the other 20, 5,000 services

That kind of, um, that ties almost more into like

the business con, you know, B C D R

W. Curtis Preston: Yeah.

Yeah,

continuity sort.

Like what are our key applications and what level of, what do we have

to do to get those online First comes back to our RPOs and RTOs, right?

W. Curtis Preston: yeah,

it's, it's,

the thing is, it's, such a

big discussion that unless you've had it cross-functionally with the

business owners and the app owners, and the infrastructure owners and the

security team, you're not in a good.

W. Curtis Preston: Yeah.

I, I think, I think it's, it's just, it's one thing to have a discussion,

again, going to Dr versus rr, um, is that it's one thing to go, well, what

are the servers we're gonna do first?

And what are, what are the servers that we're gonna do three hours later?

It's a whole other thing to say, what are the servers we're gonna do the

first couple of days, and what are the servers we're gonna do next week?

Right.

I,

And that, that's the problem, right?

You don't know until it happens.

Like if,

if you, if it's your whole environment is done right.

That is very different than, oh, we know, just, they just did this

subset of servers or whatever.

It's,

and like we were, um, The company I worked for a company

that I no longer worked there.

It was a pr uh, I was a customer and they had a, a very, they were one of the first

really, really big ransomware attacks in the news, and it was like a disaster.

I was like, wow, I'm glad I'm not on the VMware team anymore

there when this is going down.

Right.

Um, , but it really depends and you don't know what's gonna happen.

The only thing you can do is be as prepared as possible, right?

Test different recovery methods.

Um, and I love RPOs and RTOs in saying that we can meet them under a testing

scenario, but in the real world, we don't know that that's gonna happen.

W. Curtis Preston: Yeah.

One of the things on the podcast we talked about a couple

days ago was, Like Tom was mentioning, oh yeah, you just shut down your

network and you start figuring out, okay, what was affected but in what?

And you prevent everything go from going in and out.

And I was like, but how do you communicate?

Right?

And he's like, yeah, make sure you have ahead of time, sort of use cell phones.

iMessage can work.

You can set up a separate Slack instance completely outside of

the corporate environment, right?

Whatever it is to keep that ongoing communications.

like, uh, how am I supposed to use Microsoft Teams to

communicate with a security team?

Well, that might be Office 365.

That might be, okay, that's a bad example.

W. Curtis Preston: Yeah, as long as you have a, as long as you have a,

um, an internet connection, right?

Um, which is pretty easy

to get

but

like who has people's

phone numbers these days?

W. Curtis Preston: people with incident response plans, that's who

yeah, that's

But But aren't there issues though, where ransomware

actors might still have access to your Slack instance and be monitoring

what's going on from an incident

I've

seen that.

I've

seen that.

I've seen, I have seen that happen where like, they still had access.

It was teams.

I think

they still had access.

They were watching the IR

stuff happen as they were still in there hanging out.

It's like, oh yeah, Y again,

W. Curtis Preston: ransomware stuff is bad.

Melissa, I'm just gonna take that stance.

bad.

It's bad, and you don't know what's gonna happen until it happens.

Which is why, and it ties back to incident response, right?

And having an incident response firm on retainer that does this every day.

Right?

Because I, I don't care how good, even if, like, okay, let's say

you drop Melissa into X, Y, Z company and you put her in charge.

W. Curtis Preston: Do are you gonna repel down a rope from a helicopter?

Because that

Yes, I'm gonna repel down a rope from a helicopter,

drop me in, right, and say, Melissa, get ready for ransomware,

and six months later you hit me.

I would like to say that I'll be able to recover, but I don't know that.

I don't know.

That doesn't matter how good you are, you're not doing this every

day, right?

Like, so unless you're doing this every day, cuz every attack is different.

It's gonna be like, what have these people seen in the other events?

What, what ransomware gang have you been hit by?

Right?

So I can put everything into place that I think I will need

to make sure that we recover.

And yeah, honestly, we'd probably recover all our data.

I don't know if we meet our RPOs and our tails.

I, I, I'm pretty sure I could get all the data to the recoverable point,

but what was Exfiltrated, how did they get in all that kind of stuff.

you don't know, which is why you have to call the pros.

You have to call the people that do this every day.

Is there sort of a standard ransomware recovery test, but.

That kind of outlines like, Hey, here are the thing.

Because I can imagine, say you can't afford, the pros

say you can't afford the pros.

Right?

Is there sort of a, here are the testing scenarios you should be thinking

about, or here are the things that sort of get shot in the head when a

ransomware recovery or ransomware hits.

Um, Google tabletop exercises like ransomware

recovery, disaster recovery,

tabletop exercises.

Right?

That's a good place to start.

I've thought about doing like a dungeon and dragon style type,

like ransomware recovery thing.

I

Prasanna Malaiyandi: With the actual people.

Yeah, with like you get the networking security

think that would be

fun and useful.

And you know what?

When you make things fun, people actually pay a.

Yep.

right?