Immutable Backups: Your Ultimate Defense Against Ransomware
In this episode, we delve into the world of immutable backups, a crucial component of modern data protection strategies. We explore why immutable backups have become increasingly important in the face of sophisticated cyber threats, especially ransomware attacks that target backup systems. The episode covers the evolution from tape backups to disk-based systems and cloud solutions, highlighting how this shift has introduced new security challenges.
We discuss the spectrum of immutability in backup systems, from basic file system protections to fully immutable cloud storage options. The conversation touches on various implementations of immutable backups, their strengths, and potential vulnerabilities. We also address the importance of multi-factor authentication and awareness of emerging threats like AI-based voice impersonation.
Speaker:
W. Curtis Preston: Immutable backups are a critical defense against the rising
Speaker:
tide of cyber threats, and at a time when ransomware attacks are targeting not just
Speaker:
your primary data, but your backups too.
Speaker:
Understanding immutability is more critical than ever.
Speaker:
We'll explore what makes a backup truly immutable.
Speaker:
The spectrum of immutability options available and why.
Speaker:
This matters so much in your data protection strategy.
Speaker:
From tape backups to cutting edge cloud solutions, we'll cover it all.
Speaker:
Are you immutable?
Speaker:
Backups actually immutable.
Speaker:
Let's find out.
Speaker:
I.
Speaker:
By the way, if you don't know who I am, I'm w Curtis Preston, also known as Mr.
Speaker:
Backup.
Speaker:
I've been passionate about this topic for over 30 years, ever since.
Speaker:
I had to tell my boss that there were no backups of the
Speaker:
database that we had just lost.
Speaker:
I don't want that to ever happen to me again.
Speaker:
I don't want it to happen to you, and that's why we do this.
Speaker:
On this podcast, we turn unappreciated backup admins into Cyber recovery Heroes.
Speaker:
This is the backup wrap up.
Speaker:
Welcome to the show.
Speaker:
I'm your host, w Curtis Preston, AKA, Mr.
Speaker:
Backup, and I have with me my SMB backup Difficulty.
Speaker:
Sympathizer Prasanna Malaiyandi.
Speaker:
How's it going?
Speaker:
Persona.
Prasanna Malaiyandi:
I, I've learned more about dis space backups than, and tape
Prasanna Malaiyandi:
based backups in the last, like three weeks from you than any time before.
Prasanna Malaiyandi:
So yeah, how are you doing?
Prasanna Malaiyandi:
W. Curtis Preston: oh, I just like, if I could get like a, like if I
Prasanna Malaiyandi:
could have a wall here, just, you know, bang my head into the wall.
Prasanna Malaiyandi:
That's sort of how I feel like I'm doing.
Prasanna Malaiyandi:
I do
Prasanna Malaiyandi:
the problem is that you left it for 20 years
Prasanna Malaiyandi:
and now you're coming back to it.
Prasanna Malaiyandi:
And so
Prasanna Malaiyandi:
W. Curtis Preston: That's not the problem that that implies that I am old
Prasanna Malaiyandi:
or incompetent or something like that.
Prasanna Malaiyandi:
That is not the problem.
Prasanna Malaiyandi:
me going back.
Prasanna Malaiyandi:
It's like me going back and coding.
Prasanna Malaiyandi:
I could do it, but I'm not gonna be efficient.
Prasanna Malaiyandi:
W. Curtis Preston: The, the, the, um.
Prasanna Malaiyandi:
It, it, it's been, it's been really weird, right?
Prasanna Malaiyandi:
Because, you know, you change something and you change this.
Prasanna Malaiyandi:
You're like, well, let's try this.
Prasanna Malaiyandi:
And then you try that and you're like, that just doesn't work.
Prasanna Malaiyandi:
That just, you know, like the, the, the, and the really weird thing that I.
Prasanna Malaiyandi:
Has been pla plaguing me over the last, like, I don't know, several days as
Prasanna Malaiyandi:
I've been trying to make a major change to this backup configuration is that
Prasanna Malaiyandi:
the backup starts out, you know, I'm running like 20 backups at the same time.
Prasanna Malaiyandi:
Backups start out and it, and it's running like 500 megabytes per second.
Prasanna Malaiyandi:
This is great.
Prasanna Malaiyandi:
This is what I want.
Prasanna Malaiyandi:
And then about a half hour later he goes,
Prasanna Malaiyandi:
you're like.
Prasanna Malaiyandi:
To a slow drip.
Prasanna Malaiyandi:
W. Curtis Preston: happened?
Prasanna Malaiyandi:
This is not, this is not tape, this is, you know, this is not
Prasanna Malaiyandi:
like, there's not some weird tape, you know, shoe shining thing.
Prasanna Malaiyandi:
This is disc two, disc via a server in the middle.
Prasanna Malaiyandi:
It is probably some random option somewhere
Prasanna Malaiyandi:
W. Curtis Preston: random,
Prasanna Malaiyandi:
haven't done this.
Prasanna Malaiyandi:
W. Curtis Preston: data buffers thing.
Prasanna Malaiyandi:
Oh, oh, oh.
Prasanna Malaiyandi:
You're, you wanna do this on a Thursday, you need to do the, you
Prasanna Malaiyandi:
need to do the setting and, uh.
Prasanna Malaiyandi:
what I said, that because you haven't done this
Prasanna Malaiyandi:
day in to day out, you don't know.
Prasanna Malaiyandi:
It's like you lose touch of these or you don't know
Prasanna Malaiyandi:
W. Curtis Preston: Yeah.
Prasanna Malaiyandi:
And I, I will say that when I was, when I was, when I was last doing this really in
Prasanna Malaiyandi:
earnest, this backups were relatively new.
Prasanna Malaiyandi:
Um, that, you know, that really dates me, but that is, that is the case, but.
Prasanna Malaiyandi:
So, yeah.
Prasanna Malaiyandi:
And I just, you know, and I, I've been thankful for you to talk to you.
Prasanna Malaiyandi:
It's like, you know, so here's the thing, you know, I got this, I got
Prasanna Malaiyandi:
this setting, I got that setting.
Prasanna Malaiyandi:
I've been playing with jumbo frames, I've been playing with,
Prasanna Malaiyandi:
with, uh, buffer settings.
Prasanna Malaiyandi:
I've been playing with all these different things.
Prasanna Malaiyandi:
And you know, and you're like, what about the thing?
Prasanna Malaiyandi:
You know, what about this setting?
Prasanna Malaiyandi:
Oh, man.
Prasanna Malaiyandi:
And you know what?
Prasanna Malaiyandi:
In the end, once you figure out what's going on, it's probably
Prasanna Malaiyandi:
just gonna be some simple thing,
Prasanna Malaiyandi:
W. Curtis Preston: Mm-hmm.
Prasanna Malaiyandi:
right?
Prasanna Malaiyandi:
That you just didn't know existed.
Prasanna Malaiyandi:
And you're gonna fix it and be like, oh yeah, now it's like, boom.
Prasanna Malaiyandi:
W. Curtis Preston: Yeah.
Prasanna Malaiyandi:
And then it's gonna run like a thousand megabytes per second, and then,
Prasanna Malaiyandi:
we'll, everything will be amazing.
Prasanna Malaiyandi:
Um, so, we're gonna talk about some things that are really new in terms of when, when
Prasanna Malaiyandi:
I first started with backup, we really didn't talk about this concept when I, you
Prasanna Malaiyandi:
know, was doing backup back in the day.
Prasanna Malaiyandi:
There's a big thing that happened in the last.
Prasanna Malaiyandi:
Two decades.
Prasanna Malaiyandi:
That is one of the biggest reasons why we didn't use to talk about
Prasanna Malaiyandi:
cybersecurity and backups, and now we talk about it all the time.
Prasanna Malaiyandi:
Do you know what that is?
Prasanna Malaiyandi:
Persona?
Prasanna Malaiyandi:
No,
Prasanna Malaiyandi:
W. Curtis Preston: It's
Prasanna Malaiyandi:
one change.
Prasanna Malaiyandi:
No.
Prasanna Malaiyandi:
One seismic change in backups in the last 20 years.
Prasanna Malaiyandi:
Dis space backups.
Prasanna Malaiyandi:
W. Curtis Preston: Yes, exactly right.
Prasanna Malaiyandi:
Tapes weren't hackable.
Prasanna Malaiyandi:
Tapes were offline by design.
Prasanna Malaiyandi:
Right?
Prasanna Malaiyandi:
Ooh, that rhymes anytime.
Prasanna Malaiyandi:
Um, yeah.
Prasanna Malaiyandi:
So tapes were, tapes were not sitting there just files ready
Prasanna Malaiyandi:
to be deleted or encrypted.
Prasanna Malaiyandi:
Tapes were inherently hard, even if they were online, even if, if they were in a
Prasanna Malaiyandi:
tape library, they were hard to get to.
Prasanna Malaiyandi:
And then.
Prasanna Malaiyandi:
Uh, over, you know, and, and if they were offline, if they were sitting,
Prasanna Malaiyandi:
you know, in a, a tape vaulting vendor, then they were really hard to get to.
Prasanna Malaiyandi:
But the, um, what happened when we went to dis backups, we
Prasanna Malaiyandi:
basically made backups hackable.
Prasanna Malaiyandi:
And so that, and, and I don't think, in fact, I know, I know.
Prasanna Malaiyandi:
We really didn't take that through back when we were.
Prasanna Malaiyandi:
Unintended.
Prasanna Malaiyandi:
W. Curtis Preston: Yeah, we were just thinking about all the problems
Prasanna Malaiyandi:
that we were having with tape, and we wanted to fix those problems.
Prasanna Malaiyandi:
And so we went with this space backups to fix those problems.
Prasanna Malaiyandi:
And, uh, and remember, I will, we'll, we'll always say this.
Prasanna Malaiyandi:
It wasn't, the tape was too slow.
Prasanna Malaiyandi:
It was, the tape was too fast.
Prasanna Malaiyandi:
And we can, you know, if you don't know what I'm talking about, then you need
Prasanna Malaiyandi:
to, you need to listen to more episodes.
Prasanna Malaiyandi:
And we've, we've come a long way, but for a really long time backwards, were
Prasanna Malaiyandi:
just sitting there as files either in a, in a regular, uh, you know,
Prasanna Malaiyandi:
direct attached file system or an NFS or SMB attached file system that were
Prasanna Malaiyandi:
directly deletable by a privileged user.
Prasanna Malaiyandi:
Uh, as long as you, you know, as long as you had access to that drive, we've.
Prasanna Malaiyandi:
Hopefully in most cases we've gotten away from that.
Prasanna Malaiyandi:
I think still many people go and buy a, you know, pick your favorite backup,
Prasanna Malaiyandi:
you know, disc vendor, and they just mount it as a backup drive and then
Prasanna Malaiyandi:
start backing up to it, and it's, it's, it's eco back slash backups.
Prasanna Malaiyandi:
And which just like screams, come delete me.
Prasanna Malaiyandi:
And you know, I, I, I know people still do that and they shouldn't do that.
Prasanna Malaiyandi:
They should ask their backup vendor, Hey, is there a way that I can
Prasanna Malaiyandi:
transfer data to my dis space backup storage without it showing up as a,
Prasanna Malaiyandi:
as a dry letter on my uh, system or a mounted drive on my Unix system?
Prasanna Malaiyandi:
But that is a huge change in security of backups.
Prasanna Malaiyandi:
Yeah.
Prasanna Malaiyandi:
No, I agree.
Prasanna Malaiyandi:
It made life easier, right?
Prasanna Malaiyandi:
You could now do all the instant access, quickly, restore your data, do all the
Prasanna Malaiyandi:
test in dev, but there was a fatal flaw.
Prasanna Malaiyandi:
W. Curtis Preston: Yeah, it made life easier for everybody.
Prasanna Malaiyandi:
Yeah.
Prasanna Malaiyandi:
W. Curtis Preston: Yeah, absolutely.
Prasanna Malaiyandi:
And then replicated backups, right?
Prasanna Malaiyandi:
We had replicated backups, and then if you, you know, if you have a, a,
Prasanna Malaiyandi:
you know, a screw up in your backup system, it, the replication just makes
Prasanna Malaiyandi:
the screw up, uh, more effective.
Prasanna Malaiyandi:
Right.
Prasanna Malaiyandi:
Yeah.
Prasanna Malaiyandi:
W. Curtis Preston: Yeah, so this is why we have to talk about immutability.
Prasanna Malaiyandi:
Um, but before we talk about that, we'll talk about some things
Prasanna Malaiyandi:
that haven't changed at all.
Prasanna Malaiyandi:
So the thing I want to talk about is this concept of immutable
Prasanna Malaiyandi:
backups, and that really wasn't.
Prasanna Malaiyandi:
Something we talked about back in the day.
Prasanna Malaiyandi:
Well, I think a lot of it back in the day was your backups
Prasanna Malaiyandi:
were kind of already immutable, right?
Prasanna Malaiyandi:
If you were,
Prasanna Malaiyandi:
W. Curtis Preston: the,
Prasanna Malaiyandi:
were going off tape.
Prasanna Malaiyandi:
W. Curtis Preston: Yeah.
Prasanna Malaiyandi:
Right then you didn't have to worry about the scenarios
Prasanna Malaiyandi:
that led you to need immutable backups.
Prasanna Malaiyandi:
W. Curtis Preston: Yeah.
Prasanna Malaiyandi:
That, that's really true.
Prasanna Malaiyandi:
And that, and that's one of the things that we're gonna talk about, right?
Prasanna Malaiyandi:
I, I would say the biggest reason was that no one was attacking the backups.
Prasanna Malaiyandi:
No one was trying to attack the backups.
Prasanna Malaiyandi:
And the, the worst things that we were worried about were things like I.
Prasanna Malaiyandi:
Um, we were worried about a natural disaster or a terrorist
Prasanna Malaiyandi:
attack, taking out the entire place where backups were stored.
Prasanna Malaiyandi:
So we needed to make sure to have a, an air gap, a physical separation between
Prasanna Malaiyandi:
the, the backups and, uh, you know, and the, the copy of the backups, right?
Prasanna Malaiyandi:
We weren't.
Prasanna Malaiyandi:
on a tropical island.
Prasanna Malaiyandi:
W. Curtis Preston: Exactly right.
Prasanna Malaiyandi:
Exactly.
Prasanna Malaiyandi:
And so we weren't really, we, we, we weren't at all.
Prasanna Malaiyandi:
I, I could just, just absolutely tell you that we were not at all
Prasanna Malaiyandi:
ever thinking about a cyber attack.
Prasanna Malaiyandi:
Trying to, I, I don't even think we had that term.
Prasanna Malaiyandi:
I mean, we had the internet, but we didn't really have the, the term cyber attack.
Prasanna Malaiyandi:
Um, at least that I remember.
Prasanna Malaiyandi:
We certainly never used it in backup circles,
Prasanna Malaiyandi:
Or it was different, right?
Prasanna Malaiyandi:
Because you weren't necessarily worried about people stealing
Prasanna Malaiyandi:
your data or things like that.
Prasanna Malaiyandi:
I remember you were the one who recommended me to read
Prasanna Malaiyandi:
the Cuckoo's test, right?
Prasanna Malaiyandi:
And so those were the types of attacks that you sort of
Prasanna Malaiyandi:
saw back in those days, right?
Prasanna Malaiyandi:
Stealing resources, not necessarily stealing your data
Prasanna Malaiyandi:
W. Curtis Preston: Yeah, cuckoo egg, by the way.
Prasanna Malaiyandi:
Or sorry, cuckoo.
Prasanna Malaiyandi:
W. Curtis Preston: which by the way, if, if, if you're out there and you don't
Prasanna Malaiyandi:
know what we're talking about, great book.
Prasanna Malaiyandi:
Um, uh, and it's a relatively short read.
Prasanna Malaiyandi:
It's a great book to read, and, and it's a true story that, you know, involves
Prasanna Malaiyandi:
a, a cyber attack against a university.
Prasanna Malaiyandi:
It's great.
Prasanna Malaiyandi:
So before we talk about immutable backups, I want to talk about some
Prasanna Malaiyandi:
of the things that really haven't changed that are also, um, impediments
Prasanna Malaiyandi:
to having a safe backup system.
Prasanna Malaiyandi:
And the first is the fact that backups never were, never have been, and never
Prasanna Malaiyandi:
will be the sexy part of the data center.
Prasanna Malaiyandi:
Yeah, no, I a hundred percent agree.
Prasanna Malaiyandi:
No one wants to do backups.
Prasanna Malaiyandi:
No one wants to spend the money on backups, right?
Prasanna Malaiyandi:
Everyone wants to be like, Ooh, I need to do, given the recent trend
Prasanna Malaiyandi:
these days, I wanna focus on AI and do this and that and all the rest.
Prasanna Malaiyandi:
But no one wants to focus on how do I protect it other than people like Curtis.
Prasanna Malaiyandi:
W. Curtis Preston: Other than a few, few weird people.
Prasanna Malaiyandi:
I would say that it's a, there were a lot of reasons.
Prasanna Malaiyandi:
The biggest reason is easily that it is a very difficult job.
Prasanna Malaiyandi:
We, we started out this, this episode talking about how
Prasanna Malaiyandi:
sometimes, how difficult it can be.
Prasanna Malaiyandi:
You are invisible or you're in trouble.
Prasanna Malaiyandi:
No one, no one remembers the millions of backups that you got, right?
Prasanna Malaiyandi:
They only remember the one restore that you got wrong.
Prasanna Malaiyandi:
And so nobody wants to have that pressure on themselves.
Prasanna Malaiyandi:
And so they, nobody wants to raise their hand in, you know, in the,
Prasanna Malaiyandi:
uh, in the room to say, Hey, I'll, I'll be in charge of backups.
Prasanna Malaiyandi:
So this is true in terms of.
Prasanna Malaiyandi:
Um, it's not the sexy part of the data center, which means nobody
Prasanna Malaiyandi:
wants to do it, number th number one.
Prasanna Malaiyandi:
Number two, nobody wants to give that part of the data center any money
Prasanna Malaiyandi:
number, and then number three I'd say is that as a result, it goes ignored.
Prasanna Malaiyandi:
Prasanna Malaiyandi: Oh, I agree with that.
Prasanna Malaiyandi:
I think there was a study, I can't remember when it was that.
Prasanna Malaiyandi:
What percentage of an IT budget do you think goes towards backup?
Prasanna Malaiyandi:
W. Curtis Preston: It's a very small portion.
Prasanna Malaiyandi:
I think it was less than a percent.
Prasanna Malaiyandi:
W. Curtis Preston: Yeah.
Prasanna Malaiyandi:
Yeah.
Prasanna Malaiyandi:
That sounds about right.
Prasanna Malaiyandi:
Yeah.
Prasanna Malaiyandi:
So it, it goes, it goes ignored.
Prasanna Malaiyandi:
It goes ignored.
Prasanna Malaiyandi:
From a budget perspective, it goes ignored from.
Prasanna Malaiyandi:
A cybersecurity perspective, which is really what we're talking about.
Prasanna Malaiyandi:
It's what we've been talking about the last several episodes is, is
Prasanna Malaiyandi:
cybersecurity and, and backups.
Prasanna Malaiyandi:
And it, it means that it doesn't get put front of the line when we start
Prasanna Malaiyandi:
talking about putting out patches.
Prasanna Malaiyandi:
It doesn't get put front of the line in terms of looking
Prasanna Malaiyandi:
for trends that could be scary.
Prasanna Malaiyandi:
Um, it's just, it literally just goes ignored.
Prasanna Malaiyandi:
Um, and that, and that's a really scary thing.
Prasanna Malaiyandi:
I.
Prasanna Malaiyandi:
Yeah, and it's not even thought of initially when you're,
Prasanna Malaiyandi:
I know we've talked on the episodes that when you are deploying a new application,
Prasanna Malaiyandi:
you should be thinking about backup or at least have the backup person in the room.
Prasanna Malaiyandi:
But sometimes things develop so quickly that.
Prasanna Malaiyandi:
Backup is an afterthought, right?
Prasanna Malaiyandi:
And sometimes it's very painful.
Prasanna Malaiyandi:
W. Curtis Preston: Yeah, very much so.
Prasanna Malaiyandi:
I can remember, I can remember one example of that when we bought, uh, the comp, the
Prasanna Malaiyandi:
bank that I worked at bought a, an HP.
Prasanna Malaiyandi:
Was it 1000 I think was the HPT 1000.
Prasanna Malaiyandi:
That's what it was.
Prasanna Malaiyandi:
It was a T 1000 and the, it came in and it had, um, it was huge.
Prasanna Malaiyandi:
It was, are you ready?
Prasanna Malaiyandi:
A hundred gigabytes of data.
Prasanna Malaiyandi:
It was huge.
Prasanna Malaiyandi:
Humongous.
Prasanna Malaiyandi:
W. Curtis Preston: It was humongous.
Prasanna Malaiyandi:
Now your phone has more than that.
Prasanna Malaiyandi:
W. Curtis Preston: Prior to this server or prior to this server showing
Prasanna Malaiyandi:
up at our data center, the biggest server we had was like four, and then
Prasanna Malaiyandi:
suddenly we had a single server that was 25 times bigger than the next
Prasanna Malaiyandi:
biggest server, and it came with.
Prasanna Malaiyandi:
A DDS drive, which was a, a, um, four gigabyte drive.
Prasanna Malaiyandi:
And, and I, I just remember asking like, who, who's gonna stand here
Prasanna Malaiyandi:
all night long and swap tapes 25 times to get to get it back up?
Prasanna Malaiyandi:
And that was, and, and nobody had knew, nobody had even thought of that.
Prasanna Malaiyandi:
That's the point that nobody had even thought, well, maybe we should
Prasanna Malaiyandi:
have enough backup infrastructure to back up this new gigantic server.
Prasanna Malaiyandi:
And that was, by the way, how I got the, my first justification to
Prasanna Malaiyandi:
buy my first, uh, tape robot, which was the first time I sent money
Prasanna Malaiyandi:
to Spectral Logic back in the day.
Prasanna Malaiyandi:
You weren't just sitting there all night swapping tapes.
Prasanna Malaiyandi:
Curtis.
Prasanna Malaiyandi:
W. Curtis Preston: definitely not gonna do that.
Prasanna Malaiyandi:
Definitely not.
Prasanna Malaiyandi:
Um, so yeah, so that, that problem's not a new problem.
Prasanna Malaiyandi:
Uh, the other thing, and as a result of all of those things that we just
Prasanna Malaiyandi:
talked about, a very common thing for.
Prasanna Malaiyandi:
Computing infrastructure or for computing environments to do is
Prasanna Malaiyandi:
to give backups to the new person,
Prasanna Malaiyandi:
right?
Prasanna Malaiyandi:
Uh, it was the first job I ever got.
Prasanna Malaiyandi:
I, I was just editing the, the podcast.
Prasanna Malaiyandi:
So with Dwayne Lalo, he said it was the first job he ever got.
Prasanna Malaiyandi:
And the problem with that really is you have this really, really important,
Prasanna Malaiyandi:
really secure, or what should be a really secure job, and you're handing
Prasanna Malaiyandi:
it to the least experienced member of the, of the, you know, the team.
Prasanna Malaiyandi:
Right.
Prasanna Malaiyandi:
I, I mean, I, I had been at the bank,
Prasanna Malaiyandi:
you know, essentially a week.
Prasanna Malaiyandi:
I, I had been, I had been.
Prasanna Malaiyandi:
In computers, like, like commercial computers.
Prasanna Malaiyandi:
And I had been introduced to Unix for a few months.
Prasanna Malaiyandi:
I, I, I had been at this company, I, I think I had worked at this
Prasanna Malaiyandi:
company for like two months, uh, prior to going to the bank.
Prasanna Malaiyandi:
And I had been, you know, I had been given some lessons in Unix
Prasanna Malaiyandi:
and I had been demonstrated the importance of a really good backup.
Prasanna Malaiyandi:
Uh, here's, this was the first time in my career when.
Prasanna Malaiyandi:
Prasanna Malaiyandi: Theoretical versus practical.
Prasanna Malaiyandi:
W. Curtis Preston: Yeah, so we were a, uh, we were a blended call management
Prasanna Malaiyandi:
system, which you might refer to as an auto dialer, and that was a, that was a,
Prasanna Malaiyandi:
a naughty word back in the day, right?
Prasanna Malaiyandi:
We were not an auto dialer.
Prasanna Malaiyandi:
We were an intelligent call management system.
Prasanna Malaiyandi:
And, but it was essentially an Auto Tyler and it had this very fancy
Prasanna Malaiyandi:
Unix-based computer behind it.
Prasanna Malaiyandi:
And we would go out and do this, this, it was like this long involved install
Prasanna Malaiyandi:
because it was a very, this was before VoIP and all that kind of stuff.
Prasanna Malaiyandi:
We had, you know, we had literally cards that went in the server and those, each
Prasanna Malaiyandi:
card connected to a POTS line, right?
Prasanna Malaiyandi:
Mm-Hmm.
Prasanna Malaiyandi:
W. Curtis Preston: And so we had this big physical install that we had
Prasanna Malaiyandi:
to do, and it took about two weeks.
Prasanna Malaiyandi:
And then, um, you know, you, you sent a couple of SCS on site and I was one
Prasanna Malaiyandi:
of the sc I was obviously the junior se, and this other guy had been, um,
Prasanna Malaiyandi:
doing all this configuration and.
Prasanna Malaiyandi:
The job that we were supposed to do before we left was to pop in a tape,
Prasanna Malaiyandi:
by the way, a tape, a nine track
Prasanna Malaiyandi:
Ginormous.
Prasanna Malaiyandi:
Yeah.
Prasanna Malaiyandi:
W. Curtis Preston: and pop in a nine track tape and back up the OS
Prasanna Malaiyandi:
configuration that we had just finished.
Prasanna Malaiyandi:
That was the last thing that we were supposed to do, and just
Prasanna Malaiyandi:
before doing that, my, my compadre.
Prasanna Malaiyandi:
Said he was looking at the root file system and he noticed that there were
Prasanna Malaiyandi:
a couple of garbage files in the root file system and one of them started
Prasanna Malaiyandi:
with, um, question mark and had a bunch of gar dego stuff after it.
Prasanna Malaiyandi:
So he didn't want to type all of the gar dego, so he typed, um,
Prasanna Malaiyandi:
RM minus RF question mark, star
Prasanna Malaiyandi:
Oh,
Prasanna Malaiyandi:
W. Curtis Preston: from the root file system.
Prasanna Malaiyandi:
Wanting to clean up the OS before we took the backup.
Prasanna Malaiyandi:
oh geez.
Prasanna Malaiyandi:
That must have been interesting.
Prasanna Malaiyandi:
W. Curtis Preston: Yeah, so that was the, that was the day that I
Prasanna Malaiyandi:
learned the value of a good backup.
Prasanna Malaiyandi:
The best time to do a backup is yesterday.
Prasanna Malaiyandi:
The second best time is today.
Prasanna Malaiyandi:
today.
Prasanna Malaiyandi:
W. Curtis Preston: Um, this was definitely the second best time,
Prasanna Malaiyandi:
so we lost a whole bunch of work.
Prasanna Malaiyandi:
Uh, and luckily it was not my fault, so I just learned a very valuable lesson.
Prasanna Malaiyandi:
Anyway, sorry, that was bringing me, bringing me back, back in the day.
Prasanna Malaiyandi:
Um, so yeah, so.
Prasanna Malaiyandi:
It's a horrible practice that we put the most junior person in charge of the
Prasanna Malaiyandi:
backups and that should absolutely stop, but I don't know how to merge those two.
Prasanna Malaiyandi:
How do you, how do you, how do you properly reward this job that
Prasanna Malaiyandi:
is really, really important such that someone who isn't junior is
Prasanna Malaiyandi:
going to want to take the job?
Prasanna Malaiyandi:
I don't know.
Prasanna Malaiyandi:
I don't have an answer to that question.
Prasanna Malaiyandi:
Do you, do you persona.
Prasanna Malaiyandi:
Well, no, but it's No, but if you think
Prasanna Malaiyandi:
about this, this really isn't any different than say A DBA, right?
Prasanna Malaiyandi:
A database admin who starts off initially does some Basic things, right?
Prasanna Malaiyandi:
And over time they sort of become like that database architect who sort
Prasanna Malaiyandi:
of figuring out overall layouts and things like that rather than sort
Prasanna Malaiyandi:
of just administering the system.
Prasanna Malaiyandi:
Now the difference is in the case of the DBA, you're probably starting
Prasanna Malaiyandi:
off in like test and dev instances or like non-critical applications versus
Prasanna Malaiyandi:
as a backup admin, you're probably starting off on like the critical
Prasanna Malaiyandi:
infrastructure that you are protecting.
Prasanna Malaiyandi:
W. Curtis Preston: Yeah.
Prasanna Malaiyandi:
Yeah.
Prasanna Malaiyandi:
That's why I'm, I, I'm not sure I agree with your, with your comparison
Prasanna Malaiyandi:
there because, well, first off, people actually wanna be DBAs.
Prasanna Malaiyandi:
Yeah.
Prasanna Malaiyandi:
Well, and yeah, maybe.
Prasanna Malaiyandi:
Maybe what you need to do is like a cross training program where you do
Prasanna Malaiyandi:
a rotation in like a different part of the organization and then you come
Prasanna Malaiyandi:
into the backup work, because that way at least you're also get more
Prasanna Malaiyandi:
familiar with tools and other things.
Prasanna Malaiyandi:
W. Curtis Preston: Right, right.
Prasanna Malaiyandi:
Yeah.
Prasanna Malaiyandi:
Yeah.
Prasanna Malaiyandi:
That way you're not the junior person.
Prasanna Malaiyandi:
W. Curtis Preston: exactly, exactly.
Prasanna Malaiyandi:
Yeah.
Prasanna Malaiyandi:
You, you need, I, I guess the best thing we want to do here is to make sure.
Prasanna Malaiyandi:
That you're, that you're making sure that your junior person, that you're giving
Prasanna Malaiyandi:
the, the backups to has some really good cybersecurity, some basic, at least some
Prasanna Malaiyandi:
basic cybersecurity training, right?
Prasanna Malaiyandi:
Because you're handing them in order to do the backups.
Prasanna Malaiyandi:
They are a very powerful person and they have the ability to
Prasanna Malaiyandi:
read and write every file.
Prasanna Malaiyandi:
Right, which means they have the ability to overwrite every
Prasanna Malaiyandi:
file in the organization.
Prasanna Malaiyandi:
Uh, even if you're using proper modern backup design that uses role-based
Prasanna Malaiyandi:
administration and all of that, you're, you're still most likely
Prasanna Malaiyandi:
handing them the keys to the kingdom.
Prasanna Malaiyandi:
Uh,
Prasanna Malaiyandi:
Is that true though?
Prasanna Malaiyandi:
Because normally don't, you have sort of help desk restore operators and
Prasanna Malaiyandi:
then the more senior people are kind of those who are doing like the backup
Prasanna Malaiyandi:
architecture and the design, you probably have people who are figuring out
Prasanna Malaiyandi:
W. Curtis Preston: In a large environment, that would be the case, but that's not
Prasanna Malaiyandi:
the case in most companies, I don't think.
Prasanna Malaiyandi:
I think it's just one person, actually, in most companies it's a collateral duty.
Prasanna Malaiyandi:
Right.
Prasanna Malaiyandi:
So, but let's, so let's get to the meat of the episode here where we
Prasanna Malaiyandi:
wanna talk about immutable backups.
Prasanna Malaiyandi:
So first off, just generally speaking, what does immutable mean?
Prasanna Malaiyandi:
It basically means you can't change it,
Prasanna Malaiyandi:
W. Curtis Preston: Yeah.
Prasanna Malaiyandi:
It's literally, it's, it's, it's, it's a word that's used outside
Prasanna Malaiyandi:
of it, but within it, yeah.
Prasanna Malaiyandi:
It means that it cannot be changed
Prasanna Malaiyandi:
which also includes deletions.
Prasanna Malaiyandi:
W. Curtis Preston: yeah.
Prasanna Malaiyandi:
Changed.
Prasanna Malaiyandi:
Yeah.
Prasanna Malaiyandi:
Deletion is a change.
Prasanna Malaiyandi:
Right.
Prasanna Malaiyandi:
Um, and the difficulty here.
Prasanna Malaiyandi:
Is that, that sounds like a binary condition,
Prasanna Malaiyandi:
Mm-Hmm.
Prasanna Malaiyandi:
W. Curtis Preston: Like pregnant or dead.
Prasanna Malaiyandi:
Right.
Prasanna Malaiyandi:
Not the words I would've chosen, but Sure.
Prasanna Malaiyandi:
W. Curtis Preston: Both pregnancy and death are binary conditions.
Prasanna Malaiyandi:
You're either pregnant or you're not, or you're dead or alive, right?
Prasanna Malaiyandi:
These are binary conditions.
Prasanna Malaiyandi:
Um, immutable should be a binary condition that that is what is implied.
Prasanna Malaiyandi:
When we say immutable, it can't be changed.
Prasanna Malaiyandi:
It can't be deleted.
Prasanna Malaiyandi:
It can't be encrypted.
Prasanna Malaiyandi:
It can't be modified.
Prasanna Malaiyandi:
It can't be, I don't know.
Prasanna Malaiyandi:
I think I covered 'em all.
Prasanna Malaiyandi:
Yeah, but there's really a spectrum though.
Prasanna Malaiyandi:
W. Curtis Preston: it is definitely a spectrum, and that is the
Prasanna Malaiyandi:
problem, is that immutable.
Prasanna Malaiyandi:
And that's, that's the thing I wanted to just mention.
Prasanna Malaiyandi:
So let me, so let me ask you this.
Prasanna Malaiyandi:
Why, why has immutable backups, why has that really come to the fore,
Prasanna Malaiyandi:
uh, in the last say, 10 years or so?
Prasanna Malaiyandi:
Well, like you were talking about earlier
Prasanna Malaiyandi:
in the episode, cyber attacks.
Prasanna Malaiyandi:
Ransomware attacks, they not only go after your production, but they
Prasanna Malaiyandi:
realize that, hey, you have all this data sitting in your backup systems
Prasanna Malaiyandi:
that you could use to recover your environment and not pay the ransom.
Prasanna Malaiyandi:
So instead, we're gonna go attack your backup systems, delete all
Prasanna Malaiyandi:
your backups so you can't recover.
Prasanna Malaiyandi:
And then also because they're doing the double extortion where
Prasanna Malaiyandi:
they also exfiltrate your data.
Prasanna Malaiyandi:
That's a treasure trove, right?
Prasanna Malaiyandi:
They don't have to go hit 20 different systems.
Prasanna Malaiyandi:
If they get to your backup system.
Prasanna Malaiyandi:
All your data's there and backup systems are really good at restoring data so
Prasanna Malaiyandi:
they can quickly pull the data out and then blackmail you in case you decide
Prasanna Malaiyandi:
that you don't want to pay the ransom.
Prasanna Malaiyandi:
W. Curtis Preston: Yeah, exactly.
Prasanna Malaiyandi:
Now immutability won't address that second part.
Prasanna Malaiyandi:
Uh, it is important to me.
Prasanna Malaiyandi:
Right tho those are, those are the two main concerns that we have when
Prasanna Malaiyandi:
we talk about the backup system.
Prasanna Malaiyandi:
And, uh, you should definitely listen to the episode that went live.
Prasanna Malaiyandi:
Um, let's see, that would've been May 26th now, um, the one where with Dwayne Lalo,
Prasanna Malaiyandi:
where he talks a bit from the Red Team, the Red Team Cybersecurity episode that.
Prasanna Malaiyandi:
Um, where he talks about what a treasure trove the backup system is and how he
Prasanna Malaiyandi:
loves, he's a, you know, he is a red team.
Prasanna Malaiyandi:
He's a hacker, uh, that, you know, an offensive cybersecurity person.
Prasanna Malaiyandi:
Uh, he's very offensive.
Prasanna Malaiyandi:
And, and he loves going after the backup system, right?
Prasanna Malaiyandi:
When we talk about immutability, we're primarily concerned with
Prasanna Malaiyandi:
making sure that the backups themselves aren't modified, right.
Prasanna Malaiyandi:
And which, you know, as we said, you know, and, and.
Prasanna Malaiyandi:
Like you said, this wasn't a problem.
Prasanna Malaiyandi:
I don't know, even five years ago, maybe, uh, this wasn't a problem,
Prasanna Malaiyandi:
but hackers weren't going directly after the backups, but now they
Prasanna Malaiyandi:
definitely are because they're the best defense we have against ransomware.
Prasanna Malaiyandi:
Right.
Prasanna Malaiyandi:
I.
Prasanna Malaiyandi:
Yeah.
Prasanna Malaiyandi:
And and I would say before immutable backups slash immutable archiving was
Prasanna Malaiyandi:
really intended as a compliance use case.
Prasanna Malaiyandi:
W. Curtis Preston: Yeah.
Prasanna Malaiyandi:
And that's what I, that's kind of where, where I wanted to go
Prasanna Malaiyandi:
with this is let's talk about the different things that people mean
Prasanna Malaiyandi:
when they say immutable, right?
Prasanna Malaiyandi:
And prior to cyber security attacks, when we talked about the backup system
Prasanna Malaiyandi:
or the archive system, which are different things when we talk about the
Prasanna Malaiyandi:
backup system or the archive system.
Prasanna Malaiyandi:
Being immutable, what we're primarily, um, wanting to be able to say in a
Prasanna Malaiyandi:
court of law that this file, this document that I'm presenting in court,
Prasanna Malaiyandi:
is exactly the same as it was I.
Prasanna Malaiyandi:
When it was created, right.
Prasanna Malaiyandi:
This, this email that is the smoking gun we can state.
Prasanna Malaiyandi:
It is a, it is a faithful reproduction of the original email that the plaintiff
Prasanna Malaiyandi:
made or the defendant, I get It's, it's a's a faithful reproduction of
Prasanna Malaiyandi:
the email that the defendant made.
Prasanna Malaiyandi:
That shows that they were doing the thing that they weren't supposed to be doing.
Prasanna Malaiyandi:
Right?
Prasanna Malaiyandi:
And, and so the, so that the, the defense can't say, oh, well
Prasanna Malaiyandi:
that's a fake email, right?
Prasanna Malaiyandi:
You're like, no, this is a faithful reproduction of the email that this,
Prasanna Malaiyandi:
you know, and here's the, here's the chain of custodying, the chain
Prasanna Malaiyandi:
of technology to get from A to B.
Prasanna Malaiyandi:
And so, um, you know, it, it's, fine,
Prasanna Malaiyandi:
It's as good as the source.
Prasanna Malaiyandi:
It's as good as the original.
Prasanna Malaiyandi:
W. Curtis Preston: It's as good.
Prasanna Malaiyandi:
Yeah, we are.
Prasanna Malaiyandi:
We are essentially looking directly into the email system from three months
Prasanna Malaiyandi:
ago or three years ago or whatever.
Prasanna Malaiyandi:
It's right.
Prasanna Malaiyandi:
That's what we were primarily concerned with and we are still concerned with that.
Prasanna Malaiyandi:
Right.
Prasanna Malaiyandi:
We are still concerned with immutability for that reason from
Prasanna Malaiyandi:
a legal perspective, but, um.
Prasanna Malaiyandi:
What we're now more concerned with is that someone could go in and modify the backups
Prasanna Malaiyandi:
in such a way that they encrypt them.
Prasanna Malaiyandi:
They, you know, they, they delete them.
Prasanna Malaiyandi:
Uh, they could actually modify them.
Prasanna Malaiyandi:
I, I don't, I don't, I don't know of any scenario in this case, but
Prasanna Malaiyandi:
they could potentially go in and selectively modify them to mess
Prasanna Malaiyandi:
with people for whatever reason.
Prasanna Malaiyandi:
Um, but that shouldn't be able to be the case either.
Prasanna Malaiyandi:
Right?
Prasanna Malaiyandi:
Yeah, and that just seems like a lot of work to go selectively
Prasanna Malaiyandi:
modify unless you're trying to do something sneakily without being detected.
Prasanna Malaiyandi:
W. Curtis Preston: Right, right.
Prasanna Malaiyandi:
Um, so that's the, that's one way that backup systems and archive systems use
Prasanna Malaiyandi:
the term immutability is they say, I.
Prasanna Malaiyandi:
We can say that, you know, these two things were, you know, are the
Prasanna Malaiyandi:
same.
Prasanna Malaiyandi:
So the, the next thing that I see is that.
Prasanna Malaiyandi:
I see vendors that say, well, we have a A, right?
Prasanna Malaiyandi:
Only we have an append only file system that once backups are
Prasanna Malaiyandi:
written into our append only file system, they cannot be modified.
Prasanna Malaiyandi:
Have you seen backup folks saying that I.
Prasanna Malaiyandi:
Yeah, though I've seen some storage vendors, right?
Prasanna Malaiyandi:
You have companies who do backup appliances like Rubrik who say,
Prasanna Malaiyandi:
yeah, once you write your backup to our system, right, you can't
Prasanna Malaiyandi:
go and overwrite that data again.
Prasanna Malaiyandi:
W. Curtis Preston: Right,
Prasanna Malaiyandi:
of a right only and.
Prasanna Malaiyandi:
Having worked at a storage company which was famous for a write
Prasanna Malaiyandi:
Anywhere file layout, I could see the benefits of doing it that way.
Prasanna Malaiyandi:
W. Curtis Preston: Right.
Prasanna Malaiyandi:
Yeah.
Prasanna Malaiyandi:
And, and what this says is that it stops direct attacks against
Prasanna Malaiyandi:
the backup storage itself.
Prasanna Malaiyandi:
That if I, if I'm able to get into the file system, if I, if I'm able
Prasanna Malaiyandi:
to get into the operating system of that backup system, I wouldn't
Prasanna Malaiyandi:
be able to modify or delete the backup files in order to, you know.
Prasanna Malaiyandi:
Well, well essentially delete them.
Prasanna Malaiyandi:
I wouldn't be able to encrypt or delete them because this is
Prasanna Malaiyandi:
this append only file system.
Prasanna Malaiyandi:
Which is interesting because I do wonder
Prasanna Malaiyandi:
about certain scenarios like, Hey, I'm running low on space.
Prasanna Malaiyandi:
I need to go delete a whole bunch of backups or backup sets
Prasanna Malaiyandi:
in order to free things up.
Prasanna Malaiyandi:
How?
Prasanna Malaiyandi:
What do I do?
Prasanna Malaiyandi:
Am I just stuck?
Prasanna Malaiyandi:
W. Curtis Preston: Yeah, that's gonna, that's gonna be
Prasanna Malaiyandi:
depending on the product, right?
Prasanna Malaiyandi:
So the question, my, my question is.
Prasanna Malaiyandi:
With the, the product in question and that this isn't necessarily Rubrik,
Prasanna Malaiyandi:
this is just, I'm, it's just the, this is the question that we're asking
Prasanna Malaiyandi:
is, I, I understand you're saying that it's an append only file system.
Prasanna Malaiyandi:
What about the system that's controlling the backups itself?
Prasanna Malaiyandi:
What protection is there in, in that system?
Prasanna Malaiyandi:
Meaning if I am a privileged user.
Prasanna Malaiyandi:
Am I able to, um.
Prasanna Malaiyandi:
Delete
Prasanna Malaiyandi:
backups, right?
Prasanna Malaiyandi:
Am I able, am I able to prematurely expire backups?
Prasanna Malaiyandi:
Because even with immutable storage, they can have, it can
Prasanna Malaiyandi:
have an expiration date, right?
Prasanna Malaiyandi:
So you can say this, this, these backups are immutable for 90
Prasanna Malaiyandi:
days and then they can expire.
Prasanna Malaiyandi:
But if I am, you know, the super high cloud administrator of this backup system,
Prasanna Malaiyandi:
am I able to then expire backups early?
Prasanna Malaiyandi:
And if you are, then.
Prasanna Malaiyandi:
That is, again, this is why I say immutable backups.
Prasanna Malaiyandi:
Are a spectrum because the, the, the question is, if I am, you know,
Prasanna Malaiyandi:
super high up in the echelon, if I make a phone call to this person, is
Prasanna Malaiyandi:
there a way to expire older backups?
Prasanna Malaiyandi:
I, I would
Prasanna Malaiyandi:
also ask, is there a privilege to count in the operating system that allows
Prasanna Malaiyandi:
me to essentially expire all backups?
Prasanna Malaiyandi:
Yeah.
Prasanna Malaiyandi:
W. Curtis Preston: that, that, that is the case in some backup systems, that even
Prasanna Malaiyandi:
though they're immutable, if I have access to this one privileged account, I can wipe
Prasanna Malaiyandi:
out all backups with a single command.
Prasanna Malaiyandi:
Or reset the system back to a factory state, right?
Prasanna Malaiyandi:
That's sometimes common.
Prasanna Malaiyandi:
W. Curtis Preston: Right.
Prasanna Malaiyandi:
Right.
Prasanna Malaiyandi:
Um, so that, that is another one that would be called immutable.
Prasanna Malaiyandi:
Right.
Prasanna Malaiyandi:
Um, and, and I'm not saying that any of these aren't immutable, I'm just
Prasanna Malaiyandi:
saying that there are a whole bunch of things that are called immutable.
Prasanna Malaiyandi:
right.
Prasanna Malaiyandi:
right.
Prasanna Malaiyandi:
Yeah.
Prasanna Malaiyandi:
W. Curtis Preston: Um, another one, and, uh, and this is
Prasanna Malaiyandi:
where, when, and, and again.
Prasanna Malaiyandi:
This is like good, better, best, right?
Prasanna Malaiyandi:
Anything would be better than backups.
Prasanna Malaiyandi:
Sitting on just an open Windows file system would zero security checks, right?
Prasanna Malaiyandi:
Anything would be better than that,
Prasanna Malaiyandi:
and,
Prasanna Malaiyandi:
I think I have a worse scenario.
Prasanna Malaiyandi:
W. Curtis Preston: okay.
Prasanna Malaiyandi:
A window system open to the internet.
Prasanna Malaiyandi:
W. Curtis Preston: Yeah.
Prasanna Malaiyandi:
Yeah.
Prasanna Malaiyandi:
With, uh, with, uh,
Prasanna Malaiyandi:
RDP
Prasanna Malaiyandi:
W. Curtis Preston: Um, R-D-P-R-D-P enabled.
Prasanna Malaiyandi:
Yeah.
Prasanna Malaiyandi:
Uh, the, um, the next, you know, one of the most surprising things when,
Prasanna Malaiyandi:
when I listened to the episode with Dwayne again was when he said that
Prasanna Malaiyandi:
so many times that they, they, they go in with the default password.
Prasanna Malaiyandi:
And that it works.
Prasanna Malaiyandi:
That's just, that just hurts.
Prasanna Malaiyandi:
The next one is really about something.
Prasanna Malaiyandi:
So not the best, right?
Prasanna Malaiyandi:
But better than the Windows system, right?
Prasanna Malaiyandi:
Which is good.
Prasanna Malaiyandi:
Which is having a server that has been hardened and locked down and secured.
Prasanna Malaiyandi:
Such that you are protecting from most common issues and
Prasanna Malaiyandi:
security vulnerabilities, but it may not still be perfect.
Prasanna Malaiyandi:
And in this example, right, we had Dave Russell and Rick Manover
Prasanna Malaiyandi:
from Veeam over talking about the hardened backup repository.
Prasanna Malaiyandi:
I think is the official name, which is basically a server which they
Prasanna Malaiyandi:
locked down, which allows you to write Veeam backups to it, and they have an
Prasanna Malaiyandi:
immutable flag in the Linux repository such that the backups cannot just be
Prasanna Malaiyandi:
normally deleted, casually deleted.
Prasanna Malaiyandi:
I would say.
Prasanna Malaiyandi:
W. Curtis Preston: Right.
Prasanna Malaiyandi:
Yeah.
Prasanna Malaiyandi:
They can't be deleted by anyone except for root, right, and they can't even
Prasanna Malaiyandi:
be accidentally deleted by root.
Prasanna Malaiyandi:
The root would have a person with root would have to unset
Prasanna Malaiyandi:
the immutability flag on the file and then delete the file, right?
Prasanna Malaiyandi:
So.
Prasanna Malaiyandi:
Uh, this is what, you know, I agree with you that it, it is better than,
Prasanna Malaiyandi:
you know, having a Windows file system.
Prasanna Malaiyandi:
I mean, first off, it's just better in that it's Linux and not Windows.
Prasanna Malaiyandi:
And I don't, I don't just mean that Linux is better than Windows.
Prasanna Malaiyandi:
What I mean is it's something different, right?
Prasanna Malaiyandi:
And then the, uh, 'cause I definitely do not think that Linux is.
Prasanna Malaiyandi:
Perfect.
Prasanna Malaiyandi:
Right.
Prasanna Malaiyandi:
But the, uh, I do still, like the Linux way of everything is off
Prasanna Malaiyandi:
by default versus the Windows way where everything is on by default.
Prasanna Malaiyandi:
Although I think they've gotten better at that over the years.
Prasanna Malaiyandi:
The um, um.
Prasanna Malaiyandi:
It is important to understand that if someone gains root, then they're
Prasanna Malaiyandi:
able to, uh, modify these files.
Prasanna Malaiyandi:
Now they do a really good job.
Prasanna Malaiyandi:
I think, uh, specifically with the Veeam repository, they do a really good job of.
Prasanna Malaiyandi:
Giving you a list of things that you need to do where it's like essentially.
Prasanna Malaiyandi:
But once you've configured this system, you can configure it, for example, to
Prasanna Malaiyandi:
automatically update itself with security patches and that you don't need to be
Prasanna Malaiyandi:
logging into it regularly as route.
Prasanna Malaiyandi:
So you can like super secure that, um, thing.
Prasanna Malaiyandi:
So that if, if anyone ever does log into it as rude, it can set off all
Prasanna Malaiyandi:
sorts of, uh, bells and whistles so that you can know that something.
Prasanna Malaiyandi:
bad is going on.
Prasanna Malaiyandi:
Yeah, and I know we've talked in the past about
Prasanna Malaiyandi:
like patch management, go do your patch management all the time.
Prasanna Malaiyandi:
And having an automated process and keeping it up to date I think
Prasanna Malaiyandi:
is a great feature that that has.
Prasanna Malaiyandi:
W. Curtis Preston: Right.
Prasanna Malaiyandi:
And then I would say, what's the, the next, and I'm, I'm gonna
Prasanna Malaiyandi:
put this up in terms of the best as far as I can see, is that I.
Prasanna Malaiyandi:
Here's an immutable system where backups cannot be modified even by a privileged
Prasanna Malaiyandi:
account, even by somebody with root.
Prasanna Malaiyandi:
Um, the, and there would be two types of systems like that.
Prasanna Malaiyandi:
One would be a backup, a, a storage system that could be on-prem.
Prasanna Malaiyandi:
Another would be something like a cloud system like S3 with the
Prasanna Malaiyandi:
immutable flag turned onto that.
Prasanna Malaiyandi:
And I would, I would actually put.
Prasanna Malaiyandi:
Them in terms of like the, the on-prem system isn't as secure as
Prasanna Malaiyandi:
the cloud system, in my opinion.
Prasanna Malaiyandi:
Why?
Prasanna Malaiyandi:
And why would that be?
Prasanna Malaiyandi:
because you can physically access the system and
Prasanna Malaiyandi:
yank the drives and all the rest.
Prasanna Malaiyandi:
W. Curtis Preston: Yeah.
Prasanna Malaiyandi:
Physical access trumps, all right.
Prasanna Malaiyandi:
So it's funny, a lot of people are like, oh, the cloud is insecure.
Prasanna Malaiyandi:
And I'm like, well, physical system's on-prem or insecure.
Prasanna Malaiyandi:
Right, because I can easily go up to that and I can do bad things to
Prasanna Malaiyandi:
it as long as I know where it's at.
Prasanna Malaiyandi:
Yeah.
Prasanna Malaiyandi:
And even with the physical storage systems, I know at NetApp and even
Prasanna Malaiyandi:
with AWS's Object Lock, right?
Prasanna Malaiyandi:
They do have those two different modes, compliance and governance.
Prasanna Malaiyandi:
I.
Prasanna Malaiyandi:
For their immutability options.
Prasanna Malaiyandi:
One which sort of allows an admin to reset
Prasanna Malaiyandi:
W. Curtis Preston: Mm-Hmm.
Prasanna Malaiyandi:
the flag, right?
Prasanna Malaiyandi:
Like we had talked about earlier.
Prasanna Malaiyandi:
So it's good enough, but there are cases where, yeah, I need to unset it.
Prasanna Malaiyandi:
So they do give you that option.
Prasanna Malaiyandi:
And then the other one is you can't touch it unless you basically, in the case
Prasanna Malaiyandi:
of AWS, blow away your account, right?
Prasanna Malaiyandi:
So it's really nothing you could do as an admin will help you.
Prasanna Malaiyandi:
W. Curtis Preston: Yeah.
Prasanna Malaiyandi:
And you know, just realize that a, what, what I would
Prasanna Malaiyandi:
call a truly immutable system.
Prasanna Malaiyandi:
If you change your mind, you can't change your mind, right?
Prasanna Malaiyandi:
If it's, if it's truly immutable, you should not be able to, to blow it away.
Prasanna Malaiyandi:
Now, I, I still haven't done this experiment, and I, I, I really am gonna
Prasanna Malaiyandi:
do this at one point where I create an AWS account, turn on immutability, put
Prasanna Malaiyandi:
a bunch of stuff in there, and then delete the account and see what happens.
Prasanna Malaiyandi:
Um, I don't, I don't know what happens there, and then stop paying my bill.
Prasanna Malaiyandi:
No, no immutability system.
Prasanna Malaiyandi:
Outlives not paying for your, your, uh, your storage
Prasanna Malaiyandi:
Yeah, they will probably reclaim that
Prasanna Malaiyandi:
data as quickly as they can.
Prasanna Malaiyandi:
Curtis.
Prasanna Malaiyandi:
W. Curtis Preston: do garbage collection.
Prasanna Malaiyandi:
Goodbye Curtis's data.
Prasanna Malaiyandi:
The, I I mean, I will say that in that case it's kind of your own fault, but, um,
Prasanna Malaiyandi:
so this is what we mean when we say that.
Prasanna Malaiyandi:
Immutable backups.
Prasanna Malaiyandi:
You wanna make sure that a black hat, a bad actor that gets into your environment,
Prasanna Malaiyandi:
cannot do damage to the backup system.
Prasanna Malaiyandi:
And so you want to separate.
Prasanna Malaiyandi:
Backups as much as you can from the, the, the bad actor.
Prasanna Malaiyandi:
And the best way to do that is to store them in an immutable system.
Prasanna Malaiyandi:
Just realize that there is a, um, a spectrum of immutability and
Prasanna Malaiyandi:
all of them, they all say their backups are immutable, right?
Prasanna Malaiyandi:
I think it's also important to note that you may not need
Prasanna Malaiyandi:
the best solution for your use case and your needs, so also make sure you
Prasanna Malaiyandi:
understand what you're protecting against.
Prasanna Malaiyandi:
There is gonna be cost, there is gonna be complexity potentially
Prasanna Malaiyandi:
with those solutions, and maybe it isn't needed to go to that extent.
Prasanna Malaiyandi:
And maybe like better is good enough.
Prasanna Malaiyandi:
W. Curtis Preston: Yeah.
Prasanna Malaiyandi:
Never let, never let perfect be the enemy of better.
Prasanna Malaiyandi:
Right?
Prasanna Malaiyandi:
So, you know, you, you could be better.
Prasanna Malaiyandi:
Um, and that's okay.
Prasanna Malaiyandi:
Always strive to be better, but don't.
Prasanna Malaiyandi:
Don't let say, well, th you know, th this immutable system
Prasanna Malaiyandi:
doesn't work in this one scenario.
Prasanna Malaiyandi:
Don't let that prevent you from doing something.
Prasanna Malaiyandi:
Something is always better than nothing, right?
Prasanna Malaiyandi:
One thing I do want to mention is that MFA is a big part of this.
Prasanna Malaiyandi:
Some of these systems that have it's immutable except for the super powerful
Prasanna Malaiyandi:
person, or it's immutable unless you make these 17 phone calls to.
Prasanna Malaiyandi:
Um, you know, you, you've gotta really, really authenticate yourself.
Prasanna Malaiyandi:
I have seen scenarios where companies have had their email system hacked
Prasanna Malaiyandi:
in addition to their backup system being hacked, and the result was that
Prasanna Malaiyandi:
they were able to defeat MFA because they were using email as their MFA.
Prasanna Malaiyandi:
Or the other thing is also phone because now they
Prasanna Malaiyandi:
can impersonate someone, call in, they have all the access to the emails
Prasanna Malaiyandi:
and can spoof like two factor codes.
Prasanna Malaiyandi:
And talk to the person on the other end, and they don't know the difference.
Prasanna Malaiyandi:
W. Curtis Preston: Yeah, the, the, the, um, the ability of AI systems these
Prasanna Malaiyandi:
days to mimic the sound of a, of another person's voice, uh, in, in fact, uh,
Prasanna Malaiyandi:
just as a demonstration, uh, the next.
Prasanna Malaiyandi:
30 seconds of audio is completely fake.
Prasanna Malaiyandi:
How are you doing, Prasanna?
Prasanna Malaiyandi:
Prasanna Malaiyandi: I'm doing fine, Curtis.
Prasanna Malaiyandi:
What are we talking about today?
Prasanna Malaiyandi:
W. Curtis Preston: I think we're talking about immutable backups, and
Prasanna Malaiyandi:
I hear that they are on a Spectrum.
Prasanna Malaiyandi:
You mean like IBM Spectrum Protect?
Prasanna Malaiyandi:
W. Curtis Preston: No, that's a backup product.
Prasanna Malaiyandi:
You mean the Irvine Spectrum?
Prasanna Malaiyandi:
W. Curtis Preston: No, that's a mall.
Prasanna Malaiyandi:
I mean that there are many things that are called immutable, and
Prasanna Malaiyandi:
they all have different levels of protection against attacks.
Prasanna Malaiyandi:
Nice talking with you, fake Prasanna.
Prasanna Malaiyandi:
You too, fake Curtis.
Prasanna Malaiyandi:
W. Curtis Preston: So that was neither me nor persona talking.
Prasanna Malaiyandi:
For the record, that was a little tool that we use.
Prasanna Malaiyandi:
That is very useful in the right scenario, but very dangerous in the wrong scenario.
Prasanna Malaiyandi:
Um, yeah, you, you really have to think about that, right?
Prasanna Malaiyandi:
When you, when you put all these layers, if you have all these layers of
Prasanna Malaiyandi:
security between you and the ability to delete your quote immutable data, think
Prasanna Malaiyandi:
about all of the modern things that people are able to do, um, before, um.
Prasanna Malaiyandi:
You know, you subtle with one of those systems.
Prasanna Malaiyandi:
And again, don't let perfect be the enemy of the good.
Prasanna Malaiyandi:
But, but, but just think about all the things that are possible these days.
Prasanna Malaiyandi:
Uh, you know, they can fake video, they can fake audio.
Prasanna Malaiyandi:
Uh, they can certainly fake email and phones, right?
Prasanna Malaiyandi:
SMS is a horrible way to do things, right.
Prasanna Malaiyandi:
But, um, um.
Prasanna Malaiyandi:
With that, that's our discussion on immutable backups.
Prasanna Malaiyandi:
I hope you enjoyed a persona.
Prasanna Malaiyandi:
No, this was a good topic.
Prasanna Malaiyandi:
I like immutable backups.
Prasanna Malaiyandi:
W. Curtis Preston: Yeah.
Prasanna Malaiyandi:
Immutable backups.
Prasanna Malaiyandi:
Good.
Prasanna Malaiyandi:
And, uh, hope, uh, you enjoyed that as a listener.
Prasanna Malaiyandi:
Hope you learned a thing or two.
Prasanna Malaiyandi:
And with that, that's a wrap.
Listen On
