StorageCraft Outage: Lessons from a Cloud Backup Disaster
In this episode, we examine the StorageCraft outage that erased customer backup data during a botched cloud migration. We compare StorageCraft's response to Carbonite's in a similar incident and discuss the critical lessons for backup vendors and customers. Learn the importance of meticulous migration processes, potential backup resiliency strategies, and what to do if your cloud backups disappear. Don't miss these vital insights to avoid cloud backup disasters and ensure your data is always recoverable.
Stories covering this outage:
https://www.crn.com/slide-shows/storage/arcserve-ceo-storagecraft-backup-data-loss-not-acceptable
https://www.reddit.com/r/msp/comments/tgggey/just_got_a_call_from_storagecraft_cloud_data_is/
https://blocksandfiles.com/2022/03/22/arcserve-storagecraft-operation-has-lost-customer-data/
https://www.channele2e.com/news/arcserve-storagecraft-draas-suffers-cloud-data-protection-issues
Speaker:
Our latest in our cloud disaster series is the 2014 StorageCraft outage
Speaker:
that resulted in loss customer data.
Speaker:
A server decommissioning mistake during a cloud migration resulted in metadata
Speaker:
loss and StorageCraft scrambling to help customers re-seed their backups.
Speaker:
This may remind you a bit of what happened to carbonate, but this
Speaker:
is very different because here the CEO took full responsibility.
Speaker:
And the company I think went above and beyond to help customers
Speaker:
re-seed as quickly as possible.
Speaker:
Prasanna and I analyze what went wrong, the lessons learned for
Speaker:
backup providers and customers alike.
Speaker:
And what steps you should take if you ever find yourself.
Speaker:
With your backups lost in space.
Speaker:
We'll discuss the importance of migration processes, backup resiliency
Speaker:
measures and why trust is so critical in the backup and recovery business.
Speaker:
If you're not familiar with me, I'm w Curtis Preston and there's a reason
Speaker:
I'm so passionate about this subject.
Speaker:
During my first job as a backup admin, my company lost a huge
Speaker:
database and I couldn't restore it.
Speaker:
Since that nightmare, I've dedicated my career to making sure that
Speaker:
will never again happen to me or anyone who will listen to me.
Speaker:
We turn unappreciated backup admins into cyber recovery heroes.
Speaker:
This is the backup wrap-up.
Speaker:
W. Curtis Preston: Welcome to the show.
Speaker:
Hi, I'm w Curtis Preston, AKA, Mr.
Speaker:
Backup, and I have with me my career reverse trauma specialist
Speaker:
Prasanna Malaiyandi, how's it going?
Speaker:
Prasanna?
Prasanna Malaiyandi:
I am good, Curtis, how you been?
Prasanna Malaiyandi:
Are you doing a little better today?
Prasanna Malaiyandi:
W. Curtis Preston: So for the first time in a long time, I
Prasanna Malaiyandi:
find myself at the keyboard.
Prasanna Malaiyandi:
Um, you know, in a production environment administering a
Prasanna Malaiyandi:
very big net backup environment.
Prasanna Malaiyandi:
I haven't been the guy on the keyboard in a while, and it, it was.
Prasanna Malaiyandi:
Fun.
Prasanna Malaiyandi:
Um, and doing, you know, doing performance testing and all, you know, normally
Prasanna Malaiyandi:
it's been somebody else at the keyboard and I'm like advising, but this was
Prasanna Malaiyandi:
me in a data center all by myself.
Prasanna Malaiyandi:
Um, you
Prasanna Malaiyandi:
you back 30 years.
Prasanna Malaiyandi:
W. Curtis Preston: was interesting.
Prasanna Malaiyandi:
Uh, it's, it's not quite 30 years, but, but yeah, it's been, uh, it's
Prasanna Malaiyandi:
been a while since I've been the guy and, you know, and standing in what
Prasanna Malaiyandi:
we're going to call a data center.
Prasanna Malaiyandi:
It's really just a big storage closet that happens to have a, a server rack.
Prasanna Malaiyandi:
We're doing a, a backup of this, uh, customer's environment and, uh,
Prasanna Malaiyandi:
let's just say it's, it's not small.
Prasanna Malaiyandi:
It was, it was 400 terabytes of data sitting on a couple of little NAS boxes.
Prasanna Malaiyandi:
Um, so it's been, it has been interesting.
Prasanna Malaiyandi:
I will just say that, um.
Prasanna Malaiyandi:
Uh, standing in a very noisy server room talking to somebody on the phone, I could
Prasanna Malaiyandi:
barely hear them on my, you know, on
Prasanna Malaiyandi:
just be lucky.
Prasanna Malaiyandi:
You at least have cell phones now.
Prasanna Malaiyandi:
W. Curtis Preston: I, it's nice that I have cell phones, cell phones and, and
Prasanna Malaiyandi:
chat and screen connect so that like.
Prasanna Malaiyandi:
You know, they can see what I'm seeing.
Prasanna Malaiyandi:
And, um, you know, all of that.
Prasanna Malaiyandi:
And, um, yeah, it's been interesting.
Prasanna Malaiyandi:
But, um, speaking of interesting backup scenarios, we're continuing our series on
Prasanna Malaiyandi:
cloud disasters and we have another one.
Prasanna Malaiyandi:
This is very much in the vein of what happened to Carbonite.
Prasanna Malaiyandi:
It's very similar and yet different.
Prasanna Malaiyandi:
So,
Prasanna Malaiyandi:
Same.
Prasanna Malaiyandi:
Same, but
Prasanna Malaiyandi:
W. Curtis Preston: wanna do a quick summary of what happened to Carbonite?
Prasanna Malaiyandi:
So with Carbonite, what ended up happening was they had
Prasanna Malaiyandi:
their own storage that they were using.
Prasanna Malaiyandi:
Right.
Prasanna Malaiyandi:
Uh, and people would upload their backups to
Prasanna Malaiyandi:
it, and they had a double disc failure that took
Prasanna Malaiyandi:
down their storage array, and they lost all their data.
Prasanna Malaiyandi:
And they basically told customers, yeah, don't worry about it.
Prasanna Malaiyandi:
We're just gonna re-upload and do another backup and get all your data back up.
Prasanna Malaiyandi:
And during that time, there were something like, I think it was 35 customers, 34
Prasanna Malaiyandi:
customers I wanna say was the number, um, who ended up having another fault
Prasanna Malaiyandi:
in their production data, right.
Prasanna Malaiyandi:
On their laptop, didn't.
Prasanna Malaiyandi:
Crash or whatever else happened, and they were not able to restore their
Prasanna Malaiyandi:
data because there were no backups left.
Prasanna Malaiyandi:
W. Curtis Preston: Yeah, the, I I, I think the part of their story that
Prasanna Malaiyandi:
bothers me the most is that, you know, the company's reaction was our
Prasanna Malaiyandi:
stupid storage array vendor, right?
Prasanna Malaiyandi:
They blamed it on their storage array vendor and they sued their vendor.
Prasanna Malaiyandi:
Um, and so while this story.
Prasanna Malaiyandi:
Is gonna start out sounding kind of similar.
Prasanna Malaiyandi:
Uh, it, it's very similar in terms of the impact to the customers, but the response
Prasanna Malaiyandi:
from the company is completely different.
Prasanna Malaiyandi:
And so I feel completely different about the company, even though
Prasanna Malaiyandi:
the exact same thing happened on the, on the customer side.
Prasanna Malaiyandi:
So first off, we're talking about, storage craft, which.
Prasanna Malaiyandi:
Is a company that had been around for a while and then had been acquired in
Prasanna Malaiyandi:
2021 by Arcserve, which is a company that has been around a long while.
Prasanna Malaiyandi:
Arcserve goes all the way back to my early days.
Prasanna Malaiyandi:
Arcserve was actually the first backup product I ever tested as
Prasanna Malaiyandi:
a, to, to potentially buy it.
Prasanna Malaiyandi:
And running it on NetWare and that boy that brings me back.
Prasanna Malaiyandi:
But, uh, so Archer has been around a long time and, um, you know, it's
Prasanna Malaiyandi:
been sold and, and, and I don't know what, you know, spun back out
Prasanna Malaiyandi:
equity or
Prasanna Malaiyandi:
whatever.
Prasanna Malaiyandi:
Yeah.
Prasanna Malaiyandi:
W. Curtis Preston: Yeah.
Prasanna Malaiyandi:
A private equity firm.
Prasanna Malaiyandi:
They got bought by ca.
Prasanna Malaiyandi:
They got spun back out with a private equity firm and then, uh, they
Prasanna Malaiyandi:
acquired or they merged, Arcserve merged with StorageCraft back in 2021
Prasanna Malaiyandi:
and then this story unfortunately happens not long after that.
Prasanna Malaiyandi:
I think it was about five months after the acquisition what ended up happening.
Prasanna Malaiyandi:
Prasanna.
Prasanna Malaiyandi:
Yeah.
Prasanna Malaiyandi:
So like you had alluded to earlier, just like Carbonite, uh,
Prasanna Malaiyandi:
they lost customer data, right?
Prasanna Malaiyandi:
Um, but the way it happened was slightly different.
Prasanna Malaiyandi:
So as part of this, they were moving from their own facilities
Prasanna Malaiyandi:
and migrating to Google Cloud.
Prasanna Malaiyandi:
Um, and they accidentally decommissioned a server before
Prasanna Malaiyandi:
it was already up in the cloud.
Prasanna Malaiyandi:
And it wasn't just any server.
Prasanna Malaiyandi:
It was a server that contained metadata about the backups.
Prasanna Malaiyandi:
W. Curtis Preston: Yeah,
Prasanna Malaiyandi:
Yeah,
Prasanna Malaiyandi:
W. Curtis Preston: metadata is important,
Prasanna Malaiyandi:
Yeah, so they basically, um, had the same situation as
Prasanna Malaiyandi:
what we had talked about with Carbonite, where the customers lost their data.
Prasanna Malaiyandi:
Now, I think unlike Carbonite, I think that, like you said, they did
Prasanna Malaiyandi:
a much better job of accepting blame.
Prasanna Malaiyandi:
And trying to get their customers or their end users back up and
Prasanna Malaiyandi:
running as quickly as possible.
Prasanna Malaiyandi:
Doing things like receding and really going the extra distance to get them
Prasanna Malaiyandi:
up and running as quickly as possible, including things like, Hey, if you
Prasanna Malaiyandi:
have more than three terabytes, we're gonna ship you a drive so you can
Prasanna Malaiyandi:
upload to the drive and then send it to help speed up the process.
Prasanna Malaiyandi:
W. Curtis Preston: Yeah, so for those not familiar with this process,
Prasanna Malaiyandi:
when you are a cloud backup vendor, the, you know, the hardest part
Prasanna Malaiyandi:
is that initial backup, right?
Prasanna Malaiyandi:
You know, it's one
Prasanna Malaiyandi:
So
Prasanna Malaiyandi:
fast.
Prasanna Malaiyandi:
W. Curtis Preston: Yeah.
Prasanna Malaiyandi:
Yeah.
Prasanna Malaiyandi:
Damn.
Prasanna Malaiyandi:
Physics.
Prasanna Malaiyandi:
Uh, yeah.
Prasanna Malaiyandi:
So when you, you know, if, if you're back, you can back up.
Prasanna Malaiyandi:
I mean, you know, I remember, uh, you know, when I worked
Prasanna Malaiyandi:
at Druva, we had really, you know, multi petabyte customers.
Prasanna Malaiyandi:
You can back up that amount of data over the internet, but that
Prasanna Malaiyandi:
first backup is gonna be something.
Prasanna Malaiyandi:
And that's why we used the snowball edge.
Prasanna Malaiyandi:
which was a, an appliance that, that you would ship to them and, you know,
Prasanna Malaiyandi:
it already had the technology on there.
Prasanna Malaiyandi:
And then they would back up to that and then that would ship and it was
Prasanna Malaiyandi:
just mu, you know, basically a very fancy sneaker net was the way to
Prasanna Malaiyandi:
get the very first copy up there.
Prasanna Malaiyandi:
And you needed an, you needed one of those snowball edges for I think about
Prasanna Malaiyandi:
every 75 terabytes, something like that.
Prasanna Malaiyandi:
And, um, then.
Prasanna Malaiyandi:
And so that's what they're saying here.
Prasanna Malaiyandi:
And that by the way, that is referred to as the seed for those that don't
Prasanna Malaiyandi:
live in that world, that initial backup is referred to the seed.
Prasanna Malaiyandi:
So when you read the story, you're right, they did accept
Prasanna Malaiyandi:
responsibility and they said
Prasanna Malaiyandi:
the first thing that we need to get going is that seed.
Prasanna Malaiyandi:
And so, you know, you were talking about the, one of the way, just
Prasanna Malaiyandi:
like I said, the, the way, uh, my former employer would do it was they
Prasanna Malaiyandi:
would ship, uh, a snowball edge.
Prasanna Malaiyandi:
They're shipping whatever they need to ship to these customers
Prasanna Malaiyandi:
in order to make them whole.
Prasanna Malaiyandi:
One additional wrinkle, I think of this was that this was
Prasanna Malaiyandi:
specifically for their MSP customers.
Prasanna Malaiyandi:
Um, and I don't know what percentage of their business, uh, is via MSPs, but
Prasanna Malaiyandi:
that does, that does make an additional wrinkle in that not only are they
Prasanna Malaiyandi:
impacting their customer, which is the Ms.
Prasanna Malaiyandi:
P, they're impacting the end customer, which is the MSPs customer.
Prasanna Malaiyandi:
Um.
Prasanna Malaiyandi:
But again, I, you know, I want to, uh, you know, give a shout out,
Prasanna Malaiyandi:
um, to the CEO, Brandon Lacey.
Prasanna Malaiyandi:
I like that he, he doesn't try to pass the blame.
Prasanna Malaiyandi:
He doesn't try to, you know, I, I don't know.
Prasanna Malaiyandi:
I.
Prasanna Malaiyandi:
Maybe it's because he didn't have a choice, right?
Prasanna Malaiyandi:
You know, when you know the other vendor, they're saying, Hey, it was our, it
Prasanna Malaiyandi:
was our storage array, uh, that failed.
Prasanna Malaiyandi:
And that's why, you know, that's why our backups died in this case.
Prasanna Malaiyandi:
They didn't really add anything to blame it on other than, dang
Prasanna Malaiyandi:
it, why is my fingers here?
Prasanna Malaiyandi:
They didn't really have anything other to blame it on other than human failure.
Prasanna Malaiyandi:
Because it was a human that prematurely decommissioned a server before
Prasanna Malaiyandi:
that server had been migrated.
Prasanna Malaiyandi:
Yeah.
Prasanna Malaiyandi:
And that's just sort of a breakdown of processes, right?
Prasanna Malaiyandi:
It's not a technology issue.
Prasanna Malaiyandi:
Right?
Prasanna Malaiyandi:
And I'm sure now they have a process in place where they're like, Hey, make
Prasanna Malaiyandi:
sure that the data validate that the server is actually up in wherever it
Prasanna Malaiyandi:
needs to be before you decommission.
Prasanna Malaiyandi:
Because.
Prasanna Malaiyandi:
W. Curtis Preston: Yeah, that is, that is probably the
Prasanna Malaiyandi:
biggest disappointment, right?
Prasanna Malaiyandi:
Is you would think you're doing this like major migration.
Prasanna Malaiyandi:
I remember there was Datto, they did an exfiltration out of, um, Amazon and they
Prasanna Malaiyandi:
built their own private data center that was a giant, and they said they believed
Prasanna Malaiyandi:
at the time, and I believe this was Datto, they believed at the time it was
Prasanna Malaiyandi:
the single largest exfiltration project ever to leave en mass, uh, from AWS.
Prasanna Malaiyandi:
And it apparently went flawlessly.
Prasanna Malaiyandi:
You, you've got to, you know, when you're doing a migration, it's like,
Prasanna Malaiyandi:
you've gotta have that check, right?
Prasanna Malaiyandi:
You've gotta have that check.
Prasanna Malaiyandi:
We're gonna kill the thing.
Prasanna Malaiyandi:
In fact, I wouldn't even kill the thing,
Prasanna Malaiyandi:
You would just sort of, yeah.
Prasanna Malaiyandi:
W. Curtis Preston: I would, I would, uh, turn it off,
Prasanna Malaiyandi:
Yep.
Prasanna Malaiyandi:
W. Curtis Preston: right?
Prasanna Malaiyandi:
I, I would, in fact, I remember, um.
Prasanna Malaiyandi:
I remember the very first migration that I had anything to do with, and
Prasanna Malaiyandi:
I'm gonna, for those of you for the fellow gray hairs in the crowd, we were
Prasanna Malaiyandi:
migrating off, uh, an at and T three B two 4,000, which was the at t's attempt
Prasanna Malaiyandi:
at a multi-processing server, right?
Prasanna Malaiyandi:
So the three B two 1000 was the.
Prasanna Malaiyandi:
First computer ever built specifically for Unix.
Prasanna Malaiyandi:
Okay.
Prasanna Malaiyandi:
I love how you just have like these model numbers.
Prasanna Malaiyandi:
Just like right off the tongue, it just rolls right off.
Prasanna Malaiyandi:
You're like, oh yeah, everyone should know about the 3D two.
Prasanna Malaiyandi:
It's like R 2D two.
Prasanna Malaiyandi:
you might as well call it that.
Prasanna Malaiyandi:
W. Curtis Preston: Yeah.
Prasanna Malaiyandi:
What's that?
Prasanna Malaiyandi:
You might as well call it R 2D two.
Prasanna Malaiyandi:
W. Curtis Preston: Yeah, so the three B two 1000 was the first, you know, 'cause
Prasanna Malaiyandi:
I think it was the PDP 11 was the one that, uh, that Unix initially ran on.
Prasanna Malaiyandi:
But the three B two was the first computer specifically designed to run.
Prasanna Malaiyandi:
Unix and it ran, uh, system five and, um, you know, directly from
Prasanna Malaiyandi:
Bell Labs, right at and t and the three B two 4,000 was, you know, they
Prasanna Malaiyandi:
attempted a multi-processing machine and it, it just, it wasn't very good.
Prasanna Malaiyandi:
And so we were trying to migrate off of that, and I remember that.
Prasanna Malaiyandi:
Basically what we would do is we would just kill things as we
Prasanna Malaiyandi:
believed we had migrated it off.
Prasanna Malaiyandi:
We would, we would turn off a database.
Prasanna Malaiyandi:
We would turn off a server.
Prasanna Malaiyandi:
see what broke.
Prasanna Malaiyandi:
W. Curtis Preston: That's the, and well, yeah, and then see if anything broke.
Prasanna Malaiyandi:
Right.
Prasanna Malaiyandi:
You don't, I, I don't know.
Prasanna Malaiyandi:
I, I guess that's the one surprise here, is that they, they decommission
Prasanna Malaiyandi:
the server to the point of actually, you know, deleting it somehow.
Prasanna Malaiyandi:
Or, or, I, I know they talk about decommissioning.
Prasanna Malaiyandi:
I do wonder if it's sort of, um, because it's metadata, I wonder if it's like
Prasanna Malaiyandi:
a distributed system and maybe they killed the node, uh, without, and so
Prasanna Malaiyandi:
since it didn't have access to the metadata, it's sort of like maybe
Prasanna Malaiyandi:
they weren't expecting that to happen.
Prasanna Malaiyandi:
Right?
Prasanna Malaiyandi:
So even though they say
Prasanna Malaiyandi:
W. Curtis Preston: Yeah, I, yeah.
Prasanna Malaiyandi:
I don't know the details.
Prasanna Malaiyandi:
So.
Prasanna Malaiyandi:
W. Curtis Preston: Yeah, we don't know the details.
Prasanna Malaiyandi:
We're Monday morning quarterbacking, et cetera.
Prasanna Malaiyandi:
Um, but let's, but let's just talk about it.
Prasanna Malaiyandi:
When you are doing a migration, you don't kill the thing.
Prasanna Malaiyandi:
You don't, you don't kill the old one.
Prasanna Malaiyandi:
You turn off the old thing, right?
Prasanna Malaiyandi:
You turn it off and then you wait a while before you, uh, actually
Prasanna Malaiyandi:
redeploy it, delete it, whatever.
Prasanna Malaiyandi:
And then I would.
Prasanna Malaiyandi:
Then I, you know, I, I find myself wondering, well, I, I guess there wasn't,
Prasanna Malaiyandi:
there wasn't a backup of that thing.
Prasanna Malaiyandi:
There wasn't a backup of the backup, which is not that uncommon
Prasanna Malaiyandi:
Yep.
Prasanna Malaiyandi:
Backup
Prasanna Malaiyandi:
W. Curtis Preston: uh, backup.
Prasanna Malaiyandi:
Backup.
Prasanna Malaiyandi:
I.
Prasanna Malaiyandi:
Services.
Prasanna Malaiyandi:
Right?
Prasanna Malaiyandi:
Um, and as a result, when that system was deleted, it, there was no backup
Prasanna Malaiyandi:
of that backup, which I think many people might be surprised by that.
Prasanna Malaiyandi:
What do you think?
Prasanna Malaiyandi:
No.
Prasanna Malaiyandi:
Uh, I think people will be surprised because they think that backup
Prasanna Malaiyandi:
systems are highly resilient.
Prasanna Malaiyandi:
And of course someone's gonna do a backup of a backup, but you have to also realize
Prasanna Malaiyandi:
that there's additional costs associated with doing a backup of a backup.
Prasanna Malaiyandi:
And.
Prasanna Malaiyandi:
Usually backup systems, people are trying to cut costs, right?
Prasanna Malaiyandi:
Because this is like the tertiary copy.
Prasanna Malaiyandi:
And so having high availability or a replica copy of your backup system, unless
Prasanna Malaiyandi:
it's critical, which a lot of systems are, and people do replicate their backup
Prasanna Malaiyandi:
systems, but not everyone could afford to do that, and especially given that this
Prasanna Malaiyandi:
was for their MSP customers who are also trying to make money and add value, right?
Prasanna Malaiyandi:
They're looking at probably a.
Prasanna Malaiyandi:
Reasonable offering, right,
Prasanna Malaiyandi:
uh, without all the bells and whistles.
Prasanna Malaiyandi:
W. Curtis Preston: Yeah.
Prasanna Malaiyandi:
And in a world where services are often purchased on price.
Prasanna Malaiyandi:
Not necessarily alone, but where price is the primary motivator For many
Prasanna Malaiyandi:
customers, it's, this is one of those things where no one ever bought Salesforce
Prasanna Malaiyandi:
because they had a really good backup.
Prasanna Malaiyandi:
No one ever bought Oracle because it had a really good backup.
Prasanna Malaiyandi:
They bought Oracle because it had really good features and did
Prasanna Malaiyandi:
the thing they needed it to do.
Prasanna Malaiyandi:
Nobody's nobody, there's, there's nowhere, no ad in the world.
Prasanna Malaiyandi:
Said, buy Oracle, because our backup is really good.
Prasanna Malaiyandi:
Right?
Prasanna Malaiyandi:
And so the same thing is true here.
Prasanna Malaiyandi:
Buy our backup service because we also have a tertiary copy of
Prasanna Malaiyandi:
your data just in case we screw up the copy that you gave us.
Prasanna Malaiyandi:
Nobody's gonna even know that.
Prasanna Malaiyandi:
Um, and they're, and they're cer, they certainly don't want to pay for it.
Prasanna Malaiyandi:
All they see is that this one is more expensive than that one.
Prasanna Malaiyandi:
Yeah.
Prasanna Malaiyandi:
The one thing I am a little surprised about though is given this is a backup
Prasanna Malaiyandi:
system, and it looks like they did have their metadata and data split apart just
Prasanna Malaiyandi:
based on what we read in the articles, I'm surprised they didn't have a self
Prasanna Malaiyandi:
describing format for the data itself.
Prasanna Malaiyandi:
I know?
Prasanna Malaiyandi:
there are other systems which tend to do that as sort of a fail safe, so I'm not
Prasanna Malaiyandi:
necessarily going to create a copy of the data, but if I, my, something happens,
Prasanna Malaiyandi:
my metadata or there's some corruption or whatever else, I should at least be
Prasanna Malaiyandi:
able to build back all the metadata based on the data that I still have access to.
Prasanna Malaiyandi:
I.
Prasanna Malaiyandi:
W. Curtis Preston: Yeah, that's a good point.
Prasanna Malaiyandi:
And, and that is a, another really good feature to have in a backup system
Prasanna Malaiyandi:
that if the, basically the backup catalog, the image database, the, you
Prasanna Malaiyandi:
know, the database of all the backups, if that gets messed up, it would be
Prasanna Malaiyandi:
really nice if the, if the, uh, you know, if you could rebuild that.
Prasanna Malaiyandi:
And I remember back when I was doing RFPs and the, one of the questions
Prasanna Malaiyandi:
we asked was if the backup catalog, um, you know, gets corrupt, can,
Prasanna Malaiyandi:
can we rebuild it given the backups?
Prasanna Malaiyandi:
And I do remember at least one major vendor.
Prasanna Malaiyandi:
I won't call 'em out by name 'cause it might not still be the
Prasanna Malaiyandi:
answer, but their answer was.
Prasanna Malaiyandi:
Um, you know what, they, literally, the phrase was, our backup format
Prasanna Malaiyandi:
is so proprietary that even we can't read it if we don't know what's on it.
Prasanna Malaiyandi:
And they, and they saw that as like a, like a feature.
Prasanna Malaiyandi:
Uh, by the way, it turned out, interestingly enough, it turned out
Prasanna Malaiyandi:
that, uh, it wasn't that proprietary and.
Prasanna Malaiyandi:
Other vendors have figured out how to read that backup date.
Prasanna Malaiyandi:
Yeah.
Prasanna Malaiyandi:
As long as it's not, um, encrypted.
Prasanna Malaiyandi:
Right.
Prasanna Malaiyandi:
That that's a whole other, whole other thing.
Prasanna Malaiyandi:
So I don't think we've ever talked about
Prasanna Malaiyandi:
this on the podcast before.
Prasanna Malaiyandi:
And so is this something, I know we've talked previously about, Hey, here are
Prasanna Malaiyandi:
some questions you should be asking your backup vendor when trying to figure
Prasanna Malaiyandi:
out like what their solution does.
Prasanna Malaiyandi:
Do you think this is worthwhile to throw into that list of questions?
Prasanna Malaiyandi:
W. Curtis Preston: Of, do you have a backup of the backup?
Prasanna Malaiyandi:
Either a backup of the backup or if I lose
Prasanna Malaiyandi:
part of my system, like my catalog, can I still recover my data?
Prasanna Malaiyandi:
W. Curtis Preston: I, I do think that's a perfectly valid question to ask.
Prasanna Malaiyandi:
Right.
Prasanna Malaiyandi:
And I think the other question is perfectly valid as well of
Prasanna Malaiyandi:
if, what protections do you, I would point at this story, what
Prasanna Malaiyandi:
protections do you have against this happening in your environment?
Prasanna Malaiyandi:
Right.
Prasanna Malaiyandi:
Um,
Prasanna Malaiyandi:
because it may not be malicious.
Prasanna Malaiyandi:
It's just accidental things happen.
Prasanna Malaiyandi:
It's all software.
Prasanna Malaiyandi:
People are writing software, right?
Prasanna Malaiyandi:
W. Curtis Preston: Yeah, the, you know, so this, remind this, this story to
Prasanna Malaiyandi:
me is a, is a combination of the KPMG story and the Carbonite story, right?
Prasanna Malaiyandi:
Because like the KPMG story, and by the way, we have another episode on that.
Prasanna Malaiyandi:
Please feel free to check that out.
Prasanna Malaiyandi:
The KPMG story is one of failure of process as well, where.
Prasanna Malaiyandi:
You'd like to think that at a, at a company that size and when you're
Prasanna Malaiyandi:
doing something of this magnitude, I, you know, I, when I think back
Prasanna Malaiyandi:
on that story, I don't think they.
Prasanna Malaiyandi:
Thought this was of the, the, we're, I'm just trying to fix one
Prasanna Malaiyandi:
person's, one person's account.
Prasanna Malaiyandi:
They didn't think that what they were touching was a tool in this
Prasanna Malaiyandi:
case, uh, retention policies.
Prasanna Malaiyandi:
They were touching a tool that could wipe out everybody's data.
Prasanna Malaiyandi:
They, they
Prasanna Malaiyandi:
Prasanna Malaiyandi: Hundred 40,000 people.
Prasanna Malaiyandi:
W. Curtis Preston: 140,000 people's data with no backup.
Prasanna Malaiyandi:
Uh, that's a great, that's a great story.
Prasanna Malaiyandi:
Check that one out.
Prasanna Malaiyandi:
But I, the, the big thing with processes, you've, you've got to have
Prasanna Malaiyandi:
a very, uh, you know, the bigger the project, the, the bigger the process.
Prasanna Malaiyandi:
And I do remember, I.
Prasanna Malaiyandi:
When the, the biggest one I remember being involved in it was, it, was, it, it wasn't
Prasanna Malaiyandi:
a migration, it was a physical move.
Prasanna Malaiyandi:
We moved, a bunch of servers from one data center to another data center,
Prasanna Malaiyandi:
We had built a nice new big data center and we're gonna move the servers from the
Prasanna Malaiyandi:
old data center to the new data center.
Prasanna Malaiyandi:
And we had a massive process and we had tested it from one end to the other,
Prasanna Malaiyandi:
and we outsourced whatever we could.
Prasanna Malaiyandi:
So like we hired.
Prasanna Malaiyandi:
A professional moving company to move the servers we hired, uh, and paid extra
Prasanna Malaiyandi:
to have HP come in and, and, you know, unplug the servers and go ominous dominus,
Prasanna Malaiyandi:
you know, you may touch this server.
Prasanna Malaiyandi:
Right.
Prasanna Malaiyandi:
Um, and then, and then it got handed to the movers and stuff
Prasanna Malaiyandi:
and we had this big process.
Prasanna Malaiyandi:
Um, but there was one major failing and that was from the old
Prasanna Malaiyandi:
server to the new server room.
Prasanna Malaiyandi:
The new server room had, um, Twistlock.
Prasanna Malaiyandi:
You know, plugs under, underneath the floor.
Prasanna Malaiyandi:
And the old server room had the old standard, uh, north
Prasanna Malaiyandi:
American, uh, two-prong plugs.
Prasanna Malaiyandi:
And, uh, that was the, that was one day where I truly saved the day because
Prasanna Malaiyandi:
I had, uh, oh, and the electricians, that would be the ones to help us.
Prasanna Malaiyandi:
Were busy at the CEO's house, putting up his Christmas lights.
Prasanna Malaiyandi:
Um, and so, and so they didn't think we'd need electricians during our big move.
Prasanna Malaiyandi:
And I knew how to fix this problem, right?
Prasanna Malaiyandi:
I, I'm like, look, we can do this.
Prasanna Malaiyandi:
I just need to go to a gray bar, uh, electrical store.
Prasanna Malaiyandi:
And there's one down the road and it was closing in.
Prasanna Malaiyandi:
20 minutes, uh, by the time we realized what we needed to do, and I, I got in my
Prasanna Malaiyandi:
car with, with my boss and, and he looked at me and he says, I will pay the ticket.
Prasanna Malaiyandi:
Just get us there.
Prasanna Malaiyandi:
And we drove.
Prasanna Malaiyandi:
I drove and it was raining.
Prasanna Malaiyandi:
It was, I mean, there was water everywhere.
Prasanna Malaiyandi:
I'm driving to this gray bar and I've got it.
Prasanna Malaiyandi:
It's, it's, it's closing in 20 minutes and it's 15 minutes away.
Prasanna Malaiyandi:
Right.
Prasanna Malaiyandi:
I don't have time to play.
Prasanna Malaiyandi:
And I remember hydroplaning going around a big semi, and I remember
Prasanna Malaiyandi:
like, you know, swerving around a corner and all this stuff.
Prasanna Malaiyandi:
And I remember that when we got there, there's that scene in, uh, planes, trains,
Prasanna Malaiyandi:
and Automobiles where, where Steve, Steve Martin's fingers are stuck in the dash.
Prasanna Malaiyandi:
Uh, that was the way, uh, Lou was when we, when we got there.
Prasanna Malaiyandi:
But we got, we walked up to the door and it was five minutes prior to the,
Prasanna Malaiyandi:
you know, prior to the, the closing.
Prasanna Malaiyandi:
They were, the door was locked.
Prasanna Malaiyandi:
And, uh, and, and they, they pointed, they pointed at her watch.
Prasanna Malaiyandi:
They're like, they're pointing at their watch as a time.
Prasanna Malaiyandi:
And I pointed up at the clock on their wall and I was like,
Prasanna Malaiyandi:
no, it's not one o'clock yet.
Prasanna Malaiyandi:
Open your damn door.
Prasanna Malaiyandi:
And then we, you know, uh, and then I think, I think I like, I was like, please,
Prasanna Malaiyandi:
please, you know, please let me in.
Prasanna Malaiyandi:
And we got it.
Prasanna Malaiyandi:
And we bought all the stuff.
Prasanna Malaiyandi:
And I, I non.
Prasanna Malaiyandi:
Traded non whatever electrician, but I was trained as one in the Navy, um, which at
Prasanna Malaiyandi:
that time was only like five years prior.
Prasanna Malaiyandi:
Um, and then we wired adapters from.
Prasanna Malaiyandi:
A to B and save the day anyway.
Prasanna Malaiyandi:
Yeah, you gotta test your process and you gotta be ready
Prasanna Malaiyandi:
for failures in your process.
Prasanna Malaiyandi:
But I, I think that, you know, so the two big this, this one I feel very different.
Prasanna Malaiyandi:
Like I said in the beginning, I feel very different about
Prasanna Malaiyandi:
this one versus the carbonite
Prasanna Malaiyandi:
but why
Prasanna Malaiyandi:
W. Curtis Preston: because they didn't try to, they didn't try to pass the buck.
Prasanna Malaiyandi:
is that really the only difference in your mind is,
Prasanna Malaiyandi:
well, I think also in the Carbonite case, the only reason we heard about
Prasanna Malaiyandi:
it is when they try to sue the vendor.
Prasanna Malaiyandi:
W. Curtis Preston: Yeah.
Prasanna Malaiyandi:
Um, this.
Prasanna Malaiyandi:
Yeah, that's a good question.
Prasanna Malaiyandi:
If, would we have ever heard about the story if they, I, I don't know.
Prasanna Malaiyandi:
I, I, I'd like to think that it would still have made news, but all we
Prasanna Malaiyandi:
heard about was the fact that they store, they sued their storage vendor
Prasanna Malaiyandi:
in this case.
Prasanna Malaiyandi:
The article, right.
Prasanna Malaiyandi:
Um.
Prasanna Malaiyandi:
It starts out with this quote from the CEO, right?
Prasanna Malaiyandi:
In terms of magnitude is the only thing I'm thinking about waking
Prasanna Malaiyandi:
up and going to bed every night.
Prasanna Malaiyandi:
It's orders of magnitude more important than anything else that I have going
Prasanna Malaiyandi:
on in terms of operating this business.
Prasanna Malaiyandi:
Um, says the CEO right, uh, of this company trying to bounce back.
Prasanna Malaiyandi:
Now we haven't, I tried to find, I didn't find, this was almost two years ago now.
Prasanna Malaiyandi:
I didn't see.
Prasanna Malaiyandi:
An update on what happened.
Prasanna Malaiyandi:
They were saying, we're trying to get people up and running.
Prasanna Malaiyandi:
I didn't see any stories like we did with the Carbonite about some
Prasanna Malaiyandi:
customers that actually lost data because they had to restore something
Prasanna Malaiyandi:
while they were trying to be seated.
Prasanna Malaiyandi:
Uh, we don't have any of those
Prasanna Malaiyandi:
Now, I know you're a big Reddit fan, and I know you
Prasanna Malaiyandi:
probably were browsing threads about this.
Prasanna Malaiyandi:
Did you see any, like, what was the sentiment in those threads
Prasanna Malaiyandi:
from end users, customers that, if
Prasanna Malaiyandi:
you can
Prasanna Malaiyandi:
W. Curtis Preston: that's a, actually, that's a good one.
Prasanna Malaiyandi:
Let me, uh, let me pull up that thread.
Prasanna Malaiyandi:
'cause I didn't, I didn't look too much at the thread.
Prasanna Malaiyandi:
This is funny.
Prasanna Malaiyandi:
Seems like a backup company that loses backups is no longer a backup
Prasanna Malaiyandi:
company, but a data disposal service.
Prasanna Malaiyandi:
I see a pivot opportunity.
Prasanna Malaiyandi:
They should certify these backups as destroyed and issued certificates.
Prasanna Malaiyandi:
I'd say the Reddit response is typical of every response with a cloud outage.
Prasanna Malaiyandi:
There are people that are just, you know, saying, ah, the cloud sucks, and,
Prasanna Malaiyandi:
you know, and the cloud is just, it's just another computer, it's just another
Prasanna Malaiyandi:
service, and you have to verify it just like you verify every other system.
Prasanna Malaiyandi:
Right.
Prasanna Malaiyandi:
Um, and there, there were some comments in the thread saying that.
Prasanna Malaiyandi:
You know, one of the reasons that Arc Serve and StorageCraft, uh, merged was
Prasanna Malaiyandi:
that they had been going down for a while in terms of quality of service.
Prasanna Malaiyandi:
Again, these are pe other people's comments, um, and that they, that they
Prasanna Malaiyandi:
weren't that surprised at, at this event.
Prasanna Malaiyandi:
But, um, they're still around.
Prasanna Malaiyandi:
They're still running.
Prasanna Malaiyandi:
Um, just like Carbonite is still around and still running.
Prasanna Malaiyandi:
I, I, I, you know, I wish them the best.
Prasanna Malaiyandi:
Uh, this was a major migration fail.
Prasanna Malaiyandi:
Yeah, and it's also one of those things
Prasanna Malaiyandi:
people have, like once you lose that trust, it get, it's difficult,
Prasanna Malaiyandi:
especially in the space where you are that last line of defense.
Prasanna Malaiyandi:
W. Curtis Preston: Yeah.
Prasanna Malaiyandi:
I mean, people are still on LastPass.
Prasanna Malaiyandi:
Prasanna Malaiyandi: Yeah, let's not go there.
Prasanna Malaiyandi:
W. Curtis Preston: Yeah.
Prasanna Malaiyandi:
What's that?
Prasanna Malaiyandi:
not go there.
Prasanna Malaiyandi:
W. Curtis Preston: Yeah, yeah.
Prasanna Malaiyandi:
Anyway.
Prasanna Malaiyandi:
All right.
Prasanna Malaiyandi:
Well, um, not, not a ton, not some huge lessons here, but the big one
Prasanna Malaiyandi:
I think here is, you know, I, I like the Maya Culpa that the company did.
Prasanna Malaiyandi:
I.
Prasanna Malaiyandi:
Uh, and I like that they knew that their priority was getting the,
Prasanna Malaiyandi:
the next backup, but unfortunately that's gonna take a long time.
Prasanna Malaiyandi:
Um, and the, the biggest lesson here is if you're doing a migration of any
Prasanna Malaiyandi:
kind, don't go shooting the other servers until you know that it's fully migrated.
Prasanna Malaiyandi:
You know?
Prasanna Malaiyandi:
what, what?
Prasanna Malaiyandi:
Is there any lessons learned for the end users?
Prasanna Malaiyandi:
Like there's a, like what could they have done?
Prasanna Malaiyandi:
I don't think there's anything that they could have done in this situation.
Prasanna Malaiyandi:
W. Curtis Preston: yeah, the only, the only thing I, in terms of.
Prasanna Malaiyandi:
Trying to identify, I mean, what's a vendor gonna say?
Prasanna Malaiyandi:
This is the difficulty with the cloud.
Prasanna Malaiyandi:
We've discussed this in previous other things, is it's hard to actually verify
Prasanna Malaiyandi:
if you say, do you have processes in place for if somebody, if somebody over
Prasanna Malaiyandi:
there accidentally, you know, blows up a server, do you have processes in
Prasanna Malaiyandi:
place to be able to bring that server back or re of course they're gonna say,
Prasanna Malaiyandi:
yeah, that's what they're gonna say.
Prasanna Malaiyandi:
Um, the, the, the one thing you could think about, again, this is
Prasanna Malaiyandi:
expensive, but people talk about using a service that puts stuff in more
Prasanna Malaiyandi:
than one place or using more than one service, but people don't wanna pay
Prasanna Malaiyandi:
for one backup service, let alone two.
Prasanna Malaiyandi:
Could they do that?
Prasanna Malaiyandi:
Should they do that?
Prasanna Malaiyandi:
Sure.
Prasanna Malaiyandi:
Will they do that?
Prasanna Malaiyandi:
Nobody does that,
Prasanna Malaiyandi:
Yeah.
Prasanna Malaiyandi:
No one's gonna spend the money in their budget on that.
Prasanna Malaiyandi:
W. Curtis Preston: Yeah.
Prasanna Malaiyandi:
Um.
Prasanna Malaiyandi:
I mean, it's hard enough to get people to do backups in the first place.
Prasanna Malaiyandi:
It's hard enough to get them to, to properly secure those backups.
Prasanna Malaiyandi:
Um, they're not gonna pay for a completely separate, just in case they're, you know,
Prasanna Malaiyandi:
uh, I will say having something like I.
Prasanna Malaiyandi:
StorageCraft or carbonite, die and lose all your data is not
Prasanna Malaiyandi:
the same as losing all your data.
Prasanna Malaiyandi:
Right.
Prasanna Malaiyandi:
It's, it's, it's all the backups of all your data.
Prasanna Malaiyandi:
And there will be some, could be some ramifications to your business if you
Prasanna Malaiyandi:
need to restore, but statistically speaking, you're not probably
Prasanna Malaiyandi:
going to need to restore today.
Prasanna Malaiyandi:
Um, just in terms of statistically Right.
Prasanna Malaiyandi:
Um, it's not the same.
Prasanna Malaiyandi:
So, you know,
Prasanna Malaiyandi:
So, okay, so if I was an end user, I get a
Prasanna Malaiyandi:
notification from StorageCraft that says, Hey, by the way, we had an issue.
Prasanna Malaiyandi:
We lost all your backups.
Prasanna Malaiyandi:
They're gonna say, okay, I'm gonna try to help reseed.
Prasanna Malaiyandi:
Right.
Prasanna Malaiyandi:
Get all your data up there before that happens.
Prasanna Malaiyandi:
Because like you said, it could take weeks, it could take months.
Prasanna Malaiyandi:
Right.
Prasanna Malaiyandi:
What is the first thing that the end user should do at that point?
Prasanna Malaiyandi:
Right?
Prasanna Malaiyandi:
Once they get that first notification to make sure that they don't end up
Prasanna Malaiyandi:
losing all their data in case something happens to their production copy.
Prasanna Malaiyandi:
W. Curtis Preston: Well, I, I'll say this, you got any big projects,
Prasanna Malaiyandi:
uh, in process, maybe you should pause those projects, right?
Prasanna Malaiyandi:
Any big changes, any big maintenance windows.
Prasanna Malaiyandi:
Maybe you should put a pause on those maintenance windows.
Prasanna Malaiyandi:
Uh, the other is you should go back to old school backups for the
Prasanna Malaiyandi:
time being, for anything critical.
Prasanna Malaiyandi:
Uh, you know, you should make a local backup of any kind that, a backup of any
Prasanna Malaiyandi:
kind is better than a backup of no kind.
Prasanna Malaiyandi:
Uh, that's a really good question, right.
Prasanna Malaiyandi:
Um, I would put a pause on projects and I would see what I can do to get
Prasanna Malaiyandi:
some local backups of stuff, right?
Prasanna Malaiyandi:
There are plenty of open source tools out there.
Prasanna Malaiyandi:
That can, you know, surely somewhere you've got some disc and then you've
Prasanna Malaiyandi:
got some super critical servers that you really don't wanna lose.
Prasanna Malaiyandi:
Get some local backups of those.
Prasanna Malaiyandi:
Um, you know.
Prasanna Malaiyandi:
or I know you had mentioned in the past, right?
Prasanna Malaiyandi:
You really like the iDrive service, cloud service,
Prasanna Malaiyandi:
W. Curtis Preston: Right.
Prasanna Malaiyandi:
right.
Prasanna Malaiyandi:
For just doing
Prasanna Malaiyandi:
W. Curtis Preston: Well, the.
Prasanna Malaiyandi:
backup
Prasanna Malaiyandi:
W. Curtis Preston: Yeah, the, the only problem with that is, you
Prasanna Malaiyandi:
know, again, it's physics, right?
Prasanna Malaiyandi:
So I'm just trying to, I'm just trying to think of something
Prasanna Malaiyandi:
that you can do quickly.
Prasanna Malaiyandi:
Um, you know
Prasanna Malaiyandi:
Copy it to a different machine.
Prasanna Malaiyandi:
Go buy a couple external hard
Prasanna Malaiyandi:
W. Curtis Preston: Yeah.
Prasanna Malaiyandi:
Specifically for those super high value, super highly critical
Prasanna Malaiyandi:
servers, you know, uh, things like backup servers for other customers.
Prasanna Malaiyandi:
Uh uh.
Prasanna Malaiyandi:
Yeah.
Prasanna Malaiyandi:
Okay.
Prasanna Malaiyandi:
Well, all right, well, thanks for helping me walk through that.
Prasanna Malaiyandi:
That was a really good question there at the end.
Prasanna Malaiyandi:
Prasanna, thanks for helping me walk through this.
Prasanna Malaiyandi:
No, no worries.
Prasanna Malaiyandi:
That's why uh, I enjoy doing these podcasts 'cause it helps all of us, right?
Prasanna Malaiyandi:
Just figure out what should we be
Prasanna Malaiyandi:
doing in case this ever happens.
Prasanna Malaiyandi:
W. Curtis Preston: That's what we're trying to do is to pass on these
Prasanna Malaiyandi:
lessons to you, uh, the listener so that you can learn from other
Prasanna Malaiyandi:
people's struggles, unfortunately.
Prasanna Malaiyandi:
But, uh, you're why we do this.
Prasanna Malaiyandi:
We want to turn you into a cyber recovery hero.
Prasanna Malaiyandi:
That's a wrap
Prasanna Malaiyandi:
The backup wrap up is written, recorded and produced by me w Curtis Preston.
Prasanna Malaiyandi:
If you need backup or Dr.
Prasanna Malaiyandi:
Consulting content generation or expert witness work,
Prasanna Malaiyandi:
check out backup central.com.
Prasanna Malaiyandi:
You can also find links from my O'Reilly Books on the same website.
Prasanna Malaiyandi:
Remember, this is an independent podcast and any opinions that you
Prasanna Malaiyandi:
hear are those of the speaker.
Prasanna Malaiyandi:
And not necessarily an employer.
Prasanna Malaiyandi:
Thanks for listening.
Listen On
