OVHCloud Dumpster Fire (Cloud Disasters)
In this important episode of the Backup Wrap-up, W. Curtis Preston, AKA Mr. Backup, takes you on a deep dive into the shocking story of the OVHCloud data center fire of 2021. This catastrophic event left hundreds of customers scrambling to recover their precious data (often without backups), exposing the startling truth about OVHCloud's data center and backup practices and the devastating consequences of misplaced trust in cloud providers.
As usual, Curtis is joined by guest host Prasanna Malaiyandi as they unravel the complex web of controversial decisions, legal battles, and hard lessons learned from this disastrous incident. They explore the factors contributing to the fire's severity, the staggering extent of the data loss, and the eye-opening revelations about OVHCloud's backup infrastructure.
But this episode is more than just a cautionary tale. Curtis and Prasanna delve into the crucial questions every organization must ask their cloud provider to ensure their data is truly secure. They share invaluable insights and actionable advice to help you avoid falling victim to a similar fate.
Whether you're an IT professional responsible for safeguarding your company's data or an individual looking to protect your digital life, this episode is a must-listen. You'll come away with a deeper understanding of the risks associated with cloud storage, the importance of robust backup strategies, and the steps you can take to become a cyber-recovery hero.
Don't miss this opportunity to learn from one of the most significant cloud disasters in recent history. Tune in now and discover how you can keep your data out of the fire and emerge as an unsung hero in the world of backup and recovery.
Articles covering this story:
https://www.datacenterdynamics.com/en/news/ovh-fire-octave-klaba-says-ups-systems-were-ablaze/
https://www.datacenterdynamics.com/en/news/fire-could-cost-ovhcloud-105-million-ipo-filing-reveals/
https://www.datacenterdynamics.com/en/news/ovhcloud-ordered-to-pay-250k-to-two-customers-who-lost-data-in-strasbourg-data-center-fire/
https://www.datacenterdynamics.com/en/news/ovhcloud-fire-report-sbg2-data-center-had-wooden-ceilings-no-extinguisher-and-no-power-cut-out/
Election worker episode:
https://www.backupwrapup.com/election-poll-site-manager-explains-us-election-systems/
Mentioned in this episode:
Blank Midroll
Speaker:
Boy, do I have a doozy of a story for you today?
Speaker:
Picture this.
Speaker:
You're just minding your own business, trusting your cloud provider to
Speaker:
keep your data safe and sound.
Speaker:
That's when a massive fire rips through and destroys their entire data center and
Speaker:
suddenly your whole digital world is gone.
Speaker:
That's exactly what happened in 2021 to the poor customers of OVH.
Speaker:
And let me tell you the more I dug into the story, the more I
Speaker:
realized just how messed up it was.
Speaker:
We're talking about a company that claim to have top notch backup practices,
Speaker:
but in reality, they were about as reliable as a chocolate teapot.
Speaker:
This is a wake up call for anyone who's ever put their faith in the cloud.
Speaker:
This episode will be a wild ride through the ups and downs of this whole fiasco.
Speaker:
We'll talk about the Shadys decisions made by OVH cloud.
Speaker:
Y the fire got so bad in the first place.
Speaker:
Why companies lost data and the crucial lessons that every single one
Speaker:
of us needs to learn from this mess.
Speaker:
So whether you're an it hot shot or just someone who wants to
Speaker:
keep their data out of the fire.
Speaker:
You better listen up.
Speaker:
By the end of this episode, you'll know exactly what questions to ask your cloud
Speaker:
provider to make sure you never end up in the same boat as those poor OVH customers.
Speaker:
I'm w Curtis Preston, AKA Mr.
Speaker:
Backup, and no one cares about your data more than I do.
Speaker:
I want to turn you the unappreciated backup admin.
Speaker:
Into a cyber recovery hero.
Speaker:
This is the backup wrap-up.
Speaker:
W. Curtis Preston: Welcome to the show.
Speaker:
I'm w Curtis Preston, and I have with me my election worker post-Traumatic Stress
Speaker:
Counselor Prasanna Malaiyandi how's it
Prasanna Malaiyandi:
Ah, I'm doing well, Curtis, uh, how are
Prasanna Malaiyandi:
you doing or have you recovered?
Prasanna Malaiyandi:
I know that you had a pretty hectic 10 days, of which the last day was
Prasanna Malaiyandi:
prob 11 days, of which the last day was probably like a lot more
Prasanna Malaiyandi:
effort than the first 10 days, so,
Prasanna Malaiyandi:
W. Curtis Preston: Yeah.
Prasanna Malaiyandi:
Basically we got more voters on the, the actual, the actual election day.
Prasanna Malaiyandi:
It was the primary, it was super Tuesday.
Prasanna Malaiyandi:
For those of you, which for those of you not in the US is I.
Prasanna Malaiyandi:
A day when many states have their primaries for the presidential election.
Prasanna Malaiyandi:
And, uh, California is one of those, which is where I live, and
Prasanna Malaiyandi:
I happen to be an election worker.
Prasanna Malaiyandi:
Uh, it's something that I do volunteer.
Prasanna Malaiyandi:
Well, I, I volunteer to do it, but I do get paid for the time that I spend there.
Prasanna Malaiyandi:
Not a ton, but, but I do get paid for my time but you know what,
Prasanna Malaiyandi:
persona, I could, I could very easily talk about this for an hour.
Prasanna Malaiyandi:
I think we've had three episodes.
Prasanna Malaiyandi:
W. Curtis Preston: You know what?
Prasanna Malaiyandi:
I'll put in the show notes.
Prasanna Malaiyandi:
If you're curious to know my thoughts on the way elections
Prasanna Malaiyandi:
work and how they actually work versus the way you think they work.
Prasanna Malaiyandi:
Um, you know, you know, go, go check those.
Prasanna Malaiyandi:
I'll, I'll put, I'll put 'em in the show notes and you
Prasanna Malaiyandi:
and we do have also the episode with Mark.
Prasanna Malaiyandi:
Mark.
Prasanna Malaiyandi:
W. Curtis Preston: Yes.
Prasanna Malaiyandi:
And we have the episode with Mark.
Prasanna Malaiyandi:
Yeah.
Prasanna Malaiyandi:
There there's two ways that site managers run the sites.
Prasanna Malaiyandi:
Mark is of the first type, which is the, I'm here to help you vote and
Prasanna Malaiyandi:
if you have any questions other than that, like if you want to question
Prasanna Malaiyandi:
the process, here's a phone number I.
Prasanna Malaiyandi:
I'm more the, the type that will actually take the question.
Prasanna Malaiyandi:
I remember one voter, I basically was grilled for
Prasanna Malaiyandi:
probably 45 minutes by a voter.
Prasanna Malaiyandi:
I feel that she went away with a little more confidence in the, in the system.
Prasanna Malaiyandi:
And, um, what was really funny was the moment when I had two voters
Prasanna Malaiyandi:
within five minutes of each other.
Prasanna Malaiyandi:
One who said, and it was interesting, he's really, really old gentleman.
Prasanna Malaiyandi:
Like, I'm, I'm old.
Prasanna Malaiyandi:
He was old to me, so he was old, right?
Prasanna Malaiyandi:
And he, he basically said, this is crazy.
Prasanna Malaiyandi:
The, you know, with all this technology, we could just, we could
Prasanna Malaiyandi:
just replay all this stupid paper with technology and then another person.
Prasanna Malaiyandi:
Two minutes later it came up and said, there's way too much technology here.
Prasanna Malaiyandi:
We should just do this with paper.
Prasanna Malaiyandi:
And I was like, well, we are doing it with paper.
Prasanna Malaiyandi:
But you know, I don't have time to go into that.
Prasanna Malaiyandi:
But anyway, so, uh, it is time to get into our topic of the week.
Prasanna Malaiyandi:
This week in our continued series on cloud disasters where people thought
Prasanna Malaiyandi:
that they had backups or think that they don't need backups, and then
Prasanna Malaiyandi:
they found out otherwise, this one is a pretty big one, wouldn't you say?
Prasanna Malaiyandi:
Oh yeah.
Prasanna Malaiyandi:
I think it was probably one of the most impactful ones.
Prasanna Malaiyandi:
Right.
Prasanna Malaiyandi:
Just in terms of the customer
Prasanna Malaiyandi:
W. Curtis Preston: but it both, most Im, it's most impactful.
Prasanna Malaiyandi:
Customers were really messed up with this.
Prasanna Malaiyandi:
And also I.
Prasanna Malaiyandi:
You know it.
Prasanna Malaiyandi:
So, and in another way it was not impactful at all.
Prasanna Malaiyandi:
We're gonna talk about that.
Prasanna Malaiyandi:
Uh, you know, in the story we're talking about the OVH Fire.
Prasanna Malaiyandi:
Of 2021.
Prasanna Malaiyandi:
So what's interesting is, you know, we have primarily a North American audience.
Prasanna Malaiyandi:
We have, you know, plenty from other countries, but, but the bulk
Prasanna Malaiyandi:
of our audiences in North America.
Prasanna Malaiyandi:
And I'm willing to bet that prior to this fire, the average person in
Prasanna Malaiyandi:
North America had never heard of OVH.
Prasanna Malaiyandi:
Prasanna Malaiyandi: I never heard about o.
Prasanna Malaiyandi:
W. Curtis Preston: I'm willing to bet.
Prasanna Malaiyandi:
Me neither.
Prasanna Malaiyandi:
Right.
Prasanna Malaiyandi:
And I, um, I'm willing to bet that the average person still doesn't
Prasanna Malaiyandi:
know anything about opiates.
Prasanna Malaiyandi:
That this sort, that this story sort of came and went and it's kind of amazing
Prasanna Malaiyandi:
and, um, that's a little bit sad.
Prasanna Malaiyandi:
Yep.
Prasanna Malaiyandi:
W. Curtis Preston: So, um, you want to sort of tell the story of the fire?
Prasanna Malaiyandi:
Yeah, so in March.
Prasanna Malaiyandi:
So I think before we get there, so a couple things.
Prasanna Malaiyandi:
First people should probably know a little bit.
Prasanna Malaiyandi:
I know you talked about OVH isn't
Prasanna Malaiyandi:
W. Curtis Preston: Oh, that is right.
Prasanna Malaiyandi:
Yep.
Prasanna Malaiyandi:
right?
Prasanna Malaiyandi:
But OVH is a cloud provider.
Prasanna Malaiyandi:
They are the largest European headquartered cloud provider.
Prasanna Malaiyandi:
W. Curtis Preston: Right.
Prasanna Malaiyandi:
And they're headquartered in France.
Prasanna Malaiyandi:
and one of the unique things of is, at least for this data
Prasanna Malaiyandi:
center, the way that they built it, is they actually use shipping containers.
Prasanna Malaiyandi:
Right, so they're kind of stacked on top of each other.
Prasanna Malaiyandi:
They have a bunch of servers in there with power and cooling and also wooden
Prasanna Malaiyandi:
walls and framing inside as well.
Prasanna Malaiyandi:
Uh, because it is a shipping container, I've actually watched a bunch of YouTube
Prasanna Malaiyandi:
people who do a lot of container homes.
Prasanna Malaiyandi:
And so yeah, you, you need to add structure to containers in order to
Prasanna Malaiyandi:
be able to do things like put floors and walls and other aspects and
Prasanna Malaiyandi:
W. Curtis Preston: There's interesting, uh, relationship
Prasanna Malaiyandi:
there because the videos to which you referred, they also lost data.
Prasanna Malaiyandi:
Yep.
Prasanna Malaiyandi:
Yeah, the, yep.
Prasanna Malaiyandi:
The one that I watched.
Prasanna Malaiyandi:
W. Curtis Preston: we'll come maybe.
Prasanna Malaiyandi:
Yeah.
Prasanna Malaiyandi:
Maybe.
Prasanna Malaiyandi:
We'll, we'll come back to that in a later date.
Prasanna Malaiyandi:
Uh, yeah.
Prasanna Malaiyandi:
So it was a pretty big company, uh, big enough that Well, we, we'll save the
Prasanna Malaiyandi:
Yeah.
Prasanna Malaiyandi:
And, and, and they have a lot of customers and they're a lot of government
Prasanna Malaiyandi:
contracts as well, so it's not like it's like a mom and pop cloud provider.
Prasanna Malaiyandi:
Right.
Prasanna Malaiyandi:
It is a huge cloud provider
Prasanna Malaiyandi:
W. Curtis Preston: also, an important thing is that they
Prasanna Malaiyandi:
offered a backup service.
Prasanna Malaiyandi:
That you could pay extra for.
Prasanna Malaiyandi:
Um, and, uh, it happened to be hosted Veeam, right?
Prasanna Malaiyandi:
Uh, and you know, Veeam did nothing wrong here, but definitely the vendor did.
Prasanna Malaiyandi:
They offered a, a hosted backup service.
Prasanna Malaiyandi:
It's a little bit different than the average cloud backup.
Prasanna Malaiyandi:
Problem where it's like, well, you didn't, you expected your cloud vendor
Prasanna Malaiyandi:
to back up and you, they didn't back up and you should have backed up.
Prasanna Malaiyandi:
And so it's nobody's fault but your own.
Prasanna Malaiyandi:
But in this case, there were customers that actually paid for the backup service.
Prasanna Malaiyandi:
And then what happened?
Prasanna Malaiyandi:
So what ended up happening in March
Prasanna Malaiyandi:
of 2021, there was a fire and.
Prasanna Malaiyandi:
So how they have it situated is there are basically four, I don't know if
Prasanna Malaiyandi:
you'd call 'em data centers within that single site, but they had four
Prasanna Malaiyandi:
data centers, if you will, within it, and they had a fire in one.
Prasanna Malaiyandi:
And these are shipping containers.
Prasanna Malaiyandi:
And apparently they did not have the right fire suppress suppression equipment.
Prasanna Malaiyandi:
They did not have electrical cutoff capabilities.
Prasanna Malaiyandi:
And so firefighters came and they tried their best to put out that fire,
Prasanna Malaiyandi:
but the thing just spread like crazy.
Prasanna Malaiyandi:
W. Curtis Preston: just, yeah, the, the, the, the craziest thing in the report that
Prasanna Malaiyandi:
came out a year later, the, the bass rin report, it's, it said that it started at
Prasanna Malaiyandi:
an electrical inverter on the first floor.
Prasanna Malaiyandi:
But here's my favorite phrase.
Prasanna Malaiyandi:
When the firefighters came, they were met with electric arcs of
Prasanna Malaiyandi:
more than one meter around the exterior door of the energy room.
Prasanna Malaiyandi:
Like that sounds something out of like a.
Prasanna Malaiyandi:
Like a Avengers movie, you know what I mean?
Prasanna Malaiyandi:
Like this is what you call in Superman.
Prasanna Malaiyandi:
A a one meter long, that's about a yard for those of you who live in the US right?
Prasanna Malaiyandi:
Uh, electrical arcs that long, and then no suppression.
Prasanna Malaiyandi:
It said it took 'em, uh, over two hours to get the fire or to get
Prasanna Malaiyandi:
the electrical cut off because they had to do it, uh, somewhere else.
Prasanna Malaiyandi:
They couldn't do it in the
Prasanna Malaiyandi:
And it wasn't just the one data center that got
Prasanna Malaiyandi:
affected, it also impacted the other one.
Prasanna Malaiyandi:
So I think one of the other ones had some damage.
Prasanna Malaiyandi:
Right?
Prasanna Malaiyandi:
It wasn't major, right?
Prasanna Malaiyandi:
Like the first one, but there was still quite a bit of damage.
Prasanna Malaiyandi:
And then the other two, they had to bring it offline because like you mentioned,
Prasanna Malaiyandi:
they had to cut the electricity.
Prasanna Malaiyandi:
They weren't sure what else was broken.
Prasanna Malaiyandi:
Right?
Prasanna Malaiyandi:
So they basically brought everything down.
Prasanna Malaiyandi:
W. Curtis Preston: Yeah.
Prasanna Malaiyandi:
It, it, it, it's like you look at the pictures of this,
Prasanna Malaiyandi:
this was a giant fire, right?
Prasanna Malaiyandi:
And the, and initially I felt that the, um, you know, that they, I.
Prasanna Malaiyandi:
They responded well in that they, they, you know, they created, they
Prasanna Malaiyandi:
did, what we, we suggest people do is they created a status page, um,
Prasanna Malaiyandi:
and they updated that status page.
Prasanna Malaiyandi:
The, the CEO actually made a number of videos, put them out.
Prasanna Malaiyandi:
I'm not sure if he put 'em on YouTube or, uh, whatever, but he, he made it.
Prasanna Malaiyandi:
He was trying to reassure people that we're doing everything we can.
Prasanna Malaiyandi:
Recover from this, recover with a small R there.
Prasanna Malaiyandi:
Um, but um,
Prasanna Malaiyandi:
And they were transparent, right?
Prasanna Malaiyandi:
The other thing is they were like, Hey, we have other data centers.
Prasanna Malaiyandi:
They were quick trying to bring up new servers to move these customers
Prasanna Malaiyandi:
over to that new equipment, right?
Prasanna Malaiyandi:
To get them up and running as quickly as possible.
Prasanna Malaiyandi:
So they were bringing up the infrastructure, but
Prasanna Malaiyandi:
this is a big but, right?
Prasanna Malaiyandi:
What about their data?
Prasanna Malaiyandi:
W. Curtis Preston: Yeah, so apparently the backup servers were literally, you know,
Prasanna Malaiyandi:
in the same data center that they were backing up, and which, you know, is not in
Prasanna Malaiyandi:
keeping with, you know, the 3, 2, 1 rule.
Prasanna Malaiyandi:
Right?
Prasanna Malaiyandi:
So that's just, that's just a fundamental design flaw, I would say.
Prasanna Malaiyandi:
If you're offering backup services, you should not be putting the
Prasanna Malaiyandi:
backup servers directly next to the thing that they're backing up.
Prasanna Malaiyandi:
Right now they're saying that they were physically isolated, but they were just
Prasanna Malaiyandi:
literally like over in the corner, like go, you know, go sit in the corner.
Prasanna Malaiyandi:
Your backups.
Prasanna Malaiyandi:
Yeah.
Prasanna Malaiyandi:
which is not what you were, and, and I don't know how many
Prasanna Malaiyandi:
people actually understood that, right?
Prasanna Malaiyandi:
Uh, if you were managing your own backup environment in your own data
Prasanna Malaiyandi:
center, you could physically validate these things when you are trusting and
Prasanna Malaiyandi:
buying a service from a vendor, right?
Prasanna Malaiyandi:
You just have to either ask the hard questions to really figure out
Prasanna Malaiyandi:
what they're doing or just trust that they're doing the right thing.
Prasanna Malaiyandi:
And that second part got them, got a lot of customers burned.
Prasanna Malaiyandi:
Do you like what I did there?
Prasanna Malaiyandi:
W. Curtis Preston: I, I like what you just there, so about a year later
Prasanna Malaiyandi:
is when the, the fire department.
Prasanna Malaiyandi:
Uh, report came out and that's when we learned that they had no fire suppression
Prasanna Malaiyandi:
systems, they had no power cutoff systems.
Prasanna Malaiyandi:
Uh, we learned that it damaged at least two other data centers.
Prasanna Malaiyandi:
Right.
Prasanna Malaiyandi:
And the, because while they initially started out being very open.
Prasanna Malaiyandi:
OVH suddenly stopped talking roughly in May of 2021.
Prasanna Malaiyandi:
They stopped talking and, and, and they haven't spoken publicly about the
Prasanna Malaiyandi:
incident since that, and that could be because they were planning for an event
Prasanna Malaiyandi:
it was a big event.
Prasanna Malaiyandi:
W. Curtis Preston: Big event.
Prasanna Malaiyandi:
What event might they have been planning for
Prasanna Malaiyandi:
Yeah, so in October of 2021, which is seven months after
Prasanna Malaiyandi:
the fire, they went through an IPO and went public and raised a bunch of money.
Prasanna Malaiyandi:
Right.
Prasanna Malaiyandi:
It was not a it, and it wasn't a failure of an IPO either.
Prasanna Malaiyandi:
Like I think the stock actually went up.
Prasanna Malaiyandi:
They raised a good deal of money, right?
Prasanna Malaiyandi:
So it was a pretty successful IPO seven months after this incident.
Prasanna Malaiyandi:
W. Curtis Preston: I.
Prasanna Malaiyandi:
That just flabbergast me, right?
Prasanna Malaiyandi:
Like as, as someone who worked at a pre IPO company, right?
Prasanna Malaiyandi:
I can't imagine something that catastrophic happening and
Prasanna Malaiyandi:
then seven months later saying, you know what we should do?
Prasanna Malaiyandi:
We should have people
Prasanna Malaiyandi:
Well, well, well, one thing is the IPO was
Prasanna Malaiyandi:
probably already planned in the works.
Prasanna Malaiyandi:
Right?
Prasanna Malaiyandi:
Started going.
Prasanna Malaiyandi:
And they were hoping that,
Prasanna Malaiyandi:
W. Curtis Preston: they didn't wanna press pause.
Prasanna Malaiyandi:
you gotta strike while the iron's hot because remember,
Prasanna Malaiyandi:
W. Curtis Preston: The was the, was.
Prasanna Malaiyandi:
right.
Prasanna Malaiyandi:
So I think that they just took the chance, rolled the dice and figured, yeah, we're a
Prasanna Malaiyandi:
large public cloud provider in Europe that we have sort of a dominant position here.
Prasanna Malaiyandi:
We might as well raise the money while we can.
Prasanna Malaiyandi:
I don't know how things ended up there.
Prasanna Malaiyandi:
And I think like, and it's not like they've gone by the wayside, right?
Prasanna Malaiyandi:
OVH is still popular in Europe.
Prasanna Malaiyandi:
There are still a lot of people using their services.
Prasanna Malaiyandi:
Right?
Prasanna Malaiyandi:
So it's not like they just crumbled up and died.
Prasanna Malaiyandi:
W. Curtis Preston: Well, um, I would like to point out that they went out at 20.
Prasanna Malaiyandi:
Euros, right?
Prasanna Malaiyandi:
, back in, uh, October of 2022.
Prasanna Malaiyandi:
Uh, their current value is, uh, half that.
Prasanna Malaiyandi:
Okay.
Prasanna Malaiyandi:
W. Curtis Preston: So basically they have, they went up a little bit,
Prasanna Malaiyandi:
they had a little bit of a spike for about two months, and then it has
Prasanna Malaiyandi:
done nothing but go down ever since.
Prasanna Malaiyandi:
Um, so.
Prasanna Malaiyandi:
You know, it, it may not have been a failure in the beginning, but it
Prasanna Malaiyandi:
certainly doesn't look very good now.
Prasanna Malaiyandi:
It's worth less than half,
Prasanna Malaiyandi:
Yeah.
Prasanna Malaiyandi:
W. Curtis Preston: which gives me a little bit of consolation, I guess.
Prasanna Malaiyandi:
Um, maybe it's that
Prasanna Malaiyandi:
But,
Prasanna Malaiyandi:
W. Curtis Preston: the, as the news continued to come out
Prasanna Malaiyandi:
yeah.
Prasanna Malaiyandi:
W. Curtis Preston: over the, over the years.
Prasanna Malaiyandi:
Right.
Prasanna Malaiyandi:
And there was other news as well.
Prasanna Malaiyandi:
Immediately after this happened, we started hearing
Prasanna Malaiyandi:
about a class action lawsuit.
Prasanna Malaiyandi:
Right.
Prasanna Malaiyandi:
And, uh, there was a firm, Ziegler and Associates that filed, uh.
Prasanna Malaiyandi:
A class action lawsuit in September of 2021, which for the record
Prasanna Malaiyandi:
would've been a month before the IPO.
Prasanna Malaiyandi:
And they, um, they fired it, you know, for clients that had lost data,
Prasanna Malaiyandi:
uh, to the fire or due to the fire.
Prasanna Malaiyandi:
And as of the, basically about a year ago, it said there were
Prasanna Malaiyandi:
140 customers in the lawsuit.
Prasanna Malaiyandi:
And the lawsuit is still ongoing.
Prasanna Malaiyandi:
W. Curtis Preston: Yeah, it appears that, it appears that it's ongoing.
Prasanna Malaiyandi:
We don't have any information on, at least I don't have any
Prasanna Malaiyandi:
information that it ended.
Prasanna Malaiyandi:
I, I didn't see any stories that it ended, but we do have a story about two
Prasanna Malaiyandi:
customers that were apparently not in that class action law class action lawsuit.
Prasanna Malaiyandi:
What did you, uh, hear about
Prasanna Malaiyandi:
Yeah, so there were two customers.
Prasanna Malaiyandi:
Um, the first basically had, uh, sued them because they had signed
Prasanna Malaiyandi:
up for the automated service.
Prasanna Malaiyandi:
And, uh, based on the contract wording, the judge sided with them
Prasanna Malaiyandi:
saying that, yes, physically isolated means that you keep 'em separate.
Prasanna Malaiyandi:
And having it in the same building goes against like what's expected,
Prasanna Malaiyandi:
what you would do for like a backup perspective and is wrong.
Prasanna Malaiyandi:
And so they basically won a nominal amount.
Prasanna Malaiyandi:
I would say they did not.
Prasanna Malaiyandi:
I think they were suing for like 3 million euros.
Prasanna Malaiyandi:
They got like a hundred thousand euros.
Prasanna Malaiyandi:
So it was a nominal win, but it still showed that, yeah.
Prasanna Malaiyandi:
W. Curtis Preston: Yeah.
Prasanna Malaiyandi:
And, and a couple things there.
Prasanna Malaiyandi:
While I agree with the, sometimes you have the spirit of the contract
Prasanna Malaiyandi:
or the letter of the contract, I agree with what the judge says.
Prasanna Malaiyandi:
Obviously, who does that, right?
Prasanna Malaiyandi:
Who does the backup design, where the backup systems are in the same data
Prasanna Malaiyandi:
center as the, uh, primary systems.
Prasanna Malaiyandi:
Uh, having said that.
Prasanna Malaiyandi:
It that was stated in the contract and that that was the most shocking
Prasanna Malaiyandi:
part to me, is that, you know, in my review I went and found, um, you
Prasanna Malaiyandi:
know, uh, basically the both the gotta love the way back machine, right?
Prasanna Malaiyandi:
The, the internet archive.
Prasanna Malaiyandi:
Using the internet.
Prasanna Malaiyandi:
Using the internet archive, I was able to find the contract that they were quoting.
Prasanna Malaiyandi:
And, uh, what the contract says is that for the backup service that it
Prasanna Malaiyandi:
says that the storage resources for the backup service will be physically,
Prasanna Malaiyandi:
I isolated from the servers that it's backing up, and then it's like
Prasanna Malaiyandi:
somebody stopped reading right?
Prasanna Malaiyandi:
Because the very next sentence that says the storage resources will be
Prasanna Malaiyandi:
located in the same data center as the resources that it's backing up.
Prasanna Malaiyandi:
It's, it's literally right in the contract.
Prasanna Malaiyandi:
And, and so as, as like a, like a legal person, like I, I just wanna
Prasanna Malaiyandi:
say, well, how did you not read that?
Prasanna Malaiyandi:
Yeah.
Prasanna Malaiyandi:
W. Curtis Preston: I'm on the side of the customer here in that they
Prasanna Malaiyandi:
paid for a backup service, but I'm not on the side of the customer in
Prasanna Malaiyandi:
that they should have reviewed that contract and they should have said so.
Prasanna Malaiyandi:
Wait, wait.
Prasanna Malaiyandi:
What?
Prasanna Malaiyandi:
Yeah.
Prasanna Malaiyandi:
Well, and unlike a lot of other contracts right, or like SA services where they
Prasanna Malaiyandi:
talk nothing about backups, this was clearly stated what they are doing.
Prasanna Malaiyandi:
W. Curtis Preston: stated they're, and they're paying extra, right?
Prasanna Malaiyandi:
They were paying something like 2 cents a gigabyte per month
Prasanna Malaiyandi:
for this backup service and.
Prasanna Malaiyandi:
And the thing is, I think what the judge said, I, I like, this is one
Prasanna Malaiyandi:
of those things where, you know, I'm really good at arguing, and
Prasanna Malaiyandi:
I can argue both sides of this.
Prasanna Malaiyandi:
Like on one side they should have read the contract, the customer should have
Prasanna Malaiyandi:
read the contract, and they should have asked about what that sentence mean.
Prasanna Malaiyandi:
And they should have then just not paid for the backup service
Prasanna Malaiyandi:
because it wouldn't be worth it.
Prasanna Malaiyandi:
Um, on the other hand, I agree with the judge, like, who does that?
Prasanna Malaiyandi:
Right.
Prasanna Malaiyandi:
It's not like what's the point of doing a backup if it's just
Prasanna Malaiyandi:
two disk drives right next door?
Prasanna Malaiyandi:
Right.
Prasanna Malaiyandi:
And the judge was basically saying, this isn't in keeping with, you know, the
Prasanna Malaiyandi:
state of the art of the backup system.
Prasanna Malaiyandi:
Agreed, agreed, agreed.
Prasanna Malaiyandi:
And it wouldn't allow you to meet the requirements of the contract.
Prasanna Malaiyandi:
And so while the, I think basically what the judge says, you know, even though
Prasanna Malaiyandi:
the contract may have stated that.
Prasanna Malaiyandi:
What are you stupid?
Prasanna Malaiyandi:
Right?
Prasanna Malaiyandi:
You know what I mean?
Prasanna Malaiyandi:
Which is different than the second case,
Prasanna Malaiyandi:
So there were two customers who ended up suing and winning, right?
Prasanna Malaiyandi:
So the second customer also sued, uh, OVH, and this time the judge also found
Prasanna Malaiyandi:
in the favor of the customer, and this one was slightly different because in
Prasanna Malaiyandi:
their contract they had actually written out that they had a production system
Prasanna Malaiyandi:
in one data center and explicitly their backup system in a different data center.
Prasanna Malaiyandi:
In the contract, it was worded that their production and backup systems
Prasanna Malaiyandi:
should be separate, and OVH agreed to it.
Prasanna Malaiyandi:
And what ended up happening is OVH ended up keeping both copies
Prasanna Malaiyandi:
of data in the same data center.
Prasanna Malaiyandi:
But,
Prasanna Malaiyandi:
W. Curtis Preston: basically they just lied.
Prasanna Malaiyandi:
but here's the problem, is their management console
Prasanna Malaiyandi:
kept telling the customer that they were in separate data centers.
Prasanna Malaiyandi:
W. Curtis Preston: Wow.
Prasanna Malaiyandi:
Yeah.
Prasanna Malaiyandi:
That's, that's just bad.
Prasanna Malaiyandi:
And, and that's probably the worst thing that I've read about this story.
Prasanna Malaiyandi:
Because if you are a customer that's about the best you could do is say, Hey
Prasanna Malaiyandi:
listen, we're demanding that you put it separately and then the, you check the
Prasanna Malaiyandi:
console and the console sets it separate.
Prasanna Malaiyandi:
It's the cloud.
Prasanna Malaiyandi:
Like, I can't, I can't go visit it.
Prasanna Malaiyandi:
Right.
Prasanna Malaiyandi:
Um, so like they, so that in that case, the customer did
Prasanna Malaiyandi:
everything that they could.
Prasanna Malaiyandi:
And so the judge gave, uh, fined OVH 138,000 euros,
Prasanna Malaiyandi:
something like that, but that's not even the worst part of OV H'S behavior.
Prasanna Malaiyandi:
So OVH eventually found the backup server, and
Prasanna Malaiyandi:
W. Curtis Preston: right.
Prasanna Malaiyandi:
Yeah.
Prasanna Malaiyandi:
This hurts.
Prasanna Malaiyandi:
And so they were like, Hey, yeah, customer.
Prasanna Malaiyandi:
Great news.
Prasanna Malaiyandi:
We found your backup server so you can restore your data.
Prasanna Malaiyandi:
And while they were giving the customer back the backup server, they accidentally
Prasanna Malaiyandi:
deleted the data on the discs.
Prasanna Malaiyandi:
W. Curtis Preston: Uh, yeah.
Prasanna Malaiyandi:
So, um, yeah, the,
Prasanna Malaiyandi:
Like, I just wanna know your OVH, you claim
Prasanna Malaiyandi:
to be the third or the largest EU headquartered cloud provider.
Prasanna Malaiyandi:
But simple processes like this, you're not able to.
Prasanna Malaiyandi:
They, I get some of it's probably maybe they don't have the experience
Prasanna Malaiyandi:
or expertise or they never had to go through this process.
Prasanna Malaiyandi:
Hopefully now on they have better processes in place.
Prasanna Malaiyandi:
It's like a learning experience, but.
Prasanna Malaiyandi:
I feel for the customers.
Prasanna Malaiyandi:
W. Curtis Preston: Yeah, I feel for those customers, and I, and I, I,
Prasanna Malaiyandi:
I wish the best for the, the ones in the 140 class action lawsuit.
Prasanna Malaiyandi:
It doesn't sound good if the two people who specifically had it in
Prasanna Malaiyandi:
their contract only got $150,000.
Prasanna Malaiyandi:
By the way, I would state I.
Prasanna Malaiyandi:
Well, good for them.
Prasanna Malaiyandi:
Like it's sort of like.
Prasanna Malaiyandi:
I, I'm sure what they'd rather have is the data back.
Prasanna Malaiyandi:
It, it is sort of like, it's like a wrongful death lawsuit.
Prasanna Malaiyandi:
A wrongful death lawsuit.
Prasanna Malaiyandi:
Can't bring your, the, the, the deceased person back.
Prasanna Malaiyandi:
It's just in some way trying to compensate you and 150,000 Euros is not
Prasanna Malaiyandi:
gonna compensate anybody for anything.
Prasanna Malaiyandi:
Uh, I mean, I, I wouldn't mind, just for the record, if anybody wants to send me.
Prasanna Malaiyandi:
Uh, I'm good.
Prasanna Malaiyandi:
Like, you know, I, I'll take that, but for a large company that lost, you
Prasanna Malaiyandi:
know, their entire company and this is all they're getting, one interesting
Prasanna Malaiyandi:
thing that the judge, a, a point that the judge made was that there was no
Prasanna Malaiyandi:
reason, there was no incentive or, or whatever for OVH to configure backups
Prasanna Malaiyandi:
in the way that they configured them.
Prasanna Malaiyandi:
Meaning that they're not a brand new hosting company
Prasanna Malaiyandi:
that only has one data center.
Prasanna Malaiyandi:
He's saying you have data centers all over the place and you have clearly
Prasanna Malaiyandi:
bandwidth between these data centers.
Prasanna Malaiyandi:
So there was no reason to configure backups the way that you configured them.
Prasanna Malaiyandi:
You could have just as easily made sure that, that, that the backup server is
Prasanna Malaiyandi:
always in another data center, but you just couldn't be bothered doing that.
Prasanna Malaiyandi:
So that, that's just the way the judge saw things and, uh, ruled in the, you know,
Prasanna Malaiyandi:
in the favor of the, of the plaintiff.
Prasanna Malaiyandi:
So when, when we look on this, you know.
Prasanna Malaiyandi:
It's difficult to talk about lessons learned
Prasanna Malaiyandi:
Is it though?
Prasanna Malaiyandi:
W. Curtis Preston: at least, well, because like, like even the customer that
Prasanna Malaiyandi:
did what we would've said to do, right?
Prasanna Malaiyandi:
So, you know, number one, review your contract.
Prasanna Malaiyandi:
And so the customer reviewed that contract and said, Hey, what do you mean
Prasanna Malaiyandi:
you're keeping in the same data center?
Prasanna Malaiyandi:
That's nonsense.
Prasanna Malaiyandi:
You need to put it in another data center.
Prasanna Malaiyandi:
And then OVH.
Prasanna Malaiyandi:
Said, sure.
Prasanna Malaiyandi:
but so.
Prasanna Malaiyandi:
W. Curtis Preston: they put it at,
Prasanna Malaiyandi:
So, so here's, I think, how they could have done it, which
Prasanna Malaiyandi:
is by doing DR testing or restore testing,
Prasanna Malaiyandi:
W. Curtis Preston: Okay.
Prasanna Malaiyandi:
All right.
Prasanna Malaiyandi:
So, so trust and trust, but verify the old phrase from Mr.
Prasanna Malaiyandi:
Reagan there.
Prasanna Malaiyandi:
Shoot your production in the head and try to restore your data.
Prasanna Malaiyandi:
W. Curtis Preston: wait, not like the Alaska, not like.
Prasanna Malaiyandi:
Yeah.
Prasanna Malaiyandi:
Not, yeah, not like
Prasanna Malaiyandi:
W. Curtis Preston: like that episode.
Prasanna Malaiyandi:
Okay.
Prasanna Malaiyandi:
So, so simulate shooting your server in the head and then,
Prasanna Malaiyandi:
uh, and then trying to restore.
Prasanna Malaiyandi:
Uh, but well, but, but in, how do they simulate shutting down the data center?
Prasanna Malaiyandi:
Or you just prevent any access at a new data center to any
Prasanna Malaiyandi:
IP addresses in the old data center,
Prasanna Malaiyandi:
W. Curtis Preston: Like how do
Prasanna Malaiyandi:
and then try to do your restore.
Prasanna Malaiyandi:
So
Prasanna Malaiyandi:
basically.
Prasanna Malaiyandi:
W. Curtis Preston: need to verify that.
Prasanna Malaiyandi:
It's just, it is just that there is a certain degree of trust that
Prasanna Malaiyandi:
you are placing in the vendor that they're going to do the things
Prasanna Malaiyandi:
that they say they're doing.
Prasanna Malaiyandi:
Um, you need to figure out a way to verify that.
Prasanna Malaiyandi:
But I, I would say that the easier answer here is that, um.
Prasanna Malaiyandi:
Is to use a third party backup service.
Prasanna Malaiyandi:
And because while you could contractually, you could get like,
Prasanna Malaiyandi:
like you can use Amazon backup, right?
Prasanna Malaiyandi:
And you can snapshot your way to happiness and then you can replicate those
Prasanna Malaiyandi:
snapshots to another region if you want.
Prasanna Malaiyandi:
And you could probably figure out some way to verify that those
Prasanna Malaiyandi:
snapshots are, are in another region.
Prasanna Malaiyandi:
The um.
Prasanna Malaiyandi:
But if you use a third party backup service, it's like you,
Prasanna Malaiyandi:
it's like you're guaranteeing that the backups will be somewhere else
Prasanna Malaiyandi:
other than what you're backing up.
Prasanna Malaiyandi:
Because to me that's like the fundamental, you're shaking your
Prasanna Malaiyandi:
head, I know what you're thinking.
Prasanna Malaiyandi:
We could we, you can argue it in a minute.
Prasanna Malaiyandi:
I'm just saying that to me, like the fundamental thing of backup.
Prasanna Malaiyandi:
Is to make sure the backup is somewhere else.
Prasanna Malaiyandi:
Uh, so you're thinking that if a customer is running in the same
Prasanna Malaiyandi:
region,
Prasanna Malaiyandi:
No, no, no.
Prasanna Malaiyandi:
I, I wasn't, I wasn't even, I
Prasanna Malaiyandi:
W. Curtis Preston: you select what, what were you thinking?
Prasanna Malaiyandi:
I was thinking that if I'm a customer and I'm going
Prasanna Malaiyandi:
to a vendor to offload my need to have infrastructure and to manage all these
Prasanna Malaiyandi:
resources, am I as a customer going to go pay and find a backup service and manage
Prasanna Malaiyandi:
a backup service to go deal with this?
Prasanna Malaiyandi:
Right.
Prasanna Malaiyandi:
I think that that bar gets pretty high depending on how complex it is.
Prasanna Malaiyandi:
And because you also need to find a backup service that works with
Prasanna Malaiyandi:
the cloud provider you're using.
Prasanna Malaiyandi:
Right.
Prasanna Malaiyandi:
Which may be harder for the non top three clouds that you have, right?
Prasanna Malaiyandi:
non-Amazon, Azure, or
Prasanna Malaiyandi:
W. Curtis Preston: Yeah.
Prasanna Malaiyandi:
So I.
Prasanna Malaiyandi:
Okay.
Prasanna Malaiyandi:
I, I thought you were arguing something different.
Prasanna Malaiyandi:
I, I guess I would argue that that's the job,
Prasanna Malaiyandi:
It, it is.
Prasanna Malaiyandi:
I agree.
Prasanna Malaiyandi:
It is a job when you move to
Prasanna Malaiyandi:
W. Curtis Preston: you're developing infrastructure, if you're developing
Prasanna Malaiyandi:
infrastructure backup as part of the infrastructure, if you don't
Prasanna Malaiyandi:
know how to do cloud vendor A without also backing up cloud vendor
Prasanna Malaiyandi:
A, then you shouldn't use Cloud vendor A,
Prasanna Malaiyandi:
Yeah.
Prasanna Malaiyandi:
I agree with that.
Prasanna Malaiyandi:
Yeah.
Prasanna Malaiyandi:
W. Curtis Preston: okay, so what are you saying?
Prasanna Malaiyandi:
No, I, I agree.
Prasanna Malaiyandi:
But people don't really think about backup.
Prasanna Malaiyandi:
Right.
Prasanna Malaiyandi:
W. Curtis Preston: Well, that's their problem.
Prasanna Malaiyandi:
There isn't, that's why we exist, right?
Prasanna Malaiyandi:
Um, I
Prasanna Malaiyandi:
I can't
Prasanna Malaiyandi:
devil's.
Prasanna Malaiyandi:
I'm just playing devil's advocate.
Prasanna Malaiyandi:
You got a thousand things on
Prasanna Malaiyandi:
W. Curtis Preston: well, you're doing a good job.
Prasanna Malaiyandi:
You have a thousand things on your mind and like Right.
Prasanna Malaiyandi:
Backup is not gonna be your top priority.
Prasanna Malaiyandi:
W. Curtis Preston: So that, that, that's a problem as old as time.
Prasanna Malaiyandi:
No one thinks about backup until, you
Prasanna Malaiyandi:
but, but I think there is a solution though for this.
Prasanna Malaiyandi:
Right, and I think the solution is as an industry, we should hold vendors
Prasanna Malaiyandi:
accountable for providing the bare minimum needed, at least a bare minimum
Prasanna Malaiyandi:
needed for what is safe backups, right?
Prasanna Malaiyandi:
If you are saying, I'm doing backups, here's the bare minimum, right?
Prasanna Malaiyandi:
Follow the 3, 2, 1 rule.
Prasanna Malaiyandi:
Keep it out.
Prasanna Malaiyandi:
Separate in a, uh, right, that ensures that your backups are isolated, right?
Prasanna Malaiyandi:
These sort of things.
Prasanna Malaiyandi:
I think there should be a bare minimum.
Prasanna Malaiyandi:
We should hold vendors accountable to.
Prasanna Malaiyandi:
And until we get to that point, I think customers should ask their vendors,
Prasanna Malaiyandi:
Hey, here are five questions when I'm going to go use your service, and I put
Prasanna Malaiyandi:
it out and ask you questions, answer these five questions related to backup,
Prasanna Malaiyandi:
W. Curtis Preston: I, I think that's a perfectly valid thing to do.
Prasanna Malaiyandi:
I just know that with, with most of the SaaS services, the answer is no.
Prasanna Malaiyandi:
Right?
Prasanna Malaiyandi:
which is okay, but at least you have the
Prasanna Malaiyandi:
answers.
Prasanna Malaiyandi:
And then now you can start to think
Prasanna Malaiyandi:
about, yeah.
Prasanna Malaiyandi:
And now you can be like, Hey, maybe if I want to go use say ServiceNow, I should
Prasanna Malaiyandi:
figure out, okay, how am I gonna back up that data if it's important to me?
Prasanna Malaiyandi:
Because they don't offer
Prasanna Malaiyandi:
W. Curtis Preston: Agreed.
Prasanna Malaiyandi:
And maybe, maybe I'll live in a utopia that where that one day, there's two
Prasanna Malaiyandi:
products that are really impressive and one of them has backup and the other
Prasanna Malaiyandi:
one doesn't as part of the service.
Prasanna Malaiyandi:
And the one with backup wins the
Prasanna Malaiyandi:
Yeah.
Prasanna Malaiyandi:
And then the other one eventually figures out they're losing deals
Prasanna Malaiyandi:
because they don't have it, and then they start offering it, right?
Prasanna Malaiyandi:
W. Curtis Preston: That is, that's what we call the free market.
Prasanna Malaiyandi:
Um, yeah, I, I, I think that's really important.
Prasanna Malaiyandi:
Um, and I, I just think that, um, this is a really bad story because it's
Prasanna Malaiyandi:
just because some people who thought they were doing the right thing weren't
Prasanna Malaiyandi:
actually doing the right thing because the vendor wasn't doing the right thing
Prasanna Malaiyandi:
and they were actually misrepresenting.
Prasanna Malaiyandi:
That's the worst part of the story, is that they were actually
Prasanna Malaiyandi:
misrepresenting what they were doing.
Prasanna Malaiyandi:
And, um,
Prasanna Malaiyandi:
It's like, Hey, I bought a car.
Prasanna Malaiyandi:
It's supposed to get a hundred miles per gallon, but it only gets 10.
Prasanna Malaiyandi:
You know?
Prasanna Malaiyandi:
W. Curtis Preston: It's more like,
Prasanna Malaiyandi:
Prasanna Malaiyandi: Or, or, Or, or, it's, or
Prasanna Malaiyandi:
W. Curtis Preston: like it doesn't go at all.
Prasanna Malaiyandi:
no, no, no.
Prasanna Malaiyandi:
I was gonna say, or I have a car.
Prasanna Malaiyandi:
I bought it.
Prasanna Malaiyandi:
It has airbags.
Prasanna Malaiyandi:
I get into a wreck.
Prasanna Malaiyandi:
The airbags don't deploy.
Prasanna Malaiyandi:
W. Curtis Preston: Well, it turns out there's no
Prasanna Malaiyandi:
Yeah.
Prasanna Malaiyandi:
There are no airbags.
Prasanna Malaiyandi:
W. Curtis Preston: just no airbags at all.
Prasanna Malaiyandi:
And they're like, well, the airbags are only in the cars that are in the factory.
Prasanna Malaiyandi:
Uh, the demos, we, we didn't put them in
Prasanna Malaiyandi:
speaking of this, so I was actually reading a Reddit article
Prasanna Malaiyandi:
the other day, or a Reddit post and uh.
Prasanna Malaiyandi:
Someone was complaining and the story goes that they were sleeping, they were
Prasanna Malaiyandi:
renting a place, they were sleeping, and they heard a beeping and they go look.
Prasanna Malaiyandi:
And it was like one of those carbon monoxide detectors that they had
Prasanna Malaiyandi:
bought at a previous rental and brought it, but they never installed it.
Prasanna Malaiyandi:
They're like, huh, that's weird.
Prasanna Malaiyandi:
So they open all the windows and then it keeps going off and off.
Prasanna Malaiyandi:
And then they called a fire department and apparently their fireplace
Prasanna Malaiyandi:
was leaking carbon monoxide.
Prasanna Malaiyandi:
So they're like, that's at least it ca.
Prasanna Malaiyandi:
In their rental.
Prasanna Malaiyandi:
So they're like, good.
Prasanna Malaiyandi:
At least it was caught.
Prasanna Malaiyandi:
And then the fire department goes around.
Prasanna Malaiyandi:
Apparently all their smoke detectors were just shells.
Prasanna Malaiyandi:
There was no actual smoke detector inside of 'em.
Prasanna Malaiyandi:
W. Curtis Preston: So, yeah, well, at least that one is again, that's fixable.
Prasanna Malaiyandi:
You know that you can go around and test the smoke detectors in an Airbnb you're
Prasanna Malaiyandi:
saying, or staying, but that's just wrong.
Prasanna Malaiyandi:
So the person had had a battery powered,
Prasanna Malaiyandi:
Prasanna Malaiyandi: Carbon monoxide detector?
Prasanna Malaiyandi:
W. Curtis Preston: carbon monoxide detector in their like luggage or
Prasanna Malaiyandi:
No.
Prasanna Malaiyandi:
So this was a rental, right?
Prasanna Malaiyandi:
So they were renting like long term, like a year rental.
Prasanna Malaiyandi:
Right.
Prasanna Malaiyandi:
And so they had just moved into the place and from their old place,
Prasanna Malaiyandi:
they had brought whatever they had.
Prasanna Malaiyandi:
And it just happened to be just sitting in like a closet somewhere.
Prasanna Malaiyandi:
W. Curtis Preston: Wow.
Prasanna Malaiyandi:
Saved their
Prasanna Malaiyandi:
Yep.
Prasanna Malaiyandi:
Saved their life.
Prasanna Malaiyandi:
W. Curtis Preston: I.
Prasanna Malaiyandi:
Yeah.
Prasanna Malaiyandi:
Um,
Prasanna Malaiyandi:
Go replace your check your smoke detectors, folks.
Prasanna Malaiyandi:
W. Curtis Preston: your smoke detectors and your carbon monoxide
Prasanna Malaiyandi:
detectors, um, at which for the record need to be two separate devices.
Prasanna Malaiyandi:
For a while I was like, this is annoying that, uh, I mean, this is totally
Prasanna Malaiyandi:
off the subject, but for a while I was, I was saying that CO2 detectors
Prasanna Malaiyandi:
and co detector, I'm sorry, not co fire detectors or smoke detectors.
Prasanna Malaiyandi:
It should be combined.
Prasanna Malaiyandi:
And, and there are companies that sell combined smoke and
Prasanna Malaiyandi:
fire and, and co detectors.
Prasanna Malaiyandi:
Do you know why they should absolutely not be in the same place
Prasanna Malaiyandi:
or the same unit?
Prasanna Malaiyandi:
Because they rise differently.
Prasanna Malaiyandi:
W. Curtis Preston: Yeah.
Prasanna Malaiyandi:
Smoke goes up, co goes down, so co detectors should be
Prasanna Malaiyandi:
about a foot off the floor.
Prasanna Malaiyandi:
Uh, and smoke detectors should be up on the ceiling and they
Prasanna Malaiyandi:
should never, and they should therefore never be the same device.
Prasanna Malaiyandi:
Um, and how it's actually legal to sell them in one thing, I don't know.
Prasanna Malaiyandi:
Anyway.
Prasanna Malaiyandi:
So, uh, maybe I just scared the crap outta somebody and they're gonna
Prasanna Malaiyandi:
go Now buy, uh, separate CO two
Prasanna Malaiyandi:
you get those plugin ones, which is, which is perfect because
Prasanna Malaiyandi:
the outlets are at the right height.
Prasanna Malaiyandi:
W. Curtis Preston: Exactly.
Prasanna Malaiyandi:
Exactly.
Prasanna Malaiyandi:
Uh, all right.
Prasanna Malaiyandi:
Any final thoughts on the OVH Fire?
Prasanna Malaiyandi:
Uh, no.
Prasanna Malaiyandi:
I think, yeah, like you said, it's people were trying to do the right thing or
Prasanna Malaiyandi:
they assumed that the right thing was being done and they got burned and yeah.
Prasanna Malaiyandi:
I don't know what else we could recommend for them to do.
Prasanna Malaiyandi:
W. Curtis Preston: I.
Prasanna Malaiyandi:
Yeah, verify as much as you can that your backup infrastructure is separate.
Prasanna Malaiyandi:
And again, I think the easiest way to do that is to use a different vendor.
Prasanna Malaiyandi:
Um, but you know, maybe it's because I used to work for one of those
Prasanna Malaiyandi:
companies that there was a different vendor, but I don't anymore.
Prasanna Malaiyandi:
I still think that way.
Prasanna Malaiyandi:
Anyway.
Prasanna Malaiyandi:
Well, uh, thanks for the chat persona.
Prasanna Malaiyandi:
anytime.
Prasanna Malaiyandi:
Thank you, Curtis.
Prasanna Malaiyandi:
W. Curtis Preston: Thank you to our listeners.
Prasanna Malaiyandi:
Uh, you know, you're why we do this.
Prasanna Malaiyandi:
That is a wrap.
Listen On
