Surviving a Data Center Flood: Real Recovery Stories

You found the backup wrap up your go-to podcast for all things

backup recovery and cyber recovery.

In this episode, we talk with a backup admin that's been doing the same job at

the same company for almost 20 years.

He shares an incredible story about what happened when a hurricane hit

their data center on an island.

Spoiler alert, one data center was completely flooded and

Iron Mountain got hit too.

That's a double whammy if I've ever seen one.

He's a good friend of mine that went through a huge disaster.

We disguised his voice and his name so that he could tell

the story without varnish.

This is part one of two parts that cover the same story from

two different points of view.

This is a great episode from a couple of years ago, so we thought

we'd air it during the holidays.

I hope you enjoy it.

By the way, if you don't know who I am, I'm w Curtis Preston.

AKA Mr.

Backup, and I've been passionate about backup and recovery for over 30 years.

Ever since.

I had to tell my boss that our production database was gone and we had no backups.

I don't want that to happen to you, and that's why I do this podcast.

On this podcast, we turn unappreciated backup admins into Cyber Recovery Heroes.

This is the backup wrap up.

Welcome to the show.

I'm your host, W.

Curtis Preston, AKA, Mr.

Backup, and I have with me the guy that I'm now calling Mr.

Hare Persona.

How's it going?

Good, Curtis.

How are you doing?

Uh, it, it's been, it's been a busy week.

I am, I am.

Really working hard on the book.

And, uh, I actually just finished another chapter.

Uh, is there only one left?

No, there's two more left.

Um, that's impressive though.

Yeah.

And the, the chapter that I just wrote is the hardest chapter so far, uh, because

it was, it really got, it went into detail on different product categories.

So, um, I'm pretty.

We actually have a long time friend on the podcast here.

We're going to use a pseudonym for this person because we want, as

we've said before, we would love to have like real people that are

actually firing backups in anger.

Out there in the real world.

Come on the podcast.

And we've only done that a couple of times and, and we are happy to disguise

your voice and to not say where you work and to not use your real name.

And, uh, he's been in it for over 30 years and been doing backups

at the same company for almost 20 of those 30 years, which is just.

Wow.

Amazing.

Wow.

And those backups should finish any day now, but I'm so welcome

to the podcast, Harry Potter.

Thank you.

It's good to be here.

Thanks for being on the show, Harry.

Yeah, thanks, Harry.

Your company's a pretty stinking big company, right?

Yeah.

It's a multinational company.

Uh, and it makes stuff

and they make, they make stuff.

Yes, you are.

You do work for a company that actually makes stuff.

Yeah.

So that, that.

We're, we're, we're being vague on purpose, and I do apologize for

those that would like to know more.

But again, we want Harry to tell us all about his environment.

And so, uh, we're allowing him to anonymize things for that purpose.

I, I,

I will say that the company I worked for before that was one

that made big kerosene burners.

You,

you did work for a company that made big kerosene burners.

Yes, yes.

Uh, big kerosene burners that I have.

Spent quite a bit of time in.

Yeah, there you go.

And no longer are.

And no, and no longer are, oh, ugh.

Yeah.

Covid.

It comes up everywhere.

Yeah.

I do need to throw in our usual disclaimer persona.

And I do both work for Druva.

This is not a Druva podcast.

The opinions that you hear are our own.

And obviously Harry Potter is not representing Hogwarts or any

other commercial institution.

Uh, and he is, uh, speaking about his or his own opinions.

Sorry, I know that we're gonna get into what.

The environment looks like and all the rest, but how has it been

like working in the same field for 18 years at the same company?

I'm sure you've seen lots of changes.

So it's, it's been, it's been really interesting, you know, over the

last almost 20 years working with NetBackup and having seen them go

from Veritas to Semantic and back to Veritas again, and seeing them go from

having vision to not having vision.

Yeah,

not everybody will get that joke.

They used to have Veritas vision.

Actually, the original name of the con of the conference was Vera K.

Do you did?

Did you know that?

Oh,

no, I didn't.

I didn't know that.

That seems awful though.

The original name was Vera K, which in French sounds bad.

Khan I think means idiot or something like that.

So it sounds like.

Big idiot or something.

And so for that and other reasons, they chose to change the name to vision.

But then at some point, Veritas no longer had vision.

Yeah.

Because they stopped.

They stopped having the conference,

they stopped doing it all together.

Uh, and that's, that's

not a covid thing that they stopped doing it a few years ago.

Right.

Yeah.

So I, I mean, I, I started out, um, at this company several years ago using,

um, oh gosh, it was, I think it was net.

Something and now we're up to eight something.

So it's gone through a lot of changes and it's, you know, a lot of it's been.

Pretty, uh, pretty good.

I mean, I like, I like some of the things that, that they've done with it.

And we've got our own really nice reporting tool that we built

in-house, so we don't use something like Boada or something like that.

Um, and, and so it's gone through a lot of changes.

Um, I've, you know, had a number of managers that I've worked for

over the years that have been good.

I've enjoyed my time there.

What about your requirements from your users and things like that?

Like how has that shifted?

Sorry, Curtis, I know I just cut you off.

That's all right.

We're of the opinion that if you, you know, create a new

system, you know, you build a new system, deploy a new system, I.

And, and you know, we don't read minds and, uh, you have to tell us about it.

So if you don't tell us about it, we don't back it up.

Uh, and, and there have been instances where people have come to us and

said, Hey, our system just crashed and we need you to restore it.

And we, we look it up and we say, um.

Yeah, it's not in backups, you know, and, uh, if you didn't tell us about that,

then there's not a whole lot we can do.

And then they, they get really embarrassed and, and say, uh, oh

yeah, I guess we missed up on that.

Um, but we have, um, standard operating procedures where basically we say.

If you want it to be included in, uh, regular backups, you need

to give us your requirements.

You need to say, how frequently do you need it backed up, how important it is.

Because we don't have a one size fits all type of setup.

I mean, there was a, a company that I worked with, um, that did exactly that.

They said, Hey, all these systems, we don't care what they are.

Dev test fraud, doesn't matter.

They're all getting the same kind of backup, but we don't do that.

So Interesting.

So we, we, we tell them that they have to give us their requirements in order for us

to do the, you know, the backups properly.

So I have provided consulting services to your company, as I

recall, on more than one occasion.

Mm-hmm.

Right?

Oh no.

So both.

Both in the location where you currently work, as well as the location

where, boy, this sounds so cagey.

The location where I originally met you.

Right.

Yeah.

Yeah.

Right.

Yeah.

So,

so, um, you originally came to our headquarters and helped us

with some DLT tape backup tuning

dlt.

I had for forgot Yeah.

I had forgotten all about DLT.

Yeah.

What I recall about your environment that at the time was very eyeopening to me.

You had a particular manager who had.

A love affair for NetApp, um, and wanted to do all things.

Na, this, this is more the southern folks.

Um, and he, I has that, has that continued on or you still a big NetApp shop or no?

Oh, definitely.

Definitely.

Okay.

Yeah.

Um, and, and in fact, um, a lot of stuff rather than doing NDMP.

They're doing, uh, you know, SnapVault

of course, right?

It lives on, it lives on, yeah.

Snap Vault.

Yeah.

So persona, persona has a relationship, uh, a different

relationship with SnapVault there.

Yeah.

Um, and the, yeah.

The thing that, the thing that I found interesting at the time,

that was the, it was the first very large company that was using.

And was saying, listen, why would I wanna do it any other way?

Why do I need San, this was back, you know, when I first Oh, yeah.

Met your company, San was the craze, right.

And large, uh, EMC storage arrays Right.

Was the craze.

And NetApp was, uh, considered.

More like work group and that sort of stuff.

And it was the first large company that I got to where I saw them

pushing the envelope for nasa.

And that actually, that that interaction with your company, uh, influenced

my, my second book, uh, quite a bit.

The, the Sans and Nas book

San Yeah.

Yeah.

Um, and you guys have, so you do you.

So you use SnapVault where you can, it sounds like you use NetBackup quite a bit.

Yes.

Describe what that system looks like, the NetBackup side of things.

Um, okay.

So, so nowadays what we've been doing is we've, we've used, we're

using a lot of DDU storage and it's either in the form of what we're.

Primary, we're moving away from, how do I wanna put it?

Single vendor DDU storage and going to net back up appliances, which is

another single vendor, DDU storage.

But, um, but yeah, so we're, we're doing that.

And then in addition to doing that, that allows us to replicate

to a remote, uh, location.

So that's our offsite copy of our, of our backups.

Are you tapeless at this point?

Not.

Quite, there are places and one of them will talk.

Well, we're, we're gonna get to, we're

gonna get to that.

Okay.

Let me rephrase.

Are you predominantly tapeless at this point?

I'd say we're probably 70% Tapeless.

Okay.

Okay.

But there are still some places.

Yeah, we we're moving away from data domain and, and Quantum DSI and moving

to, uh, net backup, uh, appliances.

Which have

you got, you got, you know.

One or more net backup master servers, a bunch of media servers.

And then behind those are, are, are these media servers then connected to net

backup appliances or are they, is it like a media server appliance combination?

It's a media server appliance combination.

Okay.

So,

and, and, and in some cases now we're also going to media server appliances that have

a cloud storage attached so we can Okay.

We can replicate things to the cloud up to object storage.

S3.

Yes.

Yes.

Gotcha.

Per locate.

When you look at a location, like think of like your largest data centers, do

you tend to have one NetBackup master in there, or do you end up needing

multiple NetBackup masters for a location?

Lately we've been going towards an architecture where we have

master servers that, okay.

We'll replicate to each other that are located in physically different buildings.

What are they replicating?

The index?

No, everything.

You know, so it's, it's using, it's using, um, net backups air, which

is automatic image replication.

You know, so it's getting, okay, so you're literally

copying backups between.

Like entirely separate net backup domains.

Yeah.

And then, um, but, but I mean that's,

but within, but within each building, do you need one or more, uh, master servers,

um, one.

And maybe, maybe one media server to go with it.

If you did a full backup, how big would that full backup be?

That's a good question.

I think it's in, it's in the petabytes.

I just don't recall what it is right off the top of my head.

Alright.

Um, but if I, if I pulled up our homegrown reporting tool, I could probably get

some information out of it for you.

Right.

So you're managing a petabyte, essentially a petabyte of primary data.

So I, I like this feature of copying the data between the data centers.

I like that a lot.

I, I'm assuming this uses deduplication, so you're replicating Yes.

New and

unique blocks

it, sorry, I know you said it's in the same location, but two separate

physical buildings, and then you also have another copy going to

a completely different location.

Is that correct?

Yeah.

I mean that's, that's another element of it.

And, and it's not, it's not the same in each location, but it's, we're trying

to standardize on, on, um, on a, uh, um, a single design for the, uh, for this.

But,

but again, we we're, and is, is that future single design, the

cloud method that you described?

You know, that's where we're gonna go to Tapeless.

So there are places, for example, in EU where we can, um, go to a, uh, cloud

storage for the offsite part of it, and that's where we'll be going tapeless.

And we're just starting to do that.

But there are still some places where we can't do it because of, you know, whatever

rules there are within the particular countries that we're talking about.

Regulations.

Yeah, like change regulations, right?

Data must be stored on tape.

Or, or something.

Yeah.

Or it must be stored.

You can't leave the country or some sort rule like that.

Right.

You de, you definitely encounter that in certain European countries, right?

Yeah.

Yeah.

Switzerland for one, I think.

I dunno, what's that?

Switzerland?

Yeah.

I'm pretty sure Germany is too.

Yeah, may.

Yeah, it could be.

But Germany's part of the eu so they'll, you know, I think if it doesn't leave

the EU then it's okay, but I don't know.

I'm not, I'm not sure, but I don't live in Germany, so That's okay.

Yeah.

Are you, um, go

ahead.

Sorry, were you gonna continue on the data center?

I was gonna ask about if he has any cloud.

Yeah, ask whatever you want.

Go ahead.

So do you have any workloads that you are running in the cloud that

you are responsible for protecting?

Or is it mainly the data center workloads that you're.

I'm, I'm primarily concerned with the data center on, on premise

stuff and not the cloud stuff.

I mean, we do have, we do have servers in the cloud, but um, but

yeah, I'm not responsible for.

I don't directly interact.

It's not, it's not your

job.

Yeah,

yeah.

It's not my job.

Yeah.

There you go.

So, but even when you start looking at cloud as being that object store,

are you gonna be responsible for that or would you be working with another

team to sort of get access to that, for them to manage that and then

you to just dump data into a bucket?

It's part of our team, but I think the, uh, um, yeah, it's part of our team,

but it's another couple of guys on our team that, that are responsible for it.

You and I spent quite a bit of time talking about a particularly

problematic situation.

Yeah.

In a a, an island in the middle of the ocean, um, that.

Got hit by a hurricane.

Yeah.

And you had data centers there?

Yeah, we did.

And, and how, how was data, how was the, the NetBackup set up configured there?

So we had two data centers.

Each had their own master media servers.

Mm-hmm.

And also they each had their own tape libraries, but the primary

offsite copy was, you know.

The guy in the pickup truck taking it to Iron Mountain and,

uh, and the, the man in the

van?

Yeah, the man in the van.

And the other part of it was that there was replication going on

between the two data centers.

So we had Dedup devices that would, um, replicate, you know, backups

from one data center to the other.

Uh.

So as a result of that hurricane, one of these data centers got

completely, uh, drowned and, uh, the roof, uh, had, uh, leaks in

it and the data center was a mess.

And so we ended up with one of the two data centers and we had to recover.

The majority of those systems from the backups that were

in the other data center.

There was, so you were

replicating all the backups, right?

And then you also had this tape copy that you hoped you didn't have to use.

Right.

And so you The bad, go ahead.

The bad thing about the tape copy was because this island was hit so hard by

that hurricane, iron Mountain was also hit and there was a several day or week

delay in them being able to return tapes.

Ugh.

Yeah.

That, that's got, that's like that, that falls into the insult

to injury or pouring salt on a wound, whatever you wanna Exactly.

Um, it's bad enough what happened to you, but then to not be able to get your stuff.

Of course.

So you, you said you, you were gonna recover in this other data center.

So then the other problem that you might have is not being able to

get the actual servers and whatnot.

To restore, to Right.

Restore

the, the physical servers.

Yeah.

And I Yeah.

You know,

or whatever it, whatever it is you were restoring to, or storage or whatever.

Yeah.

Storage.

Yeah.

Yeah.

I, I, I, I think the main part of that was I think they were able to recover

a good number of the physical servers.

But they, it basically had to move them all out of the damage

data center and into another one.

And they, they, oh, basically, yeah.

They basically abandoned the building that they were in, but they were able to

rescue the, the servers and store it.

They were able to rescue most of them.

I.

I don't think they were able to rescue all of them.

Um, but, but did they do it

beforehand?

They didn't do it before the hurricane hit?

No, no, no.

They didn't do it before the hurricane.

Um, they did it afterwards when they were in the recovery

slash just trying to figure

out like how did, how did the data center get flooded but the

servers didn't get destroyed?

I guess that's, and it might

depends like maybe the servers at the top of the rack might be okay.

Right.

I've seen that happen.

Yeah.

Yeah.

Or might not.

The ones at the bottom of the rack might be okay because it's coming

from the top down or something.

It all depends.

But, uh, I don't, that's the part I'm not a hundred percent aware of because

they took another guy from our team and had him fly, you know, take a commercial

flight to corporate headquarters where he got on the corporate jet with a

bunch of other recovery people to go to this place to do the recovery.

Right.

And I, I, I remember at the time you, you almost were gonna go to that scenario.

I, I almost would've gone, but, but uh, but for the grace of

God, somebody else gotta go.

Yeah.

And the poor guy, because this is, um, um, a, um, an island which is.

Closer to the equator than we are.

He was, he was really upset that he didn't bring short pants.

He had long pants, and so he was, oh man.

I think, I think those short, those long pants would turn into

short pants real quick, if that.

So what, what else do you know about how that went?

You know, what was that like?

Well, there was a lot of, of, um, having to.

You know, reintroduce these systems into the, uh, network and into the, into DNS

and into the, the backup environment.

And they, they were going through a constant, um, uh, array of,

Hey, how do we get this part done?

And, you know, fortunately they had brought with them some really good

networking people and, uh, they were able to do a lot of the stuff.

But there was some things that, and I, again, it's been a while and so

I don't recall all of the, uh, the details of what they had to do, but,

you know, it was, it was pretty intense for a couple of couple of weeks.

This is similar to a past podcast that we did where it was around

disaster recovery, and it's like.

Sometimes you forget that there are other people in the infrastructure

who you need in order to bring up your production, not just the data.

Yes, exactly.

So, so that was, that's a, uh, that was the case here.

It's interesting though that they ended up replicating to a, given

that it's an island you would.

Think that they might be replicating data outside of the island to off

the island.

Yeah.

Um, I, I think the problem is just the cost of doing something where

they could get it off of the island.

That was, that was where the problem was.

So I, I think yes, they have, they have satellite up links, but they're, they're

not as fast and they're, they're pretty expensive for, for what they wanna do.

And they, you know, they, they make the decision to not spend the money on that.

So once the first data center failed, and I believe you said

that they were replicating between the two data centers, right?

Their backups,

yes.

Yeah.

So what did they do then in that?

In the only running data center, like as they were taking backups,

was it still just in one location because Iron Ma Mountain had a delay,

the other data center was gone.

What was, what were they doing?

Was it just hope nothing gets affected in the first data

or in the second data center?

Yeah, kind of.

Exactly.

Um, but, but yeah, they had to, um, uh, keep running their, their normal

backups in that data center and also do all these restores to the new,

uh, location where they had moved, you know, the servers that they had

taken out of the damaged data center.

And do you know how long it took them to recover?

That sounds like it must have been a couple of weeks at least, right?

It was

at least a couple of weeks, you know, and I, and I think, uh, my friend

was down there for about three weeks.

Wow.

I could, I can imagine the pressure and the stress because no one had

probably ever tested out that scenario.

Right.

What does it take to restore in case by data center?

It goes back to what Curtis always says, right?

It's like, test, do your backups and restores work.

Yes.

Yeah.

Yeah.

And, and it sounds like the much of the problem, or many of the problems

that you had weren't the backups, they were all of these other things that

you needed to do once you restored the system to an alternate server.

Yeah.

Yeah, exactly.

But how, how did the actual backup systems perform though?

They performed well.

And, and we didn't, I don't believe that we went back to tape on anything, um,

even though that could have happened.

Uh, but what we also found at the same time was there were little pieces

of information here and there that were on some of these servers that.

Weren't getting backed up the way that the business owners thought would've liked.

Mm, yeah.

Yeah.

Not what they had thought.

You know?

It's like they thought, oh, everything's getting backed up.

But it wasn't.

And they didn't find out.

So weren't until we started.

So you weren't using

all local drives or, or something?

No, it wasn't, it

wasn't that.

Well, partly that, but also, you know, there'd be a database server where, um.

They weren't doing a sweep of the drive that had the database

backed up to that kind of thing.

Right.

Oh, ouch.

Yeah.

So right there were, there were little things like that.

There's no way to find those things other than to do a full test recovery.

Right.

Which, which, you know, if, if you look back at.

You know, even, even when I, when I worked at this, you know, large credit card

company that we did this twice a year, but we never did a full scale recovery.

We just did a, a, a, a subset recovery of a complete system, right?

So we, we, we picked a mission critical app.

Then we had to restore all infrastructure necessary for that app to exist,

which would include things like the DNS server and DNIS server.

Remember NIS, uh oh.

Yeah.

I don't think that's, is it still around?

No one still uses that anymore, do they?

I don't know.

Yeah.

Um, but, you know, and all of the infrastructure around

it, we had to do that.

And, and as you know, I've often said we never had a fully successful restore.

In that one that went as designed and documented without having to

do something, without having Curtis

explain it.

Yeah, exactly.

So the good news is, you know, you're not alone there.

Yeah.

I mean, I remember in, in the early days at this company, we used to

pack a bunch of DLT tapes, uh, off to a data center in Arizona.

And try to recover an entire application there.

Right.

But you'd never did a full data center because that would just be insane.

Like, it'd be, it would take three weeks.

Right, right, right.

And, and in this situation, when we did these smaller tests, it was,

it was like one application, which consisted of maybe three or four

servers, like a web server, application server, database server type thing.

And then we wanted to see if it, and make it work.

Um.

Most of the time we were able to make it work from restoring

from those tape backups.

But it wasn't, it wasn't any fun as you might imagine.

Right.

And, and now of course with being able to replicate to a remote site, they can

just say, okay, I think we're gonna flip the switch and make this other site live.

And uh, and they do it that way and it's a lot quicker and a lot less stress.

And doing it.

Is this still the net backup method of rep replication or are

you talking about something else?

Um, both, uh, net backup replication and also, um, asynchronous replication

of the primary storage to another site.

Okay.

Right.

I mean, so that, you know, because there's some applications that are

just, um, have such a short RPO, uh, a daily backup isn't gonna cut it.

It's gotta be something asynchronous.

Uh, a database replication to another storage.

Right.

And then they, and then they can just kind of say, okay, we're gonna make that

the primary, and we'll go with that.

Yeah.

Without, without going into details of your company, I, I, you know,

I've spent enough time there to know that there are certain applications.

Where unique data, unique, once in a lifetime data is being created.

Yes.

Yeah, exactly.

You wouldn't be, you know, you, you might be able to redo it, but

you can't technically recreate it.

Right.

So, right.

Uh, so those, those applications would be the kind of

application I would think about.

Um, yeah, so you, so you have the primary replication as well.

Uh, you have the net backup replication because with, so, oh,

let, let's go back to this, this DR.

That you had and, and other DRS that you maybe are planning for.

Um, the ba is this still, is this VMware that we're talking about that

you're backing up and restoring?

Well, both VMware and physical.

Uh, okay, so you were restoring physical servers?

Yeah.

And with those, with those physical server, because VMware, essentially

BMR is, you know, it's included, right.

It just, you know, yeah.

It's, it's an assumed, but physical servers not so much.

How are you doing, or how, how does the system designed to do with those?

Well, I, I mean.

You know, for, for Windows systems, for example, you're backing up every,

you know, like all local drives, which includes the, um, shadow

copy components and system state.

Yep.

And when you're doing the restore of those, using that backup, depending

upon the version of the base operating system of Windows, you may have.

And you, you probably are familiar with those issues.

Yeah.

So you're, so what you, the, the difficulty would be is, is if the

backup that you have, uh, there are multiple scenarios, right?

You, you could have a backup of, you know, let's say, I'll just say Windows seven,

uh, and you're restoring to a Windows 10 server, and that's all you have.

Yeah.

Yeah.

Yes.

That's, that's what BMR, you know, bare Metal Recovery is designed to do.

Sure.

I know Net Backup has that functionality.

Yeah, but

we're not using that.

You're not using it.

Do you know why?

Yeah.

Um, it's got its own infrastructure that requires.

To do it.

And at least with net backup, it's, it's, it requires, again, a whole

set of other servers to use it.

Like you gotta have, uh, I think you have to have a, each version of Windows

system, you have to have a server running each version of Windows that

you're running that you wanna protect.

As part of the,

you looked at the infrastructure that was required to do this, and

you were like, yeah, we're not

part of it.

That's not free, is the infrastructure you have to have around it.

And as I recall, you have to have some of that infrastructure on each subnet.

You know what this reminds me of is, uh, you know, my friend Reid.

Yeah.

Shout out to re uh, Hey Reid, we know you're listening.

So Reid, Reid loves to tell the story of the free cat that his wife brought home,

and every time that that free cat, as he would make quotes in the air,

cost him money, he kept, you know, mentioning this free cat that he had.

Yeah.

So, so, yeah, so free isn't free.

So you have all that infrastructure.

So you got this up and running.

Are you doing something to protect your other data centers now that

you've gone through this experience?

Um,

in other locations, of course.

Yeah.

Well, in other locations we don't have that particular problem of, of, of

hurricanes trying to get, being surrounded

entirely by water

and, and not having data replicated off of that island.

You know, so other, other data centers that we have, we don't

have that particular problem.

Although, interestingly enough, we, we had one site that, um.

Was not an island but was in South America and we were having terrible times

backing that up to somewhere in Europe.

And um, you know, even when you did the, uh, you know, backup accelerator

or change block tracking or whatever, it's still, you had to do that initial.

Full backup in order for things to, to go from there.

And we finally just gave up with them and we sent an appliance down there.

And so the replication of that goes back to Europe and you

know, they're okay with that.

Part of it.

It's, it's, it's replication and we, we may be able to do something like

that with that island, but I, nobody's.

Nobody's really talked about it.

Yeah.

Did so, so on the island, did you end up rebuilding the data center?

Is it operational?

Did you, how did you move the workloads back, I guess is the other

thing?

So, so what I had mentioned earlier was the one that, that basically flooded or

failed, or whatever you want to call it.

Um, that was one of the first and oldest buildings on that.

Campus.

So they basically said, you know what?

This thing got damaged so badly.

We're not gonna trust it to house another data center.

We're, we've got another data center in this newer building over here, so

we're gonna put all that stuff that was there over to this newer building.

So that's what they did.

Was that just a matter of unplugging the servers and moving them, or

it, it took a while, but yeah, it was essentially that, but they had to

accelerate that timeline because of the fact that they needed those things to be

operational in a short period of time.

And then what about the, oh, go

ahead.

Of course during all this, of course there was no power, so they had to have,

you know, constant generator power Yep.

To their, to all of their operations down there.

So,

yeah.

I, I was just gonna ask, but the backups, while it was running in the

second data center, the only operating data center, I'm assuming that

those were to those local net backup appliances that were already there.

Right.

You didn't bring over the new, the old net backup appliance.

So now when you move the production workloads to that new data center,

what happened to the backups?

Did they migrate?

What did you end up doing?

Oh yeah, they migrated because we ended up moving the backup servers

into that new data center as well.

So, you know, basically the damaged data center was abandoned

and it was, everything was moved into this new, new data center.

But again, that timeline, it's actually now a swimming

pool.

Yeah.

Yeah, the original one.

Yeah.

Uh, you've been using NetBackup now for 20 years.

During which you've run into me once, or, uh, helping you to make it better, if

you could push a button and net backup, added a feature or fixed an issue.

Like what, what would that be?

Clientless?

Backup Clientless.

Oh, so so agentless backup.

Agentless backup, yeah.

That might be kind of cool.

Is that because

managing your agents on all these servers becomes a little time consuming.

It can become problematic because we don't have access to the clients.

Mm-hmm.

So, you know, if, if they wanna be updated, we have to, you know, ask them.

Pretty pleased to do that.

And there has to be good justification.

And of course there're being, you know, literally thousands of these things.

There's gonna be all kinds of pushback.

You know, this is gonna be a major project, so you know,

it be nice if everything was.

Virtual system that we didn't have to install a client on.

Mm-hmm.

But, and that's the nice thing.

Every time I get a chance to back something up as a

vm, I'm, I'm all over it.

Right.

Right.

But there are instances where it's not practical to do that because whatever

application is running on that doesn't like the momentary lapse of connection.

Yeah, exactly.

When it's doing the snapshot, it doesn't, it can't withstand that.

Huh?

So, so they tell us, Hey, these servers don't back 'em up as VMs.

We know they're VMs, but don't back them up that way,

huh?

Yeah.

So what so is, is this, this, this client issue, is it mainly at install

or you also experience it every time you need to upgrade the client?

Um.

Well, for the most part.

And then that, that's the nice thing about net backup is you can go two versions

back and still be able to back up.

So we have, we still have like clients that are running seven dot

something, even though our master servers are at eight dot something.

So they're not gonna be a problem until, and unless we go to

nine and even then they're not gonna be necessarily a problem.

But at that point.

Go ahead.

Yeah.

At that point we may have to have words with the the system owners and say,

Hey guys, you know, it would really be better if you could upgrade them.

What I would want GI to solve that problem, at least the update

problem, is it would seem that NetBackup should be able to update

its own client, like via the would same protocols that it already has.

Yeah, that would be nice.

Yeah.

As long as I know other backup products do that.

Well as, yeah.

And as long as the resulting update in that client doesn't require

a reboot of the said windows.

Yeah.

Or the old not

requiring a reboot is A is A BFD.

Yeah.

It is.

Yeah.

That's primarily where we get into issues with them.

You know, the, the, the system owners not wanting to do upgrades, especially if

they're doing, you know, even when they're doing quarterly patching, it's like, oh,

you could do it then, but because they're doing reboots all over the place when

they're doing quarterly patching, so.

But, all right.

I'm gonna, I'm gonna give you, Harry, I'm gonna give you a unique opportunity.

Are you ready for this?

Okay,

sure.

You've known me for a number of years.

Yeah.

So, yeah, tell your favorite Curtis as an idiot story.

You want me to tell the story of, of the, uh, election.

Knew you were gonna

hear

this.

I hear this.

Okay, so, so I was, I was visiting, I was visiting Curtis at his home

and, um, he, which is an Oceanside.

And, and, and I, uh, he asked me to stay over because the next day

he wanted to vote and he needed somebody to take him to the airport.

'cause he was going on a flight.

I don't know where.

Well, the way you say it, you were, you lived, the airport was on your way.

Yes.

Yes.

Alright.

Not like, okay.

I didn't ask you to

stay over so you could drive me to the airport.

No, no.

But anyway, so for some God knows why reason Curtis was not doing vote by mail

and he decided he had to go in person.

And so we leave his house before the polls open.

He gets to the place and he is like.

I forgot something.

I, I, I don't know what it was, your jacket or something.

So we had to go back to your house and get that.

And then we go back and you're, you gotta stand in line and you're standing in line.

And, uh, I don't know if we had to repeat that process of going back to your house

more than once, but it was at least once.

And, and then it was after you voted, we're like, hop in the car and go.

And.

You're driving.

Like we were

going to the LA airport, which is Yeah.

I was gonna the airport 90 minutes to two

hours away.

Yeah.

And, and it was gonna be tight.

Yeah.

And um, so you're the one who's driving Fortunately.

Yeah.

Uh, and, and, um, we get to the airport and you gimme the keys and say, just

go park it in the garage, get the keys.

Um, ran in.

For your flight and I think you missed your flight.

I think that might be possible.

Yeah, and, and the whole way from Oceanside to LA all I heard was you

should, why don't you vote by mail as you, as any normal person in

California because you know it's, anybody can register to vote by mail.

By the way, if you're not registered to vote, please register to vote.

Please, please.

You know, if you don't vote, you can't complain.

So, and if you do vote,

if you do get an absentee ballot,

get

an absentee ballot mail right away.

And, and, and if you don't do that,

your vote's important vote.

It's not worth your life.

If you have, if you have.

The possibility of voting by mail, even if, you know, 'cause most of the states

you can vote by mail for no reason.

And then there's another handful of states where you need some reason, and

Covid is being accepted as a reason.

Like if you haven't yet registered to vote by mail, now would be the time to do so.

Uh, and then the moment you receive your.

Your ballot, uh, take it to your registrar or some local box.

I don't want you dropping it in the mail.

Yeah, yeah.

Um,

Harry and I happen to listen to a podcast that has been talking about

this for quite some time, and her thing is, you know, don't, you know,

vote as quickly as you can and vote.

And don't put it in the mail, but take it, you know, if there's a Dropbox that

is a specific voting Dropbox, that's fine, but otherwise take it directly

to wherever you have to take it in your local registration area, right?

Anyway.

And that way you don't have to have your friend drive like a banshee to the

airport and like a, like a bat outta hell.

Yeah, I knew, I, I wondered which story you would pick, because

you've known me for long enough time, so I'm sure there were a few.

I hope our, our readers appreciated that story.

Or your listeners,

what did I say?

Readers?

You said readers.

So I'm such in a book mode.

So in, and the thing is, you know, I'm, I'm right.

Most of the book is written.

Voice to text.

Right.

I'm writing in Dragon, so even when I'm writing, I'm talking, which

people that know me are like, well, of course he's writing in a way

that allows him to talk anyway.

All right.

Well Harry, thank you very much for illuminating us from what

it's like in a real world to, you know, fire backups and anger.

Uh, thanks persona for joining us once again.

Thank you, Curtis.

Thank you, Harry.

Thank

you.

Yes, yes.

Go back onto the stairs.

Thanks for having me.

Yeah, I'll do that.

Thanks to the listeners.

I hope you enjoyed this Encore episode of the backup wrap-up.

That is a wrap.

You found the backup wrap up your go-to podcast for all things

backup recovery and cyber recovery.

In this episode, we talk with a backup admin that's been doing the same job at

the same company for almost 20 years.

He shares an incredible story about what happened when a hurricane hit

their data center on an island.

Spoiler alert, one data center was completely flooded and

Iron Mountain got hit too.

That's a double whammy if I've ever seen one.

He's a good friend of mine that went through a huge disaster.

We disguised his voice and his name so that he could tell

the story without any varnish.

This is part one of two parts that cover the same story.

From two different points of view this week.

Uh, we have Harry Potter.

By the way, if you don't know who I am, I'm w Curtis Preston, AKA, Mr.

Backup, and I've been passionate about backup and recovery for

over 30 years, ever since.

I had to tell my boss that we had no backups of the production

database that we had just lost.

I don't want that to happen to you, and that's why I do this.

On this podcast, we turn unappreciated backup admins into Cyber Recovery Heroes.

This is the backup wrap up.

Let's do it again.

Let's do it again.

Hmm.

You found the backup wrap up your go-to podcast for all things

backup recovery and cyber recovery.

In this episode, we talk with a backup admin that's been doing the same job at

the same company for almost 20 years.

He shares an incredible story about what happened when a hurricane hit

their data center on an island.

Spoiler alert, one data center was completely flooded and

Iron Mountain got hit too.

That's a double whammy if I've ever seen one.

He's a good friend of mine that went through a huge disaster.

We disguised his voice and his name so he could tell the story without varnish.

We call 'em Harry Potter.

This is part one of two parts that cover the same story from

two different points of view.

You'll see the second part next week.

By the way, if you don't know who I am, I'm w Curtis Preston, AKA, Mr.

Backup, and I've been passionate about backup and recovery for