Backing up databases, Part 1 (Backup to basics)
It was a dark and stormy night in 1993 when paris (the database server) went down. It would be a night the new backup admin would never forget because he couldn't restore the database from backup. The only bright side of that very sad story is that it launched a career. Yes, that's the night W. Curtis Preston started his path toward Mr. Backup. Hear him tell the story in his own words, in the middle of the backup to basics series about backing up databases. Avoid the mistake that could have (but did not) cost him his job and enjoy a good episode while you're at it!
Mentioned in this episode:
Interview ad
Hi, and welcome to Backup Central's Restore All podcast.
W. Curtis Preston:I'm a host w Curtis Preston, a k a, Mr.
W. Curtis Preston:Backup, and I have with me my Monday morning melancholy
W. Curtis Preston:minimizer Prasanna Malaiyandi?
Prasanna Malaiyandi:Wow.
Prasanna Malaiyandi:How's it going?
Prasanna Malaiyandi:So we won't get into your melancholy, but here, here's some good news for you.
Prasanna Malaiyandi:You know what this, you know what today is.
Prasanna Malaiyandi:There's two important things that happened today.
W. Curtis Preston:
Speaker:It's, no, I got nothing.
Prasanna Malaiyandi:So the first one is I now
W. Curtis Preston:Oh, do you?
Prasanna Malaiyandi:year.
W. Curtis Preston:You know the day you're three actually.
Prasanna Malaiyandi:It was actually a couple of weeks ago.
Prasanna Malaiyandi:It was, it's actually two weeks ago.
Prasanna Malaiyandi:Right.
Prasanna Malaiyandi:But I do now have a three-year-old beard and three-year-old hair.
Prasanna Malaiyandi:I don't know what you call three-year-old
W. Curtis Preston:what I call three.
Prasanna Malaiyandi:Yeah.
W. Curtis Preston:In need of a haircut.
W. Curtis Preston:That's what I, that's what I call that.
Prasanna Malaiyandi:No, it's fine.
Prasanna Malaiyandi:And.
Prasanna Malaiyandi:The second one is, this is our 200th
W. Curtis Preston:Are we really recording our 200th episode?
W. Curtis Preston:Wow.
W. Curtis Preston:That's almost four years of podcasts.
W. Curtis Preston:That's a lot of talking.
Prasanna Malaiyandi:And thank you to the listeners for listening
Prasanna Malaiyandi:to us for these last like four
W. Curtis Preston:I'm, um, I'm on another podcast that
W. Curtis Preston:doesn't have as many listeners.
W. Curtis Preston:Um, and I mean, it has very, very few listeners and it's not, you
W. Curtis Preston:know, it's just not the same.
W. Curtis Preston:Like if you, if you're producing a podcast and no one's listening to it, it's gonna.
W. Curtis Preston:You're like, why am I doing this?
W. Curtis Preston:But yeah, we've got, we've got, um, you know, um, thousands of you out there that
W. Curtis Preston:are listening to it and we appreciate it.
W. Curtis Preston:And without that, uh, this would seem like a lot of effort for nothing.
W. Curtis Preston:So, um, definitely, um,
Prasanna Malaiyandi:Yeah.
W. Curtis Preston:Well, that's good.
W. Curtis Preston:You know what?
W. Curtis Preston:See,
Prasanna Malaiyandi:
Speaker:make your Monday a little
W. Curtis Preston:brought, that, brought that, that helped me out.
W. Curtis Preston:I'm having, I'm, I'm not having the greatest Monday here and see this.
W. Curtis Preston:What, this, why, you know, you're my Monday morning melancholy
W. Curtis Preston:minimizer, but, But the, but the beard, the beard and the hair thing.
W. Curtis Preston:I know, man.
W. Curtis Preston:I mean, it, it's funny for those of you watching on the, on the, you know, you
W. Curtis Preston:can see the video of this if you go to backup central.com or you can listen to
W. Curtis Preston:it on, you know, on any of the places.
W. Curtis Preston:And, uh, but those of you looking, you, you see me, I just got a new
W. Curtis Preston:haircut talking about this nice trim saying, I've got this nice
W. Curtis Preston:trim, trim kept to my face, beard.
W. Curtis Preston:And then we have Prasanna.
W. Curtis Preston:Uh, that is the complete opposite.
Prasanna Malaiyandi:Yeah, I, I, I once had a friend, hi Jane, who
Prasanna Malaiyandi:referred to me as the caveman.
Prasanna Malaiyandi:And this is what I had a normal beer too.
Prasanna Malaiyandi:Not like this.
W. Curtis Preston:opposites in so many different ways.
W. Curtis Preston:Um, you know, like when it comes to movies and, you know, when it comes to
W. Curtis Preston:the things we enjoy eating, what, there's some stuff we like eating that's similar.
W. Curtis Preston:Um, but, and I, and
Prasanna Malaiyandi:And this is why we get along so well,
W. Curtis Preston:whole opposite attract thing.
W. Curtis Preston:Um, well, uh, we're gonna continue our, uh, backup to basic series,
W. Curtis Preston:uh, today, talking about databases.
W. Curtis Preston:Before I do that, I'll throw out our usual disclaimer.
W. Curtis Preston:I and Prasanna work at different companies.
W. Curtis Preston:He works for Zoom, I work for Druva.
W. Curtis Preston:This is not a podcast of either company.
W. Curtis Preston:Uh, the opinions that you hear ours.
W. Curtis Preston:If you'd like to join the conversation, just, uh, reach out to me, w Curtis
W. Curtis Preston:Preston gmail, or at WC preston on Twitter and also linkedin.com/iin/mr.
W. Curtis Preston:Backup.
W. Curtis Preston:You'll find me there as well.
W. Curtis Preston:And, uh, you know, um, join the conversation, right?
W. Curtis Preston:We'd love to have you on.
W. Curtis Preston:And also please rate us, uh, go to your favorite podcast or scroll down
W. Curtis Preston:to the, to the comments and um, you know, give us some stars and some
W. Curtis Preston:comments we love that keeps us going.
W. Curtis Preston:Uh, except for that one guy or the, gave us one star.
W. Curtis Preston:I don't know what his deal was, but, um, it's one person.
W. Curtis Preston:It's one person that gave us one star.
W. Curtis Preston:And he doesn't say why.
W. Curtis Preston:I'm like, Hmm, that was harsh.
W. Curtis Preston:Um, Who that is.
W. Curtis Preston:Yeah,
Prasanna Malaiyandi:tell us so we can
W. Curtis Preston:I think, you know, if you want to say, listen, I would
W. Curtis Preston:really like to listen to that podcast, but Prasanna needs to cut his hair.
W. Curtis Preston:All
Prasanna Malaiyandi:Not gonna happen.
Prasanna Malaiyandi:I'm sorry
W. Curtis Preston:know, it is what it is.
W. Curtis Preston:Well, we are continuing our, um, our backup to basic series, which
W. Curtis Preston:is based on this, uh, latest book that I wrote, which of course I
W. Curtis Preston:had lots of help from Prasanna on.
W. Curtis Preston:Um, and it's, uh, modern Data Protection from O'Reilly and Associates, and, you
W. Curtis Preston:know, you can find it at, uh, wherever.
W. Curtis Preston:Wherever books are sold, um, you can get both, uh, an ebook
W. Curtis Preston:version and a printed version.
W. Curtis Preston:And, um, So let's talk about, uh, th this is about protecting databases.
W. Curtis Preston:So I, I know I've told this story on the podcast, but it was a broken database
W. Curtis Preston:that basically started my career, right?
W. Curtis Preston:Um, that, yeah.
W. Curtis Preston:Um, it was the, the name of the database was Paris.
W. Curtis Preston:It was my, uh, my bank's purchasing database.
W. Curtis Preston:And, uh, we were backing it.
W. Curtis Preston:Via what I would now refer to as the hot backup method where you put the
W. Curtis Preston:da, you put the data files into backup mode, and then you back them up.
W. Curtis Preston:Except that we had moved, um, we had moved the server over to another, we
W. Curtis Preston:had moved the database over to another server, and nobody had told me that,
W. Curtis Preston:that I needed to put the script in place so that I could do the backups.
W. Curtis Preston:And we had been backing it up for months without, you know,
W. Curtis Preston:without getting a decent backup.
W. Curtis Preston:it was inconsistent.
W. Curtis Preston:And, um, I remember sitting there and the, the boss was like, so
W. Curtis Preston:let me, let me get this straight.
W. Curtis Preston:We have absolutely no backups of Paris whatsoever.
W. Curtis Preston:And I'm like, that is what I'm saying.
W. Curtis Preston:Uh, and she didn't fire me tr you know, chalked it up to bad training.
W. Curtis Preston:Uh, Ron Rodriguez, all your fault to this day, I'm gonna
W. Curtis Preston:throw you under the bus for that.
Prasanna Malaiyandi:You're like, you're like a name That sticks in my head.
W. Curtis Preston:good guy.
W. Curtis Preston:But yeah, just he left that out of the training.
W. Curtis Preston:So, uh, so I think it's important for you to understand databases
W. Curtis Preston:because they are, you know, I think for, for many environments they're
W. Curtis Preston:more than 50% of the data center.
W. Curtis Preston:Isn't that, isn't that normal?
W. Curtis Preston:Like for structured data essentially to be,
Prasanna Malaiyandi:I think that's changing these days.
Prasanna Malaiyandi:Oh, sorry.
Prasanna Malaiyandi:For structured data for most of it to be stored in databases.
Prasanna Malaiyandi:I think that would be a
W. Curtis Preston:So you're not necessarily agreeing that
W. Curtis Preston:most of the data in the, in the data center is structured data.
W. Curtis Preston:You're thinking that nowadays unstructured data is more because we're just, yeah.
W. Curtis Preston:Okay.
W. Curtis Preston:Yeah, yeah.
W. Curtis Preston:We
Prasanna Malaiyandi:data.
Prasanna Malaiyandi:Yeah.
Prasanna Malaiyandi:And, and if, and when you, but if you do caveat it with saying in the
Prasanna Malaiyandi:data center, I think that statement
W. Curtis Preston:You know what's funny?
W. Curtis Preston:The phrases like that come out of this old mouth, and I don't
W. Curtis Preston:even mean it when I say it.
Prasanna Malaiyandi:Yeah.
W. Curtis Preston:I don't, I mean, in the computing environment.
W. Curtis Preston:Um, so you, you're thinking that there's a lot of stuff up there.
Prasanna Malaiyandi:Just because of the amount, large amount.
Prasanna Malaiyandi:Oh yeah.
Prasanna Malaiyandi:Just the large amount of data that's out there in the world.
Prasanna Malaiyandi:Right.
Prasanna Malaiyandi:And unstructured
W. Curtis Preston:There was a time when structured data, data
W. Curtis Preston:and databases specifically was the king of the data center.
W. Curtis Preston:Right.
W. Curtis Preston:Um, and then, um, then we just started storing on just all kinds of nonsense.
W. Curtis Preston:Um, we just, we as a, as a human race, we have seemed to have a
W. Curtis Preston:never ending desire to store stuff.
Prasanna Malaiyandi:Pack rats.
Prasanna Malaiyandi:Pack rats.
Prasanna Malaiyandi:That's what we
W. Curtis Preston:Um, so I, so the, you know, if, if you don't know anything
W. Curtis Preston:about databases, the, you know, you should learn a lot in this episode.
W. Curtis Preston:If you know a a lot about databases, you may know more than I do.
W. Curtis Preston:I am not a dba, uh, I have never been a dba.
W. Curtis Preston:Uh, I have often been at war with DBAs, um, but.
Prasanna Malaiyandi:That is a database admin who, yeah, from a
Prasanna Malaiyandi:backup admin perspective, probably
W. Curtis Preston:uh, I, I've run into a few fights, you know, I've had
W. Curtis Preston:a few fights, but, um, so I think it's important to sort of divvy up the,
W. Curtis Preston:you know, the computing world into these different buckets so that we
W. Curtis Preston:understand what we're talking about.
W. Curtis Preston:And the first thing that I talked about was database delivery model.
W. Curtis Preston:And the first is, uh, is traditional database software.
W. Curtis Preston:What would you think I would mean by that Prasanna?
W. Curtis Preston:And maybe give an example.
Prasanna Malaiyandi:Yeah, so the biggest is you basically take the software, you
Prasanna Malaiyandi:download it from the vendor, you deploy it on your servers, you're managing
Prasanna Malaiyandi:it, you're doing everything else.
Prasanna Malaiyandi:This traditionally has always been like Microsoft sql.
Prasanna Malaiyandi:You have your Oracle databases.
Prasanna Malaiyandi:I think those are probably the two biggest,
W. Curtis Preston:Still, still Cybase.
W. Curtis Preston:Informix is out there and, well, and of course, yeah.
W. Curtis Preston:Yeah.
W. Curtis Preston:I don't know.
W. Curtis Preston:MySQL might actually be the biggest database out there.
W. Curtis Preston:Um, in terms of just certainly number of deployments, but many, many, you know, but
W. Curtis Preston:I wonder if we added up all the gigabytes of all the little tiny MySQL databases.
W. Curtis Preston:Um, I mean, the Backup Central has a bunch of MySQL databases behind it.
W. Curtis Preston:Uh, but yeah, that's basically, you know, that's normally what I think
W. Curtis Preston:many people think of when they think.
W. Curtis Preston:Databases, right?
W. Curtis Preston:But then if we take that database and someone else runs the database
W. Curtis Preston:application itself, right?
W. Curtis Preston:And then all I have to deal with, so basically they're gonna manage the
W. Curtis Preston:storage, the patching of, you know, Oracle, MySQL, uh, the security, perhaps
W. Curtis Preston:the security administration of it.
W. Curtis Preston:And then all I have to do is add database, add table, um, then that.
Prasanna Malaiyandi:Mm-hmm.
W. Curtis Preston:Exactly, and then start using the database.
W. Curtis Preston:That would be what we call a PaaS database or platform as a service.
W. Curtis Preston:You get the database platform, and the best example I have
W. Curtis Preston:of that is aws, r d s, right?
W. Curtis Preston:Relational database service.
W. Curtis Preston:From a backup perspective, the main difference between these two is that
W. Curtis Preston:you don't, you, you can't necessarily install, like if you want to back up,
W. Curtis Preston:um, Let's say, uh, SQL Server, you can install a SQL Server backup, uh,
W. Curtis Preston:uh, a agent right it on it, right.
W. Curtis Preston:You, you don't
W. Curtis Preston:observe it,
Prasanna Malaiyandi:And just quickly on that, I don't know if we'll talk about
Prasanna Malaiyandi:this later, but, and, but I think that's where a lot of these vendors who are
Prasanna Malaiyandi:providing PAs databases sort of try to bake in backup and some of these recovery
Prasanna Malaiyandi:operations into the platform itself
W. Curtis Preston:yeah, they, they do, they do tend to build in backup features.
W. Curtis Preston:You often need to actually execute those features.
W. Curtis Preston:You need to actually, uh, Drive the backup, but they give you the tools, they
W. Curtis Preston:give you the car, you gotta drive it.
W. Curtis Preston:Um, there is one interesting one, and I'll pick up on Amazon.
W. Curtis Preston:I'll pick on Amazon here for a minute.
W. Curtis Preston:Amazon, r d s does support, um, uh, RMAN for Oracle.
W. Curtis Preston:Let me, let me rephrase.
W. Curtis Preston:They support Armand backups for Oracle.
W. Curtis Preston:You know what?
W. Curtis Preston:They don't support.
Prasanna Malaiyandi:S B T
W. Curtis Preston:No, they don't support rman restores for Oracle.
Prasanna Malaiyandi:oh oh.
Prasanna Malaiyandi:That's.
W. Curtis Preston:You
W. Curtis Preston:can make, you can make Rman backups with Oracle, with Amazon rds.
W. Curtis Preston:But the last time I checked you cannot do am you cannot do, uh,
W. Curtis Preston:rman restores, which is just odd.
W. Curtis Preston:Uh, it is
Prasanna Malaiyandi:but I think that's where they hope that you're
Prasanna Malaiyandi:using like their built-in snapshot and capabilities and everything else like
Prasanna Malaiyandi:that to if you need to actually restore.
Prasanna Malaiyandi:Otherwise it's like, hey, you have an Armand backup and you
Prasanna Malaiyandi:could take it offsite or to your
Prasanna Malaiyandi:own instance, if you will, to do your restores.
W. Curtis Preston:I guess it's more, yeah.
W. Curtis Preston:Yeah.
W. Curtis Preston:Um, just a weird one.
W. Curtis Preston:So then the next one we have here is serverless databases.
W. Curtis Preston:You want to tackle, tackle that one?
Prasanna Malaiyandi:Yeah, so we just talked about, okay, you have.
Prasanna Malaiyandi:Sort of someone else managing the server, but you're still
Prasanna Malaiyandi:managing all the databases.
Prasanna Malaiyandi:You also typically are involved in sort of performance tuning
Prasanna Malaiyandi:at that point as well, right?
Prasanna Malaiyandi:With serverless, you're sort of getting away from all of that.
Prasanna Malaiyandi:It's like, Hey, here's a database endpoint.
Prasanna Malaiyandi:You don't even do sort of the normal, basic operations other than sort of
Prasanna Malaiyandi:accessing the data, and it automatically sort of scales up, scales down on its own.
Prasanna Malaiyandi:You don't have to worry about it.
Prasanna Malaiyandi:Now, it's funny that everyone talks about serverless, even like serverless.
Prasanna Malaiyandi:In the end,
Prasanna Malaiyandi:there's still a server somewhere, right?
W. Curtis Preston:Right.
W. Curtis Preston:But you are not
W. Curtis Preston:managing it right.
W. Curtis Preston:Yeah.
W. Curtis Preston:Um, and the, the biggest example here I would have would be DynamoDB.
W. Curtis Preston:I mean, there are a bunch of serverless databases, both on Amazon and
W. Curtis Preston:other, other, uh, cloud providers.
W. Curtis Preston:But the idea with, with Amazon db, I think, I think it, it,
W. Curtis Preston:it's a perfect example of how, how you can make it very simple.
W. Curtis Preston:With Dynamo DB , you just give it a key in value.
W. Curtis Preston:Right.
W. Curtis Preston:You have an account, you have a authentication, you give it a key and
W. Curtis Preston:value pair and it will store it somewhere.
W. Curtis Preston:You're not creating that table, you're not creating a database,
W. Curtis Preston:you're not creating any of that stuff.
W. Curtis Preston:You're just say, Hey, here's a key and a value.
W. Curtis Preston:Uh, store it for me and I'll ask for it later.
W. Curtis Preston:So that's what I'm calling the database delivery models.
W. Curtis Preston:And then we've got the database models, which are, which are like
W. Curtis Preston:the different, that's how you would, that's how you might get a database.
W. Curtis Preston:This is what kind of database you might get.
W. Curtis Preston:Um, and the biggest category, what would the biggest category be?
W. Curtis Preston:What do you think?
W. Curtis Preston:And by the
Prasanna Malaiyandi:Well, it depends.
Prasanna Malaiyandi:It
W. Curtis Preston:not even.
Prasanna Malaiyandi:Well, I think it depends on what
Prasanna Malaiyandi:you mean by biggest, right?
Prasanna Malaiyandi:Is it the number of deployments?
Prasanna Malaiyandi:Is it the sizes?
Prasanna Malaiyandi:Right.
W. Curtis Preston:I don't know.
W. Curtis Preston:I'm just talking about the, the db engines.com, the most
W. Curtis Preston:popular
Prasanna Malaiyandi:relational databases.
W. Curtis Preston:databases.
W. Curtis Preston:Yeah.
W. Curtis Preston:It looks to be over 75% of, of the other ones.
W. Curtis Preston:There are other types.
W. Curtis Preston:One of them is the one I just mentioned, the key value pair database.
W. Curtis Preston:Right.
W. Curtis Preston:There's time series, there's graph databases, document databases, a
W. Curtis Preston:database just for search engines, right?
W. Curtis Preston:Th these are the different types of databases and they tend
W. Curtis Preston:to behave differently from a backup and recovery perspective.
W. Curtis Preston:And I do think it's important for you to understand, um, you know, the
W. Curtis Preston:type of database that you're backing up when you're backing it up, right?
W. Curtis Preston:That's
Prasanna Malaiyandi:Because they each have their own nuances and differences.
Prasanna Malaiyandi:It's not like a database is a database because from backup
Prasanna Malaiyandi:and restores, they're completely
Prasanna Malaiyandi:different.
W. Curtis Preston:Exactly.
W. Curtis Preston:And then this is where I think we really get into, you really need to understand
W. Curtis Preston:this when you're doing a backup and recovery, and that is the different
W. Curtis Preston:consistency models of a database.
Prasanna Malaiyandi:What do you mean by consistency models?
W. Curtis Preston:So immediate.
W. Curtis Preston:Right versus eventual versus hybrid.
W. Curtis Preston:These are the, the, the different consistency models, right?
W. Curtis Preston:Immediate is the one that most of us think of when we, um, When we take a
W. Curtis Preston:databases, at least, I'm, I'm gonna say it's probably the most popular,
W. Curtis Preston:again, relational database management systems, RDBMSs, those are, I'm
W. Curtis Preston:pretty sure those are all immediate.
W. Curtis Preston:Wouldn't they be all immediate?
Prasanna Malaiyandi:I think so.
Prasanna Malaiyandi:yeah,
W. Curtis Preston:yeah.
W. Curtis Preston:But then eventual is the one that I struggle with.
W. Curtis Preston:It's like, why would you want that?
W. Curtis Preston:But,
Prasanna Malaiyandi:yeah.
Prasanna Malaiyandi:Well, I
W. Curtis Preston:sense once you realize how they're
W. Curtis Preston:typically deployed.
Prasanna Malaiyandi:Yeah.
Prasanna Malaiyandi:Well, and here's a good example, right?
Prasanna Malaiyandi:So you have a bank, you go make a deposit with immediate, it's like, Hey,
Prasanna Malaiyandi:the transaction shows up immediately.
Prasanna Malaiyandi:You can use it, right?
Prasanna Malaiyandi:With like eventual, it's like, Hey, you made a deposit.
Prasanna Malaiyandi:Sometime in the next 24 hours or so, your bank balance will, or
Prasanna Malaiyandi:your bank account will balance out.
Prasanna Malaiyandi:But until then, you may be able to withdraw funds.
Prasanna Malaiyandi:Sometimes you may not be able to if it hasn't cleared, right?
Prasanna Malaiyandi:So,
W. Curtis Preston:Yeah.
Prasanna Malaiyandi:That's kind of how I like to think about it.
W. Curtis Preston:The example that I used in, um, the book was I used
W. Curtis Preston:DNS as an example of the concept of eventual consistency, where, um,
W. Curtis Preston:you're gonna get a different answer depending on who you ask and when,
Prasanna Malaiyandi:Yep.
W. Curtis Preston:right?
W. Curtis Preston:You, you create a, a DNS entry and it takes a while for that entry
Prasanna Malaiyandi:to.
Prasanna Malaiyandi:Propagate
W. Curtis Preston:Propagate throughout the system.
W. Curtis Preston:Uh, and the same is true of, of, of an eventually consistent database.
W. Curtis Preston:And the thing is, you can't use the eventually consistent model.
W. Curtis Preston:If getting a wrong answer will will break the, I was gonna say, yeah, we'll break
W. Curtis Preston:everything.
W. Curtis Preston:um, but.
W. Curtis Preston:You know, it, it is weird though, right?
W. Curtis Preston:It, it's weird.
W. Curtis Preston:It's, I struggle with this idea of eventual consistency, um, and,
W. Curtis Preston:um, how it would be okay for you to get essentially a wrong answer.
Prasanna Malaiyandi:but I think it depends on the
Prasanna Malaiyandi:application, right?
Prasanna Malaiyandi:Where
W. Curtis Preston:it absolutely
Prasanna Malaiyandi:if it's, yeah, like if it's okay for you,
Prasanna Malaiyandi:cuz there is a trade-off, right?
Prasanna Malaiyandi:With, uh, immediate consistency.
Prasanna Malaiyandi:Typically there is some sort of latency associated with committing
Prasanna Malaiyandi:a transaction, especially if you're talking about multiple nodes, right?
Prasanna Malaiyandi:Which is typically where
Prasanna Malaiyandi:you see the eventual.
W. Curtis Preston:Yeah.
W. Curtis Preston:You look at dns, for example.
W. Curtis Preston:DNS would break if
Prasanna Malaiyandi:Could you
W. Curtis Preston:to be, if it had to be
W. Curtis Preston:immediately consistent across the entire world, right?
W. Curtis Preston:That's the thing.
W. Curtis Preston:Immediate consistency is very tight, right?
W. Curtis Preston:You've got to have it right.
W. Curtis Preston:And then I also listen hybrid, which is, um, somewhere between the two.
W. Curtis Preston:So here's an example I have.
W. Curtis Preston:So for example, DynamoDB users can tell DynamoDB they want a
W. Curtis Preston:strongly consistent read, and it will always read from the leader.
W. Curtis Preston:Um, where the right was initial, which is the leader is where the
W. Curtis Preston:right was initially made, even if it is still being replicated elsewhere.
W. Curtis Preston:So that's hybrid.
W. Curtis Preston:It's an, it's eventually consistent database, but you could say when
W. Curtis Preston:you create a key value pair in Dynamo DB upon read, always read
W. Curtis Preston:from the first person, right?
W. Curtis Preston:Um, yeah.
W. Curtis Preston:There, there's a bunch of a MongoDB Couchbase, uh, that support that.
W. Curtis Preston:Um, I had listed here of immediate are all the ones that we, the, the
W. Curtis Preston:ones that everybody knows their name, Oracle Sequel, you know,
W. Curtis Preston:uh, I have, uh, eventual consistency.
W. Curtis Preston:Uh, I have Cassandra and Neo four J.
W. Curtis Preston:And then I have, as hybrid, I have Mongo and DynamoDB.
W. Curtis Preston:Those are just an example.
W. Curtis Preston:Um, by the way, all that stuff comes from a great website,
W. Curtis Preston:um, called uh, db-engines.com.
W. Curtis Preston:Um, they list all that stuff in there.
W. Curtis Preston:So, um, The next one, I have a bunch of terminologies here, and I'm not
W. Curtis Preston:gonna go through a lot of them, but, um, I think one that people
W. Curtis Preston:struggle with is the difference between an instance and a database.
W. Curtis Preston:Um,
Prasanna Malaiyandi:What, what is the difference
Prasanna Malaiyandi:between an instance and a database?
W. Curtis Preston:well, an instance is essentially the way I, the way I
W. Curtis Preston:would describe it is, In many cases it's the same, but an instance is a set
W. Curtis Preston:of processes to talk to the database.
W. Curtis Preston:And the database is, well, what we normally mean when we say database, right?
W. Curtis Preston:Um, so, uh, so let me, lemme just read here, because I, I, I got this like,
W. Curtis Preston:check, you know, 50 times, you know, uh, there can be multiple databases within
W. Curtis Preston:an instance, and a database can also be distributed across multiple instances.
W. Curtis Preston:Right.
W. Curtis Preston:Um, on the same machine or on separate machines within a cluster.
W. Curtis Preston:Therefore, an instance in a database are two entirely different concepts.
W. Curtis Preston:Historically, an instance ran within a server, but modern database
W. Curtis Preston:platforms have instances that span multiple servers and node, right?
W. Curtis Preston:Um, so the instance is basically the thing that runs inside a server or a bm.
W. Curtis Preston:Talks to this thing, but a database could be across multiple
W. Curtis Preston:instances or multiple databases.
W. Curtis Preston:That instance, um, it is
W. Curtis Preston:complicated.
W. Curtis Preston:See, this is why, this is why
Prasanna Malaiyandi:but, but, here's here, here's a
W. Curtis Preston:go ahead.
Prasanna Malaiyandi:though.
Prasanna Malaiyandi:As a backup person,
Prasanna Malaiyandi:what do I need to worry about in that case?
Prasanna Malaiyandi:Do I worry about more of the instance or the
Prasanna Malaiyandi:database?
W. Curtis Preston:Well, you just need to know.
W. Curtis Preston:You need to know that there, yeah.
W. Curtis Preston:It depends.
W. Curtis Preston:You need to know that there are different things because.
W. Curtis Preston:It, it's going to dictate how you communicate with the thing that
W. Curtis Preston:you're trying to back up, right?
W. Curtis Preston:Um, in many cases, I'm, I'm gonna throw you prob again.
W. Curtis Preston:I'm gonna say in most cases they're the same, right?
W. Curtis Preston:They're not the same, but they're in the same place.
W. Curtis Preston:There's one instance and there's one database.
W. Curtis Preston:Most of the databases that I worked with, it was one
W. Curtis Preston:instance of one database, right?
W. Curtis Preston:Um, The, um, but you just need to know that that's not always the case, right?
W. Curtis Preston:We all know what a table is, right?
W. Curtis Preston:Um, you know, I, I liken it, I liken it to a spreadsheet, uh, but it's not the same.
W. Curtis Preston:The, um, and then there's this concept of a data file, which is where we stored the.
W. Curtis Preston:Right where the database stores the data.
W. Curtis Preston:Um, and this concept of a table space, which is what it sounds like, it's
W. Curtis Preston:a space where you put tables, so you could put many tables in this database.
W. Curtis Preston:Um,
Prasanna Malaiyandi:And each database does it
Prasanna Malaiyandi:slightly differently.
W. Curtis Preston:yeah, I'm, I'm trying to be very, very general here,
Prasanna Malaiyandi:Yeah.
Prasanna Malaiyandi:And I think the other question, and I know we talked about this at the
Prasanna Malaiyandi:very beginning, talking about the problem you ran into with Paris, right?
Prasanna Malaiyandi:It's just because you have a database doesn't mean you could just take all the
Prasanna Malaiyandi:database
Prasanna Malaiyandi:files and copy them out
W. Curtis Preston:No, you can't because the data, those data files
W. Curtis Preston:are what you're trying to back up quite often.
W. Curtis Preston:And if you're backing it up outside the world of the database, the database
W. Curtis Preston:is changing those data files while you're, while you're backing it up.
W. Curtis Preston:And it's, it's not gonna be consistent, you know, part, part of the, part
W. Curtis Preston:of this file that you backed up.
W. Curtis Preston:Part of it's gonna be for one point in time, part of the file is gonna
W. Curtis Preston:be from another point in time.
W. Curtis Preston:No, no.
W. Curtis Preston:Good.
W. Curtis Preston:And then, Again, going for a very generic term.
W. Curtis Preston:Here I have this concept of a master file, and that is sort
W. Curtis Preston:of the database of the database.
W. Curtis Preston:Um, a perfect example is the Oracle control file, right?
W. Curtis Preston:The, um, uh, again, this is most databases.
W. Curtis Preston:Not every database has this, um, it, it might be a J S O N file, right?
W. Curtis Preston:Um, go, what were you gonna.
Prasanna Malaiyandi:does it ever feel like inception when you're working on
Prasanna Malaiyandi:databases where it's like a database
Prasanna Malaiyandi:within a database, you know?
W. Curtis Preston:Yeah.
W. Curtis Preston:Uh, by the way, SQL Servers master Database has this concept.
W. Curtis Preston:Essentially it's the thing that's keeping track of all the
W. Curtis Preston:things that we're talking about.
W. Curtis Preston:What are all the data files?
W. Curtis Preston:What point are they at?
W. Curtis Preston:What if we have a eventually consistent, you know, what,
W. Curtis Preston:what change level are we at?
W. Curtis Preston:Um, it's the thing that keeps track of all the things.
W. Curtis Preston:And, um, there
Prasanna Malaiyandi:impressive that they actually used that piece of technology as
Prasanna Malaiyandi:a core foundation of like keeping track.
Prasanna Malaiyandi:It's like
Prasanna Malaiyandi:that's kind of
Prasanna Malaiyandi:neat.
W. Curtis Preston:Yeah.
W. Curtis Preston:Uh, and so there's going to be a different backup method for that.
W. Curtis Preston:If that file exists, there's gonna be a different backup
W. Curtis Preston:method for that file, right?
W. Curtis Preston:So you're gonna back up that database.
W. Curtis Preston:You're gonna, um,
W. Curtis Preston:In Oracle, for example, there, there's a command backup control file, right?
W. Curtis Preston:It's just a separate, uh, backup.
Prasanna Malaiyandi:They're
W. Curtis Preston:now
W. Curtis Preston:exactly special.
W. Curtis Preston:Uh, and then we have a very important concept, and that is
W. Curtis Preston:the concept of a transaction.
W. Curtis Preston:You wanna talk about what a transaction is.
Prasanna Malaiyandi:Yeah, so usually when you say make a right to a database, right,
Prasanna Malaiyandi:you just issue the SQL command or whatever to do a simple thing, but that translates
Prasanna Malaiyandi:into a whole bunch of other things.
Prasanna Malaiyandi:Now, when you actually get that before it can apply to the actual
Prasanna Malaiyandi:database, and I don't know if you wanna talk about logs right now, but
Prasanna Malaiyandi:that transaction, typically when you think about a transaction, especially
Prasanna Malaiyandi:on a relational database side, it's something that has to either complete or.
Prasanna Malaiyandi:It's either the transaction completes and the entire database rolls forward
Prasanna Malaiyandi:and everything's good, or the transaction doesn't succeed and no part of the
Prasanna Malaiyandi:transaction is successful, so basically the entire thing happens or it doesn't,
Prasanna Malaiyandi:which means it's atomic, right?
Prasanna Malaiyandi:So it all happens and everything's successful, or no part of it happens.
Prasanna Malaiyandi:So you don't want the case where it's like, Hey, I am withdrawing
Prasanna Malaiyandi:money from your bank, but I don't set the balance properly, or
Prasanna Malaiyandi:other things like that, right?
Prasanna Malaiyandi:So it either all needs to succeed or none of it
Prasanna Malaiyandi:succeeds.
Prasanna Malaiyandi:Otherwise, the world goes crazy.
W. Curtis Preston:Right.
W. Curtis Preston:And that matters most when you, because there's two
W. Curtis Preston:different types of transaction.
W. Curtis Preston:They're simple and it's complex.
W. Curtis Preston:Complex has a bunch of different parts of it.
W. Curtis Preston:And, uh, that's a, a perfect example where you were saying that you, all of those
W. Curtis Preston:parts have to work or, or none of them are allowed to work and you will then.
W. Curtis Preston:So if, if we can't finish the entire transaction, let's say.
W. Curtis Preston:Um, the database crashes.
W. Curtis Preston:It's in, it's in the middle of that.
W. Curtis Preston:What, what do we call when we make that transaction?
W. Curtis Preston:Yes, we roll
W. Curtis Preston:back the
Prasanna Malaiyandi:You roll
Prasanna Malaiyandi:back.
Prasanna Malaiyandi:Yeah.
Prasanna Malaiyandi:Which basically, so now you have to think about it though, right now that
Prasanna Malaiyandi:you have this concept of, okay, there's a transaction, it's atomic, it either
Prasanna Malaiyandi:all succeeds or it gets rolled back.
Prasanna Malaiyandi:Now if you think from the database perspective, you need to be able to
Prasanna Malaiyandi:track all of these, because how do I know how to go back to a previous state?
Prasanna Malaiyandi:It's like I need to.
Prasanna Malaiyandi:The previous state.
Prasanna Malaiyandi:Right.
Prasanna Malaiyandi:So there's so many other things a database does as a matter of just accepting a
Prasanna Malaiyandi:single transaction.
W. Curtis Preston:Yeah, exactly.
W. Curtis Preston:And, and that's why we have the transaction log, right?
W. Curtis Preston:So first, not every database has the concept of a transaction.
W. Curtis Preston:I still think.
W. Curtis Preston:Did every database has transactions, they just might not call it that.
W. Curtis Preston:Um, cuz every database is making changes, is storing data and making changes.
W. Curtis Preston:I think that generically every one of those should be called a transaction.
W. Curtis Preston:And, and by the way, this was one of my biggest challenges when I, when I
W. Curtis Preston:wrote my first, um, version of this, like, which is a really long time ago,
W. Curtis Preston:was trying to get, um, DBAs, again, of different database products to agree
W. Curtis Preston:on a generic term that would work and,
Prasanna Malaiyandi:for everything.
Prasanna Malaiyandi:yeah,
W. Curtis Preston:exactly.
W. Curtis Preston:And transaction.
W. Curtis Preston:Um, I is one of those things, right?
W. Curtis Preston:Um, but the transaction log re uh, Oracle for example, calls it a
W. Curtis Preston:redo log, some call it a, you know, um, so I ha I have the, um, It.
W. Curtis Preston:So what I do
Prasanna Malaiyandi:The T log in
W. Curtis Preston:log might not
W. Curtis Preston:be there, by The way, I'm, I'm showing that not in all NoQ databases.
W. Curtis Preston:And by the way, um, we didn't talk about no SQL when we were
W. Curtis Preston:talking about database types.
W. Curtis Preston:I thought no SQL meant that they didn't use sql.
W. Curtis Preston:It's, it's not, it's not only sql, it's what is, what no SQL stands for.
W. Curtis Preston:Um, anyway, I just thought.
W. Curtis Preston:Important, but, but the transaction log is extremely important.
W. Curtis Preston:Um, the first being the one that we've already mentioned, and
W. Curtis Preston:that is the database crashes in the midst of something, right?
W. Curtis Preston:Either the database dies on the server or the server dies.
W. Curtis Preston:Someone pulls out a plug.
W. Curtis Preston:Um, and then when the database comes back up, This is the job of the
W. Curtis Preston:master file, the transaction log.
W. Curtis Preston:So we come up with the master file and the, and the database
W. Curtis Preston:looks at all of the data files.
W. Curtis Preston:And again, I know I'm using generic terms and these aren't always going
W. Curtis Preston:to be applicable to every database, but essentially the database comes
W. Curtis Preston:up and it's got something that looks across the database and
W. Curtis Preston:says, okay, something happened.
W. Curtis Preston:We need, we need to get back to a consistent.
Prasanna Malaiyandi:
Speaker:Put Humpty Dumpty back
Prasanna Malaiyandi:
Speaker:together.
W. Curtis Preston:Yeah.
W. Curtis Preston:Right before we can let people start, we need to figure out
W. Curtis Preston:if we had anything in process.
W. Curtis Preston:We need to figure out if there were any transactions that we started recording.
W. Curtis Preston:That we didn't finish, right?
W. Curtis Preston:So the database is gonna come up and it's gonna look at each data file and
W. Curtis Preston:it's gonna look at the, the control file.
W. Curtis Preston:Like in the case of Oracle, look at the control file, otherwise
W. Curtis Preston:known as the master file.
W. Curtis Preston:And then, um, look at that transaction log and say, all right, we were
W. Curtis Preston:supposed to finish this transaction.
W. Curtis Preston:And then you're able to go to each data file and say, what Transac.
W. Curtis Preston:What?
W. Curtis Preston:Um, there's a, like, I think it's called sequence number.
W. Curtis Preston:Sequence number is what I think Oracle calls it.
W. Curtis Preston:Look at the sequence number and say, did we finish here?
W. Curtis Preston:Did we finish here?
W. Curtis Preston:Did we, oh, look at you.
W. Curtis Preston:You didn't get to the sequence number.
W. Curtis Preston:You know what, we're gonna have to roll everybody back.
W. Curtis Preston:Right?
W. Curtis Preston:That's what, that's the job of the control file is, I'm sorry, the of the
W. Curtis Preston:transaction log is to roll everybody back.
W. Curtis Preston:Um, So that, uh, we don't have, um, consistency problems
Prasanna Malaiyandi:inconsistent.
Prasanna Malaiyandi:And then potentially you could also recover the database and roll back
Prasanna Malaiyandi:forward and reapply that transaction
Prasanna Malaiyandi:and not lose data.
W. Curtis Preston:Yeah, I wasn't gonna cover that yet, but in fact, I'm sorry.
W. Curtis Preston:We're gonna cover that in part two because I'm realizing that we've been talking
W. Curtis Preston:about this for a while and we're, you know, we're coming up on 35 minutes and
W. Curtis Preston:we've just covered sort of the basics.
W. Curtis Preston:We haven't gotten to the backup and recovery.
W. Curtis Preston:In the book, by the way, in the book, you occasionally see
W. Curtis Preston:these, those little, where is it?
W. Curtis Preston:The little scorpions, right?
W. Curtis Preston:And the scorpion is meant to be like a, a warning, right?
W. Curtis Preston:So here's what I have here.
W. Curtis Preston:Please note that just like everything else in data protection and protections
W. Curtis Preston:mentioned in this section, only protect against hardware failures.
W. Curtis Preston:So when we have.
W. Curtis Preston:All the things that are built into the database.
W. Curtis Preston:If a DBA accidentally drops a crucial table or a bad actor, uh,
W. Curtis Preston:deletes or encrypts your database, your fancy replication will only
W. Curtis Preston:make it more efficient, right?
W. Curtis Preston:It will immediately replicate whatever happened everywhere else.
W. Curtis Preston:This is why we back up databases too.
W. Curtis Preston:you about to say something?
Prasanna Malaiyandi:I was just going to say that for the people
Prasanna Malaiyandi:listening, Databases are complicated.
Prasanna Malaiyandi:So if a lot of this sort of goes over your head, don't freak out.
Prasanna Malaiyandi:Don't worry.
Prasanna Malaiyandi:The database admins have been doing this for years and years and years, right?
Prasanna Malaiyandi:And just like you've learned sort of virtualization, which we'll
Prasanna Malaiyandi:talk about later, and physical or traditional, uh, sources as well, you
Prasanna Malaiyandi:just need to understand the mapping.
Prasanna Malaiyandi:And there are a couple different differences for databases.
Prasanna Malaiyandi:But once you understand that, It becomes a lot easier, so don't freak out.
Prasanna Malaiyandi:Don't worry if this all sounds like a different language, but
W. Curtis Preston:Yeah.
W. Curtis Preston:And.
Prasanna Malaiyandi:just doing that mapping your head.
W. Curtis Preston:Yeah, and I listed, um, again, going to that db engines.com,
W. Curtis Preston:there are 13 types of databases and 300 different database products listed there.
W. Curtis Preston:So don't feel overwhelmed.
W. Curtis Preston:Um, this is not the world of, oh, is it Linux or, uh, windows or Mac, right?
W. Curtis Preston:This is, well, it depends, and there are types of databases.
W. Curtis Preston:I'll just be honest.
W. Curtis Preston:There are types of databases that I don't get.
W. Curtis Preston:Graph being one of them.
W. Curtis Preston:I just don't understand what or how they do.
W. Curtis Preston:I, I think you
Prasanna Malaiyandi:I I heard I heard a new one.
Prasanna Malaiyandi:Yeah, I heard a new one recently, which was, uh, vector databases,
Prasanna Malaiyandi:which is pro apparently starting to be used for like AI ml.
W. Curtis Preston:Oh, of course.
W. Curtis Preston:The world of A I M L.
W. Curtis Preston:Um, the, the thing that, the thing that you need to understand.
W. Curtis Preston:Uh, about any database is that it has really important data
W. Curtis Preston:that needs to be protected,
Prasanna Malaiyandi:Yep.
W. Curtis Preston:What you need to know about that database is, is.
W. Curtis Preston:From a backup and recovery perspective is you need to understand all
W. Curtis Preston:of the elements of that aspect.
W. Curtis Preston:Um, the one thing I think you will struggle with, with some databases and
W. Curtis Preston:some DBAs today is the same thing that we struggle with in other parts of of it, and
W. Curtis Preston:that is, Um, some DBAs of some products, confusing availability and um, you
W. Curtis Preston:know, that concept with data protection.
W. Curtis Preston:Um, and I think of like Cassandra MongoDB where they're like, oh, we're good.
W. Curtis Preston:Like we, you know, it's replicated.
W. Curtis Preston:We got everything three different places and, um, you know,
W. Curtis Preston:it's eventually consistent.
W. Curtis Preston:Um, and, um, we c we can lose a node.
W. Curtis Preston:We're good.
W. Curtis Preston:We can lose seven nodes.
W. Curtis Preston:The database will.
W. Curtis Preston:Okay.
W. Curtis Preston:What if we lose all the notes, right?
Prasanna Malaiyandi:Or what if
Prasanna Malaiyandi:someone drops a table
W. Curtis Preston:what if somebody drops a
W. Curtis Preston:table, right?
W. Curtis Preston:What if, uh, what if a, you know, a ransomware, uh, a threat actor
W. Curtis Preston:comes in and does bad things?
W. Curtis Preston:I, I don't, this is just like the same argument that I make in the SaaS
W. Curtis Preston:world is, um, Like everything needs something that is like backup, right?
W. Curtis Preston:I, I have the broadest term, the broadest definition of backup that I think
W. Curtis Preston:anybody in the industry has, and that is anything that copies the data to another
W. Curtis Preston:place for the purposes of restore,
Prasanna Malaiyandi:Yep.
W. Curtis Preston:that's, you know, snapshots in a replication to.
W. Curtis Preston:Um, as long as we have the ability to go back in time, and as long as I
W. Curtis Preston:don't have the ability to, to attack the backup with the primary, that's my
W. Curtis Preston:biggest concern that I often have with snapshot and replication based methods.
W. Curtis Preston:Right.
W. Curtis Preston:Um, so hopefully if you're using snapshot and replication, hopefully on the other
W. Curtis Preston:end your copying that data somehow.
Prasanna Malaiyandi:Yep.
W. Curtis Preston:Um, if, if that's a, um, you know, a C D
W. Curtis Preston:P style backup, if that's tape, if that's disk, if that's cloud.
W. Curtis Preston:But just don't tell me this thing doesn't need backup,
W. Curtis Preston:that just doesn't roll with me.
W. Curtis Preston:Right.
W. Curtis Preston:Um,
Prasanna Malaiyandi:Right.
Prasanna Malaiyandi:Can I, can I challenge
W. Curtis Preston:resilient, you know, go ahead.
Prasanna Malaiyandi:Sure.
Prasanna Malaiyandi:I, and maybe we'll talk about this when we get to a chapter.
Prasanna Malaiyandi:I think we have a chapter on cloud, right?
W. Curtis Preston:Oh yeah, I'm
Prasanna Malaiyandi:when we get there, um, one of my biggest questions is, what
Prasanna Malaiyandi:about everyone who uses like Amazon s3?
W. Curtis Preston:Mm-hmm.
Prasanna Malaiyandi:Right?
Prasanna Malaiyandi:Because no one didn't, as far as I know, almost no one
Prasanna Malaiyandi:backs that.
W. Curtis Preston:I, I, it is a problem.
W. Curtis Preston:It is definitely a, a thing that needs to be discussed, and it's a
W. Curtis Preston:thing where, Um, I firmly stand on both sides of that, of that story.
W. Curtis Preston:Right.
W. Curtis Preston:Um, that, that's a, yeah, we will definitely, um, we will definitely
W. Curtis Preston:talk about that concept and you will hear me waffle back and forth.
W. Curtis Preston:And part of it is, um, it's a little bit different in that.
W. Curtis Preston:The, the multiple locations and we're get, we're got the cart before the horse.
W. Curtis Preston:But you're deal, you, you have something that is, um, you have features that
W. Curtis Preston:include both protection against node and site failure and human failure.
Prasanna Malaiyandi:Yep.
W. Curtis Preston:So if you have stuff that deals with both
W. Curtis Preston:of those, um, That, that's where I start to waffle a little bit.
Prasanna Malaiyandi:Yeah.
Prasanna Malaiyandi:And we'll get to that
Prasanna Malaiyandi:at some point.
Prasanna Malaiyandi:Yeah.
Prasanna Malaiyandi:Well, when we get to the chapter on
W. Curtis Preston:And we're not, and we're not gonna solve that.
W. Curtis Preston:We're not gonna solve that.
W. Curtis Preston:That problem when we get there.
W. Curtis Preston:But anyway, so this is the basics.
W. Curtis Preston:Uh, by the way, if you, if I think that resource db-engines.com,
W. Curtis Preston:uh, is a great resource.
W. Curtis Preston:Um, and, uh, to help you understand all these different models, um, and, and to
W. Curtis Preston:understand which model your database that.
W. Curtis Preston:That you're, uh, using, um, we didn't mention Postgres, right?
W. Curtis Preston:Postgres is another very popular, uh, I think it's gotten a lot more
W. Curtis Preston:popular over the years because
W. Curtis Preston:what?
Prasanna Malaiyandi:I was thinking like the web app development stuff.
Prasanna Malaiyandi:I think they
Prasanna Malaiyandi:use that a lot more.
W. Curtis Preston:Yeah, I, I think it's got, it's, it's, it's an open
W. Curtis Preston:source product that probably has the.
W. Curtis Preston:Like data integrity stuff built into it compared to MyQ l Um, there's complexity.
W. Curtis Preston:It goes with that, but I think that's, you know, that's the thing with that.
W. Curtis Preston:Uh, all right.
W. Curtis Preston:Well, uh, enough talking about databases, we will do part two coming up and talk
W. Curtis Preston:about how to back these things up.
W. Curtis Preston:Uh, that's, and that's really you.
W. Curtis Preston:That's really what we
Prasanna Malaiyandi:The meat and
W. Curtis Preston:That's really what we wanna talk about.
W. Curtis Preston:What's that?
W. Curtis Preston:The meat and potatoes.
W. Curtis Preston:Yeah, absolutely.
W. Curtis Preston:All right, well, uh, thanks for, you know, the usual good
W. Curtis Preston:questions, et cetera, Prasanna.
Prasanna Malaiyandi:try Curtis and I hope you have a wonderful rest of your Monday.
Prasanna Malaiyandi:Happy 200th episode.
W. Curtis Preston:Happy 200th episode.
W. Curtis Preston:And, uh, thanks to those of you that have listened on those 200 episodes.
W. Curtis Preston:By the way, if you haven't listened to all 200 episodes,
W. Curtis Preston:you got some catching up to do.
W. Curtis Preston:Um, and, uh, remember to subscribe so that you can restore it all.
Listen On
