Strategies for Disaster Recovery and Security with Senior Solutions Architect Tara Kovaleski from Tierpoint | Ep. 40

Max:00:07

‍Give me a little background, industry challenge. Like, let's actually work backwards.

‍Tara:00:12

Yeah. So we were working with this client. They are a health care organization, and they're relatively sizable. And they brought in, some new leadership. And so then the leadership realized that they really had a risk situation because they looked at all of their different applications and what was required for them to run if something were to break and realized they could only recover 25% of their key mission critical applications in a expedient manner should they need to.

Tara:00:43

And so they pinpointed the fact that this presented a tremendous amount of risk to the organization. And so they went out looking for a solution that would enable them to protect more of those applications that they needed, recover more quickly, and not leave a lot of it to be so manual. A lot of our processes were gonna be very manual, which is what was gonna take a lot of the time for recovery.

Max:01:06

So was this something that they did, like, an internal evaluation? They brought in a third party to go through and, like, help them because, I mean, actually doing a BCDR planning and then, like, figuring out risk for an organization at the if somebody hasn't done that before, I mean, it's it's, I would say it's daunting, but it is an involved, you know, like thing. Like, you you really do have to work your way through the organization of figuring out what's running and who's running what and how does it interact with each other.

Tara:01:29

That was actually the first engagement that they had with TierPoint was doing a BCDR analysis. So through a partner, they brought us into the opportunity. They were working with the client to vet out some different providers, see where they would be able to, you know, get some bids and proposals. And so the first piece of business that we won was a BCDR analysis, doing exactly that, risk mitigation, understanding what was important in the environment, talking to application owners, sort of tiering all of those applications, putting together a Doctor runbook, all of those tabletop exercises, who's going to do what in a Doctor because people are really important in a Doctor scenario, and it's something that a lot of folks forget about. You're so worried about the servers coming up, but you don't think of who's going to push what button and who's going to be available and have Internet or have access to get into the environment to be able to do the things that need to be done.

Max:02:24

What's the actual disaster that you're recovering from, and do people actually even show up? I mean, there was a I I have an example of this from SoCal, and there was a defense contractor, and they had a BCDR plan, which was their key personnel. We're going to drive to their facility. And, you know, L. A, that could, I mean, normal cases, it's like an hour, hour and a half.

Max:02:40

And they got helicopters applied to Phoenix. And it was like, well, let me think about this. Have you confirmed that every single one of those employees does not have a family because they're not getting on your helicopter. So. Okay.

Max:02:50

So new leadership comes in. We don't know what we have. Let's figure out what we have and then make decisions. Okay. So it's like a really good place to start.

Max:02:58

So to your point gets involved and it starts with, let's let's figure out what's running, and let's kinda give you a a map of what's here. How long does that process take?

Tara:03:07

It was about 6 months, and it's actually been an ongoing effort because they've increased the scope as things went along, which happens often. Because once you start getting in and looking at things and you think the scope is going to be this much, and then it ends up being a wider focus once you realize the value that you're getting out of it or you start on on, you know, turning over stones and realize that there's 5 more stones next to it that you need to look under as well. But that they they had a very solid framework after about, I would say, about 6 months.

Max:03:36

I'm I've just imagined this meeting. So you go through a process. You're probably, like, giving you know, there's, like, teaser information flowing out. And then you have this like, okay, you know, here's our initial, like, scope report. And their scope report says anything happens.

Max:03:50

You're basically offline in SOL, right? Like, I mean, it's got to be a fun meeting, right? You know, like sit in like, okay, you're a health care organization. It's kind of important stuff going on here. And not to say that if you're not in health care, it's not important, but you know, that that like connection to, you know, all the little things that can happen, you know, within a disaster and disaster response plan and then saying, okay, you know, very few of your applications are actually in a state that you could recover them in a way that was meaningful.

Max:04:18

That has to be a really interesting conversation to be having and, you know, also to be on the receiving side of.

Tara:04:24

I don't think it was as difficult of a conversation in this scenario because I think they knew, and especially the the executives in leadership whom we didn't get a lot of interface with. Right? The folks that we were dealing with were were down a level or 2, and they understood the initiative that came from the executive leadership. So it wasn't as tough of a conversation as someone who's not aware that they have a tremendous amount of risk. And so then you're coming in and telling them that all of their existing processes and what they're doing is for not and everything that they've created is is no good, that's a really tough conversation.

Tara:04:57

But when they're aware of it and they're actively seeking someone to help them try to solve a problem, I think it's a little bit easier. So in this case, it was a it was a little better because they were really eager to say, okay. We know something's broken. We really need a map for you to help us fix to fix it.

Max:05:12

So health care, not just like a line of business leader or CIO or CTO involved with this, probably, like, compliance and risk and, you know, GRC or some other function, legal probably as well, helping to to evaluate and drive this within the business?

Tara:05:28

I would think so. I don't know how much we interface with, like, legal and those folks. We have a professional services division that actually are the ones that were hands on, you know, boots on the ground talking to all of these folks to to evaluate and come up with the report. We had the initial conversations to say this is a need. Let's bring in our consultant team so that we have the right folks that can help really do the evaluation for you.

Tara:05:52

A lot of it was really focused on application level.

Max:05:55

Like, you're running this application. These people are using this application. This application has this need. It's running. It's got this interaction with these other things.

Max:06:05

If this thing goes down, this application no longer works.

Tara:06:08

Yes. If this breaks, who does this all impact?

Max:06:13

Okay. So from an organization receiving that, you end up now you've you've got I mean, the scope is expanded. Right? But you end up with this initial report that says, okay. Here were the critical applications that we started with.

Max:06:23

And these are the kind of way these applications work and they're what they're interacting, what the business what what people use. Are you presenting remediation or options for Doctor at that point, or is this just the initial assessments? Like, here's what's going on, and what do you wanna talk about next?

Tara:06:39

So we were doing it in parallel, ironically.

Max:06:40

So as we started to get information and they started to

Tara:06:40

get information from you know, architecture infrastructure teams on they knew that we needed to implement something different for their environment. So we were actually working almost in parallel in creating a design while we were working on the overall assessment at the same time.

Max:07:06

There's a lot of options that enterprise has. I mean, BCDR means different things for different people, you know, and really you're talking about like backup and then business continuity and then disaster recovery. I mean, a really almost unique domains, you know, like one, they overlap with each other a little bit, but they they've been different things. How do you align that with objectives in this process of saying, you know, are we solving a backup problem? Are we solving a continuity problem?

Max:07:35

Like, what's the line now within businesses and how do you, you know, help people say, okay, this is the road.

Tara:07:41

So a lot of times, the way that we approach it is taking a look at their applications and understanding which ones are the most critical down to the least critical. And so in a lot of cases, especially with larger clients that have 100 and 100 of servers and 100 of applications, you kind of say, okay. Well, these and you assign values to those. So you say RTO and RPO. So you say it needs to recover in this amount of time because we can't live without point of sale, for example.

Tara:08:12

We can't live without point of sale. In a EMR, you can't live without that access to that or access to the ordering systems for more than, you know, 15, 20 minutes an hour tops. So those come down in levels of criticality to things that then are less critical. Email tends to be at the top of the list very often as well, and this client had in house exchange working toward moving to an m 365, but they're not there yet. And then you look at where you can solution them.

Tara:08:40

So business objectives start with the how much of this do you need to protect? In what time do you need to protect this? How quickly does it need to come up? And then how are we gonna be able to do that? What technology do you pair with that so that you accomplish the requirements that they're asking of you?

Max:08:56

In my head, I always go to, like, a scale, you know, with fulcrum on the mill because if you ask some so RTO recovery time, RPO recovery point, like right. You know, how long does it take you to back up? How, you know, how recent is the backup? And so this is like one of those things where if you ask somebody like, well, you know, how how frequent do you want your backups and how quickly do you want to restore them? And the answer is always going to be like, well, instantaneously and instantaneous.

Max:09:18

Right. But that influences then cost. And there's there's some surprising cost factors that come into this that get missed. So when you're going through and trying to size like what an RPO and RTO should look like for somebody. You know, for example, I've seen organizations forget that if they're backing up off prem, they have to be able to download that data.

Max:09:38

And how long does that take them based on what their circus look like? So is this at this point just on a spreadsheet and you've got a formula and some math associated with it? Are you doing this on a whiteboard collaboratively with somebody? Like, how do you help shape the conversation around figuring out what an RPO looks like? You know, starting from, like, of course, we want it instantaneously, but now we have to talk about budget realities and and cost.

Tara:10:02

So I think some of that depends if you're talking wider than just this client. A lot of that depends on the client size. I have found that when clients have 100, in this case, they have 6 or 700 servers, they realize that not all those servers are equal. So that makes the conversation about which ones are the most important to the least important a little bit easier. You have more of a midsize.

Tara:10:26

That's where you start with folks that everything is equally important. So you give them the budget of what that would look like. And, ultimately, what happens is everything becomes not as equally important. And so they say, okay. Well, we could probably wait for this.

Tara:10:40

And maybe we don't need to have always on for all of these machines, or we don't need to have continuous replication for all these machines. Some of these could we could have a a day of data because they don't change that much in the daily you know, on a daily basis. And so there's no point in, you know, continuously replicating them. So then you have those types of conversations to really dive in and understand server and application based what exactly they need from each one of them, and then you can give a much more accurate reflective proposal for budget as well as you're still meeting the requirements.

Max:11:15

20, 25 years ago, this was a relatively simple problem. Right? We had a database on a SAN. We were doing triple mirroring. We were then doing, you know, freezing the data, flushing the disk, breaking the mirror, backing it up.

Max:11:26

Right. And it was going to tape and the tape was going off-site. Now there's a lot of different there's a lot of different tech and a lot of different platforms in the market that attack this. And and they do different things. I mean, it's you know, they have a they have a different approach.

Max:11:41

They're trying to solve a different pain point. They have a different target. And of course, you talk about the difference between greenfield and brownfield. Does an organization already have a piece of technology? Are they replacing that tech or are they, you know, augmenting and using it?

Max:11:55

So as you're going through and looking at this, like, you know, process of saying, okay, now we've got an idea of applications and what's running where. And then we maybe everything's critical, but now we're going to put a cost associated with it. And then we find out that some things are actually critical and some things are less critical. And then you start talking about really you get into the technology, you know, side of the conversation. Now it's like, what platform do we run?

Max:12:18

What replication options do we have? How much does this cost? You know? And and even within that, you probably have cost scales as well. Right?

Tara:12:26

Absolutely. So you look at in this case, it's a Doctor solution. So some of the decisions that you make in terms of what you're going to provide in Doctor are based off of what they're running in production. So for example, you have a client that is running all Dell. Then we can stand up an all Dell target and use some of those native capabilities in the storage to do snapshots, you know, SAN replication, if that's appropriate for, you know, certain workloads depending on the recovery time.

Tara:12:56

Then you would look at, you know, where does Zerto's obviously a huge player in in the Doctor and recovery space. So you look at where do you need to have, you know, continuous replication where your RPOs are are minutes. You're not losing very much data. Then you have a a bunch of different checkpoints to go back to should you have some type of failure or a ransomware attack for for that matter. You need to be able to go back to before you identified that, you know, that that bad actor was in your environment entirely.

Tara:13:27

I mean, sometimes backups aren't efficient for that. So the first thing we do is weigh what they currently have in place and see if you can take advantage of any of that. And if they're familiar with some of those technologies and things they wanna handle on their own versus outsourcing it to a provider to handle for them. And then you take a look at you sort of just evaluate all those all those different pieces, and it's like a puzzle.

Max:13:51

You mentioned a couple. Right? Acronis, Datto or Datto Veeam, Zerto, native replication, you know, like like, this list kind of goes on and on and on.

Tara:14:02

Yep. SQL always on availability groups.

Max:14:09

Yeah. Sand based replication, you know, snapshotting. How does a client decide between backups and Doctor? Like, like, what is what's the push for them between, hey, we've got backups and now we have a real Doctor plan?

Tara:14:23

A lot of times, it's recovery. So it's how much data if something were to fail, are you willing to lose. Majority of of companies, if they don't have databases that are being backed up on an hourly or even, you know, 15, 30 minute increment basis are only backing up once daily, maybe twice, but the majority of the servers are just being backed up once a day. So a lot of the decisions revolve around, could you lose a a day's worth of of data? And if you were in the business, you know, health care where you're processing a lot of information on a daily basis, then no.

Tara:14:59

You know, those servers that are doing all that processing can't lose 24 hours or even 8 hours of data. You know, they'd wanna only lose minutes. And so that's what that's one piece of the Doctor versus backup. The other piece is recovery. Right?

Tara:15:15

Backup can only recover so quickly. You said earlier, if you're backing up off-site, now you need to come back across the Internet to back up to your local, you know, your local servers. Some of the servers, if they're extremely large, they're gonna take a minute to recover. You know, speed of light is only speed of light, and so you're only trying to push so much data. It only goes so fast.

Max:15:36

You made a really good point I wanna talk about for a second. In this case, you know, health care, EMR, so medical records will just make this really broad and say like in a hospital setting. In a hospital setting, you have, admitting you have your finance functions, you're doing medical billing, you're doing you know, you probably have a state or federal overlay on top of that with billing codes coming in and out. You've got patient referrals, you've got third party interactions and billings. You've got prescriptions, you've got patient care going on.

Max:16:06

At the same time, you've got guests coming in and out of the facility. You've got security controls that you have to put. I mean, there's a lot happening here and some industries maybe less happening, but even even in like, you know, I think like let's go manufacturing, like manufacturing, you're still taking orders in managing something that's building something. Right. You know, it's it's mostly, you know, even if you're like milling titanium, you know, you've got CNC, you've got a PLC, you've got something that you're doing something with, and then you're producing a product, you're packaging the product, you're shipping orders out.

Max:16:38

Right. And that's a lot, you know, like there's businesses are a little more complex, and I think a lot of people you know, we start actually breaking them down of what they're actually doing. Like, there's there's a lot of stuff going on with these with with companies.

Tara:16:50

Yep. Exactly. And that's and those are the kind of conversations you have to have. I think, you know, this technology comes second to understanding the client's business and what they're doing, how they're doing it. And if something were to fail, who does that impact?

Tara:17:05

How does it impact them from being able to continue to operate?

Max:17:08

I love it, Lianne. Technology comes second. It's because really, you know, what you're talking about right was first is like, what is the objective? What's the pain? What's the problem you're trying to achieve?

Max:17:19

You know, well, it probably it's not you're not trying to achieve pain, but you're trying to remedy pain or problem. Right?

Tara:17:27

Right. Hopefully. Hopefully not creating more.

Max:17:33

Although I know some organizations where it does feel like they're trying to achieve pain and then working backwards. Now, in the modern world of, like, Doctor, so backup recovery, biz, continuity, Doctor. You know, we've we've you've got ideas around on premise options. So are you recovering the systems on premise? You know, what's your disaster that you're dealing with?

Max:17:52

And then you've got off premise recoveries and you play in both these spaces. What do you think is like the line and the over under of of an organization looking and saying, trying to decide, are they going to have an internal recovery option versus let's leverage an external recovery option? And how does that get implemented?

Tara:18:11

So I would say internal versus external. So putting it out in a data center that you don't own, even if it's just colocation. A lot of times, that decision comes down to mileage. So the a different client, but someone we're working with, their their where their primary and their secondary are very close to one another. So if there was any type of major failure within that general region, you know, we're talking, like, maybe 20 miles, it's not very far.

Tara:18:40

So one of the decisions to not have it on prem or internal is purely location based. It just doesn't you know, it poses a whole different element of risk to have everything so close. And then you could look at it goes back to the planning. So you say, okay. We're gonna do this off-site.

Tara:18:58

So we've decided we're gonna do this off-site. How far off-site? And do we wanna manage certain things, or do we wanna have a company that's going to help us? In the event that something happens, are people gonna be able to get to where they need to go? Are they gonna have act the access that they need?

Tara:19:13

Do we want them focusing on other things? We want them focusing on applications of the business and making sure things are running, and let's have someone else that's gonna take care of the infrastructure. Bring up those servers. Make sure all the lights are green. Make sure the network connectivity is there.

Tara:19:27

And now we don't have to focus on that piece of it. We can focus on the applications and the things that are running our business, you know, the finance systems, the security systems, those things that we need to make sure work so that we can continue operations. We'll need to make sure the lights are on.

Max:19:43

Declaring a disaster and not implementing, but actually executing a Doctor plan. There's a lot of moving pieces with these things usually. And because now you're talking about what's the decision to declare a disaster and start executing on the plan? Like, who who was actually involved in that decision making process and says, okay, we're going to flip the switch. And now you're you're you're turning up infrastructure somewhere else, recovering data onto that infrastructure, reporting applications and users to that new infrastructure, running that infrastructure.

Max:20:15

And then at some point in the future. Bringing all that back. You know, and and those are like the nontrivial operation to both. Go one way and then come back the other way. I think people are really surprised the first time they actually test their plan.

Max:20:33

How does to your point, how do you go through this process? Because it's one thing to say, okay, we decided to back everything up off-site and like, let's recover it. Right? Walk me through. Like, how do you actually help people go through this process?

Max:20:44

Because testing and and training and, you know, revisions, you know, it's a really important part of this process.

Tara:20:54

Absolutely. I would first start by saying that folks that purchase a Doctor solution from a provider, TierPoint or otherwise, need to make sure that they realize it's a collaborative effort. You can't hand over the keys. You're not handing over the keys to TierPoint, and we're gonna fix it and make it all come back to life without any intervention or participation by by the customer themselves. Critical is doing your annual test at least once a year.

Tara:21:20

And there are many, many organizations that we talk to that don't test either resources or don't have the plan in place. They get sidelined by, you know, all of the projects that just become a higher priority. So we highly encourage that. We offer a few different kind of tests. So it depends on what the client's requirements are.

Tara:21:42

You know, I continue to go back to that. Right? But, you know, we can do an isolated test, which means we can stand up a sandbox environment, and it won't have any outside connectivity but you can see how your servers will come up in a Doctor event. We can do what is sort of a non isolated which is still segmented from production So we're not impacting the replication that's happening in case something for real were to happen. But we're gonna give you network connectivity, which I think that's probably the best type of test to do because you need to make sure that if you fail over to the Doctor side, you can access it.

Tara:22:15

The network is always a very critical piece, and it the smallest detail on an IP address can cause you to not, you know, be able to to have access for your users. So everything might be up and running, but no one can get to it. And then the 3rd kind of test we do, and a lot of times, this is for folks that have compliance requirements or some type of security requirement is just a full failover test where they really take down production, fail everything over to the Doctor side, and then bring it back. And I think if you're doing that, number 1, there's an element of risk, and you have to do a lot of planning for that, obviously. But I think that gives you a really good picture of what would happen in a Doctor.

Tara:22:56

And then all of these types of tests go into your runbook. So you need to make sure your runbook is up to date. You need to make sure that the steps that each person takes, so to your point, as well as the client, what is each one doing along the path of recovery? And all of that is clearly documented.

Max:23:15

Testing is important for 2, I think, really two reasons. Right? The first one is, like, does it actually work? And the second reason is every single time you do a test, you find something that you missed or forgot about. Because none of these environments are static.

Max:23:30

You know, by the time that you finish an evaluation, write a Doctor plan, implement implement it and then test it, stuff has changed. And then, you know, I mean, like, I have like like thinking about doing a full scale production fail over Doctor test. Fortunately, I'm not I'm not like running it for anybody right now. Well, like, the thought of that, like, like, just makes my like like my my body is just like, you know, like, it's it's very it's it's a very discomforting thought. Right.

Max:24:05

I guess it's just what I would say. But it's it's really important. Like, a lot of people miss this stuff, and they and they find out, like, they have bad experiences down the road. You know, it's, oh, we thought we had that backed up. We thought we had that in plan.

Max:24:17

We thought we did this or tape, you know, like, I I mean, go back 20 years. Oh, our tape drives weren't act they it was running, but we weren't actually getting data on it that we could use. The tapes are all corrupted. You know? And you're like, well, we didn't know that until 6 months later when we actually needed to use the tape.

Max:24:31

And we put the tape in and there was not you know, like, there was nothing we could get on the tape. I

Tara:24:35

think it's really important to, Max, that folks, if their environment changes, they have significant changes. They add a bunch of servers. They add a new application. They move an application to SaaS. They need to communicate with your point and let us know that those things are happening because it has an impact on their Doctor recovery and what's being protected and then has an impact on the runbook.

Tara:24:58

And so even if it's not test time, we should still be talking all year and looking at different changes, firewall rule changes that happen in production that need to be replicated in Doctor. All of those things, if we maintain continuous collaboration, it makes for so much more success when you're testing. And then in the event that you have something happen, it it just increases the probability for success.

Max:25:24

This is a really interesting position for to your point of view. And because, you know, like, your services offering, you run data centers, people can co locate and re equipment in with you and be in secured facilities. You offer professional services and managed services in those data centers. You offer, you know, backup and BCDR, you know, services. Right?

Max:25:46

So, like, there's there's, you know, you can exist in lots of different, like, people want different things. They need different things. Right? But I mean, how many at some point it's like, oh, you know, we still want to have our servers on-site because we want them physically close to us because we like them in our room downstairs. And then our fire suppression system goes off and, you know, we lose our computer room and we fortunately have a D.

Max:26:13

R. Plan. And then maybe that doesn't go back on-site after that experience. You know. That that happens.

Max:26:21

Yeah, a lot. I was you know, I had to I had to write a lot of plans around earthquakes. I was never worried about earthquakes. I was always worried about a fire on a in a computer room. Like, it was always it was always the water that freaked me out.

Max:26:33

And people are like, oh, you know, it's okay. You're gonna have inert gas. It's like, no. No. Building code still requires you to have water and all these facilities.

Max:26:38

So, like, water is going to discharge. And, of course, you have regional issues like with, you know, in Florida with hurricanes, you know, I mean, hurricanes happen in other states other than Florida. Florida just is known for the hurricanes. Right? But, you know, family in Alabama, you know, they had a big hurricane issue.

Max:26:55

Right. You know, so but in your stories, of course, of like, oh, we had we had alligators in our in our in our call center. Is there a. What do you do? Right.

Max:27:06

Exactly. Exactly. Walk me through, you know, like high level. Like, walk me through what what's on a run book. And, you know, because it's like you have a you end up with a plan, right?

Max:27:17

Part of the plan is a run book, actually. I mean, walk me through the whole thing. Like, what ends up on this document? Like, how? Because we talk about this.

Max:27:22

Oh, by the way, I'm gonna I'm gonna state still terrorist thunder here for a second. Print out your runbooks. Paper. You want this stuff on paper? Trust me.

Max:27:39

Just just just take a tip here. Print print out. You want contact information. You want the whole thing. You want it printed.

Max:27:44

You want it in a binder. You want that binder all over the place. If you want it on paper, you just I'll just leave it that. I'll let her explain more.

Tara:27:52

I agree. You have no power. You can't get to it online. So what goes into a a runbook is as detailed as the client wants to make it from a collaborative standpoint. From a tier point standpoint, we're going to list out all the network connectivity, all the IP schema that's in place.

Tara:28:07

We're gonna list out all of the servers that are being protected, all the IPs associated with all those servers, the boot order in the event that, you know, it fails over. Which one do we start with and how does that flow? We're going to have the authorized contacts because one thing we talked about earlier was who can call in and who can declare disaster and when do they declare a disaster. And we have to have specific folks that are authorized to declare disaster. It can't just be anyone from the company.

Tara:28:36

It has to be someone who's been enlisted and is validated when they call in for security purposes for the client that they're not you know, someone is just not randomly calling in to do a Doctor failover. So that is well documented in the runbook as well. And then in other runbooks, I've had step And then in other runbooks, I've had steps

Max:28:51

where clients have to run scripts to ensure that certain applications

Tara:28:52

are coming up or in order to run some type of processes once the servers are up. And so we'll document what those scripts are going to be, that the client is going to do them. I've had us imp you know, imp input in the runbook what those scripts are and screenshots of what that would look like and what the output should look like. So it can be extremely detailed. It could be just, well, now the customer's going to recover, you know, from their Veeam backups, and that's a process they're going to run.

Tara:29:24

They're going to point it here, etcetera, things along those lines. But it should document all of the people. It should document all of the processes, and it should document all of the systems that are going to be touched in a recovery scenario. Who's important?

Max:29:45

Who can declare disaster? Who executes the disaster as it executes the disasters to the recovery. Right? Who says, yeah. Like, we can we can trigger the recovery.

Max:29:53

Who needs to be involved at that point? What infrastructure in what order? What things do you need to run? It's not because it's it is it's not just turning a server on. Right?

Max:30:02

Like, it's there's it's not just turning a server on. There's a lot more to it than that. Right? So, like.

Tara:30:07

There is a lot more to it than that.

Max:30:09

What else needs to happen on that server? What order? Right. Because things have to talk to each other when they boot. How do you validate that it's running and how do you connect to it after it's running?

Max:30:19

Right. Kind of kind of in that process. And then and then probably the next steps, which are how do you unwind all this and bring this all back after whatever's happened has been taken care of and you're back. So you you hit on this again. Right?

Max:30:35

Because even in saying this, you made the point. That to your point can be as basic as we're providing infrastructure to. We're helping you to. We're doing it for you. And that's a pretty big range of involvement and, you know, what kind of assistance you're giving.

Tara:30:54

I think that's what differentiates us a lot of times from the competition is the fact that we're flexible. And so we have the ability within reason, I'll say, to customize solutions for a client. It's not an all or nothing, and it's not a do you know, you guys we'll give you the servers. You know, we'll provide the infrastructure. We'll make sure the lights are on, and then we're you guys handle all of it.

Tara:31:18

We kind of we do have a wide range, but I think that allows us to be able to really help customers achieve what they wanna achieve in their budget. So folks that, you know, can't protect everything a certain way because their budget just won't allow it, but they need to make sure that their p systems that would help them at least limp along but continue to do business while the rest of it is being recovered. And that's all that their budget will allow at this point. We can help them do that.

Max:31:50

You mentioned SaaS earlier, and I'm gonna bucket SaaS and cloud. Really, I mean, cloud is just I'm just gonna use, like, external resources that you're consuming. Right? So in the Microsoft space, it's probably 365 in Azure, you know, but it could be other things. Could be AWS, could be GCP, it could be Oracle, it could be whatever.

Max:32:11

Right. A lot of people are under the impression that by going to a SaaS application or to a cloud, that that cloud or SaaS platform is providing them backups and protecting their data. But that's not necessarily true.

Tara:32:24

It is not necessarily true. I would always recommend that if you're using let's use 365 as an example. There is some level of backup that is inherent in 365 with Microsoft. But if you have the need to hold data for a longer period of time, longer than, say, 30, 60, 90 days, you need to have an additional backup support on that. And that could be for, you know, whatever business.

Tara:32:50

It doesn't have to be 7 years, which is the, you know, common compliance. But a lot of folks, they may need to hold on to data if they're interacting with a client themselves for maybe a year or 2. That in case they have to go back to communications.

Max:33:08

Let's rephrase this a little bit here because there's a difference between an outage or failure on a storage system on a cloud platform and what their obligations are or what expectations are to recovery. You know, you talk about, like, 30, 60, 90. Right? But then there you have stuff where it's like a delete is a valid operation on a platform. If a user presses a delete button, I mean, delete is valid.

Max:33:28

It's going to delete. But maybe you didn't want delete. Valid. It's going to delete. But maybe you didn't want delete.

Max:33:35

And so how you recover from delete becomes interesting for people. Right? It's not it's not, you know, because then you've got this, you know, these concepts of, like, you know, do you have the equivalent of, like, a trash bin or a recycle bin? You know, who has access to that trash bin? That recycle bin is the user?

Max:33:53

Is it the administrator? Like, how long is that retained for? Exactly. So I find that that surprises people a little bit when they get into this. So I mean, so right now, like with with this organization, you know, you're talking about on prem and and they're migrating, you know, they're in the process also going from an on prem exchange infrastructure.

Max:34:13

And I have nightmares from privityb corruption. So the movie into 365 totally makes sense. So now you're going to end up with maybe several Doctor plants and and runbooks that have to be thought about. Right? Because, like, their approach with exchange on-site versus exchange in 365, they're going to have, you know, like they wouldn't have it wouldn't be like, hey, we had a fire in our data center that's going to trigger a recovery from 365.

Max:34:39

Right. So it's going to be discrete runbooks or chapters in the runbook? Or, like, how do you keep track of this and and keep sense of this for people?

Tara:34:46

So I think when customers move to 365, they don't necessarily they can, but they don't all necessarily get the subscription or purchase the licensing through TierPoint. They can purchase it through others. We can provide some level of support for them if they'd like that. We can provide backup for it if they would like us to do that. In an outage in production, it they are out in 365.

Tara:35:08

There probably should be a mention in the runbook about 365 and testing to make sure that the mail is working. But, theoretically, the access to 365 would only be compromised if their access to public Internet was compromised. They should be able to access that from anywhere at any time unless it was an outage in Microsoft. And if there was an outage in Microsoft, then that would be something they would need to plan for and try to, you know, understand if they were gonna recover from backup or how they have their 365 structured. Make sure if they're doing backups that they have it in multiple zones so that if one zone fails, they're not subject to data loss.

Tara:35:50

They could recover from an alternate zone. All of these different things, you can plan for 10 different ways of redundancy. You could build the rocket ship, but someone has to fund the rocket ship. So it all comes down to, you know, how many bells and whistles can you put on your rocket ship that you are able to to fund. You keep saying network.

Max:36:16

I don't know. I'm I'm really curious. What are the what are the misses that people make with with network? With with BCDR? Because it's, you know, network's important.

Max:36:29

You have to connect to it. You have to have application. You you know, do you have cut new, you know, like, do you have enough speed, right? There's like they're they're kind of, like, flirting around this a little bit. And I'm getting the sense of, like, you know, there's some there's some war stories here a little bit.

Tara:36:45

Always. And I'd say over over the years, you know, I've learned to make sure that we place a huge priority on understanding the network. Because I think one of the common mistakes is you focus on from the primary site how folks if they were at the primary site, how they would access the the secondary site. But in today's world, there's a lot of remote users. There's a lot of VDI.

Tara:37:09

And so you need to make sure that you plan for that VDI infrastructure to come up at the secondary site. And there's that is entirely different segment of recovery than just recovering your servers that are running your everyday, you know, work applications. And so those are those are keys. Then you consider so when I think network, I think, you know, bandwidth. Make sure you have enough bandwidth or you have flexible bandwidth at the Doctor site.

Tara:37:36

Because if you have 500 folks that are coming into production at any given time and you're running a gig in production and then you think, well, I'm just gonna do, you know, 2 100 megs in the Doctor site because I don't really need to, and I have a direct pipe that's running from primary to secondary. But you don't think about the folks that need to use the Internet to get to that in the event of a failure. Then you create a a bottleneck unless it can expand. So it's all of these different things that you need to consider. And the next, you add in security.

Tara:38:06

So you have everything built very strictly at the production site. Everything's locked down. Are you making sure that all those same rules and policies are on the Doctor side? Because if folks have to run there even a even a couple of days, are they as well protected as they are in production?

Max:38:24

We've talked about Doctor planning from, like, traditional outages. You mentioned really early on with this talking about ransomware. I'd like to talk more about this because this ransomware is is a is a big driver for people within Doctor now. But it introduces another vector, which is identifying where your clean data actually is and then being able to, you know, you know, so like that's a little bit of a it's not as simple as just saying, hey, we're going to pull Doctor. Right?

Max:38:55

Like there's a little bit more that has to happen with that. Assume I know nothing. And from the top, like how the company is looking at this from a risk standpoint of building a plan around protection against ransomware. So that way they had it. How does that get approached and how do you figure out then?

Max:39:13

We can recover or we can't recover or we have good data or we don't have good data.

Tara:39:17

So the first thing that we want to understand is if they have cybersecurity insurance, Because if they have, some level of cybersecurity insurance and they do have a ransomware attack, we need to be very cognizant of what the rules are under that insurance. So you can't just go and enact a recovery plan. You need to make sure you're working with the insurance provider on what steps that they deem to be relevant in a recovery so that they will provide the coverage that the client is looking for. We had a you know, we've had a few situations in recent, you know, months with clients that had this scenario, and they were, you know, without operations for a little bit of time, but they had to go through the sequences that were required by the cybersecurity company, and they had to bring in consultants that were helping to map what the recovery steps would look like. The next piece to that is understanding what security systems they have in place so that, hopefully, those systems that whether they have a a SIEM or they have some type of, you know, MDR or whatever the case may be that enables us to go back into the logs to try to identify at what point they were breached because that will give you the signal to understand when you are able to go back and recover to and where you may have, you know, clean data from.

Tara:40:40

And it could be days. It could be weeks. It could be months. And so it depends. And then from there, work alongside to see what is gonna be the best way to do a recovery.

Tara:40:51

So depending on technology, a lot of clients end up having to recover from backup because that's the only data that they have that goes back that far. If you're doing a continuous replication, say you're using something like Zerto, the maximum you're saving there is 30 days of checkpoints, if you're lucky. And so that doesn't tend to be a reliable mode for recovery in a in a ransomware situation depending on how long the intruder is in your environment. There's a lot that goes into it if there's a ransomware situation. And what we do is we try to work with the client to prepare them the best that we can, sort of give them a layered approach.

Tara:41:32

You know, we'll give you if, hopefully, you can recover quickly. Hopefully, you have tools that can log and understand that your environment's been breached very quickly, so then it gives the ability to react really quickly. And so folks aren't fishing around in your environment for, you know, weeks gathering all kinds of data that they shouldn't be. So it's a com it's not even just a combination of Doctor at that point when you're putting together a full plan. It's it's security, and it's a full solution on the production side.

Max:42:01

It's an interesting kind of footnote at the end there of not trying to it wasn't it wasn't of common around focusing on preventing a breach. It was how quickly can you identify the breach and know where you have good data? And the faster that you can find, identify that, the better off you are. Because I mean, because then you get into, you know, the other side of it, which is, you know, stories and scenarios or somebody does get onto a network and breaches a network and corrupts the backup system as well. You know, and right.

Max:42:33

Because if the company, if the enterprise doesn't have a valid backup anymore, right? The ransom is more effective. Right? And so then you get into, you know, different technologies that can help try to prevent that from occurring as well. So there's all these, like, levels of risk, right?

Max:42:47

And then there's there's like risk and then there's objective and then there's cost. And and it's like like like like kind of doing, like, the wave dance. Right? Trying to figure out. Yeah.

Max:43:03

For companies that haven't had a disaster, they haven't had natural disaster, they haven't had localized a disaster, they haven't had data loss disaster, they haven't had a disaster. For companies that haven't had a disaster or a ransomware attack, you can get into a skewed perception of risk then because it hasn't occurred to you yet. And it's like, okay, this maybe isn't as big of a deal. How do you deal with that? Like, you know, because this is really important for an enterprise to have.

Max:43:27

But if they haven't experienced disaster, then it's like, how do they measure their risk and the likelihood of that risk, you know, changes because we're, you know, we're dealing with people and the perception, right, is my experience. And, you know, there's a recent, you know, a recency bias. What's the bright approach to take with that?

Tara:43:45

So there's there's 2 ways. And sometimes it's that this is a huge challenge because Doctor is an insurance policy. When push comes to shove, it's it's a cost. It's like when and I always equate it to car insurance. Right?

Tara:43:58

You don't get in your car and assume your bone could get into an accident. You think you're a good driver. You expect other folks on the road to be good drivers. And so you have no expectation of getting into any type of accident that, you would need to use your insurance, yet we all have very good car you know, we all have insurance. Doctor is very similar.

Tara:44:18

And so a lot of times, the the way we can go about it and it it depends on what level of folks you're talking to in the organization. But if you're talking on a more leadership where they can see a wider picture of the environment, looking at systems that would be down, what the cost to the business is per hour is a really good way to break it down. If you can't receive orders from clients for 1 hour, what would that cost your business on an average? If you can't have folks that are working because you have a call center and no one can receive calls, what is the cost of that for 1 hour, all of those folks sitting there idle? And sometimes when you can actually equate cost in dollars to what that would, you know, cost if you had a plan to protect yourself from that, That helps make the picture a little bit more clear.

Max:45:17

What do you think the threshold is on that in terms of actually getting budget allocated? I mean, if you if you've got a $100,000 expense per, you know, lot loss per hour, you're not gonna spend $100,000 a month to protect that. Right. You know, that's that's not like we haven't crossed the like risk versus costs, you know, benefit like threshold. Right.

Max:45:36

You know, you're not going to I mean, maybe if you even have this annual, if it was a potential loss of $100,000 it cost you $100,000 per year to protect from a $100,000, like, you still haven't crossed that line. Where do you think people end up with that in terms of understanding risk and cost with risk versus, you know, an annualized cost per to mitigate that risk?

Tara:45:54

So I think you have to estimate. And so a lot of times, you know, worst case, you say you're you could be down for 30 days, or you could say you could be down for a week, whatever metric they think is appropriate. I was, like, months to work in in round numbers. And then even if you in annualize that and say, okay. You're not gonna be down for a whole year.

Tara:46:15

But worst case, hopefully, you're not down more than a week. And so then you could break it out, you know, and you just look at the numbers. I've found it seems like depending on the organization and if those that are really not seeing the risk, maybe you can get a 10% of what their downtime if folks can quantify what that downtime would be in dollars and cents, which is very challenging. It's challenging to get those numbers. It's challenging to look at it in that fashion because in order to do that comparison, you need the client to be able to identify those metrics.

Tara:46:51

A lot of times, what I find is we start small and we and we grow. So rather than doing a full Doctor plan for someone like that, you may offer them a target place for them to recover their backups too, and you start there.

Max:47:05

I mean, some industries are easy to do this. Right? If you're in ecommerce, if you've got a retail and you know, you've been able to point of sales like you do, you already have this metric. I mean, you know, your revenue per hour because it's just something you're tracking anyways for the business. If you're doing manufacturing, you're talking about shipping order volume.

Max:47:21

Right. You kind of have an idea of like what your order volume is per day and what you're shipping. And of course, within that, you've got critical times because, you know, if you don't get a shipment out by time, like it doesn't go that day and it has whatever delay and then that causes whatever problems. So some industries are easy. Other industries a little bit harder.

Max:47:39

And even industries that I find that when you're talking about professional services where they're billing by hour and you actually have a cost of your labor force not being productive, that gets hard for people to quantify and look at. Like, it's it's I've always found that disconnect to be very interesting of, you know, you know, this building is an arc. You know, you're an architecture firm. Like, this is what your project load is. You know, this is what your, you know, your day productivity cost is for your for your employees.

Max:48:06

And this is what your billing rate is for that day for those employees. And this is what your risk is associated with an outage because I've seen this also with like network, like, you know, should we have a, you know, an adequate network redundancy strategy? Like, oh, you know, I mean, we don't spend the money on that. You're like, okay. You know, like, in the evening, our outage, you're like, oh, you know, like, maybe you maybe we want it now.

Max:48:26

It was really frustrating for me years ago, but, like, understanding of the psychology around a little bit more of, like, just that recency bias and perception and, like, not actually having like, until you until you're out roller skating and you fall and you scrape your knees, you don't necessarily wear, you know, kneepads. Right? And then the first time you do it, you're like, yeah. You know, I really should probably wear kneepads.

Tara:48:46

Yeah. And I I found it very challenging to get those numbers. Either you're not at the you're not working with the folks within the organization that would have access to that information, or sometimes they're they're not necessarily willing to share. But I think there's other reasons for that. It goes down to, you know, how much trust have you built with your client, and are you working in in collaboration and partnership in trying to find a resolution to their their issue?

Tara:49:14

And I always view it as we're trying to help. And if we need to help you build a business case in order to get you the budget that you're looking for, we'll do whatever calculations we need to or, you know, make things work to try to fit within the budget and build the story that shows why you need this, what the ramifications are if you don't do this, and this is how much it would cost, and this is the value that you're building behind it.

Max:49:39

For the most part, I find that IT teams and IT practitioners are really trying to help and, like, support the business that they're, you know, like, 99% of time, I feel when I find like the IT teams are really actually trying to do something to help the business. But I noticed consistently a a disconnect between I. T. And the business at large and we speak different languages. Right.

Max:50:01

And we get it gets insular and like the business doesn't understand it. It doesn't understand the business and vice versa. And then as a service provider, you're interfacing mostly with I. T. So you talk the I.

Max:50:10

T. Language, but you also have to talk the executive language. What is the big miss for an I. T. Team when they're trying to go through and position to to, you know, a non I.

Max:50:19

T. Leader that we need to do this? What do you think is the disconnect and how can that I. T. Team like, look at this a little differently or talk a little differently or express this a little differently that would, you know I don't know if they get the point across, but, like, bridge that gap so, you know, there's a common language being spoken.

Max:50:36

Like, how does IT get better at this?

Tara:50:38

I really think it's a matter of looking at the business from a wider lens and a more global perspective. I'm sure you know this from working with folks. If you're talking to a CIO, a CFO, they're not just concerned with one specific business unit. They're concerned with all of the business units and and in total. And I think that sometimes it comes down to IT being a cost center and being viewed as a cost center and them viewing themselves as a cost center.

Tara:51:07

And I've seen this recently that the mindset seems to be changing a little bit where they're trying to position themselves more as, I don't wanna say revenue generating because I think that's tough for IT, but contributing. So we're not just something that you have to spend money on, but there's value in what we're providing to this organization. And we can make things more efficient. We can make things more effective. And we can help the business, you know, run more effectively, more efficiently.

Tara:51:37

And that translates into dollars, and they may not know what those dollars are per se. But conveying it from a message, something along those lines, I think, is proving to be a little bit more effective for IT departments to to say upstream. You know, here's let me not just give you you know, here's 3 bids. This one is the lowest. These these 3 bids different are different because this company is going to do x.

Tara:52:02

This company is going to do y, and this company is going to do x, y, and z. And so while x, y, and z may cost a few dollars more than the y, but look at everything we're getting out of this. And I think sometimes that's the miss. It just comes down the bottom line instead of wrapping around where all the value is.

Max:52:20

I know an executive many, many, many years, and I want to say about a decade ago, he was at a company. And I don't want to say like, yeah, like he's like snap. But his reaction was really funny because it was in this normal conversation of like, oh, it's so expensive. And he's like, no, it doesn't cost anything. It's like he's like, you just bought 25 laptops.

Max:52:39

Like, you're expensive. Like it. And and the I think the epiphany for him was really understanding how to shift that mentality around, like, okay, I might run the budget, you know, the rec. I might buy you the purchasing. I might do the, you know, configuration.

Max:52:59

We might do the deployment. But, like, we didn't like like this budget doesn't live with us. You know, like if you're, you know, in revenue or marketing or operations or whatever it actually was and you went out and you hired 200 people like like, you know, I didn't I didn't hire 200 people and buy them equipment. You know, it was it was more. And it kicked off a really big effort on on his part to, I think, get everything off of his line on the spreadsheet and onto other people.

Max:53:32

So that way it was like, you know, and and it ended up working out really well for them because it became that thing in the process of doing that. The I. T. Team got very entrenched in the different business lines. And then the evaluation for the company started becoming we need to do something and how do we actually do that?

Max:53:52

And like, it was in a resource to figure out how to do that. Not like, oh, you're the trolls in the basement that are really expensive. And it really took looking at the business and how they were approaching it very differently. And, you know, and and, you know, it took him kind of having this moment of like really extreme pushback around, you know, like, no, you know, and you said it's like not revenue generating. It's like revenue enablement.

Max:54:19

Right? Like business enablement. You know, I IT is really like business enabling function. You you have to have electricity. All you have to have computers.

Max:54:28

Kids.

Tara:54:29

Right. And they have to run. And if someone can't access, there needs to be someone there to help fix it.

Max:54:35

I I'm so tempted to ask you for a horror story, but I'm I don't I don't know if I wanna go negative. I'm sure there's

Tara:54:41

No. Let's not.

Max:54:44

I'm sure there's more than one. I mean, maybe it's a stat. I don't know if you can you can say this or not, but, I mean, within the tier put tier point footprint, how many disasters or yeah. Let's say, like, how many trigger disasters are you helping with people annually? Right?

Max:54:57

Because, again, perception becomes the disaster is the hurricane that comes through Florida or the disaster is the earthquake that happens in Los Angeles or the or, you know, this or that or the next thing. But, like, disasters happen all the time. And, you know, like, what's what is like, how many of these are you seeing, you know, on on whatever period of time?

Tara:55:17

Well, I don't know a number around that, but I can give you a very recent example. And I don't know if I can use the name, but there was a very recent incident that occurred that took down a lot of folks globally. And so while we don't partner with that company, a lot of our clients do. And so that is a reflection of something that is a disaster, right, because systems were not working. Windows machines were not working.

Tara:55:45

Blue screens all over the place. And so that was a situation where folks maybe didn't need to fail over to a Doctor, but they had a very a very large issue. And so what I can tell you about that is or any of the clients that we provide OS management to, but I think we've received well over 3,000 alerts once everything you know, once that update was pushed out. And they're you know, we don't use the software. We don't partner with folks, but our customers do.

Tara:56:13

And so it became an effort from the tier point side to help our customers recover from backups, you know, reboot their Windows, revert back to an earlier version of Windows, go in and take the file out that was causing the issue. All of those types of remedies that needed to happen, our teams hopped in and supported clients to be able to get back up and running even though it was not caused by us. Then there really is no run book for that kind of scenario. Our teams were there and were able to really support our clients that had, you know, severe issues in a time that was very problematic.

Max:56:50

It's such a good example, actually. I wanna talk about this for a little bit. Okay. So the estimates 8 and a half 1000000 Windows based computers are affected by a software update from another piece of software that ends up with the machines blue screening and requiring manual intervention to recover. You read the news and you dig into this a little bit.

Max:57:11

You can see on an industry basis, some industries really affected, which of course have really big impacts down into like the day to day lives of normal people, you know, and whether that's what pick an industry. And then of course, within those industries, you could see examples of companies that had effective Doctor plans in place and others that did not have effective Doctor plans. And of course, the ones without the effect of Doctor plans are saying, hey, it wasn't our fault. It was these guys fault. And of course, the reaction to that is, is like, well, your competitors were okay because they had a plan in place.

Max:57:43

And it's it's such a good example to talk about because, you know, do you know who to call? Can you get access to the to the. Do you have paper? It's so I laugh about it. I laugh about it because it was personal experience.

Max:58:06

I can I can make fun of myself in this? Right. You know, you don't realize this stuff until it happens to you. And then you're like, oh my goodness. Like, it's really a bad idea.

Tara:58:15

And you're talking to someone that has a lot of paper on her desk because she writes everything down and, you know, I'm a document on I have my to do list over here. So I I I chuckle every time you say that, but because I'm a hold it in your hand, have the paper.

Max:58:30

I've had data loss. You only have data loss once to never ever wanna have data loss again, and then you end up like, oh, we've got, you know, you know, it's like the whole 3 to 1 backup thing. And you find out you're like, oh, no, I'm really like 752. You know, it's like. But what's fascinating to me about this D.

Max:58:46

R. Plan and this conversation with this event specifically was right. It was, do you have backups? Do you have the ability to back up or to recover from those backups? Do you have support in place to help you do that process?

Max:59:01

And who's going to actually do the work? Right. And and how important that part of this process really is. And is that who's going to that work? Is your internal team?

Max:59:13

Is it your partner? Is it to your point? Like, is that documented? Do you know what that is? Do you need to get people in front of computers in all these different locations in order to do that?

Max:59:24

Who are those people? How do you get a hold of them? What do they do when they get there? Right. And, you know, 11 company is claiming $500,000,000 in damages over this event.

Max:59:36

And, you know, and you look at competitors and they, you know, didn't have the same experience because. Right? Some of

Tara:59:44

the competitors are back up and running later that day. And, yeah, I think it also speaks to the kind of partners that you have. So whether you have a partner like Cheerpoint that was, roll up our sleeves and let's you know, we had a plan internally once it started happening and we isolated what the issue was. We had a plan to help these customers, and it may not have been documented. We had it, And I would venture to guess that a lot of folks are documenting, you know, hey.

Tara:01:00:16

If this were to happen, SharePoint manages our operating systems. Our infrastructure runs on them. They need to be the ones to get into the Windows and make these changes for us. So here's the here's the eight hundred number. Here's who we call.

Tara:01:00:30

Here's what the process would be. But I do know that we were highly successful. And, you know, you mentioned base base things that were, impacted. We have hotel clients that couldn't get keys for clients because their systems were down. They couldn't get into their room.

Tara:01:00:48

We all know about the airlines. Right?

Max:01:00:50

I was on vacation. I was in the mountains with family. So it was my family, my wife's sister's family, their parents, you know, like there was there was that there was like 13, 12, 13 of us, like in a plus the dogs. And that's the other part, right? We talk about people again.

Max:01:01:08

I'm not operationally involved with I. T. Infrastructure for, you know, like for that. But I mean, there were many years that I was and, you know, like 12 in the morning, if you if, you know, do you have key personnel that are in the mountains that may or may not have Internet connectivity or may or may not have a cell phone with service or on a plane or on a cruise ship or like whatever these different scenarios are? Have you accounted for that in your plan?

Max:01:01:34

Right. Like it says, you know, tabletop, the tabletop exercises and the tests are so important. They're so important for this stuff. It's so important. And I I kind of feel like next time I'm involved with 1, I wanna, like, at the beginning of the process, like, have somebody on the shoulder and be like, okay, you're dead.

Max:01:01:52

You know, here's you have to wear the dead vest and, like, lay on the floor. You know, like, they're doing, like, you know, the city the city drills. I'd be like, okay. Everybody figure it out. You know?

Tara:01:02:01

How many clients do you go to that they're like, yeah. Tara, she she's she's got this. She knows. Well, what what happens if Tara gets hit by a bus or Tara wins the lottery and decides she's going to an uninhabited island?

Max:01:02:18

I had this conversation yesterday, and the conversation and the IT director, we were we were talking about something and it was the reaction was, oh, it's okay. It's no big deal because all I have to do is do x, y and z and, like, we can fix this. And I and I, I looked at the business owner and I said, okay, make sure that he's never on vacation when you have an outage.

Tara:01:02:35

Exactly. You

Max:01:02:36

know, and that's like the I don't say this to, like, put anybody down. It wasn't like a slight on on their I. T. Director, but it was like, I've been on vacation with outages. You know, I mean, I was at a wedding.

Max:01:02:49

Unfortunately, I was close. I was 90 miles away from the data center and I was at the wedding. I was at the reception. Fortunately, the wedding was over. I was at the reception for the wedding and we had an event and something had happened.

Max:01:03:01

I'm trying to remember the details here. Let's see what I can can and cannot say. Anyways, we had an event. We're trying to coordinate remotely. I'm on a cell phone.

Max:01:03:08

This is before, like, you had tablets and Internet connectivity remotely. Everything's going on. And I ended up just saying, the heck with it. I got in my car and I drove the 90 miles. I'd like, you know, whatever time at night was to get to the data center, do the recovery.

Max:01:03:21

And and it worked because I was within, you know, an hour and a half drive at the right time zone, you know, like right time of day to get help to get to get there. And, you know, like, it's, you know, anyways okay. I'll I'll stop ranting about this stuff. Invest in a plan. If you need help with a plan, there's people that will help you with the plan.

Max:01:03:43

Have the plan on paper. Keep include people in the plan.

Tara:01:03:47

There's nothing else that we convey in this.

Max:01:03:50

They have a plan, have it on paper, include people you have ever done. 6. Oh, man. Don't forget that if you have computers in the desert somewhere that are vital to your you know, you have to be able to get somebody out there to touch them at some point. It's so it's such a big deal.

Max:01:04:05

Tara, thank you very much. I never cease to amaze myself with how much of a nerd I am about wanting to talk about these things. Like

Tara:01:04:12

you and me both. Oh,

Max:01:04:16

you know, I'm hoping that's, you know, it's sad. I mean, again, I don't know what to do about this perception of risk, but, like, you know, global event, everybody now knows, like, bad things can happen to their computers, you know, and like how many companies are going to have to learn that lesson the hard way themselves because they didn't spend the time to put a plan in place and, you know, test it and make sure their people know where it is and have it written down.

Tara:01:04:43

Yep. Well, hopefully, they will now.

Max:01:04:45

Hopefully, they will. Alright. Thanks, Tara.

Tara:01:04:47

Thank you. Have a good day.

‍