Disaster Recovery for Finance: UbiStor’s Secure Solutions with VP of Channel Solutions Keith Lukes | Ep. 36

Max:00:08

‍Hi. I'm Max Clark, founder of IT Broker dotcom, and this is a tech deep dive on backup, disaster recovery and business continuity. Oh, my. Keith Luke's at U Restore was kind enough to share a story and modern reality of what is going on in backup disaster recovery and business continuity and what this actually means. We get into it.

‍Max:00:24

We talk about it. We go through a real world example of a multiregional, multinational as a company, problems that they were having, what you be sort of to help them, what you should be thinking about if you're going through this, and how this all wraps together and actually impacts with a very unsexy thing for more people to talk about in the back up world. You know, let's go spend money on backups. But why it's so critical for you as an organization to have good backups and a good plan around your backups. Hope you enjoy it.

Max:00:50

This is Keith Luke's with u v store. So, Keith, thank you for joining. We're here to talk about customer of yours and how you guys helped. Give me a little bit of background, company profile, how you guys got involved, and and what was the problem that they were facing.

Keith:01:04

Yeah. For sure. So we got involved with them almost 3 years ago now. They're a global financial advisory company. They're actually publicly traded.

Keith:01:11

They're on the, S and P 500. But they had some challenges around compliance that they had for several of their locations. They needed their Zurich office to keep data within Switzerland. They wanted to keep their Singapore data in Singapore, but they also had 17 global data centers that they needed to protect. So they were looking for options to be able to manage to that locality and meet those compliance requirements, but also have a partner that could also protect regionally for them as well.

Max:01:37

So in this case, the problem is data sovereignty based on different rules, different laws in different countries and having them be able to meet that data sovereignty requirement and compliance within those facilities. Was this all their own data center assets, or were they cloud as well? Or No.

Keith:01:52

So this is all their primary infrastructure. So they operate with infrastructure at each of their locations. So, you know, while we're protecting close to 2,000 VMs globally for them, they actually operate from an IT perspective as, you know, 17 different data centers regionally that we protect for them on an on a 1 on 1 basis. So they're using their own infrastructure. They're managing that infrastructure.

Keith:02:14

They've got a whole operations team and an IT organization dedicated to that management of that compute and the resources and storage there.

Max:02:21

17 data centers, 3 major regions, thousands of VMs. This is a sophisticated IT program. Why not do it yourself? Like, if you've got that much infrastructure, like, this is not a small like, an SMB with a couple of IT teams that don't you know, I mean, they know how to manage VMs. They know how to manage data centers.

Keith:02:37

Well, really, it comes down to that's not their business. So they were looking for a partner that, you know, focuses on that technology. They wanted a combination of backup and and disaster recovery. And really, that's core to what we do as a business. Their organization and their IT team is really focused on operating the business.

Keith:02:55

And so for them, having a partner to come in and actually manage this and and and have the infrastructure that they could leverage instead of having to have the additional expense of managing that themselves and had some elasticity and flexibility to it really was what attracted them to us, and that they didn't wanna be in the backup and Doctor game. They wanted to have a partner to be able to leverage in conjunction with that.

Max:03:15

So what changed for them 3 years ago? I mean, that's, 2021, roughly. So 2020, 2021. I mean, data sovereignty was already an issue. Yep.

Max:03:24

GPR, things like that were already issues.

Keith:03:26

Yeah. And that's what what was happening is that the datasets were growing, and they really needed a strategy to get that data off-site and have a infrastructure in place to be able to recover to that was external to their business. They really didn't wanna operate internally. And so they were doing some things natively with their data center replication that just wasn't becoming efficient or cost effective for them. So they really wanted to get out of that space.

Keith:03:48

And it it was their data sets and their sizes are growing. And and just in the last 3 years, we've added, and we can talk to that a little bit more too, is is how they've evolved as a business. They've gone through an acquisition. They've added Brazil data centers. They've added data centers in the US as well.

Keith:04:04

So that growth has expanded, and that's really what they wanted to be able to do with us is they knew they had a roadmap for acquisition and expansion, and they wanted to be able to have that flexibility as they grew so they didn't have to worry about that infrastructure on their own.

Max:04:17

Larger companies usually have dedicated compliance functions. They have a GRC function. There's, you know, legal that comes into this. How much of this was driven external to the IT organization? Was this the IT organization just looking at this roadmap and their growth and requirements were coming to them and saying we need to make a change?

Max:04:32

Or was this triggered with a conversation of, you know, a compliance officer, for lack of a better word, saying we have to like, oh, look, we have this regulation that we have to adhere to now. Like, are we doing this? And what's our plan to do this?

Keith:04:45

It was a little bit of both. I think the the IT organization is sophisticated enough to know that that was coming. The initial driver was specifically around Singapore and Zurich. I mean, they have, you know, Asia Pac data centers. They have data centers in the EU.

Keith:04:58

They knew that they they wanted to keep that data, but there were specific requirements that they needed to check that were part of that. And as they were looking at the the relationship, part of it was being able to evolve with that. But I think it was it was being driven collectively in within the organization, both from the security and operations team as well as from the IT team knowing that this is road map and the requirements that they're going to be tasked with as they grew the business.

Max:05:22

Okay. So 17 data centers, Zurich and Singapore being primary drivers, lots and lots of VMs, lots of infrastructure, relatively large team or, you know, s and p 500. You know? You're talking about, like, you know, this is a large enterprise. Right?

Max:05:36

So they have we have resources. You used a phrase earlier. It's, you know, not their business. Can you expand on that for me? Let's let's talk about that a little bit.

Keith:05:44

Yeah. What we see is in the backup and Doctor space is that IT teams are stretched. Even a global organization like this, their team is is still stretched thin. They have regional operations teams for, sort of each organizational unit and also in inside of regions. So there's, like, an Asia Pac team.

Keith:06:01

There's a EU team. They've got support resources in the other regional offices. But they're stretched in just to operate the business. And so for them, and and where Yupester typically sits for most of the you know, for the customers we work with is really around taking the day to day operational need and the the the task and the lift of supporting backup in Doctor. You know, having a customer like them, while they do this as part of their business to keep it running, we're doing it every day, and we're doing it in multiple environments.

Keith:06:29

So we have a deeper understanding of the technology. The technology we use, we're their largest global Doctor reseller. So we have relationships into the organization, even for a company this size that they don't have on their own. So for us, we can operate in a much different manner than they can natively on their own. And that's where I think the differentiation comes in that all we do is back up in Doctor.

Keith:06:49

It's our priority. We wake up checking back up jobs and restoring data and helping customers, that's not what they do on a day to day basis until they need to do something relative to the backup operations. So I think that's the big differentiator and and where we're seeing more organizations come to us. We're talking, you know, about obviously a large global enterprise, but even in the mid to small enterprise space, you know, these guys are just so focused on what they're doing for the business that they tend to ignore. Backup and Doctor usually isn't a priority until you have some kind of event that needs recovery.

Keith:07:21

And that's we make it the priority. And I think that's the big differentiation we see, when it comes to doing this on your own or using a third you know, a MSP like Ubistore to be able to do that.

Max:07:31

It's actually it's a it's a really interesting phrase to use there of, like, it not being the priority until until something happens, and then it is the priority. Right? And that's probably a pretty common, like, vein of, like, oh, you know, we thought we had a backup. Yep. But we turns out we don't have a backup.

Max:07:46

And, you know, if I had a dollar for every time I've had that conversation.

Keith:07:51

Well and and that's the truth. I think what we see is, you know, one of the things we do as part of our backup services is monitor the backup jobs. And most of our companies that are running with our we call it our advanced services that have us monitoring the daily job operations are running at a 100% backup success rates, or near 100%. And so we talk about not being able to recover the data you don't protect. And so if you're not minding that store, if you're not keeping, you know, attentive to that, you can have back in your protection.

Keith:08:20

And that's what we try to avoid. Because when you go to press recover, you've gotta make sure that that data is there. Right? Before I got into the channel and and presales, I was a system admin myself for a a a Fortune, 100 company, a global system admin. And I know it.

Keith:08:35

Backups, it's the least favorite job of the IT team. You know, you're fighting fires all day. So Yeah. You see that failed backup, and you're like, oh, yeah. I gotta check that out.

Keith:08:43

Then you get busy with something else. Happens the second day, and you're like, oh, man. I gotta remember to go back and check that. And all of a sudden, you're 5, 6, a week later, and then that department calls and says, hey. Our SQL database is corrupt, and I need that SQL database from that backup.

Keith:08:57

And you're going, okay. I've lost 7 days' worth of data. And that's a huge and and it sounds so simple, but it's that's that's really where it starts, is that we start with the backup piece of it, right, and make sure that the data's there. And then we can facilitate everything off of that.

Max:09:11

I'm reminded of a of a of a situation I had. I mean, this 25 years ago now, and I was a young whippersnapper I. T. Consultant, you know, running around doing field engineering for mid market companies. And, we had this one that was in manufacturing.

Max:09:26

And I love manufacturing. But like when you walk into a plant, and they're, like, literally just taking titanium rods in off of semi trucks and, like, just putting them through, you know, CNC machines. And like you get this thing at the end of it. It's just like, it's fascinating. Like, they actually should have a YouTube channel, which is all they do is just show this stuff because it would be hugely popular.

Max:09:41

But anyways, they had an event, and they lost their ERP system. And they didn't have a backup of the ERP system. And that's how I got involved in this thing of of like and, you know, very early on into my career, I remember this vividly, like, you know, building out new infrastructure for them because they had lost their infrastructure. That's how we got caught up, you know, got the call. And they ended up hiring from a staffing agency specifically for accounting temps.

Max:10:02

And they had people on the conference rooms just going through paper and doing manual data entry of their POs, of their invoices, of their receivables, of everything just to recreate. And I couldn't tell you how far back they went, you know, if this was I try to recover 1 year or 2 years or where, you know, what the thing was. But just witnessing that, like, the actual, you know, okay, data was lost, and that's like a conceptual idea. But now you have 10 people in a conference room with file boxes.

Keith:10:27

Right.

Max:10:27

And they are typing on keyboards for weeks. And, you know, and that visual was was something else. I'm also old enough that I can remember being very excited when I, was working for a company. We bought an auto loader. And, you know, and we had a robot doing our backups, but then you had to mind the backups and change the tapes and put the tapes and faults and and and shift off.

Max:10:45

Anyways, I should talk about that another time. That should that's a probably an interesting segue here. So, you know, now we talk about, you you know, you use, like, VMs, you know, as this, like, primary idea and concept. What data were were they looking to protect? How was it actually organized across their regions, their data centers?

Max:11:01

When you guys, like, walk me through this process from an initial engagement, I hate, you know, like discovery. You know, you get a phone call like, hey, we need we need backup. Like, what does it actually look like?

Keith:11:11

Well, really, there were some complexities in that they had some Citrix infrastructure that they had. So they were using workstation desktops you know, virtual desktops for their end users that they had to be able to have recoverable that needed to sit adjacent to these applications that they were running. So it was really a combination of application data. Each location sort of has their own IT. That's where, you know, again, operating as a a global organization with 17 data centers.

Keith:11:36

It's interesting that they have a lot of regional and segmented applications that are specific to each of the offices and what they do and what they do in region. So a lot of it was, hey. Yes. We're you know, most of the infrastructure is running virtually on Windows. So really, it it came down to determining what applications they have there and then helping them with prioritization and and backup and recovery.

Keith:11:58

But really, it's an individual you know, that that's sort of they're they're interesting in that they are global in nature, but they operate very much like a bunch of small businesses or midsize organizations inside of the the operation. Right? Because these are independent entities working individually somewhat with overlays of IT organization on top of it. A lot of the conversation was, hey. We might have some weird thing over here.

Keith:12:21

Can you help us with that? You know, we have this one application that runs out of our Mumbai data center that that we need to make sure we can replicate and and protect. And so those are conversations we had just kind of facilitating what needs to be protected and how it needs to be recovered. So it was a little bit different in that it's it's not this whole kind of holistic mammoth single centralized data center supporting their thousands of users. It's very much segmented into Fother's 2,000 VMs inside the organization.

Keith:12:49

That is broken up, you know, across 17 data centers and makes it a little bit different. They have some primary, obviously, some larger ones that do, you know, Singapore is where their corporate offices are. So there's a lot more gravity in that data center. You know, for the most part, that is then distributed among the the rest of the locations.

Max:13:03

I want to talk about so much stuff here. I have so many questions I want to ask you. From a, from a presales, from a scoping, from a contracting standpoint, this sounds like a nightmare. You know, like, I'm thinking about, like, organizationally wise. And correct me if I'm wrong.

Max:13:14

Right? But, like, the primitives in terms of cost for backup and Doctor. Right? We're we're really focused on what's being backed up. Like, what's the actual unit and then how big is it?

Max:13:23

Right? So, like, you know, VM application server, you know, what is the actual thing mechanism? What tool is being used? And then how much data is is going across. But so now you take that with this environment, you say, okay, not only do we have to figure out how to scope how much data, what applications are, what tool gets used for it, but now we have to figure out there's there's 17 data centers with lots of different teams, each person having their own little fee dump.

Max:13:46

Yep. And all this information has to funnel up because you're not doing 17 different deals, I assume. Right? Like like No. That was

Keith:13:52

yeah. It's and and you're touching it. This was the other key piece for them. It's actually they knew they were going through growth mode. They wanted some easy billing.

Keith:14:00

They wanted some flexibility on how that would potentially work. One of the nice things about the technology we use, for the backups is that it's all SKU based. So we are billing them at a top level, the overall quantities of everything being used. And it just to your point, exactly how it's laid out is number of VMs, how much data is being stored in the cloud. It's you know, we're charging per VM, We're charging per gigabyte.

Keith:14:25

We're rolling that up into a single invoice to them across all of the different locations. But then we're reporting to them individually based on consumption on each data center or each organization that we can break out. So part of our design phase also included how these would be structured to make sure we can invoice in a way that made sense for them.

Max:14:45

How's it take you to go I mean, like, in this situation, like, that's not like one phone call. You know? There's a lot of stuff that has to come out in order for you to

Keith:14:51

I mean, you know, this was probably a 7 to 9 month sales cycle. Lots of presales calls. This included bringing in experts and support team from the you know, we use in this case, we're using Acronis for that technology. We brought, several resources in across the entire organization to help make sure we're architecting this correctly, include the, you know, discussions around billing, invoicing, make sure we could align with what they wanted both structurally from an organization perspective on the IT side, but also make sure it was aligned with their business operations and we could invoice for them. You know?

Keith:15:27

Are we billing in US dollars? Can we bill I mean, like, questions like that come up. That was an important one for us too is because, you know, we didn't wanna bill in euros for the EU organization and and, you know Mhmm. Asia and Australian dollars and and and US dollars. And so it was kind of a compromise in that if we could roll it up into a single invoice across all the organizations, but then allow them to see the usage and consumption inside of each organization, that way they could then bill back and and understand where the costs are.

Keith:15:55

That gave us the flexibility that we wanted and then really helped us. That was part of the, you know, part of what they were looking to is is is make it easy for me to consume your service. Right?

Max:16:05

Right.

Keith:16:05

That's the other part. And so that was that was definitely a huge part of of the discussion we had with them is is also how is this gonna be built? You know, what quantities do we do? Like, we started off at at x knowing they were going to go to y. And now we're at z.

Keith:16:19

They merged. You know, we've got we've brought on a couple more data centers. So, I mean, you know, this started off initially, we were looking probably at 800 to 900 VMs, and that's exploded to almost 2,000 in the 3 years we've been working with them.

Max:16:31

Enterprise purchasing cycles, if you haven't been involved with it, I mean, it's it's really something. I mean and little things that, like, can you give me a single bill? But then can you break down each business unit for me underneath that single bill? So that way I can then figure out internally how we're allocating costs. Yep.

Max:16:45

That's a really big deal.

Keith:16:46

Yeah. And I think part of that comes from, too, like, we operate as an MSP. Right? So we're naturally designed to have subunits and and structure like that because we're getting one bill from our partner that we then are dividing up across multiple customers as well. So a lot of the enterprise organizations that we work with, that's part of the requirement.

Keith:17:05

How do we invoice? How do we break that up? How do we ensure and understand the consumption based on each entity so we can bill back or charge back? Because that's a lot of the the the piece of the puzzle that we see. Again, there's not just a a technical requirement.

Keith:17:18

A lot of times there's an operational requirement too that you have to be delivered on when when you get an enterprise. And and so that is a much different like you said, they're a different animal to work with when you when you start operating with an enterprise like that that has multiple divisions, global in size. So I think that is a really key point that it's not just having the technology solution for them. It's really a whole, how do we integrate the 2 companies together to make sure you get the delivery and the the the solution in a way that makes sense for you as a business?

Max:17:46

Were they already Acronis users? Or was that something locations. Yeah.

Keith:17:49

They were some of their locations were Acronis users. They had some familiarity. I mean, part of I think that that alignment also helped us in that Acronis' global data, their global headquarters are in Zurich and in Singapore as well. So there's some alignment and familiarity with them. I mean, that helped us for sure.

Keith:18:06

And they had some they were using it in some of the data centers. That was it too. They were disparate technologies at all these different data centers for backup and Doctor. So part of what they wanted to do was collapse it under one single, you know, solution set for the global organization. And that's the other big key is that there was some displacement of of older technologies or different technologies because they were doing different things in different regions, like you said.

Keith:18:30

You know? Because we talked, they these were operating kind of individually, and they wanted a little bit more standardization across the entire enterprise.

Max:18:37

Ubistore is is relatively agnostic to the actual tool that's being used here. I mean, you've got Acronis. You've got lots of other tools. Can you can you give me a list real quick?

Keith:18:46

Yeah. So we work with, Commvault, Azureto, Druva, Acronis, and Datto, and Kaseya. We also have some integrations with scale computing around the pieces of of Akronis' backup and Doctor, and we do some primary compute for them too. But, yeah, those are really the the core pieces of technology we use to deliver.

Max:19:03

How much of that, like, that selection is is dictated by what's already there? Like, you're you have a brownfield, so to speak, of, you know, hey. We're already using data. We're already using a Kronos. Like

Keith:19:14

It can depend. It depends. Like, because some of the technologies we can overlay just to manage service, like Druva, Com Vault, even Datto and Acronis, depending on the version of Acronis they're using, we can come in and just bring our managed services in on top of that. A lot of times what we do is is it's a culmination of where they're at with their technology. It might be where they're at in their purchase cycle, where they're at with their licensing.

Keith:19:34

And we can look at also combining the software cost with our managed services. And a lot of time that becomes because of the volume we're at or the partnership level we have, we can give them a cost reduction and add our managed services in conjunction with that. But the technology really the reason we have this suite of services is really a lot of it is trying to solve their business problem. So we're not trying to jam our, you know, square peg into their round hole. Right?

Keith:19:58

It's really about what's your business need and what are you trying to accomplish, and what are your, you know, recover point, recover times, your compliance requirements, and then using the technologies we have in our portfolio to be able to deliver something that makes sense for them. And we're even working with customers now too who may have been working on one technology that made sense for them for 3 or 4 years, but now they're changing their operations. And so we're changing them to one of our other technologies to be able to help meet them where they're at and help them with their new workload. You know, we may be using, you know, like, I'll use Commvault as an example. They have a very extensive portfolio set around their agents and cloud capabilities, you know, protecting Azure entry ID and RDS and headless SQL, right, and Azure SQL and these agent these kind of OS, no OS systems that now reside in the cloud.

Keith:20:44

And so you've gotta protect that. You know, what was working great on your VMware environment sitting in your data center protecting Windows isn't maybe the same technology that I can use for that. And so What are what

Max:20:54

are you talking about here? You gotta back up your cloud?

Keith:20:56

Yeah. Exactly. Well, and like Microsoft 365 protection and compliance search and all you know, Google Workspace. And so, again, it it for us, it's really a a culmination of technologies that we've evolved. A lot of what we've built, you know, foundationally was built on the Commvault architecture because we've been working with them the longest.

Keith:21:13

We were one of their early MSPs. MSPs. And so the evolution is and and and really, if you look at Dan and Steve who, founded Ubistore, they've got Doctor experience going back years and really around the enterprise space. So part of what we do is we can manage to a customer like we're talking about today, but we also bring a technology set and expertise where we can take enterprise experience and drive that into the the small to midsize customer even. Because we you know, while we're talking about, a large customer today, we have customers of all sizes in our

Max:21:43

Okay. We need to talk about this more, though.

Keith:21:44

Azure, AWS, Google Workspace, 365.

Max:21:45

Right? So there is a perception that this is just done for you. The Google Workspace is backing it up. The drive is being backed up. That one, you know, SharePoint's being backed up.

Max:21:56

But you have a business practice around backing these things up. What's the disconnect here?

Keith:22:00

Well, there's you know, I look at it, and and the conversations we have around that is is Azure, AWS, Microsoft 365, Google Workspace. Right? Those are infrastructure platforms. They're designed to be highly available to deliver that service to you. You know, you were an infrastructure admin too.

Keith:22:16

Right? I was an exchange admin. Right? Like Oh, man.

Max:22:18

I started with that.

Keith:22:19

Yeah. So, I mean, everything all M365 is is an exchange platform in the cloud for you to leverage. But it's just the infrastructure. And they have some native pieces. They protect the data for 90 days.

Keith:22:30

They're adding some, obviously, some different backup capabilities now too. But but, ultimately, that your data and your infrastructure are all sitting in the same spot. And if you don't have that externally backed up, if something happens, you could be at risk. And I think the other thing is is, you know, a lot of the the ransomware attack vectors are around getting into OneDrive or SharePoint and deleting that data. Right?

Keith:22:51

Like, that's part of their process. And so if you don't have that external, trying to get that data back working with Microsoft as a single entity, you know, 1,000 user company, your support model is not they're not jumping through hoops to help you. Right? Mhmm. And that's part of it too is is having a partner that's protecting that that we can add the compliance pieces to it as well.

Keith:23:11

Because that's the other thing. You know, if you're a you know, you've got compliance concerns. If you're HIPAA or financial institution, you have to retain data for 7 years or 10 years, that applies to your backup data around your mail platform. So email, file level backups, all those things that go into Microsoft 365 protection. If you're not doing that, you're not meeting your compliance concern.

Keith:23:33

Right? And we have things too around ediscovery and search. And so the tools we use can help do higher level support and protection than what we see in there. Same thing with, like, you know, SQL infrastructure or any kind of workload you put in there. You can build backup jobs.

Keith:23:48

You can use Azure snapshotting, and you can use all the different tools inside of there, but we find that they're very expensive. They don't offer all the bells and whistles that you need sometimes from a recovery perspective or compliance perspective either, from reporting and all of the information that you wanna collect, customization of, you know, the way you're backing that up. It becomes difficult to manage to that collectively. So I think, you know, having a a tool set external to that really lends itself to giving you that sort of we talk about, like, 3, 2, 1 backup methodology. Right?

Keith:24:16

And and and having the data external from where the primary infrastructure is. Like, it's just part of a good backup strategy and and foundational to what we do. And so that's all part of the conversation you have. And so, yep, and I I see that all the time. It's like, hey, like, well, we've got it in, you know, we've gone to Microsoft 365 and I moved all my, you know, all my file data into OneDrive and SharePoint.

Keith:24:38

Okay? Like, Max leaves the company and deletes all that on his way out and you don't notice it for 90 days, then what do you do? You know, like, it's things like that that I think aren't protected natively. Or even on the way like, Max deletes that data and you notice it 5 days later and you wanna get all that back quickly. Well, you're logging in to Microsoft.

Keith:24:55

How do you you know, in the console, maybe something happened. And so I think having a quick, you know, like a partner like uStore, we can get that data back for you much more quickly than what we see in in the support models that we see from Microsoft natively.

Max:25:09

Cloud based platforms view delete as a valid operation. So if you're doing a select star delete out of your email or select star delete out of file system, and that's a valid operation going against that cloud platform. Yep. And then what do you do about that if that actually did happen and it was whoops whoops or if it was encrypted? Is you you said 321.

Max:25:28

Is 321 still a valid methodology in today's world? Right? Because first off, what's 321? I'm not gonna say it. I'm gonna make you say it.

Max:25:34

And then is 321 still, like, the the a valid best practice for for people to be, you know, striving for?

Keith:25:40

I think I still think so foundational. You know, that's 3 copies of your data, 2 different locations, one off-site. Right? So you have your primary copy. Typically, you have your second copy there adjacent to it locally, which we recommend on a different platform.

Keith:25:54

Right? So a different target. External, so you're not backing up your data to the same storage that you're running your primary compute on. And then you want that copy off-site in some other format. Used to be tape.

Keith:26:06

You know, now it's obviously cloud being what it is. It's, you know, most of what we're doing is cloud based and and eliminating tape. But I think it is still very valid. A lot of what we talk about in that 3, 2, 1 methodology is actually not only having that data off-site, but having it adjacent to compute resources built for recovery, which is now very cost effective as well. So that's adding another layer of protection to your sort of business continuity disaster recovery plan.

Keith:26:32

Right? So it's easy to have that, those 2 copies. And I'm still shocked at how many companies I'm still talking to that have, like, one copy adjacent and they're not moving things off-site. You know, you want that first copy local for quick recovery. You know, ease of use, simple sort of activities like a VM that doesn't come up or somebody deleted file folder off of the the file share that's local, that's not in like Microsoft 365.

Keith:26:56

That's easy. Once you get it off-site and protected off-site, that's really where we add that layer of compute next to it. So all the technologies we use from a backup perspective are aligned to have compute resources adjacent to them. So that way, I can take those backups if you lose a data center from a some kind of event, whether it's, you know, obviously natural disaster.

Max:27:15

I feel like we've been talking about around this a little bit. Backup, business continuity, and disaster recovery are different things. They get squished together into one thing, but they're really 3, like, uniquely different things with different objectives and and different tech you know, like, talk me through this.

Keith:27:34

Yeah. For sure. And so I think, you know, we look at backup as, like, a lot of times, it's it's protecting the data, you know, data protection. A lot of the conversation we're having around backup now is also about immutability because a lot of the attack vectors on the ransomware side are really looking at, you know, credential hijack. And part of that credential hijacking is also getting control of the the backup environment.

Keith:27:57

So they're going in and deleting your backups before they encrypt all your data. So now you're left holding a bag of nothing. So a lot of the controls we have around backup when we look at that is encryption and immutability just on the core data. So so we see that as part of backup. The other thing we see a lot of is archive and compliance.

Keith:28:16

So long term retention, data recoverability. So if you have to retain data for 7 years, where is that data stored? And that's really backup, right, and and searchability. And and and we've seen that already with customers who, you know, get an audit and they want I wanna see your files from 6 years ago from December of, you know, 20, you know, 2017. Right?

Keith:28:35

So they have to pull that data back. That's backup. You know, disaster recovery moves into the applications and environment that you run. Right? What is running the business, and how do I recover those applications and the correlated data to keep the business operating?

Keith:28:50

So that's the disaster recovery component. And then we look at business continuity as, okay, where do the employees gonna go? How are they going to connect to that information? And where are they going to work depending on what happens to the facility? And so part of our discovery around disaster recovery is that business continuity component and talking to the customers.

Keith:29:10

Okay. What kind of VPN access do you have? What kind of connectivity do you have? Where are those end users going to, you know, your customers or your end users? How are they gonna work with this new environment we've created because you've had a disaster in your primary location?

Keith:29:25

And you were talking about manufacturing too. Like, you know, every business has a different orientation for that. Like, you know, we'll talk to guys. It's like, well, how can if our manufacturing facility is down, we're dead in the water anyway. Like, there's so much operationally that we've lost that it's like, I just need a couple users to be able to handle inventory and print invoices and, like, manage the business.

Keith:29:44

Like, we're going to be so far behind that. Like, I don't need my ERP system for a lot of because my manufacturing is down. That's where business continuity ties into the operational delivery of the technology in a disaster. And that's kind of the 3 phases we look at it in that you have to make decisions around backup and archive and long term retention and how you protect that data. You protect that data.

Keith:30:05

We have to make decisions on how you wanna recover, how quickly you need to get the business back up, how much data you can lose in that transition process, talking about recover point, recover time objectives. And then, like I said, it's that once it's up, who's using it and how are they accessing it and what kind of technologies do they need or what kind of facilities do you need as a business to be able to do that? And that's kind of where we fall off is is, you know, from a business continuity perspective, we're not doing like, you know, setting up users in new offices. But I think a lot of what happened too, obviously, with work from home and remote work, we've gone through that phase. So I don't that's like that used to be a big hurdle.

Keith:30:43

Like, what are my end users gonna do when the this, you know, the office is down? Well, we figured that out. I mean, the the nation you know, the world had to figure that out in 2020. So that's not the problem anymore. It's like, how do I get connected to it?

Keith:30:55

And I think that's facilitated and accelerated a lot of different, like, Doctor strategies for customers and that they've learned that their workload doesn't need to sit adjacent to their end user.

Max:31:05

RTO and RPO. These are I mean, let's just call them what they are. These are backup nerd terms.

Keith:31:11

Yep.

Max:31:11

Right? But massive implications based on decision around RTO and RPO. Now, you know, Steve Jobs is famous for a quote of, you know, if you ask people what they want, they'll just tell you, I want a bigger phone with a bigger screen with more battery and a faster processor. Right? If you ask somebody what they want in backup, right, they're gonna tell you they want everything backed up all the time with instant recovery.

Keith:31:31

Yep.

Max:31:31

With RTO and RPO, can you talk a little bit about the difference between RTO and RPO? Right? Because that is there's a big difference. And then how crafting and designing that scale of, like, where you're going up, like, influences so many other things within this?

Keith:31:47

Yep. It all goes back to cost versus risk is how we look at that. That's really the decisions you start making on RPO and RTO. And so recover RPO. We've got your recover point objective.

Keith:31:57

How far back how much data are you willing to lose as a business when you recover in a disaster? So is are you a transactional company, or can you you know, we we were working with an energy company. Like, they basically had zero downtime requirements because they had to keep all that information up and running. And then you have your recovery time.

Max:32:15

So RPO really is two things. Right? RPO is both, like, how frequent are your backups? Someone was your last backup. So if you're doing, you know, one backup per night, in theory, your maximum exposure could be, you know, 23 hours, 59 minutes, and whatever seconds or whatever time.

Max:32:31

But it also RPO could also imply how many versions of, you know, back in time. Are you keeping 1 week, 1 month, 1 year? You know?

Keith:32:40

That is right. And that we talk about that too. That goes back to sort of the backup methodology and compliance. Like, how much how many recover points do you need? How frequent is it?

Keith:32:49

That's both for your compliance recovery because that's a recovery event. That's somebody auditing you saying, I need that data back. How long are you keeping and making those recover points available? And then you have recover time, like you said, like the how quick can I get that system back up and running? How fast does it need to be available?

Keith:33:05

And to your point, we get this all the time. It's like, well, you know, we need everything up within, you know, 1 hour. Okay? And we'll put together a methodology and a price for that. And they're like, oh, wow.

Keith:33:15

You know, a lot of sticker shock. It's like that's and we have a graphic that we use where it's like, as you get close to that point of disaster, cost goes up. And it's finding the balance. And that's where we do again, that goes back to the different technologies we have because you may need a tiered approach for your recovery. You know, what we find is a lot of times is there's a company may have a 100 VMs in the environment.

Keith:33:35

Well, at the end of the day, probably 20 of them are really driving the operation. So we'll have a different protection methodology for that subset of data and those systems relative to the other pieces. You know, we may have 10 systems that are we're replicating in near real time that fail over almost instantly, and we can support that. The rest of the business, well, then you start making decisions as an organization depending on the extent of the outage. Is it a power related issue?

Keith:34:00

Okay. My offices are gonna be down for 72 hours, or, like, fiber cut, things like connectivity that we've seen. I just need this subset up. We're not gonna go through a full disaster recovery plan because in 72 hours, I'm gonna be back up and running. I don't wanna have to bring those systems up because the other part of that process is at some point, you're gonna have to fill it back.

Keith:34:19

So you do have 2 stages in a disaster that you have to that you have to, like, mitigate to. Right? And so we do a lot of work with our customers on helping them. We do, like, a full assessment of their environment. What systems are there?

Keith:34:31

What's the criticality of it? What kind of interdependencies do they have? And then help them build a strategy. Because a lot of times, they don't know. Like you said, they go right at it.

Keith:34:38

It's like, I need it right away, instant. Everything's gotta come up right away. And that's just not typically realistic for most business operations.

Max:34:46

These are questions that probably people haven't ever really thought about beforehand of, like, how many VMs do I actually have? What's running on it? What's the dependency from one to the other? And how do these things actually RTO, also, I'm gonna use the term cloud in the sense of your infrastructure is physically located separate from where your customer's infrastructure is. AK, right, on the the cloud as everybody wants to call it nowadays.

Max:35:06

Right? But but, you know, there is a there is a factor of RTO of just, like, how fast is your customer's Internet connection? Yep. Or you know, like, if you wanna recover, you know, a petabyte of data over the Internet, like, that's do you have 10 gig connections? Do you have 100 gig connections?

Max:35:21

Yeah. You know, like, how fast is your disk? Can your disk you know, sync this data fast?

Keith:35:25

Right. There there's physics involved. Like, we talk about that all the time. Like, there's only so much data you can push through. Is your recovery multithreaded?

Keith:35:32

Right? And that's where, you know, a lot of what we do is, especially in the replication technology, is that it's that initial seed that's usually the problem. Right? Like, we have to get that data up there. Once it's there, then it's usually just block level changes or incremental changes that are being sent across the wire, and you can be pretty efficient with that.

Keith:35:47

But that's where if you have you know, we're working with a customer right now that has a large SQL infrastructure, and that needs to come up right away, and that has to be the first thing. Well Mhmm. You can't take a 100 terabyte SQL database and recover that from disk in an hour. Like, it it's just the physics don't support it. So you have to then look at a replication technology.

Keith:36:06

Or if don't wanna pay for that replication technology, your business is going to have to tolerate a 100 terabytes being egressed off of your backup data into that new infrastructure. You know, there's several different ways to look at that in you know, that's that cost risk analysis. It's the how much am I willing to tolerate and lose, you know, my cost loss as opposed to the protection I'm putting in place and what that cost my business to have it available in that time frame, if that's what I really

Max:36:32

need. Unfortunately, reality for a lot is until they have that first event, you don't really understand what that cost risk relationship is for you. Right. You know, because maybe you think that the cost wasn't worth it to you and you could tolerate the risk and then you find out that you don't want to do that again in the future.

Keith:36:47

Yeah. You don't want, you know, hundreds of employees sitting around doing nothing. Like all of a sudden you look at that and customers not being, you know, no interaction with your customer. And so I think that's part of what we do, I think, is and part of our assessment and design services on our backup and Doctor side is also talking to, you know, other people outside of IT in the organization. Like, making sure like, because sometimes what IT expects or what they want doesn't align with what business expectations are either.

Keith:37:15

So getting the, you know, the key stakeholders in the organization on the same page. Like, this is what's happening. This is how we're architected. And if it doesn't perform in the way they want, then we're supporting IT and saying, we can do that, mister CEO. Like, I can get you, you know, 1 hour RPOs and RTOs.

Keith:37:31

But here's your cost delta to the business. When we have because of the size and the the volume that we've done, you know, we bring that that goes back to why are we different than an internal IT doing it? Well, because we're doing it all the time. I mean, we're doing test events basically weekly. And most organizations, if they're lucky, are doing it one time a year.

Keith:37:50

And so it's not only that, building that plan and that strategy, you know, documenting that run book, making sure they have documented processes and how the systems will be recovered, and then making sure they test. And that's another big, huge piece of this is that when you define those RPOs and RTOs, are you testing to make sure you can hit them in an actual event? Because that's part of it too.

Max:38:11

Let's talk about testing for a little bit. Right? And this comes into the whole we thought we had a backup, but we actually didn't have a backup. You know? And and the same thing for disaster recovery and the same thing for business continuity.

Max:38:20

Right? You learn a lot of lessons when you actually try to recover data. If you're doing weekly tests, I mean, this you guys can you can test your infrastructure. How do you engage and involve your customer in this test process and have, you know, them at you know, I wanna say adhere to your schedule of, like, we have to actually test this. Like, you can say, like, oh, we have the backup, and we're able to restore it.

Max:38:40

Then the system said it restored a 100%. Right? But then somebody has to, like, turn the application on and say, did the application actually work?

Keith:38:46

Yes. We have a set of we call them our advanced Doctor services, and it's a culmination of of several different components like I mentioned. We're monitoring the backups. We're assessing the design and putting a design together. We're building the runbook with them collectively, and then we assign a technical account manager to the account as well.

Keith:39:04

So we're constantly engaged with them either quarterly or monthly to review the environment and make sure that everything's being protected. And then scheduling, it's all about that schedule and saying and an understanding we can do full scale testing. And sometimes they're not operational ready to even handle that on their own. Like, because we are pulling resources away from their team to be able to do that. And some customers have different regimented schedules.

Keith:39:27

Some are really good at it. Some are like, we wanna test quarterly, and we we wanna test a subset of systems every quarter, and we wanna do a larger scale event every year. And those are mature operations. And we try to drive that type of maturity into the organizations, but not everybody can adhere to it. Like, we do everything we can.

Keith:39:45

And again, with our customers around the advanced services, we are making sure they test at least a subset of their environment and their critical systems on an annual basis. And then we do have some organizations that even want to test failback. We're not doing that all the time, but that is another component to it. And a lot of the technologies we use now are maturing their failback operations because this is becoming more and more prevalent. Right?

Keith:40:07

Like ransomware is really and cyber threats are really changing the landscape of recoverability and and how that works.

Max:40:13

Let's talk about ransomware. Mutability factors into compliance. It factors into ransomware, Define immutability. What does this actually mean in terms of a backup platform?

Keith:40:21

Yeah. So immutability means that the data that's being backed up cannot be deleted for any reason. So once it's Can't

Max:40:28

be changed. Can't be changed.

Keith:40:30

Can't be modified. So that way, if a third party gets in and tries to delete your data, it's impossible. It's protected. If there's any kind of attack or any kind of threat that they're trying to encrypt the storage that it's on, it can't be modified with the encryption. Like, they can't touch it.

Keith:40:46

It stays in its pristine state. Now that's not to say that if there was some kind of 0 day or something, you know, something slide in, like, you can have ransomware in that backup. And we scan both on the read and the recovery of that as well to ensure you're not reinfecting a system. So that's really where we get the, you know, the immutability is is protecting that data from modification in any

Max:41:05

way. K. Good news for most ransomware is dwell time is decreasing. A few years ago, dwell time, 200 days. Now dwell time's short.

Max:41:13

Like, people get in and they do something quickly. Yep. But this this becomes this thing around, like, oh, why don't they just restore from backups? It's not just that simple of saying, like, let's go restore from backup. You know, we've had a ransomware event and we're lucky enough that we actually have backups.

Max:41:29

How do you figure out, like, which backup to go back to? Like, you know, and and because there's this a 2 part question. Right? Like, okay, we have backups, but how far back in time do we have to go? We have to go back 30 days.

Max:41:39

We have to go back 60 days. Like, how do you find your good backup in that situation?

Keith:41:42

Yeah.

Max:41:43

And the second question is, what do you do with all the changes that's occurred between that RPO and wherever you are now? Because, you know, a lot changes in 30 days for an organization.

Keith:41:52

Well and that's where a lot of what

Max:41:54

we do. So you've got a

Keith:41:55

couple of things. 1, a lot of the backup technologies now we have have capabilities around ransomware to discover and isolate. Isolate. And if they find corrupted sections of the data, they can isolate that from the recovery so we can get to the good data inside of the backup stream. So they're getting much more intelligent in the software to be able to mitigate that.

Keith:42:11

Secondly, what we see is when you have an incident, you know, the one thing people don't consider is you have an incident response team. Usually now that's also being completed by your cyber insurance company. And they're going to spend 2 days or a day or 2 forensically looking at what's happening. And part of that forensic analysis is bringing that data back into the environment, making sure it's clean. And then where do we go?

Keith:42:35

And trying to determine. So you usually have we've seen where it's 24, 40 hours before we can even press a recovery button. Because they still haven't contained it. And some of that now too is to make sure that all of the check boxes you filled out on your cyber insurance form, that you are meeting those obligations, that you are did you have 2FAF authentication setup?

Max:42:55

Did you have your EDR? Did you have your security awareness training? Did you have this? Yep.

Keith:42:58

Yep. Yep. Yep.

Max:42:58

Oh, you don't have that stuff? Like, sorry.

Keith:43:01

Sorry. Yeah. Like, hey. Good luck. We spent 2 days trying to figure that.

Keith:43:05

And and so but I think that's the other component of it. Right? And and that's part of the reason we actually have added security services too to help I mean, our core is back up in Doctor, and and and we sort of moved differently in that we're adding security services as layers of protection to the data we already protect to make it, you know, to make sure you're checking those boxes so you can recover from an event and, you know, ensure that the insurance will, you know, follow through on on their obligations. Right? But when we see the data, like, it is part of that piece.

Keith:43:34

And so a lot of our recovery too when we're using our services is we'll recover into sandbox environments and isolate the locations and recheck that infrastructure before we then put it back into primary. So that way we have a sandbox secured environment separate from production and separate from Doctor that can be scanned and checked and reviewed before we move that workload back into production. So that way we ensure that it's been scanned and cleaned, and and we can look at the dates and times that, you know, try to use combination of technologies and backups that we have to be able to get that system back up and running as quickly as possible. We talked about immutability and the importance of immutability for ransomware,

Max:44:09

also compliance. Do you have a compliance mandate? Do you have a records retention? The other one that comes up into this is e discovery. You mentioned this briefly.

Max:44:22

And for the people that have not been through any discovery process yet, I feel for you because those days are numbered. And it's it's a joy if you're not ready for it from a backup provider. Like, how are you assisting people with e discovery? Because a lot of times ediscovery is like, hey. You know, search your your mail platform for this term or it communications between these people or or your file system.

Max:44:42

So I was surprised to hear ediscovery and your list. So now if you've got a company that's going through a new discovery process, like how do you how do you help them with that? Because you're not a primary citizen.

Keith:44:51

Yeah. I mean, that starts on the front end. That's in our discovery process with the customer and saying, like, what kind of search capability do you need? Do you have an ediscovery requirement? Especially when we're dealing with health care.

Keith:45:02

So we've got I'll use m 365 because it's that's usually where they go. It's email and file. Right? I can back up and secure and provide all of the backup technologies I have around Microsoft 365. In that list that I provided, most of them do that, have search capabilities built into it.

Keith:45:17

You start moving up the stack, there's governance controls that come within, you know, like Druva and Commvault, that they're more enterprise based and have the tool sets needed. You know? Most companies don't need all that ediscovery capability. You know, that a lot of times small, midsize customers aren't really concerned with that. It's more about data protection.

Keith:45:35

That's where in our our presales process and our discovery process. It's like, do you need these requirements? We talk about that capability. Do you want need to be able to search and control legal hold, you know, if we're working with the law office? Legal hold and and those pieces.

Keith:45:48

I have those controls in some of my other technologies, and they're add ons that I can bring to the table natively in the platforms to meet that compliance requirement. That way I can have a discussion around it. Well, that's you know, your per user costs are gonna go up. So balancing again sort of the capability and technology against what I need operationally. Most of our backup technologies have search in it, but it's not the advanced search capabilities like our e b discovery tools have.

Keith:46:12

So they may have to, like, really grind inside of that. Right? Like, and really work to find and discover that they don't have all the search capabilities and functionality that they need. So we try to avoid that conversation on the front end. You know, we went through that with 1 customer that was a health care organization, and they did some demos on the different technologies.

Keith:46:29

And we're like, nope. We want that. I will pay extra for that. That makes my life so much easier. You're you're like once they saw it, it was like the light bulb went off after they looked at a couple different technologies.

Keith:46:39

It's like, oh, you can really simplify my search that way, and have governance controls that make it that easy and and handle the the legal holds? I'm like, yes. Those are checkbox options and and and much more simplistic, cape you know, they simplify what looks very complex when they do that.

Max:46:56

Ediscovery is is one of these animals where it does not matter how much it costs you or how much pain it incurs or how much time it takes. You will provide this information. Right. You have to. And the penalty for not providing that information is bad.

Max:47:10

You know? So like I said, I mean, if I feel for the if you you know, people that haven't been through your discovery yet, because it it it's eye opening. Like, all of a sudden, you're like, oh, wait. This platform I've seen this a bunch of different ways. I've seen it from the and we have a platform, and the platform doesn't, you know, support c discovery.

Max:47:23

But we never licensed that here. And now we have to go through and we have to change our licensing across our entire organization or support this feature that now we need because we have to provide this data. Otherwise, we're gonna be in a bad situation with the court. Yep. And, like, whoops.

Max:47:40

Like, we had a budget for our IT spend this year, and that budget just went Right. You know?

Keith:47:46

Yeah. And that's I mean, and and usually when you bring it up on the especially on the like, usually organizations know they need it. Like, sometimes they're like, hey. Show me what you can do around ediscovery. Like or they've been, like, through that experience.

Keith:47:58

And and that's been some of the conversations we've had too. It's like as they start, like, hey. We're moving to Microsoft 365. We need to back up this data. I also have e discovery requirements.

Keith:48:07

And and that's where, you know, some of the technologies just, you know, natively don't have all the bells and whistles you need to efficiently do that in a process. But it does come at a cost. It's an expense. No. It it is it is not free.

Keith:48:19

And that's, I think, part of it too. It's it's balancing that out. That's every conversation we have is kind of a balance on cost versus risk and and capability with our customer.

Max:48:27

We started this off talking about a customer and a compliance requirement from them, you know, data sovereignty, different regions. Was you be store already in those regions? Were you doing build outs for them? Is it something that you've seen with customers or, you know, and how granular do you have to be? Because at this point, is being in the EU granular enough?

Max:48:46

Or is it you have to be in the EU, but you also have to be in Germany? Or you have to be you know, Switzerland is a different animal. Right? Like, you have to be in Switzerland versus being somewhere in the

Keith:48:56

EU. Yep. So for us, we have 4 data centers in the US that we manage to. That's where we have like, we rely on our partners. So, like, Acronis has integrated data centers as part of their Doctor platform.

Keith:49:07

And they've got, you know, dozens upon dozens. And that's part of the advantage we see with them and as as an example and and why this works so well with this customer is that they have the footprint. You know, and their footprint also involves I mean, like, we were sending all of the data from India to Singapore, which was regionally located. Well, India has changed a little bit too, and they've got well, it's not technically a requirement. Once Acronis opened their India data centers, they're like, hey.

Keith:49:32

Can we keep that data in country? Like, we see that that's a, you know, that's a part of the conversation. It's like, now I can just keep this in region. Let's relocate our backups to that location. We also, with the backup technologies we have and the Doctor resources we have, we can integrate with AWS and Azure.

Keith:49:48

So as we start to expand globally, like, we work with a global manufacturing company that we're leveraging their footprint in Azure, and they're they've got on premise leveraging their footprint in Azure, and they're they've got on premise infrastructure in EU and Asia Pac, Australia, actually, Australia, New Zealand, and US. We can integrate with Azure, and we're using their Azure infrastructure, to be able to do that to stay within Regent. But we're managing that for them on a global scale. So we use the different cloud platforms and the capabilities of our infrastructure within our, you know, our service providers as well to be able to kind of mix and match and find the best solution for the customer. So you

Max:50:24

told me, you know, 7, 8 month 9 month sales cycle, discovery, region, technology selection, infrastructure, VMs, etcetera, contract execution. What was the rollout at that point? Were you identifying and saying, okay, here's our, like, minimum viable backup, target these things in order, you know, and attack them? I mean, this would be pretty customer specific, I would assume. But were they a, gonna go after these regions first.

Max:50:49

We go after this infrastructure first. We wanna back up. Like, what's the stage in the rollout? And how quickly before you got to a point where you're starting to do test restores with them? Yeah.

Max:50:58

You know? And what's that? Yeah.

Keith:50:59

It was definitely regionally based, and they prioritize the regions for us. I mean, they could you couldn't boil the ocean either on that and try to do that all simultaneously. But we had initiatives, we sort of had prioritization by region. So Singapore was the big one. I think, UK and Germany and their German office in UK, even though Switzerland was a, you know, important piece of the compliance piece, I think they they prioritize some of the different, you know, EU data centers.

Keith:51:24

And then same with Asia Pac. It was like, let's get Singapore done. They moved into India, and then I think they did Australia. And then, you know, as those regions were getting completed, we would start the other location and then start they wanted to start testing right after that. And there was some testing, I think, probably within 3 to 6 months after things got up and running.

Keith:51:42

And so, you know, then it's kind of phased on top of that. So as the other locations came up, we had 3 to 6 months. It was kind of like a rolling sort of piece to get going. I think it took us total probably 7 to 8 months to get every site and every location backed up.

Max:51:56

And they had Acronis and decided to standardize on Acronis and have gone through m and a cycles post this contract. I'm just curious from, like, a methodology. Are they forcing any M and A to immediately switch to Acronis? Are they keeping current infrastructure in place and just, you you know, if they had, you know, Commvault or Zerto or something else, they say, okay. Hey.

Max:52:12

Here's our you know, you're gonna use Ubusor for your backup, and we're gonna integrate on this platform. But maybe you're gonna be on something else until, you know, whatever event takes place. Or they're just saying, hey. Look. We own you now.

Max:52:21

Boom. You're on a Kronos. We're gonna

Keith:52:23

unify all this stuff. Usually, they're letting us know, like, hey. We've acquired x location. It was like, hey. We just, acquired organization in Brazil.

Keith:52:30

What is your Brazil capability? What do you have in South America? And we do happen to have a Brazilian data center that we're using. And they had some acquisitions in the Caribbean. I mean, it was like, well, where do we wanna send that?

Keith:52:40

We'll send that data to our US data center. They had a US expansion and so moved some data there. And then they had a full merger with another organization as well that we're segmenting. They're still saying, you know, kind of from a costing perspective again. How that works is that we're adding their data centers, but marking them a little bit differently right now so they can do cost backs and look at acquisition costs and everything else that comes with it.

Keith:53:02

But they're using this as a methodology. It's like, hey. We're doing this with we've got a a a programmatic approach now to backup in Doctor. We're going to bring you into that. So there wasn't too much of a time span between that acquisition and then us deploying technology and then actually integrating those data centers into their Doctor platform.

Max:53:19

So beyond, like, compliance and risk and risk assessment and all these other things that you get out of it, Right? From an m and a and an m and a cost standpoint, it sounds like now there's a we know exactly what this is gonna cost us. We can figure this out before we even get into it and standardize our operations. So, I mean, that's a huge x factor for a lot of people in m and a. So we should talk about this a little bit in terms of, like, the impact of M and A and you guys being in the business of backup and disaster recovery and continuity for organizations.

Max:53:44

And well, they have a probably enough experience where they they know this many VMs, this many noses means this much storage. Right? Like, what's it gonna cost us? But do you see that a lot with people of saying, you know, can you give us a cut sheet just so we can go through and put this in our diligence and say, answer these questions? We can attach a cost to it.

Keith:53:59

Yep. Well, for them, that was the whole thing is they knew it right away. I mean, like, when we talked to their IT team about the onboarding of this customer, he's like, but Ricky actually said it. It's like the great part is is I know what it's gonna cost me. Like, I don't have to like, he had, I mean, the the SKUs and the relative cost per device and everything else at his fingertips.

Keith:54:17

He's like and, obviously, there's when we go from backup, when we charge it on the storage in the cloud, there's a cost of you know, you get some compression and deduplication in that. He's like and he had enough data from his other sites to kind of really dial in exactly what that was gonna look like for him on on the acquisition side. And we were actually having now one of the things we're doing too is is we've had some recent conversations with, like, venture capital companies and coming in to build, like, a programmatic approach as well. It's like, hey. I know what this is gonna cost.

Keith:54:43

It's programmatic. It's very easy to use. If I fall outside of your program, what they elect about the conversations we're having there is that I've got ancillary technologies I can bring in that if they acquire a organization that just doesn't quite fit and check all the boxes that if we need to expand out of a core technology that we selected, we can do that. So having some single solution set for every problem is one of the

Max:55:05

things they

Keith:55:05

find attractive.

Max:55:05

But but,

Keith:55:05

yeah, that's that's a big part of that that process is that understanding what your costs are before things start really help. And that was one of the things they looked at.

Max:55:16

Keith, final question for you. I'm gonna make this relatively broad and open ended intentionally. Favorite backup or Doctor story. So this can be a disaster story or success and that happy ending story. You've been doing this for a while.

Max:55:27

You've seen a few things. What stands out for you?

Keith:55:30

For me, I think I always go back to my first, like, real disaster, like that first real experience. I had my own MSP for a while. It was a small financial company that we were working with. They had 10 locations. Their primary infrastructure failed in the data center we were supporting, and it was a a 48 hour recovery from tape.

Keith:55:48

And it was a hardware failure, and we went through 18 hours of trying to get the right hardware just to get the system up and running. And then we found out, we had been doing test restores of their backups, but we never tested their active directory recovery, and it was corrupted. And so I go back to the because that was like, like I said, my first real disaster, and I always kind of think of that when I talk to customers is that it was 48 hours. My wife was, you know, she was messanger and closed down to me. I was, you know, 48 hours at this customer site.

Keith:56:16

We eventually got them back up and running. The core data and all their financial information was protected. You know, we actually lost no data for them, but it took us 18 hours to get the infrastructure back up. It took us another day to rebuild their active directory from scratch and then repopulate all of their exchange and SQL data to get the business up and running. And it was interesting talking to the business owners because it's like, you know, we were managed service provider.

Keith:56:41

It's like, we're paying you guys to get up and running. Why are we down for 48 hours? And it was like, guys, if you didn't have us, I don't think you would be able to recover from this. Like, your business would have been dead in the water. And I I go back to that.

Keith:56:51

And that's kind of my thought with everything we do is we can do things so much better now. Like, I have so much more technology to be able to protect against that type of scenario. But at the core, that's what every event is. And and ransomware and everything else is changing. And, you know, we've got a bunch of great stories about customers being able to recover from those incidents, and we're seeing that the frequency of that go up.

Keith:57:10

And but I think at the core, it's like that recovery process, because it was so personal to me. Right? Like, now I'm on the presale side, so I'm not as operationally close to those events. But that was my first experience, and I think about that almost in every conversation I have with the customer because we're doing everything we can to not put them in that position. That's not where you wanna be.

Max:57:27

It's not sexy. You know, my experience I I shared earlier. Right? You're hiring people to come in and manually reenter. And and they were lucky they had paper.

Max:57:35

I mean, they were a paper driven process back. So we're talking early 2000. I mean, you know, this is circa like, you know, 992,000. 1,000. No.

Max:57:42

This would've been, like, 1998, 1999. But, otherwise, you know, it's just, like, 0. If you go to 0, what does that look like for you?

Keith:57:48

And we've talked like, we've helped like, a lot of what we do is we're also helping customers after they've had a failure, right, and struggled with their Doctor. They're looking for a different option. And that we heard we have that scenario where a customer was breached and had their backup data deleted. Like and they were fortunate in that they had moved some infrastructure and had some core, like, VMs that they had turned off that fluky luck, you know, but they lost they lost 6 months of transactions when we were talking to them. And, again, that's hard.

Keith:58:14

And that's how they they got administrative password breach. The guy went in there and then deleted their backup data, and it was like, you hate hearing that story. And so in the manual piece of it, I think, is great. I think that picture in your head of, like, what do you do to get that back? And, like you said, if you don't have paper trail now, like and most companies don't.

Keith:58:29

Everything is stored digitally. Where do you turn to? And so I think that for us, we've been doing this for over 20 years. We've had customers you know, we've got several customers that have been with us since the beginning, practically. And we've evolved with them in their business.

Keith:58:41

Like, these are people we know. These are organizations that you get to know. We do a lot of work on the implementation side to understand their business. You know, I think that personalization is very important. I think that's the level of ownership we end up with.

Keith:58:52

It's like we wanna do everything we can to avoid that scenario. And when it does happen, that's we can keep it as painless as possible. That's the hard part.

Max:59:00

Yeah. Learn from other people's pain

Keith:59:02

Yeah.

Max:59:02

And put something in place and test it. Don't experience this firsthand and build up your own scar tissue. Like, just there's there's plenty of stories about this at this point. Learn from others pain. Learn from our pain and put some in place.

Max:59:13

Keith, thank you so much. This has been a pleasure. Thank you for sharing.

Keith:59:17

No. This was great.

Max:59:18

I have so many more questions about other things, but let's let's I I'd love to do this again and do another one of these and and get into some of these. You know, I've I've been scribbling notes furiously.

Keith:59:27

Yeah. No problem. Be happy to do it. This was great.

Max:59:30

Appreciate it. Thank you, Keith.

Keith:59:31

Yeah. Thanks a lot.

Max:59:32

I'm Max Clark. Hope you enjoyed this tech deep dive with Keith Luke's Ubistore on backups, disaster recovery, and business continuity. If you haven't had an experience in a disaster yet, data loss, I'm envious of you. Most of us in the IT world have scar tissue and have experiences with disaster and data loss. The biggest thing I can tell you, if you take nothing else away from this, is invest some time and energy to understand what data is critical to you and what you're doing to back it up and how you're getting yourself a position where even if you had to re figure out how to recover from paper, let's say you actually do have paper still in your business.

Max:01:00:07

You have the ability to do that. It's going to be painful, but you can do it. There is a big difference between companies that have some backup, no backup and full backups and testing your backups are very important. If you have any questions about backup, disaster recovery, business continuity, RTO, RPO, immutable backups, archives, e discovery, how this impacts ransomware, what you can do as an organization to protect yourself and lower risk and how to actually figure out those lines between cost versus risk. Give us a call.

Max:01:00:35

Reach out. Happy to talk to you. Happy to help you get started in this journey. Figure out where you are and where you need to be. It is a conversation I will gladly have.

Max:01:00:42

We will not charge you for it. We'll get you into a position where you can decide what the next steps are. And are you ready to take the next steps and which direction to actually walk in? I'm Max Clark. Hope this helps you.

‍