Lessons on VOD, Live Streaming, and Digital Media with Mark Barron at Akamai Technologies | Ep. 41

Max:00:07

‍Let's start with VOD. And so, I mean I mean, let's let's start at the beginning. Right? Like, somebody has content and and, you know, like, a a VOD pipeline from somebody has content. They license content.

‍Max:00:24

They bought content. You know, they have content. And, yeah, let's let's just let's just start there.

Mark:00:33

Okay. I mean, I don't know if you wanna know anything about me or my background or why I'm you know, why people pointed you to me.

Max:00:42

You see how, like, out of it I am right now?

Mark:00:44

Yeah. Yeah. I mean, I I don't know how qualified I am to have a conversation here, but maybe I can tell you why, you know, other people think I am. So, yeah, I've I've been working with Akamai for about 20 years. I worked with other CDNs before then, competitors of Akamai until we got assimilated into the board.

Mark:01:05

The, you know, so I've got about 25 years in the media space, working in the streaming technology business all that time and seen it come from real networks and Windows Media all the way through Flash and Adobe, HDS, and HLS, and DASH, and CMF, and all that good stuff. So, if you ask me if things have gotten easier, I would say they've gotten harder. It's more complex. We're trying to do more with this network that doesn't really support streaming. Right?

Mark:01:41

So

Max:01:42

I I mean, so if you've been with Akamai for 20 years, I mean, that's when when was Akamai founded? Probably like what?

Mark:01:48

1998.

Max:01:48

Yeah. As I said, not much longer before that. I mean, CDN as a concept, you know, wasn't, you you know, I mean, I love I love the simplicity of the original CDN. It was like, hey. Look.

Max:01:59

You know, we're gonna buy we're gonna have servers, and we're gonna put your servers closer to where your users are, and and we're gonna make your web page go faster. And in the late nineties, that was a real big issue. AOL was trying to solve that problem with their proxies while they were trying to solve 2 problems. 1 was how do they, like, not completely oversaturate their network, but but also how to give a faster user experience. And, you know, for people that weren't on the Internet in the late nineties, I mean, we're talking about the conversion of dial up modems to digital subscriber line DSL and cable modems coming out.

Max:02:32

And and it was still, you know, latency. I mean, you know, now everybody wants to talk about latency and edge, but very different world. You know, it was nice to have a CDN.

Mark:02:41

Yeah. Yeah. And I think the the ISPs like AOL, you since you mentioned them, loved CDNs because they actually offloaded, you know, the the heavy lifting for their user base, onto a CDN caching platform, that was running inside their own local data centers. Right? So it's like, great.

Mark:03:02

We don't have to spit this stuff out of our data center just to go to an origin somewhere else. We can save on bandwidth both in transit, you know, to the origin as if you know, so it was actually a really good relationship when CDN started going into ISPs and going, hey. We can help you.

Max:03:17

Yeah. I mean, cashing notes sitting next to the d slam or sitting next to a head end, you know, really efficient architecture. I mean and, you know, like, ISP and, you know, access networks evolved over time and, you know, network capacity changes over time. But I think to your earlier point, right, like, it's, you know, it's not simpler. It's just we we have we have more going for it.

Mark:03:42

Yeah. Let's not just do caching of objects. Let's do pages that are, unique to every end user, and it has to be you know, go through a network that goes to a server that says, oh, because you're you, I need to serve this version of this page, not another version. So, yeah, it's caching is hybrid now. Right?

Max:04:02

Well, I think I think the other part of this that became, when we talk about CDNs, I I look at CDNs. It became what are you using and how do you actually determine, what edge note to get to. And this is kind this is also, I guess, foundational to this conversation, which is, are you using a BGP anycast, You know, goods and good and bad. Are you doing DNS? Good and bad.

Max:04:23

A lot of people on market right now trying to do peer to peer, good and bad. You know? Like, it's in and I guess I've been in tech long enough that when people ask me questions, you know, there's, like, my brain immediately goes to, well, it depends.

Mark:04:41

Yeah. And nobody likes that answer.

Max:04:43

Oh, no. It's it's I try to simplify it. You know, I've I've also learned over the years that, like, sometimes people are just looking for the simple simple answer, but the it depends really drives a lot of the conversation. Right? You mean, you were talking about codecs.

Max:04:56

Right? And these these conversions between, you know, Real Media and Dash and HLS and, you know, all these other things that are out there. And and that is a big it depends as well in terms of what you think about your content, how you deliver it. And, you know, there were CDMs at launch that really tried to split, and this was an architecture and efficiency. Right?

Max:05:17

Like, small object versus large object. And how do they actually optimize for that? And and then everything just kind of became everything was small and everything was large. So there was really no difference between one and the other on these things.

Mark:05:30

Yeah. And I I think all CDNs play, that game a little differently. Right? So do you do byte range requests or do you not, right, for large objects? And do you cache those byte ranges, and without going back to the origin or partially or something?

Mark:05:46

So there's there's a lot of differences that I think when you talk to companies that do multi CDN, they realize that there are so many differences in how different CDNs work and operate.

Max:06:00

Let's dig into multi CDN for a moment. And I feel like multi CDN was a precursor to what a lot of people talk about in terms of multi cloud now. And there's big push for it. Not necessarily, I feel like the the the biggest push always comes from a resiliency or redundancy or, you know, like these sorts of things as opposed to a feature set. So So when I hear people talking about multi cloud strategy, they're usually saying, oh, you know, we should put some stuff here and some stuff there as well just in case this thing goes down, and then we can go over here.

Max:06:29

And I hear that same conversation related to CDNs. Like, oh, you should have multi c now, with with the exception of geopolitical influences and specific regions and capacity limitations. Right? You know, like, multi CDN from a, like, oh, we should have redundancy always is like it feels like the wrong conversation to be in because you can get redundancy out of it, but at the same time, like, it's not you have to optimize for your CDNs differently. Like, CDNs do different things.

Mark:07:01

True. So there are definitely pieces of the workflow that are gonna be impacted like that. For example, token authentication. I don't know how deep you wanna go into that, but every CDN, has done token authentication. If you don't know what that is, that is a, a either a query string or a header or something in the URL itself that makes the URL, unique to that user.

Mark:07:28

Right? So it's a token that you get applied when you log in for streaming. For example, If you stream if you log in to a streaming, workflow through a paywall, you're gonna get a token that that CDN knows how to interpret and see what the secret and the salt are and then identify whether or not you should be able to view that stream. It's different across all CDNs until recently. Now we've got this thing called common access, token, which I know that Will Law, if anybody's ever seen him speak, he's an Akamai folk person, but he's also just a generalist in, the special the streaming world.

Mark:08:08

But common ass access token is a thing, and, we as a CDN embrace it because we hate the fact that we have to, you know, comply with or build something special that goes with this CDN or that CDN. So when you say multi CDN, there are certain components of the technology that you have to normalize. You have to limit, you know, kind of the least common to non denominator, if you will. Right? So it does rule out some advanced features that some CDNs might wanna offer you if you're across multi CDNs.

Mark:08:44

So, yeah, you have to be careful of that.

Max:08:46

Well, it also introduced another thing as well, which is you have to decide which seating to sit and send traffic to, and then what's the mechanism. Now if you're using if you have a if you're doing video and you have a player, you can put a lot of that logic into the player. If you're talking about delivering a web page, you don't have access to the player to drive that logic. You know, if you're or if it's a mobile app, you can you can put logic there. But then you get into some other logic engine that you have to route traffic through.

Max:09:09

And and this is, you know, from, like, the network engineering days. Right? You're like, you try to, like, you know, you're always trying to quash that, like, failure point. All you end up doing is introducing 37 more. So, you know, like, you know, is it is it did it solve the problem, or did you introduce a new one?

Mark:09:26

Well, with with VOD, I mean, I know you wanted to start with VOD, for streaming. Multi CDN is less of a a concern from an availability availability perspective. It's more of a contractual, you know, obligation retire, whatever your contract, commitment is. And we find that customers will bounce traffic back and forth for those reasons, but also from different geographies. You know, we all have servers in different ISPs that might perform better than others for certain end users.

Mark:10:03

So people will shape their traffic based on performance for that reason. But from an availability perspective, from a VOD, perspective, availability doesn't really come into play because you can cache the content everywhere. Right? It's, you know, popular content. The the rare use cases that VOD acts like live are like, when you release, you know, the final season of Game of Thrones or something like that.

Mark:10:30

Right? That it it becomes a live scenario where your cash is trying to fill up as fast as you can. You can gain that. You can deploy it. You can, you know, pre populate the cash.

Mark:10:41

There are certainly ways to do that. So but that's that's the VOD world. It's more of a load balance because I want I wanna save money here or there. I wanna bounce things around the world differently.

Max:10:56

I was I was having this conversation real and recently, and it's funny to me because, you know, you start as a, you know, computer network engineer, and you kinda work your way up. And right now, people ask me, and I I would still say I'm an engineer, but most of what I'm engineering is is financials. And, you you know, you're like, oh, what's our you know, and here's here's an example. Right? CDN.

Max:11:17

Like, what's our rate over here? What's our rate over here? How do we maximize our cost based on this? Oh, we have a we have a commitment tier over here that we have to fill to. Otherwise, we're you know?

Max:11:27

And, like and and then you get into, like and then you get into the other fun stuff where it's it's you know, this provider has a deal with this ISP and has capacity, and this one doesn't, and this other thing is happening. And and and and you and you it's just so funny, you know, to me always of, like, you know, gone are the days of, like, trying to get our cues working on a NICE card, you know, because it just wasn't flowing. Right? And now it's now it's like, oh, you know, we're trying to serve traffic into this re and and also, right, big difference between, you know, content that's going into, you know, tier 1 cities in the United States versus, you know, foreign markets. And then how do you deliver and how do you deal with that as well?

Max:12:11

So, like, where your content's actually going makes a makes a makes a big difference.

Mark:12:15

Yeah. And, you know, as you know, sometimes the best technical solution doesn't win because of financial reasons. Right?

Max:12:24

So that Almost every time.

Mark:12:26

Yeah. So we we can we can lay out the best technical bulletproof solution there is for any live event or live VOD scenario, and then somebody will go, yeah, but that's too expensive. So we're gonna go over here and do this. And, you know, you just have to accept that and say, okay. Well, if you're gonna do that, these are the things you should have in place just in case something goes wrong.

Mark:12:46

Right?

Max:12:47

So what are the challenges with VOD that people I mean, because VOD from a from a a high level architecture is really close to you know, as you said, it's close to any sort of traditional CDN delivery. You have an object. In this case, your object is video. But then we I guess we get into you start adding complexity to it. Right?

Max:13:03

So you have authentication. You have encoding. You have variable bit rates. You have, bit rate selection. You then have, like, ad insertions.

Max:13:13

You know, you've all these other things that start to pile into that pretty quickly. And how much of that well, you know, I mean, you know, Mark, I mean, talk me through it. Like, what within that, how much of that is offloaded onto an Akamai at this point, and how much of that is maintained and managed by the player, and and how much I don't I don't wanna use this this this phrasing to it, but, you know, there's is the intelligence happening at the player and at the at the client side or the customer side? Or is you know, are you guys driving the intelligence through the CDN? And how how much of that lift can you take off of people that wanna get into the VOD business?

Mark:13:51

Yeah. Are we are we still talking about multi CDN and

Max:13:54

No. No. No. For no. I mean, the great thing about today's world is that was 20 years ago.

Max:14:01

Right? When you 20 years ago when you started Akamai. 20 years ago, average average cost per megabit in a data center was, like, $8. So we were doing a lot of bandwidth manipulation to try to, like and it was meaningful because if you could figure out how to shunt something that was off of an $8 per per meg line down to a $6 per meg line, I mean, that made a big difference. Bandwidth rates are a little different now.

Max:14:23

I mean, it's the good news is is this usually isn't as big of a problem for people anymore.

Mark:14:28

Yeah. But mostly the way I see people traffic shape is either through automated, performance tools like, Conviva. You know, they'll have some agent in there that says, well, we're gonna shape this traffic because it's not that performing in this location, and we're gonna route these users to this location. But 9 times out of 10, we see DNS being the the main way to do it. So they have a video.company.comdomain that they will either send to 1 CDN or the other, to meet their traffic commitment, or they'll have a Akamai specific domain and a, somebody else specific domain.

Mark:15:11

And they'll just shape it in their CMS, their content management system, And they'll say, okay. We're gonna send traffic over here for these users, and we're gonna send traffic over there. Very few of our customers do it by, different format. It used to be bigger when, you know, we had flash servers and things like that. You know, you would serve for that format over a certain CDM because that CDM performed better than others.

Mark:15:36

But now that we're all HTTP based, it's more geography based, I would say, and commitment based. Right? So in the US, we're gonna satisfy this commitment that we have with this CDN or whatever. Akamai doesn't really play the geography, price points. We have a single price point in all geographies for most part, for almost every customer.

Mark:16:03

In the rare cases, we might negotiate some things, but I don't know of any. So those types of traffic traffic shaping, it's mostly DNS or CMS based from what I've seen.

Max:16:15

And the problem with DNS based traffic shaping well, I mean, again, I'm gonna date myself when we had, the the recursive DNS platforms launched. Like, the open DNS was probably the big first big one that got really huge. And, you know, now they're relatively everywhere. Right? But, I mean, I was in Los Angeles, and OpenDNS was in San Francisco.

Max:16:42

And so all of a sudden, anybody that was using DNS to to do traffic just you know? Hey. I'm in San Francisco, and I'm sitting in my office in 1 Wilshire, and all my traffic is flowing up to the bay. And you're like, I actually know that your infrastructure is right next to me on the next floor. You know?

Max:16:57

It's and that wasn't a great user experience. I mean, that's not as bad now, but it's still that's still a thing.

Mark:17:04

You talk about dating yourself.

Max:17:06

Oh, I can I'm a date myself a lot. I'm sure.

Mark:17:08

So the, AOL issue. Right? Everybody looked like they were from, Loudoun, Virginia. Right? So everybody that had an AOL login or user base, everybody was getting traffic, from that location.

Mark:17:21

It was ridiculous for a little while until that was solved. But, yeah, there's there's certain there's definitely ways to do it. So when you're talking about streaming, you have various places to check where the user is. Right? So you have the player, you have the landing page, you have the stream itself, you have all these things that should talk together to say, alright.

Mark:17:42

Based on this landing page or even this user's login based on where they usually are. There's a whole bunch of ways to do it, but, you know, DNS does seem to be the ubiquitous easy way to do it. That's where a lot of people do it.

Max:17:57

Yeah. Well, I mean, it's it it is probably the easiest. I mean, it's a it's a it's a blunt tool. I mean, you're taking and, you know, using a £5 sledgehammer to try to, like, do finish nailing. Right?

Max:18:06

It's, you're you're gonna get the rough gist of it, but not, you know, there's if if you have enough traffic, it kinda smooths itself out after time, but, you know, it's not really super granular. You know, are are people I feel like with with most of these, and with companies that are getting into streaming. Right? You start talking about, like, what do you build and what do you buy? You know?

Max:18:30

I mean, this is this is constant with everything in tech. What do you build? What do you buy? Do you buy you know? Okay.

Max:18:36

You own the content. Are you buying your own encoding platforms? Are you gonna use somebody else to do the encoding? Are you chunking it yourself? Do you use somebody else to do that?

Max:18:44

Are you you know, what DRM wrapper are you using? Do you go out and buy that? Do you use somebody else? Is the CMS platform. Right?

Max:18:50

Like, you go buy it. Do you just go chunk it together. Right? And and, again, in the VOD world, a little bit easier because you actually have the, you know, content. You upload these files to your storage system, you know, which feels like 90 out of a 100 times is on s 3, which causes other problems, but I won't I won't I won't get that.

Max:19:10

Let's let's let's not get into that that hole just yet. But even within that, I mean, how I mean, has has this market at this point, you know, I I hate using, like, the commoditized commoditized term, but, I mean, is it just you know, have we have we finally reached the point where people just go on, hey. Like, I've just got this player because this player is great. I'm just gonna use this player that's a known thing, and, like, we're off to the races.

Mark:19:35

I would say as soon as we get settled into a norm, something changes. So I'm

Max:19:41

gonna use that quote because it's so true.

Mark:19:46

So, you know, you mentioned s 3, and I'll bring up the elephant in the room. So s 3 with their egress costs, you know, I mean, it's like, okay. You you put all your content on s 3, then you're paying out the you know what if you're not caching efficiently. So, you know, there's there's companies that have built complete solutions around that, and Akamai has one of them. It's called CloudWrapper.

Mark:20:08

You know, we before we had our own cloud, we had a solution where you would only pull from the origin once, and we would cash it. And you would pay for that that layer of cash, but it was lot lower than the egress cost. Right? So

Max:20:21

Egress. As for egress is the real mode of AWS, and people that don't understand that just haven't dealt with it yet.

Mark:20:27

Right. Right. Yeah. So, and and this isn't an Akamai commercial, so I won't go into other things. But, you know, we we have solutions around that as well.

Mark:20:37

So

Max:20:37

Yeah. No. This this absolutely we should talk about this because this is I mean, it's all relevant. Right? Like, there's the origin selection and origin cost and an origin implication of that cost is a is a big factor here.

Max:20:50

Right? And, you know, if you've got 100 of terabytes or petabytes worth of of origin content, let's say, on s 3, you know, the cost of dealing with that, you know, and well, this actually, I think it it backs up a little bit. You start talking about what's your what's your encoding and ingest and DRM, like, pipeline, and what is that feeding into? And then where is that actually going after it's run through its pipeline? And then are you making a decision to shove it onto s 3 or, you know, take your pick of other alternative?

Max:21:23

You know, I hate the hyperscale name, but, you know, the public cloud providers or or or, let's call it, like, media centric or CDN friendly, you know, storage platforms that have cheap to 0 cost egress. Right?

Mark:21:38

Yeah. Yeah. And, you know, and I will bring it up. Akamai bought a cloud called the node. So, you know, the the egress cost out of there is where we're actually seeing a lot of business being taken away from other cloud providers.

Mark:21:51

But the encoding workflows, what I found is that it is very customer specific. When you get into the large players, you know, like, you know, the Disneys of the world, they they don't want anybody else touching, you know, their their secret sauce on their performance and their codec or their their encoders. Sorry. They have a specific, quality measurement because, you know, all of the Disney movies and everything, they fade to black. They do a lot of things that make encoding software a little bit challenging if, you know, you try and fit it into a one size all, someone size fits all profile.

Mark:22:30

Doesn't work.

Max:22:30

You mean Disney is not just going out and taking FFmpeg and running like

Mark:22:34

Yeah. No. They're not. They've they've built their own. So, but, you know, that that's one example of one customer.

Mark:22:43

And then other customers will outsource, to, let's say, you know, if you wanna use small to medium customers, we use Brightcove, you know, the Zencoder solution. Bitmoven's out there. You know, a lot of companies will use elemental, of course. But when when companies start using the cloud infrastructure, it's expensive, but it sucks you in and you're locked in. So it's that whole vendor lock in that, you know, with Akamai, we've you know, my being an Akamai veteran for 20 years, I've seen us get into the workflow space and then get out of it.

Mark:23:22

And as a media specialist, that's painful. So I'm like, hey. What do you wanna talk about today? All I got is delivery. But, you know, we're really good at that.

Mark:23:32

So and and we have been for a long time. So we were performance based.

Max:23:37

I mean, that's a really good point because, you know, there's there's, a lot of other companies that were big logo, big brand, big names in the CDN space that have evaporated in recent years. And you look at you talk about media as well where they had, encoding and workflow pipe pipelines, and then they got rid of those because they didn't wanna invest them anymore. And you can you can almost count that day as, like, 0 day and then just work forward to when they were gone. Because if you've if you've looked at or you used, you know, an elemental pipeline and you start looking at the actual cost of these things, like, it's very easy to be like, ah, you know, we gotta, like that that piece over there, that x, that that block in that diagram, we gotta get rid of that thing.

Mark:24:17

Yeah. We we see a lot of people using open source as well. You know, FFmpeg isn't unheard of to go out and use. I mean, it's the baseline of all encoding platforms anyway. It's just someone who's built a workflow around it.

Mark:24:31

So, you know, we've got companies that have built entire VOD and live encoding workflows on Kubernetes on the Linode platform. So we see that taking off whereas you can actually grow the scale. If you have a very large volume of, you know, content that needs to be encoded or re encoded for the next format of choice that comes out in the next couple of years, you know, you're stuck with this thing that you have to do something with. Now if you can spin up, you know, 1500 servers and, you know, over a course of a couple months and do a couple of petabytes, you're done. You know that.

Mark:25:08

And then it collapses and goes away. That's one workflow that I think is is definitely gonna evolve and keep on happening. Right? It just does. And then the packaging for DRM workflows.

Mark:25:20

Right? So you've gotta package in the different formats and different DRM schemas for, you know, to support different DRM architecture. So and technology. So EasyDRM is a perfect example that runs their licensing servers on the Akamai platform today. They're they're very happy with it because it's 0 to hardly any egress costs, but it's also just license request.

Mark:25:48

Right? It's not that much egress anyway, but it's just, you know, very high highly performant moving into the most far reaches of of the the world a little further than the hyperscalers are today. So we've kinda taken the CDN approach that we've done where we put it, you know, our CDN edge servers closest to the edge users. We're doing the same thing with cloud. We're we're trying to get into, you know, South America and, you know, in Brazil and Argentina and things like that where you don't really see the other guys.

Max:26:17

How much has, Apple and Google I mean, so Apple with Safari and, of course, the iPhone mobile devices and then Google with Chrome. I mean, now you've got relative you know, much smaller browser market. I mean, effectively. Right? Like, all the browser shares tied up with those 2.

Max:26:37

And, you know, I'm thinking back to, like, when HLS and Dash, you know, these things started launching. I mean, how much has this changed the game for people in trying to figure this stuff out?

Mark:26:48

Yeah. So Apple and iOS, they're always the the ones that kinda puts everybody kinda like, great. I gotta support that one format because iOS doesn't support. So, you know, we've we've got people that are doing LLHLS. They can't move to complete dash platforms.

Mark:27:09

They would love to for dash CTE. We're talking about latency earlier, but, LLHLS is required for iOS. So there's there are still some, workflows that require certain, codecs for devices for those specific devices if you wanna reach every device. Now very few companies that I know will alienate an entire iOS platform of devices. Right?

Mark:27:37

So they will always support whatever iOS does. Right? Now when you talk about iOS making very interesting, moves, the a v one chip that's built into the, you know, iOS 15 and above, that's pretty impressive. I think it was 15, maybe 13. I can't remember.

Mark:27:56

But, you know, that having that the ability to do hardware based decoding on the actual phone, does help battery life. It helps, you know, bandwidth, everything. So it's a it's a it's a big component of what you have to think about when you're, you know, thinking about which devices you wanna support and what players and, you know, that kind of thing. Now then then you have the Rokus of the world and the set top boxes and the devices or the TVs, the smart TVs. There's so many different devices out there that you just you really still have to play the the game.

Mark:28:32

I mean,

Max:28:33

so how many different formats are people encoding to at this point?

Mark:28:35

Well, we're we were hoping that CMF would be, you know, the the end all be all, but, you know, there's still some holdouts out there that that wanna have smooth streaming, for example, for some of the gaming consoles. So there's there's HLS, dash, and very little smooth these days. Now a new one that's actually taking off is, HESP, HESP, the high efficiency streaming protocol. That's actually been in, you know, NAB demos for the last couple years. It hasn't taken off a whole lot.

Mark:29:17

And then for the very low latency type workflows, you've got WebRTC. Right? So, you know, you have people that really wanna have two way communication conversations. That's WebRTC, but you can't really scale that from a streaming perspective. So you're not gonna see the next Olympics streamed over WebRTC ever.

Mark:29:41

So that's kinda interesting. So you got those 3 types of workflows that, you know, we have to deal with at Akamai that people are like, I need lowest latency possible. Well, do you really? I mean, let's talk about that for a second.

Max:29:55

Well, my version of that is every page has to be dynamic. And it's like, really? Does it? Every page? You can't cache anything?

Mark:30:03

Yeah. And the and the latency story is people usually want it just because they've heard about it. They're like, they have no idea what the implication of going down to 1 or 2 seconds latency from glass to glass is from an encoding workflow for live.

Max:30:16

Well, I mean, if you're doing live events, then maybe it matters. Like, if you're if you're streaming the World Cup

Mark:30:22

Only if you're also broadcasting it over some other means. Right? If not, who's gonna know but the people that are in the stadium that you're 5, 6, maybe even 15 seconds behind. Nobody's gonna know.

Max:30:34

Yeah. In most cases, they wanna delay anyways in the signal because, you know, there's other reasons for that. Right?

Mark:30:40

I mean, Janet Jackson. Right? The wardrobe malfunction.

Max:30:44

Yeah. No. Good. Good. Good.

Max:30:46

Really relevant example. Right? It's, the the the person to person live streaming, you know, the the game streaming is an interesting example of that. Right? Because now you're you're I mean, you really I mean, to your point, you have to merge a couple of things that are not very easy to merge.

Max:31:06

And then but, I mean, how much of that I mean, you've got, like, person to person streaming, and then you've got chat. But, like, you don't have a lot of people doing audio video that's more than, like, maybe 1 to 1. You know? Like, I mean, that's still pretty small. Otherwise, they're taking and using it as a source and then rebroadcasting it as a stream.

Max:31:22

Right? Like, it it changed that that technique changes a little bit.

Mark:31:26

Yeah. So so for the you might be talking about game streaming. I was I was gonna take you in a different direction, like the Twitch direction. Right? So if somebody's, you know, wanting to stream their, you know, event going on, so gaming event, Then I I was even talking to a customer just last week saying, we need 2 seconds latency.

Mark:31:50

I said, why do you need 2 seconds latency? And they said, well, we want people to be able to respond to the chat feed that's going on. I'm like, people are playing games, though. Right? They're they're playing a game, and they you think they have 2 seconds of latency of them in their brain to go over and look at a chat, and then answer that person and go, oh, so and so is asking about the the the.

Mark:32:12

It 2 seconds is not that important. It really isn't.

Max:32:15

Well, I mean, when when when we started dealing with voice over IP on a global scale, you know, there was an experiment we ran where we had and pinned, pinned a call from Los Angeles to Frankfurt and back. You know? So it was it was it was the craziest on network thing that we could do to actually simulate network, you know, latency and then and start injecting things like packet loss and jitter and all these things into the call. And at that point, I think the round trip time was, like, 270 milliseconds, if I remember correctly. I mean, it's it was it was big.

Max:32:47

You know? And and and it was fine. I mean, you couldn't tell that there was that much delay on the line. You know? You you read all these things, you're like, oh, you can't have a phone call at that kind of latency because, you know, the delay, and you're like, no.

Max:32:59

You can have a phone call just fine. It works okay.

Mark:33:01

Just watch the news, you know, and and watch one correspondent talking to another correspondent somewhere. Even in the US, like, east coast to coast, they're like, hey. What do you think about this? Yeah. I think it you know, it's it's noticeable.

Mark:33:15

I mean, that but that's that's the satellite. That's not even Internet. So, you know, we're all trying to remove that latency, that that 2 person, two way communication talking. And there's ways to do it. But So

Max:33:27

that I mean, that that creates some interesting things. Like, so acquiring Linode and bringing a compute platform and a storage platform and an alternative into the Akamai umbrella. Right? So now now you have, a commercial pressure to bring pipeline and workflow back over to you. Right?

Max:33:45

Because it's you know, can you I mean, is, you know, elemental running on Linode yet? Like, at what what's the what are the big dominoes here that change, you know, change that equation?

Mark:33:56

Yeah. So, you know, we're still going through a little bit of a maturity, and, you know, credibility phase. Right? We bought this company 2 years ago. It wasn't enterprise ready when we bought it.

Mark:34:10

We're building it out. We started this program called the Qualified Compute Program or Partners Program, where media specialists like myself and others in the company reached out to all these encoding partners and said, look. We got out of the encoding space 5 years ago. We're back in it. We we want you to come over and build your platform on us.

Mark:34:32

So, you know, ATEM, Media Excel, Capella, Bitmoven, Harmonic. You know, there's 5 big names that companies out there that are actually using those either on prem or if they're using if they're not using them on prem, they're using them in a cloud, a cloud. To go to those companies and say, hey. We want you to run that on Akamai now. We haven't we haven't earned that credibility yet.

Mark:34:58

So we're we're getting there. So that conversation is a slow conversation where we say, hey. We want you to try this with one one of your streams. Give us one of your 300 streams and, you know, move this over. And it's working.

Mark:35:12

We've got, you know, I would say, a dozen or so, companies that have started actually working with us on that. 2 or 3 of them have really made the commitment to say, this is really great. This this works perfect. So we're gonna move everything over. And it's you know, they have to wait till their commitment on their other cloud is expired.

Mark:35:33

So The

Max:35:33

cost differential on that has gotta be just absolutely bonkers.

Mark:35:37

Yeah. I mean, we're we can well, I can't share any customer names, but I can I can tell you that we've seen, like, 40% difference for some TCO calculators that we built? Top sorry. Total cost of ownership calculators.

Max:35:53

The cloud cost, I hate another term I hate. We should actually call this out here. Repatrization from cloud. Like, who comes up with this nonsense? Like like, why do we all of a sudden decide that it had to be called a hyperscaler?

Max:36:05

It's not a hyperscaler. It's a public cloud. Like, okay. And then you repatriation. Like, this is just it's crazy.

Max:36:10

Makes me nuts. Anyways, you know, industry wide people will say and use numbers, and it's like, oh, it's like 30 to 40% cheaper. You know? Okay. Like, we literally just went through a project.

Max:36:22

This is a 7 figure a month spend. It's 72% cheaper for them to move on to bare metal.

Mark:36:27

Well yeah. So bear yep.

Max:36:29

Like, it's just, you know, you can't rationalize that number when you say, you know, you're gonna you're gonna cut 700 and I mean, $1,000,000 a month. You would cut $720,000 a month.

Mark:36:40

But the argument on that so having having sold and had many conversations about cloud with companies now, the argument on that is you are paying for that bare metal. And if you think that you have to scale for, you know, lots of

Max:36:57

Can I roll my eyes?

Mark:37:00

Yeah. So so when you start to scale that and you scale it for an event or a couple of events, and then all of a sudden you don't need that infrastructure that you've just built out on prem for all that time, you know, then what?

Max:37:14

Can I give you 2 secrets that you probably already know? The first one is elasticity is a lie for the majority of of companies. Right? Because as soon as you start going through any sort of cost optimization structure, you're buying reserved instances, savings plans, you're signing an EDP, you're doing something that's committing you to a spend, and it's also usually committing you to a spend increase on an annual basis. So this idea of, like, oh, we're gonna turn stuff off.

Max:37:42

No. You're gonna share shift. You're gonna spend shift. Right? Because you've you've committed to a $20,000,000 spend this year on this platform.

Max:37:50

Like, you can go turn stuff off, but you better go spend it somewhere else because guess what? You're still writing that check. Right? So the whole that whole idea of, you know, the femorality for reduced reduction of cost in the real world, it doesn't happen. It actually causes a lot of pain for companies because then they they realize, hey.

Max:38:10

We, we rearchitected something. We saved a ton of money, and now we're gonna be we we have this $500,000 shortfall that we have to plug. What can we shove into the marketplace to make up this shortfall? Because and that's where I see multi cloud, you know, disintegrate really quickly. Because usually that second cloud platform, the test cloud doesn't have the same contractual requirements.

Max:38:33

And as soon as they hit whatever point, it's like, boom, that workflow gets yanked right back because there's a there's a spend threshold. The other secret here, and this is this is I I'm I will get off my soapbox. I promise. But the other secret is it's just because you have some percentage of infrastructure on bare metal doesn't mean you can't still scale onto the cloud. Right?

Max:38:54

And and every single one of these that we build and deploy, you know, we pick a steady state, we put it on to bare metal, and then we you know, you scale your whatever you need onto the public cloud. And it's really not hard. Like, pick your pick your cloud provider of choice. And and the bare metal providers are getting better and better at being able to provide on demand, you know, capacity. So, like, you still I mean, that's still an option.

Max:39:16

Like, there's

Mark:39:17

Well, yeah, you bring up a great point because I was talking to a broadcaster, a shopping network broadcaster. I won't mention the name. They have on prem, as well. And we're we're like, hey. Why don't you augment that with some cloud infrastructure?

Mark:39:31

They're actually using one of our QCPs, and the QCP has an orchestration layer that manages an on prem and cloud at the same time. So it's like, you won't even know the difference of where these servers are. So, you know, you could build out more channels, build out more, you know, whatever it is you wanna do. And moving they're they were in the process of migrating their on prem infrastructure some from one location to another. And they said, yeah.

Mark:39:57

We're gonna be kinda we gotta do this thing where we we bring these up and then we migrate things over slowly. And I was like, oh, if you're in the cloud, you could just, you know, then ship your stuff and then

Max:40:10

Yeah. Well, that's and that's a good example of it. Right? You know, I mean, there is, I mean, look. The there's so many and, again, it all comes back to politics usually.

Max:40:22

There's so many changes that a cloud platform does for a company that's usually alleviating internal politics and, you know, cost based budgeting and and departmental bill backs and, like, all these other things that have to happen that you go, oh, you know, like, we don't have to deal with any of that anymore. And I totally get it, and I understand it. Just don't tell me like, for the people that come at me with, like, oh, the cloud's cheaper. No. It's not.

Max:40:47

That's not the problem you're solving. Like like you know? Like, you could tell me you're solving some other problem, and I and and 100% I'm with you because I've dealt with those. I get it. I 100% get it, you know, but, like, you're not saving money.

Max:41:02

So, like, let's just let's just, you know, anyways, soapbox. I'll get out. I'll climb down. The, you know, it's I I mean, it's I mean, because the other thing with streaming that you get into when I say streaming, I'll use it like VOD streaming you get into. It's like long tail content.

Max:41:24

You know? And, like, how do you deal I mean, that now that's actually a a unique problem to VOD versus live. Live. Right? Because long tail is an animal that you you know, it's not like people are deleting their content library randomly.

Max:41:37

How do you how do you manage long tail effectively?

Mark:41:41

So there you you mentioned earlier that there's different platforms or networks or workflows for large content and small content, small pieces of content. It's the same thing for long tail and, you know, non long tail, if you will. We architect some of the caches a little bit different to hold the cache and not, age things out, as often as they, that they would normally.

Max:42:09

You should actually let's let's pause. Let's talk about that for a second. So, like, foundational architecture for CDN. Right? You know, we've got content somewhere on a on a hard drive.

Max:42:23

So it's on a cloud. It's on s 3. It's somewhere. It's on a hard drive. And and the CDN so request comes from a browser.

Max:42:31

You know? It has a URL. That URL is the CDN. You know, the CDN at that point, we're talking about, like, the CDN edge. Right?

Max:42:38

Like like, walk me through. Yeah. So walk walk me through that architecture, the CDN, you know, front to back just for people that are watching or listening to this.

Mark:42:46

Sure. Yeah. So the like you said, the content is at an origin, and this the end user might request, the content and go to an edge. The edge is closest to that user for performance reasons. Right?

Mark:42:59

Because when you cache content at the edge, you want them to have to not go through the entire workflow back to the origin if it's long ways away. What Akamai does for streaming content and many of other CDMs do is there's a, an inner edge, if you will, called the parent tier. So the edge servers will satisfy a lot of end users that are in that same geography. Right? So a single server in this it's never one single server, but and for this example, we'll call it 1 server.

Mark:43:31

We'll handle many requests, so that's why you get cache efficiency. Then that parent server will handle requests from many edge servers. So you multiply the, offload capability from a parent server because it's getting requests more often than the edge server, and the edge server may age things out because of other requests that are coming in for other things. There's only so much memory and storage that it's allocated for that caching appliance. But this middle tier, because it's serving many edge servers, will have more requests in a given month and keep content in that, location.

Mark:44:14

And then your origin, hopefully, will will receive fewer requests many fewer requests. Now for your long tail, there are architectures that you could build where you have committed parent tiers that have larger storage allocation as well as longer cache life, if you will. So won't age things out as often. So we do that for a lot of the broadcasters that we have.

Max:44:39

Is origin shielding still a big thing too as well?

Mark:44:42

So the origin shield has a couple of different use cases and names. Right? Origin Shield can be, an an echo that protects, the origin from, you know, only requests from certain IPs. And Origin Shield could be that parent tier that I'm talking about. In our case, that's called CloudWrapper.

Mark:45:04

It's the same thing. You know, fast Fastly calls it origin shield, and we call it CloudWrapper. That's the middle tier that has that larger, I guess, cash efficiency, if you will. We built that solution for Facebook, for social networks. Right?

Mark:45:22

So how many people are gonna see the videos that I upload of my cat? Well, not that many people. Right? But I don't wanna have to go back to the origin every I hate cats. I don't and I'm not a cat person.

Max:45:32

It could be a popular cat. I've seen some of those.

Mark:45:35

Yeah. I'm a flying guy. I do flying things. So but the that concept is offload that content from the customer's origin as much as possible possible so that the last mile issue is our problem, not yours. We don't charge for storage on the, for, the most for most cases, but for that use case where you definitely wanna offload from the origin.

Mark:46:00

And in this case, it might be s 3 where you're trying to prevent from a lot of, egress cost, then that middle tier is, you know, you kinda it's a shell game a little bit. You're moving the cost to someone else. But the play is it works when the egress cost outweighs the actual cost that you're paying for that tier.

Max:46:22

There was a chart that was made years ago and circulated, and it was talking about access times for CPU to, like, do work and, you know, where you sit in the cache. And I think about that a lot in this example too. Right? Because your edge cache is not a lot of storage, but very expensive very fast. You know, you're using, you know, RAM, NVMe.

Max:46:41

You know, it's it's like, what's the fastest mechanism that can send that origin sorry, on that edge. And then as you move farther and farther towards the origin, the storage becomes cheaper and slower. Right? So, like, the parent cache is probably not you know, I mean, a a mix of NVMe and SSD. Right?

Max:46:57

Then you go from that, and then you get to the next tier where it's like, okay. You've gone from SSD to SATA. Right? And and you kinda keep working now. I mean, Facebook's storing on tape drives.

Max:47:06

Like, you wanna talk about, like, crazy stuff. Like, you know, I mean, they've got data centers of tape. And, you know, that's not fast retrieval, but it's really cheap.

Mark:47:15

Yeah. No. Everything on the Akamai platform has been replaced with SSD by now. You know, we got, like, I don't know what the number is, 300,000 servers around the world or maybe 400,000. Like, I I don't pay attention to the marketing anymore.

Mark:47:31

But the the the number of servers that we have is constantly you know, we're we're swapping them out for more efficient servers. That's why it's not constantly growing. We can't just throw hardware at every problem. We have to get more efficient. So, but, you know, the architecture that I just described is the architecture that is good for long tail, something for, very high, request rate, like APIs, things like that.

Mark:48:00

Very small requests has a whole different architecture. Right? So there are different networks within the CDN that you get mapped to depending on the product that you are purchasing. So, yes, we have 400,000 servers. But in this particular product, we have, like, you know, 150,000 servers or this or you know?

Mark:48:22

And some of them do multi duty as well.

Max:48:24

So Well, API is also different. Right? You get into API or, you know, I mean, traditional REST API or GraphQL or at least different things. Right? And part of it is caching.

Max:48:34

Part of it's authentication. Part of it's, like, being able to apply, like, a WAF on top of it and protect your API, you know, from the Internet and or it's like bot. Right? You're dealing with bot. And and how do you, you know, whole whole other can of worms.

Max:48:49

Right? Like, there's all these other things with it.

Mark:48:51

Yeah. Well, we so what's interesting about what we've done with the cloud workflow that, you know, we bought this this cloud. All of a sudden, we got all these new toys to play with. You know? It's like, woah.

Mark:49:03

We're we're not committed to just the products that we build, and we go through our process of productizing. It takes, you know, Akamai, 18 months is generous. When I say from idea to finish, it's probably longer than that, in order to have some bundled solution that we can resell to many customers. Right? Now that we have the cloud, we've got partners like, Harper DB, for example, is a distributed database.

Mark:49:32

You familiar with them? I mean, we're talking, like, milliseconds of time to replicate a database that we're using for I mentioned token authentication earlier. We're doing common access token authentication with band token banning through Harper DB through for some of our larger broadcasters. It's pretty cool. And it's it's like it's not like sample sets.

Mark:49:57

Like, I'm not taking some of the requests. I'm taking every request for a TS segment and looking at a token and looking for abuse. And if I see the abuse, I can ban it or I can allow it. I mean, that's that's pretty cool.

Max:50:12

I mean, I'm it's just the cat and mouse game with this stuff. It's just so wild. You you know, so okay. Right. And then the line gets blurry because then you go into into live.

Max:50:23

Right? And there's, like, live live, which is acquisition. And then there's, like, live in the sense of, like, we're gonna broadcast something, but we're gonna record it at the same time and then put it into our VOD catalog and then deliver it. Right? Like, so, like, these lines are always kinda like it's like whack my microphone here.

Max:50:42

Lines like you know, it's like everything kind of always, like, lines up and gets blurry. Right?

Mark:50:47

Yeah. The so to me, live and simulated live are almost the same, but have different problems. The simulated live has more problems on the edge and in the network, the middle layer because it

Max:51:02

What do you mean by simulated live?

Mark:51:04

Like what you just said. It's prerecorded, but it's also playing out as if it were live. So, I don't know why I can't Tom Tom Brady's roast, for example. Right? So that was prerecorded, but it was

Max:51:21

But it has an airtime that that content goes active at whatever time. Yes.

Mark:51:25

Yeah. So so that is what I call simulated live. Right? So it's actually being kinda distributed as if it were live. It's probably VOD content might have even been going through encoders.

Mark:51:37

I don't know if it I don't think it was. But, it was probably distributed and then played back. Right? So that's

Max:51:43

So traditionally, in in a broadcast situation, in a in in in the simulated live, right, what you're talking about is, they've they have the content and they've gone to, you know, a broadcast location. Usually, like, these are these, you know, boxes with lots of satellite dishes on the roof. Right? And they put the tape in the drive, and they a lot of time, they press the play button. I mean I mean, basically.

Max:52:05

Right? You know, they press the play button, and it just it just goes out of the satellites and goes up and comes back down, then everybody takes that signal and, you know, shoves it onto whatever mechanism that they're doing to distribute it to the eyeballs.

Mark:52:16

So you're right. You so I kinda blurred the lines there a little bit. So a a true simulated live is still a live workflow where you're doing encoding. It's just coming from instead of a camera pointing to a person talking, they've actually prerecorded and probably edited and all that stuff, and then they're seeing it through live. What I'm talking about in this scenario and simulated live was probably the wrong con wrong word for it, but it's it's a VOD event that acts like a live event.

Mark:52:44

So it's a flash crowd event. Does that make sense? Okay. So, you know, I mentioned earlier the release of, you know, last season of Game of Thrones or House of the Dragons or whatever. At that time, you have what any CDN might look at as a DDoS attack, but it's not.

Mark:53:05

So it's like, well, what's going on? So you mentioned WAF. We have to be very careful. Right? So we can't actually deny requests because it looks like a DDoS attack as it might actually be a flash crowd.

Mark:53:18

Right?

Max:53:18

No. I mean, you know, you can you as a customer, right, when we were deploying infrastructure and then refreshing cash, we would go through and we would effectively spider our sites to, like you know? And we would try to do that for multiple geographies and then get, you know we kinda figure out, like, what you need to do in order to get your eyeballs the best experience, and and there were different mechanisms doing this. But, you know, I mean, a lot of that's also exposed via API. You know, CDN, you can invalidate objects with the API.

Max:53:51

Are people not, like, issuing, like, hey. You know, this is you know, by the way, you should cache this content now, you know, and get ready?

Mark:54:01

Meaning for the simulated live workflow? Yeah. So there there are sophisticated customers and there are unsophisticated customers. That's the best way I can say that. There are people that think through that and know.

Mark:54:20

And, you know, that not to change to live, but, you know, for live, there are a whole bunch of things that we used to do back in the day where, yes, you can put a live stream on your home page and have it autoplay, but but that's probably the worst thing you can do. That means anybody that goes to your home page is gonna automatically consume streaming resources that they may not otherwise have done. Right? So make them do a call to action. Make them click on it.

Mark:54:47

You know, little things like that that, you know, that are are probably natural for you and I, but not so natural for someone that's hasn't been in the live streaming game as long. Right?

Max:54:59

And for the record, I hate auto playing video on websites.

Mark:55:03

Yeah. And and especially when their audio isn't muted. Right? That's that should be a crime. Yeah.

Mark:55:11

So but so going back to the simulated the true simulated live. Right? So when you have a prerecorded thing and you're going through live yes. There is an encoder. That encoder is sending the stream to, a CDN.

Mark:55:29

In most cases, it's being pulled. So, you know

Max:55:33

I mean, from a simulated live in that and a and a from a simulated live and a live live, it's just what's the source. Is the source of tape deck or is the source of a broadcast trailer with, you know, a mixer. Right?

Mark:55:43

Yeah. But there there's also things that you can do ahead of time, like condition it for ad insertion, for example. Right? You can have if you're if you do a simulated live where you've recorded it, you can have breaks that make sense that are like, oh, we're fading black. We're doing this.

Mark:55:59

We're we're gonna go into the ads. Right? So they can do that. And live football, whatever, you do the TV timeouts and, you know, the flags and the you you kinda have to have all those ad insertion mechanisms in there. Server side ad insertions are definitely the popular way to do it now.

Mark:56:16

So, you know, those types of things, remember when I said it hasn't gotten easier? It's gotten more complicated? You know, like, how do you monetize the content? How many ads do you play? Do you only play a bumper ad?

Mark:56:27

Do you play mid roll ads? I mean, there's all kinds of

Max:56:30

Are you are you playing ads based on the market that the person is actually watching in? You know, do you have a content distribution, you know, requirement that some some contractual deal with somebody where they control the ads? You're just giving them the source? Do you give I mean yeah. I mean, again, you know, what are we actually engineering here?

Max:56:47

Video delivery or or contracts? Right.

Mark:56:49

And and then, you know, you've most of the people that come to me for live event scale type consulting, they'll be like, hey. We're worried about this stream being scalable. I'm like, okay. Well, let's talk about the stream. That is important, but what about everything else around it?

Mark:57:06

Right? What what's what is your where's your player? Where is your landing page? How are people getting tokens if you're adding handing out tokens? Do you have an API server?

Mark:57:15

Is it protected? You know, where's your DNS? Is DNS protected? That's the way I would wipe out a stream. I would take that DNS.

Mark:57:23

Remove the address. So there's all kinds of things that people don't realize that go into these live events. You know, the ad insertion model. Like, how is it being monetized? Is it a paywall?

Mark:57:35

Are you having people put in an email address? Once they do, what are you doing after that? Are you authenticating every user?

Max:57:41

Can you process a a lot of credit cards in a very short span of time? What's your fraud scoring?

Mark:57:45

Yeah. I mean, there's there's a lot of things involved. And the most important thing for a live event is you gotta have some kind of observe observability. Right? You gotta have monitoring involved because you don't want the hair on fire conversation and say, oh my god.

Mark:58:04

I got 5 complaints on Twitter. That means everything's broken. Well, wait a minute. That means that everybody in Iowa is having a problem. Everybody else is fine, so it's okay.

Mark:58:15

You know, there's there's levels of, there's levels of of of issues that you have to be careful of. Right?

Max:58:24

Akamai doesn't play in the, like, the LVC distribution world. Right? Like, these are other companies that are going and getting the signal from broadcast trailers and doing satellite distribution or or picking it up from, you know, downlink stations. I mean, so you the customer is still gonna have to go out and figure out, like, what their signal source is if they wanted to redistribute. You know?

Max:58:45

Or is it or is it at this point, can they come to Akamai directly and say, hey. You know, we wanna you know, we've got we've got the content rights to serve this traffic that's gonna be going on this day. You know, we need end to end here.

Mark:59:01

So yes and no. So we now that we have a cloud, we can, talk to our either partner base or even customers or companies that do that. Like, I'm talking point to point. I'm not talking about monetizing it for them or getting the source. But, you know, like, Zixi, for example, is deployed, on Akamai's Linode platform as well as others, so cloud platforms.

Mark:59:31

So getting content acquisition from a venue that has, Zixie, capability is fine over Akamai. And then you've got companies that want to have an encoder on-site, like a Media Excel, for example, on prem, and then send that over, terrestrial means rather than satellite to Akamai, whether it be our cloud or whoever else. But it's usually our cloud that we're because we're involved. We're talking about it. So we're talking to Media Excel and saying, hey.

Mark:01:00:05

We've got this, you know, event going on. Call it outside lands or whatever in San Francisco. I don't know if that's still going on. You know, we'll have an encoder there, and we're gonna push it to our LA server or San Francisco server, and then we'll do the, the encoding from there. So because we have this, you know, this set of toys, you know, you ask if we do this, Yeah.

Mark:01:00:30

We didn't, but doesn't mean we can't. Right? And and also doesn't mean that we're qualified to actually have those conversations. We ask we actually have other partners that actually have those conversations probably better than we do.

Max:01:00:41

You know, the major telcos all got into the space of doing content distribution. And they came up with yeah. Because, you mean, you've got fiber running into these facilities. And you're like, okay. Well, you know, here's another service we can offer.

Max:01:00:53

And that service is usually, you know, we can take and, I mean, what was it? We could take feed, you know, an SDI interface out of whatever it is and plug it into our network and then and then, you know, endpoint that anywhere else on our network. Right? But then the problem with that is, you know, the telcos that had encoding options, I mean, you still have to convert it into a signal that something else can deal with it. You know?

Max:01:01:17

And, you know, and for a long time, that was just can you convert it into RTMP, and then somebody else can at least take and transcode RTMP into something else and then shove it out the other way. Right? But you get rid of all that encoding platform, and you say, oh, you know, all we do is we just take SDI in and SDI out, and you gotta figure out the next thing. It's like, wonderful. Now, you know, like, what are we gonna do with this thing now?

Mark:01:01:38

Yeah. So so we've worked with, our customer or our partner, Attem for those types of workflows. So, Attem and MediaExcel, they both have, you know, point to point gateway solutions. So they'll deploy a server on-site, whether that customer most of our customers that want something like this, they're already using somebody like the TAM or MediaExcel. So they're the ones that tell us, hey.

Mark:01:02:04

We wanna use MediaExcel. So we're like, okay. Great. Guess what? They've been qualified to run on the node.

Mark:01:02:09

So all we have to do is deploy, you know, their their encoder. So once we get the signal, you you send it SRT to their to your encoder, and then we'll get it over whatever. You know, it'll be SRT probably from their point to our point in the node, and then we distribute it however they want to encode it from that point. Right? That's the hero encoder solution.

Max:01:02:32

With the exception of, you know, major geographic calamities. You know? Train fire in a tunnel takes on all the fiber going into Philadelphia. Right? Like, bridge collapses with us with a whatever and takes all the fiber going between, you know, Santa Cruz and San Francisco.

Max:01:02:50

You there's still there's still capacity problems and capacity planning. You know? This game has changed because the amount of what's shifted from traditional distribution of satellite or cable or over the air broadcast now is moving on to network Internet based delivery. Right? So it changes your capacity requirements.

Max:01:03:11

So, you know, we there's there's, like, that mess of it. Assuming that that's all perfect, you know, you see these big events take place that are live or simulated live that fall over. Right? And what's the experience from people when they say it fall over? They can't get onto the stream.

Max:01:03:26

They can't actually show. They end up with a low bit rate, you know, pixelation, you know, outside of just, like, you know, some aliens landing in the entire world turning at the same time to watch the same live feed. Like, what's actually going on there that, you know, has has caused that or can be, you know, done differently to alleviate that? Maybe it's a better way of phrasing it.

Mark:01:03:53

So there's there's a whole bunch of reasons in the workflow, you know, from first mile, middle mile, and last mile. The most common use case for what you're describing, and I think what we're both kind of alluding to is, unpredictable traffic that you didn't plan for. And then when that happens, you get, you know, either signal loss or rebuffering or pixelation, you know, whether that's at the last mile because neighborhoods are all watching, you know, the same thing at the same time. I personally experienced that last Friday. I was watching, you know, from my house, and, I was like, oh, this is really pixelated.

Mark:01:04:42

My buddy across the street said, should I are we allowed to mention what we're talking about?

Max:01:04:49

I mean, I don't I don't think we're getting in trouble. I mean, look. Live events are a different animal. Right? You know, World Cup, big events, major sporting.

Max:01:04:57

I mean, so it's usually major sporting events. Right? And, now, I mean, we don't know what the actual stream numbers are of the, you know, Tyson, Paul fight. Right? And I don't know if we'll ever see that disclosed.

Max:01:05:07

I've I've seen numbers everywhere between, like, 60 and a 100000000 people watching that, which is from a scale of that stream, like, is bananas. You know? Like, absolutely bananas. Like, oh, we had a you know, like, is it a 100,000,000 people watching that? Like, I'm I'm sorry.

Max:01:05:24

Nobody you you wanna talk about capacity planning? Like

Mark:01:05:27

Yeah. We usually lop off the last 0 when a customer tells us that they're gonna have that many. Right? It's like, okay. Yeah.

Mark:01:05:32

Sure. You know? But I

Max:01:05:35

mean, even at 60,000,000, that number is absolutely bananas. Like, it's

Mark:01:05:39

But they have a captive audience. They have subscribers. Right? So it's that scenario where the event is not also broadcast on TV like the Super Bowl, where people have a choice. Right?

Mark:01:05:50

You know, you have cable providers that not everybody has cut the cord. But for Netflix, you know, there is no cable channel for Netflix. So they definitely have 100% of their users streaming content. Right? And then they have, they didn't do pay per view.

Max:01:06:07

I just looked it up. The reports reports for 2024 were a 124,000,000 people watching the Super Bowl. So 20 24 broadcast Super Bowl. And that's that's broad I mean, broadcast and streaming. That's everything.

Mark:01:06:21

So that's both.

Max:01:06:21

Yeah. Global distribution, broadcast and streaming.

Mark:01:06:24

Yeah. So when you're talking 6065 to a 120,000,000 for Netflix, that's a huge amount. I mean, look at the demographics. You got people that are older that, you know, saw Tyson bite the ear off, and you got younger generation that have been growing up watching the influencer. Right?

Mark:01:06:41

So, you know, you got this this it it was the perfect storm for a lot of users. I think they got caught off guard. You know, and and whenever you build out now Netflix has super smart people working there. This is not a dig on anybody. It could have been anything in the workflow.

Mark:01:06:59

I have no idea what actually happened.

Max:01:07:01

If you have by the way, just as a little segue and pause here. If you haven't read any of the Netflix engineering, blogs on how they've actually architected their, their their their edge compute apply you know, their edge appliances and you're interested in this stuff, like, go to their engineering blog and read it. I mean, what they're doing is phenomenal. I mean, it's it's and, also, we talked we were talking earlier about, like, these, like, simulated live or, like, hot cache events. A lot of intelligence they built into their infrastructure to understand, like, okay.

Max:01:07:28

We're gonna release an app. You know, this season is now gonna drop on this date, and, like, let's warm the entire network. I mean, this really you know, it it's it's it's incredible. It's incredible engineering, and it's very sophisticated.

Mark:01:07:40

Yeah. And from a VOD perspective, they definitely got it figured out. Right? They haven't been doing live that long. So maybe and, you know, again, I think they have very smart people.

Mark:01:07:52

I think this is probably one thing that it's making the news everywhere because it happened. Right? But there there are certain things that you do in this type of scenario, old school stuff. Like, oh my god. We're rebuffering.

Mark:01:08:04

Let's lop off the top bit rate so that we can lower the actual, you know, usage in terabits per second, right, that we're actually having to to, to satisfy. And there could have been something called the waiting room. You know, when you go to a site that already has more people than they expected, you're like, hey. Hold on. We're full right now.

Mark:01:08:25

Just watch this, you know, watch this other other material while we wait for a slot to open up. Right? That kind of thing. We've been doing that for years. Right?

Mark:01:08:35

And then there's, you know, the player logic, like you say, you know, what do you do if you hit some issue on ACDN, multi CDN? Netflix doesn't do multi CDN. So, you know, this is a scenario where overflow might have helped them. But, you know, I'm sure they're gonna learn from this, and they're gonna make sure it doesn't happen again. But there's those are the the learning

Max:01:08:59

They've been doing live events, and they're gonna have more and more live events. I mean, the question is, do you have anything that, you know, generates this much traffic and at what pace do you get back to this point? Like because, I mean, if you're if you're only talking about, like, 10 to 20,000,000 people, which is still, like, bananas numbers when you start talking about, like, a live stream being consumed. Like, that's a big that's a really big event. You know?

Max:01:09:25

And we used a minute ago, like, we have too many terabits of traffic. We gotta lop off a bit rate. Right? You know? I I'm solely well, when I got on the Internet, I had a 1200 baud modem guy.

Max:01:09:35

Right? Like, I mean, now, of course now, of course, we're out there architecting and deploying, like, 800 gig backhaul for people, but, like, it's still, like, like, conceptually, like, oh, we just put we just brought up 800 gig backhaul between these two markets, you know, like, x, you know, like, every single time at, like, 1200 baud modem, like, flashes through my brain. You know?

Mark:01:09:54

Yeah. I mean, these numbers are crazy. Like, you know, we've I I can't tell you any numbers, but we do the, the streaming for the Amazon or, yeah, Amazon, Football on Thursdays over the Akamai platform. So, you know, we we get, definitely, they've you mentioned reserve capacity. You know, we have customers that have to reserve capacity because if not, it's a shared platform.

Mark:01:10:21

You know, we're gonna have other people that are impacted, so we have to make sure that we get that reserved capacity. Even on a CDN, there is no such thing as unlimited capacity.

Max:01:10:31

Who knows what the meetings are going on right now inside of Netflix talking about this? And by the time, you know, people, you know, like I mean, this will become long tail at some point. Like, 2 years down the road, maybe there's a there's a there's a retrospective that comes out and somebody talks about it. We we get a lot of details out of it. But even in the cases of Amazon streaming, you know, football, I mean, they have a platform that they may or may not have something to do with.

Max:01:10:57

Right? And they're still out there making sure they have capacity for it. I mean, at some point, like, you just we talk about, like, risk and resiliency a lot, you know, in in operations. And, like, a lot of times, like I think this is a difference between trying to have, like, trying to eliminate single points of failure in in any of, like, spaghetti diagrams to, like, actually having, like, an appropriate level of resiliency and and, you know, disaster recovery planning. Like, what do you do?

Mark:01:11:25

Yeah. It's all about building the run book so you're not making hair on fire decisions. Right? I mean, you you gotta basically predict the worst, and you're always gonna miss something. So

Max:01:11:34

I mean, but that's the other side of it. Right? Like, if your steady state is 20,000,000 streams and all of a sudden you've got a 100,000,000 going on. Right? Like, how do you 5 x your capacity and capacity planning?

Max:01:11:42

Like, nobody's nobody's planning for that. Like, the days of the days of being like, oh, we're gonna have, you know, triple or our city state. You know, we're going to we're going to, you know, we're only at 30%. Like, nobody's running their network at 30%. You know, everybody's, you know, it's just not cost efficient.

Max:01:11:59

You know, you got to be you gotta be way up there. You know?

Mark:01:12:01

You know, Akamai is not the only CDN that has, we call it the velvet rope. But, it's it's basically a capacity restriction that all of our customers are under. You know, when you sign up for a contract with us, we put the velvet rope on based on what your contract size is. If you hit that, we are gonna, prevent traffic on your behalf because you've hit it. Right?

Mark:01:12:25

And there's a little bit of a buffer that we allow people. But that's to protect all other customers. Right? We can't have one customer be the noisy neighbor on a CDN. It just doesn't happen.

Mark:01:12:36

So in Netflix case, they didn't they don't have they don't have the ability to do that. They're they're their own customer. They're their own CDN. They're not failing over to anything else. So, you know, it's

Max:01:12:50

So what do you think is going on right now? I mean, like, if if, I mean, we we we can, like, segue and not use, like, Netflix as as a as a thing here. But, like, you know, Monday morning, you come into a meeting and this has happened. Right? Like, what's going on at that conference table, and what's what's the you know, you you turn around, and first thing is, like, okay.

Max:01:13:13

Late night cable television is talking about us. All the you know, social media is talking about like, this is, like, the worst case scenario in terms of I mean, it's good and bad. Right? It's good because you're relevant in the zeitgeist. Yeah.

Max:01:13:25

Your stock's up. You're relevant in the zeitgeist. You had a huge event, but then, like, people are like, oh, it didn't work for me. Did it work for you? Like, everybody's has a story of somebody being like, oh, it worked.

Mark:01:13:32

It didn't work. Like, some everybody's talking about this thing. And, you know, like, the the first the first part

Max:01:13:32

of that becomes, you know, if that if that capacity, that stream demand was, you know, 5 times what your normal delivery is, is there even a conversation around capacity increasing on your network? Or do you just immediately just say, like, what do we do from here?

Mark:01:13:58

Yeah. So first, they're probably having a root cause analysis conversation. Right? Like, what is the cause of this? Was it some stupid configuration that we botched?

Mark:01:14:08

Or was it really legitimately more traffic than we expected? Which everybody thinks that that's what it is. So let's go with that for now. Then you then you come up with the real hard numbers of how often is this gonna happen. Is this gonna be our trajectory?

Mark:01:14:28

You know, with Akamai, we set records every year, and then it becomes the norm for the next year. It it's just it's constantly going up. So do you have the build out capacity conversation now, or do you do a hybrid of both? Do you build some out but also have some plan to say, what do we do if we get into this situation again? Do we have a failover to another CDN?

Mark:01:14:55

Do we go and lop off the top, bit rate? Or do we have a velvet rope or not a velvet rope, but a a waiting room scenario. Right? There's there are ways to prevent that high capacity event, but is it worth it to the business? Do they want those eyeballs to see that fight?

Mark:01:15:14

Absolutely. Do they want those eyeballs to see it at 300 kilobits per second? Absolutely not. Right? That's not good.

Max:01:15:22

They saw it, but they're not happy.

Mark:01:15:24

Right. Right. So, you know, there's there's ways that they could have prevented this by maybe having some time splicing. Right? So right now, they did it in Texas and where they had, you know, everybody on the East Coast.

Mark:01:15:38

I'm on the East Coast. I had to stay up till 11:30 to see the main fight. That's past my bedtime. So but I did it.

Max:01:15:47

It's hard to keep me up until 11:30 at night nowadays. Like, it's just

Mark:01:15:51

But everybody on the on the West Coast, you know, it was 8:30, whatever, in the pre fight, you know, around 6, whatever. But that's that's what you're playing with. Right? And then what about the people around the world? I mean, they they watched it.

Mark:01:16:05

They got up in the middle of the night and watched it. I don't know. I don't know what the numbers are, where their audience was from. But But this and that's

Max:01:16:11

a big problem. Right? Because, like, Olympics, like, US, you know, distribution of Olympics, I hate it because, like, oh, by the time you even wake up or even they think about broadcasting anything to show you what's going on with the Olympics, it's like, I've already all the clips are already on social media.

Mark:01:16:24

Like You've got 5 gold medals. Wow. Let me go back and look at where we got those.

Max:01:16:30

And, you know, the media, media shifted. I mean, it's been shifting for years, but now, you know, the value in media is really being expressed in live events because it like you have to sit there, you have to watch it, you have to consume it, you have to receive the advertisement or the pay gate that you've put into it in order to get it. Like, there is this, you know, we're back to this thing where, like, it's it's happening right now, and and and it has a draw. Right? So you're not gonna time shift a live event like that that, you know, causes different I mean, this was the thing that's crazy is, like, over the air distribution is massively efficient.

Max:01:17:05

Right? You have one tower and, like, everybody that's within 20 miles of that tower can or 50 miles of that tower, however tall it is, can get that signal. Right? Like, it doesn't matter if it's 1 or 50,000,000. You know?

Max:01:17:16

Like

Mark:01:17:16

You bring up a good point. So as as you know, all streaming technology today is unicast. Not all, but most of it is unicast.

Max:01:17:25

I rolled my eyes again. I'm sorry. I couldn't help it.

Mark:01:17:28

I know. So are we gonna go back to the multicast? We need to. Right? We need to have a multicast like network, whether that's true multicast or some hybrid, where you have a signal coming in like we talked about SRT.

Mark:01:17:42

And then what do you do? Do you distribute that over SRT to different geographies and do the encoding there so that you have ABR multibit rates there. So you're not distributing, like, 5 megabits per or sorry, 15, 20 megabits per second across the Internet, for all these users, or are you doing less? I don't know. Is that the answer?

Mark:01:18:02

Who knows?

Max:01:18:03

I mean, multicast has been a thought exercise on CCIE exams since, like, the dawn of time at this point. And, like, how many like like, it's it's this technology that, like, oh, it's great on the surface, but it's never ever been deployed. You know? Like, in a me you know, like

Mark:01:18:18

But the reason it hasn't is because you have routers that aren't, unicast or multicast compliant in the middle. Right? It's the Internet. But if you have a network like Netflix that has their own network, maybe you can probably build your own multicast network and make it work. I don't know.

Mark:01:18:35

You know, or CDNs may collaborate and get together. I don't know. There's a way to do it, but we haven't gotten there.

Max:01:18:42

Well, I'm thinking about, like, modern architecture here. And, you know, you still have the access layer. Right? So what is the actual access layer? And the access layer is going to be a telco.

Max:01:18:51

It's gonna be a cable company, you know, an MSO, something. It's you know? And CDNs and, content providers ultimately are trying to get access to that access network. We're getting a little redundant here. But you go to a carrier neutral facility.

Max:01:19:09

You try to peer, you know, you're a large distributor of content, so you get settlement free interconnection. You get nodes directly on their network, like, whatever those different things are. But you're still at the mercy of that access network delivering on on what your technique is from whatever that edge point is across that, you know, access network down to and, you know, and and this is I mean, this was the point of, like, peer to peer, you know, what peer to peer is trying to solve or one of the points that peer to peer tries to solve. Right? I think I think the real solve with most peer to peer for streaming is more about, you know, can you improve the quality of video delivered to an a location that has, you know, you know, limited bandwidth or, you know, that middle mile is such an issue.

Max:01:19:54

But then you but then you introduce and and inject delay into that signal and distribution. And that's a problem as well with live events. Right? You know, like, maybe if they're only delivered over the Internet, it's not a problem. Netflix, we solved your problem.

Max:01:20:10

Since you have no since you have no television broadcast, you need to release peer to peer across your network, and it will solve this problem.

Mark:01:20:18

Or you I don't know what their I think their segment size is, like, 5 seconds, but, you know, maybe they could get away with larger 5 like, larger segment sizes, but I don't think I don't think they had any problems with 5 seconds. I don't think that was their problem. But, you know, the latency conversation even with, you know, with the Netflix fight, you know, were they trying to keep low latency and why? Because the only people that would have noticed are the people that were in the venue. Right?

Mark:01:20:45

I mean, it's one of those things where

Max:01:20:48

Okay. So if you were doing a live event but did not have, traditional broadcast at the same time, so it's Internet only distribution, you know, you're competing with social media in terms of, you know, that delay. Right? I that's that's what you're talking about. Because the people at the event don't know what's going on.

Max:01:21:09

So, really, you're just talking about the people at the event that are that are on the Internet with social media distributing whatever's going on is your competition. From an architecture and delivery standpoint, delaying that feed 5 seconds, 10 seconds, 15 seconds, what does that do for you from a CDN and distribution and quality standpoint?

Mark:01:21:29

Good question. So there is a point of no return that doesn't get you much, back. But if you have low latency when I say low latency, people doing it with, like, 2 second segments. The number of requests back to the CDN from the player is compared to a 10 second segment is 5 x. So you have 5 times the request from every single user coming back and looking for for new re news, segments.

Mark:01:22:02

Because basically, the traditional buffer, in a player will buffer 3 segments, and then, you know, live segments will come in as they generate, and then you'll go back and get more. You'll go back and get more. So if you extend the life or the the length of those segments, you have fewer requests per second. We're not talking about bandwidth. We're talking about actual requests.

Mark:01:22:24

And those requests start to add up going all the way back through the chain to the encoder or to the layer that's after the encoder. And that stresses the entire workflow for all these users. Right? You get all these requests going back and forth. So, small segments is bad for very, very large events.

Mark:01:22:44

So longer segments, if you can get away with them, better.

Max:01:22:47

Longer segments would introduce I mean, with buffering though, you're gonna increase the total bandwidth requirements. So, like, are you just are you on a teeter totter here? Or

Mark:01:22:55

Yeah. A little bit. That's why I said there's a point of no re there's there's a point where that that benefit doesn't really make sense. So that's why I kinda stopped at 5 seconds. They might be okay.

Mark:01:23:04

I mean, you go up to 7 to 10 seconds. Any more than that, it's not gonna work because you get large very large files at the higher bit rates. But then you do, you know, chunked transfer. You know? So there's all kinds of techniques in there.

Mark:01:23:19

But there was another angle I was trying to think of, and I've lost my train of thought now. Darn it. Oh, so when I was watching this event, I noticed that, when I got buffering and stuff, it would buffer, but it would still play. But the DVR window was the entire length of the event. So what I noticed is when I went across the street to my buddy's house after I stopped watching it here, I went over there and I was like, what are you doing?

Mark:01:23:50

I've already seen this. He was buffered a lot further back than me, but he wasn't getting any of the pixelation. So he was actually watching at least, segments that had already been cached. So he wasn't getting any rebuffering or no, pixelation. And I said, do me a favor.

Mark:01:24:09

Fast forward to the live edge. And as he did, then we got rebuffering. So to me, that means that that, yes, they were at capacity, and they were saturating the, the actual requests that were coming in to the servers at least that we're ready. And I'm in I'm in Northern Virginia. So

Max:01:24:27

Yeah. They should have a few servers there. Yeah. That's interesting. Right?

Max:01:24:30

So then

Mark:01:24:31

So you're almost you're effectively precaching a live feed with your DVR window. Right. So,

Max:01:24:39

could you force that and a player and have a config flag and just basically force all the players to go behind time a little bit?

Mark:01:24:47

You want some users actually to be at the live edge to fill the buffer, and then you have subsequent users maybe start 15 seconds in. They'd never know the difference. Right? I mean, most people in a live stream, you can have that buffer scrub, but you can make that go away. I mean, you don't have to have it.

Mark:01:25:07

But that's all configurable.

Max:01:25:09

And then you just I mean, again, you're just talking about what you're dealing with in terms of how far behind can you go before social media becomes an issue.

Mark:01:25:16

Right. That's why I said 10 to 15 seconds is probably all you need.

Max:01:25:19

And, again, if it's not being broadcast live. Because the big issue with latency on on live is if your neighbor across the street is watching the same thing and they're cheering for goals being scored that you're behind. And and by the way, that happens between, like, satellite and cable and brand broadcast television. I mean, the big secret here is if you have the option to get an antenna on your rooftop and you wanna watch live sports, you get up much better. Everything's better.

Max:01:25:44

You get it's the encoding is better. You know? The

Mark:01:25:47

Go backwards. We're bet yeah.

Max:01:25:48

Yeah. But you get better bandwidth. You get better signal quality. You get a better you know? And and and it's actually, you know, like, the, you know, the delivery of it's it's much better.

Mark:01:25:56

Yeah. So the it's funny that the the broadcast customers that I talked to when they say, I want low latency. I'm like, well, do you really want low latency? Because you're already 8 to 10 seconds behind with your cable broadcast. They're like, yeah.

Mark:01:26:10

What what do you mean? I'm like, well, if I give you 3 seconds, and then you're gonna have people complaining that we're ahead of your broadcast. So, yeah. 8 to 10 is fine.

Max:01:26:22

This specific use case is so specific. Right? Because now you're talking about somebody who is the content owner or content creator, who then also has their own network for distribution that is trying to use their own network for distribution and not lever anything else that's available to them on the Internet to give them capacity to, like, deal with these events. Right? So, like, outside of, you know, the rematch in 2 years, like, doing this as well.

Mark:01:26:51

The smart move for them would be to have a CDN relationship as a fallback. I mean, it it's the the lowest hanging fruit method that they could do. Right? If by chance they have an event that hits this order of magnitude again, that outweighs or outpaces how fast they can build out, then they have a safety net. Maybe they never use it.

Mark:01:27:15

That's fine.

Max:01:27:16

And the level of nerdery that, like, the people that actually would care or say anything is, like, so

Mark:01:27:21

Level of people that would know. I mean

Max:01:27:24

No. There's there's there's so few of them that would be talking about it, and nobody would care. I mean, that's the thing. Nobody would care. So, I mean, you know, it's a good news for everybody else that wants to do live events.

Max:01:27:34

Right? Like, you don't have to deal with this. Just go out and and, you know, have multiple providers and and a plan to, you know, really, I mean, what I'm what I'm regurgitating is, like, the first thing is is do you have a way of actually knowing what is real, what is not? Right? So observability, Not like so and so's, you know, cousins, nephews, niece is, like, telling you that they can't watch it, you know, in the cabins, in the mountains somewhere in Peru.

Max:01:28:04

But real observe observability and and then the ability to make a decision based on that traffic and and a plan and and some sort of resiliency. Right?

Mark:01:28:12

Yeah. And it I would say for most of the live events that we do with the larger broadcasters, like Super Bowl, for example, there's 4 months of planning involved. Because for Super Bowl, it's usually a different network every year. So the last time they did it, everything's changed. We're like they're like, yeah, we're just gonna use the same encoders and the same profiles.

Mark:01:28:33

Let's talk to you about that. So, you know, there's there's a lot of planning, and there's also different technologies that have come out. Even in the last like, we have seasonal things that well, seasonal, yes, every year. But mostly, you know, everything happens in every 2 years, like Olympics and things like that. That enough things change that like, for example, when we're talking to IBM about the masters, the masters isn't until next spring.

Mark:01:29:01

But we're already talking to them about, hey. What are you doing? How are you doing it? What's changing? Things like that.

Max:01:29:07

And they have a great digital experience. I mean, you talk about, like, as an annual event that just really does a good job. And and, you know, compared to, you know, I mean, I used to watch MotoGP and, like, horrible experience. You know? Like, it's just I haven't watched MotoGP in a while, so I should probably I should probably go back and see what they're up to lately.

Mark:01:29:34

But Yeah. I mean, the the all all of the technology changes so often that you have to have some kind of plan in place. And, you know, the runbook doesn't change that often. It's what do we do if we have more capacity? Have you checked all the different points that touch the user experience?

Mark:01:29:54

Every API, every ad call, the player, the landing page, the DNS, the streaming technology, all of that needs to be scrutinized. Right? Basically, looked at with a fine tooth comb or microscope or whatever you wanna call it. And then when you have all that mapped out, then you're like, okay. Where did we come close last year or last event?

Mark:01:30:21

Where where where are the places that we're most concerned about? And then what can we do to either mitigate that in real time or mitigate that now? Right? So what are we gonna do when that happens? And then most important, who's gonna make the decision?

Mark:01:30:37

Because when when everybody's like, oh my god. We're falling over. And then somebody goes, reboot the server. And they were like, wait. Who's authorizing that?

Mark:01:30:49

It's like, wait a minute.

Max:01:30:51

Boy, there's stories there for me too. And the other good news, right, is by the time you figure out 720, then, you know, you have to deal with 4 k and you figure out 4 k and an 8 k. And by the time you have 8 k, then everybody wants to go from 422 to 444. And then you you figure that out. And then, you know, the next thing comes out.

Max:01:31:07

Right? So and and the, you know, the jump of each one of these things isn't like you know, it's it's a logarithmic jump in terms of capacity and and bandwidth requirements. Right? And, you know, I'm upgrading my house right now to 2 and a half gig of the to, you know, every station. Right?

Max:01:31:26

And, I mean, the joke about it is the only thing that actually needs it is my Xbox, but doesn't support it yet. You know? So, like and the only reason that needs it is because when I sit down to play a game, it's been so long that I wanna I have to download some stupid update. Right?

Mark:01:31:38

Yeah. And and, you know, it's funny. I used to talk to, you know, news agencies that wanted the secret sauce for their mojos. And if you don't know what a mojo is, mobile journalist. Right?

Mark:01:31:48

They're like, how do we get them to upload content directly from where they are? Like, well, today, if I had that conversation, I'd say, Elon Musk is selling a satellite dish for a $150. That's what I would do. $150 a month.

Max:01:32:02

We, I mean, I had, I mean, going back in time, you know, productions filming on the islands in Hawaii, right, where you were renting satellite transponder time and you had a window, like, you had this from from this time to this time, and that was it. And, I mean, you wanna talk about, like, do not mess that up because, like, if you miss that hour, like, you don't get another you've gotta wait till the next day.

Mark:01:32:27

Yeah. I'm sure you have well, there's there are horror stories that I've heard of, not that I've been a part of, but, like, did we talk about the time zone for that? We need like, really?

Max:01:32:41

That's not the problem. The problem is not the time zone. The problem is daylight savings time zone shifts. And by the way, it's not uniform because different locations and countries change time zones on different days of the month. And then some also have decided that they don't go a full hour.

Max:01:32:59

They maybe only go 30 minutes. And so if you've ever dealt with time, you you learn to hate it very fast because it's like

Mark:01:33:07

Even in our own US, it's I was camped out at, Lake Powell, and I was like, hey. Let's go to Antelope Canyon. And it was right around the time of Daylight Savings. They're like, oh, but on Navajo land, it's not they don't observe, but but they are within Arizona. And then, like, what the heck?

Mark:01:33:24

I left my I left the RV, and I arrived before I left. This is nuts.

Max:01:33:32

When I was in 5th grade, my my teacher was on a cruise and crossed the international date line and skipped her birthday the entire day. She skipped the entire day.

Mark:01:33:43

That's awesome. And so that's, like, worse than a leap year.

Max:01:33:49

I mean, you know, you tie you do get to these, like, weird things where it's just like, you know, I mean, this is great. You know, I, the linode the build out will be fun to continue to watch. Right? Because you know and, also, I mean, you know, bless containers. Like, really, everybody wants to talk about, you know, serverless and functions and this and that and the next thing.

Max:01:34:17

But, you know, bless containers and how and and actually making workloads portable and, like, in a meaningful way that you can actually take and move this stuff around because, you know, that's that's, like, the Nobel Prize of tech in the last few years or for most infrastructure planning.

Mark:01:34:31

Yeah. We've got a, I'm gonna draw a blank on the company name. Starts with an a. One of our qualified, excuse me, compute partners, Avisha, that, specializes in container migration because they've basically normalized all the Kubernetes infrastructure across all the clouds. So if you wanna run a cluster that crosses multiple clouds, no problem.

Mark:01:34:59

And then migrate from one to another, no problem. So, I mean, making a containerized workflow, we've got one encoding partner, I think I already mentioned them, that runs everything in a container. So they've got, a 2, what's their algorithm? If you send them a job that is 2 +1 divided by 3 or whatever it is, it'll spin up that many servers to actually process those jobs. So each server, is capable of encoding to has 2 slots for it, and then it'll automatically ratchet up and then automatically ratchet down.

Mark:01:35:44

I mean, it's just it's just so easy. It's it's done.

Max:01:35:49

You know? And that's and then you and then you get back into the financials, right, where you like it to decide. Do I wanna put this stuff on bare metal somewhere and pay for it and just have, you know, 5,000 cores just sitting because I'm gonna use it all the time or use it enough that it makes sense, or do you just go rent it? You know? And and, I mean, I I go I go back to that in terms of, like, optionality.

Max:01:36:09

You should be making decisions because you've decided to make a decision, not because you got forced into something. Right? Like, that's a that's a very different it's a very different way of dealing with the world.

Mark:01:36:19

Yeah. Do do I send all my content to encoder.com and just get it back all done? I'm done. Whatever. But encoder.com.

Max:01:36:26

So we got the Super Bowl coming up. We got a little low before the Olympics. We've got World Cup coming up. I mean, those should be world World Cup. I'm excited for it.

Max:01:36:37

And the, you you know, North America. I mean, that I mean, you wanna talk about, like, nightmares for, like, people dealing with this stuff. You're gonna have to set up broadcast at 8, 10 locations. You know? And, like, oh, by the way, like, Mexico City.

Max:01:36:53

Right? Like

Mark:01:36:54

Well and and we got people that are in Latin America that are, actually very successful in selling out our Brazil, deployment location. It's like, you know, that's where they want their content. Now the dirty little secret is, you know, a lot of the users, even in Latin America, are getting their content from Miami because the connectivity is better. But that's gonna change. Right?

Max:01:37:19

It's getting better. I mean, you know, 15, 20 years ago, I mean, the peering inside of Brazil, like, you know, one one comp you know, one network wasn't talking to another network, and they were all peering in Miami, and it was crazy. And so we learned as operators really quickly just to put everything up in Miami because it just it was better. But, it's nice to see it's nice to see Latin America building out in the same way as seeing, like, Africa, you know, like, meaningful builds going on.

Mark:01:37:47

Yeah. And and, you know, it's a it might not make sense to put the encoding workflow everywhere in all these edge locations, but all the API call, all the little things that you really need, you know, the distributed architecture for, like the token authentication. You know? Maybe that should be closest to the end user and distributed. That's the common access token thing, with the Harbor DB thing.

Mark:01:38:11

So we've got servers in Argentina. We've got servers in Colombia. I mean, there's those types of things are important. You know, you talk to Claro and America Movil, and they have their own, CDNs, you know, that that wanna interact with a little little overflow with Akamai. I mean, we've got really good solutions for them with this Linode thing.

Mark:01:38:32

So I'm excited to see how that all plays out too.

Max:01:38:35

Mark, thank you very much. It's been fantastic.

Mark:01:38:37

Yeah. Thank you. Thanks for having me.

Max:01:38:39

Really appreciate the time. And, I mean, hopefully, there's a rematch, you know, like, I'd like I, I mean, there was plenty of hype for it. So

Mark:01:38:50

Yeah. I don't know. I mean, the more and more I watch that fight, I was thinking, okay. Which fighter would have benefited at all if there was any KO? Right?

Mark:01:39:02

Like, if Jake Paul would have knocked out a 58 year old, he would have gotten ridiculed by, oh, you just knocked out an old man. You know? And And

Max:01:39:10

if he'd been knocked out?

Mark:01:39:11

Yeah. And if he'd have been knocked out, it his career would have been over.

Max:01:39:16

So I mean, this goes back to, like, you know, Floyd Mayweather's, like, deals and I mean, like, this isn't, like, this isn't real. This stuff is just entertainment. They've just they've just, you know, I mean, like, the economics are different from from a Netflix deal versus a pay per view deal, right, in terms of dollar dollars received. But I'll

Mark:01:39:37

tell you what fight I was impressed with is the pre fight. The the those girls were, like, toe to toe. I was like, holy crap. This is like Rocky. Real Rocky.

Max:01:39:46

Well, that was I mean, that's the difference between the exhibition and the real thing. Right?

Mark:01:39:50

I mean, I would have gone down after 2 of those punches, and they were, like, oh my god. That was

Max:01:39:57

amazing. Again, exhibition versus real. Right? You know? Now you're talking about just a different thing.

Max:01:40:02

I mean, you know?

Mark:01:40:03

Well, you know, I think Tyson brought the crowd for sure.

Max:01:40:06

Oh, yeah. I mean, they both I mean, it was I mean, this was definitely the spectacle of the of the year. So, you know, congratulations to everybody involved, including including everybody that's locked in a room right now trying to figure out what what to do next.

Mark:01:40:18

Yeah. I mean, it's, you know, sometimes a failure is a success, but I don't know. It might be a little too too far to reach for this one.

Max:01:40:28

Alright, man. Thank you, Ken.

Mark:01:40:29

Alright. Thanks a lot. I appreciate it. See you, Max.

‍