How do you move 50TB per day to 15,000 global engineers? with Eric Klinker of BitTorrent

Transcript

Ledge: Eric, thanks for joining us. Really cool to have you on today.

Eric: Ledge, it’s just great to be here with you guys.

Ledge: Thank you so much. Can you give a two or three minute background of you and your work? I know it’s many, varied and lots of cool stuff, so maybe the short version.

Eric: Sure. Currently the CEO and co-founder of Resilio, but probably better known as the long-time CEO of BitTorrent.

We grew BitTorrent from just a few hundred thousand users to, at one point, approaching a quarter of a billion users all over the world just doing some amazing things there.

Resilio is actually an extension of that experience. We’re taking that magical, decentralized technology, and we know the amazing things that can do, and we’re trying to put that in the hands of as many large enterprises as possible. That work has been going on for a couple of years and it’s going really, really well.

Ledge: There’s so much in the BitTorrent world, I bet most of our audience is very familiar with what the peer-to-peer and BitTorrent world was and is. How do you explain to the more business user? Just set the baseline a little bit.

Eric: I think the word you’ve got to use when you describe BitTorrent is, it’s different. Almost everything about the internet since its inception… Maybe in the early days it was more decentralized, but call it since the mid-nineties when the web took off there’s been this never-ending train that’s going towards more and more centralized.

You see that in properties like Google and Facebook and Twitter. It’s a well-trodden topic now, but BitTorrent started out and it stayed different. It was very decentralized. It was all the power at the edge. Kind of like what the internet was originally meant to be.

Through that power, we were able to do some amazing things in a completely decentralized way. There were no servers involved. There was no company of thousands. We didn’t have any data centers. Yet at one point I was moving probably 40% of the internet’s traffic volume with just 20 software engineers in a kind of run-down office in San Francisco.

That kind of power, you don’t see that in too many places, where one engineer is responsible for like whole percentage points of internet traffic. He could check something into the code base and change the nature of the internet.

That’s what decentralized power can do, and there aren’t too many examples of it but BitTorrent is certainly one of the more interesting ones, shall we say.

Ledge: It was all about moving data across that distributed network. Roughly speaking, I understand the protocol, to break up huge files into tiny pieces and have ever peer on the network have pieces of that.

Eric: That’s right.

Ledge: I did make prolific use of it, but we’re not going to talk about that.

Eric: That’s right.

Ledge: One place I’m interested what you think about this, because one place decentralization is coming up a lot is now resurfacing in the blockchain conversation. Are there parallels there?

Eric: Yes. Incidentally enough, I left BitTorrent about two years ago but stayed actively involved with the management team there. We just recently sold the company to a blockchain project.

Obviously, the blockchain space sees the synergies. I saw it from the very early days.

Unfortunately, blockchain came a little too late for BitTorrent. We were doing many of the things that you see being incubated in the blockchain world long before it was cool. Every one of these projects we probably had something very much like it at BitTorrent at one point or another.

What we had lacked at the time was this ability to decentralize the incentive mechanisms through these tokenization efforts. Now that that’s far more mainstream, there is an effort to marry blockchain and BitTorrent together. It’s underway right now and you can go check it out at that site.

There’s still hundreds of millions of users on BitTorrent. I think you’ll see BitTorrent plus blockchain being in one of the more high-scale applications of decentralized blockchain applications out there very, very soon.

Ledge: So tell the story now of Resilio. How are you taking that to the more commercial and enterprise environment? What had to change? What are the different, more accepted paradigms when you kind of hop up the ladder to big organizations?

Eric: Well, why did we even think this was a good idea to begin with?

It started with one of these applications that we had built that was decentralized called BitTorrent Sync. I don’t know if you used it, but you could think of it like a version of Dropbox but without the cloud. Completely peer-to-peer. One computer, one phone, one NAS connecting to another. You would create these very small private cloud instances.

We have a dozen, million, maybe 10 million users or so of this application, but one of the very first things that happened when we launched that was, it was a very technical product and it appealed to a very technical audience. These guys all had day jobs and they said, this thing is awesome and it’s changed my life and [sound break 00:05:28] at Caterpillar or my job at a very large fast food chain, maybe McDonald’s or KFC.

With that inbound demand, it was immediately obvious and we said, you know what, BitTorrent is a consumer product. There’s no way the CIO’s going to sign a deal with BitTorrent, but it’s clear that this technology is perfect for these enterprise applications, especially at the edge.

So if you had a very far flung operation like large retail – I think McDonald’s has 36,000 global restaurants – and they’re all in parts of the world or many of them are in parts of the world where maybe the connectivity isn’t great but, hey, BitTorrent’s been operating in these environments since the inception. In fact, that’s one of the reasons why it became so powerfully successful, was that it could operate in the poor network environment of the early internet.

Invented in 2001, became kind of popular in 2003, it’s a long time ago and the internet doesn’t look anything like it does today where you can actually stream Netflix. You couldn’t do that back then and the only network that could deliver Netflix was the post office. Except BitTorrent. BitTorrent could make the internet do what no other application could deliver.

So that power, especially if you’ve got a lot of vehicles out on the network. They may come or they may go. We have some of the largest fire departments. Cal Fire is in the news lately and they’re a Resilio customer. Large retail, I’ve just mentioned a couple of very significant fast food chains.

Then the other flipside to this argument is, well maybe I’ve got really, really good networks but I’m pushing just a tremendous amount of data. More data than you can imagine.

These might be like large video game makers where they’re doing CI/CD, they’re constantly building very large software and they’re doing it on a global basis all over the world. Maybe they have different offices in almost every geography. How do you move 50TB a day to 15,000 engineers? That’s also something that Resilio and BitTorrent technology can do.

These are just a little bit of the applications that we’ve seen. There are many, many more. We do see this trend of computing that was at the cloud beginning its long march back towards the edge. You see it manifest in terms like fog computing, grid computing, edge computing is the latest. These are all words that describe this trend.

There’s going to be a lot more data on the edge of the network than there is in the cloud, and it’s probably going to be ten times the volume that we have today. Those trends are just going to continue.

We think this is the exact perfect technology to ride that wave, and we’re super excited to be taking it out to market.

Ledge: I was a BitTorrent Sync user, and I had it on all my devices and, yeah, it was very cool how you could move stuff around in your own private cloud and there was no interstitial centralized authority there? So it made a lot of sense.

I imagine then when you’re talking about rolling that out, it gets more and more powerful when you have tens of thousands of nodes on the private network and on all the devices that go with one of those larger organization.

So, in fact if I understand the protocol correctly, it would get faster and faster and faster by nature of distributing across all those nodes.

Eric: Absolutely right. That’s what makes it different, right? Every other technology is all client server, and as you increase demand the supply has to increase in kind otherwise the application is going to fail. But when you have a peer-to-peer application, every bit of demand also brings with it a lot of supply.

It is organic in that nature. That’s what makes the protocol exceedingly scalable for all these very large data applications that enterprises is currently trying to grapple with.

It’s perfect for a world where data is going to get bigger, and of course everybody believes that it will, and it’s also perfect for a world where there’s going to be many more computers. Guess what? Moore’s Law marches on. Computing is now so cheap that it’s going to be prolific. It’s going to be everywhere. Those are both massive drivers of scalability challenges that every enterprise will have to face.

Peer-to-peer and distributed technology is the only way to solve those problems.

Ledge: Do you use a node on each of the… Let’s say you have a big public cloud, or private or hybrid type of server implementation. Do you also run them in the cloud to boost up the bandwidth of the overall…

Eric: Yep.

Ledge: So you could centralize essentially on your own systems, and use your full edge computing capability out in all the devices then.

Eric: Absolutely. That is correct. You spin up 10,000 instances and they can all be running a little piece of this protocol to make the cloud even more powerful than it is today.

It’s also an easy way to scale up demand in the cloud or to move data to or from the cloud. We focus on the edge because that’s the problem that really hasn’t been solved, but right now most of the data applications are moving from the edge into the cloud. They’re trying to figure out, well, how do I migrate this on-prem application to the cloud? Probably 50% of our customers are doing things like that. How do I get this data that used to be on the edge in both the edge and the cloud, and I want to use both? It’s, again, another perfect instance where distributed technology fits the bill very nicely.

Then once you have multi-cloud, or cloud-to-cloud, or I want to have cloud redundancy I need to keep these applications alive and well on multiple vendors so that I don’t get locked in. That’s also something that, again, distributed technology, if it’s cross-platform, is perfect for.

One of the big drivers of our business is migration to the cloud.

Ledge: So you’re talking to 10,000 engineers right now who are always interested in playing with the newest, best, most difficult problem solving technologies. How can engineers get into and deploy around this type of application who are not immediately working on enterprise problem?

Eric: It’s one of these areas where we need to do a better job of incubating the community.

Probably the best place to start is the Resilio Sync product, which is the later version of that BitTorrent Sync product that we had talked about earlier. It’s still being produced, and it’s very easy to access. It’s free to download. You can begin to use it just like you could BitTorrent Sync. That will give you a flavor, instantly, for what kinds of workflows the technology would be good for.

To engage with the enterprise product, it’s a bit of a beast. It’s all about automation and you have to have fairly strong scripting skills. If you’re a good PowerShell user of the API, it’s all API driven if you need it to be. It’s really all about automating these data driven workflows.

If I need to get data from A to B, I don’t think about that in terms of the data, I think about that in terms of the workflow. I’ve got to get it from A to B, and I might use rsycn for that. If rsync fails, I’m looking for an alternative to rsync or I’m looking for an alternative way to architect my distributed storage. Maybe if DFSR is not working the way it used to then I need more modern approach to that age-old problem of, how do I keep the data in all the places that need it when they need it?

These types of automated workflows are going to be a little more difficult to engage with, but we need to do more outreach to the community and more developer based, knowledge based activities.

We have some properties out there where if you search for them you can find it and a lot of community around it, but it’s fairly hidden and you’re going to have to do a little bit of digging. The Reddit community, the knowledge base itself, has a fairly active user base that anybody could join.

Properties like this, it’s something where if you’re in the know you know, but how do we get the word out there? This is the age-old challenge of go-to-market strategies that at Resilio we’re like BitTorrent. We’re a bunch of engineers and we’re trying to learn how to do sales and marketing and it’s always a task.

Ledge: That’s fantastic. Well, we just took a step there so…

Eric: Yes. Thank you.

Ledge: Absolutely. I ask a question of all the tech leaders and engineers that I talk to, and it’s based on, obviously we’re in the business of finding and evaluating and vetting the very best software engineers. We take that very seriously. We have a very rigorous multi-tier approach and it’s very difficult to get validated and get through into our network and work with clients.

We think we’re pretty good at that, but I always like to make sure that we’re on the cutting edge of doing that. So I like to ask everybody I talk to, what are your heuristics for identifying and hiring the best engineers? How do you measure them, and what’s the process?

What really identifies the great engineers that you want when you have many choices?

Eric: That’s an outstanding question.

We’ve kept a fairly consistent process starting all the way back from BitTorrent days.

The kind of engineer that we were looking for at BitTorrent was a little bit different. We were looking in particular for protocol engineers, and these are not as common as some of the other flavors that you have out there.

They are all great engineers but the way we would filter them, it would probably start with Brahm, first and foremost, who was fairly well integrated into the community. He loved coding challenges. He was always a puzzle guy himself. We’ve kept this process as…

If you think about a problem in terms of a puzzle, and if you can solve it, then you’re pretty good at thinking about a protocol that might solve that problem very generally. Those are always the things we’re looking for. How do you generally approach a problem? What’s your thinking process?

We were more interested in how you went about it rather than did you get the right answer at the end of the day, because the folks that we were looking for needed to approach the market very differently.

If you built up in a data center world and you were very used to scaling data centers, then we would throw you into BitTorrent where you needed to deal with something that was completely unreliable. How do you make use of an edge note that might not be there? That’s different than the skills that you would typically see in probably 99% of the jobs.

In some sense we were looking for a different kind of talent. I think the blockchain world has probably broadened this skillset quite a bit, and I think the talent pool would probably be much richer today if we were looking to staff up BitTorrent circa 2005 than it was at the time.

Those are the ways that we would evaluate talent. It’s just give them a coding challenge and seeing more importantly, how do you approach it? How do you approach the problem? If you approach it in an insightful way or a unique way, then that’s a pretty good sign that you’re going to be a very successful protocol engineer.

Ledge: So I need to ask, what was it like and how do you think about – because so many entrepreneurs face this – being ahead of the curve?

You developed a thing that was extraordinary, and people who got it really got it but then that’s just that idea that the world wasn’t ready to implement it at scale.

Looking back, how do you process that? Any advice for people who are essentially ahead of the curve?

Eric: The first thing to point out is, you’re going to make a lot of enemies if you’re successful in this.

Tiny little BitTorrent, 20 software engineers in a small office in San Francisco, we had an outsized presence on the internet but we were really trapped between two gigantic industries.

The first industry, which is obvious is Hollywood, was not ready to embrace the internet, and certainly not ready to embrace it in a way that completely took away any kind of control that they had over the content. Their entire business model was built around maintaining tight control of the product and the internet began to loosen that grip quite a bit.

They’ve since recovered quite nicely, I think, and Hollywood is doing better than ever – and everybody knew that it would once this platform had been embraced.

The other industry which you maybe didn’t expect is the telecom industry. Both of them looked at BitTorrent as pure evil. I often pictured myself as this tiny little pebble being ground between two gigantic tectonic plates of industries.

Fortunately, we had to solve both these problems otherwise BitTorrent wasn’t going to be successful. The telecom problem was manifested in debates like network neutrality and others, where clearly they were not ready to deliver the kind of capacity that an application like BitTorrent was demanding of their networks. That required a lot of CapEx to build.

They would say, throw about terms like, “Well, way too much traffic. We’ve got to stop it. We’ve got to block it. We’ve got to curtail it.”

We were able to solve that problem. Both are really political problems but we solved the telecom problem with a technical solution, which is wonderful and exactly the kinds of problems that we liked to solve.

We said, well, you know what? BitTorrent is such a distributing protocol, what you’re really complaining about is congestion. Let’s just eliminate that problem. Let’s make the protocol be less than best effort. And if any other application wants to use the network, then BitTorrent will magically get out of the way. The cost of your network is incurred only at the moment where you incur congestion, which means you have to increase capacity and build out the network. Otherwise, the network is effectively free. If there’s excess capacity, then why wouldn’t you want your user base to have access to it and use it to the maximum extent possible, because it’s going to deliver a better consumer experience to them?

So, by changing the protocol – again protocol engineers – we were able to solve a ginormous political problem with the telecom industry.

Of course there was a regulatory battle that was raging as well with network neutrality, and it has had its ups and downs since then, but it’s really not been driven by BitTorrent because at the end of the day we solved that particular problem for them.

The content industry, that was a trickier one. There wasn’t really a technical solution to what was effectively a business model problem. The business models needed to evolve, and they did eventually, to accommodate the internet. That’s why you see Netflix being probably one of the most highly valued content properties on the internet today.

Ledge: Yeah. You really drove a whole lot of disruption there.

I wonder what that’s like, looking back at that, being just 20 engineers in a room and saying, wow, we really caused quite an earthquake there.

Eric: Right. It was a unique experience, for sure. Again, I think it drove so many wonderful stories and experiences. Even though, yeah, I probably don’t have… You know, I’m only 18 I probably look like I’m 75 – it definitely had an impact on the appearances and it was stressful, but it was also just an amazing time and a great, great experience.

Ledge: Well, Eric, so cool to have you. Thanks for telling those stories. I know the audience is going to love it.

Eric: You bet. Happy to share, and it’s been great to be with you, Ledge. Thanks so much.