First Principles Thinking for Bitcoin Exchanges with Zach Dexter of LedgerX

Transcript

Ledge: Hey, welcome to the the Frontier Podcast by Gun.io. I’m Ledge, your host. Today, my guest is Zack Dexter, CTO of LedgerX.

LedgerX is the first federally regulated exchange and clearinghouse to list and clear fully-collateralized, physically-settled bitcoin swaps and options for the institutional market.

Zach led the ground-up buildout of LedgerX’s integrated exchange and clearinghouse platform, leveraging open source, cloud technology approaches including container orchestration and hardware security. He’s also a member of the CFTC Technology Advisory Committee’s cybersecurity subcommittee.

Before LedgerX, he worked as lead software engineer at a Y Combinator-backed startup and as an independent software consultant.

Zach, thanks for joining us today.

Zach: Hey, Ledge, how is it going?

David: Excellent to have you, man! Before we went on recording, you and I talked about some of the key trends that I’m seeing in various CTO interviews. Some of those regarded processes Are around the merging of ideas in the product and engineering space as well as the operations and DevOps space along with development.

So I thought we’d dive in there a little bit. You had opinions about how and where engineers need to spend their time when they’re building a highly scalable sort of MVP product but that needs to have institutional push.

Zach: Let’s take some examples from the bitcoin space. There are a lot of people building so-called “institution-facing exchanges” and to try to please the institutions, they’ve been copying features from the institutional markets in the equity space. And those include FIX APIs and low latency servers that are co-located here in New York and New Jersey.

It turns out that that stuff is not actually worth copying. There are reasons why things came up the way they did in certain industries and those reasons are often legacy.

There was a time when you could get edge by positioning your server a few meters closer to the source of the data but, in recent years, we’ve seen exchanges like IX and others come alive and people are starting to view equal access as a plus as opposed to low latency.

So we see a lot of people who are building bitcoin exchanges copying these existing features.

We really need some new thinking. Bitcoin is a new asset class; the other assets are new. There’s no reason to be copying these structures.

So in LedgerX, we’ve adopted more of a product-design approach rather than this kind of product-management approach where you take a look at the industry and say, “Oh, look! Here’s what people do to build an institutional exchange. Let’s go do that.”

We’ve started with first principles thinking instead.

David: Define “first principles thinking.” I know you, guys, have a regulated space so it’s a little bit different. Where does that fit in because you do have an entrenched sort of hierarchy, at least, from the reporting and data side that you need to satisfy?

Zach: I think “first principles thinking” is all about asking the simple questions. Simple Question #1 is “Are we allowed to do this?” And it turns out that that’s where most bitcoin exchanges today have gotten tripped up. They’ve decided to eschew regulation entirely, in many cases, or go to jurisdictions where regulation is weak or not as strong.

And we have that. We said, “Well, we are allowed to do this, to list and clear these products, if we register with the appropriate licenses and go through the appropriate processes.”

We actually did that. It sounds simple but, today, nobody else has done it. We’re still the only clearinghouse that is listing and clearing physically settled bitcoin derivatives. Period.

It’s just really interesting because this space is not exactly new these days.

And that decision came out of the first principles approach which is “What do we need to do to do this legally? If there wasn’t a lot of hand-wringing should we get registered? Should we not?”

The requirement is very clear and we took that difficult path. So it’s all about just asking the simple questions first.

David: I presume that customers really want that especially on an institutional level maybe far more than they want the magical features that are just built into some of that legacy stuff.

Zach: Yes. Those features are not very magical. The FIX is not exactly a magical protocol. A lot of the existing approaches are really complex for not very good reasons.

A good example is the fee schedule in some of these exchanges. I think you have to have a PhD in astrophysics to decipher the fee schedules on some of the bitcoin exchanges out there.

Ours is real simple. There’s no market maker rebate or after 6 p.m., they get a 25% discount if liquidity is 32% less than yesterday’s moving average and all that stuff.

People do these crazy equity-style fee schedules because there are complicated fee schedules in the equity’s world which is not a reason to do in bitcoin.

Just start over.

David: Extrapolate that to anybody working on their sort of new market approach, a disruptive approach to a market, how does anybody think about their business model, kind of that way and from a technology implementation standpoint?

Zach: In my opinion, throw everything out. Anytime you’re entering a new market, you have an incredible opportunity to build something from scratch, and it’s a lesson that can be taken far beyond the world of regulated finance. It can be taken into product design.

When we were designing our one-to-one negotiated trades which has been a huge hit, the development side, we did something that and nobody had ever done. We took the virtual pit model which was something we came up with and implemented that straight away.

How did we know to do that?

Well, we thought, okay, many years ago in Chicago, there was a trading pit that was very active; it’s sort of dead now. But people would be yelling at each other with bids and offers and everyone was kind of screaming. There was this chaos but all the traders knew each other. They really loved trading that way.

We thought we’d import that to the bitcoin world.

So there are a lot of lessons to be learned from first principles thinking on the design side, not just business strategy and regulations.

David: That’s fantastic! You made a comment that I love and I want to dig into it and it’s “Don’t build for the customers whom you think you’re going to have but build something simple that works well enough” which I think most people would resonate with in a little bit lean startup Agile kind of language.

How long does that last where you kind of say, “Don’t build for future customers?” Can you keep up that disposition in your product iteration?

Zach: If you’re building for the future customers, you’re really building for a customer that doesn’t exist. You’d want to go for the customers who do exist, the ones you have today. If you’re speculating on what people are going to want, that almost never works ─ at least, in our experience.

For us, early on, we thought that we would be catering to very large, high-frequency traders and they would be doing millions of orders a second because that’s what happens in the equity’s world.

And it turned out that wasn’t the case at all.

I think once we started to focus on the customers we did have, the early bitcoin adaptors, and build what they wanted ─ and even more than that, we took the sort of an Apple-style approach and gave them things they didn’t know they wanted.

The early bitcoin adaptors were not in the trading pits of Chicago in the 1970s and 80s. They were just growing up, maybe looking and moving to Silicon Valley to start their first tech jobs.

This was a customer base that was not familiar necessarily with the concept of a pit.

David: So it’s understanding the customer you have and not presupposing the customer that already exists. So you really carved out a new market area to go with the new market, I guess.

Zach: Yes. I think that most of the other people in this space are looking at importing not only the ideas from the old spaces in finance but also the exact customers.

And that’s not the case. In every new market, there’s a new set of buyers and sellers out there. And certainly with the bitcoin especially, I mean, the players are not just the same as they were in traditional finance.

The finance industry is being changed completely and that includes all the. It’s not just the hedge funds. It’s not just the ETFs. And it’s not just the banks.

It’s now everyone. The democratized finance is starting to be realized in a lot of ways.

David: Talk about the deep dive technology stuff. You’re a tech guy. What have you designed? What does the stack look like? What are the critical components that actually make it scale and work?

Zach: We did really something simple. We’re actually the only exchange that I’m aware of to do a real-time, netted collateral model where we net out the requirements of your entire portfolio in terms of what collateral you have to post on a per-order basis.

We’re doing this very expensive real-time computations before every single order against the matching engine. Most people will do those computations maybe once a day. They’ll have something like that.

So that’s all in C++.

The overarching principle of our system is that it’s vertically scaled. It actually doesn’t scale horizontally. Most of the systems these days, especially web apps, you can just spin up.

Now, you have the ability to support ten thousand more users. And that’s great. But, in the exchange model, it’s one of the few models that is actually centralized which is a little bit because bitcoin is about decentralization.

For that reason, we just use C++. It’s very fast and very stable. We really have no interest in any of the newer languages that are still maturing.

David: And how do you keep and how do you look to scale them? Since it’s not horizontal, do you run into any bottlenecks that, over time, the volume would increase?

Zach: At the moment, actually, most of our trades are these large one-to-one negotiated trades. We don’t really have a lot of high frequency people on the platform, but there are some.

And for that reason, we actually just need to have more raw so that means more CPU and more straightforward optimizations.

We go into the C++ standard library documentation. We use hash maps in a lot of cases. And we make a tradeoff where we decide, okay, we’re going to use more memory and we’re going to consider memory to be very cheap because it’s really cheap to go out on a server on Amazon with 64 gigs of memory. And we’re going to cache everything in the nudging engine and get really fast operations and really good use out of our CPU at the cost of using a lot of memory.

David: And so, it’s all public cloud then. You said Amazon.

Zach: Yes.

David: I just talked to a company that is still on the private data center train and loves it that way. Do you see any tradeoff there that there’s ever a reason to go back that direction?

Zach: I guess, if we wanted to spend more money for worst results, then we would go to the data center. I really do not understand why people do this. I mean, we’re a regulated exchange and clearinghouse and we tell this at CFTC and they like this answer.

The public cloud is great. There’s absolutely no reason for us to be renting racks and racks of data center space in Secaucus or wherever. There’s just no justification for it. We’re not looking to onboard our participants who are going to be doing millions of codes a second and need a few nanoseconds shaved off somewhere.

That’s not a thing in this space. It doesn’t make any sense. I’m not sure why people have done that.

David: Definitely not in your space, yes. And I tend to agree. So let me pivot for a second to one more last cool thing that you talked about when we were off-mic. I had noted that in my previous interviews this season, there seems to be a on the operations stack, that “full stack” doesn’t just mean frontend and backend anymore. It also includes DevOPs.

And you made an interesting comment that not just DevOps but, in fact, just regular operations is, in fact, coalescing as well. And I thought that was a cool story to talk about ─ how you address your regulatory and reporting needs in non-traditional ways that are driven by technology.

Zach: All of our regulatory reports are uploaded automatically and generated automatically, and reviewed automatically. We use runtime just like people who are building rocket systems to detect error.

Usually, in the past, if you’re running an exchange clearinghouse, you’d have a whole staff of people responsible for “operations” and that could be anything from market to putting together reports and regulators to turn the market on and off.

But, in our case, we’ve used a whole host of cloud-based tools and custom software to automate essentially everything.

So our operations staff is really the platform itself and if there’s any case where that is not true, we’ll work really quickly and hard to operate that particular automation.

So our platform runs itself twenty-four hours a day, seven days a week, and I think people should target ─ across all industries ─ that instead of having Excel spreadsheets operational activities, you should look to automate wherever you can.

And we’ve made heavy use of tech contractors and for that and it’s really worked that well for us because, then, we can dedicate the resources that, otherwise, would go to traditional “operations” to more software development.

David: And so, you think that’s the thing that any business can look at and should invest in heavily.

Zach: Yes. I think it’s almost like product managers and operations ─ I don’t want to say they’re categorically things of the past but maybe have your engineers talk directly to customers and do a little bit of product design Apple style; maybe err towards that especially in finance these days.

And I would personally always err towards automating operations and nailing things first 0:16:47.5 at the human level which [crosstalk]

David: ─ seems to suggest that engineers can no longer be the engineers that I remember twenty-five years ago who were all in the basement behind the screen with the lights off.

Zach: Absolutely!

David: ─ humans.

Zach: Yes, full-stack human ─ that’s a great way to put it. Our engineers are on our support tickets. They’re in a meeting with customers. They’re really trying to empathize with what customers are doing.

And that also gives us a better sense of how we prioritize. If somebody says, “Oh, I’m a product manager and I think XYZ is really important,” that’s a lot different than “Hey, I’m a customer and I feel a lot of pain here” and having an engineer really viscerally understands that. So we try to connect the two as much as possible.

David: The empathetic engineer! Awesome! Thank you. Zach, anything finishing comment? Where can people check out you and your work?

Zach: It’s ledgerx.com. And for finishing comments, I guess, I would say it to the moon.

David: ─ and back.

Zach: And possibly back. Other way, you can hedge your risk up or down.

David: Thanks, Zach, I appreciate it.