Scalable video capture and playback plus building an international team with Oliver Friedmann of Ziggeo

Transcript

Ledge: Hey, Oliver. Thanks for joining us. Really cool to have you on.

Oliver: Thank you, Ledge. Glad to be on your podcast.

Ledge: Could you just give a two or three minute background story of yourself and your work, so the audience can get to know you a little bit?

Oliver: Sure. I’m originally from Germany. I grew up there and went to school there. Studied computer science and mathematics. Did a PhD in computer science, Theory of Computation.

Did part of my studies then during my PhD time in New York, and after completing my PhD I always tried to find an excuse to come back to New York City and kind of do my own thing.

So, I started a company in New York City with an American co-founder called Ziggeo. We do an API for video recording and playback, primarily. Similar to say Twilio or SendGrid. We make it super-simple to plug in a component – in this case video recording – and playback in the browser as well as in mobile apps to enable any developer to use that. Make it as easy as adding, basically.

Ledge: Fantastic. So the point is then that, if I’m a developer, I cannot have to worry about if I want to build video recorder in my app or web or mobile or otherwise. Is that the point?

Oliver: That is the point. There are a lot of details that you need to take care of if you’re trying to build it yourself. For instance, there are obviously different types of browsers and versions, and they all need to use different types of recording technologies and different types of streaming technologies. You end up with different video formats that, once you get them, you can’t really play them back on any other device. They might be rotated, for instance.

What we do, we basically give you some UI that you can add to your website or to your app, as well as the cloud which will receive the video recordings, transcode them to whatever formats you need and different resolutions and so on, and then also stream them back to all the devices where you want to play them back.

Ledge: Very cool. Talk about some of the engineering challenges there. I know that this is critically important and so many apps are going this next direction. Video is definitely the next frontier.

Talk about, what engineering challenges did you have to overcome as you’re building the app?

Oliver: There’s a bunch of them. Starting with the client side, as I mentioned, there are many different browsers and we have customers that, for instance, still have their own customers who want to use Internet Explorer 8.

So, we need to make sure that our software works across a variety of different devices and browsers. Whenever we introduce new features we need to be sure that they don’t break anything on these old components. There’s an extensive amount of testing that goes into our software, both automated as well as manual.

On the server side, obviously people are mostly concerned about latency. How long does it take videos to upload? How long does it take for videos to be transcoded? You have to come up with an interesting pipeline on the server side to do that as quickly as possible, and also to be able to scale that as rapidly as possible. For instance, if somebody blasts out their marketing campaign where all of a sudden 10,000 people are trying to record themselves at the same time, then obviously you need to be able on the cloud side to have enough bandwidth to do that. To transcode still as quickly as before, but also to scale it down afterwards so you don’t pay for it. So we don’t pay for a crazy amount of servers that we’re running in the background.

Ledge: Yeah, understood. You really have some vertical and horizontal scaling concerns there from the infrastructure standpoint. How do you solve that to make it not just this overwhelming cloud bill every month?

Oliver: Exactly. The original prototype that we wrote was all built on AWS. We extensively used EC to autoscale, quite traditionally. But in our newer version, we’re trying to use as much of the serverless stack as possible.

We factored our application to use mostly stuff like Lambda for API endpoints that don’t use any heavy I/O. Have specific in-house built streaming servers that take care of these special protocols that we need to record from different types of devices. Then, for all the background processing, we have our own transcoding pipeline. We also use serverless features on AWS for that.

The cool thing about Lambda is that you can boot up these functions incredibly quickly, but they have the disadvantage, on the other hand, that they’re very restricted in terms of resources that you can use on them – both in terms of time, memory and storage and so on.

The way our system works, it kind of gets into the data and decides whether the resource constraints that we have on Lambda are good enough to do an actual transcoding, for instance, or to split a file, for instance into two or three and run three function instances. If that works, we do it. That can be done incredibly quickly.

If we decided, okay, it doesn’t work – the video, for instance, is too large – we launch instances of Fargate, which is a Docker execution version of AWS where you can also scale infinitely, at least in theory, with the disadvantage that the boot up time is much longer. That can be like 30 seconds.

The neat tradeoff here is, we only need to do that with videos that already have a certain size. The transcoding, in that case, will take considerably longer than just 30 seconds, so the boot up time in that case is not really a problem.

Ledge: You really had to draw on all your computer science algorithmic chops to pull all this off, and this is a pretty complicated problem.

Oliver: Yeah. It definitely comes in handy.

Ledge: How do you think about scaling your engineering organization? I can imagine that you really are going to get a lot of growth out of a proposition like this. You were the founder, and now you need to think about growing a company.

Talk about that experience.

Oliver: Exactly. We started from the ground up as a highly distributed team, so very much in line with what your guys are doing. We have engineers all around the world who work full time for our company but are not located in our office in New York.

That definitely posed a challenge. How do you communicate things? All these kinds of things. But it’s a lot of fun to build a company like that and not have the overhead or the necessity to always come to an office to meet your staff.

Ledge: That’s absolutely right. Yeah. Share some tips, because it’s not just remote. Distributed, when you’re dealing with worldwide time zones and standups or whatever your management paradigm is, most of the stuff isn’t about technology. Obviously, I imagine you have your build pipelines or CI/CD or something of that nature, but what tools do you use to overcome the human problems there? Collaboration and just time. You don’t want your guys across the world to have to be up all night.

What’s that look like?

Oliver: Yeah, you’re absolutely right. I think it depends on which part of the business people are involved in, how you structure it properly.

For instance, for our support team we have a pretty big team of support engineers who need to be really hands-on when customers reach out and need 24/7 support with our software. So, we mostly organize them on Zendesk and on chat on Slack. Our Head of Support coordinates everything that’s going on. Whenever a new ticket comes in that poses a new challenge, this is being discussed usually on Slack and then put into an internal library of particularly interesting tickets and problems on Zendesk.

Our Head of Support is mostly concerned with organizing these interesting tickets for future use, so staff can be onboarded as quickly as possible. That’s mostly how our support team works internally.

Now, our engineering team, what I found worked particularly well is, everybody who is on a remote team has a very clear objective and it doesn’t overlap crazily with what everybody else is doing.

For instance, in our case we have somebody working on everything concerning iOS and somebody working on everything concerning Android. Somebody concerning everything that we have on JavaScript and HTML and CSS in terms of the embeddings. Then all the frontend on our website; somebody for the backend, somebody for DevOps.

Obviously, they all have some overlaps but they can work reasonably well on their own. For planning we mostly use Asana, and we have weekly team meetings with everybody and then smaller meetings with the subgroups concerning the specific task of a week.

Ledge: How was it for you, going from… You probably were developing everything all the time and then trying to scale out into a team where I imagine that you had to put on more of like your management hat. That wasn’t your discipline. You studied all the way as far as you could possibly study computer science. You built, I’m sure, B1, all that stuff without other people. Then you have to turn into a worldwide manager.

What was that process like? Was that natural?

Oliver: That’s definitely an interesting process. I had some experiences coming from university working with other students and teams, and I did some teaching as well, organizing some teams, working on stuff. So, I had some experience. Obviously, as you say, it’s always hard to let go of code that you’ve written and say, okay, now somebody else has to run with it.

I think it worked reasonably well. I was very lucky to find really good talent. Whenever we bring in somebody new, I’ll try to onboard that person somewhere on the outside. Something reasonably easy to get used to the code base because we have a seriously large space. It can be intimidating at times, I think.

Whatever I think is the easiest way to get in to make people feel good about themselves and confident that they can contribute. Then, I basically try to find adjacent parts of the code to make people feel more and more at home and at some point they know most of the code.

Ledge: How did you go about finding worldwide talent? There’s billions of people and there’s a lot of engineers out there. What sources did you use? How do you find the people and make those connections such that, over that distance, you can know that really they are that great and that you want to introduce them?

Oliver: Yeah, you’re absolutely right.

We mostly use the platform called Upwork, if you’ve heard of it, to find people. It’s really a lot of going through resumes of people and then reaching out to, say, your top 10 for a particular position. Chatting a little bit with them and just checking some of the social skills, I’d say.

How quickly do they respond? Do they use way too many emojis, for instance. Once that seems fine, I usually send them engineering challenges that I came up with and that are not completely artificial but pretty close to the code style that we have in our company, and to the type of problems that we have.

Then, I just check how quickly can they solve it and how well is it solved? It doesn’t have to be solved completely. I also see it as a plus if the person reaches out and says, hey, can you clarify this? How do you really want this to be implemented? Because obviously I want people to ask questions and not just do it all by themselves and then come up with something that.

Ledge: Certainly we know about Upwork. Big time player in our space. Your strategy is absolutely right there, and a lot of times I’ll get queried and they say, “Well I could just use Upwork and I could go international and it’s going to be a lot cheaper than US based developers.”

You totally hit the nail on the head there. You need to be willing to field dozens and dozens of people and have the wherewithal from your own skillset to be able to test them.

Oliver: Exactly.

Ledge: So really what you’ve set up there is a recruiting pipeline, a testing pipeline, that’s going to allow you to identify the diamonds in the rough. That’s a great platform to do that.

I always tell people, it’s like, hey, if you’re prepared to do that you can exploit a lot of arbitrage in the labor market. But if you’re not prepared to do that, understand that the cost and the friction, you’re never going to get any work done because you don’t have the skills to run that pipeline and test it that way.

There’s this important distinction on how to apply your skills and how to apply your budget into your recruiting pipeline. If you can use a tool like that, that’s a fantastic way to get economic benefit. But if you don’t know how to do that, you’re going to be in deep trouble. I think there’s this misnomer that you just go post your job online and, great, people will apply and you just pick one and everything will be hunky-dory. It’s not that way.

What were the trials and tribulations of that method? How many people failed out?

Oliver: Oh, most of them. Particularly that people, once you post something for instance on Upwork you will immediately get some recommendation by the Upwork engine, I suppose, that say, oh, you should hire this person.

You usually can’t use these people. You really have to dig through I’d say like 100 people, and then reach out to 10 of them. Then, maybe one or two of the people that you reach out to are actually good.

As you know with coding, once you hire somebody who is not a good coder or doesn’t apply your standards, they actually make you do more work by fixing everything, all the bugs that people introduce. You need to be really careful in finding good people.

Ledge: That 1 or 2% number is an absolute, universal truth. For people that want to build a great organization, I think with any talent, is that somehow or another you’re either going to pay for the capacity to figure out how to find the 1 or 2% – like somebody else does it for you and that’s what we do – or you’re going to allocate the budget, time and expertise from your side to do the filtering down to 1 or 2%. That’s just a universal truth.

It’s so cool to hear you say that, and to have been successful with that. I myself have built businesses on freelancers on Upwork – not in software but in other functions – and that’s exactly the process. You need to be able to do that.

I think our proposition to business owners and CTOs, CEOs is, once you have the cash flow or the wherewithal to think about outsourcing that function, that’s where we’re going to step in because we can bring you the 1 or 2% right away because we already did the work.

The question there is, how to budget and think about, do you have the capacity and the ability to be able to identify the 1 or 2%? If you don’t, it’s not a good proposition for your time. It really comes down to the self-awareness of what you’re good that.

I love that you went down that road because it can be amazingly effective, but it’s not going to be effective for everybody. You need that skillset.

Oliver: I completely agree. I do like spending some time on that because I also… What’s cool about outsourcing or finding talent overseas is that, many of those people are really hungry to do something cool and to work hard. So if you spend the time and find the talent, you end up with really, really interesting and cool people, in my experience.

Ledge: You’re so right. You know, what I’ve noticed is that people tend to get these ideas that, oh, I’m not going to hire from such and such country because I heard all their developers are terrible. That’s not true. One to 2% of the developers are good everywhere. It doesn’t matter.

Oliver: Exactly.

Ledge: Now, you are right that if you’re a company that is based on English, for example, you are going to need to think very carefully about a lot of the soft skills and things like that. You might have an absolutely brilliant engineer, but if they’re speaking Russian that’s not going to help you out. Right?

Oliver: No, that’s true. No. Obviously there has to be one working language that everybody is comfortable with.

Ledge: Absolutely. You’ve totally checked all the boxes on that and it’s very cool to hear you have been successful with that.

Talk about, if you would, maybe some of the stack choices and actual deep technology choices that you had to make on the development side. I think video is still one of those areas that maybe not everybody feels great in, and I think it’s going to be more and more ubiquitous.

You’ve done a great job abstracting that, and developers maybe don’t need to think about all that because they can use their product, but if they are thinking about it, what do people do from a technology standpoint to build out some of these things?

Oliver: Sure. On the client side you kind of have to use, obviously, the languages that are there. In the browser we use HTML, JavaScript and CSS. On older browsers we have to use Flash because the access to the camera, it’s only reasonably recent that you can use plain JavaScript APIs to access the camera.

On the server side, you can use more or less whatever language you like. We started out originally, way back when we started with PHP… Which is not my favorite language at all but one of our was building a very first piece of the prototype and then we just kept on using it.

We now migrated most of our architecture to primarily Node.js. The reason that we decided to use Node.js is, when the first version of AWS Lambda came out it only supported JavaScript and we really wanted to use serverless stacks. So we said, okay, that’s fine, we’ll use Node.js.

You could use basically any language on Lambda – not any but almost any. If we would do it again we would probably use Go.

In terms of the other stack that you need for your transcoding pipeline, most transcoders will be based, including our own, on FFmpeg. That’s mostly like a C++ program that you can compile yourself, but you need to have an environment where you can execute arbitrary binaries.

For everything else, it’s kind of up to you what you’re using. Since we’re dealing with video files, and some of them are really, really large, like 20GB for instance, you need to have an I/O system and a storage system that’s capable of handling insane amounts of I/O operations.

Ledge: I love that detail. You just blew me away. I have to be honest that I can’t keep up.

I do understand that our audience would totally appreciate that, and there is a lot of chatter particularly around Go. Is this going to take off the way that people thought? It got really hot, it got a little cold, now it’s coming back around again. There’s a lot of chatter around Go in the community.

Obviously, you can’t go wrong with some Node on the backend, tons of Python. You really have a lot of options, and I do believe that most of the providers, especially on the serverless side, are realizing that people are going to use what they’re going to use and we have to adapt that way.

Last question. You had to be a technical founder and run a business. I wonder, what have you learned and what can you advise on technical founders? All the things that are not technology and not code around running a business and scaling up. Maybe some learning lessons and places or things you wish you knew before that you know now.

Oliver: Sure. I think one of the things that, particularly technical co-founders love to do is work on the technology and the product, and they think they know best what should be done with the product. But it sometimes really helps to listen to your other co-founders and also listen in to what your sales people are doing, and to listen in to the actual feedback that you’re getting from customers.

As painful as it can be, it’s really helping you to understand what people are actually asking for.

Ledge: What are some things you learned from your customers in that feedback cycle? Assumptions you made that were totally wrong and that you’re glad you had that customer feedback channel?

Oliver: Well, you sometimes think about which features to build next and you think, oh, this would be a really, really cool feature to have. Some of them are easy and some of them are hard. Sometimes, when you really listen to the customers and say, oh, what they actually want is this, that might be easier to build.

In our case, for instance, we were building some machine learning in the backend, which basically does some object identification in the video so it can tell you, hey, in minute 2, second 5 there is coffee mug. Then we tag the videos we that. We got really excited about that. Our customers were like, yeah, that’s kind of cool but what I really want to do is filter my videos automatically, whether they’re safe for work or not safe for work.

So we said, oh, okay. So we just need to detect nudity for these people.

We essentially used our object detection system to now detect nudity, and offered automatic safe or not safe remote filter, and that’s what people are looking for.

Ledge: That’s a great example. That you can totally over-engineer something and the customer is like, yeah, but I just really want this other thing.

There are so many great stories. Slack is a great example. They were like, let’s build a game. And, oh, it needs to have chat in it. Oh, wait. People only care about chat – and the rest is history.

Well, Oliver, thanks for spending the time. Super cool conversation and we’ll make sure that everybody’s aware of what you’re providing. I think it’s a really good service.

Oliver: Awesome. Thanks for having me, Ledge.