Interview with Chris Siefken, Head of Technology at Xenial

Transcript

Ledge: Hey, Chris. Thanks for joining us today.

Chris: Thanks. Thanks for having me.

Ledge: Cool. Why don’t you give a little background story of yourself and your work so the audience can get to know where you’re coming from and what you’ve done.

Chris: I’ve been working in the restaurant industry, or the restaurant technology industry, for the last roughly 12 or 13 years.

Started my own company with a couple of partners. We did loyalty and marketing automation. Worked with many of the fast food brands in America that you know, building mobile maps, doing loyalty, figuring out what people eat at fast food restaurants – which we typically call them QSRs – and helping them to understand their customers to both build better products, increase average ticket and that sort of thing.

Sold my company in 2016 to Heartland Payment Systems, which was subsequently bought by Global Payment Systems. Now we’re a business unit within Global Payment Systems, a company that’s actively trying to turn themselves into a software-led payments company, a credit card processing company. To that effect, we’re working to bring 60% of our revenue in from software sales.

Ledge: Wow. Cool. Restaurants, everybody’s been there. We all interact with it. There’s a million approaches to this point of sale, and I think we all wish sometimes it would be better.

So what’s on the cutting edge here?

Chris: For us, obviously there’s a big shift for restaurants to move to the cloud, especially enterprise restaurants. There’s some very old legacy players in this space that typically run in a Windows installed platform, and we’re in the process of building a new platform which is OS agnostic. Really modern technology. A lot of JavaScript, Angular, Node, built around a Cordova application on iOS and Android, and Electron application on other OS-es, to become the restaurant POS of the future.

It’s a serverless point of sale system that allows you to use your legacy hardware investment, which is really important to these restaurants which might have 6 or 7000 stores as a footprint. Then begin to leverage up new technologies and build a bridge to what otherwise would be some of these that looks like Revel or Square, some of the new entrants in this space.

Ledge: Before we hit record here, you talked about the applications of blockchain, which is always interesting to the audience because there’s just so much hype on blockchain. But, how are you actually using it to do real things in a scalable fashion for real businesses?

Chris: We filed a patent on using or employing the blockchain in the POS transaction ledger. Basically, older POS systems, or most POS systems, in fact the reason why they started was to has this thing called a nonresettable total. A bartender, the original POS bartender, built a kind of a button that couldn’t go backward to count the number of beers that were sold so that they could ensure that the bartenders weren’t stealing.

The POS has the need to make sure the data we’re collecting and the actors in that system are not being tampered with. Blockchain allows us to cryptographically sign the pieces and parts of those transaction records that are coming in. Things like, what am I selling, what are my time punch hours? Various other parts of the POS system, and ensure that there’s a proof of work that makes up a business day.

This allows us to ensure, hey, if the chain was tampered with or if the terminals go offline and have to come back together without a centralized service or server in the store, we’re able to do this in a seamless fashion and ensure that the integrity of the data is kept.

Ledge: I imagine that’s not the sale that you give directly to the restaurant, because they probably won’t understand what many of those things mean there. The objective there, the business case, is higher integrity and less loss of money, I guess. Right?

Chris: Yeah. It comes down to ensuring that the transactions that are recorded in the store really came from that store, and that no one went back later and tampered with the data, and potentially tried to commit any fraud.

This is a common thing that happens in restaurants, so you really want to avoid any possibility of theft and have a very secure system from that perspective.

Ledge: You talked also about the voice applications and AI. I thought that was interesting. I’d love to hear that story too.

Chris: Along with this new point of sale system we’re building, we’ve recently acquired a company called SICOM. They have a lot of hardware and drive-through technology, and we’re building what we’re calling the drive-through of the future.

If you look at QSR or that space, quick-service restaurant space, the drive-through if you think about it really from your personal experience, it hasn’t changed and hasn’t been disrupted in the last say 30 years. So what we’re working on here is really just bringing modern technology into that space.

Simple things like license plate recognition, which have been around for a long time, but connecting that to the point of sale systems. So the point of sales camera are aware of the car and then can say, “Hey, look, I realize you ordered this last time. Would you like to order that again?” and perhaps upsell you to a shake or something like that in the process. Right there on the screen you’re looking at. The menu board changes and it says, “Hey, this is what you always ordered. Do you want to order that again?” The conversation becomes much shorter.

For these restaurants, the amount of time that someone is waiting at that drive-through is a real critical part of their business. They are looking to move as many people through that line as fast as possible during that lunch rush.

The other thing we’re working on in this space is voice recognition. As you think about how Google Home and Alexa have changed our lives, in our houses and in our personal lives, bringing that to an enterprise scale for the restaurant industry means being able to take orders directly by way of voice recognition, understand what you asked for, make sure to do a consistent job as far as how they communicate with you, confirm in a digital fashion visually, “Hey this is your order, I got it right,” and take your order. It means a change in the way that restaurants employ their folks and what those team members are doing out in the stores.

We think this is going to be, at least just those two things alone let alone other things we’re doing, are going to be really big game changers for the restaurant industry and really the US economy as a whole over the very long term, if you think about it.

Ledge: Absolutely. It sounds like sort of immediate use case, depending on where you fall. To, well, this is great. I’m a business and I get to eliminate some hourly people. Of course, if you’re of the disposition that computers should not take over jobs, then that’s going to be a tough pill to swallow. I imagine you have people looking at the business with different lenses on what you’re building and how, and what the implications will be.

Chris: Some of our customers really think that the part of their brand is that customer-consumer interaction and that customer service factor that their employees bring to the equation, and so they have been less interested in bringing in things like kiosks and this kind of technology to their brand. But the increase in hourly wages and labor costs, especially in urban markets, has got everyone trying to think differently about how to run their businesses. Labor is a big factor in restaurant sales, in restaurant profitability.

Ledge: Absolutely. So, you’re taking on some major players here. Everybody knows the Squares and the Clovers and the entrenched Revels, and all these big, big players. How do you tactically go about that?

It’s easy to visualize what it ought to be on the whiteboard, but you’re really building and deploying this stuff. What does that look like from an organizational and engineering standpoint?

Chris: I think one of the most important things to note about those other POS systems is that they are targeting a smaller market in terms of just the scale – typically going after a single store or just a few stores. We are typically targeting the largest enterprise customers. Brands that you know that are all Fortune 500.

We do business with 19 of the top 40 food brands in the US, and so it’s a different product that those folks are looking for. They’re also looking for this market expertise that we bring to the table with 30 years of experience, but wanting to refresh their technology.

In terms of how we build it, in order to compete these companies have just had massive investment in this space. Many of them over $100 million, some of them over 200 million. Our focus has really been to try and scale up agile teams. We do the work onshore and off, working nearly around the clock. We take a lot of time to think about how, when we engineer products and when we engineer the next thing that we’re working on, to isolate the abilities or the needs of those teams down to the smallest common denominator, and try to make them less dependent on one another.

So we use and employ the microservices architecture model. We use a lot of AWS, Lambda and Elastic Container Service, so images are containerizing our code. Try to build contract relationships between those various development teams so that they can run independently, they can have their own sprint cycles, they can deploy when it makes sense. We use a tremendous amount of QA automation.

Those things together allow us to do things like releases twice a week, whereas in this space our competitors are having to ship Windows installer files out to the field. Really, those restaurants then have to take those Windows installer files and really look at them, and don’t have that same CI/CD modern pipeline approach that we’ve been able to employ here.

There’s a lot of work that’s been going into, how do we scale this? Making sure that, to whatever extent, our teams can run independently.

Ledge: How do you get the feedback chain back to so many engineers through the product function? Or customer feedback?

I go immediately to thinking, you’re not dealing with people on the end user front that typically would have technology degrees or maybe even be your digital native types, so small changes in interface can make a huge difference in flow. How do you assimilate all that?

Chris: We do a lot of interesting things as it goes. First, we obviously have conversations and deep relationships with these customers. Many of them span everything from technology to marketing to operations. Meaning people that actually train the team members that work out in the restaurants are part of who we talk to.

That said, we can’t just rely on that. We do a lot of logging and we aggregate those logs together. We’re pretty heavy users of Elasticsearch and Kibana, and we do a lot of anomaly detection. So we look for things like, hey, how long does it take on average a cashier to step through an order, and which cashiers do it faster, and what’s their flow? Maybe what menu setups are faster than others? We do AB testing related to that.

Then, from just a log aggregation perspective, we’re watching to see are our builds faster or slower? We do performance testing before we roll out to the field. Then once out in the field, we’re watching all these things to figure out, hey, how can we be proactive to our market?

So that instead of having someone call us – a restaurant is a running business any kind of disruption in their ability to operate is cost to our customers – and so we’ve really taken advantage of this new technology to say, hey, let’s get the anomaly reports in the morning, bring it to our customer-facing teams and pick up the phone and call places. Call restaurants and say, “Hey, I noticed that you had 12 application restarts yesterday. Let’s have a conversation about what’s going on with Terminal 4.”

Proactively help people resolve their business problems before it turns into a support ticket or a phone call and really impacts restaurant operations because, perhaps that cashier didn’t think to pick up the phone and call the help desk and that problem will continue to get worse until someone actively addresses it.

We can be the other side of that coin and provide a new value that our customers previously have not been able to see.

Ledge: I think if you’ve ever used a POS, you get this sort of – especially in a rapid environment of a high order churn like that – you’re going to get this muscle memory thing where even moving a single button makes a tremendous difference in the ability to process.

So you must have a lot of UX analysis there going on. Maybe there are some UX choices that get ingrained into the brain that you can’t undo, even though you know it could be a better choice, because undoing that muscle memory actually matters for speed.

How do you make those balances?

Chris: That’s a really good question, actually. We do employ a number of UX/UI folks, and we even employ a PhD in that field for that specific purpose – to look at the order flow and how to make sure we have the optimal configuration for speed of service.

We brought to market something called conversational ordering, where a guest can sort of tell you what you want out of order from what the system is expecting. Rather than saying, “Hey, I want that sandwich,” and then someone asking you, “What side do you want with that,” you can talk about the side first then we can load up the meal as you go. The cashier doesn’t have to go and perform specific actions in a specific way.

Or if you say, “Hey I want the number 5,” then you move on to the next meal you’re talking about, that cashier is prompted to go back and say, “Hey, what drink did you want with your number 5?”

There’s a lot of thought that goes into that, and you’re absolutely right in that sometimes you make decisions on the UI side that confuse the end user and don’t test out well and you roll it back. All of our customers, all of the folks in this space, will roll any new change like that out to a small subset of restaurants, and they will monitor those business values; drive-through time waiting, cashier input time. Just overall profitability in those stores in a small subset before making the decision to roll out new software to a broader subset.

Which again, it’s totally different than, say, the way that our competitors that are working with smaller businesses work in the space. Everyone gets the same version of Square. Well, in our case, we allow our customers by location to choose what version of software to put there. Due to the nature of the way we design the platform, it’s got a shell application that drags down all these business logic components from the cloud.

You can run a different version of software in Restaurant 123 than you run in the rest of your business, and allow that restaurant to test it and to roll back as a dropdown menu and a button click. Again, much different than the Windows reimaging of systems or potentially reinstalling components and parts in compiled software that we used to have to work with in the previous environment.

We think it’s going to be very drastic and very meaningful to our customers around cost of ownership as compared to what they’re doing today.

Ledge: Maybe tips for people that are developing high utilization UIs. This type of analytics, gathering of every single button and interaction, it’s very useful for any business. It’s not unique to the restaurant use case.

Any recommendations? It’s just like, how do you figure this out?

You mentioned having grown from 10 engineers to 400, so these practices are applicable in small teams as well. As you develop from a product perspective in order to be ready to scale, I just wonder, what are some of those learnings and how did you deal with the massive inflow of feedback and data?

Chris: I think the most important step before you go live is to take people who’ve never seen what you’re building, put them in a room… In our case we found cashiers, put them in a room, asked them to perform specific tasks around mockups for the UI and watch where they got lost so that we went out and redesigned those parts and pieces before anybody put any code into action.

The other thing that we do is watch the analytics. It’s so important to realize that you probably didn’t get it right the first time and just continuously watch that data coming back from how your user use the product. I’ve seen this all too often, where folks will come out with a brand new UI because it looks better, and forget that they did a lot of tuning and tweaking to what they have out in the field that’s gotten a conversion rate or gotten a specific user action to be as efficient and as desirable as they want it to be. You almost have to start all over again.

Those are things that you want to AB test and really think about before you make those changes, especially when you have incoming revenue and a running business. You just need to slow down and say, okay, how do I deploy this? Rather than, it’s very exciting, I’ve got this new design, we think it’s amazing internally.

We often forget that the market is really our customer and we need to make sure that it actually is better for the customer rather than just that it looks better. I think that’s something that, as technologists, we tend to forget when we get excited about that new thing we’re launching.

Ledge: Is there a level of better that just becomes reached? I think in agile we have this idea that, well, we’re just going to iterate forever and AB test to the limit of everything.

With that much data and that much input and surface area in the marketplace, do you ever reach done? It seems like you might be able to approximate the mythical done with that much input and data.

Chris: I think what you reach is a place where the returns on the investment are not strong enough to justify further investment. So perhaps you’re not done with the overall product, but one area isn’t going to generate more value for your customer or more value for us as a software partner.

You get to a place where you say, okay, I’ve got something that’s working, and now to go find more to get here I’ve got to go look somewhere else.

A prime example of that is the drive-through conversation we were just having. Where we could do a lot more work on the POS itself, but the reality is if we change the consumer experience at the drive-through, typical QSR restaurant 60% of those sales are coming through the drive-through, there’s a much more meaningful impact if we get that right than it does if we move some buttons around on the Point of Sale.

You have to look at the whole thing in context. Sometimes it takes just stepping back and saying, what’s the right thing to spend your time on? Focus is such a critical component of any product or project like this. If you’re a startup, it’s even more so. You have such a limited amount of resources. You’ve got to really focus in on what’s important, what’s going to drive the next sale, what’s going to drive that right conversation.

It’s not just, let’s iterate and get better, it’s got to be on the right thing or you won’t succeed. You won’t have a business that goes and generates revenue.

Ledge: Great thoughts. I guess I’ll leave it there.

Chris, it’s cool to have you on. Congrats on the success, and I know I look forward to having a better drive-through experience every time my kid wants a McDonald’s breakfast or something.

Chris: Thanks. It was really fun to be on.