Launching a tech startup inside of a legacy enterprise with Alex Behrens of RoomIt

Transcript

Ledge: Alex, welcome! Why don’t you give a little intro of yourself.

Alex: Absolutely! Thanks for having me. My name is Alex Behrens. I am currently the director of engineering for the hotels division within CWT. It’s a division that we the brand name of RoomIt which was established about a year ago. I’m in charge of just building an in-house engineering team within a whole new division that was established about two years ago. This is within a very large enterprise.

It’s great to be here. I’m looking forward to speaking with you a bit.

Ledge: Awesome, thanks. Off-mike, we were talking a little bit about what it’s like starting a modern software startup inside a legacy IT operation. In your LinkedIn, you served “lots of Oracle.” Obviously, you’re dealing with all kinds of stuff there: human elements, technology elements. You’re talking about building a team. Maybe tell that story a little bit.

Alex: The back story on the company was that the hotel’s division didn’t have a formal structure until about two years ago when a colleague of mine wanted to bring in an engineering team so that they could start to build and own some of the intellectual property behind the work that was needed to be done to increase revenues and margins and extend our reach further and further.

So this was kind of a huge challenge for us because we were a new office within the company. There wasn’t a Chicago office

. I didn’t mention I’m in Chicago. There was no in-house engineering team; all the software and services that were made for this division were off the shelf or third party company-produced types of things.

There was that huge task in front of us. We also had the task of helping to inspire and innovate within the very large organization and introduce mindset into the way that we produced products.

When I started, there was just me and one other developer who was a fellow right out of school; right now, we’re a team of about fourteen people which includes people of all stripes, backgrounds, a lot of former colleagues, and all the way up and down the experience ladder. So we have entry level all the way to people who are senior to me in terms of years of experience.

It was no small task to not only to build the team but to also start to innovate parts of the organization to follow our lead and to let us do the things that we wanted to do with some of the technologies and the frameworks, and the things we wanted to do.

It was kind of part and parcel of my job to start to create a difference and create a new model for the company. And those were not small tasks.

It’s an ongoing one. It’s was really what my day to day is. It’s a combination of the technology plus the people aspect of my job, and this is something a lot of companies go through especially large ones that have done the same thing for so long. They need to start to change and adjust to the new realities of the markets they’re operating in more Agile mindset that we’re bringing in as well.

Ledge: Agile mindset ─ I imagine that you probably have your eyes on enormous data sets and opportunities that they can bring machine learning, AI, maybe you’re even thinking on the blockchain realm.

What’s the cutting edge?

My guess is that the emerging technology thread remains tantalizingly out of reach while you’re trying to also build like “Oh, we’ve got to away from this legacy stuff.”

How do you navigate that?

Alex: That is definitely something that is a day-to-day challenge for us. I can even choose a very small example. We still run our own data center. We have an on-prem infrastructure but there has been a movement and a lot of expertise on my team and with myself of how to play things in the AWS environment.

We made containerization and containerized infrastructure kind of a first-class citizen in everything we’re building so even if we were working on personal machines we’re going to make it so that we would be able to transition into a containerized infrastructure using Kubernetes with a very small amount of work.

And so, one of the little battles that we’ve been having to undertake is a battle between EKS and Kops. And this is one of those philosophical belief systems that you have to make a decision about: Do you think Amazon is going to do right by the Kubernetes environment or do you want to take your chance on the community supported version of a Kubernetes deployment and maintenance infrastructure like Kops?

That is one of those kind of back and forth. The best thing that we’ve done and the way that we’ve kind of tried to make decisions on this is to have an airing of grievances, so to speak, in partnership with some of our groups in the enterprise IT side to say, “Here are two different types of things we’re looking at. Can you be partners with us in trying to figure out what’s going to best for the organization?”

For us, once again, as a new development group within a larger company, there’s been this habit of making siloes so that every team hires their own set of DevOps people; they hire their own set of engineers; and they create their own solution instead of creating a broader solution for the company and then slowly customizing it as need be for different parts and different groups.

We’ve taken a different approach on that, and that’s something that we need to do with almost every technology or problem that we look at. We’re doing the same thing with the data streaming platform. We want to bring in Kafka. We’re doing the same thing with our logging infrastructure. We brought in ELK. We’ve done the same thing with some of our routing infrastructure. We’re big users of the Nginx for just very simple routing and logging purposes and even all the way down to Mongo.

That’s going back pretty far. Mongo is not cutting-edge technology anymore but it’s our database. It’s sort of a new thing to CWT as a company.

What we try to do is to air our grievances, as it were, to let other parts of the organization take partners in trying to figure out what direction to go so that everyone has a broader sense of ownership as a result and they don’t seem to think that we’re being cavalier renegade about the way that we’re implementing technology.

Ledge: Imagine it can be kind of organizationally terrifying for some of the legacy folks who have been in your position for twenty or thirty years.

“I’m an Oracle DBA. I feel good about this. Now, we’ll allow this open-source business. What’s this and what’s that? Is it secure? What about the cloud?”

And you, guys, are probably sitting there going, “Oh, wow, we had this conversation”─

Alex: “We have to listen to this again.”

Ledge: Right.

Alex: Even on the security front, that’s another one where we’ve kind of shifted (I hate to use the term) the paradigm a little bit. It used to be very much “Send an email; fill out a very large form; try to get it approved.”

I see you laughing but that really was what it was. The

Nginx was the first thing that we brought in and that was a very long 20 plus-page document to describe what it is and why they have to pay for it.

Now, we have a security person as part of our planning, as part of our discussions about what technology we’re doing. So the barriers have really kind of reduced and we’ve worked more on a one to one as part of our ceremony basis with a lot of other groups from enterprise IT, from security, from architecture, from other groups to try to make it so that it’s less formal in some ways.

We’ve also formalized some of our architecture differently. It’s not a formal security architecture review process for us anymore. It’s “Let’s involve them as early as we can in our planning so that they have knowledge of it and aren’t surprised by anything we’re doing.”

Ledge: That would be an interesting thing to explore, too. You’re sort of an encapsulated, containerized, if you will, tech startup inside this larger organization. But I’d be curious to know ─ let’s say, there are other sort of startups inside or outside of big technology. What have you and your crew learned from the best practices that actually do come from a mature ─ we use legacy as a but there is a lot of institutional learning there on how you build massive important things that endure.

What have you taken back that maybe was not part of the mindset in the fast-moving startup kind of world?

Alex: We often think of enterprise IT or legacy as sort of a thing that you don’t want to deal with and it’s something that I don’t bear witness to often but I bear witness to it enough in that I see that and that’s the attitude that we, sometimes, especially as engineers have about someone telling us the way to do things.

A lot of times, the best thing that we can do is to have a sense of empathy about why it is that people are coming to us with that kind of disposition.

I joke a lot with one of my now best colleagues at the company because when I started off, he didn’t want me to use

Nginx for a use case where it was kind of a thing that you just use no matter what organization you’re in. His solution was “We could use JBoss for this.”

I kind of joked, “You have a JBoss tattoo. Are you being paid by the JBoss folks?”

But, over time, I learned later that the reason he was so dead set on that was because that was just what he knew; that was the way that he always did things. I thought at first that I really read him very well in that I thought I was not going to get along with him at all. But he’s been one of our staunchest allies and greatest defenders and really a partner in helping us to change the way that the organization operates.

The big lesson I have learned in this and the thing that I kind of stress to my team is that we should never see enterprise IT or any other group as an enemy, someone who is preventing us from doing things. If they’re not working in the way that we need them to, we need to say that out loud; we need to justify and articulate why the way the process is working is holding us back from achieving the objectives of our product owners and business and all that stuff.

So it’s about making a partnership to make things better for us and seeing that folks like enterprise IT, folks like middleware infrastructure teams need to be part of the solution, that they can’t be cut out entirely especially if you’re in a place that is a medium-sized or larger vision and that you ignore them at your own peril and, eventually, it will bite you back.

We’re still trying to figure this out. We’re still making our way through it. But, again, as a new development team within a large organization, there are a lot of questions about support that we don’t really have good answers for yet.

We’re a team of fourteen people and we’re doing hundreds of thousands of transactions per day. And if something goes wrong ─ obviously, you don’t want to wake up at 2 a.m. and answer an email or get on a call. So we still have to work with these enterprise groups to say, “How is our alert system working? What is the first-level support? What is the second level? What is the third level? How do these things coalesce and come together?”

And they need to be seen as partners and we need to help them improve the way that that works, too, because it’s incumbent on us as people bringing in something new to help train and mentor and work in a deliberative fashion with these groups and just not ignore them.

So that’s the big thing that I’d take back to the team. I think I’ve learned even more in this in that I shouldn’t look at those groups ─ and nobody on my team are small teams like mine within a company ─ look at them as enemies or impediments but just people who need to be influenced to think differently.

Ledge: Yes, which is really the lesson of “Hey, let’s go back to scrum master school and servant leadership is baked into the framework and there really is a mandate to serve the existing organization in a way that it does not treat that as sort of a negative thing.

It also makes me think. I’ve had some other guests on particularly in the legacy or big enterprise fintech and banking and neat conversations about “Hey, we’re obsessed with DevOps the way that we think about in a cloud now and we’re obsessed with sort of DevSecOps.”

But over there in the corner are the guys and gals who have always been Ops and make stuff run and keep stuff running. There have been a great deal of the organizational and technical infrastructure that rely on just regular old Ops.

They’re down in the basement in the dark somewhere.

But that’s stuff still there.

Alex: It is. We have a lot of that still. We have just massive infrastructure for that sort of thing.

We have a new fellow who was brought in after one of our DevOps days who is a speaker who has a lot of experience with this kind of transformational mindset and he’s on the enterprise IT side. And even for him, there’s a huge effort under way just to slowly change the direction of some of the bigger enterprise IT infrastructure matters; and it’s not an overnight thing for us.

We have to do that kind of thing and me as a consumer of those services, I also have to bear in mind that everyone’s state of affairs and experience is going to be different from mine. And then, if I do see an opportunity to say, “Maybe if we automate these alerts or maybe if we funnel the message for this into something with our runbook attached to it so that someone could troubleshoot it without our help,” we need to do our best to show them why it’s better than their current system and then, hopefully, spread that around the other parts of the organization so that they do the same thing.

But you can’t ignore those Ops people. They have to be part of the overall.

Ledge: What have you seen changed in the mindsets around ─ we’re talking about open source; we’re talking about the different stack technologies. How about down in the language front? What’s the difference between the legacy language choices or programming paradigms and even version control and the things of that nature that you see between the two approaches?

Alex: We’ve done a little bit with our choice of languages and that’s largely, sometimes, because of the difference between our finance side whether they want to capitalize everything or operational expenditure type of thing which is a level of budgetary detail that even though I understand the difference, I don’t really care about it. I just want to get my work done.

I’ll answer one part first and then the second part later.

I was fortunate enough at my last job which was a company within the advertising technology space to work for a very engineering-driven organization that wanted to have a mindset of “We’ll choose the right framework and the right language and the right solution for the task and try to leave everything in a place that it can be changed later so that you don’t bake too much into it. You break apart pieces as best you can that make sense into as small as possible component,” so on and so forth.

The downside of that is we ended up with tons of different languages, tons of different components so there’s always a balance to strike there.

But I’ve really carried that attitude forward into the new job and I’m lucky to have brought in a lot of colleagues. We have the same mindset that say if it’s a backend service that needs to have high availability and needs to have true multithreading, then we’re going to use Java.

If the frontend system will develop something with a Node.js backend, we’ll do it with an Angular frontend; we’ll try to keep Angular up to date so we don’t have to go several major versions updated.

If we’re doing a data streaming application, we’ll do something with Kafka because that’s a first class system in a data streaming world.

We’ve taken all these things on a case-to-case basis. In some cases, we had a scenario where we wanted to use Elastic backend, the Lucene infrastructure, to do some data query and type of things. One of the guys in the team has some experience with a technology called “Solr” which is based off of the same Lucene backend but it was going to be kind of hard because the rest of the team didn’t have experience to Solr even though it was based on Elastic or Lucene, to be technical about it.

And it was going to be hard to push through another new component through our enterprise IT team to say, “Please support yet another piece of infrastructure when the Elastic backend was going to suffice just fine.”

So there was always a balance to be struck there in terms of what technology we use.

A lot of other companies, sometimes, say, “We’re a completely Python shop.” I kind of scratch my head and think, okay, that’s cool but are you completely Python shop because that’s the technology that solves your problem or you’re completely a Python shop because that’s what you start out with and you never really had the capacity or the ability to change.”

It’s a little bit distressing because as an engineer myself, I’d become more and more open-minded about what technologies I think are right for every case and I’m really glad that I’m in a place where we have that freedom and flexibility as engineers to make that decision. Not every place does.

On the infrastructure side and the platform side, we finally have started transitioning into using more Sass products for a lot of things we do

. Even for our code storage, we use the GitHub Sass.

But even those have little quirks in them in terms of the way that they’re built back and the way that we, ad users, do them and the way that you account for different groups having different things.

In an organization, those are kind of case-by-case basis as to whether they make sense or not.

Personally, the Sass solutions for a lot of these things makes sense for us because it’s easier and we don’t have a huge people infrastructure to manage it. A lot of people who are in the cutting edge of some of those Sass products but every one of them has little quirks that we need to look at and we’re getting better understanding when it makes sense to go out and purchase a product versus when it makes more sense to try to build it or in-house it on-prem. And there’s no way to answer for that. It’s just that every case is different.

Ledge: You make great points on the language selection front. We have clients or people who talk about “I desperately need to write this thing in Node and we all know that Python is the best tool for that. So you’re going to rebuild a hundred different packages that you could install now because Node doesn’t do that.

We have thirty Node engineers already and I look at it and that’s a recipe for maybe in fifteen years, that’s the architecture because you just don’t know what’s going to happen later; and, of course, you do your best to advise on “Hey, maybe you should broaden your perspectives. The reality is that any of your engineers, given two weeks, could be up to speed on the proper methods to do this with a different language that’s well suited for it.”

Alex: Absolutely! In one of the previous shows, we were talking about dealing with legacy code and systems; and that’s another thing that I hear a lot about. And we have this big old thing that was maybe written in a really old version of Java or something and how are we ever going to replace it.

In that case, you have engineering resources available to start making those changes. You also need to have a leadership that’s going to recognize that changes need to be made slowly and the right kind of mindset on the product in the business side to accept that thing. You maybe need to slow down a little bit in order to remove that legacy from the equation.

Fortunately, my team is not responsible for maintaining legacy systems which is something I’m thankful for everyday but it is one of those problems that takes an art and a science and a lot of soft skills to be able to justify to other people why that thing needs to change

.

Ledge: Absolutely! I think all of us are lining up to someday be the enterprise engineer who has to deal with the legacy thing that we wrote.

Alex: Unfortunately! I know there are legacy things that I’ve written in the past jobs that I’m probably rewriting

.

Ledge: I still go back and I go, I wonder if their payroll is still running on that Bash script that we wrote 25 years ago ─ and God help them!

I’m ending every show now asking the same question, and that is, in your opinion, what is the best senior engineer? What are the profile and considerations to make the best senior engineer that you can hire?

Alex: I could probably add something about how we do or manage the way that we’ve kind to structure our expectations around work that’s not done by our in-house engineering team and that might be relevant or not. I can let you decide on that.

Ledge: Please.

Alex: Do you need to set that up? What’s best for you there?

Ledge: How do you think about adding team members potentially from a service like us where you have long-term yet temporary and outsourced or, otherwise, remote team members who need to work on something important? How do you think about adding those resources, managing them, selecting them, and making them relevant in the full-time context?

Alex: Absolutely! That is something that we need to consider quite often and the way that we’ve kind of come to grips with that is to try to make those outsourced resources, patterns, and work habits match our own down to the way that we do, the way that we expect to be written, the kinds of unit test that we expect to be written.

We want to have a well-documented understandable repeatable process about the way that we read the right code even down to the way that we record stories and score them and all that and place an expectation on any of the partners that we do work with in an outsourcing context that says, “In order for you to be part of this organization, we’re going to treat you as first-class citizens as if you were an employee. Your process will be no different than any engineer or our team.”

I think that’s worked out really well for us so far and it makes sense if you have a place that has a well thought-out process even if you don’t completely agree with their branching strategy. As long as the process is transparent, well documented, and easily followable by people no matter if they’re full time or not, then that’s something that is easily reproduceable and we can bring in new people at any point and time and just get on with it.

Ledge: You make a good point. That level of investment is necessary regardless because you’re going to be growing that team and you’re going to be bringing people of your own. So it makes sense to battle test that human infrastructure and planning against people who maybe are not traditional employees.

Alex: And it’s unfair for an organization to expect to bring new people in and not give them some of those guidelines and expect that their work is going to really be worth the money.

So that’s something I’d encourage anyone to do. If you’re doing any kind of outsourcing, have that thought out before you go out and say, “I need some people off the shelf to do some work for me.” If you don’t, then you’re kind of maybe wasting your money.”

Ledge: Got it! Alex, thank you for the thoughts. It’s been very instructive. Thanks for joining us today.

Alex: My pleasure! Thank you very much.