Interview with Brandon Smith, Director of Engineering at eVestment

Transcript

Ledge: Hey Brandon. It’s good to have you on. Thanks for joining.

Brandon: Hey, thanks for having me.

Ledge: Would you just give a quick background story of yourself and your work a little bit, so the audience can get to know you?

Brandon: Sure. I’m Director of Engineering at eVestment. We’re a financial company based out of Atlanta, Georgia.

I started out at eVestment in 2014 as a software engineer, and have grown with the team here and am now Director of Engineering. I run our application development, so web apps, single-page application. We’re really building a giant analytics suite for the institutional investor market.

We’re part of the larger Nasdaq organization, and we’re excited to be on the show.

Ledge: Great. Thanks for joining. Fintech is a huge area. So much investment going on there. Then the institutional is a whole different beast. So much going on in the space.

You talked a little bit off-mike about how you guys have had to like your engineering team over just saying as many years.

Talk to some of the scaling issues. That’s a thing that a lot of our listeners have to think about and are going through. There’s many different dimensions of that, so just tell a story a little bit on that.

Brandon: Back when I joined in 2014, I was one of maybe 15 developers. We were not immature, but we didn’t have quite the amount of infrastructure and roles in place necessary to grow to a 60-plus engineering team. As we’ve grown, those roles have found their way naturally, as well our tooling.

We used to use something called CCNET, for example. A really old CI/CD pipeline, that no one probably even knows what that is, Mercurial for source control. We, over time, migrated to Git and then TeamCity and eventually Jenkins. Just better tooling to support more hands-on keyboard, more code check-ins, getting code to production more efficiently. We didn’t want the human process to be a bottleneck.

We’ve also changed up our scrum teams. It used to be maybe not even your traditional scrum, less process. We’ve added process, not for the sake of process but for organizational clarity. Where we used to maybe have two teams of seven or eight, we’ve scaled out into 10 teams of four to five, for example.

We call them scrumlets. They’re our little scrums. They come with an embedded product donor that manages your backlog. Each team is completely self-sufficient and independent, so they can work and deploy that code to production as they see fit. They answer to themselves now as opposed to the organization and everyone’s really free to self-manage. That’s really allowed us to scale because, if each team can operate independently we can bring a new team online, and as long as they have the resources that they need to operate then they’re free to do so.

Ledge: How do you know when and around what to bring on a new team? Is that driven by, I guess the product function, the product owners are communicating out to the business side then on a regular basis?

Brandon: We have a typical investment cycle where we bring on additional teams every year, and those teams come on to work on our newest initiatives usually.

Every year, say we add two or three teams, we go through a traditional ROI process to determine what projects are the highest for revenue generation. Then those will trickle down to the teams and determine which team is working on what. If we have more money to invest than we have teams, then we’ll bring on a new team to tackle that project.

Ledge: Good problems to have, right?

Brandon: Great problems to have, yeah.

Ledge: That’s interesting. Talk about that ROI process. I love hearing that because I don’t know that a lot of engineering teams are that directly hooked up to the revenue side. That’s really interesting to hear about.

Maybe some advice on what that looks like, and then how you distill that into, okay, we actually should build X, Y or Z.

Brandon: Our product and revenue organization is very, very good at analyzing what the value prop is of a certain project. An important piece of understanding the value prop of what we’re building is, how much is it going to cost us?

The tech work is actually very tapped into that ROI process. Product team will bring an idea to the table, and the engineering team is responsible for saying, “It’s going to cost us X dollars to produce.” From those two artefacts we can produce, say, a projected revenue value of it.

That allows us to stack rank projects and say, “Well, the engineering team thinks this one’s going to take 12 months and this one’s going to take 9 months. If the projected revenue is the same, then you should probably build a 9-month project because it’s going to be cheaper at the end of the day.”

Ledge: How do you know that the engineering estimates are right? That’s the notoriously difficult piece of software, is trying to make an estimate of what something’s going to cost.

Brandon: We do estimates at different levels. At the highest level, we use a methodology called Delphi. It’s very high level but it’s a blind estimate where you get multiple subject-matter experts in a room, they spit out a number without any visibility of the other person’s number, and then we discuss, “Why did you say six months but I said nine months?” We iterate until we get down to a standard deviation we’re comfortable with. That’s at the high level.

But then as we get projects landed on to scrum teams and as we iterate down in to our sprint iterations, we’ll actually do points. We’ll try to project out the time of our project.

We try to get ahead of it. If we see a project that we thought was going to take nine months is now projecting at 19 months, we know, okay, is it really something we should be investing in? Should we continue to invest in it or should we…? We’ve got to be comfortable also killing projects if they’re going to cost us more money than we thought.

Ledge: Yeah. You’re really looking at the results of the scrum process as an ongoing burn-up , or something of that nature, to know if you’re on track. I’m guessing that you do some kind of a rigorous retro type of deal to figure out like, hey, how do we make the system better so that the next estimations actually come out right?

Brandon: Absolutely. It’s a constant recalibration to make sure we’re on track with what our assumptions are. But then, you’re absolutely right, at the end of the day when the project does shift we’ve got to look back and say, “Were our assumptions correct, not only on the estimate side but on the revenue side? If they weren’t, what can we do next time to make it better? Where did we miss cost? Where did cost creep in that we didn’t account for, and what assumptions did we make on the revenue side that maybe weren’t valid, that we can improve on?”

Ledge: Does the whole organization adhere to the scrum context? On the revenue side then, you would have expanded past what engineering and product can do. Is there a lot of buy-in then, to that whole process throughout the org?

Brandon: We like to think of ourselves as tech org through and through. So, while other parts of the organization might not do scrum, they are very comfortable with the teams that do do it and they are comfortable interfacing with scrum teams.

The tech org is the heart of the organization. We call our total umbrella the offerings organization, so products and technology together. Since the tech org works in scrums, the product team works alongside it in scrums, if that makes sense. So the whole org is very familiar with the scrum process and scheduling and working around that.

Ledge: Do you think the discipline around the data and the calculations and precision, does that come out of being a financial org? You don’t see it a whole lot. You’d that engineering would be very disciplined in that way, but I can’t say that I talk to a lot of organizations that actually do all that data stuff the right way.

Brandon: We’re a data product at the end of the day so we are a very data-driven organization. Our marketing team is even very data-driven. Every corner of the organization is just very data focused. A lot of the decisions we make are based on data. We try to make it very factually-based.

That does come some from technology, but the business that we’re in definitely feeds into that a lot and I think is a huge advantage for us.

Ledge: Talk about culture there. One thing about a data-driven organization is that it’s awesome and you get this precision-level performance out of it. The other side is that, if we get too data driven there are cultural implications, where you can derive or get down to the sort of no one’s thinking. The human side is missing there.

How do you make that balance?

Brandon: There’s definitely some analysis paralysis, as you could call it, especially on the engineering side. We can get down in the details and forget to see the goal while we’re trying to figure out the details.

It does take a mixture of personalities to prevent that. That’s something we look for in hiring. We don’t want to hire 100% of people that are just data-driven. We need the people that are comfortable making decisions with minimal data available. Then just the data-driven people to fact check them and get their back. So we’re a data-driven organization but we have very diverse personalities, which keeps us on track.

Culturally though, the data-driven aspect keeps it interesting. I’ll give you an example. Baseball season just started and we had Fantasy Baseball draft last week. The people that were participating showed up to Fantasy Baseball draft with spreadsheets of their own data and rankings for Fantasy Baseball. The data-driven organization shows its head in interesting ways.

Ledge: You’ve got a bunch of Billy Beanes there, huh?

Brandon: Yeah, very much so.

Ledge: Excellent. From a leadership perspective, how have you guys leveled out your org? I know you’re director. You think about, like some organizations do, hey, everybody still needs to commit code, or sometimes the leadership is only giving code reviews and more strategy and architecture.

How does that play out when you’re using the scrumlets – which I love that word. I haven’t heard that before.

Brandon: Under the directors, we have engineering managers. Engineering managers usually work with multiple scrum teams.

Committing code is not in the manager’s job description, but it is a very active part of their job. Even if it’s not under their own name, the commits are sitting alongside their engineers at their desk there, programming and helping them.

At the director and architect level, we do still commit code but it is more looking for opportunities to fill in gaps or help teams. I’m not embedded in a scrum team. I’m not pulling tickets from a backlog. But if they’re sort of a side project that will help with efficiency of the team or I need to move a project to cross the finish line, I will still jump in and get my hands dirty.

Our whole leadership team comes from a software engineering background. None of our leaders are just people managers or just leaders from a different walk of life. We all started as software engineers so we’re all very comfortable falling back to that, and we enjoy to sometimes, of course.

Ledge: Yeah. A lot of leadership types lament that they don’t get to do code anymore. I always like to dig into that and see how you make the balance.

I’m curious, there’s no ROI calculation, at least on the revenue side, when you’re looking at tooling and you’re looking at a greater dev efficiency and shipping faster and CI/CD, and all of those things. How do you fit that into your process? There must be some kind of an expense budget or something along that line, to say, “Well, I’ve got to go two hops from ROI and certainly two or three hops from revenue to start to say I should invest in tooling and even technical debt remediation.”

How do you fit that into the equation?

Brandon: We do have a separate technology budget that we fight very hard to protect, and the organization is pretty good about that because they understand the importance of developer efficiency. But you’re absolutely right, it’s hard to calculate an ROI if we’re using Jenkins over TeamCity but it’s actually pretty quantifiable.

If Jenkins is going down or we’re having scalability issues and we think if we cut over to something different that those issues are going to go away, we can actually calculate that the average developer is going to save 15 minutes a week, or something. You can extrapolate that out to a real dollar value and say, “Well, if it’s not costing us more than our savings, then it’s a net positive gain.”

Things like Jira. We used to use FogBugz. Moving over to Jira has allowed us to scale the organization. Getting rid of those bottlenecks, moving things to hosted, managed providers. We just have to manage on our infrastructure for our tooling, getting more things off our plate, just less maintenance time for us.

Those things are quantifiable at the end of the day, but we don’t always have to quantify them just because it’s pretty clear what the value is most of the time.

Ledge: Absolutely. How much of that feedback comes back from the developers? How do you figure out your bottlenecks? Is that an automated process or is there a qualitative nature to that?

Brandon: We do have metrics around our build process and our build pipelines. We even have metrics around code commits that result in a failed build and things like that. We’re pretty quantitative.

At the end of the day, our developers are very open and they’re good at giving feedback. We’re a very transparent organization. If there’s a bottleneck and people are feeling pain, they have no problem voicing it and we’ll know about it pretty quickly. We encourage that. If there’s something in your way that you can’t do your job, the quickest way to remove it is to let somebody know or to remove it yourself.

We typically don’t want to wait for the data to tell us we have a problem in that scenario. We’d rather know about it as soon as it becomes an issue.

Ledge: Self-managing team sounds like an awesome concept until you think, how do you…? You also have to manage some standards. Not everybody gets to build their own deploy pipeline and build pipeline.

How do you draw the lines between self-managing or things you can touch as a team and things you can’t touch because they do need to be centralized or then the efficiency would break down?

Brandon: A big way we solve that is with cross-cutting architects. We have architects that will span large swathes of the organization and work with multiple teams. They help drive standards, set standards. The team wants to start something new or try something new, they usually work with the architect to keep them headed in a solid direction on that.

We also do a ton of internal training. We will hold internal workshops to help communicate what our standards are why we do things the way that we are doing them. That really helps keep alignment. We’ll do two, three-day workshops where we get a couple of teams in a room at a time and we just dedicate the entire couple of days to nothing but go and hedge down a writing code. Everyone leaves the workshop with an understanding of, okay, this is how eVestment does X, Y, Z. Then they can go back and execute on.

Ledge: What are the biggest difficulties? There’s always room for improvement. All this stuff sounds amazing. I wonder, what keeps you up at night or what’s causing the most meetings and where are things bogging down now?

Brandon: Probably the typical answer in a large organization is, communication is always tricky. As an organization scales, communication doesn’t always scale with it so we’re constantly looking for better ways to keep our communication channels open, make sure the important people and people that need to know information are getting the information in a timely fashion.

That can be simple things. That can be things down to an important bug came in or it can be massive project decisions. Communication is important, top to bottom, from the smallest detail to the biggest project.

That’s always a struggle. If someone has a solution, I’d love to hear. It’s a bit of a constant self-improvement thing that I think a lot of organizations need to work on.

Ledge: Yeah, no doubt. What’s your communication tooling evolution been like?

Brandon: We used to be Skype for Business. Briefly moved to Teams, because Skype for Business is being phased out. Ultimately, we’re on Slack now and it’s been great for us. A lot of our tooling integrates nicely with Slack now too. Our Jenkins pipeline can tell us when a build fails right in our side channel, which is great.

Ledge: Of course, you get to that place where you’re like that fine line of alert fatigue. Those Slack channels where everything hooks up to it and it goes by so fast you’re just like, whoa. You can’t keep track of everything.

Brandon: Yeah. You’ve got to start to ignore them if there’s too many of them.

Ledge: Well, right. You get to that place where everybody just mutes the channel and then you have no alerts for anything. So it’s that constant balance.

Brandon: Yeah. We try to keep our alerts… There’s no fire hose of alerts. We try to send alerts just to specific teams – so very, very focused alerts. If only four people care about this alert, only those four people will get it. Avoid the fire hose.

Ledge: Yeah, no doubt. Last question I love to ask everybody. You mentioned, I don’t know, off-mike, on-mike, but everybody’s challenge now with hiring. Get enough engineers, get the right engineers.

First of all, how do you handle? Is it all collocated or hybrid or remote? How do you guys do that? The second part of the question is, regardless of how you do the work, talk to me about your heuristics for hiring. What makes a really great engineer, in your context?

Brandon: Sure. We are all collocated. Everyone’s here in Atlanta. The biggest things we look for in engineers that we hire is not necessarily the languages you use or even your experience with tooling, it’s more about your ability and desire to learn. Your ability to come in and be an independent thinker, and to engage with your team, and pick up our tech stack. If you have years and years of experience in our tech stack but you’re not going to come in and engage with the team and go out of your way to learn, you’re not going to effective necessarily in our organization – and, I think frankly, in a lot of organizations.

We also look for people that are just passionate about technology. Is this a job for you? Is it something that as you go through life and you encounter problems, do ideas pop in your head, “I could solve that with a quick program?” People that are truly passionate about technology and live and breathe it are of course people that we want in our teams.

Ledge: How do you interview for that stuff?

Brandon: Our interviews, we spend a lot of time just having conversations, just like this one. Getting to know people, learning about their hobbies, learning about what they like to do outside of work even. If your hobby is playing video games but you built an app to help you play video games, that’s more telling than the guy that just watches Netflix all night long. Both are fine, but they just paint a different picture of maybe what they’re going to be like at work.

Then to evaluate technical skills, we actually do a pair programming interview. An engineer will come in and work with a couple of our engineers on a real world problem. We’ll even pull an item from our backlog and work through it with somebody, hands-on. See how they think. It’s not really so much about getting it right or getting it wrong, it’s evaluating their thought process and their problem-solving skills, and seeing what kind of questions they’re going to ask. Are they going to go reach for Google and try to find answers, or they’re just going to give up? Things like that.

Ledge: Yeah. Stack Overflow and Google, right? Everybody’s best friend.

Okay. I said that was the last question, but I also want to know when you think about bringing on new people, it really sounds like you guys have a great process together and that you’ve been really deliberate about the engineering quality.

Maybe just finish this up with some tips for orgs that don’t feel like they’re there yet and don’t speak with that level of confidence around the process that they build. How do you get there, if someone’s listening and thinking, “Jeez, I really want to be data-driven and that good”?

Brandon: I think the biggest thing is, solicit feedback from your engineers. Have an open dialogue with them. Ask them what’s working well, what’s not working well. If you have senior engineers or people that you trust, don’t be afraid to solicit direct feedback from them. Encourage everyone to give their peers feedback.

Developing a culture of radical candor, if you will, and being comfortable not being too politically correct. Being okay giving people feedback and of course, receiving it as well really helps an organization grow and get to where it needs to be culturally and on an efficiency level. If you’re not comfortable talking about the things you don’t do well, then you’re not going to have a way to fix them. You got to know about them.

That usually starts with leadership. Leadership needs to be okay receiving feedback, but also giving it.

Ledge: Absolutely. Well, Brandon, really cool to have you on. You guys are doing some great stuff. A lot of good insights there. I appreciate it.

Brandon: Great. Thanks for having me.