Scaling a team to support every stack with John Isdell of Red Ventures

Transcript

Ledge: John, good to have you. Thanks for joining us.

John: Thank you, Ledge. Good to be here.

Ledge: Can you give just a little background about yourself and your work a couple of minutes, introduce yourself to the listeners?

John: Sure. My name is John Isdell. I’ve been doing IT for at least the last 20 years, and when you start talking in decades you know it’s been a long time. Started out as an engineer and then started moving into leadership about 10 years or so ago.

I’m currently with Red Ventures, right now, in Charlotte.

Ledge: Talk a little bit about Red Ventures, if you don’t mind. I did some reading but I don’t know if everyone is familiar with it. Pretty massive company doing some interesting things.

John: Red Ventures, I guess the best way to put it, is a digital portfolio company. So, Bankrate.com, CreditCards.com, Reviews, My Move, we own those and we’re using those to leverage other marketing companies. I’m sorry. We’re using it for marketing for companies, basically.

Ledge: All of us have used Bankrate at one time or another.

John: Yes, exactly.

Ledge: And never put your email in those little boxes, I’m just saying. I don’t want to cut you off at the knees there but…

John: I would greatly appreciate it if you would. That would be something we would like.

Ledge: Have done and will continue to do, just for you.

John: Yes. I do it all the time as well.

Ledge: What’s the general DevOps and engineering story behind all that? You guys have been a major player for a long time. Tons of properties. I imagine you’re dealing with all kinds of legacy infrastructure. I can remember, going way back, that a lot of your sites were there far before some of the more modern stacks came around.

So, what do you run on? How does that look? Tell us some stories.

John: From what we run on at Red Ventures, I can just say, yes, we run on everything at this moment. I came over as Director of Platform Operations, and what my team is trying to do is modernize everything. Move as much as we can to AWS, to the cloud environment, so we can get away from some on-prem servers.

But in this case right now, we have some sites in WordPress. We have some sites in Drupal. We have things written in Go. We have React. Basically, anything you can think of at the moment that is either hot in IT right now, or was hot let’s say five/six years ago, we still have.

Ledge: How do you go about arranging human resources, engineers, to even deal with all this stuff. You must have a lot of cross-disciplinary types. I imagine all types of engineering resources. Debugging the Go stack is a whole lot different than going back and debugging maybe the Enterprise Java stack or something you might have.

John: Yeah. With that being said, being in leadership for so long now with larger teams, I have a great benefit of hiring people who are so much smarter than me. Going around the fact, though, how do we deal with it?

Really, it becomes more of a prioritization issue. Do you, for example, try to troubleshoot PHP or do we work on a plan to get you to something else? Do we try to work what we can into the cloud? What is it that we’re trying to do here?

You do have a bit of an issue where you have to hire people with a lot of experience with let’s say, the Linux stack, older stack but also want to learn a lot more about AWS, Jenkins, CircleCI, even Bamboo. You name the technology, we need to find someone who has expertise in some part of it but then also wants to build more, wants to learn more and keep going and keep expanding their knowledge.

Ledge: How do you choose which thing you should migrate to in any given case? Because today’s hot stuff is going to be tomorrow’s legacy nightmare that no one wants to maintain anymore. Is that an ongoing discussion from a strategic standpoint? It’s got to change every single year.

John: Yes, absolutely. It is ongoing. Right now, you’re going to get to a point where you say, “Let’s standardize on this for the year or the next six months.”

One of the great benefits of being at Red Ventures is the culture that we have there. One of our main principles is that everything is written in pencil. If that’s the case, then we make this call, we see how it works, we start going through some changes, see if it is a good fit for everyone. And if not, we can make some modifications to it.

At Red Ventures there’s a point where sometimes you hear, “This is how our culture is and this is what we say we do,” and then you get involved in it and you realize that’s not really what happens. Here, is complete opposite. We believe in what we say. We do it the best way possible. Weave the woodpile higher. All the things, like being written in pencil, even from a leadership side you have to realize like, “Look, I made this call. I could be wrong. But let’s figure that out and let’s keep moving.”

You have to make sure that your team knows that they can talk to you about that. That they can say, “Hey man, this really… I was onboard but not anymore,” and that that’s going to be receptive. That you can make these changes.

Even sometimes when it’s not receptive, when I say, “Well, let’s just try it a little bit longer,” then I have to realize I have to give up eventually. You can’t keep doing the wrong thing. If people aren’t following suit, you have to find out why that is. Generally, it’s just a mistake in your process, it’s something that you thought was going to work but just keeps dying – which so many servers die – but just keeps going wrong.

You want to make sure that everybody gets their say and that we keep the customer happy. We’re making sure that our full stack is working and that, if in six months we need to make a change or if in six days we need to make a change, that everybody accepts that and communication gets out and we just go from there.

Ledge: What other tenets do you have besides everything is written in pencil? That’s a great tenet that drives home the, hey, we embrace change around here. What are some of the other cultural tenets that you stand by in the organization?

John: One of them that’s one of my favorites, is why I chose to come to Red Ventures, is we leave the woodpile higher.

What that really means in practice there is, Red Ventures has a lot of nonprofits that we work with. One of them, for example is Road to Hire. If you have an issue where, for example, you can’t pay for college, your family has no money, you’re struggling with high school, et cetera, for whatever reasons. You graduated, you got through, you got a GED, you want to go into tech, but maybe you can’t find a place because it’s a very tough market out there, this organization allows those students to come through an actual very competitive process, learn to code, learn to do some frontend coding, some backend coding.

Then if all goes well, they start working with Red Ventures, fulltime employees, to get more experience. That, to me, is a wonderful program to be in and work through.

Personally, I think we’ve all had those issues of do you go to school, do you not? Nowadays, with how much tuition costs, not everyone is going to be able to go. So to have a program like that is excellent.

Another one that I’m working with with some of the people, it’s called Forward787. The CEO of Red Ventures is from Puerto Rico. When the hurricane hit, he was looking and saying, “How can I help?” So he went down there doing what everyone do, bring water, things like that, but then realized that the issue of Puerto Rico at that time had to do more with a brain drain, people leaving and not coming back and couldn’t find jobs.

What this Forward787 does is, right now we have 40 people – 20 who are in Puerto Rico, 20 people who are from Puerto Rico and came to the States – and, I don’t know the full terms of it, but as long they committed to going back to Puerto Rico and starting working there, that Red Ventures, we have jobs. We’re trying to teach them how to do the business and try to build the economy in Puerto Rico again.

To me it was really cool when I heard about it. Hey, you’re interviewing or whatnot. But now to see it in practice, it’s just amazing to give everybody chances, from both the Forward787 or the Road to Hire, to give them a chance to broaden their horizons on IT and do things like I do and probably all your listeners. Just love this technology and try to figure out the next cool thing and help people along the way. It’s really amazing to me.

Ledge: That service mindset, I think, really plays into, we’re based heavily on the free and open source ethos. That’s how we got our start, being in the hacker community.

Do you guys embrace that stuff as well, contribute back to open source projects? It sounds like that would fit well with the culture.

John: Yeah. I know that we do. My team is working with a lot of open source tools, like Jenkins, for example. As we do fix things, I know that people have. I can’t tell you how many or anything like that, wish I could. But I do know that, I think we’re at something about a thousand developers or something now. That’s always the talk that I keep hearing. Again, I wish I could speak more but I know for sure we do it.

Ledge: So, what’s next as you’re planning next year’s roadmap? What are the key trends? I’m sure security, quality assurance, everything continues to level up how you set those priorities when you have a thousand engineers from the leadership perspective.

John: Absolutely. Security is really key. We all try to keep up on it. We all try our best. We’ve got, right now, three or four tools that are scanning and letting us know what’s going on.

We always bake that in as much as we can when we’re developing this platform where, basically, it’s a cloud hybrid full CI/CD. So, your developers can just say, “Oh, I want to do this and go. I need it to support 10,000 people,” et cetera.

That we have good where you’re building out these images, and make sure that we’re using Lambdas, we’re using UCSs, whatever it is we have to do. You make sure things like S3 buckets are public, for example.

Those are all things that you want to bake into your Terraform that you want to work with, and just keep checking and making sure because you’re never going to be perfect with that. But you want to have your framework and your guidelines. For my team, we work hand in hand with our security operations team.

I think I’ve heard in one of your shows prior that all these lines are being merged now. It’s not just DevOps. It’s not just security ops. It’s not just developer. We all have to work together.

If you’re not putting in a patching strategy or you’re not putting in a security strategy around anything you develop, even if it’s just a static HTML page nowadays, if you’re not doing that, you’re opening yourself up to any sorts of malfeasance or any problems out there. Because it only takes one person who just is bored and really smart and at a computer to take you down, take data.

Look at what happened to Marriott. It happens across the board. I think I heard a couple more today. It just keeps happening, and you have to stay on it and be vigilant from a security perspective.

When you ask, that’s what’s been my main driving. Especially towards the end of the year and going into next, that’s what we’ve really been focusing on.

Ledge: How do you get that security mindset across the culture? Everybody writing code needs to be thinking about that. Upstream in your ops process and downstream in your QA process, you can be scanning, you can be looking for vulnerabilities, but it does go down to each individual engineer. You’ve got to write security-compliant code. Which means you really need to be up on standards and practices that you probably weren’t two years ago because, unfortunately, not enough people were talking about security.

John: That’s very true. Fortunately, we do have a security team on its own as well. They always keep you in check, which is great.

But separately when you’re in your teams, when you’re just leadership of another department – like in this case, in platform, if you’re IT leadership in development – that always ends up becoming, at least in my side part of the scrum, part of what you’re working on this week. What you’re looking at.

You just make sure that leadership is all in a line. If you have team leads or managers that have your direct reports, you want to make sure that that’s always a constant conversation.

For me, I’ll look at different blogs, different websites. “Here’s the latest issues that we have.” Oh, from way back when, libc has an issue. You name it, this is what we have to look into.

I’ve done that continuously and whenever the next zero-day attack is, we’re all going to know about that. When you move to something like AWS and you do more things in the Cloud, it gets a little bit easier, only because the infrastructure isn’t 100% yours.

But just because you’re in the cloud, just because you’re at AWS or you’re at Azure or you’re at Google, does not mean that you can stop scanning your systems, and making sure that your passwords are okay, and using two-form identification. You have to keep doing that. You have to remain vigilant on it because the minute you don’t, somebody is going to find a way in.

Ledge: Sure. They say some enormous percentage of breeches are really just social engineering attacks. It is so much about that human culture. You can train the technology all you want, but if someone gets breached and their Twitter account uses the same password as their root password then it’s game over.

John: Very true. Part of our wind-down, our chief security officer was telling everyone to go home over the holiday and change all your passwords everywhere, just to be safe. That’s the culture there. Making sure that everyone does it.

What’s also interesting for any freelancer who works, as someone in leadership, I find myself saying the same thing 50 times a day. I always apologize for that, because I never know if I told that person three times a day, if I told the same person the same thing. But as long as you keep that message consistent, people will start following, start thinking about it and hopefully the next day they’ll be like, “Yeah, I remember you. I remember you mentioned a change of password. Let me get on that.”

Ledge: I ask everybody this. Obviously, we’re in the business of evaluating, locating, staffing, always bringing to bear the best of the best software engineers. We have very strong vetting and heuristics for doing that, proprietary systems, but in good lean fashion. We’re always trying to make that better and constant improvement.

Every tech lead that I talk to on the podcast I like to ask, what are your key heuristics for identifying just the very best software engineers that you can add to your team when you’re hiring?

John: Mainly when I look at it, I want to say team fit, culture fit, things like that. From the Red Ventures mindset, how do you feel if things change?

Some people want everything very strict. “I just do this and this is what I do.” That used to be okay 15 years ago. That was a great mindset. You want to be 100% perfect on what you do and keep that going.

Nowadays, even the work is agile. It’s not just the tickets or the scrum board, it’s, “I’m working on this right now. Oh wow, that’s incorrect. But let me make an update in the change and communicate that out. Or just build an automation that I don’t have to deal with anymore.”

I always want to keep that you are acceptable to change. In life, the only thing constant is change – I believe that my whole life. You want to keep that going.

Do you have a good sense of humor? Are you a good person? Are you looking to actually make things better? When you’re with the team, and whether it’s remote or you’re right in front of them, you end up spending more time with them than you do, generally, with your family. You just do. You have to make sure that you get along. You have to make sure that we’re all going in the same direction. We want to learn something new. We want this to be better.

One of the other tenets is, we get better every day. What does better mean? Do I learn more? If I’m a junior tech, do I learn to communicate better with other departments, and update a ticket or something and say, “Hey, I’m working on this for you. I’ll get you in a minute.”

Do I literally get better at development and from the development side I also know how to do a Lambda in AWS. It’s not just my own side of the world.

Am I looking to keep learning? There’s so many times that I had to tell people – because I think it’s human nature – they’ll say, “Well, I didn’t go to school for this. Is it okay?” Yeah, that’s fine. From what I understand, I don’t think Steve Jobs finished school. That’s cool. They learned it and so did Gates.

I’ll help you build that confidence level, as long as you’re also going to have a little bit of confidence in yourself that you can do this job. Generally, if you’ve done this for even a couple of years, you should have a little bit of confidence that, “I can knock this out.”

I’m looking for those sorts of things too, when you’re interviewing and you’re talking to somebody. In the case now that you have a larger team – and I think this is a leadership’s prerogative – but as you build a team, you start realizing, oh, I have people who are really senior at Linux, but I don’t have people who are really senior at AWS or Azure, you name it. So now I’m going to start looking for that bit of that technology or that skillset as I start interviewing people.

Sometimes that happens too. Depending on what’s going on, you might… Maybe you came in with AWS but you just hired five AWS people and now we’re starting a project on Google Cloud. So now, I’m like, “Oh, man. Okay. Maybe I need more Google Cloud people.” That just ends up happening as you grow a larger team.

Ledge: You’ve got a thousand people out there. Do you have some kind of internal mechanism where you can say, somewhere in that thousand there’s a representation of every possible permutation of all these technologies? Do you go shopping internally off your bench to say, “In fact, well, we do have those experts. They’re just doing something else right now.”

John: Yeah, absolutely. That definitely does happen. You get to a point of, no matter how big you are, resource constraints.

You’ll have one team say, “Well, I need a dedicated person,” or, “I need this person working on…” I can’t do that, because if I did that everybody would take the dedicated person. As it is right now, there’s a lot of competition on the engineering side.

Then also, you’re going to get engineers who say, “Well, no, I really just want to focus on security with AWS,” for example. “So, I don’t really want to know about PHP, WordPress. I don’t really care.” But I go, “But you need help.”

You have to build out from all those books, not just I-shaped engineers, T-shaped engineers, E-shaped engineers, et cetera. You want to try to do that because I feel that when you have people in that regard, they can talk to more people. This way, you can utilize them for many different teams and actually have a good collaborate effort around other departments.

Ledge: Okay, so last question. It sounds like quite a large engineering organization and you have had success building and managing remote, distributed workforce, probably of all different types. You’ve got consultants, freelancers, you’ve got in-house.

How do you balance all that out and make sure you have the right mix of people? How do you manage that well from a distributed standpoint?

I ask that because a lot of clients and companies in general are now struggling with that mandate, to have to go to distributed workforce. You just simply can’t get the talent unless you think that way. You hire different types of people. If they’re 1099 or W2, you hire different people. Some are working from home, some are working from offices.

How do you put that together in your management and leadership strategy, to deal with that new mandate that is really driven by a labor market that simply says the best talent is going to work where they want to work now?

John: Absolutely. Actually, as an organization, as Red Ventures, I believe now we have seven global offices. So, you have people in Brazil that you’re working with, and UK, et cetera, and across the United States.

For a lot of my career, I’ve worked with offices around the globe. I was in futures trading for a long time, so you had to go Singapore, Japan, Sydney, et cetera. It’s been something that I’ve worked on for a while.

One thing that I think has been successful is over-communication. What ends up happening when somebody isn’t in front of you is that sometimes you forget they exist. You’re talking to the guy next to you because it’s easier. But now we have Slack or we have Skype or, name the tool, where you can have communication. You want to make sure that you have a lot of that happening.

Slack is great because if you have like one team, for example platform engineering, you can have everyone in that room, and that’s who everyone’s talking to.

I’m also a big proponent, when you do remote offices, please, please, have your picture there. Have a picture of yourself, not some avatar of, I don’t know what. Because it does make it more personable when you talk to somebody, when you’re Slacking them, I know what you look like. To me, I’ve found that very helpful.

But you have to make sure everyone’s included. I remember reading some management books a while ago is that, apparently the same psychological effect of being left out in the wilderness by your tribe, happens to you when you’re not invited to a meeting. I don’t know why but I guess it’s just how humans work.

You have to make sure everyone’s included. Even if they can’t make it to a certain event, you have to let them know it’s going on and, “Hey, if you can, great. Show up. If not, here’s what we’re going to do for you.” You have to make sure that they’re always engaged and always part of the team.

Sometimes that also means giving them work that you think that they can’t do because they’re not here. That’s really a different mindset from leadership, but you also have to bake that into your team or weekly meetings or whatever it is that you do, to say like, “Hey guys, remember, this person is here and you can reach out to them whenever you want to.”

And then, as leadership, I always feel you have to make that effort more than others. Say, “Hey, I was talking to this person and they said this. Would you mind Slacking them and talking to them?” Give them the Jira, have them work on it, et cetera.

You have to be the communication catalyst or it will just fall through the cracks, because you’ve got a lot going on that day. Your engineers are very busy.

It’s kind of sad that as leadership sometimes you have to take yourself away from the tech – b because I think that’s why we all got into it. We love technology. We want to learn more. This is so cool, how this can happen. But when you start dealing with larger teams, you literally become a cheerleader and the communication person.

Ledge: Great insights, John. Thanks so much. It’s good to have you on and appreciate all the smart thinking.

John: Thank you very much. I really appreciated the time. I guess now I’m going to go play Fortnite and do the Floss dance.