Measuring the Impossible with Data Science with John Feland

Transcript

Ledge: John, thanks for joining us. Really cool to have you on today.

John: Great. Thanks, Ledge. Appreciate it. Glad to be here.

Ledge: Can you give your quick background story, a couple minutes about you and your work? I’ve seen your resume. It could go on for many, many minutes. So, maybe you have to focus here.

John: I’ll give you the short, sweet version.

The passion that drives me throughout the tortured path of my career has been how to measure things that most people think are impossible. Whether it’s trying to measure missile launches when I did intelligence work in the Air Force, or measure the impact of design on corporate performance during my pile high and deeper days at Stanford, or trying to instrument the entire consumer market in the work I did at Argus Insights.

It’s all been around, how do I find that lever of metrics that helps drive people towards taking action that improves whatever the context is, business or otherwise.

Ledge: So let’s get in there. How’d you do it?

John: Very carefully. No, I think it’s interesting. One of the things I’ve seen across the different contexts I’ve worked in is that, a lot of people think that if you have all the data, that’s all you need to do. What I’ve seen time and time again is that, actually figuring out what the right data is is actually more important. In many cases, it ends up being less than you think and it’s more around how do you tease out the pieces around the outside.

For example, one of the things we were able to do at Argus Insights was beat Wall Street estimates of iPhone sales almost every quarter for about five years running, and all based on just using the consumer reviews of iPhones across the world.

Ledge: So, you had to draw in tons of data but find the right data. I think that story happens a lot. Everybody I talk to, it used to be big data, now it’s data science and it’s machine learning and it’s AI. Really, what we’re talking about here is, 80% or 90% just nasty old ETL done in different ways. Then we get to be smart, and we get to make machines learn and have actual intelligence or artificial intelligence, whichever kind you like.

Talk a little bit about that because that’s the unglamorous belly of data.

John: I was dealing a lot with text data, some very messy text data, trying to figure out is this a real human that wrote this review or this tweet or are we dealing with another bot war?

One of the pieces I dug into within the IoT, Internet of Things, conversation within Twitter, found that 75% of all IoT tweets came from compromised accounts. So, we’re looking at millions and millions of tweets about this big B2C transformative kind of marketplace, and most of it’s fake or compromised.

Ledge: And fake tweets haven’t come up in the news at all recently, so I’m sure you haven’t thought about that.

John: No. A lot of it comes down to, what’s the pattern of clean, what’s the pattern of dirty, and how do you squeeze the right blood from the stone, if that makes any sense?

There’s limits to each data set you bring in about what you can pull from it. I was able to use the review data to figure out why certain phones were taking off or not, as the case may be. Actually saw the Amazon Fire Phone, if you remember that beast, crash within three days after launch. Before the news was reporting on the failure, we had the canaries in the coal mine saying, “This phone’s terrible,” and knew why – It’s battery life. It’s this, that and the other.

Two months later when Greenpeace staged a protest by writing bad reviews on the phone, it didn’t actually have any impact because it’s already so low anyway.

Ledge: You already knew that.

John: It’s ETL, right? It’s ETL. Then you have it cleaned off enough where you actually can do something with it. You have to be careful that you don’t oversell it. It’s about backtesting. It’s about figuring out the limitations to it. Communicate those stories in a way the client, the user, the algorithm know what they’re getting.

That’s the other piece that comes into play, is that you can get in deep in the data. Like, “Look, I cleaned the data. It’s beautiful. This is my wonderful, clean data.” And they’re like, “Yeah, but how does this impact what we’re doing at this meeting right now? How is this going to impact next quarter?” Being able to bridge those gaps between what we find in the data and what it means to the enterprise is a key feature in all this.

Ledge: You have extensive experience in writing and speaking and teaching about design thinking. I’ve got to imagine that comes in there in that, viewing the data in a context that makes sense to the user. How have you brought that together? A lot of people don’t have that shared discipline.

John: It’s part of my schizophrenia, my multiple personality disorder, for lack of a better phrase. I think it’s one of those things where key tenets of design thinking are, how do you develop that empathy and compassion for your user and start using the words and language that they know. So that as you tell them the story of what’s next, it’s a story they feel they can have a starring role in.

A lot of time, data scientists forget that part. We get excited about P values and the annealing of our convolutional networks. “Look, the weights of the neurons haven’t shifted in three decades, yes!” As opposed to what it means to the bottom line to people. That’s where that design thinking comes into play.

Along with this notion of, how do you creative bad prototypes that you can share with clients? Part of that is because it’s very easy, from a data science standpoint, to try to have everything perfect and dialed in before you share it with a client. One of the things that I find is that when you show them bad prototypes, they become co-creators with you. If you show them something that’s too polished and too pretty, they’ll say, “That’s really cool. Can we change the color on that third line on the second chart please?” When that’s just code for, “I have no idea what you’re doing and I’m just going to comment on the things I can.” Whereas, if you give them something that’s a little bit messier, it allows them to ask deeper questions and question even the direction that you’re going with it.

That’s another place where design thinking and the data science comes together.

Ledge: Obviously, we’re talking about agile and lean and all these things. The supposition being then that you can launch any product. Even if it’s not a product, it’s a data result, you can launch it and deliver it in a lean and iterative fashion to involve the user in the ownership and adoption of those final results.

John: Part of it is, if you look at how the democratization of data has enabled us to have any number of tools. I can run some model through 16 different machine learning algorithms in a matter of minutes. I can put together a Bayesian model. I can pull this library here and that library here and pull in data from census.gov and Twitter. All these could’s.

The big challenge we face as an industry is, how do we distill those to the should’s. That’s where that piece comes into play. By getting clients or users or vendors or partners involved in co-creating those should’s, you end up with a better chance of like, “Oh, wait. That means… But that… Oh my goodness, that’s what’s next. That’s great.”

Ledge: You’ve gone hardcore through the academics to the military, to design, to business, to self-employed business, back to education now. This is like a full circle story, that back to the roots, feed the next generation? What’s going on here?

John: It’s very much a give-back kind of thing. I grew up in a small town in rural Arkansas. Used to raise chickens for Tyson Foods in a past life. My parents told me, “If you do your homework, you don’t have to raise chickens for the rest of your life.” And I said, “Okay. That sounds good.”

Had some great teachers that encouraged me along the way. I got a special visa to leave the state when I was in high school and went to MIT for undergrad, did graduate work at Stanford and got stuck in the Valley. Along the way I got the teaching bug very early on.

When I was in high school, I taught 5th grade science. So any chance I’ve had a chance to get into the classroom. My last job in the Air Force was teaching at the Air Force Academy. I ran a program at the Stanford Mechanical Engineering department for a number of years.

It’s always been part of the threads, but now one of the things I started Argus Insights to do was to try to generate enough free cash flow that I could go start a school. Then I found out about the Nueva School where I’m teaching now and I’m like, “Wait a minute. You already did all the work. Can I help?” That’s what I’ve been doing for the last nine months.

Still doing the data science stuff on the side because I can’t let it go. It’s too many interesting questions to answer and too many people to try to help figure out what’s real and what’s not. But being back in the classroom it’s a joy, it’s a gift.

Ledge: That type of thread comes up a lot in community-minded folks, how do I leave time and my schedule to… I really want to contribute to opensource on the engineering side. But I got to get paid and I have to maintain at least something there.

We have some really thoughtful engineers that go, “Hey, I’m only going to bill 25 hours a week and I’m going to put 15 into my education or my volunteer work.” Maybe talk a little about that balance and how maybe, is it a thing that you ended up doing because you finally got the free cashflow and you designed a system on purpose? Or was it something that was always there that you were able to maintain?

John: It was always a little piece. I’ve got two daughters, a middle-schooler and a high-schooler. I’ve been volunteering and helping with their school programs for a while. A design contest at a local tech museum that my youngest has been doing for the last three years, now both of them are doing.

It’s been a way to give back for my kids initially, and then trying to broaden that impact over time. I think a key part of that overall piece is – I forget which pundit said it – but paying yourself first, whether it’s in social capital or financial capital, is a critical piece. I find, for myself, my financial pursuits were much more balanced and much more effective if I’d actually done something to help other people outside of me as well. It’s that mission-driven piece I picked up in the Air Force and Scouts and all those kinds of things. It’s nice to have that mission that drives it as well.

It’s like, yes, I’m doing this to pay the mortgage, but I’m also doing these other things to pay the soul.

I think that’s critical to figure out. It’s not every week I wall off 10 hours to do this because life doesn’t work that way, especially if you’re a freelancer, but it’s, how do you create that average? Go back to the data science piece. What’s my average contribution and how do I feel about that? Balance itself ends up being a dynamic, and how do you adjust the knob? Some days, work is up to 11:00. Some days, giving back is up to 11:00 – to use a tortured spinal tap reference. I think it’s about making sure that you put that as a big rock into your schedule, into your life first.

Wallow in it. Enjoy it. It’s like, I’m here because I want to be. This is great. Whether it’s contributing to an opensource project, whether it’s volunteering at a local school to help stimulate the next generate to go, “Wait. You mean I can be an engineer?” “Yep.”

That’s part of what got me started. My last year at MIT, I was a mentor coach for the local FIRST Robotics competition team. This was in early days when we were still in the gym. It had taken over the Georgia Dome and things like that. Still in a gym in Nashua, New Hampshire, wow.

Anyway, I got a chance to stimulate and spark and give active permission to these high school students from inner city Boston, but for some of them it’s too late. They were seniors. “Oh, this is so cool. I want to do this.” “What’s your GPA?” Urgh! That’s what sparked a lifelong, how do I start reaching kids earlier and having an impact earlier and give them that active permission, “You can do this. Yeah, it’s hard but that’s part of it. It’s part of the joy of it. It’s climbing those learning curves together with other people, because together you can do cool things.”

Ledge: Talk about those learning curves. One of the things that always is super popular in this show here is just the blazing, amazing failures that became the phoenix of the future successes.

What are the speed bumps along the way, that it doesn’t read like a perfect LinkedIn profile when you get to the real story?

John: No, that’s very true. Things like not having the right backend pieces together, and having an intern wipe the entire production database because he initialized a new Ruby on Rails and just said, “Yeah. Let’s just migrate the DB.” “Wait. That was… You just did what?” But we had backups for our backups for our backups.

Failures along the way, there’s more speed bumps than I can go into. Things like, I still haven’t figured out my sales team merit badge. I’ve tried four or five times and failed significantly on that piece.

Learning lessons around how selling bad news doesn’t work very well, especially when you’re trying to sell technology to marketing folks. “Wait. What do you mean that the hundreds of thousands of dollars we spent on ad campaigns the last six months have not moved the needle from a mindshare standpoint at all?” “Yup.” “Can you not tell my boss?” “Sure.”

I think the biggest thing I’ve extracted from the last several years of trying to spin data into gold or at least insights is finding a way to sell money. It’s less about cost savings, it’s more about on the revenue side. If you can find ways to help companies sell money, that’s a beautiful thing.

Ledge: Talk more about that, selling money. I’d like to buy money.

John: That’s where Wall Street does a great job, “If you give us a little bit of money, we’ll turn it into bigger money.” “Well, what’s the risk?” “Oh, let’s not talk about probabilities.”

But I found that on the Argus side, the things with the biggest impact, the things where we were able to really move the needle for clients, created the case studies and the stories and the business model that actually helped.

For example, we helped Best Buy renegotiate one of the vendor contracts with a major tablet manufacturer. They were getting huge returns. Huge returns. Anything that wasn’t an iPad was like, “That was cool but take it back,” and they had no idea why. We were able to dig into the review data and pull out what was driving people away and what was driving the returns. It turns out, in all the ads for this new tablet that wasn’t an iPad, the company was promising, you can do this and you can see pictures and videos and games, and oh my goodness, all your content from your computer comes into this tablet. It’s portable and beautiful. Imagine your life with this new device.

You get it and you buy it and you bring it home, you go to connect it to your computer and they didn’t include a cable.

In order to save their overall bomb costs, they’d made the USB cable to connect to the computer an accessory they had to buy separately. So, they boxed it back up, take it back to Best Buy and say, “Take this. I want an iPad.”

Best Buy did two things. They went through and changed the way the blue shirts interact with customers. “Oh, you want that one? Here, get this cable too.” Returns dropped. Two, they went back to the vendor and said, “Hey, we have all these returned tablets, take them back.” “No. No. We’re not doing that.” “No. Here’s the evidence where you actually messed up.” “Okay. We’ll take them back.”

So, for a small project, Best Buy literally saved millions and millions of dollars. Their return on that investment was 2000%. Who knows? It was significant. It was one of those things where it sold money.

And so, whenever project work can be that lever that drives new opportunities or new revenues to a client, that’s a beautiful thing.

It’s how you tell that story, right. The promise of saving money is tough because no one ever gets rewarded for the bad things they avoided. No CEO says, “Well, this quarter was a great quarter. We didn’t go bankrupt.” That’s your job. We don’t get rewarded for those things. We get rewarded for the new things.

So, if there’s a way to mix those pieces together where it’s some prevention but a whole lot of new as well, then you’ve earned that right to do that again for the company, over and over.

Ledge: You just told me that story. Now, I’ve been in the sales seat for about a decade and that sounds like pretty good sale to me. So, I don’t know how you failed out of your merit badge but…

John: It comes back to… Yeah. That’s a good question. When I try to scale beyond myself, that’s been the hard part. You know that a lot of times sales, especially relationship-based sales on the enterprise side, it’s a hearts and minds campaign.

Again, going back to my multiple personality disorder, I can get technical but I can also get empathetic with the customer in a way to kind of, “Well, yeah, let me think about how we twist it. Yeah, we can find that data. Sure.”

The Autonomous Vehicle Alliance was trying to figure out how first time autonomous uber riders were feeling about the experience. I’m like, “We can do that. I can think of three different ways we can get the data and do… Yeah, it’s about the scope.” Closed the deal.

Whenever I try to enable someone else to do that, without that schizophrenia, that right-left brain meld, it was much difficult. But, yeah, we’ve actually found the riders and scarily tracked them in Instagram to figure out what else they cared about. Don’t share too much on Instagram. It’s creepy what we can do.

Ledge: Noted. So we got an engineering audience here and I can tell, just every single time I talk to anybody in any type of technical leadership role, it’s empathy, empathy, empathy, empathy.

I was a software engineer when it was okay to be in the basement and all the lights were off and 12 of us were banging on code. We didn’t want to talk to the customer and nobody wanted to let us near the customer. In the last 20 years that has changed a lot.

Maybe just finish us up here with some advice on developing that secondary character, because you’re a highly credible engineer and you have this other flip side of the personality. How and where? How do people do the actual practical work to make that happen?

John: That’s a good question. The challenge we have as engineers is that we’re really good at designing for two people, ourselves and mom, because mom loves everything we do, right.

I took a page out of the guy who started doing design work at MIT a thousand years ago, that Stanford poached and started the whole Design Thank You Movement back in the fifties, John Arnold. He had the same issue with his engineering students, so he had them design for aliens from Arcturus IV as a way to force them out of their own mindset.

Anything that causes you to force yourself out of the mindset is good. You don’t have to design for aliens, though I do find that leveraging personas, interviews or even observations are really important for a couple different reasons.

I forget the name of the company but there’s a medical device company that, every time the design team meets, they have an empty chair with a picture of one of their patients there. That way, every time they’re like, “Oh, if we do this, we can cut cost,” they’ll look at the patient and say, “Wait. Is that the right thing for Bob over there? Will Bob survive more or less?”

I think the same can come true for engineers and developers working on projects. Is, how do you bring the persona of your customer in in what you’re doing? Not just demographics. I think demographics can be a path to pain for engineers. I think you have to understand the psychographic piece. What matters to them? What story do they want to have a starring role in? That becomes really important. Even if it’s some kind of middleware that eventually is going to waterfall to a real human who you’re going to make a hero. That kind of hero-based design becomes really important.

Think of yourself as Q to James Bond. What new gadgets can save the day, and how do I understand my bond – okay that could be tortured from a pun standpoint – my bond with Bond well enough to then figure out what characteristics, what situations, what scenarios, what stories to go back to a natural point of view, that can enable that?

Without that persona anchor, it’s very easy to fool ourselves the same way that entrepreneurs fool themselves in Excel spreadsheets, “Look, it’s going up!” We can fool ourselves with personas around, “Oh yeah, that’s real.” But the more you can get real people engaged and make that into a composite persona that you can leverage…

Because are asking questions. It’s not in a, it seems a little creepy from the outside, why are you talking to a piece of paper? But that conversation, that dialogue, real or imagined, helps anchor you to make sure that every decision you’re making from a design standpoint ties back to making that.

Ledge: Great insights, John. Thanks so much. It was really, really cool to listen to your story and to have you on.