Mobile App Security and Threat Defense with Domingo Guerra of Appthority

Transcript

Domingo: Thanks for having me, Ledge. It’s a pleasure to be here.

Ledge: Fantastic. Could you please give a little background story of yourself and your work and the things that you’re up to, so the audience can get to know you?

Domingo: Sure. I’m originally from Monterey, Mexico. Moved to Texas for school and then ended up in the Bay Area as this is where all the jobs were.

After some stints in engineering and the robotics group for Applied Materials and then operations group for Brocade, I decided to start a company focused on mobile app security.

We started Appthority in 2001. Really saw a big boom of how enterprises were using not just mobile devices but mobile apps increasingly for work related topics, and we wanted to help secure and protect both the employees’ and the corporate data.

After growing the company for about eight-and-a-half years, we ended up being acquired by Symantec last year, in October 2018. Now we’re really excited to be able to take our work to a global scale. Protecting enterprise but also protecting consumers, since Symantec has the ability to do both, has given us really the exciting task to look at every app on every device and see how they’re behaving, see how they’re being built, see what kind of technologies or new approaches are being implemented. It’s a pretty fun line of work.

Ledge: Congrats, first of all, on the big exit. That’s huge. Everybody dreams of that.

Tell me, mobile app security. Why’s that different? Everybody talks about security now, it’s hottest topic and nobody wants to get in an Equifax kind of situation. What is the space? Why did you jump in there and what has changed over the last several years?

Domingo: If we look at the traditional endpoints, so desktops and then eventually laptops, as an industry they’ve been getting new technology to secure, monitor and remediate threats for many decades. Then, if we look at traditional mobile phones in the workplace 10, 11, 12 years ago it was a BlackBerry-only environment. Only executives and maybe salespeople had BlackBerrys, and that was it. It was pretty locked down.

iPhone in 2007, and then shortly after that Android, really changed everything. It became every single employee, across the whole vertical enterprise but also across every type of organization or industry, started bringing their phones to work. With that, they started bringing their apps to work as well.

At first it was only receiving email but now it’s connecting to enterprise Wi-Fi, connecting to enterprise or private clouds, accessing work or personal apps. It’s really just burged into, personal security is now related to enterprise security. A lot of times, that’s not increasingly obvious to developers. They’re saying, ah, I’m just building a consumer app, I shouldn’t really care about security. But the fact that it sits on a work device or accesses work data as well, or employees use it for work if it’s maybe an organizational app or a productivity app, the importance starts increasing but it’s not always the focus is there.

The other thing that we thought was really interesting is that, in the old software model you used to buy a big DVD or CD-ROM and really install a huge package and you paid for every software you used.

In the app model where you download something very specific for a certain task – if you want this there’s an app for that, if you want something else there’s a different app there – has really pushed the developers to build lean, build small, build fast but build cheap. There’s not a big budget to spend on security tools or security analysis, and users are usually not paying for the apps either.

Maybe there’s a premium and then eventually paid services, but it’s tough for developers to monetize as well. They end up having to maybe collect some data and maybe sell that to ad networks or data brokers, which again is not always great from a security or privacy perspective.

Those were the market movers where we felt this was a great space to focus on.

Ledge: Yeah. That makes a lot of sense. It makes me wonder. The first thing that popped into my head was, you have security from the standpoint of eliminating bad actors. Then you have security from the standpoint of non-malicious app developer but just simply doesn’t have the wherewithal or the skills to use the best practices, or access the types of things that you’re talking about.

Those two different cohorts would be two different vectors for you to think about in security. I’m curious what the balance is, and how you address them. Is it literally shut it down? How do you keep the ecosystem alive in a secure way and benefit the whole community?

Domingo: That’s a great question, and it’s something where a lot of the CISOs, the Chief Information Security Officers, they struggle with the balance of how tough to be on certain aspects.

Obviously, if something’s malicious, by a malicious author, an attack… It used to be single person hackers, and then increasingly in our research we start seeing now nation type sponsored attacks, or coordinated attacks against certain companies or individuals. Obviously, for anything malicious you want to stop it. You want to eliminate the threat. Automatically remove the threat and address it.

The difficulty is that most of the threats, maybe in severity it’s malicious, but in terms of number or how likely they are to occur they’re non-malicious. So they’re good people, they’re good developers that either use the vulnerable third-party library or third-party software developer kit. Or, they’re just using a model of collecting additional data and they don’t realize that they’re also collecting enterprise data. There’s data leakage occurring, but it’s not malicious.

As a security professional, it’s hard to say let’s block every app that’s doing this because that’s most of the apps, unfortunately, sometimes on user devices. It turns out that we need to implement additional technology where, if an app is not properly encrypting traffic or sending passwords in clear text, maybe we don’t want to block the app but maybe we want to turn on a VPN to protect that traffic.

If we can work with a developer to fix it, that’s great, but sometimes we don’t know who the developer is or they’re not going to implement a change just because a certain user asked them to. It really needs to be driven by either market saying, hey, you either fix this or we’re not going to use your app or your software, or by policy – App Store or Google Play saying, hey, you need to fix this or else your app’s not going to be able to be distributed.

Ledge: How do you even figure out… You’ve got thousands of devices entering the corporate footprint, and all of them have different stuff on. I imagine you have to deal with device level and then network level and firewall. You have to touch each area then, so the type of solution you guys have to develop is not resident necessarily in any one place. Is that right?

Domingo: Yeah, that’s correct. That’s one of the interesting things with mobile, is it really came into the workplace really in the last eight to ten years that it became predominantly everywhere, but now it touches every component. It’s connecting to your corporate network. It needs to have access through firewall. It needs to have access to email. It’s now part of identity. That’s where you get your two-factor authentication codes.

Really, the whole enterprise stack goes through mobile, yet the technologies are not always there in terms of how to even monitor what’s going on. In some places, by jurisdiction, you might not be allowed to monitor a personal device so you see certain companies issuing people phones so that they can monitor and secure them.

It’s definitely a challenge, but that’s part of the opportunity, the exciting part, is that we have to rewrite the security stack with mobile in mind. A mobile-first or a mobile-only world requires that.

Ledge: Right. We used to be able to say, well, I can lock the door, but now you’ve got a thousand open windows in a building.

Domingo: You have people working from home, working remotely, connecting to coffee shops, airports, hotel’s Wi-Fi. Now it’s worrying about those networks as well. Not just your corporate network.

Ledge: Okay, you’re talking to 10,000 developers right now. Tips and tricks. Best practices from your experience building and growing a company, and from the perspective of, how do you do this right? You don’t want to be the one who gets shut down or causes a problem and gets your name on it and go out of business.

There’s a lot there, but I’d love if you’d share some stories and best practices.

Domingo: We recently conducted a threat research report. The finding, it’s a backend vulnerability. It’s called HospitalGown, just for the visual that the gown when you go to the hospital is left open and your back end is exposed. But that’s exactly what we saw.

Developers were securing their app, they’re using proper encryption, they’re encrypting traffic between the device and their backend, and they’re encrypting data stored on device, but then they forgot to lock the door on their backend. They’re using third-party tools.

We’ve seen this HospitalGown with Amazon Web Services that get left world-readable – not even a user name and password. We’ve seen it with Microsoft Azure. We’ve seen it with Google Firebase.

Really, any tool that you’re using as a developer, you have to make sure that you read and understand their security policies and their security best practices because sometimes a shortcut of saying, well, I want to be able to access this quickly, I don’t want to have to log in, means that no one else needs to log in either and bad guys are able to access your customer data, sometimes your backups. They could even delete your hard work, your app.

That’s, I think, one of the key takeaways. Just to show the scope of how this problem can propagate, a single application that’s very popular could infect tens of millions or hundreds of millions of users.

In the recent HospitalGown, we found about 500 iOS and Android apps with over 1.1 billion downloads among them. I mean, 500 apps is not that much when you’re talking about millions of apps in the App Store, but a billion downloads, that impacts a huge portion of the mobile world.

Ledge: Right. All of those devices reside in some kind of work context as well. Even if you’re not even using it, you’re sort of connected and in the sphere there.

Okay. That’s terrifying. What do we do about that?

Domingo: Just to dig a little deeper, the categories were also important. It’s not just games that might have this issue, but it’s productivity tools. Health and fitness apps are increasingly collecting more personal health information and personal identifiable information, PII. Communication. Finance apps.

In all, we found over 200 million backend databases that were exposed. Employee IDs. Credentials. Passwords. Just millions of subset of records.

How do we protect it?

Sometimes developers have security@, then the name of the app of how to reach out to them. We’ll reach out with advice and, hey, we found this please fix it. Very few developers even monitor or respond to those kinds of alerts.

One, we have to have – obviously, the big companies have bug bounties and teams to do this, but even as a solo shop we need to have security in mind. Maybe have a way to have folks reach out to us. Focus on best practices. Always encrypt. Don’t hard-code your credentials inside the app, because that’s another thing we’ve seen where developers put their admin root credentials hard-coded in the app and that’s just like leaving a Post-It note on your laptop, except shipping the laptop to a billion people that end up downloading your app.

Ledge: Sure. Or your code is checked into a repo that’s publicly available, and all your credentials and session IDs and everything are in there. Yeah. You read those stories all the time.

Domingo: Then, it sounds trivial but a lot of times our researchers, they go to the documentation on the library or STK. They go to the section that says, never do this. And we say, okay, let’s do that and see who else did exactly those [00:13:29].

No one likes documentation, but it’s there for a reason. I think it’s really important to be able to review that and think about the unintended consequences that can come from not following best practices.

Ledge: It makes me think that there must be maybe a higher bar to getting in the App Store in general. This is not a different problem than the proliferation of IoT devices that just run with ‘admin’, ‘admin’. A web server on your spoon or whatever thing someone comes up with.

It’s certainly a broad conversation that the technology has outstripped the ability, at this time, to manage at that scale. It’s kind of an existential issue for people to get their head around.

Domingo: Yeah, but I think that’s the challenge for both Apple and Google. Is, they’re already getting bombarded. The number of apps they have to review each day, each month is staggering. Obviously, they have to first focus on the malicious threats and the types of malware and really catastrophic events that could impact a user. Second, the worry about functionality – the app has to run and comply to overall terms and conditions. But looking for vulnerabilities or best practices is really low on their list of priorities that they’re reviewing for.

It takes a long time to get reviewed and approved to the App Store. It is something where some users say, well, it should be Apple’s responsibility to just be a stronger gatekeeper still and look for this, but it’s a really difficult problem to find.

We, with the Symantec’s technology, were able to do static, dynamic behavioral analysis both of the application and the backends that they’re talking to. Trying to do this from an Apple perspective would be pretty tough for them to replicate.

Ledge: So, whose job is it? You guys can find all these issues but, like you said, you can send it in to the dev/null of somebody’s email account, but do we have an accountability problem? There’s simply such a complex system that no one is in charge then?

Domingo: When we look at data privacy, we’re starting to see that it’s going to come from regulation. We’ve seen that with GDPR in Europe. California is about to pass some pretty tough privacy laws. That will definitely have an impact in terms of, both app stores are hosted in California so even if the developer doesn’t reside in California they might have to comply with some of those California rules. That’s just in terms of what data gets collected and how that data gets protected.

In terms of vulnerabilities, it’s a very difficult problem. Ultimately, when you sell a software or offer a software, I think it has to come to the developer if there’s something that should have been caught or there’s reasonable tools available to make sure that doesn’t happen, or even the documentation spelled it out on how to avoid certain issues.

I think that’s going to be interesting whether or not regulation goes that way. I hope not. I hope that, as an industry, we can take care of our code and take care of our users.

Code vulnerability analysis software has been in business for many, many years because it’s been a problem since the beginning. This just makes it tougher because now there’s, I call them Frankenstein apps. You build an app and maybe 40 or 50% max of the code is your own, most of it is third-party libraries and tools. Now you’re introducing their bugs and their vulnerabilities into your software as well.

Ledge: Right. Most of that stuff is open source, so is there a space so think about scanning the publicly available libraries then, and being able to be meaningful contributors back to the source of the issue? There’s fewer libraries than there are apps, certainly, because of reuse and distribution. That seems like a vector that might be worth exploring.

There’s increasing pressure on all the big players to make more contributions to open source because we’re all using it and all benefit from it. Is that a direction you see things going?

Domingo: Yeah, I think so. We’ve done a lot of work with the Symantec Research team to focus precisely on STKs and libraries, produce new [CVs 00:18:19] when we find them. If you’re able to fix an STK and it’s used by tens of thousands of apps, you made great progress.

It’s also then making sure that, as developers, we’re looking at the right version of the libraries that we’re using. Make sure that we’re using one with the latest fixes, bug fixes.

Then, in terms of enterprise, a lot of our customers end up submitting apps they built themselves or a lot of apps that they had a third party help them build. For this kind of issue and vulnerability before they deploy, we’re starting to see a push of not just enterprise-focused apps being reviewed, but even enterprise-to-consumer, or B2C apps.

Where maybe it’s just a marketing app – ‘sign up for this campaign’ or, ‘you’re going to an event, here is the conference’. We’re starting to see those types of apps also getting additional security reviews, but really in order to have mass effect it probably is something where either open source or tools available, maybe Apple and Google can invest in some sort of pre-scanners for vulnerability. Where you can check your app before you submit it for official review. So they don’t create a bottleneck in the normal review process, but maybe there’s a pre-health check or something. That’d be a great idea, I think.

Ledge: So now somebody just needs to own that and pay for it. Yeah, I could see where the challenge would come from on that regard.

Last question. Okay, we’re talking to a bunch of developers and business owners and the people who are deeply embedded in this ecosystem. Maybe the top three things that people should do in that seat to make sure that at least they’re doing their part.

Domingo: Absolutely. I think one is, as a developer there’s a lot of things on our minds. You’re behind schedule and over budget, and you have the new feature request and competition, and how do I monetize. I think just security is not even in the top five, let alone the top ten in our mind. That has to change.

We don’t want to wait until we’re on the headlines or we had a big breach, or worse a fine or something that can threaten the business. Just the frame of mind. It’s been proven that just being aware of some of the risks that could happen gives us a little bit more of a paranoia that’s a good, healthy paranoia. Just having it in mind as we develop, as we build so it’s not an afterthought, would be a great help.

Second, version control. Using latest versions and having some good hygiene on STKs or libraries that are not getting support or developed. Getting them from official sources. Not using third-party type of…

Like we saw with XCodeGhost on iOS a few years ago. A lot of developers, legit developers, downloaded a fake version of Xcode and ended up writing malware into the iOS App Store as a result. Millions and millions of downloads as well.

So, not trying to take shortcuts and using the official stores.

Then, when possible, when it makes sense, using third-party code review or doing some sort of peer review is helpful. Again, I understand that realistically not everyone’s going to have budget or time for it, but just trying to keep security in mind will be a great first step.

Ledge: Domingo, thank you. Gosh, that’s a lot to think about. You jammed a lot in there.

Totally appreciate your time. It’s great to have you on. I know that our audience is going to benefit a lot from the options and just the awareness of how critically important the footprint of your little tiny app is.

Domingo: Yeah. I think that’s a great way to wrap up, Ledge, is with great power comes great responsibility. The old Spider-Man quote.

Really, when we put something out there, we hope it’s going to be big and useful and folks are going to find it helpful. Sometimes it is really big, and that’s great, but it also means that the responsibility grows with that.

In certain industries, if you build a bridge as a civil engineer you need to get certified and tested and go through a lot of training. One of the beauty of software development is that anyone can do it from anywhere. You can be self-taught. It also means that we have to educate and invest in ourselves and just keep learning about security and how to best protect our users.

Ledge: Thank you for that. Great finish.

Thanks for spending time, Domingo. I really appreciate it.

Domingo: Thank you, Ledge. Have a great day.