View from inside Verizon's security SWAT team

Current Job Listings

Bryan Sartin is director of Verizon's RISK Team, the communications provider's computer forensics practice, which is also the group that helps create the annual Data Breach Investigations Report (DBIR). Network World Editor in Chief John Dix caught up with Sartin to learn more about the RISK Team, get his take on the state of enterprise security, and discuss new findings from the recently published DBIR report.

Bryan Sartin

Bryan Sartin, director, Verizon's RISK Team

You lead what looks to me to be a security SWAT team. Tell us about it.

RISK stands for Research, Investigations, Solutions and Knowledge, and we have two specific areas of focus. One is for everything that Verizon does in cloud, IT, and security. We handle incidents of a civil and criminal nature for Verizon customers, whether they are on or off the Verizon network. And that spans digital forensics, computer incident response, IT investigations, but also electronic discovery. And in that capacity we're one of the largest IT investigative entities in the world. We operate digital forensic labs in five countries and have full time investigators in 21 countries.

Our second area of focus is intelligence. Case by case, in data centers around the globe, we pick up little artifacts of intelligence from our field work and process and convert that into knowledge we use to improve products, drive innovation and secure Verizon. But we also deliver that security knowledge to clients on a regular basis.

[ DATA BREACH REPORT: Chinese cyber-espionage rising, says Verizon annual report

MORE: Verizon Enterprise chief: We're headed for cloud computing's A-list ]

Was RISK home-grown or did it stem from acquisitions?

It's grown in a variety of ways. Verizon has had security capabilities for a long time because security and Internet services just go together hand in hand. If you're going to provide someone access to the Internet, then helping them access it in a secure fashion is something that makes sense. I came in from the Cybertrust acquisition back in 2007, and a large percentage of my team did as well. I believe Cybertrust was the largest privately held information security services company in the world at the time.

We thought we had the brightest minds, the best people and the best tools at our disposal, but it was one of those things where you didn't realize what you didn't have until you became part of this great big Verizon. Then we started getting access to the assets here and people from other Verizon acquisitions over the years. So we came into an environment where there was a very established security services capability and reinforced what was there.

Today we have a little more than 100 people and four background types on the team. A good percentage is from law enforcement, another is from military or military intelligence, which plays very well into that second focus I mentioned, folks like myself have more systems engineering backgrounds, and then you have others from institutional IT type roles.

What types of things do you get called in to examine?

The most common thing is the IT investigation. We're called in when the customer believes there is enough evidence of a security breach to retain an outside professional investigations company. So typically you have employees or customers complaining of fraud, or, in the last year or two, the FBI reaching out to a company saying, "Look, here's some things you need to know. You may have suffered some type of APT attack in and around this data and time." So they call us with what they believe is hard evidence of a security breach and our job is to look at their great big network and all the moving parts and determine, did this or did this not happen?

And based upon the facts, can we prove or disprove the source, show how they got in, what they took, make sure we can stop the bleeding and contain the situation, and then finally do what's necessary to set the stage for prosecution? So we often times play a pre-law enforcement type role where we're bringing together facts and evidence and building conclusions and transitioning our findings over to law enforcement to take the final step.

Why do companies hire you versus a competitor?

A: The biggest difference is the reach of Verizon's operations. We have a true international capability and that helps us better understand the legalities and all the rigmarole that goes into international investigations. But there's also the network. I could spend hours on a white board showing you some of the ways we can derive incredible types of intelligence off the Verizon backbone that helps us do things like identify sources. We can perform entire remote investigations without even going to the customer's premises. Figure out who did it, where they came from, what tools and methods they used, and what they took. Then we can pinpoint crimes back to adversaries, link many crimes together or even turn on intrusion detection systems out in the cloud and point them at one or many networks. We have some very unique capabilities.

Speaking of competition, who are you typically up against?

Many big communication companies have a capability similar to ours, and there are more boutique-type competitors in each country we operate in. We don't really have a lot of significant international competitors across the board.

Do you get involved with the government at all?

Yes, both as a service provider and also for intelligence sharing. It's become clear that there's strength in numbers when it comes to collecting and exchanging security intelligence, especially understanding the adversaries and how they work. I mean our entire remote investigations capability is supported by intelligence collection and sharing. The more we know the more we are able to see little facets.

Somebody comes to us and says -- "Look, we've got this point of entry and we see activity on these ports at these date and times, and here's where it appears to be coming from." And with good intelligence-based research you can take little artifacts like that and convert them into an entire picture. We know who did it, where they came from, how they got in, see that this is linked to these three other investigations we've conducted, and I can tell what's under that rock before we get there. A lot of that's born out of the sharing we do with government.

So how bad is it out there?

I don't want to scare anybody with fear, uncertainty and doubt. That's certainly not the point of our data breach report. It's really about understanding the nature of the threat and how to defend against them. But the unfortunate truth is there's more concurrent criminal activity out there facing companies than ever before. It's more tumultuous and there's more diversity in the threats they face.

In your data breach report you talk about hactivism, espionage and financially motivated crime. Have the rankings of those threats changed?

Significantly, yes. Financial crimes still dominate the landscape, but last year hactivism really blew us away. People naturally associate hactivism with distributed denial of service, but suddenly hactivism accounted for more stolen records than financial crimes. And this year the big change is around cyber-espionage. People have asked us -- "Where is this APT thing I keep hearing so much about?" And no matter where we looked there just wasn't much data on it to speak of. But now in this past year cyber-espionage -- stealing intellectual property or other types of information -- accounts for about 20% percent of the cases.

Anything new on the insider threat front?

Insider threats do factor into our findings. Generally people have this perception that most of their exposures and moving parts are internal so, threat landscape-wise, the biggest danger is inside jobs. That was a myth we tried to bust in the past by showing that inside jobs are not only less than you expect, but they're considerably less.

In the last few years they've been below 5%, and in this last year they are below 1% of the overall threat landscape. It's external breaches that really hit victims the hardest, irrespective of industry. This year we've seen inside jobs jump up a bit, but it's still smaller than people expect. It's just that they do tend to hurt victims more when they happen because there is a larger average record set stolen on an inside job.

Turning back to cases where you get called in to investigate, are there any stories you can share about things you encounter?

Sure. One involved a large international company with a very recognizable brand name that had received an extortion attempt that started with a series of emails. As is typical in these things, no one notices the first five or six emails, but finally one of the execs saw this thing and thought it sounded pretty real. They notified somebody in security and ultimately we were engaged to investigate. They had us on retainer, a guaranteed 24-hour response to computer security emergencies, so they picked up the phone.

To make a long story short, the extortionist was in essence holding them up for ransom. He had some information and some intellectual property of theirs and he threatened to release this information if our customer did not meet his demands.

When you have an extortion attempt, typically you want to keep the extortionist communicating because the more communications you have the greater the possibility to identify his location or discover some other useful information. In this case the perpetrator seemed quite chatty. And what we found was he would be willing to drop the whole extortion attempt if we were willing to offer him employment. So we arranged, believe it or not, a job interview at the local airport and he in fact showed up for this meeting. Only the folks who sat down and interviewed him were members of law enforcement.

We could write a book about some of these stories. They're fascinating.

That's crazy. Any more?

There was the one more recently. A customer called us and said, "We've got a situation where an employee's credentials appear to have fallen into the Chinese government's hands. We're getting many connections in the middle of the night using this employee's credentials, only this employee is here in the United States, right in a nearby office. He's working in the office every day."

That's not unusual. Four out of five attacks in our data breach report this year involve exploited or stolen credentials. So we started to dig in, looking into how they were stolen, if others were stolen, try to see what might have been touched, and figure out how to block future access.

But we started by talking to the developer whose credentials were taken. Oftentimes the former cops on the team can sit down with somebody and get a feeling pretty quickly about whether or not they're squirrelly. And the more we pushed this guy the more squirrelly he became, and finally it turned out he had an interesting take on his job. He saw his job as an outsourcing opportunity.

He was shipping his work overseas to a software development firm in China. He would get assignments, ship those assignments out and they used his ID to log in to the systems and actually do all the staging, test and development and actually move stuff into production. He'd show up in the morning and say, "Yeah, look at what I produced," and then he'd spend the rest of the day surfing eBay.

What do you make of the oft-referenced idea that the bulk of security problems stem from the fact that security tools are not configured correctly?

That's true, to a very great extent. Many intrusion detection systems, for example, are not configured properly, but even when they are they still generate too much noise. If someone opens up on average 90 alerts a day per IDS probe and each one of those takes, what, something 3.9 minutes to vet, it's only a matter of time before they just don't notice the real important stuff breeze by.

There is a lot of talk about big data being the answer. What do you think?

A: What we're doing with the data breach report is an example of looking for commonalities or patterns inside big data, but where the rubber really hits the road is around sharing intelligence. I think sharing indicators and the TTPs -- tools, techniques and procedures -- of adversaries carries a lot more power than most people know. And it takes a big data approach to really put this kind of stuff to good use. That's something we're doing.

I mentioned that remote investigations capability, which is all based upon intelligence that we collect and we share with third parties, and we link many cases together based upon big data mining. If you put together the adversary's IP addresses, the tools, the techniques, the malcode hash patterns, you put all this stuff together in this big data mine, then when you get one little ingredient from a customer and you can go back in and say, "Well, that's interesting. They complained about activity on these ports at these times and they saw this particular source," and you can bring up those records and quickly tie that back to an adversary. 

"You know, I saw that adversary in at least 17 other crimes over the last 18 months and almost all of those crimes use this point of entry and this malcode and they were found on these kind of systems in these folders, and they stole this kind of data, and here's how they got it out." And you can build accurate predictors on what you're likely to find in the on-site investigation, all just out of that data mine.

Join the Network World communities on Facebook and LinkedIn to comment on topics that are top of mind.
Now read: Getting grounded in IoT