When Intuit’s network gets taxed, it turns to Riverbed performance management tools

Ted Turner, Senior Network Engineer at Intuit, discusses with Network World Editor in Chief John Dix, how the company’s business is changing and how it relies on different tools to keep up. 

You’re best known for TurboTax and QuickBooks. How has the business been changing of late?

Those are our bread-and-butter products, and we’re shifting to the cloud, going global, and we’ve made acquisitions like Mint to address mobile. TurboTax and QuickBooks were originally packaged software products, and that’s still the preferred method of doing business for some customers, but as we made, say, credit card processing or your personal finance stuff available on smartphones, a lot of customers have decided that having things tied to a computer is not the way to go. They like the flexibility of mobile, so that’s been a big deal for us for the last five years.

Three or four years ago we made it possible to take a picture of your W2 and upload it to us, then we run it through OCR to capture the data, because doing text entry on a phone is arduous at best. That’s an example of efforts we’ve been making over the last eight years to do more for customers, and that’s led to us building bigger data centers to house more processing power.

Two years ago our online revenues exceeded that from customers using their own servers or PCs. And everything today is driving to the cloud. Even for customers that still buy QuickBooks off the shelf at Office Depot, when they upload their data to us they can then access it from their iPhone, tablet, etc. It all works. It’s up to the customer.

Do you have data centers all over the place or are you more centralized?

We’ve got two data centers, but we partner with cloud providers so a lot of the static graphics or delivery of iPhone and Droid updates is going to come from someplace more local to the customer.

How important is it to have the Riverbed tool sets?

We measure everything from the outside. What do things look like from a customer’s point of view? What does a screen page look like? We want everything to happen in one or two seconds, and If the response is slower than we want for our customer, we dig in. What caused that? Is it the web server that’s lagging? Is it the app tier? Is it one of our downstream services? Is it the database storage?

One of the things we’ve found is there’s always about a magnitude of difference. So, just to use round numbers, if something goes from 10 milliseconds to 100 milliseconds at the database, you’re going to go from 100 ms to 1,000 ms at the app tier, and go from one second to ten seconds at the Web tier.

So when we were originally troubleshooting performance issues we would focus on the slowest thing we could find, and that was usually the Web tier. But as we dug in we would find it was really something deeper, so we would work our way through the tiers and start trying to identify what was going on. It could be a server update, it could be a network update, a load balancer or firewall problem, there are a lot of different things in the middle and that’s how we use the Riverbed SteelCentral tools to help us determine the root of the problem.

Do you have other tools as well?

We have 72 different tools within Intuit that we use for all kinds of things. From basic sys log to SNMP and all of that. If you can name a tool provider we probably have their tool inhouse.

Is it hard to keep track of all of them?

It is and that’s why I am mainly focused on the Riverbed stuff. Whenever the logging or SNMP tools point in a particular direction, we use the Riverbed tools to help figure out if we should be looking more in the app space, or more in the networking space, etc.

Do problems more typically pop up in one area vs. another?

No.  It’s a real mix.

Can you give us an example of how the tools help?

Sure. Because we’re moving so fast we had two separate mobile products, one for the iPhone and one for the Droid side. Then we developed Turbo Tax Universal (TTU) for unified development and we were evaluating that and it looked like it was working great. But the development team had all the customer experience metrics in the cloud, and in February they had difficulty bringing that data into a Big Data cluster in our data center for analysis. They wanted to figure out where the customers are; are they East Coast, are they West Coast, or do we want to serve them from different places in the cloud?

So they increased the data pumps and as more data started pumping through, it threw off balance the traffic on our 10G and 40G links to the data center. So we had this unexpected increase in traffic coming from the cloud, and we used the flow analysis of the Riverbed ARX to identify the top talkers and could then look back to see our baselines in February. We identified the jump in traffic as originating in AWS and flowing to our Big Data cluster.

We had to tell the development team not to exceed a threshold because they were stealing bandwidth from other applications, and while we will fix this, we’re in our peak season now so we don’t want to make any changes right now.

Does the Riverbed suite do everything you need, or are there other components you might have to add going forward?

Some other tools are more cloud-based which allows them to move quickly. Riverbed is making that transition now, but App Dynamics has tremendous inroads in helping figure out what’s going on in the cloud. Not so much when we get into a data center, but we’re pushing Riverbed to tell us how the cloud is doing.

Do you use any of the Riverbed WAN optimization tools?

We do not.

Any foreseeable need for that?

Yes, kind of. That’s not the area I am currently focused on. I am attached to the tax group and we’ve bantered back and forth about what could be useful. But before the acquisition of our current Riverbed tools we had got some competitor stuff in place and we’re pretty comfortable with that. We’re evaluating new things like software-defined networking tools from Big Switch and Cisco ACI, so those are a bigger focus and concern than WAN optimization.

That’s interesting. Are you using SDN now?

We do a lot of packet aggregation for our monitoring tools and security stuff. We have a number of packet analysis tools and we were using Gigamon to gather packets, but when you want to gather packets from everywhere that price point gets too high. It’s got the stability and the capability we need, and high-end things like deduplication and flow generation, but we wanted more points of input. So we decided to go with a white box solution and Big Tap from Big Switch to gather packets and forward them to the tools as needed. So we’re using software-defined networking first in non-production, in our monitoring space, and evaluating where we want to go next. It’s done well for us. We used it through our first peak of tax year 2014, which was in early February, and since that went well we’re evaluating for the next QuickBooks build what we might want to do with Cisco ACI.

For all of these tools it’s all about visibility, seeing what’s going on, especially when you’ve got stuff going back and forth between the cloud and multiple data centers.

I’m always struck by how we got along before these tools came about, what a nightmare it was.

You were kind of just stuck. If you had an outage you’re in for a long haul. That doesn’t happen anymore.