Behind the scenes of MIT's network

Network Manager/Security Architect Jeff Schiller on buying into VoIP and fiber in a big way

Massachusetts Institute of Technology Network Manager/Security Architect Jeff Schiller is leaning back in a plum-colored recliner in his office, but he isn’t relaxing. The victim of a back problem that has forced him to forgo a more traditional office chair for now, the 25-year MIT network veteran has more than enough to do, with the school forging ahead with several major network projects, including a massive VoIP rollout and its foray as a regional fiber-optic network operator. Schiller covered the gamut in a recent interview with Network World Executive News Editor Bob Brown.

How’s the VoIP project coming along?

We have 500 people on our voice-over-IP system, so we’ve really moved beyond the pilot stage to the service stage, and we’re ramping up to 1,500 users in the next couple of months, and to be a VoIP campus not too many years from now, MIT plans to switch all 15,000 of its phones to VoIP. We’ve got it going in the IT department, since you’ve got to eat your own dog food. (Some people asked if it was really wise that the phone path to the IT department would use VoIP, but we told them if the network is down, we know.) One of the arguments for having us do it by department or building is that the hard part is getting our 5ESS [phone switch] people to manually route their phone numbers to us so that people can keep their phone numbers (putting new employees on the VoIP system is much simpler, as the school uses a common name space and via a Web administration page can set up new end users with a Session Initiation Protocol address that’s the same as the e-mail address).

What’s the story behind your VoIP project?

If you would have come here a year ago you would have found that I had an ISDN phone, as we put in ISDN in 1986 [now he has a Polycom IP phone and is among the 500 initial users of VoIP at the school]. We bought a 5ESS phone switch from AT&T that went online in 1988. AT&T rewired the campus at that time and that’s how we got our first fiber plant. Around 1999 they contacted us and told us that switch would be obsolete by 2001 because they weren’t making any more software updates for it. Our CIO came to me and asked if we could do VoIP by 2001. I said “I wish I could tell you yes, but the technology is just not mature enough,” so we went and bought another 5ESS, which was hugely expensive. If you estimate a 10-year life cycle for that phone system that meant the vendor was going to be coming back to us before long to let us know we’d need to buy another one. But now voice over IP is ready, and I told our CIO about a year ago that if we want to be a voice-over-IP campus by 2010 that we’d need to start now.

What technologies are you using for the VoIP system?

We’re not 100% decided on some parts, but I’m currently using a Polycom system. The media gateways to the 5ESS are Cisco high-end voice-over-IP switches, and of course we do everything in pairs in different locations. We’re running the OpenSER SIP Express Router [MIT is also evaluating commercial offerings] on Dell 2850s redundantly, and our toilet server, which does voice mail and all the other crap, runs Asterisk software. It’s fair to say it’s mostly an open source deployment. The open source stuff not only is relatively inexpensive but we can integrate it into our infrastructure and customize it. The killer app has been sending voice mail to e-mail., something the Octel voice mail system on the 5E couldn’t do. As for the rest of the infrastructure, the voice-over-IP phones are running on a separate VLAN. We have to upgrade the general infrastructure just because it’s time to do that. We have physicists who want to send data sets of gigabytes to CERN, and the Media Lab wants to do real-time video. But voice over IP itself is not a very demanding user of the network.

What sorts of challenges or concerns have you run into with the system?

The biggest aspects of it are not technology; it’s the sheer numbers of people, it’s a staff training issue. The fact that the phone is a different color is going to be upsetting to some of them. We want to minimize disruptions and that’s why we hired a full-time project coordinator.

What about security?

One reason for having the VoIP phones on a separate VLAN is we firewall it. It turns out all these phones have Web servers — not browsers — in them and one way to configure them is to talk directly to the phone. All you need is the phone admin password, which is the same one in every phone and it’s in the manual, so we don’t let Web connections get to the VoIP phones, so security is at that level. I would love it if the phones would encrypt the voice stream. They don’t do that today and there’s nothing I can do about it except indicate to the vendors that I really want that feature and hope we’ll even get it. It’s a concern. But so are cell phones. If there hadn’t been cell phones I’d be much more worried. We don’t want to go overboard on something that’s not a real threat yet.

If I put my IETF hat back on, VoIP security in general has been a real disaster. Like everyone who does technology, the VoIP vendors don’t want to think of security when they’re designing, and they aren’t convinced the bad guys are really out there just because they’re not attacking yet (and of course they won’t attack until you have 100 million handsets out there to make it worth their while). The other problem with VoIP is that there have been a lot of Bellheads involved and they have a security model that’s completely whacked — the “trust the network” model. In the Internet space you don’t trust anybody, particularly the network. You better do end-to-end security if you care.

Then there’s the whole damn government. I don’t know this but I suspect if the Polycoms and Ciscos of the world had had these phones do end-to- end encryption on Day 1, then the U.S. government probably would have come in and tried to stop it. They want to maintain the ability to do surveillance even if we all have to walk around naked.

Getting Personal: Jeff Schiller
Organization:

Massachusetts Institute of  Technology

Title:

Network manager/Security architect (has managed the MIT Campus Computer Network since its inception in 1984)

Responsibilities:

Managing a network that has about 10,000 switches, largely a mix of Enterasys and Cisco boxes, and that supports 20,000 to 30,000 people and 50,000 to 60,000 computers. "Our backbone is 10G. Our network has a simple star configuration with 10G switches in the middle and 10G links to each router, then 10/100/1000 Ethernets from there that run the gamut depending on the building. Everything is SNMP-manageable. A couple of departments run their own networks, but we discourage that except for departments where they really should run their own, like the Media Lab."

Staff size:

7

Annual budget :

Undisclosed

Pervious jobs:

None

Education:

S.B. in electrical  engineering, MIT 1979

If he wasn't in IT he'd be doing:

Academic research

Claims to fame:

Author of  MIT's Kerberos authentication system; formerly area director for security within the Internet Engineering Steering Group; a founding member of the steering group of the New England Academic and Research Network (NEARnet), now part of Level 3.

Fun fact:

Raises guinea pigs. "Cleaning up after guinea pigs offers an interesting contrast with the high-tech world of computers, though some argue it is the same, cleaning up someone else's mess!"

OK, on to project No. 2. What’s MIT doing to become a regional optical network player?

Through an arrangement with Internet2 and their FiberCo arrangement we have a pair of fibers from Boston down through Rhode Island, Connecticut and eventually terminating at 32 Avenue of the Americas in New York City, and a redundant pair up the Hudson River and that cuts across Massachusetts. We got it at a price we could afford, so we went for it. We’re lighting it up with optical gear that will give us 72 10G waves. This means in New York City we can peer with CERN and with a lot of the major players. The contract for our fiber wasn’t 24 hours old when through the grapevine our researchers found out about it and were enthused about using it for high-speed access to various national and international assets.

MIT doesn’t already have access to high-speed links for research through Internet2 or other networks?

Internet 2 backbone now is 10G I think, and links to this part of country are around 1G. Our researchers want 10G to CERN and now we can give them that. We also did this before Internet2 announced its new network [which had gone by the working name NewNet and boasts 10G lambdas.

I’m very annoyed about the competition that emerged between Internet2 and the National LambdaRail network people. It was a national embarrassment that literally got down to name calling. The networks were going to merge but turned out to be like water and oil, so now they’re competing. A side effect was that the Internet2 people didn’t talk to us before they announced one of their NewNet nodes would be in Boston, so now we’re in the same facility as them. Even so, we’re still getting a better deal on price to get to New York City. Meanwhile, we have a history of cooperation in the Boston area with other schools, such as Boston University and Harvard, such as through the Northern Crossroads facilities. Some asked why we didn’t buy the new fiber under Northern Crossroads, but it was just a timing issue: We had the money and couldn’t wait for approvals from the others.

This all sounds too easy . . . .

It wasn’t. It’s amazingly complicated. First the IRU [Indefeasible Right to Use] agreements, and it’s Level 3 fiber, so we have to sign agreements with them. And it’s not just the fiber, you have to get space in huts along the fiber path to put in regeneration and optical amplification equipment [Nortel installs most of the equipment]. It turns out there’s paperwork to be done for each of those sites, plus lots of legal contracts. And we’re a nonprofit organization, so we need to file paperwork in every single township along the way to demonstrate this.

And this is still going to pay off?

Oh yeah. Intercity fiber is like gold. We expect by the end of January to turn it up and send data down it.

A couple of high-profile schools, UCLA and Texas, recently announced that they had data breaches. I know MIT has not been immune to breaches either, but what do you think when you hear about new breaches like these?

The problem we all have is the Microsoft patch of the week. I hate to say it, but it’s sort of the payback for universities not paying attention to security for decades or being sloppy about administrative computing. The mentality goes back to the times of disconnected, batch-oriented mainframes when the Internet was not even on the horizon and the attacks we face today were unheard of. One area in which we’ve been a little ahead of our peers is not using Social Security numbers for employee and student IDs. That goes back more than 20 years. Having said that, the SSN is used in more cases on campus than it should be, but we’re working to reduce that. The fundamental problem behind all of this is that the SSN can be so easily abused. It’s easy to learn someone’s SSN yet it is viewed as a secret by many institutions so it can be used as an authenticator. This is broken. We need legislation that says anybody who makes decisions based on authentication, which is knowledge of an SSN and a home address, they’re taking the risk in the transaction, not the consumer. People will scream: ‘But how are we going to authenticate people?’ Figure it out. Part of the solution is to have some sort of mandatory education. If I want to handle data for research on humans I need to be certified. That’s [a National Institutes of Health] requirement. I think we need to start with having a certified administrative data handler. There’s not a government agency pushing that like NIH on human research, but within institutions like ours, we could do this. I don’t think there’s a technical solution that doesn’t involve training people.

What other headaches are schools dealing with?

CALEA [Communications Assistance for Law Enforcement Act, a wiretapping law; see details at www.askcalea.net/ ] is one. [Industry trade group] Educause did us a great disservice by panicking and screaming that the sky was falling. In my view, CALEA was not targeted at higher ed. What I think it really goes back to is making sure that if the telephone companies have to be compliant then the cable guys do too. But the language used was overly broad and could include universities. The FCC chief of staff told Educause this wasn’t about universities and to go away, but Educause wouldn’t let it go and asked the FBI. And of course if you ask the FBI if they’d want cameras in every bedroom of every American citizen, they’d say of course, we could cut down on domestic violence. They woke a sleeping giant. For now, CALEA is a source of angst for IT, but the lawyers are busy.

What about dealing with wireless on campus these days?

1 2 Page 1
Page 1 of 2
The 10 most powerful companies in enterprise networking 2022