SWAT to the rescue
Duke University assembles a crack squad to tame an onslaught of PC-packing students.
She was excited, but also harried those first few days. There was a dorm room to get organized and a steady stream of mandatory and semimandatory meetings to attend. There was one speech by the university president and another by author and poet Maya Angelou. There were briefings about various courses along with freshman orientation games and some organized, late-night Carolina-bashing (strictly verbal).
One thing, at least, went like clockwork: getting her computer hooked up to DukeNet, the campus network. "I didn't know how to do it, so I called the SWAT team," Schroeder says.
"They connected it all up in about 15 minutes," she says. Just like that, she had access to university resources including library card catalogues, the Internet, games, music and the Automated Course Enrollment System (ACES), an online program that enables students to switch courses. "I spent about a week doing that. I had horrible classes," she says.
Schroeder is one of more than 6,000 students who arrived at Duke this fall with little inkling of the planning and testing that went into getting them connected in a hurry. It was a process two years in the making, started in 1995, adjusted in 1996 and further refined in 1997, when the consensus was the university really got it right.
SWAT '97 was a success financially and, more important, in terms of customer service. Three-quarters of the students, including all freshmen, were connected to the network before school started. The rest were connected within the first week of classes. In 1996, it was the third or fourth week of school before all the students got online says Ginny Cake, director of the Office of Information Technology's (OIT) Customer Support group.
And whereas the OIT spent more than $70,000 on SWAT '96, the figure dropped to $34,293 for SWAT '97. Most of the savings came from using inexpensive student labor during summer months to prepare for the fall onslaught.
But this is more than just a tale about getting kids connected to the school network. It's really a network management story, one that proves you can reduce costs and simplify administration with the proper mix of technology, planning and coordination - all without a thin client in sight.
Duke was named a cowinner in the 1997 Network World User Excellence Award competition on the strength of SWAT, but it's clear SWAT is only the tip of the iceberg when it comes to getting a handle on management at Duke.
Consider that the DukeNet FDDI backbone consists of five Cisco Systems, Inc. routers and three Cabletron Systems, Inc. chassis-based hubs, which in turn support production or test networks employing virtually every network technology there is - from simple 56K and T-1 links to remote locations to Asymmetric Digital Subscriber Line test beds, ATM networks and OC-48 links to the Internet 2 test network. In all, the backbone supports approximately 18,000 ma-chines; another 12,000 on a separate medical center network rely on DukeNet for Internet access.
Now consider that the whole lot, down to the faceplates installed in more than 200 buildings, is managed by a staff of 10 in the Duke Office of Information Technology Data Communications department.
Bob Currier, director of Data Communications, credits the management tools the group has installed and customized over the years. They include Cabletron's Spectrum management system, Remedy Corp.'s Action Request System trouble-ticket program, Ganymede Software, Inc.'s Chariot for capacity planning and Hewlett-Packard Co.'s NetMetrix probes.
Currier makes it clear his group doesn't manage servers or desktop machines; its responsibility stops at the faceplate that such equipment plugs into. But given there are hundreds of outlets in each of those 200+ buildings, that's still a tall order. "We've got a tremendous amount of hardware scattered all across campus," Currier says. "It's a fairly small group for what we do."
He has some help, of course. Currier's group works closely with the OIT's Customer Service organization, which has responsibility for the help desk. Duke has made significant advances in terms of Web-enabling its help desk, which goes a long way toward letting users help themselves.
In September, traditionally the toughest month because of the influx of students, the help desk handled 90% of all calls without having to escalate the problem to the Data Communications group or elsewhere. That's up from 50% in September 1995 and 64% in September 1996, says Philip Verghis, manager of customer services for the Customer Support department.
He credits the World Wide Web for helping to make the improvement possible. An- alyzing the number of contacts to the help desk during September in each of the past three years shows Verghis is right. The number of direct contacts via means such as phone calls and e-mail jumped dramatically, from 8,800 in September 1995 to 17,500 in 1996. Verghis says this kind of hike is to be expected. As organizations get better at offering help desk services, word spreads and the number of contacts quickly rises.
"But as you start doing a better job at self-help, the contacts at the help desk should stay about the same while the overall contacts, through the Web and other self-help initiatives, will explode," he says.
That's exactly what happened in 1997. The numbers make for convincing evidence that an awful lot of students, faculty and staff were finding the help they needed on the Web.
This kind of thinking, helping customers help themselves, comes straight from the top. "Sometimes we don't give our customers enough credit," says Betty Le Compagnon, chief information officer and vice provost for Information Technology at Duke. "If you get the tools out there, people will figure out how to use them."
SWAT '96 - The early daysStudents certainly figured out how to use the tools provided by the SWAT team, although the team made it all pretty simple.
Planning started a full year before. "We realized in the middle of SWAT '96, when we started wrapping things up at 2:30 in the morning, that we needed to reconsider the way we handled things," Currier says.
Data Communications had installed Ethernet jacks in each of the dorm rooms and set up a Dynamic Host Configuration Protocol (DHCP) server to assign TCP/IP addresses. But the server, which was based on beta code, "crashed every couple of hours," he says.
The SWAT team, which consisted of personnel on loan from virtually every one of 10 divisions within the OIT, plus a handful of students, did its best to keep up. But it was a time-consuming affair, given that SWAT had to troubleshoot network connections with rather kludgy tools, help students install Ethernet cards - an interesting chore considering many of the PCs were intended for home use - and assist in configuring systems for TCP/IP connectivity. Plug and play was not always plug and play, as Currier put it.
Oh, and there was one other thing. "We had Hurricane Fran come rolling through campus," he says. "That set us back a couple of weeks."
SWAT '97 - Getting it rightIn the spring of 1997, the effort to do better began in earnest. Two students, sophomore Kevin Cheung and senior Bill Chen, were intimately involved in the process. "They did 90% of the work," Verghis says. It started with infrastructure. In 1994, when there were only a few hundred users connected to the residential network, a single flat Ethernet sufficed. In 1995, that was broken into four segments, each attached to a Cisco 7000 router, which provided the connection to the DukeNet FDDI backbone. In 1996, each of the four Ethernets were outfitted with switched Ethernet links to the FDDI backbone.
By 1997, the Cabletron hubs supporting that structure were showing their age. So the university embarked on a project to upgrade the hubs with Cabletron 2200 switches, supplying switched Ethernet links to every port in every dorm room. Last summer, Data Communications got halfway there, upgrading some 3,000 ports, including all those in freshman dorms. "As long as the budget gods smile upon us, we'll finish this summer," Currier says.
Throughout the summer, the SWAT team tested the outlets to avoid the problems with bad jacks it faced in 1996. Using relatively inexpensive student labor, every outlet in every room was tested using a Fluke Corp. OneTouch 10/100 Network Assistant analyzer, a simple-to-use but powerful and time-saving device. Previously, SWAT team members had to boot up a laptop and ping a server to test a network connection.
"It takes time to boot that laptop," says Michael Dodd, assistant director of the Data Communica-tions group. "This [Fluke device] is streamlined. Just power it on and it's ready to go."
Now should there be a problem, the team could be reasonably sure it wasn't on the network side. The other side of the equation, then, was making sure the students were prepared.
That task fell largely to Cake's Customer Support team.
In June, the group mailed a letter to each student detailing the exact kind of hardware they should bring, complete with configuration information - including the network interface card. The letter made it clear only three operating systems would be fully supported: Windows 95, Windows NT and Macintosh OS 7.5 or higher. The letter included an order form for those who wanted to buy their equipment from the university's computer store.
The letter was effective. Only 8% of students brought a machine with an operating system other than those that were fully supported. The bulk of the stragglers, 7% of them, had Windows 3.1, for which the OIT was prepared.
Customer Support also created a CD-ROM, dubbed "Duke's Internet Survival Kit," which had all the application software students needed, including a file transfer utility, Adobe Systems, Inc.'s Adobe Acrobat document viewer, Netscape Communications Corp.'s Navigator, a virus protection tool and Simeon e-mail software. A booklet, "Duke's Internet Survival Guide," was prepared to complement the CD, providing instructions on how to use the software.
Each of these were handed out when students signed up for telephone and cable TV service, eliminating a trip to the help desk site, as had been the case in prior years.
Finally, the help desk group created Web pages to walk students through any configuration questions. Returning students who hadn't changed their configuration settings from the prior year would be all set, but others might need help getting their machines set for DHCP. The Web pages explained exactly what to do, and even provided diagrams of the Ethernet faceplate.
Returning students were strongly encouraged to take advantage of such tools before calling on the SWAT team for help. "We did not go on site until they had tried [it] themselves," Cake says.
Data Communications also built a Web page for monitoring the DHCP server, which is based on a pair of redundant Silicon Graphics, Inc. machines running Internet Software Consortium code. Technicians in the field could point their browsers at the page to see whether the server was up, a big help in troubleshooting. And a script was written to start the server every couple of minutes to ensure it wouldn't go down and stay down, as it had the year before.
Reaping rewardsThe strategy paid off. "It hasn't crashed in months," Currier says, even though the server has parceled out more than 4,100 TCP/IP addresses since August.
There also was improved communication between the Data Communications and help desk groups regarding problems that did crop up. In past years, Data Communications was informed of problems in any number of ways, from voice messages to students dropping in.
This year, students received explicit instructions to call the help desk if they had a problem. "If it doesn't come in via trouble ticket, it doesn't exist," Currier says. That helped keep things highly organized, and Data Communications was able to close most trouble tickets within 24 hours.
That kind of customer service has made the OIT a big hit with the students. Network World stopped four unsuspecting students at random, asking each for their opinion of the computer setup process. SWAT was four for four; each student said they got their computers connected to DukeNet in just a few minutes, some with SWAT's help, some on their own.
Freshman Ross Montante's response was typical. "They gave us the book and the CD, and most of it was completely self-explanatory," he says. "The whole setup was a breeze. It took me no time at all."
Behind the scenesHappy customers are a testament to good planning, and Duke does plenty of that. But the school also is on the cutting-edge when it comes to technology, as its network management setup proves.
Duke was a beta-test site for Cabletron's Spectrum when it came out some five years ago and the university has worked closely with Ca-bletron ever since on improving it. Duke uses Spectrum to manage the hubs, routers, switches and other equipment comprising DukeNet.
But you won't find a traditional network operations center at Duke, with teams of operators sitting in front of management consoles watching for trouble. Rather, the Data Communications group has taken advantage of Spectrum's client/ server architecture, giving all 10 members of the team Spectrum console software for viewing the system.
Someone is always designated as the main focal point for alerts, but anyone can tap in if they need to.
The group can get away with this strategy because it has programmed Spectrum with policies that define how the system should respond to various types of alerts. These policies include detailed instructions on which operator should be contacted, depending on time of day and type of problem, Dodd says.
The first line of defense is a paging system. If the operator on duty doesn't respond within 15 minutes, Spectrum's SpectraPhone feature will track him down, calling various preprogrammed phone numbers repeatedly until it finds its man.
Duke is now at work on giving the help desk staff read-only access to Spectrum. That will arm them with knowledge of any network problems, which should help them provide more informed responses to user queries.
And the Customer Service group is at work on some major upgrades that will marry the best features of Platinum Technology, Inc.'s Apriori problem resolution system to the best of the Remedy tool. Verghis doesn't want to divulge details until the system has been thoroughly tested, but the intent is to promote more self-help. The system ultimately should be able to automatically respond to e-mail and Web-based queries with pointers to documents that can help. A library of more than 2,500 documents has already been assembled.
By the time Kristin Schroeder graduates in the year 2001, Duke's help desk may well have automated itself right out of existence.