Managing a million page intranet: Network World, 12/15/97

By Peggy Watt
IntraNet, 12/15/97

When Northern Telecom, Ltd.'s intranet passed the million-page mark, its managers realized they had to get serious.

Employees, actively encouraged by the IT department, had begun treating the intranet as a corporate asset. But, ironically, the intranet's overseers weren't managing it as one.

It's not that they weren't paying attention to it. They knew it was growing - after all, they were the ones posting templates, building database links and adding servers to keep up with user demand. They just hadn't yet identified it as a cohesive corporate asset, says Bill Holtz, vice president of global enterprise services.

Once IT realized how invaluable the intranet had become, Holtz says he began applying standard network management practices to run it. That meant, for example, regular backups and careful traffic monitoring that alerted managers to add servers when they were needed.

At Nortel, network administrators dispatch several Web crawlers to check links, but that is almost the only Web-specific aspect of network maintenance operations. As part of its regular enterprise management routine, IT ensures that the routers are up and the links are functioning, Holtz says. It also measures capacity and usage across the corporate enterprise network and traffic crossing the proxy server to the World Wide Web. In addition, the department tracks the number and location of Web servers, how many pages exist and what major applications are running.

Although workgroups are allowed to set up their own Web servers, IT tries to keep tabs of the server population, especially since it handles enterprisewide backups. Intranet traffic doubled in 1996 and will probably do so again this year, Holtz says.

The management strategy appears to be working: CorWeb, Nortel's intranet, has swollen to 1,200 servers accommodating 45,000 users accessing 1.5 million documents. Broken down over CorWeb's eight years - it started in the beta-test days of the NCSA Mosaic browser and Apache Web server - that's an annual increase of 187,500 intranet pages.

"I don't think we could have stopped [users from embracing CorWeb] even if we'd wanted to," Holtz says.

This scenario is becoming common on Webs everywhere. Embraced by users, many intranets move toward a million pages of content.

The success is thrilling, but it comes with ever-increasing traffic, new storage demands and the need to frequently reassess the network infrastructure. The last thing intranet managers want is for network bottlenecks or aging equipment to slow the intranet's information flow.

So, for some companies, hitting the million-document milestone has forced network managers to reconsider the intranet architecture. Such was the case at Sun Microsystems, Inc., which moved from PCs to network computers (NC).

In Sun's case, indexing the intranet took four days because of the proliferation of Web servers, says Chief Information Officer Bill Raduchel, who estimates that SunWeb contains more than two million pages. By giving NCs to most of the 36,000 intranet users, Sun could store Web pages centrally.

Several common practices appear among the Webmasters who are staying in step during their million-page marches. Holtz and his peers, for example, follow good general networking administration practices and, when it comes to content, they delegate.

Like at Nortel, IT managers at GE Information Services, Inc. (GEIS), in Baltimore, Md., have empowered intranet users with the ability to update and refresh content. And GEIS goes even farther.

When a link breaks on the company's Global Village intranet, IT gets flagged. But the content manager has to fix the problem, says Wubneh Wubneh, manager of emerging technology.

"The intranet is independent, but centralized,'' Wubneh says.

As is typical for sizable intranets, IT programmers manage GEIS' corporate Web with custom tools. For example, GEIS built one tool for monitoring access. Intranet managers wanted to be able to identify usage patterns and measure response speeds.

The custom-built monitor automatically notifies administrators of status changes, such as when user traffic slows to a specified pace or when the number of people online hits a certain volume. Network managers note how often the intranet hits these milestones, adding servers or bandwidth accordingly.

Coopers & Lybrand, in New York, also took the customization route to manage growth on its intranet, which comprises about 120 Web servers and 80 Lotus Development Corp. Domino servers. IT staffers worked with a consulting firm to codevelop Surveyor, a Domino application that generates automated surveys of usage habits, compiles responses and produces statistics.

"Our experience is that most commercial management tools don't adapt well to Domino, so we have had to build our own,'' says Tracy Beverly, director of intranet services. "We need solid statistics on which business units use the intranet most, and how.''

The logs help determine, for example, whether the company should mirror frequently accessed sites or install faster pipelines for remote offices that frequently access particular databases.

"Most of our users are remote, so we just kill them if we don't get some speed for them,'' Beverly says.

GEIS' Global Village also is typical of many large intranets in its trend toward virtual pages. After an intranet grows beyond a thousand, tens of thousands or even a hundred thousand static pages, dynamic pages take over. The largest intranets generate most of their pages on the fly, drawing information from corporate databases front-ended by a Web browser.

One intranet builder - Brett Monello of Silicon Graphics, Inc. (SGI), in Mountain View, Calif. - argues that once an intranet hits about a quarter of a million pages, the numbers become moot.

At SGI, Monello's home-grown link-checker crawls constantly. It takes about one week to cover the Silicon Junction intranet, which serves SGI's 10,500 employees and has a half-million static pages.

"I don't have a goal of limiting the number of pages, but I'm concerned about what percentage of information requests can be met through the Web and how efficiently,'' Monello says.

One of Monello's tactics has been to keep the number of prepared, or canned, reports low. Instead, users generate most Web pages dynamically, drawing from Web-connected corporate databases, so they get the most current information. They may pull personnel data from human resources databases or product information from the marketing groups, for example, and then plug the results into templates of their choice.

Monello estimates that instead of some 7,000 canned reports that used to reside on a LAN, Silicon Junction stores about 500 and generates the rest as needed.

Intranet managers use the same philosophy in overseeing The Boeing Co.'s corporate Web, which circles the globe linking roughly 110,000 employees to company headquarters in Seattle. The numbers are misleading - Boeing's Web hosts more than 350,000 static pages, but it's extremely dynamic, says Dick James, deputy program manager.

Boeing had 545 Web servers before merging with McDonnell Douglas Corp.; now the number is probably double that. The intranet did not change network management practices. "The visibility of the Web just brings into focus issues we've always had,'' James says.

The technical library tracks links with a "Mom Spider'' that crawls constantly, James says. Otherwise, intranet maintenance issues are the same as with any networking infrastructure. They include replicating data across servers, conducting regular backups and running more fiber.

Same old answers

Network administrators for Hewlett-Packard Co., in Palo Alto, Calif., were prescient enough in 1989 to globally commit to TCP/IP; the company shares Boeing's sentiment.

"One lesson of a large intranet is that you build for continuous growth,'' says Joe Schneider, manager of technology infrastructure services at HP facilities in Fort Collins, Colo.

Today, HP's intranet links more than 400 sites worldwide and supports more than 100,000 PCs, about 20,000 Unix workstations and 2,000 Web servers. Although the network infrastructure keeps up with demand, Web documents - which Schneider says surely exceed one million in number - warrant watching closely for bandwidth and storage purposes.

"We've not just seen a growth in usage, but in the types of usage and types of information,'' Schneider says. "For example, what would have been a 2K-byte [File Transfer Protocol] document two years ago is now 20K of HTML code.''

Like his counterparts, Schneider's main management concerns are technology and content. "Especially when you give access to the World Wide Web, trying to anticipate all the factors involved in support is an ongoing challenge,'' he says.

For example, configuration and management of name servers on the network can affect Web server performance. "Once, a misconfigured [Domain Name System] server at [HP's] external site caused a problem, but a user's perception is that the browser broke,'' Schneider says.

An intranet user doesn't need to know the physical source of data, which may come from HP's extranet or even the corporate World Wide Web site, so Internet site management issues affect intranet performance and should be monitored, Schneider adds.

An intranet's strength often is the quantity of publicly accessible, user-provided material on it. Software vendor Autodesk, Inc., in San Rafael, Calif., splits intranet maintenance between a content manager and a technical manager, both of whom further delegate duties.

The content manager, for instance, manages 255 mini-Webmasters, each of whom is in charge of a departmental site. The IS department manages this distributed model, but does not provide guidelines for what a page needs to look like.

Large graphical documents are common at Autodesk, which runs its network on a series of Sun UltraSPARC servers. The infrastructure is solid: The headquarters campus is strung with fiber and copper, using 10Base-T and 100Base-T. Remote offices link to a WAN with speeds of at least 128K bit/sec.

When the internal Web got unwieldy, the IT department installed a series of mirror servers. Soon it had 16, and Autodesk revamped its intranet, installing proxy servers to cache data and boost access speed, says Steven Litras, intranet technologies architect.

Network managers installed Netscape Communications Corp.'s family of servers, including the Collabra groupware server and Netscape Directory Server to manage the corporate Web. They use Verity, Inc.'s Search97 as the intranet search engine and for basic spider services, but many of the staff's intranet management tools are home-grown.

The IT staff tracks usage and adds Web servers and proxies as necessary. The staff also is evaluating management software programs to find one that will allow intranet managers to automatically redistribute data across servers to achieve better load-balancing.

Reflections from the big guys

Digital Equipment Corp. has a million documents on its four-year-old intranet. Digital uses its own equipment, especially the Alta Vista search engine, for Web maintenance. All 55,000 Digital employees have access to the intranet, which runs on 1,400 servers - up from only 400 just a year ago. Alta Vista crawls and indexes weekly, and Digital archives content daily.

"This lets us do planned growth analysis,'' says Kathleen Warner, director of the Internet/Intranet Deployment Office.

Digital's various business units manage their own servers and content and keep links current. Warner says probably 60% of Digital's intranet pages are dynamic, among them transaction processing applications, multimedia and Java applets.

Similar to the tactic GEIS takes in managing its massive intranet, Digital's IT group searches for broken links, does quality assurance and sends e-mail to a business unit's Webmaster when it finds problems. Business units can buy internal services from IS to guarantee around-the-clock support, but they have to manage their own content.

Other intranet pioneers whose Webs grew quickly are also technology companies that drew from their employees' expertise. The first Web servers went up on Mi crosoft Corp.'s WAN in 1995, and usage exploded later that year with the release of Internet Explorer. Netscape, naturally, has had an intranet since its 1994 launch.

"I'm not sure anybody knows what our page count is because anyone can run a Web server,'' says Gregory Sands, a Netscape senior product manager.

Netscape's IT department rides herd over the Webmaster-filled company by maintaining central directory services and running a master list manager to help monitor usage and bandwidth needs. Similarly, Microsoft's intranet has many facets, from the administrative MSWeb to individual workgroup Web sites. The company already had a strong physical network and e-mail culture.

"At first, everyone turned their desktop machines into Web servers,'' says Fran Kottwitz, a program manager for Microsoft's IT Group, which has its own ITGWeb. "It has calmed down.''

IT encourages Microsoft's lay Webmasters to place pages on central servers, where backup is easier and there is often more space. Also, central Web management will enable IT to distribute software updates via Web technology and to optimize search functions.

Not surprisingly, those Webmasters use a lot of home-grown administrative tools that may result in commercial products - and, someday, may be something strong enough to easily manage a million.