The article by Bernie Lubitz speaks directly to what we see each day.
Re: Biggest lie in the enterprise: 'The network is down'.
N-tier applications work just fine in a laboratory setting where all servers are on the same LAN segment. They usually don't work as well when the production environment routes the remote procedure call (RPC) from New York to California. Vendors are just as guilty as internal developers when they write "chatty" applications that send streams of small packets vice designing buffering mechanisms that will increase application efficiency. Server and desktop OS support teams often omit I/O buffer size tuning and the resultant small TCP window size adds overhead that increases application response times. The majority of developers I have encountered know little about network protocols beyond the API they use to establish a socket connection.
When the "network problem" doesn't exist, then the developers' next step is often to request that we add more bandwidth to improve performance. To those who insist upon more bandwidth even though existing circuits are not overloaded, I will ask: "If you drive from one city to another, is it quicker to travel on a 2-lane road or a 4-lane road when there are no traffic jams on either road?" To those who complain that 85ms latency across country is too much for their application to handle, I explain the basic time, speed-of-light-through-fiber and distance equation and note that the laws of Physics are absolute. Lastly, for those "chatty" applications that suffer slow response / throughput I point out that if you have to move 20 people from one city to another, and you use a 5-passenger bus, it will probably take longer to complete the task. Why not use a bigger bus?
Mark Ruddeforth
Network Manager
Verizon Communications
Latest management headlines from Network World:
The fifth Golden Turkey Awards
IT sector adds jobs despite economic turmoil
Entuity's Eye watches over power consumption
|
Does Verizon's Voyager stack up to the iPhone? |
|
|
5 IT skills that won't boost your salary 
[1,407]
Women 4 times more likely than men to cough up personal info [589]
Japan's 10 funniest tech-related commercials [Videos] [407]
Throwing away a promo CD is "unauthorized distribution"? [1,265]
Adults too quick to dismiss educational video games [682]
Attack of the iPhone clones [Slideshow] [578]
10 things IT needs to know about AJAX [1,258]
This Year's 25 Geekiest 25th Anniversaries [Slideshow] [409]
|
|
Our work can help other companies
After reviewing this Network World article, I realized that Hartford Hospital's work to keep the network 'clean' should set standards for other companies.
We have a 14,000+ node network spanning much of the Connecticut area.
Our network team is only eight people who are responsible for the Infrastructure, not Servers or Desktops. When I first started, we were migrating from FDDI to ATM, and our main 'Clinical ELAN' was 4500+ nodes. We run 350+ NetWare, Unix, and Windows Servers, with Windows Desktops. Our Broadcast rate before we rolled out Windows Domain Services for this large VLAN was under 100 broadcasts/second. After we rolled out the domain, we were hitting almost 300 Broadcasts per second.
A review of the Desktop configurations via captures revealed problems with both Server and workstation configurations. When our team was done analyzing the problem, we gave the solution to our Server/Desktop teams who used Novell's ZEN to drop our Broadcast rate to an average 40 frames/sec, with IPX still enabled. We now have a Multi-GIG Nortel backbone, and our largest user segment is approximately 1600 devices.
We are running NetWare in PureIP mode, so IPX is long gone. Our average Broadcast rate on our largest user segment is eight frames/second. Even printers get adjusted to these standards.
Applications: All applications go through our team for analysis. If we see unacceptable behavior, the application is NOT allowed to go on the network without the Vendor fixing the problem. If they do not fix their application, it is not allowed onto our network. By running this so clean, our desktops are still connected at 10-half, with Modalities and Workstations connected at 100-Full or GIG. All user ports are locked to the respective speed/duplex settings to prevent negotiation issues. All ports are disabled unless activated (With required paperwork), and port recoveries are conducted on a regular basis.
We monitor our network with Plixer's WebNM, which has IpSwitch's WhatsUp embedded in it. This is visible to the NOC, and the senior IS management has access to it. So, if someone says 'The Network is Down', we say 'Check WebNM'. It usually is an authentication server that is the culprit, and WebNM will show that server on its exception page.
While I appreciate Bernie Lubitz' article in spirit, I can't help but note how easy it is to use tools such as W&G's LinkView Classic to show Management the Broadcast rates in nice, pretty graphs, use WhatsUP to show what's down, various capture programs to analyze application, and basic MS skills such as turning off Computer Browser on all clients.
Using a Desktop support tool like ZEN helps make these changes globally, including at remote sites.
Throwing bandwidth at the problem is only masking it, not fixing it.
Some time and research can do wonders to cure the problem.
Comment, our work can help other companies
David, it's great that you have the funds and the cooperation to run such a clean network. Unfortunately, not all Network Groups receive such favorable acceptance to their research and recomendations.
Response - Our work can help other companies
Bernie, most of our tools are nearly 10 years old or are free. We purchase WebNM, HP OpenView, and a couple of Wireless tools. The rest are relics in this day and age, but they work.
A few of the items that we established early on:
1) We do all IP address assignments. This got us off of spreasheets really early. We use Nortel's ADM (Formerly NetID) for this, and have been using this for eight years.
2) All ports are deactivated upon install. All activations are on Tuesdays and Thursdays, and require documentation including MAC addresses.
3) Showing the problems to Managment using graphs they can understand, showing the $$ required to upgrade vs. the cost of cleanup (invariably less to cleanup when it is the same problem on thousands of desktops - ZEN works!), and having Managment realize they have the power to implement change works quite well.
4) Our Enterprise IP scheme was planned out almost 10 years go. This is still a valid template, with the only modification is the addition of some VLSM segments. There is room for expansion in our design, as well as OSPF area and route summary capability.
Personally, I used to be the IS manager of a Law firm. If you thought a non-profit Hospital would be stingy, work for a Doctor's Practice Group or a Law office! Yet, once I showed how desktop image cleanup improved efficiency, reduced calls, and reduced complaints about the 'Network is slow!', I was able to get the funding I needed for other upgrades and projects.
Your Comment on The Biggest Lie
Mark, your comments are on right on the money. Telecommunication carriers have a rough go of attempting to meet the clients unrealistic expectations. I particularly like the bigger bus.
Post new comment