Test-drive your network designs

The days of overengineering for bandwidth are numbered. Few companies can afford to throw extra megabits per second at a project when the budget calls for accurate, robust and economical network designs from the start.

To help you stay lean and mean with confidence, we put four network simulation products through the paces and found this year's models are easy to use and accurate, and don't require a Ph.D. to run.

All performed well, though MIL 3's IT DecisionGuru outperformed the rest and earned our World Class Award with a near-perfect score. IT DecisionGuru delivers the whole package: accuracy; good performance; customization capabilities; superb documentation; and, except for some quirks with the license manager, easy installation.

For network managers with a smaller budget and less-demanding simulation needs, NetCracker Technology's user-friendly NetCracker Professional 2.0 fits the bill. NetCracker combines presentation and drawing tools with a simulation engine.

CACI's Comnet Predictor and Baseliner 1.1 left us with mixed feelings. While the simulation engine is robust and the results we obtained were useful, the package felt a little dated. Some of the terminology, such as "network node" and "processing node," is ambiguous, and the graphics don't stack up to those of the other products. We hope CACI's revised Predictor 2.0, still in beta during our tests but shipping this month, will resolve some of these shortcomings.

Analytical Engine's NetRule 2.0 is the only product we reviewed that's written in Java. NetRule suffered slightly in the performance category because of the interpreted nature of Java. NetRule doesn't have the depth of features or extensive protocol libraries of IT DecisionGuru or NetCracker, but it doesn't cost as much either.

SMARTEr than the average simulator

Network simulators typically use one of two simulation methods: discrete event or analytical. Discrete-event simulators create an extremely detailed, packet-by-packet model of predicted network activity but require extensive calculations to simulate a very brief period. Typical discrete-event simulations can take several hours or even days to complete.

Analytical simulators use mathematical equations to predict network and application performance. These packages sacrifice accuracy to provide faster performance.

IT DecisionGuru uses a hybrid simulation technology combining discrete-event and analytical simulation techniques that lets network managers control the amount of detail provided. Turn up the level of detail, and run-time increases proportionally. Scale it back, and run-time is shortened.

IT DecisionGuru performed extremely well at all levels of detail. With the out-of-the-box settings, most simulations finished in a minute or two on our Sun UltraSparc 2.0. Performance was almost identical on our Windows NT platform. MIL 3 states that the performance of IT DecisionGuru running under Windows NT should be slightly better than the Solaris version, but we saw very little difference.

In our most complex test scenario - an enterprisewide collection of subnets all sending heavy HTTP and File Transfer Protocol traffic across a 10M bit/sec link - IT DecisionGuru took 45 minutes to simulate 1 hour of network activity. This is a good example of its hybrid technology. A pure discrete-event simulator would have taken many hours to process this data, while an analytical tool would have finished in a few minutes.

We compared IT DecisionGuru's performance statistics to results we obtained with Ganymede Software's Chariot 2.0, a network testing and modeling tool (NW, Jan. 25, page 65). We used Chariot 2.0 to provide a benchmark for accuracy assessments. IT DecisionGuru predicted performance of approximately 2.5M bit/sec on a 10M bit/sec link over the course of an hour, a metric that closely matched the Chariot test findings and our historical observations of similar segments of the university network.

A feature unique to IT DecisionGuru - and one that makes it particularly appropriate for organizations with enterprise resource planning programs such as SAP R/3, PeopleSoft and Baan - is MIL 3's Simulation Methodology for Application Response Time Engineering (SMARTE).

Instead of depending on canned application models, SMARTE lets you profile the exact behavior of an application by capturing packet traces, adding background traffic levels and then investigating what-if scenarios. SMARTE lets you tailor a simulation to reflect your real network, rather than a network that the creators of the program envisioned.

Our runner-up, NetCracker Professional 2.0, is based on an analytical engine. NetCracker performed well on the NT platform but didn't let us choose run-time intervals. While IT DecisionGuru and CACI's Predictor let you select the simulation period in minutes, hours or days, NetCracker runs until you stop it.

This lack of interval selection makes NetCracker less useful to managers who want to model network activity over set periods to predict growth over time. However, NetCracker Technology says it plans to include this feature in future releases of the product.

On our enterprisewide performance test, NetCracker estimated 1.25M bit/sec utilization, which was not as close to actual performance figures as IT DecisionGuru's 2.5M bit/sec estimation. The differing results are most likely the result of the difference between NetCracker's strictly analytical engine and IT DecisionGuru's hybrid engine, and the extremely granular nature of IT DecisionGuru's settings.

Overall, we found NetCracker's reports to be clear and easy to understand; the product's response time on our NT test system was excellent. While not up to the level of IT DecisionGuru in its protocol models and application profiling tools, NetCracker would make an excellent presentation package for sales engineers or network managers making budget presentations.

CACI's Comnet Predictor and Baseliner uses a proprietary technology called Flow Decomposition as its underlying engine. Flow Decomposition analyzes network queuing and traffic flow instead of tracking each network event, as would a discrete-event simulator. Avoiding packet-by-packet analysis provides much snappier response times. We found that most of our simulations ran in just a few seconds on our 400-MHz Pentium PC.

Comnet Predictor and Baseliner's performance reports were very clear, making it easy to find the areas of the network that had the potential to be oversubscribed. The reports were also accurate, falling within a few percentage points of what we expected to find based on our Chariot testing and historical records. This result was particularly impressive given the short time it took Predictor to perform the calculations.

We were also impressed with Predictor's easy tuning features that let you tweak settings to try different scenarios.

A step below the other three packages, Analytical Engine's NetRule 2.0 is a basic, entry-level product. Analytical Engine considers NetRule to be a modeling package and notes that simulation is just one of many techniques for modeling a network's performance. The company's intent is to provide an easy-to-use product with an uncomplicated interface.

In fact, NetRule simplifies device models to the extent that we were concerned about its accuracy. NetRule provides various computer models, such as a file server, generic server, Internet server and 200-MHz Pentium PC, but all start with the same device attributes. Company representatives say it's the user's responsibility to properly tune each device to reflect its true performance characteristics. We found this to be disconcerting for a product that's aimed at novice modeling and simulation users. Performance estimates can be flawed from the beginning if a user fails to properly tune each device, leading to inconsistent predicted and actual results.

On the plus side, NetRule's cross-platform portability is a useful feature. Because it is implemented in Java, NetRule runs on a variety of platforms without a problem.

Customization counts

All the programs we tested offer a reasonable selection of network devices and protocols for building network models; the ability to customize these components is where the products showed their

differences.

IT DecisionGuru, clearly the leader in this area, lets users create "derived models" by modifying the attributes of an existing node or link model. If the router you're trying to model isn't included in the supplied libraries, you can simply pick the best approximation and change the attributes. In addition, MIL 3 stores a large collection of user-created derived models on its Web site.

What's missing from IT DecisionGuru is the ability to create an entirely new device or protocol from scratch. For this task, you need to purchase MIL 3's OPNET Planner, which lets you create, edit and save custom models. However, MIL 3's device library is excellent and, when coupled with the free customer-derived models, should be enough for average users to design a corporate network.

NetCracker Technology provides the second-best customization options. You can't control as many device attributes as you can with IT DecisionGuru, but NetCracker's wide selection of network devices and configuration options makes this a secondary concern.

Configuring devices with NetCracker was awkward at first, but we learned to appreciate its unique approach. For example, if you're adding a router to a design, NetCracker first creates the router without any interfaces for you to place in the diagram. Then you have to drag and drop the appropriate interfaces on top of the empty router. Once we got used to this approach, we found it easy and flexible.

Designing our test network with CACI's Comnet Predictor and Baseliner was a bit challenging because the program first makes you choose a generic model for each device, such as "network node." Once you've copied a base object to the new diagram, you can select from a range of products. The finished model works well, but it looks a bit less polished than the other packages.

Comnet Predictor and Baseliner's library includes a good selection of devices, from the 3Com CoreBuilder 3500 to the Zynx ZX318 server switch. You can modify some of the attributes, such as supported protocols and packet rate per protocol. Like NetCracker, Comnet Predictor and Baseliner doesn't allow you to set as many parameters as IT DecisionGuru, but we thought its options were sufficient for most users.

Comnet Predictor and Baseliner, like MIL 3's IT DecisionGuru, is based on an existing high-end simulation package. CACI's Comnet III discrete-event simulator is the parent product, targeted at hardware engineers designing new products. Users can import models generated by Comnet Predictor and Baseliner directly into Comnet III for a more detailed analysis.

Analytical Engine's NetRule includes a basic set of network devices. But we found the settings for very different devices to be too similar, as noted above. NetRule does provide a variety of modifiable attributes, and once a device is properly configured, you can store the newly created device model in a library file. With enough time, a user could probably build a library of devices that accurately reflects a given network.

Getting started

All the products installed easily, though we did experience a few minor hang-ups. For example, NetCracker reported that it had expired immediately after we installed it. A quick call to technical support got us a new CD-ROM and license key, which solved the problem.

IT DecisionGuru's license server caused a few problems. We originally installed the software on an NT platform and registered it through MIL 3's Web page without any trouble.

But when it came time to install the product on our UltraSparc, we found that it would take many steps to unregister the original installation and then re-register the UltraSparc. We elected to leave the license server on the NT machine and run the client on the UltraSparc. Unfortunately, we had moved the NT box to a different Class B network, and when we did, the license server wouldn't operate. MIL 3 confirmed that the license server must remain on the same network subnet. This is a potential problem for large corporations or universities.

In the field, all the products we tested provided useful results, and all have distinguishing features. Based on its combination of features, performance, technical support, documentation and customization capabilities, MIL 3's IT DecisionGuru is our top choice. If you want the Ferrari of simulators, this is the product to purchase.

Currier is director of data communications at Duke University in Durham, N.C., and the 1998 Grand Prize winner in the Excellence in Campus Networking competition sponsored by CAUSE, a user group for computer professionals in higher education. He can be reached at robert.currier@duke.edu.