We received a flurry of passionate responses to our recent blogs about protocol analysis tools. It reminded us of a story Mike Pennacchi of Network Protocol Specialists told us at SHARKFEST. If your house is burning, you welcome fire fighters bursting into your living room to pour 100 gallons of water a minute over your possessions. But if the local fire marshal asks for an appointment to inspect your house for fire code violations, you probably look for an excuse to be out of town. When the network is broken or has a serious performance problem, protocol analysts are the swashbuckling network firefighters who tap into a network and record packet traces by the gigabyte.
"Most of us here at SHARKFEST are firefighters because companies will spend money to fix problems, but not to avoid them," Pennacchi said. When it comes to firefighting, he knows whereof he speaks. As a volunteer fireman in Groveland, California in the late 1990's he didn't see much action because the town's very forceful fire marshal made sure buildings were safe. Based on your many responses there is a cadre of network firefighters strongly attached to their fire trucks and hoses.
However unpopular and unsexy the position might be, someone in your enterprise should be the network equivalent of the fire marshal. Otherwise, who will worry about the processes and tools needed to avoid a network or application performance emergency? We advocate ongoing performance management rather than emergency performance repair. Bring on the fire marshals!
Performance management requires a system of processes, procedures and reporting - and it requires a dedicated staff to keep the system in good working order. Furthermore, that staff needs proper tools and training - and that requires investment. The system must evolve as network and application infrastructure changes and it must adjust to shifts in infrastructure jurisdiction and ownership as the outsourcing/insourcing pendulum swings to and fro. Finally the system must stay current with changes in management and operations technology.
In an ideal world institutionalized performance management, should obviate the need for network analysts to fight fires. But we will never live in a perfect world, so those of you who love to fight fires will still have jobs - and your service will be invaluable for debugging during infrastructure changes and upgrades.
The question is who will design, build and staff the performance management system? This calls for the unglamorous capabilities of an accountant with a clipboard. How do we get the staid accountant to work with the petulant network analyst?
One final thought. Both the firefighters and the fire marshal work for the same fire department. Furthermore the most expert fire marshals were once firefighters. Are you tired of fighting fires? Are you ready to move to a "desk job"? How about putting on the newly minted badge of network fire marshal?
The Leader in Network Knowledge
A Good Comparison
I'm a "fire marshal" often doing "fire fighter" job - and really between us (we are many) called a fire fighter in this (computer) business. The unfortunate truth is that upfront planning, not just in networks but in whole systems, is often seen too expensive and companies are taking the risks - always(?) ending paying for it. And not just them but everyone - a fire (SPAM, viruses, other network related problems as performance problems and even identity theft) have a tendency to spread!
Performance? My favorite! We fight the problems if they happen (every time!)
Yes, I'm tired of fire fighting but don't see any way out. I'd rather do a preventive planning but find very few opportunities - the current corporate attitude seems to be that if we start a fire in next block, it's their problem not realizing that the fire comes back and burns your premises. Or, even seeing that(?), it's cheaper to take an insurance against loss than to prevent it - is it really?
Now - I don't agree it needs headcount (staff) as much as clear thinking. The idea to have more people to solve a problem reminds me always of the "Mythical Man Month" or the the nine month period to make baby. One dedicated person should be enough.
Great Piece -- Exactly right
Great Piece. I actually just wrote a blog post myself on this recently. Proactive management (fire marshal) is more effective and less expensive, but less glorious. This is often why it is overlooked. In a recent conversation I was actually told that when people prevent fires, they get fired because people think that the fire marshal isn't needed. Thanks for this piece, it's exactly what more people need to articulate.
Fire Marshal = ITIL problem manager!
This is a great post and I am in full agreement that enterprises require fire marshals, which is the ITIL equivalence of a problem manager. One of the important functions a problem manager is to review the major incidents that have occurred (and in so doing incrementally improve the quality of service being provided.) They do this as fire marshals as the firefighters do not investigate the underlaying causes, which may not manifest themselves immediately but only after a number of fires.
In my opinion network fire marshalls are not being appointed in large numbers as IT managers think incorrectly that IT systems are not prone to failure or that failures are a cause of product deficiencies. The impact of process and people seems to be negated.
You are correct that fires will never be totally prevented. A guy named H.W. Heinrich proved in 1931 (later reconfimed seperatedly in 1969) that there is a constant pyramid ratio between incidents categorized as minor, moderate and major. A big drive in quality is to increase the mean time between incidents and not to eliminate them! This improvement is largely addressed with people and process.