Network Operational Investigating 101 (NOI-101): A new Cisco Lab Rat subseries

Put your detective hat on your head and your Network Operational Investigator badge on your lapel.  It’s time for a new subseries – Network Operational Investigating 101, NOI-101 for short.  This series will focus on the detective work (troubleshooting side) of our jobs as network engineers. 

But first… a little fun. 

CLUE: The Network Edition 

Introducing CLUE: The Network Edition – the classic “who done it” is back with a modern twist!

CLUE, the classic “who done it” game is back and this time the crime scene is in your network!  Was it…. the WAN router dropping packets? ….. that old remote router still processing switching and with high CPU? ….  A data center Nexus 7K someone misconfigured?     It’s the most challenging edition of CLUE yet!  Hours and hours of entertainment and fun for the whole team!

Okay… maybe not always so much fun.  But that is reality right?  And we all need to step up and troubleshoot sometimes.  Essentially, we all need to be a “Network Operational Investigator”

Network Operational Investigator

Welcome to the world of being a “Network Operational Investigator”.  It isn’t an easy job.  When a “crime” happens (translation – an outage or failure in your network) you need to be right there interviewing the suspects, surveying the crime scene, asking the right questions.  Trying to quickly figure out what is happening, where it is happening, and why it is happening.  This can be very difficult.   The “crime scene” is so large and our potential suspects pool so many and so diverse.  And, of course, with everyone doing more and more with less and less these days, there isn’t a lot of training on being a “Network Operational Investigator”.  Sometimes there seems barely time just to keep up with all the varying technologies. 

 ….. enter the NOI-101 Cisco Lab Rat subseries.

Network Operational Investigating 101

This will be a new subseries I’ll run on this blog.  The title will always be in the form of Network Operational Investigating 101: Whatever Topic.  So, what kinds of things will I be covering here?  Well,  what would you need to know if you wanted to be a great detective?  What would you need to know to be like a Columbo?  A Hercule Poirot? A Sherlock Holmes?

How about things like interviewing techniques…..  knowing how to ask the right questions….  problem solving….  noticing things… letting the evidence guide you and not jumping to conclusions? 

It’s great when, in our network, a failure occurs and the “action” box below is addressed by hardware or by designed and configured redundancy in our network.  Then, a human doesn’t need to be involved.   But when that yellow action box below is for action/reaction/troubleshooting by a human… wow can it be a while before we get to recovery sometimes.  

So back to the question “What kinds of things will I be covering here?” Answer: Anything that helps when a human is involved with that action box.    Anything from something as simple as what “!.!.!.!.!.!.!.!” means when you ping to having an easy to read “map” to the crime scene (your diagram). 

Until that first post?  Remember one thing 

“Let the evidence guide you”

Gil GrissomCSI (Las Vegas)