Stuff I learned from my 3 biggest network screw ups

My grandmother used to tell me that while painful, mistakes always make the most interesting stories and teach the best lessons. I certainly, do not go out to an account hoping to trash out someones network so I can learn stuff. If that was the case, the only thing I would learn is resume building 101. The one credo I have lived my life by is: "If anything is worth doing, it's worth overdoing" and that includes mistakes. Man, I have made some whoppers! In effort to further the field, purge my guilt and hopefully spin a good yarn, I present to you three big ole network hose jobs brought on directly by my mistakes. The names have been changed to protect me from getting my tail kicked... again... Oops 00x01: Unix file removal So there I am. A fresh IT goober with about 15 minutes of experience. Fresh on the heels of figuring out how symbolic links work and making my own cron jobs, I am pumped up with confidence. So I did the WORST thing a noob can do: I started reading the manual of my beige colored HP-UX server. This is equal to a hypercondriac being restricted to only surf to WebMD. I came across the command rm. I could end the story there couldn't I? Hey IT is all about service, and MY users could easily use more disk space... So I started deleting files with old file dates, thinking IF I made a mistake, I had a backup right? What! That is only the data and not the OS? Where did I put those 900 system floppies... 6 hours of angry calls later... Lesson Learned: Just because you have Read The Friggen Manual doesn't mean you UNDERSTAND The Friggen Manual. Maybe that's why my wife hasn't bought me that; "Physicians Desktop Reference" I have been wanting... Oops 00x02: 911 call centers and redundancy I'm a proud Dad. I admit it. I mean my kids always keep their bail bond under 200 bucks and they only drag race on backroads. Now, part of any dues-paying process for an IT geek is being on call. This on call the weekend, I had to go into a 911 call center to make sure certain jobs ran, tapes where full and just a general system walk through. It's the 911 system so up time is critical. Especially in this town. Man it was so rough they had an ammo reload station at every door exiting the building. So my son wanted to go into work with me. Man...I shed a tear, thinking of my son getting hooked on IT at an early age and going on to develop cool new stuff...(sob) that's my boy! So there I am plugging away in the MMC of my DEC VAX when all the sudden, the screen goes blank. Then it flickers to life, then blank again, then flickers, then blank again. This must be a loose cable on the CRT... a phone started to ring, then two, then all of the lines. The 911 center was offline and as I turned, I saw my son turning the CPU unit rocker switch, OFF then On. He liked that snapping sound it made. Isn't that cute? Luckily VAX is very resilient and only half the 911 center went down. But bringing the other half back up was a HUGE job, made the newspaper and I learned some new cuss words that day. Lesson Learned:Your kid is cute to YOU. Do not bring them into someplace as mega awesome as the data center with all the blinking lights and buttons, then be a irresponsible and not watch them. I have tunnel vision when it comes to troubleshooting...not Dad vision... In short; No kids or magnets in the Data Center Oops: 00x03: Windows is NOT Unix I am not a Microsoft hater. I know that folks need to make money. To me Windows is THE reason that computing is at the level it is now. Let's face it, as geeks we are comfortable with the command line. I believe that many folks would not even have a system if all they had was a Bourne shell. That being said, as a Unix goober, I am guilty of looking at Microsoft systems like I look at a 3 year old's drawing of a dog...awwwww...isn't that cute! It has DNS! Yeah, that punk attitude cost me at a customer bigtime. I was called out to work on a DNS issue. I know BIND well, this is Microsoft, HOW HARD CAN IT BE... I went about updating the zone file directly like I would on any BIND machine and that's where the problems started. Apparently, not all updates are in the main zone file. But in the process of troubleshooting this, I wondered; "why is the DHCPClient enabled on a server with a static IP?" Let's disable that obvious misconfig. Bad idea, seems Windows DNS uses this service to dynamically update it's A, SRV and CNAME record through this channel. A little time passed and all of the sudden folks cannot connect to Exchange! Apparently, the Exchange Global Catalog does not like not knowing where stuff is in the hierarchy. Microsoft Support showed me a Windows command I never forgot: c:\netdiag /v /l /test:dns This showed me which servers have active DNS records. A fail on ANY line is problem. Lesson Learned:Don't be an OS snob. Features may have the same names, but work completely different. Oh and, use the primary interface of the OS. I learned to love GUI on MS. Now back to you! What mistakes have you learned some of your most valuable lessons from? Post 'um in and share the love! As for me, I have to go fix a Unix file access problem... chmod -R 777 / should work right? Jimmy Ray Purser Trivia File Transfer Protocol More than 350 new animal species were discovered in the eastern Himalayas, including the world's smallest deer and a flying frog.