Can you get your shop under control?

Getting a new shop under control isn't easy at all, but it's worth it if you just stick with it.

If you're like me you've got a set of scripts that you take from job to job.  I've got scripts for everything from setting up backups to troubleshooting blocks.  And of course the dream is always that when you go into a new shop you'll whip out your script library at the first sign of trouble and save the day.  Of course, it rarely works out that way.  Often times things are so specialized in a shop that a lot of your scripts are rendered useless, or nearly anyway.  It's not really the troubleshooting scripts you can't use because thankfully you can do much to alter how blocks get reported, or how replication works.  So you've got some good solidarity there.  It's really the setup scripts that get blown up first. 

I've played this game in several places;  I go in thinking I'll have the environment under control in no time because I've got my scripts and nothing can hold me back.  However, there are many political and physical factors that stand in my way every time.  Oh y, it would be nice to just drop in all my stuff and be up and running, but that's not how it works.  I've been able to do it a couple times, but I'm convinced those were special circumstances where I was the only DBA they'd ever had so nothing was in place.  At any other gig however, I've had some pretty decent obstacles getting even backups going well.

One of the biggest problems I run into is other DBAs.  Quite often I'm brought in as a team lead or an enterprise overseer kinda role and everyone doesn't want to play along.  And it's totally the company's fault.  They put me in a position to be in charge of all the data and even make me responsible for it, but they don't make everyone play along.  Here's a good example.  A couple gigs ago I was brought in as the Sr. Enterprise DBA.  I was to make policy, standardize backups, get all the shops compliant, etc.  However, the shops I was supposed to be in charge of weren't all in our division and they all had their own divisional VPs.  So when I tried to get their DBAs to do things a certain way, their VP just said no.  And the board of directors didn't back me up.  So without the support of the board, and without buy-in from the regional VPs I was just another DBA in the company doing things my own way.  So now I had to figure out a way to get everyone to play ball.  Because it was that same old story, right?  The board agreed that something needed to be done because they were spending too much time and money on data quality and auditing, but they refused to consider it important enough to simply tell their VPs to get on the ball.

So here I am trying to figure out what to do about this because mind you, I'm still being held accountable for this data.  And in my mind I'm being held only do a degree because if they refuse to do what it takes then they can't really expect me to get it done.  Anyway though... I considered it a challenge to see if I could get these shops in-line.  So what I did was I used the audits as a mechanism to get what I want.  I took a couple months and wrote a set of scripts and processes that satisfied all of our audits and I put them in place in the offices I controlled.  I then went through a couple of those audits and tweaked my scripts as I went along.  Now I was ready.  Because many of those other shops were failing their audits because they simply didn't know how to do what they needed to do.  And the board saw this.  They saw that my offices were passing audits and that the others weren't.  And they saw that the ones that were making really good faith efforts to pass the audits were asking for like 100K for software to do it.  Then why were we able to do it over here for free, but we can't do it over there?

And then it happened.  One day I found myself in the elevator with the CEO.  He said how pleased he was with my efforts on bringing all the offices in my division to full compliance and he asked if I could do the same with the other divisions.  I promptly said no.  Honestly, he was shocked.  I could see it on his face.  He said, why can't you do it for them?  I said, well, because I was brought in here to get all this in order, but nobody on the board will back me up.  All the divisional VPs tell their DBAs they don't have to play along so I decided to concentrate on my own division.  You've actually put me in a position that can't be won.  He had no idea this was going on.  He was never told that the other VPs had shut me out.  I said, well, I've brought it up to the CTO and he said there's nothing that can be done because they've been here too long and nobody has the power to go against them.  He actually got mad.  So you mean to tell me that I'm paying 40K/mo in fines at 3 facilities because we're not compliant and all because nobody will let you do your job.  I said, I couldn't put it better myself.  He then asked.  Well, if I cleared your roadblocks how long would it take you to get these shops compliant.  I said, well, I can have the scripts in place right away, but it really depends on the control we're talking about.  Some controls just have to be in place at the time of the audit while others have to be in place for a certain amount of time beforehand.  But I can guarantee you that none of the shops with my processes in them will never fail another audit.  And in just a few months they'll pass with flying colors.  I handle all the audits myself for shops with my scripts in them. 

It was actually later that day when I started getting emails from DBAs saying I had been given admin rights on all their boxes and I was free to do whatever I needed to make them compliant.  And what I found once I got in there was horrendous.  Yeah, these guys weren't DBAs.  They were coders quite often, or windows admins who had been given the task of managing the DBs.  They had no idea what they were doing.  Quite often the backups they'd had in place wouldn't restore from the simplest issues.  I even had one shop where the windows admin, who know NOTHING about DBs, rebuilt the entire server about twice a week for blocking or connectivity issues.  Guys, I only wish I were kidding.  So I had to put a stop to that right away.  These guys had such a poor opinion of SQL Server, and they were supposed to be a 24/7 shop, but they spent 2 days every week down.  And they did this for like 2yrs before I got there.  After a couple rounds I finally got the guy to start calling me when they had DB issues and after the 2nd or 3rd time I had them back up in just a couple mins he started listening.  I even got them to change the code that was causing the issue and we never had it again.

So yeah, you can come into a shop with your scripts ready to go, but you may not be able to use them right away.  I had to gut every process these guys had in place to get things working properly.  But when I was done I had 80-something shops all working on the same scripts and I knew exactly what everything was doing and how it worked.  Now that's having a shop under control.  And I'm working to do that in my current gig, but it takes time, right?

Join the Network World communities on Facebook and LinkedIn to comment on topics that are top of mind.

Copyright © 2010 IDG Communications, Inc.

SD-WAN buyers guide: Key questions to ask vendors (and yourself)