Take control of your servers

As a DBA you have to control the situation when production issues arise.

Today I'd like to talk about gathering evidence.  As a DBA you have to gather evidence for all kinds of things all the time.  Sometimes you're chasing a rogue query through the system, sometimes you're trying to find out which user is doing something, sometimes you're trying to find which process is deadlocking or blocking, etc.  Ferreting evidence is a large part of our job.  And I use the word evidence for a reason; because that's what we're looking for.  And we wouldn't be doing a very good job if all we did was take the first thing we saw and say that was the cause. 

Let's say you're looking for a process that's blocking others.  And you run a query from sysprocesses and you pull the first query you see and say that's causing the blocking.  You have no evidence to back that up, nor do you have any real data to support it.  So now you've declared what the problem is and everyone goes about trying to fix this issue.  The problem is there's nothing to fix.  You've turned everyone onto the wrong path, and for what?  Because you were too lazy or too stupid to look at it properly?  And yeah, I know the language is strong, but I'm doing it for a reason. 

This is the kind of thing we get from end users all the time.  They know nothing about DBs, but they want to play amateur detective and dig into performance issues that they have no business being in the middle of.  They want to help, I get that, but it's not their job.  Now, I understand the implication of what I just said.  It's not their job.  I'm really not one of those who believes that people have their specific job and should never cross over.  During a crisis especially, everyone should pitch in to get the job done, and I'll do whatever modest office it takes to get production moving again.  But there has to be a line right?  You wouldn't let end users into your firewall or your router to poke around would you?  I think the answer to that is a resounding Hell No.  So why do you let them into your database? 

Part of the reason is because SQL is so well documented and the client is so freely available, that everyone automatically feels like they're entitled to access.  I've had many many users in the past insist on having access so that they can help look into issues when they come up.  And of course management buys into it and made me give them access.  So the next time there was an issue and we were all on the troubleshooting call, one of these end users would see something completely benign and state that he found the problem.  Then no matter what I said, the entire call focused on fixing this non-issue while the real issue went unresolved.  And this is the real danger of letting users have that kind of access to your system.  In my shop anyways it has nothing to do with wanting to be secretive or not wanting them to know what's going on in the DB, because I have a transparency policy where I'll let anyone know what's going on at any time.  However, if they're not trained in what to look for and what's important, then they just make things worse.

It's funny though isn't it?  Nobody would ever dream of letting anyone into the SAN or the router without training, but they see nothing at all wrong with letting them into the DB.  I'm going to tell you guys something that not everyone knows.  Taking a T-SQL course for a week does NOT qualify you to diagnose production issues.  Hell, it barely qualifies you to write code.

And as DBAs we have to hold ourselves to a higher standard.  We can't troubleshoot the same way our end users do.  If we did, then what makes us the DBA?  And at the same time when this happens we have to be strong.  Don't let the end users run the show.  Don't be afraid to stand up and say, NO, that's not the problem.  Now nobody do anything else until I get a chance to look at this.  Take charge of your server.  And with just a little authority you can keep your users in line and solve the problem with little commotion.

Good luck.

Join the Network World communities on Facebook and LinkedIn to comment on topics that are top of mind.
Take IDG’s 2020 IT Salary Survey: You’ll provide important data and have a chance to win $500.