You can systematically get rid of the superstition and conjecture that plagues your office.
In my last post I talked about the magic threshold that gets thrown up whenever someone doesn’t know how or doesn’t want to troubleshoot an issue. And at the end I suggested that knowledge is really the best way to avoid that type of irrational thinking. I started thinking about it though and there are some other ways you can avoid these types of things and the reason I didn’t mention them before was that I was concentrating on the individual and how one can avoid that tendency, but now I want to turn to the company, or at least the group and see how you can turn things around on a more global scale. Because let’s face it, you may be the one sitting in the corner who knows that it’s BS to suggest a magic threshold, but unless you can communicate that effectively, then you’ll never get rid of that cancer. So here are some of the ways you can help alleviate this in your environment.
Evangelize
This one is probably the most important. Many DBAs prefer to go back to their desk and complain about how things are instead of working from within to change them. Every day while you’re at work, a big portion of your job is to evangelize and increase knowledge about your product. So for example, when you have an issue where a large table is involved and someone is saying that there’s too much data for SQL Server to be able to handle, you can disagree but they probably won’t listen. What you have to do is to find working examples outside of that issue where there are large tables that perform really well. Get through the current issue first though. During a time of crisis is not the time to pick a fight. Find the trouble and fix it, then look for ways to evangelize later. Specifically seek out the people who think the table is too large to be handled by the DB and show them other examples of large tables. And then explain to them what actually happened during the incident and why. And then explain to them that this issue could easily have happened on a much smaller table and the results would have been the same. Don’t do it in an I-told-you-so manner though. You’re just trying to prove to them that the issue wasn’t size. And once you prove it to them, then they’ll actually help you evangelize it in the future because now it’s something tangible to them. I’ve even gone so far as to produce a similar table and fill it with double the amount of data the production table has and run the same queries against it and shown them that the DB can clearly handle a lot more than what we’re currently doing. This is a really powerful demo because it shows them without a doubt that we’re nowhere near a threshold of any kind.
Train
Training your personnel is another very powerful tool in fighting superstition. I like to hold monthly training classes for my devs and in those sessions I discuss whatever is on our minds as a group at the time. So quite often I end up discussing the methods we use to troubleshoot slowness issues and why we make the decisions we do. If you demystify this stuff for them then all of a sudden it becomes something tangible to them again and they can grasp what’s going on.
Get everyone involved
This is something my company does that is right on the money. I wish I had something to do with it, but it was in place long before I got here. What we do is whenever we have a performance issue on one of our tier 1 systems, someone from every corner of support is required to get on a conference bridge and we all troubleshoot it together. So instead of everyone pointing fingers, everyone is there and checking their own turf on the spot. And we go around the call checking with everyone. DBAs, is it the DB? No, it doesn’t look like it. Citrix, how about you? No, it doesn’t look like it’s us either. Network, how does it look? So far it looks fine but… [Citrix] wait, I just found something. I think it’s us. Now, it’s kind of a pain to have to stop what you’re doing, but it shortens the troubleshooting time because everyone is in direct contact. So it’s really one of the things we get right. In fact, and this is no lie, I’m actually sitting on a bridge right now which is what spawned this post. And yeah, you’ll have those people who are stubborn about their views no matter how many hard numbers you throw at them. And frankly, that just amazes me, but that’s another post isn’t it? So there you have it.
I’ll summarize just for the word count.
1. Evangelize – show people that their misconceptions are unwarranted.
2. Train – teach them how to troubleshoot the issues and it’ll cease to be voodoo to them.
3. Get everyone together – when you have a big issue, get everyone on the same call and troubleshoot as a team and not a bunch of finger-pointing silos.
