Don't let vendors take advantage of you

Don't do anything just because the vendor says it's a good idea. Here's why.

It's amazing to me how things come about sometimes in a large company and how strong you have to be as a DBA to keep things going the way they should.  I'm going to tell you the story of a vendor that happened just yesterday.  Now vendors are notorious for not having the brightest staff and in healthcare even more so.

Ok, so yesterday I was more or less minding my own business when I got an email asking me to rebuild a SQL box.  I checked and I had never even heard of this box before and my group didn't even have rights to it.  I replied to the email asking why the box was being rebuilt and they told me that the SQL install was massively corrupt and was unusable.  I said, wow, that's a big statement.  In what way is it unusable?  Well, the vendor told me this.  I don't know how it's corrupt.  Well, I asked, is it down?  I mean, nobody is able to connect or anything?  She said, no, it's still running and people are working.  I said, ok then how is it unusable?  She said, because it's corrupt and unstable.  I asked again, ok in what way is it unstable.  She said, let me just forward you the email from the vendor.  And the email I got next seemed like a joke at first.  I kept thinking someone is messing with me because they like to watch me explode on vendors.  After about 3 more emails of her assuring me that this was not a joke, I insisted on a meeting with the vendor.  I need to ask more questions I told her.  She set up the call and while we were waiting, I asked her a little about the history of this server.  She said that they've had to rebuild about 3 times a year because of this type of instability and she couldn't understand why they were always having this kind of luck when other groups run for years without any trouble.  I assured her that we were going to get to the bottom of this and I'm pretty sure we can make it so they don't have anything unstabling their box again.

So we get on the call and I told the guy, we'll call him Barney, I told him that I read his email to her but I wanted to hear it straight from him.  Here's the overall discussion we had:

B:  This server has always had a number of problems and this time they were brought about by the log backups failing.

S:  Ok, I'm listening.

B:  Well, the log backups were failing and the log filled up the drive.  So in order to prevent that, I had to put the DB into AutoShrink mode.

S:  Why didn't you just fix the log backups?

B:  Well, I was trying to, but I needed the time to work and this bought me some time.

S:  Did it now?  So how did this buy you time?  Did you actually get the DB back online with this?

B:  Yeah, I was able to get the DB back online for about 2hrs.

S:  And have you fixed the log backups yet?

B:  That's actually where I ran into the problems.  The problem is that while I was trying to free-up space for the log backups, I noticed a massive corruption in the DB and that's why we need to rebuild.  It's not causing an immediate problem, but it will and soon.  So we need to get it before it causes any real damage.

S:  Ok, well, I've got a login now so tell me where you noticed this corruption and I'll take a look.

B:  You don't need to.  I've already looked at it and all I need you to do is rebuild SQL and then restore the backups.

S:  The backups that are failing?

B:  Well, they're going to have to lose some data.  There's nothing that can be done about that.  But we have to rebuild right away.

S:  Ok, so I'm not going to rebuild just because you say so.  I'm the DBA here and I say when something gets rebuilt.  So either you share your findings with me or nothing is going to happen. 

B:  But I'm the one who manages this box and I have for the past 2yrs.  So I know this system and I know the server.  And we've had this problem off and on since we installed it.  We know the drill.

S:  That's great, but you need to explain it to me so I understand what's wrong with this system.

B:  What happened to the other DBA I used to work with?  He always just rebuilt it when I asked him to.

S:  He's gone.  Now explain it to me.

B:  Ok fine.  The problem is that while I was looking at the backups, I noticed that the TempDB backup was failing.  And I know from experience that that's the start of a major corruption.

S:  Yeah, I read that in your email.

B:  So why did you need me to explain it to you if you already read it?

S:  I really just wanted to see if you had the guts to be that stupid out loud.

B:  Ummm... ummmm... what are you talking about?

S:  You can't backup tempdb, stupid.  It's recreated every time SQL starts so it doesn't allow you to back it up.  You'll always get an error.  And what upsets me is that you've rebuilt this system and lost these peoples' data so many times over the past 2yrs for no other reason than you're just stupid.  Massive corruption... yeah, there's a corruption alright.

B:  Look, I don't think it's necessary to...

S:  Shut up, yes it is.  And what pisses me off the most is that you've been billing for this "support".  Trust me when I say you'll never touch this box again.  And turning on AutoShrink to fix a log backup issue.  Well that's just dumb.  The reason it worked for a little while is probably because the data and logs are on the same drive and it shrunk the data file a little and gave the log a little more room to grow.  But that's not even close to fixing the issue, and had they been split like they're supposed to it wouldn't have helped at all.  But a single shrink would have fixed that too.  Turning on AutoShrink is the worst thing you could do.  Look, you sound like a good kid.  And I'm sure you're just trying to do your best, but this is unacceptable.  You need to pick up a book and learn something about SQL before you go rebuilding boxes and losing people's data.  Your company has no business putting you on the front line like this.  And I'm sorry if I'm coming off as harsh, but the question now is what are you going to do about it?  Are you going to take this as a wake-up call and start learning something, or are you going to just go away and call me names?  It's up to you.

There was a little more to the call, but that was the interesting part.  My app group downstairs was shocked both at how I handled it and at how they could have rebuilt the system so many times for no reason.  I did have to bring them up to my desk and show them on 3 other servers that tempdb never backs up and it's just a stupid thing to say.  They were shocked.  But they were ultimately grateful that I stopped him and they've since given my group full control over that server and we've got the log backups and everything else going again. 

I'd say as a DBA I'm quite outspoken.  I don't put up with vendors messing around with our data and I certainly don't put up with vendors charging us so much money for support only to do us more harm than good.  This group would have been better off simply taking their chances on not having any backups at all.  You have to be outspoken in a large company though.  You have to walk a pretty fine line though between offending people and telling the truth.  And sometimes that can be a hard line to walk because you've got to lay down the law from time to time.  The only deciding factor in your success is whether the upper brass will back you up and let you make the right decisions or undermine you and let politics and stupidity rule.  My guess is that most of the time it'll be the latter, but you'll win battles from time to time and maybe you can slowly turn your ship away from a few icebergs.

Join the Network World communities on Facebook and LinkedIn to comment on topics that are top of mind.
Related:

Copyright © 2010 IDG Communications, Inc.

IT Salary Survey: The results are in