Benchmarks aren’t the only test

When you think about testing your apps, what comes to mind?

One thing that's no joke in IT is testing applications.  It's something that gets overlooked so often and when it is done, it's not done right.  And even discovering where your danger spots are can be difficult. 

Now, normally when I say testing I mean benchmarking, and that's certainly a part of it, but it's not the only part.  There are actually 3 different levels of testing your DB and your app that you have to consider if you're going to be really thorough.  And those levels are performance, regression, and data purity.

So let's take a look at each one of these real quick and we'll go in order of least to most difficult.


And of course by performance I mean benchmarking.  This is the classic scenario that all of the DB vendors push down our throats every chance they get.  Press releases with one common message are plastered all over our magazines and websites:  Our database can beat up your database, or your database is a pile of boogers next to ours... or something like that anyway.  So as it turns out, benchmarking is actually pretty easy.  Generating users can be a tiny challenge, especially if you need to simulate hundreds or even thousands it can be a little tricky, but for the smaller scale tests it's not really that hard.  That's where the performance benchmarking vendors come into play.  They can generate lots of users very easily and even allow you to distribute the load easily.  So there's definitely value added there.  Once you get your user problem worked out the next issue is simulating an exact workload.  And depending on the complexity of your app, it can be a tricky thing to do... even for the vendors who live in this space.  The problem of course is with the timing between the processes.  If you need to coordinate something precisely it's not an easy task.  You can capture a workload from your profiling tool, but you lose the dynamic nature of your benchmark if you make any changes.  And of course you also lose your timing.  However, even with its minor problems, you can still do some very nice simulations and you can get a good (but not generally exact) picture of how your app will scale.


This is a fancy way of saying you wanna make sure nothing in the app has broken while you were fixing something else.  So do all the buttons and menus still do what they're supposed to?  To a large degree a lot of this testing is done by companies either not at all, or informally by hand.  If you're lucky your company has developed a checklist for testers to go through so they don't forget anything, but you're still relying on them to actually perform the test.  We had a situation last year where the guy doing our testing for us cleared the app upgrade and when it went to prod it bombed.  When we went back to the test results they showed everything cleared, but it clearly wasn't working.  He didn't actually perform the test.  When it all came down to it, he thought there wouldn't be any way for this piece to break because there were no enhancements to it.  Little did he know that the dev commented out the code to test something else, and just forgot to change it back... OOPS.  The truth on the ground is that level of testing is boring and nobody wants to do it.  Even professional testers don't want to be bothered with that boring stuff.  And this is why they make frontend testing apps for this type of thing.  You can program in all the tests you want and it'll run them every time without complaining or lying about it.  So if you really want to test your frontends, there's just no better way to go than getting a tool.  Visual Studio 2010 has some really good stuff in this area.  Some others are WinRunner and AppLoader.

Data Purity

This one is just tough.  So you've benchmarked your DB and you know it'll perform.  And you've regressed your frontend and you know all the buttons work the way they're supposed to.  But what about the data?  Does every SP and every query return what it's supposed to?  Did the format of the phone numbers change slightly?  Did the last names get truncated?  Is the logic right?  When you choose all customers in a single region is that really what you're getting with nobody extra and nobody left out?  And the more complex the logic the harder it is to make sure you got it right.  Currently I don't know of any apps that can help you with this type of testing.  You've got to compare resultsets from the new version with resultsets from the old version.  Or you've got to make sure that new SPs are doing what they're supposed to do.  Wow, that's tough.  Currently this is still a manual process which is why it's mostly left to the individual devs who write the procedures.  The problem comes when they may not even fully understand the logic themselves or they may just give it a cursory glance and not a full inspection.  And if you've got a big app they could have a lot of code to go through.  At my last gig our big home-grown app had accumulated over 3,000 SPs alone, and over 4,000 views.  Go ahead and try to check all of those.

The point here is that there are different levels of testing and each one comes with its own challenges.  It's not easy running a dev shop, but if you're going to take on the task you need to try to do it right.  Get yourself a testing suite of some kind and program your tests once and be done with it.  If you think they're expensive, well you may be right, but balance that against hiring a couple people and all the overhead involved with that.  Tools pay for themselves pretty quickly especially when they can work 24/7.  Testing tools aren't going to forget to test something, or be tired one day and half-ass it.

Join the Network World communities on Facebook and LinkedIn to comment on topics that are top of mind.

Copyright © 2010 IDG Communications, Inc.