How much time do you spend on product evaluation?

A couple of weeks ago I had the privilege of participating in Postini’s user conference in San Francisco. The event was very well managed and provided lots of informative content in a variety of breakout and general sessions. However, one session I attended really stood out as an excellent example of what user organizations ought to do when selecting new messaging capabilities.

The user company that presented at this particular session cannot be identified, but it is a Fortune 500 company focused primarily on manufacturing. The company used an anti-spam appliance that was permitting too much spam to reach its end users. The IT department realized it needed a better solution to address the company’s growing spam problem, and so undertook a very comprehensive analysis of a variety of appliances and managed services.

The company ran a pilot of each product or service under consideration in order to gather detailed information on each solution. A committee was then organized to determine the various criteria on which the offerings would be evaluated, to weight each of these criteria in terms of how important each was to the company, and then to rate each offering on a scale of 1 to 10 for each of the criteria. The committee members would discuss each offering’s rating on a particular criterion and hammer out any differences that they had on particular ratings. The final result, after a great deal of hard work, was a matrix that provided a total score for each of the several offerings.

Obviously, since this was a Postini user conference, the winner of the competition was Postini, or why else would the user company be presenting. With before and after stats provided by this company, the speakers showed that the Postini solution was able to dramatically lower the amount of spam reaching its end users.

What impressed me about this company’s efforts was how comprehensive they were and how much hard and tedious work this project represented. The effort that this company put forth represented true due diligence in an effort to solve a quite serious spam problem. What I’d like to find out from you is this: When you’re evaluating new messaging capabilities, how many person-hours go into the entire process, including evaluations of various solutions, testing, piloting and so forth; and how long does the entire process take from start to finish? Please drop me a line.