Novell Knowlegebase praised

A month ago I castigated Novell for what I thought was a decided decline in its online support, specifically the Novell Knowledgebase and its associated search functions. Soon after I finished writing that newsletter, it was announced that Novell had been cited as one of the “Ten Best Web Support Sites of 2003” by the Association of Support Professionals (ASP).

I still stand by my criticism of the Knowledgebase, so how can we reconcile that with this award?

According to the ASP’s guideline, “The award winners were selected by a panel of judges with expertise in Web support design and implementation, using a scoring system based on 25 separate performance criteria. The evaluation process covers the following areas:

* Overall usability, design, and navigation.

* Knowledgebase and search implementation.

* Interactive features.

* Personalization.

* The major site development challenge.”

Well, “knowledgebase” is second on the list. How can that be?

Some time next month the ASP will publish full details of the judging and a list of the winners but I was able to sneak a peek at a preliminary version. Concerning the knowledgebase, there were four questions that judges considered. I’ve reproduced them below, with the guidelines for assigning scores (from 0 to 4.0) and the approx. score Novell received (final tallies weren’t certified when I saw the results).

First, judges were asked “What’s your impression of the site’s overall knowledgebase content (tech notes, articles, tutorials, FAQs, etc.)?”

Guidelines were:

* Comprehensive, in-depth collection of several different kinds of content (4.0).

* Good collection of problem-resolution tech notes, not much else (2.0).

* Content seems skimpy (0.0).

Novell scored 3.75. I don’t disagree with that finding; there is a plethora of information. My objection was to the completeness, accuracy and applicability of the information on Novell’s Knowlegebase.

Next the judges were asked, “What’s your impression of the Search and Browse functions?” The guidelines were:

* Retrieves documents ranked by relevance; clear titles and summaries (4.0).

* Retrieves many documents, but some are off-topic or poorly described (2.0).

* No Search/Browse function, or function is ineffective (0.0).

Novell scored an impressive 3.8. While the Novell Knowledgebase search does retrieve documents with clear titles and summaries and does show a “relevance ranking,” it’s very easy to find off-topic results returned. Here I question the methodology, which simply looks at quantity rather than quality of the search results.

The third question was “What’s your impression of other knowledgebase options (e.g., natural language queries, troubleshooting wizards)?” Here the guidelines were very straightforward:

* Well implemented, very useful (4.0).

* Available, but implementation needs work (2.0).

* No other knowledgebase options are offered (0.0).

I would have given Novell a 3.75 on this based on the guidelines but the ASP judges only awarded 3.5 points.

Finally, the judges were asked: “How would you describe the writing and editing of the knowledgebase text?” The guidelines offered were:

* Well written, easy to follow, step-by-step (4.0).

* Adequate, but occasionally hard to follow (2.0).

* Documents are poorly edited and often confusing (0.0).

Novell received 3.25 points from the judges. Rather high in my estimation, (I might have awarded a 2.5), but there is another consideration.

While we can hope that the judges award points based on objective measurements the reality is that the points are awarded subjectively and judges can be swayed by comparisons to other sites they are judging. The result is what we called “grading on the curve” in school – rather than award everyone a grade between 2.1 and 1.9, you scale the result to a range of 3.5 to 2.0.

Novell does maintain an excellent support site. When judged against other support sites (Microsoft and Dell come to mind, but there are many) it’s really outstanding. Novell does deserve the award. But compared to what it could be, well that’s another story and one I already told in “Getting your problem solved at Novell.com is hard work.” Don’t rest on your laurels, Novell.