Performance Testing Revisited

How much of a difference would it make if Cisco changed all of its certification exams to performance-based testing? Answer: A lot.

A few weeks ago, Wendell Odom asked this question in his NWW blog about the potential for changing all Associate- and Professional-level Cisco certification testing from its current format to performance-based testing. In a blog I guest-wrote for Wendell in December 2007, I expressed some thoughts on the value of certification to both candidates and employers, in an attempt to bring some perspective to reports that the value (in dollars) of technical certifications is slipping in the industry. In today’s post, I’d like to tie these concepts together and revisit ways that certification vendors can make their certs more valuable.

First, let’s look at what’s important to vendors in their certification programs. Generally they care about three concepts:

• That their certifications are perceived in a positive and valuable light in their industry. There’s a prestige element to this as well as a practical one. Think “branding.”
• That their certifications actually mean something—have relevance—to candidates, and to the employers and potential employers of those candidates. It’s more than just a little nice, too, if the industry press likes a vendor’s certifications, because this generates a lot of revenue and publicity opportunity. There’s a great synergy between Cisco Systems and Cisco Press, for example.
• The huge win for any certification vendor is that it drives preference, at the individual level, toward that vendor’s products and solutions. Almost by definition, a person who puts a lot of their own skin in the game by earning a professional certification shows significant dedication to that vendor. After all, the vendor has certified that person’s skills, which is an explicit validation of the individual’s efforts. Those certified individuals almost certainly will show their appreciation by preferring that vendor over others—and bringing formidable skill and knowledge to the game. In effect, certified people become an extension of the vendor’s workforce, bringing prestige and commitment—with no direct costs like salaries and benefits—to the vendor.

Think of the most well-known vendors in terms of their technical certifications; the list is pretty short. Now consider the number of certifications these vendors offer and the value of their “better” certifications in the marketplace. Then multiply by thousands and thousands of individuals. Based on this, it’s easy to see why it’s in each certification vendor’s interest to keep the reputations of their certifications at the highest possible level.

For these reasons, every certification vendor needs to worry about the technical rigor of their certifications, exam security, freshness of content, exam delivery, training content, and other key spokes in the certification wheel. Let’s focus on exam delivery, with an eye toward the types of questions on the exams.

Each exam must test the candidate’s technical knowledge in a way that results in low scores for people who don’t know the material—good questions penalize guessing. However, bad test questions, those that simply obscure the correct answer(s) or take too long to answer, also penalize good candidates. The trick is to deliver exams in a way that rewards those who understand the material without hurting those same candidates by giving them questions that are simply difficult or slow to answer, even if they understand the content. For this reason, most certification exams use several types of exam questions to balance test-taking skill with technical knowledge.

Performance-based testing conceptually requires candidates to show real-world skills for a successful outcome. The trick, once again, is balance. Cisco achieves this balance for expert-level certifications by using two exams—one multiple-choice and multiple-answer (the qualification exam), and the other strictly performance-based (the lab exam). This is a potent one-two punch that results in a highly respected certification that also requires a lot of commitment from candidates. Most certifications, however, use a single exam (or several exams on different topics), with blended question types, to arrive at the same result. What, then, is the right balance for these exams?

Focusing on Cisco’s Associate- and Professional-level certification exams, I think they have the balance about right. Cisco mixes several different types of questions, including performance-based questions, on these exams. Coming back to Wendell’s point in his earlier post, it may be true that solely performance-based exams—those made up of only simulation or “simlet” scenario questions—might do the best job of identifying the most qualified candidates. The difficulty with that approach is that the test-taking skill aspect of an exam with solely performance-based questions become perhaps as important as knowing the material. For example, someone who is a very strong critical thinker, with good reading and good organization skills, and whose native language is the same as that of the exam, will do comparatively much better at performance-based questions than someone who is weak in any of those areas. It’s not necessarily bad that this is the case, but you need to zoom out and consider what the exam is testing for—is it the technical skill or is it a reading and speed test? The answer, of course, is “yes,” but there’s a balance point that exam developers must find.

I think that for the entry-level certifications, the balance should be more toward the technical skill evaluation and less along the lines of complementary test-taking skills. As you climb the certification ladder, the balance should lead toward a more well-rounded, professional skill set—higher level technical competency along with a solid complement of analytical skills and the organizational ability to handle complex scenarios in a high-pressure environment. In other words, what you’d see at work every day. Therefore, higher-level exams should be driven by more performance-based questions.

A certification program profile that meets this description should hit all three goals I mentioned early in this post. Assuming, that is, that you agree with my points on the topic. What do you think is the right balance of performance-based questions for each level of certification testing? Which vendors get it right and which ones miss the mark?