Insights about How Cisco Maintains Certifications

“I Love My Wife!”, but does Cisco Really Care?

I really do love my wife. But unlike some people, I don't write that in comments section when taking a Cisco exam. Yes, Cisco reads those comments we make when taking Cisco exams - all of them, including the comments like "I love my wife". Today I'll look at Cisco's answer to the question "What do you do with all those comments?", along with another question we had about potential Data Center certifications.

This post is the second post about the questions we all collected a few weeks ago, which I then asked about at Cisco Live!, aka Netwokers. Fred Weiller, Director of Marketing for Learning@Cisco, was gracious enough to answer questions. I also asked some of the questions informally amongst others at the show.

The first question for today's post:

What do you do with the feedback we type on an exam?

For anyone who's taken a Cisco exam, it's obvious that the comment feature in the exam is not intended for 2-way communication. There's no place to leave contact info, other than to type it as part of the comment. Personally, I think that's perfectly reasonable, but surely someone at least reads the comments, right? I asked Fred what they did with them. Summarizing what I understood Fred to say:

• The Product Manager for the exam reads every comment
• The PM can weed out the surprising but unrelated comments like "I love my wife"
• The comments remain available for later review, e.g., they can look at all comments on Question X
• They might act to change/delete a question based on comments
• They might act to change/delete a question based on some stats, and use the comments to better understand the question

There were two surprises, for me at least, with Fred's answer. First, they read every comment. I figured it would be reactive, e.g., a problem is scoring poorly (too high a percentage of correct, too high a percentage incorrect).  But they read them all. The second and much funnier surprise was that people really do write completely unrelated comments, like "I love my wife" and "I wish this exam was over".  I wonder if there's an opportunity to use positive reinforcement, typing, "I'm going to pass" as a comment on every question, as a way to keep the test taker confident when taking the exam.

But this conversation with Fred led to others, from which I draw this conclusion: Cisco wants each exam question to fairly and effectively assess whether the test taker has the knowledge and skills specified for that test. Cisco can't do that by only looking at the comments, so I asked more about the more analytical side of maintaining the question database for an exam. I'm describing it here, but I'm sure I'll lose some exactness in my attempt to summarize - but I think you'll get the general idea.

Cisco runs stats against every question, in fact, against every answer. They look for underperforming questions and answers. But "underperforming" is a loaded word. Part of the logic relates to something called Item Response Theory (IRT). Generally, the idea is that each question should be measured as to whether candidates that should fail answer incorrectly, and candidates that should pass answer correctly. Admittedly a layman's interpretation so let me expound a bit.

First, consider 100 ICND1 exams. Cisco has a passing score for the exam. Then consider question X, which was on all 100 of this sample set. If you scored the question on the same 300-1000 scale Cisco uses for how they score the exam, a well-performing question gets a score around where the cut score (the passing score) is for the exam. If question X scored a 500, and the passing score is 800, then it may be underperforming. If it scores a 990, it may be underperforming as well.

It may at first appear that this boils down to what questions are too hard and what questions are too easy, but it's not just that. Question X may be difficult, but it may be that the right percentages of test takers happen to know that topic very well. Or the question may be simple, but none of the available study resources happened to include that topic. Or, maybe the topic is actually out-of-scope compared to the exam topics, and it somehow got into the exam database anyway. For example, maybe it's a tough question, but the Exam Cert Guide for that exam does a great job of going over a very similar scenario.

After digesting all this, I draw two major conclusions. First, Cisco spends considerable effort to maintain the integrity and accuracy of the question databases for each exam. It's not just a couple of geeks that know the topics that react if someone complains loudly, but instead, they take a much more analytical approach to maintenance.

The second conclusion is that even with all this work, it is probably impossible to find and fix all potentially ambiguous questions. I'm not talking about totally busted questions, e.g.,  "What does RIP use for a metric", and "hop count" is not an answer. I'm not talking about questions where you don't happen to know the topic well enough to interpret the meaning of the question and answers. I'm talking about questions where you know the topic well, you read the question, and you wonder what the question really means.

The second question I'll review in today's post is:

Will there be a Data Center professional certification?

The answer was a process answer, which in itself was interesting. Fred said that the typical path to create a new career certification - i.e., CCNA and CCNP like certifications - was to first start by adding exams and certifications in Cisco's specialization certs. Then, Cisco monitors the growth rate, and considers when to create a "career" certification track.

If I were guessing, I'd have to say that they would be here one day. Moving beyond Fred's process-oriented answer, the list of such Data Center specialization certs is already long. I have no idea about volume of folks getting these specialization certs yet, but there's already a CCIE storage cert which could be morphed into a Data Center exam.

No survey this post, but some questions to spark discussion. After reading today's post, will you change how you supply comments to Cisco on your next exam? Do you have more confidence in the accuracy of the questions on Cisco exams, or not? And what's the next technology for which you think Cisco will add a CCNA and CCNP level cert - Data Center, or something else?

• Careers
• Cisco
• CCENT
• CCIE
• CCNA
• CCNP
• Cisco labs
• Cisco Press
• Wendell Odom