Wow, I wrote my last post on a bit of a whim, and we got a great discussion going. Less than 10% of us think it’d make the Cisco Associate and Professional exams worse by moving to completely CLI tests. More surprising to me was that close to 70% thought that an all CLI test improved their chance of passing, with only 16% thinking it would decrease their chance of passing. Also, around 30% of you through that a 75 minute exam could hold 15 or more Sim/Simlet questions! (See here for the surveys.) From our little corner of the world, it looks like most people think that a big move towards hands-on questions on Cisco exams would be better for the test takers and better for the certifications.
Today I’ll wrap up some thoughts on how this change could help defeat cheating to some degree. It seems to me that a change to an all-CLI exam – Sims (fix the config) and Simlets (answer MC questions by using the Sim) – gives Cisco the opportunity to defeat the memorization benefits when using brain dumps, making it more efficient to learn the materials using legal study tools. (If you want to read more about the issues related to cheating and preventing cheating, check out Robert Williams’ Certification Integrity blog.)
One of the advantages when using a brain dump is that you can memorize the questions, so that on the exam, you could easily recognize and remember the question and answers, and confidently and quickly answer it. I think the Sim/Simlet question type gives the vendor a chance to make dozens of questions that outwardly look identical, with the differences being only on the CLI ad in which answers are correct. The result would be that the person using the brain dumps would both be required to learn all the related commands and concepts anyway. And if you have to learn the materials, it usually more efficient to learn it the good old fashioned way – study – rather than repeating a bunch of questions for which there’s no good explanation as to why one answer is right and another wrong.
Also, using such a strategy would make it way easy to better defeat brain dumps by changing the exam bank more frequently. You create 25 questions from the same base question, put 5 in the database, and replace them every 2 months or so. Outwardly, the new questions look almost if not completely identical. However, the right answers are different because the initial configurations in the simulated gear are different. Not only do you have questions that looked like what someone saw in a brain dump – they’re all different, with different answers, so you can potentially fake-out the person using the brain dump.
For example, imagine a question that shows three routers in a triangle of serial links, with 1 switch and 1 PC off each router’s single Fa0/0 interface. Then figure is part of a simlet question that simply asks something like “what IP routes should exist on router R2?”. The simlet only gives you access to routers R1 and R3, but not R2. The answers include things like subnets and prefixes listed in R2’s routing table, next-hop IP addresses, source of routing info (eg RIP, OSPF), outgoing interface(s), Admin distance values.
Then, Cisco could take that same base question: the same figure, same question text, and come up with a set of say 15 answers. Then, to create a new question, they change the initial config in 1 or more devices. By changing the initial configs, and prudently choosing the pool of answers, you could have similar questions for which each answer was right on some questions, and wrong on others.
These different questions then require the user to look at the routers, know what commands to look for, what status to look for in show commands, predict the behavior on R2 in this case. Also, you can disallow access to enable mode, so the user can’t even display the initial configurations on R1 and R3 – which makes the brain dump folks have to then analyze the question, deduce the device’s initial configs, and causes them more headaches.
For example, the various questions could vary the following in the initial configurations:
- The IP addresses/masks (and resultant subnets)
- Whether a router is at the boundary of a classful network or not (and the effects of auto-summary)
- The routing protocol used
- The user of passive-interface or not
- The configuration of one of several settings that impacts whether EIGRP or OSPF forms neighbor relationships (Hello, Dead, AND/process-id, subnet mismatch, etc)
- The user of secondary IP addresses
- RIP versions
- Authentication configuration
With this style of question, the person who used brain dumps gets no real clue as to the answer by just looking at the questions and answers, or better yet, gets a false sense of what should be right when it’s not. To get it right, the user has to learn the commands well. This method would help negate the benefits of having memorized the questions and answers.
Honestly, the motivation for changing to all hands-on would be to create a stronger certification (IMHO), with the benefit of helping reduce the benefits of using brain dumps being just a side-effect. What do ya’ll think – does it help at all? Or if Cisco makes this change, should they just not even bother to create dozens of variations on these types of questions?
