Wow, I wrote my last post on a bit of a whim, and we got a great discussion going. Less than 10% of us think it'd make the Cisco Associate and Professional exams worse by moving to completely CLI tests. More surprising to me was that close to 70% thought that an all CLI test improved their chance of passing, with only 16% thinking it would decrease their chance of passing. Also, around 30% of you through that a 75 minute exam could hold 15 or more Sim/Simlet questions! (See here for the surveys.) From our little corner of the world, it looks like most people think that a big move towards hands-on questions on Cisco exams would be better for the test takers and better for the certifications.
Today I'll wrap up some thoughts on how this change could help defeat cheating to some degree. It seems to me that a change to an all-CLI exam - Sims (fix the config) and Simlets (answer MC questions by using the Sim) - gives Cisco the opportunity to defeat the memorization benefits when using brain dumps, making it more efficient to learn the materials using legal study tools. (If you want to read more about the issues related to cheating and preventing cheating, check out Robert Williams' Certification Integrity blog.)
One of the advantages when using a brain dump is that you can memorize the questions, so that on the exam, you could easily recognize and remember the question and answers, and confidently and quickly answer it. I think the Sim/Simlet question type gives the vendor a chance to make dozens of questions that outwardly look identical, with the differences being only on the CLI ad in which answers are correct. The result would be that the person using the brain dumps would both be required to learn all the related commands and concepts anyway. And if you have to learn the materials, it usually more efficient to learn it the good old fashioned way - study - rather than repeating a bunch of questions for which there's no good explanation as to why one answer is right and another wrong.
Also, using such a strategy would make it way easy to better defeat brain dumps by changing the exam bank more frequently. You create 25 questions from the same base question, put 5 in the database, and replace them every 2 months or so. Outwardly, the new questions look almost if not completely identical. However, the right answers are different because the initial configurations in the simulated gear are different. Not only do you have questions that looked like what someone saw in a brain dump - they're all different, with different answers, so you can potentially fake-out the person using the brain dump.
For example, imagine a question that shows three routers in a triangle of serial links, with 1 switch and 1 PC off each router's single Fa0/0 interface. Then figure is part of a simlet question that simply asks something like "what IP routes should exist on router R2?". The simlet only gives you access to routers R1 and R3, but not R2. The answers include things like subnets and prefixes listed in R2's routing table, next-hop IP addresses, source of routing info (eg RIP, OSPF), outgoing interface(s), Admin distance values.
Then, Cisco could take that same base question: the same figure, same question text, and come up with a set of say 15 answers. Then, to create a new question, they change the initial config in 1 or more devices. By changing the initial configs, and prudently choosing the pool of answers, you could have similar questions for which each answer was right on some questions, and wrong on others.
These different questions then require the user to look at the routers, know what commands to look for, what status to look for in show commands, predict the behavior on R2 in this case. Also, you can disallow access to enable mode, so the user can't even display the initial configurations on R1 and R3 - which makes the brain dump folks have to then analyze the question, deduce the device's initial configs, and causes them more headaches.
For example, the various questions could vary the following in the initial configurations:
With this style of question, the person who used brain dumps gets no real clue as to the answer by just looking at the questions and answers, or better yet, gets a false sense of what should be right when it's not. To get it right, the user has to learn the commands well. This method would help negate the benefits of having memorized the questions and answers.
Honestly, the motivation for changing to all hands-on would be to create a stronger certification (IMHO), with the benefit of helping reduce the benefits of using brain dumps being just a side-effect. What do ya'll think - does it help at all? Or if Cisco makes this change, should they just not even bother to create dozens of variations on these types of questions?
Odom, CCIE No, 1624, splits time between writing books for Cisco Press and teaching classes for Skyline ATS. In his 25-ish years in the networking industry, he has worked as as a pre-sale and post-sale SE for a few networking vendors, as well as a network engineer implementing network technology. Wendell has spent the majority of the last 15 years teaching, consulting, and writing about networking technologies, most of which in some way relate to Cisco products. His books include titles on QoS, CCIE R/S, as well as several titles related to CCNA certification, including the September 2007 book CCNA Official Exam Certification Library (CCNA Exam 640-802) (Read a sneak peek of chapter 7). Click for the list of current titles by Wendell.
|
|
How Come?
Wendell, you write the bloody Exam Certification Guides, how come you aren't writing the exam, or at the very least, being heavily consulted on the content, etc?
On the question of Sims (fix the config) and Simlets.
I'm not entirely sure if the people calling for 15 or more of them per exam, have actually taken the official exam.
In the current CCNA exams, a sim/simlet should take you 10-15 minutes to complete. It is pretty obvious from the time available in total, and the total number of questions given, that this is in fact the amount of time Cisco expect you to take on these questions.
I think you can all do the maths on that one.
Even at 5 minutes per sim/simlet, it's still 75 minutes for a 15 sim/simlet question exam.
I hope there is no theory, or the people are really fast typists, and none of them even begin to get stuck on a single question.
Jeremy Cioara a double CCIE iirc, admits that even he had trouble with the amount of questions he had to get through on the current 640-802 exam. In his CBT Nuggets, he advises that you take the ICND1 & ICND2 tests separately for this very reason.
Maybe that's the key, have 2 tests. 1 theory, 1 sim/simlet based practical, who knows?
To be honest, this doesn't affect me at all in my situation, but I cannot help but to comment on what I see to be serious deficiencies in some peoples reasoning, due to what may be lack of experience in taking Cisco Certification exams.
Why?
Hey Shaun,
Yep, 15's a lot on a 75 minute test. I think the right number for a 75 minute test, if the Sims/Simlets on CCNA are the same style/depth/difficulty, is 11 or 12. that's why I was so surprised at the number of 15+ choices. But then, even with the 2 exam path, is 24 Sim/Simlets enough to get to the breadth? Maybe not. Such amove might require a pair of 2-hour tests, 20 sim/simlets each, that kind of thing, and it mayput it over the edge.
Maybe the 1 theory/1 practial concept for CCNA would work - interesting thought.
I haven't seen stats for the actual exams, and percentage of people that seem to be using the 1-exam versus 2-exam path, but from some stats I've seen (the details of which I can't share), certainly more people are going for the 2-exam choice today versus with the old exams.
I can't really expound much on your initial "how come...", other than to say Pearson (the owner of the Cisco Press brand) and Cisco have a strong business relationship, which helps all us Cisco Press authors when we write. I can wax philosophic a bit and say that for any one person to have detailed knowledge of the actual exam questions also write primary study sources like the CCNA ECG book is probably a bad idea for the certification long-term. Sure, I'm occasionally asked my opinion about things, but Cisco has a lot of well-qualified folks that define all aspects of what happens with CCNA. I know it's in good hands. Sure, I could probably get someone's attention in the cert group at Cisco, just to ask questions and float ideas, and I figure Networkers is a good place to ask about the whole all-CLI thing I started last week.
From my POV, this is the thing...
If test creators start allocating only 5 minutes per sim/simulet, test takers are going to be forced to whack out the answers as quickly as possible, where the primary concern will be, how quickly you can jam commands into the CLI.
There will be little to no chance to linger on any theoretical concerns that actually test whether you understand what the hell you are doing, and we will be right back where we started, except people won't be complaining about other people simply memorising the theoretical answers to questions, they will be complaining about how people are simply memorising the commands to answer the questions.
What will be worse is, people who do understand what is going on, and really do have good problem solving ability, are going to be competing for jobs with people who can memorise a few commands and can type really fast, at employers who are going to think are extremely qualified, all because they managed to pass Cisco's new whizz-bang supposedly impossible to fool CCNA exam!
On a final note Wendell, have you taken any of the Cisco CCNA exams, or can you get access to take at least some of the Sims/Simulets currently available and give us your honest opinion on how much time if should take somebody who should make a good CCNA to complete 1 or more of them?
I'd be interested to find out how long the guy who writes the Exam Certification books, takes on average, to do these Sims/Simulets.
Time per question
If Cisco ever did go to an all Sim/Simlet style test, I really wouldn't worry about them setting unrealistic number of questions for the available time. Cisco would go through testing where they would work on the balance of the number and difficulty of questions in relation the the time alloted.
I can name that Sim in... X seconds
Back in the 1970's in the USA, there was a relatively popular game show called "Name that Tune". Quizzers would claim that they could name a tune only hearing some number of notes. (They got a clue as to what the song was as well). Your post sounded like that to me "I can do that sim in 3 minutes"; "I can so that Sim in 10 seconds"; "Do that Sim"!
Anyway...
I've never done Sims on CCNA/CCNP to go as fast as I could, partly because I got my CCIE before CCNA/CCNP even came out - so there wasn't a lot of motivation to take them other than to prep to write about them. However, if I were to guess, and was trying to anwer the Sims ASAP, I'd guess the fastest would be 90 seconds, slowest 180 seconds. The Simlets take me a little longer, since they typically have a net larger number of words to read/comprehend in the question, and that takes more think time.
I agree, it's not a perfect solution. I don't think that typing speed makes a ton of difference, particularly if you practice enough to be good at the shorter versions of the commands and the use of the tab key. A slow typist that does a sh ip ei ne will go a lot faster than someone who quicly types "show ip eigrp ?" to find the next parameter if they don't recall it. But certainly, the exam could facor those who memorize configuration checklists even if they don't know what they're doing. I think the best mix would be around 2/3 simlet, 1/3 sim, with the Simlets having the capability of testing the concepts/theory much better than Sims.
I think for the CCNA exam, you should allot 7 minutes per Sim/Simlet. I've not taken the CCNP exams recently enough to have a good estimate.
Thanks again Shaun for the posts!
Now Wendell....
You are showing your age by talking about Name That Tune! :) (And I am by admitting I used to watch it)
Part of the problem is that we have a different perspective on these simlets because of the experiences and tortures we have had to go through for the CCIE exam.
When just getting started in the networking world though, it's a different ballgame. Many people coming up have trouble enough with subnetting (good, bad or otherwise is irrelevant, they just do), so anticipating the number of routes may take a while to work through.
I applaud the idea. I think it's a wonderful idea, but the timeframe should be revisited. The two exam approach certainly isn't a bad idea. But let's take that a step further. What about all the CCNAs that are out there right now who may have snuck on in before this raising of the bar?
Cisco, in their infinite desire to be backwards-compatible is unlikely to tell existing CCNAs that they aren't really CCNAs any longer due to the rules changing.
With two exams, it makes a simple rule adjustment to force the CCNA recertification process require the Sim-oriented exam.
Like you, I do training classes. I specifically do CCIE Lab Prep training. I have PLENTY of people coming through classes who IMHO miss some basic concepts about how routers think along the way. I definitely support the idea of the pseudo-hands-on simulations and agree with you on the simple changes that could be done.
But split into two exams. Possibly increase the time required for the exam. If 75 minutes isn't long enough, make it longer!
Having recently sat through a CCDE beta exam pegged at almost four hours, I don't see a problem with the long timeframe!
Scott
Dating myself
Scott,
Yep, I'm dating myself (;-) with the Name That Tune reference. Almost as good as the Gong show - that's one that should be added to the lab exam - when you're 3 hours in, and you're already hopelessly lost, with no way to get enough points in time, they pound the gong and put you out of your misery.
With the breadth of CCNA today, I think you're right that making the exams a little longer in an effort to make them stronger would play well for the legitimacy of CCNA. On the other hand, for jobs focused on networking, you've gotta go past CCNA anyway, so maybe CCNA's role - and length of the exams - needs to stay where it is. Thankfully, I get to speculate, and then avoid making those tough calls.
Thanks for the post.
Wendell
The vote
I didn't vote for that reason, I haven't taken a Cisco exam yet so I couldn't vote on how many or how long for questions.
I don't think having the time really tight is a good idea people are nervous etc on exam day and being over concerned with the time allowed would be a bad thing I think.
Time pressure vs/ fairness
David,
Historically, most people who are medium-prepared or less feel time pressure on the current exams. For CCNA in particular, if you're well prepared for all aspects, you should be able to finish in the time alloted without too much pressure. In CCNA's case, that means doing subnetting in your head and relatively quickly, and the ability to use the commands covered on CCNA without doing a lot of question marks from the CLI. Of course, you need to know the usual facts/concepts.
Thanks for the post,
Wendell
Why All Hands-on Cisco Exam won't Negates Brain Dumps!
Sorry Wendell, I forgot to answer the original premise of the article.
An all hands on Cisco Exam won't negate brain dumps, not at the CCNA level, in fact it is likely to make matters worse.
Why?
Because the amount of commands a CCNA is expected to know is ridiculously small, if I recall correctly, it's only around the 30-40 overall commands.
Compare that to the CCNP where the number of commands blows out to up to around 160 commands, I think most people would be able to work out, it wouldn't be all that hard to give worked examples of just about anything a CCNA level candidate would be expected to do.
I guess the one thing it would stop is people memorising brain dump answers, and thus being forced to learn how to subnet for real, instead of just memorising the answers.
I've been led to believe that the current CCNA exams, focus heavily on subnetting anyhow. So long as the number of questions were high and properly randomised, that ought to be enough to defeat most brain dump users, as the possible combinations for subnetting is practically limitless.