Don't panic. Don't rejoice. Don't send John Chambers a letter. But at the end of the post, I wanna know - did the title make you think "hoorah" or "oh @%$&"?
Let me be clear: this is just pure speculation on my part, but I found it to be interesting speculation, so I wanted to share it with you. I got started down this path by noticing Randy Muller's blog on the Microsoft Subnet. Paraphrasing, Randy's blog started to discuss how much real/virtualized/simulated/emulated content could be or should be on some of the Microsoft exams. That got me thinking about that same thing for Cisco exams... and that got me thinking about cheating, and all the good stuff Robert Williams talks about in his blog. (Good thing I don't have a boss - I'd be admitting I need more to do if I've got that much time to read!) Anyway, back to the main point, which I'll throw out as an assertion, and let you comment:
Cisco should change the Associate and Professional level cert exams to use only Sim and Simlet questions today, and strengthen it further by converting to using real IOS images on the testing platform through virtualization tools.
Quick background: Simlets use a simulator, but with multiple-guess questions. Sometimes, you don't get access to enable mode. Simlets require you to choose commands and interpret their output in order to answer the multiguess questions. Sim questions require you to change the config, so Sim questions always boil down to "the config is broken - fix it".
In month's past in this blog, when we've talked about the value of certifications, one of the overriding and most heart-felt opinions ended up as one of my blog titles: " Skills Rule, Certs (without Skills) Drool". Skills matter in real life, with the certs being 1 of many things that can give evidence to someone's actual skills. Certs without skills ... are bad, and from comments I've heard over the years, a cert without comparable skills makes people look worse than just not having the skills. So, my first argument is simple: an exam that requires the use of the CLI for majority of the exam time requires more skills at making the gear work, which in turn increases the value of the certifications, and the perception of those certifications in the job market.
But it's much more than a perception issue. I contend that if you made the exams all Sim/Simlets - and made each instance of the exam cover as many as possible of the related show and configuration commands for that certification - it would be a much greater proof of skills. Ultimately, it's a better gauge of the skills Cisco wants to see in their partner engineers.
I think you could still ask about concepts and theory with Simlet questions, and replace 80% of the multi-guess and drag-n-drop questions with Simlet questions. That's just a guess, of course. For the other 20% - things that might be impossible to ask via a Sim or Simlet - I'd argue that the benefits of a totally CLI-oriented test might outweigh the loss of coverage of some theoretical and conceptual items. How to test the theory with Simlet questions, requiring the user do some command? Most of the concepts and theory is reflected somewhere in the output of a show command.
For example, if router R1 connects to 2 different classful networks, uses RIP-2 but leaves the auto-summary command as default, R1 will auto-summarize routes to the classful network boundary. A Simlet question could start with those details in the initial configuration, and not even give the user access to router R1. The Simlet would then require the user to find the summarized route in router R2, as learned from R1, and know that auto-summary would be one reason that R2 only has a route for the entire classful network, instead of the particular subnet.
The biggest downside I see with such a move would be whether Cisco could touch enough topics on a single exam. So, for those of you who have taken a Cisco cert test: How many Sim/Simlet questions could you cram into one exam? Take the CCNA 640-802 (CCNA) exam, for instance. It's listed as a 90 minute test, but 15 minutes of that is so you can get settled in, take the exam tutorial, etc. So... how many Sim and Simlet questions would be comparable time pressure to the current mix of CCNA exam questions? 8? 10? 12? And if you knew it was all related to CLI in some way... would 15 be out of the question in 75 minutes? And even with 15 questions, could Cisco cover anywhere near as many topics as they do on a single current CCNA exam?
So, survey time. Give me your posts, your survey clicks, your rants about how absolutely crazy this idea is. And let me state again - if I had heard this from Cisco, no way I'd be able to post it here. So, it's just Wendell's ramble! Let me know your thoughts.
Later in the week, I'll wrap this up, address what you bring up, and talk about how I think making a change to all CLI-based questions solves the braindump cheating problem.
Odom, CCIE No, 1624, splits time between writing books for Cisco Press and teaching classes for Skyline ATS. In his 25-ish years in the networking industry, he has worked as as a pre-sale and post-sale SE for a few networking vendors, as well as a network engineer implementing network technology. Wendell has spent the majority of the last 15 years teaching, consulting, and writing about networking technologies, most of which in some way relate to Cisco products. His books include titles on QoS, CCIE R/S, as well as several titles related to CCNA certification, including the September 2007 book CCNA Official Exam Certification Library (CCNA Exam 640-802) (Read a sneak peek of chapter 7). Click for the list of current titles by Wendell.
The Leader in Network Knowledge
Although sim/simlets are a
Although sim/simlets are a better thing than today's "silly" questions, i believe the use of dynamips and real ios would be the best solution.
Dynamips
Cisco needs to put an end to speculations among engineers whether use of Dynamips is breaking Cisco licensing policy or not. It's good that you bring it up in the context of the exams. I think it will only be to Cisco's "huge" advantage to partner up with the creators of Dynamips as it grows ever more popular.
Incorporate Dynamips into the exam somehow; use it for everyday testing; use it for learning; and darn it, use it for baking a cake if it can.. sorry.. i really like Dynamips. :D
Baking a Cake? Web 2.0?
Arageon,
Maybe a combo of Dynamips and some web 2.0 portal could indeed allow us to achieve what was formerly unthinkable... Maybe we'll have a Dynamips Bake-off! ;-)
W
Aragoen, Cisco is using an
Aragoen,
Cisco is using an emulator already internally, but we may not see it make it into the test environment for a while.
See here:
http://brokenpipes.blogspot.com/2008/03/how-to-become-ccie-v2.html
"In fact, I did all my practice for CCIE SP only with (censored), something similar as dynamips. Censored = internal info to Cisco employees heheh."
It Would Be GREAT!
Long time lurker, first time poster. AWESOME blog, Wendell.
I think it would be fantastic. Not only would it prove that the certification holder can apply his knowledge hands on, but it would make the tests easier for those who understand and are able to apply knowledge (versus the book geniuii who can memorize everything, but have no clue what to do with it).
I would really like it if Cisco truly implemented this.
Yes it would
Colby,
I also think that this would be fantastic. It really does show you can be hands on, heck, you practically need to be hands on be think ing about taking the exam this way.
Lurk away
Colby,
Glad you like the blog! Who knows, maybe it'll at least migrate that direction...
Wendell
The problem with a short all CLI exam
Hi Wendell
I see several problems with making the NA & NP level exams primarily CLI based.
1. Instead of turning out Network Engineers, it will only end up turning out Network Mechanics who know how to do things, without knowing why they are doing it.
2. It will end up swapping 1 type of stupidity for another. The exam will go from having people who can memorise a whole bunch of theoretical knowledge without necessarily understanding what it all means, to a bunch of people who can remember Command Line procedures, without necessarily understanding what they are doing. i.e. If you change the ip mtu then you need to adjust the ip tcp adjust-mss by X without actually understanding either of them are manipulating.
3. If you turn your Network Engineers into mechanics who simply types commands in by rote, the chasm between the NP & IE will become even worse/harder/larger than it is now, especially when faced with really difficult real world or even worse, totally off the wall made up problems that you tend to encounter in a CCIE practical situation.
Finally, I am not against having a lot more simulation questions for the NA & NP certification tests, but there has to be a balance between making sure the certification exam tests both practical & theoretical knowledge.
Yes, I acknowledge it is not easy, if it was, we would have the solution already, but I felt it was worthwhile pointing out the issues with overreacting to the current certification crisis, which has been around for quite some time now anyway.
Cheers
Shaun
P.S. Wendell, whatever happened to the prizes for the CCNP Lab recommendations?
But I want to IGNORE the details ;-)
Shaun,
In reality, a wanton move to all sim/simlets would have some problems, including those you pointed out. Maybe I'll float the idea in person at Networkers in a few weeks, and see if they've already ruled out such a drastic move for these reasons or others.
I don't know if I'd agree with point 3, though. I don't think this change would increase the gap from NP to IE - I think it'd decrease the gap. I think the gap is in how to use to commands to accomplish a goal, so if NP required complete mastery of the commands covered in those exams, I think you'd be closer to passing the practical. I think that the all CLI exams could test a large part of the theory covered on the current exams, while lots more CLI, by doing what the IE lab does to some extent - that is, require you to apply theory to choose the right feature, right commands, and config them right. But, I could be wrong on this point.
Thanks for the through post on the CCNP labs, by the way. However, you actually posted after the contest deadline. Here's the link to the post where I announced the winner:
CCNP Lab Suggestions Winner – Free Book!
As usual, Shaun, thanks for the well considered post.
Wendell
CCIE versus CCNP exams
My understanding of the CCIE Lab exam is, not only do you have to have an excellent grasp of the commandline, but to have any chance of passing, you really really need to know your theory thoroughly as well.
More importantly, you can get away without memorising every command down to the letter, as you always have access to the documentation CD, but without a thorough grasp of the theory and concepts, you would be dead in the water, especially in the CCIE Lab exam, where they actually do try to trip you up. Things like, the obvious solution to 1 section breaking functionality in another or no configuration prompt in global config mode (the infamous no service prompt config command) etc.
As you have pointed out, it is going to be extremely difficult to increase the number of sims, and be thorough, and still make the test viable in terms of time allocated, and cost, which is why passing the 8 hour CCIE Lab exam is so highly valued.
DOH!
I thought I was a shoe in for something there, since I contributed quite bit of the the hardware ideas.
Oh well, netx time I guess.