The Cert Exam Problem

Cert exams often test for the “best” answer instead of the correct one, which makes it increasingly possible that individuals are only able to perform in a certain set of scenarios and oftentimes, the wrong ones. How do we fix this? Get rid of them - ALL of them. Just kidding. I think there is a very practical way of fixing them, however I think it cuts into the revenue streams of the vendor and the testing companies, because in theory, people would have more success.

Quick disclaimer, this blog entry is based on my anecdotal experience, having tested through at least 5 vendors at various levels through my career. However, I know of several other very intelligent individuals who would echo some of my sentiment.

Elephant in the Room

Everyone is different. If you haven’t figured that out by now, no one will save you. Every person is going to read and more importantly understand a problem differently. When presented with an exam question, some people don’t even read it the same way (more on this later). In short - exam providers MUST understand this. If you’ve tested in the last 10 years or so, you’ve probably been given ear plugs, a sheet of laminated paper and dry erase marker, maybe headphones. The testing centers certainly understand that everyone is different, so why does it seem like exam providers fail at this?

Test Knowledge, Not Reading Skill

Okay, I think it’s important everyone knows how to read. We Americans have had the largest drop in reading scores since 1990 just a few years ago. I don’t think we need to make exams out of pictures or make each question no more than 10 words… but it seems the trend is to take a question, and force as many red herrings into it as possible. There is a clear difference between testing someone’s knowledge of a problem, and making them jump through “hoops of understanding” just to pass the exam.

Again, I am actually for making tests challenging. I don’t think someone should be able to simply pay for an exam slot a month before and, with zero prior knowledge, pass the exam (I’m guilty of this, oops). But in each of my last several exams alone, I’ve had to re-read a question multiple times and a couple of those times, I’ve come out more confused than my first time reading it through.

What is the Point of an Exam?

If we really think about the intent of a vendor exam, I think it comes down to a few items of intent.

Knowledge of Subject Matter

This one is obvious. Vendors are testing an individual on the subject at hand and giving them something which informs others that the individual “knows the subject”. If I sign up to take a Microsoft Office Excel exam, I’m being tested on Excel. That makes complete sense.

Credentials

“GIVE ME SOME RESPECT BOY, YOU’RE TALKING TO A CCIE, CISSP, MS MVP, CRM, IRT, ABCDEFGHIKLMNOPQREEEEEEEEEEEEEE”

I’m being facetious here, and this is more of a societal thing… but who wouldn’t want to show off their top tier credentials they spent months or even years trying to earn? I will never knock someone who is proud of their accomplishments. Though there are some individuals who let letters at the end of their name or a credential on their resume get to their head.

Career Change or Improvement

For some the credential helps individuals move up or out. I know a lot of the cybersecurity space has been pushing users to “Go get the Sec+, and get into Cybersecurity” (I have opinions on this, maybe it will be another blog post). Of course a huge perk is proving to your boss, a prospective organization, and yourself that you’re capable of both improvement and taking the next step in your career.

Do Vendors Excel at Serving the Candidate?

For the most part, I think some vendors do a good job. I actually think highly specific vendors do a better job than the big vendors. If you’re taking a highly specialized exam, like Veeam, Zscaler, Crowdstrike, etc, the test material likely is put together by the teams who help create the product. Think about it - the vendors themselves are providing their user base with knowledge on their own product, and this knowledge should (hopefully) reduce service requests, and also provide their customers with an investment route into their product.

On the other hand, I think some of the larger vendors (Microsoft being one of these) might not take my above points into account. Their product catalog is longer than a CVS receipt, it’s probably impossible for exam quality to be consistent throughout their certifications. Additionally, some of the bigger vendors have multiple facets of products. Using Microsoft as an example, you have, Virtualization, M365 Applications, Development, Automation, AI, Data, Endpoint, Security, the list goes on. I can’t exactly be mad at them for at least trying to offer training and certifications for all of these… perhaps the exam candidate just needs to understand the quality breakdown with larger orgs?

Some vendors offer beta exams, but the target audience is likely individuals who have taken their exams before and know the exam game.

The Real World

The beauty of this all is that exams are not necessary. Do I need to know how to double encapsulate the flux capacitor with LACP, while adhering to SOC2 compliance? No. But not because that’s not a real thing, I, just like everyone else with an internet connection, have Google. The likelihood of me not having an internet connection is relatively small, and so when problems arise, Google is there. Additionally, with the ever growing age of AI, I can probably find the answer just by asking Copilot.

Now I’m not advocating for individuals to forgo certifications entirely, or rely on Google to do their job (though it is a skill and also probably the reason AI is flourishing). I think what I’ve outlined above gives anyone enough ammo to go after a certification. The value is (usually) there.

Solving the Problem

The unfortunate reality is that vendors need to put more time and effort into validating exams prior to making them public, this includes beta exams.

A vendor I previously worked with had a learning and certification manager who would reach out on a regular basis to me. The same was said of others. The learning team actually cared about their exam candidates and were always looking for feedback. This vendor was the cream of the crop when it came to exams. I’d passed a couple of their exams and failed a couple too; my failures I had direct feedback on and I knew why I failed. There wasn’t guesswork on what the question was asking, I knew what was being asked, I didn’t have to find the “best” answer. The real world answer was right in front of me (whether I knew it or not :P).

Beta exams are nice, but as previously mentioned, sometimes they’re directed at individuals who’ve already passed a previous exam. I can understand that there are large unknowns in pushing a beta exam to someone who’s never taken your exams before or perhaps never taken an exam period.

Vendors simply need to push more of their cashflow into a dedicated team, who not only manages the content but engages with the community. They also need to be aware of the Elephant in the Room, and not look for the “most correct” answer, but the real world answer.

In Summary

Perhaps I’ve answers some unknown questions I’ve had for some time… that Certification Exam Problems really exist, or maybe are exacerbated, by the vendor organizations which create the exams. The bigger the vendor organization, the more likely the exam may not exactly be highly tailored to the product, or the more likely that the quality would be “shared” amongst the vendor’s product line. With these larger organizations, Cisco, Microsoft, Google, Amazon, etc, the unfortunate reality is that the exams carry a lot of weight in the community and I believe a lot of the reason the tests are “hard” isn’t because of the knowledge required to pass them.