Why propose an advanced certification in software testing?

A couple of weeks ago, I posted A proposal for an advanced certification in software testing. There were plenty of comments, on the blog, on Twitter, and in private email to me.

I think the best way to respond to these is with a series of posts, each one focused on a different issue. This first one goes to the fundamental question, Why should we create such a thing?

I used to see certifications as irrelevant (and misleading)

For a long time, when people asked me whether they should get certified in software testing, I said no. I would say that, in my opinion, there is no value in the current certifications.

I know more good testers who are not certified than good ones who are certified. I feel as though I’ve met a whole lot of clueless fools who carry testing certifications.

Many of the exam-review courses teach to the exam and present an oversimplified and outdated view of the field. I think that, from a what-will-you-learn perspective, taking them is a waste of time and money.

It used to seem obvious to me that certification must be irrelevant to a tester’s career.

The market proved me wrong

Unfortunately, my predictions that the community would see the ISTQB/ASQ/QAI-type credential as irrelevant were proved wrong.

The fact that hundreds of thousands of people in our field have decided to get certified demonstrates, in and of itself, that the credential is widely perceived as relevant.

I think that a willingness to discover and publish that you were mistaken is one of the critical traits of a scientist. The history of science is the story of of a never-ending stream of ideas that were well supported at the time—but were proved wrong. They were replaced with better ideas that were more useful and better-supported—and proved wrong too.

It seems to me that I can’t be a great tester (or an adequate scientist) if I am reluctant to subject my beliefs and ideas to the same level of criticism that I apply to the work of others.

In retrospect, I realize that I misread the evolution of certification in 1990 through 2010.

  • The testing community had a growing core of people who had decided to do this work as a career. They believed they were committed to doing good work and that they were good at what they did.
  • The demand for testing services was exploding, with floods of new people who had little background, varying levels of commitment and increasingly inflated salary expectations.
  • Many of the people who saw themselves as professionals were getting tired of being characterized as unskilled, clueless bureaucrats by so many other people in the development community.
  • Many of the people involved in recruiting testers or setting their pay scales don’t know enough about testing to tell the good ones from incompetents who can spin persuasive resumes and interviews.
  • In this environment, even if you are a test manager with really good hiring instincts, you still have the challenge of justifying the salaries you want to pay to people who don’t understand your staff.

Certification was sold as a formal credential, something that demonstrates (at a minimum) that you are committed enough to the field to go through the hassle of getting certified. And as proof that you are at least familiar with the basics of the field and that you are good enough at precision reading to be able to pass a formal exam.

If there is no stronger credential in the field, it is easy to see this as better than nothing.

I think that some of the get-certified sales pitches goes far beyond than this, saying or implying that certification demonstrates that a person has genuine professional competence. I think that goes far beyond what any of these certifications could possibly attest to, but I think that’s the impression that is sometimes encouraged.

We can argue about the motivation and about the marketing. We can speculate endlessly about why someone would spend good money on exam-prep courses so they could get one or more of these certifications.

I think it is more useful to ask whether we can give them better value for their time and money.

One approach: Open Certification

My interest in creating a better alternative to the current certifications is not new. Back in 2006, Mike Kelly and I started hosting workshops to plan an “Open Certification”. The idea was to create a huge, open pool of multiple-choice questions and to examine candidates via a random stratified sample of questions from the pool. However, there were some insurmountable problems:

  • We were determined to not be tied to one proprietary body of knowledge. But consider this example: Suppose we are willing to accept six different widely-used definitions of “test case.” Which one is the right one for this exam? And what if the student encounters (and answers on the basis of) Definition 7? How do we say that one is wrong?
    • The obvious way to deal with this is to write the question to say “Famous Person 1’s definition of test case is …” but what do people have to do to prepare for such an exam? Do they have to memorize 6 different definitions and the names of the people we tie those definitions to? Almost no one could pass such an exam. An even if you could pass it, all the memorizing you would have to do in order to pass it would be an abuse of your time.
  • We were determined, back then, to do something extremely cheap or free. But the development and maintenance costs for the software and questions were going to be very high. Even if we could get volunteer labor to create the first drafts of the exam (and exam site), we would need to do a lot of sustaining engineering. People were going to have to be paid.
  • The exam would be free but with this complex a series of questions, how long would it be before training companies started selling exam prep courses? The cost of the exams is not the big cost factor in the other certifications. It is the cost of the training. Were we kidding ourselves about making a difference here?
  • Finally, there was the most difficult problem. Even if the exam was successful, it would still be a bunch of multiple-choice questions. Our approach to certification wouldn’t be offering any better evidence of deep knowledge or skill than the others.

I forget his exact words, but Mike laid out an important criterion early in the project. If we couldn’t be confident of developing something clearly better than the alternative we were replacing, we shouldn’t bother doing it. As we proceeded, it became clearer and clearer that we were creating something that might be cheaper, but that probably wasn’t better.

Eventually, we pulled the plug on Open Certification.

But that was not abandonment of the idea of a better certification. It was a recognition that we didn’t have a better idea, yet.

In parallel with the Open Certification project, I was transforming BBST from a purely academic course to a very student-challenging industrial course.

One of the really valuable outcomes of the Open Certification meetings was a “standard” for drafting challenging multiple-choice test questions. I applied this to the BBST courses, creating a suite of quiz questions that BBST’s graduates have come to know and love.

But we didn’t stop with multiple-choice. We used multiple-choice as a tutorial tool, not as the core examiner. BBST demanded a much higher level of knowledge and skill than I knew how to get from multiple-choice exams. I concluded that something along these lines was a better way to go.

Another alternative

Rather than trying to replace the ASQ/ISTQB/QAI approach,  I think we can build on it.

  • Let people get one of those credentials. Or let them get some other credential that is challenging but that approaches the field in less simplistic terms. Treat their credential-from-training as a baseline.
  • From here, let the tester present a portfolio of evidence that s/he can do more than just pass an exam or two—that s/he can actually do competent work in the field.

The person who can demonstrate both, mastery of basic training and a competent portfolio gets an advanced certification.

I think this gives us two important advances:

  • It breaks out of the ideological stranglehold that a few vendors have had on credentialing in our field.
  • It presents a richer view of the capabilities and contributions of the person who carries the credential.

This isn’t perfect, but it’s better. I think that has some value.


5 Responses to “Why propose an advanced certification in software testing?”

  1. Dexx says:

    I have to admit that when I read your initial post on certification my first (and pretty much only) reaction was annoyance. Yet another nerd who imagines himself to have discovered a unifying theme to software testing that you can brand, package, and sell.

    Thanks for proving me wrong with the newest addition. For starters, you made it clear that you understand that you’re aiming for better rather than the unattainable perfect. What’s more important is that you emphasize that although the state of testing certificates is abysmal at the moment that doesn’t mean we should condemn certification as a whole. Just because the barrel is full of bad apples doesn’t mean the next batch has to be equally disgusting. Thanks for reminding me of that.

  2. Jim Hazen says:

    Your alternative sounds like a return to the Petition based method that the QAI CSTE was originally, and I can support that. As I mentioned in my previous post I fully support this idea/direction.

    I used to teach Scuba Diving, and amongst the Instructor ranks we used to have a saying about students (and other instructors too), “Certified doesn’t mean qualified”. Meaning that you may have gone through a class and passed, but without some level of other experience (real world application) you’re not ready to go to the next level. I mean I did teach practical skills in the pool and validated them in the ocean before I turned people loose. But also I pushed people to practice their skills by diving and diving often.

    And that is where I see the correlation with Testing; we need to learn the basics and then apply them in real world situations, and then based on that experience apply it to further education to improve and broaden our knowledgebase so that we can turnaround again and apply it. It’s that OJT (on the job training) that also helps. By having a certification like you propose that OJT along with book knowledge can be turned into street smarts, and that helps us to have credibility in our industry.

    I hope it all goes well, I’ll be interested to see it and give it a shot.


    Jim Hazen

  3. Peter says:

    No certification will tell you the true competency of an employee. It’s only real world experience that’ll give you a clue. But a certification is deemed relevant because it’s the only standard way to judge people on a somewhat level playing field.So a certification could make the difference in getting an interview, but it’s the interview and perhaps testing process that will make the difference in getting or keeping the job. That’s why probation periods are also extremely important for determining competency levels.

  4. Chris K says:

    One of the problems I have when thinking about testing certifications is separating why I’d want to take one (I can’t say that I would) versus why I’d like to see the certification system become better.

    My only experience with industrial certification is with teaching Scuba Diving. Realistically you can’t go diving without having a basic scuba certification but you could certainly test or get a job without a testing certification. So what’s the point, what’s the value? Perhaps this is not a fundamental question you intend to cover but it is one of the first questions that come to mind.

    The second part is, would I like to see the current certification system become better? I’m more comfortable answering that question because I think a different ideological view, a more transparent process, and an examination similar to BBST that requires a greater depth of knowledge would be hugely beneficial to community and the certification process as a whole.


    I’ve stopped wondering whether I would like to see a certification system. What seems more important to me is that a few hundred thousand testers have bought into this. Given that it is a significant factor in the community, my goal is to improve it.

    — Cem

  5. […] Why propose an advanced certification in software testing? – Cem Kaner – https://kaner.com/?p=401 […]