Credentialing in Software Testing: Elaborating on my STPCon Keynote

A couple of weeks ago, I talked about the state of software testing education (and software testing certification) in the keynote panel at STPCon. My comments on high-volume test automation and qualitative methods were more widely noticed, but I think the educational issues are more significant.

Here is a summary:

  1. The North American educational systems are in a state of transition.
  2. We might see a decoupling of formal instruction from credentialing.
  3. We are likely to see a dispersion of credentialing—-more organizations will issue more diverse credentials.
  4. Industrial credentials are likely to play a more significant role in the American economy (and probably have an increased or continued-high influence in many other places).

If these four predictions are accurate, then we have thinking to do about the kinds of credentialing available to software testers.


For much of the American population, the traditional university model is financially unsustainable. We are on the verge of a national credit crisis because of the immensity of student loan debt.

As a society, we are experimenting with a diverse set of instructional systems, including:

  • MOOCs (massive open online courses)
  • Traditionally-structured online courses with an enormous diversity of standards
  • Low-cost face-to-face courses (e.g. community colleges)
  • Industrial courses that are accepted for university credit
  • Traditional face-to-face courses

Across these, we see the full range from easy to hard, from no engagement with the instructor to intense personal engagement, from little student activity and little meaningful feedback to lots of both. There is huge diversity of standards between course structures and institutions and significant diversity within institutions.

  • Many courses are essentially self-study. Students learn from a book or a lecturer but they get no significant assignments, feedback or assessments. Many people can learn some topics this way. Some people can learn many topics this way. For most people, this isn’t a complete solution, but it could be a partial one.
  • Some of my students prosper most when I give them free rein, friendly feedback and low risk. In an environment that is supportive, provides personalized feedback by a human, but is not demanding, some students will take advantage of the flexibility by doing nothing, some students will get lost, and some students will do their best work.
  • The students who don’t do well in a low-demand situation often do better in a higher-demand course, and in my experience, many students need both—-flexibility in fields that capture their imagination and structure/demand in fields that are less engrossing or that a little farther beyond the student’s current knowledge/ability than she can comfortably stretch to.

There is increasing (enormous) political pressure to allow students to take really-inexpensive MOOCs and get course credit for these at more expensive universities. More generally, there is increasing pressure to allow students to transfer courses across institutions. Most universities allow students to transfer in a few courses, but they impose limits in order to ensure that they transfer their culture to their students and to protect their standards. However, I suspect strongly that the traditional limits are about to collapse. The traditional model is financially unsustainable and so, somewhere, somehow, it has to crack. We will see a few reputable universities pressured (or legislated) into accepting many more credits. Once a few do it, others will follow.

In a situation like this, schools will have to find some other way to preserve their standards—-their reputations, and thus the value of their degree for their graduates.

Seems likely to me that some schools will start offering degrees based on students’ performance on exit exams.

  • A high-standards institution might give a long and complex set of exams. Imagine paying $15,000 to take the exam series (and get grades and feedback) and another $15,000 if you pass, to get the degree.
  • At the other extreme, an institution might offer a suite of multiple-guess exams that can be machine-graded at a much lower cost.

The credibility of the degree would depend on the reputation of the exam (determined by “standards” combined with a bunch of marketing).

Once this system got working, we might see students take a series of courses (from a diverse collection of providers) and then take several degrees.

Maybe things won’t happen this way. But the traditional system is financially unsustainable. Something will have to change, and not just a little.

Decoupling Instruction from Credentialing

The vision above reflects a complete decoupling of instruction from credentialing. It might not be this extreme, but any level of decoupling creates new credentialing pressures / opportunities in industrial settings.


Instruction consists of the courses, the coaching, the internships, and any other activities the students engage in to learn.


Credentials are independently-verifiable evidence that a person has some attribute, such as a skill, a type of knowledge, or a privilege.

There are several types of credentials:

  • A certification attests to some level of competency or privilege. For example,
    • A license to practice law, or to do plumbing, is a certification.
    • An organization might certify a person as competent to repair their equipment.
    • An organization might certify that, in their opinion, a person is competent to practice a profession.
  • A certificate attests that someone completed an activity
    • A certificate of completion of a course is a certificate
    • A university degree is a certificate
  • There are also formal recognitions (I’m sure there’s a better name for this…)
    • Awards from professional societies are recognitions
    • Granting someone an advanced type of membership (Senior Member or Fellow) in a professional society is a recognition
    • Election to some organizations (such as the American Law Institute or the Royal Academy of Science) is a recognition
    • I think I would class medals in this group
  • There are peer recognitions
    • Think of the nice things people say about you on Linked-In or Entaggle
  • There are workproducts or results of work that are seen as honors
    • You have published X many publications
    • You worked on the development team for X

The primary credentials issued by universities are certificates (degrees). Sometimes, those are also certifications.

Dispersion of Credentialing

Anyone can issue a credential. However, the prestige, credibility, and power of credentials vary enormously.

  • If you need a specific credential to practice a profession, then no matter who endorses some other credential, or how nicely named that other credential is, it still won’t entitle you to practice that profession.
  • Advertising that you have a specific credential might make you seem more prestigious to some people and less prestigious to other people.

It is already the case that university degrees vary enormously in meaning and prestige. As schools further decouple instruction from degrees, I suspect that this variation will be taken even more seriously. Students of mine from Asia, and some consultants, tell me this is already the case in some Asian countries. Because of the enormous variation in quality among universities, and the large number of universities, a professional certificate or certification is often taken more seriously than a degree from a university that an employer does not know and respect.

Industrial Credentials

How does this relate to software testing? Well, if my analysis is correct (and it might well not be), then we’ll see an increase in the importance and value of credentialing by private organizations (companies, rather than universities).

I don’t believe that we’ll see a universally-accepted credential for software testers. The field is too diverse and the divisions in the field are too deep.

I hope we’ll see several credentialing systems that operate in parallel, reflecting different visions of what people should know, what they should believe, what they should be able to do, what agreements they are willing to make (and be bound by) in terms of professional ethics, and what methods of assessing these things are appropriate and in what depth.

Rather than seeing these as mutually-exclusive competing standards, I imagine that some people will choose to obtain several credentials.

A Few Comments On Our Current State

Software Testing has several types of credentials today. Here are notes on a few. I am intentionally skipping several that feel (to me) redundant with these or about which I have nothing useful to say. My goal is to trigger thoughts, not survey the field.


ISTQB is currently the leading provider of testing certifications in the world. ISTQB is the front end of a community that creates and sells courseware, courses, exams and credentials that align with their vision of the software testing field and the role of education within it. I am not personally fond of the Body of Knowledge that ISTQB bases its exams on. Nor am I fond of their approach to examinations (standardized tests that, to my eyes, emphasize memorization over comprehension and skill). I think they should call their credentials certificates rather than certifications. And my opinion of their marketing efforts is that they are probably not legally actionable, but I think they are misleading. (Apart from those minor flaws, I think ISTQB’s leadership includes many nice people.)

It seems to me that the right way to deal with ISTQB is to treat them as a participant in a marketplace. They sell what they sell. The best way to beat it is to sell something better. Some people are surprised to hear me say that because I have published plenty of criticisms of ISTQB. I think there is lots to criticize. But at some point, adding more criticism is just waste. Or worse, distraction. People are buying ISTQB credentials because they perceive a need. Their perception is often legitimate. If ISTQB is the best credential available to fill their need, they’ll buy it. So, to ISTQB’s critics, I offer this suggestion.

Industrial credentialing will probably get more important, not less important, over the next 20 years. Rather than wasting everyone’s time whining about the shortcomings of current credentials, do the work needed to create a viable alternative.

Before ending my comments on ISTQB, let me note some personal history.

Before ASTQB (American ISTQB) formed, a group of senior people in the community invited me into a series of meetings focused on creating a training-and-credentialing business in the United States. This was a private meeting, so I’m not going to say who sponsored it. The discussion revolved around a goal of providing one or more certification-like credentials for software testers that would be (this is my summary-list, not theirs, but I think it reflects their goals):

  • reasonably attainable (people could affort to get the credential, and reasonably smart people who worked hard could earn it),
  • credible (intellectually and professionally supported by senior people in the field who have earned good reputations),
  • scalable (it is feasible to build an infrastructure to provide the relevant training and assessment to many people), and
  • commercially viable (sufficient income to support instructors, maintainers of the courseware and associated documentation, assessors (such as graders of the students and evaluators of the courses), some level of marketing (because a credential that no one knows about isn’t worth much), and in the case of this group, money left over for profit. Note that many dimensions of “commercial viability” come into play even if there is absolutely no profit motive—-the effort has to support itself, somehow).

I think these are reasonable requirements for a strong credential of this kind.

By this point, ISEB (the precursor to ISTQB) had achieved significant commercial success and gained wide acceptance. It was on people’s minds, but the committee gave me plenty of time to speak:

  • I talked about multiple-choice exams and why I didn’t like them.
  • I talked about the desirability of skill-based exams like Cisco’s, and the challenges of creating courses to support preparation for those types of exams.
  • I talked about some of the thinking that some of us had done on how to create a skill-based cert for testers, especially back when we were writing Lessons Learned.

But there was a problem in this. My pals and I had lots of scattered ideas about how to create the kind of certification system that we would like, but we had never figured out how to make it practical. The ideas that I thought were really good were unscalable or too expensive. And we knew it. If you ask today why there is no certification for context-driven testing, you might hear a lot of reasons, including principled-sounding attacks on the whole notion of certification. But back then, the only reason we didn’t have a context-driven certification was that we had no idea how to create one that we could believe in.

So, what I could not provide to the committee was a reasonably attainable, credible, scalable, commercially viable system—-or a plan to create one.

The committee, quite reasonably, chose to seek a practical path toward a credential that they could actually create. I left the committee. I was not party to their later discussions, but I was not surprised that ASTQB formed and some of these folks chose to work with it. I have never forgotten that they gave me every chance to propose an alternative and I did not have a practical alternative to propose.

(Not long after that, I started an alternative project, Open Certification, to see if we could implement some of my ideas. We did a lot of work in that project, but it failed. They really weren’t practical. We learned a lot, which in turn helped me create great courseware—-BBST—-and other ideas about certification that I might talk about more in the future. But the point that I am trying to emphasize here is that the people who founded ASTQB were open to better ideas, but they didn’t get them. I don’t see a reason to be outraged against them for that.)

The Old Boys’ Club

To some degree, your advancement in a profession is not based on what you know. It’s based on who you know and how much they like you.

We have several systems that record who likes like you, including commercial ones (LinkedIn), noncommercial ones (Entaggle), and various types of marketing structures created by individuals or businesses.

There are advantages and disadvantages to systems based on whether the “right” people like you. Networking will never go away, and never should, but it seems to me that

Credentials based on what you know, what you can do, or what you have actually done are a lot more egalitarian than those based on who says they respect you.

I value personal references and referrals, but I think that reliance on these as our main credentialing system is a sure path to cronyism and an enemy of independent thinking.

My impression is that some people in the community have become big fans of reputation-systems as the field’s primary source of credentials. In at least some of the specific cases, I think the individuals would have liked the system a whole lot less when they were less influential.


I’ve been delighted to see that the Miagi-do school has finally come public.

Michael Larsen states a key view succinctly:

I distrust any certification or course of study that doesn’t, in some way, actually have a tester demonstrate their skills, or have a chance to defend their reasoning or rationale behind those skills.

In terms of the four criteria that I mentioned above, I think this approach is probably reasonably attainable, and to me, it is definitely credible. Whether it scalable and commercially viable has yet to be seen.

I think this is a clear and important alternative to ISTQB-style credentialing. I hope it is successful.

Other Ideas on the Horizon

There are other ideas on the horizon. I’m aware of a few of them and there are undoubtedly many others.

It is easy to criticize any specific credentialing system. All of them, now known or coming soon, have flaws.

What I am suggesting here is:

  • Industrial credentialing is likely to get more important whether you like it or not.
  • If you don’t like the current options, complaining won’t do much good. If you want to improve things, create something better.

This post is partially based on work supported by NSF research grant CCLI-0717613 ―Adaptation & Implementation of an Activity-Based Online or Hybrid Course in Software Testing. Any opinions, findings and conclusions or recommendations expressed in this post are those of the author and do not necessarily reflect the views of the National Science Foundation.

12 Responses to “Credentialing in Software Testing: Elaborating on my STPCon Keynote”

  1. As for Miagi-Do I would like to add that we don’t follow any commercial aspect with it right now. If it becomes ever a crucial alternative to ISTQB in the heads of the hiring people around, with public courses, scaling up to 100 people, I am sure the idea and essence of Miagi-Do will be dead, and the name will no longer mean what it means right now. In fact, I think our FAQ states that we will close the thing then. We throw in a lot of our private time right now, and we don’t sell courses, courseware or do marketing. That said, I don’t see the commercially viable criteria coming up for Miagi-Do school.

    We are working on the scalable thing, though. 🙂

    I understand what you are saying about commercial viability. But even for non-commercial activities, there is the question of how the activity sustains itself over time. This is a problem for open source software, for example. We explored the problem (and summarized years of thinking / discussions) for BBST in this paper:
    Part of the challenge of sustaining the activity is economic. The plan for the economic part is the plan for commercial viability.

    — cem

  2. Matt says:

    I read Michael’s explanation of Miagi-do, and what caught my attention is his references to Aikido.

    I’ve been practicing Aikido for about 5 years now, have observed quite a few gradings and been through a few myself. I like the idea of this kind of credentialling, as long as the people running the tests are themselves qualified. There is some risk of the cronyism you mentioned in “The Old Boys Club”, unless those who are executing the tests are aware of their biases and actively manage them. As well, there can be some perception of cronyism from those who aren’t familiar with the martial arts model that Miagi-do is using (ie. that testing is primarily demonstrative and based on merit).

    I don’t think that it will be commercially viable, but I’m not sure that’s a bad thing necessarily. Many Aikido dojos are not-for-profit, their instructors are volunteers. This stems from a philosophy that those who have been taught have a responsibility to pass on that teaching. When a dojo is run as a for-profit business, the conflict between the need to make money and the need to make sure that people you “pass” meet real criteria can be significant. Keeping the Miagi-do school as a strictly non-commercial venture means it is more likely to focus on the credibility of their “certification” than on raking in the money.

    As for scalability, I think it depends on how you define scalable. Can it be grown to a large organization, and maintain at least some of it integrity? I think the answer is yes. Can it be done quickly? Probably not, if for no other reason than it takes time to develop a person in to a “black belt”, whether in a martial art or Miagi-do testing.

  3. Cem, thank you very much for mentioning Miagi-do in this post, and capturing the spirit of what we have intended. Matt and Markus, thanks for your comments, and I agree, I think that a lot of what we offer is more than a “credentialing”, we are looking to be mentors and help develop others who want to make a difference in the software testing world. I also agree that I would much rather be part of the solution than just complain about the deficiencies.

    We are a grassroots movement, and by design, we are currently small. If, however, we can help develop those individuals over time with demonstrable skills and also a desire to “pay it forward” for others, then yes, I see the potential of Miagi-do scaling well. It will certainly take time, but if we can somehow shape the narrative or get organizations to change their minds about their approach, I’ll consider it time well spent. It also, so far, looks to be time spent with some excellent company :).

  4. […] Credentialing in Software Testing: Elaborating on my STPCon Keynote Written by: Cem Kaner […]

  5. I think there is another and more likely future, one where credentials are rendered obsolete by portfolios.

    The rise of distributed version control systems (i.e. Git) and corresponding free, social, online code repository services that support them, has given programmers instant, publicly available portfolios that exhibit their code with an emphasis on peer review e.g. accepted pull requests, project followers etc. Such portfolios have replaced programmer credentials in my corner of the world and this practice is reportedly spreading fast:

    Given their utility, accuracy and growing popularity for hiring programmers, I’d like to see (and expect others will too) similar portfolios spreading to other roles in software development, especially testers. As these online repository services include issue trackers & wikis, testers’ output, whether it be test plans, exploratory testing notes, automated tests, code reviews, issues raised etc. are already catered for.

    Given any tester today, irrespective of their location, experience, education or finances, can now, without invitation or permission, copy (fork/clone) an interesting/innovative/high-profile open source project, test it to the best of their abilities, easily store the artefacts of their work online for all to see, as well as submit their work back to the project via peer review…all at no financial cost!…it’s hard to imagine increasing demand for industry credentials, especially from an increasingly fragmented group of suppliers, when it’s now possible and much simpler to just look at a person’s body of work instead.

    • My apologies for a partially off-topic reply, as far as the original post is concerned, since this only touches the original subject indirectly.

      There is a pretty crucial problem with tester portfolios, as I see it:

      Future generations are able to see the final product say, Mona Lisa, long after it was completed, but what they can’t see is the work and input of the people who helped the artist finalize the painting to the way it is now.

      How this relates to testing is simple: Code is visible, applications are visible, well-functioning websites, online stores, insurance services, hotel reservation systems, flight-control systems and x-ray machines are all final products but they do not show the work of the testers – they only show the end result after the input testers have given to the developers has helped them fix errors and finalize the product.

      So my question is: How would you build a tester portfolio that clearly shows the work you did, as a tester? Simply saying “I tested this” is no good in my view, because code can be altered AFTER it was tested and, all of a sudden, your brilliant testing work is wiped clean with a new, buggy version. Similarly, I could simply lie in my LinkedIn profile (which is one kind of a portfolio of the work I’ve done) and 99.99% of the world wouldn’t know any better.

      Please, don’t get me wrong here: I don’t think it’s a bad idea, I just think it might be difficult to make it work – at least in a credible way.

      Let me start by agreeing with you. Much of my best testing work would not show up cleanly in a portfolio. First, most of the work was confidential and so I could not show it anyway. But even if I decided to make a significant investment of unpaid labor in an open source project for the purpose of creating an advertisement that I could call a “portfolio”, much of my work would not be code. It would be investigation of the market and of the situation of the development group, analysis of the product, and multidimensional evaluation of the design. To show these in a “portfolio” entry, I would have to write a book. So, I don’t think the idea of a portfolio is workable as a One-True-Way for credentialing.

      On the other hand, building a collection of examples of what you have done is a useful way to show some of your skills and some of your knowledge and some of your experience. You might do this with code samples, test design documents, articles, books, presentations, etc.

      In my case, the published examples still don’t reach to things that I’ve been extensively trained in but haven’t yet used in a publishable way. So a portfolio would give a useful but incomplete picture of what I know and what I can do.

      In addition, some types of credentials carry responsibility. For example, as an attorney, I am bound by a Code of Ethics. Violation of that code can entitle people to sue me or prosecute me (or violating California’s Business & Professions Code) or discipline me (for violating rules of conduct as a California attorney–the discipline can include a fine of any (unlimited) amount of money.) Demonstrations of what I can do don’t say anything about the nature of responsibilities that I have undertaken.

      On it’s own, a demonstration-of-work portfolio is probably a bad idea for testing (and for most other fields, including software development). But a collection of work products, as part of a broader presentation that includes other types of credentials, is probably useful.

      — cem kaner

  6. […] stay out of it this time.  Two things tipped my hand: (1) Miagi-Do went public, and (2) Dr. Kaner mentioned us. In a blog post. On credentialing in software […]

  7. Nischal says:

    it was mentioned that a credentialing system should allow the Software Testers to defend their reasoning and rationale behind their skills, what “skills” are we talking about?

    Is there a specific skill set that is needed for a Software Tester? Can they be quantified or defined qualitatively?

    If this can be determined and agreed upon globally, then maybe there can be a viable credentialing system.

    Is that what the current “Certifications” are doing?

    There is no globally-recognized set of skills for software testing. Nor should there be. A tester working on the accuracy of a complex financial system would need very different skills from a tester working on the playability of a real-time game.

    But, so what? Rather than looking for The One True Credential, we can accept the idea that different credentials tell us about different things. One credential might be more relevant for one job. Another credential might be more relevant for another. Some combinations of credentials might illustrate an ability to cross traditional boundaries.

    It’s much the same in other fields. My doctorate in Psychology came from a university that is widely respected for empirical methods. My work involved significant amounts of mathematical and physiological modeling and hands-on mathematical and physiological research. My brother’s doctorate in Psychology came from a university focused on clinical skills. He was one of the early advocates of analyzing organizational dynamics in the same ways we analyze family dynamics. We both have Ph.D.’s in Psychology, we have both had impacts on our field that directly applied the knowledge and skills we learned and practiced in school, but neither of us would be competent doing the other’s work. Does that make our credentials bad? No. It makes them different.

    We live in a complex world. It won’t change itself into a less complex, less diverse, less interesting place. Rather than demanding a single, standardized solution, I prefer to embrace the world’s diversity.

    — cem kaner

    • Nischal says:

      I completely agree with what you said about the diversity in Software Testing AND with the notion that different does not mean bad. There can be certificates for specific skills but there can’t be a true certification for Software Testing (unless there is a Grand Unified Theory that explains everything including certification for testing!).

      I’m sorry, but I think you’re saying a silly thing. In the two professions that I know very well — Psychology (where I have a Ph.D.) and Law (where I have a J.D. and a license to practice law in California) — people are certified every day even though neither field has a Grand Unified Theory. And probably, neither field will ever have one.

      Last night, I read some fascinating articles on the difficulties economists face in coming up with a definition of “money”. For example, see “Money” is as basic a concept to economists as “test case” is to testers.

      People can be an experts in a field even though the field is so diverse that widely-recognized experts can have fundamental disagreements. To say that there can be no type of certification for a field’s experts is, to my way of thinking, a silly thing.

      It is easy to propose impossible criteria. But in the end, I think that such criteria are not useful. (And for criteria that demand unification or standardization of the field, I think they are also undesirable.)

      — cem kaner

      • Nischal says:

        Thank you for your time on clarifying this for me.

        Oh and the article about money… fascinating indeed! Thank you for sharing!

        – Nischal

  8. […] level perspective and voice of reason in the increasingly heated debate on testing certification. Leave it to testing dean Cem Kaner to put the entire discussion into […]

  9. Oliver Erlewein says:

    Hi Cem,
    I’ve been interviewing dozens of testers over the last couple of years. My stance on what I want from a tester has changed significantly over this time. I have pretty much arrived at a similar conclusion. There is no practical way to make a useful testing certification. That is the main reason for me condemning ISTQB. They cannot “win”. It’s not a matter of improving them but a fact that there is no such thing. And as you point out, what do you do instead?

    When interviewing for testers and judging their abilities and skills I try to see how they deal with situations rather than what they actually answer or not. How clever are their problem solving tactics? Do they have methodologies they use? Are they applying them consciously or unconsciously? Can they listen to what I am saying and focus on what I actually want? Can they notice that there is a difference in what I say and what I give them to do in written form (intentional)? Can they convince me that they know what testing is?

    All those things are specific and they somehow aren’t. Basically what I am looking for is a fitting canvas and not a painting. I am fully aware that the painting will emerge from what I will train the person to do and what they will learn in the context of the work they do. The odd thing is, that those people that actively promote their ISTQB are also the people I am usually the least impressed with. To my surprise that also holds true when they have years of experience. There has so far only been one memorable exception and I’d rather attribute that to the actual person. ISTQB somehow seems to manage to ruin the canvas.

    The “Old Boy’sClub” is unfair and tainted and whatever but it does put people on my radar, that I do like to employ. Sure, there will be people that I’d need that are outside of that club. So it isn’t perfect or PC but it does get the job done better than most other systems. So for the time being, until we have a better, bright idea why not work on that? I think conferences and workshops do that quite well.

    I think the main point is, that certification, credentials and the such are closer to an impossibility than they are to a hard task. I respect the people that try. But I absolutely loathe people that give you the notion that it’s easy.

    Cheers Oliver


    When you say that credentials are closer to an impossibility than a hard task, I think that the same thing could be said of psychology, medicine, and law. They are cognitively complex fields, subject to great internal controversies. But there are many useful credentials in these fields.

    When you say that ISTQB training “ruins the canvas”, you are saying that you see a distinctive effect of ISTQB training and certification. You might not like that effect, but you seem to be able to recognize it. Someone who has a strong opinion about the approaches and attitudes that you think are common to ISTQB-certified testers would find it useful to know whether an applicant is ISTQB-certified. The one who likes the ISTQB approach/attitude would favor ISTQB-certified applicants. Someone who finds ISTQB repulsive might avoid hiring someone with that credential.

    Here’s another influence-of-training example. I worked for a brief time as a prosecutor. Working with other attorneys, I could tell, often quickly, whether they had graduated from a law school that was accredited by the American Bar Association. Students from some of the non-accredited schools in Northern California were sometimes able to pass the Bar Exam and become lawyers but they had different attitudes, skills, and vulnerabilities. Different types of training left their mark.

    I think that in software testing, we have distinctive educational approaches. I think it can be useful to understand how well someone has learned the approach(es) they studied, including how well they can apply what they learned.