A couple of weeks ago, I talked about the state of software testing education (and software testing certification) in the keynote panel at STPCon. My comments on high-volume test automation and qualitative methods were more widely noticed, but I think the educational issues are more significant.
Here is a summary:
- The North American educational systems are in a state of transition.
- We might see a decoupling of formal instruction from credentialing.
- We are likely to see a dispersion of credentialing—-more organizations will issue more diverse credentials.
- Industrial credentials are likely to play a more significant role in the American economy (and probably have an increased or continued-high influence in many other places).
If these four predictions are accurate, then we have thinking to do about the kinds of credentialing available to software testers.
For much of the American population, the traditional university model is financially unsustainable. We are on the verge of a national credit crisis because of the immensity of student loan debt.
As a society, we are experimenting with a diverse set of instructional systems, including:
- MOOCs (massive open online courses)
- Traditionally-structured online courses with an enormous diversity of standards
- Low-cost face-to-face courses (e.g. community colleges)
- Industrial courses that are accepted for university credit
- Traditional face-to-face courses
Across these, we see the full range from easy to hard, from no engagement with the instructor to intense personal engagement, from little student activity and little meaningful feedback to lots of both. There is huge diversity of standards between course structures and institutions and significant diversity within institutions.
- Many courses are essentially self-study. Students learn from a book or a lecturer but they get no significant assignments, feedback or assessments. Many people can learn some topics this way. Some people can learn many topics this way. For most people, this isn’t a complete solution, but it could be a partial one.
- Some of my students prosper most when I give them free rein, friendly feedback and low risk. In an environment that is supportive, provides personalized feedback by a human, but is not demanding, some students will take advantage of the flexibility by doing nothing, some students will get lost, and some students will do their best work.
- The students who don’t do well in a low-demand situation often do better in a higher-demand course, and in my experience, many students need both—-flexibility in fields that capture their imagination and structure/demand in fields that are less engrossing or that a little farther beyond the student’s current knowledge/ability than she can comfortably stretch to.
There is increasing (enormous) political pressure to allow students to take really-inexpensive MOOCs and get course credit for these at more expensive universities. More generally, there is increasing pressure to allow students to transfer courses across institutions. Most universities allow students to transfer in a few courses, but they impose limits in order to ensure that they transfer their culture to their students and to protect their standards. However, I suspect strongly that the traditional limits are about to collapse. The traditional model is financially unsustainable and so, somewhere, somehow, it has to crack. We will see a few reputable universities pressured (or legislated) into accepting many more credits. Once a few do it, others will follow.
In a situation like this, schools will have to find some other way to preserve their standards—-their reputations, and thus the value of their degree for their graduates.
Seems likely to me that some schools will start offering degrees based on students’ performance on exit exams.
- A high-standards institution might give a long and complex set of exams. Imagine paying $15,000 to take the exam series (and get grades and feedback) and another $15,000 if you pass, to get the degree.
- At the other extreme, an institution might offer a suite of multiple-guess exams that can be machine-graded at a much lower cost.
The credibility of the degree would depend on the reputation of the exam (determined by “standards” combined with a bunch of marketing).
Once this system got working, we might see students take a series of courses (from a diverse collection of providers) and then take several degrees.
Maybe things won’t happen this way. But the traditional system is financially unsustainable. Something will have to change, and not just a little.
Decoupling Instruction from Credentialing
The vision above reflects a complete decoupling of instruction from credentialing. It might not be this extreme, but any level of decoupling creates new credentialing pressures / opportunities in industrial settings.
Instruction consists of the courses, the coaching, the internships, and any other activities the students engage in to learn.
Credentials are independently-verifiable evidence that a person has some attribute, such as a skill, a type of knowledge, or a privilege.
There are several types of credentials:
- A certification attests to some level of competency or privilege. For example,
- A license to practice law, or to do plumbing, is a certification.
- An organization might certify a person as competent to repair their equipment.
- An organization might certify that, in their opinion, a person is competent to practice a profession.
- A certificate attests that someone completed an activity
- A certificate of completion of a course is a certificate
- A university degree is a certificate
- There are also formal recognitions (I’m sure there’s a better name for this…)
- Awards from professional societies are recognitions
- Granting someone an advanced type of membership (Senior Member or Fellow) in a professional society is a recognition
- Election to some organizations (such as the American Law Institute or the Royal Academy of Science) is a recognition
- I think I would class medals in this group
- There are peer recognitions
- Think of the nice things people say about you on Linked-In or Entaggle
- There are workproducts or results of work that are seen as honors
- You have published X many publications
- You worked on the development team for X
The primary credentials issued by universities are certificates (degrees). Sometimes, those are also certifications.
Dispersion of Credentialing
Anyone can issue a credential. However, the prestige, credibility, and power of credentials vary enormously.
- If you need a specific credential to practice a profession, then no matter who endorses some other credential, or how nicely named that other credential is, it still won’t entitle you to practice that profession.
- Advertising that you have a specific credential might make you seem more prestigious to some people and less prestigious to other people.
It is already the case that university degrees vary enormously in meaning and prestige. As schools further decouple instruction from degrees, I suspect that this variation will be taken even more seriously. Students of mine from Asia, and some consultants, tell me this is already the case in some Asian countries. Because of the enormous variation in quality among universities, and the large number of universities, a professional certificate or certification is often taken more seriously than a degree from a university that an employer does not know and respect.
How does this relate to software testing? Well, if my analysis is correct (and it might well not be), then we’ll see an increase in the importance and value of credentialing by private organizations (companies, rather than universities).
I don’t believe that we’ll see a universally-accepted credential for software testers. The field is too diverse and the divisions in the field are too deep.
I hope we’ll see several credentialing systems that operate in parallel, reflecting different visions of what people should know, what they should believe, what they should be able to do, what agreements they are willing to make (and be bound by) in terms of professional ethics, and what methods of assessing these things are appropriate and in what depth.
Rather than seeing these as mutually-exclusive competing standards, I imagine that some people will choose to obtain several credentials.
A Few Comments On Our Current State
Software Testing has several types of credentials today. Here are notes on a few. I am intentionally skipping several that feel (to me) redundant with these or about which I have nothing useful to say. My goal is to trigger thoughts, not survey the field.
ISTQB is currently the leading provider of testing certifications in the world. ISTQB is the front end of a community that creates and sells courseware, courses, exams and credentials that align with their vision of the software testing field and the role of education within it. I am not personally fond of the Body of Knowledge that ISTQB bases its exams on. Nor am I fond of their approach to examinations (standardized tests that, to my eyes, emphasize memorization over comprehension and skill). I think they should call their credentials certificates rather than certifications. And my opinion of their marketing efforts is that they are probably not legally actionable, but I think they are misleading. (Apart from those minor flaws, I think ISTQB’s leadership includes many nice people.)
It seems to me that the right way to deal with ISTQB is to treat them as a participant in a marketplace. They sell what they sell. The best way to beat it is to sell something better. Some people are surprised to hear me say that because I have published plenty of criticisms of ISTQB. I think there is lots to criticize. But at some point, adding more criticism is just waste. Or worse, distraction. People are buying ISTQB credentials because they perceive a need. Their perception is often legitimate. If ISTQB is the best credential available to fill their need, they’ll buy it. So, to ISTQB’s critics, I offer this suggestion.
Industrial credentialing will probably get more important, not less important, over the next 20 years. Rather than wasting everyone’s time whining about the shortcomings of current credentials, do the work needed to create a viable alternative.
Before ending my comments on ISTQB, let me note some personal history.
Before ASTQB (American ISTQB) formed, a group of senior people in the community invited me into a series of meetings focused on creating a training-and-credentialing business in the United States. This was a private meeting, so I’m not going to say who sponsored it. The discussion revolved around a goal of providing one or more certification-like credentials for software testers that would be (this is my summary-list, not theirs, but I think it reflects their goals):
- reasonably attainable (people could affort to get the credential, and reasonably smart people who worked hard could earn it),
- credible (intellectually and professionally supported by senior people in the field who have earned good reputations),
- scalable (it is feasible to build an infrastructure to provide the relevant training and assessment to many people), and
- commercially viable (sufficient income to support instructors, maintainers of the courseware and associated documentation, assessors (such as graders of the students and evaluators of the courses), some level of marketing (because a credential that no one knows about isn’t worth much), and in the case of this group, money left over for profit. Note that many dimensions of “commercial viability” come into play even if there is absolutely no profit motive—-the effort has to support itself, somehow).
I think these are reasonable requirements for a strong credential of this kind.
By this point, ISEB (the precursor to ISTQB) had achieved significant commercial success and gained wide acceptance. It was on people’s minds, but the committee gave me plenty of time to speak:
- I talked about multiple-choice exams and why I didn’t like them.
- I talked about the desirability of skill-based exams like Cisco’s, and the challenges of creating courses to support preparation for those types of exams.
- I talked about some of the thinking that some of us had done on how to create a skill-based cert for testers, especially back when we were writing Lessons Learned.
But there was a problem in this. My pals and I had lots of scattered ideas about how to create the kind of certification system that we would like, but we had never figured out how to make it practical. The ideas that I thought were really good were unscalable or too expensive. And we knew it. If you ask today why there is no certification for context-driven testing, you might hear a lot of reasons, including principled-sounding attacks on the whole notion of certification. But back then, the only reason we didn’t have a context-driven certification was that we had no idea how to create one that we could believe in.
So, what I could not provide to the committee was a reasonably attainable, credible, scalable, commercially viable system—-or a plan to create one.
The committee, quite reasonably, chose to seek a practical path toward a credential that they could actually create. I left the committee. I was not party to their later discussions, but I was not surprised that ASTQB formed and some of these folks chose to work with it. I have never forgotten that they gave me every chance to propose an alternative and I did not have a practical alternative to propose.
(Not long after that, I started an alternative project, Open Certification, to see if we could implement some of my ideas. We did a lot of work in that project, but it failed. They really weren’t practical. We learned a lot, which in turn helped me create great courseware—-BBST—-and other ideas about certification that I might talk about more in the future. But the point that I am trying to emphasize here is that the people who founded ASTQB were open to better ideas, but they didn’t get them. I don’t see a reason to be outraged against them for that.)
The Old Boys’ Club
To some degree, your advancement in a profession is not based on what you know. It’s based on who you know and how much they like you.
We have several systems that record who likes like you, including commercial ones (LinkedIn), noncommercial ones (Entaggle), and various types of marketing structures created by individuals or businesses.
There are advantages and disadvantages to systems based on whether the “right” people like you. Networking will never go away, and never should, but it seems to me that
Credentials based on what you know, what you can do, or what you have actually done are a lot more egalitarian than those based on who says they respect you.
I value personal references and referrals, but I think that reliance on these as our main credentialing system is a sure path to cronyism and an enemy of independent thinking.
My impression is that some people in the community have become big fans of reputation-systems as the field’s primary source of credentials. In at least some of the specific cases, I think the individuals would have liked the system a whole lot less when they were less influential.
I distrust any certification or course of study that doesn’t, in some way, actually have a tester demonstrate their skills, or have a chance to defend their reasoning or rationale behind those skills.
In terms of the four criteria that I mentioned above, I think this approach is probably reasonably attainable, and to me, it is definitely credible. Whether it scalable and commercially viable has yet to be seen.
I think this is a clear and important alternative to ISTQB-style credentialing. I hope it is successful.
Other Ideas on the Horizon
There are other ideas on the horizon. I’m aware of a few of them and there are undoubtedly many others.
It is easy to criticize any specific credentialing system. All of them, now known or coming soon, have flaws.
What I am suggesting here is:
- Industrial credentialing is likely to get more important whether you like it or not.
- If you don’t like the current options, complaining won’t do much good. If you want to improve things, create something better.
This post is partially based on work supported by NSF research grant CCLI-0717613 ―Adaptation & Implementation of an Activity-Based Online or Hybrid Course in Software Testing. Any opinions, findings and conclusions or recommendations expressed in this post are those of the author and do not necessarily reflect the views of the National Science Foundation.