This is a draft of a proposal to create a more advanced, more credible credential (certification) in software testing.
The core idea is a certification based on a multidimensional collection of evidence of education, experience, skill and good character.
- I think it is important to develop a credential that is useful and informative.
- I think we damage the reputation of the field if we create a certification that requires only a shallow knowledge of software testing.
- I think we damage the value of the certification if we exaggerate how much knowledge or skill is required to obtain it.
- I think it is important to find a way to tolerate different approaches to software testing, and different approaches to training software testers. This proposal is not based on any one favored “body of knowledge” and it is not tied to any one ideology or group of vendors.
The idea presented here is imperfect—as are the other certifications in our field. It can be gamed—as can the others. Someone who is intent on gaining a credential via cheating and fraud can probably get away with it for a while—but the others have security risks too. This certification does not assure that the certified person is competent—neither do the others. The certification does not subject the certified person to formal professional accountability for their work—neither do the others—and even though certificate holders say that they will follow a code of ethics, we have no mechanism for assuring that they do or punishing them if they don’t—and neither do the others.
With all these we-don’t-do-thises and we-don’t-promise-thats, you might think I’m kidding about this being a real proposal. I’m not.
Even if we agree that this proposed certification lacks the kinds of powers that could be bestowed by law or magic, I think it can provide useful information and that it can create incentives that favor higher ethics in job-seeking and, eventually, professional practice. It is not perfect, but I think it is far better than what we have now.
This credential is based on a collection of several different types of evidence that, taken together, indicate that the certificate holder has the knowledge and skill needed to competently perform the usual services provided by a software tester.
Here are the types of evidence. As you read this, imagine that the Certification Body hosts a website that will permanently post a publicly-viewable dossier (a collection of files) for every person certified by that body. The dossier would include everything submitted by an applicant for certification, plus some additional material. Here’s what we’d find in the file.
Authorization by the Applicant
As part of the application, the applicant for Certification would grant the Certification Board permission to publish all of the following materials. The applicant would also sign a legal waiver that would shield the Board from all types of legal action by the applicant / Certified Tester arising out of publication of the materials described below. The waiver will also authorize the Board to exercise its judgment in assessing the application and will shield the Board from legal action by the applicant if the Board decides, in its unfettered discretion, to reject the applicant’s application or to later cancel the applicant’s Certification.
The Certified Tester should have at least a minimum level of formal education. The baseline that I imagine is a bachelor’s-level degree in a field relevant to software testing.
- Some fields, such as software engineering, are obviously relevant to software testing. But what about others like accounting, mathematics, philosophy, physics, psychology, or technical writing? We would resolve this by requiring the applicant for certification to explain in writing how and why her or his education has proved to be relevant to her or his experiences as a tester and why it should be seen as relevant education for someone in the field.
- The requirement for formal education should be waived if the applicant requests waiver and justifies the request on the basis of a sufficient mix of practical education and professional achievement.
The Certified Tester should have successfully completed a significant amount of practical training in software testing. Most of this training would typically be course-based, typically commercial training. Some academic courses in software testing would also qualify. A non-negotiable requirement is successful completion of at least some courses that are considered advanced. “Successful” completion means that the student completed an exam or capstone project that a student who doesn’t know the material would not pass.
- There is an obvious accreditation issue here. Someone has to decide which courses are suitable and which are advanced.
- I think that many different types of courses and different topics might be suitable as part of the practical training. For example, suppose we required 100 classroom-hours of training (1 training day = 6 classroom hours). Perhaps 60 of those hours could be in related fields (programming, software metrics, software-related law, project accounting, etc.) but a core would have to be explicitly focused on testing.
- I think the advanced course hours (24 classroom hours?) would have to be explicitly advanced software testing courses.
- There is no requirement that these courses come from any particular vendor or that they follow any particular software testing or software development ideology.
The Certified Tester should have successfully completed a proctored, advanced, examination in software testing.
- This requirement anticipates competing exams offered by several different groups that endorse different approaches to software testing. Our field does not have agreement on one approach or even one vocabulary. The appearance of agreement that shows up in industry “standards” is illusory. As a matter of practice (I think, often good practice), the standards are routinely ignored by practitioners. Examinations that adopt or endorse these standards should be welcome but not mandatory.
Which exams are suitable and which are advanced?
There is an obvious accreditation issue here. Someone has to decide which exams are suitable and which are advanced.
I am inclined to tentatively define an advanced exam as one that requires as minimum prerequisites (a) successful completion of a specified prior exam and (b) additional education and experience. For example, ISTQB Foundations would not qualify but an ISTQB Advanced or Expert exam might. Similarly, BBST:Foundations would not qualify but BBST:Bug Advocacy might and BBST:Domain Testing definitely should.
An exam might be separate from a course or it might be a final exam in a sufficiently advanced course.
For an exam to be used by a Certified Tester, the organization that offers and grades the exam must provide the Certification Board with a copy of a sample exam. The organization must attest under penalty of perjury that they believe the sample is fairly representative of the scope and difficulty of the actual current exam. This sample will appear on the Certification Board’s website, and be accessible as a link from the Certified Tester’s dossier. (Thus, the dossier doesn’t show the Certified Tester’s actual exam but it does show an exam that is comparable to the actual one.)
What about the reliability and the validity of the exams?
Let me illustrate the problem with two contrasting examples:
- I think it is fair to characterize ISTQB as an organization that is striving to create highly reliable exams. To achieve this, they are driven toward questions that have unambiguously correct answers. Even in sample essay questions I have seen for the Expert exam, the questions and the expected answers are well-grounded in a published, relatively short, body of knowledge. I think this is a reasonable and respectable approach to assessment and I think that exams written this way should be considered acceptable for this certification.
- The BBST assessment philosophy emphasizes several other principles over reliability. We expect answers to be clearly written, tightly focused on the question that was asked, with a strong logical argument in favor of whatever position the examinee takes in her or his answer, that demonstrates relevant knowledge of the field. We expect a diversity of points of view. I think it gives the examiner greater insight into the creativity and depth of knowledge of the examinee. I think this is also a reasonable and respectable approach to assessment that we should also consider acceptable for this certification.
There is a tradeoff between these approaches. Approaches like ISTQB’s are focused on the reliability of the exam, especially on between-grader reliability. This is an important goal. The BBST exams are not focused on this. For certification purposes, we would expect to improve BBST reliability by using paired grading (two examiners) but this is imperfect. I would not expect the same level of reliability in BBST exams that ISTQB achieves. However, in my view of the assessment of cognitively complex skills, I believe the BBST approach achieves greater validity. Complicating the issue, there are problems in the measurement of both, reliability and validity, of education-related exams.
The difference here is not just a difference of examination style. I believe it reflects a difference in ideology.
Somehow, the Certification Board will have to find a way to accredit some exams as “sufficiently serious” tests of knowledge even though one is obviously more reliable than the other, one is obviously more tightly based on a published body of knowledge than the other, etc.
Somehow, the Certification Board will have to find a way to refuse to accredit some exams even though they have the superficial form of an exam. In general, I suspect that the Certification Board will cast a relatively broad net and that if groups like ASQ and QAI offer advanced exams, those exams will probably qualify. Similarly, I suspect that a final exam in a graduate-level university course that is an “advanced” software testing course (prerequisite being successful completion of an earlier graduate-level course in testing) would qualify.
Professional achievements include publications, honors (such as awards), and other things that indicate that the candidate did something at a professional level.
An applicant for certification does not have to include any professional achievements. However, if the applicant provides them, they will become part of the applicant’s dossier and will be publicly visible.
Some decisions will lie in the discretion of the Certification Board. For example, the Certification Board:
- might or might not accept an applicant’s academic background as sufficiently relevant (or as sufficiently complete)
- might or might not accept an applicant’s training-experience portfolio as sufficient or as containing enough courses that are sufficiently related to software testing
In such cases, the Certification Board will consider the applicant’s professional achievements as additional evidence of the applicant’s knowledge of the field.
The applicant will provide at least three letters of endorsement from other people who have stature in the field. These letters will be public, part of the Certified Tester’s dossier. An endorsement is a statement from a person that, in that person’s opinion, the applicant has the knowledge, skills and character needed to competently provide the services of a professional software tester. The letter should provide additional details that establish that the endorser knows the knowledge, skill and character of the applicant well enough to credibly make an endorsement.
- A person of stature is someone who is experienced in the field and respected. For example, the person might be (this is not a complete list)
- personally known to the Certification Board
- a Certified Tester
- a Senior Member or Distinguished Member or Fellow of ACM, ASQ, or IEEE
- If one of the endorsers withdraws his or her endorsement, that withdrawal will be published in the Certified Tester’s dossier along with the original endorsement (now marked “withdrawn”) and the Certified Tester will be required to get a new endorser.
- If one of the apparent endorsers contacts the Certification Board and asserts that s/he did not write an endorsement for an applicant and that s/he does not endorse the applicant, and if the apparent endorser provides credible proof of identify, that letter will be published in the Certified Tester’s dossier along with the original letter (now marked “disputed”).
The applicant will provide a detailed description of his or her professional history that includes at least N years of relevant experience.
- The applicant must attest that this description is true and not materially incomplete. It will be published as part of the dossier. Potential future employers will be able to check the claims made here against the claims made in the applicant’s application for work with them.
- The descriptions of relevant positions will include descriptions of the applicant’s role(s) and responsibilities, including typical tasks s/he performed in that position
- The applicant’s years of relevant experience and years of formal education will interact: Someone with more formal education that is relevant to the field will be able to become certified with less relevant experience (but never less than K years of experience).
The candidate must engage in professional activities, including ongoing study, to keep the certification.
Code of Ethics
The candidates must agree to abide by a specific Code of Ethics, such as the ACM code. We should foresee this as a prelude to creating an enforcement structure in which a Certified Tester might be censured or certification might be publicly canceled for unethical conduct.
Somehow, we have to form a Certification Board. The Board will have to charge a fee for application because the website, the accrediting activities, evaluation of applications, marketing of the certification, etc., will cost money.
This collection of material does not guarantee competence, but it does present a multidimensional view of the capability of an experienced person in the field. It speaks to a level of education and professional involvement and to the credibility of self-assertions made when someone applies for a job, submits a paper for publication, etc. I think that the public association of the endorser with the people s/he endorses will encourage most possible endorsers to think carefully about who they want to be permanently publicly identified with. I think the existence of the dossier will discourage exaggeration and fraud by the Certified Tester.
It is not perfect, but I think it will be useful, and better than what I think we have now.
This is not a certification of a baseline of competence in the way that certifications (licenses) work in fields like law, engineering, plumbing, and cosmetology. Those are regulated professions in which the certified person is subject to penalties and civil litigation for conduct that falls below baseline. Software engineering (including software testing) is not a regulated profession, there is no such cause of action in the courts as “software engineering malpractice,” and there are no established penalties for incompetence. There is broad disagreement in the field about whether such regulations should exist (for example, the Association for Computing Machinery strongly opposes the licensing of software engineers while the IEEE seems inclined to support it) and the creation of this certification does not address the desirability of such regulation.
The Current Goal: A Constructive Discussion
This article is a call for discussion. It is not yet a call for action, though I expect we’ll get there soon.
This article follows up an article I wrote last May about credentialing systems. I identified several types of credentials in use in our field and suggested four criteria for a better credential:
- reasonably attainable (people could affort to get the credential, and reasonably smart people who worked hard could earn it),
- credible (intellectually and professionally supported by senior people in the field who have earned good reputations),
- scalable (it is feasible to build an infrastructure to provide the relevant training and assessment to many people), and
- commercially viable (sufficient income to support instructors, maintainers of the courseware and associated documentation, assessors (such as graders of the students and evaluators of the courses), some level of marketing (because a credential that no one knows about isn’t worth much), and in the case of this group, money left over for profit. Note that many dimensions of “commercial viability” come into play even if there is absolutely no profit motive—-the effort has to support itself, somehow).
I think the proposal in this article sketches a system that would meet those criteria.
A more detailed draft of this proposal was reviewed at the 2014 Workshop on Teaching Software Testing. We did not debate alternative proposals or attempt to reach consensus. The ideas in this paper are not the product of WTST. Nor are they the responsibility of any participant at WTST. However, I am here acknowledging the feedback I got at that meeting and thanking the participants: Scott Allman, Janaka Balasooriya, Rex Black, Jennifer Brock, Reetika Datta, Casey Doran, Rebecca L. Fiedler, Scott Fuller, Keith Gallagher, Dan Gold, Douglas Hoffman, Nawwar Kabbani, Chris Kenst, Michael Larsen, Jacek Okrojek, Carol Oliver, Rob Sabourin, Mike Sowers, and Andy Tinkham. Payson Hall has also questioned the reasoning and offered useful suggestions.
To this point, we have been discussing whether these ideas are worthwhile in principle. That’s important and that discussion should continue.
We have not yet begun to tackle the governance and implementation issues raised by this proposal. It is probably time to start thinking about that.
- I’m positively impressed by (what I know of) the governance model of ISTQB and wonder whether we should follow that model.
- I would expect to be an active supporter/contributor to the governance of this project (for example an active member of the governing Board). However—just as I helped found AST but steadfastly refused to run for President of AST—I believe we can find a better choice than me for chief executive of the project.