Rebecca Fiedler and I have just completed a major round of updates to BBST, the Black Box Software Testing course. This creates what we consider a stable release, which we expect to be the final release of BBST Version 3. I am starting to work on Version 4 (initial discussion post: next week).

The course videos are unchanged. The home page for BBST (with the best course descriptions) is still http://bbst.info. Foundations, Bug Advocacy, and Test Design are still available for free at http://testingeducation.org/BBST along with course slides and other supporting materials.

These are the videos that the Association for Software Testing helped develop, and teaches from. I will probably continue to use these in the Software Engineering degree programs at Florida Tech for a few years. For now, we teach from these videos in private courses for commercial clients, through Kaner Fiedler Associates and in conjunction with Altom, which offers the latest updates to our courses to the public.

My course version identifier will stay at Version 3. (Foundations (2010) offers the third generation of Foundations videos, Bug Advocacy (2008) videos are second generation, Test Design (2011) are mixed, second and third. Collectively: Version 3).

Here’s what’s new:

(1) We published course workbooks for each of the three courses. The workbooks provide:

• An edited transcript of each lecture
• Updated (or completely replaced) activities
• Detailed feedback for most activities, based on patterns in student work that we’ve seen over the 12 years since the first BBST course videos.
• Often-detailed commentary on the lectures and activities. Our primary focus in these notes is the instructional effectiveness of the materials: treat the intended content and skill development as given. How well does the courseware support them? Occasionally, we felt compelled to acknowledge some other issues in the field that we decided were unignorably relevant to the courses.

The workbooks are:

• Foundations of Software Testing: A BBST Workbook
• Bug Advocacy: A BBST Workbook
• Test Design: A BBST Workbook

I am making the final publishing-support revisions for Test Design this week. It should be available at Amazon before the end of July.

These books run 230-to-400 pages long. They took a lot of work.

(2) We have overhauled the multiple-choice review questions in all three courses.

These were a source of some irritation in BBST. The new questions are still hard, but student feedback is that they are much better.

(3) We revised our model for instructor-student feedback.

When we planned Version 3, one of our key concerns was that instructors were volunteers. There is only so much work you can give to volunteers before they burn out. We designed the course-interaction model to have as much peer review as possible. This would give students feedback on the quality of their work while keeping the instructor workload under control. Overall, that model has worked pretty well.

• Feedback quality is, of course, variable. The feedback comes from people who don’t necessarily understand the material themselves. They are learning as much from giving the feedback as the students who receive it.
• We had some problems with bullying from some students who seemed more motivated to insist on their doctrinaire views than to learn anything new. This was a nuisance to manage when it arose, but I think most instructors figured out how to handle it well enough.
• Students who needed additional personal feedback often received private feedback via instructor emails or Skypes. My role as an instructional coach for AST completed a few years ago, but my sense is that AST’s instructors are pretty attentive to this need.

The same concerns for volunteer-taught courses apply to large-enrollment courses in universities and colleges. Detailed personalized feedback is just not an instructional option.

However, students’ expectations are appropriately different when they attend commercial versions of the course (paying the usual rates for commercial instruction) or for small-enrollment university courses. They expect better feedback.

Starting in about 2011, Professor Keith Gallagher helped me adapt his ideas on interactive grading to my courses. An interactive grading session is a one-on-one meeting, student with instructor, that lasts 1-to-2 hours (occasionally, 3 hours, ouch) and is focused on a specific exam or major assignment. Usually the focus is on the course’s key piece of work. The intent of the session is coaching, identifying ways the student can improve their work and praising the strengths. Some sessions, of course, are more difficult. In the commercial BBST courses,

• We offer one interactive grading session per student. There are usually two instructors (sometimes three) in the commercial courses. The student typically picks which instructor to meet with.
• We draw student attention to the feedback available on assignments in the course Workbook. For assignments that are not well-covered in the Workbook, the instructor writes a detailed class-wide feedback. We provide personalized feedback on aspects of the student’s work that are not covered in the general feedback.
• The instructor provides summary grading for each piece of work (exceptional, acceptable, disappointing, not-submitted or no-value). (Most work is graded “acceptable.”)

I don’t think this level of feedback is sustainable in a volunteer-taught course. It is more easily sustainable if the instructor uses the Workbook as a course text and can rely on the students having their own copy. But it is still more work than I would ever want to assign to volunteers.

If you are teaching or taking BBST, we suggest that you use the Workbooks

The Workbooks are not Creative Commons licensed. They are not free, but they are inexpensive. When we originally planned these, we intended to publish through commercial publishers, but the prices they planned for the Workbooks were way too high. We created Context-Driven Press so that we could control the quality and price of the books. You can get the books from Amazon for less than $25 each (much less, for Kindle editions).

If you are teaching a BBST course, we strongly recommend that you use the course Workbook as a course text. It changes the student experience, and the feedback we continually get is that the change is strongly positive.

I think these updates mark the completion of this version of BBST.

Online BBST started in 2004. It reflects my understanding of the industry circa 2001 (when we published Lessons Learned in Software Testing.) BBST Version 3 presents the same ideas. The polish from Versions 1 through 3 came more in the instructional design than in the perspective. I am proud of Version 3, and I think it will provide a useful foundation for several years to come. However, our field’s methods, technologies, and social structures have evolved. I think it is time for some bigger changes in what I teach.

There is a fourth course in the BBST series, BBST-Domain Testing. This is currently available only through Kaner Fiedler Associates, though I suspect Altom will offer it soon. The Domain Testing course includes the core sequence of videos, but also includes a set of supplementary videos that have demonstrations and supporting discussions. We spent more time on the supplementary videos than on the core sequence. In BBST Version 4, we intend to exploit some of the opportunities that supplementary videos provide.

I’ll be posting some notes on (Rebecca’s and my ideas for) Foundations 4.0, and asking for your help in planning the update. Those will come in about a week.

We’re putting the finishing touches on BBST Domain Testing before the online pilot starts June 22. We have a handful of seats still available. (Sorry, the class is now full and we have had to close registration CK — 5/23/2014)

Apply to participate here

If you are selected, we will require a $50 non-refundable deposit to defray the cost of hosting the class. You will also need The Domain Testing Workbook (either print or electronic format available at contextdrivenpress.com or Amazon) to use in the class.

You can read Chris Kenst’s report on his experience in January’s face-to-face pilot at http://www.testingcircus.com/?wpdmact=process&did=Mi5ob3RsaW5r

As with all the BBST courses, this course comes with lots of homework. You should expect to work 10-15 hours per week on the course.

If you have questions, please contact us at info@bbst.info.

With our first teaching of the new BBST:Domain Testing course (based on The Domain Testing Workbook) and our revision of BBST:Foundations with the Foundations of Software Testing workbook, Rebecca Fiedler and I have started to introduce the next generation of BBST. Recently, we’ve been getting requests for papers or interviews on where BBST came from and where it’s going.

• This note is a short summary of the history of BBST. You can find many more details in the articles I’ve been posting to this blog over the last decade, as I tried to think through the course design’s strengths, weaknesses and potential.
• My next post, and an upcoming article in Testing Circus, look at the road ahead.

What is BBST™ ?

BBST is a series of courses on Black Box Software Testing. The overall goal of the series is to improve the state of the practice in software testing by helping testers develop useful testing skills and deeper insights into the challenges of the field.

(Note: BBST is a registered trademark of Kaner, Fiedler & Associates.)

Today’s BBST Courses

Today, most people familiar with the BBST courses think of a four-week, fully online course. Rebecca Fiedler and I started working on the instructional design for the BBST online courses back in 2004, with funding from the National Science Foundation. The courses have gotten excellent reviews. We’ve taken Foundations through three major revisions. Bug Advocacy and Test Design have had two. We’re working on our next major update now. You’ll read more about that in my next post.

The typical instructor-led course is organized around six lectures (about six hours of talk, divided into one-hour parts), with a collection of activities. To successfully complete a typical instructor-led course, a student spends about 12-15 hours per week for 4 weeks (48-60 hours total). Most of the course time is spent on the activities:

• Orientation activities introduce students to a key challenge considered in a lecture. The student puzzles through the activity for 30 to 90 minutes, typically before watching the lecture, then sees how the lecture approaches this type of problem. The typical Foundations course has two to four of these.
• Application activities call for two to six hours of work. It applies ideas or techniques presented in a lecture or developed over several lectures. The typical Foundations course has one to two of these.
• Multiple-choice quizzes help students identify gaps in their knowledge or understanding of key concepts in the course. These questions are tough because they are designed to be instructional or diagnostic (to teach you something, to deepen your knowledge of something, or to help you recognize that you don’t understand something) rather than to fairly grade you.
• Various other discussions that help the students get to know each other better, chew on the course’s multiple-choice quiz questions, or consider other topics of current interest.
• An essay-style final exam.

In the instructor-led course, students get feedback on the quality of their work from each other and, to a lesser or greater degree (depends on who’s teaching the course), they get feedback from the instructors. Students in our commercial courses (which we offer through Kaner, Fiedler & Associates) get a lot of feedback. Students in courses taught by unpaid volunteer instructors are more likely to get most of their feedback from the other students.

So, that’s today (in the online course).

However, BBST has actually been around for 20 years.

Background on the BBST Course Design

I started teaching BBST in 1994, with Hung Quoc Nguyen, for the American Society for Quality in Silicon Valley. This was the commercial version of the course (taught to people working as testers). Development of the course was significantly influenced by:

• Detailed peer reviews of the live class and of the circulating slide decks. The reviews included detailed critiques from colleagues when I made significant course updates (I offered free beta-review classes to test the updates).
• Co-teaching the material with colleagues. We would learn together by cross-teaching material, often challenging points in each other’s slides or lecture in front of the students. For example, I taught with James Bach, Elisabeth Hendrickson, Doug Hoffman and (for the metrics material) Pat Bond, a professor at Florida Tech.
• Rational Software, which contracted with me to create a customized version of BBST to support testing under the Rational Unified Process. They criticized the course in detail over several pilot teachings, and allowed me to apply what I learned back to the original course.

In 1999, I decided that if I wanted to learn how to significantly improve the instructional value of the course, I was going to have to see how teachers help students learn complex topics and skills in university. My sense was, and is, that good university instruction goes much deeper and demands more from the students than most commercial training.

Florida Tech hired me in 2000 to teach software engineering and encouraged me to evolve BBST down two parallel tracks:

• a core university course that would be challenging for our graduate students and a good resume-builder when our students looked for jobs
• a stronger commercial course that demanded more from the students.

We correctly expected that the two tracks would continually inform each other. Getting feedback from practitioners would help us keep the academic stuff real and useful. Trying out instructional ideas in the classroom would give us ideas for redesigning the learning experience of commercial students.

By 2003, I realized that most of my students were doing most of their learning outside the classroom. They claimed to like my lectures, but they were learning from assignments and discussions that happened out of the classroom. In 2004, I decided to try taping the lectures. The students could watch these at home, while we did activities in the classroom that had previously been done out of class. This went well, and in 2005, I created a full set of course videos.

I used the 2005 videos in my own classes. I put a Creative Commons license on the videos and posted them, along with other supporting materials, on my lab’s website. Rebecca Fiedler and I also started giving talks to educators about our results, such as these two papers (Association for Educational & Communications Technology conference and the Sloan Conference on Asynchronous Learning Networks in 2005).

These days, what we were doing has a name (“flipping“) and the Open Courseware concept is old news. Back then, it was still hard to find examples of other people doing this. Even though many other people were experimenting with the same ideas, not many people were yet publishing and so we had to puzzle through the instructional ideas by reading way too much stuff and thinking way too hard about way too many conflicting opinions and results. We summarized our own design ideas in the 2005 presentations (cited above). A good sample of the literature we were reading appeared in our applications for funding to the National Science Foundation, such as the one that was funded (2007), which gave us the money to pay graduate students to help with evaluation and redesign of the course (yielding the current public version).

For readers interested in the “science” that informed our course design, I’m including an excerpt from the 2007 Grant Proposal at the end of this article.

The Collaboration with AST

While we were sketching the first BBST videos, we were also working to form AST (the Association for Software Testing). AST incorporated in 2004. Perhaps a year later, Rebecca and I decided that the academic version of online BBST could probably be adapted for working testers. The AST activists at that time were among my closest professional friends, so it was natural to bring this idea to them.

We began informally, of course. We started by posting a set of videos on a website, but people kept asking for instructor support—for a “real” class. By this point (late 2006), the Florida Tech course was maturing and I was confident (in retrospect, laughably overconfident) that I could translate what was working in a mixed online-plus-face-to-face university class to a fully online course for practitioners located all over the world. The result worked so badly that everyone dropped out (even the instructors).

We learned a lot from the details of the failure, looked more carefully at how other university instructors had redesigned traditional academic courses to make them effective for remote students who had full-time jobs and who probably hadn’t sat in an academic classroom for several years (so their academic skills were rusty). After a bunch more pilot-testing, I offered the first BBST:Foundations as a one-month class (essentially the modern structure) in October, 2007.

We offered BBST:Foundations through AST, adding BBST:Bug Advocacy in 2008, redoing BBST:Foundations (new slides, videos, etc.) in 2010, and adding BBST:Test Design in 2012.

AST was our learning lab for commercial courseware. Florida Tech’s testing courses, and my graduate research assistants at Florida Tech, were my learning lab for the academic courseware. I would try new ideas at Florida Tech and bring the ones that seemed promising into the AST courses as minor or major updates. All the while, I was publishing the courseware at my lab’s website, testingeducation.org, and encouraging other people to use the material in their courses.

We trained and supervised a crew of volunteer instructors for AST’s BBST, but other people were teaching the course (or parts of it) too. This included professors, commercial trainers, managers teaching their own staff how to test, etc. Becky created an instructor’s course (how to teach BBST), which we offered as an instructor-led course through AST but which we also offered as a free learning experience on the web (study it yourself at your own pace). In 2012, we published a 357-page Instructor’s Manual for BBST. We published the book as a Technical Report (a publication method available to university professors) so that we could supply it to the public for free.

Underlying much of the AST collaboration was a hope that we could create an open courseware community that would function like some of the successful open software communities.

• In the open source software world, many of the volunteers who maintain and enhance open source software are able to charge people for support services. That is, the software (courseware) is free but if you want support, you have to pay for it. The support money creates an income stream that makes it possible for skilled people to spend time improving the software.
• We hoped that we could create a similar type of structure for open source courseware (the BBST courses). You can see the thinking, for example, in a 2008 paper that Rebecca and I wrote with Scott Barber, Building a free courseware community around an online software testing curriculum.

It turns out that this is a very complex idea. It is probably too complex for a small professional society that handles most of its affairs pretty informally.

For now, Rebecca and I have formed Kaner, Fiedler & Associates to sustain BBST instead. That is, KFA sells commercial BBST training and the income stream makes it possible for us to make new-and-improved versions of BBST.

AST might also create its own project to maintain and enhance BBST. If so, we’ll probably see the evolution of contrasting designs for the next generations of the courses. We think we’d learn a lot from that dynamic and we hope that it happens.

An Excerpt from our 2007 Grant Proposal

This is from our application for NSF Award CCLI-0717613, Adaptation and Implementation of an Activity-Based Online or Hybrid Course in Software Testing. (When we acknowledge support from NSF, we are required to remind you that National Science Foundation does not endorse any opinions, findings, conclusions or recommendations that arose out of NSF-funded research.) The full application is available online but it is very concisely written, structured according to very specific NSF guidelines, and packed with points that address NSF-specific concerns. Here is the most relevant section of that 56-page document here, in terms of explaining our approach and literature review for the course’s instructional design.

3. Our Current Course (Black Box Software Testing—BBST)

We adopted the new teaching method in Spring 2005 after pilot work in 2004. Our new approach spends precious student contact hours on active learning experiences (more projects, seminars and labs) that involve real-world problems, communication skills, critical thinking, and instructor scaffolding [129, 136] without losing the instructional benefits of polished lectures. Central to a problem-based learning environment is that students focus on “becoming a practitioner, not simply learning about practice” [122, p. 3]

Anderson et al.’s [11] update to Bloom’s taxonomy [20] is two-dimensional, knowledge and cognitive processing.

• On the Knowledge dimension, the levels are Factual Knowledge (such as the definition of a software testing technique), Conceptual Knowledge (such as the theoretical model that predicts that a given test technique is useful for finding certain kinds of bugs), Procedural Knowledge (how to apply the technique), and Metacognitive Knowledge (example: the tester decides to study new techniques on realizing that the ones s/he currently knows don’t apply well to the current situation.)
• On the Cognitive Process dimension, the levels are Remembering (such as remembering the name of a software test technique that is described to you), Understanding (such as being able to describe a technique and compare it with another one), Applying (actually doing the technique), Analyzing (from a description of a case in which a test technique was used to find a bug, being able to strip away the irrelevant facts and describe what technique was used and how), Evaluating (such as determining whether a technique was applied well, and defending the answer), and Creating (such as designing a new type of test.).

For most of the material in these classes, we want students to be able to explain it (conceptual knowledge, remembering, understanding), apply it (procedural knowledge, application), explain why their application is a good illustration of how this technique or method should be applied (understanding, application, evaluation), and explain why they would use this technique instead of some other (analysis).

3.1 We organize classes around learning units that typically include:

• Video lecture and lecture slides. Students watch lectures before coming to class. Lectures can convey the lecturer’s enthusiasm, which improves student satisfaction [158] and provide memorable examples to help students learn complex concepts, tasks, or cultural norms [47, 51, 115]. They are less effective for teaching behavioral skills, promoting higher-level thinking, or changing attitudes or values [19]. In terms of Bloom’s taxonomy [11, 20], lectures would be most appropriate for conveying factual and conceptual knowledge at the remembering and understanding levels. Our students need to learn the material at these levels, but as part of the process of learning how to analyze situations and problems, apply techniques, and evaluate their own work and the work of their peers. Stored lectures are common in distance learning programs [138]. Some students prefer live lectures [45, 121] but on average, students learn as well from video as live lecture [19, 139]. Students can replay videos [53] which can help students whose first language is not English. Web-based lecture segments supplement some computer science courses [34, 44]. Studio-taped, rehearsed lectures with synchronously presented slides (like ours) have been done before [29]. Many instructors tape live lectures, but Day and Foley [30-34] report their students prefer studio-produced lectures over recorded live lectures. We prefer studio-produced lectures because they have no unscripted interruptions and we can edit them to remove errors and digressions.
• Application to a product under test. Each student joins an open source software project (such as Open Office or Firefox) and files work with the project (such as bug reports in the project’s bug database) that they can show and discuss during employment interviews. This helps make concepts “real” to students by situating them in the development of well-regarded products [118]. It facilitates transfer of knowledge and skills to the workplace, because students are doing the same tasks and facing the same problems they would face with commercial software [25]. As long as the assignments are not too far beyond the skill and knowledge level of the learner, authentic assignments yield positive effects on retention, motivation, and transfer [48, 52, 119, 153].
• Classroom activities. We teach in a lab with one computer per student. Students work in groups. Activities are open book, open web. The teacher moves from group to group asking questions, giving feedback, or offering supplementary readings that relate to the direction taken by an individual group. Classroom activities vary. Students might apply ideas, practice skills, try out a test tool, explore ideas from lecture, or debate a question from the study guide. Students may present results to the class in the last 15 minutes of the 75-minute class. They often hand in work for (sympathetic) grading: we use activity grades to get attention [141] and give feedback, not for high-stakes assessment. We want students laughing together about their mistakes in activities, not mourning their grades [134].
• Examples. These supplementary readings or videos illustrate application of a test technique to a shipping product. Worked examples can be powerful teaching tools [25], especially when motivated by real-life situations. They are fundamental for some learning styles [43]. Exemplars play an important role in the development and recollection of simple and complex concepts [23, 126, 146]. The lasting popularity of problem books, such as the Schaum’s Outline series and more complex texts like Sveshnikov [148] attests to the value of example-driven learning, at least for some learners. However, examples are not enough to carry a course. In our initial work under NSF Award EIA-0113539 ITR/SY+PE: Improving the Education of Software Testers, we expected to be able to bring testing students to mastery of some techniques through practice with a broad set of examples. Padmanabhan [113, 132] applied this to domain testing in her Master’s thesis project at Florida Tech, providing students with 18 classroom hours of instruction, including lecture, outlines of ways to solve problems, many practice exercises and exams. Students learned exactly what they were taught. They could solve new problems similar to those solved in class. However, in their final exam, we included a slightly more complicated problem that required them to apply their knowledge in a way that had been described in lecture but not specifically practiced. The students did the same things well, in almost exactly the same ways. However, they all failed to notice problems that should have been obvious to them but that only required a small stretch from their previous drills. This result was a primary motivator for us to redesign the testing course from a lecture course heavy with stories, examples and practice to more heavily emphasize more complex activities.
• Assigned readings.
• Assignments, which may come with grading rubrics. These are more complex tasks than in-class activities. Students typically work together over a two-week period.
• Study guide questions. At the start of the course, we give students a list of 100 questions. All midterm and final exam questions come from this pool. We discuss use and grading of these questions in [60] and make that paper available to students. We encourage group study, especially comparison of competing drafts of answers. We even host study sessions in a café off campus (buying cappuccinos for whoever shows up). We encourage students to work through relevant questions in the guide at each new section of the class. These help self-regulated learners monitor their progress and understanding—and seek additional help as needed. They can focus their studying and appraise the depth and quality of their answers before they write a high-stakes exam. Our experience of our students is consistent with Taraban, Rynearson, & Kerr’s [149]—many students seem not to be very effective readers or studiers, nor very strategic in the way they spend their study time—as a result, they don’t do as well on exams as we believe they could. Our approach gives students time to prepare thoughtful, well-organized, peer-reviewed answers. In turn, this allows us to require thoughtful, well-organized answers on time-limited exams. This maps directly to one of our objectives (tightly focused technical writing). We can also give students complex questions that require time to carefully read and analyze, but that don’t discriminate against students whose first language is not English because these students have the questions well in advance and can seek guidance on the meaning of a question.

Excerpt from the Proposal’s references:

11. Anderson, L.W., Krathwohl, D.R., Airasian, P.W., Cruikshank, K.A., Mayer, R.A., Pintrich, P.R., Raths, J. and Wittrock, M.C. A Taxonomy for Learning, Teaching &  Assessing: A Revision of Bloom’s Taxonomy of Educational Objectives. Longman, New York, 2001.

19. Bligh, D.A. What’s the Use of Lectures? Jossey-Bass, San Francisco, 2000.

20. Bloom, B.S. (ed.), Taxonomy of Educational Objectives: Book 1 Cognitive Domain. Longman, New York, 1956

23. Brooks, L.R. Non-analytic concept formation and memory for instances. in Rosch, E. and Lloyd, B.B. eds. Cognition and categorization, Erlbaum, Hillsdale, NJ, 1978, 169-211.

25. Clark, R.C. and Mayer, R.E. e-Learning and the Science of Instruction. Jossey-Bass/Pfeiffer, San Francisco, CA, 2003.

29. Dannenberg, R.P. Just-In-Time Lectures, undated.

30. Day, J.A. and Foley, J. Enhancing the classroom learning experience with Web lectures: A quasi-experiment GVU Technical Report GVU-05-30, 2005.

31. Day, J.A. and Foley, J., Evaluating Web Lectures: A Case Study from HCI. in CHI ’06 (Extended Abstracts on Human Factors in Computing Systems), (Quebec, Canada, 2006), ACM Press, 195-200. Retrieved January 4, 2007, from http://www3.cc.gatech.edu/grads/d/Jason.Day/documents/er703-day.pdf

32. Day, J.A., Foley, J., Groeneweg, R. and Van Der Mast, C., Enhancing the classroom learning experience with Web lectures. inInternational Conference of Computers in Education, (Singapore, 2005), 638-641. Retrieved January 4, 2007, fromhttp://www3.cc.gatech.edu/grads/d/Jason.Day/documents/ICCE2005_Day_Short.pdf

33. Day, J.A., Foley, J., Groeneweg, R. and Van Der Mast, C. Enhancing the classroom learning experience with Web lecturesGVU Technical Report GVU-04-18, 2004.

34. Day, J.A. and Foley, J.D. Evaluating a web lecture intervention in a human-computer interaction course. IEEE Transactions on Education, 49 (4). 420-431. Retrieved December 31, 2006.

43. Felder, R.M. and Silverman, L.K. Learning and teaching styles in engineering education. Engineering Education, 78 (7). 674-681.

44. Fintan, C., Lecturelets: web based Java enabled lectures. in Proceedings of the 5th annual SIGCSE/SIGCUE ITiCSE Conference on Innovation and technology in computer science education, ( Helsinki, Finland, 2000), 5-8.

45. Firstman, A. A comparison of traditional and television lectures as a means of instruction in biology at a community college., ERIC, 1983.

47. Forsyth, D., R. The Professor’s Guide to Teaching: Psychological Principles and Practices. American Psychological Association, Washington, D.C., 2003.

48. Gagne, E.D., Yekovich, C.W. and Yekovich, F.R. The Cognitive Psychology of School Learning. HarperCollins, New York, 1994.

51. Hamer, L. A folkloristic approach to understanding teachers as storytellers. International Journal of Qualitative Studies in Education, 12 (4). 363-380, from http://ejournals.ebsco.com/direct.asp?ArticleID=NLAW20N8B16TQKHDEECM.

52. Haskell, R.E. Transfer of learning: Cognition, instruction, and reasoning. Academic Press, San Diego, 2001.

53. He, L., Gupta, A., White, S.A. and Grudin, J. Corporate Deployment of On-demand Video: Usage, Benefits, and Lessons, Microsoft Research, Redmond, WA, 1998, 12.

60. Kaner, C., Assessment in the software testing course. in Workshop on the Teaching of Software Testing (WTST), (Melbourne, FL, 2003), from https://kaner.com/pdfs/AssessmentTestingCourse.pdf

113. Kaner, C. and Padmanabhan, S., Practice and transfer of learning in the teaching of software testing. in Conference on Software Engineering Education & Training, (Dublin, 2007).

115. Kaufman, J.C. and Bristol, A.S. When Allport met Freud: Using anecdotes in the teaching of Psychology. Teaching of Psychology, 28 (1). 44-46.

118. Lave, J. and Wenger, E. Situated Learning: Legitimate Peripheral Participation. Cambridge University Press, Cambridge, England, 1991.

119. Lesh, R.A. and Lamon, S.J. (eds.). Assessment of authentic performance in school mathematics. AAAS Press, Washington, DC, 1992.

121. Maki, W.S. and Maki, R.H. Multimedia comprehension skill predicts differential outcomes of web-based and lecture courses.Journal of Experimental Psychology: Applied, 8 (2). 85-98.

122. MaKinster, J.G., Barab, S.A. and Keating, T.M. Design and implementation of an on-line professional development community: A project-based learning approach in a graduate seminar Electronic Journal of Science Education, 2001.

126. Medin, D. and Schaffer, M.M. Context theory of classification learning. Psychological Review, 85 (207-238)

129. National Panel Report. Greater Expectations: A New Vision for Learning as a Nation Goes to College, Association of American Colleges and Universities, Washington, D.C., 2002.

132. Padmanabhan, S. Domain Testing: Divide & Conquer Department of Computer Sciences, Florida Institute of Technology, Melbourne, FL, 2004.

134. Paris, S.G. Why learner-centered assessment is better than high-stakes testing. in Lambert, N.M. and McCombs, B.L. eds.How Students Learn: Reforming Schools Through Learner-Centered Education, American Psychological Association, Washington, DC, 1998.

136. Project Kaleidoscope. Project Kaleidoscope Report on Reports: Recommendations for Action in support of Undergraduate Science, Technology, Engineering, and Mathematics. Investing in Human Potential: Science and Engineering at the Crossroads, Washington, D.C., 2002. Retrieved January 16, 2006, from http://www.pkal.org/documents/RecommentdationsForActionInSupportOfSTEM.cfm.

138. Rossman, M.H. Successful online teaching using an asynchronous learner discussion forum. Journal of Asynchronous Learning Networks, 3 (2), from http://www.sloan-c.org/publications/jaln/v3n2/v3n2_rossman.asp.

139. Saba, F. Distance education theory, methodology, and epistemology: A pragmatic paradigm. in Moore, M.G. and Anderson, W.G. eds. Handbook of Distance Education, Lawrence Erlbaum Associates, Mahwah, New Jersey, 2003, 3-20.

141. Savery, J.R. and Duffy, T.M. Problem Based Learning: An Instructional Model and Its Constructivist Framework, Indiana University, Bloomington, IN, 2001.

146. Smith, D.J. Wanted: A New Psychology of Exemplars. Canadian Journal of Psychology, 59 (1). 47-55

148. Sveshnikov, A.A. Problems in probability theory, mathematical statistics and theory of random functions. Saunders, Philadelphia, 1968

149. Taraban, R., Rynearson, K. and Kerr, M. College students’ academic performance and self-reports of comprehension strategy use. Reading Psychology, 21 (4). 283-308.

153. Van Merrienboer, J.J.G. Training complex cognitive skills: A four-component instructional design model for technical training. Educational Technology Publications, Englewood Cliffs, NJ, 1997.

158. Williams, R.G. and Ware, J.E. An extended visit with Dr. Fox: Validity of student satisfaction with instruction ratings after repeated exposures to a lecturer. American Educational Research Journal, 14 (4). 449-457.

Sowmya Padmanabhan, Doug Hoffman and I just published a new book together, The Domain Testing Workbook.

The book focuses on one (1) testing technique. Domain Testing is the name of a generalized approach to Equivalence Class Analysis and Boundary Testing. Our goal is to help you develop skill with this one technique. There are plenty of overviews of this technique: it is the most widely taught, and probably the best understood, technique in the field. However, we’re not aware of any other presentations that are focused on helping you go beyond an understanding of the technique to achieving the ability to use it competently.

This is the first of a series of books that are coming along slowly. Probably the next one will be on scenario testing, with Dr. Rebecca Fiedler, merging her deep knowledge of qualitative methodology with my hands-on use of scenarios in software design, software testing, and software-related human factors. Also in the works are books on risk-based testing and specification-based testing. We’ve been working on all of these for years. We learned much from the Domain Testing Workbook about how challenging it is to write a book with a development-of-skill instructional design goal. At this point, we have no idea what the schedule is for the next three. When the soup is ready, we’ll serve it.

This work comes out of a research proposal that I sent to the United States’ National Science Foundation (NSF) in 2001 (Improving the Education of Software Testers), which had, among its objectives:

• “Identify skills involved in software testing.”
• “Identify types of exercises that support the development of specific testing skills.”
• “Create and publish a collection of reusable materials for exercises and tests.”
• “Prototype a web-based course on software testing.”
• “Create a series of workshops that focus on the teaching of software testing”

NSF approved the project, which made it possible for us to open the Center for Software Testing Education & Research (CSTER). The web-based course we had in mind is now available as the BBST series. The Workshops on Teaching Software Testing are now in their 13th year. And the Domain Testing Workbook is our primary contribution focused on “exercises that support the development of specific testing skills.”

When we started this project, we knew that domain testing would be the easiest technique to write this kind of book about. Over the years, we had two surprises that caused us to fundamentally rethink the structure of this book (and of other books that I’m still working on that are focused on scenario testing, risk-based testing, and specification-based testing):

• Our first surprise (or better put, our first shock) was that we could teach students a broad collection of examples of the application of domain testing, confirm that they could apply what they had learned to similar problems, and yet these students would fail miserably when we gave them a slightly more complex problem that was a little different than they had seen before. This was the core finding of Sowmya’s M.Sc. thesis. What we had tripped over was the transfer problem, which is probably the central instructional-design problem in Science, Technology, Engineering & Mathematics (STEM) instruction. The classic example is of the student who did well in her or his calculus course but cannot figure out how to apply calculus to problems (like modeling acceleration) in their introductory physics course. These students can’t transfer what they learned in one course to another course or to more complex situations (such as using it at their job). We had believed that we could work around the transfer problem by building our instruction around realistic exercises/examples. We were mistaken.
  • Ultimately, we concluded that teaching through exercises is still our best shot at helping people develop skill, but that we needed to provide a conceptual structure for the exercises that could give students a strategy for approaching new problems. We created a schema—an 18-step cognitive structure that describes how we do a domain analysis—and we present every exercise and every example in the context of that schema. We haven’t done the formal, experimental research to check this that Sowmya was able to do with our initial approach—an experiment like that is time-consuming and expensive and our funding for that type of work ran out long ago. However, we have inflicted many drafts of the schema on our students at Florida Tech and we believe it improves their performance and organizes their approach to tasks like exams.
• Our next surprise was that domain testing is harder to apply than we expected. Doug and I are experienced with this technique. We think we’re pretty good at it, and we’ve thought that for a long time. Many other testers perceive us that way too. For example, Testing Computer Software opens with an example of domain testing and talks about the technique in more detail throughout the book. That presentation has been well received. So, when we decided to write a book with a few dozen examples that increased in complexity, we were confident that we could work through the examples quickly. It might take more time to write our analysis in a way that readers could understand, but doing the analysis would be straightforward. We were sooooooo wrong. Yes, we could quickly get to the core of the problem, identifying how we would approach identifying equivalence classes and boundaries (or non-boundary most-powerful test cases) for each class. Yes, we could quickly list several good tests. We got far enough along that we could impress other people with what we knew and how quickly we could get there, but not far enough to complete the problem. We were often getting through about a third of the analysis before getting stuck. Doug would fly to Palm Bay (Florida, where I live) and we would work problems. Some problems that we expected to take a day to solve and explain took us a week.
  • As we came to understand what skills and knowledge we were actually applying when we slogged through the harder problems, we added more description of our background knowledge to the book. A presentation of our Domain Testing Schema—and the thinking behind it—grew from an expected length of about 30 pages to 200. Our presentation of 30 worked examples grew from an expected length of maybe 120 pages (4 pages each, with diagrams) to 190 pages.

We got a lot of help with the book. Our acknowledgments list 91 people who helped us think through the ideas in the book. Many others helped indirectly, such as many participants in the WTST workshops who taught us critical lessons about the instructional issues we were beating our heads against.

Perhaps the main advance in the clarity of the presentation came out of gentle-but-firm, collegial prodding by Paul Gerrard. Paul’s point was that domain testing is really a way for a tester to model the software. The tests that come out of the analysis are not necessarily tied to the actual code. They are tied to the tester’s mental model of the code. Our first reactions to these comments were that Paul was saying something obvious. But over time we came to understand his point—it might be obvious to us, but presentations of domain testing, including ours, were doing an inadequate job of making it obvious to our readers. This led us to import the idea of a notional variable from financial analysis as a way of describing the type of model that domain testing was leading us to make. We wrote in the book:

“A notional variable reflects the tester’s “notion” of what is in the code. The tester analyzes the program as if this variable was part of the code…. The notional variable is part of a model, created by the tester, to describe how the program behaves or should behave. The model won’t be perfect. That exact variable is probably not in the code. However, the notional variable is probably functionally equivalent to (works the same way as) a variable that is in the code or functionally equivalent to a group of variables that interact to produce the same behavior as this variable.”

This in turn helped us rethink parts of the Schema in ways that improved its clarity. And it helped us clarify our thinking about domain-related exploratory testing as a way of refining the modeling of the notional variables (bringing them into increasingly accurate alignment with the program implementation or, if the implementation is inadequate, with how the program should behave).

I hope you have a chance to read The Domain Testing Workbook and if you do, that you’ll post your reactions here or as a review on Amazon.

Becky and I are working on a new version of the BBST courses (BBST-2014). In the interim, to support the universities teaching the course, the AST, and the many people who are studying the materials on their own, we’re publishing some of the ideas we have for clarifying the course. The first two in this series focused on oracles in the Foundations course and on interactive grading as a way to give students more detailed feedback. This post is about follow-up testing, which we cover in the current BBST-BA’s slides 44-68.

Some failures are significant and they occur under circumstances that seem fairly commonplace. These will either be fixed, or there will be a compelling reason to not fix them.

Other failures need work. When you see them, there’s ambiguity about the significance of the underlying bug:

• Some failures look minor. Underlying the failure is a bug that might (or might not) be more serious than you’d imagine from the first failure.
• Some failures look rare or isolated. Underlying that failure is a bug that might (or might not) cause failures more often, or under a wider range of conditions, than you’d imagine from the first failure.

Investigating whether a minor-looking failure reflects a more serious bug

To find out whether the underlying bugs are more serious than the first failures make them look, we can do follow-up testing. But what tests?

That’s a question that Jack Falk, Hung Quoc Nguyen and I wrestled with many times. You can see a list of suggestions in Testing Computer Software. Slides 44-56 sort those ideas (and add a few) into four categories:

1. Vary your behavior
2. Vary the options and settings of the program
3. Vary data that you load into the program
4. Vary the software and hardware environment

Some students get confused by the categories (which is not shocking, because the categories are a somewhat idiosyncratic attempt to organize a broad collection of test ideas), confused enough that they don’t do a good job on the exam of generating test ideas from the categories.

For example, they have trouble with this question:

Suppose that you find a reproducible failure that doesn’t look very serious.

• Describe the four tactics presented in the lecture for testing whether the defect is more serious than it first appeared.
• As a particular example, suppose that the display got a little corrupted (stray dots on the screen, an unexpected font change, that kind of stuff) in Impress when you drag the scroll bar up and down. Describe four follow-up tests that you would run, one for each of the tactics that you listed above.

I’m not going to solve this puzzle for you, but the solution should be straightforward if you understand the categories.

The slides work through a slightly different example:

A program unexpectedly but slightly scrolls the display when you add two numbers:

• The task is entering numbers and adding
• The failure is the scrolling.

Let’s consider the categories in terms of this example

1. Vary your behavior

When you run a test, you intentionally do some things as part of the test. For example, you might:

• enter some data into input variables
• write some data into data files tha tthe program will read
• give the program commands

You might change any of these as part of your follow-up testing. (What happens if I do this instead of that?)

These follow-up tests might include changing the data or adding steps, substituting steps, or taking steps away.

For example, if adding one pair of numbers causes unexpected scrolling, suppose you try adding two numbers many times. Will the program scroll more, or scroll differently, as you repeat the test?

Suppose we modified the example so the program reads (and then adds) two numbers from a data file. Changing that data file would be another example of varying your behavior.

The slides give several additional examples.

2. Vary the options and settings of the program

Think about Microsoft Word. Here are some examples of its options:

• Show (or don’t show) formatting marks on the screen
• Check spelling (or don’t check it) as you type

In addition, you might change a template that controls the formatting of new documents. Some examples of variables you might change in the template are:

• the default typeface
• the spacing between tab stops
• the color of the text

Which of these are “options” and which of these are “settings”? It doesn’t matter. The terminology will change from program to program. What doesn’t change is that these are persistent variables. Their value stays with the program from one document to another.

3. Vary data that you load into the program

This isn’t well worded. Students confuse what I’m talking about with the basic test data.

Imagine again testing Microsoft Word. Suppose that you are focused on the format of the document, so your test documents have lots of headings and tables and columns and headers and footers (etc.). If you change the document, save it, and load the revised one, that is data that you load into the program, but I think of that type of change as part of 1. Vary your behavior.

When Word starts up, it also loads some files that might not be intentional parts of your test. For example, it loads a dictionary, a help file, a template, and probably other stuff. Often, we don’t even think of these files (and how what they hold might affect memory or performance) when we design tests. Sometimes, changing one of these files can reveal interesting information.

4. Vary the software and hardware environment

For example,

• Change the operating system’s settings, such as the language settings
• Change the hardware (a different video card) or how the operating system works with the hardware (a different driver)
• Change hardware settings (same video card, different display resolution)

This is a type of configuration testing, but the goal here is to try to demonstrate a more impressive failure, not to assess the range of circumstances under which the bug will appear.

Investigating whether an obscure-looking failure will arise under more circumstances

In this case, the failure shows up under what appear to be special circumstances. Does it only show up under special circumstances?

Slides 58-68 discuss this, but some additional points are made on other slides or in the taped lecture. Bringing them together…

1. Uncorner your corner cases
2. Look for configuration dependence
3. Check whether the bug is new to this version
4. Check whether failures like this one already appear in the bug database
5. Check whether bugs of this kind appear in other programs

Here are a few notes:

1. Uncorner your corner cases

Risk-based tests often use extreme values (boundary conditions). These are good for exposing a failure, but once you find the failure, try less extreme values. Demonstration of failure with a less extreme test will yield a more credible report.

2. Look for configuration dependence

In this case, the question is whether the failure will appear on many configurations or just this one. Try it with more memory, with another version of operating system (or on another OS altogether), etc.

3. Check whether the bug is new to this version

Does this bug appear in earlier versions of the program? If so, did users (or others) complain about it?

If the bug is new, especially if it was a side-effect of a fix to another bug, some people will take it much more seriously than a bug that has been around a long time but rarely complained about.

4. Check whether bugs like this one already appear in the bug database

One of the obvious ways to investigate whether a failure appears under more general circumstances than the ones in a specific test is to check the bug tracking database to see whether the failure has in fact occurred under other circumstances. It often takes some creativity and patience to search out reports of related failures (because they probably aren’t reported in exactly the same way as you’re thinking of the current failure), but if your database has lots of not-yet-fixed bugs, such a search is often worthwhile.

5. Check whether bugs of this kind appear in other programs

The failure you’re seeing might be caused by the code specifically written by this programmer, or the bug might be in a library used by the programmer. If it’s in the library, the same bug will be in other programs. Similarly, some programmers model (or copy) their code from textbook descriptions or from code they find on the web. If that code is incorrect, the error will probably show up in other programs.

If the error does appear in other programs, you might find discussions of the error (how to find it, when it appears, how serious it is) in discussions of those programs.

This post is partially based on work supported by NSF research grant CCLI-0717613 ―Adaptation & Implementation of an Activity-Based Online or Hybrid Course in Software Testing. Any opinions, findings and conclusions or recommendations expressed in this post are those of the author and do not necessarily reflect the views of the National Science Foundation.

Summary: This overview describes the general concept of high volume automated testing (HiVAT) and twelve examples of HiVAT techniques. The common thread of these techniques is automated generation, execution and evaluation of arbitrarily many tests. The individual tests are often weak, but taken together, they can expose problems that individually-crafted tests will miss. The simplest techniques offer high coverage: they can reach a program’s weakness in handling specific special cases. Other techniques take the program through long sequences of tests and can expose programs that build gradually (e.g. memory leaks, stack corruption or memory corruption) or that involve unexpected timing of events. In my experience, these sequences have exposed serious problems in embedded software, especially in systems that involved interaction among a few processors. As we embed more software into more consumer systems, including systems that pose life-critical risks, I believe these tests will become increasingly important.

I am writing this note as an intentionally rough draft. It servers as an introduction for a course on HiVAT at Florida Institute of Technology. It provides a structure for work in the course. It is intentionally under-referenced. One of the students’ tasks in the course is to dig through a highly disjointed literature and map research papers, practitioner papers, and conference presentations to the techniques listed here. Students might also add HiVAT techniques, with associated papers, or add sections on directly-relevant automation-support technology or directly-relevant surveys of test automation strategies / results. I will replace this post with later drafts that are more academically complete as we make progress in the course.

Consider automated regression testing. We reuse a regression test several times–perhaps running it on every build. But are these really automated? The computer might execute the tests and do a simple evaluation of the results, but a human probably designed that test, a human probably wrote the test code that the computer executes, a human probably provided the test input data by coding it directly into the test or by specifying parameters in an input file, a human probably provided the expected results that the program uses to evaluate the test result, and if there appears that there might be a problem, it will be a human who inspects the results, does the troubleshooting and either writes a bug report (if the program is broken) or rewrites the test. All that work by humans is manual.

Notice, by the way, that this interplay of human work and computer work applies whether the tests are run at the system level using a tool like QuickTest Pro, at the unit level using a tool like jUnit, or at a hybrid level, using a tool like FIT.

So, automated regression test are actually manual tests with an automated tint.

And every manual software test is actually automated. When you run a manual test, you might type in the inputs and look at the outputs, but everything that happens from the acceptance of those inputs to the display of the results of processing them is done by a computer under the control of a program–that’s automated.

The difference between “manual” and “automated” is a matter of degree, not a matter of principle. Some tests are more automated, others are less automated, but there is no “all” and there is no “none” for test automation.

High-Volume Tests Focused on Inputs

High-Volume Parametric Variation

We can make the automated regression tests more automated by transforming some of the human tasks to computer tasks. For example, imagine testing a method that takes two inputs (FIRST, SECOND) and returns their sum (SUM). A typical regression test would include specific values for FIRST, SECOND, and EXPECTED_SUM. But suppose we replace the specific test values with a test data generator that supplies random values for FIRST and SECOND and calculates the value of EXPECTED_SUM. We can generate billions of tests this way, each a little different, with almost no increase in human effort.

This is one of the simplest examples of high volume automated testing. A human supplied the algorithm, but the test tool applies that algorithm to create, run, and evaluate the results of arbitrarily many tests.

In this example, those tests are all pretty similar. Why bother running them all? The traditional answer is “don’t bother.” Domain testing is most widely used software testing technique. The point of domain testing is to help the tester minimize test redundancy by selecting a small number of inputs to represent the larger set of possibilities. Usually, this works well. Occasionally, a program has a problem with a small number of specific inputs. For example, the program might be optimized in a way that processes a few specific values specially. Or it might be vulnerable to small calculation errors that are usually too small to notice, but occasionally have a visible effect (see Doug Hoffman’s report at http://www.testingeducation.org/BBST/foundations/Hoffman_Exhaust_Options.pdf for an example.

High-Volume Combination Testing

Rather than thinking of testing a single function that takes a few inputs, imagine something more complex. A program processes several inputs, perhaps with a series of functions, and reports an output. Again, the traditional approach is to minimize redundancy. When we do combination tests (test several variables together), the set of possible tests is usually huge. If the variables are independent, we can minimize the number of combination tests with combinatorial testing (e.g. all-pairs or all-triples). If the variables are related, we can use cause-effect graphing instead. But if we suspect that the program will fail only on a small number of specific combinations (and we don’t know which few those are), we have to test a larger set of combinations, generating input values and calculating the expected result for each combination.

There are different strategies for generating the input values:

• Exhaustive sampling. This tests all the combinations but the set might be impossibly large.
• Random sampling. Generate random values for the inputs, stopping when some large number have been tested.
• Optimized sampling. Use an algorithm that optimizes the set of combinations in some way. As a very simple example, if you are going to test a million combinations, you could divide the space of combinations into a million same-size, non-overlapping subsets and sample one value of each. Or you could use a sequential algorithm that assigns values for the next combination by creating a test that is distant (according to some distance function) from all previous tests.

Input Fuzzing

So far, I’ve presented tests that generate both, the inputs and the expected value of the test. The expected value serves as an oracle, a mechanism for deciding whether the program passed or failed the test. Test oracles are incomplete, but they are useful for automation. Even if we can’t detect all possible failures, we can detect any failures of a certain kind (such as a calculation error).

“Fuzzing” refers to a family of high-volume automated tests that vary the inputs but have no oracle. The test runs until the program crashes or fails in some other unmissable way.

Hostile Data Stream Testing

Alan Jorgensen tested for many types of security errors by taking a good file in a standard format (e.g. PDF) and corrupting by substituting one string in the file with another. In Jorgensen’s work, the new string was typically much longer or was syntactically different. Jorgensen would then open the now-corrupt file with the application under test. The program might reject the file as corrupt or accept it. If it accepted the file, Jorgensen could ask the program to use the file in some way (e.g. display it) and look for misbehavior. In some cases, Jorgensen could exploit a failure to recognize a corrupt file by embedding executable code that the program would then inappropriately execute. Jorgensen’s search for corruptions that would escape detection was an intensely automated activity. His analysis of exploitability was not.

There are several published variations of file fuzzing (variation of the contents or format of an input file) that are suited for different kinds of risks.

High-Volume Tests that Exploit the Availability of an Oracle

Having an oracle gives you a strong basis for high-volume automated testing. All that’s required is to generate inputs to the program under test and to drive the program to process the inputs. Given a result, you can use the oracle to check whether the program passed the test. No oracle is perfect: the program might fail the test even though its output matches the oracle’s. But even though you can’t learn everything that might possibly be interesting by relying on an oracle, you have a basis for running a boatload of tests that collectively check thoroughly for some types of failures and give you an opportunity to stumble onto some other types of failures (e.g. crashes from memory leaks) that you didn’t design the tests to look for. For any other ways that you can imagine the program might fail, you can design tailored manual tests as needed.

High-volume testing with an oracle won’t completely test the program (against all possible risks), but it will provide you with a level of coverage that you can’t achieve by hand.

Function Equivalence Testing

Function equivalence testing starts with an oracle–a reference function that should work the same way as the function under test. Given the reference function, you can feed inputs to the function under test, get the results, and check whether the reference function gives the same results. You can test with as many input values as you want, perhaps generating a large random set of inputs or (if you have enough available computer time) testing every possible input.

The final exam in my Programmer-Testing course illustrates function equivalence testing. The question that students have to answer is whether Open Office Calc does its calculations the same way as Microsoft Excel. We adopt a quality criterion: If Calc does it the same way as Excel, that’s good enough, even if Excel makes a few calculation errors.

To answer the question, the students:

• test several functions individually
• then test those functions together by testing formulas that combine several functions

To do this,

• The students pick several individual functions in Calc
• They test each by feeding random inputs to the Calc function and the same inputs to the corresponding Excel function.
• Then they create random formulas that combine the functions, feed random data to the functions in the formula, and compare results.

If you test enough inputs, and the Calc results are always the same as Excel’s (allowing a little rounding error), it is reasonable to conclude that the calculation in Calc is equivalent to the calculation in Excel.

Constraint Checks

We use a constraint oracle to check for impossible values or impossible relationships.

For example an American ZIP code must be 5 or 9 digits. If you are working with a program that reads (or looks up or otherwise processes) ZIP codes, you can check every code that it processes. If it accepts (treats as a ZIP code) anything that has non-numeric characters or the wrong number of characters, then it has a bug. If you can find a way to drive the program so that it reads (or does whatever it does with) lots of ZIP codes, you have a basis for a set of high-volume automated tests.

Inverse Operations

Imagine taking a list that is sorted from high to low, sorting it low to high, then sorting it back (high to low). If you can give this program enough lists (enough sizes, enough diversity of values), you can eventually conclude that it sorts correctly (or not). Any operation that you can invert, you can build a high volume test series against.

State-Model Based Testing (SMBT)

If you have a state model (including a way to decide where the program should go if you give it an input and a way to determine whether the program actually got there), you can feed the program an arbitrarily long sequence of inputs and check the results.

I think the most common way to do SMBT is with a deterministic series of tests, typically the shortest series that will achieve a specified level of coverage. The typical coverage goal is every transition from each possible state to every state that it can reach next. You can turn this into a high-volume series by selecting states and inputs randomly and running the sequence for an arbitrarily long time. Ben Simo cautions that this has to be monitored because some programs will get locked into relatively tight loops, never reaching some states or some transitions. If you write your test execution system to check for this, though, you can force it out of the loop and into a not-yet hit state.

Diagnostics-Based Testing

I worked at a company that designed telephone systems (Telenova). Our phones gave customers a menu-driven interface to 108 voice features and 110 data features. Imagine running a system test with 100 phones calling each other, putting each other on hold, transferring calls from one phone to another, then conferencing in outside lines, etc. We were never able to create a full state model for our system tests. They were just too complex.

Instead, the Telenova staff (programmers, testers, and hardware engineers) designed a simulator that could drive the phones from state to state with specified or random inputs. They wrote probes into the code to check whether the system was behaving in unexpected ways. A probe is like an assert command, but if the program triggers the probe, it logs an error instead of halting the program. A probe might check whether a variable had an unexpected value, whether a set of variables had unexpected values relative to each other, whether the program went through a series of states in an unexpected order, etc.

Because these probes checked the internal state of the system, and might check any aspect of the system, we called them diagnostics.

Implementing this type of testing required a lot of collaboration. The programmers wrote probes into their code. Testers did the first evaluations of the test logs and did extensive troubleshooting, looking for simple replication conditions for events that showed up in the logs. Testers and programmers worked together to fix bugs, change the probes, and specify the next test series.

As challenging as the implementation was, this testing revealed a remarkable number of interesting problems, including problems that would have been very hard to find in traditional ways but had the potential to cause serious failures in the field. This is what convinced me of the value of high-volume automated testing.

High-Volume Tests that Exploit the Availability of Existing Tests or Tools

Sometimes, the best reason to adopt a high-volume automated technique is that the adoption will be relatively easy. That is, large parts of the job are already done or expensive tools that can be used to do the job are already in place.

Long-Sequence Regression Testing (LSRT)

For example, Pat McGee and I wrote about a well-known company that repurposed its huge collection of regression tests of its office-automation products’ firmware. (In deference to the company’s desire not to be named, we called it Mentsville.)

When you do LSRT, you start by running the regression tests against the current build. From that set, pick only tests that the program passes. Run these in random order until the program fails (or you’ve run them for a long enough time). The original regression tests were designed to reveal functional problems, but we got past those by using only tests that we knew the program could pass when you ran them one-at-a-time. The bugs we found came from running the tests in a long series. For example, the program might run a series of 1000 tests, thirty of them the same as the first (Test 1), but it might not fail Test 1 until that 30th run. Why did it fail on time 30 and not before?

• Sometimes, the problem was a gradual build-up of bad data in the stack or memory.
• Sometimes, the problem involved timing. For example, sometimes one processor would stay busy (from the last test) for an unexpectedly long time and wouldn’t be ready when it was expected for this test. Or sometimes the firmware would expect a location in memory to have been updated, but in this unusual sequence, the process or processor wouldn’t yet have completed the relevant calculation.
• Sometimes the problem was insufficient memory (memory leak) and some of the leaks were subtle, requiring a specific sequence of events rather than a simple call to a single function.

These were similar to the kinds of problems we found at Telenova (running sequences of diagnostics-supported tests overnight).

Troubleshooting the failures was a challenge because it was hard to tell when the underlying failure actually occurred. Something could happen early in testing that wouldn’t cause an immediate failure but would cause gradual corruption of memory until hours later, the system crashed. That early result was what was needed for replicating the failure. To make troubleshooting easier, we started running diagnostics between tests, checking the state of memory, or how long a task had taken to execute, or whether a processor was still busy, or any of over 1000 other available system checks. We only ran a few diagnostics between tests. (Each diagnostic changed the state of the system. We felt that running too many diagnostics would change the state too much for us to get an accurate picture of the the effect of running several regression tests in a row.) But those few tests could tell us a great deal. And if we needed more information, we could run the same 1000-test sequence again, but with different diagnostics between tests.

High-Volume Protocol Testing

A protocol specifies the rules for communication between two programs or two systems. For example, an ecommerce site might interact with VISA to bill a customer’s VISA card for a purchase. The protocol specifies what commands (messages) the site can send to VISA, what their structure should be, what types of data should appear in the messages and where, and what responses are possible (and what they mean). A protocol test sends a command to the remote system and evaluates the results (or sends a related series of commands and evaluates the series of responses back).

With popular systems, like VISA, lots of programs want to test whether they work with the system (whether they’ve implemented the protocol correctly and how the remote system actually responds). A tool to run these types of tests might be already available, or pieces of it might be available. If enough is available, it might be easy to extend it, to generate long random sequences of commands to the remote system and to process the system’s responses to each one.

Load-Enhanced Functional Testing

Common lore back in the 1970’s was that a system behaved differently, functionally differently, when it was running under load. Tasks that it could perform correctly under “normal” load were done incorrectly when the system got busy.

Alberto Savoia described his experience of this in a presentation at the STAR conference in 2000. Segue (a test automation tool developer) built a service around it. As I understand their results, a system could appear 50%-busy (busy, but not saturated) but still be unable to correctly run some regression tests. The value of this type of testing is that it can expose functional weaknesses in the system. For example, suppose that a system running low on memory will rely more heavily on virtual memory and so the timing of its actions slows down. Suppose that a program running on the system spreads tasks across processors in a way that makes it vulnerable to race conditions. Suppose that Task 1 on Processor 1 will always finish before Task 2 on Processor 2 when the system is running under normal load, but under some heavy loads, Processor 1 will get tied up and won’t process work as quickly as Processor 2. If the program assumes that Task 1 will always get done before Task 2, that assumption will fail under load (and so might the program).

In the field, bugs like this produce hard-to-reproduce failures. Unless you find ways to play with the system’s timing, you might never replicate problems like this in the lab. They just become mystery-failures that a few customers call to complain about.

If you are already load-testing a program and if you already have a regression suite, then it might be very easy to turn this into a long-sequence regression test running with a moderate-load load test in parallel. The oracles include whatever comes with each regression test (to tell you whether the program passed that test or not) plus crashes, obvious long delays, or warnings from diagnostics (if you add those in).

A Few More Notes on History

I think that most of the high-volume work has been done in industry and not published in the traditional academic literature. For example, the best-known (best-publicized) family of high-volume techniques are “fuzzing”, attributed to Professor Barton Miller at the University of Wisconsin in 1988.

• But I was seeing this industrial demonstrations of this approach when I first moved to Silicon Valley in 1983, and I’ve been told of applications as early as 1966 (the “Evil” program at Hewlett-Packard).
• Long-sequence regression testing was initially developed in 1984 or 1985. I played a minor role in developing Telenova’s diagnostics-based approach in 1987; my impression was that we were applying ideas already implemented years ago at other telephone companies.
• The testing staff at WordStar (a word-processing company) used a commercially available tool to feed long sequences of tests from one computer to another in 1984 or 1985. The first computer generated the tests and analyzed the test results. The second machine ran the program under test (WordStar). They were hooked up so that commands from the first machine looked like keyboard inputs to the second, and outputs that the second machine intended for its display actually went to the second machine for analysis. As an example of the types of bugs they found with this setup, they were able to replicate a seemingly-irreproducible crash that turned out to result from a memory leak. If you boldfaced a selection of text and then italicized it, there was a leak. If you italicized first, then applied bold, no leak. The test that exposed this involved a long random sequence of commands. I think this was a normal way to use that type of tool (long sequences with some level of randomization).
• We also used random input generators in the telephone world. I think of a tool called The Hammer but there were earlier tools of this class. Hammer Technologies was formed in 1991 and I was hearing about these types of tools back in the mid-to-late-1980’s while I was at Telenova.

I’ve heard of related work at Texas Instruments, Microsoft, AT&T, Rolm, and other telephone companies, in the testing of FAA-regulated systems, at some auto makers, and at some other companies. I’m very confident that the work actually done is much more broadly spread than this small group of companies and includes many other techniques than I’ve listed here. However, most of what I’ve seen has been by semi-private demonstrations or descriptions or via demonstrations and descriptions at practitioner conferences. Most of the interesting applications that I have personally heard of have involved firmware or other software that controls hardware systems.

As far as I can tell, there is no common vocabulary for these techniques. Much of what has been published has been lost because many practitioner conference proceedings are not widely available and because old papers describing techniques under nonstandard names are just not being found in searches. I hope that we’ll fill in some of these gaps over the next several months. But even for the techniques that we can’t find in publications, it’s important to recognize how advanced the state of the practice was 25 years ago. I think we are looking for a way to make sometimes-closely-held industrial practices more widely known and more widely adopted, rather than inventing a new area.

This post is partially based on work supported by NSF research grant CCLI-0717613 ―Adaptation & Implementation of an Activity-Based Online or Hybrid Course in Software Testing. Any opinions, findings and conclusions or recommendations expressed in this post are those of the author and do not necessarily reflect the views of the National Science Foundation.

The Workshop on Teaching Software Testing is concerned with the practical aspects of teaching university-caliber software testing courses to academic or commercial students.

WTST 2013 is focused on high volume automated testing (HiVAT). Our goal is to bring together instructors who have experience teaching high-volume techniques or who have given serious consideration to how to teach these techniques. We also welcome participants focused on the teaching of complex cognitive concepts and the transfer of what was learned to industrial practice.

As at all WTST workshops, we reserve some seats for senior students who are strongly interested in teaching and for faculty who are starting their careers in this area or beginning a research program connected with teaching this type of material.

There is no fee to attend this meeting. You pay for your seat through the value of your participation. Participation in the workshop is by invitation based on a proposal. We expect to accept 15 participants with an absolute upper bound of 25.

BACKGROUND ON THE WORKSHOP TOPIC

High volume automated testing involves automated generation, execution and evaluation of the results of a large set of tests. This contrasts with more traditional “automated” testing that involves automated execution of a relatively small number of human-created tests.

Here are four examples of the types of problems that underlie the need for HiVAT:

1. Many types of code weakness (such as timing-related problems) yield intermittent failures and are hard to detect with traditional testing techniques
2. There are immense numbers of possible combination tests of several variables together. Some combinations are unique (a failure appears only on a particular combination)
3. Some failures occur primarily when a system under test is under load and so detection and characterization is essentially a statistical challenge.
4. Characterizing the reliability of software requires a statistically useful set of tests.

In the academic community, the most commonly discussed HiVAT family of techniques is called “fuzzing” – but fuzzing as we know it involves very simplistic evaluation of the test results—essentially run the software until it crashes or fails in some other very obvious way. Other HiVAT techniques rely on more powerful oracles and can therefore find other kinds of bugs.

WTST is about teaching testing, not creating new techniques. The challenge we are trying to address in this WTST is that many of these techniques are known but not widely applied. We believe this is because they are poorly taught. As far as we can tell, most testing courses don’t even mention these techniques. Of those that do (and go beyond fuzzing), our impression is that students come out baffled about how to actually DO that type of testing in their work.

At Florida Tech, we’re trying to address this by creating “reference implementations” for several techniques—open source demonstrations of them, with commentary on the design and implementation. We’re hoping that WTST will provide examples of other good approaches.

TO ATTEND AS A PRESENTER

Please send a proposal BY DECEMBER 1, 2012 to Cem Kaner <kaner@cs.fit.edu> that identifies who you are, what your background is, what you would like to present, how long the presentation will take, any special equipment needs, and what written materials you will provide. Along with traditional presentations, we will gladly consider proposed activities and interactive demonstrations.

We will begin reviewing proposals immediately. We encourage early submissions. It is unlikely but possible that we will have accepted a full set of presentation proposals by December 1.

Proposals should be between two and four pages long, in PDF format. We will post accepted proposals to http://www.wtst.org.

We review proposals in terms of their contribution to knowledge of HOW TO TEACH software testing. Proposals that present a purely theoretical advance in software testing, with weak ties to teaching and application, will not be accepted. Presentations that reiterate materials you have presented elsewhere might be welcome, but it is imperative that you identify the publication history of such work.

By submitting your proposal, you agree that, if we accept your proposal, you will submit a scholarly paper for discussion at the workshop by January 8, 2013. Workshop papers may be of any length and follow any standard scholarly style. We will post these at http://www.wtst.org as they are received, for workshop participants to review before the workshop.

TO ATTEND AS A NON-PRESENTING PARTICIPANT:

Please send a message by DECEMBER 1, 2012, to Cem Kaner <kaner@cs.fit.edu> that describes your background and interest in teaching software testing. What skills or knowledge do you bring to the meeting that would be of interest to the other participants?

The hosts of the meeting are:

Cem Kaner (https://kaner.com and http://www.testingeducation.org)
Rebecca Fiedler (http://bbst.info)
Michael Kelly (http://www.developertown.com/)

HOW THE MEETING WILL WORK

WTST is a workshop, not a typical conference. It is a peer conference in the tradition of The Los Altos Workshops on Software Testing (http://lawst.com). Our presentations serve to drive discussion. The target readers of workshop papers are the other participants, not archival readers. We are glad to start from already-published papers, if they are presented by the author and they would serve as a strong focus for valuable discussion.

In a typical presentation, the presenter speaks 10 to 90 minutes, followed by discussion. There is no fixed time for discussion. Past sessions’ discussions have run from 1 minute to 4 hours. During the discussion, a participant might ask the presenter simple or detailed questions, describe consistent or contrary experiences or data, present a different approach to the same problem, or (respectfully and collegially) argue with the presenter. In 20 hours of formal sessions, we expect to cover six to eight presentations. Some of our sessions will be activities, such as brainstorming sessions, collaborative searching for information, creating examples, evaluating ideas or work products. We also have lightning presentations, time-limited to 5 minutes (plus discussion). These are fun and they often stimulate extended discussions over lunch and at night.

Presenters must provide materials that they share with the workshop under a Creative Commons license, allowing reuse by other teachers. Such materials will be posted at http://www.wtst.org.

Our agenda will evolve during the workshop. If we start making significant progress on something, we are likely to stick with it even if that means cutting or time boxing some other activities or presentations.

LOCATION AND TRAVEL INFORMATION

We will hold the meetings at

Harris Center for Assured Information, Room 327

Florida Institute of Technology

150 W University Blvd

Melbourne, FL 32901

Airport

Melbourne International Airport is 3 miles from the hotel and the meeting site. It is served by Delta Airlines and US Airways. Alternatively, the Orlando International Airport offers more flights and more non-stops but is 65 miles from the meeting location.

Hotel

We recommend the Courtyard by Marriott – West Melbourne located at 2101 W. New Haven Avenue in Melbourne, FL.

Please call 1-800-321-2211 or 321-724-6400 to book your room by December 24, 2012. Be sure to ask for the special WTST rates of $93 per night. Tax is an additional 11%. All reservations must be guaranteed with a credit card by Tuesday, December 24, 2013. If rooms are not reserved, they will be released for general sale. Following that date reservations can only be made based upon availability.

For additional hotel information, please visit the hotel website at http://www.marriott.com/hotels/travel/mlbch-courtyard-melbourne-west/

Kaner, Fiedler & Associates has completed the process of registering BBST as our trademark. We will be revising the course slides soon to show BBST ® (the R in the circle) and to include an acknowledgment in the acknowledgement paragraph on the front slide of each course. Please acknowledge the trademark when you publicize or write about the BBST courses. The proper notice is “BBST is a registered trademark of Kaner, Fiedler & Associates ,LLC”).

We decided to register the trademark a long time ago, when we saw a few training companies marketing “BBST training” even though they didn’t have any connection with us and, as far as we could tell, their principals had not taken BBST courses. That was reinforced as people we didn’t know approached us with marketing ideas that seemed highly inappropriate. Registration protects “BBST” from being hijacked for other purposes. Registration has finally been completed and so it is time to start applying it.

The BBST courses, including the instructor’s course and the instructor’s manual continue to be Creative Commons-licensed. Anyone can teach those materials. Anyone can advertise that they teach those materials (as long as those are the materials they actually teach). Anyone can modify the course as they see fit and advertise that they teach a course that is partially based on those materials.  Nothing in the registration of BBST limits any of these things. It merely protects “BBST” from inappropriate use (i.e. use that goes beyond these extremely broad limits).

If you have specific questions about the BBST trademark, please ask.

The BBST course series is open source — anyone can download the materials. Anyone can teach the courses.

Dr. Rebecca L. Fiedler designed the BBST Instructors Course to teach people who had taken BBST how to teach it. Over several years, we’ve been working on an Instructors’ Manual to support the course, publishing several drafts for review.

Dr. Fiedler, Doug Hoffman and I have finally finished the manual, a 357-page book. You can find it here, at the web page for the BBST Instructors Course. (Like BBST, the BBST instructors course materials are available to the world for free. Enjoy!)

As with all of the BBST work, we thank the National Science Foundation, for its support: grants EIA-0113539 ITR/SY+PE Improving the Education of Software Testers and CCLI-0717613 Adaptation & Implementation of an Activity-Based Online or Hybrid Course in Software Testing. (The views expressed in BBST and this blog reflect the opinions of the authors and not NSF.)

When I first studied testing, I learned that a test involved comparison of the test result to an expected result. The expected result was the oracle: the thing that would tell you whether the program passed or failed the test. We generalized this a bit, especially for automated testing–we would look to a reference program as our oracle. The reference program is a program that generates the expected results, rather than the results themselves. But the idea was the same: testing involved comparison with a known result.

The Oracle Problem

Oracle is an interesting choice of terminology, because the oracles of Greece (the original “oracles”) were mythological. And Greek tragedies are full of stories of people who misinterpreted what an oracle told them, and behaved (on the basis of their understanding) in ways that brought disaster on them.

If we define a software testing oracle as a tool that tells you whether the program passed your test, we are describing a myth–something that doesn’t exist. Relying on the oracle, you might make either of the classic mistakes of decision theory:

• The miss: you believe the program has passed even though it did something wrong.
• The false alarm: you believe the program has failed even though it has behaved appropriately.

So we soften the definition: a software testing oracle is a tool that helps you decide whether the program passed your test.

Seen this way, oracles are heuristic devices: they are useful tools that help us make decisions, but sometimes they point us to the wrong decision.

If you don’t have authoritative oracles (“authoritative” = an oracle that is always correct), then how can you test? How can you specify a test in a way that a junior tester or a computer can run the test and correctly tell you whether the program passed it?

The Instructional Problem

I’ve been emphasizing the oracle problem in my testing courses for about a dozen years. I see this as one of the defining problems of software testing: one of the reasons that skilled testing is a complex cognitive activity rather than a routine activity. Most of the time, I start my courses with a survey of the fundamental challenges of software testing, including an extended discussion of the oracle problem.

If you’ve seen the BBST-Foundations courses (for example, the Association for Software Testing teaches a version of this course), you’ve seen my introduction to the oracle problem and to heuristic oracles. Students typically work through one or more theoretical and/or practical labs in BBST. Once they understand the oracle problem, the course presents two approaches for using oracles:

• One approach, that I associate with James Bach and Michael Bolton, lists 8 general types of expectations. For example, we expect a product to operate consistently across all of its features. We expect it to operate consistently from version to version, etc. (I’ll list the set of consistencies later.)
• A different approach, that I associate with Doug Hoffman (but I think it’s been independently developed and followed by lots of people), lists specific heuristics. None of them is complete. Each focuses on a specific prediction about the results of a test and ignores other aspects of the test. For example, if we are testing a program that does calculations, checking whether it says 2+3=5 (5 is the expectation) is not complete, but it is useful. Testing whether we can invert an operation (take the square root of a square of a number) isn’t a complete test of a square-a-number function, but it is useful. (More examples below…)

Bach and Bolton have done a good job of explaining their approach. Mike Kelly provides a great summary of it. I follow an explanatory approach that I learned from an early version of Bach’s RST course in my lectures and it works well. Students understand the consistencies and (generally) find them compelling. The list feels complete–any specific oracle you can think of can be classified as an example of one of their consistencies.

I think there are three problems with Bach and Bolton’s consistencies:

1. These provide a useful way to think about a bug after you find it and try to report it. The list of consistencies can structure your thinking as you try to figure out how to explain to someone else why a particular program behavior feels wrong (what feels wrong about it?). However, even though I do find them helpful for evaluating test results, I don’t find them helpful for designing tests.
2. I think they are particularly worthless for designing automated tests. Automated testing depends on oracles–the automated-testing-program that runs a zillion tests has to decide whether the software under test passed each test or not. The consistencies don’t guide testers (not me, not the testers I know, not the students I teach) toward oracle ideas that are specific enough to be programmed and used by an automaton.
3. In my courses, the consistencies capture my students’ imagination and interfere with their thinking about oracles that would be useful for designing tests, especially automated tests.

It is the third problem that I have been wrestling with for several years and that will cause me to rewrite the BBST-Foundations course.

In exam after exam, when I give students a specific scenario that clearly involves automated testing and ask them to suggest oracles they would use to support their automated testing and how they would use them, they ramble through a memorized list of consistency heuristics and don’t come up with ideas–some students don’t come up with any ideas–for oracles that support the automation.

• I tried to correct this problem by raising the problem in supplementary lectures. It didn’t work.
• I went even further, telling students that this was a classic problem in this course and they needed to answer questions about oracles in test design with specific oracles. It didn’t work.
• I went even further, telling students as part of the exam question itself that this question called for specific ideas about the oracles they would design into specific tests and they shouldn’t rely on general descriptions of consistencies. It didn’t work.
• Even when I give a set of exam questions to students in advance, and they draft an answer in advance with full benefit of time, course notes, lecture notes and videos, discussions with each other, and anything they can find on the web–even when these questions have cautionary notes about this being a question about test automation and they shouldn’t just present a consistency oracle–it still didn’t work.

They just keep giving back worthless ideas about test design and flunk that part of the exam.

Feh!

An Important Instructional Heuristic

When a few students give bad answers on an exam, the problem is in the students. They don’t understand the material well enough.

When a lot of students give bad answers on the exam, the problem is in the instruction. It’s the responsibility of the teacher to troubleshoot and fix this.

When a lot of students give bad answers that are weak in a consistent way, something specific in the instruction leads them down that path. In my experience, that something is often something the instructor is particularly attached to.

I like the consistencies a lot. But I think that in the Foundations course, they are an attractive nuisance (an almost-irresistable invitation to take a hazardous or counterproductive path).

Therefore

• In the next generation of the Foundations course, I will probably drop the oracle consistencies approach altogether.
• In the next generation of the Bug Advocacy course, I will probably add the oracle consistencies as a useful tool for persuasive bug report writing.

Appendix: More Details on Oracles

1. The underlying problem: Oracles are necessarily incomplete
2. Oracles are heuristics
3. Bach and Bolton’s consistencies
4. Hoffman’s approach
5. Applying this to test automation

Oracles are Necessarily Incomplete

Back when dinosaurs roamed the earth, some testasauruses theorized that a properly designed software test involves:

• a set of preconditions that specify the state of the software and system when you start the test
• a set of procedures that specify what you do when you do the test
• comparison of what the software under test does with a set of postconditions: the predicted state of the system under test after you run the test. This set of postconditions make up the expected results of the test.

We can call the postconditions the oracle or we can say that a program that generates the expected results is the oracle, but in either case, the testasauruses said, good testing involves comparing the program’s test behavior to expected results, and to do good testing, you need an oracle. (Fossils from this era have been preserved in IEEE Standard 829 on software test documentation.)

Elaine Weyuker’s (1980) On Testing Nontestable Programs shattered that view. Weyuker argued that “it is unusual for … an oracle to be pragmatically attainable or even to exist” (p. 3). Instead, she said, testers rely on partial oracles. For example:

• A tester might recognize a result of a calculation as impossibly large even though she doesn’t know what the exact result should be. (You might not know offhand what 1.465732 x 2.74312 is, but if a program said 7,000,000 you could reject that as obviously wrong without doing any calculations.)
• A tester might recognize behavior as inappropriate, even if she doesn’t know exactly how the program should behave.

Weyuker’s paper wasn’t widely noticed in the practitioner community. I don’t think we appreciated the extent of this problem until the Quality Week conference in 1998, when Doug Hoffman (A Taxonomy for Test Oracles) explained this problem and its implications this way:

Suppose that we specify a test by describing

• the starting state of the system under test
• the test inputs (the data and operations you use to carry out the test)
• the expected test outputs

We can still make mistakes in interpreting the test results.

• We might incorrectly decide that the program passed the test because its outputs matched the expected outputs but it misbehaved in some other way. For example, a program that adds 2+2 might get 4, but it is clearly broken in some way if it takes 10 hours to get that result of 4.
• We might incorrectly decide that the program failed the test because its outputs did not match the expected results, but on more careful examination, we might realize that it did the right thing. For example, imagine testing to a network printer with the expectation that the printer will page a specific page within 1 minute–but during the test, another computer sent a long document to the printer and so it didn’t actually get to the test document for a long time. This might be the exactly correct behavior under the circumstances, but it doesn’t match the expectation.

Most testers, doing manual testing, would probably not make either mistake. But an automated test would make both mistakes. So would a manual tester who was trying to exactly follow a fully-detailed script.

Doug argued that both types of mistakes were inevitable in testing because no one could fully specify the starting state of the system and no one could fully specify the ending state of the system. There are too many potentially-relevant variables. For example, suppose in your 2+2 test, you do specify the expected time for the test to complete:

• Did you specify the contents of the stack? What if the program adds stuff to the stack but doesn’t remove it, or corrupts the stack in some other way?
• Did you specify the contents of memory? Memory leaks are common bugs. And buffer overflows are a common example of a class of bug that corrupts memory.
• Did you specify the contents of the hard disk? What if the program saves something or deletes something?
• Did you specify what the printer would do during the test? What if the program sends something to the printer, even though it is not supposed to, or sends unauthorized email, etc.?

If you don’t have experience thinking about the diversity of ways that something can go wrong, but you have a bit of technical savvy, the Hewlett-Packard printer diagnostics can be eye-opening. You can find documentation of these in Management Information Bases (MIB’s) published by HP. I find these at https://spp.austin.hp.com/SPP/Public/Sdk/SdkPublicDownload.aspx but if this source goes away, you can find third party sites like OiDView. For example, the MIB file for the LaserJet 9250c runs 8506 lines, documenting 176 commands, many of them with many possible parameters. A program can go wrong in hundreds (or thousands) of different ways.

From a diagnostic point of view, imagine running a test and checking the state of the printer. For example, you might check how much free memory there is, or how long the last command took to execute, or the most recent internal error code. Each diagnostic command that you run changes the state of the machine, and so the results of the next diagnostic are no longer looking at the system as it was right after the test completed.

So in practical terms, even if you could fully specify the state of the system after a test (you can’t, but pretend that you could), you still couldn’t check whether the system actually reached that state after the test because each of the diagnostics that you would run to check the state of the system would change the state. The next diagnostic tests the machine that is now in a different state. In practical terms, you can only run a few diagnostics as part of a test (maybe just one) before the diagnostics stop being informative. If  these diagnostics don’t look for a problem in the right places, you won’t see it. This is sometimes called the Heisenbug problem, in honor of the Heisenberg Uncertainty Principle.

Oracles are Heuristics

Hoffman argued that no oracle can fully specify the postcondition state of the system under test and therefore no oracle is complete. Given that an oracle is incomplete, you might use the oracle and incorrectly conclude that the program failed the test when it didn’t or passed the test when it didn’t. Either way, reliance on an oracle can lead you to the wrong conclusion.

A decision rule that is useful but not always correct is called a heuristic.

My favorite presentations of the ideas underlying Heuristics were written by Billy V. Koen. See his book (I prefer the shorter and simpler ASEE early edition used in introductory engineering courses, but the current version is good too) and a wonderful historical article that he wrote for BBST.

The Bach / Bolton Consistency Heuristics

Imagine running a test. The program misbehaves. The tester notices the behavior and recognizes that something is wrong. What is it that makes the tester decide this is wrong behavior?

In Bach’s view (as I understand it from talking with him and teaching about this with him), what happens is that the tester makes a comparison between the behavior and some expectations about the ways the program should (or should not) behave. These comparisons might be conscious or unconscious, but Bach posits that they must happen because every explanation of why a program’s behavior has been evaluated as a misbehavior can be mapped to one of these types of consistency.

Here’s the list:

• Consistent within product: Function behavior consistent with behavior of comparable functions or functional patterns within the product.
• Consistent with comparable products: Function behavior consistent with that of similar functions in comparable products.
• Consistent with history: Present behavior consistent with past behavior.
• Consistent with our image: Behavior consistent with an image the organization wants to project.
• Consistent with claims: Behavior consistent with documentation, specifications, or ads.
• Consistent with standards or regulations: Behavior consistent with externally-imposed requirements.
• Consistent with user’s expectations: Behavior consistent with what we think users want.
• Consistent with purpose: Behavior consistent with product or function’s apparent purpose.

(If you reorder the list, you can use a mnemonic abbreviation to memorize it: HICCUPPS.)

For example, imagine that there is a program specification and that the program behaves differently from what you would predict from the specification. The behavior might be reasonable, but if it contradicts the specification, you should probably write a bug report. Your explanation of the problem in the report wouldn’t be “this is bad”. It would be “this is bad because it is inconsistent with the specification.”

The list is designed to cover every type of consistency-expectation that testers rely on. If they realize the list is incomplete, they add a new type.

For the sake of argument, I will assume that this list is complete, i.e. that every rationale that a tester provides for why a program is misbehaving can be mapped to one of these 8 types of consistency.

I have seen it argued (mainly on Twitter) that this is the “right” list. That every other oracle can be mapped to this list (this oracle tests for this type of inconsistency) and therefore they are all special cases. If you know this list, the argument goes, you can derive (or imagine) (or something) all the oracles from it.

As far as I know, there is no empirical research to support the claim that testers in fact always rely on comparisons to expectations or that these particular categories of expectations map to the comparisons that go on in testers’ heads.

• That assertion does not match my subjective impression of what happens in my head when I test. It seems to me that misbehaviors often strike me as obvious without any reference to an alternative expectation. One could counter this by saying that the comparison is implicit (unconscious) and maybe it is. But there is no empirical evidence of this, and until there is, I get to group the assertion with Santa Claus and the Tooth Fairy. Interesting, useful, but not necessarily true.
• The assertion also does not match my biases about the nature of concept formation and categorical reasoning. As a graduate student, I studied cognition with Professor Lee R. Brooks. Some of his most famous work was on nonanalytic concept formation (see his 1978 chapter In Rosch & Lloyd’s classic Cognition & Categorization or his paper with Larry Jacoby (1984) on Nonanalytic cognition. A traditional view of cognition holds that we make many types of judgment on the basis of rules that put things into categories–something is this or that because of a set of rules that we consult either consciously or unconsciously. Bach and Bolton’s consistencies are examples of the kinds of categories that I think of when I think of this tradition. A very different view holds that we make judgments on the basis of similarity to exemplars. (An exemplar is a memorable example.) A person can learn arbitrarily many exemplars. Experts have probably learned many more than nonexperts and so they make better evaluations. One of the most interesting experiments in Lee’s lab required the experimental subject to make a judgment (saying which category something belonged to) and explain the judgment. The subjects in the experiments described what they said were their decision rules to explain each choice. But over a long series of decisions, you can ask whether these rules actually describe the judgments being made. The answer was negative. The subject would describe a rule that recently he hadn’t followed and that he would again not follow later. Instead, the more accurate predictor of his decisions was the similarity of the thing he was categorizing to other things he had previously categorized. It appeared that unconscious processing was going on, but it was  nonanalytic (similarity-based), not analytic (rule-based). I found, and still find, this line of results persuasive.

A list can be useful as a heuristic device, as a tool that helps you consciously think about a problem, whether the list describes the actual underlying psychology of testing or not.

But if it is to be a good heuristic device, it has to be more useful than not. As a tool for teaching oracles as part of test design, my experience is that the consistency list fails the utility criterion.

I don’t have any scientific research to back up my conclusion, just a lot of personal experience. But when dealing with a heuristic device that is not backed up by any scientific research (just a lot of personal experience), I get to rely on what I’ve got.

Doug Hoffman’s Approach

I first saw Doug talk about oracles in 1998, at Quality Week. That was the start of a long series of publications on oracles and the use of oracles in test automation. Along with the papers, I have the benefit of having taught courses on test automation with Doug and having talked at length with him while he struggled to get his ideas on paper.

Doug made two key points in 1998:

• All oracles are heuristic (we’ve already covered that ground)
• There are a lot of incomplete oracles available. Given that we have to rely on incomplete oracles (because no oracles are complete), we should think about what combinations of oracles we can use to learn interesting things about the software.

Doug’s work was so striking that we opened the Fifth Los Altos Workshop on Software Testing with it. That meeting became an intense, 2-day long, moderated debate between Doug and James Bach. We learned so much about managing difficult debates in that meeting that we were able to create what I think of as the current structure of LAWST, adopted in LAWST 6.

Doug has published several lists of specific types of oracles. Unfortunately, each of the ones I’ve read has its own idiosyncrasies that can be confusing, so I won’t try to restate them. Instead, I’ll work from BBST-Foundations-2013 (in preparation), which refines a list that I prepared for the current BBST Foundations with Doug’s coaching.

• We use the constraint oracle to check for impossible values or impossible relationships. For example an American ZIP code must be 5 or 9 digits. If you see something that is non-numeric or some other number of digits, it cannot be a ZIP code. A program that produces such a thing as a ZIP code has a bug.
• We use the regression oracle to check results of the current test against results of execution of the same test on a previous version of the product.
• We use self-verifying data as an oracle. In this case, we embed the correct answer in the test data. For example, if a protocol specifies that when a program sends a message to another program, the other one will return a specific response (or one of a few possible responses), the test could include the acceptable responses. An automated test would generate the message, then check whether the response was in the list or was the specific one in the list that is expected for this message under this circumstance.
• We use a physical model as an oracle when we test a software simulation of a physical process. For example, does the movement of a character or object in a game violate the laws of gravity?
• We use a business model the same way we use a physical model. If we have a model of a system, we can make predictions about what will happen when events X take place. The model makes predictions. If the software emulates the business process as we intend, it should give us behavior that is consistent with those predictions. Of course, as with all heuristics, if the program “fails” the test, it might be the model that is wrong.
• We use a statistical model to tell us that a certain behavior or sequence of behaviors is very unlikely, or very unlikely in response to a specific action. The behavior is not impossible, but it is suspicious. We can test whether the actual behavior in the test is within the tolerance limits predicted by the model. This is often useful for looking for patterns in larger sets of data (longer sequences of tests). For example, suppose we expect an eCommerce website to get 80% of its customers from the local area, but in beta trials of its customer-analysis software, the software reports that 70% of the transactions that day were from far away. Maybe this was a special day, but probably this software has a bug. If we can predict a statistical pattern (correlations among variables, for example), we can check for it.
• Another type of statistical oracle starts with an input stream that has known statistical characteristics and then check the output stream to see if it has the same characteristics. For example, send a stream of random packets, compute statistics of the set, and then have the target system send back the statistics of the data it received. If this is a large data set, this can save a lot of transmission time. Testing transmission using checksums is an example of this approach. (Of course, if a message has a checksum built into the message, that is self-verifying data.)
• We use a state model to specify what the program does in response to an input that happens when it is in a known state. A full state model specifies, for every state the program can be in, how the program will respond (what state it will transition to) for every input.
• We can build an interaction model to help us test the interaction between this program and another one. The model specifies how that program will behave in response to events in (actions of) this program and how this program will behave in response to actions of the other program. The automaton triggers the action, then checks the expected behavior.
• We use calculation oracles to check the calculations of a program. For example, if the program adds 5 numbers, we can use some other program to add the 5 numbers and see what we get. Or we can add the numbers and then successively subtract one at a time to see if we get a zero.
• The inverse oracle is often a special case of a calculation oracle (the square of the square root of 2 should be 2) but not always. For example, imagine taking a list that is sorted low to high, sorting it high to low and then sorting it low to high. Do we get back the same list?
• The reference program generates the same responses to a set of inputs as the software under test. Of course, the behavior of the reference program will differ from the software under test in some ways (they would be identical in all ways only if they were the same program). For example, the time it takes to add 1000 numbers might be different in the reference program versus the software under test, but if they ultimately yield the same sum, we can say that the software under test passed the test.

You can probably imagine lots of other possibilities for this list.

Applying This to Automation

What is special about these oracles is that they are programmable. You can create automated tests that will check the behavior of the program against the result predicted by (or predicted against by) any of these oracles or by any (well, probably, almost any) combination of these oracles.

Given a programmable oracle you can do high volume automated testing. Have the test-design-and-execution program randomly generate inputs to the software under test and check whether the software responds the way the oracle predicts. You might use some type of model to drive the random number generator (making some events more likely than others). You might randomly create a long random sequence of tests (e.g. regression tests) by randomly selecting which test to run next from a pool of already-built tests (each of which has an expected result that you can check against). Given an oracle, you can detect whatever failures that oracle can expose. For example, you might test with several oracles:

• one oracle predicts how long an operation should take (or a range of possibility). If the program takes substantially more or less time, that’s a problem.
• another oracle can predict the calculation result of the operation (or the functional result if you’re doing something else, like sorting, that isn’t exactly a calculation)
• another oracle might predict the amount of free memory, or at least might tell you whether a large data set (or memory-intensive calculation) should fit in memory. If so, you can detect memory leaks this way.

No matter what combination you choose, you will miss some types of errors. You cannot test all the dimensions of the result of a test with any oracle or any combination of oracles. But if you test a feature by machine using some oracles, then when you test that feature with a human painstakingly designing and running each test individually, that person will know that she doesn’t have to waste time checking whether certain types of bugs are there or not, because if they were there, the automaton would already have exposed them.

An Exam Question

Here’s an exam question (that students in the current version of BBST Foundations have often handled poorly):

Suppose you have written a test tool that allows you to feed commands and data to Microsoft Excel and to Open Office Calc and to see the results. The test tool is complete, and it works correctly. You have been asked to test a new version of Calc and told to automate all of your testing. What oracles would you use to help you find bugs? What types of information would you expect to get with each oracle?

Note: Don’t just echo back a consistency heuristic. Be specific in your description of a relevant oracle and of the types of information or bugs that you expect.

Look back at the Hoffman list and think of what oracles you could use for this test, to facilitate extensive automated testing.

Now look further back to the Bach / Bolton list and think of what oracles their lists suggests, that would work well for designing extensive automated testing.

For me, the Hoffman list works better. (And if I thought about additional oracles that were specific enough to support automation, I would add them to the Hoffman list, growing it into something that gets longer and longer the more I use it.) What about for you?

This post is partially based on work supported by NSF research grant CCLI-0717613 ―Adaptation & Implementation of an Activity-Based Online or Hybrid Course in Software Testing. Any opinions, findings and conclusions or recommendations expressed in this post are those of the author and do not necessarily reflect the views of the National Science Foundation.

This week, Florida Tech’s Center for Software Testing Education & Research (my lab) published a bibliography of dissertations and theses focused on software testing (http://www.zotero.org/groups/cster_dissertations). To use the bibliography, you need an open source (free) tool called Zotero (http://www.zotero.org/)

What are these documents?

Theses are research reports written by graduate students as a final requirement for graduation. Doctoral theses are also called dissertations.

The typical thesis includes:

• a literature review that describes a significant set of related research published by others
• an idea (for example, an idea about how to improve testing, or how to create and assess a testing tool, or how to study how testing is really done, or how to teach it more effectively, or how to prove that some other idea about testing is wrong)
• a description of the thesis methodology and technology (for example, how a test tool was designed, implemented, and studied)
• a description of the results of the study.

Theses are evaluated by professors who are experts in the discipline and at least one who is not. For example, the supervisory committee for a doctoral student in Computer Science might include three professors of Computer Science, one professor of Biology and one of Business Administration. Of the three computer scientists, one (or two) would typically be expert in the subject matter of the thesis (e.g. software testing) and the others would probably be experts in other areas of computing that the work depends on. for example, a dissertation focused on testing databases might be supervised by an expert in testing, an expert in databases and an expert in research design.

Why I recommend them

Theses are designed to be read by someone who is not an expert in the field. Therefore, a thesis will typically organize a testing problem–including the most relevant research papers–in a way that a student or a mid-level testing practitioner can understand.

Of course, theses vary in quality. Some are written poorly. Some are researched poorly. Many present half-baked ideas (this is student work, not the work of an experienced practitioner or a professional researcher). But overall, I have found them good starting points when I start working in a new area or when I assign a student to an area that is new to her.

What’s in the bibliography

We have over 700 references, most before 2007.

Each reference includes the basic bibliographic information (author, title, etc.). It also includes:

• a URL.
  • If we found a free copy of the thesis online, we point to that.
  • If not, then if the thesis is listed in WorldCat, we point to that. WorldCat indexes many of the world’s public libraries. If your public or university library is on the Interlibrary Loan system, WorldCat will tell your reference librarian what library has a copy of the thesis, so you can borrow it. Interlibrary Loans are often free to the borrower. It’s not as convenient as free-on-the-web, but it’s still free.
  • If it’s not listed on WorldCat, we point to ProQuest (we often point to ProQuest in the notes as well). You might know this branch of ProQuest as University Microfilms. You can order dissertations from ProQuest but this is not always free (PQDT Open and ProQuest with Google Scholar publish some dissertations for free, possibly including some that we thought were available only commercially). Prices vary. I think $39 is a typical number. Because theses are of variable quality, I strongly suggest that you preview as much as you can (you can often download a chapter for free from ProQuest) or read an article that summarizes the thesis (see next section) before paying $39.
• a related reference
  • Students often publish an academic paper that summarizes their thesis. For example, Computer Science theses are often summarized in papers published through the Association for Computing Machinery, or the Institute of Electrical and Electronics Engineers, the Human Factors and Ergonomics Society, or the American Society for Quality. If we found a paper that appears to be based on a thesis, we point to it in the thesis’ Notes in Zotero.
• an abstract (a short summary of the thesis)
  • We chose not to copy abstracts from Dissertations Abstracts (ProQuest) because we don’t want to risk a copyright fight. If we found a copy of a thesis online or if an author posted a copy of their thesis abstract online, we copied that abstract into the bibliographic record for the thesis.

Digging up this extra information takes a lot of time and painstaking work. We’re continuing to add more recent work, and expect to grow the collection significantly over the summer.

Acknowledgments

This bibliography was created primarily by Karishma Bhatia, Casey Doran, Pat McGee, Kasey Powers, Andy Tinkham, and Patricia Terol Tolsa.

This bibliography is a product of research that was supported by NSF Grants EIA-0113539 ITR/SY+PE: “Improving the Education of Software Testers” and CCLI-0717613 “Adaptation & Implementation of an Activity-Based Online or Hybrid Course in Software Testing.” Any opinions, findings and conclusions or recommendations expressed in this material are those of the author(s) and do not necessarily reflect the views of the National Science Foundation.

Juha Itkonen will be defending his dissertation on “Empirical Studies on Exploratory Software Testing” this Friday. I haven’t read the entire document, but what I have read looks very interesting.

Juha has been studying real-world approaches to software testing for about a decade (maybe longer–when I met him almost a decade ago, his knowledge of the field was quite sophisticated). I’m delighted to see this quality of academic work and wish him well in his final oral exam.

For a list of soon-to-be-Dr. Itkonen’s publications, see https://wiki.aalto.fi/display/~jitkonen@aalto.fi/Full+list+of+publications.

Becky and I are getting closer to rolling out Test Design. Here’s our current summary of the course:

Learning Objectives for Test Design

This is an introductory survey of test design. The course introduces students to:

• Many (over 100) test techniques at a superficial level (what the technique is).
• A detailed-level of familiarity with a few techniques:
  • function testing
  • testing tours
  • risk-based testing
  • specification-based testing
  • scenario testing
  • domain testing
  • combination testing.
• Ways to compare strengths of different techniques and select complementary techniques to form an effective testing strategy
• Using the Heuristic Test Strategy Model for specification analysis and risk analysis
• Using concept mapping tools for test planning.

I’m still spending about 6 hours per night on video edits, but our most important work is on the assessments. To a very large degree, my course designs are driven by my assessments. That’s because there’s such a strong conviction in the education community–which I share–that students learn much more from the assessments (from all the activities that demand that they generate stuff and get feedback on it) than from lectures or informal discussions. The lectures and slides are an enabling backdrop for the students’ activities, rather than the core of the course.

In terms of design decisions, deciding what I will hold my students accountable for knowing requires me to decide what I will hold myself accountable for teaching well.

If you’re intrigued by that way of thinking about course design, check out:

• Angelo & Cross (1993, 2nd Ed.), Classroom Assessment Techniques: A Handbook for College Teachers. Josey-Bass.
• Wiggins & McTighe (2005, 2nd Ed.), Understanding by Design. Prentice Hall

I tested the course’s two main assignments in university classrooms several times before starting on the course slides (and wrote 104 first-draft multiple-quess questions and maybe 200 essay questions). But now that the course content is almost complete, we’re revisiting (and of course rewriting) these materials. In the process, we’ve been gaining perspective.

I think the most striking feature of the new course is its emphasis on content.

Let me draw the contrast with a chart that compares the BBST courses (Foundations, Bug Advocacy, and Test Design) and some other courses still on the drawing boards:

A few definitions:

• Course Skills: How to be an effective student. Working effectively in online courses. Taking tests. Managing your time.
• Social Skills: Working together in groups. Peer reviews. Using collaboration tools (e.g. wikis).
• Learning Skills: How to gather, understand, organize and be able to apply new information. Using lectures, slides, and readings effectively. Searching for supplementary information. Using these materials to form and defend your own opinion.
• Testing Knowledge: Definitions. Facts and concepts of testing. Structures for organizing testing knowledge.
• Testing Skills: How to actually do things. Getting better (through practice and feedback) at actually doing them.
• Computing Fundamentals: Facts and concepts of computer science and computing-relevant discrete mathematics.

As we designed the early courses, Becky Fiedler and I placed a high emphasis on course skills and learning skills. Students needed to (re)learn how to get value from online video instruction, how to take tests, how to give peer-to-peer feedback, etc.

The second course, Bug Advocacy, emphasizes specific testing skills–but the specific skills are the ones involved in bug investigation and reporting. Even though these critical thinking, research, and communication skill have strong application to testing, they are foundational for knowledge-related work.

Test Design is much more about the content (testing knowledge). We survey (depends on how you count) 70 to 150 test techniques. We look for ways to compare and contrast them. We consider how to organize projects around combinations of a few techniques that complement each other (make up for each other’s weaknesses and blindnesses). The learning skills component is active reading–This is certainly generally useful, but its context and application is specification analysis.

Test Design is more like the traditional Software Testing Course firehose. Way too much material in way too little time, with lots of reference material to help students explore the underemphasized parts of the course when they need it on the job.

The difference is that we are relying on the students’ improved learning skills. The assignments are challenging. The labs are still works-in-progress and might not be polished until the third iteration of the course, but labs-plus-assignments being home a bunch of lessons.

Whether the students’ skills are advanced enough to process over 500 slides efficiently, integrate material across sections, integrate required readings, and apply them to software — all within the course’s 4-week timeframe — remains to be seen.

This post is partially based on work supported by NSF research grant CCLI-0717613 ―Adaptation & Implementation of an Activity-Based Online or Hybrid Course in Software Testing. Any opinions, findings and conclusions or recommendations expressed in this post are those of the author and do not necessarily reflect the views of the National Science Foundation.

As I finish up the post-production and assessment-design for the Test Design course, I’m writing these articles as a set of retrospectives on the instructional design of the series.

For me, this is a transition point. The planned BBST series is complete with lessons to harvest as we create courseware on software metrics, development of skills with specific test techniques, computer-related law/ethics (in progress), cybersecurity, research methods and instrumentation applied to quantitative finance, qualitative research methods, and analysis of requirements.

Instructional Design

I think course design is a multidimensional challenge, focused around seven core questions:

1. Content: What information, or types of information do I want the students to learn?
2. Skills: What skills do I want the students to develop?
3. Level of Learning: What level of depth do I want to students to learn this material at?
4. Learning activities: How will the course’s activities (including the assessments, such as assignments and exams) that I use support the students’ learning?
5. Instructional Technologies: What technologies will I use to support the course?
6. Assessment: How will I assess the course: How will I find out what the students have learned, and at what level?
7. Improvement: How will I use the assessment results to guide my improvement of the course?

This collection of articles will probably skip around in these questions as I take inventory of my last 7 years of notes on online and hybrid course development.

Objectives of the BBST Courses

The changing nature of the objectives of the BBST courses.

Courses differ in more than content. They differ in the other things you hope students learn along with the content.

The BBST courses include a variety of types of assessment: quizzes, labs, assignments and exams. For instructional designers, the advantage of assessments is that we can find out what the students know (and don’t know) and what they can apply.

The BBST courses have gone through several iterations. Becky Fiedler and I used the performance data to guide the evolutions.

Based on what we learned, we place a higher emphasis in the early courses on course skills and learning skills and a greater emphasis in the later courses on testing skills.

A few definitions:

• Course Skills: How to be an effective student. Working effectively in online courses. Taking tests. Managing your time.
• Social Skills: Working together in groups. Peer reviews. Using collaboration tools (e.g. wikis).
• Learning Skills: How to gather, understand, organize and be able to apply new information. Using lectures, slides, and readings effectively. Searching for supplementary information. Using these materials to form and defend your own opinion.
• Testing Knowledge: Definitions. Facts and concepts of testing. Structures for organizing testing knowledge.
• Testing Skills: How to actually do things. Getting better (through practice and feedback) at actually doing them.
• Computing Fundamentals: Facts and concepts of computer science and computing-relevant discrete mathematics.

You can see the evolution of emphasis in the course’s specific learning objectives.

Learning Objectives of the 3-Course BBST Set

• Understand key testing challenges that demand thoughtful tradeoffs by test designers and managers.
• Develop skills with several test techniques.
• Choose effective techniques for a given objective under your constraints.
• Improve the critical thinking and rapid learning skills that underlie good testing.
• Communicate your findings effectively.
• Work effectively online with remote collaborators.
• Plan investments (in documentation, tools, and process improvement) to meet your actual needs.
• Create work products that you can use in job interviews to demonstrate testing skill.

Learning Objectives for the First Course (Foundations)

This is the first of the BBST series. We address:

• How to succeed in online classes
• Fundamental concepts and definitions
• Fundamental challenges in software testing

Improve academic skills

• Work with online collaboration tools
  • Forums
  • Wikis
• Improve students’ precision in reading
• Create clear, well-structured communication
• Provide (and accept) effective peer reviews
• Cope calmly and effectively with formative assessments (such as tests designed to help students learn).

Learn about testing

• Key challenges of testing
  • Information objectives drive the testing mission and strategy
  • Oracles are heuristic
  • Coverage is multidimensional
  • Complete testing is impossible
  • Measurement is important, but hard
• Introduce you to:Basic vocabulary of the field
  • Basic facts of data storage and manipulation in computing
  • Diversity of viewpoints
  • Viewpoints drive vocabulary

Learning Objectives for the Second Course (Bug Advocacy)

Bug reports are not just neutral technical reports. They are persuasive documents. The key goal of the bug report author is to provide high-quality information, well written, to help stakeholders make wise decisions about which bugs to fix.

Key aspects of the content of this course include:

• Defining key concepts (such as software error, quality, and the bug processing workflow)
• The scope of bug reporting (what to report as bugs, and what information to include)
• Bug reporting as persuasive writing
• Bug investigation to discover harsher failures and simpler replication conditions
• Excuses and reasons for not fixing bugs
• Making bugs reproducible
• Lessons from the psychology of decision-making: bug-handling as a multiple-decision process dominated by heuristics and biases.
• Style and structure of well-written reports

Our learning objectives include this content, plus improving your abilities / skills to:

• Evaluate bug reports written by others
• Revise / strengthen reports written by others
• Write more persuasively (considering the interests and concerns of your audience)
• Participate effectively in distributed, multinational workgroup projects that are slightly more complex than the one in Foundations

Learning Objectives for the Third Course (Test Design)

This is an introductory survey of test design. The course introduces students to:

• Many (nearly 200) test techniques at a superficial level (what the technique is).
• A detailed-level of familiarity with a few techniques:
  • function testing
  • testing tours
  • risk-based testing
  • specification-based testing
  • scenario testing
  • domain testing
  • combination testing.
• Ways to compare strengths of different techniques and select complementary techniques to form an effective testing strategy
• Using the Heuristic Test Strategy Model for specification analysis and risk analysis
• Using concept mapping tools for test planning.

 
This post is partially based on work supported by NSF research grant CCLI-0717613 ―Adaptation & Implementation of an Activity-Based Online or Hybrid Course in Software Testing. Any opinions, findings and conclusions or recommendations expressed in this post are those of the author and do not necessarily reflect the views of the National Science Foundation.

Back in 2004, I started developing course videos on software testing and computer-related law/ethics. Originally, these were for my courses at Florida Tech, but I published them under a Creative Commons license so that people could incorporate the materials in their own courses.

Soon after that, a group of us (mainly, I think, Scott Barber, Doug Hoffman, Mike Kelly, Pat McGee, Hung Nguyen, Andy Tinkham, and Ben Simo) started planning the repurposing of the academic course videos for professional development. I put together some (failing) prototypes and Becky Fiedler took over the instructional design.

• We published the first version of BBST-Foundations and taught the courses through AST (Association for Software Testing). It had a lot of rough edges, but people liked it at lot.
• So Becky and I created course #2, Bug Advocacy, with a lot of help from Scott Barber (and many other colleagues). This was new material, a more professional effort than Foundations, but it took a lot of time.

That took us to a fork in the road.

• I was working with students on developing skills with specific techniques (I worked with Giri Vijayaraghavan and Ajay Jha on risk-based testing; Sowmya Padmanabhan on domain testing; and several students with not-quite-successful efforts on scenario testing). Sowmya and I proved (not that we were trying to) that developing students’ testing skills was more complex than I’d been thinking. So, Becky and I were rethinking our skills-development-course designs.
• On the other hand, AST’s Board wanted to pull together a “complete” introductory series in black box testing. Ultimately, we went that way.

The goal was a three-part series:

1. A reworked Foundations that fixed many of the weaknesses of Version 1. We completed that one about a year ago (Becky and Doug Hoffman were my main co-creators, with a lot of design guidance from Scott Barber).
2. Bug Advocacy, and
3. a new course in Test Design.

Test Design is (finally) almost done (many thanks to Michael Bolton and Doug Hoffman). I’ll publish the lectures as we finish post-production on the videos. Lecture 1 should be up around Saturday.

Test Design is a survey course. We cover a lot of ground. And we rely heavily on references, because we sure don’t know everything there is to know about all these techniques.

To support the release of the videos, I’m publishing our references now. (The final course slides will have the references too, but those won’t be done until we complete editing the last video.):

• As always, it has been tremendously valuable reading books and papers suggested by colleagues and rereading stuff I’ve read before. A lot of careful thinking has gone into the development and analysis of these techniques.
• As always, I’ve learned a lot from people whose views differ strongly from my own. Looking for the correctness in their views–what makes them right, within their perspective and analysis, even if I disagree with that perspective–is something I see as a basic professional skill.
• And as always, I’ve not only learned new things: I’ve discovered that several things I thought I knew were outdated or wrong. I can be confident that the video is packed with errors–but plenty fewer than there would have been a year ago and none that I know about now.

So… here’s the reference list. Video editing will take a few weeks to complete–if you think we should include some other sources, please let me know. I’ll read them and, if appropriate, I’ll gladly include them in the list.

Active reading (see also Specification-based testing and Concept mapping)

• Adler, M. (1940). How to mark a book. http://academics.keene.edu/tmendham/documents/AdlerMortimerHowToMarkABook_20060802.pdf
• Adler, M., & Van Doren, C. (1972). How to Read a Book. Touchstone.
• Beacon Learning Center. (Undated). Just Read Now. http://www.justreadnow.com/strategies/active.htm
• Active reading (summarizing Bean, J. Engaging Ideas). (Undated). http://titan.iwu.edu/~writcent/Active_Reading.htm
• Gause, D.C., & Weinberg, G.M. (1989). Exploring Requirements: Quality Before Design. Dorset House.
• W.D. Hurley (1989) “A generative taxonomy of application domains based on interaction semantics” at http://dl.acm.org/citation.cfm?id=75960. (see also Jha’s and Vijayaraghavan’s writing on generative taxonomy in the section on Failure Mode Analysis.)
• PLAN: Predict/Locate/Add/Note. (Undated). http://www.somers.k12.ny.us/intranet/reading/PLAN.html
• MindTools. (Undated). Essential skills for an excellent career. http://www.mindtools.com
• Penn State University Learning Centers. (Undated). Active Reading. http://istudy.psu.edu/FirstYearModules/Reading/Materials.html
• Weill, P. (Undated). Reading Strategies for Content Areas: Part 1 After Reading. http://www.douglasesd.k12.or.us/LinkClick.aspx?fileticket=6FrvrocjBnk%3d&​tabid=1573

All-pairs testing

See http://www.pairwise.org/ for more references generally and http://www.pairwise.org/tools.asp for a list of tools.

• Bach, J., & Schroeder, P. (2004). Pairwise Testing: A Best Practice that Isn’t. Proceedings of the 22nd Pacific Northwest Software Quality Conference, 180–196. http://www.testingeducation.org/wtst5/​PairwisePNSQC2004.pdf. For Bach’s tool, see http://www.satisfice.com/tools/pairs.zip
• Bolton, M. (2007). Pairwise testing. http://www.developsense.com/pairwiseTesting.html
• Chateauneuf, M. (2000). Covering Arrays. Ph.D. Dissertation (Mathematics). Michigan Technological University.
• Cohen, D. M., Dalal, S. R., Fredman, M. L., & Patton, G. C. (1997). The AETG system: An approach to testing based on combinatorial design. IEEE Transactions on Software Engineering, 23(7). http://aetgweb.argreenhouse.com/papers/1997-tse.html. For more references, see http://aetgweb.argreenhouse.com/papers.shtml
• Czerwonka, J. (2008). Pairwise testing in the real world: Practical extensions to test-case scenarios. http://msdn.microsoft.com/en-us/library/cc150619.aspx. Microsoft’s PICT tool is at http://download.microsoft.com/download/f/5/5/f55484df-8494-48fa-8dbd-8c6f76cc014b/pict33.msi
• Jorgensen, P. (2008, 3rd Ed.). Software Testing: A Craftsman’s Approach. Auerbach Publications.
• Kuhn, D. R. & Okun, V. (2006). Pseudo-exhaustive testing for software. 30th Annual IEEE/NASA Software Engineering Workshop. http://csrc.nist.gov/acts/PID258305.pdf
• Zimmerer, P. (2004). Combinatorial testing experiences, tools, and solutions. International Conference on Software Testing, Analysis & Review (STAR West). http://www.stickyminds.com

Alpha testing

See references on tests by programmers of their own code, or on relatively early testing by development groups. For a good overview from the viewpoint of the test group, see Schultz, C.P., Bryant, R., & Langdell, T. (2005). Game Testing All in One. Thomson Press

Ambiguity analysis (See also specification-based testing)

• Bender, R. (Undated). The Ambiguity Review Process, http://benderrbt.com/Ambiguityprocess.pdf
• Berry, D.M., Kamisties, E., & Krieger, M.M. (2003) From contract drafting to software specification: Linguistic sources of ambiguity. http://se.uwaterloo.ca/~dberry/handbook/ambiguityHandbook.pdf
• Fabbrini, F., Fusani, M., Gnesi, S., & Lami, G. (2000) Quality evaluation of software requirements specifications. Thirteenth International Software & Internet .Quality Week. http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.98.4333&rep=rep1&type=pdf
• Spector, C.C. (1997) Saying One Thing, Meaning Another, Eau Claire, WI: Thinking Publications (reprint at http://www.superduperinc.com/products/view.aspx?pid=tpx12901)
• Spector, C.C. (2001). As Far As Words Go: Activities for Understanding Ambiguous Language and Humor, Baltimore: Brookes Publishing.

Best representative testing (See domain testing)

Beta testing

• Bolton, M. (2001). Effective beta testing. http://www.developsense.com/EffectiveBetaTesting.html
• Fine, M.R. (2002). Beta Testing for Better Software. Wiley.
• Schultz, C.P., Bryant, R., & Langdell, T. (2005). Game Testing All in One. Thomson Press.
• Spolsky, J. (2004). Top twelve tips for running a beta test. http://www.joelonsoftware.com/articles/BetaTest.html
• Wilson, R. (2005). Expert user validation testing. Unpublished manuscript.

Boundary testing (See domain testing)

Bug bashes

• Berkun, S. (2008). How to run a bug bash. http://www.scottberkun.com/blog/2008/how-to-run-a-bug-bash/
• Powell, C. (2009). Bug bash. http://blog.abakas.com/2009/01/bug-bash.html
• Schroeder, P.J. & Rothe, D. (2007). Lessons learned at the stomp. Conference of the Association for Software Testing, http://www.associationforsoftwaretesting.org/?dl_name=Lessons_Learned_at_the_Stomp.pdf

Build verification

• Guckenheimer, S. & Perez, J. (2006). Software Engineering with Microsoft Visual Studio Team System. Addison Wesley.
• Page, A., Johnston, K., & Rollison, B.J. (2009). How We Test Software at Microsoft. Microsoft Press.
• Raj, S. (2009). Maximize your investment in automation tools. Software Testing Analysis & Review. http://www.stickyminds.com

Calculations

Note: There is a significant, relevant field: Numerical Analysis. The list here merely points you to a few sources I have personally found helpful, not necessarily to the top references in the field.

• Arsham, H. (2010) Solving system of linear equations with application to matrix inversion. http://home.ubalt.edu/ntsbarsh/business-stat/otherapplets/SysEq.htm
• Boisvert, R.F., Pozo, R., Remington, K., Barrett , R.F., & Dongarra, J.J. (1997) Matrix Market: A web resource for test matrix collections. In Boisvert, R.F. (1997) (Ed.) Quality of Numerical Software: Assessment and Enhancement. Chapman & Hall.
• Einarsson, B. (2005). Accuracy and Reliability in Scientific Computing. Society for Industrial and Applied Mathematics (SIAM).
• Gregory, R.T. & Karney, D.L. (1969). A Collection of Matrices for Testing Computational Algorithms. Wiley.
• Kaw, A.K. (2008), Introduction to Matrix Algebra. Available from http://www.lulu.com/browse/preview.php?fCID=2143570. Chapter 9, Adequacy of Solutions, http://numericalmethods.eng.usf.edu/mws/gen/04sle/mws_gen_sle_spe_adequacy.pdf

Combinatorial testing. See All-Pairs Testing

Concept mapping

• Hyerle, D.N. (2008, 2nd Ed.). Visual Tools for Transforming Information into Knowledge, Corwin.
• Margulies, N., & Maal, N. (2001, 2nd Ed.) Mapping Inner Space: Learning and Teaching Visual Mapping. Corwin.
• McMillan, D. (2010). Tales from the trenches: Lean test case design. http://www.bettertesting.co.uk/content/?p=253
• McMillan, D. (2011). Mind Mapping 101. http://www.bettertesting.co.uk/content/?p=956
• Moon, B.M., Hoffman, R.R., Novak, J.D., & Canas, A.J. (Eds., 2011). Applied Concept Mapping: Capturing, Analyzing, and Organizing Knowledge. CRC Press.
• Nast, J. (2006). Idea Mapping: How to Access Your Hidden Brain Power, Learn Faster, Remember More, and Achieve Success in Business. Wiley.
• Sabourin, R. (2006). X marks the test case: Using mind maps for software design. Better Software. http://www.stickyminds.com/BetterSoftware/magazine.asp?fn=cifea&id=90

Concept mapping tools:

• http://www.graphic.org/
• http://www.innovationtools.com/resources/mindmapping.asp
• 
  http://www.inspiration.com
• 
  http://www.mindjet.com
• 
  http://www.mindtools.com/mindmaps.html
• 
  http://www.novamind.com
• 
  http://users.edte.utwente.nl/lanzing/cm_bibli.htm
• 
  http://www.xmind.net
• 
  http://en.wikipedia.org/wiki/List_of_concept_mapping_and_mind_mapping_software

Configuration coverage

• Black, R. (2002, 2nd Ed.). Managing the Testing Process. Wiley.
• Kaner, C. (1996). Software negligence and testing coverage. Software Testing, Analysis & Review Conference (STAR). https://kaner.com/pdfs/negligence_and_testing_coverage.pdf
• Pawson, M. (2001). The test matrix. Software Testing & Quality Engineering, 3(6). http://www.stickyminds.com/getfile.asp?ot=XML&id=5005&fn=Smzr1XDD3339filelistfilename1%2Epdf

Configuration / compatibility testing

• Frye, C. (2009). Configuration testing: QA pros discuss 10 things you may not know. http://searchsoftwarequality.techtarget.com/news/1361051/Configuration-testing-QA-pros-discuss-10-things-you-may-not-know
• Kaner, C., Falk, J., & Nguyen, H.Q. (2nd Edition, 2000). Testing Computer Software. Wiley.
• McCaffrey, J., & Despe, P. (2008). Configuration testing with virtual server, part 2. MSDN Magazine. December. http://msdn.microsoft.com/en-us/magazine/dd252952.aspx
• Patton, Ron. (2006, 2nd Ed.). Software Testing. SAMS.

Constraint checks

See our notes in BBST Foundation’s presentation of Hoffman’s collection of oracles.

• Hoffman, D. (1999). Heuristic test oracles. Software Testing & Quality Engineering, 1(2), 29-32. http://www.softwarequalitymethods.com/Papers/STQE%20Heuristic.pdf

Constraints

• Jorgensen, A.A. (1999). Software Design Based on Operational Modes. Doctoral Dissertation, Florida Institute of Technology. https://cs.fit.edu/Projects/tech_reports/cs-2002-10.pdf
• Whittaker, J.A. (2002) How to Break Software, Addison Wesley.

Diagnostics-based testing

• Al-Yami, A.M. (1996). Fault-Oriented Automated Test Data Generation. Ph.D. Dissertation, Illinois Institute of Technology.
• Kaner, C., Bond, W.P., & McGee, P.(2004). High volume test automation. Keynote address: International Conference on Software Testing Analysis & Review (STAR East 2004). Orlando. https://kaner.com/pdfs/HVAT_STAR.pdf (The Telenova and Mentsville cases are both examples of diagnostics-based testing.)

Domain testing

• Abramowitz & Stegun (1964), Handbook of Mathematical Functions. http://people.math.sfu.ca/~cbm/aands/frameindex.htm
• Beizer, B. (1990). Software Testing Techniques (2nd Ed.). Van Nostrand Reinhold.
• Beizer, B. (1995). Black-Box Testing. Wiley.
• Binder, R. (2000). Testing Object-Oriented Systems: Addison-Wesley.
• Black, R. (2009). Using domain analysis for testing. Quality Matters, Q3, 16-20. http://www.rbcs-us.com/images/documents/quality-matters-q3-2009-rb-article.pdf
• Copeland, L. (2004). A Practitioner’s Guide to Software Test Design. Artech House.
• Clarke, L.A. (1976). A system to generate test data and symbolically execute programs. IEEE Transactions on Software Engineering, 2, 208-215.
• Clarke, L. A. Hassel, J., & Richardson, D. J. (1982). A close look at domain testing. IEEE Transactions on Software Engineering, 2, 380-390.
• Craig, R. D., & Jaskiel, S. P. (2002). Systematic Software Testing. Artech House.
• Hamlet, D. & Taylor, R. (1990). Partition testing does not inspire confidence. IEEE Transactions on Software Engineering, 16(12), 1402-1411.
• Hayes, J.H. (1999). Input Validation Testing: A System-Level, Early Lifecycle Technique. Ph.D. Dissertation (Computer Science), George Mason University.
• Howden, W. E. (1980). Functional testing and design abstractions. Journal of Systems & Software, 1, 307-313.
• Jeng, B. & Weyuker, E.J. (1994). A simplified domain-testing strategy. ACM Transactions on Software Engineering, 3(3), 254-270.
• Jorgensen, P. C. (2008). Software Testing: A Craftsman’s Approach (3rd ed.). Taylor & Francis.
• Kaner, C. (2004a). Teaching domain testing: A status report. Paper presented at the Conference on Software Engineering Education & Training. https://kaner.com/pdfs/teaching_sw_testing.pdf
• Kaner, C., Padmanabhan, S., & Hoffman, D. (2012) Domain Testing: A Workbook, in preparation.
• Myers, G. J. (1979). The Art of Software Testing. Wiley.
• Ostrand, T. J., & Balcer, M. J. (1988). The category-partition method for specifying and generating functional tests. Communications of the ACM, 31(6), 676-686.
• Padmanabhan, S. (2004). Domain Testing: Divide and Conquer. M.Sc. Thesis, Florida Institute of Technology. http://www.testingeducation.org/a/DTD&C.pdf
• Schroeder, P.J. (2001). Black-box test reduction using input-output analysis. Ph.D. Dissertation (Computer Science). Illinois Institute of Technology.
• Weyuker, E. J., & Jeng, B. (1991). Analyzing partition testing strategies. IEEE Transactions on Software Engineering, 17(7), 703-711.
• Weyuker, E.J., & Ostrand, T.J. (1980). Theories of program testing and the application of revealing subdomains. IEEE Transactions on Software Engineering, 6(3), 236-245.
• White, L. J., Cohen, E.I., & Zeil, S.J. (1981). A domain strategy for computer program testing. In Chandrasekaran, B., & Radicchi, S. (Ed.), Computer Program Testing (pp. 103-112). North Holland Publishing.
• http://www.wikipedia.org/wiki/Stratified_sampling

Dumb monkey testing

• Arnold, T. (1998), Visual Test 6. Wiley.
• Nyman, N. (1998). Application testing with dumb monkeys. International Conference on Software Testing Analysis & Review (STAR West).
• Nyman, N. (2000), Using monkey test tools. Software Testing & Quality Engineering, 2(1), 18-20
• Nyman, N. (2004). In defense of monkey testing. http://www.softtest.org/sigs/material/nnyman2.htm

Eating your own dogfood

• Page, A., Johnston, K., & Rollison, B.J. (2009). How We Test Software at Microsoft. Microsoft Press.

Equivalence class analysis (see Domain testing)

Experimental design

• Popper, K.R. (2002, 2nd Ed.). Conjectures and Refutations: The Growth of Scientific Knowledge. Routledge.
• Shadish, W.R., Cook, T.D., & Campbell, D.T. (2002). Experimental and Quasi-Experimental Designs for Generalized Causal Inference, 2nd Ed. Wadsworth.

Exploratory testing

• Bach, J. (1999). General functionality and stability test procedure. http://www.satisfice.com/tools/procedure.pdf
• Bach, J. (2000). Session-based test management. Software Testing & Quality Engineering. http://www.satisfice.com/articles/sbtm.pdf
• Bach, J. (2007). Exploratory testing explained. In Ryber,. T. (2007). Essential Software Test Design. Fearless Consulting. http://www.satisfice.com/articles/et-article.pdf 
• Bach, J., Bach, J. & Bolton, M. (2009). Exploratory testing dynamics (version 2.2). http://www.satisfice.com/blog/wp-content/uploads/2009/10/et-dynamics22.pdf
• Bolton, M. (2011). Resources on Exploratory Testing, Metrics and Other Stuff. http://www.developsense.com/resources.html#exploratory
• Cox, R., Duisters, P., & van de Laar (2011). Testing in the medical domain. Testing Experience, (March),6-9. http://www.improveqs.nl/files/testingexperience13_03_11_Cox_Duisters_Laar.pdf
• Hendrickson, E. (2011). Exploratory Testing in an Agile Context. http://www.agilistry.com/downloads/ETinAgile-agile2011-final.pdf
• Kaner, C. (2006). Exploratory testing after 23 years. Conference of the Association for Software Testing. https://kaner.com/pdfs/ETat23.pdf
• Kaner, C. & Hoffman, D. (2010). Introduction to exploratory test automation. https://kaner.com/pdfs/VISTACONexploratoryTestAutmation.pdf
• Kohl, J. (2007). Getting started with exploratory testing –Parts 1-4. http://www.kohl.ca/blog/archives/000185.html
• Kohl, J. (2007). Getting started with exploratory testing –Part 2. http://www.kohl.ca/blog/archives/000186.html
• Kohl, J. (2007). Getting started with exploratory testing –Part 3. http://www.kohl.ca/blog/archives/000187.html
• Kohl, J. (2007). Getting started with exploratory testing –Part 4. http://www.kohl.ca/blog/archives/000188.html
• Kohl, J. (2007). Man and machine: Combining the power of the human mind with automation tools. Better Software, December, 20-25. http://www.kohl.ca/articles/ManandMachine_BetterSoftware_Dec2007.pdf
• Kohl, J. (2011). Documenting exploratory testing. Better Software, May/June, 22-25. http://www.nxtbook.com/nxtbooks/sqe/bettersoftware_0511/#/23/OnePage
• Robinson, H. (2010). Exploratory test automation. Conference of the Association for Software Testing. http://www.harryrobinson.net/ExploratoryTestAutomation-CAST.pdf

Failure mode analysis: see also Guidewords and Risk-Based Testing.

• Cheit, R.E. (1990). Setting Safety Standards: Regulation in the Public and Private Sectors. University of California Press. http://ark.cdlib.org/ark:/13030/ft8f59p27j/
• Department of Defense (1980). Procedures for Performing a Failure Mode, Effects and Criticality Analysis: MIL-STD-1629A. http://sre.org/pubs/Mil-Std-1629A.pdf
• Department of Defense Patient Safety Center (2004). Failure mode and effects analysis (FMEA): An advisor’s guide. AF Patient Safety Program. http://www.fmeainfocentre.com/handbooks/FMEA_Guide_V1.pdf
• FMEA Info Center (undated). FMEA Info Center: Everything You Want to Know About Failure Mode and Effect Analysis. http://www.fmeainfocentre.com/index.htm (this is a large collection of resources on FMEA).
• FMEA-FMECA.COM (undated). FMEA Examples. http://fmea-fmeca.com/fmea-examples.html
• Goddard, P.L. (2000). Software FMEA techniques. Proceedings of the Reliability and Maintainability Symposium, 118-123.
• W.D. Hurley (1989) “A generative taxonomy of application domains based on interaction semantics” at http://dl.acm.org/citation.cfm?id=75960
• Jha, A. (2007). A Risk Catalog for Mobile Applications. (Master’s Thesis in Software Engineering) Department of Computer Sciences at Florida Institute of Technology. http://www.testingeducation.org/articles/AjayJha_Thesis.pdf
• Kaner, C., Falk, J., & Nguyen, H.Q. (2nd Edition, 2000b). Bug Taxonomy (Appendix) in Testing Computer Software. Wiley. http://www.logigear.com/logi_media_dir/Documents/whitepapers/Common_Software_Errors.pdf
• Pentti, H. & Atte, H. (2002). Failure mode and effects analysis of software-based automation systems. http://www.fmeainfocentre.com/handbooks/softwarefmea.pdf
• SoftRel (undated) Software FMEA service. http://www.softrel.com/fmea.htm
• Vijayaraghavan, G. (2002). A Taxonomy of E-Commerce Risks and Failures. (Master’s Thesis) Department of Computer Sciences at Florida Institute of Technology. http://www.testingeducation.org/a/tecrf.pdf
• Vijayaraghavan, G., & Kaner, C. (2002). Bugs in your shopping cart: A taxonomy. 15th International Software Quality Conference (Quality Week). San Francisco, CA. (Best Paper Award.) http://www.testingeducation.org/a/bsct.pdf
• Vijayaraghavan, G., & Kaner, C.(2003). Bug taxonomies: Use them to generate better tests. Software Testing, Analysis & Review Conference (Star East). Orlando, FL. (Best Paper Award). http://www.testingeducation.org/a/bugtax.pdf
• http://en.wikipedia.org/wiki/Taxonomy
• http://en.wikipedia.org/wiki/Folk_taxonomy

Feature integration testing

• Overbaugh, J. (2007). How to do integration testing. http://searchsoftwarequality.techtarget.com/answer/How-to-do-integration-testing
• Van Tongeren, T. (2001). Functional integration test planning. http://www.stickyminds.com/getfile.asp?ot=XML&id=3693&fn=XUS2004669file1%2Epdf

Function testing

• Bolton, M. (2006). The factors of function testing. Better Software. http://www.developsense.com/articles/2006-07-TheFactorsOfFunctionTesting.pdf
• Craig, R.D., & Jaskiel, S.P. (2002). Systematic Software Testing. See Chapter 5, Test Design (the discussion of inventories). Artech House.

Function equivalence testing

• Hoffman, D. (2003). Exhausting your test options. Software Testing & Quality Engineering, 5(4), 10-11
• Kaner, C., Falk, J., & Nguyen, H.Q. (2nd Edition, 2000). Testing Computer Software. Wiley.

Functional testing below the GUI

• Marick, B. (2002). Bypassing the GUI. Software Testing & Quality Engineering, Sept/Oct. 41-47. http://www.exampler.com/testing-com/writings/bypassing-the-gui.pdf

Guerilla testing

• Kaner, C., Falk, J., & Nguyen, H.Q. (2nd Edition, 2000). Testing Computer Software. Wiley.

Guidewords

• Bach, J. (2006). Heuristic test strategy model, Version 4.8. http://www.satisfice.com/tools/satisfice-tsm-4p.pdf
• Broomfield, E.J. & Chung, P.W.H. (1994). Hazard identification in programmable systems: A methodology and case study. ACM SIGAPP Applied Computing Review, 2(1), 7-14.
• Falla, M. (Ed.) (1997). Advances in Safety Critical Systems: Results and Achievements from the DTI/EPSRC R&D Programme in Safety Critical Systems. Chapter 3: Hazard Analysis. http://www.comp.lancs.ac.uk/computing/resources/scs/
• Fenelon, P. & Hebbron, B. (1994). Applying HAZOP to software engineering models. Risk Management and Critical Protective Systems: Proceedings of SARSS 1994. http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.56.6073&rep=rep1&type=pdf
• Fenelon, P., McDermid, J.A., Nicholson, M. & Pumfrey, D.J. (1994). Towards integrated safety analysis and design. ACM SIGAPP Applied Computing Review, 2(1), 21-32. http://www-users.cs.york.ac.uk/~djp/publications/djp-acm.pdf
• Fields, R. Paterno, F., Santoro, C. & Tahmassebi, S. (1999). Comparing design options for allocating communication media in cooperative safety-critical contexts: A method and a case study. ACM Transactions on Computer-Human Interaction, 6(4), 370–398.
• Gagnat, A.K. & Jansen, V. (2006). Incorporating human factors and interface design into safety-critical systems development. M.Sc. Thesis, Østfold University College, Faculty of Computer Science, http://www.hiof.no/neted/upload/attachment/site/group12/Ann_Katrin_Gagnat_&_Vegar_Jansen_Incorporating_human_factors_and_interface_design_into_safety_critical_systems_development.pdf
• HAZOP Guidelines (2008). Hazardous Industry Planning Advisory Paper No. 8, NSW Government Department of Planning. http://www.planning.nsw.gov.au/plansforaction/pdf/hazards/haz_hipap8_rev2008.pdf
• Hewitt, R. (2005). Information-based risk assessment software architecture. Proceedings of the 2005 IEEE Engineering Management Conference, 574-578.
• Lutz, R. & Nikora, A. (2005, November). Failure Assessment. 1st Int’l Forum on Integrated System Health Engineering and Management for Aerospace (ISHEM’05). http://www.cs.iastate.edu/~rlutz/publications/ishem05.pdf
• Reinhardt, D. (2006). Certification criteria for emulation technology in the Australian Defence Force Military Avionics Context. 11th Australian Workshop on Safety Related Programmable Systems (SCS’06). Conferences in Research and Practice in Information Technology. 69, 79-92. http://crpit.com/confpapers/CRPITV69Reinhardt.pdf
• Stone, G.R. (2005). On arguing the safety of large systems. 10th Australian Workshop on Safety related Programmable Systems. Conferences in Research and Practice in Information Technology, 55, 69-75. http://crpit.com/confpapers/CRPITV55Stone.pdf
• Ye, F. & Kelly, T. (1994). Contract-based justification for COTS component within safety-critical applications. 9th Australian Workshop on Safety Related Programmable Systems (SCS’ 04). Conferences in Research & Practice in Information Technology, 47, 13-22. http://crpit.com/confpapers/CRPITV47Ye.pdf

Installation testing

• Agruss, C. (2000). Software installation testing: How to automate tests for smooth system installation. Software Testing & Quality Engineering, 2 (4). http://www.stickyminds.com/getfile.asp?ot=XML&id=5001&fn=Smzr1XDD1806filelistfilename1%2Epdf
• Bach, J. (1999), Heuristic risk-based testing, Software Testing & Quality Engineering, 1 (6), 22-29. http://www.satisfice.com/articles/hrbt.pdf
• Kumar, P. (2010). Installation testing – why and how? Testing Circus, 1(2), 7-11. http://testingcircus.com/OctoberNovember2010.aspx
• Noggle, B.J. (2011). Testing the installer. The Testing Planet. March, http://wiki.softwaretestingclub.com/w/file/fetch/39449474/TheTestingPlanet-Issue4-March2011.pdf
• Shinde, V. (2007). Software installation/uninstallation testing. http://www.softwaretestinghelp.com/software-installationuninstallation-testing

Interoperability testing

• European Telecommunications Standards Institute (undated). Interoperability test specification. http://portal.etsi.org/mbs/testing/interop/interop.htm
• Simulations Interoperability Standards Organization (2010). Commercial Off-the-Shelf (COTS) Simulation Package Interoperability (CSPI) Reference Mode. http://www.sisostds.org/DigitalLibrary.aspx?Command=Core_Download&EntryId=30829

Load testing

• Asbock, S. (2000). Load Testing for eConfidence. Segue.
• Barber, S. (2006). Remember yesterday. Software Test & Performance Magazine. January, 42-43. http://www.perftestplus.com/resources/016PeakPerf.pdf
• Savoia, A. (2000). The science and art of web site load testing. International Conference on Software Testing Analysis & Review (STAR East), Orlando. www.stickyminds.com/getfile.asp?ot=XML&id=1939&​fn=XDD1939filelistfilename1.pdf 
• Savoia, A. (2001). Three web load testing blunders and how to avoid them. Software Testing & Quality Engineering, 3(3), 54-59. http://www.stickyminds.com/s.asp?F=S5034_MAGAZINE_2

Localization testing

• Bolton, M. (2006, April). Where in the world? Better Software. http://www.developsense.com/articles/2006-04-WhereInTheWorld.pdf
• Chandler, H.M. & Deming, S.O (2nd Ed. in press). The Game Localization Handbook. Jones & Bartlett Learning.
• Ratzmann, M., & De Young, C. (2003). Galileo Computing: Software Testing and Internationalization. Lemoine International and the Localization Industry Standards Association. http://www.automation.org.uk/downloads/documentation/galileo_computing-software_testing.pdf
• Savourel, Y. (2001). XML Internationalization and Localization. Sams Press.
• Singh, N. & Pereira, A. (2005). The Culturally Customized Web Site: Customizing Web Sites for the Global Marketplace. Butterworth-Heinemann.
• Smith-Ferrier, G. (2006). .NET Internationalization: The Developer’s Guide to Building Global Windows and Web Applications. Addison-Wesley Professional.
• Uren, E., Howard, R. & Perinotti, T. (1993). Software Internationalization and Localization. Van Nostrand Reinhold.

Logical expression testing

• Amman, P., & Offutt, J. (2008). Introduction to Software Testing. Cambridge University Press.
• Beizer, B. (1990). Software Testing Techniques (2nd Ed.). Van Nostrand Reinhold.
• Copeland, L. (2004). A Practitioner’s Guide to Software Test Design. Artech House (see Chapter 5 on decision tables).
• Jorgensen, P. (2008, 3rd Ed.). Software Testing: A Craftsman’s Approach. Auerbach Publications (see Chapter 7 on decision tables).
• Brian Marick (2000) modeled testing of logical expressions by considering common mistakes in designing/coding a series of related decisions. Testing for Programmers. http://www.exampler.com/testing-com/writings/half-day-programmer.pdf.
• MULTI. Marick implemented his approach to testing logical expressions in a program, MULTI. Tim Coulter and his colleagues extended MULTI and published it (with Marick’s permission) at http://sourceforge.net/projects/multi/

Long-sequence testing

• Claessen, K. (undated). QuickCheck 2.4.0.1: Automatic testing of Haskell programs. http://hackage.haskell.org/cgi-bin/hackage-scripts/package/QuickCheck
• Koopman, P. (undated). The Ballista ® Project: COTS software robustness testing. http://www.cs.cmu.edu/afs/cs/project/edrc-ballista/www/
• McGee, P. & Kaner, C. (2004). Experiments with high volume test automation.” Workshop on Empirical Research in Software Testing, International Symposium on Software Testing and Analysis. https://kaner.com/pdfs/MentsvillePM-CK.pdf

Mathematical oracle

See our notes in BBST Foundation’s presentation of Hoffman’s collection of oracles.

• Abramowitz & Stegun (1964), Handbook of Mathematical Functions. http://people.math.sfu.ca/~cbm/aands/frameindex.htm
• Hoffman, D. (1999). Heuristic test oracles. Software Testing & Quality Engineering, 1(2), 29-32. http://www.softwarequalitymethods.com/Papers/STQE%20Heuristic.pdf

Numerical analysis (see Calculations)

Paired testing

• Kaner, C. & Bach, J. (2001). Exploratory testing in pairs. Software Testing Analysis & Review (STAR West). https://kaner.com/pdfs/exptest.pdf
• Kohl, J. (2004). Pair testing: How I brought developers into the test lab. Better Software, 6 (January), 14-16. http://www.kohl.ca/articles/pairtesting.html
• Lambert, R. (2009). Pair testing. http://www.softwaretestingclub.com/forum/topics/pair-testing

Pairwise testing (see All-Pairs testing)

Performance testing

• Barber, S. (2010) PerfTestPlus website, http://www.perftestplus.com/pubs.htm
• du Plessis, J. (2009). Resource monitoring during performance testing. http://www.stickyminds.com/getfile.asp?ot=XML&id=14959&fn=XUS231346159file1%2Epdf
• Meier, J.D., Farre, C., Bansode, P., Barber, S., & Rea, D. (2007). Performance Testing Guidance for Web Applications. Redmond: Microsoft Press.

Programming or software design

• Roberts, E. (2005, 20th Ed.). Thinking Recursively with Java. Wiley.

Psychological considerations

• Bendor, J. (2005). The perfect is the enemy of the best: Adaptive versus optimal organizational reliability. Journal of Theoretical Politics. 17(1), 5-39.
• Rohlman, D.S. (1992). The Role of Problem Representation and Expertise in Hypothesis Testing: A Software Testing Analogue. Ph.D. Dissertation, Bowling Green State University.
• Teasley, B.E., Leventhal, L.M., Mynatt, C.R., & Rohlman, D.S. (1994). Why software testing is sometimes ineffective: Two applied studies of positive test strategy. Journal of Applied Psychology, 79(1), 142-155.
• Whittaker, J.A. (2000). What is software testing? And why is it so hard? IEEE Software, Jan-Feb. 70-79.

Quicktests

• Andrews, M., & Whittaker, J.A. (2006), How to Break Web Software, Addison Wesley.
• Hendrickson, E. (2006), Test heuristics cheat sheet. http://testobsessed.com/wp-content/uploads/2011/04/testheuristicscheatsheetv1.pdf
• Hunter, M. J. (2010). You are not done yet. http://www.thebraidytester.com/downloads/YouAreNotDoneYet.pdf
• Jorgensen, A.A. (1999). Software Design Based on Operational Modes. Doctoral Dissertation, Florida Institute of Technology. https://cs.fit.edu/Projects/tech_reports/cs-2002-10.pdf
• Jorgensen, A.A. (2003). Testing with hostile data streams. ACM SIGSOFT Software Engineering Notes, 28(2). http://cs.fit.edu/media/TechnicalReports/cs-2003-03.pdf
• Kaner, C. (1988), Testing Computer Software, McGraw-Hill, presented a list of 480 common software problems. http://logigear.com/articles-by-logigear-staff/445-common-software-errors.html
• Kaner, C. & Johnson, B. (1999) Styles of exploration, 7th Los Altos Workshop on Software Testing. https://kaner.com/pdfs/LAWST7StylesOfExploration.pdf
• Nguyen, H.Q., Johnson, B., & Hackett, M. (2003, 2nd ed), Testing Applications on the Web. Wiley.
• Whittaker, J.A. (2002) How to Break Software, Addison Wesley.
• Whittaker, J.A. & Thompson, H.H. (2004). How to Break Software Security. Addison Wesley.

Random testing

• Ciupa, I., Leitner, A., Oriol, M., & Meyer, B. (2007). Experimental Assessment of Random Testing for Object-Oriented Software. International Symposium on Software Testing and Analysis. http://staff.unak.is/not/andy/MScTesting0708/Lectures/ExpRandomTestingOOEiffel.pdf
• Hamlet, D. (2006). When only random testing will do. Proceedings of the 1st International Workshop on Random Testing. http://web.cecs.pdx.edu/~hamlet/rt.pdf
• Hamlet, D. (2002) Random testing. Encyclopedia of Software Engineering. http://web.cecs.pdx.edu/~hamlet/random.pdf
• Robinson, H. (2004). Things that find bugs in the night. http://www.stickyminds.com/s.asp?F=S7331_COL_2

Regression testing

• Bach, J. (1999). Test automation snake oil. http://www.satisfice.com/articles/test_automation_snake_oil.pdf
• Buwalda, H. (undated). Key success factors for keyword-driven testing. http://logigear.com/articles-by-logigear-staff/389–key-success-factors-for-keyword-driven-testing.html
• Engstrom, E., Skoglund, M. & Runeson, P. (2008). Empirical evaluations of regression test selection techniques: A systematic review. Conference on Empirical Software Engineering and Measurement. 22-31. http://portal.acm.org/citation.cfm?doid=1414004.1414011
• Groder, C. (1999). Building maintainable GUI tests. In Fewster, M. & Graham, D. (1999). Software Test Automation. Addison-Wesley
• Harrold, M.J. & Orso, A. (2008). Retesting software during development and maintenance. Frontiers of Software Maintenance. http://pleuma.cc.gatech.edu/aristotle/pdffiles/harrold_orso_fosm08.pdf
• Hoffman, D. (2007). Avoiding the “test and test again” syndrome. Conference of the Association for Software Testing. http://www.softwarequalitymethods.com/H-Papers.html#TestTest2
• Kaner, C. (1997). Improving the maintainability of automated test suites. Software QA, (4)(4). https://kaner.com/pdfs/autosqa.pdf
• Kaner, C. (1998). Avoiding shelfware: A manager’s view of automated GUI testing.” Software Testing Analysis & Review (STAR East). https://kaner.com/pdfs/shelfwar.pdf
• Kaner, C. (2009). The value of checklists and the danger of scripts: What legal training suggests for testers. Conference of the Association for Software Testing. https://kaner.com/pdfs/ValueOfChecklists.pdf
• Leung, H.K.N. & White, L.J. (1989(. Insights into regression testing. Proceedings of the International Conference on Software Maintenance. 60-69.
• Marick, B.M. (1999). When should a test be automated? International Conference on Software Testing Analysis and Review (STAR East). http://www.exampler.com/testing-com/writings/automate.pdf
• Marick, B.M. (2005). Working your way out of the automated GUI testing tarpit (parts 1, 2, 3). http://www.testingreflections.com/node/view/3059
• Marick, B.M. (undated). How many bugs do regression tests find? http://www.sqa.fyicenter.com/art/How_Many_Bugs_Do_Regression_Tests_Find.html
• Memon, A.M. & Xie, Q. (2004) Empirical evaluation of the fault-detection effectiveness of smoke regression test cases for GUI-based software. Proceedings of the International Conference on Software Maintenance. http://www.cs.umd.edu/~atif/papers/MemonICSM2004.pdf
• Mugridge, R. & Cunningham. W. (2005) Fit for Developing Software: Framework for Integrated Tests. Prentice Hall.
• Nguyen, H.Q., Hackett, M., & Whitlock, B.K. (2006). Happy About Global Test Automation. Happy About books.
• Onoma, A.K., Tsai, W.T., Poonawala, M.H., & Sugunama, H. (1998). Regression testing in an industrial environment. Communications of the ACM. 41(5), 81-86
• Pettichord, B. (2001a). Seven steps to test automation success. http://www.io.com/~wazmo/papers/seven_steps.html
• Pettichord, B. (2001b). Success with test automation. http://www.io.com/~wazmo/succpap.htm
• Rothermel, G. & Harrold, M. (1997). Experience with regression test selection. Empirical Software Engineering, 2(2), 178-188. http://www.cc.gatech.edu/aristotle/Publications/Papers/wess96.ps

Requirements-based testing

• Bach, J. (1999). Risk and requirements-based testing. IEEE Computer, June, 113-114. http://www.satisfice.com/articles/requirements_based_testing.pdf
• Bender, R. (2009). Requirements Based Testing Process Overview. http://benderrbt.com/Bender-Requirements%20Based%20Testing%20Process%20Overview.pdf
• Culbertson, R., Brown, C., & Cobb, G. (2002). Rapid Testing. Prentice Hall.

Requirements-based testing (continued)

• Whalen, M.W., Rajan, A., Heimdahl, M.P.E., & Miller, S.P. )2006). Coverage metrics for requirements-based testing. Proceedings of the 2006 International Symposium on Software Testing and Analysis. http://portal.acm.org/citation.cfm?id=1146242
• Wiegers, K.E. (1999). Software Requirements. Microsoft Press.

Risk-based testing

• Bach, J. (1999). Heuristic risk-based testing. Software Testing & Quality Engineering. http://www.satisfice.com/articles/hrbt.pdf
• Bach, J. (2000a). Heuristic test planning: Context model. http://www.satisfice.com/tools/satisfice-cm.pdf
• Bach, J. (2000b). SQA for new technology projects. http://www.satisfice.com/articles/sqafnt.pdf
• Bach, J. (2003). Troubleshooting risk-based testing. Software Testing & Quality Engineering, May/June, 28-32. http://www.satisfice.com/articles/rbt-trouble.pdf
• Becker, S.A. & Berkemeyer, A. (1999). The application of a software testing technique to uncover data errors in a database system. Proceedings of the 20th Annual Pacific Northwest Software Quality Conference, 173-183.
• Berkovich, Y. (2000). Software quality prediction using case-based reasoning. M.Sc. Thesis (Computer Science). Florida Atlantic University.
• Bernstein, P.L. (1998). Against the Gods: The Remarkable Story of Risk. Wiley.
• Black, R. (2007). Pragmatic Software Testing: Becoming an Effective and Efficient Test Professional. Wiley.
• Clemen, R.T. (1996, 2nd ed.) Making Hard Decisions: An Introduction to Decision Analysis. Cengage Learning.
• Copeland, L. (2004). A Practitioner’s Guide to Software Test Design. Artech House.
• DeMarco, T. & Lister, T. (2003). Waltzing with Bears. Managing Risk on Software Projects. Dorset House.
• Dorner, D. (1997). The Logic of Failure. Basic Books.
• Gerrard, P. & Thompson, N. (2002). Risk-Based E-Business Testing. Artech House.
• HAZOP Guidelines (2008). Hazardous Industry Planning Advisory Paper No. 8, NSW Government Department of Planning. http://www.planning.nsw.gov.au/plansforaction/pdf/hazards/haz_hipap8_rev2008.pdf
• Hillson, D. & Murray-Webster, R. (2007). Understanding and Managing Risk Attitude. (2nd Ed.). Gower. http://www.risk-attitude.com/
• Hubbard, D.W. (2009). The Failure of Risk Management: Why It’s Broken and How to Fix It. Wiley.
• Jorgensen, A.A. (2003). Testing with hostile data streams. ACM SIGSOFT Software Engineering Notes, 28(2). http://cs.fit.edu/media/TechnicalReports/cs-2003-03.pdf
• Jorgensen, A.A. & Tilley, S.R. (2003). On the security risks of not adopting hostile data stream testing techniques. 3rd International Workshop on Adoption-Centric Software Engineering (ACSE 2003), p. 99-103. http://www.sei.cmu.edu/reports/03sr004.pdf
• Kaner, C. (2008). Improve the power of your tests with risk-based test design. Quality Assurance Institute QUEST conference. https://kaner.com/pdfs/QAIriskKeynote2008.pdf
• Kaner, C., Falk, J., & Nguyen, H.Q. (2nd Edition, 2000a). Testing Computer Software. Wiley.
• Neumann, P.G. (undated). The Risks Digest: Forum on Risks to the Public in Computers and Related Systems. http://catless.ncl.ac.uk/risks
• Perrow, C. (1999). Normal Accidents: Living with High-Risk Technologies. Princeton University Press (but read this in conjunction with Robert Hedges’ review of the book on Amazon.com).
• Petroski, H. (1992). To Engineer is Human: The Role of Failure in Successful Design. Vintage.
• Petroski, H. (2004). Small Things Considered: Why There is No Perfect Design. Vintage.
• Petroski, H. (2008). Success Through Failure: The Paradox of Design. Princeton University Press.
• Pettichord, B. (2001). The role of information in risk-based testing. International Conference on Software Testing Analysis & Review (STAR East). http://www.stickyminds.com
• Reason, J. T.  (1997). Managing the Risks of Organizational Accident. Ashgate Publishing.
• Schultz, C.P., Bryant, R., & Langdell, T. (2005). Game Testing All in One. Thomson Press (discussion of defect triggers).
• Software Engineering Institute’s collection of papers on project management, with extensive discussion of project risks. https://seir.sei.cmu.edu/seir/
• Weinberg, G. (1993). Quality Software Management. Volume 2: First Order Measurement. Dorset House.

Rounding errors (see Calculations)

Scenario testing (See also Use-case-based testing)

• Anggreeni, I., & van der Voort, M. (2007) Tracing the scenarios in scenario-based product design: A study to support scenario generation. Technical Report TR-CTIT-07-70, Centre for Telematics and Information Technology, University of Twente, Enschede. ISSN 1381-3625. http://eprints.eemcs.utwente.nl/11231/01/TR-CTIT-07-70.pdf
• Alexander, D. (2000). Scenario methodology for teaching principles of emergency management. Disaster Prevention & Management, Vol. 9(2), pp. 89-97.
• Alexander, I., & Maiden, N. (2004). Scenarios, Stories, Use Cases: Through the Systems Development Life-Cycle. Wiley.
• Bolton, M. (2007). Users we don’t like. Developsense.com. http://www.developsense.com/articles/2007-08-UsersWeDontLike.pdf
• Bolton, M. (2007). Why we do scenario testing. Developsense.com. http://www.developsense.com/blog/2010/05/why-we-do-scenario-testing/
• Buwalda, H. (2000a). The three holy grails of test development. presented at EuroSTAR conference.
• Buwalda, H. (2000b). Soap Opera Testing. International Software Quality Week Europe conference, Brussels. http://www.logigear.com/resource-center/software-testing-articles-by-logigear-staff/246-soap-opera-testing.html
• Buwalda, H. (2007a). Key principles of test design. http://www.logigear.com/newsletter-2007/304-key-principles-of-test-design.html
• Buwalda, H. (2007b). The first holy grail of test design. http://www.logigear.com/newsletter-2007/305-the-first-holy-grail-of-test-design.html
• Buwalda, H. (2007c). The second holy grail of test design. http://www.logigear.com/newsletter-2007/306-the-second-holy-grail-of-test-design.html
• Buwalda H. (2007d). The third holy grail of test design. http://www.logigear.com/newsletter-2007/307-the-third-holy-grail-of-test-design.html
• Buwalda, H. (2008). The potential and risks of keyword based testing. http://logigear.com/newsletter-2008/342-the-potential-and-risks-of-keyword-based-testing.html
• Buwalda, H. (undated). Key success factors for keyword-driven testing. http://logigear.com/articles-by-logigear-staff/389–key-success-factors-for-keyword-driven-testing.html
• Buwalda, H., Janssen, D. & Pinkster, I. (2002). Integrated Test Design and Automation: Using the TestFrame Method. Addison-Wesley.
• Carroll, J.M. (ed.) (1995). Scenario-Based Design. Wiley.
• Carroll, J.M. (1995). Development, Scenario-Based Design: Envisioning Work and Technology in System. Wiley.
• Carroll, J.M. (1999). Five reasons for scenario-based design. Proceedings of the 32nd Hawaii International Conference on System Sciences, http://www.massey.ac.nz/~hryu/157.757/Scenario.pdf
• Guckenheimer, S. & Perez, J. (2006). Software Engineering with Microsoft Visual Studio Team System. Addison Wesley.
• Heijden, Kes van der (1996). Scenarios: The Art of Strategic Conversation. Wiley.
• Kahn, H. (1967). The use of scenarios. In Kahn, Herman & Wiener, Anthony (1967). The Year 2000: A Framework for Speculation on the Next Thirty-Three Years, pp. 262-264. http://www.hudson.org/index.cfm?fuseaction=publication_details&id=2214
• Kaner, C. (2003). An introduction to scenario testing. https://kaner.com/pdfs/ScenarioIntroVer4.pdf
• Memon, A.M., Pollack, M.E., & Soffa, M.L. (2000). A planning-based approach to GUI testing. International Software Quality Week, San Francisco. http://www.cs.umd.edu/~atif/papers/MemonSQW2000.pdf; http://www.cs.umd.edu/~atif/presentations/SQW2000.pdf
• Ringland, G. (1998). Scenario Planning: Managing for the Future. Wiley.
• Rippel, M. & Teply, P. (2009). Operational Risk — Scenario Analysis. Working Papers IES 2008/15, Charles University Prague, Faculty of Social Sciences, Institute of Economic Studies, revised Sep 2008. http://ideas.repec.org/p/fau/wpaper/wp2008_15.html
• Rosson, M.B., & Carroll, J.M. (2002). Usability Engineering: Morgan Kaufmann.
• Rothman, J., & Lawrence, B. (1999). Testing in the dark. Software Quality & Design Engineering, 1(2). Pp. 34-39 http://www.jrothman.com/Papers/Pragmaticstrategies.html
• Wack, P. (1985a). Scenarios: Uncharted waters ahead. Harvard Business Review 63(5), 74-89. http://tuvalu.santafe.edu/events/workshops/images/d/d9/Wack.pdf
• Wack, P. (1985b). Scenarios: Shooting the rapids. Harvard Business Review 63(6), 139-150. http://www.scribd.com/doc/4489875/Wack-Shooting-the-rapids
• Walker, W.E. (1994). The use of scenarios and gaming in crisis management planning and training. Presented at the conference, The Use of Scenarios for Crisis Management, Netherlands Ministry of Home Affairs, at the Netherlands Institute for Fire Service & Disaster Mgmt, Arnhem, November,16-18.
• Weber, B. (2008) Information Commerce 1997 – Scenario Mapping Changes Beliefs. http://www.strategykinetics.com/scenario_planning/

Self-verifying data

• Nyman, N. (1999). Self-verifying data: Testing without an oracle. International Conference on Software Testing Analysis & Research (STAR East), Orlando. http://www.stickyminds.com/getfile.asp?ot=XML&id=2918&fn=XDD2918filelistfilename1%2Epdf
• Knaus, R. Aougab, H., & Bentahar, N. (2004). Software Reliability: A Preliminary Handbook. McLean, VA: United States Department of Transportation: Federal Highway Administration (see the discussions of wrapping, especially Chapter 6). http://www.fhwa.dot.gov/publications/research/safety/04080/04080.pdf
• Romero, G. & Gauvin, C. (1998). Self-verifying communications testing. United States Patent US7409618, http://www.patents.com/self-verifying-communications-testing-7409618.html

Specification-based testing (See also active reading; See also ambiguity analysis)

• Bach, J. (1999). Reframing requirements analysis. IEEE Computer, 32(2), 120-122. http://www.satisfice.com/articles/reframing_requirements.pdf
• Bach, J. (2006). Heuristic test strategy model, Version 4.8. http://www.satisfice.com/tools/satisfice-tsm-4p.pdf
• Black, R. (2007). Pragmatic Software Testing: Becoming an Effective and Efficient Test Professional. Wiley.
• Donat, M. R. (1998). A Discipline of Specification-Based Test Derivation, Ph.D. Dissertation (Computer Science). University of British Columbia. http://www.nlc-bnc.ca/obj/s4/f2/dsk2/ftp02/NQ34519.pdf
• Hayes, J.H. (1999). Input Validation Testing: A System-Level, Early Lifecycle Technique. Ph.D. Dissertation (Computer Science), George Mason University.
• Jacky, J., Veanes, M., Campbell, C. & Schulte, W. (2008). Model-Based Software Testing and Analysis with C#. Cambridge University Press.
• Kaner, C. (1998). Liability for product incompatibility. Software QA, 5(4), p. 33ff. https://kaner.com/pdfs/liability.pdf
• Kaner, C. (2003). Liability for defective documentation. Conference of the ACM SIGDOC. https://kaner.com/pdfs/liability_sigdoc.pdf
• Lawrence, B. (undated). http://www.coyotevalley.com/tools.htm

State-model-based testing

• Auer, A.J. (1997). State Testing of Embedded Software. Ph.D. Dissertation (Computer Science). Oulun Yliopisto (Finland).
• Becker, S.A. & Whittaker, J.A. (1997). Cleanroom Software Engineering Practices. IDEA Group Publishing.
• Buwalda, H. (2003). Action figures. Software Testing & Quality Engineering. March/April 42-27. http://www.logigear.com/articles-by-logigear-staff/245-action-figures.html
• El-Far, I. K. (1999), Automated Construction of Software Behavior Models, Masters Thesis, Florida Institute of Technology, 1999.
• El-Far, I. K. & Whittaker, J.A. (2001), Model-based software testing, in Marciniak, J.J. (2001). Encyclopedia of Software Engineering, Wiley. http://testoptimal.com/ref/Model-based Software Testing.pdf
• Jorgensen, A.A. (1999). Software Design Based on Operational Modes. Doctoral Dissertation, Florida Institute of Technology. https://cs.fit.edu/Projects/tech_reports/cs-2002-10.pdf
• Katara, M., Kervinen, A., Maunumaa, M., Paakkonen, T., & Jaaskelainen, A. (2007). Can I have some model-based GUI tests please? Providing a model-based testing service through a web interface. Conference of the Association for Software Testing. http://practise.cs.tut.fi/files/publications/TEMA/cast07-final.pdf
• Mallery, C.J. (2005). On the Feasibility of Using FSM Approaches to Test Large Web Applications. M.Sc. Thesis (EECS). Washington State University.
• Page, A., Johnston, K., & Rollison, B.J. (2009). How We Test Software at Microsoft. Microsoft Press.
• Robinson, H. (1999a). Finite state model-based testing on a shoestring. http://www.stickyminds.com/getfile.asp?ot=XML&id=2156&fn=XDD2156filelistfilename1%2Epdf
• Robinson, H. (1999b). Graph theory techniques in model-based testing. International Conference on Testing Computer Software. http://sqa.fyicenter.com/art/Graph-Theory-Techniques-in-Model-Based-Testing.html
• Robinson, H. Model-Based Testing Home Page. http://www.geocities.com/model_based_testing/
• Rosaria, S., & Robinson, H. (2000). Applying models in your testing process. Information & Software Technology, 42(12), 815-24. http://www.harryrobinson.net/ApplyingModels.pdf
• Schultz, C.P., Bryant, R., & Langdell, T. (2005). Game Testing All in One. Thomson Press.
• Utting, M., & Legeard, B. (2007). Practical Model-Based Testing: A Tools Approach. Morgan Kaufmann.
• Vagoun, T. (1994). State-Based Software Testing. Ph.D. Dissertation (Computer Science). University of Maryland College Park.
• Whittaker, J.A. (1992). Markov Chain Techniques for Software Testing and Reliability Analysis. Ph.D. Dissertation (Computer Science). University of Tennessee.
• Whittaker, J.A. (1997). Stochastic software testing. Annals of Software Engineering, 4, 115-131.

Stress testing

• http://en.wikipedia.org/wiki/Stress_testing
• Beizer, B. (1984). Software System Testing and Quality Assurance. Van Nostrand. See also: http://www.faqs.org/faqs/software-eng/testing-faq/section-15.html

Task analysis (see also Scenario testing and Use-case-based testing)

• Crandall, B., Klein, G., & Hoffman, R.B. (2006). Working Minds: A Practitioner’s Guide to Cognitive Task Analysis. MIT Press.
• Draper, D. & Stanton, N. (2004). The Handbook of Task Analysis for Human-Computer Interaction. Lawrence Erlbaum.
• Ericsson, K.A. & Simon, H.A. (1993). Protocol Analysis: Verbal Reports as Data (Revised Edition). MIT Press.
• Gause, D.C., & Weinberg, G.M. (1989). Exploring Requirements: Quality Before Design. Dorset House.
• Hackos, J.T. & Redish, J.C. (1998). User and Task Analysis for Interface Design. Wiley.
• Jonassen, D.H., Tessmer, M., & Hannum, W.H. (1999). Task Analysis Methods for Instructional Design.
• Robertson, S. & Robertson, J. C. (2006, 2nd Ed.). Mastering the Requirements Process. Addison-Wesley Professional.
• Schraagen, J.M., Chipman, S.F., & Shalin, V.I. (2000). Cognitive Task Analysis. Lawrence Erlbaum.
• Shepard, A. (2001). Hierarchical Task Analysis. Taylor & Francis.

Test design / test techniques (in general)

• Bach, J. (2000). Heuristic Test Planning: Context Model. http://www.satisfice.com/tools/satisfice-cm.pdf
• Bach, J. (2006). Heuristic test strategy model. Version 4.8. http://www.satisfice.com/tools/satisfice-tsm-4p.pdf
• Black, R. (2007). Pragmatic Software Testing: Becoming an Effective and Efficient Test Professional. Wiley.
• Buwalda, H. (2007). Key principles of test design. http://logigear.com/newsletter-2007/304-key-principles-of-test-design.html
• Copeland, L. (2004). A Practitioner’s Guide to Software Test Design. Artech House.
• Collard, R. (2001). Test Design Fundamentals. International Conference on Software Testing Analysis & Review. (STAR West)
• Desikan, S. & Gopalaswamy, R. (2006). Software Testing: Principles and Practices. Pearson Education.
• Edgren, R. (2011). The Little Black Book on Test Design. http://thetesteye.com/blog/2011/09/the-little-black-book-on-test-design/ 
• Edgren, R. (2010). Review of properties in Kaner’s What is a Good Test Case? http://thetesteye.com/blog/2010/05/review-of-properties-in-kaners-what-is-a-good-test-case/
• Emilsson, H., Jansson, M., Edgren, R. (2010). Software Quality Characteristics 1.0. http://thetesteye.com/posters/TheTestEye_SoftwareQualityCharacteristics.pdf
• Jorgensen, P. (2008, 3rd Ed.). Software Testing: A Craftsman’s Approach. Auerbach Publications.
• Kaner, C. (2003). What is a good test case? https://kaner.com/pdfs/GoodTest.pdf
• Kaner, C. (2004b). The ongoing revolution in software testing. Software Test & Performance conference. https://kaner.com/pdfs/TheOngoingRevolution.pdf
• Kaner, C., Bach, J., & Pettichord, B. (2001). Lessons Learned in Software Testing: Chapter 3: Test Techniques. http://media.techtarget.com/searchSoftwareQuality/downloads/Lessons_Learned_in_SW_testingCh3.pdf
• Test design / test techniques (in general)
• Kelly, M.D. (2007). Specialists and other myths: Because you aren’t a specialist doesn’t mean you can’t do it. Conference of the Association for Software Testing. http://www.michaeldkelly.com/pdfs/CAST2007_​SpecialistsAndOtherMyths.pdf
• Kelly, M.D. (2009). Software testing: Assessing risk and scope. http://searchsoftwarequality.techtarget.com/podcast/Software-Testing-Assessing-risk-and-scope
• Loveland, S., Miller, G. Prewitt, R., & Shannon, M. (2005). Software Testing Techniques: Finding the Techniques that Matter. Charles River Media.
• Nguyen, H.Q., Johnson, B., & Hackett, M. (2003). Testing Applications on the Web, 2nd Ed. Wiley.
• Page, A., Johnston, K., & Rollison, B.J. (2009). How We Test Software at Microsoft. Microsoft Press.
• Perry, W.E. (2006). Effective Methods for Software Testing. Wiley.
• Rajani, R., & Oak, P. (2004). Software Testing: Effective Methods, Tools & Techniques. Tata McGraw-Hill.
• Ryber,. T. (2007). Essential Software Test Design. Fearless Consulting.
• Schultz, C.P., Bryant, R., & Langdell, T. (2005). Game Testing All in One. Thomson Press.
• Sutton, M., Greene, A., & Amini, P. (2007). Fuzzing: Brute Force Vulnerability Discovery. Addison Wesley.
• Takanen, A., DeMott, J., & Miller, C. (2008). Fuzzing for Software Security Testing and Quality Assurance. Artech House.
• Whittaker, J.A. (2002). How to Break Software. Addison Wesley.

Test idea catalogs

• Edgren, R. (2009). More and better test ideas. EuroSTAR. http://www.thetesteye.com/papers/redgren_moreandbettertestideas.pdf
• Edgren, R. (2011). The Little Black Book on Test Design. http://thetesteye.com/blog/2011/09/the-little-black-book-on-test-design/
• Hendrickson, E. (2006). Test Heuristics Cheat Sheet. http://testobsessed.com/wp-content/uploads/2011/04/testheuristicscheatsheetv1.pdf
• Hunter, M. J. (2010). You are not done yet. http://www.thebraidytester.com/downloads/YouAreNotDoneYet.pdf
• Kaner, C., Padmanabhan, S., & Hoffman, D. (2012) Domain Testing: A Workbook, in press.
• Marick, B.M. (1994). The Craft of Software Testing: Subsystems Testing Including Object-Based and Object-Oriented Testing. Prentice-Hall. Updated catalog: http://www.exampler.com/testing-com/writings/catalog.pdf
• Marick, B.M. (undated). A short catalog of test ideas for… http://www.exampler.com/testing-com/writings/short-catalog.pdf
• Nguyen, H.Q., Johnson, B., & Hackett, M. (2003, 2nd ed), Testing Applications on the Web. Wiley (Appendices D through H).
• Sabourin, R. (2008). Ten things you might not know about sources of testing ideas. Better Software, April, 46. http://www.amibugshare.com/articles/Article_10_Sources_of_Testing_Ideas.pdf

Testing skill

Many of the references in this collection are about the development of testing skill. However, a few papers stand out, to me, as exemplars of papers that focus on activities or structures designed to help testers improve their day-to-day testing skills. We need more of these.

• Bach, J. (2006). Heuristic test strategy model, Version 4.8. http://www.satisfice.com/tools/satisfice-tsm-4p.pdf
• Gärtner, M. (2011) Testing Dojos: Craftsmanship in Software Testing: Blog at http://www.testingdojo.org/tiki-index.php. Expanded from Gaertner, M. (2010, Winter). Testing dojos: Another way to gain software testing experience. Methods & Tools. http://www.methodsandtools.com/mt/download.php?winter10
• Hendrickson, E. (2006), Test heuristics cheat sheet. http://testobsessed.com/wp-content/uploads/2011/04/testheuristicscheatsheetv1.pdf
• Hunter, M. J. (2010). You are not done yet. http://www.thebraidytester.com/downloads/YouAreNotDoneYet.pdf

Tours

• Bolton, M. (2009). Of testing tours and dashboards. http://www.developsense.com/blog/2009/04/of-testing-tours-and-dashboards
• Craig, R. D., & Jaskiel, S. P. (2002). Systematic Software Testing. Artech House. (see their discussions of inventories)
• Goucher, A. (2009). ‘Exploratory Software Testing’ – a cheat of a book. http://adam.goucher.ca/?p=1225
• Kaner, C. (2011). Testing tours: Research for best practices? https://kaner.com/?p=96
• Kelly, M.D. (2005). Touring Heuristic. http://www.michaeldkelly.com/archives/50
• Kelly, M.D. (2006). Taking a tour through test country. A guide to tours to take on your next test project. Software Test and Performance Magazine, February, 20-25. http://michaeldkelly.com/pdfs/Taking%20a%20Tour%20Through%20Test%20Country.pdf; http://www.softwaretestpro.com/Item/2752/Taking-a-Tour-Through-Test-Country/Test-and-QA-Video-Exploratory-Strategy-Performance-Functional-Software-Testing-Unit-User-Web
• Kohl, J. (2006). Modeling test heuristics. http://www.kohl.ca/blog/archives/000179.html
• Laplante, P. (2009). Exploratory testing for mission critical, real-time, and embedded systems. Part of the IEEE Reliability Society 2009 Annual Technology Report. http://paris.utdallas.edu/IEEE-RS-ATR/document/2009/2009-08.pdf
• Whittaker, J.A. (2009). Exploratory Software Testing: Tips, Tricks, Tours, and Techniques to Guide Test Design. Addison-Wesley.

Usability testing

• Cooper, A. (2004). The Inmates are Running the Asylum: Why High Tech Products Drive Us Crazy and How to Restore the Sanity. Pearson Education.
• Cooper, A., Reimann, R. & Cronin, D. (2007). About Face 3: The Essentials of Interaction Design. Wiley.
• Dumas, J.S. & Loring, B.A. (2008). Moderating Usability Tests: Principles and Practices for Interacting. Morgan Kaufmann.
• Fiedler, R.L., & Kaner, C. (2009). “Putting the context in context-driven testing (an application of Cultural Historical Activity Theory).” Conference of the Association for Software Testing. https://kaner.com/pdfs/FiedlerKanerCast2009.pdf
• Ives, B., Olson, M.H., & Baroudi, J.J. (1983). The measurement of user information systems. Communications of the ACM, 26(10), 785-793. http://portal.acm.org/citation.cfm?id=358430
• Krug, S. (2005, 2nd Ed.). Don’t Make Me Think: A Common Sense Approach to Web Usability. New Riders Press.
• Kuniavsky, M. (2003). Observing the User Experience: A Practitioner’s Guide to User Research. Morgan Kaufmann.
• Lazar, J., Fend, J.H., & Hochheiser, H. (2010). Research Methods in Human-Computer Interaction. Wiley.
• Nielsen, J. (1994). Guerrilla HCI: Using discount usability engineering to penetrate the intimidation barrier. http://www.useit.com/papers/guerrilla_hci.html
• Nielsen, J. (1999). Designing Web Usability. Peachpit Press.
• Nielson, J. & Loranger, H. (2006). Prioritize Web Usability. MIT Press.
• Norman, D.A. (2010). Living with Complexity. MIT Press.
• Norman, D.A. (1994). Things that Make Us Smart: Defending Human Attributes in the Age of the Machine. Basic Books.
• Norman, D.A. & Draper, S.W. (1986). User Centered System Design: New Perspectives on Human-Computer Interaction. CRC Press.
• Patel, M. & Loring, B. (2001). Handling awkward usability testing situations. Proceedings of the Human Factors and Ergonomics Society 45th Annual Meeting. 1772-1776.
• Platt, D.S. (2006). Why Software Sucks. Addison-Wesley.
• Rubin, J., Chisnell, D. & Spool, J. (2008). Handbook of Usability Testing: How to Plan, Design, and Conduct Effective Tests. Wiley.
• Smilowitz, E.D., Darnell, M.J., & Benson, A.E. (1993). Are we overlooking some usability testing methods? A comparison of lab, beta, and forum tests. Proceedings of the Human Factors and Ergonomics Society 37th Annual Meeting, 300-303.
• Stone, D., Jarrett, C., Woodroffe, M. & Minocha, S. (2005). User Interface Design and Evaluation. Morgan Kaufmann.
• Tullis, T. & Albert, W. (2008). Measuring the User Experience: Collecting, Analyzing, and Presenting Usability Metrics (Interactive Technologies). Morgan Kaufmann.

Use-case based testing (see also Scenario testing and Task analysis)

• Adolph, S. & Bramble, P. (2003). Patterns for Effective Use Cases. Addison-Wesley.
• Alexander, Ian & Maiden, Neil. Scenarios, Stories, Use Cases: Through the Systems Development Life-Cycle.
• Alsumait, A. (2004). User Interface Requirements Engineering: A scenario-based framework. Ph.D. dissertation (Computer Science), Concordia University.
• Berger, Bernie (2001) “The dangers of use cases employed as test cases,” STAR West conference, San Jose, CA. http://www.testassured.com/docs/Dangers.htm
• Charles, F.A. (2009). Modeling scenarios using data. STP Magazine. http://www.quality-intelligence.com/articles/Modelling%20Scenarios%20Using%20Data_Paper_Fiona%20Charles_CAST%202009_Final.pdf
• Cockburn, A.(2001). Writing Effective Use Cases. Addison-Wesley.
• Cohn, M. (2004). User Stories Applied: For Agile Software Development. Pearson Education.
• Collard, R. (July/August 1999). Test design: Developing test cases from use cases. Software Testing & Quality Engineering, 31-36.
• Hsia, P., Samuel, J. Gao, J. Kung, D., Toyoshima, Y. & Chen, C. (1994). Formal approach to scenario analysis. IEEE Software, 11(2), 33-41.
• Jacobson, I. (1995). The use-case construct in object-oriented software engineering. In John Carroll (ed.) (1995). Scenario-Based Design. Wiley.
• Jacobson, I., Booch, G. & Rumbaugh, J. (1999). The Unified Software Development Process. Addison-Wesley.
• Jacobson, I. & Bylund, S. (2000) The Road to the Unified Software Development Process. Cambridge University Press.
• Kim, Y. C. (2000). A Use Case Approach to Test Plan Generation During Design. Ph.D. Dissertation (Computer Science). Illinois Institute of Technology.
• Kruchten, P. (2003, 3rd Ed.). The Rational Unified Process: An Introduction. Addison-Wesley.
• Samuel, J. (1994). Scenario analysis in requirements elicitation and software testing. M.Sc. Thesis (Computer Science), University of Texas at Arlington.
• Utting, M., & Legeard, B. (2007). Practical Model-Based Testing: A Tools Approach. Morgan Kaufmann.
• Van der Poll, J.A., Kotze, P., Seffah, A., Radhakrishnan, T., & Alsumait, A. (2003). Combining UCMs and formal methods for representing and checking the validity of scenarios as user requirements. Proceedings of the South African Institute of Computer Scientists and Information Technologists on Enablement Through Technology. http://dl.acm.org/citation.cfm?id=954014.954021
• Zielczynski, P. (2006). Traceability from use cases to test cases. http://www.ibm.com/developerworks/rational/library/04/r-3217/

User interface testing

• Adzic, G. (2007). Effective user interface testing. http://gojko.net/2007/09/25/effective-user-interface-testing/
• Apple Computer (1993). Macintosh Human Interface Guidelines. Apple.
• Apple Computer (2011). Apple Human Interface Guidelines: User Experience. http://developer.apple.com/library/mac/#documentation/UserExperience/Conceptual/AppleHIGuidelines
• Dudziak, T. (2005). How to unit test the user interface of web applications. ApacheCon US. http://floyd.openqa.org/HowToUnitTestTheUserInterfaceOfWebApplications.pdf
• McKay, E.N. (1999). Developing User Interfaces for Microsoft Windows. Microsoft Press.
• Microsoft (2010). Windows User Experience Guidelines for Windows 7 and Windows Vista. http://msdn.microsoft.com/en-us/library/aa511258.aspx
• Nielsen, J. (2001). Coordinating User Interfaces for Consistency. Morgan Kaufmann.
• Olson, D.R. (1998). Developing User Interfaces. Morgan Kaufmann.
• Olson, D.R. (2009). Building Interactive Systems: Principles for Human-Computer Interaction. Course Technology.
• Visual Studio 2010 (undated). Testing the user interface with automated UI tests. http://msdn.microsoft.com/en-us/library/dd286726%28VS.100%29.aspx

User testing (see beta testing)

• Albert, W., Tullis, T. & Tedesco, D. (2010). Beyond the Usability Lab: Conducting Large-Scale Online User Experience Studies. Morgan Kaufmann.
• Wang, E., & Caldwell, B. (2002). An empirical study of usability testing: Heuristic evaluation vs. user testing. Proceedings of the Human Factors and Ergonomics Society 46th Annual Meeting. 774-778.

This post is partially based on work supported by NSF research grant CCLI-0717613 ―Adaptation & Implementation of an Activity-Based Online or Hybrid Course in Software Testing. Any opinions, findings and conclusions or recommendations expressed in this post are those of the author and do not necessarily reflect the views of the National Science Foundation.

A few years ago, Michael Bolton wrote a blog post on “Testing Tours and Dashboards.” Back then, it had recently become fashionable to talk about “tours” as a fundamental class of tool in exploratory testing. Michael reminded readers of the unacknowledged history of this work.

Michael’s post also mentioned that some people were talking about running experiments on testing tours. Around that time, I heard a bunch about experiments that purported to compare testing tours. This didn’t sound like very good work to me, so I ignored it. Bad ideas often enjoy a burst of publicity, but if you leave them alone, they often fade away over time.

This topic has come up again recently, repeatedly. A discussion with someone today motivated me to finally publish a comment.

The comments have come up mainly in discussions or reviews of a test design course that I’m creating. The course (video) starts with a demonstration of a feature tour, followed by a inventory of many of the tours I learned from Mike Kelly, Elisabeth Hendrickson, James Bach and Mike Bolton.

Why, the commentator asks, do I not explain which tours are better and which are worse? After all, hasn’t there been some research that shows that Tour X is better than Tour Y? This continues down one of two tracks:

• Isn’t it irresponsible to ignore scientific research that demonstrates that some tours are much better than others or that some tours are ineffective?
• Shouldn’t I be recommending that people do more of this kind of research? Wouldn’t this be a promising line of research? Couldn’t someone propose it to a corporate research department or a government agency that funds research? After all, this could be a scientific way to establish some Best Practices for exploratory testing (use the best tours, skip the worst ones).

This idea, using experiments to rank tours from best to worst, can certainly be made to sound impressive.

I don’t think this is a good idea. I’ll say that more strongly: even though this idea might be seductive to people who have training in empirical methods (or who are easily impressed by descriptions of empirical work), I think it reflects a fundamental lack of understanding of exploratory testing and of touring as a class of exploratory tools.

A tour is a directed search through the program. Find all the capabilities. Find all the claims about the product. Find all the variables. Find all the intended benefits. Find all the ways to get from A to B. Find all the X. Or maybe not ALL, but find a bunch.

This helps the tester achieve a few things:

1. It creates an inventory of a class of attributes of the product under test. Later, the tester can work through the inventory, testing each one to some intended level of depth. This is what “coverage”-oriented testing is about. You can test N% of the program’s statements, or N% of the program’s features, or N% of the claims made for the product in its documentation–if you can list it, you can test it and check it off the list.
2. It familiarizes the tester with this aspect of the product. Testing is about discovering quality-related information about the product. An important part of the process of discovery is learning what is in the product, how people can / will use it, and how it works. Tours give us different angles on that multidimensional learning problem.
3. It provides a way for the tester to explain to someone else what she has studied in the product so far, what types of things she has learned and what basics haven’t yet been explored. In a field with few trustworthy metrics, this gives us a useful basis for reporting progress, especially progress early in the testing effort.

So which tour is better?

From a test-coverage perspective, I think that depends a lot on contract, regulation, and risk.

1. To the extent that you have to know (and have to be able to tell people in writing) that all the X’s have been tested and all the X’s work, you need to know what all the X’s are and how to find them in the program. That calls for an X-tour. Which tour is better? The one you need for this product. That probably varies from product to product, no?
2. Some programmers are more likely to make X-bugs than Y-bugs. Some programmers are sloppy about initializing variables. Some are sloppy about boundary conditions. Some are sloppy about thread interactions. Some are good coders but they design user interactions that are too confusing. If I’m testing Joe X’s code, I want to look for X-bugs. If Joe blows boundaries, I want to do a variable tour, to find all the variables so I can test all the boundaries. But if Joe’s problem is incomprehensibility, I want to do a benefit tour, to see what benefits people _should_ get from the program and how hard/confusing it is for users to actually get them. Which tour is better? That depends on which risks we are trying to mitigate, which bugs we are trying to find. And that varies from program to program, programmer to programmer, and on the same project, from time to time.

From a tester-learning perspective, people learn differently from each other.

1. If I set 10 people with the task of learning what’s in a program, how it can be used, and how it works, those people would look for different types of information. They would be confused by different things. They would each find some things more interesting than others. They would already know some things that their colleagues didn’t.
2. Which tour is better? The tour that helps you learn something you’re trying to learn today. Tomorrow, the best tour will be something different.

Testing is an infinite task. Define “distinct tests” this way: two tests are distinct if each can expose at least one bug that the other would miss. For a non-trivial program, there is an infinite number of distinct potential tests. The central problem of test design is boiling down this infinite set to a (relative to infinity) tiny collection of tests that we will actually use. Each test technique highlights a different subset of this infinity. In effect, each test technique represents a different sampling strategy from the space of possible tests.

Tours help the tester gain insight into the multidimensional nature of this complex, infinite space. They help us imagine, envision, and as we gain experience on the tour, prioritize the different sampling strategies we could use when we do more thorough, more intense testing after finishing the tours. So which tour is best? The ones that give the testers more insight and that achieve a greater stretch of the testers’ imagination. For this, some tours will work better for me, others for you.

The best tour isn’t necessarily the one that finds the most bugs. Or covers the most statements (or take your pick of what coverage attribute) of the product. The best tour is the one that helps the individual human tester learn something new and useful. (That’s why we call it exploration. New and useful knowledge.) And that depends on what you already know, on what risks and complexities characterize this program, and what the priorities are in this project.

A tour that is useful to you might be worthless to me. But that doesn’t stop it from being useful for you.

• Rather than looking for a few “best” tours, I think it would be more interesting to develop guidance on how to do tour X given that you want to. (Do you know how to do a tour of uses of the program that trigger multithreaded operations in the system? Would it be interesting to know how?)
• Rather than looking for a few “best” tours, I think it would be more interesting to develop a more diverse collection of tours that we can do, with more insight into what each can teach us.
• Rather than seeking to objectify and quantify tours, I think we should embrace their subjectivity and the qualitative nature of the benefits they provide.

Pseudo-academics and best-practicers trivialize enough aspects of testing. They should leave this one alone.

I’ll be keynoting at CAST on investment modeling and exploratory test automation.

In its essence, exploratory testing is about learning new things about the quality of the software under test. Exploratory test automation is about using software to help with that learning.

Testing the value of investment models is an interesting illustration because it is all about the quality of the software, it is intensely automated, and none of the tests are regression tests. This is one illustration of automated exploration. I’ll point to others in my talk and paper.

THERE’S STILL TIME TO SUBMIT YOUR OWN PROPOSAL FOR A PRESENTATION. (The official deadline is March 20, but if you’re a few days late, I bet the conference committee will still read your proposal.)

Here’s the link: http://conferences.associationforsoftwaretesting.org/

What’s different about CAST is that we genuinely welcome discussion and debate. Any participant can ask questions of any speaker. Any participant can state her or his counter-argument to a speaker. We’ll keep taking questions all day. Artificial time limits are not used to cut off discussion.  After all, the point of a conference is “conferring.”

See you there…