Sowmya Padmanabhan, Doug Hoffman and I just published a new book together, The Domain Testing Workbook.
The book focuses on one (1) testing technique. Domain Testing is the name of a generalized approach to Equivalence Class Analysis and Boundary Testing. Our goal is to help you develop skill with this one technique. There are plenty of overviews of this technique: it is the most widely taught, and probably the best understood, technique in the field. However, we’re not aware of any other presentations that are focused on helping you go beyond an understanding of the technique to achieving the ability to use it competently.
This is the first of a series of books that are coming along slowly. Probably the next one will be on scenario testing, with Dr. Rebecca Fiedler, merging her deep knowledge of qualitative methodology with my hands-on use of scenarios in software design, software testing, and software-related human factors. Also in the works are books on risk-based testing and specification-based testing. We’ve been working on all of these for years. We learned much from the Domain Testing Workbook about how challenging it is to write a book with a development-of-skill instructional design goal. At this point, we have no idea what the schedule is for the next three. When the soup is ready, we’ll serve it.
This work comes out of a research proposal that I sent to the United States’ National Science Foundation (NSF) in 2001 (Improving the Education of Software Testers), which had, among its objectives:
- “Identify skills involved in software testing.”
- “Identify types of exercises that support the development of specific testing skills.”
- “Create and publish a collection of reusable materials for exercises and tests.”
- “Prototype a web-based course on software testing.”
- “Create a series of workshops that focus on the teaching of software testing”
NSF approved the project, which made it possible for us to open the Center for Software Testing Education & Research (CSTER). The web-based course we had in mind is now available as the BBST series. The Workshops on Teaching Software Testing are now in their 13th year. And the Domain Testing Workbook is our primary contribution focused on “exercises that support the development of specific testing skills.”
When we started this project, we knew that domain testing would be the easiest technique to write this kind of book about. Over the years, we had two surprises that caused us to fundamentally rethink the structure of this book (and of other books that I’m still working on that are focused on scenario testing, risk-based testing, and specification-based testing):
- Our first surprise (or better put, our first shock) was that we could teach students a broad collection of examples of the application of domain testing, confirm that they could apply what they had learned to similar problems, and yet these students would fail miserably when we gave them a slightly more complex problem that was a little different than they had seen before. This was the core finding of Sowmya’s M.Sc. thesis. What we had tripped over was the transfer problem, which is probably the central instructional-design problem in Science, Technology, Engineering & Mathematics (STEM) instruction. The classic example is of the student who did well in her or his calculus course but cannot figure out how to apply calculus to problems (like modeling acceleration) in their introductory physics course. These students can’t transfer what they learned in one course to another course or to more complex situations (such as using it at their job). We had believed that we could work around the transfer problem by building our instruction around realistic exercises/examples. We were mistaken.
- Ultimately, we concluded that teaching through exercises is still our best shot at helping people develop skill, but that we needed to provide a conceptual structure for the exercises that could give students a strategy for approaching new problems. We created a schema—an 18-step cognitive structure that describes how we do a domain analysis—and we present every exercise and every example in the context of that schema. We haven’t done the formal, experimental research to check this that Sowmya was able to do with our initial approach—an experiment like that is time-consuming and expensive and our funding for that type of work ran out long ago. However, we have inflicted many drafts of the schema on our students at Florida Tech and we believe it improves their performance and organizes their approach to tasks like exams.
- Our next surprise was that domain testing is harder to apply than we expected. Doug and I are experienced with this technique. We think we’re pretty good at it, and we’ve thought that for a long time. Many other testers perceive us that way too. For example, Testing Computer Software opens with an example of domain testing and talks about the technique in more detail throughout the book. That presentation has been well received. So, when we decided to write a book with a few dozen examples that increased in complexity, we were confident that we could work through the examples quickly. It might take more time to write our analysis in a way that readers could understand, but doing the analysis would be straightforward. We were sooooooo wrong. Yes, we could quickly get to the core of the problem, identifying how we would approach identifying equivalence classes and boundaries (or non-boundary most-powerful test cases) for each class. Yes, we could quickly list several good tests. We got far enough along that we could impress other people with what we knew and how quickly we could get there, but not far enough to complete the problem. We were often getting through about a third of the analysis before getting stuck. Doug would fly to Palm Bay (Florida, where I live) and we would work problems. Some problems that we expected to take a day to solve and explain took us a week.
- As we came to understand what skills and knowledge we were actually applying when we slogged through the harder problems, we added more description of our background knowledge to the book. A presentation of our Domain Testing Schema—and the thinking behind it—grew from an expected length of about 30 pages to 200. Our presentation of 30 worked examples grew from an expected length of maybe 120 pages (4 pages each, with diagrams) to 190 pages.
We got a lot of help with the book. Our acknowledgments list 91 people who helped us think through the ideas in the book. Many others helped indirectly, such as many participants in the WTST workshops who taught us critical lessons about the instructional issues we were beating our heads against.
Perhaps the main advance in the clarity of the presentation came out of gentle-but-firm, collegial prodding by Paul Gerrard. Paul’s point was that domain testing is really a way for a tester to model the software. The tests that come out of the analysis are not necessarily tied to the actual code. They are tied to the tester’s mental model of the code. Our first reactions to these comments were that Paul was saying something obvious. But over time we came to understand his point—it might be obvious to us, but presentations of domain testing, including ours, were doing an inadequate job of making it obvious to our readers. This led us to import the idea of a notional variable from financial analysis as a way of describing the type of model that domain testing was leading us to make. We wrote in the book:
“A notional variable reflects the tester’s “notion” of what is in the code. The tester analyzes the program as if this variable was part of the code…. The notional variable is part of a model, created by the tester, to describe how the program behaves or should behave. The model won’t be perfect. That exact variable is probably not in the code. However, the notional variable is probably functionally equivalent to (works the same way as) a variable that is in the code or functionally equivalent to a group of variables that interact to produce the same behavior as this variable.”
This in turn helped us rethink parts of the Schema in ways that improved its clarity. And it helped us clarify our thinking about domain-related exploratory testing as a way of refining the modeling of the notional variables (bringing them into increasingly accurate alignment with the program implementation or, if the implementation is inadequate, with how the program should behave).
I hope you have a chance to read The Domain Testing Workbook and if you do, that you’ll post your reactions here or as a review on Amazon.