Archive for June, 2016

An Update to BBST

Monday, June 27th, 2016

Rebecca Fiedler and I have just completed a major round of updates to BBST, the Black Box Software Testing course. This creates what we consider a stable release, which we expect to be the final release of BBST Version 3. I am starting to work on Version 4 (initial discussion post: next week).

The course videos are unchanged. The home page for BBST (with the best course descriptions) is still Foundations, Bug Advocacy, and Test Design are still available for free at along with course slides and other supporting materials.

These are the videos that the Association for Software Testing helped develop, and teaches from. I will probably continue to use these in the Software Engineering degree programs at Florida Tech for a few years. For now, we teach from these videos in private courses for commercial clients, through Kaner Fiedler Associates and in conjunction with Altom, which offers the latest updates to our courses to the public.

My course version identifier will stay at Version 3. (Foundations (2010) offers the third generation of Foundations videos, Bug Advocacy (2008) videos are second generation, Test Design (2011) are mixed, second and third. Collectively: Version 3).

Here’s what’s new:

(1) We published course workbooks for each of the three courses. The workbooks provide:

  • An edited transcript of each lecture
  • Updated (or completely replaced) activities
  • Detailed feedback for most activities, based on patterns in student work that we’ve seen over the 12 years since the first BBST course videos.
  • Often-detailed commentary on the lectures and activities. Our primary focus in these notes is the instructional effectiveness of the materials: treat the intended content and skill development as given. How well does the courseware support them? Occasionally, we felt compelled to acknowledge some other issues in the field that we decided were unignorably relevant to the courses.

The workbooks are:

I am making the final publishing-support revisions for Test Design this week. It should be available at Amazon before the end of July.

These books run 230-to-400 pages long. They took a lot of work.

(2) We have overhauled the multiple-choice review questions in all three courses.

These were a source of some irritation in BBST. The new questions are still hard, but student feedback is that they are much better.

(3) We revised our model for instructor-student feedback.

When we planned Version 3, one of our key concerns was that instructors were volunteers. There is only so much work you can give to volunteers before they burn out. We designed the course-interaction model to have as much peer review as possible. This would give students feedback on the quality of their work while keeping the instructor workload under control. Overall, that model has worked pretty well.

  • Feedback quality is, of course, variable. The feedback comes from people who don’t necessarily understand the material themselves. They are learning as much from giving the feedback as the students who receive it.
  • We had some problems with bullying from some students who seemed more motivated to insist on their doctrinaire views than to learn anything new. This was a nuisance to manage when it arose, but I think most instructors figured out how to handle it well enough.
  • Students who needed additional personal feedback often received private feedback via instructor emails or Skypes. My role as an instructional coach for AST completed a few years ago, but my sense is that AST’s instructors are pretty attentive to this need.

The same concerns for volunteer-taught courses apply to large-enrollment courses in universities and colleges. Detailed personalized feedback is just not an instructional option.

However, students’ expectations are appropriately different when they attend commercial versions of the course (paying the usual rates for commercial instruction) or for small-enrollment university courses. They expect better feedback.

Starting in about 2011, Professor Keith Gallagher helped me adapt his ideas on interactive grading to my courses. An interactive grading session is a one-on-one meeting, student with instructor, that lasts 1-to-2 hours (occasionally, 3 hours, ouch) and is focused on a specific exam or major assignment. Usually the focus is on the course’s key piece of work. The intent of the session is coaching, identifying ways the student can improve their work and praising the strengths. Some sessions, of course, are more difficult. In the commercial BBST courses,

  • We offer one interactive grading session per student. There are usually two instructors (sometimes three) in the commercial courses. The student typically picks which instructor to meet with.
  • We draw student attention to the feedback available on assignments in the course Workbook. For assignments that are not well-covered in the Workbook, the instructor writes a detailed class-wide feedback. We provide personalized feedback on aspects of the student’s work that are not covered in the general feedback.
  • The instructor provides summary grading for each piece of work (exceptional, acceptable, disappointing, not-submitted or no-value). (Most work is graded “acceptable.”)

I don’t think this level of feedback is sustainable in a volunteer-taught course. It is more easily sustainable if the instructor uses the Workbook as a course text and can rely on the students having their own copy. But it is still more work than I would ever want to assign to volunteers.

If you are teaching or taking BBST, we suggest that you use the Workbooks

The Workbooks are not Creative Commons licensed. They are not free, but they are inexpensive. When we originally planned these, we intended to publish through commercial publishers, but the prices they planned for the Workbooks were way too high. We created Context-Driven Press so that we could control the quality and price of the books. You can get the books from Amazon for less than $25 each (much less, for Kindle editions).

If you are teaching a BBST course, we strongly recommend that you use the course Workbook as a course text. It changes the student experience, and the feedback we continually get is that the change is strongly positive.

I think these updates mark the completion of this version of BBST.

Online BBST started in 2004. It reflects my understanding of the industry circa 2001 (when we published Lessons Learned in Software Testing.) BBST Version 3 presents the same ideas. The polish from Versions 1 through 3 came more in the instructional design than in the perspective. I am proud of Version 3, and I think it will provide a useful foundation for several years to come. However, our field’s methods, technologies, and social structures have evolved. I think it is time for some bigger changes in what I teach.

There is a fourth course in the BBST series, BBST-Domain Testing. This is currently available only through Kaner Fiedler Associates, though I suspect Altom will offer it soon. The Domain Testing course includes the core sequence of videos, but also includes a set of supplementary videos that have demonstrations and supporting discussions. We spent more time on the supplementary videos than on the core sequence. In BBST Version 4, we intend to exploit some of the opportunities that supplementary videos provide.

I’ll be posting some notes on (Rebecca’s and my ideas for) Foundations 4.0, and asking for your help in planning the update. Those will come in about a week.