Archive for the ‘testing’ Category

A proposal for an advanced certification in software testing

Monday, March 3rd, 2014

This is a draft of a proposal to create a more advanced, more credible credential (certification) in software testing.

The core idea is a certification based on a multidimensional collection of evidence of education, experience, skill and good character.

  • I think it is important to develop a credential that is useful and informative.
    • I think we damage the reputation of the field if we create a certification that requires only a shallow knowledge of software testing.
    • I think we damage the value of the certification if we exaggerate how much knowledge or skill is required to obtain it.
  • I think it is important to find a way to tolerate different approaches to software testing, and different approaches to training software testers. This proposal is not based on any one favored “body of knowledge” and it is not tied to any one ideology or group of vendors.

The idea presented here is imperfect—as are the other certifications in our field. It can be gamed—as can the others. Someone who is intent on gaining a credential via cheating and fraud can probably get away with it for a while—but the others have security risks too. This certification does not assure that the certified person is competent—neither do the others. The certification does not subject the certified person to formal professional accountability for their work—neither do the others—and even though certificate holders say that they will follow a code of ethics, we have no mechanism for assuring that they do or punishing them if they don’t—and neither do the others.

With all these we-don’t-do-thises and we-don’t-promise-thats, you might think I’m kidding about this being a real proposal. I’m not.

Even if we agree that this proposed certification lacks the kinds of powers that could be bestowed by law or magic, I think it can provide useful information and that it can create incentives that favor higher ethics in job-seeking and, eventually, professional practice. It is not perfect, but I think it is far better than what we have now.

The Proposal

This credential is based on a collection of several different types of evidence that, taken together, indicate that the certificate holder has the knowledge and skill needed to competently perform the usual services provided by a software tester.

Here are the types of evidence. As you read this, imagine that the Certification Body hosts a website that will permanently post a publicly-viewable dossier (a collection of files) for every person certified by that body. The dossier would include everything submitted by an applicant for certification, plus some additional material. Here’s what we’d find in the file.

Authorization by the Applicant

As part of the application, the applicant for Certification would grant the Certification Board permission to publish all of the following materials. The applicant would also sign a legal waiver that would shield the Board from all types of legal action by the applicant / Certified Tester arising out of publication of the materials described below. The waiver will also authorize the Board to exercise its judgment in assessing the application and will shield the Board from legal action by the applicant if the Board decides, in its unfettered discretion, to reject the applicant’s application or to later cancel the applicant’s Certification.

Education (Academic)

The Certified Tester should have at least a minimum level of formal education. The baseline that I imagine is a bachelor’s-level degree in a field relevant to software testing.

  • Some fields, such as software engineering, are obviously relevant to software testing. But what about others like accounting, mathematics, philosophy, physics, psychology, or technical writing? We would resolve this by requiring the applicant for certification to explain in writing how and why her or his education has proved to be relevant to her or his experiences as a tester and why it should be seen as relevant education for someone in the field.
  • The requirement for formal education should be waived if the applicant requests waiver and justifies the request on the basis of a sufficient mix of practical education and professional achievement.

Education (Practical)

The Certified Tester should have successfully completed a significant amount of practical training in software testing. Most of this training would typically be course-based, typically commercial training. Some academic courses in software testing would also qualify. A non-negotiable requirement is successful completion of at least some courses that are considered advanced. “Successful” completion means that the student completed an exam or capstone project that a student who doesn’t know the material would not pass.

  • There is an obvious accreditation issue here. Someone has to decide which courses are suitable and which are advanced.
  • I think that many different types of courses and different topics might be suitable as part of the practical training. For example, suppose we required 100 classroom-hours of training (1 training day = 6 classroom hours). Perhaps 60 of those hours could be in related fields (programming, software metrics, software-related law, project accounting, etc.) but a core would have to be explicitly focused on testing.
  • I think the advanced course hours (24 classroom hours?) would have to be explicitly advanced software testing courses.
  • There is no requirement that these courses come from any particular vendor or that they follow any particular software testing or software development ideology.

Examination

The Certified Tester should have successfully completed a proctored, advanced, examination in software testing.

  • This requirement anticipates competing exams offered by several different groups that endorse different approaches to software testing. Our field does not have agreement on one approach or even one vocabulary. The appearance of agreement that shows up in industry “standards” is illusory. As a matter of practice (I think, often good practice), the standards are routinely ignored by practitioners. Examinations that adopt or endorse these standards should be welcome but not mandatory.

Which exams are suitable and which are advanced?

There is an obvious accreditation issue here. Someone has to decide which exams are suitable and which are advanced.

I am inclined to tentatively define an advanced exam as one that requires as minimum prerequisites (a) successful completion of a specified prior exam and (b) additional education and experience. For example, ISTQB Foundations would not qualify but an ISTQB Advanced or Expert exam might. Similarly, BBST:Foundations would not qualify but BBST:Bug Advocacy might and BBST:Domain Testing definitely should.

An exam might be separate from a course or it might be a final exam in a sufficiently advanced course.

For an exam to be used by a Certified Tester, the organization that offers and grades the exam must provide the Certification Board with a copy of a sample exam. The organization must attest under penalty of perjury that they believe the sample is fairly representative of the scope and difficulty of the actual current exam. This sample will appear on the Certification Board’s website, and be accessible as a link from the Certified Tester’s dossier. (Thus, the dossier doesn’t show the Certified Tester’s actual exam but it does show an exam that is comparable to the actual one.)

What about the reliability and the validity of the exams?

Let me illustrate the problem with two contrasting examples:

  • I think it is fair to characterize ISTQB as an organization that is striving to create highly reliable exams. To achieve this, they are driven toward questions that have unambiguously correct answers. Even in sample essay questions I have seen for the Expert exam, the questions and the expected answers are well-grounded in a published, relatively short, body of knowledge. I think this is a reasonable and respectable approach to assessment and I think that exams written this way should be considered acceptable for this certification.
  • The BBST assessment philosophy emphasizes several other principles over reliability. We expect answers to be clearly written, tightly focused on the question that was asked, with a strong logical argument in favor of whatever position the examinee takes in her or his answer, that demonstrates relevant knowledge of the field. We expect a diversity of points of view. I think it gives the examiner greater insight into the creativity and depth of knowledge of the examinee. I think this is also a reasonable and respectable approach to assessment that we should also consider acceptable for this certification.

There is a tradeoff between these approaches. Approaches like ISTQB’s are focused on the reliability of the exam, especially on between-grader reliability. This is an important goal. The BBST exams are not focused on this. For certification purposes, we would expect to improve BBST reliability by using paired grading (two examiners) but this is imperfect. I would not expect the same level of reliability in BBST exams that ISTQB achieves. However, in my view of the assessment of cognitively complex skills, I believe the BBST approach achieves greater validity. Complicating the issue, there are problems in the measurement of both, reliability and validity, of education-related exams.

The difference here is not just a difference of examination style. I believe it reflects a difference in ideology.

Somehow, the Certification Board will have to find a way to accredit some exams as “sufficiently serious” tests of knowledge even though one is obviously more reliable than the other, one is obviously more tightly based on a published body of knowledge than the other, etc.

Somehow, the Certification Board will have to find a way to refuse to accredit some exams even though they have the superficial form of an exam. In general, I suspect that the Certification Board will cast a relatively broad net and that if groups like ASQ and QAI offer advanced exams, those exams will probably qualify. Similarly, I suspect that a final exam in a graduate-level university course that is an “advanced” software testing course (prerequisite being successful completion of an earlier graduate-level course in testing) would qualify.

Professional Achievement

Professional achievements include publications, honors (such as awards), and other things that indicate that the candidate did something at a professional level.

An applicant for certification does not have to include any professional achievements. However, if the applicant provides them, they will become part of the applicant’s dossier and will be publicly visible.

Some decisions will lie in the discretion of the Certification Board. For example, the Certification Board:

  • might or might not accept an applicant’s academic background as sufficiently relevant (or as sufficiently complete)
  • might or might not accept an applicant’s training-experience portfolio as sufficient or as containing enough courses that are sufficiently related to software testing

In such cases, the Certification Board will consider the applicant’s professional achievements as additional evidence of the applicant’s knowledge of the field.

References

The applicant will provide at least three letters of endorsement from other people who have stature in the field. These letters will be public, part of the Certified Tester’s dossier. An endorsement is a statement from a person that, in that person’s opinion, the applicant has the knowledge, skills and character needed to competently provide the services of a professional software tester. The letter should provide additional details that establish that the endorser knows the knowledge, skill and character of the applicant well enough to credibly make an endorsement.

  • A person of stature is someone who is experienced in the field and respected. For example, the person might be (this is not a complete list)
    • personally known to the Certification Board
    • a Certified Tester
    • a Senior Member or Distinguished Member or Fellow of ACM, ASQ, or IEEE
  • If one of the endorsers withdraws his or her endorsement, that withdrawal will be published in the Certified Tester’s dossier along with the original endorsement (now marked “withdrawn”) and the Certified Tester will be required to get a new endorser.
  • If one of the apparent endorsers contacts the Certification Board and asserts that s/he did not write an endorsement for an applicant and that s/he does not endorse the applicant, and if the apparent endorser provides credible proof of identify, that letter will be published in the Certified Tester’s dossier along with the original letter (now marked “disputed”).

Professional Experience

The applicant will provide a detailed description of his or her professional history that includes at least N years of relevant experience.

  • The applicant must attest that this description is true and not materially incomplete. It will be published as part of the dossier. Potential future employers will be able to check the claims made here against the claims made in the applicant’s application for work with them.
  • The descriptions of relevant positions will include descriptions of the applicant’s role(s) and responsibilities, including typical tasks s/he performed in that position
  • The applicant’s years of relevant experience and years of formal education will interact: Someone with more formal education that is relevant to the field will be able to become certified with less relevant experience (but never less than K years of experience).

Continuing Education

The candidate must engage in professional activities, including ongoing study, to keep the certification.

Code of Ethics

The candidates must agree to abide by a specific Code of Ethics, such as the ACM code. We should foresee this as a prelude to creating an enforcement structure in which a Certified Tester might be censured or certification might be publicly canceled for unethical conduct.

Administrative Issues

Somehow, we have to form a Certification Board. The Board will have to charge a fee for application because the website, the accrediting activities, evaluation of applications, marketing of the certification, etc., will cost money.

Benefits

This collection of material does not guarantee competence, but it does present a multidimensional view of the capability of an experienced person in the field. It speaks to a level of education and professional involvement and to the credibility of self-assertions made when someone applies for a job, submits a paper for publication, etc. I think that the public association of the endorser with the people s/he endorses will encourage most possible endorsers to think carefully about who they want to be permanently publicly identified with. I think the existence of the dossier will discourage exaggeration and fraud by the Certified Tester.

It is not perfect, but I think it will be useful, and better than what I think we have now.

This is not a certification of a baseline of competence in the way that certifications (licenses) work in fields like law, engineering, plumbing, and cosmetology. Those are regulated professions in which the certified person is subject to penalties and civil litigation for conduct that falls below baseline. Software engineering (including software testing) is not a regulated profession, there is no such cause of action in the courts as “software engineering malpractice,” and there are no established penalties for incompetence. There is broad disagreement in the field about whether such regulations should exist (for example, the Association for Computing Machinery strongly opposes the licensing of software engineers while the IEEE seems inclined to support it) and the creation of this certification does not address the desirability of such regulation.

The Current Goal: A Constructive Discussion

This article is a call for discussion. It is not yet a call for action, though I expect we’ll get there soon.

This article follows up an article I wrote last May about credentialing systems. I identified several types of credentials in use in our field and suggested four criteria for a better credential:

  • reasonably attainable (people could affort to get the credential, and reasonably smart people who worked hard could earn it),
  • credible (intellectually and professionally supported by senior people in the field who have earned good reputations),
  • scalable (it is feasible to build an infrastructure to provide the relevant training and assessment to many people), and
  • commercially viable (sufficient income to support instructors, maintainers of the courseware and associated documentation, assessors (such as graders of the students and evaluators of the courses), some level of marketing (because a credential that no one knows about isn’t worth much), and in the case of this group, money left over for profit. Note that many dimensions of “commercial viability” come into play even if there is absolutely no profit motive—-the effort has to support itself, somehow).

I think the proposal in this article sketches a system that would meet those criteria.

A more detailed draft of this proposal was reviewed at the 2014 Workshop on Teaching Software Testing. We did not debate alternative proposals or attempt to reach consensus. The ideas in this paper are not the product of WTST. Nor are they the responsibility of any participant at WTST. However, I am here acknowledging the feedback I got at that meeting and thanking the participants: Scott Allman, Janaka Balasooriya, Rex Black, Jennifer Brock, Reetika Datta, Casey Doran, Rebecca L. Fiedler, Scott Fuller, Keith Gallagher, Dan Gold, Douglas Hoffman, Nawwar Kabbani, Chris Kenst, Michael Larsen, Jacek Okrojek, Carol Oliver, Rob Sabourin, Mike Sowers, and Andy Tinkham. Payson Hall has also questioned the reasoning and offered useful suggestions.

To this point, we have been discussing whether these ideas are worthwhile in principle. That’s important and that discussion should continue.

We have not yet begun to tackle the governance and implementation issues raised by this proposal. It is probably time to start thinking about that.

  • I’m positively impressed by (what I know of) the governance model of ISTQB and wonder whether we should follow that model.
  • I would expect to be an active supporter/contributor to the governance of this project (for example an active member of the governing Board). However—just as I helped found AST but steadfastly refused to run for President of AST—I believe we can find a better choice than me for chief executive of the project.

Comments?

New Book: Foundations of Software Testing–A BBST Workbook

Friday, February 14th, 2014

New Book: Foundations of Software Testing—A BBST Workbook

Rebecca Fiedler and I just published our first book together, Foundations of Software Testing—A BBST Workbook.

Becky and I started working on the instructional design for the online version of the BBST (Black Box Software Testing) course in 2004. Since then,Foundations has gone through three major revisions. Bug Advocacy and Test Design have gone through two.

Our Workbooks mark our first major step toward the next generation of BBST™.

We are creating the new versions of BBST through Kaner, Fiedler & Associates, a training company that we formed to provide an income stream for ongoing evolution of these courses. BBST is a registered trademark of Kaner, Fiedler & Associates.

What’s in the Book

The Workbook includes slides, lecture transcripts, orientation activities and feedback, application activities, exam advice, and author reflections. Here are some some details:

All the course slides

Foundations has 304 slides. Some of these are out of date. We provide notes on these in the Author’s Reflections.

A transcript of the six lectures.

The transcripts are almost word-for-word the same as the spoken lecture. They actually reproduce the script that I wrote for the lecture. In a few cases, my scripts are a little longer than what actually made it past the video edits. We lay the transcript and the slides out together, side-by-side. In an 8.5×11 printed book, this is a great format for taking notes. Unfortunately, it doesn’t translate to Kindle well, so there is no Kindle edition of the book.

Four Orientation Activities

Orientation activities introduce students to a key challenge considered in a lecture. The student puzzles through the activity for 30 to 90 minutes, typically before watching the lecture, then sees how the lecture approaches this type of problem. The typical Foundations course has two to four of these.

The workbook presents the instructions for four activities, along with detailed feedback on them, based on past performance of students in the online and university courses.

I revised, rewrote or added (new) all of these activities for this Workbook. Because, in my opinion, the most important learning in BBST comes from what the students actually do in the class, the new Orientation and Application activities create a substantial revision to the course.

In my university courses, I practice continuous quality improvement, revising all of them every term in response to (a) my sense (and to what ever relevant data I have collected) about strengths and weaknesses that showed up in previous of the course or (b) ideas that have demonstrated their value in other courses and can be imported into this one. Most of the updates are grounded in a long series of revisions that I used and evaluated in my university-course version of BBST.

Two Application Activities

An application activity applies ideas or techniques presented in a lecture or developed over several lectures. The typical application activity calls for two to six hours of work, per student. The typical Foundations course has one to two of these.

One of these is revised from the public BBST, the other completely rewritten.

Advice on answering our essay-style exam questions

The advice runs 11 pages. I also provide a practice question and detailed feedback on the structure of the answer.

I think the advice is good for anyone taking the course, but it is particularly focused on university students who are preparing for an exam that will yield graded results (A, B, Pass-with-distinction, etc.). The commercial versions of BBST are typically pass-fail, so some of the fine details in this advice are beyond the needs of those readers. If you are a university student, I recommend this as a tighter and more polished presentation than the exam-preparation essay included in the public course.

Author reflections

My reflections present my sense of the strengths and weaknesses of the current course, the ways we are addressing those with the new activities, and some of the changes we see coming in the next generation of videos.

Because Foundations is written to introduce students to the fundamental challenges in software testing, some of my reflections add commentary on widely-debated issues in the field. Some of these might become focus points for the usual crowd to practice their Sturm und Drang on Twitter.

Who the Book is For

We want to support three groups with the BBST Workbooks:

  • Self-studiers. Many people watch the course videos on their own. The course videos for the current version of Foundations are available for free, along with the slides, the course readings, and the public-course versions of four activities and the study guide (list of essay questions for the exam). The Workbook updates the activities, and provides detailed feedback for all of the orientation activities, and provides several design notes on the orientation and application activities. If you are studying BBST on your own or with a few friends, we believe this provides much better support than the videos alone.
  • In-house trainers. If you are planning to teach BBST to staff at your company, the Workbook is an inexpensive textbook to support the students. The feedback for the activities provides a detailed survey of common issues and ideas raised in each activity. If your trainees submit their work to you for review, you might want to supplement these notes with comments that are specific to each student’s work. The comments in the workbook should cover most of the comments that you would otherwise repeat from student to student. The instructors’ reflections will, we hope, give you ideas about how to tailor the application activities (or replace them) to make them suitable for your company.
  • Students in instructor-led courses. The BBST Foundations in Software Testing Workbook is an affordably-priced (retail price $19.99) supplement to any instructor-led course. Students will appreciate the convenience of print versions of the course slides and lectures for ongoing reference. Instructors will appreciate the level of feedback provided to students in the workbook.

Buy the Book
Foundations of Software Testing: A BBST Workbook is available from:

Evolution of the BBST Courses

Thursday, February 13th, 2014

With our first teaching of the new BBST:Domain Testing course (based on The Domain Testing Workbook) and our revision of BBST:Foundations with the Foundations of Software Testing workbook, Rebecca Fiedler and I have started to introduce the next generation of BBST. Recently, we’ve been getting requests for papers or interviews on where BBST came from and where it’s going.

  • This note is a short summary of the history of BBST. You can find many more details in the articles I’ve been posting to this blog over the last decade, as I tried to think through the course design’s strengths, weaknesses and potential.
  • My next post, and an upcoming article in Testing Circus, look at the road ahead.

What is BBST™ ?

BBST is a series of courses on Black Box Software Testing. The overall goal of the series is to improve the state of the practice in software testing by helping testers develop useful testing skills and deeper insights into the challenges of the field.

(Note: BBST is a registered trademark of Kaner, Fiedler & Associates.)

Today’s BBST Courses

Today, most people familiar with the BBST courses think of a four-week, fully online course. Rebecca Fiedler and I started working on the instructional design for the BBST online courses back in 2004, with funding from the National Science Foundation. The courses have gotten excellent reviews. We’ve taken Foundations through three major revisions. Bug Advocacy and Test Design have had two. We’re working on our next major update now. You’ll read more about that in my next post.

The typical instructor-led course is organized around six lectures (about six hours of talk, divided into one-hour parts), with a collection of activities. To successfully complete a typical instructor-led course, a student spends about 12-15 hours per week for 4 weeks (48-60 hours total). Most of the course time is spent on the activities:

  • Orientation activities introduce students to a key challenge considered in a lecture. The student puzzles through the activity for 30 to 90 minutes, typically before watching the lecture, then sees how the lecture approaches this type of problem. The typical Foundations course has two to four of these.
  • Application activities call for two to six hours of work. It applies ideas or techniques presented in a lecture or developed over several lectures. The typical Foundations course has one to two of these.
  • Multiple-choice quizzes help students identify gaps in their knowledge or understanding of key concepts in the course. These questions are tough because they are designed to be instructional or diagnostic (to teach you something, to deepen your knowledge of something, or to help you recognize that you don’t understand something) rather than to fairly grade you.
  • Various other discussions that help the students get to know each other better, chew on the course’s multiple-choice quiz questions, or consider other topics of current interest.
  • An essay-style final exam.

In the instructor-led course, students get feedback on the quality of their work from each other and, to a lesser or greater degree (depends on who’s teaching the course), they get feedback from the instructors. Students in our commercial courses (which we offer through Kaner, Fiedler & Associates) get a lot of feedback. Students in courses taught by unpaid volunteer instructors are more likely to get most of their feedback from the other students.

So, that’s today (in the online course).

However, BBST has actually been around for 20 years.

Background on the BBST Course Design

I started teaching BBST in 1994, with Hung Quoc Nguyen, for the American Society for Quality in Silicon Valley. This was the commercial version of the course (taught to people working as testers). Development of the course was significantly influenced by:

  • Detailed peer reviews of the live class and of the circulating slide decks. The reviews included detailed critiques from colleagues when I made significant course updates (I offered free beta-review classes to test the updates).
  • Co-teaching the material with colleagues. We would learn together by cross-teaching material, often challenging points in each other’s slides or lecture in front of the students. For example, I taught with James Bach, Elisabeth Hendrickson, Doug Hoffman and (for the metrics material) Pat Bond, a professor at Florida Tech.
  • Rational Software, which contracted with me to create a customized version of BBST to support testing under the Rational Unified Process. They criticized the course in detail over several pilot teachings, and allowed me to apply what I learned back to the original course.

In 1999, I decided that if I wanted to learn how to significantly improve the instructional value of the course, I was going to have to see how teachers help students learn complex topics and skills in university. My sense was, and is, that good university instruction goes much deeper and demands more from the students than most commercial training.

Florida Tech hired me in 2000 to teach software engineering and encouraged me to evolve BBST down two parallel tracks:

  • a core university course that would be challenging for our graduate students and a good resume-builder when our students looked for jobs
  • a stronger commercial course that demanded more from the students.

We correctly expected that the two tracks would continually inform each other. Getting feedback from practitioners would help us keep the academic stuff real and useful. Trying out instructional ideas in the classroom would give us ideas for redesigning the learning experience of commercial students.

By 2003, I realized that most of my students were doing most of their learning outside the classroom. They claimed to like my lectures, but they were learning from assignments and discussions that happened out of the classroom. In 2004, I decided to try taping the lectures. The students could watch these at home, while we did activities in the classroom that had previously been done out of class. This went well, and in 2005, I created a full set of course videos.

I used the 2005 videos in my own classes. I put a Creative Commons license on the videos and posted them, along with other supporting materials, on my lab’s website. Rebecca Fiedler and I also started giving talks to educators about our results, such as these two papers (Association for Educational & Communications Technology conference and the Sloan Conference on Asynchronous Learning Networks in 2005).

These days, what we were doing has a name (“flipping“) and the Open Courseware concept is old news. Back then, it was still hard to find examples of other people doing this. Even though many other people were experimenting with the same ideas, not many people were yet publishing and so we had to puzzle through the instructional ideas by reading way too much stuff and thinking way too hard about way too many conflicting opinions and results. We summarized our own design ideas in the 2005 presentations (cited above). A good sample of the literature we were reading appeared in our applications for funding to the National Science Foundation, such as the one that was funded (2007), which gave us the money to pay graduate students to help with evaluation and redesign of the course (yielding the current public version).

For readers interested in the “science” that informed our course design, I’m including an excerpt from the 2007 Grant Proposal at the end of this article.

The Collaboration with AST

While we were sketching the first BBST videos, we were also working to form AST (the Association for Software Testing). AST incorporated in 2004. Perhaps a year later, Rebecca and I decided that the academic version of online BBST could probably be adapted for working testers. The AST activists at that time were among my closest professional friends, so it was natural to bring this idea to them.

We began informally, of course. We started by posting a set of videos on a website, but people kept asking for instructor support—for a “real” class. By this point (late 2006), the Florida Tech course was maturing and I was confident (in retrospect, laughably overconfident) that I could translate what was working in a mixed online-plus-face-to-face university class to a fully online course for practitioners located all over the world. The result worked so badly that everyone dropped out (even the instructors).

We learned a lot from the details of the failure, looked more carefully at how other university instructors had redesigned traditional academic courses to make them effective for remote students who had full-time jobs and who probably hadn’t sat in an academic classroom for several years (so their academic skills were rusty). After a bunch more pilot-testing, I offered the first BBST:Foundations as a one-month class (essentially the modern structure) in October, 2007.

We offered BBST:Foundations through AST, adding BBST:Bug Advocacy in 2008, redoing BBST:Foundations (new slides, videos, etc.) in 2010, and adding BBST:Test Design in 2012.

AST was our learning lab for commercial courseware. Florida Tech’s testing courses, and my graduate research assistants at Florida Tech, were my learning lab for the academic courseware. I would try new ideas at Florida Tech and bring the ones that seemed promising into the AST courses as minor or major updates. All the while, I was publishing the courseware at my lab’s website, testingeducation.org, and encouraging other people to use the material in their courses.

We trained and supervised a crew of volunteer instructors for AST’s BBST, but other people were teaching the course (or parts of it) too. This included professors, commercial trainers, managers teaching their own staff how to test, etc. Becky created an instructor’s course (how to teach BBST), which we offered as an instructor-led course through AST but which we also offered as a free learning experience on the web (study it yourself at your own pace). In 2012, we published a 357-page Instructor’s Manual for BBST. We published the book as a Technical Report (a publication method available to university professors) so that we could supply it to the public for free.

Underlying much of the AST collaboration was a hope that we could create an open courseware community that would function like some of the successful open software communities.

  • In the open source software world, many of the volunteers who maintain and enhance open source software are able to charge people for support services. That is, the software (courseware) is free but if you want support, you have to pay for it. The support money creates an income stream that makes it possible for skilled people to spend time improving the software.
  • We hoped that we could create a similar type of structure for open source courseware (the BBST courses). You can see the thinking, for example, in a 2008 paper that Rebecca and I wrote with Scott Barber, Building a free courseware community around an online software testing curriculum.

It turns out that this is a very complex idea. It is probably too complex for a small professional society that handles most of its affairs pretty informally.

For now, Rebecca and I have formed Kaner, Fiedler & Associates to sustain BBST instead. That is, KFA sells commercial BBST training and the income stream makes it possible for us to make new-and-improved versions of BBST.

AST might also create its own project to maintain and enhance BBST. If so, we’ll probably see the evolution of contrasting designs for the next generations of the courses. We think we’d learn a lot from that dynamic and we hope that it happens.

An Excerpt from our 2007 Grant Proposal

This is from our application for NSF Award CCLI-0717613, Adaptation and Implementation of an Activity-Based Online or Hybrid Course in Software Testing. (When we acknowledge support from NSF, we are required to remind you that National Science Foundation does not endorse any opinions, findings, conclusions or recommendations that arose out of NSF-funded research.) The full application is available online but it is very concisely written, structured according to very specific NSF guidelines, and packed with points that address NSF-specific concerns. Here is the most relevant section of that 56-page document here, in terms of explaining our approach and literature review for the course’s instructional design.

3. Our Current Course (Black Box Software Testing—BBST)

We adopted the new teaching method in Spring 2005 after pilot work in 2004. Our new approach spends precious student contact hours on active learning experiences (more projects, seminars and labs) that involve real-world problems, communication skills, critical thinking, and instructor scaffolding [129, 136] without losing the instructional benefits of polished lectures. Central to a problem-based learning environment is that students focus on “becoming a practitioner, not simply learning about practice” [122, p. 3]

Anderson et al.’s [11] update to Bloom’s taxonomy [20] is two-dimensional, knowledge and cognitive processing.

  • On the Knowledge dimension, the levels are Factual Knowledge (such as the definition of a software testing technique), Conceptual Knowledge (such as the theoretical model that predicts that a given test technique is useful for finding certain kinds of bugs), Procedural Knowledge (how to apply the technique), and Metacognitive Knowledge (example: the tester decides to study new techniques on realizing that the ones s/he currently knows don’t apply well to the current situation.)
  • On the Cognitive Process dimension, the levels are Remembering (such as remembering the name of a software test technique that is described to you), Understanding (such as being able to describe a technique and compare it with another one), Applying (actually doing the technique), Analyzing (from a description of a case in which a test technique was used to find a bug, being able to strip away the irrelevant facts and describe what technique was used and how), Evaluating (such as determining whether a technique was applied well, and defending the answer), and Creating (such as designing a new type of test.).

For most of the material in these classes, we want students to be able to explain it (conceptual knowledge, remembering, understanding), apply it (procedural knowledge, application), explain why their application is a good illustration of how this technique or method should be applied (understanding, application, evaluation), and explain why they would use this technique instead of some other (analysis).

3.1 We organize classes around learning units that typically include:

  • Video lecture and lecture slides. Students watch lectures before coming to class. Lectures can convey the lecturer’s enthusiasm, which improves student satisfaction [158] and provide memorable examples to help students learn complex concepts, tasks, or cultural norms [47, 51, 115]. They are less effective for teaching behavioral skills, promoting higher-level thinking, or changing attitudes or values [19]. In terms of Bloom’s taxonomy [11, 20], lectures would be most appropriate for conveying factual and conceptual knowledge at the remembering and understanding levels. Our students need to learn the material at these levels, but as part of the process of learning how to analyze situations and problems, apply techniques, and evaluate their own work and the work of their peers. Stored lectures are common in distance learning programs [138]. Some students prefer live lectures [45, 121] but on average, students learn as well from video as live lecture [19, 139]. Students can replay videos [53] which can help students whose first language is not English. Web-based lecture segments supplement some computer science courses [34, 44]. Studio-taped, rehearsed lectures with synchronously presented slides (like ours) have been done before [29]. Many instructors tape live lectures, but Day and Foley [30-34] report their students prefer studio-produced lectures over recorded live lectures. We prefer studio-produced lectures because they have no unscripted interruptions and we can edit them to remove errors and digressions.
  • Application to a product under test. Each student joins an open source software project (such as Open Office or Firefox) and files work with the project (such as bug reports in the project’s bug database) that they can show and discuss during employment interviews. This helps make concepts “real” to students by situating them in the development of well-regarded products [118]. It facilitates transfer of knowledge and skills to the workplace, because students are doing the same tasks and facing the same problems they would face with commercial software [25]. As long as the assignments are not too far beyond the skill and knowledge level of the learner, authentic assignments yield positive effects on retention, motivation, and transfer [48, 52, 119, 153].
  • Classroom activities. We teach in a lab with one computer per student. Students work in groups. Activities are open book, open web. The teacher moves from group to group asking questions, giving feedback, or offering supplementary readings that relate to the direction taken by an individual group. Classroom activities vary. Students might apply ideas, practice skills, try out a test tool, explore ideas from lecture, or debate a question from the study guide. Students may present results to the class in the last 15 minutes of the 75-minute class. They often hand in work for (sympathetic) grading: we use activity grades to get attention [141] and give feedback, not for high-stakes assessment. We want students laughing together about their mistakes in activities, not mourning their grades [134].
  • Examples. These supplementary readings or videos illustrate application of a test technique to a shipping product. Worked examples can be powerful teaching tools [25], especially when motivated by real-life situations. They are fundamental for some learning styles [43]. Exemplars play an important role in the development and recollection of simple and complex concepts [23, 126, 146]. The lasting popularity of problem books, such as the Schaum’s Outline series and more complex texts like Sveshnikov [148] attests to the value of example-driven learning, at least for some learners. However, examples are not enough to carry a course. In our initial work under NSF Award EIA-0113539 ITR/SY+PE: Improving the Education of Software Testers, we expected to be able to bring testing students to mastery of some techniques through practice with a broad set of examples. Padmanabhan [113, 132] applied this to domain testing in her Master’s thesis project at Florida Tech, providing students with 18 classroom hours of instruction, including lecture, outlines of ways to solve problems, many practice exercises and exams. Students learned exactly what they were taught. They could solve new problems similar to those solved in class. However, in their final exam, we included a slightly more complicated problem that required them to apply their knowledge in a way that had been described in lecture but not specifically practiced. The students did the same things well, in almost exactly the same ways. However, they all failed to notice problems that should have been obvious to them but that only required a small stretch from their previous drills. This result was a primary motivator for us to redesign the testing course from a lecture course heavy with stories, examples and practice to more heavily emphasize more complex activities.
  • Assigned readings.
  • Assignments, which may come with grading rubrics. These are more complex tasks than in-class activities. Students typically work together over a two-week period.
  • Study guide questions. At the start of the course, we give students a list of 100 questions. All midterm and final exam questions come from this pool. We discuss use and grading of these questions in [60] and make that paper available to students. We encourage group study, especially comparison of competing drafts of answers. We even host study sessions in a café off campus (buying cappuccinos for whoever shows up). We encourage students to work through relevant questions in the guide at each new section of the class. These help self-regulated learners monitor their progress and understanding—and seek additional help as needed. They can focus their studying and appraise the depth and quality of their answers before they write a high-stakes exam. Our experience of our students is consistent with Taraban, Rynearson, & Kerr’s [149]—many students seem not to be very effective readers or studiers, nor very strategic in the way they spend their study time—as a result, they don’t do as well on exams as we believe they could. Our approach gives students time to prepare thoughtful, well-organized, peer-reviewed answers. In turn, this allows us to require thoughtful, well-organized answers on time-limited exams. This maps directly to one of our objectives (tightly focused technical writing). We can also give students complex questions that require time to carefully read and analyze, but that don’t discriminate against students whose first language is not English because these students have the questions well in advance and can seek guidance on the meaning of a question.

Excerpt from the Proposal’s references:

11. Anderson, L.W., Krathwohl, D.R., Airasian, P.W., Cruikshank, K.A., Mayer, R.A., Pintrich, P.R., Raths, J. and Wittrock, M.C. A Taxonomy for Learning, Teaching &  Assessing: A Revision of Bloom’s Taxonomy of Educational Objectives. Longman, New York, 2001.

19. Bligh, D.A. What’s the Use of Lectures? Jossey-Bass, San Francisco, 2000.

20. Bloom, B.S. (ed.), Taxonomy of Educational Objectives: Book 1 Cognitive Domain. Longman, New York, 1956

23. Brooks, L.R. Non-analytic concept formation and memory for instances. in Rosch, E. and Lloyd, B.B. eds. Cognition and categorization, Erlbaum, Hillsdale, NJ, 1978, 169-211.

25. Clark, R.C. and Mayer, R.E. e-Learning and the Science of Instruction. Jossey-Bass/Pfeiffer, San Francisco, CA, 2003.

29. Dannenberg, R.P. Just-In-Time Lectures, undated.

30. Day, J.A. and Foley, J. Enhancing the classroom learning experience with Web lectures: A quasi-experiment GVU Technical Report GVU-05-30, 2005.

31. Day, J.A. and Foley, J., Evaluating Web Lectures: A Case Study from HCI. in CHI ’06 (Extended Abstracts on Human Factors in Computing Systems), (Quebec, Canada, 2006), ACM Press, 195-200. Retrieved January 4, 2007, from http://www3.cc.gatech.edu/grads/d/Jason.Day/documents/er703-day.pdf

32. Day, J.A., Foley, J., Groeneweg, R. and Van Der Mast, C., Enhancing the classroom learning experience with Web lectures. inInternational Conference of Computers in Education, (Singapore, 2005), 638-641. Retrieved January 4, 2007, fromhttp://www3.cc.gatech.edu/grads/d/Jason.Day/documents/ICCE2005_Day_Short.pdf

33. Day, J.A., Foley, J., Groeneweg, R. and Van Der Mast, C. Enhancing the classroom learning experience with Web lecturesGVU Technical Report GVU-04-18, 2004.

34. Day, J.A. and Foley, J.D. Evaluating a web lecture intervention in a human-computer interaction course. IEEE Transactions on Education49 (4). 420-431. Retrieved December 31, 2006.

43. Felder, R.M. and Silverman, L.K. Learning and teaching styles in engineering education. Engineering Education78 (7). 674-681.

44. Fintan, C., Lecturelets: web based Java enabled lectures. in Proceedings of the 5th annual SIGCSE/SIGCUE ITiCSE Conference on Innovation and technology in computer science education, ( Helsinki, Finland, 2000), 5-8.

45. Firstman, A. A comparison of traditional and television lectures as a means of instruction in biology at a community college., ERIC, 1983.

47. Forsyth, D., R. The Professor’s Guide to Teaching: Psychological Principles and Practices. American Psychological Association, Washington, D.C., 2003.

48. Gagne, E.D., Yekovich, C.W. and Yekovich, F.R. The Cognitive Psychology of School Learning. HarperCollins, New York, 1994.

51. Hamer, L. A folkloristic approach to understanding teachers as storytellers. International Journal of Qualitative Studies in Education12 (4). 363-380, from http://ejournals.ebsco.com/direct.asp?ArticleID=NLAW20N8B16TQKHDEECM.

52. Haskell, R.E. Transfer of learning: Cognition, instruction, and reasoning. Academic Press, San Diego, 2001.

53. He, L., Gupta, A., White, S.A. and Grudin, J. Corporate Deployment of On-demand Video: Usage, Benefits, and Lessons, Microsoft Research, Redmond, WA, 1998, 12.

60. Kaner, C., Assessment in the software testing course. in Workshop on the Teaching of Software Testing (WTST), (Melbourne, FL, 2003), from http://www.kaner.com/pdfs/AssessmentTestingCourse.pdf

113. Kaner, C. and Padmanabhan, S., Practice and transfer of learning in the teaching of software testing. in Conference on Software Engineering Education & Training, (Dublin, 2007).

115. Kaufman, J.C. and Bristol, A.S. When Allport met Freud: Using anecdotes in the teaching of Psychology. Teaching of Psychology28 (1). 44-46.

118. Lave, J. and Wenger, E. Situated Learning: Legitimate Peripheral Participation. Cambridge University Press, Cambridge, England, 1991.

119. Lesh, R.A. and Lamon, S.J. (eds.). Assessment of authentic performance in school mathematics. AAAS Press, Washington, DC, 1992.

121. Maki, W.S. and Maki, R.H. Multimedia comprehension skill predicts differential outcomes of web-based and lecture courses.Journal of Experimental Psychology: Applied8 (2). 85-98.

122. MaKinster, J.G., Barab, S.A. and Keating, T.M. Design and implementation of an on-line professional development community: A project-based learning approach in a graduate seminar Electronic Journal of Science Education, 2001.

126. Medin, D. and Schaffer, M.M. Context theory of classification learning. Psychological Review85 (207-238)

129. National Panel Report. Greater Expectations: A New Vision for Learning as a Nation Goes to College, Association of American Colleges and Universities, Washington, D.C., 2002.

132. Padmanabhan, S. Domain Testing: Divide & Conquer Department of Computer Sciences, Florida Institute of Technology, Melbourne, FL, 2004.

134. Paris, S.G. Why learner-centered assessment is better than high-stakes testing. in Lambert, N.M. and McCombs, B.L. eds.How Students Learn: Reforming Schools Through Learner-Centered Education, American Psychological Association, Washington, DC, 1998.

136. Project Kaleidoscope. Project Kaleidoscope Report on Reports: Recommendations for Action in support of Undergraduate Science, Technology, Engineering, and Mathematics. Investing in Human Potential: Science and Engineering at the Crossroads, Washington, D.C., 2002. Retrieved January 16, 2006, from http://www.pkal.org/documents/RecommentdationsForActionInSupportOfSTEM.cfm.

138. Rossman, M.H. Successful online teaching using an asynchronous learner discussion forum. Journal of Asynchronous Learning Networks3 (2), from http://www.sloan-c.org/publications/jaln/v3n2/v3n2_rossman.asp.

139. Saba, F. Distance education theory, methodology, and epistemology: A pragmatic paradigm. in Moore, M.G. and Anderson, W.G. eds. Handbook of Distance Education, Lawrence Erlbaum Associates, Mahwah, New Jersey, 2003, 3-20.

141. Savery, J.R. and Duffy, T.M. Problem Based Learning: An Instructional Model and Its Constructivist Framework, Indiana University, Bloomington, IN, 2001.

146. Smith, D.J. Wanted: A New Psychology of Exemplars. Canadian Journal of Psychology59 (1). 47-55

148. Sveshnikov, A.A. Problems in probability theory, mathematical statistics and theory of random functions. Saunders, Philadelphia, 1968

149. Taraban, R., Rynearson, K. and Kerr, M. College students’ academic performance and self-reports of comprehension strategy use. Reading Psychology21 (4). 283-308.

153. Van Merrienboer, J.J.G. Training complex cognitive skills: A four-component instructional design model for technical training. Educational Technology Publications, Englewood Cliffs, NJ, 1997.

158. Williams, R.G. and Ware, J.E. An extended visit with Dr. Fox: Validity of student satisfaction with instruction ratings after repeated exposures to a lecturer. American Educational Research Journal14 (4). 449-457.

On the design of advanced courses in software testing

Sunday, January 19th, 2014

This year’s Workshop on Teaching Software Testing (WTST 2014) is on teaching advanced courses in software testing. During the workshop, I expect we will compare notes on how we design/evaluate advanced courses in testing and how we recognize people who have completed advanced training.

This post is an overview of one of the two presentations I am planning for WTST.

This presentation will consider the design of the courses. The actual presentation will rely heavily on examples, mainly from BBST (Foundations, Bug Advocacy, Test Design), from our new Domain Testing course, and from some of my non-testing courses, especially statistics and metrics. The slides that go with these notes will appear at the WTST site in late January or early February.

In the education community, a discussion like this would come as part of a discussion of curriculum design. That discussion would look more broadly at the context of the curriculum decisions, often considering several historical, political, socioeconomic, and psychological issues. My discussion is more narrowly focused on the selection of materials, assessment methods and teaching-style tradeoffs in a specialized course in a technical field. The broader issues come into play, but I find it more personally useful to think along six narrower dimensions:

  • content
  • institutional considerations
  • skill development
  • instructional style
  • expectations of student performance
  • credentialing

Content

In terms of the definition of “advanced”, I think the primary agreement in the instructional community is that there is no agreement about the substance of advanced courses. A course can be called advanced if it builds on other courses. Under this institutional definition, the ordering of topics and skills (introductory to advanced) determines what is advanced, but that ordering is often determined by preference or politics rather than by principle.

I am NOT just talking here about fields whose curricula involve a lot of controversy. Let me give an example. I am currently teaching Applied Statistics (Computer Science 2410). This is parallel in prerequisites and difficulty to the Math department’s course on Mathematical Statistics (MTH 2400). When I started teaching this, I made several assumptions about what my students would know, based on years of experience with the (1970′s to 1990′s) Canadian curriculum. I assumed incorrectly that students would learn very early about the axioms underlying algebra—this was often taught as Math 100 (1st course in the university curriculum). Here, it seems common to find that material in 3rd year. I also assumed incorrectly that my students would be very experienced in the basics of proving theorems. Again mistaken, and to my shock, many CS students will graduate, having taken several required math courses, with minimal skills in formal logic or theorem proof. I’m uncomfortable with these choices (in the “somebody moved my cheese” sense of the word “uncomfortable”)—it doesn’t feel right, but I am confident that these students studied other topics instead, topics that I would consider 3rd-year or 4th-year. Even in math, curriculum design is fluid and topics that some of us consider foundational, others consider advanced.

In a field like ours (testing) that is far more encumbered with controversy, there is a strong argument for humility when discussing what is “foundational” and what is “advanced”.

Institutional Considerations

In my experience, one of the challenges in teaching advanced topics is that many students will sign up who lack basic knowledge and skills, or who expect to use this course as an opportunity to relitigate what they learned in their basic course(s). This is a problem in commercial and university courses, but in my experience, it is much easier to manage in a university because of the strength and visibility of the institutional support.

To make space for advanced courses, institutions that designate a courses as advanced are likely to

  • state and enforce prerequisites (courses that must be taken, or knowledge/skill that must be demonstrated before the student can enrol in the advanced course)
  • accept transfer credit (a course can be designated as equivalent to one of the institution’s courses and serve as a prerequisite for the advanced course)

The designation sets expectations. Typically, this gives instructors room to:

  1. limit class time spent rehashing foundational material
  2. address topics that go beyond the foundational material (whatever material this institution has designated as foundational)
  3. tell students who do not know the foundational material (or who cannot apply it to the content of the advanced course) that it is their responsibility to catch up to the rest of the class, not the course’s responsibility to slow down for them
  4. demand an increased level of individual performance from the students (not just work products on harder topics, but better work products that the student produces with less handholding from the instructor)

Note clearly that in an institution like a university, the decisions about what is foundational, what is advanced, and what prerequisites are required for a particular course are made by groups of instructors, not by the administrators of the institution. This is an idealized model–it is common for institutional administrators to push back, encouraging instructors to minimize the number of prerequisites they demand for any particular course and encouraging instructors to take a broader view of equivalence when evaluating transfer credits. But at its core, the administration adopts structures that support the four benefits that I listed above (and probably others). I think this is the essence of what we mean by “protecting the standards” of the institution.

Skill Development

I think of a skill as a type of knowledge that you can apply (you use it, rather than describe it) and your application (your peformance) improves with deliberate practice.

Students don’t just learn content in courses. They learn how to learn, how to investigate and find/create new ideas or knowledge on their own, how to find and understand the technical material of their field, how to critically evaluate ideas and data, how to communicate what they know, how to work with other students, and so on. Every course teaches some of these to some degree. Some courses are focused on these learning skills.

Competent performance in a professional field involves skills that go beyond the learning skills. For example, skills we must often apply in software testing include:

  • many test design techniques (domain testing, specification-based testing, etc.). Testers get better with these through a combination of theoretical instruction, practice, and critical feedback
  • many operational tasks (setting up test systems, running tests, noting what happened)
  • many advanced communication skills (writing that combines technical, persuasive and financial considerations)

Taxonomies like Bloom’s make the distinction between memorizable knowledge and application (which I’m describing as skill here). Some courses, and some exams, are primarily about memorizable knowledge and some are primarily about application.

In general, in my own teaching, I think of courses that focus on memorizable knowledge as survey courses (broad and shallow). I think of survey courses as foundational rather than advanced.

Most survey courses involve some application. The student learns to apply some of the content. In many cases, the student can’t understand the content without learning to apply it at least to simple cases. (In our field, I think domain testing–boundary and equivalence class analysis–is like this.) It seems to me that courses run on a continuum, how much emphasis on learning things you can remember and describe versus learning ways to apply knowledge more effectively. I think of a course that is primarily a survey course as a survey course, even if it includes some application.

Instructional Style

Lecture courses are probably the easiest to design and the easiest to sell. Commercial and university students seem to prefer courses that involve a high proportion of live lecture.

Lectures are effective for introducing students to a field. They introduce vocabulary (not that students remember much of it–they forget most of what they learn in lecture). They convey attitudes and introduce students to the culture of the field. They can give students the sense that this material is approachable and worth studying. And they entertain.

Lectures are poor vehicles for application of the material (there’s little space for students to try things out, get feedback and try them again).

In my experience, they are usually also poor vehicles for critical thinking (evaluating the material). Some lecturers develop a style that demands critical thinking from the students (think of law schools) but I think this requires very strong cultural support. Students understand, in law school, that they will flunk out if they come to class unprepared and are unwilling or unable to present and defend ideas quickly, in response to questions that might come from a professor at any time. Lawyers view the ability to analyze, articulate and defend in real time as a core skill in their field and so this approach to teaching is considered appropriate. In other fields that don’t prioritize oral argumentation so highly, a professor who relied on this teaching style and demanded high performance from every student, would be treated as unusual and perhaps inappropriate.

As students progress from basic to advanced, the core experiences they need to support further progress also change, from lecture to activities that require them to do more–more applications to increasingly complex tasks, more critical evaluation of what they are doing, what others are doing, and what they are being told to do or to accept as correct or wise. Fewer things are correct. More are better-for-these-reasons or better-for-these-purposes.

Expectations of Student Performance

More advanced courses demand that students take more responsibility for the quality of their work:

  • The students expect, and tolerate, less specific instructions. If they don’t understand the instructions, the students understand that it is their responsibility to ask for clarification or to do other research to fill in the blanks.
  • The students don’t expect (or know they are not likely to get) worked examples that they can model their answers from or rubrics (step-by-step evaluation guides) that they can use to structure their answers. These are examples of scaffolding, instructional support structures to help junior students accomplish new things. They are like the training wheels on bicycles. Eventually, students have to learn to ride without them. Not just how to ride down this street for these three blocks, but how to ride anywhere without them. Losing the scaffolding is painful for many students and some students protest emphatically that it is unfair to take these away. I think the trend in many universities has been to provide more scaffolding for longer. This cuts back on student appeals and seems to please accreditors (university evaluators) but I think this delays students’ maturation in their field (and generally in their education).

One of the puzzles of commercial instruction is how to assess student performance. We often think of assessment in terms of passing or failing a course. However, assessment is more broadly important, for giving a student feedback on how well she knows the material or how well she does a task. There has been so much emphasis on high-stakes assessment (you pass or you fail) in academic instruction that many students don’t understand the concept of formative assessment (assessment primarly done to give the student feedback in order to help the student learn). This is a big issue in university instruction too, but my experience is that commercial students are more likely to be upset and offended when they are given tough tasks and told they didn’t perform well on them. My experience is that they will make more vehement demands for training wheels in the name of fairness, without being willing to accept the idea that they will learn more from harder and less-well-specified tasks.

Things are not so well specified at work. More advanced instruction prepares students more effectively for the uncertainties and demands of real life. I believe that preparation involves putting students into uncertain and demanding situations, helping them accept this as normal, and helping them learn to cope with situations like these more effectively.

Credentialing

Several groups offer credentials in our field. I wrote recently about credentialing in software testing at http://kaner.com/?p=317. My thoughts on that will come in a separate note to WTST participants, and a separate presentation.

Last call for WTST 2014

Sunday, November 24th, 2013

This year’s Workshop on Teaching Software Testing is focused on designing and teaching advanced courses in software testing. It is in sunny Florida, in late January 2014. Right after WTST, we will teach a 5-day pilot of the Domain Testing course. You can apply to attend either one.

We expect the WTST discussion to flow down two paths. At this point, we are not sure which will dominate:

1. What are the characteristics of a genuinely “advanced” testing course?

What are people teaching or designing at this level and what design decisions and assessment decisions are they making? What courses should we be designing?

2. What should the characteristics be for an advanced certification in software testing?

I’ve been criticizing the low bar set by ISTQB’s, QAI’s, and ASQ’s certifications for over 15 years. From about 1996 to (maybe it was) 2003, I worked with several colleagues on ideas for a better certification. As I pointed out recently, those ideas failed. We couldn’t find a cost-effective solution that met our quality standards. I moved on to other challenges, such as creating the BBST series. Some others adopted a more critical posture toward certification in general.

Looking back, I think the same problems that motivated thousands of testers (and employers) to seek a credentialing system for software testers are still with us. The question, I think, is not whether we need a good credentialing system. The question is whether we can get a credentialing system that is good.

From some discussions about advanced course design, I think we are likely to see a discussion of advanced credentialing at WTST. The idea that ties this discussion to WTST is that the credential would be based at least partially on performance in advanced courses.

I don’t know whether this discussion will go very far, whether it will be a big part of the meeting itself or just the after-meeting dinners, or whether anyone will come to any agreements. But if you are interested in participating in a constructive discussion about a very hard problem, this might be a good meeting.

To apply to come to WTST, please send me a note (kaner@cs.fit.edu).

For more information about WTST, see http://wtst.org/. For more on the first pilot teaching of the Domain Testing course, which we will teach immediately following WTST, see http://bbst.info.

New Book: The Domain Testing Workbook

Saturday, November 16th, 2013

Sowmya Padmanabhan, Doug Hoffman and I just published a new book together, The Domain Testing Workbook.

The book focuses on one (1) testing technique. Domain Testing is the name of a generalized approach to Equivalence Class Analysis and Boundary Testing. Our goal is to help you develop skill with this one technique. There are plenty of overviews of this technique: it is the most widely taught, and probably the best understood, technique in the field. However, we’re not aware of any other presentations that are focused on helping you go beyond an understanding of the technique to achieving the ability to use it competently.

This is the first of a series of books that are coming along slowly. Probably the next one will be on scenario testing, with Dr. Rebecca Fiedler, merging her deep knowledge of qualitative methodology with my hands-on use of scenarios in software design, software testing, and software-related human factors. Also in the works are books on risk-based testing and specification-based testing. We’ve been working on all of these for years. We learned much from the Domain Testing Workbook about how challenging it is to write a book with a development-of-skill instructional design goal. At this point, we have no idea what the schedule is for the next three. When the soup is ready, we’ll serve it.

This work comes out of a research proposal that I sent to the United States’ National Science Foundation (NSF) in 2001 (Improving the Education of Software Testers), which had, among its objectives:

  • “Identify skills involved in software testing.”
  • “Identify types of exercises that support the development of specific testing skills.”
  • “Create and publish a collection of reusable materials for exercises and tests.”
  • “Prototype a web-based course on software testing.”
  • “Create a series of workshops that focus on the teaching of software testing”

NSF approved the project, which made it possible for us to open the Center for Software Testing Education & Research (CSTER). The web-based course we had in mind is now available as the BBST series. The Workshops on Teaching Software Testing are now in their 13th year. And the Domain Testing Workbook is our primary contribution focused on “exercises that support the development of specific testing skills.”

When we started this project, we knew that domain testing would be the easiest technique to write this kind of book about. Over the years, we had two surprises that caused us to fundamentally rethink the structure of this book (and of other books that I’m still working on that are focused on scenario testing, risk-based testing, and specification-based testing):

  • Our first surprise (or better put, our first shock) was that we could teach students a broad collection of examples of the application of domain testing, confirm that they could apply what they had learned to similar problems, and yet these students would fail miserably when we gave them a slightly more complex problem that was a little different than they had seen before. This was the core finding of Sowmya’s M.Sc. thesis. What we had tripped over was the transfer problem, which is probably the central instructional-design problem in Science, Technology, Engineering & Mathematics (STEM) instruction. The classic example is of the student who did well in her or his calculus course but cannot figure out how to apply calculus to problems (like modeling acceleration) in their introductory physics course. These students can’t transfer what they learned in one course to another course or to more complex situations (such as using it at their job). We had believed that we could work around the transfer problem by building our instruction around realistic exercises/examples. We were mistaken.
    • Ultimately, we concluded that teaching through exercises is still our best shot at helping people develop skill, but that we needed to provide a conceptual structure for the exercises that could give students a strategy for approaching new problems. We created a schema—an 18-step cognitive structure that describes how we do a domain analysis—and we present every exercise and every example in the context of that schema. We haven’t done the formal, experimental research to check this that Sowmya was able to do with our initial approach—an experiment like that is time-consuming and expensive and our funding for that type of work ran out long ago. However, we have inflicted many drafts of the schema on our students at Florida Tech and we believe it improves their performance and organizes their approach to tasks like exams.
  • Our next surprise was that domain testing is harder to apply than we expected. Doug and I are experienced with this technique. We think we’re pretty good at it, and we’ve thought that for a long time. Many other testers perceive us that way too. For example, Testing Computer Software opens with an example of domain testing and talks about the technique in more detail throughout the book. That presentation has been well received. So, when we decided to write a book with a few dozen examples that increased in complexity, we were confident that we could work through the examples quickly. It might take more time to write our analysis in a way that readers could understand, but doing the analysis would be straightforward. We were sooooooo wrong. Yes, we could quickly get to the core of the problem, identifying how we would approach identifying equivalence classes and boundaries (or non-boundary most-powerful test cases) for each class. Yes, we could quickly list several good tests. We got far enough along that we could impress other people with what we knew and how quickly we could get there, but not far enough to complete the problem. We were often getting through about a third of the analysis before getting stuck. Doug would fly to Palm Bay (Florida, where I live) and we would work problems. Some problems that we expected to take a day to solve and explain took us a week.
    • As we came to understand what skills and knowledge we were actually applying when we slogged through the harder problems, we added more description of our background knowledge to the book. A presentation of our Domain Testing Schema—and the thinking behind it—grew from an expected length of about 30 pages to 200. Our presentation of 30 worked examples grew from an expected length of maybe 120 pages (4 pages each, with diagrams) to 190 pages.

We got a lot of help with the book. Our acknowledgments list 91 people who helped us think through the ideas in the book. Many others helped indirectly, such as many participants in the WTST workshops who taught us critical lessons about the instructional issues we were beating our heads against.

Perhaps the main advance in the clarity of the presentation came out of gentle-but-firm, collegial prodding by Paul Gerrard. Paul’s point was that domain testing is really a way for a tester to model the software. The tests that come out of the analysis are not necessarily tied to the actual code. They are tied to the tester’s mental model of the code. Our first reactions to these comments were that Paul was saying something obvious. But over time we came to understand his point—it might be obvious to us, but presentations of domain testing, including ours, were doing an inadequate job of making it obvious to our readers. This led us to import the idea of a notional variable from financial analysis as a way of describing the type of model that domain testing was leading us to make. We wrote in the book:

“A notional variable reflects the tester’s “notion” of what is in the code. The tester analyzes the program as if this variable was part of the code…. The notional variable is part of a model, created by the tester, to describe how the program behaves or should behave. The model won’t be perfect. That exact variable is probably not in the code. However, the notional variable is probably functionally equivalent to (works the same way as) a variable that is in the code or functionally equivalent to a group of variables that interact to produce the same behavior as this variable.”

This in turn helped us rethink parts of the Schema in ways that improved its clarity. And it helped us clarify our thinking about domain-related exploratory testing as a way of refining the modeling of the notional variables (bringing them into increasingly accurate alignment with the program implementation or, if the implementation is inadequate, with how the program should behave).

I hope you have a chance to read The Domain Testing Workbook and if you do, that you’ll post your reactions here or as a review on Amazon.

Follow-Up Testing in the BBST Bug Advocacy Course

Sunday, February 10th, 2013

Becky and I are working on a new version of the BBST courses (BBST-2014). In the interim, to support the universities teaching the course, the AST, and the many people who are studying the materials on their own, we’re publishing some of the ideas we have for clarifying the course. The first two in this series focused on oracles in the Foundations course and on interactive grading as a way to give students more detailed feedback. This post is about follow-up testing, which we cover in the current BBST-BA’s slides 44-68.

Some failures are significant and they occur under circumstances that seem fairly commonplace. These will either be fixed, or there will be a compelling reason to not fix them.

Other failures need work. When you see them, there’s ambiguity about the significance of the underlying bug:

  • Some failures look minor. Underlying the failure is a bug that might (or might not) be more serious than you’d imagine from the first failure.
  • Some failures look rare or isolated. Underlying that failure is a bug that might (or might not) cause failures more often, or under a wider range of conditions, than you’d imagine from the first failure.

Investigating whether a minor-looking failure reflects a more serious bug

To find out whether the underlying bugs are more serious than the first failures make them look, we can do follow-up testing. But what tests?

That’s a question that Jack Falk, Hung Quoc Nguyen and I wrestled with many times. You can see a list of suggestions in Testing Computer Software. Slides 44-56 sort those ideas (and add a few) into four categories:

  1. Vary your behavior
  2. Vary the options and settings of the program
  3. Vary data that you load into the program
  4. Vary the software and hardware environment

Some students get confused by the categories (which is not shocking, because the categories are a somewhat idiosyncratic attempt to organize a broad collection of test ideas), confused enough that they don’t do a good job on the exam of generating test ideas from the categories.

For example, they have trouble with this question:

Suppose that you find a reproducible failure that doesn’t look very serious.

  • Describe the four tactics presented in the lecture for testing whether the defect is more serious than it first appeared.
  • As a particular example, suppose that the display got a little corrupted (stray dots on the screen, an unexpected font change, that kind of stuff) in Impress when you drag the scroll bar up and down. Describe four follow-up tests that you would run, one for each of the tactics that you listed above.

I’m not going to solve this puzzle for you, but the solution should be straightforward if you understand the categories.

The slides work through a slightly different example:

A program unexpectedly but slightly scrolls the display when you add two numbers:

  • The task is entering numbers and adding
  • The failure is the scrolling.

Let’s consider the categories in terms of this example

1. Vary your behavior

When you run a test, you intentionally do some things as part of the test. For example, you might:

  • enter some data into input variables
  • write some data into data files tha tthe program will read
  • give the program commands

You might change any of these as part of your follow-up testing. (What happens if I do this instead of that?)

These follow-up tests might include changing the data or adding steps, substituting steps, or taking steps away.

For example, if adding one pair of numbers causes unexpected scrolling, suppose you try adding two numbers many times. Will the program scroll more, or scroll differently, as you repeat the test?

Suppose we modified the example so the program reads (and then adds) two numbers from a data file. Changing that data file would be another example of varying your behavior.

The slides give several additional examples.

2. Vary the options and settings of the program

Think about Microsoft Word. Here are some examples of its options:

  • Show (or don’t show) formatting marks on the screen
  • Check spelling (or don’t check it) as you type

In addition, you might change a template that controls the formatting of new documents. Some examples of variables you might change in the template are:

  • the default typeface
  • the spacing between tab stops
  • the color of the text

Which of these are “options” and which of these are “settings”? It doesn’t matter. The terminology will change from program to program. What doesn’t change is that these are persistent variables. Their value stays with the program from one document to another.

3. Vary data that you load into the program

This isn’t well worded. Students confuse what I’m talking about with the basic test data.

Imagine again testing Microsoft Word. Suppose that you are focused on the format of the document, so your test documents have lots of headings and tables and columns and headers and footers (etc.). If you change the document, save it, and load the revised one, that is data that you load into the program, but I think of that type of change as part of 1. Vary your behavior.

When Word starts up, it also loads some files that might not be intentional parts of your test. For example, it loads a dictionary, a help file, a template, and probably other stuff. Often, we don’t even think of these files (and how what they hold might affect memory or performance) when we design tests. Sometimes, changing one of these files can reveal interesting information.

4. Vary the software and hardware environment

For example,

  • Change the operating system’s settings, such as the language settings
  • Change the hardware (a different video card) or how the operating system works with the hardware (a different driver)
  • Change hardware settings (same video card, different display resolution)

This is a type of configuration testing, but the goal here is to try to demonstrate a more impressive failure, not to assess the range of circumstances under which the bug will appear.

Investigating whether an obscure-looking failure will arise under more circumstances

In this case, the failure shows up under what appear to be special circumstances. Does it only show up under special circumstances?

Slides 58-68 discuss this, but some additional points are made on other slides or in the taped lecture. Bringing them together…

  1. Uncorner your corner cases
  2. Look for configuration dependence
  3. Check whether the bug is new to this version
  4. Check whether failures like this one already appear in the bug database
  5. Check whether bugs of this kind appear in other programs

Here are a few notes:

1. Uncorner your corner cases

Risk-based tests often use extreme values (boundary conditions). These are good for exposing a failure, but once you find the failure, try less extreme values. Demonstration of failure with a less extreme test will yield a more credible report.

2. Look for configuration dependence

In this case, the question is whether the failure will appear on many configurations or just this one. Try it with more memory, with another version of operating system (or on another OS altogether), etc.

3. Check whether the bug is new to this version

Does this bug appear in earlier versions of the program? If so, did users (or others) complain about it?

If the bug is new, especially if it was a side-effect of a fix to another bug, some people will take it much more seriously than a bug that has been around a long time but rarely complained about.

4. Check whether bugs like this one already appear in the bug database

One of the obvious ways to investigate whether a failure appears under more general circumstances than the ones in a specific test is to check the bug tracking database to see whether the failure has in fact occurred under other circumstances. It often takes some creativity and patience to search out reports of related failures (because they probably aren’t reported in exactly the same way as you’re thinking of the current failure), but if your database has lots of not-yet-fixed bugs, such a search is often worthwhile.

5. Check whether bugs of this kind appear in other programs

The failure you’re seeing might be caused by the code specifically written by this programmer, or the bug might be in a library used by the programmer. If it’s in the library, the same bug will be in other programs. Similarly, some programmers model (or copy) their code from textbook descriptions or from code they find on the web. If that code is incorrect, the error will probably show up in other programs.

If the error does appear in other programs, you might find discussions of the error (how to find it, when it appears, how serious it is) in discussions of those programs.

This post is partially based on work supported by NSF research grant CCLI-0717613 ―Adaptation & Implementation of an Activity-Based Online or Hybrid Course in Software Testing. Any opinions, findings and conclusions or recommendations expressed in this post are those of the author and do not necessarily reflect the views of the National Science Foundation.

An Overview of High Volume Automated Testing

Monday, January 14th, 2013

Summary: This overview describes the general concept of high volume automated testing (HiVAT) and twelve examples of HiVAT techniques. The common thread of these techniques is automated generation, execution and evaluation of arbitrarily many tests. The individual tests are often weak, but taken together, they can expose problems that individually-crafted tests will miss. The simplest techniques offer high coverage: they can reach a program’s weakness in handling specific special cases. Other techniques take the program through long sequences of tests and can expose programs that build gradually (e.g. memory leaks, stack corruption or memory corruption) or that involve unexpected timing of events. In my experience, these sequences have exposed serious problems in embedded software, especially in systems that involved interaction among a few processors. As we embed more software into more consumer systems, including systems that pose life-critical risks, I believe these tests will become increasingly important.

I am writing this note as an intentionally rough draft. It servers as an introduction for a course on HiVAT at Florida Institute of Technology. It provides a structure for work in the course. It is intentionally under-referenced. One of the students’ tasks in the course is to dig through a highly disjointed literature and map research papers, practitioner papers, and conference presentations to the techniques listed here. Students might also add HiVAT techniques, with associated papers, or add sections on directly-relevant automation-support technology or directly-relevant surveys of test automation strategies / results. I will replace this post with later drafts that are more academically complete as we make progress in the course.

Consider automated regression testing. We reuse a regression test several times–perhaps running it on every build. But are these really automated? The computer might execute the tests and do a simple evaluation of the results, but a human probably designed that test, a human probably wrote the test code that the computer executes, a human probably provided the test input data by coding it directly into the test or by specifying parameters in an input file, a human probably provided the expected results that the program uses to evaluate the test result, and if there appears that there might be a problem, it will be a human who inspects the results, does the troubleshooting and either writes a bug report (if the program is broken) or rewrites the test. All that work by humans is manual.

Notice, by the way, that this interplay of human work and computer work applies whether the tests are run at the system level using a tool like QuickTest Pro, at the unit level using a tool like jUnit, or at a hybrid level, using a tool like FIT.

So, automated regression test are actually manual tests with an automated tint.

And every manual software test is actually automated. When you run a manual test, you might type in the inputs and look at the outputs, but everything that happens from the acceptance of those inputs to the display of the results of processing them is done by a computer under the control of a program–that’s automated.

The difference between “manual” and “automated” is a matter of degree, not a matter of principle. Some tests are more automated, others are less automated, but there is no “all” and there is no “none” for test automation.

High-Volume Tests Focused on Inputs

High-Volume Parametric Variation

We can make the automated regression tests more automated by transforming some of the human tasks to computer tasks. For example, imagine testing a method that takes two inputs (FIRST, SECOND) and returns their sum (SUM). A typical regression test would include specific values for FIRST, SECOND, and EXPECTED_SUM. But suppose we replace the specific test values with a test data generator that supplies random values for FIRST and SECOND and calculates the value of EXPECTED_SUM. We can generate billions of tests this way, each a little different, with almost no increase in human effort.

This is one of the simplest examples of high volume automated testing. A human supplied the algorithm, but the test tool applies that algorithm to create, run, and evaluate the results of arbitrarily many tests.

In this example, those tests are all pretty similar. Why bother running them all? The traditional answer is “don’t bother.” Domain testing is most widely used software testing technique. The point of domain testing is to help the tester minimize test redundancy by selecting a small number of inputs to represent the larger set of possibilities. Usually, this works well. Occasionally, a program has a problem with a small number of specific inputs. For example, the program might be optimized in a way that processes a few specific values specially. Or it might be vulnerable to small calculation errors that are usually too small to notice, but occasionally have a visible effect (see Doug Hoffman’s report at http://www.testingeducation.org/BBST/foundations/Hoffman_Exhaust_Options.pdf for an example.

High-Volume Combination Testing

Rather than thinking of testing a single function that takes a few inputs, imagine something more complex. A program processes several inputs, perhaps with a series of functions, and reports an output. Again, the traditional approach is to minimize redundancy. When we do combination tests (test several variables together), the set of possible tests is usually huge. If the variables are independent, we can minimize the number of combination tests with combinatorial testing (e.g. all-pairs or all-triples). If the variables are related, we can use cause-effect graphing instead. But if we suspect that the program will fail only on a small number of specific combinations (and we don’t know which few those are), we have to test a larger set of combinations, generating input values and calculating the expected result for each combination.

There are different strategies for generating the input values:

  • Exhaustive sampling. This tests all the combinations but the set might be impossibly large.
  • Random sampling. Generate random values for the inputs, stopping when some large number have been tested.
  • Optimized sampling. Use an algorithm that optimizes the set of combinations in some way. As a very simple example, if you are going to test a million combinations, you could divide the space of combinations into a million same-size, non-overlapping subsets and sample one value of each. Or you could use a sequential algorithm that assigns values for the next combination by creating a test that is distant (according to some distance function) from all previous tests.

Input Fuzzing

So far, I’ve presented tests that generate both, the inputs and the expected value of the test. The expected value serves as an oracle, a mechanism for deciding whether the program passed or failed the test. Test oracles are incomplete, but they are useful for automation. Even if we can’t detect all possible failures, we can detect any failures of a certain kind (such as a calculation error).

“Fuzzing” refers to a family of high-volume automated tests that vary the inputs but have no oracle. The test runs until the program crashes or fails in some other unmissable way.

Hostile Data Stream Testing

Alan Jorgensen tested for many types of security errors by taking a good file in a standard format (e.g. PDF) and corrupting by substituting one string in the file with another. In Jorgensen’s work, the new string was typically much longer or was syntactically different. Jorgensen would then open the now-corrupt file with the application under test. The program might reject the file as corrupt or accept it. If it accepted the file, Jorgensen could ask the program to use the file in some way (e.g. display it) and look for misbehavior. In some cases, Jorgensen could exploit a failure to recognize a corrupt file by embedding executable code that the program would then inappropriately execute. Jorgensen’s search for corruptions that would escape detection was an intensely automated activity. His analysis of exploitability was not.

There are several published variations of file fuzzing (variation of the contents or format of an input file) that are suited for different kinds of risks.

High-Volume Tests that Exploit the Availability of an Oracle

Having an oracle gives you a strong basis for high-volume automated testing. All that’s required is to generate inputs to the program under test and to drive the program to process the inputs. Given a result, you can use the oracle to check whether the program passed the test. No oracle is perfect: the program might fail the test even though its output matches the oracle’s. But even though you can’t learn everything that might possibly be interesting by relying on an oracle, you have a basis for running a boatload of tests that collectively check thoroughly for some types of failures and give you an opportunity to stumble onto some other types of failures (e.g. crashes from memory leaks) that you didn’t design the tests to look for. For any other ways that you can imagine the program might fail, you can design tailored manual tests as needed.

High-volume testing with an oracle won’t completely test the program (against all possible risks), but it will provide you with a level of coverage that you can’t achieve by hand.

Function Equivalence Testing

Function equivalence testing starts with an oracle–a reference function that should work the same way as the function under test. Given the reference function, you can feed inputs to the function under test, get the results, and check whether the reference function gives the same results. You can test with as many input values as you want, perhaps generating a large random set of inputs or (if you have enough available computer time) testing every possible input.

The final exam in my Programmer-Testing course illustrates function equivalence testing. The question that students have to answer is whether Open Office Calc does its calculations the same way as Microsoft Excel. We adopt a quality criterion: If Calc does it the same way as Excel, that’s good enough, even if Excel makes a few calculation errors.

To answer the question, the students:

  • test several functions individually
  • then test those functions together by testing formulas that combine several functions

To do this,

  • The students pick several individual functions in Calc
  • They test each by feeding random inputs to the Calc function and the same inputs to the corresponding Excel function.
  • Then they create random formulas that combine the functions, feed random data to the functions in the formula, and compare results.

If you test enough inputs, and the Calc results are always the same as Excel’s (allowing a little rounding error), it is reasonable to conclude that the calculation in Calc is equivalent to the calculation in Excel.

Constraint Checks

We use a constraint oracle to check for impossible values or impossible relationships.

For example an American ZIP code must be 5 or 9 digits. If you are working with a program that reads (or looks up or otherwise processes) ZIP codes, you can check every code that it processes. If it accepts (treats as a ZIP code) anything that has non-numeric characters or the wrong number of characters, then it has a bug. If you can find a way to drive the program so that it reads (or does whatever it does with) lots of ZIP codes, you have a basis for a set of high-volume automated tests.

Inverse Operations

Imagine taking a list that is sorted from high to low, sorting it low to high, then sorting it back (high to low). If you can give this program enough lists (enough sizes, enough diversity of values), you can eventually conclude that it sorts correctly (or not). Any operation that you can invert, you can build a high volume test series against.

State-Model Based Testing (SMBT)

If you have a state model (including a way to decide where the program should go if you give it an input and a way to determine whether the program actually got there), you can feed the program an arbitrarily long sequence of inputs and check the results.

I think the most common way to do SMBT is with a deterministic series of tests, typically the shortest series that will achieve a specified level of coverage. The typical coverage goal is every transition from each possible state to every state that it can reach next. You can turn this into a high-volume series by selecting states and inputs randomly and running the sequence for an arbitrarily long time. Ben Simo cautions that this has to be monitored because some programs will get locked into relatively tight loops, never reaching some states or some transitions. If you write your test execution system to check for this, though, you can force it out of the loop and into a not-yet hit state.

Diagnostics-Based Testing

I worked at a company that designed telephone systems (Telenova). Our phones gave customers a menu-driven interface to 108 voice features and 110 data features. Imagine running a system test with 100 phones calling each other, putting each other on hold, transferring calls from one phone to another, then conferencing in outside lines, etc. We were never able to create a full state model for our system tests. They were just too complex.

Instead, the Telenova staff (programmers, testers, and hardware engineers) designed a simulator that could drive the phones from state to state with specified or random inputs. They wrote probes into the code to check whether the system was behaving in unexpected ways. A probe is like an assert command, but if the program triggers the probe, it logs an error instead of halting the program. A probe might check whether a variable had an unexpected value, whether a set of variables had unexpected values relative to each other, whether the program went through a series of states in an unexpected order, etc.

Because these probes checked the internal state of the system, and might check any aspect of the system, we called them diagnostics.

Implementing this type of testing required a lot of collaboration. The programmers wrote probes into their code. Testers did the first evaluations of the test logs and did extensive troubleshooting, looking for simple replication conditions for events that showed up in the logs. Testers and programmers worked together to fix bugs, change the probes, and specify the next test series.

As challenging as the implementation was, this testing revealed a remarkable number of interesting problems, including problems that would have been very hard to find in traditional ways but had the potential to cause serious failures in the field. This is what convinced me of the value of high-volume automated testing.

High-Volume Tests that Exploit the Availability of Existing Tests or Tools

Sometimes, the best reason to adopt a high-volume automated technique is that the adoption will be relatively easy. That is, large parts of the job are already done or expensive tools that can be used to do the job are already in place.

Long-Sequence Regression Testing (LSRT)

For example, Pat McGee and I wrote about a well-known company that repurposed its huge collection of regression tests of its office-automation products’ firmware. (In deference to the company’s desire not to be named, we called it Mentsville.)

When you do LSRT, you start by running the regression tests against the current build. From that set, pick only tests that the program passes. Run these in random order until the program fails (or you’ve run them for a long enough time). The original regression tests were designed to reveal functional problems, but we got past those by using only tests that we knew the program could pass when you ran them one-at-a-time. The bugs we found came from running the tests in a long series. For example, the program might run a series of 1000 tests, thirty of them the same as the first (Test 1), but it might not fail Test 1 until that 30th run. Why did it fail on time 30 and not before?

  • Sometimes, the problem was a gradual build-up of bad data in the stack or memory.
  • Sometimes, the problem involved timing. For example, sometimes one processor would stay busy (from the last test) for an unexpectedly long time and wouldn’t be ready when it was expected for this test. Or sometimes the firmware would expect a location in memory to have been updated, but in this unusual sequence, the process or processor wouldn’t yet have completed the relevant calculation.
  • Sometimes the problem was insufficient memory (memory leak) and some of the leaks were subtle, requiring a specific sequence of events rather than a simple call to a single function.

These were similar to the kinds of problems we found at Telenova (running sequences of diagnostics-supported tests overnight).

Troubleshooting the failures was a challenge because it was hard to tell when the underlying failure actually occurred. Something could happen early in testing that wouldn’t cause an immediate failure but would cause gradual corruption of memory until hours later, the system crashed. That early result was what was needed for replicating the failure. To make troubleshooting easier, we started running diagnostics between tests, checking the state of memory, or how long a task had taken to execute, or whether a processor was still busy, or any of over 1000 other available system checks. We only ran a few diagnostics between tests. (Each diagnostic changed the state of the system. We felt that running too many diagnostics would change the state too much for us to get an accurate picture of the the effect of running several regression tests in a row.) But those few tests could tell us a great deal. And if we needed more information, we could run the same 1000-test sequence again, but with different diagnostics between tests.

High-Volume Protocol Testing

A protocol specifies the rules for communication between two programs or two systems. For example, an ecommerce site might interact with VISA to bill a customer’s VISA card for a purchase. The protocol specifies what commands (messages) the site can send to VISA, what their structure should be, what types of data should appear in the messages and where, and what responses are possible (and what they mean). A protocol test sends a command to the remote system and evaluates the results (or sends a related series of commands and evaluates the series of responses back).

With popular systems, like VISA, lots of programs want to test whether they work with the system (whether they’ve implemented the protocol correctly and how the remote system actually responds). A tool to run these types of tests might be already available, or pieces of it might be available. If enough is available, it might be easy to extend it, to generate long random sequences of commands to the remote system and to process the system’s responses to each one.

Load-Enhanced Functional Testing

Common lore back in the 1970′s was that a system behaved differently, functionally differently, when it was running under load. Tasks that it could perform correctly under “normal” load were done incorrectly when the system got busy.

Alberto Savoia described his experience of this in a presentation at the STAR conference in 2000. Segue (a test automation tool developer) built a service around it. As I understand their results, a system could appear 50%-busy (busy, but not saturated) but still be unable to correctly run some regression tests. The value of this type of testing is that it can expose functional weaknesses in the system. For example, suppose that a system running low on memory will rely more heavily on virtual memory and so the timing of its actions slows down. Suppose that a program running on the system spreads tasks across processors in a way that makes it vulnerable to race conditions. Suppose that Task 1 on Processor 1 will always finish before Task 2 on Processor 2 when the system is running under normal load, but under some heavy loads, Processor 1 will get tied up and won’t process work as quickly as Processor 2. If the program assumes that Task 1 will always get done before Task 2, that assumption will fail under load (and so might the program).

In the field, bugs like this produce hard-to-reproduce failures. Unless you find ways to play with the system’s timing, you might never replicate problems like this in the lab. They just become mystery-failures that a few customers call to complain about.

If you are already load-testing a program and if you already have a regression suite, then it might be very easy to turn this into a long-sequence regression test running with a moderate-load load test in parallel. The oracles include whatever comes with each regression test (to tell you whether the program passed that test or not) plus crashes, obvious long delays, or warnings from diagnostics (if you add those in).

A Few More Notes on History

I think that most of the high-volume work has been done in industry and not published in the traditional academic literature. For example, the best-known (best-publicized) family of high-volume techniques are “fuzzing”, attributed to Professor Barton Miller at the University of Wisconsin in 1988.

  • But I was seeing this industrial demonstrations of this approach when I first moved to Silicon Valley in 1983, and I’ve been told of applications as early as 1966 (the “Evil” program at Hewlett-Packard).
  • Long-sequence regression testing was initially developed in 1984 or 1985. I played a minor role in developing Telenova’s diagnostics-based approach in 1987; my impression was that we were applying ideas already implemented years ago at other telephone companies.
  • The testing staff at WordStar (a word-processing company) used a commercially available tool to feed long sequences of tests from one computer to another in 1984 or 1985. The first computer generated the tests and analyzed the test results. The second machine ran the program under test (WordStar). They were hooked up so that commands from the first machine looked like keyboard inputs to the second, and outputs that the second machine intended for its display actually went to the second machine for analysis. As an example of the types of bugs they found with this setup, they were able to replicate a seemingly-irreproducible crash that turned out to result from a memory leak. If you boldfaced a selection of text and then italicized it, there was a leak. If you italicized first, then applied bold, no leak. The test that exposed this involved a long random sequence of commands. I think this was a normal way to use that type of tool (long sequences with some level of randomization).
  • We also used random input generators in the telephone world. I think of a tool called The Hammer but there were earlier tools of this class. Hammer Technologies was formed in 1991 and I was hearing about these types of tools back in the mid-to-late-1980′s while I was at Telenova.

I’ve heard of related work at Texas Instruments, Microsoft, AT&T, Rolm, and other telephone companies, in the testing of FAA-regulated systems, at some auto makers, and at some other companies. I’m very confident that the work actually done is much more broadly spread than this small group of companies and includes many other techniques than I’ve listed here. However, most of what I’ve seen has been by semi-private demonstrations or descriptions or via demonstrations and descriptions at practitioner conferences. Most of the interesting applications that I have personally heard of have involved firmware or other software that controls hardware systems.

As far as I can tell, there is no common vocabulary for these techniques. Much of what has been published has been lost because many practitioner conference proceedings are not widely available and because old papers describing techniques under nonstandard names are just not being found in searches. I hope that we’ll fill in some of these gaps over the next several months. But even for the techniques that we can’t find in publications, it’s important to recognize how advanced the state of the practice was 25 years ago. I think we are looking for a way to make sometimes-closely-held industrial practices more widely known and more widely adopted, rather than inventing a new area.

This post is partially based on work supported by NSF research grant CCLI-0717613 ―Adaptation & Implementation of an Activity-Based Online or Hybrid Course in Software Testing. Any opinions, findings and conclusions or recommendations expressed in this post are those of the author and do not necessarily reflect the views of the National Science Foundation.

 

Interactive Grading in University and Practitioner Classes: An Experience Report

Monday, January 14th, 2013

Summary: Graders typically work in private with no opportunity to ask even the simplest questions about a student’s submitted work. Interactive Grading is a technique that requires the student to participate in the grading of their work. The aim of this post is to share my experiences with interactive grading, with some tips for others who want to try it. I start with an overview and then provide three detailed examples, with suggestions for the conduct of the session. Let me stress one point: care must be exercised to keep the student comfortable and engaged, and not let the session degenerate into another lecture. In terms of results, most students told me that interactive grading was a good use of their time and that it helped them improve their performance. Several asked me to add more of it to my courses. A few viewed at as more indoctrination. In terms of impact on student grades, they showed marginal improvement.

Interactive grading is not a new idea. I think most instructors have done this for some students at some times. Certainly, I have. So when professor Keith Gallagher described it to me as an important part of his teaching, I didn’t initially understand its significance. I decided to try it myself after several of Keith’s students talked favorably about their classes with him. It wasn’t until I tried it that I realized what I had been missing.

That start came 15 months ago. Since then, I’ve used interactive grading in the online (professional-development) BBST classes, hybrid (online + face-to-face) university software testing classes, and a face-to-face university software metrics course. I’ve used it for exams, essays, and assignments (take-home complex tasks). Overall, it’s been a positive change.

These notes describe my personal experiences and reflections as an instructor. I emphasize this by writing in the very-obvious first person. Your experiences might be different.

What is Interactive Grading?

When I teach, I assign tasks and students submit their work to me for grading.

Usually I review student work in private and give them feedback after I have completed my review.

  • When I do interactive grading:
    • I meet with the student before I review the work.
    • I read the work for the first time while I meet with the student.
    • I ask the student questions, often open-ended questions that help me understand what the student was trying to say or achieve. Students often demonstrate that they understood the material better than their submitted work suggests. If they misunderstood part of the task, we can get to the bottom of the misunderstanding and they can try to demonstrate, during this meeting, their ability to do the task that was actually assigned.
    • I often coach the student, offering suggestions to improve the student’s strategy or demonstrating how to do parts of the task.
    • I typically show the student a grading rubric early in the meeting and assign the grade at the end of the meeting.
  • When I explicitly build interactive grading into my class:
    • It becomes part of the normal process, rather than an exception for a student who needs special attention. This changes the nature and tone of the discussions.
    • Every student knows well in advance what work will be interactively graded and that every student’s work will be handled the same way. This changes how they prepare for the meetings and how they interpret the meetings.
    • I can plan later parts of the course around the fact that the students have already had this experience. This changes my course design.

Costs and Benefits of Interactive Grading

Here is a summary of my conclusions. I’ll support this summary later in this report, with more detailed descriptions of what we did and what happened.

Costs

Interactive grading feels like it takes more time:

  • It takes time to prepare a grading structure that the students can understand (and therefore that I can use effectively when we have the meeting)
  • Scheduling can take a lot of time.
  • The meetings sometimes run long. It feels as though it takes longer to have the meeting than it would take to grade the work normally.

When I’ve checked my grading times for exams and assignments that I do in the traditional way, I think I actually spend the same amount of time (or more). I also do the same level of preparation. (Note: I do a lot of pre-grading preparation. The contrast might be greater for a less formal grader.)

As far as I can tell, the actual difference for me is not time, it is that interactive grading meetings are more stressful for me than grading in a quiet, comfortable home office. That makes it feel longer.

Benefits for the Students

During interactive grading, I can ask questions like,

  • What were you thinking?
  • What do you think this word in the question means? If I gave you a different explanation of what this word means, how would that affect your answer to the question?
  • Give me an example of what you are describing?
  • Can you give me a real-life example of what you are describing? For example, suppose we were working with OpenOffice. How would this come up in that project?
  • Can you explain this with a diagram? Show me on my whiteboard.
  • How would you answer this if I changed the question’s wording this way?
  • How would someone actually do that?
  • Why would anyone want to do that task that way? Isn’t there a simpler way to do the same thing?

I will raise the grade for a student who does a good job with these questions. I might say to the student,

“If I was only grading the written answer, you would get a ‘D’. But with your explanation, I am giving you a ‘B’. We need to talk about how you can present what you know better, so that you can get a ‘B’ again on the next exam, when I grade your written answers without an interactive supplement.”

If a student performs poorly on an exam (or an assignment, or an essay), the problem might be weak competence or weak performance.

  • A student who doesn’t know the material has no competence.
  • A student who knows the material but gives a poor answer on an exam despite that is showing poor performance. For example, you won’t get a good answer from a knowledgeable student who writes poorly or in a disorganized way or who misunderstands the question.

These focusing questions:

  • give the student who knows the material a chance to give a much better explanation or a much better defense of their answer.
  • give the student who knows the material a chance but performs poorly some indicators of the types of performance improvements they need to work on.
  • give the student who doesn’t know the material a clear form of feedback on why they are getting a poor grade.

Let’s consider competence problems. Students might lack the knowledge or the skills they are supposed to be learning for several reasons:

  • Some students simply don’t take the time or make the effort to do good work. Interactive grading probably won’t do much for them, beyond helping some of them understand the standards better.
  • Some students memorize words that they don’t really understand. The interactive grading discussion helps (some of) them understand a bit better the differences between memorized words and understanding something well enough to explain it in their own words and to explain how to do it or use it or why it’s important. It gives them a path to a different type of answer when they ask themselves while studying, “Do I know this well enough?”
  • Some students lack basic student-skills (how to study, how to look things up online, how to use the library, etc.) I can demonstrate these activities during the discussion, having the student do them with me. Students don’t become experts overnight with these, but as I’ll discuss below, I think this sometimes leads to noticeable improvements.

Some of the tasks that I assign to students can be done in a professional way. I choose tasks that I can demonstrate at a professional level of skill. In the give-and-take of interactive grading, I can say, “Let me show you how a professional would do that.” There is a risk of hijacking the discussion, turning it into yet-another-lecture. But judiciously used, this is a very personalized type of coaching of complex skills.

Now consider performance problems. The student understands the material but provides an exam answer or submits an assigned paper that doesn’t adequately reflect what they know, at their level of sophistication. A student with an “A” level of knowledge might look like a “C” student. A student with a “C” level of knowledge might look like a “D” or “F” student. These students are often puzzled by their poor grades. The interactive grading format makes it possible for me to show a student example after example after example of things they are doing (incomprehensible sentences, uninterpretable diagrams, confusing structure, confusing formatting, etc.) and how these make it hard on the reader. Here are two examples:

  • Some of my students speak English as a second language. Some speak/write it well; some are trying hard to communicate well but make grammar/spelling errors that don’t interfere with my ability to understand their writing. Some write sentences that I cannot understand. They expect me to guess their meaning and they expect me to bias my guessing in their favor. During interactive grading, I can read a sentence, realize that I can’t understand it, and then ask the student what it means. In some cases, the student had no idea (they were, as far as I can tell, bluffing, hoping I would give them points for nothing). In other cases, the student intended a meaning but they realized during the discussion that they were not conveying that meaning. For some students, this is a surprise. They didn’t realize that they were failing to communicate. Some have said to me that they had previously thought they were being downgraded for minor errors in their writing (spelling, grammar) rather than for writing something that the instructor could not understand. For some students, I think this changes their motivation to improve their writing.
  • Some students write disorganized answers, or answers that are organized in a fundamentally different way from the structure explicitly requested by the question. Some students do this strategically (with the goal of disguising their ignorance). Others are simply communicating poorly. In my experience, retraining these students is very hard, but this gives me another opportunity to highlight the problems in the work, to demonstrate how the problems affect how I analyze and evaluate their work, and how they could do it differently. (For more on this, see the discussion in the next section, Benefits for Me).

Benefits for Me

It’s easy to grade “A” work. The student “gets it” and so they get a high grade. When I recognize quickly that the student has met my objectives for a specific exam question, I stop analyzing it, award a very high grade, and move to the next question. Very fast.

In contrast, a typical “C” or “D” answer takes a lot longer to grade. The answer is typically disorganized, confused, hard to understand, rewords the question in an effort to present the question as its answer, seems to make inappropriate assumptions about what I should find obvious, has some mistakes, and/or has contradictions or inconsistencies that make the answer incoherent even though each individual component could be argued to be not-necessarily-wrong.

When I say, “disorganized”, I mean (for example) that if the question asks for parts (1), (2), (3), (4) and (5), the student will give three sections instead that address (in the first one) (1), (3) and (5), (in the second one) (1) and (2) and (in the third one) (3) and (5) with a couple of words from the question about part (4) but no added information.

I waste a lot of time trying to understand these answers. When I grade privately, I read a bad answer over several times, muttering as I try to parse the sentences and map the content to the question that was asked. It’s uncertain work–I constantly question whether I am actually understanding what the student meant and trying to figure out how much benefit of how much doubt I should give the student.

When I do interactive grading with a student, I don’t have to guess. I can say, “The question asked for Part 1. I don’t see an answer directly to Part 1. Can you explain how your answer maps to this part of the question?” I insist that the student map their actual words on the exam to the question that was asked. If they can’t sort it out for me, they can flunk. Next time, based on this experience, they can write in a way that will make it easier to map the answer to the question (make it easier for me to grade; get them a better grade).

This discussion is difficult, it can be unpleasant for the student and for me, I have to be diplomatic and somewhat encouraging or it will become a mess. But I don’t have to struggle alone with not understanding what was written. I can put the burden back on the student.

Some other benefits:

  • Students who write essays by copying content without understanding it demonstrate their cluelessness when we grade the essay interactively.
  • Students who cheat on a take-home exam (in my experience so far) avoid the interactive grading session, where they would probably demonstrate that they don’t understand their own answers.
  • Students who want to haggle with me about their grade have an opportunity to do so without driving me crazy. I now have a way to say, Show me what makes this a ‘B’ instead of arguing with them about individual points or about their situational need for a higher grade.

Maybe the most important benefits:

  • (Most) students tell me they like it and many of them ask to do it again (to transform a subsequent task into an interactive-graded one or to add interactive grading in another course)
  • Their performance seems to improve, sometimes significantly, which makes the next work easier to grade
  • This is highly personalized instruction. The student is getting one-on-one attention from the professor. Many students feel as though they don’t get enough personal attention and this addresses that feeling. It also makes some students a little more confortable with dropping by my office for advice at other times.

A More Detailed Report

Someone who is just trying to learn what interactive grading should stop here. What follows is “nuts and bolts”.

I’ve done interactive grading for three types of work:

  • midterm exams
  • practical assignments (homework that requires the student to apply something they have learned to a real-life task)
  • research essays

The overall process for interactive grading is the same for all three. But I’ve also noticed some differences. Marking exams isn’t the same as marking essays; neither is grading them interactively.

If you’re going to try interactive grading for yourself, the differences among descriptions of how it worked for these might help you make a faster and surer start.

Structural/Logistical Matters

I tell students what “interactive grading” is at the start of the course and I tell them which pieces of their work will be graded interactively. In most courses (in all of my university courses for credit), I tell them that this is a mandatory activity.

Some students choose not to participate in interactive grading sessions. Until now, I would tolerate this and grade their work in the traditional way. However, no one has ever given me a good reason for avoiding these (and I have run into several bad ones). The larger problem for me is that later in the course, when I want to do something that assumes that every student has had the interactive grading experience, it is inappropriate for students who skipped it. In the future, in academic courses, I will reinforce the “mandatory” nature of the activity by assigning a grade of zero on the work to a student who (after being warned of this) chooses not to schedule a grading meeting.

Scheduling the meetings is a challenge. To simplify it, I suggest a Doodle poll (www.doodle.com). Post the times you can be available and let each student sign up for a time (from your list) that s/he is available.

I schedule the meetings for 1.5 hours. They often end at 1 hour. Some drag on. If the meeting’s going to run beyond 2 hours, I usually force the meeting to a conclusion. If I think there will be enough value for the student, I offer to schedule a follow-up meeting to go over the rest of the work. If not, then either I announce the grade at the end of the session or I tell the student that I will grade the rest of the work in the traditional way and get back to them with the total.

During the meeting, the student sits across a desk from me. I use a computer with 3 monitors. Two of the monitors show the same information. I look at one and turn the other toward the student. Thus, the student and I can see the same things without having to crowd together to look over each other’s shoulder at one screen. The third screen faces me. I use it to look at any information that I don’t want to share with the student. I use 27″ monitors (small enough to fit on my desk, big enough to be readable for most students) with 1980×1020 resolution. This easily fits two readable word-processing windows side-by-side, such as the student’s submitted work in one window and the grading guide in the next window.

Example 1: Midterm Exams

In some of my courses, I give the students a list of questions well before the exam and draw the exam questions from the list. In my Software Testing 1 course, for example (the “Black Box Testing Course”), I give students a 100-question subset of this list: http://www.testingeducation.org/BBST/takingexams/ExamEssayQuestions2010.pdf

I outlined the costs and benefits of this approach at WTST 2003, in: http://kaner.com/pdfs/AssessmentTestingCourse.pdf. For our purposes, the most important benefit is that students have time before the exam to prepare an answer to each question. They can’t consult their prepared answers during the actual exam, but this lets them come to the exam well-prepared, with a clear idea of what the question means, how to organize the answer, and what points they want to make in it.

I typically give 2 or 3 midterms in a course and I am typically willing to drop the worst one. Thus the student can do poorly on the first midterm, use that experience to learn how to improve their study strategy and writing, and then do better on the next one(s). I do the interactive grading with the first midterm.

Students sometimes submit exams on paper (traditional, handwritten supervised exam), sometimes in a word processor file (supervised exam where students type at a university-owned computer in a university class/exam-room), sometimes in a word processor file (unsupervised takehome exam). For our purposes, assume that the student submitted an electronic exam.

Before I start grading any student’s work, I prepare a grading guide that identifies the types of information that I expect to see in the answer and the points that are possible for that particular type of info. If you’ve never seen that type of grading structure, look at my slides and videos (“How we grade exams”) at http://www.testingeducation.org/BBST/takingexams/.

Many of my questions are (intentionally) subject to some interpretation. Different students can answer them differently, even reaching contradictory conclusions or covering different technical information, but earn full points. The grading guide will allow for this, offering points for several different clusters of information instead of showing only One True Answer.

During the meeting, I rely frequently on the following documents, which I drag on and off of the shared display:

  • The student’s exam
  • The grading guide for that exam
  • The set of all of the course slides
  • The transcripts of the (videotaped) lectures (some or all of my lectures are available to the students on video, rather than given live)
  • The assigned papers, if any, that are directly relevant to exam questions.

We work through the exam one question at a time. The inital display is a copy of the exam question and a copy of the student’s answer. I skim it, often running my mouse pointer over the document to show where I am reading. If I stop to focus, I might select a block of text with the mouse, to show what I am working on now.

  • If the answer is well done, I’ll just say “good”, announce the grade (“That’s a 10 out of 10″) Then I skip to the next answer.
  • If the answer is confusing or sometimes if it is incomplete, I might start the discussion by saying to the student, “Tell me about this.” Without having seen my grading guide, the student tells me about the answer. I ask follow-up questions. For example, sometimes the student makes relevant points that aren’t in the answer itself. I tell the student it’s a good point and ask where that idea is in the written answer. I listed many of my other questions near the start of this report.
  • At some point, often early in the discussion, I display my grading guide beside the student’s answer, explain what points I was looking for, and either identify things that are missing or wrong in the student’s answer or ask the student to map their answer to the guide.
    • Typically, this makes it clear that the point is not in the answer, and what the grading cost is for that.
    • Some students try to haggle with the grading.
      • Some simply beg for more points. I ask them to justify the higher grade by showing how their answer provides the information that the grading guide says is required or creditable for this question.
      • Some tell me that I should infer from what they did write that they must have known the other points that they didn’t write about and so I should give them credit. I explain that I can only grade what they say, not what they don’t say but I think they know anyway.
      • Some agree that their answer as written is incomplete or unclear but they show in this meeting’s discussion that they understand the material much better than their answer suggests. I often give them additional credit, but we’ll talk about why it is that they missed writing down that part of the answer, and how they would structure an answer to a question like this in the future so that their performance on the next exam is better.
      • Some students argue with the analysis I present in the guide. Usually this doesn’t work, but sometimes I give strong points for an analysis that is different from mine but justifiable. I might add their analysis to the grading guide as another path to high points for that answer.
  • The student might claim that I am expecting the student to know a specific detail (e.g. definition or fact) that wasn’t taught in the course. This is when I search the slides and lecture transcripts and readings, highlighting the various places that the required information appeared. I try to lead from here to a different discussion–How did you miss this? What is the hole in your study strategy?
  • Sometimes at the end of the discussion, I ask the student to go to the whiteboard and present a good answer to the question. I do this when I think it will help the student tie together the ideas we’ve discussed, and from doing that, see how to structure answers better in the future. A good presentation (many of them are good enough) might take the assigned grade for the question from a low grade to a higher one. I might say to the student, “That’s a good analysis. Your answer on paper was worth 3/10. I’m going to record a 7/10 to reflect how much better a job you can actually do, but next time we won’t have interactive grading so you’ll have to show me this quality in what you write, not in the meeting. If you give an answer this good on the next exam, you’ll get a 9 or 10.

In a 1.5 hour meeting, we can only spend a few minutes on each question. A long discussion for a single question runs 20 minutes. One of my tasks is to move the discussion along.

Students often show the same weakness in question after question. Rather than working through the same thing in detail each time, I’ll simply note it. As a common example, I might say to a student

Here’s another case where you answered only 2 of the 3 parts of the question. I think you need to make a habit of writing an outline of your answer, check that the outline covers every part of the question, and then fill in the outline.

Other students were simply unprepared for the exam and most of their answers are simply light on knowledge. Once it’s clear that lack of preparation was the problem (often it becomes clear because the student tells me that was the problem), I speed up the meeting, looking for things to ask or say that might add value (for example complimenting a good structure, even though it is short on details). There is no value in dragging out the meeting. The student knows the work was bad. The grade will be bad. Any time that doesn’t add value will feel like scolding or punishment, rather than instruction.

The goal of the meeting is constructive. I am trying to teach the student how to write the next exam better. We might talk about how the student studied, how the student used peer review of draft answers, how the student outlined or wrote the answer, how the student resolved ambiguities in the question or in the course material — and how the student might do this differently next time.

Especially if the student achieved a weak grade, I remind the student that this is the first of three midterms and that the course grade is based on the best two. So far, the student has lost nothing. If they can do the next two well, they can get a stellar grade. For many students, this is a pleasant and reassuring way to conclude the meeting.

The statistical results of this are unimpressive. For example, in a recent course, 11 students completed the course.

  • On midterm 1 (interactively graded) their average grade was 76.7.
  • On midterm 2, the average grade was 81.5
  • On midterm 3, the average grade was 76.3
  • On the final exam, the average grade was 77.8

Remember that I gave students added credit for their oral presentation during interactive grading, which probably added about 10 points to the average grade for midterm 1. Also, I think I grade a little more strictly toward the end of the course and so a B (8/10) answer for midterm 2 might be a B- (7.5) for midterm 3 or the final. Therefore, even though these numbers are flat, my subjective impression of the underlying performance was that it was improving, but this was not a powerful trend.

At the end of the meetings, I asked students whether they felt this was a good use of their time and whether they felt it had helped them. They all told me that it did. In another class (metrics), some students who had gone through interactive grading in the testing course asked for interactive grading of the first metrics midterm. This seemed to be another indicator that they thought the meetings had been helpful and sufficiently pleasant experiences.

This is not a silver bullet, but the students and I feel that it was helpful.

Example 2: Practical Assignments

In my software testing class, I assign tasks that people would do in actual practice. Here are two examples that we have taken to interactive grading:

  1. The student joins the OpenOffice (OOo) project (https://blogs.apache.org/OOo/entry/you_can_help_us_improve) and reviews unconfirmed bug reports. An unconfirmed bug has been submitted but not yet replicated. The student tries to replicate it and adds notes to the report, perhaps providing a simpler set of steps to get to the failure or information that the failure shows up only on a specific configuration or with specific data. The student also writes a separate report to our class, evaluating the communication quality and the technical quality of the original report. An example of this assignment is here: http://www.testingeducation.org/BBST/bugadvocacy/AssignmentBugEvaluationv11.3.pdf
  2. The student picks a single variable in OpenOffice Writer (such as the number of rows in a table) and does a domain analysis of it. In the process, the student imagines several (about 20) ways the program could fail as a consequence of an attempt to assign a value to the variable or an attempt to use the variable once that value has been assigned. For each risk (way the program could fail), the student divides the values of the variable into equivalence classes (all the values within the same class should cause the test with that variable and that value to behave the same way) and then decides which one test should be used from each class (typically a boundary value). An example of this assignment is here: http://www.testingeducation.org/BBST/testdesign/AssignmentRiskDomainTestingFall2011.pdf

The Bug Assignment

When I meet with the student about the bug assignment, we review the bug report (and the student’s additions to it and evaluation of it). I start by asking the student to tell me about the bug report. In the discussion, I have several follow-up questions, such as

  • how they tried to replicate the report
  • why they stopped testing when they did
  • why they tested on the configurations they did
  • what happened when they looked for similar (often equivalent) bugs in the OOo bug database and whether they learned anything from those other reports,

In many cases, especially if the student’s description is a little confusing, I will bring up OpenOffice and try to replicate the bug myself. Everything I do is on the shared screen.They see what I do while I give a running commentary.

  • Sometimes I ask them to walk me through the bug. They tell me what to do. I type what they say. Eventually, they realize that the instructions they wrote into the bug report aren’t as clear or as accurate as they thought.
  • Sometimes I try variations on the steps they tried, especially if they failed to replicate the bug.

My commentary might include anecdotes about things that have happened (that I saw or did) at real companies or comments/demonstration of two ways to do essentially the same thing, with the second being simpler or more effective.

When I ask students about similar bugs in the database, most don’t know how to do a good search, so I show them. Then we look at the reports we found and decide whether any are actually relevant.

I also ask the student how the program should work and why they think so. What did they do to learn more about how the program should work? I might search for specifications or try other programs and see what they do.

I will also comment on the clarity and tone of the comments the student added to the OOo bug report, asking the student why they said something a certain way or why they included some details and left out others.

Overall, I am providing different types of feedback:

  • Did the student actually do the work that was assigned? Often they don’t do the whole thing. Sometimes they miss critical tasks. Sometimes they misunderstand the task.
  • Did the student do the work well? Bug reporting (and therefore this assignment) involves a mixture of persuasive technical writing and technical troubleshooting.
    • How well did they communicate? How much did they improve the communication of the original report? How well did they evaluate the communication quality of the original report?
    • How well did they troubleshoot? What types of information did they look for? What other types could they have looked for that would probably have been helpful? What parameters did they manipulate and how wise were their choices of values for those parameters?

Thinking again about the distinction between competence and performance,

  • Some students show performance problems (for example, they do the task poorly because they habitually follow instructions sloppily). For these students, the main feedback is about their performance problems.
  • Some students show competence problems–they are good at following instructions and they can write English sentences, but they need to learn how to do a better job of bug reporting. For these students, the main feedback is what they are already doing well and what professional-quality work of this type looks like.

This is a 4-phase assignment. I try to schedule the interactive grading sessions soon after Phase 1, because Phase 3 is a more sophisticated repetition of Phase 1 (similar task on a different bug report). The ideal case is Phase 1 — Feedback — Phase 3. Students who were able to schedule the sessions this way have told me that the feedback helped them do a much better job on Phase 3.

The Domain Testing Analysis

Every test technique is like a lens that you look through to see the program. It brings some aspects of the program into clear focus, and you test those in a certain way. It pretty much makes the other aspect of the program invisible.

In domain testing, you see a world of variables. Each variable stands out as a distinct individual. Each variable has sets of possible values:

  • The set of values that users might try to assign to the variable (the values you might enter into a dialog box, for example). Some of these values are invalid — the variable is not supposed to take on these values and the program should reject them.
  • The set of values that the variable might actually take on — other parts of the program will use this variable and so it is interesting to see whether this variable can take on any (“valid”) values that these parts of the program can’t actually handle.
  • The set of values that might be output, when this variable is displayed, printed, saved to disk, etc.

These sets overlap, but it is useful to recognize that they often don’t map perfectly onto each other. A test of a specific “invalid” value might be sometimes useful, sometimes uninteresting and sometimes impossible.

For most variables, each of these sets is large and so you would not want to test every value in the set. Instead, domain testing has you group values as equivalent (they will lead to the same test result) and then sample only one or two values from every set of equivalents. This is also called equivalence-class analyis and boundary testing.

One of the most challenging aspects of this task is adopting the narrow focus of the technique. People aren’t used to doing this. Even working professionals — even very skilled and experienced working professionals — may not be used to doing this and might find it very hard to do. (Some did find it very hard, in the BBST:Test Design course that I taught through the Association for Software Testing and in some private corporate classes.)

Imagine an assignment that asks you to generate 15 good tests that are all derived from the same technique. Suppose you imagine a very powerful test (or a test that is interesting for some other reason) that tests that variable of that program (the variable you are focusing on). Is it a good test? Yes. Is it a domain test? Maybe not. If not, then the assignment is telling you to ignore some perfectly good tests that involve the designated variable, maybe in order to generate other tests that look more boring or less powerful. Some people find this confusing. But this is why there are so many test techniques (BBST: Test Design catalogs over 100). Each technique is better for some things and worse for others. If you are trying to learn a specific technique (you can practice a different one tomorrow), then tests that are not generated by that technique are irrelevant to your learning, no matter how good they are in the general scheme of things.

We run into a conflict of intuitions in the practitioner community here. The view that I hold, that you see in BBST, is that the path to high-skill testing is through learning many different techniques at a high level of skill. To generate a diverse collection of good tests, use a diverse set of techniques. To generate a few tests that are optimized for a specific goal, use a technique that is appropriate for that goal. Different goals, different techniques. But to do this, you have to apply a mental discipline while learning. You have to narrow your focus and ask, “If I was a diehard domain tester who had no use for any other technique, how would I analyze this situation and generate my next tests?” Some people have told me this feels narrow-minded, that it is more important to train testers to create good tests and to get in touch with their inner creativity, than to cramp their style with a narrow vision of testing.

As I see it, this type of work doesn’t make you narrow-minded. It doesn’t stop you from using other techniques when you actually do testing at work. This is not your testing at work. It is your practice, to get good enough to be really good at work. Think of practicing baseball. When you are in batting practice, trying to improve your hitting, you don’t do it by playing catch (throwing and catching the ball), not even if it is a very challenging round of catch. That might be good practice, but it is not good batting practice.

The domain testing technique helps you select a few optimal test values from a much larger set of possibilities. That’s what it’s good for. We use it to pick the most useful specific values to test for a given variable. Here’s a heuristic: If you design a test that would probably yield the same result (pass/fail) now matter what value is in the variable, you are probably focused on testing a feature rather than a variable, and you are almost certainly not designing using domain testing.

Keeping in mind the distinction between competence and performance,

  • Some students show performance problems (for example, they do the task poorly because they habitually follow instructions sloppily). For these students, the main feedback is about their performance problems.
  • The students who show competence problems are generally having trouble with the idea of looking at the world through the lens of a single technique. I don’t have a formula for dealing with these students. It takes individualized questioning and example-creating, that doesn’t always work. Here are some of the types of questions:
    • Does this test depend on the value of the variable? Does the specific value matter? If not, then why do we care which values of the variable we test with? Why do a domain analysis for this?
    • Does this test mainly depend on the value of this variable or the value of some other variable?
    • What parts of the program would care if this variable had this value instead of that value?
    • What makes this specific value of the variable better for testing than the others?

From my perspective as the teacher, these are the hardest interactive grading discussions.

  • The instructions are very detailed but they are a magnet for performance problems that dominate the discussions with the weaker student.
  • The technique is not terribly hard, if you are willing to let yourself apply it in a straightforward way. The problem is that people are not used to explicitly using a cognitive lens, which is essentially what a test technique is.

Student feedback on this has been polite but mixed. Many students are enthusiastic about interactive grading and (say they) feel that they finally understood what I was talking about after the discussion that applied it to their assignment. Other students came away feeling that I had an agenda, or that I was inflexible.

Example 3: Research Essays

In my Software Metrics course, I require students to write two essays. In each essay, the student is required to select one (1) software metric and to review it. I give students an outline for the essay that has 28 sections and subsections, requiring them to analyze the validity and utility of the metric from several angles. They are to look in the research literature to find information for each section, and if they cannot find it, to report how they searched for that information (what search terms in which electronic databases) and summarize the results. Then they are to extrapolate from the other information they have learned to speculate what the right information probably is.

These are 4th year undergraduates or graduate students. A large percentage of these students lack basic library skills. They do not know how to do focused searches in electronic databases for scholarly information and they do not know how to assess its credibility or deal with conflicting results and conflicting conclusions. Many of them lack basic skills in citing references. Few are skilled at structuring a paper longer than 2 or 3 pages. I am describing good students at a well-respected American university that is plenty hard to get into. We have forced them to take compulsory courses in writing, but those courses only went so far and the students only paid so much attention. My understanding, having talked at length with faculty at other schools, is that this is typical of American computer science students.

My primary goal is to help students learn how to cut through the crap written about most metrics (wild claims pro and con, combined with an almost-shocking lack of basic information) so that they can do an analysis as needed on the job.

Metrics are important. They are necessary for management of projects and groups. Working with metrics will be demanded of most people who want to rise beyond junior-level manager or mid-level programmer and of plenty of people whose careers will dead-end below that level. But badly-used metrics can do more harm than good. Given that most (or all) of the software metrics are poorly researched, have serious problems of validity (or at least have little or no supporting evidence of validity), and have serious risk of causing side-effects (measurement dysfunction) to the organization that uses them, there is no simple answer to what basket of metrics should be used for a given context. On the other hand, we can look at metrics as imperfect tools. People can be pretty good at finding information if they understand what they are looking for and they understand the strengths and weaknesses of their tools. And they can be pretty good at limiting the risks of risky things. So rather than encouraging my students to adopt a simplistic, self-destructive (or worse, pseudo-moralistic) attitudeof rejection of metrics, I push them to learn their tools, their limits, their risks, and some ways to mitigate risk.

My secondary goal is to deal with the serious performance problems that these students have with writing essays.

I assign two essays so that they can do one, get detailed feedback via interactive grading, and then do another. I have done this in one (1) course.

Before the first essay was due, we had a special class with a reference librarian who gave a presentation on finding software-metrics research literature in the online databases of Florida Tech’s library and we had several class discussions on the essay requirements.

The results were statistically unimpressive: the average grade went from 74.0 to 74.4. Underneath the numbers, though, is the reality that I enforced a much higher standard on the second essay. Most students did better work on Essay 2, often substantially.

The process for interactive grading was a little different. I kept the papers for a week before starting interactive grading, so that I could check them for plagiarism. I use a variety of techniques for this (if you are curious, see the video course on plagiarism-detection at http://www.testingeducation.org/BBST/engethics/). I do not do interactive grading sessions with plagiarists. The meeting with them is a disciplinary meeting and the grade is zero.

Some students skirted the plagiarism line, some intentionally and others not. In the interactive grading session, I made a point of raising this issue, giving these students feedback on what I saw, how it could be interpreted and how/why to avoid it in the future.

For me, doing the plagiarism check first is a necessary prerequisite to grading an essay. That way, I have put the question behind me about whether this is actually the student’s work. From here, I can focus on what the student did well and how to improve it.

A plagiarism check is not content-focused. I don’t actually read the essay (or at least, I don’t note or evaluate its content, I ignore its structure, I don’t care about its style apart from looking for markers of copying, and I don’t pay attention to who wrote it). I just hunt for a specific class of possible problems. Thus, when I meet with the student for interactive grading, I am still reading the paper for the first time.

During the interactive grading session, I ask the student to tell me about the metric.

One of the requirements that I impose on the students (most don’t do it) is that they apply the metric to real code that they know, so that they can see how it works. As computer science students, they have code, so if they understand the metric, they can do this. I ask them what they did and what they found. In the rare case that the student has actually done this, I ask about whether they did any experimenting, changing the code a little to see its effect on the metric. With some students, this leads to a good discussion about investigating your tools. With the students who didn’t do it, I remind them that I will be unforgiving about this in Essay 2. (Astonishingly, most students still didn’t do this in Essay 2. This cost each of them a one-letter-grade (10-point) penalty.)

From there, I typically read the paper section by section, highlighting the section that I am reading so that the student can follow my progress by looking at the display. Within about 1/2 page of reading, I make comments (like, “this is really well-researched”) or ask questions (like, “this source must have been hard to find, what led you to it?”)

The students learned some library skills in the meeting with the librarian. The undergrads have also taken a 1-credit library skills course. But most of them have little experience. So at various points, it is clear that the student has not been able to find any (or much) relevant information. This is sometimes an opportunity for me to tell the student how I would search for this type of info, and them demonstrate that search. Sometimes it works, sometimes I find nothing useful either.

In many cases, the student finds only the cheerleading of a few people who created a metric or who enthuse about it in their consulting practices. In those reports, all is sunny and light. People might have some problems trying to use the metric, but overall (these reports say), someone who is skilled in its use can avoid those problems and rely on it. In the discussion, I ask skeptical questions. I demonstrate searches for more critical commentary on the metric. I point out that many of the claims by the cheerleaders lack data. They often present summaries of not-well-described experiments or case studies. They often present tables with numbers that came from not-exactly-clear-where. They often present reassuring experience reports. Experience reports (like the one you are reading right now) are useful. But they don’t have a lot of evidence-value. A well-written experience report should encourage you to try something yourself, giving you tips on how to do that, and encourage you to compare your experiences with the reporter’s. For experience reports to provide enough evidence of some claim to let you draw some conclusions about it, I suggest that you should require several reports that came from people not associated with each other and that made essentially the same claim, or had essentially the same thing/problem arise, or reached the same conclusion. I also suggest that you should look for counter-examples. During the meeting, I might make this same point and then hunt for more experience reports, probably in parallel with the student who is on their own computer while I run the search on mine. This type of attack on the data foundation of the metric is a bit familiar to these students because we use Bossavit’s book The Leprechauns of Software Engineering: How folklore turns into fact and what to do about it, as one of our texts.

In the essay, all students have performance problems (that is, all students have problems with the generic task of writing essays and the generic task of finding information) and so (in my extensive experience — 1 course), the discussion with all students switches back and forth between the content (the analysis of the metric and of the research data) and the generic parts of the assignment.

A Few More Thoughts

You can use interactive grading with anything that you can review. For example, Keith Gallagher uses interactive grading in his programming courses, reading the student’s code with them and asking questions about the architecture, style, syntax, and ability to actually provide the benefits the program is supposed to provide. His students have spoken enthusiastically about this to me.

Dr. Gallagher ends his sessions a little differently than I do. Rather than telling students what their grade is, he asks them what they think their grade should be. In the discussion that follows, he requires the student to view the work from his perspective, justifying their evaluation in terms of his grading structure for the work. It takes some skill to pull this off, but done well, it can give a student who is disappointed with their grade even more insight into what needs improvement.

Done with a little less skill, interactive grading can easily become an interaction that is uncomfortable or unpleasant for the student. It can feel to a student who did poorly as if the instructor is bullying them. It can be an interaction in which the instructor does all the talking. If you want to achieve your objectives, you have to intentionally manage the tone of the meeting in a way that serves your objectives.

Overall, I’m pleased with my decision to introduce interactive grading to my classes. As far as I can tell, interactive grading takes about the same amount of time as more traditional grading approaches. There are several benefits to students. Those with writing or language problems have the opportunity to demonstrate more skill or knowledge than their written work might otherwise suggest. They get highly personalized, interactive coaching that seems to help them submit better work on future assignments. Most students like interactive grading and consider it a worthwhile use of their time. Finally, I think it is important to have students feel they are getting a good value for their tuition dollar. This level of constructive personal attention contributes to that feeling.

This post is partially based on work supported by NSF research grant CCLI-0717613 ―Adaptation & Implementation of an Activity-Based Online or Hybrid Course in Software Testing. Any opinions, findings and conclusions or recommendations expressed in this post are those of the author and do not necessarily reflect the views of the National Science Foundation.

WTST 2013 Call for Participation: Teaching High Volume Automated Testing (HiVAT)

Tuesday, October 30th, 2012

Hello,

This is the WTST 2013 Call for Participation. Do you know someone who has been working in this area? If so, please pass this information along to them.

Thanks!

==============
12th WORKSHOP ON TEACHING SOFTWARE TESTING (WTST 2013)
JANUARY 25-27, 2013
MELBOURNE, FLORIDA
at the HARRIS INSTITUTE FOR ASSURED INFORMATION

 

WTST CALL FOR PARTICIPATION

TEACHING HIGH VOLUME AUTOMATED TESTING (HiVAT)

The Workshop on Teaching Software Testing is concerned with the practical aspects of teaching university-caliber software testing courses to academic or commercial students.

WTST 2013 is focused on high volume automated testing (HiVAT). Our goal is to bring together instructors who have experience teaching high-volume techniques or who have given serious consideration to how to teach these techniques. We also welcome participants focused on the teaching of complex cognitive concepts and the transfer of what was learned to industrial practice.

As at all WTST workshops, we reserve some seats for senior students who are strongly interested in teaching and for faculty who are starting their careers in this area or beginning a research program connected with teaching this type of material.

There is no fee to attend this meeting. You pay for your seat through the value of your participation. Participation in the workshop is by invitation based on a proposal. We expect to accept 15 participants with an absolute upper bound of 25.

BACKGROUND ON THE WORKSHOP TOPIC

High volume automated testing involves automated generation, execution and evaluation of the results of a large set of tests. This contrasts with more traditional “automated” testing that involves automated execution of a relatively small number of human-created tests.

Here are four examples of the types of problems that underlie the need for HiVAT:

  1. Many types of code weakness (such as timing-related problems) yield intermittent failures and are hard to detect with traditional testing techniques
  2. There are immense numbers of possible combination tests of several variables together. Some combinations are unique (a failure appears only on a particular combination)
  3. Some failures occur primarily when a system under test is under load and so detection and characterization is essentially a statistical challenge.
  4. Characterizing the reliability of software requires a statistically useful set of tests.

In the academic community, the most commonly discussed HiVAT family of techniques is called “fuzzing” – but fuzzing as we know it involves very simplistic evaluation of the test results—essentially run the software until it crashes or fails in some other very obvious way. Other HiVAT techniques rely on more powerful oracles and can therefore find other kinds of bugs.

WTST is about teaching testing, not creating new techniques. The challenge we are trying to address in this WTST is that many of these techniques are known but not widely applied. We believe this is because they are poorly taught. As far as we can tell, most testing courses don’t even mention these techniques. Of those that do (and go beyond fuzzing), our impression is that students come out baffled about how to actually DO that type of testing in their work.

At Florida Tech, we’re trying to address this by creating “reference implementations” for several techniques—open source demonstrations of them, with commentary on the design and implementation. We’re hoping that WTST will provide examples of other good approaches.

TO ATTEND AS A PRESENTER

Please send a proposal BY DECEMBER 1, 2012 to Cem Kaner <kaner@cs.fit.edu> that identifies who you are, what your background is, what you would like to present, how long the presentation will take, any special equipment needs, and what written materials you will provide. Along with traditional presentations, we will gladly consider proposed activities and interactive demonstrations.

We will begin reviewing proposals immediately. We encourage early submissions. It is unlikely but possible that we will have accepted a full set of presentation proposals by December 1.

Proposals should be between two and four pages long, in PDF format. We will post accepted proposals to http://www.wtst.org.

We review proposals in terms of their contribution to knowledge of HOW TO TEACH software testing. Proposals that present a purely theoretical advance in software testing, with weak ties to teaching and application, will not be accepted. Presentations that reiterate materials you have presented elsewhere might be welcome, but it is imperative that you identify the publication history of such work.

By submitting your proposal, you agree that, if we accept your proposal, you will submit a scholarly paper for discussion at the workshop by January 8, 2013. Workshop papers may be of any length and follow any standard scholarly style. We will post these at http://www.wtst.org as they are received, for workshop participants to review before the workshop.

TO ATTEND AS A NON-PRESENTING PARTICIPANT:

Please send a message by DECEMBER 1, 2012, to Cem Kaner <kaner@cs.fit.edu> that describes your background and interest in teaching software testing. What skills or knowledge do you bring to the meeting that would be of interest to the other participants?

The hosts of the meeting are:

Cem Kaner (http://kaner.com and http://www.testingeducation.org)
Rebecca Fiedler (http://bbst.info)
Michael Kelly (http://www.developertown.com/)

HOW THE MEETING WILL WORK

WTST is a workshop, not a typical conference. It is a peer conference in the tradition of The Los Altos Workshops on Software Testing (http://lawst.com). Our presentations serve to drive discussion. The target readers of workshop papers are the other participants, not archival readers. We are glad to start from already-published papers, if they are presented by the author and they would serve as a strong focus for valuable discussion.

In a typical presentation, the presenter speaks 10 to 90 minutes, followed by discussion. There is no fixed time for discussion. Past sessions’ discussions have run from 1 minute to 4 hours. During the discussion, a participant might ask the presenter simple or detailed questions, describe consistent or contrary experiences or data, present a different approach to the same problem, or (respectfully and collegially) argue with the presenter. In 20 hours of formal sessions, we expect to cover six to eight presentations. Some of our sessions will be activities, such as brainstorming sessions, collaborative searching for information, creating examples, evaluating ideas or work products. We also have lightning presentations, time-limited to 5 minutes (plus discussion). These are fun and they often stimulate extended discussions over lunch and at night.

Presenters must provide materials that they share with the workshop under a Creative Commons license, allowing reuse by other teachers. Such materials will be posted at http://www.wtst.org.

Our agenda will evolve during the workshop. If we start making significant progress on something, we are likely to stick with it even if that means cutting or time boxing some other activities or presentations.

LOCATION AND TRAVEL INFORMATION

We will hold the meetings at

Harris Center for Assured Information, Room 327

Florida Institute of Technology

150 W University Blvd

Melbourne, FL 32901

Airport

Melbourne International Airport is 3 miles from the hotel and the meeting site. It is served by Delta Airlines and US Airways. Alternatively, the Orlando International Airport offers more flights and more non-stops but is 65 miles from the meeting location.

Hotel

We recommend the Courtyard by Marriott – West Melbourne located at 2101 W. New Haven Avenue in Melbourne, FL.

Please call 1-800-321-2211 or 321-724-6400 to book your room by December 24, 2012. Be sure to ask for the special WTST rates of $93 per night. Tax is an additional 11%. All reservations must be guaranteed with a credit card by Tuesday, December 24, 2013. If rooms are not reserved, they will be released for general sale. Following that date reservations can only be made based upon availability.

For additional hotel information, please visit the hotel website at http://www.marriott.com/hotels/travel/mlbch-courtyard-melbourne-west/