Archive for the ‘Uncategorized’ Category

Racial Profiling in Ferguson Missouri? A Note on Statistical Interpretation

Thursday, August 21st, 2014

As I write this, there is serious community tension in Ferguson, Missouri over the shooting and killing of a unarmed black teenager by a white police officer, and over the response to that shooting by the local police force.

The narrative in much of the press is that this is yet another incident that illustrates a serious problem of racism in the United States, especially in our police. For example, many newspapers cite data from the Missouri 2013 Vehicle Stops Report. The overall report covers data in every county in Missouri and presents years of historical comparisons. The report specifically for Ferguson is here.

Here are some examples of press coverage that seem representative of what I’ve seen in many papers online:

“Last year, 86 percent of the cars stopped by Ferguson police officers were being driven by African-Americans, according to the state’s annual racial profiling report. Once pulled over in Ferguson, African-American drivers were twice as likely to be searched, according to the report.”

“Last year, for the 11th time in the 14 years that data has been collected, the disparity index that measures potential racial profiling by law enforcement in the state got worse. Black Missourians were 66 percent more likely in 2013 to be stopped by police, and blacks and Hispanics were both more likely to be searched, even though the likelihood of finding contraband was higher among whites.”

Of course, there are some counter-examples. Some news reports (what I’ve seen on Fox, for example) seem to ignore these data completely and instead appear to me to present the events in terms of violent bad black people who deserve whatever violent treatment the police provide for them. There is nothing useful to learn about data evaluation from these reports, so I will ignore them for the rest of this note.

I should state my bias: My personal (nonexpert) impression is that the shooting was unjustified and that the St. Louis County police response has been inappropriate. I have no insight into the motivation of anyone involved.

However, if you look at the actual numbers from Ferguson, it is not clear to me that the conclusions of racial profiling, conclusions like the ones quoted above, that have appeared in every news source that I respect, are justified by the data.

The focus of this blog is on the teaching of software engineering topics, primarily software testing and measurement (and thus too, statistical analysis).

The data from Ferguson provide an interesting example for caution in the interpretation of such data.

First, some of the numbers that are consistent with the summaries. According to the Attorney General’s report

  • Ferguson’s population (age 16 and over) is 15,865, of whom 63% are black.
  • 4632 of 5384 vehicle stops (86%) were of blacks, a much higher percentage than the 63% of the population
  • 562 of the 611 searches (92%) were of blacks
  • 483 of the 521 arrests (93%) were of blacks
  • 12.13% of the blacks who were stopped were searched, compared to only 6.85% of the whites
  • 21.71% of the blacks who were searched had contraband (drugs, weapons, stolen property) compared to 34.04% of the whites.

These data appear to suggest two conclusions:

  1. Blacks are being stopped, searched and arrested at a higher rate than their representation in the population
  2. Many more searches of blacks than whites are unproductive, suggesting the police would find more contraband if they searched fewer blacks and more whites.

If Ferguson’s police are continuing to search blacks at a much higher frequency than whites, even though searched whites have contraband at a higher frequency than searched blacks, this appears to suggest a pattern that is racist and counterproductive (less protective of public safety).

That conclusion, I think, is the conclusion the newspapers are inviting us to draw.

Let’s look at some more data.

  • 66% (369/562) of the searches and 76% (369/483) of the arrests of blacks involved an outstanding warrant
  • 30% (14/47) of the searches and 39% (14/36) of the arrests of whites involved an outstanding warrant

Searches and arrests involving warrants don’t involve much exercising of judgment on the part of the officer who is searching or arresting someone.

  • A warrant is an order from a court to arrest someone. The officer is supposed to stop and arrest a person if there is a warrant out for them.
  • When a police officer arrests someone, they must search the person. Among the many important reasons for this rule is the safety of the officer: arresting someone and then not checking them carefully for weapons would be extremely unwise.

In a community of only 15,865 people, it would not be surprising for the local police to be aware of most of the people who have warrants outstanding against them or for these police to recognize those people on the street.

Because the police are supposed to arrest people who have warrants against them and supposed to search people they arrest, I don’t think we should count these numbers of stops, searches and arrests against the police.

If you look only at the stops that didn’t involve outstanding warrants,

  • 34% of the times that police searched a black person, and 24% of the times the police arrested a black person, the search did not involve an outstanding warrant.

In contrast

  • 70% of the times that the police searched a white person, and 61% of the time they arrested a white person, the search did not involve an outstanding warrant.

The conclusions that these numbers suggest to me are that:

  • the Ferguson police appear to have been making discretionary stops (stops in which they were exercising their own judgment, rather than executing a court order) of white people at almost twice the rate as for black people
  • the higher contraband-find rate for whites than blacks might be because a higher proportion of whites were searched on the basis of police suspicion of contraband, compared to a higher proportion of blacks being searched as part of an arrest that involved a warrant (past bad behavior, not currently suspicious behavior). Considered this way, the disparity (higher rate of contraband-finds for whites versus blacks) seems unsurprising and not at all suggestive of bad police work.

I don’t know what truth underlies these numbers. I think that, for me to interpret them with any confidence, I would have to do other studies, such as riding along with Ferguson police and learning how they decide who to stop and what post-stop behaviors trigger further investigation (such as searches or checks for outstanding warrants).

What does seem clear to me is that the first conclusion (racially-motivated differences in the police officers’ decisions to search people) is not supported by these data. That motivation might be present but—despite first appearances—these data do not seem to be evidence of it.

I wrote this note because it suggests two important lessons for students of statistics and research design:

  1. In many cases (as here), data may show statistically significant (large, probably consistent) differences. However, interpretation of those differences is almost always open to further investigation.
    • The numbers don’t tell you what they mean. Even the most statistically significant trends must be interpreted by people.
  2. In many cases (as here), the data support alternative interpretations.
    • Whenever possible, you should look at your data in many ways, to see if they tell you the same story. If they don’t, you need to investigate further, and maybe fix your model.
    • A few numbers in isolation tell you very little, often much less than you would initially imagine.
    • If you design your research (or your management) so that you will see only a few numbers at the end, you are designing tunnel vision into your work. You are creating your own context for bad interpretations and bad decisions.

New Book: Foundations of Software Testing–A BBST Workbook

Friday, February 14th, 2014

New Book: Foundations of Software Testing—A BBST Workbook

Rebecca Fiedler and I just published our first book together, Foundations of Software Testing—A BBST Workbook.

Becky and I started working on the instructional design for the online version of the BBST (Black Box Software Testing) course in 2004. Since then,Foundations has gone through three major revisions. Bug Advocacy and Test Design have gone through two.

Our Workbooks mark our first major step toward the next generation of BBST™.

We are creating the new versions of BBST through Kaner, Fiedler & Associates, a training company that we formed to provide an income stream for ongoing evolution of these courses. BBST is a registered trademark of Kaner, Fiedler & Associates.

What’s in the Book

The Workbook includes slides, lecture transcripts, orientation activities and feedback, application activities, exam advice, and author reflections. Here are some some details:

All the course slides

Foundations has 304 slides. Some of these are out of date. We provide notes on these in the Author’s Reflections.

A transcript of the six lectures.

The transcripts are almost word-for-word the same as the spoken lecture. They actually reproduce the script that I wrote for the lecture. In a few cases, my scripts are a little longer than what actually made it past the video edits. We lay the transcript and the slides out together, side-by-side. In an 8.5×11 printed book, this is a great format for taking notes. Unfortunately, it doesn’t translate to Kindle well, so there is no Kindle edition of the book.

Four Orientation Activities

Orientation activities introduce students to a key challenge considered in a lecture. The student puzzles through the activity for 30 to 90 minutes, typically before watching the lecture, then sees how the lecture approaches this type of problem. The typical Foundations course has two to four of these.

The workbook presents the instructions for four activities, along with detailed feedback on them, based on past performance of students in the online and university courses.

I revised, rewrote or added (new) all of these activities for this Workbook. Because, in my opinion, the most important learning in BBST comes from what the students actually do in the class, the new Orientation and Application activities create a substantial revision to the course.

In my university courses, I practice continuous quality improvement, revising all of them every term in response to (a) my sense (and to what ever relevant data I have collected) about strengths and weaknesses that showed up in previous of the course or (b) ideas that have demonstrated their value in other courses and can be imported into this one. Most of the updates are grounded in a long series of revisions that I used and evaluated in my university-course version of BBST.

Two Application Activities

An application activity applies ideas or techniques presented in a lecture or developed over several lectures. The typical application activity calls for two to six hours of work, per student. The typical Foundations course has one to two of these.

One of these is revised from the public BBST, the other completely rewritten.

Advice on answering our essay-style exam questions

The advice runs 11 pages. I also provide a practice question and detailed feedback on the structure of the answer.

I think the advice is good for anyone taking the course, but it is particularly focused on university students who are preparing for an exam that will yield graded results (A, B, Pass-with-distinction, etc.). The commercial versions of BBST are typically pass-fail, so some of the fine details in this advice are beyond the needs of those readers. If you are a university student, I recommend this as a tighter and more polished presentation than the exam-preparation essay included in the public course.

Author reflections

My reflections present my sense of the strengths and weaknesses of the current course, the ways we are addressing those with the new activities, and some of the changes we see coming in the next generation of videos.

Because Foundations is written to introduce students to the fundamental challenges in software testing, some of my reflections add commentary on widely-debated issues in the field. Some of these might become focus points for the usual crowd to practice their Sturm und Drang on Twitter.

Who the Book is For

We want to support three groups with the BBST Workbooks:

  • Self-studiers. Many people watch the course videos on their own. The course videos for the current version of Foundations are available for free, along with the slides, the course readings, and the public-course versions of four activities and the study guide (list of essay questions for the exam). The Workbook updates the activities, and provides detailed feedback for all of the orientation activities, and provides several design notes on the orientation and application activities. If you are studying BBST on your own or with a few friends, we believe this provides much better support than the videos alone.
  • In-house trainers. If you are planning to teach BBST to staff at your company, the Workbook is an inexpensive textbook to support the students. The feedback for the activities provides a detailed survey of common issues and ideas raised in each activity. If your trainees submit their work to you for review, you might want to supplement these notes with comments that are specific to each student’s work. The comments in the workbook should cover most of the comments that you would otherwise repeat from student to student. The instructors’ reflections will, we hope, give you ideas about how to tailor the application activities (or replace them) to make them suitable for your company.
  • Students in instructor-led courses. The BBST Foundations in Software Testing Workbook is an affordably-priced (retail price $19.99) supplement to any instructor-led course. Students will appreciate the convenience of print versions of the course slides and lectures for ongoing reference. Instructors will appreciate the level of feedback provided to students in the workbook.

Buy the Book
Foundations of Software Testing: A BBST Workbook is available from:

Thinking about the fraud against Target

Wednesday, January 22nd, 2014

I read an interesting article in the Wall Street Journal today:

Basically, the theory presented in the article is that there are these wonderful credit/debit cards with embedded chips that are much more secure than the current system. If only Target (and other retailers) had adopted these, we would have less fraud. Apparently, the fault lies with Target.

I imagine that the expected response to this article is “What were they thinking?” as the reader realizes that more-effective technology was at hand at what might have been a reasonable price.

I got to watch some of this play out in the late 1990′s. At the time, I was working as a technology-focused lawyer and one of the areas I worked on was electronic payment systems. I published a few papers on this. One available from my website appeared in 1998 in the Journal of Electronic Commerce, called “SPLAT! Requirements bugs on the information superhighway“, see

The issues I wrote about in this (and related papers) involved the use of public-key encryption systems to guarantee identity. The same commercial-liability issues were coming up for chip cards, with the same rationale.

These systems offered the potential of significantly reducing fraud in consumer transactions. Fraud was seen as a big problem. With these savings of billions of dollars of losses, some credit card company representatives spoke of being able to noticeably lower their fees and interest rates. Who wouldn’t want that?

Unfortunately, some financial services firms (and some other folks) saw two opportunities here.

  1. They hated paying money to criminals committing fraud
  2. They hated guaranteeing every credit card transaction in the event of fraud—they wanted to put this risk back on the consumer but current legislation wouldn’t let them

The proposals to adopt encryption-based identification systems in commerce tied these together. The proposed laws would:

  1. authorize the use of encryption-based identification as equivalent to an ink signature
  2. treat the encryption-based identification as absolutely authoritative, so that if someone successfully impersonated you, you would bear all the loss. Current law sticks the financial-services firms with the risk of credit-card fraud losses because they design the system and decide how much security to build into it. The proposed new system would be an alternative to the consumer-protected credit-card system. It would flip the risk to the consumer.

I think legislation would have easily passed that provided incentives to adopt encryption-based identification. For example, the legislation could have created a “rebuttable presumption” — an instruction to a court to assume that a message encrypted with your key came from you and if you wanted to deny that, you would have to prove it.  This legislation would have reduced fraud, which would benefit everyone. (Well, everyone but the criminals…)

Unfortunately, the demand went further. Even if you could prove that you were the victim of identity theft that was in no way your fault, you would still be held accountable for the loss. 

The lawyers advocating for incentivizing encryption-based identification weren’t willing to separate the proposals. The result of their inflexibility was opposition to encryption-based payment-related identification systems (including chip cards). One dimension of the opposition was technical–the security of the payment systems was almost certainly less (and therefore the risk of fraud that was created by the system and not by negligence of the consumer was greater) than the most enthusiastic proponents imagined. Another dimension was irritation with what was perceived as greed and unwillingness to compromise.

Back then, I saw this play out because I was helping a committee write the Uniform Electronic Transactions Act (UETA). This eventually passed in most states and was then federalized under the name ESIGN. ESIGN now governs electronic payments in the United States. The multi-year drafting process that yielded UETA/ESIGN offered a unique opportunity to write incentives for stronger identification systems into the laws governing electronic payments. Instead, we chose to write legislation that accepted a status quo that involved too much fraud, with prospects of much worse fraud to come. I was one of the people who successfully encouraged the UETA drafting committee to take this less-secure route because there was no politically-feasible path to what seemed like the obvious compromise.

Our economy has benefited enormously from legislation that lets you buy something by clicking “I agree”, without having to sign a physical piece of paper with a physical ink-pen. We could have done this better. Instead, we accepted the predictable future outcome that the United States would continue to use insecure payment systems, that would result in ongoing fraud, like the latest attacks on Target, Neiman Marcus, and (apparently, according to recent reports) at least six other national retailers.

The 2014 Workshop on Teaching Software Testing is Expanded

Monday, October 28th, 2013

The 2014 Workshop on Teaching Software Testing is scheduled for January 24-26 in Melbourne, FL This year’s focus will be on advanced courses in software testing. You can read the details at

In addition, we will immediately follow WTST with the first live pilot of the Domain Testing class. This will be our first of the next generation of BBST classes. Rebecca Fiedler and I have been working on a new course design and we plan to use some new technology. Our first pilot will run in Melbourne from Jan 27-31, 2014. Our second live pilot is scheduled for May 12-16 at DeveloperTown in Indianapolis.

Are you interested in attending one of the pilot courses? If so, please apply here. If you are accepted as a Domain Testing Workshop participant, there will be a non-refundable deposit of $100 to defray the cost of meals, snacks, and coffee service. We’ll publish more details about the pilot courses as they become available.


DOMA–Time for a change

Thursday, March 28th, 2013








Some people have asked me why I posted this graphic. It is a token of solidarity with the Human Rights Campaign. This is one of several images that HRC supporters posted on their websites while the United States Supreme Court was hearing arguments about the Defense of Marriage Act.

Please update your links to this blog

Sunday, November 20th, 2011

A few new posts will be coming soon. I’m hoping they won’t be missed.

I moved my blog to over a year ago — If you’re still linking to the old site, please update it.


AST Instructors’ Tutorial at CAST in Toronto

Wednesday, May 28th, 2008

You’ve read about the Association for Software Testing’s free software testing courses. Now find out how you can get involved in teaching these for AST, for your company, or independently. This workshop will use presentations, lectures, and hands-on exercises to address the challenges of teaching online: Becky Fiedler, Scott Barber and I will host the Live! AST Instructors Orientation Course Jumpstart Tutorial On July 17, 2008, in conjunction with this year’s Conference of the AST (CAST).

To register for this Tutorial, go to

To register for CAST (early registration ends June 1)  go to

More Details

AST is developing a series of online courses, free to our members, each taught by a team of volunteer instructors. AST will grant a certificate in software testing to members who have completed 10 courses. (We hope to develop our 10th course by July 2009).

AST trains and certifies instructors: The main requirements for an AST member to be certified to teach an AST course are (a) completing the AST course on teaching online, (b) teaching the same course three times under supervision, (c) approval by the currently-certified instructors for that course, and agreement to teach the course a few times for AST (for free).

As a certified instructor, you can offer the course for AST credit: for AST (for free), for your company, or on your own. You or your company can charge fees without paying AST any royalties or other fees. (AST can only offer each free course a few times per year–if demand outstrips that supply, instructors will have a business opportunity to fill that gap.)

This Tutorial, the day after CAST, satisfies the Instructors Orientation Course requirement for prospective AST-certified instructors.

This workshop will use presentations, lectures, and hands-on exercises to address the challenges of teaching online. (Bring your laptop and wireless card if you can.) The presenters will merge instructional theory and assessment theory to show you how they developed the AST-BBST online instructional model. Over lunch, Scott Barber will lead a panel discussion of AST members who are working on AST Instructor Certification.

Your registration includes a boxed lunch and light snacks in the morning and afternoon.

This workshop is partially based on research that was supported by NSF Grants EIA-0113539 ITR/SY+PE:“Improving the Education of Software Testers and CCLI-0717613 “Adaptation & Implementation of an Activity-Based Online or Hybrid Course in Software Testing. Any opinions, findings and conclusions or recommendations expressed in this workshop are those of the presenter(s) and do not necessarily reflect the views of the National Science Foundation.

Sponsorship opportunities for CAST2008 still available

Friday, May 23rd, 2008

There are still sponsorship opportunities available for CAST08, the annual conference of the Association for Software Testing.The AST is a non-profit professional association with a mission of advancing the understanding and practice of software testing through conferences, LAWSTâ„¢ style peer workshops, publications, training courses, web sites, and other services.

This is an exciting year for CAST. We are planning for 350 participants to join us this year at 89 Chestnut in Toronto, Canada. We’ve secured Gerald M. Weinberg, Cem Kaner, Robert Sabourin, and Brian Fish as keynote speakers related to the conference theme of Beyond the Boundaries: Interdisciplinary Approaches to Software Testing. Pre-conference tutorials are being given by Jerry Weinberg, Scott Barber, Hung Nguyen, and Julian Harty. The rest of the program is equally impressive in terms of content and speaker quality. As an added bonus, Jerry has scheduled the official launch of his new book Perfect Software and Other Testing Myths to be at CAST. These are just some of the reasons that CAST attracts some of the most well known and influential testers, testing consultants, and trainers of testers from around the globe to attend as participants.

CAST’s Sponsorship and Exhibitor Programs include what you’d expect from a conference in our industry, but probably at a lower price than you might be used to.  There are packages where promotional material is delivered to participants on your behalf, meal and artifact (participant backpacks, proceedings CDs, etc.) sponsorship opportunities, as well as exhibitor booth options. Additionally, this year we have added a “Vendors & Service Providers Only” track at the conference.  This is where you get to come and discuss, demonstrate, answer questions about, and/or introduce your products and services to interested CAST participants. We do require vendors to follow the same 2/3, 1/3 rule that CAST mandates for all sessions, that is: You present for up to 2/3 of the total time slot, you leave at least 1/3 of the time for facilitated questions and answers. Last year we piloted this idea with 2 sessions. Participants *loved* that the vendors were presenting then sticking around to field hard questions and the vendors who presented reported that in addition to high quality leads and contacts, they gained significantly more valuable market information than they get as exhibitors or vendor-presenters at other conferences.

I hope you find all the information you need to decide what level of sponsorship is right for you in the link below, but if you do not find what you need, please do not hesitate to contact AST’s Executive Director and this year’s CAST Sponsorship Chairperson, Scott Barber at

CAST 2008 Sponsorship and Exhibitor Program:

Program published for CAST (Conference of the Association for Software Testing)

Friday, April 25th, 2008

The CAST program is at

CAST is a special conference because of its emphasis on open discussion and critical thinking.

For example, all speakers, including keynotes, are subject to unlimited questioning. If the discussion of a keynote goes beyond the end of the keynote time, we either move the next session to a new room, or move the keynote discussion to a new room. For example, the “questioning” for a keynote by James Bach included a full critical counterpresentation by one of the people in the audience, followed by a long debate. Several 1-hour track presentations have morphed into 2-hour (moderated) discussions.

Conference of the Association for Software Testing

Tuesday, May 29th, 2007

CAST (Conference of the Association for Software Testing) early bird
registration ends this week. Register at

I just got back from ICSE (the International Conference on Software
Engineering). ICSE is the premier conference for academic software
engineers–the conference acceptance standards are so high (only 5% of
submitted papers are accepted) that a paper at ICSE is ranked by
academic bean counters on a par with most computer science journals.

Many (20%?) of the ICSE sessions involved software testing. It was
frustrating to sit through them because the speaker would talk for a
while, say some outrageous things, some interesting things and some
confusing things–and then at the end of the talk, we got 3 minutes to
ask questions and argue with the speaker and the other participants.
Some talks (such as an invited address on the Future of Software
Testing) were clearly designed to be thought-provoking but with only 5
minutes for discussion, their impact evaporated quickly.

In AST, we think that testers, of all people, should critically evaluate
the ideas that are presented to them and we think that testers’
conferences should encourage testers to be testers.

We designed CAST to foster discussion. Every talk–every keynote, every
track talk, every tutorial–has plenty of time for discussion. If a talk
provokes a strong discussion, we delay the next talk. If it looks as
though the discussion will go for a long time, we move the next talk to
another room, so people can choose to stay with the discussion or go to
the next talk. I haven’t seen this at any other conference. It
contributes tremendously to the overall quality of the sessions.

By the way, if you want to come to the meeting to challenge what one of
the speakers is saying, you can prepare some remarks in advance. It is
important that your response be courteous and appreciative of the effort
the speaker has put into her presentation. If you have that, give Jon
Bach some notice and you will almost certainly get floor time right
after that speaker.

Another interesting event at CAST will be the full-day “tutorial” /
workshop / debate on certification. We had a keynote last year on
certification that stirred up so much discussion it nearly took over the
rest of the day. This time, it looks like we’ll have representatives of
all the main groups who certify software testers, plus an academic
curmudgeon (might be me) who is skeptical of the value of all of them.
If you have ever considered favoring certified software testers in your
hiring practices or taking a certification exam to improve your
marketability or credibility, this day will provide a six-way discussion
that compares and contrasts the different approaches to credentialing
software testers. I don’t think there has ever been a session like this
at any other testing conference.

Before CAST, the Workshop on Heuristic & Exploratory Techniques will
explore boundary analysis (domain testing), how we actually decide what
boundaries to consider and what values to use. This might sound
straightforward, but in doing task analyses at some other meetings, we
discovered that this is a remarkably complex problem. For more
information, contact Jon Bach.

After CAST, the Workshop on Open Certification will meet to develop
certification questions and work out the logistics of developing the
Exam Server (we already have a Question Server running for the project).
If you want to contribute to this project, contact Mike Kelly or Cem
Kaner. For more info, see