As I write this, there is serious community tension in Ferguson, Missouri over the shooting and killing of a unarmed black teenager by a white police officer, and over the response to that shooting by the local police force.
The narrative in much of the press is that this is yet another incident that illustrates a serious problem of racism in the United States, especially in our police. For example, many newspapers cite data from the Missouri 2013 Vehicle Stops Report. The overall report covers data in every county in Missouri and presents years of historical comparisons. The report specifically for Ferguson is here.
Here are some examples of press coverage that seem representative of what I’ve seen in many papers online:
“Last year, 86 percent of the cars stopped by Ferguson police officers were being driven by African-Americans, according to the state’s annual racial profiling report. Once pulled over in Ferguson, African-American drivers were twice as likely to be searched, according to the report.”
http://www.mcclatchydc.com/2014/08/19/237001_feds-could-go-several-ways-in.html?rh=1#storylink=cpy
“Last year, for the 11th time in the 14 years that data has been collected, the disparity index that measures potential racial profiling by law enforcement in the state got worse. Black Missourians were 66 percent more likely in 2013 to be stopped by police, and blacks and Hispanics were both more likely to be searched, even though the likelihood of finding contraband was higher among whites.”
Of course, there are some counter-examples. Some news reports (what I’ve seen on Fox, for example) seem to ignore these data completely and instead appear to me to present the events in terms of violent bad black people who deserve whatever violent treatment the police provide for them. There is nothing useful to learn about data evaluation from these reports, so I will ignore them for the rest of this note.
I should state my bias: My personal (nonexpert) impression is that the shooting was unjustified and that the St. Louis County police response has been inappropriate. I have no insight into the motivation of anyone involved.
However, if you look at the actual numbers from Ferguson, it is not clear to me that the conclusions of racial profiling, conclusions like the ones quoted above, that have appeared in every news source that I respect, are justified by the data.
The focus of this blog is on the teaching of software engineering topics, primarily software testing and measurement (and thus too, statistical analysis).
The data from Ferguson provide an interesting example for caution in the interpretation of such data.
First, some of the numbers that are consistent with the summaries. According to the Attorney General’s report
- Ferguson’s population (age 16 and over) is 15,865, of whom 63% are black.
- 4632 of 5384 vehicle stops (86%) were of blacks, a much higher percentage than the 63% of the population
- 562 of the 611 searches (92%) were of blacks
- 483 of the 521 arrests (93%) were of blacks
- 12.13% of the blacks who were stopped were searched, compared to only 6.85% of the whites
- 21.71% of the blacks who were searched had contraband (drugs, weapons, stolen property) compared to 34.04% of the whites.
These data appear to suggest two conclusions:
- Blacks are being stopped, searched and arrested at a higher rate than their representation in the population
- Many more searches of blacks than whites are unproductive, suggesting the police would find more contraband if they searched fewer blacks and more whites.
If Ferguson’s police are continuing to search blacks at a much higher frequency than whites, even though searched whites have contraband at a higher frequency than searched blacks, this appears to suggest a pattern that is racist and counterproductive (less protective of public safety).
That conclusion, I think, is the conclusion the newspapers are inviting us to draw.
Let’s look at some more data.
- 66% (369/562) of the searches and 76% (369/483) of the arrests of blacks involved an outstanding warrant
- 30% (14/47) of the searches and 39% (14/36) of the arrests of whites involved an outstanding warrant
Searches and arrests involving warrants don’t involve much exercising of judgment on the part of the officer who is searching or arresting someone.
- A warrant is an order from a court to arrest someone. The officer is supposed to stop and arrest a person if there is a warrant out for them.
- When a police officer arrests someone, they must search the person. Among the many important reasons for this rule is the safety of the officer: arresting someone and then not checking them carefully for weapons would be extremely unwise.
In a community of only 15,865 people, it would not be surprising for the local police to be aware of most of the people who have warrants outstanding against them or for these police to recognize those people on the street.
Because the police are supposed to arrest people who have warrants against them and supposed to search people they arrest, I don’t think we should count these numbers of stops, searches and arrests against the police.
If you look only at the stops that didn’t involve outstanding warrants,
- 34% of the times that police searched a black person, and 24% of the times the police arrested a black person, the search did not involve an outstanding warrant.
In contrast
- 70% of the times that the police searched a white person, and 61% of the time they arrested a white person, the search did not involve an outstanding warrant.
The conclusions that these numbers suggest to me are that:
- the Ferguson police appear to have been making discretionary stops (stops in which they were exercising their own judgment, rather than executing a court order) of white people at almost twice the rate as for black people
- the higher contraband-find rate for whites than blacks might be because a higher proportion of whites were searched on the basis of police suspicion of contraband, compared to a higher proportion of blacks being searched as part of an arrest that involved a warrant (past bad behavior, not currently suspicious behavior). Considered this way, the disparity (higher rate of contraband-finds for whites versus blacks) seems unsurprising and not at all suggestive of bad police work.
I don’t know what truth underlies these numbers. I think that, for me to interpret them with any confidence, I would have to do other studies, such as riding along with Ferguson police and learning how they decide who to stop and what post-stop behaviors trigger further investigation (such as searches or checks for outstanding warrants).
What does seem clear to me is that the first conclusion (racially-motivated differences in the police officers’ decisions to search people) is not supported by these data. That motivation might be present but—despite first appearances—these data do not seem to be evidence of it.
I wrote this note because it suggests two important lessons for students of statistics and research design:
- In many cases (as here), data may show statistically significant (large, probably consistent) differences. However, interpretation of those differences is almost always open to further investigation.
- The numbers don’t tell you what they mean. Even the most statistically significant trends must be interpreted by people.
- In many cases (as here), the data support alternative interpretations.
- Whenever possible, you should look at your data in many ways, to see if they tell you the same story. If they don’t, you need to investigate further, and maybe fix your model.
- A few numbers in isolation tell you very little, often much less than you would initially imagine.
- If you design your research (or your management) so that you will see only a few numbers at the end, you are designing tunnel vision into your work. You are creating your own context for bad interpretations and bad decisions.