The New York Times, on Monday, showcased (.htm) an NBER working paper (.pdf) that proposed that “blacks are 23.8 percent less likely to be shot at by police relative to whites.” (p.22)
The paper involved a monumental data collection effort to address an important societal question. The analyses are rigorous, clever and transparently reported. Nevertheless, I do not believe the above conclusions are justified from the evidence. Relying on additional results reported in the paper, I show here the data are consistent with police shootings being biased against Blacks, but too noisy to confidently conclude either way [1],[2].
Teenagers in bikinis
As others have noted [3], an interesting empirical challenge for interpreting the shares of Whites vs Blacks shot by police while being arrested is that biased officers, those overestimating the threat posted by a Black civilian, will arrest less dangerous Blacks on average. They will arrest those posing a real threat, but also some not posing a real threat, resulting in lower average threat among those arrested by biased officers [4].
For example, a biased officer may be more likely to perceive a Black teenager in a bikini as a physical threat (YouTube) than a non-biased officer would, lowering the average threat of his arrestees. If teenagers in bikinis, in turn, are less likely to be shot by police than armed criminals are, racial bias will cause a smaller share of Black arrestees to be shot. A spurious association showing no bias precisely because there is bias.
A closer look at the table behind the result that Blacks are 23.8% less likely to be shot, leads me to suspect the finding is indeed spurious.
Let’s focus on the red rectangle (the other columns don't control for threat posed by arrestee). It reports odds ratios for Black relative to White arrestees being shot, controlling for more and more variables. The numbers tell us how many Blacks are shot for every White that is. The first number, .762 is where the result that Blacks are 23.8% less likely to be shot comes from (1-.762=.238). It controls for nothing, criminals and teenagers in bikinis are placed in the same pool .
The highlighted Row 4 shows what happens when we control for, among other things, how much of a threat the arrestee posed (namely, whether s/he drew a weapon). The odds ratio jumps from .76 to 1.1. The evidence suggesting discrimination in favor of Blacks disappears, exactly what you expect if the result is driven by selection bias (by metaphorical teenagers in bikinis lowering the average threat of arrestees).
Given how noisy the results are, big standard errors (see next point), I don't read much from the fact that the estimate goes over 1.0 (showing discrimination against Blacks), I do make much of the fact that the estimate is so unstable, and it moves dramatically in the predicted direction by the “it is driven by selection-bias” explanation.
Noisy estimates
The above discussion took the estimates at face value, but they have very large standard errors, to the point they provide virtually no signal. For example:
Row 4 is compatible with Blacks being 50% less likely to be shot than Whites, but
Row 4 is compatible with Blacks being 80% more likely to be shot than Whites.
These results do not justify updating our beliefs on the matter one way or the other.
How threatening was the person shot at?
Because the interest in the topic is sparked by videos showing Black civilians killed by police officers despite posing no obvious threat to them, I would define the research question as follows:
When a police officer interacts with a civilian, is a Black civilian more likely to be shot than a White civilian is, for a given level of actual threat to the police officer and the public?
The better we can measure and take into account threat, the better we can answer that research question.
The NBER paper includes analyses that answer this question better than the analyses covered by The New York Times do. For instance, Table 8 (.png) focuses on civilians shot by police and asks: Did they have a weapon? If there is bias against Blacks, we expect fewer of them to have had a weapon when shot, and that's what the table reports [5].
14.9% of White civilians shot by White officers did not have a weapon.
19.0% of Black civilians shot by White officers did not have a weapon.
The observed difference is 4.1 percentage points, or about 1/3 the baseline (a larger effect size than the 23.8% behind the NY Times story). As before the estimates are noisy and not statistically significant.
When big effect size estimates are not statistically significant we don't learn the effect is zero, we learn the sample is too small,the results inconclusive; not newsworthy.
Ideas for more precise estimates
One solution is larger samples. Obviously, but sometimes hard to achieve.
Collecting additional proxies for threat could help too. For example, the arrestee's criminal record, the reason for the arrest, and the origin of the officer-civilian interaction (e.g., routine traffic stop vs responding to 911 call). What kind of weapon did the civilian have and was it within easy reach. Etc.
The data used for the NBER article includes long narrative accounts written by the police about the interactions. These could be stripped of race identifying information and raters be asked to subjectively evaluate the threat level right before the shooting takes place.
Finally, I'd argue we expect not just a main effect of threat, one to be controlled with a covariate, but an interaction. In high-threat situations the use of force may be unambiguously appropriate. Racial bias may be play a larger role in lower-threat situations.
Author feedback.
Our policy is to contact authors whose work we discuss to request feedback and give an opportunity to respond within our original post. Roland Fryer, the Harvard economist author of the NBER article, generously and very promptly responded providing valuable feedback that I hope to have been able to adequately incorporate, including the last paragraphs with constructive suggestions. (I am especially grateful given how many people must be contacting him right after the New York Times articles came out.)
PS: Josh Miller (.htm) from Bocconi had a similar set of reactions that are discussed in today's blogpost by Andrew Gelman (.htm).
Footnotes.
- The paper and supplement add to nearly 100 pages, by necessity I focus on the subset of analyses most directly relevant to the 23.8% result. [↩]
- The result most inconsistent with my interpretation of the data are reported in Table 6 (.png), page 24 in the paper, comparing the share of police officers who self-reported, after the fact, whether they shot before or after being attacked. The results show an unbelievably large difference favoring Blacks; officers self-report being 44% less likely to shoot before being attacked by Black vs White arrestees. [↩]
- See e.g. these tweets by political scientist Matt Blackwell .pdf [↩]
- The intuition behind this selection bias is commonly relied on to test for discrimination. It dates back at least to Becker (1957) "The Economics of Discrimination"; it's been used in empirical papers examining discrimination in real estate, bank loans, traffic stops, teaching evaluations, etc. [↩]
- The table reports vast differences in the behavior of White and Black officers; I suspect this means the analyses need to include more controls. [↩]