You are here
Football Zebras > News > Study that claims to reveal officials’ bias reveals the study has bias against officials

Study that claims to reveal officials’ bias reveals the study has bias against officials

When it comes to officiating, there are a minefield of perceived biases. Visitors claim that officials succumb to the cheers and jeers of the home team crowd. Fans claim that the NFL pushes officials to favor a chosen team. At the end of the day, both teams may leave convinced that the officials were out to get them. However, a recent article by Noah Davis and Dr. Michael Lopez may have overplayed its hand when when it claimed, “sideline bias in the NFL is real, and it’s spectacular.”

The FiveThirtyEight article is based on a 2016 paper by Lopez, an assistant professor of statistics at Skidmore College, which looked at whether plays toward a sideline resulted in a higher proportion of fouls beneficial to that sideline. Four types of penalties, including holding and pass interference, were included in the analysis, which appeared in the October 2016 issue of the economics journal Economic Inquiry. While the paper finds that there is an imbalance between the number of fouls called to each sideline, a critical review finds that the absolute difference is quite small. The paper then errs when it asserts that officials must be the cause for the discrepancy while discounting the possibility that the difference might be a natural consequence of normal play.

Issues start early on when, during the presentation of the paper’s hypotheses, Lopez asserts that any difference found would necessarily be attributable to pressure to call or not call a penalty which benefits the team whose sideline the official is on. This assertion, which is unnecessary to the paper’s core hypotheses, presupposes that the singular cause is the officiating. By doing so at the beginning of its methods section, it leads the reader to believe that the author has analyzed a “control” group which factors for  all other causes.

The paper utilizes raw data sourced from Armchair Analysis and Football Outsiders, both reputable subscription services. From this data, Lopez concludes that there is a statistically significant difference in the frequency of calls to each sideline, but only for defensive pass interference and aggressive defensive fouls. He further concludes that location relative to the goal line has an effect on whether fouls for offensive holding are called on runs toward the defense’s sideline (though there is no statistically significant difference overall).

Lopez, a former collegiate offensive lineman, has a keen interest in using statistics to analyze sports, especially football and hockey. Lopez has written several reviews, including an analysis of when penalties such as offensive holding are more likely to be called during a game. In all, it’s not Lopez’s statistics which are flawed, but rather the conclusion that he jumps to with those statistics. The paper finds some statistically significant correlations, but then attempts to present the findings as showing a causal link, which is beyond the reach of this type of study. This gap between correlation and cause would be acceptable for a discussion at the local watering hole, where there are lower editorial standards than a peer-reviewed journal article.

Notably, Lopez does not consider whether the fouls called are actually warranted. The NFL issues grades to every official each game, breaking down each play and judging both calls and no-calls. By the NFL’s accounting, officials get over 97% of the calls correct every game. Fewer than 5 calls per game are graded either as a miscall or a “support” (a call in a  grey area and not judged wrong). These ratings by the NFL stand in contradiction to claims that the officials exhibit a sideline bias, as an observer grading the game after the fact is not subject to the supposed influence of the sideline. In fact, an official who does display a bias, whether it be toward or against a team or sideline, is likely to be out of a job with a tarnished reputation — although the league’s extensive scouting and vetting process would identify any of these tendencies before such an official is hired.

In addition, the magnitude of the difference is blown out of proportion by the article. The graphs published in the paper and displayed by FiveThirtyEight give the appearance of bigger disparity than bears scrutiny. While FiveThirtyEight claims that defensive penalties are called about 50% more often on the offensive team’s sideline, over the 160 plays in an average NFL game, the actual figure comes out to less than 1 foul per game. At these levels, the differences are more mundane than they are spectacular. The chart from FiveThirtyEight below (“defensive aggressive penalties”) shows areas of uncertainty that are separated by less than  ¼ of a penalty for every 1,000 pass plays, but the scale of the chart gives the appearance of a wide gulf.


In fact, the discrepancy may have a more natural cause: the players. A defensive player, playing on the offensive sideline, may play more aggressively. “Friendly sideline banter” may increase the likelihood that a defensive player, who doesn’t have the luxury of knowing the play beforehand, will commit a foul when on the offensive sideline.

Granted, the article does include a discussion of possible of player-performance factors, admitting that the effect on sideline plays is most relevant for aggressive defensive penalties. But it discards this explanation because it is an inadequate justification for offensive holding calls, one of the more weakly correlated results. He further discounts this player-sideline effect for DPI, claiming that, because DPI could occur anywhere in the outer third of the field, the effect would be minor. Here Lopez errs in failing to appreciate the time cornerbacks spend near to the opposing sideline before and at the start of plays. In discarding this explanation, Lopez goes all-in on the officials as the reason for the difference, but the existence of a difference alone doesn’t necessarily demonstrate that the cause was official bias.

It’s easy to place the blame at the feet of the officials who point out the fouls. We want officials to “let the players play” when a foul is called on our team. At the same time, we want the officials to stop the other team from “getting away with murder.” But officials tend to work by one common sense rule: they don’t hand out flags; the players earn them.

(adsbygoogle = window.adsbygoogle || []).push({}); (adsbygoogle = window.adsbygoogle || []).push({});

Similar Articles

3 thoughts on “Study that claims to reveal officials’ bias reveals the study has bias against officials

  1. A foul is a foul is a foul. A foul is the first 5 minutes is the same as a foul in the last 5 minutes. A foul in the end zone is the same as a foul at the 50 yard line. A foul near a sideline is the same as a foul near the other sideline.
    Now, what’s different is how it is managed. As an official, I am taught to pay careful attention to fouls that occur near an opponent’s bench. Retaliation is a big factor when your teammate is wronged right in front of you. The official should step in immediately, diffuse the situation by his presence, and get back to the game.

  2. You can paint it anyway you want the refs cheat and are biased the Jaguars have been playing against the refs since 2003. They make calls on the Jaguars yet their opponents do the same exact thing and no call…heck we had that dumb broad and she stood with her hands crossed through 3 PI’s on Allen Robinson and finally in the 4th quarter the flag came from the other side of the field for another PI. It was awful the Greenbay game and the San Diego was called by the same ref and if he could find a penalty he made one up then to make the amount of calls even and they thought the games were out of reach they finally called on the Jags opponents. So tired of these old me and dumb woman screwing up games so that the “chosen” teams would win 🙁

Comments are closed.