Where on the Diamond do Defensive Statistics Agree?

Defensive metrics in baseball can be tricky to really grasp. Fortunately, as defensive metrics have become more popularized over the last handful of seasons, it has been correspondingly easier to at least contextualize them. OAA? DRS? These are metrics that one might not have ready and complete definitions for, but are now more regularly recognized and generally made sense of.

One place these metrics are discussed more often is MLB Trade Rumors. For anyone who reads MLB Trade Rumors, one might notice that it seems increasingly common for writers covering position player news to provide some insight into any newly signed, or traded, players’ defensive value according to these metrics. Something along the lines of “X player’s defensive work at Y position was seen as strong by OAA and DRS, but UZR wasn’t so sure”.

I, for one, find these brief recaps helpful if only to reinforce the frequencies of both relative consensus and variation across defensive metrics. It is also interesting to note sometimes considerable swings in values assigned to players from season to season. It is a good reminder that not all metrics necessarily align and that no metric is perfect (lest it become the one metric to rule them all?).

The three metrics that seem to appear most often are OAA (outs above average), DRS (defensive runs saved), and UZR (ultimate zone rating). Interestingly though, nearly as regularly as they appear, they are at odds. I touched on this awhile back, but now (following an offseason of trades and signings peppered with these metrics being juxtaposed) wonder how these three “flagship” defensive metrics vary across defensive positions.

Put another way, is there more consensus among OAA/DRS/UZR for second basemen or for center fielders? For defensive positions that correspond to greater variability between metrics, the value of those metrics might be taken with a (slight larger) grain of salt relative to metrics at other positions.

So, as is often the case, addressing a question such as this begins with a FanGraphs leaderboard. Data for the last four seasons was pulled, split by player, position, and season. Only data for 750+ innings was considered here given that 750 innings corresponds to roughly 83 (9 inning) games. This filter was applied so only defensive metrics for players that spent about half a full season at any given position were to be considered. This filter also removed all player-seasons from the shortened 2020 season.

Visually, the following table illustrates the top 10 rows, sorted by player name, of the dataset that is used here.

From here, the goal is pretty simple: determine whether those three key metrics align, on average, more often for some positions than for others.

However, OAA, DRS, and UZR aren’t necessarily on the same scale. 5 DRS might not translate to 5 OAA, making comparisons across metrics as they currently stand a bit misleading. To illustrate this, the mirror histogram below juxtaposes the distribution of UZR (in maroon) and OAA (in orange).

Across the timeframe examined, UZR clearly has more extreme values and is less tightly centralized around 0 than OAA, which appears a bit more normally distributed.

To address this discrepancy in distributions, all three metrics were min-max rescaled, and those figures were compared. That way, a player with an OAA figure that was in the 100th percentile can exactly match an UZR that is also in the 100th percentile, regardless of the relative upper bounds of those original statistics.

The standard deviation was then taken across all three rescaled metrics for each player-season. Taking a standard deviation across three datapoints doesn’t offer much insight into real variability of course, but for these purposes it is intended simply to be descriptive. Put simply, how much do these three metrics vary, for each qualifying player-season?

These results are, in a word, perplexing. Average standard deviation, provided in the rightmost column, is greatest among first shortstops and least among first basemen. So, the three defensive metrics measured here align most for first baseman, and least for shortstops. One could pretty easily rationalize this, as shortstops must cover more ground and generally are part of a wider varieties of plays.

Beyond that however, things are muddled. Centerfield, arguably the next most demanding position among those listed here and where the most ground is covered, offers more consensus than the outfield corners. Additionally, the standard deviations among 2B, 3B, RF, LF are nearly identical. There seems to be no systematically greater or lesser variation between infield and outfield, or corners versus up the middle, as I might have anticipated.

Of course, these results aren’t intended to be rigorous and the variability between these averages could be in a large part due to chance alone; what it does do is offer a glimpse into the variability among these measures and in doing so underscores their nuanced nature. And interestingly that variation is greatest when evaluating a position so important as shortstop.