2019 NFL OC Analytics Scorecard

A Look into Coach’s Statistical Impact

The outcome of football games are determined by manly things: team skill, play execution  and referee interaction to name a few. Yet, if we judged this by the amount of fans yelling at their screen, it’d be fair to think coach play-calling and scheme also play a large part in outcomes. Whether addressing the armchair QB’s claims or to help handicappers adjust their predictive models, this piece is the first in a series that will start to answer the questions: What Impact do Coaches Objectively have on NFL outcomes, and Who does it best?

Currently, there are some industry leaders that are doing very interesting things with objectively measuring coach decision making and their impact on the game, notably EDJ Sports (link) and their work with comparing the change in game winning percentages based on a coach’s 4th down and timeout usage decisions. Our hope is to expand on this type of work and look into as many areas as we can to measure coach decision making. We will compare such “analytics-community-accepted-value-adds” like pre-snap motion, and play-action rate to provide a more data-rich, robust answer to the evaluation of coach impact.

In this, Part I of III, we wanted to create a baseline level of understanding, only concerning ourselves with the Offensive side of the ball, strictly from 2019  (more data is better than less, recent data) on 5 items:

1. Pre-Snap Motion
2. Play-Action Rate
3. 4th Down Aggressiveness
4. Adaptability (Rates relative to opponent’s strength vs pass/run)
   1. Change in Pass usage
   2. Change in Run usage

To ensure we use the most relevant information, all data collected was restricted to weeks 1-16, excluded 4th downs (mostly), game situations where 16 or less points separated teams and only plays within the middle 80 yards of the field were included. It should also be noted, although we will present the information together, and form an aggregate scorecard, because of the complexity of all the variables, each variable in itself is NOT mutually exclusive to the next. Given the power of big numbers, most of the spillover should be seeped away in volume of data, but we still need to make sure we are clear, there could be some overlap in these variables.

In this study, the first thing we wanted to look into was the impact of the decisions themselves, instead of the coaches making those decisions. We looked on average, how much do certain decisions actually impact outcomes, over the course of a season measured by the change in SIS’ Game Winning Percentage (GW%) (LINK). Below you will see how this shaped up for the 2019 season comparing the average GW% gained when using the variable in question vs. how much is lost from doing the opposite/null variable (i.e. Play Action (PA) Rate compares dropbacks when using PA, vs dropbacks that do not use PA).

Strat.	Win%	Win% null	Net	% Change
Pre Snap Motion	0.23%	0.22%	0.01%	5%
Play Action Rate	0.60%	0.34%	0.26%	76%
4th and 2-0 to go Go for it	1.97%	1.34%	0.63%	46.87%

Although all variables considered are statistically significant, we separated the “standard” decisions from the “reactionary” decisions. Below, you will see how altering your pass/run ratio, relative to the strength of opponent vs pass/run (based on SIS EPA, LINK) correlates to a change in game winning percentage points.

Reactionary Strat.	Correl. To Opponent Strength/Rank
Passing Rate relative to Def. Rk.	32.07%
Rushing Rate relative Def. Rk.	25.75%

All 5 items showed a significant impact on the average outcome of football games from last year, with what we believe to be very interesting findings for the “Reactionary Strategies” that we plan to investigate further in Part II. 

Altogether, in 2019 here is how each team fared with regards to all 5 of these variables (Green indicating a top 8 rank, and red a bottom 8).

Team	Pre Snap Motion	Play Action Rate	4th Down Rate	POA	ROA
ARI	0.49	0.25	0.86	-0.16	0.04
ATL	0.30	0.24	0.93	-0.06	0.47
BAL	0.63	0.35	0.88	0.43	-0.34
BUF	0.69	0.32	0.90	-0.18	0.22
CAR	0.34	0.23	0.78	-0.24	0.07
CHI	0.42	0.27	0.72	-0.12	0.10
CIN	0.36	0.32	0.83	0.25	-0.20
CLE	0.23	0.16	0.88	-0.20	0.18
DAL	0.34	0.24	0.85	-0.02	-0.31
DEN	0.38	0.26	0.69	-0.46	-0.43
DET	0.37	0.27	0.78	-0.11	0.48
GB	0.32	0.32	0.77	0.17	0.35
HOU	0.34	0.24	0.90	-0.19	0.21
IND	0.44	0.36	0.87	-0.05	0.29
JAX	0.36	0.31	0.78	-0.09	0.14
KC	0.63	0.27	0.82	0.23	0.24
LA	0.45	0.15	0.76	0.13	0.26
LAC	0.53	0.24	0.79	-0.26	0.28
MIA	0.37	0.32	0.82	-0.07	-0.15
MIN	0.33	0.28	0.76	-0.69	0.21
NE	0.32	0.33	0.76	-0.01	0.29
NO	0.48	0.24	0.77	0.06	-0.29
NYG	0.25	0.23	0.82	-0.01	0.51
NYJ	0.31	0.28	0.82	0.10	-0.36
OAK	0.38	0.24	0.84	-0.30	0.02
PHI	0.56	0.34	0.91	0.09	0.14
PIT	0.35	0.20	0.76	-0.60	0.07
SEA	0.46	0.27	0.79	-0.14	0.49
SF	0.54	0.36	0.77	0.04	0.05
TB	0.60	0.33	0.80	-0.01	0.09
TEN	0.44	0.18	0.85	-0.09	0.14
WAS	0.38	0.27	0.76	0.25	0.51

*ROA and POA is Run and Pass Opponent Adjustment rating.

If we look at the same dataset, but with regards to the individual team variances due to the specific strategy here is how that shaped up. Put another way, what teams generated the most/least value when deploying the specific stratagem. 

Team	PSM	PA	4th Dn.	Total	Rk.
ARI	0.02%	1.07%	83.90%	28.33%	7
ATL	0.26%	0.28%	91.64%	30.72%	1
BAL	0.15%	0.31%	85.22%	28.56%	6
BUF	-0.43%	0.67%	87.90%	29.38%	3
CAR	0.37%	-0.13%	76.19%	25.48%	21
CHI	-0.44%	0.18%	70.74%	23.50%	31
CIN	-0.43%	-0.02%	81.69%	27.08%	13
CLE	0.13%	0.69%	85.45%	28.75%	5
DAL	-0.26%	-0.13%	83.61%	27.74%	10
DEN	-0.07%	0.37%	67.86%	22.72%	32
DET	0.30%	0.24%	75.78%	25.44%	22
GB	0.07%	-0.04%	75.33%	25.12%	23
HOU	-0.06%	0.03%	86.95%	28.97%	4
IND	0.28%	0.33%	83.60%	28.07%	8
JAX	0.33%	0.44%	76.26%	25.68%	19
KC	0.46%	0.10%	81.15%	27.24%	12
LA	-0.33%	-0.11%	73.32%	24.29%	30
LAC	0.34%	0.47%	78.68%	26.50%	16
MIA	-0.03%	-0.02%	78.51%	26.15%	18
MIN	-0.15%	0.23%	73.36%	24.48%	28
NE	0.27%	-0.06%	74.07%	24.76%	26
NO	0.11%	0.70%	74.08%	24.97%	25
NYG	0.13%	0.47%	81.39%	27.33%	11
NYJ	-0.05%	0.25%	80.45%	26.88%	14
OAK	-0.04%	-0.09%	80.60%	26.82%	15
PHI	-0.32%	-0.57%	89.95%	29.69%	2
PIT	-0.19%	0.47%	75.05%	25.11%	24
SEA	0.69%	-0.50%	76.53%	25.57%	20
SF	0.06%	0.67%	72.60%	24.44%	29
TB	-0.46%	0.81%	78.61%	26.32%	17
TEN	-0.14%	0.66%	83.18%	27.90%	9
WAS	-0.26%	0.54%	73.83%	24.70%	27

There is plenty to glean from all this information, but we are still far from collecting a predictive model, or even set of variables to use as a base adjustment for handicappers/fans alike. Our hope is with including the 2020 data (adjusting for coach changes) and adding more “reactionary variables” (personnel use vs opponent strength, deep vs short passing vs opp. Strength, etc) we will be much closer to providing a valid “coach adjustment” to predictive models. Stay tuned.