Using the TPI to Set Realistic Expectations at Aston Villa

Plenty has been written about the unreasonable expectations of Aston Villa’s management team and some of their supporters since they began their search for a new manager. Chris Nee wrote a great piece at Two Footed Tackle, detailing Villa’s history in the Premier League. He makes the case that their “expectations may not have been unrealistic fifteen or even ten years ago, and maybe they haven’t changed. But football has.” Simon Clancy wrote in the NY Times Goal blog of the recent management carousel at the club which was kept alive by the appointment of Alex McLeish given that some fans don’t want him. I even spilled some digital ink on the topic, identifying several Aston Villa managers as large over-performers when it comes to the expectations set by the club’s transfer expenditures.

Just how unrealistic are these expectations given how much Villa is willing or able to pay for transfers? The Transfer Price Index (TPI) allows us to quantify just that point.

First, have a look at how the two most recent managers – Martin O’Neill and Gerard Houllier – performed against expectations related to their average starting XI transfer costs over an entire season (termed as £XI, given that the figures take into account TPI inflation). Such performance can be quantified via the regression equations that were created earlier this year and have been updated with the 2011 Transfer Price Index data.

Save for Martin O’Neill’s first season at the club when he was cleaning up the mess from an overpriced side that had finished 16th the year before, Aston Villa has outperformed table position expectations their transfer expenditures should have set. If Villa’s ownership was indeed upset about the return-on-investment that they were getting under O’Neill, they need to take a look at the table above and realize how well he did given modern English football economics. Their squad and starting XI expenditures were in line with 9th to 11th place finishes, and O’Neill’s utilization rate was right at the average for the seasons in which he managed. O’Neill did just about as good as anyone could have asked of him, and the only man to do better with such meagre transfer expenditures is Arsene Wenger at Arsenal. The replacement of O’Neill with Houllier, who saw a lower utilization rate and average £XI, saw Villa regress this season but still outperform transfer expenditure expectations.

Another way to assess O’Neill’s and Houllier’s performance versus financial expectations is via their match day performance. In an earlier post a model was created to estimate match points vs. venue (home/away) and match starting XI cost ratios (denoted by “m£XIR). That model can be used to generate the expected points from a match, with the difference between the actual match points and the expected match points representing over or under performance versus the model for that specific match. If the cumulative difference to the model is observed over the 38 matches in a season one begins to understand where any streaks of over- or underperformance exist within a season. In the case of underperformance, such a streak at the wrong time of a season can seal a manager’s fate just as much as a season long run that results in relegation or near-relegation. The graph below shows the cumulative point differential for O’Neill’s and Houllier’s tenures at Aston Villa.

The graph shows the two most disappointing seasons occur in O’Neill’s first and Houllier’s only seasons with the club. Even O’Neill’s first season showed a good bit of promise over the last eight matches where he improved his cumulative performance versus the m£XIR model by 5 points to nearly break even on the year.  Each of O’Neill’s other three seasons saw him steadily climb through positive territory to be 10+ points to the positive by the end of the season.

The only major disappointment with O’Neill’s performance could have been in 2008/09. On February 7th, 2009 Aston Villa defeated Blackburn and went in to the international break 18 points to the good versus their £XI expectations and were sitting third in the table – a gurranteed Champions League berth. A tough nine matches after the break saw Villa go 0-4-5, drop to a point differential of 10.5, and drop to the fifth table position.  During that nine match run a combination of tolerable losses to clubs like Chelsea, Manchester City, Liverpool, and Manchester United were compounded by unexpected ties or losses to Stoke City, Tottenham Hotspur, Everton, West Ham United, and Bolton. Other than this nine match run, Aston Villa never saw such poor form under O’Neill’s.

Houllier, on the other hand, was a disaster almost from day one. The acquisition of Darren Bent in the January transfer window certainly stemmed the slide, and boosted Villa’s average match £XI by nearly £20M. But nothing could save Houllier at the end of the season. Having invested so much in the club during O’Neill’s tenure, Villa ownership could not stomach another regression the likes of O’Neill’s first season and sacked Houllier at the conclusion of the 2010/11 campaign.

Benefits of the m£XIR model

Astute readers will notice that this result (under-performance versus the m£XIR model) is in contrast to the over-performance versus the average M£XI model.  There are several reasons for this.

First, notice the final table for the 2010/11 season.  Had just one of Villa’s wins ended up in a draw or loss they would have sat at 46 or 45 points, and thus would have finished 14th, a clear underperformance versus the M£XI model.  Thus, the evaluation of performance against the M£XI model can be highly sensitive to small changes in point totals – perhaps it’s not the most robust model for evaluating manager performance in the short term.

Second, the M£XI model uses an “average of averages” in its construction – it looks at the average of a club’s finish and average M£XI and makes a regression relationship between the two.  Such simplifications are acceptable for long-term projections, but presents some challenges in short-term evaluations.  Most importantly, it ignores the reality of shifts in £XI from match-to-match due to injury, transfer, poor form leading to a player’s relegation to a substitute role or the reserve squad, or any other reason the cost of the starting XI may change.  It’s this variability that successful managers manage, and less successful ones do not.  It’s variability in £XI that can wreak havoc on the best laid managerial plans, and looking at individual match data rather than a season or entire Premier League average provides such insight into £XI variability.  The table below summarizes variability via the standard deviation (σ) of the club’s £XI for each season of the managers’ tenures.

The volatility seen under Houllier is certainly exacerbated by the fact that the club shelled out £29.3M for Darren Bent and Jean Makoun in the January 2011 transfer window. However, even taken as two half seasons Houllier’s variation would still rank in the top of the five year period (σ Aug-Dec = £15.9m, σ Jan-May = £13.9m). O’Neill only comes close to this in his first year. By the time O’Neill was in his final year he had achieved his lowest variation in starting XI cost, as well as his most consistent over performance versus the m£XIR model by nearly 10 points.

Clearly, for O’Neill, the key to outperforming £XI expectations was the consistency of the starting XI. [Although it can lead to fatigue later in the campaign – a regular criticism of the club's performance under the Ulsterman.]

Given that Aston Villa couldn’t see eye-to-eye with Martin O’Neill on long term spending and Gerard Houllier underperformed, what’s next for Alex McLeish and Villa? Short of McLeish being the next Arsene Wenger, how much money must Aston Villa spend in the transfer market to make Champions League qualification a realistic outcome each season (i.e. the long-term)? Just such an estimate can be provided using the M£XI and MSq£ regression equations referenced earlier in the post.

The table below summarizes the M£XI’s required for various odds of Champions League qualification using 2011 TPI data. The qualification odds are based upon the 50th and 95th percentile prediction intervals for the regression equation (the 50% odds are from the base regression equation). The “Required 2011/12 £XI (M)” column is a product of the “Required M£XI” column and a projected 2011/12 £XI based upon historical inflation rates within the Premier League.

Let’s give Villa the benefit of the doubt and assume they achieve the 50% utilization seen during O’Neill’s final two years – in other words, on average, half of their transfer outlay makes it into the starting line-up each week – so whatever increase in £XI is required must be doubled when it comes to actual talent purchased* – this total is reflected in the “Required Investment” column. [*Plenty of players by all managers fail to succeed, either because they just don't perform as expected, or the fact that only eleven members of the squad can start any one game.]