Several weeks ago Simon Kuper used his Financial Times column to publish the results of the latest study by Stefan Szymanski on the correlation between wages and club performance in English soccer over the last forty years. The study is an extension of Kuper’s and Szymanski’s wages versus table position study in Soccernomics, and one should not be surprised to see this latest study included in the second edition of the book due this spring. Chapter 3 of the first edition of Soccernomics laid out a telling story of how 89% of league finish position variation can be measured by the relative wages of a club over the long term. Some may dislike the model as such an argument may make the game a bit too deterministic for many to accept, but it does make sense given the brutally efficient and relatively free market for soccer players.
In almost a nod to such concern over pre-determination, Szymanski’s latest study attempts to identify the mangers who most over and under performed versus wage expectations set by his model. The overall rankings included 251 managers who had managed five seasons or more. There are a few expected results as to who shows up in the ranks of the over performers – Paisley, Ferguson, Robson, Wenger, and Dalglish round out the top five. There are also some relatively unknown managers in the top 20, including Paul Sturrock who is profiled in Kuper’s article. Overall it is a very compelling article as to which few managers add value to their clubs beyond what wages already provide.
A similar approach can be taken with the Transfer Price Index (TPI). Nearly a year and a half of work in the index has yielded a bevy of new metrics, the latest of which is the m₤XIR metric. For the last eight months that metric has been used to look at individual club and manager performance versus transfer expenditure expectations as well as a wholesale re-ordering of the Premier League’s nineteen tables. As was hinted at in the alternative tables post, it is now time to turn the attention on the careers of the individual managers.
Differences and Similarities in Approach and Outcome
Similar to Szymanski’s wage-based model, a compilation of each manager’s performance against the m₤XIR metric has been made. There are a few key differences between the models beyond the simple fact that Szymanski’s studies focus on wages and the entire Football League while the TPI focuses on transfer expenditures and only the Premier League.
Perhaps the biggest difference between the two models is how their inputs are calculated. Szymanski’s study looks at total team payroll and uses it as an input to his model. This approach looks at the cost of all the talent assembled on the team, regardless of how much time each player spends on the pitch. The approach Szymanski’s study takes is very similar to the Sq£ approach taken in Pay As You Play, and the subsequent MSq£ model that has been developed at this blog. While such approaches help one understand how well a manager performs given the total theoretical talent at his disposal, they don’t take into account the talent the manager actually puts on the pitch from match to match. Injuries and the realities of only having eleven spots on the pitch mean the manager must juggle a lot of expensive talent, especially given the reality of extra-league competitions each club faces. This blog’s own study of the TPI data indicates that utilizing a match-based, rather than squad-based, approach yields better correlation to actual competitive results. The foundational data of the m£XIR model is built upon a match-by-match analysis using data from each club’s starting lineups.
From reading Kuper’s article, it also appears that Szymanski’s approach is to take a longer-term view of a manager’s performance than the m₤XIR metric. Szymanski’s model looks to measure manager performance in the table versus a season’s wage expenditures, which means there is a single data point for each manager each season they were at the helm of a club. The m£XIR metric looks at the current transfer purchase price (CTPP) of the starting XI for a club (£XI) in an individual match, does the same for the club’s opponent, generates a ratio of the starting XI CTTP’s (m£XIR), factors in which club is playing at home, and provides the expected points for each of the clubs in the match. This provides 38 data points per manager per season back to the 1995-96 season, and 42 for the first three years of the Premier League. This increase in sample size against a metric that is well understood (match points) allows for a very intuitive tallying of manager performance versus transfer expenditures. The result is the m£XIR residual (actual points – expected points), which is often normalized to a points per match (PPM) basis to make comparisons much easier. This provides more granularity as to manager performance, and conforms with statistical principals that an increased sample size leads to a closer sampling of the true population distribution.
Another difference between Szymanski’s model and the TPI approach is in the number of samples required for inclusion in each study. Szymanski required a manager to complete at least five full seasons to be included in his study. The rationale is that one needs a relatively decent sample size to draw from to make good statistical conclusions, and managers who don’t last as long in the Football League likely did not meet the experience prerequisite due to performance related issues. The result is that Szymanski’s model contains 251 of the 699 managers for which club payroll data was available. Given the m£XIR’s approach of sampling each individual match and the volatility associated with match-to-match performance, the threshold for sample size for inclusion in this manager study is much higher. Performance versus the m₤XIR metric tends to settle to within 10% of their long term average by their 15th match. A manager must have a career with 38 or more matches under his belt to be included in this study. The benefit is that a 38 match sample size not only guarantees most managers get a least a full season worth of data before they’re judged, but any managers with a large number of short stints are also included in the study.
Where there are similarities between the models is in the outcome. Managers at clubs who consistently finish at the top of the table tend to dominate the upper rungs of both Szymanki’s and the m£XIR rankings. Kuper’s observation of the reason for such a phenomenon in Szymanki’s Football League-wide study rings true even when comparing the lower and upper echelons of the Premier League:
A manager in League Two who has the 90th smallest budget in England but manages to finish 80th nationwide is overachieving. However, a manager with the third-highest budget in England who wins the Premier League is probably overachieving even more.
At football’s summit, competition is fiercer, the amount of money typically required to jump places is higher, and so managers of giant clubs predominate at the top of Szymanski’s rankings.
Recall from the post analyzing the effects of transfer spending on long-term performance in the EPL table that only 71% of the variation in table position can be explained by transfer spending. The other 30% comes from a variety of sources. As Kuper and Szymanski have pointed out, perhaps a large bit of that variation is due to player wages. Another may be due to club aura and the relative ease a Manchester United or an Arsenal have in attracting talent due to their reputations as perennial Champions League participants and Premier League championship contenders. Some of it, however, does come down to club management. The Premier League is too cutthroat and its expenditures on players too high to discount the managerial talent it takes to meld such high priced and self-confident players into a championship contending squad year after year.
With all of those caveats in place, it is time to review the all-time top managers against the m₤XIR metric. What follows is a discussion of a subset of the top performers.
The TPI database contains 162 unique manager combinations over the 19 seasons analyzed. There are 14,932 samples within the 19 seasons, two per match given each club that is participating. Each sample has an expected point value assigned to it based upon the m£XIR model, the actual points earned in the match, and the residual points for the match (actual – expected). Of the 162 manager combinations, 96 meet the 38 or greater match threshold. It is these 96 managers upon which this study is conducted.
The average residual points per match are calculated for each the 96 managers included in the study. That average value is then also converted into a projected residual over a 38 match season to give some sense of how many points per season the manager may be adding or subtracting from the expectations set by club transfer spending. Finally, the manager’s percentile relative to the expected population variation is provided. This percentile is generated from their average residual points per match, and provides a method for comparing the non-linear nature of points contribution when the outer edges of the performance distribution are reached.
A summary of the top 30% of managers is provided below, and will be discussed in greater detail later in this post. The full list of 96 managers’ performances can be found here. The color coding of managers is based upon a 10%/80%/10% categorization. The top 10% have been colored green, the middle 80% yellow, and the bottom 10% red. This is not to say that any percentage should be used as an arbitrary cutoff of over/under performance, but rather the use of established methods in the corporate world for identifying employee performance in a regimented ranking system. Detailed discussions of key mangers follow.
The top 10% of managers include a number of expected names. Alex Ferguson tops the list with Arsene Wenger, with each adding more than 16 points per season each beyond what their £XI’s would deliver. While this is impressive on its own, it’s even more impressive when compared to other managers. Ferguson’s tenure at Manchester United prevents any comparisons within that club, but Arsenal provides two managers for comparison to Wenger. Bruce Rioch comes in 27th overall, adding only 1.9 points (72nd percentile) per season beyond what transfer funds suggest the team should have earned. George Graham finished 43rd on the list (55th percentile), earning 1.4 points less than he should have. Wenger not only delivers above average value compared to his contemporaries, but also his competition at Arsenal during the Premier League Era. While he earned trophies during his first half of his career at Arsenal, he’s especially outperformed the model since 2004-2005.
Three out of the top ten slots are occupied by Chelsea managers. While Jose Mourinho’s overperformance is expected in spite of his expenditures on transfers, Claudio Ranieri may represent the best example of the fickleness of the Abramovich era at the club. While Ranieri’s tenure ranks him sixth best, his overall average is definitely helped by his outstanding 2003-2004 season average of 0.567 points per match over performance. He was only bested by Arsene Wenger and Arsenal’s perfect season that produced a 0.96 per match over performance (best overall single season performance). Nevertheless, expectations at Chelsea were such that Ranieri was sacked and Mourinho replaced him. Mourinho’s average ends up being lower than Ranieri’s single season in the post-Abramovich Era, with Mourinho only outperforming Ranieri on a seasonal basis in 2004-2005. One could argue that such performance during that first season under Mourinho may have been expected of the club even if Ranieri had remained at the helm.
Preceding Ranieri’s tenure was Gianluca Vialli. Initially serving as a player-manager, Vialli would go on to to manage 94 Premier League matches between February 1998 and September 2000. During his two seasons Vialli managed a club with an average £XI of £100M – only slightly more than half of the value of the championship winning clubs of Mourinho. Vialli’s performance of earning more than 11 points more per season than transfer expenditures would predict ranks him better than all but 3% of the managers in the Premier League Era.
Several Liverpool managers also pop up on the list. The number four position on the list will continue to stoke the ever present Rafael Benitez debate. There is no doubt that Benitez’s final season was his worst at Liverpool when compared to expectations, yet even then he was able to earn more than 7 points greater than his transfer resources would suggest. In every other year but his first at the club he was well north of 10 points to the good versus the m£XIR model. Benitez was forced to rebuild the team, both its youth system that suffered under the later years of Houllier and via transfers to make up for the lack of youth. In spite of this, he is fourth best overall in the Premier League era in getting more from his money. Houllier over performed greatly from 1999/2000 until 2001/2002, averaging nearly 15 points better than the m£XIR model would have predicted. He average barely better than 6 points higher than predicted in his final two seasons, which would have been good enough to place him 14th overall. By the time he went to Aston Villa he underperformed by more than 6 points.
While currently at Liverpool and certainly overachieving versus the record of his immediate predecessor, Kenny Dalglish’s record in the Premier League era is mostly filled with managerial stints at Blackburn and Newcastle. Taking a side from the Championship to Premier League champions in three years, Dalglish averaged nearly 15 points more per game than should have expected due to transfers. Dalglish’s tenure at Newcastle dragged down his record that would have seen him finish third in the rankings without it. He finished 8 points to the good with Newcastle in 1996/97, but the disastrous 1997/1998 campaign saw him finish nearly 15 points lower than his side should have based upon £XI’s. Dalglish’s first half year at Liverpool saw him reclaim the old magic, outperforming his predecessor by nearly 22 points if judged over a full 38 match season.
Finally, we come to Frank Clark and Roberto Mancini. Frank Clark’s inclusion in this list is due solely to his tenure at Nottingham Forrest during the early-90’s. While Dalglish was getting Blackburn into the top flight and winning championships through costly transfers, Clark was getting Forrest re-promoted and outpacing his meager transfer outlays. Clark did so well that his Forrest side displaced Blackburn’s single championship in 1994/95 if transfer expenditures are taken into account. Alas, Forrest would be relegated again in 1997, with Clark getting sacked nearly halfway through the season.
Mancini is primarily the beneficiary of form matching investments. There is no doubt that he provided an instant improvement in Manchester City’s performance when he arrived during the 2009/10 season. The season prior Mark Hughes had underperformed by nearly 5 points, while he was on track for another negative performance when sacked in December 2009. Since then Mancini has had the club invest heavily and has the results to show for it. Over the 2010/11 season he earned 11 points more than expected of the transfer fees paid for the players on the pitch, securing a spot in the 2011/12 Champions League for City.
With the expected over performers now understood, it is time to move on to the lesser known managers who should get better recognition for what they’ve done with their club’s player investments.
The Next 20% of Managers
Before diving into the particulars of the managers who finished between the 10th and 30th percentiles it is worth noting just how hard it is to succeed long-term in the Premier League. Readers should note the very steep fall off in points per season earned by the managers as a transition from 1st to 11th place is made. Given that the data is normally distributed, the fall off in points will be much more rapid than the fall of in percentile. To move from 11th to 1st position, an increase of only 10% of the overall managers’ performances, requires a more than 2.5 fold increase in over performance. There’s a reason men like Ferguson, Wenger, and Benitez spend far more time at a single club than the average Premier League manager. Nonetheless, there are managers who have made their marks on multiple clubs, and thus deserve some recognition for their over performance.
Leaving aside two managers who over peformed at successful clubs (Ancelotti at Chelsea and Eriksson at Manchester City), the first manager of interest is David O’Leary. His ranking of 13th on the list is easily the tale of two diametrically opposed management tenures. At Leeds United, prior to his sacking in the Summer of 2002, O’Leary led the club to Top 6 finishes each year for regular qualification for European competitions. While the spending on players was clearly unsustainable in retrospect, the assumption of continual European competition that they rested upon was best served by O’Leary’s leadership. As Leeds soon found out, such consistent performance is very rare. O’Leary’s average over performance of 0.317 points per match (+12.05 points per 38 match season) at Leeds would have placed him 5th on the list. O’Leary would pop up again at Aston Villa but find much less success with only one of three seasons being to the good of the m£XIR model and an average underperformance of 0.123 points per match (-4.67 points per season). With his Aston Villa performance dragging down his overall average, O’Leary finishes 13th in the overall rankings.
Dropping down the list just a few notches further brings us to the names thrown out in casual conversation when the topic is “best bang for the manager buck/pound.” David Moyes’ tenure at Everton was covered in a previous blog post, as was Martin O’Neill’s at Aston Villa. Of note is O’Neill’s recent form since taking over at Sunderland, which will surely raise him in the rankings if he’s able to maintain anything close to the club’s current form. Perhaps the most interesting story of this lot of three managers is that of Sam Allardyce.
Allardyce has a well-earned reputation of managing well above financial expectations. His final four years of his six year tenure at Bolton saw him improve team performance versus the m£XIR model each year, with his final year generating a 0.522 points per match overperformance (+19.84 points over a 38 match season). Allardyce’s half season at Newcastle United was tumultuous, with a poor run of form around Christmas sealing his fate. Even with that poor run of form he was still on track to earn 0.226 points per match more than his transfer expenditures suggested (+8.59 points per season). Perhaps the greatest misalignment between expectations and performance came at Blackburn. Allardyce took a club that was sitting 19th in the table and led them to a 15th place finish. He followed up with a 10th place finish the next season. With the purchase of the club by the Venky group in November 2011, Allardyce was sacked on December 13th with the club sitting in the 13th position and 5 points clear of the relegation zone. The outrage amongst the media and other managers was loud and well placed. All Allardyce had done was steadily improve the club’s fortunes against the expectations set by transfer expenditures: -0.185 in 2008/09, -.089 in 2009/10, and -0.04 in 2010/11. Blackburn would end up finishing 15th in 2010/11 without Allardyce, while Allardyce would move on to relegated West Ham in the summer. As of this writing, Blackburn sits 17th in the Premier League table and tied on points for relegation while Allardyce has West Ham comfortably at the top of the Championship.
There are many more managers in the 10th to 30th percentile with fewer than 200 matches under their belts, and covering all of them would produce a very lengthy article. The remainder of this article will focus on those managers with more than 200 matches to their record.
Kevin Keegan’s 21st position in the rankings proves a cautionary tale of going out on top and then staying retired. When he nominally retired from managerial duties when leaving Newcastle United on January 7, 1997, Keegan’s m£XIR average match residual was +0.310 (+11.78 points over a season). His return to management via Manchester City was far less successful. While he did earn the club promotion after a single season in the Championship, Keegan never was to the positive during his nearly three years leading City in the Premier League. He ended up leaving City with an average residual of -0.071 (-2.70 points per season). This dragged him down from what would have been 4th overall position to 21st.
Mark Hughes’ over performance early in his career earned him steady promotions in the Premier League. Having started out at Blackburn in 2004, Hughes exhibited steady improvement just like Allardyce did at Bolton. Hughes first season saw the team finished with an average residual of -0.394 (-17.67 points on the season), but the club avoided the drop. By the time Hughes’ left in 2008 he had the club over performing by +0.523 points per match (+19.87 points per season), a 37.5 point turnaround versus the model and good enough for 7th position in that season’s table. Moving to Manchester City in the summer of 2008, Hughes would have a rough start in the 2008/2009 season by finishing 10th with a residual of -0.123 (-4.67 per season). He would start the 2009/10 season much better, but would still get sacked in December 2009 even though he was earning 0.512 points more than expected (+19.46 points per season). Outside of his first season at Blackburn, Hughes’ least successful campaign versus the m£XIR model came with Fulham in 2010/11 (-0.183 per match, -6.95 per season), lowering his average further to the current ranking of 23rd. He left after one season to manage QPR this season.
The most experienced manager in terms of Premier League matches, outside of Alex Ferguson and Arsene Wenger, is Harry Redknapp. Save for a single season at Southampton, Redknapp has spent at least three seasons at each of the other three clubs he’s managed. Spurs fans must hope that the pattern Redknapp set at West Ham and Portsmouth is not repeated at their club. In both instances Redknapp built up the clubs to overachieve by more than a third of a point per match (+12.54 points per season) in the middle of his tenure. Such overachievement was followed by rapid fall offs in performance to either break even (Portsmouth) or negative territories (West Ham) versus the m£XIR model. In both cases, the clubs were relegated from the Premier League within several seasons of Redknapp’s departure. Redknapp’s performance at Spurs so far has been similar – a huge deficit in 2008/09 (-0.553 per match/-21.01 per season), a nearly equal over performance in 2009/10 that saw Spurs qualify for Champions League (+0.417 per match/+15.83 per season), and a backwards slide in 2010/11 (+0.209 per match/+7.94 per season). This season’s solid third position as of this writing would help solidify Redknapp’s record of over performance at Spurs. It’s these performances the last two seasons at Spurs that have helped raised Redknapp into 24th overall.
The rankings in this article are based solely upon performance relative to transfer expectations. The analysis rests upon the theory that the vast majority of players at an English Premier League club are transfers – either of the free or non-free variety. Wages are certainly important in garnering player interest in a club, and in keeping a player at a club once they approach the final year of their current contract. But as has been seen time and again, big spenders in the transfer market also happen to often be big spenders on wages. Thus, one must be willing to spend in both markets.
There is a difference in the results between Szymanski’s wage-based model and the TPI’s transfer based model when it comes to ranking managers. A quick analysis comparing rank order between Szymanski’s results and the TPI’s m£XIR results, where they overlap, shows less than a 31% correlation between them, making the correlation not statistically significant. The m£XIR model only takes into account the transfer cost in current pounds of the talent that makes it onto the pitch, while Szymanski’s analysis looks at the total payroll of the clubs. The lists can be viewed as complimentary to each other, with the TPI list being a bit more focused on the Premier League Era.
The manager rankings based upon the m£XIR provide some insight into who has consistently succeeded at big clubs, and which of the mid-level club’s managers may be ready for a shot at a big club. With speculation indicating Harry Redknapp being the next manager of England, men like Martin O’Neill or Sam Allardyce might be good candidates for the Tottenham Hotspur job. They’ve proven their worth at extracting maximum value from mid-level clubs, and should be considered for the job if it were available. Certainly their tactical outlook must match the style of play the club’s senior management desires, but but realizing better value from a limited budget may be what such a club like Spurs are looking for given they don’t have a benefactor like Chelsea or Manchester City. Or perhaps they look to someone like Rafael Benitez. He had a very good record at Liverpool when it came to extracting value from transfers, and Spurs total value of transfers is very similar to Liverpool’s when Benitez was still running the show. This may be all idle speculation if Redknapp doesn’t move on, but it does give an example of the options available to clubs looking to screen potential managers when considering the constraints a Fair Play era will place on them.
While the study inherently excludes managers who have no Premier League experience, it does provide an interesting contrast. The top 10% of managers is light on management with prior English top flight experience. In most cases, the clubs who hired those managers took chances on non-UK management talent in an increasingly global game. Contrast this with the majority of the managers in the 11% to 30% window who come from a substantial UK background. Perhaps there is something to be said for worldly experience providing better insight into player purchases, or maybe there is some sort of anti-English bias at the top clubs of the Premier League when it comes to managers.
Beyond this separation between the first and second tiers, there’s also a distinct change in performance before and after the arrival of Roman Abramovich at Chelsea. The introduction of the non-English benefactor model over the last decade of the league has led to both concentration of championships within the big transfer spenders and increased opportunity to beat the market. Further discussion on this difference will be taken up in a later article, as will a discussion of the worst management performances versus transfer expectations.