Managing Replay: The New Market Inefficiency

posted in: Featured | 4

If you click on an umpire’s name in this piece, you can see an hilarious blown call they’ve made (with more in the footnotes)! Enjoy!

When the new instant replay system was announced last week, much of the discussion was focused on what was not in it. Neighbourhood play at 2B? Nope. Trap on the infield? Move along! Fair/foul on the infield? Why are you even asking?! Ball and strikes? You’re still safe, Eric Greggs of the world! Suffice to say, the new plan didn’t exactly cover all the bases[1]. But what interested me the most about the new plan wasn’t the exclusions, it was the replay procedure. In order for a replay to be examined in the first six innings, a manager must throw a challenge flag. Each manager gets only one challenge(unless they’re successful, in which case they get a second). By limiting the number of challenges, baseball is attempting to control the pace of the game. What MLB ended up doing was creating the newest way for smart teams/managers to impact the outcome of the game.

The generally accepted view is that it takes approximately 10 net runs to add one win over the course of season. Unfortunately, that information is completely independent of context. Within any individual game, one run can be the difference between winning and losing. By carefully managing the use of replay challenges, a smart manager can greatly increase the odds of scoring more (or allowing fewer) runs, making a win much more likely.

By my estimation there are three major factors that should determine the usage of a challenge[2]. The first and most important of these is run expectancy[3].

Run expectancy data for the 2013 season

As you can see from the table above, certain game situations just aren’t very likely to lead to runs. If a manager uses a challenge on an unlikely scoring situation (say a 2-out bases empty grounder), he may not have it later for a situation with real impact. A general understanding of which scenarios lead to the most runs should help teams avoid this problem.

The second factor is the inning. The rules state that replays for the first six innings may only be initiated by manager challenges, after which replays may be initiated by the crew chief if a team is out of challenges. Couple this with the relative infrequency of blown calls[4] and the closer you get to the seventh inning, the more necessary it is that you challenge a close play. The odds simply get too small that there will be more than one blown call. In fact, with no punishment for a failed challenge other than losing the right to make another, if umpires begin to show a real willingness to call for their own replays late in a game an argument could be made that teams should ALWAYS challenge close plays at the end of the sixth inning.

Finally, it is an inescapable fact that some umpires are simply better than others. Teams that keep track of which umpires blow the most calls can use this to their advantage. If Jim Joyce or Jeff Nelson, two widely respected, mostly accurate umpires[5] are calling the game, you probably want to challenge any bad call. But if historically bad umpires like Tim Welke, CB Bucknor, Jerry Meals (two videos!) or – as no blue jays fan will forget- Bob Davidson are involved, bad calls could be coming in droves. When those guys are on the field the smart managers will be more careful with their challenges by focusing on the information above.

Using these three factors(run expectancy, inning and umpire) together, teams should be able to get the most out of their challenges[6]. But this is a Blue Jays website, so the question becomes, how likely is John Gibbons to take advantage of all this? Also, how will he compare to the rest of what is supposed to be a pretty smart division? To answer these questions, I decided to examine all five teams using four categories that have quantifiable impact on the run scoring environment of a game: sacrifice bunts, intentional walks, pitchouts and defensive shifts.


As you can see from the table above, a sac bunt lowers the expected numbers of runs in every situation(excluding squeeze bunts). In short, they’re almost always a bad idea[7]. So how did the AL East perform? The Red Sox and Rays tied for the 3rd fewest with 24 sac bunts each while the Orioles (27) and Jays (29)[8] were close behind with the 10th and 8th most, respectively. The Yankees, with 36 sacs, had the most in the AL East and fifth most in the AL (just one behind third place Anaheim and KC). So right off the bat all the AL East teams other than the Yankees are showing an aptitude for understanding run expectancy by (mostly) staying away from sac bunts. With the exception of a low total from John Farrell that could very well be due to lineup strength(over 30 attempts both seasons in Toronto), this played out mostly as expected. Joe Girardi has always liked bunts for some reason.


Like bunts, intentional walks always carry a negative run expectancy. Of course, this is obvious because you’re putting someone on base without exchanging it for an out. But while some people[9] rail against giving out intentional walks, they are not always, a bad idea. In fact, an IBB can be just fine if the key idea is to prevent a single run from scoring as opposed to limiting maximum damage. That’s why these free passes are only excusable in high leverage situations. In any other situation, adding baserunners is a bad idea. With that in mind, I pulled intentional walk data for all situations described as “low leverage” by Fangraphs[10]. Gibby comes out smelling like roses here; in these “low leverage situations” the Blue Jays issued only one intentional walk, the fewest in the AL[11].  Boston also looks like they know what they’re doing with the 2nd fewest in the league with two. Baltimore is in the middle of the pack with six, but Tampa (14) and New York (13) are way up at the top of the list with the 3rd and 4th most, respectively. This looks similarly to the bunt data, as Boston looks to really understand run expectancy, Toronto and Baltimore also appear strong and New York looks a little confused. Tampa’s reliance on the IBB probably surprises some people, but Joe Maddon does have a bit of a reputation for occasionally over-managing. Still, Maddon is universally considered among the smartest managers in the game, if not the smartest, so he gets a pass.

(Information courtesy of Brooks Baseball)

Last year, Sam Miller at Baseball Prospectus wrote a very good piece examining the value of the pitchout. While I absolutely encourage you to read the article, the gist of it is that while pitchouts are generally not advantageous, the impact is likely only 1-2 runs over the course of the entire season. While this would suggest that pitchouts aren’t worth mentioning, I included the information partially just because the research is really interesting, but mostly because it’s something we can quantify. This information doesn’t do much to separate the managers though. The Jays and Yankees had four of them in 2013, while the other three teams in the division had five, and they were all near the bottom of baseball. What it does suggest is that at least some of them are paying attention to modern research. Both Farrell and Girardi showed drastic decreases from their 2012 pitchout totals[12].

(Information courtesy of Baseball Info Solutions)

The value of a shift should be pretty obvious. Using batted ball data, teams can place their fielders where opposing hitters are most likely to hit the ball, greatly increasing the odds of getting an out. It’s such an obvious advantage that use of the shift was up approximately 70% over 2012 and up 230% (!!) over 2011. So let’s see how the big boys of AL East performed. Note: If you’re a Jays fan who doesn’t like bad news, this would be the time where you might want to click away to another article[13]. This year the Orioles shifted 595 times on balls in play, ranking first in the league. The Rays were right behind them (2nd) with 555 shifts. The Red Sox and Yankees shifted 477 and 475 times, respectively (7th and 8th) and the Toronto Blue Jays used this obvious defensive tool…249 times. Those 249 times were good for just 15th in the league and represented a drop of 187 shifts from 2012. By all accounts the 2012 Jays were very successful with their shifting. In fact, the 3B-to-shallow RF shifts were SO successful that they broke DRS (Defensive Runs Saved, an advance metric used to measure defense). Baseball Info Solutions had to completely alter how DRS accounts for shifts because Brett Lawrie was showing up as the best 3B in the history of baseball. So why would the Jays alter their approach? I wish I could say there is a good justification for the philosophy change, but I just can’t see it. The 2013 team was really bad defensively and even worse on the mound. Why on earth would they not do everything possible to increase their odds of turning batted balls into outs? Honestly, this is a question that needs to be asked to the administration. The Rays and Orioles get A+ for their shifting work, Boston and New York get high passing grades and the Jays receive a grade of “fails to complete assignment.” Just awful.

Everything was looking good for Toronto and its potential to exploit the new replay system[14]. John Gibbons doesn’t bunt too much and he almost never pitches out or intentionally walks people in low leverage spots. All of those things show an inherent understanding of run expectancy. Gibbons also exploits platoon advantages and knows how to run a bullpen, which show the ability to adjust as the game moves into the later innings. But hoo boy that shift information…It’s almost a perfect parallel for the umpire information. We’re talking about a whole bunch of raw data with very clear applications, that is mostly being ignored. Given Gibbons’ strength in other areas I’m still confident that the Blue Jays can take advantage of the new replay system, but it’s far from a sure thing.

An enormous thank you is owed to the great people at Baseball Info Solutions and at Brooks Baseball for their assistance in gathering information and for their expertise.  This post doesn’t happen without them. Everybody should check out the amazing work they do.


[1]Hiyo!! Sorry…I’ll show myself out.
[2] Aside from the obvious: making sure the call will actually be overturned.
[3] For those who are new to RE, it just shows how many runs on average were scored in each baserunner/out scenario.
[4]Despite what sports news shows would have you believe.
[5]These plays notwithstanding.
[6]Of course, if Angel Hernandez is manning the replay booth, all bets are off.
[7] For an American League team. The presence of the pitcher in NL lineups makes bunts far more excusable.
[8] A whopping TEN of these were by Munenori Kawasaki, so there is a chance that some were bunt hit attempts that turned into sacs.
[9]*Cough* Keith Law *Cough*
[10]I left out the mid-leverage data because it’s too ambiguous.
[11]Once again, NL data is useless for comparison because of the pitcher spot.
[12]Obviously there are no 2012 totals for Gibbons, as he wasn’t managing in the big leagues.
[13]Another article on BJP, of course!
[14]Unfortunately, this was also true of every other AL East team except the Yankees.

  • JefQ27BlueJayz

    Holy CRAP! You guys just don’t stop do ya?? Anyway, what about the former Gibby player interviews? Another great article fellas! Wow!.

    • Ewan Ross

      That’s the beauty of having a sizeable staff, and the reason why we brought all our blogs together. This way we have constant content for our super fans like you.

      As to the Missions on Gibbons series, Isaac tells me we have one more coming, not sure when, he just started correspondence with him recently.

  • Justin Jay

    I wonder what the average is for number of bad calls made during a game, regardless of crew. Just an overall standard.

    I feel that replays in the 7th and 8th will be met with reluctance by umpires, but in the 9th, nobody wants to blow a game in the 9th, so maybe they’ll be more open to using it. I’m sure the umpires, like a Jim Joyce, that have made crucial mistakes in the past will be more likely to use it. I’m hoping anyway.

    Great article Josh. I wondered why Gibby didn’t use the shift more as well. To prevent the bunt down the 3rd base line? Avoid getting burned by that unlikely occasion? Or was the middle infield just so bad at one point, he just felt it didn’t matter? haha

  • Pingback: Baseball Blogs Weigh In: Norris, Bucs, D'Backs()