After a weekend of listening to football broadcasters, I need to detox. I still feel this zinger from a few weeks ago during the Sunday Night Football broadcast: the commentator noted that Adam Vinatieri, the kicker for the Indianapolis Colts, had a career record of 18 of 36 when attempting field goals over 50 yards, and a season record of 2 of 4 at that distance. No matter how you look at it, he said, it’s a 50-50 proposition. Then Vinatieri kicked the field goal, upping his record on the season to 3 out of 5 from long range. Then the commentator quipped, “So I bet he misses the next one.”
It is ironic that while the sports world is finally embracing the scientific method of numerical analysis, the marketing department exhibits and even celebrates statistical stupidity. The talking heads will tell us Peyton Manning’s Total QBR, Yasiel Puig’s WAR, and other advanced metrics, but then tell us Vinatieri will miss the next one, and then cut to a commercial telling us that “it’s only weird if it doesn’t work,” which translates to: “Superstition is only stupid if you’re not drinking beer.”
If, like me, you cringe when you hear the commentator say that Vinatieri is “due” to miss the next one, then you probably don’t need to read the rest of this blog post, but you should anyway, because I may succeed in finally giving you the explanation for something that you already know to be true, but which no drunk sports fan can ever correctly explain. And you may one day be that drunk sports fan.
First, we’re going to have to switch this discussion to coin flips. (If we go with the statistician’s other favorite—balls in urns—then we would have to convince the commentator why we want to sample with replacement, and we know that NFL broadcasters will resist the use of replacements.) The reason we have to use coin flips in our discussion is that in the case of Vinatieri, the broadcaster is assuming that Vinatieri’s probability of success is 0.5, when in fact Vinatieri’s success probability (denoted p) is an unknown parameter that we need to guess. The broadcaster’s logic is that if we know that the success probability p=0.5, then a missed field goal on the next opportunity would restore the 3 of 5 record to a 3 of 6 record, i.e., a 50% success rate.
Forgetting about the “due theory” for a second, a glaring problem here is that the argument is not logically consistent. The only reason he thinks Vinatieri is a 50% kicker from long range is from Vinatieri’s historical kicking data. But now that we see Vinatieri is 3 of 5 for the season (60%) or 19 of 37 (51.35%) for his career, we should update our estimate of his success probability p. His true probability of success is some unknown number, but we would now estimate it to be something between 51.35% and 60% (depending on whether we think recent results are a better measure of his current ability), in which case the broadcaster should announce that Vinatieri is a favorite to make the next one. Anyway, this avenue leads us to philosophical debates about Bayesian statistics, prior distributions (esp. empirical Bayes priors), and other deep concepts that distract us from our simple mission: to show the simple reason that “due theory” is bogus, and reconcile that bogusness with other seemingly contradictory beliefs.
So let’s keep this simple with a fair coin, an object with a known success probability of p=0.5. Let’s suppose we have flipped the coin 100 times, and it has come up 65 heads and 35 tails. Present this data to a civilian, or even an AP, and let him even inspect the coin. Ask him to describe his beliefs about the current situation. He will state three beliefs, and we will stipulate that these are facts:
- The coin is fair, giving a 0.5 probability of heads on any given flip.
- Each flip of the coin has no effect on any other flip. Because the flips are independent, the prior results do not help us predict the next flip.
- If we flip the coin many times, the fraction of heads observed must converge to 0.5 (the success probability given in Fact #1).
Interestingly, these three commonplace notions form the foundation of much of statistical work, and hence have precise technical terminology and definitions. The first two facts are what a statistician means when he says that the coin flips are i.i.d.—independent and identically distributed. The third fact is what he means when he says that our estimator is consistent. An “estimator” is just a rule or way of coming up with a guess. In this case, if we had to guess Vinatieri’s success rate, or the coin’s heads rate, our guess is just the number of successes as a fraction of the total number of attempts. So, our rule generates the estimate 65/100 = 0.65 for the heads probability of the coin, which we know from inspection has a 0.5 heads probability. To say that our estimator or rule is “consistent” means that if we collect data from a large number of trials, our guess will get closer and closer to the correct answer. So, consistency means that the fraction of heads is going to get closer and closer to 0.5 as we keep flipping that coin.
So now we have these three facts, and we also have a sample of 65 heads out of a 100 flips. What will happen on the next 100 flips? Most people, even APs, are unable to reconcile the meaningless of prior flips (Fact #2) with the necessity of convergence to a 50% fraction (Fact #3). They say they don’t believe in “due,” but they also “know” that the fraction has to gravitate to ½, and if you actually watch their AP activity, you will see that their actions belie their true beliefs. I have seen AP machine players who don’t believe in “due,” but who switch to the machine that has not hit a Royal Flush for a while. I have seen card counters who are on a winning streak at a casino who become afraid to play there, because they feel they are due for a loss, even though they pay lip service to the AP Creed of not believing in “due.” Though we have not eliminated superstition in the AP world, at least embarrassment over superstition is a start.
Though they say they know the coin is not due, in their heart of hearts they expect the coin to restore the 50% fraction in the following way: Over the next 100 flips, if tails is supposedly due, then we fix the 65/100 fraction by updating it to (65+35)/(100+100) = 100/200 = 0.5. Now of course, they realize that it is silly to expect 65 tails in the next 100 flips. After all, that’s 15 extra tails in only 100 flips. They may then reason that heads got that lucky in the first 100, so it’s not inconceivable for tails to run that lucky in the next 100. Still, they usually feel guilty about worshipping the Due idol, and they also know that Fact #3 somehow invokes the “long run,” so then they decide that the better way to restore order to the universe is to make tails due over the next 10000 flips. If we just get 5015 tails, or 15 extra, over the next 10000 flips, then our overall heads fraction is (65+4985)/(100+10000) = 0.5. This way tails only needs to be slightly lucky for a while to restore order.
Either way, what these people are trying to do is to fix the heads fraction in the numerator. They are trying to cancel out the 15 extra heads by inserting 15 extra tails into the numerator, so that the fraction becomes ½. Here is the secret about Due: the heads fraction corrects itself in the denominator, not the numerator. While the coin showed 15 extra heads in the first 100 flips, what happens to our 0.65 fraction if the next 100 flips are even? Then the fraction becomes (65+50)/(100+100) = 0.575. Then it becomes (65+50+50)/(100+100+100) = 0.55 if we stay even in the next 100 flips. After 10000 additional even flips after the initial hundred, we would arrive at (65+5000)/(100+10000) = 0.5015. Look how close we are to 50%!
To rectify the fact that heads ran lucky at the beginning, we do not need tails to get lucky. If in future flips, the coin just stays true to its nature, with tails getting only its fair share, then the original excess of 15 heads gets constantly diluted by the ever-expanding denominator. The 0.65 fraction cannot be maintained; heads cannot keep getting extra lucky given Facts #1 and #2. No fixed excess of heads can survive getting diluted down to the 50% success rate as the denominator grows.
You may not like the fact that as we get closer and closer to 50% in this example, we are always on the high side. I have two answers. First of all, this is just an example. If you collected real data on millions of flips, you would see the fraction bounce around, but the bounces would get smaller and smaller and the growing denominator would still wipe out any excess of heads or tails in the numerator. Facts #1 and #2 guarantee that the future balance of heads and tails will dilute any prior excess of heads or tails in the numerator as the denominator grows. Second, statisticians’ main priority is having a guarantee that as the data sample size grows, our guess will get really close to the truth. Whether the guess is always high or always low along the way is usually not that big a deal. Who cares if we are too high if the overestimate is shrinking and shrinking, and we are getting superclose to the truth as the sample grows. Too high? No problem—just get more data and the overestimate shrinks in magnitude. Anyway, we’ll save a discussion of the relative merits of “unbiasedness” and “consistency” for another day (or not!).
Are you betting football professionally?
Excellent explanation, and I’m saying “Doh, it is the denominator” about now. It is hard to get someone who believes in “due” to accept the concept of independent events (although those same people would likely be perfectly willing to accept two consecutive royal flushes in video poker if they happened to hit them). However about anyone can understand the simple math you use here to explain why it’s so. Good job.
My interest in football (pro/SEC) is more personal than professional. (Not so personal that I would poison trees and then confess by saying “Roll damn Tide” on the radio, but passionate enough that I can enjoy watching a game without having a bet on it!)
You must not be a Colin Cowturd fan 🙂
Thanks for the support! I told Anthony Curtis that I could write a blog if there is interest and if he could make it as easy as possible, so LVA and Orion Network Solutions customized a page for me. So far so good.
I wonder about the efficacy of using a pile of equations to illustrate a trivially obvious concept: that Vinatieri’s past field goal attempts, made and missed, do not reach with some invisible, ghostly hand into the present and swipe at the football, thus affecting the outcome. I mean, if you believe in luck (another nonexistent element) or “due”ness, then your brain is so atrophied that proving your beliefs to be, like your brain, complete mush is a futile exercise. In general, I’ve, through painful attempt after painful attempt, been forced to the conclusion that trying to educate the tiny-brained folk is futile.
I appreciate your illustrating the concept in such a cogent way, but doing so seems to me something like proving to the roulette aficionado that there are no invisible fairies dancing on the roulette wheel and thus, any attempts to propitiate them are hopeless, and the house edge remains 5.26%.
I guess I didn’t make my point clearly. I am not trying to use math to convert those who believe in “due”–I know that some people are beyond salvation; rather, I am actually trying to make a point ABOUT the math. There are people who correctly don’t believe in “due,” but who get stuck on the math: if the numerator has a +15, how can we achieve the 1/2 fraction unless we insert a -15 into the numerator? My point IS a math point–that we leave the imbalance in the numerator alone, and we fix the situation by expanding the denominator to dilute the imbalance until we get arbitrarily close to 1/2.
I like your comments because I feel most sports announcers are idiots. Using the example you gave, if a rookie kicker made his first three kicks this announcer should then announce he will be in the nfl hall of fame because he’s never going to miss.
Thanks for trying to educate us “tiny-brain” folk.
I really enjoy your posts, little did I expect to see Adam Vinatieri’s name in a post. Much less a reference to Harvey Updyke in the comments section.
So you’ve probably heard about “A Mathematician Reads the Newspapers”; how about “A Non-Mathematician Watches the NFL”? Should be great reading fun either with or without Bill Bel’s team in the playoffs/Super Bowl.
Once-a -year LV visitor here. AP or not (perhaps “plays with intelligence” would better describe me), my experience has been like ‘heads’ in the example. I’m ahead in my casino lifetime, and not all of it can be accounted for by “intelligent play”. For example, while I play only positive VP, I’ve hit far more bonus quads than expected.
I’ve never believed anything was “due”. I’ll play a VP machine with the knowledge that a Royal appeared on it five minutes ago – I will not switch machines just because the current one isn’t paying – every hand is independent. But I always thought that my so-far abundance of bonus quads must be balanced out by fewer-than-expected in the future. I never considered that (while short-term results will invariably ‘bounce around’) it’s possible for me to have the same positive $ lifetime result even after a large amount of future play.
Of course, I could get my ass handed to be for the next 10 years and end up deep in the red (hmmm … 2012 … lots of doomsday movies that year, and LV certainly took a big bite out of my ass that year). You never know. But thank you for pointing out that just because I’ve had ‘good luck’ in the past, it doesn’t mean I have to have ‘bad luck’ in the future.
The problem is to cpotume the approximate probability that in a room of n people, at least two have the same birthday. For simplicity, disregard variations in the distribution, such as leap years, twins, seasonal variations, etc., and assume that the 365 possible birthdays are equally likely. Real-life birthday distributions are not uniform since not all dates are equally likely.If P(A) is the probability of at least two people in the room having the same birthday, it may be simpler to calculate P(A’), the probability of there not being any two people having the same birthday. Then, because P(A) and P(A’) are the only two possibilities and are also mutually exclusive, P(A’) = 1 − P(A).In deference to widely published solutions concluding that 23 is the number of people necessary to have a P(A) that is greater than 50%, the following calculation of P(A) will use 23 people as an example.When events are independent of each other, the probability of all of the events occurring is equal to a product of the probabilities of each of the events occurring. Therefore, if P(A’) can be described as 23 independent events, P(A’) could be calculated as P(1) d7 P(2) d7 P(3) d7 d7 P(23).The 23 independent events correspond to the 23 people, and can be defined in order. Each event can be defined as the corresponding person not sharing their birthday with any of the previously analyzed people. For Event 1, there are no previously analyzed people. Therefore, the probability, P(1), that person number 1 does not share his/her birthday with previously analyzed people is 1, or 100%. Ignoring leap years for this analysis, the probability of 1 can also be written as 365/365, for reasons that will become clear below.For Event 2, the only previously analyzed people are Person 1. Assuming that birthdays are equally likely to happen on each of the 365 days of the year, the probability, P(2), that Person 2 has a different birthday than Person 1 is 364/365. This is because, if Person 2 was born on any of the other 364 days of the year, Persons 1 and 2 will not share the same birthday.Similarly, if Person 3 is born on any of the 363 days of the year other than the birthdays of Persons 1 and 2, Person 3 will not share their birthday. This makes the probability P(3) = 363/365.This analysis continues until Person 23 is reached, whose probability of not sharing their birthday with people analyzed before, P(23), is 343/365.P(A’) is equal to the product of these individual probabilities: (1) P(A’) = 365/365 d7 364/365 d7 363/365 d7 362/365 d7 d7 343/365The terms of equation (1) can be collected to arrive at: (2) P(A’) = (1/365)b2b3 d7 (365 d7 364 d7 363 d7 d7 343)Evaluating equation (2) gives P(A’) = 0.492703Therefore, P(A) = 1 − 0.492703 = 0.507297 (50.7297%) or the chances are better than 1 in 2 that two people in a group of 23 will share a birthday.