I just found this really cool paper on the phenomenon of the “hot hand” in sports. The idea behind the “hot hand” (also called the “clustering illusion”) is that success breeds success. In other words, when you are successful (you win games, you make free throws, you get hits) you will continue to be successful. In sports, it has frequently been observed that events are close to independent, meaning that the “hot hand” is just an illusion.
In the paper, the authors downloaded all the data on NBA free throws for the 2005/2006 through the 2009/2010 seasons. They cleaned up the data, then analyzed changes in conditional probability. Their analysis suggested that free throw success was not an independent event. They go on to explain:
However, while statistical traces of this phenomenon are observed in the data, an open question still remains: are these non random patterns a result of “success breeds success” and “failure breeds failure” mechanisms or simply “better” and “worse” periods? Although free throws data is not adequate to answer this question in a definite way, we speculate based on it, that the latter is the dominant cause behind the appearance of the “hot hand” phenomenon in the data.
The things I like about the paper are that they explain things very simply, use a lot of real data they obtained themselves, and are very careful in their conclusions.