Precognition and 5%

Many areas of scientific research such as psychology, medicine, and economics make heavy use of tests of statistical significance. A popular significance level is 5%; this means that the experiment results are less than 5% likely to have occurred by random chance. While these tests are arguably valid and useful, with the wide availability of automated experimentation and computerized data mining they become ever easier to unintentionally and intentionally game.

One well-known mistake is to omit necessary corrections when using the same data set to test multiple hypothesis. If a single hypothesis is 5% likely to be true by random chance, then by testing 14 hypothesis against the same data set it is over 50% likely [1 - (1 - .05)^14 = 0.51] to find at least one false positive result. Besides correction factors, a common solution is to have two separate data sets, where one set is used for data mining for possible hypothesis, and a second set is used to verify the result. On a large scale, however, poor hypothesis can still pass this second filter. This is especially true when the hypothesis themselves are being automatically generated and not based on any previously known plausible physical mechanisms.

A related problem is known as the "file drawer effect": positive results are published, while negative results remain in the "file drawer". This creates a serial version of the problem above, where if the same hypothesis is tested 14 times then by random chance it is over 50% likely to be confirmed at least once. The negative results are never published. There is a movement to publish negative results, but this is also problematic because the negative results may be due to recognized poor experimental procedure. These problems reduce the usefulness of meta-studies, since they are summarizing and aggregating the results of positive studies without having accurate information about how many other negative studies were never published.

These issues are becoming more well known, as problematic results in areas such as clinical drug trials are found. A useful "canary in the coal mine" for statistical techniques is parapsychology. While confirmation of abilities such as precognition (seeing the future) is possible, it is much more likely to indicate that statistical standards for research and publication in a field (in this case psychology) have fallen too low. Recently the paper "Feeling the Future: Experimental Evidence for Anomalous Retroactive Influences on Cognition and Affect" by Daryl J. Bem was accepted for publication in a prominent psychology journal. Critics claim the author has made numerous mistakes, including those discussed above. Bem found subjects could predict a future result correctly 53.1% of the time (where random chance was 50%). But as Wagenmakers, etc. have have pointed out a similar large-scale test has been running for a long time: casino roulette. In European roulette the house edge is 2.7% [36 to 1 odds against winning; 35 to 1 payout], so gamblers with Bem's 3.1% edge would have cleaned out the casinos already. It is suspicious that in most studies which do find psychic abilities it hovers at the edge of the statistical significance value chosen for the study.

Bem's response to critics is amusing. A major part of his defense is appeal to authority: the referees of his paper accepted it for publication in a prominent journal, and his statistical techniques are commonly accepted in psychology. He defers to another author (Radin) for defense of the historical record of parapsychological research. Radin cites meta-studies and makes an even more dubious appeal to authority: the U.S. patent office.

Black Swans in the Market

Modern portfolio theory is based on the assumption that changes in stock prices are normally distributed (follow a bell curve). Benoit Mandelbrot and Nassim Nicholas Taleb disagree with this assumption, and believe market crashes like that of 1987 are much more likely to occur than standard theory predicts.

For my spring 2006 U.C. Berkeley Extension Statistics class I created a presentation Are Daily Changes in Stock Prices Normally Distributed? I analyzed MSFT (used in the standard MBA Finance textbook as an example of normality!), BRKA, and the S&P 500 index over various date ranges. I found none of them satisfied statistical tests for normality.

Black Swans

"Black Swans", as unlikely or unexpected economic events, are currently in the news. This usage was popularized by Nassim Nicholas Taleb. It was based on the old European belief that all swans were white, which was contradicted by the discovery of black swans in Australia. But as I discussed earlier genus and species are human-defined categories. There is no logical contradiction inherent in a black swan, and probabilities related to finding one are really probabilities of finding individuals (animal instances) fitting into ill-defined categories.

Note the first black swan was discovered in 1697, over 150 years before publication of Darwin's On the Origin of Species (1859) and over 250 years before Watson and Crick's A Structure for Deoxyribose Nucleic Acid (DNA) (1953). So seventeenth century notions of species would not include modern concepts of evolutionary descent or DNA similarity. Instead they were based on ideas of fixed categories inherited from Aristotle.

Fooled by Randomness

In Breaking the Spell: Religion as a Natural Phenomenon Daniel Dennett cites Julian Jaynes "brilliant but quirky and unreliable book" The Origins of Consciousness in the Breakdown of the Bicameral Mind as noting "that the very idea of randomness or chance is of quite recent origin" (Dennet, p. 133):

"Sortilege or the casting of lots differs from omens in that it is active and designed to provoke the gods' answers to specific questions in novel situations. It consisted of throwing marked sticks, stones, bones, or beans upon the ground, or picking one out of a group held in a bowl, or tossing such markers in the lap of a tunic until one fell out. Sometimes it was to answer yes or no, at other time to choose one out of a group of men, plots, or alternatives. But this simplicity -- even triviality to us -- should not blind us from seeing the profound psychological problem involved, as well as appreciating its remarkable historical significance. We are so used to the huge variety of games of chance, of throwing dice, roulette wheels, etc., all of the vestiges of the ancient practice of divination by lots, that we find it difficult to really appreciate the significance of this practice historically. It is a help here to realize that there was no concept of chance whatever until very recent times. Therefore, the discovery (how odd to think of it as a discovery!) of deciding an issue by throwing sticks or beans on the ground was an extremely momentous one for the future of mankind. For, because there was no chance, the result had to be caused by the gods whose intentions were being divined." (Jaynes, p. 339-240)